How to execute thousands of promises in series with batching systems

Sometimes we need to process a thousands of data. It will be very exhausted if we process one by one. Especially if we need to hit API for each data. When we hit the API with thousands of request, the server will be unavailable and our script failed too.

So, How to hit thousands of API request with Javascript Promise?

Let say we have 10,000 data and to process all of it we need to hit 10,000 API requests. It is a huge amount if we do it at the same time. It will leads to server down and unaccessible service which cause worse effect.

We can use simple approach, we send the API request 50 by 50. So we will need to process 10,000 / 50 = 200 times. But this amount still exhausted if we doit manually. So, is there any better approach?

Yes it is. We can make a Promise in javascript for 50 data. Then make next 50 data if and only if the current 50 data are successfully executed.

Our program will wait current 50 data executed, then it will executed the next 50 data. You can see more detail through the following code.

const suspendedUsers = fs
  .readFileSync("file.csv", "utf8")
  .split("\n");

let promises = [];
let count = 0;

function exec(users) {
    for (const user of users) {
        promises.push(suspendUser(user));
    }
    
    Promise.allSettled(promises)
    .then((response) => {
        count = count + users.length;
        console.log(`Processed ${count} users`)
        if(suspendedUsers.length > 0) {
            exec(suspendedUsers.splice(0, 50));
        }
    });
}

exec(suspendedUsers.splice(0, 50));

Once Promise.allSettled is done go to the next batch

Leave a Comment