Efficiently making thousands of http requests (api calls) in JavaScript.

Often we come across scenarios to retrieve a large number of data from a remote data source.

In Javascript, operations like I/O, R/W happen in an asynchronous manner. Likewise our api requests which are http requests will also do the same. Now with RxJS using observables & streams we can easily handle such operations by a well defined asynchronous program.

Eg. Let’s consider that we want to fetch ’n’ records through api requests, and here each record is associated with a unique Id.

Now let’s assume there are 2000 such “Id’s” in incremental order (eg. 1…2000), so in that case the value of ’n’ = 2000 (i.e. 2000 api calls for 2000 records).

Prerequisites:

• Basic understanding of api requests and javascript.
• Basic understanding of observables and streams.
• Node.Js installed on your system. https://nodejs.org/en/download/

The examples are written in node.js. But nowadays modern frameworks like angular have built-in capability of observables and streams.

Install RxJS module through following command

npm i rxjs

Method 1: Invoking all the 2000 requests at once. (Error Prone).

In this scenario, javascript will keep making one request after another without waiting for the response of the previous request. So here all 2000 requests are invoked asynchronously without waiting for their response.

In the above code we used:

• range(): It creates an observable that emits a sequence of numbers from 1 to 2000.
• flatMap(): It is an operator that projects a source value, to which mapping and manipulation can be done, and returns an observable. In this case returns an observable stream for the http request’s promise object.
• from(): creates an observable from the promise of the http request.
• Here npm’s “request()” module is used to make an http request.

In the above code we append an id which is emitted by the range to the given uri to get the expected record from the remote data source.

Now the above code, after a certain number of requests will throw exceptions like “ECONNRESET” (connection reset with remote source) , “ETIMEDOUT” (connection timeout with remote source). This happens because, javascript can handle only a certain number of requests on a limited number of sockets, which is exhausted when it is jammed with a large number of requests.

Method 2: Making one request at one time. (Precise, but slow).

In this scenario, javascript will make a request only when response for the previous request is received. This kind of appears to be synchronous, but it is not. Yet the request<->response part is triggered one after another for each api call.

Here we just slightly tweaked the code from “Method 1”. We replaced the “flatMap()” with “concatMap()”. The most important difference between both is that:

flatMap() creates an observable for projected source value without waiting for the previous observable to complete.

And,

concatMap() waits for the previous observable to complete before creating a new one.

In the above example, each api call will be made only when the response of the previous api call is handled. Thus the approach is less error prone but too slow.

Method 3: Divide and Conquer (Efficient)

So to improve performance and maintain precision, we equally divide all the requests in sets. For eg. we equally divide 2000 requests into 100 sets, where each set will contain 20 requests. Based on this logic, the program will invoke a set only when requests and responses of the previous set are completed.

Here we have combined the best attributes i.e “Performance” from Method 1 and “Precision” from Method 2. This is achieved through the coexistence of the RxJS operators like “flatMap()” and “concatMap()”.

To create the sets of 20 requests each we used the RxJS operator “bufferCount()”.

• bufferCount(): Buffers the values from source observables until the size hits the given “bufferSize”. In our case bufferSize = 20, hence it buffers 20 id’s each time from the source observable and then emits it as an array. This is how we divide the 2000 requests into 100 sets.
• concatMap(): takes this emitted array as projected source value and returns an observable. Thus here concatMap() achieves precision by creating observable only when the previous observable i.e the previous set is completed
• flatMap(): takes every id from the set as projected source value, creates its http request promise and returns it as an observable. Thus here flatMap() achieves performance by invoking an http request without waiting for the completion of the previous request.

Finally, with method 3 we have conquered our shortcomings, and in contrast to method 1 & method 2 we have better results and improved performance.

Note: Remember to fine tune the value in bufferCount for the most optimal result.