NodeJs Axios: Make Concurrent Requests

In this guide for The NodeJs Web Scraping Playbook, we will look at how to configure the NodeJS Axios to make concurrent requests so that you can increase the speed of your scrapers.

The more concurrent threads you have, the more requests you can have active in parallel, and the faster you can scrape.

So in this guide we will walk you through the best way to send concurrent requests with NodeJS Axios:

Make Concurrent Requests Using Promise.All() & Bottleneck
Adding Concurrency To ScrapeOps Scrapers

Let's begin...

If you prefer to follow along with a video then check out the video tutorial version here:

Make Concurrent Requests Using Promise.all() & Bottleneck

The first approach to making concurrent requests with NodeJS Axios is to use Javascript's Promise.all() functionality and the bottleneck package to control the concurrency.

Here is an example:

const axios = require('axios');
const cheerio = require('cheerio');
const Bottleneck = require('bottleneck');

const NUM_THREADS = 5;

// Example list of URLs to scrape
const list_of_urls = [
  'http://quotes.toscrape.com/page/1/',
  'http://quotes.toscrape.com/page/2/',
  'http://quotes.toscrape.com/page/3/',
  'http://quotes.toscrape.com/page/4/',
  'http://quotes.toscrape.com/page/5/'
];

const output_data_list = [];
const limiter = new Bottleneck({ maxConcurrent: NUM_THREADS });

async function scrape_page(url) {
  try {
    const response = await axios.get(url);
    if (response.status === 200) {
      const $ = cheerio.load(response.data);

      const title = $('h1').text();

      // Add scraped data to "output_data_list" array
      output_data_list.push({
        'title': title
      });
    }
  } catch (error) {
    console.log('Error', error);
  }
}

(async () => {
  await Promise.all(
    list_of_urls.map(url =>
      limiter.schedule(() => scrape_page(url))
    )
  );

  console.log(output_data_list);
})();

Here we:

We import the necessary libraries: axios, cheerio, and Bottleneck. axios is used for making HTTP requests, cheerio is used for parsing the HTML response, and Bottleneck is used for limiting the number of concurrent threads.
We define the NUM_THREADS constant, which represents the maximum number of concurrent threads we want to allow for scraping.
We create an array list_of_urls containing the URLs we want to scrape.
We define an empty array output_data_list to store the scraped data. In your code, you could have this data being saved to a queue.
We create an instance of Bottleneck called limiter with maxConcurrent option set to NUM_THREADS. This ensures that at most NUM_THREADS requests are executed concurrently.
We define the scrape_page function, which takes a url as an argument and performs the scraping for that url. Inside the function:

We use axios.get method from axios to send an HTTP GET request to the URL and obtain the response.
If the response's status code is 200 (indicating a successful response), we load the response into cheerio using cheerio.load(response.data).
We extract the title from the HTML using $('h1').text() and add it to the output_data_list array as an object with a 'title' property.

We use Promise.all() to map each URL in list_of_urls to a call to limiter.schedule(). This schedules the scrape_page function for each URL, enforcing the concurrency limit set by limiter.
limiter.schedule() returns a promise for each URL, and Promise.all() waits for all the promises to resolve. Once all the promises are resolved, we reach the await line, and execution continues to the next line.
We log the output_data_list to the console, which contains the scraped data from all the URLs.

Overall, the code sets a concurrency limit using Bottleneck and utilizes axios and cheerio for making HTTP requests and parsing the HTML response, respectively. The scraping is done concurrently for multiple URLs, and the scraped data is collected in an array.

Using this approach we can significantly increase the speed at which we can make requests with NodeJs Axios.

Adding Concurrency To ScrapeOps Scrapers

The following is an example sending requests to the ScrapeOps Proxy API, which enables you to use all the available threads your proxy plan allows you to make.

Just change the NUM_THREADS value to the number of concurrent threads your proxy plan allows.

const querystring = require('querystring')
const axios = require('axios');
const cheerio = require('cheerio');
const Bottleneck = require('bottleneck');

const NUM_THREADS = 5;

const SCRAPEOPS_API_KEY = 'your_api_key'; // Replace with your actual API key

function get_scrapeops_url(url) {

  const payload = {
    api_key: SCRAPEOPS_API_KEY,
    url: url
  };

  const proxy_url = `https://proxy.scrapeops.io/v1/?${querystring.stringify(payload)}`;
  return proxy_url;
}

// Example list of URLs to scrape
const list_of_urls = [
  'http://quotes.toscrape.com/page/1/',
  'http://quotes.toscrape.com/page/2/',
  'http://quotes.toscrape.com/page/3/',
  'http://quotes.toscrape.com/page/4/',
  'http://quotes.toscrape.com/page/5/'
];

const output_data_list = [];
const limiter = new Bottleneck({ maxConcurrent: NUM_THREADS });

async function scrape_page(url) {
  try {
    const response = await axios.get(get_scrapeops_url(url));

    if (response.status === 200) {
      const $ = cheerio.load(response.data);

      const title = $('h1').text();

      // Add scraped data to "output_data_list" array
      output_data_list.push({
        'title': title
      });
    }
  } catch (error) {
    console.log('Error', error);
  }
}

(async () => {
  await Promise.all(
    list_of_urls.map(url =>
      limiter.schedule(() => scrape_page(url))
    )
  );

  console.log(output_data_list);
})();

You can get your own free API key with 1,000 free requests by signing up here.

NodeJs Axios: Make Concurrent Requests

Make Concurrent Requests Using Promise.all() & Bottleneck​

Adding Concurrency To ScrapeOps Scrapers​

More Web Scraping Tutorials​

Make Concurrent Requests Using Promise.all() & Bottleneck

Adding Concurrency To ScrapeOps Scrapers

More Web Scraping Tutorials