NodeJs Got: How to Use & Rotate Proxies

import got from 'got';
import { HttpProxyAgent, HttpsProxyAgent } from 'hpagent';

const requestUrl = 'https://httpbin.org/ip';
const proxyUrl = 'http://proxy.example.com:8080';
const options = {
  agent: {
    https: new HttpsProxyAgent({
       proxy: proxyUrl
    })
  }
};
(async () => {
    try {
        const response = await got.get(requestUrl, options);

        console.log(response.body);
    } catch (error){
        console.log('error', error);
    }
})();

In this guide for The Nodejs Web Scraping Playbook, we will look at how to integrate the 3 most common types of proxies into our Nodejs Got based web scraper.

Using proxies with the Got library allows you to spread your requests over multiple IP addresses making it harder for websites to detect & block your web scrapers.

In this guide we will walk you through the 3 most common proxy integration methods and show you how to use them with Nodejs Got:

• Using Proxy IPs With Got
• Proxy Authentication With Got
• The 3 Most Common Proxy Formats
• Proxy Integration #1: Rotating Through Proxy IP List
• Proxy Integration #2: Using Proxy Gateway
• Proxy Integration #3: Using Proxy API Endpoint

Let's begin...

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

Using Proxy IPs With Got​

Using a proxy with Got is quite straightforward. We simply need to create a new HttpProxyAgent instance by passing proxy url as proxy param and set it to options.agent.https field.

import got from 'got';
import { HttpsProxyAgent } from 'hpagent';

const options = {
  agent: {
    https: new HttpsProxyAgent({
       proxy: 'http://proxy.example.com:8080';
    })
  }
};
(async () => {
    try {
        const response = await got.get('https://httpbin.org/ip', options);

        console.log(response.body);
    } catch (error){
        console.log('error', error);
    }
})();

This method will work for all request methods Got supports: get, post, put, delete, patch, head.

Proxy Authentication With Got​

Some proxy IPs require authentication in the form of a username and password to use the proxy.

To add authentication to the proxy we can simply add the username and password to the proxy strings:

import got from 'got';
import { HttpsProxyAgent } from 'hpagent';

const proxyUrl = 'http://USERNAME:PASSWORD@proxy.example.com:8080';
const options = {
  agent: {
    https: new HttpsProxyAgent({
       proxy: proxyUrl
    })
  }
};
(async () => {
    try {
        const response = await got.get('https://httpbin.org/ip', options);

        console.log(response.body);
    } catch (error){
        console.log('error', error)
    }
})();

The 3 Most Common Proxy Formats​

That covered the basics of integrating a proxy into Got. In the next sections we will show you how to integrate Got into the 3 most common proxy formats:

• Rotating Through List of Proxy IPs
• Using Proxy Gateways
• Using Proxy APIs

A couple years ago, proxy providers would sell you a list of proxy IP addresses and you would configure your scraper to rotate through these IP addresses and use a new one with each request.

However, today more and more proxy providers don't sell raw lists of proxy IP addresses anymore. Instead providing access to their proxy pools via proxy gateways or proxy API endpoints.

We will look at how to integrate with all 3 proxy formats.

Proxy Integration #1: Rotating Through Proxy IP List​

Here a proxy provider will normally provide you with a list of proxy IP addresses that you will need to configure your scraper to rotate through and select a new IP address for every request.

The proxy list you recieve will look something like this:

[
    'http://Username:Password@85.237.57.198:20000',
    'http://Username:Password@85.237.57.198:21000',
    'http://Username:Password@85.237.57.198:22000',
    'http://Username:Password@85.237.57.198:23000',
]

To integrate them into our scrapers we need to configure our code to pick a random proxy from this list everytime we make a request.

In our NodeJS Got scraper we could do it like this:

import got from 'got';
import { HttpsProxyAgent } from 'hpagent';

(async () => {
  const proxyList = [
    'https://Username:Password@85.237.57.198:20000',
    'https://Username:Password@85.237.57.198:21000',
    'https://Username:Password@85.237.57.198:22000',
    'https://Username:Password@85.237.57.198:23000',
  ];

  const randomProxy = proxyList[Math.floor(Math.random() * proxyList.length)];

  const options = {
    agent: {
      https: new HttpsProxyAgent({
        proxy: randomProxy
      })
    }
  };
  
  try {
    const response = await got.get('https://httpbin.org/ip', options)

    console.log(response.body);
  } catch (error) {
    console.log('error', error)
  }
})();

This is a simplistic example, as when scraping at scale we would also need to build a mechanism to monitor the performance of each individual IP address and remove it from the proxy rotation if it got banned or blocked.

Proxy Integration #2: Using Proxy Gateway​

Increasingly, a lot of proxy providers aren't selling lists of proxy IP addresses anymore. Instead, they give you access to their proxy pools via a proxy gateway.

Here, you only have to integrate a single proxy into your Got scraper and the proxy provider will manage the proxy rotation, selection, cleaning, etc. on their end for you.

This is the most common way to use residential and mobile proxies, and is becoming increasingly common when using datacenter proxies too.

Here is an example of how to integrate a BrightData's residential proxy gateway into our Got scraper:

import got from 'got';
import { HttpsProxyAgent } from 'hpagent';

(async () => {
  const options = {
    agent: {
      https: new HttpsProxyAgent({
        proxy: 'http://USERNAME:PASSWORD@zproxy.lum-superproxy.io:22225'
      })
    }
  };
  
  try {
    const response = await got.get('https://httpbin.org/ip', options);

    console.log(response.body);
  } catch (error) {
    console.log('error', error);
  }
})();

As you can see, it is much easier to integrate than using a proxy list as you don't have to worry about implementing all the proxy rotation logic.

Proxy Integration #3: Using Proxy API Endpoint​

Recently, a lot of proxy providers have started offering smart proxy APIs that take care of managing your proxy infrastructure for you by rotating proxies and headers for you so you can focus on extracting the data you need.

Here you typically send the URL you want to scrape to their API endpoint and then they will return the HTML response to you.

Although every proxy API provider has a slightly different API integration, they are all very similar and are very easy to integrate with.

Here is an example of how to integrate with the ScrapeOps Proxy Manager:

import got from 'got';

(async () => {
    const SCRAPEOPS_API_KEY = 'YOUR_API_KEY';
    const url = 'http://httpbin.org/ip';

    try{
        const response = await got.get(`https://proxy.scrapeops.io/v1?api_key=${SCRAPEOPS_API_KEY}&url=${url}`)
        console.log(response.body);
    } catch (error){
        console.log('error', error);
    }
})();

Here you simply send the url you want to scrape to the ScrapeOps API endpoint in the url query parameter, along with your API key in the api_key query parameter, and ScrapeOps will deal with finding the best proxy for that domain and return the HTML response to you.

Note that we did not have to configure the proxy url in request options since ScrapeOps proxy API takes care of that.

You can get your own free API key with 1,000 free requests by signing up here.

More Web Scraping Tutorials​

So that's how you can integrate proxies into your Got scrapers.

If you would like to learn more about Web Scraping, then be sure to check out The Web Scraping Playbook.

Or check out one of our more in-depth guides:

• How to Scrape The Web Without Getting Blocked Guide
• The State of Web Scraping 2020
• The Ethics of Web Scraping