Skip to main content

NodeJs Puppeteer Examples

The following are code examples on how to integrate the ScrapeOps Proxy Aggregator with your NodeJs Puppeteer Scrapers.

Authorisation - API Key​

To use the ScrapeOps proxy, you first need an API key which you can get by signing up for a free account here.

Your API key must be included with every request using the password proxy port parameter otherwise the proxy port will return a 403 Forbidden Access status code.

Residential Proxy Port​

The following is some example code to send a URL to the ScrapeOps Residential Proxy port:


"http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181"

Here are the individual connection details:

  • Proxy: residential-proxy.scrapeops.io
  • Port: 8181
  • Username: scrapeops
  • Password: YOUR_API_KEY

To enable extra/advanced functionality, you can pass parameters by adding them to username, separated by periods.

For example, if you want to enable Country Geotargeting with a request, the username would be scrapeops.country=us.


"http://scrapeops.country=us:YOUR_API_KEY@residential-proxy.scrapeops.io:8181"

Also, multiple parameters can be included by separating them with periods.

Integrating With Puppeteer Scrapers​

To integrate our proxy with your Puppeteer you just need to define the proxy port settings, set Puppeteer to ignore HTTPS errors and configure the proxy authorization:


const puppeteer = require('puppeteer');
const cheerio = require('cheerio');

// ScrapeOps proxy configuration
PROXY_USERNAME = 'scrapeops';
PROXY_PASSWORD = 'YOUR_API_KEY'; // <-- enter your API_Key here
PROXY_SERVER = 'residential-proxy.scrapeops.io';
PROXY_SERVER_PORT = '8181';

(async () => {
    const browser = await puppeteer.launch({
        ignoreHTTPSErrors: true,
        args: [
            `--proxy-server=http://${PROXY_SERVER}:${PROXY_SERVER_PORT}`
        ]
    });
    const page = await browser.newPage();
    await page.authenticate({
        username: PROXY_USERNAME,
        password: PROXY_PASSWORD,
      });
    

    try {
        await page.goto('https://quotes.toscrape.com/page/1/', {timeout: 180000});
        let bodyHTML = await page.evaluate(() => document.body.innerHTML);
        let $ = cheerio.load(bodyHTML);

        // find H1 text
        let h1Text = $('h1').text()
        console.log('h1Text:', h1Text)

    } catch(err) {
        console.log(err);
    }
    
    await browser.close();

})();

ScrapeOps will take care of the proxy selection and rotation for you so you just need to send us the URL you want to scrape.

Response Format​

After receiving a response from one of our proxy providers the ScrapeOps Residential Proxy Aggregator will then respond with the raw HTML content of the target URL along with a response code:


<html>
  <head>
     ...
  </head>
  <body>
     ...
  </body>
</html>

Status Codes​

The ScrapeOps Residential Proxy Aggregator will return the status code returned by the target website.

However, if the proxy port will return the following status codes if there are specific errors in your request:

Status CodeBilledDescription
400NoBad request. Either your url or query parameters are incorrectly formatted.
401NoYou have consumed all your credits. Either turn off your scraper, or upgrade to a larger plan.
403NoEither no api_key included on request, or api_key is invalid.

Here is the full list of status codes the Proxy Port returns.

Limiting API Requests

When you use a headless browser to scrape a page, it will generate 10's or 100's of additional requests to download extra JS, CSS and image files and query external APIs. These additional requests will be treated as additional requests to our Proxy Port so it is recommended you configure your Playwright scraper to only make the requests that are absolutely necessary to retrieve the data you want to extract.