Skip to main content

Proxy Aggregator Quick Start

ScrapeOps Proxy Aggregator is an easy to use proxy that gives you access to the best performing proxies via a single endpoint. We take care of finding the best proxies, so you can focus on the data.

Authorisation - API Key

To use the ScrapeOps proxy, you first need an API key which you can get by signing up for a free account here.

Your API key must be included with every request using the api_key query parameter otherwise the API will return a 403 Forbidden Access status code.


Making Requests

To make requests you need send the URL you want to scrape to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/ by adding your API Key and URL to the request using the api_key and url query parameter:


curl "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything"

The ScrapeOps Proxy supports GET requests (POST requests coming soon).

The following is some example Python code to use with Proxy API:


import requests

response = requests.get(
url='https://proxy.scrapeops.io/v1/',
params={
'api_key': 'YOUR_API_KEY',
'url': 'http://httpbin.org/ip',
},
)

print('Body: ', response.content)


ScrapeOps will take care of the proxy selection and rotation for you so you just need to send us the URL you want to scrape.


Response Format

After recieving a response from one of our proxy providers the ScrapeOps Proxy API will then respond with the raw HTML content of the target URL along with a response code:


<html>
<head>
...
</head>
<body>
...
</body>
</html>

The ScrapeOps Proxy API will return a 200 status code when it successfully got a response from the website that also passed response validation, or a 404 status code if the website responds with a 404 status code. Both of these status codes are considered successful requests.

The following is a list of possible status codes:

Status CodeBilledDescription
200YesSuccessful response.
404YesPage requested does not exist.
400NoBad request. Either your url or query parameters are incorrectly formatted.
401NoYou have consumed all your credits. Either turn off your scraper, or upgrade to a larger plan.
403NoEither no api_key included on request, or api_key is invalid.
429NoExceeded your concurrency limit.
500NoAfter retrying for up to 2 minutes, the API was unable to receive a successful response.

Advanced Functionality

To enable other API functionality when using the Proxy API endpoint you need to add the appropriate query parameters to the ScrapeOps Proxy URL.

For example, if you want to enable Javascript rendering with a request, then add render_js=true to the request:


curl "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything&render_js=true"

The API will accept the following parameters:

ParameterDescription
render_jsRequest with Javascript rendering enabled. Example: render_js=true
residentialRequest using residential proxy pools. Example: residential=true
countryMake requests from specific country. Example: country=us

Timeout

The ScrapeOps proxy keeps retrying a request for up to 2 minutes before returning a failed response to you.

To use the Proxy correctly, you should set the timeout on your request to a least 2 minutes to avoid you getting charged for any successful request that you timed out on your end before the Proxy API responded.


Dashboard

You can monitor your scraping performance using the Proxy Dashboard.


Usage Endpoint

You can programmatically monitor your ScrapeOps Proxy API credit consumption and concurrency usage using the usage endpoint.


curl "https://proxy.scrapeops.io/v1/account?api_key=YOUR_API_KEY"

Example response:


{
"plan_api_credits": 1000000,
"used_api_credits": 455332,
"plan_max_concurrency": 100,
"active_concurrency": 15,

}