Skip to main content

Request Optimization

Certain domains are very hard to scrape and require you to use more advanced/expensive functionality to scrape them reliably at scale.

The ScrapeOps Proxy API Aggregator provides an automatic Request Optimization functionality that when enabled will tell the API to find the optimal request settings to give you the best performance at the lowest cost.

Instead of you having to decide which features and proxies to use, the ScrapeOps Proxy API Aggregator will enable/disable the following features for you to give you the best performance at the lowest cost:

To enable Request Optimization, simply add optimize_request=true to your request and the Proxy API will take care of the rest.


curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything&optimize_request=true"

Functionality Priority

It is recommended not to manually enable/disable other functionality when using optimize_request=true as it can interfer with how the Proxy API optimizes the request. However, if you need to enable certain functionality to get the correct data, then give it a try and if the data isn't what you expected then just reach out to us at info@scrapeops.io and we can look into it for you.

If you set optimize_request=true but also enable other functionality on your request, then depending on which functionality you have enabled the functionality enabled by the Request Optimization will get the priority.

Learning Time​

If the Proxy API hasn't seen the domain you are trying to request before, then it might need some time to learn and split-test different proxy configurations to determine the optimal setup.

So if your first few requests don't get successful responses then send a few more as this will help our optimization algorithm learn. However, if after 10-20 requests it still isn't giving good performance then just reach out to us at info@scrapeops.io and we will look into it for you.

Request Costs​

When using the Request Optimization functionality, request costs can vary from 1 API credit to 50 API credits per request, depending on what features & proxies are required to successfully scrape the data.

If there is the option of using a lower cost versus a higher cost request setup, then it will try the lower cost request setup first before falling back to the higher cost request setup if the cheaper option fails.

Target Performance & Costs

The Proxy API will aim to give you the lowest costs whilst maintaining a 90-95% success rate for your requests.

Certain domains are very hard to scrape, so trying to maintain a +99% success rate could result in unnecessarily inflating your costs as you would end up using the most expensive option for every request. Whereas if you were okay with a 90-95% success rate then you could scrape at a much lower cost and you can simply retry any failed requests on your end.

Max Request Costs​

If you want to control your costs, you can set a maximum request cost you want to incur using the max_request_cost parameter.

For example, if you set max_request_cost=30 then the Proxy API will only use request setups that would cost you up to and including 30 API credits per request.


curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://httpbin.org/anything&optimize_request=true&max_request_cost=30"

Setting a too low max_request_cost can mean that all your requests fail as any proxy setups that work for your target domain cost more than your max_request_cost. Or it can also lower your performance as your requests can't be fully optimized.