Certain domains are very hard to scrape and require you to use more advanced/expensive functionality to scrape them reliably at scale.
The ScrapeOps Proxy API provides an automatic Request Optimization functionality that when enabled will tell the API to find the optimal request settings to give you the best performance at the lowest cost.
Instead of you having to decide which features and proxies to use, the ScrapeOps Proxy API will enable/disable the following features for you to give you the best performance at the lowest cost:
- JS Rendering
- Country Geotargeting
- Premium Proxies
- Residential Proxies
- Mobile Proxies
- Anti-Bot Bypasses
To enable Request Optimization, simply add
optimize_request=true to your request and the Proxy API will take care of the rest.
It is recommended not to manually enable/disable other functionality when using
optimize_request=true as it can interfer with how the Proxy API optimizes the request. However, if you need to enable certain functionality to get the correct data, then give it a try and if the data isn't what you expected then just reach out to us at firstname.lastname@example.org and we can look into it for you.
If you set
optimize_request=true but also enable other functionality on your request, then depending on which functionality you have enabled the functionality enabled by the Request Optimization will get the priority.
If the Proxy API hasn't seen the domain you are trying to request before, then it might need some time to learn and split-test different proxy configurations to determine the optimal setup.
So if your first few requests don't get successful responses then send a few more as this will help our optimization algorithm learn. However, if after 10-20 requests it still isn't giving good performance then just reach out to us at email@example.com and we will look into it for you.
When using the Request Optimization functionality, request costs can vary from 1 API credit to 50 API credits per request, depending on what features & proxies are required to successfully scrape the data.
If there is the option of using a lower cost versus a higher cost request setup, then it will try the lower cost request setup first before falling back to the higher cost request setup if the cheaper option fails.
The Proxy API will aim to give you the lowest costs whilst maintaining a 90-95% success rate for your requests.
Certain domains are very hard to scrape, so trying to maintain a +99% success rate could result in unnecessarily inflating your costs as you would end up using the most expensive option for every request. Whereas if you were okay with a 90-95% success rate then you could scrape at a much lower cost and you can simply retry any failed requests on your end.
Max Request Costs
If you want to control your costs, you can set a maximum request cost you want to incur using the
For example, if you set
max_request_cost=30 then the Proxy API will only use request setups that would cost you up to and including 30 API credits per request.
Setting a too low
max_request_cost can mean that all your requests fail as any proxy setups that work for your target domain cost more than your
max_request_cost. Or it can also lower your performance as your requests can't be fully optimized.