Anti-Bot Bypasses
The ScrapeOps Proxy API enables you to bypass most anti-bot systems that are designed to prevent developers from scraping data from websites.
The default Proxy API is able to bypass most common techniques websites use to detect and block scrapers, however, some websites use more advanced anti-scraping techniques or 3rd party anti-bot solutions like Cloudflare, Incapsula, and PerimeterX to block scraping.
These more advanced techniques often involve TLS fingerprinting, browser fingerprinting, and JS challenges, to detect and block scrapers. To bypass these more advanced techniques you will have to activate one of our in-built bypasses by adding the bypass query parameter to your request.
| Bypass | API Credits | Description |
|---|---|---|
generic_level_1 | 10 | The generic level 1 anti-bot bypass is designed to bypass simple anti-bots. |
generic_level_2 | 35 | Generic anti-bot bypass that is designed to bypass most anti-bots. On large plans the credit multiple will be increased to maintain a flat rate of $3.50 per thousand requests. |
generic_level_3 | 50 | Generic anti-bot bypass that is designed to bypass complex anti-bot systems. On large plans the credit multiple will be increased to maintain a flat rate of $4 per thousand requests. |
generic_level_4 | 85 | Generic anti-bot bypass that is designed to bypass the most extreme anti-bot systems. On large plans the credit multiple will be increased to maintain a flat rate of $8 per thousand requests. |
cloudflare_level_1 | 10 | Use to bypass Cloudflare protected sites with low security settings enabled. |
cloudflare_level_2 | 35 | Use to bypass Cloudflare protected sites with medium security settings enabled. On large plans the credit multiple will be increased to maintain a flat rate of $3.50 per thousand requests. |
cloudflare_level_3 | 50 | Use to bypass Cloudflare protected sites with high security settings enabled. On large plans the credit multiple will be increased to maintain a flat rate of $4 per thousand requests. |
incapsula | 35 | Use to bypass Incapsula protected sites. On large plans the credit multiple will be increased to maintain a flat rate of $3.50 per thousand requests. |
perimeterx | 35 | Use to bypass PerimeterX protected sites. On large plans the credit multiple will be increased to maintain a flat rate of $3.50 per thousand requests. |
datadome | 50 | Use to bypass DataDome protected sites. On large plans the credit multiple will be increased to maintain a flat rate of $4 per thousand requests. |
As always, you only get charged for successful requests. So if the bypass doesn't work then you won't get charged.
These bypasses rely on using headless browsers to simulate real user fingerprints and behaviours to successfully scrape protected websites. As a result, they cost more than using normal HTTP requests (1 API Credit).
To keep your costs as low as possible, you should first check to see if you can get good results without any bypass activated on your request.
Then, if the performance isn't good enough for your use case you can try a anti-bot specific bypass or one of our generic bypasses. Different bypasses use different browser, proxy, and bypass configurations so if one bypass doesn't work then another one might.
Activate Anti-Bot Bypass
For example, to scrape a Cloudflare protected website, you can enable our Cloudflare bypass by adding bypass=cloudflare_level_1 to your request.
curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://example.com/&bypass=cloudflare_level_1"
We don't support logging into websites whilst also using the bypass functionality as it makes the requests much more detectable and can degrade the performance of the bypasses for other users.
Cloudflare Bypasses
Cloudflare is the most common anti-bot system being used by websites today, and bypassing it depends on which security settings the website has enabled.
To combat this, we offer 3 different Cloudflare bypasses designed to solve the Cloudflare challenges at each security level.
| Security Level | Bypass | API Credits | Description |
|---|---|---|---|
| Low | cloudflare_level_1 | 10 | Use to bypass Cloudflare protected sites with low security settings enabled. |
| Medium | cloudflare_level_2 | 35 | Use to bypass Cloudflare protected sites with medium security settings enabled. On large plans the credit multiple will be increased to maintain a flat rate of $3.50 per thousand requests. |
| High | cloudflare_level_3 | 50 | Use to bypass Cloudflare protected sites with high security settings enabled. On large plans the credit multiple will be increased to maintain a flat rate of $4 per thousand requests. |