Wait For Element

The ScrapeOps Proxy API Aggregator enables you to tell the headless browser to wait for a specific element to appear on the page before returning the HTML response.

This is useful if you are scraping dynamic websites that take a while to load all the data onto the page.

To enable the ScrapeOps Proxy's Wait Functionality, you just need to add the wait_for query parameter to your request, along with a CSS selector of the page element you want to wait for.

For example, adding wait_for=.loading-done to the request will tell the headless browser to wait until the element with the CSS selector .loading-done before returning the HTML response. Giving the website enough time to load the target data onto the page.

curl -k "https://proxy.scrapeops.io/v1/?api_key=YOUR_API_KEY&url=http://example.com/&wait_for=.loading-done"

Automatically Enables Javascript Rendering

Adding a wait_for to the request will automatically enable a headless browser (equivalent to adding render_js=true to the request). Using the Headless Browser functionality will consume 10 API Credits.