Skip to main content

Custom Parser

Overview​

ScrapeOps Custom Parser lets you generate production‑ready parsers with AI from the UI, then run them at scale via the ScrapeOps platform. Today we support e‑commerce product detail pages and e‑commerce search/listing pages. Additional page types are on the way.

Create a Custom Parser in the UI (AI‑assisted)​

  1. Open the Scraper Generator in the ScrapeOps app and paste one or more example URLs.
  2. Choose your preferred language: Python or Node.js.
  3. Click Generate. Our AI builds a parser tailored to the target site and page type.
  4. Review and download the generated code, or use it directly via our APIs (no hosting required).

Notes

  • Python and Node.js are supported for generated code.
  • Page types currently supported: E‑commerce Product and E‑commerce Search/Listing. Others are coming soon.

Using Your Custom Parser via ScrapeOps​

You can run your Custom Parser in two ways. Both use the same query parameter: custom_parser_id.

Option 1: Proxy API Aggregator (URL fetch + parse)​

Use the ScrapeOps Proxy API Aggregator to fetch the page HTML first, then automatically parse it with your Custom Parser in one request.

  • Enable auto_extract=true.
  • Add custom_parser_id=YOUR_PARSER_ID and api_key=YOUR_API_KEY.

Example (cURL)

curl "https://proxy.scrapeops.io/v1/?api_key=YOUR_SCRAPEOPS_API_KEY&url=https%3A%2F%2Fwww.example.com%2Fproduct%2F123&auto_extract=true&custom_parser_id=YOUR_PARSER_ID"

Key points

  • Advanced request options are available for better scraping (headers, cookies, JS rendering, geotargeting, sessions, retries, and more).
  • Parser usage is included in proxy credits (no additional parser charge when using this method).
  • See the Proxy API docs and Auto‑Extract for details: Proxy API Quickstart · Auto‑Extract

Option 2: Parser API (parse only, you provide HTML)​

Send raw HTML you fetched by any method to the ScrapeOps Parser API. The API parses it with your Custom Parser and returns structured JSON.

  • Include custom_parser_id=YOUR_SCRAPEOPS_API_KEY.

Example (cURL)

curl -X POST "https://parser.scrapeops.io/v1/custom-parser?api_key=YOUR_API_KEY&custom_parser_id=YOUR_PARSER_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "html": "<html>...</html>"
  }'

Key points

  • Ideal when you already have HTML (e.g., your own crawler or archived pages).
  • Parsing usage is billed on the Parser API. See: Parser API Overview

Supported Page Types (Current)​

  • E‑commerce Product Detail pages
  • E‑commerce Search/Listing pages

We are actively adding support for additional page types.

Languages​

  • Python
  • Node.js