Skip to main content

ScrapeOps Quickstart Guide: Proxy APIs, Monitoring & Scheduling

Welcome to the ScrapeOps Documentation pages. Here you will find info on how to integrate and use our products.

💻 Explore the ScrapeOps Dashboard: Interactive Demo​

🔗 ScrapeOps Dashboard Demo

💻 ScrapeOps Proxy API Aggregator​

ScrapeOps Proxy API Aggregator is an easy to use proxy that gives you access to the best performing Proxy APIs via a single endpoint. We take care of finding the best proxies, so you can focus on the data.

To use the ScrapeOps Proxy API Aggregator, you first need an API key which you can get by signing up for a free account here.

🚀 Getting Started​

To make requests you need send the URL you want to scrape to the ScrapeOps Proxy API endpoint https://proxy.scrapeops.io/v1/ by adding your API Key and URL to the request using the api_key and url query parameter:

import requests

response = requests.get(
    'https://proxy.scrapeops.io/v1/',
    params={
        'api_key': 'YOUR_API_KEY',
        'url': 'http://httpbin.org/anything'
    }
)
print(response.text)  # Returns the HTML content

With the ScrapeOps Proxy API Aggregator you are only charged for successful requests (200 and 404 status codes).

To learn how to use the ScrapeOps Proxy API Aggregator and customise it to your requirement then check out the QuickStart Guide.

🏠 ScrapeOps Residential Proxy Aggregator​

ScrapeOps Residential Proxy Aggregator is an easy to use proxy that gives you access to the best performing Residential Proxy providers via a single proxy port. We take care of finding the best proxies, so you can focus on the data.

To use the ScrapeOps Residential Proxy Aggregator, you first need an API key which you can get by signing up for a free account here.

🚀 Getting Started​

To make requests you need send the URL you want to scrape to set their proxy port to the ScrapeOps Residential Proxy Port http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181

The username for the proxy is scrapeops and the password is your API key.

import requests

proxies = {
    'http': 'http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181',
    'https': 'http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181'
}

response = requests.get('https://httpbin.org/ip', proxies=proxies)
print(response.text)

Here are the individual connection details:

  • Proxy: residential-proxy.scrapeops.io
  • Port: 8181
  • Username: scrapeops
  • Password: YOUR_API_KEY

With the ScrapeOps Residential Proxy Aggregator you are charged for bandwidth consumed.

To learn how to use the ScrapeOps Residential Proxy Aggregator and customise it to your requirement then check out the QuickStart Guide.

📊 ScrapeOps Monitoring​

ScrapeOps Monitoring is a monitoring tool purpose built for web scraping. With a simple 30 seconds install of one of our SDKs, your scraper's performance & error stats will be automatically aggregated and shipped to your ScrapeOps dashboard.

⭐ Features & Functionality​

ScrapeOps Monitoring gives you the following features & functionality:

  • Scrapy Job Stats & Visualisation

    • 📈 Individual Job Progress Stats
    • 📊 Compare Jobs versus Historical Jobs
    • 💯 Job Stats Tracked
      • ✅ Pages Scraped & Missed
      • ✅ Items Parsed & Missed
      • ✅ Item Field Coverage
      • ✅ Runtimes
      • ✅ Response Status Codes
      • ✅ Success Rates & Average Latencies
      • ✅ Errors & Warnings
      • ✅ Bandwidth

  • Health Checks & Alerts

    • 🔍 Custom Spider & Job Health Checks
    • 📦 Out of the Box Alerts - Slack (More coming soon!)
    • 📑 Daily Scraping Reports

🚀 Getting Started​

To use ScrapeOps you first need to create a free account and get your free API_KEY.

Currently ScrapeOps integrates with both Python Requests & Python Scrapy scrapers:

  1. Python Requests Integration
  2. Python Scrapy Integration

More ScrapeOps Monitoring integrations are on the way.

⏰ ScrapeOps Server Manager & Scheduler​

ScrapeOps Server Manager & Job Scheduler is a easy to use server integration that enables you to deploy, manage and schedule your scrapers from the ScrapeOps dashboard.

There are two options to integrate ScrapeOps with your servers:

  1. Via SSH (Recommended)
  2. Via Scrapyd Server HTTP Endpoints (Only Applicable to Python Scrapy)

⭐ Features & Functionality​

ScrapeOps Server Manager & Job Scheduler gives you the following features & functionality:

  • SSH Server Management
    • 🔗 Integrate With Any SSH Capably Server
    • 🕷 Deploy scrapers directly from GitHub to your servers.
    • ⏰ Schedule Periodic Jobs
  • ScrapyD Cluster Management
    • 🔗 Integrate With ScrapyD Servers
    • ⏰ Schedule Periodic Jobs
    • 💯 All Scrapyd JSON API Supported
    • 🔐 Secure Your ScrapyD with BasicAuth, HTTPS or Whitelisted IPs

To learn how to setup the integrate ScrapeOps with your servers with this guide.

🤖 ScrapeOps n8n Node​

ScrapeOps n8n Node is a powerful integration that brings all of ScrapeOps' web scraping capabilities into the n8n workflow automation platform. Perfect for no-code builders and developers who want to integrate web scraping into their automation workflows.

⭐ Features & Functionality​

The ScrapeOps n8n node provides access to:

  • Proxy API Integration
    • 🌐 Smart proxy rotation and management
    • 🛡 Anti-bot bypass capabilities
    • 💻 JavaScript rendering support
    • 🌎 Geo-targeting options
  • Parser API Access
    • 🛒 E-commerce parsers (Amazon, eBay, Walmart)
    • 💼 Job site parsers (Indeed)
    • 🏠 Real estate parsers (Redfin)
    • 📦 Returns structured JSON data
  • Data API Direct Access
    • ⚡ Amazon Product API
    • 🔍 Amazon Search API
    • 📈 More APIs coming soon

🚀 Getting Started​

To use the ScrapeOps n8n node:

  1. Install the node from the n8n community nodes
  2. Get your API key from ScrapeOps
  3. Configure credentials in n8n
  4. Start building powerful scraping workflows

Learn more in our comprehensive n8n documentation.

🔌 ScrapeOps MCP Server​

ScrapeOps MCP Server exposes the ScrapeOps Proxy API to MCP-compatible IDEs (Cursor, Claude Desktop, VS Code, Windsurf). It gives AI agents first-class scraping and browsing tools with proxy support, anti-bot bypass, JavaScript rendering, screenshots, and LLM-powered extraction.

⭐ Features & Functionality​

The ScrapeOps MCP Server provides:

  • Web Browsing Tools
    • 🌐 Browse any URL with proxy support and geo-targeting
    • 🛡 Anti-bot bypass (Cloudflare, DataDome, PerimeterX)
    • 💻 JavaScript rendering for dynamic sites
    • 📷 Screenshots (base64)
  • Data Extraction
    • ⚡ Structured extraction (auto or LLM schema-based)
    • 🔗 Link discovery and normalization
    • 🎯 Optimize-request logic to auto-test and reuse best-performing proxies

🚀 Getting Started​

Run the MCP server with npx:

env SCRAPEOPS_API_KEY=YOUR_API_KEY npx -y @scrapeops/mcp

Then configure your IDE (Cursor, Claude Desktop, VS Code, or Windsurf) to connect to the MCP server:

{
  "mcpServers": {
    "@scrapeops/mcp": {
      "command": "npx",
      "args": ["-y", "@scrapeops/mcp"],
      "env": { "SCRAPEOPS_API_KEY": "YOUR_API_KEY" }
    }
  }
}

Learn more in our comprehensive MCP Server documentation.

⚡ ScrapeOps AI Scraper Builder​

ScrapeOps AI Scraper Builder automatically generates production-ready web scrapers from any e-commerce product page URL using AI. Simply provide URLs, choose your language, and get a working scraper that outputs structured JSON data.

⭐ Supported Page Types​

The AI Scraper Builder currently supports:

  • Product Details: Individual product pages (name, price, images, reviews, specifications)
  • Product Search: Search results pages (product listings, pagination, search metadata)
  • Product Category: Category/browse pages (product listings, subcategories, filters)

🚀 Getting Started​

To use the AI Scraper Builder, you first need to create a free account and get your free API_KEY.

Then navigate to the AI Assistant → Scraper Generator in the dashboard:

  1. Enter up to 5 URLs from the same website
  2. Choose your language (Python or Node.js) and library
  3. Click Generate and get production-ready scraper code

Learn more in our comprehensive AI Scraper Builder documentation.

📄 ScrapeOps Parser API​

ScrapeOps Parser API takes raw HTML pages and parses them into structured JSON format. Simply send us the HTML page for supported websites, and we'll extract the data for you.

⭐ Supported Parsers​

We currently support parsing for:

  • Ecommerce: Amazon, Walmart, eBay, Target
  • Real Estate: Redfin, Zillow
  • Job Portals: Indeed
  • Search Engines: Google (Maps, Patents, Scholar), Bing, Yandex

🚀 Getting Started​

To use the Parser API, you first need to create a free account and get your free API_KEY.

Send HTML to the Parser API endpoint and receive structured JSON:

import requests

# First get the HTML from the target page
html = requests.get('https://www.amazon.com/dp/B08WM3LMJF').text

# Send HTML to the Parser API
response = requests.post(
    'https://parser.scrapeops.io/v2/amazon?api_key=YOUR_API_KEY',
    data={'html': html}
)
print(response.json())

Learn more in our comprehensive Parser API documentation.

📦 ScrapeOps Data APIs​

ScrapeOps Data APIs are a collection of APIs that allow you to access structured JSON data from popular websites using dedicated endpoints. No need to build or maintain your own parsers—just call the API and get the data.

⭐ Available APIs​

The following Data APIs are available:

  • Amazon: Product API, Product Search API
  • Redfin: Sale Search, Rent Search, Sale Detail, Rent Detail, Building Detail, State Search, Agent Search, Agent Profile
  • Indeed: Job Search, Job Detail, Company Search, Top Companies, Company Snapshot, Company About, Company Reviews, Company Jobs
  • eBay: Product API, Search API, Category API, Store API, Feedback API
  • Walmart: Product API, Product Search, Category, Review, Shop, Browse APIs

🚀 Getting Started​

To use the Data APIs, you first need to create a free account and get your free API_KEY.

Query the API by passing in the product ID or URL:

import requests

response = requests.get(
    'https://proxy.scrapeops.io/v1/structured-data/amazon/product',
    params={
        'api_key': 'YOUR_API_KEY',
        'asin': 'B0BNLTS1T3',
        'country': 'us'
    }
)
data = response.json()
print(data['title'])   # "Apple AirPods Pro (2nd Generation)"
print(data['price'])   # "$249.00"

Learn more in our comprehensive Data APIs documentation.