Skip to main content

Powerful Job Monitoring & Scheduling for Web Scraping

Schedule your scraping jobs, monitor their performance, conduct health checks & get alerts with ScrapeOps.

In return for your feedback, Beta users get free access for 1 year!

ScrapeOps Dashboard Demo

Real-Time & Historical Job Stats

Easily monitor jobs in real-time, compare jobs to previous jobs run, spot trends forming and catch problems early before your data feeds go down.

Item & Page Vaildation

ScrapeOps checks pages for CAPTCHAs & bans, and the data quality of every item scraped so you can detect broken parsers without having to query your DB.

Job Scheduling & Management

Plan, schedule, manage and run all your jobs across multiple servers with a single, easy to use scraper management dashboard. Integrates via Scrapyd and SSH.

Automatic Health Checks

ScrapeOps automatically checks every job versus historical averages and custom health checks, then tells you if your scrapers are healthy or not.

Custom Checks, Alerts & Reports

Easily configure your own custom health checks and reports, and get alerts how and when you want them. Alerts via Slack, email and more.

Proxy Analytics

Monitor your current proxy usage & performance, and get alerts when you are running out of credits or there is a problem.

Real-Time Job Monitoring

Easily Monitor Your Scrapers

ScrapeOps automatically logs and ships your scraping performance stats, so you can monitor everything in a single dashboard.

Effortlessly compare pages & items scraped, runtimes, status codes, success rates and errors versus previous job runs to identify potential issues with your scrapers.

ScrapeOps Real-Time Scraper Monitoring
ScrapeOps Job Scheduler

Health Checks & Alerts

Built-In Health Checks & Alerts

Out of the box, ScrapeOps automatically checks the health of every job versus its historical moving average to see if it is healthy or not. Alerting you when and how you want it.

If that isn't enough, you can configure custom checks, alerts and reports on any job or spider.

Job Scheduling & Management

Simplify Job Management

Easily setup and manage your web scrapers across multiple servers, from one easy to use job management dashboard.

Schedule, run, pause and re-run your scraping jobs with our full Scrapyd integration.

ScrapeOps Job Scheduler

Integration

Integrate With Scrapy in 30 Seconds

Web scraping can be painful enough without having to build your own custom monitoring and alerting system from scratch.

With just 3 lines of code, ScrapeOps gives you all the monitoring and alerting options you will ever need straight out of the box.

Install the ScrapeOps SDK:
pip install scrapeops-scrapy
Add 3 lines to your settings.py file:
## settings.py

# Add Your ScrapeOps API key
SCRAPEOPS_API_KEY = 'YOUR_API_KEY'


# Add In The ScrapeOps Extension
EXTENSIONS = {
'scrapeops_scrapy.extension.ScrapeOpsMonitor': 500,
}


# Update The Download Middlewares
DOWNLOADER_MIDDLEWARES = {
'scrapeops_scrapy.middleware.retry.RetryMiddleware': 550,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
}

## DONE!

More Integrations On The Way

Python Requests

Python Requests

Python's most popular HTTP request library.

Python Selenium

Python Selenium

Python's most popular headless browser.

NodeJs

NodeJs

NodeJs popular Requests, Axios, fetch libraries.

Puppeteer

Puppeteer

NodeJs most popular headless browser library.

Get Started For Free

If you have a hobby project or need only basic monitoring then use our Community Plan for free, forever!.

Community Plan

Unlimited Usage (Jobs, Spiders, Requests)
Real-Time Job Statistics
Historical Job Analytics
Job Scheduling via ScrapyD
Automatic Job Health Checks & Alerts
Basic Proxy Analytics
Community Support
COMMUNITY PLAN
Free, Forever
pip install scrapeops-scrapy

Unlock Premium Features

If web scraped data is mission critical for your business, then ScrapeOps premium features will make your life a whole lot easier.

Hobby Plan

$7
per month

Hobby Plan

1M Pages per month
Real-Time Monitoring & Alerts
Data Quality Validation
Captcha/Ban Detection & Retries
Custom Job Health Checks & Alerts
Custom Periodic Reports
Custom Scraper Events & Tags
Job Scheduling via SSH
ETL Pipeline Support
Distributed Crawls Support
Email support

Startup Plan

$24
per month

Startup Plan

5M Pages per month
Real-Time Monitoring & Alerts
Data Quality Validation
Captcha/Ban Detection & Retries
Custom Job Health Checks & Alerts
Custom Periodic Reports
Custom Scraper Events & Tags
Job Scheduling via SSH
ETL Pipeline Support
Distributed Crawls Support
Email support

Business Plan

$49
per month

Business Plan

15M Pages per month
Real-Time Monitoring & Alerts
Data Quality Validation
Captcha/Ban Detection & Retries
Custom Job Health Checks & Alerts
Custom Periodic Reports
Custom Scraper Events & Tags
Job Scheduling via SSH
ETL Pipeline Support
Distributed Crawls Support
Priority support

Need a bigger plan? Contact us here.

Premium Features Explained

Real-Time Monitoring & Alerts

ScrapeOps will monitor your jobs in real-time, and alert you if their performance drops instead of only at job finish.

Data Quality Validation

Monitor your items field coverage and setup custom unit tests so you can detect parser issues straight away.

Captcha/Ban Detection & Retries

ScrapeOps will check every scraped page for CAPTCHAs & bans and trigger an event so you can retry the request.

Custom Health Checks & Alerts

Create custom real-time scraper health checks for all your scrapers so you detect unhealthy jobs straight away. Simple to setup with Slack, email or phone.

Custom Periodic Reports

Automate your daily scraping checks by scheduling ScrapeOps to check your spiders & jobs every couple hours and send you a report if any issues are detected.

Custom Scraper Events & Tags

Trigger custom events in your scrapers to log any invalid HTML responses, geotargeting or data quality issues that are specific to your own scrapers.

Scheduling via SSH

Give ScrapeOps SSH access to your servers, and you will be able to schedule and run any type of scraper from the dashboard. No more setting up CRON jobs.

ETL Pipeline Support

Monitor scraping and data parsing across multiple servers even when you have split HTML extraction and data parsing between different processes.

Distributed Crawl Support

Monitor your scrapers even when you have a distributed scraping infrastructure, where multiple scraping processes are working from the same queue.

Ready to make web scraping less painful?

Sign up for a free account today.