01
Real-time & historical job stats
Easily monitor jobs in real-time, compare jobs to previous jobs run, spot trends forming and catch problems early before your data feeds go down.
Monitoring & Scheduling
Powerful job monitoring & scheduling for web scraping
Schedule your scraping jobs, monitor their performance & get alerts with ScrapeOps Monitoring & Scheduling.
Monitor unlimited scrapers and pages with the free account.
What you get
Everything you need to keep your scrapers healthy
02
Item & page validation
ScrapeOps checks pages for CAPTCHAs & bans, and the data quality of every item scraped so you can detect broken parsers without having to query your DB.
03
Error logging
The ScrapeOps SDK logs any Warnings or Errors raised in your jobs and aggregates them on your dashboard, so you can see your errors without having to check your logs.
04
Automatic health checks
ScrapeOps automatically checks every job versus historical averages and custom health checks, then tells you if your scrapers are healthy or not.
05
Custom checks, alerts & reports
Easily configure your own custom health checks and reports, and get alerts how and when you want them. Alerts via Slack, email and more.
06
Job scheduling & management
Plan, schedule, manage and run all your jobs across multiple servers with a single, easy to use scraper management dashboard. Integrates via Scrapyd and SSH.
Real-time job monitoring
Easily monitor your scrapers
ScrapeOps automatically logs and ships your scraping performance stats, so you can monitor everything in a single dashboard.
Effortlessly compare pages & items scraped, runtimes, status codes, success rates and errors versus previous job runs to identify potential issues with your scrapers.
amazon_products
Job #4182 vs average of last 5 runs
MetricThis jobAverageChange
Pages scraped48,21047,880+0.7%
Items scraped47,96047,500+1.0%
Runtime38m 12s37m 40s+1.4%
Success rate99.2%99.0%+0.2%
Errors1422−36%
Status codes
20096.8%4041.4%4031.0%4290.8%
Health checks & alerts
Built-in health checks & alerts
Out of the box, ScrapeOps automatically checks the health of every job versus its historical moving average to see if it is healthy or not. Alerting you when and how you want it.
If that isn’t enough, you can configure custom checks, alerts and reports on any job or spider.
zillow_listings
Job #2219 · 5 health checks
✓Pages scraped vs 7-day average98%
✕Items scraped vs 7-day average61%
✓Error rate below 2%0.6%
✓Runtime within 25% of average+8%
✕Ban pages below 1%3.4%
ScrapeOps Alerts#scraper-alerts · Slack
zillow_listings #2219 is unhealthy. 2 checks failed: items scraped at 61% of average, ban pages at 3.4%.
Job scheduling & management
Simplified job management
Connect your servers & GitHub with ScrapeOps, and manage your web scrapers across multiple servers from one easy to use job management dashboard.
Deploy, schedule, run, pause and re-run your scraping jobs with our SSH server & GitHub integration.
Learn more →scraper-01DigitalOcean · Onlinescraper-02AWS · Onlineacme/scrapersGitHub · main · Deployed 2h ago
SpiderScheduleNext runStatus
amazon_productsscraper-010 */6 * * *in 2h 14mActiveRunPause
google_serpscraper-01*/15 * * * *in 6mActiveRunPause
zillow_listingsscraper-0230 2 * * *Tomorrow 02:30ActiveRunPause
ebay_pricesscraper-020 0 * * 1Mon 00:00PausedRunResume
4 scheduled jobs across 2 servers+ Schedule job
Integration
Integrate with Python scrapers in 30 seconds
With just 3 lines of code, ScrapeOps gives you all the monitoring and alerting options you will ever need straight out of the box.
Python Scrapy docs →terminal
$ pip install scrapeops-scrapysettings.py
## settings.py# Add Your ScrapeOps API keySCRAPEOPS_API_KEY = 'YOUR_API_KEY'# Add In The ScrapeOps ExtensionEXTENSIONS = {'scrapeops_scrapy.extension.ScrapeOpsMonitor': 500,}# Update The Download MiddlewaresDOWNLOADER_MIDDLEWARES = {'scrapeops_scrapy.middleware.retry.RetryMiddleware': 550,'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,}
More integrations on the way
Python SeleniumPython’s most popular headless browser.
Node.jsNode.js popular Requests, Axios, fetch libraries.
PuppeteerNode.js most popular headless browser library.
PlaywrightNode.js’ hot new headless browser library.Click any integration to register interest.
Pricing
Free forever, with premium features when you need them
If web scraped data is mission critical for your business, then ScrapeOps premium features will make your life a whole lot easier.
Community plan
Free
forever
If you have a hobby project or need only basic monitoring then use our Community Plan for free, forever.
- ✓Unlimited usage (jobs, spiders, requests)
- ✓Real-time job statistics
- ✓Historical job analytics
- ✓Job scheduling via Scrapyd
- ✓Automatic job health checks & alerts
- ✓Basic proxy analytics
- ✓Community support
Hobby plan
$4.95per month
1M pages per month
All premium features, plus email support.
Get free trialStartup plan
Popular$14.95per month
3M pages per month
All premium features, plus email support.
Get free trialBusiness plan
$19.95per month
5M pages per month
All premium features, plus priority support.
Get free trialNeed a bigger plan? Contact us here.
Premium features explained
Included in every Hobby, Startup and Business plan.
Error monitoring
ScrapeOps will log any errors & warnings in your scrapers and will display them in your dashboard, along with the tracebacks.
Server provisioning
Directly link your hosting provider with ScrapeOps, then provision and setup new servers from the ScrapeOps dashboard.
Scheduling via SSH
Give ScrapeOps SSH access to your servers, and you will be able to schedule and run any type of scraper from the dashboard.
Code deployment
Link your GitHub repos to ScrapeOps and deploy new scrapers to your servers directly from the ScrapeOps dashboard.
Custom stats groups
Group jobs together with your own custom parameters, and monitor their grouped stats on your dashboard.
Custom health checks & alerts
Create custom real-time scraper health checks for all your scrapers so you detect unhealthy jobs straight away.
Custom periodic reports
Automate your daily scraping checks by scheduling ScrapeOps to check your spiders & jobs every couple hours and send you a report if any issues are detected.
Custom scraper events & tags
Trigger custom events in your scrapers to log any invalid HTML responses, geotargeting or data quality issues that are specific to your own scrapers.
Distributed crawl support
Monitor your scrapers even when you have a distributed scraping infrastructure, where multiple scraping processes are working from the same queue.
FAQ
Questions about Monitoring & Scheduling
Everything you need to know about ScrapeOps Monitoring & Scheduling.
Still have a question?
The docs cover SDK setup, health checks, alerts and server scheduling.
Read the docs01What is ScrapeOps Monitoring & Scheduling?
ScrapeOps Monitoring & Scheduling is a DevOps tool designed specifically for web scraping. It allows you to monitor your scraper performance in real-time, track historical statistics, receive automated alerts, and schedule and manage scraping jobs across multiple servers, all from a single dashboard.
02What web scraping frameworks are supported?
We currently have SDKs for Python Scrapy and Python Requests, with more integrations on the way. The SDK integration takes just a few lines of code to set up and immediately starts logging your scraper performance data.
03What metrics does ScrapeOps track?
ScrapeOps automatically tracks pages scraped, items extracted, runtime, status codes, success rates, error rates, request latency, and more. You can compare current job runs against historical averages to quickly identify issues.
04How do the health checks and alerts work?
Out of the box, ScrapeOps automatically checks every job’s health against its historical moving average. If something looks off (like a significant drop in items scraped or spike in errors) you get alerted via email, Slack, or webhook. You can also configure custom checks and alert thresholds.
05Can I schedule scraping jobs with ScrapeOps?
Yes! With the job scheduling feature, you can connect your servers and GitHub repos to ScrapeOps, then deploy, schedule, run, pause, and re-run your scraping jobs from our dashboard. It supports cron-style scheduling and manual triggers.
06Is there a free plan?
Yes! The Community Plan is free forever and lets you monitor unlimited scrapers and pages. It includes basic monitoring, built-in health checks, and email alerts. Premium plans unlock advanced features like custom alerts, job scheduling, and extended data retention.
07How long does integration take?
Integration takes about 30 seconds. For Scrapy, it’s just 3 lines added to your settings.py file. For Python Requests, you wrap your requests with our SDK. No code changes to your scraping logic are required.
08Can I manage scrapers running on multiple servers?
Yes! ScrapeOps allows you to connect multiple servers via SSH and manage all your scraping jobs from one central dashboard, regardless of where they’re running. This is available on Business and Enterprise plans.