Skip to main content

Server Management & Scheduling Overview

The following is an overview of how to integrate ScrapeOps with your servers so that you can deploy, manage and schedule your scrapers from the ScrapeOps dashboard.

There are two options to integrate ScrapeOps with your servers:

  1. Via SSH (Recommended)
  2. Via Scrapyd Server HTTP Endpoints (Only Applicable to Python Scrapy)

💻 Demo​

Login to the Demo Account and navigate to the Scheduler or Server Pages.

⭐ SSH Server Integration​

The SSH server integration allows you to deploy, schedule & run any type of scraper (Python, NodeJs, Scrapy, etc) on a server that is accessible via SSH. Which includes most major server & VM providers (Digital Ocean, Vultr, AWS, etc.).

The ScrapeOps server manager allows you to:

  • 🔗 Integrate With Any SSH Capably Server
  • 🕷 Deploy scrapers directly from GitHub to your servers.
  • ⏰ Schedule Periodic Jobs

To learn how to setup the SSH server integration follow this guides:

Get $100 Free Credits with Vultr using this link, and $100 Free Credits with Digital Ocean using this link.

⭐ Scrapyd Server Integration​

The Scrapyd server integration allows you to:

  • 🔗 Integrate With ScrapyD Servers
  • ⏰ Schedule Periodic Jobs
  • 💯 All Scrapyd JSON API Supported
  • 🔐 Secure Your ScrapyD with BasicAuth, HTTPS or Whitelisted IPs

To learn how to setup the Scrapyd integration follow this guide.