Skip to main content

ScrapeOps API Documentation

The ScrapeOps API allows you to manage your servers, spiders, and scheduled jobs. You can create, retrieve, update, and delete resources through a series of endpoints. Authentication is required for all endpoints using an API key.

Base URL​

https://backend.scrapeops.io/v1/client

Authentication​

To use the ScrapeOps API, you first need an API key which you can get by signing up for a free account here.

Your API key must be included with every request the X-API-KEY header otherwise the API will return a 403 Forbidden Access status code.

All requests must include the X-API-KEY header:

X-API-KEY: {{x-api-key}}

General Notes​

  1. Optional Parameters:
    • settings, args, and commandToRun are optional parameters for applicable endpoints.
  2. Integration:
    • These APIs currently support Scrapy VM integrations only.

Endpoints​

1. List Servers​

Retrieve a list of all servers in your ScrapeOps account.

Request

  • Method: GET
  • URL: https://backend.scrapeops.io/v1/client/servers

Response

  • A list of servers.

Description
Use this endpoint to retrieve all servers associated with your account.

2. List Spiders by Server ID​

Retrieve a list of spiders for a specific server.

Request

  • Method: GET
  • URL: https://backend.scrapeops.io/v1/client/servers/{serverId}/spiders
  • Path Parameter:
    • serverId (integer): ID of the server.

Description
Retrieve all spiders belonging to a specific server. Use the List Servers endpoint to find the serverId.

3. List All Scheduled Jobs​

Retrieve all scheduled jobs.

Request

  • Method: GET
  • URL: https://backend.scrapeops.io/v1/client/scheduled-jobs

Description
Lists all scheduled jobs in your account.

4. List Scheduled Jobs by Server ID​

Retrieve all scheduled jobs for a specific server.

Request

  • Method: GET
  • URL: https://backend.scrapeops.io/v1/client/servers/{serverId}/scheduled-jobs
  • Path Parameter:
    • serverId (integer): ID of the server.

Description
Lists all scheduled jobs associated with a specific server.

5. Create a Scheduled Job​

Schedule a new job for a specific server and spider.

Request

  • Method: POST
  • URL: https://backend.scrapeops.io/v1/client/scheduled-jobs
  • Body: 
    {
      "serverId": 2,
      "serverSpiderId": 1,
      "cronToken": "* * * * *",
      "settings": [
        {
          "key": "lang",
          "value": "python"
        }
      ],
      "args": [
        {
          "key": "lang",
          "value": "python"
        }
      ],
      "commandToRun": ""
    }
    Optional Fields:
    • settings, args, and commandToRun.

Description
Create a scheduled job with optional settings, arguments, and commands.

6. Delete Scheduled Job​

Delete an existing scheduled job.

Request

  • Method: DELETE
  • URL: https://backend.scrapeops.io/v1/client/scheduled-jobs/{jobId}
  • Path Parameter:
    • jobId (integer): ID of the scheduled job.

Description
Deletes the scheduled job with the specified ID.

7. Run a Spider​

Run a spider on a specific server.

Request

  • Method: POST
  • URL: https://backend.scrapeops.io/v1/client/spiders/run
  • Body: 
    {
      "serverId": 2,
      "selectedSpiderId": 1,
      "settings": [
        {
          "key": "test",
          "value": "test"
        }
      ],
      "args": [
        {
          "key": "test",
          "value": "test"
        }
      ],
      "commandToRun": ""
    }
    Optional Fields:
    • settings, args, and commandToRun.

Description
Executes a spider on a given server with optional settings, arguments, and commands.

Postman Collection​

To help you quickly test and integrate with the ScrapeOps API, we provide a comprehensive Postman collection containing pre-configured requests for all available endpoints.

Download & Setup​

  1. Download the Postman collection: Postman Collection

  2. Import the collection into Postman:

    • Open Postman
    • Click "Import" in the top left
    • Drag and drop the downloaded JSON file or browse to select it
    • Click "Import" to complete

  3. Configure your environment:

    • Create a new environment in Postman
    • Add the following variables: 
      host: https://backend.scrapeops.io/v1/client
      x-api-key: YOUR_API_KEY
    • Select your environment from the environment dropdown

What's Included​

The postman collection contains ready-to-use requests for all API endpoints:

  • List Servers
  • List Spiders by Server ID
  • List All Scheduled Jobs
  • List Scheduled Jobs by Server ID
  • Create Scheduled Jobs
  • Delete Scheduled Jobs
  • Run Spiders

Each request includes:

  • Proper HTTP method and endpoint configuration
  • Required headers and authentication
  • Example request bodies where applicable
  • Path parameter examples
  • Helpful descriptions and usage notes

Error Handling​

Ensure proper error handling for responses with status codes other than 200 OK. The API will return appropriate HTTP status codes and error messages in the response body.