Skip to main content

Claude Code Plugin

The ScrapeOps Claude Code Plugin brings the AI Scraper Builder directly into Claude Code. Instead of switching to the dashboard to generate or fix a scraper, you do it in conversation with Claude. Describe what you want, answer a couple of questions, and the finished scraper lands as a file in your project, ready to run.

⭐ What You Can Do​

The plugin adds four slash commands to Claude Code:

CommandWhat it does
/generate-scraperBuild a single-URL scraper. You give Claude one or more URLs from the same site; it returns a runnable script that takes a URL as a CLI argument and writes results to a JSONL file.
/generate-crawler-scraperBuild a crawler + scraper pair from a natural-language description (e.g. "Amazon, searching for men's t-shirts"). The crawler discovers product URLs (with pagination), the scraper extracts the details, and they hand off via a JSONL file. Supports plain scripts and full Scrapy projects.
/fix-scraperFix or extend an existing local scraper. Point Claude at a parser file and tell it what's wrong (empty field, broken selector, missing data, new field to add). It diagnoses, edits, re-runs, and validates autonomously. Works for parser issues and crawler/pagination issues.
/scrapeops-setupSave or update your ScrapeOps API key. Most users never need to run this manually. Claude will ask for the key the first time it's needed.

🚀 Getting Started​

Prerequisites​

Install​

git clone https://github.com/ScrapeOps/scrapeops-scraping-assistant-claude-plugin.git
cd scrapeops-scraping-assistant-claude-plugin
./setup.sh

Restart Claude Code after the script finishes. That's it. Open any project and run /generate-scraper.

The first time a command needs your API key, Claude will ask for it and save it locally to ~/.claude/settings.json. You don't need to run any extra setup step.

Update​

./setup.sh --uninstall && git pull origin main && ./setup.sh

Your saved API key is preserved.

💡 Typical Workflows​

Generate a scraper for a single page​

You:    /generate-scraper
Claude: Which language do you want the scraper in?
You:    Python
Claude: Which Python library?
You:    BeautifulSoup
Claude: Paste 1–5 URLs from the target site.
You:    https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
        [10–15 minute generation]
Claude: Done. Saved books_toscrape_com_scraper.py + README.
        Run it with: python books_toscrape_com_scraper.py "<url>"

The generated script accepts a URL as a CLI argument, fetches HTML through the ScrapeOps proxy, and writes one JSON line per page to an output file.

Build a crawler + scraper for a search​

You:    /generate-crawler-scraper Amazon, searching for "men's t-shirts", 3 pages
Claude: [discovers the search URL, generates crawler.py and scraper.py]
        Done. Run them in order:
          python crawler.py    → writes product_urls.jsonl
          python scraper.py    → reads product_urls.jsonl, writes products.jsonl

Crawler and scraper are kept as separate files by default so you can re-run either one independently. A README is generated with the exact run commands.

Fix a broken field​

You:    /fix-scraper the price field is coming back empty
Claude: Found books_toscrape_com_scraper.py and page_1.html.
        [diagnoses, edits selectors, re-runs, validates]
Claude: Done. Price extracts correctly now.

You can also ask Claude to add a new field, change which selector is used, or fix pagination that's missing pages: same command, different task description.

If you don't have an HTML file locally, just provide a URL and Claude will fetch a fresh copy through the ScrapeOps proxy before fixing.

⚙ Tips​

  • Multiple URLs help accuracy. For /generate-scraper, providing 2–5 URLs from the same domain produces a more robust parser than a single URL.
  • Pick the right library for the page. Static HTML → BeautifulSoup / Cheerio / equivalent. JavaScript-heavy SPA → Playwright / Selenium / Puppeteer. Claude will warn you if the choice doesn't match what it sees on the page.
  • Fix vs. regenerate. Use /fix-scraper when one or two fields break. Re-run /generate-scraper if the site has been redesigned end-to-end.
  • Same quota as the dashboard. Generations consumed by the plugin count against your account's plan limit, just like generations from the web app.
  • Check or update your key. Run /scrapeops-setup any time to view or replace your API key.

🔒 Privacy & Security​

  • Your API key is stored locally in ~/.claude/settings.json and never leaves your machine except when authenticating with the ScrapeOps backend.
  • Page content is sent to the ScrapeOps backend for AI processing, the same data path as the web dashboard.
  • Generated scraper code runs entirely on your machine.

Resources​