ScrapeOps
ScrapeOps Playbook · 15 guidesPython Web Scraping Playbook: Video Tutorials
Video tutorials for the Python web scraping playbook, from the first request through parsing, browsers, and avoiding bans.
Python For Beginners Series
Create Your First Python Spider - Python Requests/BS4 Beginners Series [Part 1]
In Part 1 of the series, we go over the basics of how to build a scraper with Python using Python Requests & BeautifulSoup.
Cleaning Dirty Data & Dealing With Edge Cases - Python Requests/BS4 Beginners Series [Part 2]
In Part 2 of the series, we're going to show you how to make your scraper more robust and reliable.
Storing Data in AWS S3, MySQL & Postgres DBs - Python Requests/BS4 Beginners Series [Part 3]
In Part 3 of the series, we'll explore several different ways we can store the data and talk about their pros, and cons and in which situations you would use them.
Managing Retries & Concurrency - Python Requests/BS4 Beginners Series [Part 4]
In Part 4 of the series, we make our scraper more robust and scalable by handling failed requests and using concurrency.
Faking User-Agents & Browser Headers - Python Requests/BS4 Beginners Series [Part 5]
In Part 5 of the series, we make our scraper production ready by using fake user agents & browser headers to make our scrapers look more like real users.
How To Scrape with Python Requests Series
How to Scrape Amazon With Python Requests and BeautifulSoup
In this video, we'll take you through how to scrape Amazon using Python Requests and BeautifulSoup.
How to Scrape g2 With Python Requests and BeautifulSoup
In this video, we'll take you through how to scrape g2 using Python Requests and BeautifulSoup.
Python Requests
Python Requests: Web Scraping Guide
In this guide, we walk through how you should set up your Python Request scrapers to avoid getting blocked, retrying failed requests and scaling up with concurrency.
Python Requests: How to Send POST Requests
In this guide, we walk through how to send POST requests with Python Requests. Including how to POST form data and JSON data.
Python Requests: How to Use & Rotate Proxies
In this guide, we walk through how to use proxies with Python Requests. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python Requests scrapers.
Python Requests: Setting Fake User-Agents
In this guide, we walk through how to use fake user-agents with Python Requests to prevent your scrapers from getting blocked.
Python Requests: Retry Failed Requests
In this guide, we walk through how to configure Python Requests to retry failed requests so you can build a more reliable system.
Python Requests: Make Concurrent Requests
In this guide, we walk through how to configure Python Requests to make concurrent requests so that you can increase the speed of your scrapers.
