The
Python
Web Scraping Playbook

In Part 1 of the series, we go over the basics of how to build a scraper with Python using Python Requests & BeautifulSoup.

In Part 2 of the series, we're going to show you how to make your scraper more robust and reliable.

In Part 3 of the series, we'll explore several different ways we can store the data and talk about their pros, and cons and in which situations you would use them.

In Part 4 of the series, we make our scraper more robust and scalable by handling failed requests and using concurrency.

In Part 5 of the series, we make our scraper production ready by using fake user agents & browser headers to make our scrapers look more like real users.

In Part 6 of the series, we'll explore how to use proxies to bypass anti-bot systems by hiding your real IP address and location.

Should you use Python Selenium or Python Requests? In this guide we compare both options and tell you when you should use each.

Should you use Python Selenium or Python Scrapy? In this guide we compare both options and tell you when you should use each.

Should you use Python Scrapy or Python Pyppeteer? In this guide we compare both options and tell you when you should use each.

Should you use Python Selenium or NodeJS Playwright? In this guide we compare both options and tell you when you should use each.

In this guide we show you how to create and manage fake user agents when scraping in Python so you don't get blocked.

In this guide, we use the Python CloudScraper library to scrape Cloudflare protected websites.

In this guide we show you how to setup and use a FlareSolverr server to bypass Cloudflare when scraping.

In this guide we show you how to use Python Pyppeteer, the Puppeteer library for Python, to render and scrape Javascript heavy websites.

In this guide we will go through the best javascript rendering libraries for Scrapy so you can scrape modern websites with ease.

In this guide we show you how to use Scrapy Playwright to render and scrape Javascript heavy websites.

In this guide we show you how to setup and use Scrapy Splash in your Spider to extract JS rendered data from webpages.

In this guide we show you how to setup and use Scrapy Selenium in your Spider to extract JS rendered data from webpages.

In this guide, we compare the 5 best Python HTML parsing libraries available in 2023 - BeautifulSoup, lxml, html5lib, requests-html, and pyquery.

In this guide, we walk through the Python Newspaper3k library and how to use it to scrape & curate articles.

In this guide, we walk through the Python FeedParser library and how to parse RSS, Atom & RDF feeds.

In this guide, we walk through how you should set up your Python Request scrapers to avoid getting blocked, retrying failed requests and scaling up with concurrency.

In this guide, we walk through how to send POST requests with Python Requests. Including how to POST form data and JSON data.

In this guide, we walk through how to use proxies with Python Requests. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python Requests scrapers.

In this guide, we walk through how to use fake user-agents with Python Requests to prevent your scrapers from getting blocked.

In this guide, we walk through how to configure Python Requests to retry failed requests so you can build a more reliable system.

In this guide, we walk through how to configure Python Requests to make concurrent requests so that you can increase the speed of your scrapers.

In this guide, we walk through how you should set up your Python hRequests scrapers to avoid getting blocked, retrying failed requests and scaling up with concurrency.

In this guide, we walk through how to send POST requests with Python hRequests. Including how to POST form data and JSON data.

In this guide, we walk through how to use fake user-agents with Python hRequests to prevent your scrapers from getting blocked.

In this guide, we walk through how to configure Python hRequests to retry failed requests so you can build a more reliable system.

In this guide, we walk through how to configure Python hRequests to make concurrent requests so that you can increase the speed of your scrapers.

In this guide, we walk through how to use proxies with Python HTTPX. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python HTTPX scrapers.

In this guide, we walk through how to use fake user-agents with Python HTTPX to prevent your scrapers from getting blocked.

In this guide, we walk through how to send POST requests with Python HTTPX. Including how to POST form data and JSON data.

In this guide, we walk through how to configure Python HTTPX to retry failed requests so you can build a more reliable system.

In this guide, we walk through how to use proxies with Python aiohttp. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python aiohttp scrapers.

In this guide, we walk through how to send POST requests with Python aiohttp. Including how to POST form data and JSON data.

In this guide, we walk through how to use cURL in Python using PycURL. Including how to make GET, POST requests, use proxies, user-agents, and more.

In this guide, we walk through how to use BeautifulSoup to scrape data from HTML websites and files.

In this guide, we walk through how to install and use BeautifulSoup on Windows, MacOS, and Linux machines.

In this guide, we walk through how to use BeautifulSoup's find_all() method to find the first page element by class, id, text, regex, and more.

In this guide, we walk through how to use BeautifulSoup's find_all() method to find a list of page elements by class, id, text, regex, and more.

In this guide, we walk through how to fix your code when BeautifulSoup returns an empty list or value.

In this guide, we walk through how to use BeautifulSoup to remove HTML tags like span, script, etc. from HTML files.

In this guide, we walk through how to use Parsel to scrape data from HTML websites and files.