The
Python
Web Scraping Playbook

In Part 1 of the series, we go over the basics of how to build a scraper with Python using Python Requests & BeautifulSoup.

In Part 2 of the series, we're going to show you how to make your scraper more robust and reliable.

In Part 3 of the series, we'll explore several different ways we can store the data and talk about their pros, and cons and in which situations you would use them.

In Part 4 of the series, we make our scraper more robust and scalable by handling failed requests and using concurrency.

In Part 5 of the series, we make our scraper production ready by using fake user agents & browser headers to make our scrapers look more like real users.

In this video, we'll take you through how to scrape Amazon using Python Requests and BeautifulSoup.

In this video, we'll take you through how to scrape g2 using Python Requests and BeautifulSoup.

In this guide, we walk through how you should set up your Python Request scrapers to avoid getting blocked, retrying failed requests and scaling up with concurrency.

In this guide, we walk through how to send POST requests with Python Requests. Including how to POST form data and JSON data.

In this guide, we walk through how to use proxies with Python Requests. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python Requests scrapers.

In this guide, we walk through how to use fake user-agents with Python Requests to prevent your scrapers from getting blocked.

In this guide, we walk through how to configure Python Requests to retry failed requests so you can build a more reliable system.

In this guide, we walk through how to configure Python Requests to make concurrent requests so that you can increase the speed of your scrapers.

In this guide we show you how to use Python Pyppeteer, the Puppeteer library for Python, to render and scrape Javascript heavy websites.

In this guide, we compare the 5 best Python HTML parsing libraries available in 2023 - BeautifulSoup, lxml, html5lib, requests-html, and pyquery.