ScrapeOps
ScrapeOps Playbook · 53 guidesPython Web Scraping Guides and Tutorials
Python scraping guides for Requests, HTML parsers, headless browsers, proxies, user-agents, logins, and choosing a library for the job.
Guides by topic
Browse the Playbook
01
Intro to Python
2 guides
02
Comparison of Python Libraries for Web Scraping
4 guides
Python Selenium VS Python Requests ComparedShould you use Python Selenium or Python Requests? In this guide we compare both options and tell you when you should use each.→Python Scrapy VS Python Selenium ComparedShould you use Python Selenium or Python Scrapy? In this guide we compare both options and tell you when you should use each.→ Python Scrapy vs Python Pyppeteer ComparedShould you use Python Scrapy or Python Pyppeteer? In this guide we compare both options and tell you when you should use each.→ Python Selenium vs NodeJS Playwright ComparedShould you use Python Selenium or NodeJS Playwright? In this guide we compare both options and tell you when you should use each.→
03
Proxies, User-Agents & Avoiding Bans
5 guides
Python Fake User-Agents: How to Manage User Agents When ScrapingIn this guide we show you how to create and manage fake user agents when scraping in Python so you don't get blocked.→Python CloudScraper: Scrape Cloudflare Protected WebsitesIn this guide, we use the Python CloudScraper library to scrape Cloudflare protected websites.→FlareSolverr Guide: Bypassing Cloudflare Made SimpleIn this guide we show you how to setup and use a FlareSolverr server to bypass Cloudflare when scraping.→How To Bypass Anti-Bots With PythonIn this guide, we'll explore different methods to bypass anti-bot measures using Python.→How To Solve CAPTCHAs with PythonIn this guide, we will explore various strategies to programmatically solve CAPTCHAs using Python.→
05
Python Headless Browsers & Javascript Rendering
6 guides
The Python Pyppeteer Guide - Use Puppeteer With PythonIn this guide we show you how to use Python Pyppeteer, the Puppeteer library for Python, to render and scrape Javascript heavy websites.→Scrapy Javascript Rendering: The 4 Best Scrapy Libraries to Scrape JS Heavy WebsitesIn this guide we will go through the best javascript rendering libraries for Scrapy so you can scrape modern websites with ease.→Scrapy Playwright Guide: Render & Scrape JS Heavy WebsitesIn this guide we show you how to use Scrapy Playwright to render and scrape Javascript heavy websites.→Scrapy Splash Guide: A JS Rendering Service For Web ScrapingIn this guide we show you how to setup and use Scrapy Splash in your Spider to extract JS rendered data from webpages.→Scrapy Selenium Guide: Integrating Selenium Into Your Scrapy SpidersIn this guide we show you how to setup and use Scrapy Selenium in your Spider to extract JS rendered data from webpages.→The Best Python Headless Browsers For Web Scraping in 2024 In this article, we'll walk through the process of infinite scrolling with Selenium, and we'll attempt to scrape an infinite scroller with plain old Requests and also the ScrapeOps Headless Browser.→
06
HTML Parser Libraries
3 guides
The 5 Best Python HTML Parsing Libraries ComparedIn this guide, we compare the 5 best Python HTML parsing libraries available in 2023 - BeautifulSoup, lxml, html5lib, requests-html, and pyquery.→Newspaper3k Guide: Scrape Articles Using AIIn this guide, we walk through the Python Newspaper3k library and how to use it to scrape & curate articles.→FeedParser Guide: Parse RSS, Atom & RDF Feeds With PythonIn this guide, we walk through the Python FeedParser library and how to parse RSS, Atom & RDF feeds.→
07
Python Requests
7 guides
Python Requests: Web Scraping GuideIn this guide, we walk through how you should set up your Python Request scrapers to avoid getting blocked, retrying failed requests and scaling up with concurrency.→Python Requests: How to Send POST RequestsIn this guide, we walk through how to send POST requests with Python Requests. Including how to POST form data and JSON data.→Python Requests: How to Use & Rotate ProxiesIn this guide, we walk through how to use proxies with Python Requests. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python Requests scrapers.→Python Requests: Setting Fake User-AgentsIn this guide, we walk through how to use fake user-agents with Python Requests to prevent your scrapers from getting blocked.→Python Requests: Retry Failed RequestsIn this guide, we walk through how to configure Python Requests to retry failed requests so you can build a more reliable system.→Python Requests: Make Concurrent RequestsIn this guide, we walk through how to configure Python Requests to make concurrent requests so that you can increase the speed of your scrapers.→Python Requests: Fix SSL Errors in Python RequestsExplore alternative methods for bypassing the SSL errors and understanding its root cause.→
08
Python hRequests
5 guides
Python hRequests: Web Scraping GuideIn this guide, we walk through how you should set up your Python hRequests scrapers to avoid getting blocked, retrying failed requests and scaling up with concurrency.→Python hRequests: How to Send POST RequestsIn this guide, we walk through how to send POST requests with Python hRequests. Including how to POST form data and JSON data.→Python hRequests: Setting Fake User-AgentsIn this guide, we walk through how to use fake user-agents with Python hRequests to prevent your scrapers from getting blocked.→Python hRequests: Retry Failed RequestsIn this guide, we walk through how to configure Python hRequests to retry failed requests so you can build a more reliable system.→Python hRequests: Make Concurrent RequestsIn this guide, we walk through how to configure Python hRequests to make concurrent requests so that you can increase the speed of your scrapers.→
09
Python HTTPX
4 guides
Python HTTPX: How to Use & Rotate ProxiesIn this guide, we walk through how to use proxies with Python HTTPX. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python HTTPX scrapers.→Python HTTPX: Setting Fake User-AgentsIn this guide, we walk through how to use fake user-agents with Python HTTPX to prevent your scrapers from getting blocked.→Python HTTPX: How to Send POST RequestsIn this guide, we walk through how to send POST requests with Python HTTPX. Including how to POST form data and JSON data.→Python HTTPX: Retry Failed RequestsIn this guide, we walk through how to configure Python HTTPX to retry failed requests so you can build a more reliable system.→
10
Python aiohttp
2 guides
Python aiohttp - How to Use & Rotate ProxiesIn this guide, we walk through how to use proxies with Python aiohttp. Including how to rotate proxies in a list, use proxy gateways, and proxy APIs with our Python aiohttp scrapers.→Python aiohttp: How to Send POST RequestsIn this guide, we walk through how to send POST requests with Python aiohttp. Including how to POST form data and JSON data.→
11
Python PycURL
1 guide
12
Python BeautifulSoup
6 guides
BeautifulSoup Guide: Scraping HTML Pages With PythonIn this guide, we walk through how to use BeautifulSoup to scrape data from HTML websites and files.→How To Install BeautifulSoupIn this guide, we walk through how to install and use BeautifulSoup on Windows, MacOS, and Linux machines.→How To Use BeautifulSoup's find() MethodIn this guide, we walk through how to use BeautifulSoup's find_all() method to find the first page element by class, id, text, regex, and more.→How To Use BeautifulSoup's find_all() MethodIn this guide, we walk through how to use BeautifulSoup's find_all() method to find a list of page elements by class, id, text, regex, and more.→Fix BeautifulSoup Returns Empty List or ValueIn this guide, we walk through how to fix your code when BeautifulSoup returns an empty list or value.→How To Eliminate Span & Other HTML Tags With BeautifulSoupIn this guide, we walk through how to use BeautifulSoup to remove HTML tags like span, script, etc. from HTML files.→
13
Python Parsel
1 guide
14
More Guides
5 guides
Python aiohttp - Setting Fake User-AgentsIn this guide, we walk through how to use fake user-agents with Python aiohttp to prevent your scrapers from getting blocked.→How To Download Images With PythonIn this guide, we will explore various methods to download images using Python, offering detailed steps and examples for each approach.→How To Submit Forms With Python RequestsIn this guide, we will dive into various approaches for submitting forms using Python and provide detailed steps and examples for each method.→Python Scrapy vs Requests with Beautiful Soup ComparedIn this article, we're going to explore two popular Python libraries—Scrapy and Requests with Beautiful Soup—and review their strengths, weaknesses, and use cases.→Python Selenium vs Python Pyppeteer ComparedShould you use Python Selenium or Python Pyppeteer? In this guide we compare both options and tell you when you should use each.→
Need an easy way to monitor your scrapers?
Sign up for a free ScrapeOps account today.