The 7 Best Web Scraping SubReddits
Reddit is a treasure trove of great resources and smart developers willing to help a fellow web scraper out.
You just need to know where to find them.
In this guide we're going to share with you the complete list of subreddits that every serious web scraper should be a part of.
Need help scraping the web?
Then check out ScrapeOps, the complete toolkit for web scraping.
r/WebScraping
Number 1 on this list is r/WebScraping the largest subreddit dedicated to everything web scraping. With over 9,000 members and a pretty active community it is a great place to keep up to date on the latest news in web scraping and get help if you run into trouble with a particular website.
You can find resources/advice on anything from:
- How to scrape specific websites?
- How to bypass specific anti-bots?
- What are the best web scraping proxy providers?
- How to use particular web scraping libraries?
If you can't find a answer to your question then, just ask your own and someone in the community will help you out.
r/Scrapy
Next on our list is r/Scrapy, a subreddit dedicated to everything Python Scrapy. The leading web scraping framework for Python.
The r/Scrapy is the offical community center for the Scrapy framework, and the best place to share cool articles, spiders, Scrapy extensions and whatnots.
A number of the original creators and current maintainers of Scrapy are actively daily on the subreddit so if you have a Scrapy specific issue then the Scrapy subreddit is a great place to go.
The moderators are very open to members asking for help, advice and code reviews so it is a great place to go if you need any help with:
- Getting started with Scrapy
- Using some of Scrapy's less well-known functionality
- Customising Scrapy for your specific use cases.
- Picking Scrapy extensions or web scraping stacks.
r/Selenium
Whilst, not dedicated to web scraping, given how popular Selenium is amongst the web scraping community as a headless browser the r/Selenium subreddit is a great community to be a part of if you use Selenium as part of your web scraping stack.
r/Selenium is a large and active subreddit, dedicated to helping you to use Selenium in your automation and web scraping stacks.
In this subreddit, you can get help with Selenium specific setup, configuration and scaling issues from a experienced and helpful community.
r/Puppeteer
Like the Puppeteer subreddit, r/Puppeteer is a small but extremely helpful subreddit dedicated to helping you setup, configure and use Puppeteer in your automation and web scraping stacks.
The r/Puppeteer community are great for helping you with topics like:
- Getting started with Puppeteer.
- Debugging Puppeteer specific issues.
- How to deploy your Puppeteer headless browsers to production.
- Efficient scaling strategies for your Puppeteer headless browsers.
r/ProxiesTalk
Although it is a small subreddit, r/ProxiesTalk can be a great place to find good proxies that work but which won't cost you an arm and a leg.
Oftentimes, when we look for web scraping proxies we end up at the same places Bright Data, Oxylabs, Scrapingbee, ScraperAPI. However, there are hundreds of other proxy providers out there that give similar performance but at a fraction of the cost.
r/ProxiesTalk is a great place to find them. So if you need a particular type of proxy (SOCKS, IPV$, IPV6, etc.) and are open to looking at smaller players then be sure to check out r/ProxiesTalk.
r/ProxyLists
Another subreddit dedicated to web scraping proxies is r/ProxyLists. On the surface it is dedicated to sharing free proxy lists you can use with your scrapers, however, proxy companies have kinda hijacked it and started spamming it with their own promotional content.
You can still find some good free proxy lists there, but you do have to wade through a good bit of spammy posts.
r/ShoeBots
Although not dedicated to web scraping, the r/ShoeBots subreddit can be a good source of info if you are primarily focused on scraping e-commerce sites.
The primary discussion topic in r/ShoeBots is helping people get into sneaker botting and finding the best bots for specific website. Sometimes there is good discussions about general web scraping topics (about proxies and selecting proxies for specific websites).
Websites that are frequent targets of sneaker botting are often the most heavily protected with anti-bot countermeasures, so the sneaker bot community have a very good idea which proxies are the best for particular websites.
Honorable Mentions
There are many more subreddits that regularly mention web scraping topics like:
If Discord is more your thing then be sure to check out our list of the best web scraping discord communities.
If you would like to learn more about web scraping in general, then be sure to check out The Web Scraping Playbook. Or check out some our other popular articles like: