Web Scraping in 2026: Six Insights From Pierluigi of The Web Scraping Club
ScrapeOps Blog10 min read
What AI is improving, why reliable collection remains difficult, and how customer demand is changing, drawn from our AMA with Pierluigi Vinciguerra.
Scraping has never been more accessible.
AI writes the first version of the code. Scraping APIs handle extraction.
Browser agents click through the interactive parts.
But a scraper that runs once isn't a data pipeline. The harder question is what changes when you need complete, correct data every day.
As The Web Scraping Club's founder Pierluigi Vinciguerra put it during our recent Reddit AMA:
What become harder is the antibot bypass, since they evolved from a couple of years to now.
The tooling, he added, has become easier. There are far more tools, and AI makes them much easier to plug into a pipeline.
Pierluigi has worked in scraping for at least 12 years. He started when he founded his first web data company.
Five years ago he launched The Web Scraping Club newsletter to share notes with other professionals. Today he is co-founder and CTO of Data Boutique, a marketplace for web-scraped data.
That gives him three views at once. He builds scrapers, covers the industry, and sees what customers actually buy.
In our fourteenth r/WebScrapingInsider AMA, we asked Pierluigi what has become easier and what is still hard. We also asked how costs are shifting, and what that means for people selling datasets.
Here are the six biggest insights from the discussion.
1. Building Scrapers Is Getting Easier. Reliability Still Takes Engineering.
One redditor asked what has become easier in the last two years, and what has become harder. A lot of people have the same question.
Pierluigi's answer was uneven progress. Tooling and integration have improved a lot, but so have anti-bot defenses.
That makes sense once you split a scraping project into stages:
- Getting the first scraper working.
- Keeping access to the site over time.
- Handling changes to the site.
- Checking that the dataset is actually what you needed.
AI clearly speeds up the first two. It doesn't automatically handle the last two.
Pierluigi was explicit about where reliability comes from:
Keep in mind that reliability of data pipeline is made by data quality controls and best practices
He also separated routine fixes from the hardest targets. On heavily protected websites, he said, the work is closer to R&D than maintenance.
None of this means every pipeline is getting harder to run. Progress just varies across the workflow.
A scraper you built in an afternoon still needs the same checks as one that took a week.
2. Choose the Scraping Approach Around the Workload
Several redditors asked some version of "which tool should I use?"
Pierluigi kept answering with another question: what does the workload look like?
His main split is between two jobs. One is a few records from many sources.
The other is a whole website, collected again and again.
For broad scraping, think of a news aggregator: many websites, a few pages from each. He said this used to need a scraper per source, and scraping APIs now remove most of that work.
For heavy scraping, he still recommends dedicated scrapers. In Python, his standard stack is Scrapy, curl_cffi and a browser solution for harder websites.
Here AI helps write and maintain the code. It doesn't extract every record.
Using an AI tool to collect a full retailer catalogue, in his words, "would be too expensive."
Browser agents are a third category:
They’re different use cases: browser agent for interaction with websites, storing user’s sessions, small scraping tasks. Traditional scraping for massive data gathering, for creating larger datasets
Put together, the options look like this:
| Workload | Approach Pierluigi describes |
|---|---|
| Many websites, a few records from each | Scraping APIs and AI-assisted extraction |
| Large, repeated collections | Dedicated scraping pipelines |
| Interactive tasks involving sessions | Browser agents |
| An existing dataset meets the requirement | Consider buying the collected data |
His examples mentioned runs of 500,000+ rows. That shows scale, not a universal break-even point.
Where your line sits depends on your sources, your refresh rate and what each option costs.
3. AI Maintenance Starts With Documented Procedures
Asked about his team's AI stack, Pierluigi didn't start with a model or a tool. He started with onboarding:
I'm treating my AI stack as a newly hired dev.
He has written his best practices, standards, workflows and tool docs into skills. So when a scraper breaks, the AI already knows the order to investigate.
First, is it an IP issue? Then, is it a selector issue, or does the site need better tooling?
The preparation is a big part of the story. He said the setup took a couple of months.
The workflows had to become skills and procedures. The goal was an AI that behaves almost deterministically on issues.
He thinks it was worth it. For standard websites, he estimates maintenance now averages a couple of minutes per scraper.
That figure spreads the hour a fix takes across the months when nothing breaks.
He was also careful about the label "self-healing":
Well, not really self-heal; I want to be in the loop to understand what's going on, but that's my choice.
It doesn't cover everything either. The top 10% of websites with the hardest bot protection, such as Chinese e-commerce sites, still need manual work.
He didn't share a full architecture. He also didn't describe fully autonomous repair in production.
What he did describe can become a starting checklist. This is our reading of his approach, not his spec:
- Document the failure types you see repeatedly.
- Define the diagnostic steps and which tool to use for each.
- Specify how repaired output gets checked.
- Decide which changes need a human to review them.
4. A Successful Run Can Still Miss Most of the Data
A repair can get a scraper running again. It doesn't tell you whether the whole dataset came back.
A QA-minded redditor pointed out that a run can be green while the output is wrong. Pierluigi agreed, and named the hardest version of the problem:
The most complex challenge is understanding if the extraction is complete or not. Maybe you didn’t get any error but you scraped just 2000 items instead of 10000.
Every row in that run can look perfect. The problem is the pages and categories that never showed up.
He said the data type matters. Text and unstructured data are hard to judge, while numbers are easier because you can set thresholds.
For e-commerce, the checks he described are:
- Totals. Compare extracted counts with the site's category or brand totals.
- History. Compare the current run with past runs and similar extractions.
- Outliers. Flag prices that look too high or too low.
- Ground truth. Use manual checks to estimate how many items to expect.
Pagination is where coverage often goes missing. Asked what scraping API providers should improve, he pointed to the ones he uses for some Data Boutique projects:
When scraping a product catalog, most of them break when going from page 1 to page 2, because there are a lot of edge cases on these websites.
These checks have limits. Not every site shows totals, and he gave no universal alert threshold.
Historical comparisons tell you when to look closer. They don't prove a run is correct.
When his team finds bad data, he said, they delete it rather than keep it in the trend.
5. Lower Engineering Effort Doesn't Guarantee a Lower Scraping Bill
Proxy prices have dropped, and there are more scraping tools than ever. So has scraping got cheaper?
I didn't see the invoices going down. They just moved.
More websites are protecting themselves. So teams use more proxies and more tools.
When pushed on total cost, he split it into two sides:
The engineering time, I think, is going down for most of the websites with no complex anti-bot solutions, but again, the bills for tools are going up instead.
There's a second effect. Cheaper tasks don't mean a smaller bill if you run more of them.
Pierluigi noted that AI tools make one-off and non-critical scrapes worth doing. Nobody would have written code for those before.
Was the hard part getting through, or doing it reliably at a sensible cost? He chose the second: "the hard part is finding the tradeoff between reliability and costs."
The AMA has no controlled comparison of the same dataset over time. So this isn't proof that scraping costs more overall, just an experienced operator saying the spend has moved.
6. Easier Extraction Is Changing What Customers Buy
Those costs shape whether customers build, use a service or buy finished data. At Data Boutique, Pierluigi has watched the mix change.
Take a few hundred Nike products from Zalando. AI tools can mostly handle that now, so buyers rarely ask for it.
The full Zalando site is around a million records. That needs a working scraper, so you either build it or buy the result.
So I've seen a shift from more targeted requests to full-scope websites.
He reported slightly growing demand for full-site datasets. Demand for small, targeted scrapes is falling.
Many Data Boutique buyers lack the time or skills to build scrapers. Even with AI, he said, they wouldn't write the code.
For sellers, his first rule was about demand:
Just don't start if you don't have a validation from the market. At Databoutique, we have buyers who can post requests for websites they can't find on the platform, so answering these will lead to a sure sale.
The other route is listing data you already collect for paying customers. If you scrape Zalando for a client, the same dataset can become a second revenue stream.
Ideally, the price comes in below what a buyer would spend building the scraper. He admitted that doesn't always happen.
Extraction isn't always where the value ends. The hardest e-commerce problem he described was matching products across retailers:
Matching the same product from different sites is definitely the most painful process, especially in industries where the original product code is not shown.
In fashion, his team built an internal tool and hired experts to match by hand, with AI assisting. He said AI still can't do it alone.
That's one example of specialist work beyond collection. It isn't proof that matching is a moat for every business.
Conclusion: Easier Scraping Moves the Work, It Doesn't Remove It
As first-time collection gets easier, the effort moves. It goes to the parts AI doesn't solve on its own.
Pierluigi's answers point to five priorities:
- Match the approach to the workload.
- Turn maintenance knowledge into documented procedures.
- Check completeness, not just record accuracy.
- Measure the total cost of delivery, not the cost of one tool.
- Validate demand before building a dataset business.
AI can cut the work of building and maintaining scrapers. Dependable, useful data still takes careful operational and commercial decisions.
Asked for tips on maintaining 100+ sources, Pierluigi summarized:
Standardize, Document, Best practices. And then create all the skills about processes and your documentation, so that an AI could help you.
To see Pierluigi's answers in context, read the full AMA thread on r/WebScrapingInsider.
Then check out ScrapeOps, the complete toolkit for web scraping.