Skip to main content

2 posts tagged with "anti-bot"

View All Tags

· 15 min read

The Hardest Part of a Browser Agent Isn't Browsing. It's Knowing It Worked.

From the AMA

An agent can click every button correctly and still fail the task. This AMA is about what sits underneath that: persistent identity, stealth that never holds still, and the harder problem of verifying a run actually did what it was supposed to.

Browser automation used to have a simple job. Give a program a browser, define the clicks, keep the selectors alive when the site changes.

Browser agents are making that job description obsolete. Current models can write browser code, recover from their own mistakes, and chain together workflows that used to take real engineering hours.

None of that turns out to be the hard part.

The hard part is what happens after the agent acts. Did it keep the identity it started with? Did the IP change under it mid-task? Did the run actually finish the job, or did it just perform a sequence of actions that looked right?

That split came up again and again in a Reddit AMA with Alexander Yue, who works on evaluations at Browser Use.

As Browser Use's Alexander Yue put it during our recent Reddit AMA:

Yue works on evaluations at Browser Use and on physics and CS research at Stanford SLAC. Browser Use's open source harness has passed 107k GitHub stars, and its newer v4 platform runs agents on hosted, persistent browser sessions instead of throwaway headless instances. He was joined in the thread by Reagan Hsu, Browser Use's founding engineer, though the questions below were answered by Yue.

In our fourth r/WebScrapingInsider AMA, we asked Yue how much of the browser an agent should actually control, what identity survives between runs, why stealth resists a single score, and how anyone proves a run did the job.

Here are the six biggest insights from the discussion.

· 14 min read

Scraping Shock - Why Web Data Is Getting Too Expensive to Scrape

Something's breaking in web scraping.

Success rates are slipping. Costs are spiralling. Teams are struggling to keep up.

The proxies are cheaper. The infrastructure is more sophisticated.

But the math no longer works.

Proxies that once cost $30 per GB now go for $1.

Yet the cost of a successful scrape, one clean, validated payload, has doubled or tripled or 10X.

Web scraping hasn't gotten harder because of access.

It's gotten harder because of economics.

Retries, JS rendering, and anti-bot bypasses now consume more budget than bandwidth.

Every website is still technically scrapable, but fewer make financial sense to scrape at scale.

The barrier isn't access anymore. It's affordability.

This is Scraping Shock, the moment cheap access collides with expensive success.

In this article, we will dive deep into the most important trend affecting web scraping today: