2 posts tagged with "reverse engineering"

Akamai, DataDome, Incapsula and Kasada each ship JavaScript that inspects the visiting client, encodes what it finds and sends it back to be judged. Much of that code is heavily obfuscated, and some of it runs inside a custom virtual machine designed to keep it unreadable.

Reverse engineering one of these systems means working out what the script collects, how it encodes it, and which values the server actually checks. Then the vendor ships a new build, and on some systems that happens daily.

Patrick van den Bosch and his team have been working through that cycle since 2020. He is the founder of Hyper Solutions, which builds request-based solutions for all four vendors. During our recent Reddit AMA, he described where his team's time really goes:

>

Most of the time is usually spent testing, understanding how the antibot updates and how to maintain it at scale.

The Hyper Solutions team started out as sneaker developers. According to the company, its API now handles more than 2.5 billion requests a month.

In our thirteenth r/WebScrapingInsider AMA, we asked Patrick how he approaches an unfamiliar protection, why bypasses that work in testing fail later, where AI helps, and how to judge whether a bypass is worth the cost.

Here are the nine biggest lessons from the discussion.

Your scraper gets blocked. So you swap the proxies.

Still blocked. You add a stealth plugin, then a new browser, then a CAPTCHA solver.

Each change feels like progress. But each one is a guess, and some of those guesses make things worse. A scraper that rotates its fingerprint every few requests can get blocked faster than one that changes nothing.

The engineers who get consistent results start somewhere else. Before they change anything, they work out what actually failed, and ask one question: does this change address the actual cause?

As freelance anti-bot engineer Ibrahim El Khalil Mlata put it during our recent Reddit AMA:

>

I treat it like a decision tree, isolate variables one by one, most people jump straight to replacing proxies, but in my experience its often the fingerprint or the scraping behavior

Ibrahim is a web scraping and anti-bot engineer based in Algeria who has spent the last seven years reverse engineering anti-bot defenses across retail pricing, legaltech, logistics, hospitality and AI-training-data pipelines. Among the work he has done: running 1,000+ spiders across 300 retailers, reviving a dead 160-spider fleet, matching 100K+ hotel reviews a month, and taking mobile apps apart with Frida when the website is locked down. You can find his work on GitHub.

In our eleventh r/WebScrapingInsider AMA, we asked Ibrahim how he diagnoses blocks, when reverse engineering is worth the effort, and what keeps a scraping system useful once it is in production.

Here are the nine biggest insights from the discussion.