Skip to main content

Residential Proxies Explained: How You Can Scrape Without Getting Blocked

Not too long ago, when you wanted to scrape a website you would go online, find a proxy provider and get yourself some data center proxies to use with your web scraper and everything would be great. The data would come back and you could continue with your life.

Unfortunately, web scraping today isn't as simple. Websites and anti-scraping technologies have gotten more sophisticated and are increasinly able to identify and ban suspisious requests from web scrapers.

This is lead to an explosive growth in demand for residential proxies, a type of proxy that tilts the balance of power back in favour of the web scraper and makes it much harder for websites to stop you accessing the public data on their site.

So that begs the question, what are residential proxies and what makes them so powerful?

If you prefer to follow along with a video then check out the video tutorial version here:

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

What Are Residential Proxies?​

Residential proxies are proxies that route their traffic through someones home or office internet router, not a datacenter. Making it look like the request is coming from someones house or local business.

Residential proxies differ from data center proxies, in the fact the underlying IP address is own by a Internet Service Provider (ISPs) and has been assigned to a specific device, such as a computer, mobile phone or tablet.

When to Use Residential Proxies?​

Residential proxies have a lot of benefits over datacenter proxies that make them a great option in certain situations. But they also have some drawbacks, namely speed and cost.

Residential proxies are significantly more expensive than data center proxies (upwards of 10X more expensive) and often have higher latencies too.

As a result, residential proxies shouldn't be your go-to proxy option, instead, you should focus on using them if you encounter one of the following situations:

Getting Blocked & Anti-Bots​

The number 1 reason why you should consider using residential proxies over data center proxies is when your data center proxies are continuously getting blocked by the website you are trying to scrape.

Businesses are increasingly starting to use more and more sophisticated anti-scraping technologies like Cloudflare, DataDome, and PerimeterX to detect and block people scraping their website.

These anti-scraping technologies use a number of request & behaviour profiling techniques to detect and ban requests from certain IPs.

The advantage of using residential proxies is that it makes it much harder for these anti-scraping technologies to determine if a request is coming from a scraper or an ordinary user. So the success rates are typically much higher when compared to data center proxies.

Country & City Level Geotargeting​

Another advantage of using residential proxies, is that they are great when you need to make your requests look like they are coming from a specific country, city, or even postcode, something that is very hard to achieve with data center proxies.

Because residential proxies are real IP addresses in peoples homes and businesses, they give a huge diversity in the places they are. As a result, some proxy providers like BrightData have been able to index all their residential IPs to specific countries and cities, and allow their users to decide from where their requests come from.

Bots & Automation​

Residential proxies also really have an edge in the area of automated bots that simulate the behaviour of real users to login to websites and take actions. Be it buy a Sneaker from a eCommerce store during a sale, or automate a Facebook or Instagram account.

Requests from static residential proxies are much harder to detect than data center proxies when the bot is logging into the website, so they are a great choice for these use cases.

How To Use Residential Proxies​

Residential proxies are very easy to use. Most providers give you access to their residential proxy pools through a single endpoint proxy, and where they manage the selection and management of those proxies on their end.

Meaning you just need to send your requests to a proxy endpoint like BrightData's:


http://USERNAME:PASSWORD@zproxy.lum-superproxy.io:22225

And they will route your request through their residential proxy network for you, and return the HTTP response after the request has been completed.

Integrating this proxy endpoint into your web scrapers is very easy, as it normally is just a parameter you add to the request. No need to worry about rotating proxies or managing bans, etc.

Here is a simple example using Python:


import requests

proxies = {
   'http': 'http://zproxy.lum-superproxy.io:22225',
   'https': 'http://zproxy.lum-superproxy.io:22225',
}

url = 'http://example.com/'

response = requests.get(url, proxies=proxies, auth=('USERNAME', 'PASSWORD'))


Most other web scraping tools like ScrapeBox, Apify, PhantomBuster, etc. all support this type of proxy integration out of the box making it a very simple setup.

Where Do Residential Proxies Come From?​

Where proxy providers get their residential proxies can be a bit opaque.

Some proxy providers treat it as a closely held secret, whilst others are very open about they get their proxies.

In truth there are 5 main ways proxy providers get their residential proxies:

  • Free VPNs
  • Proxy SDKs In Apps & Chrome Extensions
  • Paid Bandwidth Products
  • Device Proxy Farms
  • Proxy Reselling

If would like to learn more about this topic then we have written a whole guide on Where Do Residential Proxies Come From?.

The Legality Of Residential Proxies​

When proxy providers first started offering residential proxies to their users, there were legitimate questions over how did they acquire access to these residential IP addresses and do they have permission from the owners to use them.

Thankfully, in the last few years things have significantly moved out of the shadows and the proxy provider industry have become much more transparent about how they get their residential proxies and how they have explicit permission to use them.

Sites like Honeygrain and PacketStream pay users to let them use their IP address as a residential proxy and then they sell access to this proxy network to web scrapers and other proxy providers.

More Web Scraping Guides​

In this guide, we went through what are residential proxies, why you should use them and how you can get started using them.

If you would like to learn more about web scraping in general, then be sure to check out The Web Scraping Playbook, or check out one of our more in-depth guides: