anyIP Residential Proxies: Web Scraping Guide

anyIP is one of the more reputable options when it comes to residential proxy providers. They boast over 100,000,000 IP addresses, a 98.% success rate, and a response time of 0.6 seconds.

In today's article, we're going to go through the process of getting setup and using anyIP from start to finish.

• TLDR: How to Integrate anyIP With Python
• Understanding Residential Proxies
• Why Use anyIP?
• anyIP Pricing
• Setting Up With anyIP
• Authentication
• Basic Requests
• Country Geotargeting
• City Geotargeting
• Static Proxies
• Error Codes
• KYC Verification
• Implementing anyIP Proxy Ports
• Case Study: Amazon.es
• Alternative: ScrapeOps Residential Proxies
• Legal and Ethical Considerations
• Conclusion
• More Web Scraping Guides

TLDR: How to Integrate anyIP Residential Proxy?​

To use anyIP:

1. You need to create an account.
2. Then, you need to retrieve your username, password and ip_address_and_port.
3. Once you've got those, save them to a config.json file.

{
    "anyIP": {
        "username": "your-anyIP-username,type_residential",
        "password": "your-anyIP-password",
        "ip_address_and_port": "your-domain-and-port-number"
    }
}

Once you've got it setup, you can test your connection with the code below.

import requests
import json
from bs4 import BeautifulSoup
from urllib.parse import urlencode

username = ""
password = ""
ip_address_and_port = ""

with open("config.json", "r") as config_file:
    config = json.load(config_file)
    username = config["anyIP"]["username"]
    password = config["anyIP"]["password"]
    ip_address_and_port = config["anyIP"]["ip_address_and_port"]


proxies = {
    "http": f"http://{username}:{password}@{ip_address_and_port}",
    "https": f"http://{username}:{password}@{ip_address_and_port}",
}

if __name__ == "__main__":

    url = "https://api.ipify.org?format=json"

    response = requests.get(url)

    ip = response.json()["ip"]

    print("real ip address:", ip)

    proxied_response = requests.get(url, proxies=proxies)

    proxied_ip = proxied_response.json()["ip"]

    print("proxy ip address:", proxied_ip)

The code above is meant for testing.

• First, it reads your config file to hook up to the proxy port connection.
• Then it uses the ipify API to check your address without the proxy, and rechecks it using the proxy.
• Both IP addresses get printed to your terminal.
• If the IP addresses are different, your proxy connection is working correctly.

Understanding Residential Proxies​

Residential proxies are a staple in the proxy service industry. Sometimes referred to as premium proxies, when you use a service like ScrapeOps, an initial try usually goes through a datacenter IP. If the request fails using the datacenter IP, your service will typically fall back on a residential proxy to retrieve the data.

It's a common practice for some servers to ban datacenter IP addresses. To avoid the retry logic completely (whether in your scraper or from your proxy service), it can be a great idea to just integrate with a residential proxy service.

Types of Residential Proxies​

When using residential proxies, there are two main types: Rotating Proxies and Static Proxies.

Rotating Residential Proxies​

When using Rotating Proxies, your proxy service will rotate between multiple IP addresses so your requests are all coming from different places.

Pros:

• Anonymity & Security: The frequent change of IP addresses makes it harder to trace or ban your activity.
• Ideal for web scraping: Since the IP changes regularly, websites have difficulty detecting and blocking your requests, making them great for large-scale scraping.
• Avoiding bans: Constant rotation makes it easier to avoid being flagged or banned by target websites.

Cons:

• Consistency: If you need to keep the same IP for a session (e.g., logging into the same account), rotating proxies can cause issues with session persistence.
• Speed fluctuations: With frequent IP changes, the speed and reliability of the connection might vary, depending on the network quality of the new IP.

Static Residential Proxies​

With Static Proxies, you continue to resuse the same IP address.

Pros:

• Consistency: Since the IP address remains constant, they are excellent for applications requiring session-based usage, like managing social media accounts or logging into specific services.
• Lower chance of flagging: Since these IPs belong to real residential devices, they're often less likely to be flagged as proxies by websites.
• Reliable connection: Static proxies can offer more stable speeds compared to rotating ones.

Cons:

• Risk of IP bans: With a static IP, if your activity is flagged or blocked by a website, that single IP will be banned, making it unusable.
• Less anonymity: The lack of frequent IP changes means your browsing habits can be tracked more easily if the proxy is detected.

Rotating Proxies are best for avoiding anti-bot mechanisms, Static Proxies are best for authenticated sessions (such as logging into a website).

Residential vs. Data Center Proxies​

Residential proxies use real residential IP addresses for higher trust and less likelihood of detection but come at a higher cost and slower speeds, while data center proxies offer faster, cheaper, and more scalable options but are more easily detected and blocked by websites.

Here's a comparison to help you make an informed decision:

Use Cases​

Residential proxies are beneficial in various scenarios, including:

• Web scraping on websites with strong anti-bot measures, as residential IPs are less likely to be detected and blocked.
• Accessing geo-restricted content, allowing users to bypass location-based restrictions on streaming platforms, websites, and services.
• Managing multiple social media accounts without getting flagged for suspicious activity, as they provide real residential IPs.
• E-commerce monitoring and price tracking, where residential proxies help avoid detection by platforms that block data center proxies.
• Ad verification, ensuring that digital ads are displayed correctly across different locations.

Why Use anyIP Residential Proxies?​

Using anyIP for residential proxies is a great decision for reliability and bypassing anti-bots.

anyIP offers rotating proxies with over 100,000,000 IP addresses and a 98.6% success rate.

With anyIP, we get the option to choose between using a residential or a mobile IP address. By default, their service uses both. This gets us our data as quickly and reliably as possible.

With anyIP, we can also use sessions which allow us to keep a session in tact for up to 7 days!

We also get the ability to finely control our geolocation and internet provider.

anyIP Residential Proxy Pricing​

With anyIP, we get a number of different plans we can choose.

If you look at the options below, plans start as low as $25/month go as high as $1000/month. You should choose a plan based on your usage.

You can view a detailed breakdown of the pricing and bandwidth in the table below.

Comparison to Other Providers​

anyIP's pricing tends to be on the higher end of the spectrum compared to other residential proxy providers.

• Cheap Providers: Providers offering plans around $2-3 per GB are considered cheaper.
• Expensive Providers: Those with smaller plans priced in the $6-8 per GB range are more expensive.

For a comprehensive comparison of residential proxy providers, including pricing and features, you can refer to the ScrapeOps Proxy Comparison page.

Setting Up anyIP Residential Proxies​

Getting started with anyIP is pretty easy.

1. You can create an account with an email and password, or just simply create an account with Google.
2. Once you've made an account, you'll be taken to their dashboard.

The dashboard is simple, yet intuitive. Navigation is straightforward and it's easy to find everything.

Then, you need to buy a plan. We're going to try out their micro plan.

If you click the option to pay with a credit card, you get a popup similar to the one below. It's pretty intuitive and modern. The payment processor seems legitimate.

If you choose to pay with cryptocurrency, you'll be redirected to NOWPayments which gives you the option to pay in over 70 different cryptocurrencies.

Depending on your choice of blockchain, speed of payment may vary from just a few seconds all the way up to 30 minutes. This is not anyIP's fault and it's pretty normal for a crypto gateway.

Here is a screenshot of their NOWPayments page.

Authentication​

Authentication with anyIP is pretty straightforward. We need our username, password, and ip_address_and_port.

If you scroll down on the dashboard page to the Team Members section, you should see your account.

Next to your account there should be a button titled, use. Click this button.

After clicking this button, you'll receive a popup with your private information near the bottom. You can even get a sample cURL request.

In the username section, you should see something like ,session_randSession{some-random-number} at the end of the url. Feel free to remove this from your username. This is only necessary if you plan to use a Static Proxy (Sticky Session).

For best practice, setup a config.json file with the following information.

{
  "anyIP": {
        "username": "{your-username},type_residential",
        "password": "{your-password}",
        "ip_address_and_port": "portal.anyip.io:{your-port-number}"
    }
}

Basic Request Using anyIP Residential Proxies​

To make requests, we need to setup a proxy port URL. They are constructed with the variables we saved in the config file: username, password, ip_address_and_port.

The full proxy URL will look like this:

http://{username}:{password}@{ip_address_and_port}

Take a look at the code example below to get started.

import requests
import json
from bs4 import BeautifulSoup
from urllib.parse import urlencode

username = ""
password = ""
ip_address_and_port = ""

with open("config.json", "r") as config_file:
    config = json.load(config_file)
    username = config["anyIP"]["username"]
    password = config["anyIP"]["password"]
    ip_address_and_port = config["anyIP"]["ip_address_and_port"]


proxies = {
    "http": f"http://{username}:{password}@{ip_address_and_port}",
    "https": f"http://{username}:{password}@{ip_address_and_port}",
}

if __name__ == "__main__":

    url = "https://api.ipify.org?format=json"

    response = requests.get(url)

    ip = response.json()["ip"]

    print("real ip address:", ip)

    proxied_response = requests.get(url, proxies=proxies)

    proxied_ip = proxied_response.json()["ip"]

    print("proxy ip address:", proxied_ip)

In the example above, we:

• Create a dict to hold our http and https proxy settings.
• Save our url as a variable, https://api.ipify.org?format=json. This API call will return our IP address.
• We check our IP both with and without the proxy and print them to the console so we can compare them.

Here are the results from the code.

Here is the location of our proxy IP address. As you can see, our location is showing up in Vietnam. The proxy is working!

Country Geotargeting​

When you're scraping the web, country level geotargeting is a must. Depending on your location, different websites will give you different results!

We don't want to scrape products and get our prices in dollars for some results and euros for others.

Top 10 Countries Supported by anyIP Residential Proxies​

AnyIP residential proxies support a wide range of geolocations for users.

This document lists the top 10 markets using AnyIP residential proxies, along with the number of IPs available in each region.

To add country geotargeting, we just need to adjust our URL a little bit. We need to add ,country_{COUNTRY_CODE} to our proxy URL.

The full URL looks like this:

You can view their full list of countries and their codes here.

Here is our full code.

import requests
import json
from bs4 import BeautifulSoup
from urllib.parse import urlencode

username = ""
password = ""
ip_address_and_port = ""

with open("config.json", "r") as config_file:
    config = json.load(config_file)
    username = config["anyIP"]["username"]
    password = config["anyIP"]["password"]
    ip_address_and_port = config["anyIP"]["ip_address_and_port"]



if __name__ == "__main__":

    COUNTRY_CODE = "US"

    proxies = {
        "http": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
        "https": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
    }

    url = "https://api.ipify.org?format=json"

    response = requests.get(url)

    ip = response.json()["ip"]

    print("real ip address:", ip)

    proxied_response = requests.get(url, proxies=proxies)

    proxied_ip = proxied_response.json()["ip"]

    print("proxy ip address:", proxied_ip)

• We move our proxies into main so that we can tweak them from inside the main block.
• We add a COUNTRY_CODE so the user can easily change their location.
• The COUNTRY_CODE is now included in our proxy url.

Here are is the terminal output.

Now, to verify our that our address is in the country we chose (US).

As you can see above, we chose US as a country code and our proxy location is in Texas. Everything is working!

City Geotargeting​

Some proxy providers give us the ability to narrow our location down to a specific city. anyIP is not one of those providers. Honestly, this really isn't a bad thing.

When it comes to city level geotargeting, you tend to lose some reliability. If you're trying to request a URL and it fails, it's much easier to switch to another IP that's in the same country.

It's quite a bit more difficult to switch to another IP address within the same city because the pool of addresses is so much smaller.

Error Codes​

anyIP does not give a list of status codes in their documentation.

Thanks to the people over at Mozilla, we can give you a list of common codes used in web development. This list doesn't include every status code you might run into, but it should cover the vast majority of common codes.

Most importantly, if you're receiving anything other than a 200, you should lookup what it means. 200 means that everything is working.

If you're receiving a code other than 200, there might be an issue with either your scraper or the server.

You can view the full list of standard codes from Mozilla here.

KYC (Know-Your-Customer) Verification​

To use anyIP, there isn't a strict KYC process. You can signup with Google, or simply with an email address and password.

Once you've got an account and you've chosen a plan, you're all good to go!

This is far more convenient than some other residential providers. Some providers even require a video call to speak with you before you can use their product.

Implementing anyIP Residential Proxies in Web Scraping​

To use anyIP with pretty much any service, you need to use a library or framework that supports proxy port integration.

You saw our proxy URL earlier when we used Python Requests.

In this section, we'll give you a little snippet with how to get started using various libraries.

Python Requests​

import requests
import json
from bs4 import BeautifulSoup
from urllib.parse import urlencode

username = ""
password = ""
ip_address_and_port = ""

with open("config.json", "r") as config_file:
    config = json.load(config_file)
    username = config["anyIP"]["username"]
    password = config["anyIP"]["password"]
    ip_address_and_port = config["anyIP"]["ip_address_and_port"]



if __name__ == "__main__":

    COUNTRY_CODE = "US"

    proxies = {
        "http": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
        "https": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
    }

    url = "https://api.ipify.org?format=json"

    response = requests.get(url)

    ip = response.json()["ip"]

    print("real ip address:", ip)

    proxied_response = requests.get(url, proxies=proxies)

    proxied_ip = proxied_response.json()["ip"]

    print("proxy ip address:", proxied_ip)

Python Scrapy​

import scrapy
import json

class IpifySpider(scrapy.Spider):
    name = "ipify_spider"

    def start_requests(self):
        with open("config.json", "r") as config_file:
            config = json.load(config_file)
            username = config["anyIP"]["username"]
            password = config["anyIP"]["password"]
            ip_address_and_port = config["anyIP"]["ip_address_and_port"]

        COUNTRY_CODE = "US"

        proxy = f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}"

        # Setting the proxy for the request
        meta = {
            "proxy": proxy
        }

        url = "https://api.ipify.org?format=json"

        yield scrapy.Request(url, callback=self.parse_ip, meta=meta)

    def parse_ip(self, response):
        data = response.json()
        print("Proxy IP Address:", data['ip'])

NodeJs Puppeteer​

const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
    const config = JSON.parse(fs.readFileSync('config.json', 'utf8'));
    const username = config.anyIP.username;
    const password = config.anyIP.password;
    const ipAddressAndPort = config.anyIP.ip_address_and_port;

    const COUNTRY_CODE = 'US';

    const proxyUrl = `http://${username},country_${COUNTRY_CODE}:${password}@${ipAddressAndPort}`;

    const browser = await puppeteer.launch({
        args: [`--proxy-server=${proxyUrl}`],
    });
    const page = await browser.newPage();

    await page.authenticate({
        username: username,
        password: password,
    });

    await page.goto('https://api.ipify.org?format=json');
    const ip = await page.evaluate(() => document.body.innerText);

    console.log('Proxy IP Address:', ip);

    await browser.close();
})();

NodeJs Playwright​

const { chromium } = require('playwright');
const fs = require('fs');

(async () => {
    const config = JSON.parse(fs.readFileSync('config.json', 'utf8'));
    const username = config.anyIP.username;
    const password = config.anyIP.password;
    const ipAddressAndPort = config.anyIP.ip_address_and_port;

    const COUNTRY_CODE = 'US';

    const proxyUrl = `http://${username},country_${COUNTRY_CODE}:${password}@${ipAddressAndPort}`;

    const browser = await chromium.launch({
        proxy: {
            server: `http://${ipAddressAndPort}`,
            username: username,
            password: password,
        }
    });
    const page = await browser.newPage();

    await page.goto('https://api.ipify.org?format=json');
    const ip = await page.evaluate(() => document.body.innerText);

    console.log('Proxy IP Address:', ip);

    await browser.close();
})();

Case Study - Scrape Amazon.es​

Here, we're going to do a small case study: scrape Amazon.es.

We'll scrape from two different proxy locations, Great Britain and Spain.

The code below first requests the page using a proxy code in Britain. Then, it requests the same page using a proxy in Spain.

We'll write both pages to files so we can open them up and review them in the browser.

Here is the code.

import requests
import json
from bs4 import BeautifulSoup
from urllib.parse import urlencode

username = ""
password = ""
ip_address_and_port = ""

with open("config.json", "r") as config_file:
    config = json.load(config_file)
    username = config["anyIP"]["username"]
    password = config["anyIP"]["password"]
    ip_address_and_port = config["anyIP"]["ip_address_and_port"]



if __name__ == "__main__":

    COUNTRY_CODE = "GB"

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
    }

    proxies = {
        "http": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
        "https": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
    }

    url = "https://www.amazon.es/s?k=portátil"

    response = requests.get(url, headers=headers, proxies=proxies)

    with open("british.html", "w") as file:
        file.write(response.text)


    COUNTRY_CODE = "ES"

    proxies = {
        "http": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
        "https": f"http://{username},country_{COUNTRY_CODE}:{password}@{ip_address_and_port}",
    }

    response = requests.get(url, headers=headers, proxies=proxies)
    with open("spanish.html", "w") as file:
        file.write(response.text)

Here are some screenshots of the page from a British location.

Next, you can look at some shots from our scrape using the Spanish proxy.

While the prices are pretty much the same. Look at the locations in the shots above. Amazon is detecting that we're in separate locations. The first is in the United Kingdom and the second is in Spain.

Alternative: ScrapeOps Residential Proxy Aggregator​

As opposed to signing up for anyIP, you can easily sign up for a free trial with the ScrapeOps Residential Proxy Aggregator. When you sign up, you get 500 MB for free!

On top of that, ScrapeOps offers similar pricing and a larger selection of plans than anyIP.

You can view the available ScrapeOps plans below.

The ScrapeOps Residential Proxy Aggregator is highly valuable for several reasons:

1. Access to Multiple Proxy Networks: It combines various proxy providers, offering access to millions of residential IPs for diverse scraping needs.

2. Cost Efficiency: Aggregating proxies from multiple providers lowers the overall cost compared to direct purchases from a single provider.

3. Simplified Management: It offers a centralized dashboard for seamless proxy management and switching between different networks.

4. Increased Success Rates: By rotating between multiple proxy pools, it reduces the chances of getting blocked, enhancing scraping success rates.

5. Geo-targeting & Session Management: It provides geo-targeting features and supports persistent sessions with static proxies for stable, localized scraping tasks.

Once you've signed up for your free trial, put your API key in a config file like we did earlier. You can get connected and test it out with the code example below.

Once again, it checks your proxied IP address against your actual IP address.

import requests
import json
from bs4 import BeautifulSoup
from urllib.parse import urlencode

API_KEY = ""
with open("config.json", "r") as config_file:
    config = json.load(config_file)
    API_KEY = config["scrapeops_api_key"]


proxies = {
    "http": f"http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181",
    "https": f"http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181",
}

if __name__ == "__main__":

    url = "https://api.ipify.org?format=json"

    response = requests.get(url)

    ip = response.json()["ip"]

    print("real ip address:", ip)

    proxied_response = requests.get(url, proxies=proxies)

    proxied_ip = proxied_response.json()["ip"]

    print("proxy ip address:", proxied_ip)

Ethical Considerations and Legal Guidelines​

Whenever we scrape the web (or even browse it for that matter), we need to do so with a conscience. In this tutorial, we scraped public data.

Public data is public information and therefore free to scrape. However, if you decide to scrape private data, you're subject to a completely different set of rules and regulations.

When we scrape, we also need to consider the policies of the site we're scraping.

While it might be legal, certain sites might not appreciate it when you scrape their data and the might even ban you!

Always be aware of your target site's Terms and Conditions and their robots.txt.

Conclusion​

In conclusion, residential proxies are a very important part of the proxy world.

When you hook up to a residential proxy port, you get the best possible IP addresses and you can just set it and forget it... simply pass proxies=proxies into Python Requests.

Take this new knowledge and go build something with it! You can now set a custom geolocation and bypass anti-bots!

More Python Web Scraping Guides​

Want to take your scraping skills to the next level?

Check out Python Web Scraping Playbook or take a look at the links below!

• Residential Proxies Explained
• The Crazy Economics of Residential and Mobile Proxies