IPRoyal Datacenter Proxies: Web Scraping Guide

• TLDR: How To Integrate IPRoyal Datacenter Proxy
• Understanding Datacenter Proxies
• Why Use Datacenter Proxies?
• Datacenter Proxy Pricing
• Setting Up IPRoyal Datacenter Proxies
• Authentication
• Basic Request Using IPRoyal Datacenter Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing IPRoyal Datacenter Proxies
• Case Study: Scraping The Guardian
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

TLDR: How to Integrate IPRoyal Datacenter Proxy?

1. We import random along with our typical proxy setup.
2. We read iproyal-proxies.txt into an array.
3. Next, we use random.choice() to select a random proxy from the list as our proxy_url.
4. Then, like other proxy integrations, we assign our proxy_url to both the http and https protocols of our proxies dict.

import requestsimport random
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(result.text)

• First, we create an empty array.
• Next, we use Python's file functionality to read iproyal-proxies.txt and load each line of the file into our array.
• proxy_url = f"http://{random.choice(proxy_list)}" choose a random proxy from the list to use as our proxy_url.
• We create a dict object to hold our http and https proxies. We then assign both of them to our proxy_url.
• We then print result.text to our terminal, revealing the location information of our proxy.

Understanding Datacenter Proxies

Datacenter vs Residential

Datacenter

• Price: No matter which provider you're choosing, datacenter proxies tend to be very affordable. When using residential proxies, it's not uncommon to pay up to $8/GB.
• Speed: When you use this type of proxy, it gets hosted inside an actual datacenter. Datacenters are often equipped with some of the best internet connections and hardware. Datacenter proxies offer unparalleled performance.
• Availability: Datacenters are huge. Each machine in a datacenter normally gets its own IP address. This can give us a seemingly endless proxy pool to work with.

• Blocking: As touched on briefly, some webites block all datacenter IPs by default. This can can make some websites far more difficult to scrape.
• Less Geotargeting Support: We usually do get geotargeting support with datacenter proxies. However, you still show up with a datacenter IP... no matter which country you choose. With datacenter proxies, our locations are limited.
• Less Anonymity: With datacenter proxies, our location always shows up inside of a datacenter. In comparison to a residential proxy, this can really stick out.

Residential

• Anonymity: We get a real IP address assigned to a real residential device. When dealing with more difficult sites, this allows us to blend in more easily.
• Better Access: When using a residential IP address, it doesn't matter if your target site blocks datacenter IP addresses. Your traffic is coming from somebody's real device in a real home.

• Price: Residential proxies are expensive. It's not uncommon to pay between $5 and $8 per GB! Datacenter proxies, on the other hand, can be 10x cheaper!
• Speed: Datacenter proxies use top of the line hardware and internet connections. When using a residential proxy, you're far more likely to get a low-end device on a subpar internet connection. This can drastically reduce the performance of our scraper.

Why Use IPRoyal Datacenter Proxies?

• IPRoyal's Datacenter Proxies are quite affordable.
• When setting up our proxies, we get to choose their location.

IPRoyal Datacenter Proxy Pricing

• On their highest price plan, you do actually buy proxies at $1.57 per IP.
• On their lowest tier, you're actually paying $1.80 per IP address.

Setting Up IPRoyal Datacenter Proxies

Authentication

import requestsimport random
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(result.text)

• All of our variables (username, password, hostname, port`) are already baked into the url inside of our text file.
• Because of this, there is no need to write any code for authentication. IPRoyal already took care of that for us.
• We use a dict to hold both our http and https proxies. We set each one to our proxy_url.
• Finally, make a request to the Lumtest API for our location information and print it to the terminal.

Basic Request Using IPRoyal Datacenter Proxies

import requestsimport random
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(result.text)

Country Geotargeting

import requestsimport randomfrom urllib.parse import urlencode
API_KEY = "your-scrapeops-api-key"SCRAPEOPS_LOCATION = "us"
def get_scrapeops_url(url, location=SCRAPEOPS_LOCATION):    payload = {        "api_key": API_KEY,        "url": url,        "country": location    }    proxy_url = "https://proxy.scrapeops.io/v1/?" + urlencode(payload)    return proxy_url
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"
proxies = {    "http": proxy_url,    "https": proxy_url}
print("-------------------------UK-----------------------")result = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(result.text)
print("-------------------------US------------------------")result = requests.get(get_scrapeops_url("http://lumtest.com/myip.json"))print(result.text)

• Our ScrapeOps API_KEY and SCRAPEOPS_LOCATION. This API_KEY gets used for authentication with ScrapeOps. SCRAPEOPS_LOCATION is used to tell ScrapeOps where we'd like to appear.
• We write a function called get_scrapeops_url(). This function takes in our API_KEY, location, and target url. It then uses urlencode to wrap all of this information into a proxied url that we can ping using Requests (or anything for that matter).
• We check our location information with IPRoyal just like we did before.
• After our IPRoyal check, we go through and connect to the same API endpoint using our ScrapeOps proxy.
• We print the location information from each request to the console.

• Using our British proxy from IPRoyal, our location shows up in GB with a timezone of Europe/London.
• With our ScrapeOps proxy in the US, our location shows up as US with a timezome of America/Chicago.

City Geotargeting

• Local Ads
• Local Businesses
• Local Social Media
• Local Events

Error Codes

KYC Verification

Implementing IPRoyal Datacenter Proxies in Web Scraping

Python Requests

import requestsimport random
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(result.text)

• Our credentials are already inside each url from iproyal-proxies.txt.
• We read load proxy_list from our file and use random.choice() to select a random proxy.
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriverimport random
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"

## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We build our url exactly how we did above with Python Requests: http://{random.choice(proxy_list)}. This takes a random choice from our proxy_list.
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options.

Python Scrapy

scrapy startproject datacenter

import scrapyimport osimport random
proxy_list = []
file_path = os.path.join(os.path.dirname(__file__), "iproyal-proxies.txt")

with open(file_path) as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"

class ExampleSpider(scrapy.Spider):    name = "datacenter_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

scrapy crawl datacenter_proxy

• First, we use os.path.join() to ensure that our spider can find the path to the file.
• Once again, we create the same proxy_url: http://{random.choice(proxy_list)}. This allows us to select a random proxy from the list.
• From inside start_requests, we assign our proxy_url to request.meta['proxy']. This tells Scrapy that all of this spider's requests are to be made through the proxy_url we created earlier.

NodeJS Puppeteer

mkdir puppeteer-datacenter

cd puppeteer-datacenternpm init --y

npm install puppeteer

const puppeteer = require("puppeteer");const fs = require("fs");
const proxyFileData = fs.readFileSync("iproyal-proxies.txt", "utf8");const proxyArray = proxyFileData.split('\n').map(line => line.trim()).filter(line => line.length > 0);const proxyUrl = proxyArray[Math.floor(Math.random() * proxyArray.length)];const proxyHost = proxyUrl.split("@");const creds = proxyHost[0].split(":");

(async () => {  const browser = await puppeteer.launch({    args: [`--proxy-server=http://${proxyHost[proxyHost.length-1]}`]  });
  const page = await browser.newPage();
  await page.authenticate({    username: creds[0],    password: creds[1]  });
  await page.goto('http://lumtest.com/myip.json');  await page.screenshot({path: 'puppeteer.png'});
  await browser.close();})();

• First, we read our proxy file: fs.readFileSync("iproyal-proxies.txt", "utf8").
• Next, we use string splitting to separate each line read from the file.
• proxyArray[Math.floor(Math.random() * proxyArray.length)] chooses a random proxy from our list.
• We then use some more string splitting to separate our hostname and credentials from the full proxy string.
• We also add our password to the password field for authentication: password: password.
  • http://${proxyHost[proxyHost.length-1]} yields our server.
  • proxyHost[0].split(":") gives our proxy credentials.
  • We set our username with: username: creds[0].
  • Our password is set with: password: creds[1].

NodeJS Playwright

mkdir playwright-datacenter

cd playwright-datacenternpm init --y

npm install playwrightnpx playwright install

const playwright = require("playwright");const fs = require("fs");
const proxyFileData = fs.readFileSync("iproyal-proxies.txt", "utf8");const proxyArray = proxyFileData.split('\n').map(line => line.trim()).filter(line => line.length > 0);const proxyUrl = proxyArray[Math.floor(Math.random() * proxyArray.length)];const proxyHost = proxyUrl.split("@");const creds = proxyHost[0].split(":");

const options = {    proxy: {        server: `http://${proxyHost[proxyHost.length-1]}`,        username: creds[0],        password: creds[1]    }};
(async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
    await page.goto('http://lumtest.com/myip.json');
    await page.screenshot({ path: "playwright.png" })
    await browser.close();})();

• Like our Puppeteer example, we first setup our configuration variables. This process requires a bit of extra code to split up our proxyUrl.
• We create a proxy object with the following fields:
• server: `http://${proxyHost[proxyHost.length-1]}
• username: creds[0]
• password: creds[1]

Case Study: Scrape The Guardian

1. We use our IPRoyal proxies to setup a connection in the UK.
2. We use the ScrapeOps Proxy Aggregator to setup a connection based in the US.
3. With each of these proxies, we make a GET request to the Lumtest API and then we make one to The Guardian.
4. We print our results to the terminal for comparison.

import requestsimport randomfrom urllib.parse import urlencodefrom bs4 import BeautifulSoup
API_KEY = "your-scrapeops-api-key"LOCATION = "us"
def get_scrapeops_url(url, location=LOCATION):    payload = {        "api_key": API_KEY,        "url": url,        "country": location    }    proxy_url = "https://proxy.scrapeops.io/v1/?" + urlencode(payload)    return proxy_url
proxy_list = []
with open("iproyal-proxies.txt") as file:    proxy_list = file.read().splitlines()
proxy_url = f"http://{random.choice(proxy_list)}"
proxies = {    "http": proxy_url,    "https": proxy_url}


print("----------------------US---------------------")

location_info = requests.get(get_scrapeops_url("http://lumtest.com/myip.json"))print(location_info.text)
response = requests.get(get_scrapeops_url("https://www.theguardian.com/"))
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")print(subnav.text)

print("----------------------UK---------------------")
location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get("https://www.theguardian.com/", proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")
print(subnav.text)

• With each proxy connection, we make our request to the Lumtest API is used specifically to verify our proxy connection. It is not the end result of our code.
• On each run, we use soup.select_one("div[data-testid='sub-nav']") to find and return the navbar on the page.

• ScrapeOps gives us access to a REST API. This allows us to wrap all of our parameters into a proxied url.
• Since we're using a REST API, we don't need to setup a proxy connection manually using Requests. Instead, we just need to wrap our url properly.

• us: USUS elections 2024WorldEnvironmentUkraineSoccerBusinessTechScienceNewslettersWellness
• gb: UKWorldClimate crisisUkraineFootballNewslettersBusinessEnvironmentUK politicsEducationSocietyScienceTechGlobal developmentObituaries

• us: US | US elections 2024 | World | Environment | Ukraine | Soccer | Business | Tech | Science | Newsletters | Wellness
• gb: UK | World | Climate crisis | Ukraine | Football | Newsletters | Business | Environment | UK politics | Education | Society | Science | Tech | Global development | Obituaries

Alternative: ScrapeOps Proxy Aggregator

• JavaScript Rendering
• Screenshot Capability
• Country Geotargeting
• Residential and Mobile Proxies
• Anti-bot Bypass
• Custom Headers
• Sticky Sessions

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"LOCATION = "us"
def get_scrapeops_url(url, location=LOCATION):    payload = {        "api_key": API_KEY,        "url": url,        "country": location    }    proxy_url = "https://proxy.scrapeops.io/v1/?" + urlencode(payload)    return proxy_url
response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json"))print(response.text)

• Create our configuration variables: API_KEY and LOCATION.
• Write a get_scrapeops_url() function. This function takes all of our parameters along with a target url and wraps it into a ScrapeOps Proxied url. This is an incredibly easy way to scrape and it makes our proxy code much more modular.
• Check our IP info with response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json")).
• Finally, we print it to the terminal. You should get an output similar to this.

{"country":"US","asn":{"asnum":31898,"org_name":"ORACLE-BMC-31898"},"geo":{"city":"San Jose","region":"CA","region_name":"California","postal_code":"95119","latitude":37.2379,"longitude":-121.7946,"tz":"America/Los_Angeles","lum_city":"sanjose","lum_region":"ca"}}

Ethical Considerations and Legal Guidelines

Legal

• Don't use residential proxies to access illegal content: These actions can come with intense legal penalties that can even include prison or jail time depending on severity.
• Don't scrape and disseminate other people's private data: Depending on what your jurisdiction, this is also a highly illegal and dangerous practice. Doxxing private data can also lead to heavy fines and possibly jail/prison time.

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement.

Conclusion

More Cool Articles

• How To Bypass Anti-Bots
• Bright Data Residential Proxies