Oxylabs Datacenter Proxies: Web Scraping Guide

• TLDR: How To Integrate Oxylabs Datacenter Proxy
• Understanding Datacenter Proxies
• Why Use Datacenter Proxies?
• Datacenter Proxy Pricing
• Setting Up Oxylabs Datacenter Proxies
• Authentication
• Basic Request Using Oxylabs Datacenter Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing Oxylabs Datacenter Proxies
• Case Study: Scraping The Guardian
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

TLDR: How to Integrate Oxylabs Datacenter Proxy?

1. Once you've created your proxies, just make sure you've got your credentials: USERNAME, PASSWORD, PORT.
2. If you direct your requests to port 8000, Oxylabs will even manage your proxies for you!
3. In the code below, we perform a simple check on our location when using a proxy.
4. First, we import our dependencies.
5. Then, we read our credentials from a config file. You can technically hardcode your API keys into your file, but it is not considered best practice to do so.

import requestsimport json
config = {}
with open("config.json") as file:    config = json.load(file)["oxylabs"]
USERNAME = config["username"]PASSWORD = config["password"]HOSTNAME = "dc.oxylabs.io"PORT = 8000
proxy_url = f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.text)

• Set our credentials: USERNAME, PASSWORD, PORT.
• Set our proxy_url: http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}. We then create a dict that sets this proxy as our http and https proxies.
• We use the proxy with requests.get(url, proxies=proxies). proxies=proxies assigns our new dict to the proxy protocol used in Python Requests.
• Once we receive our response back, we go ahead and print it to the terminal.

Understanding Datacenter Proxies

Datacenter vs Residential

Datacenter

• Price: Datacenter proxies are cheap... really cheap. When you use datacenter bandwidth, you usually start out at less than $1/GB.
• Speed: These proxies are hosted on real machines inside actual datacenters. Datacenters typically get you top of the line connection speed and hardware.
• Availability: Datacenter proxies usually operate with a much larger pool of IP addresses.

• Blocking: Some sites block datacenter IPs by default. Ideally, these are the sites that you'd scrape using a Residential Proxy.
• Less Geotargeting Support: When we use datacenter proxies, we often do get the option to use geotargeting. However, we aren't able to target very specifically and our IP will show up as a datacenter IP.
• Less Anonymity: Your IP address is tied to that of a datacenter provider, not that of an actual residential neighborhood. While you do still receive anonymity, your traffic tends to look a bit odd on the server end.

Residential

• Anonymity: Residential proxies offer much more anonymity. When your requests appear as though they're coming from an actual house or mobile device, they're much less likely to stick out.
• Better Access: As previously mentioned, some sites block datacenter IP addresses. If you're trying to access a site that blocks them, you need to use a residential IP address.

• Price: Residential Proxies are far more expensive than their Datacenter counterparts. Oxylabs charges $8/GB on their Pay As You Go plan!
• Speed: Residential proxies are often slower than their datacenter counterparts. When you're on a residential proxy, your traffic is going from your machine, to another server, to a residential machine. Then when a response is received, it needs to make it back through that entire chain again. If you factor in varying hardware, that can bottleneck pretty quickly.

Why Use Oxylabs Datacenter Proxies?

• Datacenter Proxies with Oxylabs are both flexible and affordable.
• We can use geotargeting to select which country we want to appear in.

Oxylabs Datacenter Proxy Pricing

Pay Per IP Address

Pay Per Bandwidth

Setting Up Oxylabs Datacenter Proxies

Authentication

import requestsimport json
config = {}
with open("config.json") as file:    config = json.load(file)["oxylabs"]
USERNAME = config["username"]PASSWORD = config["password"]HOSTNAME = "dc.oxylabs.io"PORT = 8000
proxy_url = f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.text)

• Our username is always passed in as user-{USERNAME}.
• This is followed by :{PASSWORD}.

Basic Request Using Oxylabs Datacenter Proxies

import requestsimport json
config = {}
with open("config.json") as file:    config = json.load(file)["oxylabs"]
USERNAME = config["username"]PASSWORD = config["password"]HOSTNAME = "dc.oxylabs.io"PORT = 8000
proxy_url = f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.text)

• We get started by importing our dependencies and reading our credentials from a config file.
• Our HOSTNAME is dc.oxylabs.io and our PORT is 8000.
• We combine our credentials with our HOSTNAME and PORT to create our proxy_url: f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}".
• We then make a dict called proxies and assign this proxy_url to the http and https keys in the dict.
• When we make requests, we pass in proxies=proxies and this tells Python Requests to route our requests through these specific proxies.

Country Geotargeting

import requestsimport json
config = {}
with open("config.json") as file:    config = json.load(file)["oxylabs"]
USERNAME = config["username"]PASSWORD = config["password"]HOSTNAME = "dc.oxylabs.io"PORT = 8000COUNTRY = "us"
proxy_url = f"http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.text)

• COUNTRY = "us" allows us to save our country as a variable. This makes our code easier to tweak in the future.
• Our url now includes the country flag: http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@{HOSTNAME}:{PORT}". This tells Oxylabs that we want to use a proxy located in a specific country.

City Geotargeting

• Advertisements
• Localized Social Media (like Nextdoor or Marketplace)
• Local Store Pricing
• General town sentiment

Error Codes

KYC Verification

Implementing Oxylabs Datacenter Proxies in Web Scraping

Python Requests

import requestsimport json

USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "dc.oxylabs.io"PORT = 8000
proxy_url = f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.text)

• Once we've got our credentials, our proxy_url gets setup like this: http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriver
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "dc.oxylabs.io"PORT = 8000
proxy_url = f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We setup our url with the same structure we used with Python Requests: http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}.
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options,

Python Scrapy

scrapy startproject datacenter

import scrapy
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "dc.oxylabs.io"PORT = 8000
proxy_url = f"http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
class ScrapyExampleSpider(scrapy.Spider):    name = "datacenter_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

• We construct our url the same way we did in the other previous two examples: http://user-{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}.
• Inside of our start_requests method, we assign our proxy_url to request.meta['proxy'].

NodeJS Puppeteer

mkdir puppeteer-datacenter

cd puppeteer-datacenternpm init --y

npm install puppeteer

const puppeteer = require('puppeteer');
const PORT = 8000;const USERNAME = "your-username";const PASSWORD = "your-password";
(async () => {  const browser = await puppeteer.launch({    args: [`--proxy-server=dc.oxylabs.io:${PORT}`]  });
  const page = await browser.newPage();
  await page.authenticate({    username: `user-${USERNAME}`,    password: PASSWORD  });
  await page.goto('http://lumtest.com/myip.json');  await page.screenshot({path: 'puppeteer.png'});
  await browser.close();})();

• First, we declare all of our configuration variables as constants: PORT, USERNAME, and PASSWORD.
• We set our url when we launch our browser with this arg added args: [`--proxy-server=`dc.oxylabs.io:${PORT}`].
• We add our USERNAME to the authentication: user-${USERNAME}.
• We also add our PASSWORD to the authentication: password: PASSWORD.

NodeJS Playwright

mkdir playwright-datacenter

cd playwright-datacenternpm init --y

npm install playwrightnpx playwright install

const playwright = require('playwright');
const PORT = 8000;const USERNAME = "randomuser_4U4Cw";const PASSWORD = "Some_bullshit1234";
const options = {    proxy: {        server: `http://dc.oxylabs.io:${PORT}`,        username: `user-${USERNAME}`,        password: PASSWORD    }};
(async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
    await page.goto('http://lumtest.com/myip.json');
    await page.screenshot({ path: "playwright.png" })
    await browser.close();})();

• Like our Puppeteer example, we first setup our configuration variables: PORT, USERNAME, and PASSWORD.
• We create a proxy object with the following fields:
• server: `http://dc.oxylabs.io:${PORT}
• username: `user-${USERNAME}
• password: PASSWORD

Case Study: Scraping The Guardian

1. In the code below, we'll first setup a proxy in the US.
2. We perform a GET to verify our location information and then we make a GET to The Guardian.
3. We print our location information and we find the navbar from the Guardian's front page.
4. After this initial run, we reset our connection to use an IP inside of the UK.
5. If our proxies are working, we'll receive different output from each proxy.

import requestsfrom bs4 import BeautifulSoup
import requestsimport json
config = {}
with open("config.json") as file:    config = json.load(file)["oxylabs"]
USERNAME = config["username"]PASSWORD = config["password"]HOSTNAME = "dc.oxylabs.io"PORT = 8000COUNTRY = "us"
proxy_url = f"http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}


print("----------------------US---------------------")

location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get('https://www.theguardian.com/', proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")print(subnav.text)

print("----------------------UK---------------------")
COUNTRY = "gb"
proxy_url = f"http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get('https://www.theguardian.com/', proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")
print(subnav.text)

• Earlier, we were constantly checking our location info. location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies) does this once again. Once we've got our location info, we print it to the console so that we can verify our results.
• COUNTRY: This variable changes later on in the code so we can reset the proxy.
• proxies: After resetting our country, we need to reset our proxies.

• us: USUS elections 2024WorldEnvironmentUkraineSoccerBusinessTechScienceNewslettersWellness
• gb: WorldUKClimate crisisUkraineEnvironmentScienceGlobal developmentFootballTechBusinessObituaries

• us: US | US elections | 2024 | World | Environment | Ukraine | Soccer | Business | Tech | Science | Newsletters | Wellness
• gb: World | UK | Climate crisis | Ukraine | Environment | Science | Global development | Football | Tech | Business | Obituaries

Alternative: ScrapeOps Residential Proxy Aggregator

• JavaScript Rendering
• Screenshot Capability
• Country Geotargeting
• Residential and Mobile Proxies
• Anti-bot Bypass
• Custom Headers
• Sticky Sessions

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"LOCATION = "us"
def get_scrapeops_url(url, location=LOCATION):    payload = {        "api_key": API_KEY,        "url": url,        "country": location    }    proxy_url = "https://proxy.scrapeops.io/v1/?" + urlencode(payload)    return proxy_url
response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json"))print(response.text)

• Create our configuration variables: API_KEY and LOCATION.
• Write a get_scrapeops_url() function. This function takes all of our parameters along with a target url and wraps it into a ScrapeOps Proxied url. This is an incredibly easy way to scrape and it makes our proxy code much more modular.
• Check our IP info with response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json")).
• Finally, we print it to the terminal. You should get an output similar to this.

Ethical Considerations and Legal Guidelines

Legal

• Don't use proxies to access illegal content: These actions can come with intense legal penalties and even prison or jail time depending on the severity of the offense.
• Don't scrape and disseminate other people's private data: Depending on what jurisdiction you're dealing with, this is also a highly illegal and dangerous practice. Doxxing private user data can also lead to heavy fines and possibly jail/prison time,

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement. In most cases, a terms of service agreement is legally binding.

Conclusion

More Cool Articles

• Python Web Scraping Playbook
• NodeJS Web Scraping Playbook
• Scrapy Web Scraping Playbook
• Python Selenium Web Scraping Playbook