Geonode Datacenter Proxies: Web Scraping Guide

• TLDR: How To Integrate Geonode Datacenter Proxy
• Understanding Datacenter Proxies
• Why Use Datacenter Proxies?
• Datacenter Proxy Pricing
• Setting Up Geonode Datacenter Proxies
• Authentication
• Basic Request Using Geonode Datacenter Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing Geonode Datacenter Proxies
• Case Study: Scraping The Guardian
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

TLDR: How to Integrate Geonode Datacenter Proxy?

1. Once you've got your USERNAME, HOSTNAME, PORT, and PASSWORD, simply put them into the file below.
2. This code sets up a proxy connection.
3. Then, it checks your location information to ensure that everything's working correctly.
4. If the output is not your actual location, the proxy connection is working correctly.

import requestsimport json
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9000
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.json())

• First, we setup our configuration variables: USERNAME, PASSWORD, HOSTNAME, PORT.
• Next, we create a proxy_url. It is comprised of the configuration variables mentioned above.
• We create a dict object holding our http and https proxies. We set them both to our proxy_url.
• We print our proxied_ip to the terminal. If the location printed here is different from your actual location, your proxy is working correctly.

Understanding Datacenter Proxies

Datacenter vs Residential

Datacenter

• Price: Datacenter proxies are cheap... really cheap. When using residential proxies, it's not uncommon to pay up to $8/GB.
• Speed: These proxies are hosted inside actual datacenters. Datacenters usually use top of the line internet connections and hardware. This can really increase our speed and efficiency.
• Availability: Datacenter proxies usually operate with a much larger pool of IP addresses.

• Blocking: Some sites block datacenter IPs by default. This makes some sites more difficult to scrape when using a datacenter proxy.
• Less Geotargeting Support: While we often do get the option to choose our location with datacenter proxies, they still don't appear quite as normal as a residential proxy. We choose our location, but it still shows up as a datacenter.
• Less Anonymity: Since you're not traced to an individual residential IP address, your proxy location can be traced easily to a datacenter. This doesn't reveal your identity, but it does reveal that the request is not coming from a standard residential location. Your request isn't tied to some random household, it's tied to a company.

Residential

• Anonymity: Residential proxies do give us a higher degree of anonymity. Since you're using an IP address tied to an actual house, your traffic blends in much more.
• Better Access: There are quite a few sites that block datacenter IP addresses. If you're trying to access a site that blocks them, you need to use a residential IP address.

• Price: Residential Proxies are far more expensive than their Datacenter counterparts. Geonode charges $4/GB on their Pay As You Go plan!
• Speed: Residential proxies are often slower than their datacenter counterparts. You're not always tied to a state of the art machine with the best connection. You are tied to a real residential device using a residential internet connection.

Why Use Geonode Datacenter Proxies?

• Datacenter Proxies with Geonode are very affordable.
• We can use geotargeting to select which country we want to appear in.

Geonode Datacenter Proxy Pricing

Setting Up Geonode Datacenter Proxies

Authentication

import requestsimport json
USERNAME = "your-username"PASSWORD = "your-super-secret-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9000
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.json())

Basic Request Using Geonode Datacenter Proxies

import requestsimport json
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9000
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.json())

Country Geotargeting

import requestsimport json
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9001
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.json())

City Geotargeting

import requestsimport json
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9002
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.json())

• Local Ads
• Local Businesses
• Local Social Media
• Local Events

Error Codes

KYC Verification

Implementing Geonode Datacenter Proxies in Web Scraping

Python Requests

import requestsimport json
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9000
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
url = "http://lumtest.com/myip.json"
proxied_ip = requests.get(url, proxies=proxies)print("Proxy Location:", proxied_ip.json())

• Once we've got our credentials, our proxy_url looks like this: f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}".
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriver
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9000
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"

## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We build our url exactly the same way we did with Python Requests: f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}".
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options,

Python Scrapy

scrapy startproject datacenter

import scrapy
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"PORT = 9001
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}"

class BrightdataScrapyExampleSpider(scrapy.Spider):    name = "datacenter_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

• Once again, our proxy_url gets formatted the same way: f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{PORT}".
• Inside of our start_requests method, we assign our proxy_url to request.meta['proxy'].

NodeJS Puppeteer

mkdir puppeteer-datacenter

cd puppeteer-datacenternpm init --y

npm install puppeteer

const puppeteer = require('puppeteer');
const USERNAME = "your-username";const PASSWORD = "your-password";const HOSTNAME = "shared-datacenter.geonode.com";const PORT = 9000;
(async () => {  const browser = await puppeteer.launch({    args: [`--proxy-server=${HOSTNAME}:${PORT}`]  });
  const page = await browser.newPage();
  await page.authenticate({    username: USERNAME,    password: PASSWORD  });
  await page.goto('http://lumtest.com/myip.json');  await page.screenshot({path: 'puppeteer.png'});
  await browser.close();})();

• First, we declare all of our configuration variables as constants: PORT, USERNAME, PASSWORD, HOSTNAME.
• We set our url when we launch our browser with this arg added --proxy-server=${HOSTNAME}:${PORT}.
• We add our USERNAME to the authentication: username: USERNAME.
• We also add our PASSWORD to the authentication: password: PASSWORD.

NodeJS Playwright

mkdir playwright-datacenter

cd playwright-datacenternpm init --y

npm install playwrightnpx playwright install

const playwright = require('playwright');
const USERNAME = "your-username";const PASSWORD = "your-password";const HOSTNAME = "shared-datacenter.geonode.com";const PORT = 9000;
const options = {    proxy: {        server: `http://${HOSTNAME}:${PORT}`,        username: USERNAME,        password: PASSWORD    }};
(async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
    await page.goto('http://lumtest.com/myip.json');
    await page.screenshot({ path: "playwright.png" })
    await browser.close();})();

• Like our Puppeteer example, we first setup our configuration variables: PORT, USERNAME, PASSWORD, HOSTNAME.
• We create a proxy object with the following fields:
• server: `http://${HOSTNAME}:${PORT}
• username: USERNAME
• password: PASSWORD

Case Study: Scrape The Guardian

1. In our code below, we first setup a proxy based in the US.
2. We make a GET to verify our location information and then we make a GET to The Guardian.
3. We print our location information and then we find the navbar from the Guardian's front page.
4. After our initial run, we reset our proxy connection so it uses an IP inside of the UK.
5. If our proxies are working, we'll receive different output from each proxy.

import requestsfrom bs4 import BeautifulSoup
USERNAME = "your-username"PASSWORD = "your-password"HOSTNAME = "shared-datacenter.geonode.com"UK_PORT = 9001US_PORT = 9002
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{US_PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}


print("----------------------US---------------------")

location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get('https://www.theguardian.com/', proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")print(subnav.text)

print("----------------------UK---------------------")
proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOSTNAME}:{UK_PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get("https://www.theguardian.com/", proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")
print(subnav.text)

• location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies) is used after setting up each proxy connection. We make this request just for verification purposes.
• We have two different ports setup. If you remember in our geotargeting examples, we setup one proxy in the UK and then we set another up for New York City. The NYC proxy doesn't always show up in New York, but it does consistently show up inside the US. This will be more than sufficient for our needs here.
• proxies: After resetting our country, we need to reset our proxies.

• us: USUS elections 2024WorldEnvironmentUkraineSoccerBusinessTechScienceNewslettersWellness
• gb: UKWorldClimate crisisUkraineFootballNewslettersBusinessEnvironmentUK politicsEducationSocietyScienceTechGlobal developmentObituaries

• us: US | US elections 2024 | World | Environment | Ukraine | Soccer | Business | Tech | Science | Newsletters | Wellness
• gb: UK | World | Climate crisis | Ukraine | Football | Newsletters | Business | Environment | UK politics | Education | Society | Science | Tech | Global development | Obituaries

Alternative: ScrapeOps Proxy Aggregator

• JavaScript Rendering
• Screenshot Capability
• Country Geotargeting
• Residential and Mobile Proxies
• Anti-bot Bypass
• Custom Headers
• Sticky Sessions

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"LOCATION = "us"
def get_scrapeops_url(url, location=LOCATION):    payload = {        "api_key": API_KEY,        "url": url,        "country": location    }    proxy_url = "https://proxy.scrapeops.io/v1/?" + urlencode(payload)    return proxy_url
response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json"))print(response.text)

• Create our configuration variables: API_KEY and LOCATION.
• Write a get_scrapeops_url() function. This function takes all of our parameters along with a target url and wraps it into a ScrapeOps Proxied url. This is an incredibly easy way to scrape and it makes our proxy code much more modular.
• Check our IP info with response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json")).
• Finally, we print it to the terminal. You should get an output similar to this.

{"country":"US","asn":{"asnum":36352,"org_name":"AS-COLOCROSSING"},"geo":{"city":"Buffalo","region":"NY","region_name":"New York","postal_code":"14205","latitude":42.8856,"longitude":-78.8736,"tz":"America/New_York","lum_city":"buffalo","lum_region":"ny"}}

Ethical Considerations and Legal Guidelines

Legal

• Don't use residential proxies to access illegal content: These actions can come with intense legal penalties and even prison or jail time depending on the severity of the offense.
• Don't scrape and disseminate other people's private data: Depending on what jurisdiction you're dealing with, this is also a highly illegal and dangerous practice. Doxxing private data can also lead to heavy fines and possibly jail/prison time,

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement.

Conclusion

More Cool Articles

• How To Scrape Google Reviews
• Bright Data Residential Proxies