Geonode Residential Proxies: Web Scraping Guide

• TLDR: How to Integrate Geonode Residential Proxy
• Understanding Residential Proxies
• Why Use Geonode Residential Proxies?
• Geonode Residential Proxy Pricing
• Setting Up Geonode Residential Proxies
• Authentication
• Basic Request Using Geonode Residential Proxies
• Country Geotargeting
• City Geotargeting
• How To Use Static Proxies
• Error Codes
• Implementing Geonode Residential Proxies in Web Scraping
• Case Study - Scraping Amazon.es with Geo-Location Targeting
• Advanced Techniques and Proxy Rotation
• Ethical Considerations and Legal Guidelines
• Conclusion

TLDR: How to Integrate Geonode Residential Proxy

• Sign up for a Geonode account and complete the KYC process.
• Obtain your credentials (username and password) from the dashboard.
• Use this code snippet to make requests via Geonode’s proxy:

import requests
# Proxy configurationusername = "YOUR_USERNAME"password = "YOUR_PASSWORD"endpoint = "YOUR_ENDPOINT"
proxy_url = f"http://{username}:{password}@{endpoint}"
proxies = {    "http": proxy_url,    "https": proxy_url}
# Make a requesturl = "http://google.com"response = requests.get(url, proxies=proxies)
print(f"Status Code: {response.status_code}")print(f"Content: {response.text}")

Understanding Residential Proxies

Why Are Residential Proxies Important?

• Market Research: Scraping pricing and competitor data without being flagged.
• Ad Verification: Ensuring ads are displayed correctly across different regions.
• Accessing Geo-restricted Content: Viewing content restricted to specific locations.
• E-commerce: Managing multiple accounts or monitoring product availability.
• SEO & Analytics: Collecting accurate SERP data without triggering captcha or blocks.

Rotating Residential Proxies

• IPs originate from real residential devices, offering high authenticity.
• Access to a broad range of IPs across multiple countries.
• Easily bypasses geo-restrictions and anti-scraping defenses.

• Frequent IP changes can disrupt session consistency, which may be an issue for tasks needing continuous sessions.
• Typically slower and costlier than data center proxies.

Static Residential Proxies

• ISP-assigned IPs offer enhanced stability and dependability.
• Faster and generally more reliable than rotating residential proxies.
• High-quality IPs that improve performance.

• More costly due to the ISP-provided IPs.
• Limited country coverage compared to rotating proxies.

Residential vs. Data Center Proxies

• Residential Proxies: Best for tasks requiring undetectable, authentic IPs, especially for geo-sensitive or heavily monitored platforms.
• Datacenter Proxies: Ideal for cost-effective, high-speed automation where detection isn’t a major concern.

When Are Residential Proxies Useful?

Web Scraping and Data Collection

SEO and SERP Analysis

Social Media Monitoring

Ad Verification

Geo-Restricted Content Access

Why Use Geonode Residential Proxies?

• Large IP Pool: Access millions of residential IP addresses worldwide, enabling diverse IP use for price comparisons, market research, and large-scale scraping without blocks.
• Worldwide Coverage: Connect from over 160 countries, perfect for localized data collection and geo-targeting across regions.
• Unlimited Concurrent Requests: Run multiple scraping tasks simultaneously, improving efficiency by monitoring numerous websites at once.
• Flexible Session Control: Choose between sticky or rotating IP sessions, minimizing detection risk for large-scale scraping operations.
• SOCKS5 & HTTPS Protocols: Supports advanced protocols for secure, high-performance streaming and scraping, ensuring faster and more reliable connections.
• Customizable Rotation: Adjustable IP rotation based on time or request count, reducing potential detection by aligning with website rate limits.
• Ethical Sourcing: IPs are ethically sourced from consenting users worldwide, ensuring compliance with global standards like GDPR and CCPA.
• High Uptime & Reliability: 99.9% uptime ensures continuous data collection with minimal interruptions, maximizing efficiency.
• Pay-as-You-Go Pricing: Flexible, usage-based billing provides affordable access without subscription commitments, fitting both large and small projects.
• Reputation & Trust: Highly rated across trusted review platforms, offering a reliable and ethical proxy solution.
• Global Network: Proxies sourced from 190+ countries, ensuring accurate data collection and regional targeting.
• ZenShield and Repocket Partnerships: Ethical sourcing solutions through partnerships, supporting trust and integrity in proxy usage.

Geonode Residential Proxy Pricing

Pricing Structure

• Pay-As-You-Go (PAYG): Only pay for the data you actually use, making this a cost-effective choice for varying data needs without committing to a monthly subscription.
• Membership Discounts: Save up to 40% with membership plans, offering substantial discounts for high-volume users.
• Static & Rotating Options: Choose between IPs that remain static for up to 24 hours or rotating proxies, based on your project’s requirements.

Pricing Table

Flexible Billing and Plan Modifications

Comparison to Other Providers

Setting Up Geonode Residential Proxies

Creating a Geonode Account

1. Visit the Geonode website and sign up using your Google account or email.

2. Before being able to utilize the proxies, you need to subscrible or start a 1$ Trial. Click on the arrow button in the Residential Proxies Card.

3. This will take you to Premium Residential Window. To start a trial you can click on "Try with $1 Trial". For any of the subscription types.

4. Now Enter your card details and start the trial.
5. Now your trial is started (for 3 days) but you still need to purchase the data that you want to utilzie (In trial you can purchase for free).
6. Since you are on a trial, you will have $5 in you wallet to buy data and try it out. Select the wallet, agree to terms and conditions and click Confirm payment. Subsequently if you want more data, you can buy through card or crypto as well.

Configuring Proxies

Access Your Zones:

• To access different zones or locations throughout the world, you can go to "Proxy configuration" on the dashboard. And from there you can select any country, it's state or city. And specify the port from 9000 to 9010. The port would be later used when we get the endpoints for the proxies.

Configure Your Zone:

• Now based on the port you selected for a zone/location you can select an ENDPOINT with that port. You will use that ENDPOINT later when you configure the proxy.

Proxy Integration

Steps to Set Up a Proxy Rotator with Geonode

1. Get your ENDPOINT from the dashboard.

2. Integrating with Python's requests Library: Use the requests library to send HTTP requests via Geonode’s proxies. This setup lets you control the IP address for each request, facilitating web scraping and bypassing location restrictions.
3. Monitoring and Managing Proxies: Geonode provides tools to track proxy performance, allowing you to make real-time adjustments based on success rates and response times.

Basic Python Proxy Integration Example

import requests
# Define the proxy server with IP and portproxies = {    'http': 'http://YOUR_ENDPOINT',    'https': 'https://YOUR_ENDPOINT'}
# Send a GET request using the proxyresponse = requests.get('https://www.example.com', proxies=proxies)
# Check response status and print the contentprint(response.status_code)print(response.text)

• A proxies dictionary specifies the proxy server for both HTTP and HTTPS requests.
• We pass proxies as a parameter in the requests.get() method.
• Finally, the response’s status code and content are printed, which confirms whether the proxy setup is functioning correctly.

Additional Features and Use Cases

Authentication

import requestsfrom bs4 import BeautifulSoup
# Proxy configurationusername = "YOUR_USERNAME"password = "YOUR_PASSWORD"endpoint = "premium-residential.geonode.com:9001"
proxy_url = f"http://{username}:{password}@{endpoint}"
proxies = {    "http": proxy_url,    "https": proxy_url}
# Make a requesturl = "http://google.com"response = requests.get(url, proxies=proxies)
s = BeautifulSoup(response.content, 'html.parser')print(response.status_code)print(s.text)
# From here you can process the data as needed

• We import the requests and BeautifulSoup libraries.
• Initialize username, password, and endpoint variables (with placeholders for credentials).
• Send an HTTP GET request using the proxy.
• Parse the HTML response using BeautifulSoup.
• Print the status code and parsed text content.

Basic Request Using Geonode Residential Proxies

1. Install the requests Library: Ensure you have the requests library installed.
2. Set Up Proxy Details: Use your Geonode credentials, including username, password, and endpoint URL, to configure the proxy.
3. Send the Request: Use requests to route your HTTP requests through the Geonode proxy.

import requestsfrom bs4 import BeautifulSoup
# Geonode Proxy configurationusername = "YOUR_USERNAME"  # Replace with your Geonode usernamepassword = "YOUR_PASSWORD"  # Replace with your Geonode passwordendpoint = "premium-residential.geonode.com:9001"
proxy_url = f"http://{username}:{password}@{endpoint}"
proxies = {    "http": proxy_url,    "https": proxy_url}
# Target URLurl = "http://books.toscrape.com/"response = requests.get(url, proxies=proxies)
# Parse and extract data using BeautifulSoupsoup = BeautifulSoup(response.content, 'html.parser')links = soup.find_all('a')for link in links:    print(link.get('href'))

Country Geotargeting

Using Country-Specific Proxies

import requestsfrom bs4 import BeautifulSoup
# Geonode Proxy configurationusername = "YOUR_USERNAME"   # Replace with your Geonode usernamepassword = "YOUR_PASSWORD"   # Replace with your Geonode passwordendpoint = "premium-residential.geonode.com:9001"  # Use assigned country-specific port if applicable
proxy_url = f"http://{username}:{password}@{endpoint}"
proxies = {    "http": proxy_url,    "https": proxy_url}
response_ip = requests.get("https://httpbin.org/ip", proxies=proxies)ip = response_ip.json()['origin']
# Get detailed inforesponse_info = requests.get(f"http://ipinfo.io/{ip}", proxies=proxies)info = response_info.json()
print(f"IP: {info['ip']}")print(f"Country: {info['country']}")

• Configure Proxy: The proxy URL is generated using Geonode credentials (username, password, and endpoint). For country-specific access, select the required country in Geonode’s dashboard and assign it to the chosen port.
• Make a Request: Set up both HTTP and HTTPS proxy options, then send the request.
• Parse HTML: Use BeautifulSoup to parse and extract data from the target page.

City Geotargeting

• You can select the state and the city from where you select a country.
• Then specify the port.
• Use that port in the ENDPOINT.

How To Use Static Proxies

Key Benefits of Static Proxies

• Consistency and Reliability: Provide a fixed IP address, ensuring that your connection remains stable over time.
• Better Performance for Long Sessions: Better suited for extended, uninterrupted sessions where a persistent connection is needed.
• Avoiding Geo-Blocking or IP Bans: Help you avoid being blocked or banned from websites that limit access based on changing IPs or that may flag IPs that are frequently rotated.

Common Use Cases for Static Proxies

• SEO and SERP Tracking: For consistently tracking search engine results from the same location.
• Social Media Automation: Managing multiple social media accounts from a fixed location.
• Web Scraping: When scraping data from websites that may block rotating IPs or when a consistent IP is needed to avoid detection.
• Testing: Static proxies are useful for testing applications that require a stable connection from a specific location.

Example of Using Static Proxies with Python

1. Creating a Proxy Zone

2. Configuring Your Proxy

3. Making Requests with Sticky Proxies

import requests
# Geonode Proxy configurationusername = "YOUR_USERNAME"   # Replace with your Geonode usernamepassword = "YOUR_PASSWORD"   # Replace with your Geonode passwordendpoint = "premium-residential.geonode.com:10000"  # Sticky endpoint
proxy_url = f"http://{username}:{password}@{endpoint}"
proxies = {    "http": proxy_url,    "https": proxy_url}
# Make request to check IPresponse_ip = requests.get("https://httpbin.org/ip", proxies=proxies)ip = response_ip.json()['origin']
# Get detailed inforesponse_info = requests.get(f"http://ipinfo.io/{ip}", proxies=proxies)info = response_info.json()
print(f"IP: {info['ip']}")print(f"Country: {info['country']}")

• Configure Proxy: Create the proxy URL using your Geonode credentials and the assigned endpoint.
• Request Static IP: Make requests to httpbin.org/ip to confirm the IP is stable across calls.
• Detailed Information: Use the IP from ipinfo.io to verify geolocation accuracy.

import pprintimport randomimport requests
# Geonode Proxy configuration (replace with your own credentials)username = "YOUR_USERNAME"  # Replace with your Geonode usernamepassword = "YOUR_PASSWRORD"  # Replace with your Geonode password
def get_proxies_from_file(file_path):    """Reads proxies from the specified file and returns a list of proxies."""    with open(file_path, 'r') as file:        proxies = [line.strip() for line in file.readlines()]    return proxies
def make_request_with_proxy(proxy):    """Makes a request to check IP and fetches geolocation information."""    # Construct the proxy URL with authentication    proxy_url = f'http://{username}:{password}@{proxy}'        proxies = {        'http': proxy_url,        'https': proxy_url    }
    # Get IP address from httpbin    ip_response = requests.get("https://httpbin.org/ip", proxies=proxies)    ip_data = ip_response.json()    print(f"IP Address: {ip_data['origin']}")
    # Get detailed info from IP geolocation service    geo_response = requests.get("http://ipinfo.io", proxies=proxies)    pprint.pprint(geo_response.json())
def main():    proxy_file = 'RESIDENTIAL-PREMIUM-sticky.txt'  # Path to your .txt file with Geonode proxies    proxies = get_proxies_from_file(proxy_file)        if not proxies:        print("No proxies found in the file.")        return        selected_proxy = random.choice(proxies)    print(f"Using proxy: {selected_proxy}")        make_request_with_proxy(selected_proxy)
if __name__ == "__main__":    main()

1. get_proxies_from_file: Reads the proxy list from RESIDENTIAL-PREMIUM-sticky.txt. Each line should contain a proxy in the format host:port.
2. make_request_with_proxy: Adds the Geonode username and password directly to each proxy URL for authentication. This function:
   • Retrieves the IP address from the proxy.
   • Retrieves additional geolocation data using the IP.
3. main function: Manages reading the proxies, selecting one at random, and initiating the request.

• Each line in RESIDENTIAL-PREMIUM-sticky.txt should have the format:
  host:port
• Replace username and password in the code with your Geonode account details for authentication.

Error Codes

KYC Verification

• Personal Review: A compliance officer personally evaluates each customer’s intended use case.
• Ongoing Monitoring: User accounts are monitored continually to ensure compliance with declared use cases.

• Adult content
• Gambling
• Cryptocurrency-related activities

Implementing Geonode Residential Proxies in Web Scraping

Python Requests

import requestsfrom bs4 import BeautifulSoup
# Geonode Proxy configurationusername = "YOUR_USERNAME"  # Replace with your Geonode usernamepassword = "YOUR_PASSWORD"  # Replace with your Geonode passwordendpoint = "premium-residential.geonode.com:10000"  # Geonode endpoint for premium residential proxies
# Define the proxy URLproxy_url = f"http://{username}:{password}@{endpoint}"
# Set up proxiesproxies = {    "http": proxy_url,    "https": proxy_url}
# Define the URL to scrapeurl = "http://books.toscrape.com/"
# Send a GET request through the Geonode proxyresponse = requests.get(url, proxies=proxies)
# Parse the HTML content with BeautifulSoupsoup = BeautifulSoup(response.content, "html.parser")
# Find all the links on the web pagelinks = soup.find_all("a")
# Print each link foundfor link in links:    print(link.get("href"))

• We configure the Geonode proxy by setting up the proxy URL with the username and password.
• Requests are sent through the specified proxy, enabling access to restricted sites and allowing IP rotation if needed.
• The code fetches and parses all links from the target website using BeautifulSoup.

Python Selenium

Python Scrapy

1. Create a new Scrapy project:

   scrapy startproject <project_name>

2. Inside your Scrapy project, create a spider using Geonode proxies:

   import scrapy
   # Geonode Proxy configurationusername = "YOUR_USERNAME"password = "YOUR_PASSWORD"
   class GeonodeScrapyExampleSpider(scrapy.Spider):    name = "GeonodeScrapyExample"
       def start_requests(self):        request = scrapy.Request(url="http://example.com", callback=self.parse)        request.meta['proxy'] = f"http://{username}:{password}@premium-residential.geonode.com:10000"        yield request
       def parse(self, response):        print(response.body)

scrapy runspider <Pythonfilename.py>

Node.js Playwright

1. Install Playwright:

   npm install playwright

2. Set up Playwright with Geonode proxy settings:

   const playwright = require('playwright');
   const options = {    proxy: {        server: 'http://premium-residential.geonode.com:10000',        username: 'YOUR_USERNAME',        password: 'YOUR_PASSWORD'    }};
   (async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
       await page.goto('http://example.com');
       const content = await page.content();    console.log(content);
       await browser.close();})();

node scrape.js

NodeJS Puppeteer

1. Install Puppeteer:

   npm install puppeteer

2. Set up the script with Geonode proxy authentication:

   const puppeteer = require('puppeteer');
   (async () => {  const browser = await puppeteer.launch({    headless: false,    args: ['--proxy-server=premium-residential.geonode.com:10000']  });
     const page = await browser.newPage();
     await page.authenticate({    username: 'YOUR_USERNAME',    password: 'YOUR_PASSWORD'  });
     await page.goto('http://example.com');  await page.screenshot({ path: 'example.png' });
     await browser.close();})();

node scrape.js

Case Study - Scraping Amazon.es

import requestsfrom bs4 import BeautifulSoupfrom dataclasses import dataclassfrom typing import Dict, Tuple
# Proxy configurationPROXY_CONFIG = {    "host": "premium-residential.geonode.com",    "port": "10000",    "username": "YOUR_USERNAME",  # Replace with your Geonode username    "password": "YOUR_PASSWORD"   # Replace with your Geonode password}
# Product URLPRODUCT_URL = 'https://www.amazon.es/Taurus-WC12T-termoel%C3%A9ctrica-Aislamiento-Temperatura/dp/B093GXXKRL/ref=lp_14565165031_1_2'
@dataclassclass ProductInfo:    title: str    price: str    availability: str
def get_proxy_url(country: str) -> str:    # Generate Geonode proxy URL with country targeting    return f"http://{PROXY_CONFIG['username']}-country-{country}:{PROXY_CONFIG['password']}@{PROXY_CONFIG['host']}:{PROXY_CONFIG['port']}"
def get_headers() -> Dict[str, str]:    # HTTP headers to simulate a browser request    return {        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',        'Accept-Language': 'es-ES,es;q=0.9',        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',        'Accept-Encoding': 'gzip, deflate, br',        'Referer': 'https://www.amazon.es/'    }
def scrape_amazon(country: str) -> ProductInfo:    proxy_url = get_proxy_url(country)    proxies = {"http": proxy_url, "https": proxy_url}        try:        response = requests.get(PRODUCT_URL, proxies=proxies, headers=get_headers(), verify=False, timeout=30)        response.raise_for_status()                soup = BeautifulSoup(response.content, 'html.parser')                title = soup.select_one('#productTitle').text.strip() if soup.select_one('#productTitle') else "Title not found"        price = soup.select_one('.a-price-whole').text.strip() if soup.select_one('.a-price-whole') else "Price not found"        availability = soup.select_one('#availability span').text.strip() if soup.select_one('#availability span') else "Availability not found"                return ProductInfo(title, price, availability)        except requests.RequestException as e:        print(f"An error occurred while scraping with {country} IP: {str(e)}")        return ProductInfo("Error", "Error", "Error")
def main():    countries = {        "SPAIN": "es",        "PORTUGAL": "pt"    }        results = {country_code: scrape_amazon(country_code) for country_code in countries.values()}        for country_name, country_code in countries.items():        info = results[country_code]        print(f"\n{country_name} IP Results:")        print(f"Title: {info.title}")        print(f"Price: {info.price}€")        print(f"Availability: {info.availability}")
if __name__ == "__main__":    main()

• We observe that the product price is slightly higher when accessed from a Portuguese IP (222€) compared to a Spanish IP (199€).
• This demonstrates Amazon's dynamic pricing strategy, where prices may vary based on the user's location.

Tips for Troubleshooting Common Issues

• Verify Proxy Configuration: Double-check that the proxy details (host, port, username, and password) are correct. Incorrect or outdated proxy credentials often lead to connection errors.
• Confirm Proxy Location: Ensure the proxy is located in a region supported by the target website. Using proxies from unsupported regions might not bypass geo-restrictions effectively.
• Monitor IP Rotation: Geonode proxies may rotate IP addresses. Make sure the proxy in use is stable and provides consistent access from the intended region.
• Implement Error Handling for Proxy Failures: Add error handling to your scraping code to handle cases where a proxy might fail or return an invalid response. This can help you troubleshoot issues more efficiently.
• Test Multiple Proxies: If one proxy doesn’t work, try different proxies to determine if the issue lies with the specific proxy server or your configuration.

Alternative: ScrapeOps Residential Proxy Aggregator

Why Choose ScrapeOps Residential Proxy Aggregator?

Using ScrapeOps Residential Proxy Aggregator with Python Requests

import requests
api_key = 'YOUR_API_KEY'target_url = 'https://httpbin.org/ip'proxy_url = f'http://scrapeops:{api_key}@residential-proxy.scrapeops.io:8181'
proxies = {    'http': proxy_url,    'https': proxy_url,}
response = requests.get(    url=target_url,    proxies=proxies,    timeout=120,)
print('Body:', response.content)

Ethical Considerations and Legal Guidelines

• Respect Website Terms of Service: Websites establish terms of service to govern access and usage of their data. Scraping activities should align with these terms to prevent legal issues or bans. While Geonode’s proxies provide anonymity, users are still obligated to follow the legal and ethical guidelines of each target site.
• Avoid Abusive Practices: Users should refrain from actions that may be perceived as abusive, such as excessive scraping that could overload a website's servers or invasive data extraction. Responsible proxy use means respecting the limits set by target websites.
• Prioritize Transparency and Consent: While Geonode ensures ethically sourced proxies, users should also be transparent about their data collection activities when feasible. Informing site owners about the purpose and scope of data collection can foster trust and open channels for potential collaboration.

Importance of Scraping Responsibly

Conclusion

More Python Web Scraping Guides

• How To Create And Rotate User Agents To Evade Detection
• How to Scrape The Web Without Getting Blocked Guide
• Scrape Google Cache Version