IPRoyal Residential Proxies: Web Scraping Guide

• TLDR: How to Integrate IPRoyal Residential Proxy?
• Understanding Residential Proxies
• Why Choose IPRoyal Residential Proxies?
• IPRoyal Residential Proxy Pricing
• Setting Up IPRoyal Residential Proxies
• Authentication
• Basic Request Using IPRoyal Residential Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing IPRoyal Residential Proxies in Web Scraping
• Case Study: Scrape Amazon Prices
• Alternative: ScrapeOps Residential Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Web Scraping Guides

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

TLDR: How to Integrate IPRoyal Residential Proxy?

1. Install the Python requests package:

   pip install requests

2. Set Up IPRoyal Residential Proxy:

   import requests
   username = "your_proxy_username"password = "your_proxy_password"port = "your_proxy_port"proxy = f"geo.iproyal.com:{port}"
   proxies = {    "http": f"http://{username}:{password}@{proxy}",	"https": f"http://{username}:{password}@{proxy}"}
   response = requests.get("https://example.com", proxies=proxies)print(response.text)

1. Set Credentials and Proxy: Provide your IPRoyal credentials (username and password) along with the proxy address (geo.iproyal.com:<port>).
2. Configure Proxies: Create a proxies dictionary to set up HTTP and HTTPS proxies using your credentials.
3. Send a Request: Use the requests.get method to send a request to the target website (https://example.com) via the configured proxies.
4. Print the Response: Finally, print the response text received from the target website.

Understanding Residential Proxies

What Are Residential Proxies?

Types of Residential Proxies

• Rotating Residential Proxies: These proxies automatically change the IP address with each request or at regular intervals, providing high anonymity but potentially slower speeds.
• Static Residential Proxies: These proxies maintain the same IP address for the duration of a session, offering consistent performance but with a higher risk of detection.

Residential vs. Data Center Proxies

When Are Residential Proxies Useful?

• Web Scraping and Data Collection: Ensures accurate data without getting blocked by anti-scraping mechanisms.
• SEO and SERP Analysis: Gathers precise search engine results from different locations.
• Social Media Monitoring: Tracks trends and activities on social media platforms without being flagged.
• Ad Verification: Checks the correct display of ads in different regions.
• Geo-Restricted Content Access: Allows access to content limited to specific geographical areas.

Why Choose IPRoyal Residential Proxies?

• Exceptional Coverage: Access a diverse range of IP addresses worldwide, reducing detection risks and enhancing data collection accuracy.
• Precise Targeting: Select proxies from any country, state, or city effortlessly, ensuring your data is as specific as needed without additional costs.
• Flexible and Reliable: Enjoy pay-as-you-go pricing with no contracts or expiration on purchased traffic, combined with 24/7 support and advanced technical features for seamless integration.

IPRoyal Residential Proxy Pricing

Pricing Table

Pricing Comparison

• Cheap Providers: Providers offering plans around $2-3 per GB are considered cheaper.
• Expensive Providers: Those with smaller plans priced in the $6-8 per GB range are more expensive.

Setting Up IPRoyal Residential Proxies

Creating an IPRoyal Account

Selecting and Purchasing Residential Proxies

Authentication

1. username and password
2. IP whitelisting

Method 1: Username & Password Authentication

1. Install Required Packages:

   pip install requests python-dotenv

2. Set Up Environment Variables: Create a .env file in your project directory to securely store your IPRoyal credentials:

   IPROYAL_USERNAME=your_usernameIPROYAL_PASSWORD=your_passwordIPROYAL_PORT=your_proxy_port

Loading and Configuring Proxy Settings

1. Load Environment Variables: Use the python-dotenv package to load your credentials:

   from dotenv import load_dotenvimport os
   load_dotenv()
   username = os.getenv("IPROYAL_USERNAME")password = os.getenv("IPROYAL_PASSWORD")port = os.getenv("IPROYAL_PORT")proxy = f"geo.iproyal.com:{port}"

2. Set Up Proxies Dictionary: Create a dictionary that contains your proxy settings:

proxies = {    "http": f"http://{username}:{password}@{proxy}:{port}",    "https": f"http://{username}:{password}@{proxy}:{port}"}

3. Send a Request Through the Proxy: Use the requests library to send a request via the configured proxy:

import requestsfrom dotenv import load_dotenvimport os      load_dotenv()      username = os.getenv("IPROYAL_USERNAME")password = os.getenv("IPROYAL_PASSWORD")proxy = os.getenv("IPROYAL_HOST")port = os.getenv("IPROYAL_PORT")
import requests
proxies = {   "http": f"http://{username}:{password}@{proxy}:{port}",   "https": f"http://{username}:{password}@{proxy}:{port}"}   response = requests.get("https://httpbin.org/ip", proxies=proxies)print(response.text)

Method 2: IP Whitelisting Authentication

1. Log in to Your IPRoyal Account:
   • Navigate to the Royal Residential proxies configuration page in your IPRoyal dashboard.
2. Add Your IP Address:
   • Scroll down to find authentication options and click the "Whitelist" button.
   • On the IP whitelist configuration page, click "Add".
   • Configure your proxies as needed (country, state, proxy type, session type).
   • Enter the IP address you want to whitelist.
   • Click "Create" to add the IP to your whitelist.
3. Configure Proxy Without Username and Password: Once your IP is whitelisted, you can set up your proxies without including credentials. In the "Formatted proxy list" section of your dashboard, select your whitelisted IP and copy the IP:PORT information. Then use it like this:

   proxy = "copied_ip:copied_port"
   proxies = {    'http': f'http://{proxy}',    'https': f'http://{proxy}'}

4. Send a Request: You can now send requests through the proxy as usual:

   import requests
   proxy = "copied_ip:copied_port"
   proxies = {    'http': f'http://{proxy}',    'https': f'http://{proxy}'}
   response = requests.get("https://example.com", proxies=proxies)print(response.text)

Basic Request Using IPRoyal Residential Proxies

1. Install Dependencies: Ensure you have the required packages installed:

   pip install requests beautifulsoup4

2. Set Up the Proxy: Configure the proxy settings as demonstrated earlier in the Authentication section.
3. Send the Request and Parse the HTML: Use the requests library to send a request and the BeautifulSoup library to parse the HTML content:

import requestsfrom bs4 import BeautifulSoup
url = 'https://ipv4.icanhazip.com'proxy = 'geo.iproyal.com'port_number = 12321proxy_auth = 'YOUR-USERNAME:YOUR-PASSWORD'proxies = {    'http': f'http://{proxy_auth}@{proxy}:{port_number}',    'https': f'http://{proxy_auth}@{proxy}:{port_number}'}print(proxies)
response = requests.get("https://quotes.toscrape.com", proxies=proxies)soup = BeautifulSoup(response.content, 'html.parser')   # Print the page titleprint(soup.title.text)

1. Set Up Proxies: We configure the proxies dictionary with the necessary credentials and proxy address.
2. Send a Request: We use requests.get() to fetch the content of the target website.
3. Parse the HTML: We utilize BeautifulSoup to parse the HTML content and extract information like the page title.

Handling Proxy Errors

try:    response = requests.get("https://example.com", proxies=proxies, timeout=10)    response.raise_for_status()except requests.exceptions.ProxyError:    print("Proxy error occurred. Please check your proxy settings.")except requests.exceptions.Timeout:    print("The request timed out. Try increasing the timeout or check your internet connection.")except requests.exceptions.RequestException as e:    print(f"An error occurred: {e}")

Country Geotargeting

Top 10 Countries Supported by IPRoyal

Using Country-Specific Proxies

import requests
url = 'https://ipv4.icanhazip.com'proxy = 'geo.iproyal.com:12321'proxy_auth = 'YOUR-USERNAME:YOUR-PASSWORD'country = 'us'proxies = {    'http': f'http://{proxy_auth}_country-{country}@{proxy}',    'https': f'http://{proxy_auth}_country-{country}@{proxy}'}
response = requests.get(url, proxies=proxies)print(response.text)

• We have set up country geo-targeting by adding the _country flag to our url..
• This flag allows us to route our request through the specified country (in this case, "us").

City Geotargeting

Top 10 Cities Supported by IPRoyal

Using City-Specific Proxies

import requests
url = 'https://ipv4.icanhazip.com'proxy = 'geo.iproyal.com:12321'proxy_auth = 'YOUR-USERNAME:YOUR-PASSWORD'country = 'us'city = 'detroit'proxies = {    'http': f'http://{proxy_auth}_country-{country}_city-{city}@{proxy}',    'https': f'http://{proxy_auth}_country-{country}_city-{city}@{proxy}'}
response = requests.get(url, proxies=proxies)print(response.text)

• We have implemented city geo-targeting by using both _country and _city flags.
• The _country flag specifies the country code (e.g., 'us' for the United States), while the _city flag designates the city (e.g., 'detroit').

Error Codes

KYC (Know-Your-Customer) Verification

• IPRoyal requires KYC validation prior to granting full access to their proxy services.
• New accounts have limited access to services until KYC verification is completed.

1. Identity Verification:
   • Customers must provide an image of a government-issued ID (passport, driver's license, or national ID).
   • A selfie is required to confirm the identity matches the provided document.
2. Document Authentication:
   • IPRoyal uses iDenfy, a third-party platform, to verify documents and conduct identity checks.
   • Advanced algorithms detect anomalies or inconsistencies in the provided documents.
   • Facial recognition technology compares the photo on the ID with the provided selfie.
   • Information may be cross-referenced with government databases for accuracy.
3. Ongoing Monitoring:
   • IPRoyal continuously monitors their network to prevent abuse.
   • They reserve the right to refuse service based on risk level assessment.

• IPRoyal states they will refuse service for suspicious, inappropriate, or unethical use cases.

Implementing IPRoyal Residential Proxies in Web Scraping

Python Requests

import requestsfrom dotenv import load_dotenvimport os      load_dotenv()      username = os.getenv("IPROYAL_USERNAME")password = os.getenv("IPROYAL_PASSWORD")proxy = os.getenv("IPROYAL_HOST")port = os.getenv("IPROYAL_PORT")
import requests
proxies = {   "http": f"http://{username}:{password}@{proxy}:{port}",   "https": f"http://{username}:{password}@{proxy}:{port}"}   response = requests.get("https://httpbin.org/ip", proxies=proxies)print(response.text)

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriver
username = "your-username"password = "your-password"proxy = "geo.iproyal.com"port = 12321

## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": f"http://{username}:{password}@{proxy}:{port}",        "https": f"http://{username}:{password}@{proxy}:{port}",        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We setup our url the same way we did with Python Requests: f"http://{username}:{password}@{proxy}:{port}".
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options,

Python Scrapy

scrapy startproject iproyal_scraper

import scrapy
class MySpider(scrapy.Spider):    name = 'myspider'        def start_requests(self):        #proxy configuration        proxy = 'geo.iproyal.com:12321'        proxy_auth = 'YOUR-USERNAME:YOUR-PASSWORD'        country = 'us'        proxy = f'http://{proxy_auth}_country-{country}_city-detroit@{proxy}'                urls = [            'https://httpbin.org/ip',        ]                for url in urls:            yield scrapy.Request(url=url,                                  callback=self.parse,                                 meta={'proxy': proxy})  # Set proxy for this specific request
    def parse(self, response):        # Process the response here        self.log(f"Visited {response.url}, response IP: {response.text}")

scrapy crawl myspider

NodeJS Puppeteer

mkdir puppeteer-scraper

cd puppeteer-scrapernpm init --y

npm install puppeteer

const puppeteer = require('puppeteer');
//proxy configurationPROXY_USERNAME = 'YOUR-USERNAME';PROXY_PASSWORD = 'YOUR-PASSWORD';PROXY_SERVER = 'geo.iproyal.com';PROXY_SERVER_PORT = '12321';
(async () => {    const browser = await puppeteer.launch({        ignoreHTTPSErrors: true,        args: [            `--proxy-server=http://${PROXY_SERVER}:${PROXY_SERVER_PORT}`        ]    });    const page = await browser.newPage();    await page.authenticate({        username: PROXY_USERNAME,        password: PROXY_PASSWORD,      });    
    try {        await page.goto('https://httpbin.org/ip', {timeout: 180000});                await page.screenshot({path: "ip-1.png"})
    } catch(err) {        console.log(err);    }        await browser.close();
})();

NodeJs Playwright

mkdir playwright-scraper

cd playwright-scrapernpm init --y

npm install playwright

npx playwright install

const playwright = require('playwright');
//proxy configurationPROXY_USERNAME = 'YOUR-USERNAME';PROXY_PASSWORD = 'YOUR-PASSWORD';PROXY_SERVER = 'geo.iproyal.com';PROXY_SERVER_PORT = '12321';
(async () => {
    const browser = await playwright.chromium.launch({        headless: true,        // ignoreHTTPSErrors: true,        proxy: {            server: `http://${PROXY_SERVER}:${PROXY_SERVER_PORT}`,            username: PROXY_USERNAME,            password: PROXY_PASSWORD        }            });
    const context = await browser.newContext({ignoreHTTPSErrors: true });
    const page = await context.newPage();
    try {        await page.goto('https://httpbin.org/ip', {timeout: 180000});
        await page.screenshot({path: "ip.png"})
    } catch(err) {        console.log(err);    }        await browser.close();
})();

Case Study: Scrape Amazon Prices

Code Example

import requestsfrom bs4 import BeautifulSoup
country = "pt"proxy_auth = 'YOUR-USERNAME:YOUR-PASSWORD'
proxies = {  "http": f"http://{proxy_auth}_country-{country}@geo.iproyal.com:12321",  "https": f"http://{proxy_auth}_country-{country}@geo.iproyal.com:12321"}
headers = {    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"}
print("----------------------PT-------------------")
response = requests.get('https://www.amazon.es/s?k=port%C3%A1til', proxies=proxies, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")
location = soup.select_one("span[id='glow-ingress-line2']")
print("Location:", location.text)
first_price_holder = soup.select_one("span[class='a-price']")first_price = first_price_holder.select_one("span[class='a-offscreen']")
print("First Price:", first_price.text)

print("----------------------ES---------------------")
country = "es"
proxies = {  "http": f"http://{proxy_auth}_country-{country}@geo.iproyal.com:12321",  "https": f"http://{proxy_auth}_country-{country}@geo.iproyal.com:12321"}
response = requests.get('https://www.amazon.es/s?k=port%C3%A1til', proxies=proxies, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")
location = soup.select_one("div[id='glow-ingress-block']")
print("Location:", location.text.strip())
first_price_holder = soup.select_one("span[class='a-price']")first_price = first_price_holder.select_one("span[class='a-offscreen']")
print("First Price:", first_price.text)

Explanation

• Scraping Logic: We extract our location with soup.select_one("div[id='glow-ingress-block']"). We extract our first price holder on the page with soup.select_one("span[class='a-price']"). Then we extract the price with first_price_holder.select_one("span[class='a-offscreen']").

Initial Attempt: Scrape Location and Price from Amazon.es

country = "pt"

Second Attempt: Scrape Location and Price from Amazon.es

Analysis: Price Differences

----------------------PT-------------------Location:                    Portugal                First Price: 406,60 €----------------------ES---------------------Location: Entrega en Barcelona 08035                
                   Actualizar ubicaciónFirst Price: 399,99 €

• On our first, run, our location was detected as Portugal and our first price on the page was 406,60 €.
• After we change the location, we were detected in Entrega en Barcelona 08035 (Barcelona, Spain) and our first price on the page was 399,99 €.

Alternative: ScrapeOps Residential Proxy Aggregator

Top 3 Reasons to Choose ScrapeOps Residential Proxy Aggregator

1. Competitive Pricing: ScrapeOps offers lower pricing, allowing you to maximize your budget while maintaining high quality.
2. Flexible Plans: With ScrapeOps, you have access to a wider variety of plans, including smaller, more affordable options tailored to your needs. The best part? You can start using the proxies with a free trial account.
3. Enhanced Reliability: By leveraging multiple proxy providers through a single port, ScrapeOps offers greater reliability. If one provider faces issues, your requests can seamlessly switch to another, ensuring continuous access.

Code Example Using Python Requests

import requests
API_KEY = "YOUR-SUPER-SECRET-API-KEY"
proxies = {  "http": f"http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181",  "https": f"http://scrapeops:{API_KEY}@residential-proxy.scrapeops.io:8181"}response = requests.get('https://httpbin.org/ip', proxies=proxies, verify=False)print(response.text)

• Proxy Configuration: We configure the ScrapeOps residential proxy by setting up the proxies dictionary. This includes the proxy server address, port, and necessary authentication details, allowing you to route our requests through the residential proxy.
• Output: We print our IP address to the terminal so we can check and verify that our proxy is configured properly.

Ethical Considerations and Legal Guidelines

1. Compliance with Terms of Service: Adhere to IPRoyal's terms of service, which prohibit illegal activities and abuse of their proxy network.
2. Data Protection: Handle any collected data in compliance with relevant data protection laws, such as GDPR. This includes proper storage, processing, and deletion of personal data if collected.
3. Handle Personal Data Carefully: If your scraping activities involve collecting personal data, ensure you have a legal basis for doing so and that you're complying with relevant data protection regulations like GDPR.
4. Respect Robots.txt: Always check and adhere to the target website's robots.txt file, which specifies which parts of the site can be crawled.
5. Use User-Agents Responsibly: While it's common practice to rotate user-agents, ensure they are recent and commonly used. Misrepresenting your scraper as a different browser could be seen as unethical.

Conclusion

More Web Scraping Guides

• Scrape.do: Web Scraping Integration Guide
• ScrapingAnt: Web Scraping Integration Guide