NetNut Residential Proxies: Web Scraping Guide
Residential proxies use real IP addresses from actual devices to make your scraping requests look like regular user traffic. A standout provider in this space is NetNut, known for its reliable web scraping, data collection, and online anonymity services. It’s distinguished by high performance, security, and global coverage.
This guide will walk you through setting up and integrating NetNut’s residential proxies into your web scraping projects.
- TLDR: How to Integrate NetNut Residential Proxy?
- Understanding Residential Proxies
- Why Use NetNut Residential Proxies?
- NetNut Residential Proxy Pricing
- Setting Up NetNut Residential Proxies
- Authentication
- Basic Request Using NetNut Residential Proxies
- Country Geotargeting
- City Geotargeting
- How To Use Static Proxies
- Error Codes
- KYC Verification
- Implementing NetNut Residential Proxies in Web Scraping
- Case Study - Scraping Ebay.es with Geo-Location Targeting
- Alternative: ScrapeOps Residential Proxy Aggregator
- Ethical Considerations and Legal Guidelines
- Conclusion
- More Web Scraping Guides
TLDR: How to Integrate NetNut Residential Proxy?
To integrate NetNut Residential Proxy into your Python web scraping project:
-
Sign up for a NetNut account and complete the verification process.
-
To configure your proxy, provide your username, password, server, and port details.
-
In the username variable, you can indicate a specific country and city using the format
username-res-sc-country_state_city
. For example, to target Dallas, Texas, the format would beusername-res_sc-us_texas_dallas
. -
Use the provided code snippet to send requests through the proxy.
import requests
username = 'username-res-any' # or `username-res_sc-us_texas_dallas` to target dallas texas
password = 'password'
server = 'gw.netnut.net'
port = '5959'
# Proxy configuration for both http and https
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
# API URL to check IP and location
url = 'http://ip-api.com/json'
# Making the request
try:
response = requests.get(url, proxies=proxy, timeout=10)
# Checking if the request was successful
if response.status_code == 200:
print(response.text) # Print the response content
else:
print(f"Error: Received status code {response.status_code}")
print(response.text) # Print the response content for debugging
except Exception as e:
print(f"Error: {e}")
In the code snippet above:
-
The username and password for the NetNut proxy are defined. You can use a general username or specify a particular location (e.g., Dallas, Texas).
-
A proxy dictionary is created for both HTTP and HTTPS requests, incorporating the username, password, server, and port.
-
The script sets the URL to an API endpoint (http://ip-api.com/json) that provides IP and location information.
-
A GET request is sent to the API using the specified proxy, with a timeout of 10 seconds.
-
If the request is successful (status code 200), the response content is printed.
Understanding Residential Proxies
Residential proxies serves as intermediaries that route your web traffic through real devices, each associated with a legitimate IP address from an Internet Service Provider (ISP).
Unlike data center proxies that originate from cloud servers, residential proxies offer greater authenticity by mimicking genuine user behavior. This increased realism makes it significantly harder for websites to detect and block scraping activities.
By masking your actual IP address with one from a real residential location, these proxies enable your web scraping requests to appear as if they originate from regular users.
This capability allows you to effortlessly bypass anti-scraping measures, geographic restrictions, and CAPTCHA challenges, making residential proxies an ideal choice for reliable and undetected data collection.
Types of Residential Proxies
There are two main types of residential proxies used in web scraping, each with its own benefits and use cases:
Rotating Residential Proxies
Rotating residential proxies assign a different IP address for each request or after a set period of time. This constant IP rotation reduces the chances of being detected or blocked by websites. T
hey are ideal for large-scale web scraping projects, where you need to gather vast amounts of data without getting flagged. Rotating proxies are particularly useful for scraping highly restricted or heavily monitored sites.
Pros:
- High anonymity: IP changes frequently, reducing the chances of detection.
- Bypass restrictions: Effective for accessing sites with strict anti-scraping measures.
- Ideal for large-scale scraping: Great for handling thousands of requests without bans.
Cons:
- Session instability: Difficult to maintain long-term sessions or logins.
- Reduced speed: Rotating proxies often exhibit slower connection speeds compared to static proxies due to the constant IP switching process.
Static Residential Proxies
Unlike rotating proxies, static residential proxies provide you with a consistent IP address for all your requests.
This gives the impression that all traffic is coming from the same device over time, making it ideal for activities that require a stable identity, such as managing multiple accounts, performing localized testing, or accessing region-specific content.
Static residential proxies are valuable when long-term sessions or consistent connections are needed.
Pros:
- Session stability: Same IP address is retained, making it ideal for long-term sessions.
- Consistent identity: Perfect for localized testing or tasks requiring a stable location.
- Cost-effective for small projects: Affordable when IP rotation isn’t necessary.
Cons:
- Higher detection risk: Repeated use of the same IP increases the chance of blocking.
- Limited scalability: Not suitable for large-scale scraping due to higher risk of bans.
- Geographic limitations: Tied to one location, limiting flexibility for global scraping.
Residential vs. Data Center Proxies
Residential and data center proxies are the two most common types of proxies used for web scraping. Each has its unique advantages and disadvantages, depending on the use case.
Here's a comparison to help you understand how they differ:
Feature | Residential Proxies | Data Center Proxies |
---|---|---|
IP Source | Real devices with IPs from ISPs | Virtual machines in data centers with shared IPs |
Anonymity | High – harder to detect and block due to genuine IPs | Moderate – easily detected and blocked by websites |
Speed | Slower – routes through real user devices | Faster – hosted on high-speed servers |
Cost | More expensive due to the authenticity of the IPs | More affordable because of their virtual nature |
Reliability | Very reliable for bypassing strict anti-scraping measures | Less reliable on heavily monitored websites |
Best Use Case | Scraping restricted or geo-blocked websites | High-volume scraping where detection is less of an issue |
Risk of Blocking | Low – harder to detect and block | High – easily flagged by advanced security systems |
Scalability | Limited by availability of real IPs | Highly scalable due to large pools of virtual IPs |
Use Cases
Residential proxies are highly versatile and can be used in a variety of applications where anonymity, access to geo-blocked content, and avoiding detection are essential.
Here are some common use cases:
Web Scraping
Residential proxies are ideal for web scraping projects that target websites with strong anti-scraping mechanisms. The use of real IP addresses helps bypass detection and allows for smooth data extraction from even the most restrictive websites.
Price Comparison
Online retailers and travel platforms often show different prices based on a user’s location. Residential proxies enable businesses to scrape these websites from various geographic regions, ensuring they get accurate pricing data for competitive analysis.
Ad Verification
Many advertising companies use residential proxies to verify that their ads are displayed correctly and in the right locations. Since these proxies simulate users in different locations, marketers can ensure their campaigns are being delivered as intended.
SEO Monitoring
Residential proxies allow digital marketers to track keyword rankings and search engine results from different locations. This helps them optimize local SEO strategies and ensures accurate ranking data from various regions.
Accessing Geo-Restricted Content
Residential proxies are useful for accessing content that is restricted to specific regions, such as streaming services, localized websites, or regional news sources. By routing traffic through local IPs, you can bypass geo-blocks and access the desired content.
Market Research
Businesses use residential proxies to gather data about competitors, consumer behavior, and market trends without being detected. This helps companies make informed decisions based on real, unfiltered data from different regions.
Social Media Management
Residential proxies help manage multiple social media accounts without triggering platform restrictions. By using different IPs, businesses can run various accounts for marketing, customer engagement, and brand promotion without being flagged for any suspicious activity.
Why Use NetNut Residential Proxies?
NetNut is a top provider in the residential proxy market, known for delivering reliable, high-performance solutions for web scraping, data collection, and online privacy.
Here’s why NetNut is the preferred choice for businesses and professionals alike:
Global Coverage
With over 85 million IP addresses across 200+ countries, NetNut ensures you have access to a vast pool of global IPs. This allows you to easily bypass geo-restrictions and access data from anywhere in the world, ensuring wide-ranging and accurate data collection.
Ultra-Fast Speeds
NetNut’s residential proxies are optimized for speed. With ultra-fast data transfers and no bandwidth throttling, you can scrape and gather data without any delays, making your projects run smoother and more efficiently.
High-Level Security
NetNut provides top-tier security with encrypted data transfers, anonymous IP addresses, and a strict no-logging policy. Whether you're managing sensitive data or performing large-scale operations, your online activities remain private and secure.
Scalability and Flexibility
With NetNut, you can scale your operations without limitations. Whether you're a startup or an enterprise, their network is built to grow with your needs. Plus, the real-time static proxy control and customizable settings allow you to tailor your proxy performance for maximum efficiency.
Easy Integration & Support
NetNut’s residential proxies are easy to integrate with popular programming languages and business tools. Plus, their dedicated account managers provide hands-on support, including full integration assistance and remote sessions, so you can hit the ground running without any technical barriers.
NetNut Residential Proxy Pricing
NetNut offers various pricing plans for both rotating and static residential proxies, catering to different usage needs. You can opt for either monthly or annual billing options, with charges based on bandwidth usage rather than per IP address or concurrency.
All plans include high-speed, reliable IPs that effectively bypass restrictions, along with additional features such as city/state selection, API access, IP whitelisting, and support from a dedicated account manager.
Residential Proxies - Monthly Billing Plan
Plan | Price per GB | Total Data | Monthly Cost |
---|---|---|---|
Starter | $7.07/GB | 14GB | $99 |
Advanced | $6.94/GB | 36GB | $250 |
Production | $6.65/GB | 75GB | $499 |
Semi-Pro | $5.70/GB | 175GB | $999 |
Professional | $4.99/GB | 400GB | $1,999 |
Master | $3.75/GB | 1TB | $3,750 |
Residential Proxies - Annual Billing Plan
Plan | Price per GB | Total Data | Annual Cost | Monthly Cost |
---|---|---|---|---|
Starter | $6.00/GB | 14GB | $1,008 | $84 |
Advanced | $5.91/GB | 36GB | $2,556 | $213 |
Production | $5.65/GB | 75GB | $5,088 | $424 |
Semi-Pro | $4.85/GB | 175GB | $10,188 | $849 |
Professional | $4.24/GB | 400GB | $20,388 | $1,699 |
Master | $3.18/GB | 1TB | $38,256 | $3,188 |
Static Proxies - Monthly Billing Plan
Plan | Price per GB | Total Data | Monthly Cost |
---|---|---|---|
Starter | $14.40/GB | 7GB | $99 |
Advanced | $12.50/GB | 20GB | $250 |
Production | $9.98/GB | 50GB | $499 |
Semi-Pro | $7.99/GB | 125GB | $999 |
Professional | $5.71/GB | 350GB | $1,999 |
Master | $4.50/GB | 1TB | $4,500 |
Static Proxies - Annual Billing Plan
Plan | Price per GB | Total Data | Annual Cost | Monthly Cost |
---|---|---|---|---|
Starter | $12.00/GB | 7GB | $1,008 | $84 |
Advanced | $10.65/GB | 20GB | $2,556 | $213 |
Production | $8.48/GB | 50GB | $5,088 | $424 |
Semi-Pro | $6.79/GB | 125GB | $10,188 | $849 |
Professional | $4.85/GB | 350GB | $20,388 | $1,699 |
Master | $3.82/GB | 1TB | $45,900 | $3,825 |
Comparison to Other Providers
NetNut's pricing is on the higher side for smaller plans, starting at $7.07/GB, which is above the $6-8 per GB range typically considered expensive.
For larger plans, it becomes more competitive, with prices dropping to $3.75/GB. Generally, it's more affordable for higher data usage but expensive for smaller plans.
Feel free to check out our ScrapeOps Proxy Comparison page to see how different residential proxy providers compare on pricing and features, so you can make a more informed decision about which one suits your needs best.
Setting Up NetNut Residential Proxies
To set up NetNut residential proxies, start by creating and verifying your account. Once that's complete, choose a proxy plan that suits your needs. From there, configure your proxies, set up authentication, and integrate them into your web scraping scripts.
Let’s walk through each step to make this process easy. Keep in mind that, as of this writing, NetNut only offers free trials to companies or organizations.
Creating a NetNut Account
To start using NetNut residential proxies, here’s how to create and activate your account:
-
Visit the NetNut Registration Page and fill out the registration form with your first name, last name, and social account details.
-
After creating your account and verifying your email, you’ll be redirected to a page that prompts you to contact NetNut's sales department. They’ll help activate your account so you can qualify for a free trial or purchase a plan.
- Once your account is activated, log in to access the dashboard and navigate to the
Products
tab on your dashboard.
-
Choose a plan that best fits your requirements.
-
After selecting a plan, you’ll be redirected to the billing section. Fill in your billing details and click “Confirm.”
- In the checkout section, enter your card details and click "Subscribe Now" to confirm your purchase.
- Your purchased plan should now be visible on your dashboard, showing your active status and available data.
Note: To fully access NetNut’s proxies, you must qualify for a free trial or purchase a plan.
Creating and Configuring Proxies
In this section, you'll learn how to set up your NetNut residential proxies. This includes generating your proxy password and integrating it with other details like the username, server, and port.
Let’s go through each step to make the integration process simple and straightforward.
Generating Proxy Password:
- To generate a proxy password, navigate to the
Settings
Tab and click onAccount
.
-
Scroll down to the bottom of the page to find the proxy details.
-
Click on
Generate Password
, then clickUpdate
.
- Make sure to note down your proxy Username and Password, as you'll need them to configure your proxy. It's also important to note that it may take around 10-15 minutes for the generated proxy password to take effect.
Proxy Integration
Once your password is set, you’re ready to use your NetNut credentials and proxy server details to integrate into your web scraping scripts.
Here are the details you’ll need:
- Username: Format should follow
username-res-any
- Password: The proxy-password you generated in the previous section
- Server:
gw.netnut.net
- Port:
5959
For example, if you're using Python with the requests
library, you can configure the proxy like this:
import requests
import pprint
username = 'username-res-any'
password = 'proxy-password'
server = 'gw.netnut.net'
port = '5959'
# Proxy configuration for both http and https
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
# API URL to check IP and location
url = 'http://ip-api.com/json'
# Pretty printer instance
pp = pprint.PrettyPrinter(indent=4)
# Making the request
try:
response = requests.get(url, proxies=proxy, timeout=10)
# Checking if the request was successful
if response.status_code == 200:
pp.pprint(response.json()) # Pretty print the JSON response
else:
print(f"Error: Received status code {response.status_code}")
pp.pprint(response.text) # Pretty print the response content for debugging
except Exception as e:
print(f"Error: {e}")
In the code above:
-
The script sets up the proxy using the provided
username
,password
,server
, andport
values, specifically formatted for use withrequests
. It applies these proxy settings to both HTTP and HTTPS requests. -
The target URL (
http://ip-api.com/json
) is an API that returns the IP address and geolocation data for the incoming request, allowing you to verify that the proxy is active and routing traffic as expected.
This setup is useful for validating the proxy's functionality and IP location before integrating it into larger web scraping tasks.
The screenshot below shows the results after running the script above:
Authentication
NetNut offers two primary authentication methods for using residential proxies: Username & Password and IP Whitelisting.
These options help you manage and control proxy access, allowing for secure and efficient use. Below, we’ll review each method with examples to help you integrate them into your web scraping projects.
1. Username & Password Authentication
This method uses a unique username and a proxy password, provided by NetNut, to authenticate your requests.
To set up this method, make sure you’ve generated your proxy password and have saved your credentials securely. Here’s a sample code to implement this using python request:
import requests
# Define the proxy server and credentials
username = 'username-res-any' # username used upon registration
password = 'proxy-password' # proxy password generated from your account settings
server = 'gw.netnut.net' # Netnut's Proxy URL
port = '5959' # Netnut's Proxy Port
# Configure the proxies dictionary
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
try:
response = requests.get(url, proxies=proxy, timeout=10)
# Checking if the request was successful
if response.status_code == 200:
pp.pprint(response.json()) # Pretty print the JSON response
else:
print(f"Error: Received status code {response.status_code}")
pp.pprint(response.text) # Pretty print the response content for debugging
except Exception as e:
print(f"Error: {e}")
2. IP Whitelisting Authentication
You can connect to NetNut's residential proxies via whitelisted IPs without needing a username or password. To set up your whitelist, follow these steps:
-
Log in to your Dashboard
-
Go to Settings and select Whitelist IP to open the section for managing IP whitelisting.
-
Input the IP addresses you want to whitelist. Make sure each IP is a direct IP (not from a proxy or VPN service) and in the IPv4 format.
-
After entering the IPs, submit the list. NetNut will then send you the ports associated with the countries you need access to.
You can adjust the whitelist as needed, adding or removing IPs whenever necessary. Once set up, your approved IPs will allow direct access to NetNut’s proxies, skipping the need for login credentials.
Once you receive your approved ports you can authenticate your proxies using the following code:
import requests
# sample list of proxies
proxies_list = [
"gw.netnut.net:approved_port",
"gw.netnut.net:approved_port",
]
# Test the proxies by making a request
for proxy in proxies_list:
proxy_dict = {
"http": f"http://{proxy}",
"https": f"http://{proxy}"
}
try:
response = requests.get("http://ip-api.com/json", proxies=proxy_dict, timeout=10)
data = response.json()
print(f"Proxy: {proxy}, Country: {data['country']}, IP: {data['query']}")
except Exception as e:
print(f"Failed to connect using {proxy}: {e}")
Basic Request Using NetNut Residential Proxies
To use NetNut residential proxies for making requests, you should first generate a proxy password as described in the previous section of this article and save it along with your proxy username, server, and port information.
Making a Request
-
Set Up Your Proxy Configuration: Start by defining your proxy details. You’ll need to include your username, password, server address, and port. Here's an example of how to set this up in Python using the
requests
library:import requests
# Define proxy credentials
username = 'your_username-res-any' # Replace with your proxy username
password = 'your_password' # Replace with your proxy password
server = 'gw.netnut.net'
port = '5959'
# Create proxy configuration
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
# Making a request
response = requests.get('http://example.com', proxies=proxy)
# Check response status
if response.status_code == 200:
print('Request was successful!')
print(response.content)
else:
print('Request failed with status code:', response.status_code)
The screenshot below demonstrates that we successfully made a basic request using our residential proxy.
Country Geotargeting
Country geotargeting is the ability to select or assign IP addresses based on a specific country.
This allows users to access the internet as if they are located in a particular country. Businesses or individuals often use this feature for various purposes, such as:
- Content Access: Accessing geo-restricted content that is only available to users in specific countries.
- Ad Verification: Ensuring that ads are correctly displayed to users in different countries, as they might be shown differently based on geographic location.
- SEO and Market Research: Monitoring search engine rankings, market trends, or competitor activity in specific regions. Price Scraping: Checking how prices and deals vary across countries.
- Testing and Localization: Testing websites, applications, and services to ensure proper localization (e.g., currency, language, or user experience differences) based on the region.
- Fraud Prevention: Detecting fraudulent activities by comparing IP location with the declared location of the user.
NetNut offers residential proxies with access to over 85 million IP addresses across more than 200 countries, enabling precise geotargeting capabilities.This feature allows users to access region-specific content as if they were physically present there.
Top 10 Countries Supported by NetNut Residential Proxies
Here are the top 10 countries where NetNut's residential proxies are most supported, based on proxy pool size, connection stability, and reliability.
Country | Number of IPs |
---|---|
USA | 2,316,791 |
India | 1,087,258 |
France | 967,890 |
UK | 956,262 |
Germany | 864,182 |
Poland | 712,261 |
Italy | 696,832 |
Mexico | 556,454 |
Canada | 588,437 |
Argentina | 580,293 |
To target specific countries, replace any
in your username
configuration with the desired country code. For example, to target the U.S., use the following configuration:
import requests
username = 'username-res-US' # replaced "any" with "US"
password = 'password'
server = 'gw.netnut.net'
port = '5959'
# Create proxy configuration
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
try:
response = requests.get("http://ip-api.com/json", proxies=proxy, timeout=10)
data = response.json()
print(f"Country: {data['country']}, IP: {data['query']}")
except Exception as e:
print(f"Failed to connect using : {e}")
In this code sample:
- We imported the necessary libraries: requests and BeautifulSoup.
- Defined the proxy credentials, which include the username, password, server, and port for NetNut.
- Set up the proxy settings for both HTTP and HTTPS.
- The script sends a GET request to the http://ip-api.com/json endpoint using the previously defined proxy settings.
- Extracted the returned IP address and country from the response using ipinfo.io.
- The results confirm the output, displaying both the IP address and the corresponding country.
City Geotargeting
City geotargeting is a more granular form of geotargeting that allows users to select or assign IP addresses based on specific cities, rather than just countries.
This enables users to browse the internet or access services as if they are located in a particular city. City geotargeting is valuable for more localized and precise needs, such as:
- Localized Ad Testing: Verifying how ads appear to users in a specific city, as ad campaigns can vary not just by country but by city as well.
- SEO and Local Search Optimization: Monitoring how websites rank in search engines or how competitors perform in specific cities, which can be crucial for local businesses.
- Local Market Research: Gathering insights into user behavior, preferences, and trends in specific urban areas, which can differ widely from one city to another.
- Content Customization: Testing localized content for websites or apps, ensuring that location-based features such as weather, event information, or city-specific promotions are functioning correctly.
- Price and Availability Monitoring: Observing how prices, deals, or product availability might vary between cities.
- User Experience Testing: Ensuring that websites or applications provide a consistent user experience across different cities, especially for services that depend on localized data (e.g., delivery services, ride-sharing apps).
- Geographic Fraud Detection: Comparing a user’s claimed location with their actual city-level IP address to detect potential fraud or security issues.
NetNut offers advanced city-level geotargeting through its extensive network of residential proxies, enabling users to select IP addresses from specific cities.
To target specific cities, replace any
in your username
configuration with res_sc-countrycode_state_city
. For example, to target Dallas, Texas, in the U.S., use the following configuration in Python:
import requests
username = 'username-res_sc-us_texas_dallas'
password = 'password'
server = 'gw.netnut.net'
port = '5959'
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
try:
response = requests.get("http://ip-api.com/json", proxies=proxy, timeout=10)
data = response.json()
print(f"Country: {data['country']}, IP: {data['query']}, State: {data['regionName']}, City: {data['city']}")
except Exception as e:
print(f"Failed to connect using : {e}")
In the code above, we:
- Configured the proxy connection by replacing the default username with the specific details required to target Dallas, Texas in the U.S. i.e
username-res_sc-us_texas_dallas
- Created both http and https proxy configurations using the provided credentials.
- Sent a GET request to http://ip-api.com/json to retrieve the location data and IP of the connection, verifying that the proxy is functioning and accurately geotargeting the selected city.
- Extracted and printed the country, IP address, state, and city from the API response to confirm the connection’s details.
Below is a sample output from running the code:
How To Use Static Proxies
NetNut allows you to maintain a static session by adding a session ID to the username
configuration. This ensures that requests consistently use the same IP address.
Here’s an example configuration:
Username Format: username = username-res-us-sid-<any_number>
The session ID can be any number up to 8 digits (between 1 and 9), preferably 7–8 digits. Changing the session ID will rotate the IP to a new static one.
Making Requests Using Static Residential Proxies
To implement a static session, add a session ID in your username. The following Python script demonstrates how to hold a static session, allowing all requests to come from the same IP:
import requests
username = 'username-res-us-sid-1122445'
password = 'password'
server = 'gw.netnut.net'
port = '5959'
# Create proxy configuration
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
try:
# Send multiple requests to check IP consistency
for _ in range(5):
response = requests.get("http://ip-api.com/json", proxies=proxy, timeout=10)
data = response.json()
print(f"Country: {data['country']}, IP: {data['query']}, State: {data['regionName']}, City: {data['city']}")
except Exception as e:
print(f"Failed to connect: {e}")
In the code above:
- We set a session ID in the username (username-res-us-sid-1121445), ensuring that each request routes through the same static IP address. This session ID keeps the IP constant across multiple requests.
- Next, the proxy credentials (username, password, server, and port) are set up in the proxy dictionary, applying the proxy to both http and https requests.
- A GET request is made to http://ip-api.com/json, which retrieves IP location data for verification purposes.
- To confirm the IP’s consistency, a for loop repeats the request five times, displaying the IP address, country, state, and city.
The output below verifies that the IP remains unchanged due to the static session configuration.
Error Codes
When using NetNut residential proxies, understanding common error codes can help troubleshoot issues that may arise during requests.
Here’s a breakdown of frequently encountered error codes, their meanings, and some recommended solutions.
Error Code | Meaning | Description | Potential Solution |
---|---|---|---|
400 | Bad Request | Indicates an invalid request format or missing host in the URL. | Verify the URL format and ensure correct request syntax. |
403 | Access Denied | Access restricted by NetNut, often due to content limitations. | Review NetNut’s content policy or contact support for further guidance. |
407 | Proxy Authentication Error | Occurs due to invalid username or password during proxy authentication. | Double-check that the proxy credentials (username and password) are correct. |
490 | Invalid Access Point | Indicates no available IPs for the selected country or group, or a non-existent group. | Ensure the access point, location, and group settings are accurate, and confirm IP availability in that region. |
500 | Internal Server Error | Server-side issue originating from NetNut, generally temporary. | Retry the request after a brief delay, as the issue may resolve automatically. |
502 | Bad Gateway | Indicates that the assigned IP is no longer available. | Wait for a new IP assignment or change the session ID to receive a new IP instantly. |
503 | Service Unavailable | Temporary proxy error or connection refused by the target website. | Wait and retry the request later, as the target may be temporarily denying access. |
522 | Connection Timeout | Proxy server did not receive a response from the target in the expected timeframe. | Retry the request, as network latency or server delay may be causing the timeout. |
KYC (Know-Your-Customer) Verification
To begin using NetNut’s proxies, a Know-Your-Customer (KYC) verification is required. This process typically involves contacting their sales team through Telegram, WhatsApp, Skype, or email.
During verification, you’ll be asked to provide basic information, such as your email address, details about your intended use case, and an estimate of your monthly data requirements in GB.
Once these details are reviewed and approved, NetNut will send a confirmation email indicating that your account is active and ready for use.
Implementing NetNut Residential Proxies in Web Scraping
NetNut residential proxies are compatible with various web scraping libraries and frameworks, making them versatile for web scraping and automation.
In this section we'll look at some the basics of setting up these proxies with different tools, including examples of U.S. geotargeting and automatic proxy rotation.
Python Requests
Setting up NetNut proxies with Python Requests is straightforward. Here's a quick guide on how to set it up:
import requests
import time
username = 'username-res-us'
password = 'password'
server = 'gw.netnut.net'
port = '5959'
# Create proxy configuration
proxy = {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}',
}
# Make five requests to demonstrate proxy rotation
for i in range(5):
try:
# Make a request
response = requests.get("http://ip-api.com/json", proxies=proxy, timeout=10)
data = response.json()
# Print the response (country, IP, state, city)
print(f"Request {i+1}: Country: {data['country']}, IP: {data['query']}, State: {data['regionName']}, City: {data['city']}")
# Pause briefly between requests to mimic real-world usage
time.sleep(2)
except Exception as e:
print(f"Request {i+1} failed: {e}")
In the script above, we:
-
Defined proxy credentials with the username, password, server, and port for NetNut.
-
Created a dictionary that includes both http and https protocols for the proxy configuration.
-
Used a loop to make 5 requests to http://ip-api.com/json, indicating that each request utilizes a different IP from the proxy service.
The screenshot below shows that each request utilizes a different IP address, effectively demonstrating the proxy rotation.
Python Selenium
Integrating proxies with Selenium enhances your web scraping, testing, and automation projects significantly.
Here’s a concise guide on how to set up and utilize NetNut proxies with Selenium.
pip install --upgrade undetected_chromedriver
Now, configure the WebDriver as follows:
import undetected_chromedriver as uc
from selenium.webdriver.common.by import By
import time
import json
# Proxy configuration
proxy_host = "gw.netnut.net"
proxy_port = "5959"
proxy_username = "username-res-us"
proxy_password = "password"
proxy = f"{proxy_host}:{proxy_port}"
# Initialize Chrome options with the proxy server
chrome_options = uc.ChromeOptions()
chrome_options.add_argument(f'--proxy-server=http://{proxy}')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
# Initialize undetected_chromedriver with configured options
driver = uc.Chrome(options=chrome_options)
# Navigate to ipinfo.io to verify the IP and location details
driver.get("https://ipinfo.io/json")
time.sleep(5) # Wait for the page to load
# Retrieve the JSON response from the page
page_source = driver.find_element(By.TAG_NAME, "pre").text
data = json.loads(page_source)
# Print IP information
print("IP Information from ipinfo.io (using proxy):")
for key, value in data.items():
print(f"{key}: {value}")
# Close the browser
driver.quit()
In the script above, we:
-
Configured the Chrome options to use a proxy server for authentication by specifying the proxy host, port, username, and password.
-
Navigated to the ipinfo.io website to retrieve IP and location information in JSON format.
-
Retrieved and parsed the JSON response from the page to access the IP information.
-
Printed the IP information retrieved from the API in a formatted manner.
Below is an example of the result you may see after running the code.
Python Scrapy
In this section, we’ll see how to set up NetNut's residential proxy with Scrapy.
- To begin, create a new Scrapy project by opening your terminal and executing the following command:
scrapy startproject <project_name>
This command will create a new folder with your project name and set up the basic configuration for a Scrapy project inside it
-
Next, set up a Scrapy spider to work with the NetNut proxy. Head over to your project_name\spiders folder and create a new file, like
NetNut.py
-
In this file, you'll want to create a new Scrapy spider by making a Python class that extends scrapy.Spider. Check out this example of how your code could look:
import scrapy
import json
class NetNutScrapyExampleSpider(scrapy.Spider):
name = "NetNutScrapyExample"
def start_requests(self):
# URL to be accessed with different proxies
url = "http://ip-api.com/json"
username = 'username-res-us'
password = 'password'
server = 'gw.netnut.net'
port = '5959'
# Create the proxy URL
proxy = f"http://{username}:{password}@{server}:{port}"
# Create requests with the proxy
request = scrapy.Request(url=url, callback=self.parse, meta={'proxy': proxy})
request.dont_filter = True
yield request
def parse(self, response):
try:
# Decode JSON response
data = response.json()
ip = data.get("query")
country = data.get("country")
city = data.get("city")
region = data.get("regionName")
isp = data.get("isp")
org = data.get("org")
# Log the details
self.log(f"Proxy Used: {response.meta['proxy']}")
self.log(f"IP Address: {ip}, Country: {country}, City: {city}, Region: {region}, ISP: {isp}, Org: {org}")
self.log(f"Status Code: {response.status}")
except json.JSONDecodeError:
self.log(f"Failed to decode JSON from {response.url}")
self.log(f"Response Text: {response.text[:100]}")
-
Go ahead and open the
project_name/settings.py
file and changeROBOTSTXT_OBEY
toFalse
. -
Next, head over to your terminal, go to the project directory (where the scrapy.cfg file is), and run this command to kick off the spider:
scrapy crawl spider_name
Make sure to swap out spider_name with the actual name you gave your spider class, here, it's NetNutScrapyExample
.
Once you run it, you should see results that look a lot like the screenshot below.
Now, let’s take a closer look at the important parts of this code.:
-
We set up a basic Scrapy spider class which builds on scrapy.Spider, laying out the fundamental framework for our scraping tasks
-
In the start_requests method, we create a request to our target URL. This method initiates the request to the specified URL.
-
We set up our proxy by defining variables for the username, password, server, and port.
-
Now, we create a Scrapy request with
request = scrapy.Request(url=url, callback=self.parse, meta={'proxy': proxy})
-
Finally, we send the request off with yield request. This hands the request over to Scrapy for processing.
NodeJs Puppeteer
To connect NetNut residential proxies with Puppeteer for browser automation, follow these steps:
- First, make sure you have Puppeteer installed. You can install it by running the command:
npm install puppeteer
- Configure your Puppeteer settings to use NetNut proxies.
const puppeteer = require('puppeteer');
async function scrapeData() {
// Proxy details
const proxyHost = 'gw.netnut.net';
const proxyPort = '5959';
const proxyUsername = 'username-res-us'; // Your proxy username
const proxyPassword = 'password'; // Your proxy password
// Launch a browser instance with the proxy configuration
const browser = await puppeteer.launch({
args: [
`--proxy-server=${proxyHost}:${proxyPort}`,
'--no-sandbox',
'--disable-setuid-sandbox'
],
headless: false // Set to true for production
});
// Create a new page
const page = await browser.newPage();
// Set up proxy authentication
await page.authenticate({
username: proxyUsername,
password: proxyPassword
});
try {
// Visit the target page
await page.goto('http://ip-api.com/json', { timeout: 60000 });
// Extract the IP the request comes from and print it
const body = await page.waitForSelector('body');
const ip = await body.evaluate(el => el.textContent);
console.log(ip); // { "origin": "your-proxy-ip" }
} catch (error) {
console.error('An error occurred:', error);
} finally {
await browser.close();
}
}
scrapeData();
- Save the script as a file, such as
netnut.js
, and execute it with Node.js.
node netnut.js
Let's break down the key parts of this code:
-
We import the Puppeteer library to control a Chromium browser.
-
We define the proxy server's host, port, username, and password to authenticate the connection.
-
The code navigates to 'http://ip-api.com/json', waits for the page to load, and extracts the IP address from the response.
Here’s an example of the result you might see after running the code.
NodeJs Playwright
To set up NetNut residential proxies with Playwright for browser automation, follow these steps:
- First, install Playwright using the command:
npm install playwright
npx playwright install
- Configure your Playwright settings to include NetNut proxies.
const { chromium } = require('playwright');
async function scrapeData() {
// Proxy details
const proxyHost = 'gw.netnut.net';
const proxyPort = '5959';
const proxyUsername = 'username-res-us';
const proxyPassword = 'password';
const proxyServer = `http://${proxyHost}:${proxyPort}`;
try {
// Launch a browser instance with the proxy configuration
const browser = await chromium.launch({
proxy: {
server: proxyServer,
username: proxyUsername,
password: proxyPassword
},
headless: false // Set to true for production
});
// Create a new context and page
const context = await browser.newContext();
const page = await context.newPage();
// Visit the target page
await page.goto('http://ip-api.com/json', { timeout: 60000 });
// Extract the IP information and print it
const content = await page.content();
const bodyText = await page.evaluate(() => document.body.innerText);
console.log('Page content:', content);
console.log('Body text:', bodyText);
// Parse the JSON response
try {
const ipInfo = JSON.parse(bodyText);
console.log('IP Information:', ipInfo);
} catch (parseError) {
console.error('Error parsing JSON:', parseError);
}
await browser.close();
} catch (error) {
console.error('An error occurred:', error);
if (error.message.includes('NS_ERROR_PROXY_CONNECTION_REFUSED')) {
console.error('Proxy connection refused. Please check your proxy settings and ensure the proxy server is accessible.');
} else if (error.message.includes('ERR_PROXY_CONNECTION_FAILED')) {
console.error('Proxy connection failed. This might be due to incorrect proxy credentials or network issues.');
}
}
}
scrapeData();
- Save the script as netnut.js and run it with Node.js.
node netnut.js
Here’s a breakdown of the key parts of this code:
-
First, we define the NetNut proxy details, including host, port, username, and password, and assemble them into a proxy server URL.
-
Then, we start a Chromium browser instance, connect through the specified proxy, and run in visible mode for testing (headless: true for production).
-
After that, we create a browsing context and open a new page within the browser.
-
We then navigate to http://ip-api.com/json to fetch IP information, with a 60-second timeout.
-
Next, we extract and log the page content and specific body text containing IP details, parsing the JSON to get readable IP information.
Below is an example of the result you might see after running the code.
Case Study - Scraping Amazon.es with Geo-Location Targeting
In this section, we’ll demonstrate how to scrape product pages on Amazon.es using NetNut's residential proxies to enable geo-targeting for price and availability data from Spain and Portugal.
You’ll see how product details, like prices or availability, can vary between these two locations.
Setup
- First, we'll configure our proxy credentials to use both locations of Spain and Portugal, we can do this using the python code below
import requests
from bs4 import BeautifulSoup
def get_proxy_config(country_code):
username_map = {
'es': 'username-res-es', # Spain uses US proxy
'pt': 'username-res-pt' # Portugal
}
password = 'password'
server = 'gw.netnut.net'
port = '5959'
username = username_map.get(country_code.lower())
if not username:
raise ValueError(f"Unsupported country code: {country_code}")
return {
'http': f'http://{username}:{password}@{server}:{port}',
'https': f'http://{username}:{password}@{server}:{port}'
}
def get_headers():
return {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'https://www.amazon.es/'
}
def scrape_amazon(url, country_code):
try:
proxy = get_proxy_config(country_code)
headers = get_headers()
response = requests.get(url, headers=headers, proxies=proxy, timeout=10)
response.raise_for_status() # Raises an HTTPError for bad responses
soup = BeautifulSoup(response.content, 'html.parser')
title = soup.select_one('#productTitle').text.strip() if soup.select_one('#productTitle') else "Title not found"
price = soup.select_one('.a-price-whole').text.strip() if soup.select_one('.a-price-whole') else "Price not found"
availability = soup.select_one('#availability span').text.strip() if soup.select_one('#availability span') else "Availability not found"
return {
'title': title,
'price': price,
'availability': availability
}
except requests.RequestException as e:
return {'error': f"Request failed: {str(e)}"}
except Exception as e:
return {'error': f"An error occurred: {str(e)}"}
def main():
url = "https://www.amazon.es/Taurus-WC12T-termoel%C3%A9ctrica-Aislamiento-Temperatura/dp/B093GXXKRL/ref=lp_14565165031_1_2"
countries = ['es', 'pt']
for country in countries:
print(f"\nResults for {country.upper()}:")
result = scrape_amazon(url, country)
for key, value in result.items():
print(f"{key.capitalize()}: {value}")
if __name__ == "__main__":
main()
From the code above:
-
We created a
get_proxy_config
function that selects proxy settings based on the country code: Spain uses a Spanish proxy (es), and Portugal uses a Portuguese proxy (pt). -
In
get_headers
, we defined browser headers, like the User-Agent and language preferences, to make the request appear as if it’s coming from a real browser. -
The main scraping function,
scrape_amazon
, handles accessing the Amazon product page using the proxy and headers configured for the specified country. -
In
scrape_amazon
, we also set up the proxy and headers for each request to fetch the page from either Spain or Portugal based on the input. -
The function then sends a GET request to Amazon with a timeout of 10 seconds to prevent long waits.
-
With BeautifulSoup, the function extracts key elements, such as the
product title
,price
, andavailability
. -
Finally, the main function runs
scrape_amazon
for each country and prints the product data for both Spain and Portugal, showing a comparison of what's available based on location.
Results and Analysis
When we execute this script, the output may look something like this:
We see a clear difference in the product's price when accessed from Spanish and Portuguese IPs. The price is €219 from Spain and €222 from Portugal, indicating a small but noteworthy price variation based on geographical location.
This illustrates how e-commerce platforms may implement different pricing strategies depending on the user's location.
By utilizing IP addresses from both Spain and Portugal, businesses and researchers can obtain important insights into how pricing varies across markets.
Tips for Troubleshooting Common Issues
Here are some tips for troubleshooting common issues when using residential proxies:
-
Ensure that the username, password, and other credentials are correct. Double-check for typos.
-
Make sure the proxy host, port, and type (HTTP/HTTPS) are properly configured in your application.
-
Use tools like curl or online proxy testers to confirm that the proxy is operational and can connect to the internet.
-
If you're experiencing connection issues, verify if the proxy IP has been blocked by the target website. Switch to a different proxy if needed.
-
If you're making multiple requests, consider rotating proxies to avoid being flagged or rate-limited by the target site.
Alternative: ScrapeOps Residential Proxy Aggregator
For a budget-friendly web scraping solution, the ScrapeOps Residential Proxy Aggregator serves as an excellent alternative to conventional proxy providers.
This service compiles proxies from various sources, granting you access to an extensive range of IP addresses while ensuring exceptional flexibility and dependability.
For full setup instructions and guidance on using the ScrapeOps Residential Proxy Aggregator, check out the ScrapeOps Documentation.
Why Use ScrapeOps Residential Proxy Aggregator?
-
All-in-One Access: ScrapeOps Residential Proxy Aggregator combines proxies from over 20 top providers, including BrightData, NetNut, anyIP, Oxylabs, IPRoyal, Asocks, Smartproxy, Geonode, HydraProxy, Infatica, WTFProxy, Froxy, and more. This allows you to work with a vast pool of IP addresses from a single, convenient proxy port.
-
Automatic Proxy Selection: ScrapeOps automatically picks the best-performing and most cost-effective proxy provider for each request, optimizing speed and reliability without manual input.
-
Cost Efficiency: ScrapeOps’ system ensures you never overpay by choosing the most affordable proxy with the desired performance. They also offer a price match guarantee if you find a better deal elsewhere.
-
Location Flexibility: ScrapeOps offers IPs from multiple countries, allowing users to geo-target and access region-specific data seamlessly, which is essential for competitive market analysis.
-
Anti-Ban Protection: With automated proxy rotation and switching, ScrapeOps minimizes risks of blocks, CAPTCHAs, and bans, keeping scraping sessions uninterrupted.
-
Flexible Pricing: ScrapeOps provides various pricing tiers, starting with a free trial that includes 500MB of bandwidth. Plans scale with usage, catering to both small and large scraping projects.
-
Continuous Optimization: They continuously monitor and adjust based on proxy performance to use the best possible proxies for each request.
-
Easy Integration: The aggregator is compatible with HTTP/SOCKS5, and it’s easy to integrate with popular programming languages like Python, Node.js, Java, and more.
Using ScrapeOps Residential Proxy Aggregator with Python Requests
Here's an example of how to use the ScrapeOps Residential Proxy Aggregator with Python's requests library:
import requests
api_key = 'YOUR_API_KEY'
target_url = 'http://httpbin.org/ip'
proxy_url = f'http://scrapeops:{api_key}@residential-proxy.scrapeops.io:8181'
proxies = {
'http': proxy_url,
'https': proxy_url,
}
response = requests.get(
url=target_url,
proxies=proxies,
timeout=120,
)
print('Body:', response.content)
The code above shows an example of how to send a URL request through the ScrapeOps Proxy port. ScrapeOps will manage proxy selection and rotation automatically—simply send the URL you'd like to scrape.
Check out the free trial with 500MB of Bandwidth Credits here. No credit card required.
Ethical Considerations and Legal Guidelines
Web scraping, while a powerful tool for data extraction, raises several ethical and legal concerns, especially when conducted using residential proxies. Here are some key considerations:
Terms of Service Compliance
Always review the terms of service (ToS) of the website being scraped. Many sites explicitly prohibit scraping or automated data collection. Ignoring these terms can result in account bans or legal consequences.
User Privacy
Scraping data that involves personal information (such as emails, contact details, or any identifiable data) without consent can infringe on privacy rights. This is especially important in jurisdictions with strict data protection laws (e.g., GDPR in Europe).
Impact on Website Performance
Excessive scraping can put a strain on a website's server resources, affecting its performance and user experience. Ethical scraping practices involve implementing polite scraping techniques, such as respecting rate limits and minimizing requests.
Transparency and Accountability
If applicable, maintain transparency about your scraping practices. This could involve disclosing scraping activities to relevant stakeholders or ensuring compliance with internal ethical guidelines.
Conclusion
With residential proxies, you can seamlessly manage web scraping tasks while preserving your anonymity and scaling projects efficiently. These proxies allow you to circumvent IP bans and geo-restrictions, facilitating smoother operations and broader access to various data sources.
By employing this approach, you enhance your scraping success and maintain compliance with ethical standards, ensuring your competitiveness in the ever-evolving data landscape.
More Python Web Scraping Guides
For more web scraping tutorials, check out the Python Web Scraping Playbook along with these useful guides: