Smartproxy Datacenter Proxies: Web Scraping Guide

• TLDR: How To Integrate Smartproxy Datacenter Proxy
• Understanding Datacenter Proxies
• Why Use Datacenter Proxies?
• Datacenter Proxy Pricing
• Setting Up Smartproxy Datacenter Proxies
• Authentication
• Basic Request Using Smartproxy Datacenter Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing Smartproxy Datacenter Proxies
• Case Study: Scraping The Guardian
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

TLDR: How to Integrate Smartproxy Datacenter Proxy?

import requests
username = "your-username"password = "your-password"port = 10000proxy_url = f"http://{username}:{password}@dc.smartproxy.com:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

• First, we setup our configuration variables: username, password, port.
• Next, we create a proxy_url using the configuration from above.
• We create a dict object to hold our http and https proxies. We then assign both of them to our proxy_url.
• We finally print our result.text to the terminal, revealing the location information of our proxy.

Understanding Datacenter Proxies

Datacenter vs Residential

Datacenter

• Price: No matter which provider you're choosing, datacenter proxies tend to be very affordable. When using residential proxies, it's not uncommon to pay up to $8/GB.
• Speed: When you use a proxy like this, it gets hosted inside an actual datacenter. Datacenters usually use the best of the best internet connections and hardware. As long as you're not getting blocked, datacenter proxies offer unparalleled performance.
• Availability: Datacenters are huge. Each machine in a datacenter typically has its own IP address. This gives us an enormous proxy pool to work with.

• Blocking: As we mentioned earlier, some webites block all datacenter IPs by default. This can cause unending headache when scraping more difficult sites.
• Less Geotargeting Support: Most datacenter proxies do give us the option to use geotargeting. However, you still show up with a datacenter IP no matter which country you choose. When we use geotargeting with datacenter proxies, we often don't get the pinpoint accuracy that we can achieve with residential proxies.
• Less Anonymity: When you use a datacenter proxy, your location is always going to show inside a datacenter. This doesn't give you the ability to blend in with normal residential and mobile users.

Residential

• Anonymity: When we use a residential proxy, we get an actual IP address assigned to an actual residential device. When dealing with more difficult sites, this is incredibly important.
• Better Access: When using a residential IP address, it doesn't matter if your target site blocks datacenter IP addresses. Your traffic is coming from somebody's real device in a real home.

• Price: When using a residential proxy service, it's not uncommon to pay between $5 and $8 per GB! Datacenter proxies, on the other hand, can be 10x cheaper!
• Speed: As we mentioned earlier, datacenters tend to use the best of the best internet connections and hardware. When you use a residential proxy, you can often be stuck with a low grade internet connection on a low grade device. This setup will still get you the access you need, but at the cost of both price and performance.

Why Use Smartproxy Datacenter Proxies?

• Smartproxy's Datacenter Proxies tend to be very cost effective.
• We can use geotargeting to select which country we want to appear in.

Smartproxy Datacenter Proxy Pricing

Setting Up Smartproxy Datacenter Proxies

Authentication

import requests
username = "your-username"password = "your-password"port = 10000proxy_url = f"http://{username}:{password}@dc.smartproxy.com:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

• We save our configuration variables: username, password, port. This way, they're easy to tweak.
• All of our HTTP requests get routed through a custom url: http://{username}:{password}@dc.smartproxy.com:{port}.
• We use a dict to hold both our http and https proxies. We set each one to our proxy_url.
• Finally, we query Smartproxy's API to get our location information.

Basic Request Using Smartproxy Datacenter Proxies

import requests
username = "your-username"password = "your-password"port = 10000proxy_url = f"http://{username}:{password}@dc.smartproxy.com:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

Country Geotargeting

import requests
username = "your-username"password = "your-username"country = "us"port = 10000

proxy_url = f"http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

• Inside of our code, we add the country flag followed by the country of our choosing.
• We also add the user flag to our url. This allows their service to differentiate between our country and our username.
• Our full url now looks like this: http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}.

{    "browser": {        "name": "",        "version": ""    },    "platform": {        "os": "undefined undefined"    },    "engine": {},    "isp": {        "isp": "M247 Europe",        "asn": 9009,        "domain": "",        "organization": "M247 Europe"    },    "city": {        "name": "",        "code": "",        "state": "",        "time_zone": "America/Chicago",        "zip_code": "",        "latitude": 37.751,        "longitude": -97.822    },    "proxy": {        "ip": "198.56.0.20"    },    "country": {        "code": "US",        "name": "United States",        "continent": "North America"    }}

• isp: This holds data about our internet service provider. As odd as it sounds, our service provider is in Europe.
• country: Our country shows up in the United States of America with a code of US. As odd as it might seem, even though our ISP is located in Europe, our actual location is within the United States.
• city: There isn't much information here other than our latitude, longitude and timezone. Our timezone shows up in Chicago. If you lookup these coordinates, it actually places us inside the Cheney Resevoir in Kansas... definitely within the US. You can verify that here with Google Maps.

City Geotargeting

• Local Ads
• Local Businesses
• Local Social Media
• Local Events

Error Codes

KYC Verification

Implementing Smartproxy Datacenter Proxies in Web Scraping

Python Requests

import requests
username = "your-username"password = "your-password"country = "us"port = 10000

proxy_url = f"http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

• Once we've got our credentials, we use them to build our proxy_url: http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}.
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriver
username = "your-username"password = "your-password"country = "us"port = 10000

proxy_url = f"http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}"

## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We build our url exactly how we did above with Python Requests: http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}.
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options.

Python Scrapy

scrapy startproject datacenter

import scrapy
username = "your-username"password = "your-password"country = "us"port = 10000

proxy_url = f"http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}"

class ExampleSpider(scrapy.Spider):    name = "datacenter_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

scrapy crawl datacenter_proxy

• Once again, we create the same proxy_url: http://user-{username}-country-{country}:{password}@dc.smartproxy.com:{port}.
• From inside start_requests, we assign our proxy_url to request.meta['proxy']. This tells Scrapy that all of this spider's requests are to be made through the proxy_url we created earlier.

NodeJS Puppeteer

mkdir puppeteer-datacenter

cd puppeteer-datacenternpm init --y

npm install puppeteer

const puppeteer = require('puppeteer');
const username = "your-username";const password = "your-password";const country = "us";const port = 10000;const hostname = "dc.smartproxy.com";
(async () => {  const browser = await puppeteer.launch({    args: [`--proxy-server=${hostname}:${port}`]  });
  const page = await browser.newPage();
  await page.authenticate({    username: `user-${username}-country-${country}`,    password: password  });
  await page.goto('http://lumtest.com/myip.json');  await page.screenshot({path: 'puppeteer.png'});
  await browser.close();})();

• First, we declare all of our configuration variables as constants: username, password, country, and hostname.
• We set our url when we launch our browser with this arg added --proxy-server=${hostname}:${port}.
• We add our username and country to the username field: user-${username}-country-${country}.
• We also add our password to the password field for authentication: password: password.

NodeJS Playwright

mkdir playwright-datacenter

cd playwright-datacenternpm init --y

npm install playwrightnpx playwright install

const playwright = require('playwright');
const username = "your-username";const password = "your-password";const country = "us";const port = 10000;const hostname = "dc.smartproxy.com";
const options = {    proxy: {        server: `http://${hostname}:${port}`,        username: `user-${username}-country-${country}`,        password: password    }};
(async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
    await page.goto('http://lumtest.com/myip.json');
    await page.screenshot({ path: "playwright.png" })
    await browser.close();})();

• Like our Puppeteer example, we first setup our configuration variables: port, username, password, hostname.
• We create a proxy object with the following fields:
• server: `http://${hostname}:${port}
• username: user-${username}-country-${country}
• password: password

Case Study: Scrape The Guardian

1. We first setup a proxy based in the US. We make a GET to verify our location information and then we make our GET request to The Guardian.
2. Next, we print our location info and the navbar from The Guardian.
3. After the first run, we reset our proxy connection to use a proxy form the UK.
4. If our proxies are working, we'll receive different output from each proxy.

import requestsfrom bs4 import BeautifulSoup
username = "your-username"password = "your-password"country = "us"hostname = "dc.smartproxy.com"port = 10000

proxy_url = f"http://user-{username}-country-{country}:{password}@{hostname}:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}


print("----------------------US---------------------")

location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get('https://www.theguardian.com/', proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")print(subnav.text)

print("----------------------UK---------------------")
country = "gb"
proxy_url = f"http://user-{username}-country-{country}:{password}@{hostname}:{port}"
proxies = {    "http": proxy_url,    "https": proxy_url}
location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies)print(location_info.text)
response = requests.get("https://www.theguardian.com/", proxies=proxies)
soup = BeautifulSoup(response.text, "html.parser")
subnav = soup.select_one("div[data-testid='sub-nav']")
print(subnav.text)

• location_info = requests.get("http://lumtest.com/myip.json", proxies=proxies) is used after setting up each proxy connection. When we make this request, we do it simply so we can verify our location.
• We're using two different locations. First, we set our country to "us", and later on, we set it to "gb".
• Once we've reset our country, we need to reset our proxy_url.
• proxies: After resetting our country, we need to reset our proxies.

• us: USUS elections 2024WorldEnvironmentUkraineSoccerBusinessTechScienceNewslettersWellness
• gb: UKWorldClimate crisisUkraineFootballNewslettersBusinessEnvironmentUK politicsEducationSocietyScienceTechGlobal developmentObituaries

• us: US | US elections 2024 | World | Environment | Ukraine | Soccer | Business | Tech | Science | Newsletters | Wellness
• gb: UK | World | Climate crisis | Ukraine | Football | Newsletters | Business | Environment | UK politics | Education | Society | Science | Tech | Global development | Obituaries

• On the US version of the site, the top left hand corner of the navbar holds US followed by US Elections.
• In the UK, the viewer's attention is prioritized a bit differently, it holds UK followed by World.

Alternative: ScrapeOps Proxy Aggregator

• JavaScript Rendering
• Screenshot Capability
• Country Geotargeting
• Residential and Mobile Proxies
• Anti-bot Bypass
• Custom Headers
• Sticky Sessions

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"LOCATION = "us"
def get_scrapeops_url(url, location=LOCATION):    payload = {        "api_key": API_KEY,        "url": url,        "country": location    }    proxy_url = "https://proxy.scrapeops.io/v1/?" + urlencode(payload)    return proxy_url
response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json"))print(response.text)

• Create our configuration variables: API_KEY and LOCATION.
• Write a get_scrapeops_url() function. This function takes all of our parameters along with a target url and wraps it into a ScrapeOps Proxied url. This is an incredibly easy way to scrape and it makes our proxy code much more modular.
• Check our IP info with response = requests.get(get_scrapeops_url("http://lumtest.com/myip.json")).
• Finally, we print it to the terminal. You should get an output similar to this.

{"country":"US","asn":{"asnum":16509,"org_name":"AMAZON-02"},"geo":{"city":"Columbus","region":"OH","region_name":"Ohio","postal_code":"43215","latitude":39.9625,"longitude":-83.0061,"tz":"America/New_York","lum_city":"columbus","lum_region":"oh"}}

Ethical Considerations and Legal Guidelines

Legal

• Don't use residential proxies to access illegal content: These actions can come with intense legal penalties that can even prison or jail time depending on severity.
• Don't scrape and disseminate other people's private data: Depending on what jurisdiction you're dealing with, this is also a highly illegal and dangerous practice. Doxxing private data can also lead to heavy fines and possibly jail/prison time.

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement.

Conclusion

More Cool Articles

• How To Scrape Quora
• Bright Data Residential Proxies