Smartproxy Mobile Proxies: Web Scraping Guide

• TLDR: How To Integrate Smartproxy Mobile Proxy
• Understanding Mobile Proxies
• Why Use Smartproxy Mobile Proxies?
• Proxy Pricing
• Setting Up Smartproxy Mobile Proxies
• Authentication
• Basic Request Using Smartproxy Mobile Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing Smartproxy Mobile Proxies
• Case Study: Scraping Weather.com
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

TLDR: How to Integrate Smartproxy Mobile Proxy?

import requests
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

• First, we need to create our connection variables: USERNAME, and PASSWORD.
• We use these connection variables to create our proxy_url: http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000.
• We create a dict object, which holds our http and https proxies. We then assign both of them to our proxy_url.
• We send a GET request to the url using the proxies keyword argument.
• Once we've received our response, we print it to the terminal.

Understanding Mobile Proxies

What Are Mobile Proxies?

• They use 3G, 4G, or 5G connections, which makes them more trustworthy compared to residential or datacenter proxies.
• Mobile IPs often change frequently due to the nature of mobile carrier systems (NAT - Network Address Translation), which adds an extra layer of anonymity.
• Websites and services are less likely to block mobile IPs because they are associated with legitimate mobile users.

Why Are Mobile Proxies Important?

• Ad Verification: Ensuring that ads appear correctly and are not subject to fraud.
• Web Scraping: Accessing data from websites that impose strict bans on IPs that don't appear organic.
• Bypassing Geo-Restrictions: Accessing content restricted to specific regions.
• Social Media Management: Managing multiple accounts on platforms that limit actions from a single IP.
• SEO and Marketing Research: Conducting localized keyword research and competitor analysis.

Datacenter vs Premium Proxies

Datacenter Proxies

• Price: The best thing about Datacenter proxies is the price. When you purchase Datacenter proxies from Smartproxy, you get the option to pay for bandwidth, or to pay per IP address. Their bandwidth plans range in price from $0.60/GB to $0.50/GB. If you go based on IP, your plans range from $0.10 per to $0.09 per IP address.
• Speed: The performance of Datacenter proxies is virtually unbeatable. When you choose a Datacenter Proxy, you'll get quality hardware with a speedy internet connection.
• Availability: Datacenters tend to hold warehouses of machines and each machine gets its own IP address. Combine this with the fast, stable internet connection mentioned above, and you'll have one of the most reliable proxy connections on the market.

• Blocking: More difficult sites tend to block datacenter IP addresses. They want real people to visit their site. If a request is coming from a datacenter, it is likely some type of bot.
• Less Geotargeting Support: Datacenters do offer geotargeting support, but it's often limited since there aren't all that many datacenters around the world.
• Less Anonymity: With a datacenter, we don't show up as a normal user. This makes it much more difficult for us to blend in with normal human traffic.

Premium

• Anonymity: With a Premium Proxy, you're given an actual residential or mobile IP address. Alongside that, your request gets forwarded through the device of an actual human user.
• Better Access: With Premium Proxies, we tend to blend in much better. To the site we're trying to scrape, our traffic looks completely human.

• Price: Premium Proxies are far more expensive. With Residential and Mobile proxies from Smartproxy, plans start at $7/GB and $8/GB respectively. Compared to $0.60/GB, these proxies cost more than 10 times their datacenter counterparts.
• Speed: With Premium Proxies, performance can often be a little lacking. With this type of proxy, you're tied to a real device (often an outdated smartphone or laptop) on a real home network.

Mobile vs Residential Proxies

Residential Proxies

• Authenticity: With a Residential Proxy, you get a real residential IP address tied to someone's actual house or apartment. Your traffic can't look much more human than this.
• Geotargeting: With Residential Proxies, we often get much finer control over our geotargeting. Sometimes you can even choose which city you want to show up in!

• Relatively Static: ISPs (internet service providers) do rotate IP addresses, but they don't do so very often. Whether your provider rotates weekly, monthly, or even yearly, we need to rotate IP addresses to get past IP blocks. This is the achilles heel of Dedicated Residential Proxy services.
• Sometimes More Detectable: It's not common practice by any means, but there are some sites that check for a mobile only environment. They do this by examining both your IP address and user-agent string.

Mobile Proxies

• Dynamic IP Addresses: When you're on a mobile phone, it's changing IP addresses all the time. When you walk down the street, your phone probably jumps from your home network to cellular data, and then possibly to a public network you've used before. On top of this, when using cellular data, your provider will often rotate IP addresses throughout their entire mobile network.
• Carrier Network Address Translation: This is quite different from the IP rotation we mentioned above. Some carriers actually keep all their devices connected through the same IP mobile address. This type of privacy actually works quite different from the method mentioned above. Since there are a large number of devices using this IP address at any given time, you blend in with everyone else on the network.
• Social Media/ Mobile First Platforms: It's now the year 2024 and people rarely use actual computers. Their smartphone provides everything they need. Because of this, many sites (like social media based sites), have adopted a mobile first approach. Mobile proxies make us look far more legitimate on these sites.

• Reliability: One of the pros of Mobile Proxies, our IPs are always changing. This is actually also a con. Because of the constant network switching, this can lead to some unreliability with our proxy connection.
• Price: At Smartproxy, Mobile Proxies start at $8/GB on the Pay As You Go plan. This is more than 10 times the price of their Datacenter offerings and 12.5% more than their standard Residential offerings.

Why Use Smartproxy Mobile Proxies?

Smartproxy Mobile Proxy Pricing

• From there, the Self-Service plans go down by $0.50/GB with each tier.
• The highest Self-Service plan comes in at $300/month with a price of $6/GB.

• Their Enterprise plans begin at $550/month and $5.50/GB.
• The highest tier comes in at $2,250/month and costs at total of $4.50/GB.

Setting Up Smartproxy Mobile Proxies

1. First, head on over to Smartproxy. As you can see in the screenshot below, you can create an account using either your email address or your Google account.

2. Once you've got your account, you'll be taken to their dashbaord. On the left side of the page, click Proxies. Then, click the Residential dropdown. Your next two options will be Residential and Mobile. Click Mobile.

3. Now, you'll be prompted to choose a mobile plan. Since we're just trying it out, we'll go with Pay As You Go. When you're ready, click Continue to checkout.

4. Next, you'll be prompted to actually purchase your plan.

5. After making your payment, it takes a second to confirm.

6. Once your payment's been confirmed, you'll be prompted to setup your new proxy.

7. Now, you need to go through and configure your proxy. We're going with almost the default setup. Instead of a Static proxy, we use a Rotating one.

8. Once you're finished with all that, you can test your proxy with a the command below. Make sure to replace YOUR_USERNAME and YOUR_PASSWORD with your actual username and password.

curl -U "YOUR_USERNAME:YOUR_PASSWORD" -x "gate.smartproxy.com:10001" "https://ip.smartproxy.com/json"

{    "browser": {        "name": "",        "version": ""    },    "platform": {        "os": "undefined undefined"    },    "engine": {},    "isp": {        "isp": "Vivo",        "asn": 26599,        "domain": "vivozap.com.br",        "organization": "Vivo"    },    "city": {        "name": "Blumenau",        "code": "SC",        "state": "Santa Catarina",        "time_zone": "America/Sao_Paulo",        "zip_code": "89000",        "latitude": -26.8766,        "longitude": -49.0969    },    "proxy": {        "ip": "177.174.250.237"    },    "country": {        "code": "BR",        "name": "Brazil",        "continent": "South America"    }}

Authentication

• Our authentication string always goes as follows: http://{USERNAME}:{PASSWORD}.
• Our actual URL starts with our username.
• Then, we append it with :, and we append our password to that: {USERNAME}:{PASSWORD}.

Basic Request Using Smartproxy Mobile Proxies

• First, we save our USERNAME and PASSWORD variables.
• They are then used to create our proxy_url: http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000.
• We use a dict called proxies to hold our http and https protocols.
• We assign our proxy_url to both of these protocols.

import requests
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

{    "browser": {        "name": "",        "version": ""    },    "platform": {        "os": "undefined undefined"    },    "engine": {},    "isp": {        "isp": "Comcast Cable",        "asn": 7922,        "domain": "comcast.net",        "organization": "Comcast Cable"    },    "city": {        "name": "Indianapolis",        "code": "IN",        "state": "Indiana",        "time_zone": "America/Indiana/Indianapolis",        "zip_code": "46236",        "latitude": 39.8685,        "longitude": -85.969    },    "proxy": {        "ip": "73.146.135.168"    },    "country": {        "code": "US",        "name": "United States",        "continent": "North America"    }}

Country Geotargeting

import requests
USERNAME = "your-username"PASSWORD = "your-password"COUNTRY = "IT"
proxy_url = f"http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@gate.smartproxy.com:7000"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

{    "browser": {        "name": "",        "version": ""    },    "platform": {        "os": "undefined undefined"    },    "engine": {},    "isp": {        "isp": "Vodafone Italia",        "asn": 30722,        "domain": "vodafone.it",        "organization": "Vodafone Italia"    },    "city": {        "name": "Rimini",        "code": "45",        "state": "Emilia-Romagna",        "time_zone": "Europe/Rome",        "zip_code": "47921",        "latitude": 44.0545,        "longitude": 12.5686    },    "proxy": {        "ip": "5.90.141.72"    },    "country": {        "code": "IT",        "name": "Italy",        "continent": "Europe"    }}

City Geotargeting

import requests
USERNAME = "your-username"PASSWORD = "your-password"PORT = 21051
proxy_url = f"http://{USERNAME}:{PASSWORD}@city.smartproxy.com:{PORT}"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

{    "browser": {        "name": "",        "version": ""    },    "platform": {        "os": "undefined undefined"    },    "engine": {},    "isp": {        "isp": "Spectrum",        "asn": 20001,        "domain": "spectrum.com",        "organization": "Spectrum"    },    "city": {        "name": "Los Angeles",        "code": "CA",        "state": "California",        "time_zone": "America/Los_Angeles",        "zip_code": "90044",        "latitude": 33.9555,        "longitude": -118.2863    },    "proxy": {        "ip": "76.32.131.65"    },    "country": {        "code": "US",        "name": "United States",        "continent": "North America"    }}

• Local Ads
• Local Businesses
• Local Social Media
• Local Events

Error Codes

KYC Verification

Implementing Smartproxy Mobile Proxies in Web Scraping

Python Requests

1. First, we need to create our environment variables. We'll use those variables to create our proxy_url.
2. Then, we'll create a dict called proxies.
3. Within proxies, we assign our proxy_url to both the http and https protocols.
4. Whenever we make an HTTP request, we pass proxies=proxies in as a keyword arguments.
5. The proxies kwarg tells Python Requests to forward all these requests through our proxy_url.

import requests
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.smartproxy.com/json", proxies=proxies)print(result.text)

• Just like in our previous examples, we setup our configuration variables.
• Our proxy_url is composed from those variables: http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000.
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.
• Once we get our response back, we print it to the terminal.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it will not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriverimport json
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000"
## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We build our proxy_url using our config variables, just like we did in earlier examples.
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options.

Python Scrapy

scrapy startproject mobile

import scrapy
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://{USERNAME}:{PASSWORD}@gate.smartproxy.com:7000"

class ExampleSpider(scrapy.Spider):    name = "mobile_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

scrapy crawl mobile_proxy

• As always, we create our configuration variables.
• Once again, we use these config variables to create our proxy_url.
• From inside of start_requests, we use proxy_url = request.meta['proxy']. This tells Scrapy that all of our requests should go through the proxy we created.

NodeJS Puppeteer

mkdir puppeteer-mobile

cd puppeteer-mobilenpm init --y

npm install puppeteer

const puppeteer = require("puppeteer");
const USERNAME = "your-username";const PASSWORD = "your-password";const proxyUrl = "gate.smartproxy.com:7000";
(async () => {  const browser = await puppeteer.launch({    args: [`--proxy-server=http://${proxyUrl}`]  });
  const page = await browser.newPage();
  await page.authenticate({    username: USERNAME,    password: PASSWORD  });
  await page.goto('http://lumtest.com/myip.json');  await page.screenshot({path: 'puppeteer.png'});
  await browser.close();})();

• First, we create our config variables: USERNAME, and PASSWORD. We declare all of them using the const keyword.
• We add our proxy url to our proxy-args: args: [`--proxy-server=http://${proxy_url}`].
• We pass the following credentials into page.authenticate():
  • username: USERNAME
  • password: PASSWORD

NodeJS Playwright

mkdir playwright-mobile

cd playwright-mobilenpm init --y

npm install playwrightnpx playwright install

const playwright = require("playwright");
const USERNAME = "your-username";const PASSWORD = "your-password";const proxyUrl = "gate.smartproxy.com:7000";
const options = {    proxy: {        server: proxyUrl,        username: USERNAME,        password: PASSWORD    }};
(async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
    await page.goto('http://lumtest.com/myip.json');
    await page.screenshot({ path: "playwright.png" })
    await browser.close();})();

• Just like with Puppeteer, we first setup our configuration variables.
• We create a proxy object with the following fields:
• server: http://${proxy_url}
• username: FULL_USER
• password: PASSWORD
• Instead of using page.authenticate(), with Playwright, we pass our options into chromium.launch().

Case Study: Scrape Weather.com

• First, we'll perform this scrape with a Portuguese mobile proxy.
• Then, we'll repeat the process with a mobile proxy in the US.

1. Weather is different all over the world.
2. Weather.com should give us different content based on our location.

1. We start by using urllib3 to disable some SSL warnings and make it easier to read our output.
2. As we talked about above, we setup our first proxy connection in Portugal.
3. With our Portuguese proxy, we scrape the site description. We also count the amount of tries it takes for a successful response.
4. Afterward, we perform the same scrape using a US based proxy. This allows us to compare the quality of both proxy connections.

import requestsimport urllib3from bs4 import BeautifulSoup
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
USERNAME = "your-username"PASSWORD = "your-password"COUNTRY = "PT"
proxy_url = f"http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@gate.smartproxy.com:7000"
proxies = {  "http": proxy_url,  "https": proxy_url}
headers = {    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"}
print("----------------------PT-------------------")
success = False
tries = 1while not success:    response = requests.get("https://weather.com", proxies=proxies, headers=headers, verify=False)    if response.status_code != 200:        print(f"Failed to get site, Status Code: {response.status_code}")        tries+=1        continue
    soup = BeautifulSoup(response.text, "html.parser")    description_tag = soup.select_one("meta[name='description']")    description = description_tag.get("content")
    print("Description:", description)    print("Total tries:", tries)    success = True

print("----------------------US---------------------")
COUNTRY = "US"
proxy_url = f"http://user-{USERNAME}-country-{COUNTRY}:{PASSWORD}@gate.smartproxy.com:7000"
proxies = {  "http": proxy_url,  "https": proxy_url}
success = Falsetries = 1while not success:    response = requests.get("https://weather.com", proxies=proxies, headers=headers, verify=False)    if response.status_code != 200:        print(f"Failed to get site, Status Code: {response.status_code}")        tries+=1        continue
    soup = BeautifulSoup(response.text, "html.parser")    description_tag = soup.select_one("meta[name='description']")    description = description_tag.get("content")
    print("Description:", description)    print("Total tries", tries)    success = True

• urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) is used to disable our SSL warnings. When using a proxy port, SSL errors can be common and the pollute the output we're trying to read.
• After setting up each proxy connection, we do the following:
• Then, we create our success and tries variables.
• For each attempt, we use a while loop. After each attempt, we increment our tries. Once we get a successful response, we set success to True and this allows us to exit the loop.
• We locate the page description with soup.select_one("meta[name='description']").
• We finally print our tries and description to the terminal.

• Portugal: Previsão e condições meteorológicas e radar Doppler de hoje e de hoje à noite para Socorro, Lisboa por The Weather Channel e Weather.com.
• United States: The Weather Channel and weather.com provide a national and local weather forecast for cities, as well as weather radar, report and hurricane coverage.

• Portugal: 1
• United States: 1

Alternative: ScrapeOps Residential Proxy Aggregator

• Access to the top 20 residential proxy providers, including Smartproxy, Bright Data, and Oxylabs.
• 98% success rate due to automatic proxy switching.
• Bypasses anti-bot measures and avoids blocks.
• Optimizes performance and cost by monitoring proxy performance and pricing.
• Flexible pricing plans starting at $15 per month, with up to $999 for higher usage.
• 500 MB of free bandwidth credits to start. Our plans range in price from free (yes... FREE) all the way to $999 per month. We have 8 different choices available.

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"
proxy_url = f"http://scrapeops.mobile=true:{API_KEY}@residential-proxy.scrapeops.io:8181"
proxies = {    "http": proxy_url,    "https": proxy_url}
response = requests.get("https://lumtest.com/myip.json", proxies=proxies, verify=False)print(response.text)

• With ScrapeOps, we only need our API_KEY.
• We use our API_KEY to build our proxy_url: http://scrapeops.mobile=true:{API_KEY}@residential-proxy.scrapeops.io:8181.
• The flag, mobile=true, tells ScrapeOps that we want a mobile IP address.
• We check our IP information by making a GET: requests.get("https://lumtest.com/myip.json", proxies=proxies, verify=False).
• Finally, we print it to the terminal. You should get an output similar to what you see below. We removed the SSL warning to make the output more legible.

{"country":"KH","asn":{"asnum":17976,"org_name":"CAMGSM Company Ltd"},"geo":{"city":"Phnom Penh","region":"12","region_name":"Phnom Penh","postal_code":"","latitude":11.5583,"longitude":104.9121,"tz":"Asia/Phnom_Penh","lum_city":"phnompenh","lum_region":"12"}}

Ethical Considerations and Legal Guidelines

Legal

• Don't use proxies to access illegal content: This can bring you legal fines or even jail time depending on what you're trying to access.
• Don't scrape and disseminate other people's private data: It varies by jurisdiction, but this is also a highly illegal and dangerous practice. Doxxing private data can lead to heavy fines and/or possibly jail/prison time.

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement.

Conclusion

More Cool Articles

• How To Scrape Trustpilot
• Bright Data Residential Proxies