Oxylabs Mobile Proxies: Web Scraping Guide

• TLDR: How To Integrate Oxylabs Mobile Proxy
• Understanding Mobile Proxies
• Why Use Oxylabs Mobile Proxies?
• Proxy Pricing
• Setting Up Oxylabs Mobile Proxies
• Authentication
• Basic Request Using Oxylabs Mobile Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing Oxylabs Mobile Proxies
• Case Study: Scraping Weather.com
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

TLDR: How to Integrate Oxylabs Mobile Proxy?

1. Once you've got an account and you've created your Mobile Proxy, you need to save your USERNAME and PASSWORD.
2. These are used to construct your proxy_url: http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777.

1. Once we've got our proxy_url, we create a dict object to represent our proxies.
2. Within this dict, we set our http and https protocols to our proxy_url.
3. As we make HTTP any requests, we pass proxies=proxies in as a keyword argument. This tells requests to use that url for our proxy connection.

import requests
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.oxylabs.io/location", proxies=proxies)print(result.text)

• First, we setup our connection variables: USERNAME, and PASSWORD.
• We combine these variables to create our proxy_url: http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777.
• We create a dict object to hold our http and https proxies. We then assign both of them to our proxy_url.
• We send a GET request to the url using the proxies keyword argument.
• Once we've received our response, we print it to the terminal.

Understanding Mobile Proxies

What Are Mobile Proxies?

Why Are Mobile Proxies Important?

Datacenter vs Premium

1. Datacenter proxies are hosted indside an actual datacenter.
2. Premium proxies are composed of both Residential and Mobile proxies.
3. Managed proxies usually use a combination of datacenter and premium IP addresses.

Datacenter

• Price: Datacenter Proxies are often very cheap. Oxylabs Datacenter Proxies start at $0.65/GB and as your plan tier increases, they get even cheaper. You also get the option to pay per IP address as well. When paying per IP, plans start at $1.20 per IP.
• Speed: Datacenter proxies offer unmatched performance. With a Datacenter Proxy, you're guaranteed quality hardware with a decent internet connection.
• Availability: Datacenters host warehouses of machines each with their own IP addresses. With a stable internet connection and so many machines to use, datacenter proxies give us access to very large IP pools with unmatched reliability.

• Blocking: When dealing with more complex sites, they sometimes block datacenter IP addresses. This can make some sites very difficult to scrape if you're limited to datacenter IP addresses.
• Less Geotargeting Support: When using a datacenter proxy, you'll often get support for geotargeting, but it is very limited compared to what you can get when using a premium proxy.
• Less Anonymity: Our IP always shows up inside of a datacenter. Because of this, we don't blend in as well with normal traffic.

Premium

• Anonymity: Premium Proxies give you access to a regular IP address on a real device. This makes it relatively easy for your scraper to blend in with other traffic.
• Better Access: As we mentioned earlier, datacenter proxies can get blocked. Sites typically don't block Premium Proxies because they're composed of real users with standard internet connections.

• Price: Premium Proxies are far more expensive than datacenter solutions. If you're on a bandwidth based plan, 1GB costs $1.65 when using a Datacenter Proxy. That same bandwidth from Oxylabs costs about $9 when using a Premium Proxy.
• Speed: When it comes to Premium Proxies, performance is often subpar. When you use a premium proxy, you're getting a regular IP address on a real device. This means that we're limited by not only the compute power (possibly an outdated smartphone), but internet connection as well.

Mobile vs Residential

Residential

• Authenticity: Residential proxies give you a real IP address. This makes you far more difficult to detect in comparison to a Datacenter proxy.
• Geotargeting: When using residential proxies, you often get much finer control over your location. Sometimes, you can even narrow down your location to a specific city.

• Relatively Static: ISPs (internet service providers) sometimes rotate IP addresses. However, they don't do this often. Whether you keep your IP for a week, a month, or even a year, this can often make an IP ban difficult to get past... especially if you're using a dedicated proxy.
• Sometimes More Detectable: It's a very uncommon practice, but there are some sites that check for a mobile only environment. They do this by checking both your IP address and user-agent string.

Mobile

• Dynamic IP Addresses: Mobile phones receive new IP addresses all the time. Phones move when people move. This causes them to change connection all the time. Whenever your cell phone connects to a different tower, you get a new IP address. If you walk from one store into another, you often get a new IP address.
• Carrier Network Address Translation: This method is a bit different than the IP rotation we mentioned above. With some carriers, many devices sometimes share the same network IP address. This gives you a large anonymity pool because there are loads of other phones all using your same IP address.
• Social Media/ Mobile First Platforms: In 2024, regular people rarely use a computer. Their smartphone provides everything they need. Because of this, many sites (like social media based sites), have adopted a mobile first approach. Mobile proxies make us look far more legitimate on these sites.

• Reliability: As we've already talked about, mobile phones change IP addresses and networks all the time. This can lead to reliability issues.
• Price: Oxylabs supposedly charges $8/GB for Mobile Proxies when using the Pay As You Go plan. However, when purchasing 1GB, it cost me $9. The extra dollar isn't a big deal. Perhaps it is a round-up after adding tax or maybe they need to update the site.

Why Use Oxylabs Mobile Proxies?

Oxylabs Mobile Proxy Pricing

Setting Up Oxylabs Mobile Proxies

1. For starters, if you don't have one already, you'll need to setup an account. As you might have noticed in the screenshot from their front page, click Try Oxylabs today.

2. Afterward, you can create an account using either Google or with a traditional email and password.

3. Once you've got an account, you'll be taken to your account dashboard. Here, you can navigate to the Mobile Proxy option under the Proxies dropdown. Once you've selected it, go ahead and click Buy now.

4. After clicking the Buy now button, you'll be prompted to setup your plan. In the example below, we select the $9/GB Pay As You Go plan.

5. Next, you'll be prompted to actually purchase your traffic. Go ahead and click Continue.

6. Next, you'll be prompted to select a payment method. Choose whichever method works for you and enter your required information. After payment, your payment takes a second to process. Once its done processing, you'll be prompted to setup your new Mobile Proxy. Click on the button title, Let's start.

7. After you've finished setting up your username and password for the proxy, you'll be given a cURL command that you can use to test everything out. Copy and paste the command into your terminal and hit Enter.

8. After running the cURL command listed above, you should receive a response similar to the one below. If you do, this means that everything is working.

{"ip":"172.58.180.148","providers":{"dbip":{"country":"US","asn":"AS21928","org_name":"T-Mobile USA, Inc.","city":"Jenks","zip_code":"","time_zone":"","meta":"\u003ca href='https://db-ip.com'\u003eIP Geolocation by DB-IP\u003c/a\u003e"},"ip2location":{"country":"US","asn":"","org_name":"","city":"Dallas","zip_code":"75201","time_zone":"-06:00","meta":"This site or product includes IP2Location LITE data available from \u003ca href=\"https://lite.ip2location.com\"\u003ehttps://lite.ip2location.com\u003c/a\u003e."},"ipinfo":{"country":"US","asn":"AS21928","org_name":"T-Mobile USA, Inc.","city":"","zip_code":"","time_zone":"","meta":"\u003cp\u003eIP address data powered by \u003ca href=\"https://ipinfo.io\" \u003eIPinfo\u003c/a\u003e\u003c/p\u003e"},"maxmind":{"country":"US","asn":"AS21928","org_name":"T-MOBILE-AS21928","city":"Dallas","zip_code":"","time_zone":"-06:00","meta":"This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com."}}}

Authentication

.

• Our username is always laid out as follows: customer-YOUR_USERNAME.
• We first put a flag for our username, customer and then we attach our username to it -YOUR_USERNAME.
• Then, we append it with :, and we append our password to that: customer-{USERNAME}:{PASSWORD}.

Basic Request Using Oxylabs Mobile Proxies

1. First, we save our USERNAME and PASSWORD variables.
2. Then, we use them to build our proxy_url: http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777. We use a dict and assign this url to both our http and https protocols.
3. Finally, we pass this dict into our proxies argument: proxies=proxies.

import requests
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.oxylabs.io/location", proxies=proxies)print(result.text)

{"ip":"24.206.23.54","providers":{"dbip":{"country":"BS","asn":"AS15146","org_name":"Cable Bahamas","city":"Killarney","zip_code":"","time_zone":"","meta":"\u003ca href='https://db-ip.com'\u003eIP Geolocation by DB-IP\u003c/a\u003e"},"ip2location":{"country":"BS","asn":"","org_name":"","city":"Nassau","zip_code":"-","time_zone":"-05:00","meta":"This site or product includes IP2Location LITE data available from \u003ca href=\"https://lite.ip2location.com\"\u003ehttps://lite.ip2location.com\u003c/a\u003e."},"ipinfo":{"country":"BS","asn":"AS15146","org_name":"Cable Bahamas","city":"","zip_code":"","time_zone":"","meta":"\u003cp\u003eIP address data powered by \u003ca href=\"https://ipinfo.io\" \u003eIPinfo\u003c/a\u003e\u003c/p\u003e"},"maxmind":{"country":"BS","asn":"AS15146","org_name":"CABLEBAHAMAS","city":"Nassau","zip_code":"","time_zone":"-05:00","meta":"This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com."}}}

Country Geotargeting

• Access Geo-Restricted Content: Unlock websites, services, or platforms restricted to specific countries.
• Localized Marketing and SEO: Test ads, keywords, and content tailored for regional audiences.
• Ad Verification: Ensure ads display correctly in target countries while detecting fraud.
• Market Research: Gather accurate insights on regional competitors and trends.
• Bypass Regional Restrictions: Overcome censorship and firewalls in restricted regions.
• Social Media and E-Commerce Management: Safely manage accounts and campaigns for specific countries.
• Improved Anonymity and Security: Appear as a legitimate user from the selected region while maintaining privacy.

import requests
USERNAME = "your-username"PASSWORD = "your-password"COUNTRY = "IT"
proxy_url = f"http://customer-{USERNAME}-cc-{COUNTRY}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.oxylabs.io/location", proxies=proxies)print(result.text)

{"ip":"5.91.137.97","providers":{"dbip":{"country":"IT","asn":"AS30722","org_name":"Vodafone Italia S.p.A.","city":"Rome","zip_code":"","time_zone":"","meta":"\u003ca href='https://db-ip.com'\u003eIP Geolocation by DB-IP\u003c/a\u003e"},"ip2location":{"country":"IT","asn":"","org_name":"","city":"Ripa","zip_code":"00135","time_zone":"+01:00","meta":"This site or product includes IP2Location LITE data available from \u003ca href=\"https://lite.ip2location.com\"\u003ehttps://lite.ip2location.com\u003c/a\u003e."},"ipinfo":{"country":"IT","asn":"AS30722","org_name":"Vodafone Italia S.p.A.","city":"","zip_code":"","time_zone":"","meta":"\u003cp\u003eIP address data powered by \u003ca href=\"https://ipinfo.io\" \u003eIPinfo\u003c/a\u003e\u003c/p\u003e"},"maxmind":{"country":"IT","asn":"AS30722","org_name":"Vodafone Italia S.p.A.","city":"Rome","zip_code":"","time_zone":"+01:00","meta":"This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com."}}}

City Geotargeting

import requests
USERNAME = "your-username"PASSWORD = "your-password"COUNTRY = "US"CITY = "los_angeles"
proxy_url = f"http://customer-{USERNAME}-cc-{COUNTRY}-city-{CITY}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.oxylabs.io/location", proxies=proxies)print(result.text)

{"ip":"166.199.100.19","providers":{"dbip":{"country":"US","asn":"AS7018","org_name":"AT\u0026T Services, Inc.","city":"Tallahassee","zip_code":"","time_zone":"","meta":"\u003ca href='https://db-ip.com'\u003eIP Geolocation by DB-IP\u003c/a\u003e"},"ip2location":{"country":"US","asn":"","org_name":"","city":"Gainesville","zip_code":"32601","time_zone":"-05:00","meta":"This site or product includes IP2Location LITE data available from \u003ca href=\"https://lite.ip2location.com\"\u003ehttps://lite.ip2location.com\u003c/a\u003e."},"ipinfo":{"country":"US","asn":"AS7018","org_name":"AT\u0026T Services, Inc.","city":"","zip_code":"","time_zone":"","meta":"\u003cp\u003eIP address data powered by \u003ca href=\"https://ipinfo.io\" \u003eIPinfo\u003c/a\u003e\u003c/p\u003e"},"maxmind":{"country":"US","asn":"AS7018","org_name":"ATT-INTERNET4","city":"Los Angeles","zip_code":"","time_zone":"-08:00","meta":"This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com."}}}

• Local Ads
• Local Businesses
• Local Social Media
• Local Events

Error Codes

KYC Verification

Implementing Oxylabs Mobile Proxies in Web Scraping

Python Requests

1. We create our environment variables, then we use them to build a proxy_url.
2. Then, we assign that proxy_url to our http and https protocols of our proxies dict.
3. When making our requests, we pass proxies=proxies in as one of our keyword arguments.

import requests
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("https://ip.oxylabs.io/location", proxies=proxies)print(result.text)

• Like before, we setup our configuration variables.
• We create a proxy_url that holds those variables: http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777.
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.
• Once we get our response back, we print it to the terminal.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriverimport json
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777"
## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We build our proxy_url exactly the way we have throughout this article.
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options.

Python Scrapy

scrapy startproject mobile

import scrapy
USERNAME = "your-username"PASSWORD = "your-password"proxy_url = f"http://customer-{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777"


class ExampleSpider(scrapy.Spider):    name = "mobile_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

scrapy crawl mobile_proxy

• First, we create our configuration variables, just like we did before.
• Once again, we create the same basic proxy_url.
• From inside our start_requests method, we assign our proxy_url to request.meta['proxy']. This tells Scrapy to forward all of this spider's requests through the proxy_url we created earlier.

NodeJS Puppeteer

mkdir puppeteer-mobile

cd puppeteer-mobilenpm init --y

npm install puppeteer

const puppeteer = require("puppeteer");
const USERNAME = "your-username";const PASSWORD = "your-password";const proxy_url = "pr.oxylabs.io:7777";
const FULL_USER = `customer-${USERNAME}`;


(async () => {  const browser = await puppeteer.launch({    args: [`--proxy-server=http://${proxy_url}`]  });
  const page = await browser.newPage();
  await page.authenticate({    username: FULL_USER,    password: PASSWORD  });
  await page.goto('http://lumtest.com/myip.json');  await page.screenshot({path: 'puppeteer.png'});
  await browser.close();})();

• First, we create our config variables: USERNAME, and PASSWORD. We declare all of these as constants.
• Next, we create a FULL_USER, which combines our USERNAME string with our -customer flag.
• We add our proxy url to our proxy-args: args: [`--proxy-server=http://${proxy_url}`].
• We pass the following credentials into page.authenticate():
  • username: FULL_USER
  • password: PASSWORD

NodeJS Playwright

mkdir playwright-mobile

cd playwright-mobilenpm init --y

npm install playwrightnpx playwright install

const playwright = require("playwright");

const USERNAME = "your-username";const PASSWORD = "your-password";const proxy_url = "pr.oxylabs.io:7777";
const FULL_USER = `customer-${USERNAME}`;


const options = {    proxy: {        server: `http://${proxy_url}`,        username: FULL_USER,        password: PASSWORD    }};
(async () => {    const browser = await playwright.chromium.launch(options);    const page = await browser.newPage();
    await page.goto('http://lumtest.com/myip.json');
    await page.screenshot({ path: "playwright.png" })
    await browser.close();})();

• Just like with Puppeteer, we first setup our configuration variables.
• We create a proxy object with the following fields:
• server: http://${proxy_url}
• username: FULL_USER
• password: PASSWORD

Case Study: Scrape Weather.com

1. First, we should get different content based on our location.
2. We're also going to count our tries as well. This will allow us to benchmark the proxy quality in each country. Ideally, both proxies should get our content on the first try, but that isn't always the case.

1. We start by using urllib3 to disable some SSL warnings that make it difficult to read our output.
2. In the code below, we setup our first proxy connection in Portugal.
3. Then, we scrape the site description. We also count the amount of tries it takes for a successful response.
4. Afterward, we perform the same scrape using a US based proxy. This allows us to compare the quality of both proxy connections.

import requestsimport urllib3from bs4 import BeautifulSoup
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
USERNAME = "your-username"PASSWORD = "your-password"COUNTRY = "PT"
proxy_url = f"http://customer-{USERNAME}-cc-{COUNTRY}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {  "http": proxy_url,  "https": proxy_url}
headers = {    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"}
print("----------------------PT-------------------")
success = False
tries = 1while not success:    response = requests.get("https://weather.com", proxies=proxies, headers=headers, verify=False)    if response.status_code != 200:        print(f"Failed to get site, Status Code: {response.status_code}")        tries+=1        continue
    soup = BeautifulSoup(response.text, "html.parser")    description_tag = soup.select_one("meta[name='description']")    description = description_tag.get("content")
    print("Description:", description)    print("Total tries:", tries)    success = True

print("----------------------US---------------------")
COUNTRY = "US"
proxy_url = f"http://customer-{USERNAME}-cc-{COUNTRY}:{PASSWORD}@pr.oxylabs.io:7777"
proxies = {  "http": proxy_url,  "https": proxy_url}
success = Falsetries = 1while not success:    response = requests.get("https://weather.com", proxies=proxies, headers=headers, verify=False)    if response.status_code != 200:        print(f"Failed to get site, Status Code: {response.status_code}")        tries+=1        continue
    soup = BeautifulSoup(response.text, "html.parser")    description_tag = soup.select_one("meta[name='description']")    description = description_tag.get("content")
    print("Description:", description)    print("Total tries", tries)    success = True

• urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) disables SSL warnings. This will make our output a bit easier to read.
• After setting up each proxy connection, we do the following:
• Then, we create our success and tries variables.
• As long as our operation hasn't succeeded, we continue trying and we increment tries for each unsuccessful try.
• Once successful, we locate the page description with soup.select_one("meta[name='description']").
• We finally print our tries and description to the terminal.

• Portugal: Previsão e condições meteorológicas e radar Doppler de hoje e de hoje à noite para Socorro, Lisboa por The Weather Channel e Weather.com.
• United States: The Weather Channel and weather.com provide a national and local weather forecast for cities, as well as weather radar, report and hurricane coverage.

• Portugal: 1
• United States: 1

Alternative: ScrapeOps Residential Proxy Aggregator

• Access to the top 20 residential proxy providers, including Smartproxy, Bright Data, and Oxylabs.
• 98% success rate due to automatic proxy switching.
• Bypasses anti-bot measures and avoids blocks.
• Optimizes performance and cost by monitoring proxy performance and pricing.
• Flexible pricing plans starting at $15 per month, with up to $999 for higher usage.
• 500 MB of free bandwidth credits to start.

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"
proxy_url = f"http://scrapeops.mobile=true:{API_KEY}@residential-proxy.scrapeops.io:8181"
proxies = {    "http": proxy_url,    "https": proxy_url}
response = requests.get("https://lumtest.com/myip.json", proxies=proxies, verify=False)print(response.text)

• Our only config variable is our API_KEY.
• We use our API_KEY to build our proxy_url: http://scrapeops.mobile=true:{API_KEY}@residential-proxy.scrapeops.io:8181.
• The flag, mobile=true, tells ScrapeOps that we want a mobile IP address.
• We check our IP information by making a GET: requests.get("https://lumtest.com/myip.json", proxies=proxies, verify=False).
• Finally, we print it to the terminal. You should get an output similar to what you see below. We removed the SSL warning to make the output more legible.

{"country":"VN","asn":{"asnum":45899,"org_name":"VNPT Corp"},"geo":{"city":"Huế","region":"26","region_name":"Thừa Thiên Huế Province","postal_code":"530000","latitude":16.3322,"longitude":107.5864,"tz":"Asia/Bangkok","lum_city":"hue","lum_region":"26"}}

Ethical Considerations and Legal Guidelines

Legal

• Don't use proxies to access illegal content: This can bring you legal fines or even jail time depending on what you're trying to access.
• Don't scrape and disseminate other people's private data: It varies by jurisdiction, but this is also a highly illegal and dangerous practice. Doxxing private data can lead to heavy fines and/or possibly jail/prison time.

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement.

Conclusion

More Cool Articles

• Scrape.do
• Bright Data Residential Proxies