Bright Data Mobile Proxies: Web Scraping Guide

• TLDR: How To Integrate Bright Data Mobile Proxy
• Understanding Mobile Proxies
• Why Use Bright Data Mobile Proxies?
• Proxy Pricing
• Setting Up Bright Data Mobile Proxies
• Authentication
• Basic Request Using Bright Data Mobile Proxies
• Country Geotargeting
• City Geotargeting
• Error Codes
• KYC Verification
• Implementing Bright Data Mobile Proxies
• Case Study: Scraping Weather.com
• Alternative: ScrapeOps Proxy Aggregator
• Ethical Considerations and Legal Guidelines
• Conclusion
• More Cool Articles

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

TLDR: How to Integrate Bright Data Mobile Proxy?

import requestsimport json
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225"
proxies = {    "http": proxy_url,    "https": proxy_url}
success = False
while not success:    result = requests.get("http://geo.brdtest.com/mygeo.json", proxies=proxies)    try:        if result.status_code != 200:            raise Exception("Failed to find proxies")        print(result.text)        success = True    except Exception as e:        print(f"Request Failed, {e}, Status: {result.status_code}")

• First, we setup our connection variables: USERNAME, ZONE, and PASSWORD.
• We then put them together to create a proxy_url: http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225.
• We create a dict object to hold our http and https proxies. We then assign both of them to our proxy_url.
• while we have not received a successful response, we attempt to do the following:
  • Send a GET request to the url.
  • Within our try block, we:
    • Check the status of the response.
    • If the status code is not 200, we throw an error.
    • If we receive a 200, we print the result.
  • Our except block is used to handle any errors.
    • We print an error message and our status code: Request Failed, {e}, Status: {result.status_code}.

Understanding Mobile Proxies

What Are Mobile Proxies?

Why Are Mobile Proxies Important?

Datacenter vs Premium

1. Datacenter proxies are hosted indside a datacenter.
2. Premium proxies are either residential or mobile, these proxies are best for sites that practice IP blocking.
3. Managed proxies tend to use a combination of datacenter and premium IP addresses.

Datacenter

• Price: Datacenter proxies are incredibly cheap. If you choose to use their Datacenter proxies, Bright Data only charges $0.60 per GB.
• Speed: Datacenter proxies give us unparalleled performance. When you use this type of proxy, it gets hosted in an actual datacenter. Datacenters use some of the best hardware with a fast and stable internet connection.
• Availability: Datacenters are huge. They host tons and tons of machines each with their own IP addresses. With a stable internet connection and so many machines to use, datacenter proxies give us access to very large IP pools with unmatched reliability.

• Blocking: When you're dealing with more difficult sites, they tend to block datacenter IP addresses. If you don't have a residential or mobile IP, they assume you're up to no good.
• Less Geotargeting Support: When using a datacenter proxy, you'll often get support for geotargeting, but it is very limited compared to what you can get when using a premium proxy.
• Less Anonymity: Our IP always shows up inside of a datacenter. Because of this, we don't blend in as well with normal traffic.

Premium

• Anonymity: With a Premium proxy, you get a regular IP address on a real device. This makes it relatively easy to blend in.
• Better Access: As mentioned earlier, datacenter proxies can often get blocked. Sites generally don't block residential or mobile IPs because this is the traffic they want.

• Price: Premium proxies are far more expensive than their datacenter counterparts. With Bright Data, both their residential and mobile proxies can cost as much as $8.40 per GB. This is more than 10 times the price of their Datacenter products.
• Speed: With premium proxies, speed is not as good. When you're connected through a premium proxy, you're getting a regular IP address on a real device. This means that we're limited by not only the compute power (possibly an outdated smartphone), but internet connection as well.

Mobile vs Residential

Residential

• Authenticity: Residential proxies give you a real IP address. This makes you far more difficult to detect.
• Geotargeting: When using residential proxies, you often get much finer control over your location. Sometimes, you can even narrow down your location to a specific city.

• Relatively Static: ISPs (internet service providers) do sometimes rotate our IP addresses. However, they don't do this often. Whether you keep your IP for a week, a month or even a year, this can often make an IP ban difficult to get past... especially if you're using a dedicated proxy.
• Sometimes More Detectable: This is an incredibly uncommon practice, but there are some sites that look for a mobile only environment. They do this by checking both your IP address and user-agent string.

Mobile

• Rotating IP Addresses: Phones get new IP addresses all the time. Phones move around alot. This causes them to change connection all the time. Whenever your cell phone connects to a different tower, you get a new IP address. If you walk from one store into another, you often get a new IP address.
• Carrier Network Address Translation: This method is a bit different than the IP rotation mentioned above. With this feature, many devices share the same network IP address. This gives you a high anonymity pool because there are loads of other phones all using your same IP address.
• Social Media/ Mobile First Platforms: Nowadays, most normal people rarely use a computer. Their smartphone provides everything they need internet wise. Because of this, many sites (social media in particular), have adopted a mobile first approach. Mobile proxies make us look far more legitimate on these sites.

• Reliability: As mentioned earlier, mobile phones change IP addresses and networks all the time. This leads to reliability issues. As you might have noticed in our TLDR section, it took us several retries to even get a good response through the proxy.
• Price: While Bright Data charges $8.40/GB for both its Mobile and Residential products, some providers chage more for mobile. As we've already talked about, we need to make multiple requests. Depending on your provider, this can eat up your bandwidth, which leads to higher cost.

Why Use Bright Data Mobile Proxies?

Bright Data Mobile Proxy Pricing

1. Pay As You Go,
2. 69GB,
3. 158GB, and
4. 339GB.

Setting Up Bright Data Mobile Proxies

1. To start, head on over to https://brightdata.com. You can create an account using a username and password, or you can simply sign up using your Google account.

2. Next, once you're logged in, click on the Add button and select Mobile proxies.

3. Now, you should be taken to a screen where you can configure your mobile proxies. In the shot below, we make a new zone called, mobile_proxy1 and we just keep the default settings. Once you're finished, click on the Add button.

4. If you look at our final screenshot below, you'll see a basic request builder and some of our basic proxy information. There is also a toggle switch to turn it on and off. In the example below, it is switched off. Make sure you toggle your proxy on.

5. For convenience, here's a closer look at the cURL command in the screenshot above. If you choose to run it, make sure to replace YOUR_USERNAME with your actual username. Also make sure to replace YOUR_PASSWORD with your actual password.

curl -i --proxy brd.superproxy.io:22225 --proxy-user brd-customer-YOUR_USERNAME-zone-mobile_proxy1:YOUR_PASSWORD -k "https://geo.brdtest.com/welcome.txt"

6. If you run the cURL command listed above, you'll get a response similar to this. As you can see below, we get all sorts of information about our proxy and some basic usage examples as well. They also give us information on how to receive a JSON response and a link to their docs.

Authentication

• Our username is always laid out as follows: brd-customer-YOUR_USERNAME.
• We first put a flag for our username, brd-customer and then we attach our username to it -YOUR_USERNAME.
• We also have a zone flag followed by the name of the zone we created when doing our initial setup.
• : is then followed by our password.

Basic Request Using Bright Data Mobile Proxies

import requestsimport random
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225"
proxies = {    "http": proxy_url,    "https": proxy_url}
result = requests.get("http://geo.brdtest.com/mygeo.json", proxies=proxies)print(result.text)

{"country":"IT","asn":{"asnum":30722,"org_name":"Vodafone Italia S.p.A."},"geo":{"city":"Rome","region":"62","region_name":"Lazio","postal_code":"00125","latitude":41.8904,"longitude":12.5126,"tz":"Europe/Rome","lum_city":"rome","lum_region":"62"}}

Country Geotargeting

import requestsimport json
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"COUNTRY = "us"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}-country-{COUNTRY}:{PASSWORD}@brd.superproxy.io:22225"
proxies = {    "http": proxy_url,    "https": proxy_url}
success = False
while not success:    result = requests.get("http://geo.brdtest.com/mygeo.json", proxies=proxies)    try:        if result.status_code != 200:            raise Exception("Failed to find proxies")        print(result.text)        success = True    except Exception as e:        print(f"Request Failed, {e}, Status: {result.status_code}")

City Geotargeting

• Local Ads
• Local Businesses
• Local Social Media
• Local Events

Error Codes

KYC Verification

Implementing Bright Data Mobile Proxies in Web Scraping

Python Requests

import requestsimport json
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225"
proxies = {    "http": proxy_url,    "https": proxy_url}
success = False
while not success:    result = requests.get("http://geo.brdtest.com/mygeo.json", proxies=proxies)    try:        if result.status_code != 200:            raise Exception("Failed to find proxies")        print(result.text)        success = True    except Exception as e:        print(f"Request Failed, {e}, Status: {result.status_code}")

• Like before, we setup our configuration variables.
• We combine all these variables into our proxy_url: http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225.
• We then create a dict object that holds both our http and https proxies.
• When making our requests, we make sure to pass proxies=proxies. This tells Python Requests to use the dict object we created for our proxy settings.
• Since our connection is prone to 502 errors, we use a while loop with some basic retry logic to keep trying our request until we get a proper response.

Python Selenium

• Security: Browsers are updated with security patches regularly. Without these patches, your browser will have holes in the security that have been fixed in other browsers such as Chromedriver or Geckodriver.
• Dependency Issues: SeleniumWire is no longer maintained. In time, it may not be able to keep up with its dependencies as they get updated. Broken dependencies can be a source of unending headache for anyone in software development.
• Compatibility: As the web itself gets updated, SeleniumWire doesn't. Regular browsers are updated all the time. Since SeleniumWire no longer receives updates, you may experience broken functionality and unexpected behavior.

from seleniumwire import webdriverimport json
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225"
## Define Your Proxy Endpointsproxy_options = {    "proxy": {        "http": proxy_url,        "https": proxy_url,        "no_proxy": "localhost:127.0.0.1"    }}
## Set Up Selenium Chrome driverdriver = webdriver.Chrome(seleniumwire_options=proxy_options)
## Send Request Using Proxydriver.get('https://httpbin.org/ip')

• We build our proxy_url exactly the way we have throughout this article.
• We assign this url to both the http and https protocols of our proxy settings.
• driver = webdriver.Chrome(seleniumwire_options=proxy_options) tells webdriver to open Chrome with our custom seleniumwire_options.

Python Scrapy

scrapy startproject mobile

import scrapy
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}:{PASSWORD}@brd.superproxy.io:22225"

class ExampleSpider(scrapy.Spider):    name = "mobile_proxy"
    def start_requests(self):        request = scrapy.Request(url="https://httpbin.org/ip", callback=self.parse)        request.meta['proxy'] = proxy_url        yield request
    def parse(self, response):        print(response.body)

scrapy crawl mobile_proxy

• First, we use create configuration variables.
• Once again, we create the same `proxy_url.
• From inside start_requests, we assign our proxy_url to request.meta['proxy']. This tells Scrapy that all of this spider's requests are to be made through the proxy_url we created earlier.
• Scrapy has builtin retry logic, we don't need to code any ourselves.

NodeJS Puppeteer

mkdir puppeteer-mobile

cd puppeteer-mobilenpm init --y

npm install puppeteer

const puppeteer = require("puppeteer");
const USERNAME = "your-username";const ZONE = "mobile_proxy1";const PASSWORD = "your-password";
const FULL_USER = `brd-customer-${USERNAME}-zone-${ZONE}`;
(async () => {    var success = false;    const browser = await puppeteer.launch({    args: ["--proxy-server=http://brd.superproxy.io:22225"]  });    while (!success) {
    try {      const page = await browser.newPage();
    await page.authenticate({      username: FULL_USER,      password: PASSWORD    });
    var badResponse = false;
    page.on("response", async (response) => {      const status = response.status();      if (status !== 200) {        badResponse = true;        console.log(`Failed to get the page, Status Code: ${status}`);      }    });
    await page.goto('http://lumtest.com/myip.json');
    if (badResponse) {      throw new Error("bad response");    }
    await page.screenshot({path: 'puppeteer.png'});    success = true;    console.log("Screenshot successful");
    } catch {      console.log("Failed to connect to mobile proxy, retrying");    }
  } 
  await browser.close();
})();

• First, we create our config variables: USERNAME, ZONE, PASSWORD. All of these get declared as constants.
• Next, we create a FULL_USER, which combines our USERNAME string along with our ZONE.
• We add our proxy url to our proxy-args: args: ["--proxy-server=http://brd.superproxy.io:22225"].
• We pass the following credentials into page.authenticate():
  • username: FULL_USER
  • password: PASSWORD
• We use page.on() to analyze our response:
  • If our response.status() is not 200, we set badResponse to true.
• This page.on() logic executes whenever we GET a page. If badResponse is true, we throw an error.

NodeJS Playwright

mkdir playwright-mobile

cd playwright-mobilenpm init --y

npm install playwrightnpx playwright install

const playwright = require("playwright");
const USERNAME = "your-username";const ZONE = "mobile_proxy1";const PASSWORD = "your-password";
const FULL_USER = `brd-customer-${USERNAME}-zone-${ZONE}`;
const options = {    proxy: {        server: "http://brd.superproxy.io:22225",        username: FULL_USER,        password: PASSWORD    }};
(async () => {    const browser = await playwright.chromium.launch(options);        var success = false;        while (!success) {        const page = await browser.newPage();
        try {            const response = await page.goto('http://lumtest.com/myip.json');            if (response.status() !== 200) {                throw new Error(`Failed respoonse, Status Code: ${response.status()}`);            }            await page.screenshot({ path: "playwright.png" });            console.log("screenshot successful");            success = true;        } catch (error) {            console.log("ERROR:", error);        } finally {            await page.close();        }
    }    await browser.close();    })();

• Like our Puppeteer example, we first setup our configuration variables.
• We create a proxy object with the following fields:
• server: "http://brd.superproxy.io:22225"
• username: FULL_USER
• password: PASSWORD
• Our retry logic is quite a bit easier to implement with Playwright. With Playwright, page.goto() returns a response object. This allows us to check the status code in a much easier way (like we did with Requests).

Case Study: Scrape Weather.com

1. First, we should get different content based on our location.
2. As well, we're going to count our tries. This will allow us to benchmark the proxy quality in each country.

• We start by disabling some warnings to prevent SSL errors from clogging up the console. If you wish, you can also manually download and install Bright Data's SSL certificate here.
• In the code below, we setup our first proxy connection in Portugal.
• Then, we scrape the site description.
• We count the tries and then print both the description and tries taken to the terminal.
• Afterward, we perform the same exercise using a US based proxy. This allows us to compare the quality of both proxies.

import requestsimport urllib3from bs4 import BeautifulSoup
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
USERNAME = "your-username"ZONE = "mobile_proxy1"PASSWORD = "your-password"COUNTRY = "pt"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}-country-{COUNTRY}:{PASSWORD}@brd.superproxy.io:33335"
proxies = {  "http": proxy_url,  "https": proxy_url}
headers = {    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"}
print("----------------------PT-------------------")
success = False
tries = 1while not success:    response = requests.get("https://weather.com", proxies=proxies, headers=headers, verify=False)    if response.status_code != 200:        print(f"Failed to get site, Status Code: {response.status_code}")        tries+=1        continue
    soup = BeautifulSoup(response.text, "html.parser")    description_tag = soup.select_one("meta[name='description']")    description = description_tag.get("content")
    print("Description:", description)    print("Total tries:", tries)    success = True

print("----------------------US---------------------")
COUNTRY = "us"
proxy_url = f"http://brd-customer-{USERNAME}-zone-{ZONE}-country-{COUNTRY}:{PASSWORD}@brd.superproxy.io:33335"
proxies = {  "http": proxy_url,  "https": proxy_url}
success = Falsetries = 1while not success:    response = requests.get("https://weather.com", proxies=proxies, headers=headers, verify=False)    if response.status_code != 200:        print(f"Failed to get site, Status Code: {response.status_code}")        tries+=1        continue
    soup = BeautifulSoup(response.text, "html.parser")    description_tag = soup.select_one("meta[name='description']")    description = description_tag.get("content")
    print("Description:", description)    print("Total tries", tries)    success = True

• urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) is used to disable any SSL warnings we might receive.
• After setting up each proxy connection, we do the following:
• Then, we set both our success and tries variables.
• While our operation hasn't succeeded, we continue trying and we increment tries for each unsuccessful try.
• Once our try is successful, we find the page description with soup.select_one("meta[name='description']").
• We finally print our tries and description to the terminal.

• Portugal: Previsão e condições meteorológicas e radar Doppler de hoje e de hoje à noite para Socorro, Lisboa por The Weather Channel e Weather.com.
• United States: The Weather Channel and weather.com provide a national and local weather forecast for cities, as well as weather radar, report and hurricane coverage.

• Portugal: 1
• United States: 3

1. When using a Portuguese proxy:

• Our site comes written in Portuguese.
• Our Portuguese proxy was able to get a proper response on the first try.

2. With our US based proxy:

• Our site comes written in English.
• The US based proxy took three trials to get a proper response.

Alternative: ScrapeOps Residential Proxy Aggregator

• Access to the top 20 residential proxy providers, including Smartproxy, Bright Data, and Oxylabs.
• 98% success rate due to automatic proxy switching.
• Bypasses anti-bot measures and avoids blocks.
• Optimizes performance and cost by monitoring proxy performance and pricing.
• Flexible pricing plans starting at $15 per month, with up to $999 for higher usage.
• 500 MB of free bandwidth credits to start.

import requestsfrom urllib.parse import urlencode
API_KEY = "your-super-secret-api-key"
proxy_url = f"http://scrapeops.mobile=true:{API_KEY}@residential-proxy.scrapeops.io:8181"
proxies = {    "http": proxy_url,    "https": proxy_url}
response = requests.get("https://lumtest.com/myip.json", proxies=proxies, verify=False)print(response.text)

• Create our configuration variable: API_KEY.
• We then build our proxy_url: http://scrapeops.mobile=true:{API_KEY}@residential-proxy.scrapeops.io:8181.
• The flag, mobile=true, tells ScrapeOps that we want a mobile IP address.
• We check our IP information by making a GET: requests.get("https://lumtest.com/myip.json", proxies=proxies, verify=False).
• Finally, we print it to the terminal. You should get an output similar to what you see below. We removed the SSL warning to make the output more legible.

{"country":"VN","asn":{"asnum":7552,"org_name":"Viettel Group"},"geo":{"city":"Ho Chi Minh City","region":"SG","region_name":"Ho Chi Minh","postal_code":"700000","latitude":10.8217,"longitude":106.6254,"tz":"Asia/Ho_Chi_Minh","lum_city":"hochiminhcity","lum_region":"sg"}}

Ethical Considerations and Legal Guidelines

Legal

• Don't use proxies to access illegal content: This can being you legal fines or even jail time.
• Don't scrape and disseminate other people's private data: Depending on what your jurisdiction, this is also a highly illegal and dangerous practice. Doxxing private data can also lead to heavy fines and possibly jail/prison time.

Ethical

• Social Media Monitoring: Social media stalking can be a very destructive and disrespectful behavior. How would you feel if someone used data collection methods on your account?
• Respect Site Policies: Failure to respect a site's policies can get your account suspended/banned. It can even lead to legal troubles for those of you who sign and violate a terms of service agreement.

Conclusion

More Cool Articles

• How To Scrape LinkedIn Jobs
• Bright Data Residential Proxies