Skip to main content

AdsPower Integration

How to Integrate ScrapeOps Residential Proxies with AdsPower

Launched in 2018, AdsPower is a tool for creating custom browsers and automation. With AdsPower, you get to choose pretty much everything about your browser such as operating system, user-agents, language and much more.

AdsPower is known for being user-friendly and quite affordable in comparison to similar tools. Proxy integration with AdsPower is quite straightforward, and once you know how to do it, you can be pretty unstoppable on the web. If you decide to follow along with this guide, you'll be up and running in no time.

Introduction​

With AdsPower and ScrapeOps Residential Proxies, we get a very powerful combination of tools. When you master them both, you'll be able to accomplish all sorts of valuable tasks on the web.

AdsPower​

As we mentioned earlier, AdsPower has been around since 2018. Since their inception, they've been very good about adding features and improving the user experience.

With AdsPower, you can create custom browser profiles, automated tasks and proxies. If you want to get past geoblocks on the web, AdsPower is a great way to do this.

It's also a great way to automate multiple accounts for different sites. Whether you're looking to get past a geoblocker, or automate account management, AdsPower is a great skill to have in your scraping toolbox.

AdsPower Home Page

AdsPower is an amazing tool, but it doesn't come with proxies out of the box.

In order to get the most out of AdsPower, you need to integrate it with a proxy service. This allows you to get the results you want, no matter where you are in the world. Good proxies not only get the site you want, but they protect your privacy and security without sacrificing performance.

ScrapeOps Residential Proxies​

A proxy acts as a middleman between your browser and the rest of the world. Normally, when you visit a website, your browser sends an HTTP request to that website.

When you're using a proxy, your browser actually makes the request to the proxy server. The server then fetches the website for you. This is a great way to remain anonymous on the web and also enhance your security. You're not interfacing directly with potentially malicious sites because the proxy is the one interacting with them.

ScrapeOps Residential Proxies take it a step further. Not only do you get the middleman of a proxy, your HTTP requests actually go through real residential networks.

Many proxy services give you a datacenter IP and this often sticks out like a sore thumb on the web. With ScrapeOps Residential Proxies, your browser sends a request to our server. Then, our server sends that request through someone's actual home network.

Once it receives a valid response, our server sends the response back to you. You can actually use a website without ever directly interacting with it.

Prerequisites​

To make it through this tutorial, you're going to need a couple of things: a computer, an AdsPower account, and a ScrapeOps account. AdsPower offers a free tier which is what we'll be using today. You can also create a ScrapeOps account with a free trial as well.

In these next few sections, we'll show you how to do all of that.

Creating an AdsPower Account​

First, head over to the AdsPower registration page.

Enter the information required to create your account. Have your email ready, they'll send you a verification code before you even click Sign up.

AdsPower Registration

Once you've created your account, you're ready to go.

Creating a ScrapeOps Account​

Now, you need a ScrapeOps account. If you've already got one, feel free to skip this section, just make sure you've got your API key.

To create a ScrapeOps account, go to our registration page.

ScrapeOps Registration

Once you've registered, you'll arrive at our dashboard. On the left hand side, click Proxy Aggregator. Next, click Residential Proxy Aggregator as you see in the shot below. If you click Request Builder, it will reveal your API key. Store this key somewhere safe.

ScrapeOps Dashboard

Once you've got your ScrapeOps API key and your AdsPower account, you're good to go!

Setting Up AdsPower With ScrapeOps Residential Proxies​

Now, we'll go through and set everything up. First, we'll download AdsPower. Then we'll set up our first browser profile. While we're setting it up, we'll hook it up to our Residential Proxy Aggregator.

If you follow the steps we've outlined below, you'll be up and running in just a few minutes.

  1. First, go to AdsPower's downloads page. Download the version matching your operating system.

AdsPower Download Page

  1. If you're on Windows or Mac, just run the installer. For Debian based versions of Linux (like Ubuntu), cd into your Downloads folder and run the command below.
sudo dpkg -i AdsPower-Global-6.10.20-x64.deb
  1. After you're finished installing, you need to create a new browser profile. The browser profile holds all the custom settings you'll want to use with your browser. Choose the version of SunBrowser (Chrome) that you'd like to use.

Go ahead and choose an operating system you'd like to appear as too. This is not the operating system you're running, this is the operating system that website servers see you running.

Proxy Setup

  1. If you scroll down, you can enter your proxy details.
  • Our Proxy type is HTTP.
  • The Host is residential-proxy.scrapeops.io.
  • Our Port is 8181.
  • Your Proxy username is scrapeops.
  • In the Proxy Password section, enter your ScrapeOps API key.

Proxy Setup

  1. Once you're finished entering your proxy details, click the Check Proxy button. If everything is correct, a green message should pop up. It should begin with the phrase, "Connection test passed!"

Proxy Check

Testing the Integration​

AdsPower comes with a built-in browserscan. This is what we'll use to test our actual proxy integration. If our browser shows up in a location other than our own, the proxy is working correctly.

As soon as we open our custom browser, our location and IP information get scanned from the browser.

In the Profiles section, find the browser profile you just created and click the Open button.

Open the Browser

As you can see in the screenshot below, we're showing up in London. Our proxy is working correctly!

AdsPower Location Info

Configuring AdsPower for Proxy Integration​

We can fine-tune our browser by simply editing our Proxy username. In the next couple sections, we'll add geotargeting and then we'll connect using a sticky session.

When we use geotargeting, we actually get to choose the location of our browser. When we use a sticky session, we can reuse an IP address over multiple requests.

We make these tweaks by adding different flags to our Proxy username. When you add these flags, our proxy server reads them and uses them as instructions.

If you can follow along in the examples below, you'll be able to control SunBrowser like a pro.

Geotargeting​

We can use geotargeting by adding the country flag to our username. Our proxy server sees country and it knows that you want your requests routed through a custom country.

If you look at the example below, we changed our Proxy username to scrapeops.country=us. us is the country code for the US. You can view a list of fully supported countries here.

Edit Country

Look at the screenshot below. Our location is showing up as Atlanta, Georgia, US. Our geotargeting is working perfectly.

Verify Country

To use geotargeting properly, you need to append the country flag to your username. You follow it up with your two letter country code.

  • If you wish to appear in the US, your username would be scrapeops.country=us.
  • If you wish to appear in the UK, your username would be scrapeops.country=uk.

Using Static Proxies​

Static proxies (sticky sessions) allow us to reuse the same IP address over the course of multiple requests. If you need to remain logged in, or keep a session intact for any other reason, these sticky sessions are imperative.

To use a sticky session, you'll append your username with the sticky_session flag. When our proxy server sees sticky_session, it knows that you want to keep a browsing session intact. You also need to give your session a number. This can be any number between 0 and 10000. Sessions are valid for a total of 10 minutes.

In this next image, we edit our Proxy username. It's now scrapeops.sticky_session=7. sticky_session tells the ScrapeOps server that we're using a sticky session. 7 is the ID number of the session. Any requests including sticky_session=7 will all go to the same IP that we receive here.

If we make a new request and use sticky_session=6, this will create a new session on a new machine with the ID, 6.

Edit Sticky Session

To test our sticky session, click Open like you did earlier. As you can see in the image below, we're showing up in Portugal. We've got an IPv6 address ending in 50a0.

Sticky Session: First Load

If you look at our next image, we're showing up in Portugal and we still have an IPv6 address ending with 50a0.

Sticky Session: Second Load

To use a static proxy, you need to use the sticky_session flag on your username. Each session gets assigned a number anywhere between 0 and 10000. If your Proxy username is scrapeops.sticky_session=7, this creates a sticky session with a session number, 7.

Common Issues and Troubleshooting​

With AdsPower, there are a number of issues we can run into. There are some general problems we might have with AdsPower such as a browser not being installed, or we're getting our display in an incorrect language.

There are also some proxy related issues we might run into such as failed connection, geotargeting problems, and issues related to sticky sessions.

Browser Not Installed​

If you've got an incorrect browser version, you can actually see it in your profile settings. AdsPower will give you a prompt to install the correct browser version.

Download Browser Prompt

Once your browser has been downloaded, click the Install button. After it's finished installing, you should be all set to run your new browser instance.

Install the Browser

Incorrect Language​

By default, our language is automatically set based on our location. This can create some major problems if you're getting responses in a language you don't understand. To fix this, once again, you need to edit your browser profile. Scroll down to Language. The default setting is Based on IP. You need to change it to Custom.

Custom Language Settings

After you select Custom, you'll see your options to choose from. If your desired language isn't listed, click Add Language.

Choosing a Language

Proxy Fails To Connect​

If your proxy isn't working at all, you need to open up your proxy settings in the browser profile. Make sure all of your settings are correct. You can double-check them against the settings listed below.

  • Proxy type: HTTP
  • Host:Port: residential-proxy.scrapeops.io:8181
  • Proxy username: scrapeops
  • Proxy Password: Your ScrapeOps API key

Geotargeting Issues​

When geotargeting, you should have the country flag appended to your username. Pass in a two letter country code along with country. To show up in the US, your username will be scrapeops.country=us.

To make sure your country of choice is fully supported, double-check it against our list here.

Static Proxy Issues​

First, double-check your username. It should be scrapeops.sticky_session={your-session-number}. If your username is correct, you might be running into expiration problems. Even if you're using a sticky session, your IPs will get rotated every 10 minutes.

We rotate these IP addresses to keep our proxies healthy. This also helps keep your identity safe and protects all of our users as a whole. If you're running into expiration issues, the only thing you can do is speed up your browsing. Make sure that you accomplish everything within the allotted 10 minute window.

Conclusion​

You did it! You now know how to use AdsPower and also how to use ScrapeOps Residential Proxies. This is a really powerful combination. You can enjoy the benefits of anonymous browsing, geotargeting and sticky sessions. Use these new skills to your advantage. Go forth and collect all the data you can find!

If you'd like to know more about these tools, take a look at the links below.

More Python Web Scraping Guides​

We've got all sorts of awesome stuff here at ScrapeOps. We love scraping so much that we've written multiple playbooks on it. You can view our general Python Web Scraping Playbook here.

We also have playbooks for Selenium and Scrapy. If you're looking to write some JavaScript, take a look at our Puppeteer and Playwright playbooks.

For a taste of some of our other content, check out the links below.