Skip to main content

Python aiohttp - Setting Fake User-Agents

Python aiohttp: Setting Fake User-Agents

Using Fake User-Agents With Python aiohttp

To use fake user-agents with Python aiohttp first create a session object using the aiohttp.ClientSession() method and then use the get() method on that session object. Then you just need to define a user-agent in a headers dictionary and pass it into the headers attribute of your request.


import aiohttp
import asyncio

headers = {"User-Agent": "Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"}

async def make_request():
    async with aiohttp.ClientSession() as session:
        response = await session.get('http://httpbin.org/headers', headers=headers)
        print(await response.json())

asyncio.run(make_request())


One of the most common reasons for getting blocked whilst web scraping is using bad user-agents.

However, integrating fake user-agents into your Python web scrapers is very easy.

So in this guide, we will go through:

First, let's quickly go over some the very basics.

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

What Are Fake User-Agents?​

User Agents are strings that let the website you are scraping identify the application, operating system (OSX/Windows/Linux), browser (Chrome/Firefox/Internet Explorer), etc. of the user sending a request to their website. They are sent to the server as part of the request headers.

Here is an example User agent sent when you visit a website with a Chrome browser:


'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36'

When scraping a website, you also need to set user-agents on every request as otherwise the website may block your requests because it knows you aren't a real user.

In the case of most Python HTTP clients like Python aiohttp, when you send a request the default settings clearly identify that the request is being made with Python aiohttp in the user-agent string.


'User-Agent': 'Python/3.9 aiohttp/3.7.4',

This user-agent will clearly identify your requests are being made by the Python aiohttp library, so the website can easily block you from scraping the site.

That is why we need to manage the user-agents we use with Python aiohttp when we send requests.

How To Set A Fake User-Agent In Python aiohttp​

Setting Python aiohttp to use a fake user-agent is very easy. We need to use aiohttp.ClientSession() to create a new instance of the ClientSession class. The async with statement is used to create a context manager for the aiohttp.ClientSession() to manage the life cycle of the session object. Next, we make a GET request using the session.get() method. Then we just need to define it in a headers dictionary and add it to the request using the headers parameter.


import aiohttp
import asyncio

headers = {"User-Agent": "Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"}

async def make_request():
    async with aiohttp.ClientSession() as session:
        response = await session.get('http://httpbin.org/headers', headers=headers)
        print(await response.json())

asyncio.run(make_request())

Link to the official documentation.

How To Set A Fake User-Agent In Python aiohttp Sessions​

When using Python aiohttp Sessions functionality, we just need to define headers on the session instance.


import aiohttp
import asyncio

headers = {"User-Agent": "Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"}

async def make_request():
    async with aiohttp.ClientSession() as session:
        response = await session.get('http://httpbin.org/headers', headers=headers)
        print(await response.json())

asyncio.run(make_request())

How To Rotate User-Agents​

Rotating through user-agents is also pretty straightforward when using Python aiohttp. We just need a list of user-agents in our scraper and use a random one with every request.

import aiohttp
import asyncio
import random

user_agent_list = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
    'Mozilla/5.0 (iPhone; CPU iPhone OS 14_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1',
    'Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363',
]

headers={"User-Agent": user_agent_list[random.randint(0, len(user_agent_list)-1)]}

async def make_request():
    async with aiohttp.ClientSession() as session:
        response = await session.get('http://httpbin.org/headers', headers=headers)
        print(await response.json())

asyncio.run(make_request())

This works but it has drawbacks as we would need to build & keep an up-to-date list of user-agents ourselves.

How To Manage Thousands of Fake User-Agents​

A better approach would be to use a free user-agent API like ScrapeOps Fake User-Agent API to download an up-to-date user-agent list when your scraper starts up and then pick a random user-agent for each request.

To use the ScrapeOps Fake User-Agents API you just need to send a request to the API endpoint to retrieve a list of user-agents.


http://headers.scrapeops.io/v1/user-agents?api_key=YOUR_API_KEY

To use the ScrapeOps Fake User-Agent API, you first need an API key which you can get by signing up for a free account here.

Example response from the API:


{
  "result": [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Windows; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8",
    "Mozilla/5.0 (Windows NT 10.0; Windows; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15",
    "Mozilla/5.0 (Windows NT 10.0; Windows; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36"
  ]
}

To integrate the Fake User-Agent API you should configure your scraper to retrieve a batch of the most up-to-date user-agents when the scraper starts and then configure your scraper to pick a random user-agent from this list for each request.

Here is an example Python aiohttp scraper integration:


import aiohttp
import asyncio
import random

SCRAPEOPS_API_KEY = 'YOUR_API_KEY'

async def get_user_agent_list():
    async with aiohttp.ClientSession() as session:
        url = f'http://headers.scrapeops.io/v1/user-agents?api_key={SCRAPEOPS_API_KEY}'
        async with session.get(url) as response:
            json_response = await response.json()
            return json_response.get('result', [])

async def get_random_user_agent(user_agent_list):
    random_index = random.randint(0, len(user_agent_list) - 1)
    return user_agent_list[random_index]

async def make_request(url, user_agent_list):
    ## Add Random User-Agent To Headers
    headers = {'User-Agent': await get_random_user_agent(user_agent_list)}
    async with aiohttp.ClientSession() as session:
        async with session.get(url, headers=headers) as response:
            return await response.text()

async def main():
    ## Retrieve User-Agent List From ScrapeOps
    user_agent_list = await get_user_agent_list()

    url_list = [
        'https://example.com/1',
        'https://example.com/2',
        'https://example.com/3',
    ]

    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in url_list:
            task = asyncio.create_task(make_request(url, user_agent_list))
            tasks.append(task)

        responses = await asyncio.gather(*tasks)
        for response in responses:
            print(response)

asyncio.run(main())

Here the scraper will use a random user-agent for each request.

Why Use Fake Browser Headers​

For simple websites, simply setting an up-to-date user-agent should allow you to scrape a website pretty reliably.

However, a lot of popular websites are increasingly using sophisticated anti-bot technologies to try and prevent developer from scraping data from their websites.

These anti-bot solutions not only look at your requests user-agent when analysing the request, but also the other headers a real browser normally sends.

By using a full set of browser headers you make your requests look more like real user requests, and as a result harder to detect.

Here are example headers when using a Chrome browser on a MacOS machine:

sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

As we can see, real browsers don't just send User-Agent strings but also a number of other headers that are used to identify and customize the request.

So to improve the reliability of our scrapers we should also include these headers when making requests.

You could build a list of fake browser headers yourself, or you could use the ScrapeOps Fake Browser Headers API to get an up-to-date list every time your scraper starts up.

ScrapeOps Fake Browser Headers API​

The ScrapeOps Fake Browser Headers API is a free API that returns a list of optimized fake browser headers that you can use in your web scrapers to avoid blocks/bans and improve the reliability of your scrapers.

API Endpoint:


http://headers.scrapeops.io/v1/browser-headers?api_key=YOUR_API_KEY

Response:


{
  "result": [
    {
      "upgrade-insecure-requests": "1",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Windows; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "sec-ch-ua": "\".Not/A)Brand\";v=\"99\", \"Google Chrome\";v=\"103\", \"Chromium\";v=\"103\"",
      "sec-ch-ua-mobile": "?0",
      "sec-ch-ua-platform": "\"Windows\"",
      "sec-fetch-site": "none",
      "sec-fetch-mode": "",
      "sec-fetch-user": "?1",
      "accept-encoding": "gzip, deflate, br",
      "accept-language": "bg-BG,bg;q=0.9,en-US;q=0.8,en;q=0.7"
    },
    {
      "upgrade-insecure-requests": "1",
      "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36",
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "sec-ch-ua": "\".Not/A)Brand\";v=\"99\", \"Google Chrome\";v=\"103\", \"Chromium\";v=\"103\"",
      "sec-ch-ua-mobile": "?0",
      "sec-ch-ua-platform": "\"Linux\"",
      "sec-fetch-site": "none",
      "sec-fetch-mode": "",
      "sec-fetch-user": "?1",
      "accept-encoding": "gzip, deflate, br",
      "accept-language": "fr-CH,fr;q=0.9,en-US;q=0.8,en;q=0.7"
    }
  ]
}

To use the ScrapeOps Fake Browser Headers API, you first need an API key which you can get by signing up for a free account here.

To integrate the Fake Browser Headers API you should configure your scraper to retrieve a batch of the most up-to-date headers when the scraper starts and then configure your scraper to pick a random header from this list for each request.

Here is an example Python aiohttp scraper integration:


import aiohttp
import asyncio
from random import randint

SCRAPEOPS_API_KEY = 'YOUR_API_KEY'

async def get_headers_list():
    async with aiohttp.ClientSession() as session:
        url = f'http://headers.scrapeops.io/v1/browser-headers?api_key={SCRAPEOPS_API_KEY}'
        async with session.get(url) as response:
            json_response = await response.json()
            return json_response.get('result', [])

def get_random_header(header_list):
  random_index = randint(0, len(header_list) - 1)
  return header_list[random_index]

async def make_request(url, header_list):
    headers = get_random_header(header_list)
    async with aiohttp.ClientSession() as session:
        async with session.get(url, headers=headers) as response:
            return await response.text()

async def main():
    header_list = await get_headers_list()

    url_list = [
        'https://example.com/1',
        'https://example.com/2',
        'https://example.com/3',
    ]

    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in url_list:
            task = asyncio.create_task(make_request(url, header_list))
            tasks.append(task)

        responses = await asyncio.gather(*tasks)
        for response in responses:
            print(response)

asyncio.run(main())

For more information on check out the Fake Browser Headers API documenation.

More Web Scraping Tutorials​

So that's how you can set fake user-agents when scraping with Python aiohttp.

If you would like to learn more about Web Scraping, then be sure to check out The Web Scraping Playbook.

Or check out one of our more in-depth guides: