Python Requests: How To Retry Failed Requests (2026 Guide)

TL;DR — Python Requests does not retry failed requests automatically. Mount a urllib3.util.Retry strategy on a requests.Session via an HTTPAdapter, or decorate your call with tenacity.@retry. Always include exponential backoff with jitter, only retry on 408, 429 and 5xx by default, and respect the Retry-After header.

In this guide for The Python Web Scraping Playbook, we'll look at how to configure the Python Requests library to retry failed requests so you can build a more reliable system. There are several ways to do this and we'll walk through each in turn:

• Which status codes should I retry?
• Retry With Sessions & HTTPAdapter (urllib3 Retry) — the built-in option
• Retry With Tenacity — the most flexible option
• Build Your Own Retry Logic Wrapper — the simplest custom option
• Respecting the Retry-After Header
• Retrying Async Requests with httpx
• Idempotency: When You Shouldn't Retry
• Frequently Asked Questions

Let's begin...

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Proxy Manager

Scraper Monitoring

Job Scheduling

Which status codes should I retry?​

Before you wire up any retry library, decide which failures actually deserve a retry. Retrying a 404 is pointless. Retrying a 403 just gets you blocked harder. The right default set in 2026 is:

A short rule of thumb: retry on connection errors, 408, 425, 429 and 5xx — and nothing else. Everything else needs a code fix, not a retry.

Retry Failed Requests Using Sessions & HTTPAdapter​

If you are okay with using Python Sessions, then you can define the retry logic using a HTTPAdapter. This is the built-in option — no extra dependencies — and it's almost always the right starting point.

Here is the minimal example:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = requests.Session()

retries = Retry(
    total=5,
    backoff_factor=1,
    status_forcelist=[408, 429, 500, 502, 503, 504],
    allowed_methods=["HEAD", "GET", "OPTIONS"],
    respect_retry_after_header=True,
)

s.mount("http://", HTTPAdapter(max_retries=retries))
s.mount("https://", HTTPAdapter(max_retries=retries))

response = s.get("http://quotes.toscrape.com/")

Here we:

1. Create a retry strategy with urllib3's Retry util, telling it how many retries to make, which status codes to retry on (status_forcelist), and which HTTP methods to retry (allowed_methods — important for idempotency, see below).
2. Tell it to honour the Retry-After header when the server sends one back.
3. Add this retry strategy to an HTTPAdapter and mount it on the session for both http:// and https://.

Exponential backoff with backoff_factor​

The backoff_factor controls how long we wait between retries. urllib3 uses the formula:

{backoff_factor} * (2 ** ({number_retries} - 1))

Here are some example sleep sequences different backoff factors produce:

## backoff_factor = 1
0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256

## backoff_factor = 2
1, 2, 4, 8, 16, 32, 64, 128, 256, 512

## backoff_factor = 3
5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560

backoff_factor=1 is a sensible default for web scraping. If you need jitter (a small random offset on each delay, recommended for high-concurrency workloads), urllib3 added a backoff_jitter parameter in 2.0:

retries = Retry(
    total=5,
    backoff_factor=1,
    backoff_jitter=0.5,  # add up to 0.5s of random jitter
    status_forcelist=[408, 429, 500, 502, 503, 504],
)

Retry Failed Requests Using Tenacity​

Tenacity is the most popular general-purpose retry library in the Python ecosystem (it's a fork of the now-archived retrying library). It works as a decorator and is much more flexible than the urllib3 Retry adapter — you can retry based on exceptions, response content, custom predicates, or all three together.

Install it with pip install tenacity and then:

import requests
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential_jitter,
    retry_if_exception_type,
    retry_if_result,
)

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}

def _is_retryable_response(response):
    return response is not None and response.status_code in RETRYABLE_STATUS

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=1, max=60, jitter=2),
    retry=(
        retry_if_exception_type(requests.exceptions.RequestException)
        | retry_if_result(_is_retryable_response)
    ),
    reraise=True,
)
def fetch(url, **kwargs):
    response = requests.get(url, timeout=30, **kwargs)
    return response

response = fetch("http://quotes.toscrape.com/")
print(response.status_code)

Why reach for tenacity instead of the urllib3 Retry adapter?

• Retry on the response body, not just the status code. You can plug in any callable — for example, retrying when the HTML looks like an anti-bot challenge page even though the response is 200 OK.
• Custom wait strategies like wait_exponential_jitter, wait_random_exponential, or wait_chain for staged backoff.
• Works the same way for sync and async code, so you can keep a single retry helper as you migrate to httpx and asyncio.
• Logs and callbacks — before_sleep, after, retry_error_callback make it easy to wire retries into your observability stack.

A common scraping pattern is to retry both on transport errors and on HTML that looks like a soft block:

from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_result

def _looks_like_ban_page(response):
    if response is None:
        return False
    if response.status_code != 200:
        return response.status_code in {408, 425, 429, 500, 502, 503, 504}
    return "Robot or human?" in response.text or "Just a moment" in response.text

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=1, max=30),
    retry=retry_if_result(_looks_like_ban_page),
    reraise=True,
)
def fetch(url):
    return requests.get(url, timeout=30)

Respecting the Retry-After Header​

When a server returns 429 Too Many Requests or 503 Service Unavailable it often includes a Retry-After header telling you exactly when it's safe to come back. Ignoring it is the fastest way to get your IP throttled harder.

• If you're using the urllib3 Retry adapter, set respect_retry_after_header=True (it's true by default in modern urllib3, but be explicit).
• If you're rolling your own loop or using tenacity, read the header yourself:

import time
import requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def _retry_after_seconds(response, default=5):
    header = response.headers.get("Retry-After")
    if not header:
        return default
    if header.isdigit():
        return int(header)
    # HTTP-date format
    try:
        when = parsedate_to_datetime(header)
        return max(0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return default

response = requests.get("http://quotes.toscrape.com/")
if response.status_code in (429, 503):
    wait = _retry_after_seconds(response)
    time.sleep(wait)
    response = requests.get("http://quotes.toscrape.com/")

The Retry-After header can be either an integer number of seconds or an HTTP-date — handle both. Cap the maximum wait to something sensible (e.g. 5 minutes) so a misconfigured server can't stall your crawl indefinitely.

Retrying Async Requests with httpx​

If you've moved to httpx for async scraping, the cleanest approach is to combine tenacity's async decorator with an httpx.AsyncClient:

import asyncio
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=1, max=30),
    retry=retry_if_exception_type((httpx.TransportError, httpx.HTTPStatusError)),
    reraise=True,
)
async def fetch(client, url):
    response = await client.get(url, timeout=30)
    response.raise_for_status()
    return response

async def main():
    async with httpx.AsyncClient() as client:
        response = await fetch(client, "http://quotes.toscrape.com/")
        print(response.status_code)

asyncio.run(main())

response.raise_for_status() converts any 4xx/5xx response into an HTTPStatusError, which the retry_if_exception_type predicate then catches. If you want to retry only specific status codes, raise a more specific exception or use retry_if_result against response.status_code.

httpx also supports custom transports, so you can pin a retry policy directly to a client via httpx.HTTPTransport(retries=N) — but that only retries connection errors, not status codes, so tenacity remains the more capable option.

Build Your Own Retry Logic Wrapper​

Another method of retrying failed requests with Python Requests is to build your own retry logic around your request functions.

import requests

NUM_RETRIES = 3
for _ in range(NUM_RETRIES):
    try:
        response = requests.get('http://quotes.toscrape.com/')
        if response.status_code in [200, 404]:
            ## Escape for loop if returns a successful response
            break
    except requests.exceptions.ConnectionError:
        pass
    
## Do something with successful response
if response is not None and response.status_code == 200:
    pass

The advantage of this approach is that you have a lot of control over what is a failed response.

Above we are only look at the response code to see if we should retry the request, however, we could adapt this so that we also check the response to make sure the HTML response is valid.

Below we will add an additional check to make sure the HTML response doesn't contain a ban page.

import requests

NUM_RETRIES = 3
for _ in range(NUM_RETRIES):
    try:
        response = requests.get('http://quotes.toscrape.com/')
        if response.status_code in [200, 404]:
            if response.status_code == 200 and '<title>Robot or human?</title>' not in response.text:
                break
    except requests.exceptions.ConnectionError:
        pass
    
## Do something with successful response
if response is not None and response.status_code == 200:
    pass

We could also wrap this logic into our own request_retry function if we like:

import requests

def request_retry(url, num_retries=3, success_list=[200, 404], **kwargs):
    for _ in range(num_retries):
        try:
            response = requests.get(url, **kwargs)
            if response.status_code in success_list:
                ## Return response if successful
                return response
        except requests.exceptions.ConnectionError:
            pass
    return None

response = request_retry('http://quotes.toscrape.com/')

Idempotency: When You Shouldn't Retry​

Retrying a request that has side effects can create duplicates. The HTTP spec classes some methods as idempotent — repeating them produces the same end state — and some as not:

urllib3.util.Retry respects this by default: its allowed_methods parameter lists only the idempotent methods (HEAD, GET, OPTIONS, PUT, DELETE, TRACE). Tenacity puts the responsibility on you — if you wrap a POST helper with @retry, retrying on a 5xx will happily re-submit the body.

For POST requests in scraping pipelines (e.g. submitting a form, calling a webhook), the safest pattern is:

import requests
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=1, max=30),
    # Only retry on transport errors — never on 4xx/5xx after the server may have processed the POST.
    retry=retry_if_exception_type(requests.exceptions.ConnectionError),
    reraise=True,
)
def post_form(url, data):
    return requests.post(url, data=data, timeout=30)

If the server you're calling supports an Idempotency-Key header (Stripe, GitHub and most modern APIs do), send a stable UUID per logical request so the server can deduplicate retried POSTs for you.

Frequently Asked Questions​

More Web Scraping Tutorials​

So that's how you can configure Python Requests to automatically retry failed requests.

If you would like to learn more about Web Scraping, then be sure to check out The Web Scraping Playbook.

Or check out one of our more in-depth guides:

• How To Solve 403 Forbidden Errors When Web Scraping — when retries aren't enough and the server is actively blocking your scraper.
• How To Bypass Cloudflare — for sites where retries return 403 or a challenge page.
• How to Scrape The Web Without Getting Blocked Guide
• The Ethics of Web Scraping