Skip to main content

Java OkHttp: Retry Failed Requests

Java OkHttp: Retry Failed Requests

In this guide for The Java Web Scraping Playbook, we will look at how to configure the Java OkHttp library to retry failed requests so you can build a more reliable system.

There are a couple of ways to approach this, so in this guide we will walk you through the 2 most common ways to retry failed requests and show you how to use them with the Java OkHttp:

Let's begin...

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Retry Failed Requests Using Retry Library​

Here we use the Retry4j package to define the retry logic and trigger any retries on failed requests.

Here is an example:


import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.Callable;

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;

import com.evanlennick.retry4j.CallExecutor;
import com.evanlennick.retry4j.CallExecutorBuilder;
import com.evanlennick.retry4j.Status;
import com.evanlennick.retry4j.config.RetryConfig;
import com.evanlennick.retry4j.config.RetryConfigBuilder;

public class RetryFailedRequests {
    public static List<Integer> badStatusCodes = new ArrayList<>(Arrays.asList(429, 500, 502, 503, 504));
    public static void main(String[] args) throws Exception {
        OkHttpClient client = new OkHttpClient();

        Callable<Void> makeRequest = () -> {
            Request request = new Request.Builder()
                    .url("https://quotes.toscrape.com")
                    .build();
            Response response = client.newCall(request).execute();
            int statusCode = response.code();
            if (badStatusCodes.contains(statusCode)) {
                throw new Exception("Bad status code: " + statusCode);
            }
            String responseBodyText = response.body().string();
            System.out.println("Response body: " + responseBodyText);
            return null;
        };
        RetryConfig config = new RetryConfigBuilder()
                .retryOnAnyException()
                .withMaxNumberOfTries(5)
                .withDelayBetweenTries(10, ChronoUnit.SECONDS)
                .withExponentialBackoff()
                .build();

        CallExecutor callExecutor = new CallExecutorBuilder()
                .config(config)
                .onFailureListener((Status s) -> {
                    System.out.println("Maximum number of retries reached.");

                })
                .afterFailedTryListener((Status s) -> {
                    System.out.println("Total tries: " + s.getTotalTries());
                })
                .build();
        callExecutor.execute(makeRequest);
    }
}

In the above code, we use the java OkHttp library to send HTTP requests with retry functionality. We also utilize the retry4j package to control the retry behavior.

First, we initialize badStatusCodes to a list of status codes which should trigger retry. We then initialize a Callable named makeRequest to contain our request logic. Inside makeRequest, we check if the response status code is in badStatusCodes list. If it is, we throw an exception in order to trigger retry.

Next, we use RetryConfigBuilder to define the our retry config parameters, including the maximum number of retries, the condition that triggers retry, the delay between retries, and the backoff function to calculate each successive delay between retries:

  • retryOnAnyException(): This specifies that the retry mechanism should trigger a retry for any exception that occurs during the operation.

  • withMaxNumberOfTries(5): This sets the maximum number of retry attempts to 5.

  • withDelayBetweenTries(10, ChronoUnit.SECONDS): This sets a delay of 10 seconds between each retry attempt.

  • withExponentialBackoff(): This indicates that exponential backoff function should be used for calculating the delay between retry attempts. This approach is often used to avoid overwhelming a service with rapid retries when it's already experiencing issues.

Next we use CallExecutorBuilder, also from retry4j, to create a callExecutor for controlling and executing our retry operation based on config we just defined. We also configure the callExecutor with these listeners:

  • onFailureListener: This listener gets called after we reach maximum number of retries and the operation still hasn't succeeded.

  • afterFailedTryListener: It gets called after each failed retry attempt. Here we print the total number of tries made so far.

Finally we use callExecutor.execute(makeRequest) to run our request through the retry mechanism.

Build Your Own Retry Logic Wrapper​

Another method of retrying failed requests with Java OkHttp is to build your own retry logic around your request functions.


import java.net.ConnectException;

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;

public class CustomRetryLogic {
    final public static int NUM_RETRIES = 3;

    public static void main(String[] args) throws Exception {
        // initialization
        Response response = null;
        OkHttpClient client = new OkHttpClient();

        for (int i = 0; i < NUM_RETRIES; i++) {
            try {
                Request request = new Request.Builder()
                    .url("https://quotes.toscrape.com")
                    .build();
                response = client.newCall(request).execute();

                int status = response.code();
                if (status == 200 || status == 404) {
                    // escape the loop if a successful response is returned or the page is not found
                    break;
                }
            } catch (Exception e) {
                boolean connectionError = e instanceof ConnectException;
                if (connectionError) {
                    // handle connection exceptions
                }
            } finally {
                System.out.println("Total tries: " + (i + 1));
            }
        }

        if (response != null && response.code() == 200) {
            // do something with the successful response
            System.out.println("Response body: " + response.body().string());
        } else {
            System.out.println("No valid response after maximum number of tries");
        }
    }
}

In the above code, we use OkHttp library to send HTTP requests and use our custom retry wrapper logic to handle retries. We first initialize a variable response to store the response from the successful request.

We then use a for loop with a maximum of NUM_RETRIES iterations. Inside the loop, we make a GET request to the specified URL. If the response status code is either 200 or 404, we break out of the loop.

If a connection error occurs, we catch the error and continue to the next iteration.

Finally, after the loop, we check whether the response variable is not null and has a status code of 200. If these conditions are met, you can perform actions with the successful response.

The advantage of this approach is that you have a lot of control over what is a failed response.

Previously, we examined only the response code to determine whether we should retry the request. However, we could modify this to include a check for the response's validity by examining its HTML content

Below we will add an additional check to make sure the HTML response doesn't contain a ban page.


import java.net.ConnectException;

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import org.jsoup.Jsoup;

public class CustomRetryLogic {
    final public static int NUM_RETRIES = 3;

    public static void main(String[] args) throws Exception {
        // initialization
        Response response = null;
        String responseBodyText = null;
        boolean validResponse = false;
        OkHttpClient client = new OkHttpClient();

        for (int i = 0; i < NUM_RETRIES; i++) {
            try {
                Request request = new Request.Builder()
                    .url("https://quotes.toscrape.com")
                    .build();
                response = client.newCall(request).execute();
                responseBodyText = response.body().string();

                //// use Jsoup to parse pageTitle
                String pageTitle = Jsoup.parse(responseBodyText).title();

                int status = response.code();
                boolean validStatus = status == 200 || status == 404;
                if (validStatus && !pageTitle.contains("Robot or human?")) {
                    // escape the loop if valid status code is returned and the expected content is not present
                    validResponse = true;
                    break;
                }
            } catch (Exception e) {
                boolean connectionError = e instanceof ConnectException;
                if (connectionError) {
                    // handle connection exceptions
                }
            } finally {
                System.out.println("Total tries: " + (i + 1));
            }
        }

        if (response != null && validResponse && response.code() == 200) {
            // do something with the successful response
            System.out.println("Response body: " + responseBodyText);
        } else {
            System.out.println("No valid response after maximum number of tries");
        }
    }
}

In this example, we also check the successful 200 status code responses to make sure they don't contain a ban page.


"<title>Robot or human?</title>"

If it does then the code will retry the request.

More Web Scraping Tutorials​

So that's how you can configure Java OkHttp Library to automatically retry failed requests.

If you would like to learn more about Web Scraping, then be sure to check out The Web Scraping Playbook.

Or check out one of our more in-depth guides: