Skip to main content

Java Code Examples

The following are code examples on how to integrate the ScrapeOps Residential Proxy Aggregator with your Java scrapers using the Java HttpClient library.

Authorisation - API Key​

To use the ScrapeOps proxy, you first need an API key which you can get by signing up for a free account here.

Your API key must be included with every request using the password proxy port parameter otherwise the proxy port will return a 403 Forbidden Access status code.

Basic Request​

The following is some example code to send a URL to the ScrapeOps Proxy port http://scrapeops:YOUR_API_KEY@residential-proxy.scrapeops.io:8181:


import java.io.IOException;
import java.net.*;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.List;

public class Main {
    public static void main(String[] args) {
        String apiKey = "YOUR_API_KEY";
        String targetUrl = "https://httpbin.org/ip";
        String proxyUrl = "http://scrapeops:" + apiKey + "@residential-proxy.scrapeops.io:8181";

        ProxySelector proxySelector = new ProxySelector() {
            @Override
            public List<Proxy> select(URI uri) {
                return List.of(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("residential-proxy.scrapeops.io", 8181)));
            }

            @Override
            public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
                System.err.println("Proxy connection failed: " + ioe.getMessage());
            }
        };

        HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(120))
                .proxy(proxySelector)
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(targetUrl))
                .timeout(Duration.ofSeconds(120))
                .header("Proxy-Authorization", "Basic " + Base64.getEncoder().encodeToString(("scrapeops:" + apiKey).getBytes()))
                .GET()
                .build();

        try {
            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
            System.out.println("Body: " + response.body());
        } catch (IOException | InterruptedException e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}


ScrapeOps will take care of the proxy selection and rotation for you so you just need to send us the URL you want to scrape.

Response Format​

After receiving a response from one of our proxy providers the ScrapeOps Residential Proxy Aggregator will then respond with the raw HTML content of the target URL along with a response code:


<html>
  <head>
     ...
  </head>
  <body>
     ...
  </body>
</html>

Status Codes​

The ScrapeOps Residential Proxy Aggregator will return the status code returned by the target website.

However, if the proxy port will return the following status codes if there are specific errors in your request:

Status CodeBilledDescription
400NoBad request. Either your url or query parameters are incorrectly formatted.
401NoYou have consumed all your credits. Either turn off your scraper, or upgrade to a larger plan.
403NoEither no api_key included on request, or api_key is invalid.

Here is the full list of status codes the Proxy Port returns.

Advanced Functionality​

To enable advanced proxy functionality when using the Residential Proxy endpoint you need to pass parameters by adding them to username, separated by periods.

For example, if you want to enable Country Geotargeting with a request, the username would be scrapeops.country=us.


import java.io.IOException;
import java.net.*;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.List;

public class Main {
    public static void main(String[] args) {
        String apiKey = "YOUR_API_KEY";
        String targetUrl = "https://httpbin.org/ip";
        String proxyUrl = "http://scrapeops.country=us:" + apiKey + "@residential-proxy.scrapeops.io:8181";

        ProxySelector proxySelector = new ProxySelector() {
            @Override
            public List<Proxy> select(URI uri) {
                return List.of(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("residential-proxy.scrapeops.io", 8181)));
            }

            @Override
            public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
                System.err.println("Proxy connection failed: " + ioe.getMessage());
            }
        };

        HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(120))
                .proxy(proxySelector)
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(targetUrl))
                .timeout(Duration.ofSeconds(120))
                .header("Proxy-Authorization", "Basic " + Base64.getEncoder().encodeToString(("scrapeops:" + apiKey).getBytes()))
                .GET()
                .build();

        try {
            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
            System.out.println("Body: " + response.body());
        } catch (IOException | InterruptedException e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}



Check out this guide to see the full list of advanced functionality available.

Timeout​

The ScrapeOps proxy keeps retrying a request for up to 2 minutes before returning a failed response to you.

To use the Proxy correctly, you should set the timeout on your request to a least 2 minutes to avoid you getting charged for any bandwidth that you consumed before the Proxy responded.