URL Encoding

When using the ScrapeOps Proxy Aggregator API integration method, you should always encode your target URL.

This is because if you send an unencoded URL that contains query parameters then the API can think those query parameters are meant for the API and not part of your URL.

Here is an example unencoded URL:

'https://example.com/?test=hello'

And this is the same URL encoded:

'https%3A%2F%2Fexample.com%2F%3Ftest%3Dhello'

You can use an online tool like MeyerWeb's URL Decoder/Encoder for encoding once off URLs, however, you should implement the encoding into your code.

The following are examples on to encode your URLs in popular programming languages.

---

URL Encoding With Python

Here are some options of how to encode a URL in Python using urllib.parse.quote:

import urllib.parse
encoded_url = urllib.parse.quote('https://example.com/?test=hello')

Also, you can encode the query parameters dictionary that you would pass to a Python Requests request using urlencode:

import requests
from urllib.parse import urlencode

proxy_params = {
      'api_key': 'YOUR_API_KEY',
      'url': 'https://example.com/?test=hello', 
  }

response = requests.get(
  url='https://proxy.scrapeops.io/v1/',
  params=urlencode(proxy_params),
  timeout=120,
)

---

URL Encoding With NodeJs

This is how you would encode a URL with encodeURIComponent:

const encodedUrl = encodeURIComponent('https://example.com/?test=hello');

Here is an alternative method using the querystring library in a request-promise scraper.

const rp = require('request-promise');
const qs = require('querystring');

const proxyParams = {
  api_key: 'YOUR_API_KEY',
  url: 'https://example.com/?test=hello'
};

const proxyUrl = 'https://proxy.scrapeops.io/v1/?' + qs.stringify(proxyParams);
const requestOptions = {
  uri: proxyUrl,
  timeout: 120000
};

rp(requestOptions)
  .then(response => {
    console.log(response);
  })
  .catch(error => {
    console.error(error);
  });

---

URL Encoding With Java

This is how you would encode a URL with URLEncoder:

import java.net.URLEncoder;

public class UrlEncoderExample {

    public static void main(String[] args) {
        String encodedUrl = URLEncoder.encode("https://example.com/?test=hello", "UTF-8");
        System.out.println("Encoded URL: " + encodedUrl);

        // Now you can use the URL in a Java HTTP client library, such as HttpURLConnection or OkHttp
    }
}

The following is a Java code example to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:

import java.io.IOException;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;

public class Main {
    public static void main(String[] args) {
        String apiKey = "YOUR_API_KEY";
        String url = "https://example.com/?test=hello";
        String proxyUrl = "https://proxy.scrapeops.io/v1/";

        String queryString = "api_key=" + apiKey + "&url=" + URLEncoder.encode(url, StandardCharsets.UTF_8);

        HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(120))
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(proxyUrl + "?" + queryString))
                .timeout(Duration.ofSeconds(120))
                .GET()
                .build();

        try {
            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
            System.out.println("Body: " + response.body());
        } catch (IOException | InterruptedException e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

---

URL Encoding With PHP

This is how you would encode a URL with urlencode:

$encodedUrl = urlencode('https://example.com/?test=hello');
echo "Encoded URL: " . $encodedUrl . "\n";

The following is a PHP code example to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:

<?php

$api_key = 'YOUR_API_KEY';
$url = 'https://example.com/?test=hello';
$proxy_url = 'https://proxy.scrapeops.io/v1/';
$encodedUrl = urlencode($url);

$proxy_params = array(
    'api_key' => $api_key,
    'url' => $encodedUrl,
    'render_js' => true
);

$query_string = http_build_query($proxy_params);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $proxy_url . '?' . $query_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);

$response = curl_exec($ch);

if (curl_errno($ch)) {
    echo 'Error: ' . curl_error($ch);
} else {
    echo 'Body: ' . $response;
}

curl_close($ch);

?>

---

URL Encoding With Golang

This is how you would encode a URL with url.QueryEscape:

package main

import (
	"fmt"
	"net/url"
)

func main() {
	encodedUrl := url.QueryEscape("https://example.com/?test=hello")
	fmt.Println("Encoded URL:", encodedUrl)
}

The following is a Golang code example to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

func main() {
	apiKey := "YOUR_API_KEY"
	targetURL := "https://example.com/?test=hello"
	proxyURL := "https://proxy.scrapeops.io/v1/"

	queryParams := url.Values{}
	queryParams.Add("api_key", apiKey)
	queryParams.Add("url", targetURL)

	client := &http.Client{
		Timeout: time.Duration(120 * time.Second),
	}

	req, err := http.NewRequest("GET", proxyURL+"?"+queryParams.Encode(), nil)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	response, err := client.Do(req)
	if err != nil {
		fmt.Println("Error making request:", err)
		return
	}
	defer response.Body.Close()

	if response.StatusCode == http.StatusOK {
		bodyBytes, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Println("Error reading response body:", err)
			return
		}
		fmt.Println("Body:", string(bodyBytes))
	} else {
		fmt.Println("Error:", response.StatusCode)
	}
}

---

URL Encoding With Ruby

This is how you would encode a URL with uri:

require 'uri'

encoded_url = URI.encode_www_form_component('https://example.com/?test=hello')
puts "Encoded URL: #{encoded_url}"

The following is a Ruby code example to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:

require 'httparty'
require 'cgi'

response = HTTParty.get(
  'https://proxy.scrapeops.io/v1/',
  query: {
    'api_key' => 'YOUR_API_KEY',
    'url' => CGI.escape('https://example.com/?test=hello'),
  },
  timeout: 120,
)

puts 'Body: ' + response.body

---

URL Encoding With C#

This is how you would encode a URL in C#:

using System;

class Program
{
    static void Main(string[] args)
    {
        string encodedUrl = Uri.EscapeDataString("https://example.com/?test=hello");
        Console.WriteLine("Encoded URL: " + encodedUrl);

        // Now you can use the URL in a C# HTTP client library, such as HttpClient or WebClient
    }
}

The following is a C# code example to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:

using System;
using System.Net.Http;
using System.Threading.Tasks;

public class ScrapeOpsProxyExample
{
    public static async Task Main(string[] args)
    {
        string apiKey = "YOUR_API_KEY";
        string targetUrl = "https://example.com/?test=hello";
        string proxyUrl = $"https://proxy.scrapeops.io/v1/?api_key={apiKey}&url={Uri.EscapeDataString(targetUrl)}";

        using (HttpClient client = new HttpClient())
        {
            client.Timeout = TimeSpan.FromSeconds(120);
            HttpResponseMessage response = await client.GetAsync(proxyUrl);
            string responseBody = await response.Content.ReadAsStringAsync();

            Console.WriteLine("Body: " + responseBody);
        }
    }
}

---

URL Encoding With R

This is how you would encode a URL with URLencode:

encoded_url <- URLencode('https://example.com/?test=hello')
cat(paste("Encoded URL:", encoded_url, "\n"))

The following is a R code example to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:

library(httr)

api_key <- 'YOUR_API_KEY'
url <- 'https://example.com/?test=hello'
proxy_url <- 'https://proxy.scrapeops.io/v1/'

query_string <- paste0('api_key=', api_key, '&url=', URLencode(url), '&render_js=true')

response <- GET(
  url = paste0(proxy_url, '?', query_string),
  timeout(120)
)

if (status_code(response) == 200) {
  content <- content(response, as = "text")
  cat('Body:', content, '\n')
} else {
  cat('Error:', status_code(response), '\n')
}