Skip to main content

Golang Code Examples

The following are code examples on how to integrate the ScrapeOps Proxy Aggregator with your Golang based web scrapers.

Authorisation - API Key​

To use the ScrapeOps proxy, you first need an API key which you can get by signing up for a free account here.

Your API key must be included with every request using the api_key query parameter otherwise the API will return a 403 Forbidden Access status code.

Basic Request​

The following is some example Golang code to send a URL to the ScrapeOps Proxy endpoint https://proxy.scrapeops.io/v1/:


package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

func main() {
	apiKey := "YOUR_API_KEY"
	targetURL := "https://httpbin.org/ip"
	proxyURL := "https://proxy.scrapeops.io/v1/"

	queryParams := url.Values{}
	queryParams.Add("api_key", apiKey)
	queryParams.Add("url", targetURL)

	client := &http.Client{
		Timeout: time.Duration(120 * time.Second),
	}

	req, err := http.NewRequest("GET", proxyURL+"?"+queryParams.Encode(), nil)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	response, err := client.Do(req)
	if err != nil {
		fmt.Println("Error making request:", err)
		return
	}
	defer response.Body.Close()

	if response.StatusCode == http.StatusOK {
		bodyBytes, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Println("Error reading response body:", err)
			return
		}
		fmt.Println("Body:", string(bodyBytes))
	} else {
		fmt.Println("Error:", response.StatusCode)
	}
}

This Go code uses the net/http and net/url packages to send an HTTP GET request with the specified query string and timeout value. The query string is created using url.Values, and the Encode() method is used to URL-encode the query string parameters.

From here, ScrapeOps will take care of the proxy selection and rotation for you so you just need to send us the URL you want to scrape.

Response Format​

After receiving a response from one of our proxy providers the ScrapeOps Proxy API Aggregator will then respond with the raw HTML content of the target URL along with a response code:


<html>
  <head>
     ...
  </head>
  <body>
     ...
  </body>
</html>

The ScrapeOps Proxy API Aggregator will return a 200 status code when it successfully got a response from the website that also passed response validation, or a 404 status code if the website responds with a 404 status code. Both of these status codes are considered successful requests.

Here is the full list of status codes the Proxy API returns.

Advanced Functionality​

To enable other API functionality when using the Proxy API endpoint you need to add the appropriate query parameters to the ScrapeOps Proxy URL.

For example, if you want to enable Javascript rendering with a request, then add render_js=true to the request:


package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"time"
)

func main() {
	apiKey := "YOUR_API_KEY"
	targetURL := "https://httpbin.org/ip"
	proxyURL := "https://proxy.scrapeops.io/v1/"

	queryParams := url.Values{}
	queryParams.Add("api_key", apiKey)
	queryParams.Add("url", targetURL)
	queryParams.Add("render_js", "true")

	client := &http.Client{
		Timeout: time.Duration(120 * time.Second),
	}

	req, err := http.NewRequest("GET", proxyURL+"?"+queryParams.Encode(), nil)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	response, err := client.Do(req)
	if err != nil {
		fmt.Println("Error making request:", err)
		return
	}
	defer response.Body.Close()

	if response.StatusCode == http.StatusOK {
		bodyBytes, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Println("Error reading response body:", err)
			return
		}
		fmt.Println("Body:", string(bodyBytes))
	} else {
		fmt.Println("Error:", response.StatusCode)
	}
}

Check out this guide to see the full list of advanced functionality available.

Timeout​

The ScrapeOps proxy keeps retrying a request for up to 2 minutes before returning a failed response to you.

To use the Proxy correctly, you should set the timeout on your request to a least 2 minutes to avoid you getting charged for any successful request that you timed out on your end before the Proxy API responded.