Skip to main content

Fix SSL Errors in Python Requests

The Python Requests library is the most popular Python package for making HTTP requests. By default, the Requests library acts like a web browser, at the moment of making an HTTP request it verifies the web server’s Secure Sockets Layer (SSL) certificate. If the library is unable to verify the certificate, it will throw an SSLError.

In this article, we’ll cover the topics that will help you to handle effectively and securely SSL errors. Find them outlined below:

Before getting to the actual Python code, it is important to understand what happens behind the hood when making an HTTP request, and what kind of validation steps exist to validate the SSL certificate. Hence we’ll begin by analysing the Secure Sockets Layer communications.

TLDR: How to Fix SSL Errors in Python​

The shortest, easiest, and most effective solution to fix SSL errors in Python requests is to disable SSL verification. However, this approach should be used cautiously as it may expose your application to security risks.

Here's how to do it:

import requests

response = requests.get("https://www.amazon.com/", verify=False)

By setting the verify parameter to False, you instruct Python requests to skip SSL certificate verification.

While this can quickly resolve SSL errors, it's important to note that it compromises the security of your application by allowing potential man-in-the-middle attacks.

Therefore, use this solution only if you fully understand the risks and if verifying the SSL certificate is not critical for your use case.

An Introduction to SSL Connection​

Websites that prioritize secure connections use a technology called SSL (Secure Sockets Layer) or its successor, TLS (Transport Layer Security). These secure connections are identified by "HTTPS" at the beginning of the URL. This is why SSL errors, encountered when using Python's Requests library, only occur with HTTPS websites.

An SSL connection uses encryption to secure the data that travels between the user and the web server, ensuring that no one has access to sensitive information being transmitted between the two parties.

For an SSL connection to occur, a client (web browser) needs to make a HTTP request to a server having an SSL certificate. For instance, with Python’s request library, we can make a GET request to a website and initiate an SSL connection through a process called SSL/TLS handshake, which involves the following steps:

  1. Client Hello: The client initiates the SSL handshake by sending a message to the server (website). This message includes the supported versions of the security protocol, a list of preferred encryption algorithms (cipher suites), and a random string of data (client random).

  2. Server Hello: The server picks the cipher suite that both will use to communicate. It then sends it to the client, along with the supported version of the security protocol, a random message (server random) and its SSL certificate.

  3. Authentication: The client verifies the server's SSL certificate. This certificate contains information about the server, like its domain name, and is issued by a trusted certificate authority (CA). By checking the certificate and CA signature, the client confirms the server's identity.

  4. Key Exchange: Both sides establish a secret key for encryption. This key exchange typically involves the client sending another random string encrypted with the server's public key obtained from the certificate.

  5. Finished: Finally, both sides send a "finished" message containing a hash of all previous messages exchanged during the handshake, encrypted with the established secret key.

Once the handshake is complete, a secure encrypted connection is established, and the data exchanged between the client and the server is protected.

SSL Certificates​

The SSL certificates prove the identity of the web server, containing the website’s public key, which is used to make encryption and authentication possible. These certificates are issued by trusted third-party organizations known as Certificate Authorities (CAs).

In a nutshell, SSL certificates are responsible for:

  • Encrypt communication: They safeguard sensitive information like credit card details and personal information between the website and the user.

  • Credibility: Issuing a certificate ensures the user they are interacting with a legitimate website and not a fraudulent one.

  • Verification: The certificate contains the ID of the website and its unique public key, which proves its uniqueness on the internet.

  • Prevent man-in-the-middle attacks: By encrypting communication, certificates ensure hackers cannot intercept the information sent and received.

There are various types of SSL certificates, such as the following:

  • Single-domain: As the name says, these types of certificates are only available for one single website.

  • Wildcard: Very similar to single-domain certificates, the only difference is that it also applies to subdomains.

  • Multi-domain: Unlike the other two domains, this type of certificate applies to multiple domains.

SSL/TLS certificates play a crucial role in securing connections, and several Certificate Authorities offer them while giving further explanation about each type. However, our focus now shifts to the Python requests library and how it interacts with these encrypted connections.

What Are SSL Errors​

"SSL errors" refer to issues encountered during the Secure Sockets Layer (SSL) or Transport Layer Security (TLS) handshake process, which is crucial for establishing secure communication over the internet.

When working with Python requests or any HTTP client library, SSL errors may occur due to various reasons.

These errors can disrupt the normal flow of communication between the client and the server, potentially leading to application malfunctions or security vulnerabilities.

  • Functionality: SSL errors may prevent the application from establishing a secure connection with the server, leading to failed requests, timeouts, or unexpected behavior.
  • Security: Ignoring or mishandling SSL errors can expose the application to various security risks, such as man-in-the-middle attacks, data interception, or server impersonation.

Common SSL errors include:

  • Certificate Validation Errors:
    • These occur when the server's SSL certificate is invalid, expired, or not trusted by the client. It indicates a potential security risk as the identity of the server cannot be verified.
    • Error Message: "certificate verify failed"
    • Error Code: SSL_ERROR_CERTIFICATE_VERIFY_FAILED
  • Hostname Verification Failures:
    • SSL/TLS certificates are typically issued for specific domain names (hostnames). If the hostname in the certificate does not match the hostname of the server the client is trying to connect to, a hostname verification failure occurs.
    • Error Message: "hostname 'example.com' doesn't match 'www.example.com'"
    • Error Code: SSL_ERROR_BAD_CERT_DOMAIN
  • Handshake Failures:
    • SSL/TLS handshake is the initial process where the client and server establish a secure connection. Handshake failures can happen due to incompatible cipher suites, protocol versions, or other configuration issues.
    • Error Message: "handshake failure"
    • Error Code: SSL_ERROR_HANDSHAKE_FAILURE

Identifying SSL Errors​

The requests library verifies the SSL certificates during the HTTPS requests, just like a web browser does. It ensures the website’s authenticity by looking at its built-in list of trusted Certificate Authorities (CAs), if the certificate was issued by one of those in the list, the data transmission is encrypted and protected, otherwise, it will prompt an error.

Identifying SSL errors in Python Requests involves implementing methods to detect and handle issues related to SSL/TLS certificate validation.

Here are some common methods for identifying SSL errors:

Certificate Validation​

One of the most common methods for identifying SSL errors is by enabling certificate validation in Requests. The verify parameter in Requests determines whether SSL certificate validation is performed.

By default, verify is set to True, meaning Requests will validate SSL certificates against the system's CA bundle. If certificate validation fails, Requests raises an "SSLError".

import requests

try:
    response = requests.get('https://example.com', verify=True)
except requests.exceptions.SSLError as e:
    print(f"SSL Error: {e}")

Exception Handling​

Catching the requests.exceptions.SSLError exception allows you to handle SSL errors gracefully in your Python code. You can log the error message, retry the connection, or prompt the user for further action.

import requests
from requests.exceptions import SSLError

try:
    response = requests.get('https://example.com')
except SSLError as e:
    print(f"SSL Error: {e}")

Inspecting Response Attributes​

After making a request, you can inspect the response object to check for SSL-related issues. The response.ok attribute indicates whether the request was successful (status code 200-299).

If SSL validation fails, the response object may contain additional information about the error in the response.reason attribute.

import requests

response = requests.get('https://example.com')
if not response.ok:
    print(f"Request failed: {response.reason}")

How to Fix SSL Errors​

As mentioned before, when the Python requests library cannot validate the website’s certificate, an SSLerror is triggered. This happens because the company that issued the certificate (the Certificate Authority or CA) isn't on the built-in list of trusted CAs that the library uses.

Let’s take a look at the different methods to fix the problem:

Option 1: Update SSL/TLS Libraries and Dependencies​

The first approach to this issue is to make sure you have the latest version of the Python requests library. You can use to following command to update it:

pip install requests --upgrade

Also, make sure you have all the dependencies you need to operate the requests library:

pip install PyOpenSSL cryptography ndg-httpsclient --upgrade

Option 2: Installing CA Certificates​

If the first solution doesn't sove the error, and disabling verification isn't an option because you need to keep the communication secured, then you need to obtain the website's certificate, add it to your repository, and update your code like this:

import requests
 
response = requests.get('https://www.amazon.com', verify ='<path_to_certificate>')

You can get certificates from any legitimate Certificate Authority (CA).

There are free options like Let’s Encrypt, a popular choice for many websites, which is great for personal projects or small businesses.

Paid CAs often offer additional features like higher warranty amounts or specialized certificates for specific needs.

The certificates themselves typically come in two main formats:

  1. DER (Distinguished Encoding Rules) and
  2. PEM (Privacy-Enhanced Mail).

They often use extensions like .der, .cer, .crt for DER, and .pem for PEM.

While both formats contain the same certificate information, PEM is the more common format used with the requests library. If you encounter a DER certificate, you might need to convert it to PEM before using it.

Option 3: Disable SSL Verification​

One easy way to bypass the error is by setting the SSL verification to false. However, note that this option should be used with caution as it may expose the application to security risks.

import requests

response = requests.get("https://www.amazon.com/", verify=False)

Before applying this method, make sure you’re not sending/retrieving any sensitive information.

Option 4: Handling Self-Signed Certificates​

When using self-signed SSL certificates, which are not signed by a trusted Certificate Authority (CA), you need to explicitly configure your Python application to trust these certificates. Otherwise, your application may raise SSL verification errors when trying to establish a connection with a server using a self-signed certificate.

Suppose you have obtained the self-signed certificate self_signed.crt and want to configure your Python application to trust it. You can use the verify parameter in Python Requests to specify the path to the self-signed certificate:

import requests

# Path to the self-signed certificate
certificate_path = '/path/to/self_signed.crt'

# Make a request with SSL verification using the self-signed certificate
response = requests.get('https://example.com', verify=certificate_path)

Option 5: Dealing With Hostname Verification Failures​

Hostname verification failures occur when the hostname specified in the URL does not match the Common Name (CN) or Subject Alternative Name (SAN) fields in the server's SSL certificate. This can happen due to misconfiguration or when the server's certificate is not issued for the hostname being accessed.

When making HTTPS requests in your Python application, ensure that you're using the correct hostname in the URL. Mismatched hostnames can lead to hostname verification failures, so it's essential to use the hostname exactly as it appears in the SSL certificate.

Configure Python Requests to perform hostname verification during SSL/TLS handshake. By default, Requests performs hostname verification, but you can explicitly enable it to ensure that the hostname matches the one in the SSL certificate.

import requests

# Make a request with hostname verification enabled (default behavior)
response = requests.get('https://example.com')

# Make a request with explicit hostname verification enabled
response = requests.get('https://example.com', verify=True)

Best Practices for SSL Handling​

Implementing secure SSL/TLS handling is crucial for ensuring the confidentiality, integrity, and authenticity of data transmitted over the internet.

Here are some best practices to follow:

Implement Secure Configurations:

  • Use the latest versions of SSL/TLS protocols (TLS 1.2 or higher) to benefit from the latest security features and mitigate known vulnerabilities.
  • Disable deprecated and insecure cipher suites, such as SSLv3 and weak encryption algorithms (e.g., RC4, SHA-1).
  • Configure SSL/TLS settings to enforce strong encryption, perfect forward secrecy (PFS), and secure renegotiation.

Regularly Update Libraries and Certificates:

  • Keep SSL/TLS libraries and dependencies (e.g., OpenSSL, pyOpenSSL) up-to-date to patch security vulnerabilities and ensure compatibility with the latest standards.
  • Regularly update CA certificates to maintain trust with Certificate Authorities and prevent SSL verification errors.
  • Monitor security advisories and announcements for any vulnerabilities or updates related to SSL/TLS libraries and certificates.

Validate Server Identities:

  • Always validate the identity of the server by checking its SSL certificate during the SSL/TLS handshake process.
  • Verify that the certificate is issued by a trusted Certificate Authority (CA) and has not expired or been revoked.
  • Perform hostname verification to ensure that the hostname in the URL matches the Common Name (CN) or Subject Alternative Name (SAN) in the server's certificate.

Utilize Trusted Certificate Authorities (CAs):

  • Obtain SSL certificates from reputable and trusted CAs to establish trust with clients and ensure the authenticity of your server's identity.
  • Avoid self-signed certificates for production use unless necessary, as they may not be trusted by clients by default and can lead to SSL verification errors.

Monitor SSL/TLS Connections:

  • Implement logging and monitoring mechanisms to track SSL/TLS connections and detect any suspicious or anomalous behavior.
  • Monitor for SSL/TLS handshake failures, certificate validation errors, and unexpected changes in SSL/TLS configurations.
  • Regularly audit SSL/TLS configurations and certificates to identify potential security risks and compliance issues.

By following these best practices, you can strengthen the security of your SSL/TLS implementations, reduce the risk of vulnerabilities and attacks, and ensure the integrity and confidentiality of data transmitted over secure connections

Conclusion​

In this article, we explored various techniques for handling SSL errors in Python applications, emphasizing the importance of proactive SSL management for ensuring security.

We discussed different approaches to addressing SSL errors, including updating SSL/TLS libraries, installing CA certificates, configuring SSL verification settings, handling self-signed certificates, and dealing with hostname verification failures.

By updating SSL/TLS libraries and dependencies, ensuring the validity of server certificates, and implementing secure configurations, you can mitigate potential SSL vulnerabilities and establish robust and secure communication channels.

For more information on Python requests and SSL/TLS handling, check the official documentation of Python Requests.

More Web Scraping Guides​

If you would like to get more information about web scraping with Python, check out our extensive Python Web Scraping Playbook or other tutorials: