Python HTTPX - How to Send POST Requests

In this guide for The Python Web Scraping Playbook, we will look at how to make POST requests with the Python HTTPX library.

In this guide we will walk you through the most common ways of sending POST requests with Python HTTPX library:

Let's begin...

POST JSON Data Using Python HTTPX

A common scenario for using POST requests is to send JSON data to an API endpoint, etc. Python HTTPX makes it straightforward to accomplish this task.

We simply just need to add the data to the request using the json parameter of the POST request:

import httpx

url = 'https://httpbin.org/post'
data = {'key': 'value'}

# Send POST request with JSON data using the json parameter
response = httpx.post(url, json=data)

# Print the response
print(response.json())

The Python HTTPX library will automatically encode the data as JSON and set the Content-Type header to application/json.

This approach can be simpler and more concise than manually encoding the data and setting the headers. Additionally, it may offer some performance benefits, as the HTTPX library can use a more efficient encoding method for JSON data.

POST Form Data Using Python HTTPX

Another common use case for using POST requests is to send form data to an endpoint.

We simply just need to add the data to the request using the data parameter of the POST request:

import httpx

url = 'https://httpbin.org/post'
data = {'key': 'value'}

# Send POST request with FORM data using the data parameter
response = httpx.post(url, data=data)

# Print the response
print(response.text)

The HTTPX library will automatically encode the data as JSON and set the Content-Type header to application/x-www-form-urlencoded so you don't have to set any headers.

Configuring Data Types

As we've seen above when you use the data or json parameter to send data with the POST request is defaults the Content-Type header to either application/json or application/x-www-form-urlencoded.

However, if you would like to override this or send data with another Content-Type then you can do so by just adding the Content-Type header to the POST request.

In the following example, we will send JSON data using the data parameter instead of the json parameter as we did previously.

import httpx
import json

url = 'https://httpbin.org/post'
data = {'key': 'value'}

# Convert data to JSON format
json_data = json.dumps(data)

# Set the Content-Type header to application/json
headers = {'Content-Type': 'application/json'}

# Send POST request with JSON data using httpx
response = httpx.post(url, data=json_data, headers=headers)

# Print the response
print(response.json())

Using POST Requests With Sessions

You can also use Python HTTPX's Session functionality to send POST requests. Using sessions can be useful if you need to send multiple requests to the same server, as sessions can keep track of cookies and other stateful information across requests.

Here's an example of how to make a POST request with JSON data using a session in Python HTTPX:

import httpx

url = 'https://httpbin.org/post'
data = {'key': 'value'}

# Create an httpx session
session = httpx.Client()

# Set the Content-Type header to application/json for all requests in the session
session.headers.update({'Content-Type': 'application/json'})

# Send a POST request with JSON data using the session object
response = session.post(url, json=data)

# Print the response
print(response.json())

In this example, we create a Session object using httpx.Client(). We then update the headers attribute of the session object to set the Content-Type header to application/json. This will ensure that all requests sent through the session will have this header.

Finally, we use the session.post() method to send a POST request with JSON data, passing the URL and the data dictionary as arguments. The session will automatically encode the data as JSON and set the Content-Type header to application/json.

The session.post() method returns a Response object, just like the regular httpx.post() method.

More Web Scraping Tutorials

So that's how you can send POST requests using Python HTTPX.

If you would like to learn more about Web Scraping, then be sure to check out The Web Scraping Playbook.

Or check out one of our more in-depth guides:

• How to Scrape The Web Without Getting Blocked Guide
• The State of Web Scraping 2020
• The Ethics of Web Scraping