Explore Python Libraries: HTTPX

The HTTPX library provides important features available in the Requests library as well as support for both HTTP/1.1 and HTTP/2. You can have a look at HTTPX's website to know what features HTTPX has in its pocket. In this guide, you will explore the following HTTPX topics with practical examples:

All of the code in this guide assumes that the HTTPX library has been installed and successfully imported in your current working environment as import httpx.

Suppose you want to fetch the stock stats of a company from the Yahoo! Finance website. The associated URL will be https://finance.yahoo.com/quote/ABXY/key-statistics?p=ABXY where "ABXY" is the company's registered abbreviation. Notice that the URL consists of a key-value pair where the key is "p" and the value is "ABXY". So if you want to fetch stock stats for another company, all you need to do is replace "ABXY" with your required company's registered abbreviation.

To learn this concept practically, you can fetch the stock stats of PS (NASDAQ: Pluralsight). Start by forming a dictionary as shown:

And finally, use httpx.get method and provide the complete URL including the eq variable:

You can view the resulted URL using url attribute as shown:

If you want to let the client and server pass additional data with an HTTP request, you can set custom HTTP headers using HTTPX library. For instance, consider YouTube whose headers can be extracted as shown:

The result of the above code is the following YouTube headers:

Notice, the default value of the content-encoding header is gzip. What if you want to go with any other alternative like br?

To update the value of a header, store the new value in a dictionary and pass the dictionary to the headers argument of the get method as shown:

This updated header is shown below:

You can use the HTTPX library to fetch text, image, and JSON data. In this section, you will learn to fetch text and image data both small and large in size. For larger data you can perform streaming, which will be explained further in this section.

Suppose you want to fetch data in the HTML format from a webpage that consists of not more than 5-10 lines of text as shown in the image:

Source: Dua, D. and Graff, C. (2019). UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science.

To do so, pass the URL inside the httpx.get method and later use the text attribute as shown:

Output:

To fetch large text data, you can perform streaming, i.e. fetch a limited amount of data at a time until you reach the end of the page. To understand this topic, suppose you want to extract all the text data from the Wikipedia page about Ghostbusters. To do so, you could fetch data line by line inside a for loop that is nested inside the httpx.stream method as shown:

In the above code, the print(line) statement is commented. You can uncomment it and observe the output as it streams. The last line print(count) describes the total number of lines (1166) fetched during the stream.

The process of fetching an image data is similar to that of a text data. However, the only difference is how you read it once you get it in your Python environment. Consider that you need to fetch a penguin image from the Pixabay website. To do so, pass the image URL inside the httpx.get method and later use PIL and io methods to read and display the data as shown:

Output image

Source: Pixabay

You can use the httpx.stream method to fetch large image data. The only difference from that of streaming text data is that this time the for loop iterates over iter_bytes as shown:

HTTPX can be used to post data inside a form available on a webpage. In this section, you will learn how to upload text and file data to a webpage.

Suppose you want to write "Hello, World!" on a webpage using HTTPX. The input area would look like the one as shown here in the green color:

To do so, use the post method and provide the information inside the data argument as shown:

The above code results in an HTML output. Since the output is too big, given below is the properly formatted webpage with the "Hello, World!" string in the desired box:

There are many websites where you have to upload a file on their server. Here's one instance as highlighted in the green box:

To upload a complete file rather than just writing a string, you can use the files argument of the post method. Assume you have a HTML file upload.html stored on your drive. You can upload it on this webpage using the given code:

You can verify that the data has been successfully uploaded by checking the console result.

Have you ever noticed that when you type "https://msn.com" in a browser, it may not necessarily retain its URL as-it-is? There are good chances that you may be redirected based on the geographical region from where you are trying to access the website. For instance, run the following code and check your output:

For me it is https://www.msn.com/en-in/, but for you, the last two characters may differ. But, the point is the original URL has been changed and you are redirected to a new one. However, the good news is that you can control whether redirection should take place or not. If you don't need a redirection from your original URL then use allow_redirects=False inside the get method as shown:

Also, you can control how long should it take for a website to load before the program throws a timeout error. You can specify custom timeout value by using the timeout parameter as shown:

In the above code, if the URL is not reached before one second, the program will result in a ConnectError.

You have now learned the basics of HTTPX library by exploring its get, post, and stream methods.