Tour of a HTTP request in Rust

TL;DR

When we talk about a web service, we, more often than not, mean deployed code which listens on a certain IP address and port and responds to HTTP messages. There are many steps involved for two parties to be able to communicate with each other. Application developers are mainly confronted with two pieces of this process: TCP and HTTP.

TCP is a protocol which two parties use to establish a connection. They follow a certain pattern (three-way-handshake) where they send and receive short messages to negotiate the connection details. After establishing this connection, you can receive and send HTTP messages to the other party. HTTP is a stateless protocol which demands a response for every request, and defines, besides other options, the size and format of the data it is sending.

There is a great advantage in knowing how exactly the communication with TCP works and how and where HTTP is added. Having this information can help you improve the performance of your server application and make it more secure. If, for some reason, you want to choose any other protocol than TCP or HTTP, you know where and how to replace it.

Rust and the OSI model

Web services are deployed on computers connected to the internet. These computers have IP addresses and open ports they listen to for new messages. Applications running on these machines are signaling interest on certain messages so they can process and answer these.

A message sent to a server has to go through multiple layers and geographically different locations for it to arrive. To get a better understanding of these different layers, the OSI model was created. It is a conceptual framework to standardize the communications.

As we can see in the following figure, a client can’t just send over data to a server. It needs to go through many different routers to find the right server. For it not to get lost, a user application and the kernel are adding several headers to it, so each layer of the communication process knows where to route it to.

A packet goes through these different layers, and almost all of them add an extra header on top so the next layer knows how to deal with the information. Your application adds a HTTP header on top of the data it wants to send, before the kernel adds the TCP, IP and Ethernet header.

The receiving server goes through the same process but in reverse. It has to dismantle each header until it can read the data inside of it.

The added header sizes are standardized, so the operating system and the kernel know how many bytes they have to strip out until they can read the data. We can use this information now to get a better grasp on our data we receive.

In the OSI overview figure earlier, we see that the ethernet and IP header are pretty non-negotiable. But everything else is more in our control. We can choose UDP for example instead of TCP and can use our own protocol instead of using HTTP. We can, for example, have security reasons to implement our own protocol (with an own header size), so intruders who are reading our messages can’t make sense of them.

Where does Rust come into play? A web service has to support the following mechanism for it to be able to create connection, receiving messages and sending responses:

• Opening a connection to another client
• Support the different layers (TCP and HTTP)
• Hold connections
• Parse receiving messages
• Send proper HTTP messages back

Many other programming languages include a rich standard library to create these HTTP servers. Rust is however a little bit different. Being a Systems Programming Language, Rust wants to be as small as possible and also functioning well on micro controllers for example who don’t always want to communicate via HTTP with their peers.

Therefore, Rust decided just to include a basic understanding of TCP in the standard library, and no build-in support for HTTP. The blue parts (TCP/IP) are included in the Rust standard library. If you want to create web server which supports HTTP, you have to create your own. Luckily, this is a common scenario, so the community already built some battle-tested web server implementations in the past. The crate hyper for example is widely used as a http server.

There are also crates for web frameworks, which include all the layers beneath them (HTTP, TCP etc.) and offer all the modern ergonomics like parsing URL query parameter, reading and returning JSON and so on.

This also gives you a greater choice: If you just want a minimal functioning application server without much bloat doing one thing, you can create the few functions you need by hand and have a lightweight solution afterwards.

If you are coming from Go, NodeJS or Java, this means a shift in perspective. You probably have to look for a library which supports your needs from the start instead of going a few more miles without thinking about help from the community.

In addition to HTTP, you also need to make sure the connection between client and server is secure. This is handled via TLS (Transport Layer Security), a successor of SSL. Rust also hasn’t built-in TLS support, but there exist a few packages which support you in enabling TLS in your application.

Opening a connection

We look at an example where a browser application is sending a HTTP request to our web service which is written in Rust. We will dive shortly into how exactly the bytes arrive at the kernel, and how our Rust application is getting the bytes delivered into the running application. Note that this is all abstracted away through libraries, but you can later on choose not to use such library and implement something via the Rust core library itself.

In addition, it is helpful to know or at least heart about it once how exactly the flow of bytes in a web service works, so you can spot bugs, bottlenecks and other misconfigurations later on in your running application.

When the client sends a HTTP request, the kernel is wrapping the data in a package with a HTTP and TCP header attached to it. It arrives on our server at the so called NIC (network interface card). The client first has to establish a TCP connection to our server. Once done, our kernel opened a socket to which is listening to this address for incoming messages.

If you want to dig deeper into the kernel side of networking, I highly recommend Beej's Guide to Network Programming.

When we run a web server in Rust, we also have a socket to the operating system side where we can listen to incoming messages. The kernel’s job is to copy the data from the incoming TCP message onto our internal socket and notifies us when new data arrived.

In detail, the kernel reads the incoming messages and figures out which TCP connection (IP address and port) it is associated with, looks up the corresponding socket and copies the data to a receive buffer.

It notifies the process which is listening to new data to this socket and copies the data to a new buffer once the process is signaling interest. It copies the data from the receive buffer into the read buffer so that your server application can get the bytes out of the kernel into your program.

We learned earlier that Rust supports TCP right out of the box. Therefore, we can create, open and listen to a TCP socket within Rust. Once we receive a message, we can also answer back on the same socket. We can basically send any text back to the socket, we just have to be aware that the other side can interpret what we are sending.

Let’s open a socket, so the kernel knows where to forward incoming requests to. Each socket has to know the protocol being used (TCP in our case), the IP address and the port. In Rust, the TcpListener is handling the job for us, and we can use bind to tell the kernel the address and port we are listening to.

use std::net::TcpListener;

fn main() {
	let listener = TcpListener::bind("127.0.0.1:8080").unwrap(); 
	for stream in listener.incoming() {
            let stream = stream.unwrap(); 
            println!("stream accepted {:?}", stream);
}

If you use cargo run to start the server, open a browser and navigate to localhost:8080, you see that we print something like this:

stream accepted TcpStream { addr: 127.0.0.1:8080, peer: 127.0.0.1:56931, fd: 4 }

This is a step in the right direction. But why don’t we see any data or HTTP headers? It’s because we receive a stream and print it on the console. We actually have to read the content from the stream.

When reading from the stream like that, the baseline we expect is UTF8 encoded text. At this point, the kernel already stripped away the TCP header and all we have left is the data encapsulated in it. This can either be HTTP headers + data or some other headers attached to the data.

We marked the Session layer with an asterisk (*), because technically, the Linux Kernel doesn’t concern itself with the OSI model and implements its own mechanics. However, the idea of keeping a connection is handled in the kernel, and when copying the data over to the socket for the Rust application, the TCP header will be removed.

Parsing our stream content, we should see the headers and also the data in plain text, and it is on us to strip away the headers to get to the real data of the message. Headers however play an important role: They help us interpret the data we receive.

There are plenty of different HTTP header, and the server is in charge to interpret them in the right way and work with them.

When using a web framework later on, all the details are abstracted away. However it is vital to understand the flow how information arrives at your application so later on, you can choose asynchronous strategies, your own protocol and where to look for optimizations.

Adding HTTP

The TcpListener gave us a stream, which we need to read and interpret. We have to somehow take this stream and read what’s in it. For this, we need a few components. First, we need to create a new function which takes an incoming stream and writes the bytes back to a local buffer. From there we can parse the data accordingly and send back an answer.

We add a helper function which does exactly that for us.

…

fn handle_stream(mut stream: TcpStream) {
    let mut buffer = [0; 1024];
    stream.read(&mut buffer).unwrap();
    println!("Request: {}", String::from_utf8_lossy(&buffer[..]));
}
…

All we have to do is to call this function in our iterator.

use std::net::{TcpListener, TcpStream}; 
use std::io::prelude::*; 

…

fn main() {
	let listener = TcpListener::bind("127.0.0.1:8080").unwrap(); 
	for stream in listener.incoming() {
	    let stream = stream.unwrap();
	    handle_stream(stream); 
	}
}

After starting the application again with cargo run, you can open a new browser window and navigate to the website localhost:8080 and see what your application is printing onto the console.

It will vary with your browser of choice, but the current version of Safari will send multiple requests which look like the following:

GET / HTTP/1.1
Host: localhost:8080
Upgrade-Insecure-Requests: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
Accept-Language: en-us

It includes:

• GET: The HTTP method
• /: The Path on the server
• HTTP/1.1: The version of the HTTP protocol
• HOST: The host/domain of the server we want to request data from
• Accept-Language: Which human language we prefer and understand

Instead of just printing out the stream, we can start to look at the HTTP specification, store the content in an array and iterate over it line by line, and create a HTTP struct out of it. This work is not trivial since we need to check the length of the message from the HTTP header and build the full message ourselves.

Thankfully there are already crates published in the Rust ecosystem which help you with this task. So, deploying a http server in production is much less work than we do here by hand.

After you opened your browser and navigated to localhost:8080, you saw an error page. That’s because we don’t return an answer yet, which the HTTP protocol requests we do. To solve this problem, we can write onto the stream and send bytes back to the client.

...

fn handle_stream(mut stream: TcpStream) {
    let mut buffer = [0; 512];
    stream.read(&mut buffer).unwrap();
    println!("{}", String::from_utf8_lossy(&buffer[..]));

    let response = "HTTP/1.1 200 OK\r\n\r\n";
    stream.write(response.as_bytes()).unwrap();
    stream.flush().unwrap();
}

...

When you open your browser now and navigate to localhost:8080, you will get a blank page instead of an error. We successfully communicated via HTTP to another application in just a few lines of code.