Have you ever wondered what happens behind the scenes when making an HTTP
request?
Well, it's something that I've wondered. :D
So I decided to try to dig a little bit into that and see what I would find.
This article of mine is what I learned, and I'm going to do my best to explain how an HTTP request travels around the internet.
First, let's have a look at what makes computers able to interact with each other. To communicate through the internet efficiently, they use protocols, a set of rules and standards used to communicate between computers.
These protocols are all within the Internet protocol suite, commonly known as TCP/IP (Transmission Control Protocol/Internet Protocol) model.
This model splits into four different layers. They are the application layer, transport layer, internet layer and link layer.
Here are the layers with some example protocols:
| Layer | Protocol |
|---|---|
| Application | HTTP(S) |
| Transport | TCP |
| Internet (or Network) | IP |
| Link (or Network Access/Interface) | IEEE 802 (WiFi) |
When a message travels from one computer to another, it goes through all
the layers in both computers.
Starting at the application layer from the sender computer (referred to as
a client), through the transport, internet and link layers and then in
reverse order in the receiving computer (referred to as a server).
Let's start with a high level overview of what happens and then go into more details.
The application (Browser, Postman, etc.) queries a Domain Name System (DNS) server to get an IP address for the domain name ("www.example.com").
A reliable connection is made with receiving computer with a TCP handshake.
If using HTTPS, a TLS handshake is performed to make sure the connection is secure.
The data is split into segments, and information like the senders and receivers IP addresses added. Turned into packets with the size and sequence order (to help the receiver reassembles the segments) attached.
The packets travel through the internet using different paths.
When the receiver gets the packets, it error checks them, make sure it has all the packets and reassembles the segments before sending them to the application, which performs the request.
The receiver sends a response back, and the connection closes.
Now let's see what happens at each layer when we make an HTTP request.
Let's say we have clicked on an item on a website, which will be at "www.example.com/items/2". The first thing that the application (Chrome, Firefox, Postman) we're using needs to do is to find which computer on the internet has this information, meaning finding its IP address.
To do this, it needs to get the domain name, "www.example.com", and translate it to an IP address. It does this with a DNS (Domain Name System) lookup.
The Domain Name System (DNS) is like a phonebook for the internet. Since IP addresses are hard to remember for humans, we use domain names like example.com.
But computers need IP addresses to connect with each other. So the DNS helps the applications convert domain names into IP addresses.
For example, one of the IP addresses for facebook.com is 157.240.21.35. So if you type that IP address in your browser, it will direct you to facebook.com.
So the application will sends a request to the DNS server and ask for the IP address for the specified domain name.
Your internet service provider will provide the IP address of the DNS server to query. Usually, the server is geographically close to you, so that the query will take the shortest amount of time possible.
For more information on DNS, have a look at this article.
Now once we have the IP address, the application will add a port number to it. This port number tells the server (receiving computer) which protocol we want to communicate through.
For example, for HTTP, it will be port 80, and for HTTPS, it will be port 443. Another common one is port 25, for the Simple Mail Transfer Protocol (SMTP) or email.
UDP Protocol
When making a DNS lookup, it's usually through the User Datagram Protocol (UDP) instead of Transmission Control Protocol (TCP), which most requests go through.
TCP is a more reliable connection than UDP; we'll see why that is below and has error checking and recovery. But UDP has none of that, and because of that, it's faster than TCP.
Also, the DNS query is usually sent over an unencrypted connection (UDP), even if you visit a site over HTTPS, which is done to save time.
However, the DNS query can be done over an encrypted connection using the DoH (DNS over HTTPS) protocol.Learn more about UDP here
Now that the application layer has the message we want to send, the server IP address and the port number, it sends that information to the transport layer.
The transport layer divides the data it gets up into segments.
It also adds a TCP header to each segment with the client and server
ports, segment ordering information and a data field known as a checksum
to verify the data has transferred without error.
Before the transport layer can send the segments off, it needs to make sure it has a connection with the server. It does so with a TCP handshake.
A TCP handshake is a process that makes sure the connection between the client and server is reliable. It's also known as a three-way handshake, mainly because it involves three steps.
To read more about the TCP handshake, see this article.
Once the connection is stable, the client can start sending the actual message.
TLS (SSL) Handshake
If the request uses HTTPS protocol, port 443, it also needs to perform a TLS (SSL) handshake to verify that the connection is secure.
It's a bit more complicated than the TCP handshake, and if you want to learn more about that, I recommend this article.
The next action of the transport layer is to transmit the segments to the internet layer.
The internet layer adds an IP header to the segments and formats them into
a packet.
The IP header includes the client and server IP addresses and the packets'
length and sequence order.
Then the packets are handed over to the last layer, which is the link
layer.
The last layer, the link layer, turns the packets into a frame, which means attaching the third header and a footer to "frame" the packet. The header includes a field called cyclical redundancy check (CRC) and is used for error checking.
It then defines how the data should be sent physically through the network.
And voila, the data is on its way.
Now, the internet is a bunch of computers and networks of computers connected together.
An example of a computer network is your home network. Your laptop, phone, printer, and Alexa are connected to your router. Your ISP then makes sure your router is linked to other routers that are connected to other routers and so on.
Eventually, all computers on the internet are connected through multiple routers.
So the client's data travelling to the server can take many paths. And the packets will most likely take different paths from each other because it's faster that way, instead of going the same route and possibly causing a "traffic jam".
But that also means that the packets can arrive at the server at different
times and in a separate order, then sent out.
The server can put it correctly together once it has all the packets
because it includes the ordering information in the headers attached.
When the data arrives at the server, it will travel through the layers in reverse order.
Here's what will happen:
Once it has performed the request, it will send a response back. The connection between the computers will then close when response data has transferred back to the client.
Hopefully, this helped you understand a little bit better how an HTTP request travels the internet.
I had great fun digging around and learning about it.
Thank you for reading.