How the MITM Proxy works

Many blogs explain the what is the MITM, how to configure it to the client and server, there is no such blog easily available which explains How MITM works, so I thought of writing a blog.

Before getting into the How MITM works, let’s see what is Proxy server and How it works in short.

What is a Proxy Server?

A proxy server is a software system running on a computer or any other hardware that acts as an intermediary between server and end-users, such as a computer, mobile, and Base server. An Internet server hosting resources like web pages, files, videos.

When the user/client requests the web page or data hosted on that Web/Base server, it sends the request to the base server, Upon receiving the request the web server provides the response with its destination address. So this is the normal procedure of accessing the web pages or response from the webserver to the requesting client.

Simple communication between client and server

So is this safe? not at all because someone in the middle can hack your IP address and they can attack your Client device/PC or terminal, so what’s the solution?

So the solution is the Proxy Server.

When the proxy server comes in between Client and Base server then it hides the IP address of the client, let’s take below example the internet user with IP address 25.25.10.10 when communicates to base server with IP address 22.10.10.1 then in between proxy server works as an intermediary creates own IP address that is 35.45.10.10. When an intermediary proxy receives response data from the Web server then stores that data in its own cache memory and then deliver it to the client if someone tries to hack the connection or client IP address then hacker gets an IP address of the proxy i.e 35.45.10.10 so here proxy saves client user from the hacker. Proxy protects from such attacks by hiding the actual IP of the client. This way we can avoid client IP address to be hacked using a Proxy server.

Communication between client apps and base server using the Proxy server

If another client with a different IP address(25.25.10.11 as shown in image 2) which already belongs to this Proxy server then the Proxy server allows the connection. After setting up the connection Proxy server checks what this client requires, if this client requires the data which already present in the Proxy server cached memory then it returns data without making the connection to the backend server, in this way Proxy server saves the internet bandwidth which is unnecessary.

If some external client tries to connect to the same proxy server with different IP addresses (28.25.10.11 as shown in image 2) which don’t belong to the Proxy server then the Proxy server authenticates the user and restricts the access.

Benefits of Proxy Server?

1. Security — Hides User Identity from Hackers and other agencies.
2. Speed — Increase Speed using cache memory.
3. Save Bandwidth — Using Cache helps to reduce the Internet bandwidth requirement.
4. Authentication — Proxy server enables user authentication.
5. Site Filtering — Using a proxy, particular websites can be allowed or restricted.
6. Surf log — Can keep a record of surfing done by users.

So, that is all about What is Proxy server and How it works. Let’s jump to the main topic MITM proxy’s proxy mechanism in detail and How it works internally.

Let’s see the working of conventional proxy

HTTP call

The simple HTTP call from your computer or Mobile to the MITM proxy and base server is the simplest and most reliable way to intercept the traffic between both client/computer and base server which we will be monitored on the MITM proxy monitor. In this simple call, the client connects directly to the proxy and makes a request that looks like this

GET http://example.com/page.html HTTP/1.1

This is a proxy GET request that includes a schema and host specification, and it includes all the information MITM proxy requires to proceed.

Simple HTTP call between client and server in the presence of a conventional proxy

1. The client connects to the proxy and makes a request.
2. MITM proxy connects to the upstream/base server and simply forwards the request further.

HTTPS call

Unlike HTTP calls the HTTPS call is different. Here the client connects to the proxy and makes a request that looks like this

CONNET example.com:443 HTTP/1.1

A conventional proxy can neither view nor manipulate a TLS(Transport Layer Security) encrypted data stream, so a CONNECT request simply asks the MITM proxy to open a pipe between the client and server. The proxy is just an interface/facilitator between them, it blindly forwards data in both directions without knowing anything about the contents. The negotiation of the TLS connection happens over the pipe, and the subsequent flow of requests and responses are completely opaque to the proxy.

How does the MITM proxy work?

MITM proxy and Charles proxy is an HTTP proxy/HTTP monitor that enables developers to view all of the HTTP and SSL/HTTPS traffic between the client and the server. The MITM in its name stands for Man-In-The-Middle, the basic idea behind it is to pretend to be the server to the client and pretend to be the client to the server, while in the middle we can observe the traffic coming from both sides on MITM proxy monitor.

The tricky part is that the Certificate Authority system is to create secure connections to a server via the Internet, by allowing a trusted-party to cryptographically sign a server’s certificates to verify that they are legit. As we know if the server has a Certificate authority with both public and private keys if the client which has these keys can only connect securely to the Sever. If this signature doesn’t match a secure client will simply drop the connection and refuse to proceed, to overcome from this Mitmproxy includes a full CA implementation that generates interception certificates on the fly. To get the client to trust these certificates, we register Mitmproxy as a trusted CA with the device manually.

Let’s look into some issues while creating a dummy certificate for MITM proxy to access the server because a dummy certificate is must needed to form a pipe between client and server.

1. Remote hostname

To proceed to generate dummy certificate, we need to know the domain name to use in interception certificate, the client will verify that the certificate is for the domain it’s connecting to, and abort if this is not the case. Let’s check the below request it seems that the CONNECT request above gives us all we need in this example, both of these values are “example.com”. But what if the client had initiated the connection as below

CONNECT 10.1.1.1:443 HTTP/1.1

Using the IP address is perfectly legitimate because it gives us enough information to initiate the pipe, even though it doesn’t reveal the remote hostname.

MITM proxy has a cunning mechanism that as soon as we see the CONNECT request, we pause the client part of the conversation, and initiate a simultaneous connection to the server. We complete the TLS handshake with the server and inspect certificates it used, Now, we use the Common name in the upstream certificates to generate the dummy certificate for the client. So, we have the correct hostname to present to the client, even it was never specified. This way we can generate dummy certificate for the client to connect to the server

2. Subject alternative Name

Sometimes Subject Alternative Name creates the problem, the hostname that the client is connecting to gets dropped because of SAN. The connection between client and domain gets rejected because of the SAN field present in the certificate(certificate of the server) that allows an arbitrary number of alternative domains to be specified. If the expected domain matches any of these then the client will proceed, even though the domain doesn’t match the certificate CN. So here what does Mitm Proxy do, the proxy extract the CN from the upstream certificate, and also the SANs and add them to the generated dummy certificate.

3. Server Name Indication

One of the big limitations of the TLS is that each certificate requires its own IP address. This means that you couldn’t do virtual hosting where multiple domains with independent certificates share the same IP address. So we have a solution in the form of the Server Name Indication extension to the TLS protocols. This lets the client specify the remote server name at the start of the TLS handshake which then lets the base server select the right certificate to complete the process.

SNI breaks our upstream certificate sniffing process because when we connect without using SNI, we get served a default certificate that may have nothing to do with the certificate expected by the client. The solution is another tricky complication to the client connection process. After the client connects, we allow the TLS handshake to continue until just after the SNI value has been passed to us. Now we can pause the conversation, and initiate an upstream connection using the correct SNI value, which then serves us the correct upstream certificate, from which we can extract the expected CN and SANs to create a dummy certificate for the client.

Let’s check above all mechanism in below diagram

HTTPS request call from client to server through a MITM proxy

1. The client makes a connection to MITM proxy, and issues an HTTP CONNECT request.
2. MITM proxy responds with a 200 Connection Established, as if it has set up the CONNECT pipe.
3. The client believes it’s talking to the remote server and initiates the TLS connection. It uses SNI to indicate the hostname it is connecting to.
4. MITM proxy connects to the server and establishes a TLS connection using the SNI hostname indicated by the client.
5. The server responds with the matching certificate, which contains the CN and SAN values needed to generate the interception certificate.
6. MITM proxy generates the interception certificate and continues the client TLS handshake paused in step 3.
7. The client sends the request over the established TLS connection.
8. MITM proxy passes the request on to the server over the TLS connection initiated in step 4.

So, this is how MITM proxy forms the dummy certificate to form a pipe connection between client and server.

I have mentioned anything about the configuration of these MITM proxies or Charles proxies to our clients and servers. That’s all for now on How the MITM proxy works.