In one of my earlier post I explained how to use Microsoft Network Monitor to debug a networking problem. Network trace tools aren’t very useful in debugging problems when the channel is secured (HTTPS) and you need to view the data to make your conclusions. However you can still debug SSL handshake failures using network monitor. Here’s a scenario.
A client (Not browser) is trying to connect to a IIS web server by sending its client certificate to post some data. However, the IIS machine always rejects the authentication.
The first step was to take a network trace as usual. For instructions on how to capture simultaneous traces, see this post
Analyzing the network trace:
I filtered the traffic by the keyword SSL. In the display filter tab, type SSL and click on Apply button.
Here are some of the frames the we picked from the capture:
During SSL connection negotiation process, the client and server can mutually exchange certificates for authentication. The client authentication can be optional. To understand more about how the SSL negotiation takes place, please see these Microsoft KBs
Description of the Secure Sockets Layer (SSL) Handshake
Description of the Server Authentication Process During the SSL Handshake
Essentially, what is going on here can be summarized as:
While performing any type of debugging, we need to follow the data; we need to look for something that is “interesting”. In this case an “alert” from the server is being sent. So we looked at what this alert is.
Since our “interesting” frame is 32, we looked more at the headers and the details in the frame. Here is the frame in detail
Frame: Number = 32, Captured Frame Length = 61, MediaType = ETHERNET + Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[MAC Address],SourceAddress:[Source MAC Address] + Ipv4: Src = ServerIP, Dest = ClientIP, Next Protocol = TCP, Packet ID = 5022, Total IP Length = 47 + Tcp: Flags=...AP..., SrcPort=5443, DstPort=1100, PayloadLen=7, Seq=2764379044 - 2764379051, Ack=3896915131, Win=64240 (scale factor 0x0) = 64240 - Ssl: Encrypted Alert. - TlsRecordLayer: ContentType: Encrypted Alert - Version: TLS 1.0 Major: 3 (0x3) Minor: 1 (0x1) Length: 2 (0x2) EncryptedData: Binary Large Object (2 Bytes)
Here is the handshake data in Hex (This data isn't encrypted yet because the handshake is still in progress)
00 1E 68 0F 3E 80 00 30 48 7E 1B 90 08 00 45 00 00 2F 13 9E 40 00 3D 06 D2 0E 0A F4 40 0F 0A 33 02 E7 15 43 04 4C A4 C5 13 A4 E8 46 34 BB 50 18 FA F0 2B B8 00 00 15 03 01 00 02 02 2A
We can conclude a lot from this frame. We know that this is an ethernet packet. The TCP flags set are AP, which means transfer data to the end application on the client without buffering at the TCP level, the communication port on the server is 5443, the client port is 1100, the window size for data transmission is 64240 bytes and so forth. The most interesting set of data is the TLS record layer… It shows the major and minor versions in use after negotiation and the length of the data, which is 2 bytes. This data is encrypted.
So here’s the trick now. This is an encrypted alert and RFC defines the alert descriptions. You can find it here for TLS 1.0 (Transport layer security)
Take the hex value of the 2 bytes of this message, which is 2A (The last 2 values, all values before this are headers in hex and we can ignore that because network monitor already gave us that information)
Using Windows calculator, convert hex 2A into decimal. We get: 42.
Now, let’s go look at the TLS 1.0 RFC to find out what what decimal value 42 represents for Alert Description. You can search for “AlertDescription” on the page. You should get:
decompression_failure(30), handshake_failure(40), bad_certificate(42), unsupported_certificate(43), certificate_revoked(44), certificate_expired(45), certificate_unknown(46), illegal_parameter(47), unknown_ca(48), access_denied(49), decode_error(50), decrypt_error(51), export_restriction(60), protocol_version(70), insufficient_security(71), internal_error(80), user_canceled(90), no_renegotiation(100), (255) } AlertDescription;
Thus, we can see that 42 translates to bad_certificate. So we can say that the client sent a bad certificate to the server and therefore the server rejected the connection request. Armed with this information, we checked the client certificate in the certificate store and indeed, we had a bad certificate. Replacing it with a good certificate fixed the problem!
Software has some a long way and enabled people around the world to do many things. It has also made our lives so much easier. However these new capabilities have also created new problems and the answers to those problems are software tools which are excellent utilities for debugging these problems. I want to once again thank all those developers who made these tools, continue to improve and build new tools. It makes the job of a support professional much easier and helps a lot of customers resolve complex problems. Thank you!