Max Euston
Troubleshooting SSL communications using network dumps
While there are some products (notably WireShark) that do a good job of protocol analysis, you don't always have access to them. In these cases, your only option is to look directly at hex dumps from your favorite network trace tool (tcpdump in the case below, but it's the same thing for snoop and other tools).
You should refer to The TLS Protocol (RFC 4346) (especially Appendix A) for details of the individual message formats.
We'll start with the simple case that is used for common https. Then, we'll discuss the additional handshaking steps for a server that requires client authentication (X.509 certificate). When problems occur, we'll see how to show that it's either the client that isn't sending the certificate, or the server that isn't requesting one (often hard to troubleshoot between 2 parties when "finger pointing" occurs, and you only control one side of the connection).
A connection where only the "server" is authenticated (common https)
Let's start with a simple HTTP request over SSL to www.google.com (from my web server). We send a 'GET /' and Google redirects (302) us to port 80 (http).
www$ openssl s_client -connect www.google.com:443 -quiet GET / HTTP/1.0 HTTP/1.0 302 Found Location: http://www.google.com Date: Sun, 02 Mar 2008 23:45:25 GMT Content-Type: text/html; charset=UTF-8 Server: GFE/1.3 Connection: Close Content-Length: 218 <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="http://www.google.com">here</A>. </BODY></HTML>
Now, we didn't need to know what was sent, but let's look at the SSL negotiation "on the wire" and "piece it together" (from my firewall).
gw# tcpdump -s 1500 -lntX host www.google.com and port 443
The general flow of the communication will be as follows (for new connections with full negotiation):
| Step | Client | Server |
| 1 | Open TCP connection | |
| 2 | ClientHello | |
| 3 | ServerHello Certificate ServerHelloDone | |
| 4 | ClientKeyExchange ChangeCipherSpec Finished | |
| 5 | ChangeCipherSpec Finished | |
| 6 | HTTP request (encrypted) | |
| 7 | HTTP response (encrypted) |
Now, let's look at this in detail. (Timing and TCP options have been removed from the traces below for clarity).
- TCP "3-way" handshake (not part of SSL)
- Client sends SYN packet
IP 216.182.45.246.63875 > 64.233.169.99.443: S 2327361744:2327361744(0) win 65535 0x0000: 4500 003c 8b01 4000 4006 bec1 d8b6 2df6 E..<..@.@.....-. 0x0010: 40e9 a963 f983 01bb 8ab8 b8d0 0000 0000 @..c............ 0x0020: a002 ffff 63ae 0000 0204 05b4 0103 0301 ....c........... 0x0030: 0101 080a 0981 ae16 0000 0000 ............Let's do a quick TCP/IP review here. The '5' (above) says that the IP header is 20 (5*4) bytes long. The '6' says it's a TCP packet. At an offset of 20 (5*4) plus 12 bytes (32 or 0x20 hex), we have the length of the TCP header. The 'a' (10) says the TCP header is 40 (10*4) bytes long. Adding both header lengths (IP=20 and TCP=40), we get 60 (or 0x3c) bytes, which is where the SSL data starts (and is after the end of the above packet, because it is just a SYN (no data) in this case).
You can refer to the RFCs for IP and TCP header formats (section 3.1 in both documents).
- Server responds with SYN-ACK packet (IP and TCP headers not shown for ACK-only packets from this point forward)
IP 64.233.169.99.443 > 216.182.45.246.63875: S 306717471:306717471(0) ack 2327361745 win 5672
- Client responds with ACK packet
IP 216.182.45.246.63875 > 64.233.169.99.443: . ack 1 win 33323
- Client sends SYN packet
- Client sends the SSL (version 2) "ClientHello" message (142 bytes, not counting the TCP header)
IP 216.182.45.246.63875 > 64.233.169.99.443: P 1:143(142) ack 1 win 33323 0x0000: 4500 00c2 8b05 4000 4006 be37 d8b6 2df6 E.....@.@..7..-. 0x0010: 40e9 a963 f983 01bb 8ab8 b8d1 1248 2320 @..c.........H#. 0x0020: 8018 822b 7d75 0000 0101 080a 0981 ae1f ...+}u.......... 0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... 0x0040: 0039 0000 3800 0035 0000 1600 0013 0000 .9..8..5........ 0x0050: 0a07 00c0 0000 3300 0032 0000 2f03 0080 ......3..2../... 0x0060: 0000 6600 0005 0000 0401 0080 0800 8000 ..f............. 0x0070: 0063 0000 6200 0061 0000 1500 0012 0000 .c..b..a........ 0x0080: 0906 0040 0000 6500 0064 0000 6000 0014 ...@..e..d..`... 0x0090: 0000 1100 0008 0000 0604 0080 0000 0302 ................ 0x00a0: 0080 fb38 cd42 50d8 4a97 5ecf 7554 9d9c ...8.BP.J.^.uT.. 0x00b0: 898d 1113 b314 1567 505b d7b9 273e 3e4b .......gP[..'>>K 0x00c0: 257a %zWhile the exact structure of this depends on the SSL/TLS versions supported by the client, the important thing to remember is that this first packet is from the client and is the "ClientHello" message. Often, this is as far as we need to go (but we'll break it apart here anyway).
This particular message (which will work with SSLv2 or above servers) consists of the following information:
- The packet length (0x008c => 140 bytes - the "high bit" indicates the format is SSLv2)
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... ^^^^ - The message type (01 => client_hello)
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... ^^ - Highest protocol version supported (03 01 => 3.1 - TLSv1)
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... ^^ ^^ - Length of CipherSuite structure (0x0063 => 33 items, 3 bytes each for SSLv2)
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... ^^ ^^ - Length of Session ID (0x0000 => not present)
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... ^^ ^^ - Length of Challenge (0x0020 => 32 bytes)
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... ^^ ^^ - Cipher Suites supported
0x0030: 1814 400e 808c 0103 0100 6300 0000 2000 ..@.......c..... 0x0040: 0039 0000 3800 0035 0000 1600 0013 0000 .9..8..5........ 0x0050: 0a07 00c0 0000 3300 0032 0000 2f03 0080 ......3..2../... 0x0060: 0000 6600 0005 0000 0401 0080 0800 8000 ..f............. 0x0070: 0063 0000 6200 0061 0000 1500 0012 0000 .c..b..a........ 0x0080: 0906 0040 0000 6500 0064 0000 6000 0014 ...@..e..d..`... 0x0090: 0000 1100 0008 0000 0604 0080 0000 0302 ................ 0x00a0: 0080 fb38 cd42 50d8 4a97 5ecf 7554 9d9c ...8.BP.J.^.uT..
These translate to the following ciphers (in order of preference).
00,00,39 TLS1_CK_DHE_RSA_WITH_AES_256_SHA 00,00,38 TLS1_CK_DHE_DSS_WITH_AES_256_SHA 00,00,35 TLS1_CK_RSA_WITH_AES_256_SHA 00,00,16 SSL3_CK_EDH_RSA_DES_192_CBC3_SHA 00,00,13 SSL3_CK_EDH_DSS_DES_192_CBC3_SHA 07,00,c0 SSL2_CK_DES_192_EDE3_CBC_WITH_MD5 00,00,33 TLS1_CK_DHE_RSA_WITH_AES_128_SHA 00,00,32 TLS1_CK_DHE_DSS_WITH_AES_128_SHA 03,00,80 SSL2_CK_RC2_128_CBC_WITH_MD5 00,00,66 TLS1_CK_DHE_DSS_WITH_RC4_128_SHA 00,00,05 SSL3_CK_RSA_RC4_128_SHA 00,00,04 SSL3_CK_RSA_RC4_128_MD5 01,00,80 SSL2_CK_RC4_128_WITH_MD5 08,00,80 SSL2_CK_RC4_64_WITH_MD5 00,00,63 TLS1_CK_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA 00,00,62 TLS1_CK_RSA_EXPORT1024_WITH_DES_CBC_SHA 00,00,61 TLS1_CK_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5 00,00,15 SSL3_CK_EDH_RSA_DES_64_CBC_SHA 00,00,12 SSL3_CK_EDH_DSS_DES_64_CBC_SHA 00,00,09 SSL3_CK_RSA_DES_64_CBC_SHA 06,00,40 SSL2_CK_DES_64_CBC_WITH_MD5 00,00,65 TLS1_CK_DHE_DSS_EXPORT1024_WITH_RC4_56_SHA 00,00,64 TLS1_CK_RSA_EXPORT1024_WITH_RC4_56_SHA 00,00,60 TLS1_CK_RSA_EXPORT1024_WITH_RC4_56_MD5 00,00,14 SSL3_CK_EDH_RSA_DES_40_CBC_SHA 00,00,11 SSL3_CK_EDH_DSS_DES_40_CBC_SHA 00,00,08 SSL3_CK_RSA_DES_40_CBC_SHA 00,00,06 SSL3_CK_RSA_RC2_40_MD5 04,00,80 SSL2_CK_RC2_128_CBC_EXPORT40_WITH_MD5 00,00,03 SSL3_CK_RSA_RC4_40_MD5 02,00,80 SSL2_CK_RC4_128_EXPORT40_WITH_MD5
See the complete list (as of this writing) to lookup any particular value.
- SSLv2 challenge
0x00a0: 0080 fb38 cd42 50d8 4a97 5ecf 7554 9d9c ...8.BP.J.^.uT.. 0x00b0: 898d 1113 b314 1567 505b d7b9 273e 3e4b .......gP[..'>>K 0x00c0: 257a %z
Server sends ACK to "ClientHello"
IP 64.233.169.99.443 > 216.182.45.246.63875: . ack 143 win 106
- The packet length (0x008c => 140 bytes - the "high bit" indicates the format is SSLv2)
- The next packet from server arrives "out of order" (data not shown here - we'll look at it below)
IP 64.233.169.99.443 > 216.182.45.246.63875: P 1419:1719(300) ack 143 win 106
Client tells the server that there is missing data
IP 216.182.45.246.63875 > 64.233.169.99.443: . ack 1 win 33323
The "missing" packet arrives (no retransmit occurred, the packets simply arrived out of order)
IP 64.233.169.99.443 > 216.182.45.246.63875: . 1:1419(1418) ack 143 win 106
The client now tells the server that is has this and the "out of order" packet
IP 216.182.45.246.63875 > 64.233.169.99.443: . ack 1719 win 32464
These 2 packets (from the server) consist of multiple SSL messages (only ACKs (no data) came from the client).
Even though the client started talking SSLv2, the ClientHello message stated that it supported up to version 0x0301 (TLSv1). The server picks the highest version that both the client and server support, which in this case is not SSLv2. So, the ServerHello message is a SSLv3 format message, which is a bit easier to decode.
IP 64.233.169.99.443 > 216.182.45.246.63875: . 1:1419(1418) ack 143 win 106 0x0000: 4500 05be 82ae 0000 3506 0c93 40e9 a963 E.......5...@..c 0x0010: d8b6 2df6 01bb f983 1248 2320 8ab8 b95f ..-......H#...._ 0x0020: 8010 006a 760a 0000 0101 080a 1814 4044 ...jv.........@D 0x0030: 0981 ae1f 1603 0100 4a02 0000 4603 0147 ........J...F..G 0x0040: cb3d b134 45b5 c9f5 9b06 c53c a8f5 6353 .=.4E......<..cS 0x0050: 8b60 2d3e b4d6 f3cc 7e0b 3b7c 3f08 6320 .`->....~.;|?.c. 0x0060: 2160 0f97 9736 fb60 1786 0197 d974 7cf2 !`...6.`.....t|. 0x0070: 34ab 413c 3cc1 fcd8 c5a0 0991 8aba c316 4.A<<........... 0x0080: 0035 0016 0301 0659 0b00 0655 0006 5200 .5.....Y...U..R.Here is the breakdown of the first record from the server
- ContentType (0x16 => 22 => 'Handshake')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x004a = 74 bytes)
Note: Since there is more data (after the next 74 bytes) from the server, we know there are more messages to process (shown below).- Handshake Type (0x02 => Server Hello)
- Length of below message (0x000046 = 70 bytes)
- Version (0x0301 => 3.1 - TLSv1)
- Timestamp (0x47cb3db1 => 1204501937 => 'Sun Mar 2 23:52:17 GMT 2008')
Note: This does NOT need to be an accurate timestamp, some servers don't have a clock. - 28 Random bytes (0x3445b5c9f59b06c53ca8f563538b602d3eb4d6f3cc7e0b3b7c3f0863)
- Session ID length (0x20 => 32 bytes)
- 32-byte Session ID (0x21600f979736fb6017860197d9747cf234ab413c3cc1fcd8c5a009918abac316)
Note: The server selected this value to identify a new session. The value is otherwise meaningless (no hidden information).
For subsequent TCP connections, the client can simply reference this Session ID to quickly resume the SSL/TLS session (i.e. no complex public key calculations are needed for continued communications using the same SSL/TLS session). - Cypher suite (0x0035 => TLS1_CK_RSA_WITH_AES_256_SHA)
Note: This is the first entry from the CipherSuite list (in the ClientHello message) that the server also supported. - Compression method (0x00 => 'No compression')
Notice that the next few bytes below (0x160301) look familiar... They should, they are the ContentType (0x16) and Version (0x0301), (just like above), followed by a Record Length (0x0659). Look past this portion of the packet dump to see the decoded information. You can see that there appear to be X.509 certificate(s) in these packets.
0x0080: 0035 0016 0301 0659 0b00 0655 0006 5200 .5.....Y...U..R. 0x0090: 0325 3082 0321 3082 028a a003 0201 0202 .%0..!0......... 0x00a0: 1068 7664 383d 496e 2ef5 e319 9842 e07c .hvd8=In.....B.| 0x00b0: ee30 0d06 092a 8648 86f7 0d01 0105 0500 .0...*.H........ 0x00c0: 304c 310b 3009 0603 5504 0613 025a 4131 0L1.0...U....ZA1 0x00d0: 2530 2306 0355 040a 131c 5468 6177 7465 %0#..U....Thawte 0x00e0: 2043 6f6e 7375 6c74 696e 6720 2850 7479 .Consulting.(Pty 0x00f0: 2920 4c74 642e 3116 3014 0603 5504 0313 ).Ltd.1.0...U... 0x0100: 0d54 6861 7774 6520 5347 4320 4341 301e .Thawte.SGC.CA0. 0x0110: 170d 3037 3035 3033 3135 3334 3538 5a17 ..070503153458Z. 0x0120: 0d30 3830 3531 3432 3331 3831 315a 3068 .080514231811Z0h 0x0130: 310b 3009 0603 5504 0613 0255 5331 1330 1.0...U....US1.0 0x0140: 1106 0355 0408 130a 4361 6c69 666f 726e ...U....Californ 0x0150: 6961 3116 3014 0603 5504 0713 0d4d 6f75 ia1.0...U....Mou 0x0160: 6e74 6169 6e20 5669 6577 3113 3011 0603 ntain.View1.0... 0x0170: 5504 0a13 0a47 6f6f 676c 6520 496e 6331 U....Google.Inc1 0x0180: 1730 1506 0355 0403 130e 7777 772e 676f .0...U....www.go 0x0190: 6f67 6c65 2e63 6f6d 3081 9f30 0d06 092a ogle.com0..0...* 0x01a0: 8648 86f7 0d01 0101 0500 0381 8d00 3081 .H............0. 0x01b0: 8902 8181 00e6 c5c6 8dcd 0ba3 0304 dcae ................ 0x01c0: ccc9 46be bdcc 9dbc 7334 48fe d375 64d0 ..F.....s4H..ud. 0x01d0: c9c9 7627 720f a996 1a3b 81f3 14f6 ae90 ..v'r....;...... 0x01e0: 56e7 19d2 7368 a785 a4ae ca24 1430 00ba V...sh.....$.0.. 0x01f0: e836 5d81 733a 7105 8fb1 af11 87da 5cf1 .6].s:q.......\. 0x0200: 3ebf 5351 846f 440e b7e8 26d7 2fb2 6ff2 >.SQ.oD...&./.o. 0x0210: f25d dfa7 cf8c a5e9 1e6f 3048 9421 0b01 .].......o0H.!.. 0x0220: adba 0e71 010d 10ef bfee 2cd3 8dfe 54a8 ...q......,...T. 0x0230: fed3 978f cb02 0301 0001 a381 e730 81e4 .............0.. 0x0240: 3028 0603 551d 2504 2130 1f06 082b 0601 0(..U.%.!0...+.. 0x0250: 0505 0703 0106 082b 0601 0505 0703 0206 .......+........ 0x0260: 0960 8648 0186 f842 0401 3036 0603 551d .`.H...B..06..U. 0x0270: 1f04 2f30 2d30 2ba0 29a0 2786 2568 7474 ../0-0+.).'.%htt 0x0280: 703a 2f2f 6372 6c2e 7468 6177 7465 2e63 p://crl.thawte.c 0x0290: 6f6d 2f54 6861 7774 6553 4743 4341 2e63 om/ThawteSGCCA.c 0x02a0: 726c 3072 0608 2b06 0105 0507 0101 0466 rl0r..+........f 0x02b0: 3064 3022 0608 2b06 0105 0507 3001 8616 0d0"..+.....0... 0x02c0: 6874 7470 3a2f 2f6f 6373 702e 7468 6177 http://ocsp.thaw 0x02d0: 7465 2e63 6f6d 303e 0608 2b06 0105 0507 te.com0>..+..... 0x02e0: 3002 8632 6874 7470 3a2f 2f77 7777 2e74 0..2http://www.t 0x02f0: 6861 7774 652e 636f 6d2f 7265 706f 7369 hawte.com/reposi 0x0300: 746f 7279 2f54 6861 7774 655f 5347 435f tory/Thawte_SGC_ 0x0310: 4341 2e63 7274 300c 0603 551d 1301 01ff CA.crt0...U..... 0x0320: 0402 3000 300d 0609 2a86 4886 f70d 0101 ..0.0...*.H..... 0x0330: 0505 0003 8181 0093 a48e 059d 7d8a f3f8 ............}... 0x0340: 32d0 3b9c 21ce d2e8 55fd 80b5 bbd5 2b54 2.;.!...U.....+T 0x0350: 7a25 acaf 7318 0af9 b77a 995c 1623 4657 z%..s....z.\.#FW 0x0360: fc31 195b 8bf2 0479 73ee b4b2 566b dfd7 .1.[...ys...Vk.. 0x0370: f7d8 56d5 b7aa cde8 9cc8 99f3 764b 6407 ..V.........vKd. 0x0380: adea 9a2b 2092 e692 9b32 847c 8262 779a ...+.....2.|.bw. 0x0390: 15a0 d721 adc8 d98c bb31 829b 1086 a941 ...!.....1.....A 0x03a0: 7a12 e001 5609 06d8 639a 50ee 44ad de75 z...V...c.P.D..u 0x03b0: 4101 7a69 5349 8a00 0327 3082 0323 3082 A.ziSI...'0..#0. 0x03c0: 028c a003 0201 0202 0430 0000 0230 0d06 .........0...0.. 0x03d0: 092a 8648 86f7 0d01 0105 0500 305f 310b .*.H........0_1. 0x03e0: 3009 0603 5504 0613 0255 5331 1730 1506 0...U....US1.0.. 0x03f0: 0355 040a 130e 5665 7269 5369 676e 2c20 .U....VeriSign,. 0x0400: 496e 632e 3137 3035 0603 5504 0b13 2e43 Inc.1705..U....C 0x0410: 6c61 7373 2033 2050 7562 6c69 6320 5072 lass.3.Public.Pr 0x0420: 696d 6172 7920 4365 7274 6966 6963 6174 imary.Certificat 0x0430: 696f 6e20 4175 7468 6f72 6974 7930 1e17 ion.Authority0.. 0x0440: 0d30 3430 3531 3330 3030 3030 305a 170d .040513000000Z.. 0x0450: 3134 3035 3132 3233 3539 3539 5a30 4c31 140512235959Z0L1 0x0460: 0b30 0906 0355 0406 1302 5a41 3125 3023 .0...U....ZA1%0# 0x0470: 0603 5504 0a13 1c54 6861 7774 6520 436f ..U....Thawte.Co 0x0480: 6e73 756c 7469 6e67 2028 5074 7929 204c nsulting.(Pty).L 0x0490: 7464 2e31 1630 1406 0355 0403 130d 5468 td.1.0...U....Th 0x04a0: 6177 7465 2053 4743 2043 4130 819f 300d awte.SGC.CA0..0. 0x04b0: 0609 2a86 4886 f70d 0101 0105 0003 818d ..*.H........... 0x04c0: 0030 8189 0281 8100 d4d3 67d0 8d15 7fae .0........g..... 0x04d0: cd31 fe7d 1d91 a13f 0b71 3cac ccc8 64fb .1.}...?.q<...d. 0x04e0: 63fc 324b 0794 bd6f 80ba 2fe1 0493 c033 c.2K...o../....3 0x04f0: fc09 3323 e90b 742b 71c4 03c6 d2cd e22f ..3#..t+q....../ 0x0500: f509 63cd ff48 a500 bfe0 e7f3 88b7 2d32 ..c..H........-2 0x0510: de98 36e6 0aad 007b c464 4a3b 8475 03f2 ..6....{.dJ;.u.. 0x0520: 7092 7d0e 62f5 21ab 6936 8431 7590 f8bf p.}.b.!.i6.1u... 0x0530: c76c 881b 0695 7cc9 e5a8 de75 a12c 7a68 .l....|....u.,zh 0x0540: dfd5 ca1c 8758 6019 0203 0100 01a3 81fe .....X`......... 0x0550: 3081 fb30 1206 0355 1d13 0101 ff04 0830 0..0...U.......0 0x0560: 0601 01ff 0201 0030 0b06 0355 1d0f 0404 .......0...U.... 0x0570: 0302 0106 3011 0609 6086 4801 86f8 4201 ....0...`.H...B. 0x0580: 0104 0403 0201 0630 2806 0355 1d11 0421 .......0(..U...! 0x0590: 301f a41d 301b 3119 3017 0603 5504 0313 0...0.1.0...U... 0x05a0: 1050 7269 7661 7465 4c61 6265 6c33 2d31 .PrivateLabel3-1 0x05b0: 3530 3106 0355 1d1f 042a 3028 3026 501..U...*0(0&And here is the data (minus the IP and TCP headers) from the "out of order" packet0x0030: a024 a022 8620 6874 7470 3a2f .$."..http:/ 0x0040: 2f63 726c 2e76 6572 6973 6967 6e2e 636f /crl.verisign.co 0x0050: 6d2f 7063 6133 2e63 726c 3032 0608 2b06 m/pca3.crl02..+. 0x0060: 0105 0507 0101 0426 3024 3022 0608 2b06 .......&0$0"..+. 0x0070: 0105 0507 3001 8616 6874 7470 3a2f 2f6f ....0...http://o 0x0080: 6373 702e 7468 6177 7465 2e63 6f6d 3034 csp.thawte.com04 0x0090: 0603 551d 2504 2d30 2b06 082b 0601 0505 ..U.%.-0+..+.... 0x00a0: 0703 0106 082b 0601 0505 0703 0206 0960 .....+.........` 0x00b0: 8648 0186 f842 0401 060a 6086 4801 86f8 .H...B....`.H... 0x00c0: 4501 0801 300d 0609 2a86 4886 f70d 0101 E...0...*.H..... 0x00d0: 0505 0003 8181 0055 ac63 eade a1dd d290 .......U.c...... 0x00e0: 5f9f 0bce 76be 1351 8f93 d905 2bc8 1b77 _...v..Q....+..w 0x00f0: 4bad 6950 a1ee dedc fddb 07e9 e839 94dc K.iP.........9.. 0x0100: ab72 792f 06bf ab81 70c4 a8ed ea53 34ed .ry/....p....S4. 0x0110: ef1e 53d9 06c7 562b d15c f4d1 8a8e b42b ..S...V+.\.....+ 0x0120: b137 9048 0842 25c5 3e8a cb7f eb6f 04d1 .7.H.B%.>....o.. 0x0130: 6dc5 74a2 f7a2 7c7b 603c 77cd 0ece 4802 m.t...|{`<w...H. 0x0140: 7f01 2fb6 9b37 e02a 2a36 dcd5 85d6 ace5 ../..7.**6...... 0x0150: 3f54 6f96 1e05 af16 0301 0004 0e00 0000 ?To.............Here is the breakdown of the second record from the server
- ContentType (0x16 => 22 => 'Handshake')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0659 = 1625 bytes)
- Handshake Type (0x0b => 11 => Certificate)
- Length of below message (0x000655 = 1621 bytes)
- Length of certificate chain (0x000652 => 1618 bytes)
- Length of X.509 Certificate #1 (0x000325 => 805 bytes)
- X.509 Certificate #1 (805 bytes)
- Length of X.509 Certificate #2 (0x000327 => 807 bytes)
- X.509 Certificate #2 (807 bytes)
- Length of certificate chain (0x000652 => 1618 bytes)
Again, notice that the next few bytes below look familiar (0x160301 and a length)
0x0150: 3f54 6f96 1e05 af16 0301 0004 0e00 0000 ?To.............Here is the breakdown of the third record from the server
- ContentType (0x16 => 22 => 'Handshake')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0004 = 4 bytes)
- Handshake Type (0x0e => 14 => Server Hello Done)
- Length of below message (0x000000 = 0 bytes)
(Empty message)
- No more data comes from the server at this point. The next packet is from the client.
IP 216.182.45.246.63875 > 64.233.169.99.443: P 143:282(139) ack 1719 win 33323 0x0000: 4500 00bf 8b8d 4000 4006 bdb2 d8b6 2df6 E.....@.@.....-. 0x0010: 40e9 a963 f983 01bb 8ab8 b95f 1248 29d6 @..c......._.H). 0x0020: 8018 822b 063c 0000 0101 080a 0981 ae3d ...+.<.........= 0x0030: 1814 4044 1603 0100 8610 0000 8200 808f ..@D............ 0x0040: 0904 e0c3 ae63 8977 abec b463 f3d8 877f .....c.w...c.... 0x0050: bec5 899b 72cd f0df 4d36 5b7a 1f2c 46b5 ....r...M6[z.,F. 0x0060: 2791 4814 0881 7613 a07f 731b d637 f5db '.H...v...s..7.. 0x0070: e055 cc53 a70a 291f 7043 123a 9c84 1ddf .U.S..).pC.:.... 0x0080: 42ac 42bf 2e0a 2918 1c3d 8b38 bac8 b2f9 B.B...)..=.8.... 0x0090: bd06 cc12 b4b4 ddd5 b0fe 17ed baad 9c2d ...............- 0x00a0: 9675 2a80 25aa afdb 1f78 5590 b28c 635d .u*.%....xU...c] 0x00b0: f4ea 079f bb67 2b29 1411 b495 efad a8 .....g+).......Here is the breakdown of the second record from the client
- ContentType (0x16 => 22 => 'Handshake')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0086 = 134 bytes)
- Handshake Type (0x10 => 16 => Client Key Exchange)
- Length of below message (0x000082 = 130 bytes)
- Length of Encrypted Pre-Master-Secret (0x0080 => 128 bytes)
Note: The RSA format of the message (as opposed to DH) is used since the Server Certificate and chosen CipherSuite specified RSA encryption. Also note that "Block Mode" ciphers (Cipher Block Chaining or CBC) can cause the output to be padded to the next block size. - Pre-Master-Secret encrypted with server's public key
- Length of Encrypted Pre-Master-Secret (0x0080 => 128 bytes)
The server then sends a TCP ACK to this packet.
IP 64.233.169.99.443 > 216.182.45.246.63875: . ack 282 win 123
Next, the client sends another message.
IP 216.182.45.246.63875 > 64.233.169.99.443: P 282:341(59) ack 1719 win 33323 0x0000: 4500 006f 8b99 4000 4006 bdf6 d8b6 2df6 E..o..@.@.....-. 0x0010: 40e9 a963 f983 01bb 8ab8 b9ea 1248 29d6 @..c.........H). 0x0020: 8018 822b 9f8e 0000 0101 080a 0981 ae45 ...+...........E 0x0030: 1814 4193 1403 0100 0101 1603 0100 30ae ..A...........0.Here is the breakdown of the third record from the client
- ContentType (0x14 => 20 => 'ChangeCipherSpec')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0001 = 1 byte)
- change_cipher_spec (0x01)
Again, since there is more data from the client, then there must be another message.
0x0030: 1814 4193 1403 0100 0101 1603 0100 30ae ..A...........0. 0x0040: cf4d 600d b2c4 b0d6 d36a ef15 349b ffbd .M`......j..4... 0x0050: d371 fc8f 0777 ed7e e086 f3d2 f445 4cf4 .q...w.~.....EL. 0x0060: 20aa 28bb e8a1 7a7a 1763 6134 a32d 4c ..(...zz.ca4.-LHere is the breakdown of the fourth record from the client
- ContentType (0x16 => 22 => 'Handshake')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0030 = 48 bytes)
- *** Unfortunately, we can not interpret this further. Any more data from the client (after a 'change_cipher_spec'
message) is encrypted using the agreed-upon parameters (shared secret). However, we can still see that no
'Application'-layer data has passed (yet).
- *** Unfortunately, we can not interpret this further. Any more data from the client (after a 'change_cipher_spec'
message) is encrypted using the agreed-upon parameters (shared secret). However, we can still see that no
'Application'-layer data has passed (yet).
Again, the server then sends a TCP ACK to this packet.
IP 64.233.169.99.443 > 216.182.45.246.63875: . ack 341 win 123
- Now, the server sends another message
IP 64.233.169.99.443 > 216.182.45.246.63875: P 1719:1778(59) ack 341 win 123 0x0000: 4500 006f 82b6 0000 3506 11da 40e9 a963 E..o....5...@..c 0x0010: d8b6 2df6 01bb f983 1248 29d6 8ab8 ba25 ..-......H)....% 0x0020: 8018 007b 6f86 0000 0101 080a 1814 41be ...{o.........A. 0x0030: 0981 ae45 1403 0100 0101 1603 0100 305a ...E..........0ZJust like the client, now the server will switch to encrypted communication
- ContentType (0x14 => 20 => 'ChangeCipherSpec')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0001 = 1 byte)
- change_cipher_spec (0x01)
Similarly, we can not see inside this next Handshake message from the server
0x0030: 0981 ae45 1403 0100 0101 1603 0100 305a ...E..........0Z 0x0040: cc13 fabd 7154 c98d b8c0 4c96 8f7e 9ce3 ....qT....L..~.. 0x0050: 555a 7157 bf85 e21b bc4c f234 b7c1 72f7 UZqW.....L.4..r. 0x0060: ecf0 0762 b914 1bde ef1b 4036 72b6 4a ...b......@6r.J- ContentType (0x16 => 22 => 'Handshake')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0030 = 48 bytes)
- *** Encrypted Handshake Message
- The client sends more packets of encrypted data (and the server sends TCP ACKs).
Note: Even though the connection is encrypted, we can see something different (a new record type) here.
ContentType 0x17 (23) is for 'ApplicationData' (Version and Length fields are still available, but the encapsulated data is encrypted).IP 216.182.45.246.63875 > 64.233.169.99.443: P 341:431(90) ack 1778 win 33323 0x0000: 4500 008e 8bd9 4000 4006 bd97 d8b6 2df6 E.....@.@.....-. 0x0010: 40e9 a963 f983 01bb 8ab8 ba25 1248 2a11 @..c.......%.H*. 0x0020: 8018 822b ce37 0000 0101 080a 0981 ae50 ...+.7.........P 0x0030: 1814 41be 1703 0100 20d3 42c9 f8a7 2e69 ..A.......B....i 0x0040: 9baf 5bad 9264 a529 0363 6ce3 95d6 22b8 ..[..d.).cl...". 0x0050: eaba 4a47 8739 6141 a817 0301 0030 f251 ..JG.9aA.....0.Q 0x0060: 87f5 38b2 3d53 dfa7 ae95 cede 66de 6903 ..8.=S......f.i. 0x0070: 285a 9d1e 021b f59e 514c 83a0 8e4f 1581 (Z......QL...O.. 0x0080: 0229 fd21 330d f28f e819 97bb ebb4 .).!3......... IP 64.233.169.99.443 > 216.182.45.246.63875: . ack 431 win 123 IP 216.182.45.246.63875 > 64.233.169.99.443: P 431:505(74) ack 1778 win 33323 0x0000: 4500 007e 8be2 4000 4006 bd9e d8b6 2df6 E..~..@.@.....-. 0x0010: 40e9 a963 f983 01bb 8ab8 ba7f 1248 2a11 @..c.........H*. 0x0020: 8018 822b 0732 0000 0101 080a 0981 ae58 ...+.2.........X 0x0030: 1814 4251 1703 0100 2002 75a5 7e3a a45d ..BQ......u.~:.] 0x0040: 6420 8041 1665 b78d bdf4 a69b ab06 ddff d..A.e.......... 0x0050: 790c 0b3e af03 13b8 d217 0301 0020 7727 y..>..........w' 0x0060: f5e6 8b11 b364 50a4 0e4b f102 959c cd6c .....dP..K.....l 0x0070: 52f9 767b decf 486e e22b 0632 a906 R.v{..Hn.+.2.. IP 64.233.169.99.443 > 216.182.45.246.63875: . ack 505 win 123Even though we can't see inside these packets, we know (from our test, and the small packet size) that it is from our "GET" request. However, there seems to be more data here than our simple 16-byte 'GET' request. Remember that in addition to encryption, SSL/TLS provides authentication. In order to ensure that some "3rd party" isn't injecting "bogus" packets into the connection, each encrypted message includes a Message Authentication Code (MAC). While this adds a small overhead (for non-trivial examples) to the application data, it ensures that no tampering (data modification/deletion/duplication) occurred in the process.
- The next data from the server should be the HTTP response, but a (FIN) packet arrives out-of-order.
IP 64.233.169.99.443 > 216.182.45.246.63875: F 2215:2215(0) ack 505 win 123 0x0000: 4500 0034 82be 0000 3506 120d 40e9 a963 E..4....5...@..c 0x0010: d8b6 2df6 01bb f983 1248 2bc6 8ab8 bac9 ..-......H+..... 0x0020: 8011 007b f40f 0000 0101 080a 1814 427b ...{..........B{ 0x0030: 0981 ae58 ...XLike before, the client tells the server that there is missing data.
IP 216.182.45.246.63875 > 64.233.169.99.443: . ack 1778 win 33323
Finally, the HTTP response arrives.
IP 64.233.169.99.443 > 216.182.45.246.63875: P 1778:2215(437) ack 505 win 123 0x0000: 4500 01e9 82bc 0000 3506 105a 40e9 a963 E.......5..Z@..c 0x0010: d8b6 2df6 01bb f983 1248 2a11 8ab8 bac9 ..-......H*..... 0x0020: 8018 007b 2ff6 0000 0101 080a 1814 427b ...{/.........B{ 0x0030: 0981 ae58 1703 0101 b0b6 1faa 8072 9c12 ...X.........r.. 0x0040: 4a44 e029 832b 2e0d 8969 4cbe fad5 5516 JD.).+...iL...U.(most encrypted data not shown)0x01c0: 1f1c 0533 3180 60a0 4c68 38e0 68df 49c2 ...31.`.Lh8.h.I. 0x01d0: 089e e102 01a4 88e3 90fc cc91 9602 669d ..............f. 0x01e0: b3a7 fa2b ea6d 6cac b6 ...+.ml..The client sends a TCP ACK for the above data (and the out-of-order FIN packet, which closed the server side of the connection).
IP 216.182.45.246.63875 > 64.233.169.99.443: . ack 2216 win 33104
- Finally, the client sends another new record type.
IP 216.182.45.246.63875 > 64.233.169.99.443: P 505:542(37) ack 2216 win 33323 0x0000: 4500 0059 8bf7 4000 4006 bdae d8b6 2df6 E..Y..@.@.....-. 0x0010: 40e9 a963 f983 01bb 8ab8 bac9 1248 2bc7 @..c.........H+. 0x0020: 8018 822b cd53 0000 0101 080a 0981 ae60 ...+.S.........` 0x0030: 1814 427b 1503 0100 20e7 ba28 eb31 e583 ..B{.......(.1.. 0x0040: 04ca bddb 3673 53e4 211c b52d 0c6d 89b7 ....6sS.!..-.m.. 0x0050: 4098 b802 6f5b e7aa da @...o[...- ContentType (0x15 => 21 => 'Alert')
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x0020 = 32 bytes)
- *** Encrypted Alert Message
While the server should have sent a TLS "Close Alert" first, it simply closed the connection. Additionally, it responded to the client's 'Alert' and FIN packets with TCP RST (Reset) packets. Even though this is not proper operation, in today's world of Distributed Denial Of Service (DDOS) attacks, this practice has become commonplace (but still wrong).
IP 216.182.45.246.63875 > 64.233.169.99.443: F 542:542(0) ack 2216 win 33323 IP 64.233.169.99.443 > 216.182.45.246.63875: R 306719687:306719687(0) win 0 IP 64.233.169.99.443 > 216.182.45.246.63875: R 306719687:306719687(0) win 0
A connection that uses a "client" certificate (mutual authentication)
Now that we have seen how "common https" works, let's look at the additional steps (noted * below) required for client authentication.
| Step | Client | Server |
| 1 | Open TCP connection | |
| 2 | ClientHello | |
| 3 | ServerHello Certificate ServerKeyExchange* CertificateRequest* ServerHelloDone | |
| 4 | Certificate* ClientKeyExchange CertificateVerify* ChangeCipherSpec Finished | |
| 5 | ChangeCipherSpec Finished | |
| 6 | HTTP request (encrypted) | |
| 7 | HTTP response (encrypted) |
Here are exceprts from a trace with another server (one that requires client authentication via certificate).
Note that unlike the previous example (where each and every TLS "message" was within its own TLS "record"), this time the server placed multiple TLS "messages" inside one TLS "record".
0x0030: 00d2 5d6e 1603 0119 a202 0000 4603 0100 ..]n........F...
Here is the basic structure of this "one large record"
- ContentType (0x16 => 22 => 'Handshake' record)
- Version (0x0301 => 3.1 - TLSv1)
- Length of below record (0x19a2 => 6562 bytes)
- Handshake Type (0x02 => ServerHello)
- Length of below message (0x000046 => 69 bytes)
- Version (0x0301 => 3.1 - TLSv1)
- Timestamp (0x00000047 - Note that this is not "real", but it isn't required to be)
- 28 Random bytes (0xa516d0d18ff0370478b468407086ebc420d63f5beaf8dd1e727c7c8e)
- Session ID length (0x20 => 32 bytes)
- 32-byte Session ID (0x00000143256394d7b750c385cc363112ec86df205858585847d540360000ccc7)
Note: This server is storing some "hidden" information inside the session ID - this is NOT advised! - Cypher suite (0x0035 => TLS1_CK_RSA_WITH_AES_256_SHA)
- Compression method (0x00 => 'No compression')
- Handshake Type (0x0b => 11 => Certificate)
- Length of below message (0x000c14 => 3092 bytes)
- Length of certificate chain (0x000c11 => 3089 bytes)
- Length of X.509 Certificate #1 (0x000528 => 1320 bytes)
- Certificate #1 (1320 bytes)
- Length of X.509 Certificate #2 (0x0004a0 => 1184 bytes)
- Certificate #2 (1184 bytes)
- Length of X.509 Certificate #3 (0x000240 => 576 bytes)
- Certificate #3 (576 bytes)
- Length of certificate chain (0x000c11 => 3089 bytes)
- Handshake Type (0x0d => 13 => CertificateRequest)
- Length of below message (0x000d38 => 3384 bytes)
- Length of Certificate Type List (0x01 => 1 byte)
- Certificate Types #1 (0x01 - rsa_sign)
- Length of Certificate Authorities DN List (0x0d34 => 3380 bytes)
- Length of Certificate Authority DN #1 (0x0061 => 97 bytes)
- Certificate Authority DN #1
Using 'openssl asn1parse -inform der', you get the following0:d=0 hl=2 l= 95 cons: SEQUENCE 2:d=1 hl=2 l= 11 cons: SET 4:d=2 hl=2 l= 9 cons: SEQUENCE 6:d=3 hl=2 l= 3 prim: OBJECT :countryName 11:d=3 hl=2 l= 2 prim: PRINTABLESTRING :US 15:d=1 hl=2 l= 23 cons: SET 17:d=2 hl=2 l= 21 cons: SEQUENCE 19:d=3 hl=2 l= 3 prim: OBJECT :organizationName 24:d=3 hl=2 l= 14 prim: PRINTABLESTRING :VeriSign, Inc. 40:d=1 hl=2 l= 55 cons: SET 42:d=2 hl=2 l= 53 cons: SEQUENCE 44:d=3 hl=2 l= 3 prim: OBJECT :organizationalUnitName 49:d=3 hl=2 l= 46 prim: PRINTABLESTRING :Class 3 Public Primary Certification Authority - (Remaining Certificate Authority DNs)
- Handshake Type (0x0e => 14 => Server Hello Done)
- Length of below message (0x000000 = 0 bytes)
(Empty message)
0x0030: 00d2 5d6e 1603 0119 a202 0000 4603 0100 ..]n........F...
0x0040: 0000 47a5 16d0 d18f f037 0478 b468 4070 ..G......7.x.h@p
0x0050: 86eb c420 d63f 5bea f8dd 1e72 7c7c 8e20 .....?[....r||..
0x0060: 0000 0143 2563 94d7 b750 c385 cc36 3112 ...C%c...P...61.
0x0070: ec86 df20 5858 5858 47d5 4036 0000 ccc7 ....XXXXG.@6....
0x0080: 0035 000b 000c 1400 0c11 0005 2830 8205 .5..........(0..
(Certificate #1)
0x05b0: 42a9 11d8 c700 04a0 3082 049c 3082 0405 B.......0...0...
0x05c0: a003 0201 0202 1075 .......u
(next packet - IP and TCP headers not shown)
0x0030: 337d 9ab0 e123 3bae 2d7d e446 3}...#;.-}.F
(Certificate #2)
0x04c0: e603 900c 0002 4030 8202 3c30 8201 a502 ......@0..<0....
(Certificate #3)
0x05c0: 6669 6361 7469 6f6e fication
(next packet - IP and TCP headers not shown)
0x0030: 2041 7574 686f 7269 7479 3081 .Authority0.
(Certificate #3 - continued)
0x0170: 640d 640d 000d 3801 010d 3400 6130 5f31 d.d...8...4.a0_1
0x0180: 0b30 0906 0355 0406 1302 5553 3117 3015 .0...U....US1.0.
0x0190: 0603 5504 0a13 0e56 6572 6953 6967 6e2c ..U....VeriSign,
0x01a0: 2049 6e63 2e31 3730 3506 0355 040b 132e .Inc.1705..U....
0x01b0: 436c 6173 7320 3320 5075 626c 6963 2050 Class.3.Public.P
0x01c0: 7269 6d61 7279 2043 6572 7469 6669 6361 rimary.Certifica
0x01d0: 7469 6f6e 2041 7574 686f 7269 7479 0061 tion.Authority.a
(Remaining Certificate Authority DNs)
0x05c0: 7465 2043 6f6e 7375 te.Consu
(next packet - IP and TCP headers not shown)
0x0030: 6c74 696e 6731 2830 2606 0355 lting1(0&..U
(Remaining Certificate Authority DNs)
0x05c0: 0313 2156 6572 6953 ..!VeriS
(next packet - IP and TCP headers not shown)
0x0030: 6967 6e20 436c 6173 7320 3320 ign.Class.3.
(Remaining Certificate Authority DNs)
0x0380: 6172 6477 6172 650e 0000 00 ardware....
The 'ServerKeyExchange' is very much like the 'ClientKeyExchange'.
The Client 'Certificate' is exactly like the Server 'Certificate'.
The 'CertificateVerify' message is simply a signature and is formatted as follows:
- Handshake Type (0x0f => 15 => CertificateVerify)
- Length of below message (0x000014/0x000024 => 20/36 bytes)
Note: There are 2 formats (20 bytes for DSA and 36 bytes for RSA)- Signature value
When you have problems getting mutual authentication to work, you have to ensure that all the pieces are working correctly:
- The server is presenting a complete and correct certificate chain
Use: openssl s_client -connect server:443 -showcerts
While the root CA is optional (since by definition, both sides must already have it and trust it), you should make sure that the server presents all the remaining certificates (especially any "intermediate" CAs). - The server is requesting a client certificate
Look for the CertificateRequest Handshake Message (type 0x0d) from the server - The server should be providing the correct list of acceptable CAs
Look inside the CertificateRequest Message for the desired CAs. If you don't see the one you expect, ensure that you you have all chains correctly installed in the server's keystore (whatever format it uses). Be especially aware of the common mistake of installing the root CA and the server certificate while missing any intermediate CA (and make sure that they aren't "expired"). - The client should be sending the Certificate (type 0x0b) and CertificateVerify
(type 0x0f) Messages.
Similar to the previous check (server sending CertificateRequest), make sure that the client has all the correct certificate chains in its keystore. Again, be especially aware of the common mistake of installing the root CA and the client certificate while missing any intermediate CA (and make sure that they aren't "expired").
