How to detect an incoming SSL (https) handshake (SSL wire format)?

Question

I'm writing a server which is accepting incoming TCP connections. Let's suppose the server has accepted a TCP connection, and has already received 16 (or so) bytes from the client. Knowing those 16 bytes how can the server detect whether the client wants to initiate an SSL handshake?

I've made an experiment, which showed that on my Linux system connecting to localhost (either 127.0.0.1 or AF_UNIX) via SSL makes the client send the following handshake (hexdump), followed by 16 seemingly random bytes:

8064010301004b0000001000003900003800003500001600001300000a07
00c000003300003200002f03008000000500000401008000001500001200
0009060040000014000011000008000006040080000003020080

How should the server probe these first few bytes, just to be able to determine whether the client is sending an SSL handshake? The probe must return true for all valid SSL handshakes, and it must return false with high probability for a message sent by the client which is not an SSL handshake. It is not allowed to use any libraries (like OpenSSL) for the probe. The probe must be a simple code (like a few dozen lines in C or Python).

Answer 1

The client always sends so called HelloClient message first. It can be in SSL 2 format or SSL 3.0 format (the same format as in TLS 1.0, 1.1 and 1.2).

And there is also possibility that SSL 3.0/TLS 1.0/1.1/1.2 clients send HelloClient with the older format (SSL 2), just with the higher version number in the data. So, detection of SSL 2 HelloClient is neccessary for newer clients too. (For example Java SSL implementation does so)

Let's say 'b' is your buffer. I tried to diagram the message format.

SSL 2

+-----------------+------+-------
| 2 byte header   | 0x01 | etc.
+-----------------|------+-------

• b[0] & 0x80 == 0x80 (it means most significant bit of b[0] is '1')

• ((b[0] & 0x7f) << 8 | b[1]) > 9 (It menas the low 7 bits of b[0] together with b[1] are length of data. You can have less in your buffer, so you cannot check them. But from the message format we know there are 3 field of 2 bytes (length fields), and at least one item in cipher list field (of size 3). So there should be at least 9 bytes (data length >= 9).

• b[2] must be 0x01 (message type "ClientHello")

SSL 3.0 or TLS 1.0, 1.1 and 1.2

+-------+------------------+------------------+--------+------
| 0x16  | 2 bytes version  |  2 bytes length  |  0x01  |  etc.
+-------+------------------+------------------+--------+------

• b[0] == 0x16 (message type "SSL handshake")

• b[1] should be 0x03 (currently newest major version, but who knows in future?)

• b[5] must be 0x01 (handshake protocol message "HelloClient")

For reference, you can see http://www.mozilla.org/projects/security/pki/nss/ssl/draft02.html and http://tools.ietf.org/html/rfc4346