SSL Handshaking With Older Clients Using SSLEngine (JSSE)

Question

This is a follow-up question to "SSL Handshaking Using Self-Signed Certs and SSLEngine (JSSE)".

I have implemented a NIO Webserver that can process SSL and non-SSL messages on the same port. In order to distinguish between SSL and non-SSL messages, I check the first byte of the inbound request to see if it is a SSL/TLS message. Example:

byte a = read(buf);
if (totalBytesRead==1 && (a>19 && a<25)){
    parseTLS(buf);
}

In the parseTLS() method I instantiate an SSLEngine, initiate the handshake, wrap/unwrap messages, etc. Everything seems to work fine for most modern web browsers (Firefox 10, IE 9, Safari 5, etc).

Problem is that older web browsers like IE 6 and libraries like Java's URLConnection class seem to initiate the SSL/TLS handshake differently. For example, the first few bytes from IE 6 look something like this (hex values):

80 4F 01 03 00 ...

If I pass the message to the SSLEngine, it doesn't seem to recognize the message and throws an Exception.

javax.net.ssl.SSLException: Unsupported record version Unknown-0.0

So what exactly is IE 6 and Java's URLConnection class sending over? Is this a valid SSL/TLS message that the JSSE SSLEngine can support? Do I have to do some pre-processing or negotiate with the client to send a different message?

Thanks in advance!

UPDATE

Thanks to Bruno and EJP and some further debugging I have a much better understanding of what's going on. As Bruno correctly pointed out, the IE6 and Java 6 clients are sending over a SSLv2 ClientHello. Contrary to one of my earlier comments, the SSLEngine in Java 1.6 can in fact unwrap the SSLv2 message and generate a valid response to send back to the client. The SSLException I reported earlier was an error on my side and has nothing to do with the SSLEngine (I incorrectly assumed that the client was done sending data over and I ended up with an empty ByteBuffer when the SSLEngine was expecting more data to unwrap).

Answer 1

This looks like an SSLv2 Client Hello (see TLS specification):

TLS 1.1 clients that support SSL Version 2.0 servers MUST send SSL Version 2.0 client hello messages [SSL2]. TLS servers SHOULD accept either client hello format if they wish to support SSL 2.0 clients on the same connection port. The only deviations from the Version 2.0 specification are the ability to specify a version with a value of three and the support for more ciphering types in the CipherSpec.

• 80 4F is the length and the high bit must be set to 1 (see msg_length description).
• 01 is the message type (Client Hello)
• 03 00 is the highest supported version (SSLv3 here)

Since Java 7, this is now disabled by default.

EDIT:

Just to clarify, this isn't really an SSLv2 Client Hello, this is a Client Hello for SSLv3 in the SSLv2 format. In this case, the server will reply with a (proper) SSLv3 Server Hello (corresponding to the 03 00 requested version number). The same also works for TLS 1.0, 1.1 and 1.2, although the usage of this format is progressively deprecated.

A JSSE 7 SSLServerSocket will still understand such a Client Hello and reply appropriately with the SSLv3/TLS1.x Server Hello.