SSL Handshaking Using Self-Signed Certs and SSLEngine (JSSE)

Question

I have been tasked to implement a custom/standalone Java webserver that can process SSL and non-SSL messages on the same port.

I have implemented an NIO server and its working quite well for non-SSL requests. I am having a heck of a time with the SSL piece and could really use some guidance.

Here's what I have done so far.

In order to distinguish between SSL and non-SSL messages, I check the first byte of the inbound request to see if it is a SSL/TLS message. Example:

   byte a = read(buf);
   if (totalBytesRead==1 && (a>19 && a<25)){
       parseTLS(buf);
   }

In the parseTLS() method I instantiate an SSLEngine like this:

   java.security.KeyStore ks = java.security.KeyStore.getInstance("JKS");
   java.security.KeyStore ts = java.security.KeyStore.getInstance("JKS");

   ks.load(new java.io.FileInputStream(keyStoreFile), passphrase);
   ts.load(new java.io.FileInputStream(trustStoreFile), passphrase);

   KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
   kmf.init(ks, passphrase);

   TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
   tmf.init(ts);

   SSLContext sslc = SSLContext.getInstance("TLS");     
   sslc.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);


   SSLEngine serverEngine = sslc.createSSLEngine();
   serverEngine.setUseClientMode(false);
   serverEngine.setEnableSessionCreation(true);
   serverEngine.setWantClientAuth(true);

Once the SSLEngine is instantiated, I process the inbound data using the unwrap/wrap methods using code straight out of the official JSSE Samples:

   log("----");

   serverResult = serverEngine.unwrap(inNetData, inAppData);
   log("server unwrap: ", serverResult);
   runDelegatedTasks(serverResult, serverEngine);

   log("----");

   serverResult = serverEngine.wrap(outAppData, outNetData);
   log("server wrap: ", serverResult);
   runDelegatedTasks(serverResult, serverEngine);

The first part of the handshake seems to work just fine. The client sends a handshake message and the server responds with a message with 4 records:

handshake (22)
- server_hello (2) 
- certificate (11) 
- server_key_exchange (12)
- certificate_request (13) 
- server_hello_done (14)

Next, the client sends a message with three parts:

handshake (22)
 - certificate (11)
 - client_key_exchange (16)

change_cipher_spec (20)
 - client_hello (1)

handshake (22)
 *** Encrypted Message ****

The SSLEngine unwraps the client request and parses the records but the wrap method produces 0 bytes with a handshake status of OK/NEED_UNWRAP. In other words, there's nothing for me to send back to the client and the handshake comes to a screeching halt.

This is where I am stuck.

In the debugger, I can see that the SSLEngine, specifically the ServerHandshaker, doesn't find any peer certs. This is rather obvious when I look at the certificate record from the client which is 0 bytes long. But why?

I can only assume that there's something wrong with the HelloServer response but I can't seem to put my finger on it. The server seems to be sending a valid cert but the client isn't sending anything back. Is there a problem with my keystore? Or is it the truststore? Or does it have something to do with the way I'm instantiating the SSLEngine? I'm stumped.

Couple other points:

• The keystore and truststore referenced in the code snippit above were created using the following tutorial: http://www.techbrainwave.com/?p=953
• I'm using Firefox 10 and IE 9 as the client to test the server. Same results for both web clients.
• I'm using Sun/Oracle JDK 6 and Java Secure Socket Extension (JSSE) that comes bundled with it.

I look forward to any guidance you might have but please don't tell me I'm nuts or to use Netty or Grizzly or some other existing solution. Its just not an option at this time. I just want to understand what I'm doing wrong.

Thanks in Advance!

Answer 1

You got NEED_UNWRAP, so do an unwrap. That in turn might give you BUFFER_UNDERFLOW, which means you have to do a read and retry the unwrap.

Similarly when you get NEED_WRAP, do a wrap: that in turn might give you BUFFER_OVERFLOW, which means you have to do a write and retry the wrap.

That wrap or unwrap might in turn might tell you to do another operation: wrap or unwrap.

Just do what it tells you to do.