SSL handshake alert: unrecognized_name error since upgrade to Java 1.7.0

Question

I upgraded from Java 1.6 to Java 1.7 today. Since then an error occur when I try to establish a connection to my webserver over SSL:

javax.net.ssl.SSLProtocolException: handshake alert:  unrecognized_name
    at sun.security.ssl.ClientHandshaker.handshakeAlert(ClientHandshaker.java:1288)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1904)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1027)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1262)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1289)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1273)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:523)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1296)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at java.net.URL.openStream(URL.java:1035)

Here is the code:

SAXBuilder builder = new SAXBuilder();
Document document = null;

try {
    url = new URL(https://some url);
    document = (Document) builder.build(url.openStream());
} catch (NoSuchAlgorithmException ex) {
    Logger.getLogger(DownloadLoadiciousComputer.class.getName()).log(Level.SEVERE, null, ex);  
}

Its only a test project thats why I allow and use untrusted certificates with the code:

TrustManager[] trustAllCerts = new TrustManager[]{
    new X509TrustManager() {

        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
            return null;
        }

        public void checkClientTrusted(
                java.security.cert.X509Certificate[] certs, String authType) {
        }

        public void checkServerTrusted(
                java.security.cert.X509Certificate[] certs, String authType) {
        }
    }
};

try {

    SSLContext sc = SSLContext.getInstance("SSL");
    sc.init(null, trustAllCerts, new java.security.SecureRandom());
    HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
} catch (Exception e) {

    Logger.getLogger(DownloadManager.class.getName()).log(Level.SEVERE, null, e);
} 

I sucessfully tried to connect to https://google.com. where is my fault?

Thanks.

Answer 1

Java 7 introduced SNI support which is enabled by default. I have found out that certain misconfigured servers send an "Unrecognized Name" warning in the SSL handshake which is ignored by most clients... except for Java. As @Bob Kerns mentioned, the Oracle engineers refuse to "fix" this bug/feature.

As workaround, they suggest to set the jsse.enableSNIExtension property. To allow your programs to work without re-compiling, run your app as:

java -Djsse.enableSNIExtension=false yourClass

The property can also be set in the Java code, but it must be set before any SSL actions. Once the SSL library has loaded, you can change the property, but it won't have any effect on the SNI status. To disable SNI on runtime (with the aforementioned limitations), use:

System.setProperty("jsse.enableSNIExtension", "false");

The disadvantage of setting this flag is that SNI is disabled everywhere in the application. In order to make use of SNI and still support misconfigured servers:

1. Create a SSLSocket with the host name you want to connect to. Let's name this sslsock.
2. Try to run sslsock.startHandshake(). This will block until it is done or throw an exception on error. Whenever an error occurred in startHandshake(), get the exception message. If it equals to handshake alert: unrecognized_name, then you have found a misconfigured server.
3. When you have received the unrecognized_name warning (fatal in Java), retry opening a SSLSocket, but this time without a host name. This effectively disables SNI (after all, the SNI extension is about adding a host name to the ClientHello message).

For the Webscarab SSL proxy, this commit implements the fall-back setup.