Java 11 and 12 SSL sockets fail on a handshake_failure error with TLSv1.3 enabled

java ssl sslhandshakeexception tls1.3
Corentor picture Corentor · Aug 22, 2019 · Viewed 12.6k times · Source

While developing a SSL-based (non HTTP) server in Java 12, I met unexpected difficulties for having the server and the client talk together. The connection was always interrupted with a handshake_failure error. After lots of efforts (initially I was incriminating my certificates), I was able to isolate the issue thanks to a HelloWorld SSL client/server pair at the socket level: the handshake failure was caused by the presence of TLSv1.3 among the protocols enabled by default in the JDK.

So far, my solution to avoid the handshake error is to disable TLSv1.3 completely (server-side), but I am not satisfied by this solution. I would like to understand what's going on (assuming it's not a bug in the JDK or in the system but in my application or my environment).

I provide some code below. Can anyone check and tell me what I am doing wrong? Thanks and best regards.

Here is my configuration: OSX 10.14 (Mojave) I have tested with the following versions of Java:

EDIT: I added Java 11 to my tests and the problem occurs also for this version

  • Java 8 (oracle64-1.8.0.181): no handshake failure
  • Java 10 (oracle64-10.0.2 2018-07-17): no handshake failure
  • Java 11 (openjdk version "11.0.4" 2019-07-16): handshake failure
  • Java 12 (oracle64-12.0.1 2019-04-16): handshake failure
  • Java 12 (openjdk 12.0.2 2019-07-16): handshake failure
// HelloSSLServer.java

import java.net.*;
import javax.net.*;
import javax.net.ssl.*;
import java.io.*;

public class HelloSSLServer {

    public static void main(String args[]) throws Exception {
        int port = 1234;
        boolean needClientAuth = false;
        ServerSocketFactory factory = SSLServerSocketFactory.getDefault();
        try (ServerSocket ss = factory.createServerSocket(port)) {
            SSLServerSocket ssl = (SSLServerSocket) ss;
            ssl.setNeedClientAuth(needClientAuth);
            ssl.setEnabledProtocols(new String[] { "TLSv1.2" }); // fails with TLSv1.3 (or with the default, i.e. no call)
            while (true) {
                try (Socket socket = ss.accept()) {
                    PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
                    out.println("Hello World!");
                }
            }

        }
    }
}

// HelloSSLClient.java

import java.net.*;
import javax.net.*;
import javax.net.ssl.*;
import java.io.*;

public class HelloSSLClient {

    public static void main(String args[]) throws Exception {
        String host = "localhost";
        int port = 1234;
        SocketFactory factory = SSLSocketFactory.getDefault();
        try (Socket connection = factory.createSocket(host, port)) {
            SSLSocket ssl = (SSLSocket) connection;

            SSLParameters sslParams = new SSLParameters();
            sslParams.setEndpointIdentificationAlgorithm("HTTPS");
            ssl.setSSLParameters(sslParams);

            BufferedReader input =
                new BufferedReader(new InputStreamReader(connection.getInputStream()));
            String message = input.readLine();
            System.out.println("Got the message: " + message);
        }
    }
}

(a server's certificate will also be needed in the server's keystore and in the client's truststore for testing any SSL application)

Here is the outcome (when the error occurs):

$ java -Djavax.net.ssl.keyStore=serverkeystore.ks -Djavax.net.ssl.keyStorePassword=xxxxxxx HelloSSLServer

$ java -Djavax.net.ssl.trustStore=clienttruststore.ks -Djavax.net.ssl.trustStorePassword=xxxxxx HelloSSLClient
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
    at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:180)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1180)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1091)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:721)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:804)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
    at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
    at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
    at HelloSSLClient.main(HelloSSLClient.java:21)

As suggested by @SvetlinZarev and @user207421, here are some snippets of the output with the -Djavax.net.debug=ssl:handshake enabled. Since the code works when one forces the TLSv1.2 protocol, it is more interesting to see what's going on when one simply uses the default (call to ssl.setEnabledProtocols commented out):

NOTE: the project is personal and experimental and the server's certificate is fake, self-signed, and added to the client's truststore (the keystore and the truststore too are dummy), and would be public anyway, so I do not fear to release any sensitive information

From the client side:

java -Djavax.net.debug=ssl:handshake -Djavax.net.ssl.trustStore=clienttruststore.ks -Djavax.net.ssl.trustStorePassword=xxxxxxxxx HelloSSLClient
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.229 CEST|SSLCipher.java:463|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.451 CEST|ServerNameExtension.java:261|Unable to indicate server name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.451 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: server_name
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.455 CEST|SignatureScheme.java:282|Signature algorithm, ed25519, is not supported by the underlying providers
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.455 CEST|SignatureScheme.java:282|Signature algorithm, ed448, is not supported by the underlying providers
javax.net.ssl|INFO|01|main|2019-08-22 09:54:22.459 CEST|AlpnExtension.java:161|No available application protocols
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.460 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.460 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: cookie
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.477 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: renegotiation_info
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.477 CEST|PreSharedKeyExtension.java:633|No session to resume.
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.478 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: pre_shared_key
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.482 CEST|ClientHello.java:653|Produced ClientHello handshake message (
"ClientHello": {
  "client version"      : "TLSv1.2",
  "random"              : "8D 95 FA BE D8 F4 BC AC E6 15 36 FE FE A2 57 C4 DD EF F6 53 B8 54 1D 4A ED AE C6 0A CD 92 E0 A4",
  "session id"          : "CC EE 0F 29 F8 9A 3B 72 61 61 99 46 AA 69 CF 23 4F E9 05 13 2A 52 B8 1D 34 18 FA DF 26 1B 46 87",
  "cipher suites"       : "[TLS_AES_128_GCM_SHA256(0x1301), TLS_AES_256_GCM_SHA384(0x1302), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B), TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA9), TLS_DHE_DSS_WITH_AES_128_CBC_SHA256(0x0040), 

[...]

TLS_ECDH_RSA_WITH_AES_128_CBC_SHA(0xC00E), TLS_DHE_RSA_WITH_AES_128_CBC_SHA(0x0033), TLS_DHE_DSS_WITH_AES_128_CBC_SHA(0x0032), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.528 CEST|ServerHello.java:871|Consuming ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "B5 27 FE 28 29 85 AC 1A C4 62 57 28 45 12 63 BA 4D CC 4B E0 02 A4 A9 7A ED 9F A3 8D A6 98 85 BE",
  "session id"          : "CC EE 0F 29 F8 9A 3B 72 61 61 99 46 AA 69 CF 23 4F E9 05 13 2A 52 B8 1D 34 18 FA DF 26 1B 46 87",
  "cipher suite"        : "TLS_AES_128_GCM_SHA256(0x1301)",
  "compression methods" : "00",
  "extensions"          : [
    "supported_versions (43)": {
      "selected version": [TLSv1.3]
    },
    "key_share (51)": {
      "server_share": {

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.528 CEST|SSLExtensions.java:189|Consumed extension: supported_versions
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.529 CEST|ServerHello.java:967|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.530 CEST|SSLExtensions.java:160|Ignore unsupported extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.530 CEST|SSLExtensions.java:160|Ignore unsupported extension: max_fragment_length
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.530 CEST|SSLExtensions.java:160|Ignore unsupported extension: status_request

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.582 CEST|SSLExtensions.java:170|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.582 CEST|SSLExtensions.java:170|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:189|Consumed extension: supported_groups
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:204|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:204|Ignore unavailable extension: max_fragment_length
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:212|Ignore impact of unsupported extension: supported_groups
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:204|Ignore unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.585 CEST|Alert.java:238|Received alert message (
"Alert": {
  "level"      : "fatal",
  "description": "handshake_failure"
}
)
javax.net.ssl|ERROR|01|main|2019-08-22 09:54:22.586 CEST|TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): Received fatal alert: handshake_failure (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)

And here is the server side:

java -Djavax.net.debug=ssl,handshake -Djavax.net.ssl.keyStore=serverkeystore.ks -Djavax.net.ssl.keyStorePassword=xxxxxxxxx HelloSSLServer
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:55.869 CEST|SSLCipher.java:463|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.697 CEST|SignatureScheme.java:282|Signature algorithm, ed25519, is not supported by the underlying providers
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.698 CEST|SignatureScheme.java:282|Signature algorithm, ed448, is not supported by the underlying providers
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.704 CEST|ClientHello.java:809|Consuming ClientHello handshake message (
"ClientHello": {
  "client version"      : "TLSv1.2",
  "random"              : "0D E6 53 8A B0 E4 E7 9A 80 93 49 84 AD 88 0A 5F D5 7F 29 37 C3 86 A7 28 A7 D9 C6 7D EB DF 6A 3D",
  "session id"          : "5B 9A 18 25 31 65 8C 8F E8 E6 93 DA F5 AA 50 45 A8 C9 20 D1 9D 67 35 9B 7B D3 46 D5 CA C0 FC 85",
  "cipher suites"       : "[TLS_AES_128_GCM_SHA256(0x1301), TLS_AES_256_GCM_SHA384(0x1302), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B),

[...]

TLS_DHE_DSS_WITH_AES_128_CBC_SHA(0x0032), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",
  "compression methods" : "00",
  "extensions"          : [
    "status_request (5)": {
      "certificate status type": ocsp
      "OCSP status request": {
        "responder_id": <empty>
        "request extensions": {
          <empty>
        }
      }
    },
    "supported_groups (10)": {
      "versions": [secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
    },
    "ec_point_formats (11)": {
      "formats": [uncompressed]
    },
    "signature_algorithms (13)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },
    "signature_algorithms_cert (50)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.704 CEST|SSLExtensions.java:189|Consumed extension: supported_versions
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.705 CEST|ClientHello.java:839|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.705 CEST|SSLExtensions.java:189|Consumed extension: psk_key_exchange_modes
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|PreSharedKeyExtension.java:805|Handling pre_shared_key absence.
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|SSLExtensions.java:170|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|SSLExtensions.java:170|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|SSLExtensions.java:189|Consumed extension: status_request
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:189|Consumed extension: supported_groups
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:160|Ignore unsupported extension: ec_point_formats
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:189|Consumed extension: signature_algorithms
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:189|Consumed extension: signature_algorithms_cert

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.739 CEST|ServerHello.java:576|Produced ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "FD FD 39 0C 3A D8 F0 E8 38 F8 08 D8 19 94 7A FA 4A 68 71 F8 4C 32 EB 7A D0 53 96 E5 9F E8 0A 3B",
  "session id"          : "5B 9A 18 25 31 65 8C 8F E8 E6 93 DA F5 AA 50 45 A8 C9 20 D1 9D 67 35 9B 7B D3 46 D5 CA C0 FC 85",
  "cipher suite"        : "TLS_AES_128_GCM_SHA256(0x1301)",
  "compression methods" : "00",
  "extensions"          : [
    "supported_versions (43)": {
      "selected version": [TLSv1.3]
    },
    "key_share (51)": {
      "server_share": {

[...]

javax.net.ssl|ALL|01|main|2019-08-22 12:23:59.785 CEST|X509Authentication.java:243|No X.509 cert selected for EC
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.785 CEST|CertificateMessage.java:1055|Unavailable authentication scheme: ecdsa_sha1
javax.net.ssl|ALL|01|main|2019-08-22 12:23:59.785 CEST|X509Authentication.java:243|No X.509 cert selected for RSA
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.785 CEST|CertificateMessage.java:1055|Unavailable authentication scheme: rsa_pkcs1_sha1
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.785 CEST|CertificateMessage.java:1065|No available authentication scheme
javax.net.ssl|ERROR|01|main|2019-08-22 12:23:59.788 CEST|TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): No available authentication scheme (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: No available authentication scheme
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)

[...]

Answer

Corentor picture Corentor · Aug 22, 2019

Here is the solution: one needs the -keyalg flag with keytool to generate certificates, otherwise, the key will be ciphered with the old default DSA, that is not allowed anymore with TLS1.3. With RSA it works.

Conclusion:

  • keytool ciphers by default with DSA for backwards compatibility, unless -keyalg is provided
  • Since Java 11, TLS1.3 is the new default encryption scheme for SSL sockets in JSSE, when it can be negotiated.
  • DSA is not supported anymore in TLS1.3

It is therefore recommended to always generate certificates with explicit -keyalg RSA (or any other supported TLS1.3 algorithm) to avoid surprises.

Edit 19 nov 2020: the issue of the keytool's default algorithm was discussed on Red Hat's Bugzilla a long time ago ( https://bugzilla.redhat.com/show_bug.cgi?id=1582253 ). It seems that RSA is the keytool's new default algorithm in OpenJDK 14, but it is not guaranteed for other JDK implementations to my knowledge (unless the JDK specification says anything about it).

Related questions

Received fatal alert: handshake_failure through SSLHandshakeException

I have a problem with authorized SSL connection. I have created Struts Action that connects to external server with Client Authorized SSL certificate. In my Action I am trying to send some data to bank server but without any luck, …

Read More
Java client certificates over HTTPS/SSL

I am using Java 6 and am trying to create an HttpsURLConnection against a remote server, using a client certificate. The server is using an selfsigned root certificate, and requires that a password-protected client certificate is presented. I've added the server …

Read More
Java SSLHandshakeException "no cipher suites in common"

I'm using an SSLServerSocket to accept client connections on my openSUSE server, but none of them can connect. I always get an SSLHandshakeException saying no cipher suites in common. I've activated all of the possible suites, enabled multiple protocols, tried …

Read More