Getting Could not generate DH keypair Exception

Question

I am getting this exception when vCenter(5.5) try to download zip file from a https server. VCenter server has JRE 1.6.0.31. I am aware of bug related to same issue in Java 1.6 and 1.7. This exception is caused by java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive).

I can change the SSL certificate from the https server. I tried to solve this issue by generating new certificates with OpenSSL (openssl req -new -x509 -newkey rsa:1024 -days 365 -out server.crt -extensions usr_cert ) and other options as well. I looked into sun.security.ssl.ClientHandshaker code, There is switch case which executes following lines of code and generate exception.

case K_DHE_DSS:
case K_DHE_RSA:
    this.serverKeyExchange(new DH_ServerKeyExchange(input, serverKey,
    clnt_random.random_bytes, svr_random.random_bytes,messageLen));

is there any way to generate SSL certificate which will avoid this code execution (may be, SSL certificate without DH keypair) ?

Thanks in advance!

Answer 1

The keyexchange is DHE_RSA which means Diffie Hellman Ephemeral authenticated by RSA. The DHE key, and even the DHE parameters, are not in the certificate, which is longterm (FSVO long) not ephemeral. The key in the certificate is RSA, and Java has no problem dealing with RSA at secure sizes, which today is more than 1024 and conventionally 2048.

1. The direct fix is to use old-Java-compatible weak DH(E) parameters. There are probably a dozen or more SSL/TLS server programs that use OpenSSL, or otherwise use PEM format files for the cert and (RSA) privatekey. (Lots of software uses or can handle PEM for cert, but mostly OpenSSL does so for privatekey.) Many of these server programs allow configuring the DH(E) parameters, but all that I've seen do so differently, and you don't identify yours.

2. A workaround is to avoid DHE_anything and use plain-RSA keyexchange. All SSL/TLS implementations including OpenSSL implement this. However, most applications and/or middlewares no longer prefer it and some don't even allow it, because it doesn't provide Forward Secrecy. How you control the ciphersuites, and thus keyexchange, again depends on the unidentified server program.

3. A better workaround is to enable ECDHE (and specifically ECDHE_RSA, since your cert/key is RSA) if you can.

   3A. Java6 JSSE implements ECDHE protocols, but enables them only if there is a JCE "provider" for ECC primitives in your JRE -- and Oracle/Sun JRE6 as delivered (and TTBOMK the OpenJDK one also) does not have an ECC provider. To add ECC provider to JRE6

   • download bcprov-jdk15on-$version.jar from http://www.bouncycastle.org/latest_releases.html and put it in your JRE/lib/ext

   • edit JRE/lib/security/java.security to add to the list of providers a line like security.provider.N=org.bouncycastle.jce.provider.BouncyCastleProvider where N is the next available number.

   JRE7 does include ECC provider, and supports ECDHE out-of-the-box, if that is an option.

   3B. If your server uses OpenSSL 1.0.0 or higher (except for certain older RedHat builds) it implements ECDHE, but it can be used only if (1) the ciphersuites are enabled, which is true by default but could be disabled by the server program or its configuration, and (2) the server program sets tmp_ecdh parameters (or in 1.0.2 enables auto-setting). Both of these depend on the unidentified server program, and if the server program doesn't use OpenSSL the answer may be very different.

If you identify your server program and the SSL/TLS-related parts of its configuration, I (or perhaps someone else) can be more specific.

EDIT server is nginx:

(1) nginx should be able to use DHE params Java(6,7) can handle, see http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam and create the file with

 openssl dhparam 1024 >mydhfile

(or if you really want, a smaller DSA-1 size = 512 to 1024 and a multiple of 64, but you don't need and I don't recommend smaller than 1024).

(2) alernatively, to disable DHE, don't add it but do add at least some kRSA (or just RSA) in the ssl_ciphers string. Change the string in your comment at least to

 EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH:kRSA:!RC4:!aNU‌​LL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

or because unmodified JRE6 won't offer ECDHE (or TLSv1.2) and we haven't added any PSK or SRP or DSS or aNULL that need to be deleted, and you distrust 3DES (why?) and RC4 and Java doesn't implement SEED or Camellia, you could use much more simply just

 AES128-SHA:AES256-SHA 

(These are actually TLS_RSA_WITH_AES*_CBC_SHA but for hysterical raisins the OpenSSL name omits the RSA, and the CBC.)

Tomcat by default uses Java (JSSE) to handle HTTPS connections, but depending on the packaging/install often can use APR also called "Tomcat native" or just "native" which is actually OpenSSL. If Tomcat/JSSE is run on JRE6 or 7, it uses DHE size 768 which JRE6,7 client can handle; if run on JRE8 it defaults to size 1024 which JRE6,7 client can handle. I don't know what Tomcat/APR uses (and can't easily test) but it could well be 1024 or less. If you want to find out and have Tomcat/APR running and openssl 1.0.2 available, use openssl s_client -connect host:port -tls1 -cipher EDH+AES; when it connects enter Q, return; and look about 20 lines up for "Server Temp Key".