Decrypting an OpenSSL PEM Encoded RSA private key with Java?

Question

I have an encrypted private key and I know the password.

I need to decrypt it using a Java library.

I'd prefer not to use BouncyCastle though, unless there is no other option. Based on previous experience, there is too much change and not enough documentation.

The private key is in this form:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,56F3A98D9CFFA77A

X5h7SUDStF1tL16lRM+AfZb1UBDQ0D1YbQ6vmIlXiK....
.....
/KK5CZmIGw==
-----END RSA PRIVATE KEY-----

I believe the key data is Base64 encoded since I see \r\n after 64 characters.

I tried the following to decrypt the key:

import java.security.Key;
import java.security.KeyFactory;
import java.security.PrivateKey;
import java.security.spec.PKCS8EncodedKeySpec;
import javax.crypto.EncryptedPrivateKeyInfo;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;

public String decrypt(String keyDataStr, String passwordStr){
  // This key data start from "X5... to ==" 
  char [] password=passwordStr.toCharArray();
  byte [] keyDataBytes=com.sun.jersey.core.util.Base64.decode(keyDataStr);

  PBEKeySpec pbeSpec = new PBEKeySpec(password);
  EncryptedPrivateKeyInfo pkinfo = new EncryptedPrivateKeyInfo(keyDataBytes);
  SecretKeyFactory skf = SecretKeyFactory.getInstance(pkinfo.getAlgName());
  Key secret = skf.generateSecret(pbeSpec);
  PKCS8EncodedKeySpec keySpec = pkinfo.getKeySpec(secret);
  KeyFactory kf = KeyFactory.getInstance("RSA");
  PrivateKey pk=kf.generatePrivate(keySpec);
  return pk.toString();
}

I get this Exception

java.io.IOException: DerInputStream.getLength(): lengthTag=50, too big.
    at sun.security.util.DerInputStream.getLength(DerInputStream.java:561)
    at sun.security.util.DerValue.init(DerValue.java:365)
    at sun.security.util.DerValue.<init>(DerValue.java:294)
    at javax.crypto.EncryptedPrivateKeyInfo.<init> (EncryptedPrivateKeyInfo.java:84)

Am I passing the right parameter to EncryptedPrivateKeyInfo constructor?

How can I make this work?

I tried what Ericsonn suggested, with one small change since I am working Java 7, I could not use Base64.getMimeCoder() instead I used Base64.decode and I am getting this error I am getting an error like this Input length must be multiple of 8 when decrypting with padded cipher at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:750)

static RSAPrivateKey decrypt(String keyDataStr, String ivHex, String password)
            throws GeneralSecurityException, UnsupportedEncodingException
          {
            byte[] pw = password.getBytes(StandardCharsets.UTF_8);
            byte[] iv = h2b(ivHex);
            SecretKey secret = opensslKDF(pw, iv);
            Cipher cipher = Cipher.getInstance("DESede/CBC/NoPadding");
            cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
            byte [] keyBytes=Base64.decode(keyDataStr.getBytes("UTF-8"));
            byte[] pkcs1 = cipher.doFinal(keyBytes);
            /* See note for definition of "decodeRSAPrivatePKCS1" */
            RSAPrivateCrtKeySpec spec = decodeRSAPrivatePKCS1(pkcs1);
            KeyFactory rsa = KeyFactory.getInstance("RSA");
            return (RSAPrivateKey) rsa.generatePrivate(spec);
          }

          private static SecretKey opensslKDF(byte[] pw, byte[] iv)
            throws NoSuchAlgorithmException
          {
            MessageDigest md5 = MessageDigest.getInstance("MD5");
            md5.update(pw);
            md5.update(iv);
            byte[] d0 = md5.digest();
            md5.update(d0);
            md5.update(pw);
            md5.update(iv);
            byte[] d1 = md5.digest();
            byte[] key = new byte[24];
            System.arraycopy(d0, 0, key, 0, 16);
            System.arraycopy(d1, 0, key, 16, 8);
            return new SecretKeySpec(key, "DESede");
          }

          private static byte[] h2b(CharSequence s)
          {
            int len = s.length();
            byte[] b = new byte[len / 2];
            for (int src = 0, dst = 0; src < len; ++dst) {
              int hi = Character.digit(s.charAt(src++), 16);
              int lo = Character.digit(s.charAt(src++), 16);
              b[dst] = (byte) (hi << 4 | lo);
            }
            return b;
          }
          static RSAPrivateCrtKeySpec decodeRSAPrivatePKCS1(byte[] encoded)
          {
            ByteBuffer input = ByteBuffer.wrap(encoded);
            if (der(input, 0x30) != input.remaining())
              throw new IllegalArgumentException("Excess data");
            if (!BigInteger.ZERO.equals(derint(input)))
              throw new IllegalArgumentException("Unsupported version");
            BigInteger n = derint(input);
            BigInteger e = derint(input);
            BigInteger d = derint(input);
            BigInteger p = derint(input);
            BigInteger q = derint(input);
            BigInteger ep = derint(input);
            BigInteger eq = derint(input);
            BigInteger c = derint(input);
            return new RSAPrivateCrtKeySpec(n, e, d, p, q, ep, eq, c);
          }

          private static BigInteger derint(ByteBuffer input)
          {
            byte[] value = new byte[der(input, 0x02)];
            input.get(value);
            return new BigInteger(+1, value);
          }


          private static int der(ByteBuffer input, int exp)
          {
            int tag = input.get() & 0xFF;
            if (tag != exp)
              throw new IllegalArgumentException("Unexpected tag");
            int n = input.get() & 0xFF;
            if (n < 128)
              return n;
            n &= 0x7F;
            if ((n < 1) || (n > 2))
              throw new IllegalArgumentException("Invalid length");
            int len = 0;
            while (n-- > 0) {
              len <<= 8;
              len |= input.get() & 0xFF;
            }
            return len;
          }

1640 is keyDataStr.length() and 1228 is keyBytes.length

Answer 1

You need to use a non-standard, OpenSSL method for deriving the decryption key. Then use that to decrypt the PKCS-#1–encoded key—what you are working with is not a PKCS #8 envelope. You'll also need the IV from the header as input to these processes.

It looks something like this:

  static RSAPrivateKey decrypt(String keyDataStr, String ivHex, String password)
    throws GeneralSecurityException
  {
    byte[] pw = password.getBytes(StandardCharsets.UTF_8);
    byte[] iv = h2b(ivHex);
    SecretKey secret = opensslKDF(pw, iv);
    Cipher cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
    cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
    byte[] pkcs1 = cipher.doFinal(Base64.getMimeDecoder().decode(keyDataStr));
    /* See note for definition of "decodeRSAPrivatePKCS1" */
    RSAPrivateCrtKeySpec spec = decodeRSAPrivatePKCS1(pkcs1);
    KeyFactory rsa = KeyFactory.getInstance("RSA");
    return (RSAPrivateKey) rsa.generatePrivate(spec);
  }

  private static SecretKey opensslKDF(byte[] pw, byte[] iv)
    throws NoSuchAlgorithmException
  {
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    md5.update(pw);
    md5.update(iv);
    byte[] d0 = md5.digest();
    md5.update(d0);
    md5.update(pw);
    md5.update(iv);
    byte[] d1 = md5.digest();
    byte[] key = new byte[24];
    System.arraycopy(d0, 0, key, 0, 16);
    System.arraycopy(d1, 0, key, 16, 8);
    return new SecretKeySpec(key, "DESede");
  }

  private static byte[] h2b(CharSequence s)
  {
    int len = s.length();
    byte[] b = new byte[len / 2];
    for (int src = 0, dst = 0; src < len; ++dst) {
      int hi = Character.digit(s.charAt(src++), 16);
      int lo = Character.digit(s.charAt(src++), 16);
      b[dst] = (byte) (hi << 4 | lo);
    }
    return b;
  }

This is already a lot of code, so I will link to another answer for the definition of the decodeRSAPrivatePKCS1() method.