Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?

Question

I want to generate a RSA-SHA256 signature in Java, but I can't get it to produce the same signature as with OpenSSL on the console.

This is what I did with OpenSSL (following this tutorial):

Generate key pair:

openssl genrsa -out private.pem 1024

Extract public key:

openssl rsa -in private.pem -out public.pem -outform PEM -pubout

Create hash of data:

echo 'data to sign' > data.txt
openssl dgst -sha256 < data.txt > hash

The generated hash file starts with (stdin)= what I removed by hand (first forgot to mention it, thanks mata).

Sign hash:

openssl rsautl -sign -inkey private.pem -keyform PEM -in hash  > signature

To reproduce the results in Java I first converted the private key from PEM to DER:

openssl pkcs8 -topk8 -inform PEM -outform DER -in private.pem -nocrypt > private.der

Now I wrote this Java class to generate the same signature:

public class RSATest {

    public static void main(String[] args) throws IOException,
            NoSuchAlgorithmException, InvalidKeySpecException,
            InvalidKeyException, SignatureException {

        byte[] encodedPrivateKey = readFile("private.der");
        byte[] content = readFile("data.txt");

        KeyFactory keyFactory = KeyFactory.getInstance("RSA");
        PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encodedPrivateKey);
        RSAPrivateKey privateKey = (RSAPrivateKey) keyFactory
                .generatePrivate(keySpec);

        Signature signature = Signature.getInstance("SHA256withRSA");
        signature.initSign(privateKey);
        signature.update(content);
        byte[] signatureBytes = signature.sign();

        FileOutputStream fos = new FileOutputStream("signature-java");
        fos.write(signatureBytes);
        fos.close();
    }

    private static byte[] readFile(String filename) throws IOException {
        File file = new File(filename);
        BufferedInputStream bis = new BufferedInputStream(new FileInputStream(
                file));
        byte[] bytes = new byte[(int) file.length()];
        bis.read(bytes);
        bis.close();
        return bytes;
    }

}

Unfortunately the results are not the same, so I think I must have done something wrong, but I can't figure out what. Can someone of you help me to find the bug?

Answer 1

openssl dgst -sha256 < data.txt

produces something like:

(stdin)= b39eaeb437e33087132f01c2abc60c6a16904ee3771cd7b0d622d01061b40729

notice the (stdin)='? you don't want that to be part of your hash, if you need to create a digest, use the -binary option.

try using this to sign your data:

openssl sha -sha256 -sign private.pem < data.txt

This does everything you need.

---

edit - a little more explanations:

let's create a digest and show it

$ openssl dgst -sha256 -binary < data.txt > digest
$ hd digest
00000000  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...|
00000010  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|

now we take this digest and sign int using rsautl:

$ openssl rsautl -sign -inkey private.pem < digest > sign1
$ hd sign1
00000000  1b 7a cf a4 8d 41 8e 04  ed 3a bb ba 86 f1 f8 e0  |.z...A...:......|
00000010  df f7 47 3e d7 a7 f4 90  7a 05 f8 7f 45 e5 29 e7  |..G>....z...E.).|
00000020  9f f4 2c 91 97 2f e7 26  69 9f 6a 07 a3 48 1b 85  |..,../.&i.j..H..|
00000030  2e f8 ee 44 4d 25 9f ae  05 95 81 c9 e3 07 68 ad  |...DM%........h.|

now let's sign the same file using dgst directly:

$ openssl dgst -sha256 -sign private.pem < data.txt > sign2
$ hd sign2
00000000  15 c2 94 87 eb e6 cb 45  c8 63 0c 97 60 d3 07 f3  |.......E.c..`...|
00000010  dc 65 32 ad 44 1c c2 2a  7f a3 e1 fc dd 84 27 8c  |.e2.D..*......'.|
00000020  77 a6 97 2b 33 6b c6 d7  7d e1 1d 39 5c 48 b6 48  |w..+3k..}..9\H.H|
00000030  cb 18 be bf 6a 66 90 d3  88 89 52 6c dd d1 b9 99  |....jf....Rl....|

So what's different here? To see that, we can verify the signature and show the raw output. Both files do contain the digest, but the metadata and padding is different:

$ openssl rsautl -raw -verify -inkey private.pem < sign1 | hd
00000000  00 01 ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff 00  |................|
00000020  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...|
00000030  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|

$ openssl rsautl -raw -verify -inkey private.pem < sign2 | hd
00000000  00 01 ff ff ff ff ff ff  ff ff ff ff 00 30 31 30  |.............010|
00000010  0d 06 09 60 86 48 01 65  03 04 02 01 05 00 04 20  |...`.H.e....... |
00000020  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...|
00000030  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|

To see this more clearly, we can try to use the -asn1parse flag, which won't work for the first signature, but for the second it shows the correct structure of the signature:

$ openssl rsautl -verify -inkey private.pem -asn1parse < sign1
Error in encoding
139931349546656:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:142:

$ openssl rsautl -verify -inkey private.pem -asn1parse < sign2
    0:d=0  hl=2 l=  49 cons: SEQUENCE          
    2:d=1  hl=2 l=  13 cons:  SEQUENCE          
    4:d=2  hl=2 l=   9 prim:   OBJECT            :sha256
   15:d=2  hl=2 l=   0 prim:   NULL              
   17:d=1  hl=2 l=  32 prim:  OCTET STRING      
      0000 - 26 3b 0a a1 2e b9 32 db-b8 dc d3 6f 37 94 0b 05   &;....2....o7...
      0010 - 71 9c ba 79 46 34 28 9f-5c 5b 98 9a 64 61 c9 ec   q..yF4(.\[..da..