openssl- decrypting a base64 string with a key and IV

Question

I'm trying to decrypt a base64 string which has been encrypted with aes256 in openssl. I was given the session key and IV, which were encrypted with my key. I converted them to hexadecimal so that I can use the following openssl command:

openssl enc -d -aes256 -iv iv.hex -K sessionkey.hex -in message.b64 -out message.txt

I get the error saying the IV is a non-hex value. I started out with IV and session key in base64, which was encrypted with my key. So I did the following:

//convert base64 to binary

openssl base64 -d -in iv.b64 -out iv.bin
openssl base64 -d -in sessionkey.b64 -out sessionkey.bin

//decrypt using my private key

openssl rsautl -decrypt -inkey mykey.pem -in sessionkey.bin -out sessionkey_out.bin
openssl rsautl -decrypt -inkey mykey.pem -in iv.bin -out iv_out.bin

//convert to hex using the following C code:

include

main()
{
 int c;
        while ((c=getchar())!=EOF)
                printf("%02X",c);
}

//use the hex IV and key to decrypt the message

openssl enc -d -aes256 -iv iv.hex -K sessionkey.hex -in message.b64 -out message.txt

I get the error at the last step, saying the IV is non-hex. Any ideas?

Answer 1

The problem is that you are specifying files for your IV and key while openssl expects the values to be provided as hex on the command line.

For example, I was attempting to decrypt an M3U8 stream (HLS) and the key 16 byte file contained non-printable characters that I couldn't input via the keyboard at run time (omitting -K takes the key from the keyboard).

-rw-r--r--@ 1 Mufasa  staff       16 Apr 17 10:45 sequence146094144.key
-rw-r--r--  1 Mufasa  staff  3272528 Apr 17 10:48 sequence146094161.ts

So I converted the key file to hex:

hexdump -e '16/1 "%02x" "\n"' sequence146094144.key 
8d2aeccbefb0955ec9a75f2f051faa6e

And my IV was provided in hex already, so I just removed the 0x:

IV=0x00000000000000000000000008B53851

Resulting with this command that successfully decrypted the .ts file:

openssl aes-128-cbc -d -in sequence146094161.ts -out output.ts -iv 00000000000000000000000008B53851 -K 8d2aeccbefb0955ec9a75f2f051faa6e

Checking the output with ffprobe:

ffprobe output.ts 
ffprobe version 2.8.git Copyright (c) 2007-2016 the FFmpeg developers
  built with Apple LLVM version 7.0.2 (clang-700.1.81)

...

Input #0, mpegts, from 'output.ts':
  Duration: 00:00:10.04, start: 8414.107644, bitrate: 2607 kb/s
  Program 1 
    Stream #0:0[0x1e1]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 960x540 [SAR 1:1 DAR 16:9], Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x1e2](und): Audio: aac (HE-AAC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 60 kb/s
    Stream #0:2[0x100]: Unknown: none ([134][0][0][0] / 0x0086)

And my file played in VLC. In your case, if the iv.bin file you generated is a plain text hex string, use the hex value on the command line as is without further conversion. If it appears to be anything else but hex convert it to HEX direct from the file. Same logic goes the sessionkey.bin file you generated.