RC4 doesn't work correctly with openssl command?

Question

I need to encode the result of a execution with RC4. Before to do the bash script, I'm testing how to crypt the data.

I'm using the next command:

echo -ne "test" | openssl rc4 -k test -nosalt -e -nopad | xxd

And the output is:

0000000: bdb1 7f03                                ....

now, If I try to do the same with this online RC4 encoder http://www.fyneworks.com/encryption/rc4-encryption/index.asp the output is: DA EA 54 65

Different output, with the same data and same key?? Data: "test" key: "test"

Also I checked with a small program that I have coded in C, and the output is the same that the online encoder... so, the question is, what I'm doing wrong with the command openssl??

Thanks!

Answer 1

RC4 has variable-length keys, and OpenSSL's enc utility forces you to pick a key size. These other implementations you're testing against make no such restriction, so your keys don't match.

The documentation for the enc utility describes the allowed key sizes for the cipher:

    rc4                128 bit RC4
    rc4-64             64 bit RC4
    rc4-40             40 bit RC4

So RC4 works only on a 128-bit (16-byte) key. Also, the -k option means to derive a key from the given passphrase. It does this internally using the EVP_BytesToKey function, which implements a Key Derivation Function (KDF).

Anyway, long story short, your RC4 implementations aren't using the same key. Use the -p option to have OpenSSL print the actual key it is using:

$ echo -ne "test" | openssl rc4 -k test -nosalt -e -nopad -p
key=098F6BCD4621D373CADE4E832627B4F6

Further, since it's expecting 16-byte keys, it'll zero-pad shorter keys even if you specify a short key with the -K (uppercase K) option. You can use xxd to find the ascii hex values of "test" and -p again to see OpenSSL's key:

$ echo -ne "test" | xxd
0000000: 7465 7374                                test
$ echo -ne "test" | openssl rc4 -K 74657374 -nosalt -e -nopad -p
key=74657374000000000000000000000000

So you must match key lengths and specify a hex-value key with the -K option and you'll see the RC4 implementations are equivalent. E.g., here I use RC-40 to restrict the key length to 5 bytes and use the 5-byte key "tests", or 74 65 73 74 73.

$ echo -ne "test" | openssl rc4-40 -K 7465737473 -nosalt -e -nopad | xxd
0000000: dd9b 5cb9   

You'll find that your web implementation gets the same result when given the key "tests".