I need to implemented security for client-server communication. I have implemented the following hybrid cryptosystem
To encrypt a message addressed to Alice in a hybrid cryptosystem, Bob does the following:
To decrypt this hybrid ciphertext, Alice does the following:
I am using RSA For a public-key cryptosystem, and AES for symmetric-key cryptosystem. Every thing works fine, but I am not sure how to handle AES initialization vector. Currently, I am concatenating the AES key and initialization vector encrypting it with the public key and sending that to server.
I just wanted to get some opinions about this approach. How this problem is solved by other communication protocols SSL etc.
Thanks.
You don't encrypt the IV. Bundle it with the encrypted key and send it (in the clear) to the recipient.
Standards for this do exist. This scheme is called "KeyTransRecipientInfo" in CMS (upon which S/MIME is based), and PGP offers a similar mode. TLS also includes the initialization vector as a parameter in the key encryption algorithm identifier, using the same ASN.1 syntax as CMS. A robust, open-source library to perform this operation is available for many, many platforms.
At the very least, studying the CMS specification might help avoid some of the many pitfalls in a home-brew implementation. See §6.1 and §6.2.1 of RFC 3369.