Windows CryptoAPI: CryptSignHash with CALG_SHA_256 and private key from MY keystore

Question

I am trying to generate digital signatures on Windows (from XP SP3, but currently testing with Windows 7) with CryptoAPI that will be compatible with the following openssl commands:

openssl dgst -sha256 -sign <parameters> (for signing)
openssl dgst -sha256 -verify <parameters> (for validation)

I want to use a private key from the Windows "MY" keystore for signing.

I managed to sign files using the SHA1 digest algorithm by using the following CryptoAPI functions (omitting parameters for brevity):

CertOpenStore
CertFindCertificateInStore
CryptAcquireCertificatePrivateKey
CryptCreateHash (with CALG_SHA1)
CryptHashData
CryptSignHash

The generated signature is compatible with "openssl dgst -sha1 -verify" (once the byte order is reversed).

My problem is: when I try to use CALG_SHA_256 with CryptCreateHash, it fails with error 80090008 (NTE_BAD_ALGID). By googling around, I found that I needed to use a specific provider (PROV_RSA_AES) instead of the default one. Since I would have a provider handle, I would also need to replace CryptAcquireCertificatePrivateKey by CryptGetUserKey. So I modified my program to look like:

CryptAcquireContext (with PROV_RSA_AES)
CertOpenStore
CertFindCertificateInStore
CryptGetUserKey
CryptCreateHash (with CALG_SHA256)
CryptHashData
CryptSignHash

Unfortunately, this didn't work as expected: CryptGetUserKey failed with error 8009000D (NTE_NO_KEY). If I remove the CryptGetUserKey call, the program runs until CryptSignHash, which fails with error 80090016 (NTE_BAD_KEYSET). I know the keyset does exist and works fine, since I was able to use it to sign the SHA1 digest.

I tried acquiring the context again with information from the certificate context I got from CertFindCertificateInStore: the best I could do was a successful CryptGetUserKey call, but CryptSignHash would always fail with the same error.

The private key I am trying to use is 2048 bits long, but I don't expect it to be a problem since it works with the SHA1 digest. I am at a loss, so any suggestion would be very welcome!

Answer 1

80090008 is caused because of Base provider does not support SHA256, SHA384 and SHA512, you got to use CryptAcquireContext(hProv, NULL, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, 0);