Hi all,
Some time ago a customer of mine was getting the following CryptographicException when calling EncryptedXml.DecryptDocument method to decrypt an XmlDocument in their .NET app:
[System.Security.Cryptography.CryptographicException] = {"The data to be decrypted exceeds the maximum for this modulus of 0 bytes"}_HResult = -2146233296_HResult = 0x80131430No Inner Exception
This was only happening with a smart card certificate associated to a specific third-party CSP (non-MS).
In order to troubleshoot this, I used my CryptoAPI Tracer script to get some traces while reproducing the issue. Thanks to my script I could see these calls that .NET is making to the third-party CSP behind the scenes (I will omit key container and CSP names as they are not needed to understand the issue):
>>>>>>>>>>>>>>>>>>>>>> CryptAcquireContextA (0x570) IN pszContainer 001bf1a0 "Cert Container Name" pszProvider 001b4698 "Third-party CSP" dwProvType PROV_RSA_FULL dwFlags 0 OUT hProv 0x1bea28 RESULT CryptAcquireContextA (0x570) SUCCEEDED <<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> CryptGetUserKey (0x570) IN hProv 0x1bea28 dwKeySpec AT_KEYEXCHANGE OUT hUserKey 0x1bf1a0 RESULT CryptGetUserKey (0x570) SUCCEEDED <<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> CryptGetKeyParam (0x570) IN hKey 0x1bf1a0 dwParam KP_ALGID pbData NULL dwDataLen 0 dwFlags 0 OUT dwDataLen 4 RESULT CryptGetKeyParam (0x570) SUCCEEDED <<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> CryptGetKeyParam (0x570) IN hKey 0x1bf1a0 dwParam KP_ALGID pbData 0x1bd228 dwDataLen 4 dwFlags 0 OUT bData 001bd228 0000a400 dwDataLen 4 RESULT CryptGetKeyParam (0x570) SUCCEEDED <<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> CryptGetKeyParam (0x570) IN hKey 0x1bf1a0 dwParam KP_KEYLEN pbData NULL dwDataLen 0 dwFlags 0 OUT dwDataLen 4 RESULT CryptGetKeyParam (0x570) SUCCEEDED <<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>> CryptGetKeyParam (0x570) IN hKey 0x1bf1a0 dwParam KP_KEYLEN pbData 0x1bd228 dwDataLen 4 dwFlags 0 OUT bData 001bd228 00000000 dwDataLen 4 RESULT CryptGetKeyParam (0x570) SUCCEEDED <<<<<<<<<<<<<<<<<<<<<<
These calls mean the following: before .NET tries to decrypt the XML, it asks the CSP for some information on the key it will use to decrypt, like its length (CryptGetKeyParam API with KP_KEYLEN flag). For some unknown reason the third-party CSP is returning a length of 0 for the key (pbData points to 0), which is invalid, thus .NET won't continue with the decryption and will raise the exception we've seen. Why is the CSP returning an invalid length? Only the CSP developers can answer that question.
In my customer's particular case, the CSP developers quickly localized and fixed the issue.
I hope this helps.
Regards,
Alex (Alejandro Campos Magencio)