Hi all,

Some time ago a customer of mine was getting the following CryptographicException when calling EncryptedXml.DecryptDocument method to decrypt an XmlDocument in their .NET app:

 [System.Security.Cryptography.CryptographicException] = {"The data to be decrypted exceeds the maximum for this modulus of 0 bytes"}
_HResult = -2146233296
_HResult = 0x80131430
No Inner Exception

This was only happening with a smart card certificate associated to a specific third-party CSP (non-MS).

In order to troubleshoot this, I used my CryptoAPI Tracer script to get some traces while reproducing the issue. Thanks to my script I could see these calls that .NET is making to the third-party CSP behind the scenes (I will omit key container and CSP names as they are not needed to understand the issue):

>>>>>>>>>>>>>>>>>>>>>>

CryptAcquireContextA (0x570)

IN
pszContainer
001bf1a0 "Cert Container Name"

pszProvider
001b4698 "Third-party CSP"

dwProvType
PROV_RSA_FULL

dwFlags
0

OUT
hProv
0x1bea28

RESULT
CryptAcquireContextA (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetUserKey (0x570)

IN
hProv
0x1bea28

dwKeySpec
AT_KEYEXCHANGE

OUT
hUserKey
0x1bf1a0

RESULT
CryptGetUserKey (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_ALGID

pbData
NULL

dwDataLen
0

dwFlags
0

OUT
dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_ALGID

pbData
0x1bd228

dwDataLen
4

dwFlags
0

OUT
bData
001bd228 0000a400

dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_KEYLEN

pbData
NULL

dwDataLen
0

dwFlags
0

OUT
dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_KEYLEN

pbData
0x1bd228

dwDataLen
4

dwFlags
0

OUT
bData
001bd228 00000000

dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

These calls mean the following: before .NET tries to decrypt the XML, it asks the CSP for some information on the key it will use to decrypt, like its length (CryptGetKeyParam API with KP_KEYLEN flag). For some unknown reason the third-party CSP is returning a length of 0 for the key (pbData points to 0), which is invalid, thus .NET won't continue with the decryption and will raise the exception we've seen. Why is the CSP returning an invalid length? Only the CSP developers can answer that question. 

In my customer's particular case, the CSP developers quickly localized and fixed the issue.

I hope this helps.

Regards,

 

Alex (Alejandro Campos Magencio)