The RSACryptoServiceProvider class provides two methods, Encrypt and Decrypt which seem to be the managed counterparts to CAPI's CryptEncrypt and CryptDecrypt functions. However, if you try to encrypt using CAPI and decrypt using managed code, you'll end up with a CryptographicException saying "bad data". What's going on here?
Since v1.0 of the runtime, the output from RSACryptoServiceProvider::Encrypt has been reversed in memory. Input to RSACryptoServiceProvider::Decrypt is also swapped before it's processed. Since the two reversals undo each other, using the RSA class to round trip data works perfectly.
However, if you use CAPI for one of the steps, the memory reverse won't undo itself, and the output won't contain valid padding -- in this case leading to a bad data exception. The solution is to introduce the reversal yourself. To interop, you'd need to either:
Which gets the output from CAPI setup so that Decrypt's own memory reverse puts the bytes in a state that they can be decrypted. Going the other direction would of course be:
The Reverse in this direction undoes the reversal that Encrypt does. Reversing the memory can be done with a call to Array.Reverse();
Why not change the behavior of Encrypt and Decrypt now? Doing that would prevent people using the same code on different versions of the runtime from being able to interop with each other. Even worse, if you wrote some data out to a file while your app was running on a version of the runtime that had the old behavior, and before your app started up again the computer admin uninstalled the old version of the runtime and installed a new version which didn't have this behavior, when your app started back up it wouldn't be able to read its file anymore.
Instead, when writing code that needs to interact with CryptEncrypt and CryptDecrypt, you need to remember to apply this transform to your data in order for your code to work properly.
Updated 11:19: Added a reference to Array.Reverse