Random things about .NET, CLR, C++, and work
I'm seeing many people reporting that they are seeing strange P/invoke issues when they moved their code to VS 2012. Typically, they have P/invokes like this:
[DllImport("Win32Project2.dll", PreserveSig = true, CharSet = CharSet.Unicode)]
static extern int MyPInvoke(out string ret);
If you attach a native debugger (or enable native debugging), with the right symbols (public symbol would be just fine), you would see that it is actually crashing inside combase.dll!CoTaskMemFree:
> ntdll.dll!_RtlReportCriticalFailure@8() + 0x33 bytes ntdll.dll!_RtlpReportHeapFailure@4() + 0x21 bytes ntdll.dll!_RtlpLogHeapFailure@24() + 0xa2 bytes ntdll.dll!_RtlFreeHeap@12() + 0x2677b bytes combase.dll!CoTaskMemFree(void * pv) Line 475 C++
Problem like this are usually caused by a mismatch between the managed declaration and the native implementation. The usual suspect here is the 'out string' signature. In .NET world, having a out string in P/invoke means:
So, if MyPInvoke pass a LPWSTR that is not allocated from CoTaskMemAlloc, say from HeapAlloc, new, or a string literal, you are potentially asking CLR to do CoTaskMemFree on your pointer, which would result in undefined behavior.
Basically, this means the P/invoke was already broken before.
But why it didn't crash before and suddenly starts to crash now after upgrading?
This is exactly the kind of thing you could expect with undefined behavior, because that could change anytime. It turns out in Vista, Windows guys decided to make CoTaskMemFree a tiny bit better: by actually reporting errors. Before that, since it returns void, it would silently fail if a random pointer is passed to CoTaskMemFree (because CoTaskMemAlloc has some book keeping, so in 99.99% of the case it would know it is not theirs). In Vista, if the app is indeed compiled for Vista, which means its subsystem version would be 6.0 in the PE file, you would get the better behavior: you would crash!
How's VS 2012 involved in this puzzle? It actually pass /subsystemversion:6.0 to C# compiler (this is a new switch), which would enable this new crash-if-error behavior.
Fixing this issue is actually pretty straight-forward:
For more information, you can refer to an MSDN article that I wrote about 4 years ago here: http://msdn.microsoft.com/en-us/magazine/cc164193.aspx
Let's first start by looking at a small code snippet:
if (Marshal.GetHRForException(myException) == E_SOME_COM_ERROR)
This looks perfectly fine, right? Not really. It turns out this API is actually poorly named, and it actually does more than just retrieving the HR from the exception object. Using this API incorrectly could give you some weird problems, such as throwing out an incorrect, old exception.
So what does the API actually do?
Besides returning you the HRESULT, this function also sets the current thread IErrorInfo object to be the exception object. Every thread has a assoicated IErrorInfo COM object, defaulting to NULL. This IErrorInfo object represents the failure from the last COM API call, which is sort of like the GetLastError() Win32 API, or errno C api. From the IErrorInfo object, you can get more information such as the error description, help file context, etc, that gives you more information about the error. By setting IErrorInfo object to the exception object (actually setting to the IErrorInfo implementation of the exception object, which is shared by all managed objects), you are essentially telling your COM callers, "hey, there is a failure, it's this exception object, and you should do something about it", which is obviously a bad idea if you only want to retrieve the HR.
In many cases, this could go unnoticed, if your COM caller/callee handles IErrorInfo correctly. By "correctly", I mean they follow the COM IErrorInfo protocol, which are basically:
1. The caller use IErrorInfo from the current thread when calling a COM interface that explicily says "I support IErrorInfo"
2. The callee (which implements the COM interface) that says "I support IErrorInfo", must clear/set IErrorInfo before returning. This is very similar to GetLastError()/errno, right?
If this is the case, the incorrectly set IErrorInfo in the thread, would typically either be overwritten/cleared by some interface implementation that supports IErrorInfo, or ignored.
However, because the protocol I described is actually not very well understood by every COM developer, I've often seen cases where the COM component just say "sure, I support IErrorInfo on every COM interface", and they don't set/clear IErrorInfo at all. And then, the caller of this poorly implmented COM interface, would end up using the IErrorInfo set by GetHRForException call earlier, and is lead to believe that there is an error. For example, let's consider the following scenario:
Now, you might be wondering: What is this API really intended for in the first place? It is actually used in places in COM interop, where you want to return the HRESULT and set IErrorInfo by yourself. This doesn't happen very often - most people would be happy enough to let COM interop takes care of that part by converting Exception to HR and IErrorInfo (which, happens to call Marshal.GetHRForException). So unless you really know what you are doing, my advice is to stay away from Marshal.GetHRForException.
What if you really, really want to just get the HRESULT for a specific Exception? You should use Exception.HResult property.