Yi Zhang's MSDN Blog

Everything about the CLR - interop / WinRT / and other random stuff

Random things about .NET, CLR, C++, and work

  • Yi Zhang's MSDN Blog

    WARNING: VS 2012 might break your broken P/Invoke


    I'm seeing many people reporting that they are seeing strange P/invoke issues when they moved their code to VS 2012. Typically, they have P/invokes like this:

    [DllImport("Win32Project2.dll", PreserveSig = true, CharSet = CharSet.Unicode)]
    static extern int MyPInvoke(out string ret);

    If you attach a native debugger (or enable native debugging), with the right symbols (public symbol would be just fine), you would see that it is actually crashing inside combase.dll!CoTaskMemFree:

    >    ntdll.dll!_RtlReportCriticalFailure@8()  + 0x33 bytes   
         ntdll.dll!_RtlpReportHeapFailure@4()  + 0x21 bytes   
         ntdll.dll!_RtlpLogHeapFailure@24()  + 0xa2 bytes   
         ntdll.dll!_RtlFreeHeap@12()  + 0x2677b bytes   
         combase.dll!CoTaskMemFree(void * pv)  Line 475    C++

    Problem like this are usually caused by a mismatch between the managed declaration and the native implementation. The usual suspect here is the 'out string' signature. In .NET world, having a out string in P/invoke means:

    1. Native code pass a LPWSTR out to managed code, allocating the memory using CoTaskMemAlloc
    2. CLR creates a managed System.String instance from LPWSTR, and frees the LPWSTR using CoTaskMemFree (because the ownership has already transferred from native code to managed code, which the 'out' implies)

    So, if MyPInvoke pass a LPWSTR that is not allocated from CoTaskMemAlloc, say from HeapAlloc, new, or a string literal, you are potentially asking CLR to do CoTaskMemFree on your pointer, which would result in undefined behavior.

    Basically, this means the P/invoke was already broken before.

    But why it didn't crash before and suddenly starts to crash now after upgrading?

    This is exactly the kind of thing you could expect with undefined behavior, because that could change anytime. It turns out in Vista, Windows guys decided to make CoTaskMemFree a tiny bit better: by actually reporting errors. Before that, since it returns void, it would silently fail if a random pointer is passed to CoTaskMemFree (because CoTaskMemAlloc has some book keeping, so in 99.99% of the case it would know it is not theirs). In Vista, if the app is indeed compiled for Vista, which means its subsystem version would be 6.0 in the PE file, you would get the better behavior: you would crash!

    How's VS 2012 involved in this puzzle? It actually pass /subsystemversion:6.0 to C# compiler (this is a new switch), which would enable this new crash-if-error behavior.

    Fixing this issue is actually pretty straight-forward:

    1. If you can change the native implementation, change the native function to allocate the memory using CoTaskMemAlloc
    2. Otherwise, you'll need to change the P/invoke declaration in managed code. Since the only there are probably a thousand kind of allocators out there, and we only support one (CoTaskMemAlloc/CoTaskMemFree), your best bet is to pass it as IntPtr, and then free it (or simply not free it, if it is a const string, for example) yourself, either by using one of the Marshal APIs, or expose the specific free function of the allocator using another P/invoke.

    For more information, you can refer to an MSDN article that I wrote about 4 years ago here: http://msdn.microsoft.com/en-us/magazine/cc164193.aspx

  • Yi Zhang's MSDN Blog

    Marshal.GetHRForException does more than just Get-HR-For-Exception


    Let's first start by looking at a small code snippet:

    if (Marshal.GetHRForException(myException) == E_SOME_COM_ERROR)

    This looks perfectly fine, right? Not really. It turns out this API is actually poorly named, and it actually does more than just retrieving the HR from the exception object. Using this API incorrectly could give you some weird problems, such as throwing out an incorrect, old exception.

    So what does the API actually do?

    Besides returning you the HRESULT, this function also sets the current thread IErrorInfo object to be the exception object. Every thread has a assoicated IErrorInfo COM object, defaulting to NULL. This IErrorInfo object represents the failure from the last COM API call, which is sort of like the GetLastError() Win32 API, or errno C api. From the IErrorInfo object, you can get more information such as the error description, help file context, etc, that gives you more information about the error. By setting IErrorInfo object to the exception object (actually setting to the IErrorInfo implementation of the exception object, which is shared by all managed objects), you are essentially telling your COM callers, "hey, there is a failure, it's this exception object, and you should do something about it", which is obviously a bad idea if you only want to retrieve the HR.

    In many cases, this could go unnoticed, if your COM caller/callee handles IErrorInfo correctly. By "correctly", I mean they follow the COM IErrorInfo protocol, which are basically:

    1. The caller use IErrorInfo from the current thread when calling a COM interface that explicily says "I support IErrorInfo"

    2. The callee (which implements the COM interface) that says "I support IErrorInfo", must clear/set IErrorInfo before returning. This is very similar to GetLastError()/errno, right?

    If this is the case, the incorrectly set IErrorInfo in the thread, would typically either be overwritten/cleared by some interface implementation that supports IErrorInfo, or ignored.

    However, because the protocol I described is actually not very well understood by every COM developer, I've often seen cases where the COM component just say "sure, I support IErrorInfo on every COM interface", and they don't set/clear IErrorInfo at all. And then, the caller of this poorly implmented COM interface, would end up using the IErrorInfo set by GetHRForException call earlier, and is lead to believe that there is an error. For example, let's consider the following scenario:

    1. Some COM component A calls into .NET code
    2. .NET code throw an exception, say IOException
    3. .NET code catches the exception, and call Marshal.GetHRForException, which sets IErrorInfo to this exception IOException
    4. .NET code calls into another COM component B, calls into IB.Func
    5. IB.Func failed and return some HR failure code, say E_OUTOFMEMORY
    6. CLR sees the HR failure code, and ask the COM component B "do you support IErrorInfo", and B answers "sure, why not", except it doesn't really support it and it doesn't set/clear IErrorInfo
    7. CLR retrieve the IErrorInfo, and sees that it is actually a managed exception, and throws out the IOException exception instead of OutOfMemoryException.

    Now, you might be wondering: What is this API really intended for in the first place? It is actually used in places in COM interop, where you want to return the HRESULT and set IErrorInfo by yourself. This doesn't happen very often - most people would be happy enough to let COM interop takes care of that part by converting Exception to HR and IErrorInfo (which, happens to call Marshal.GetHRForException). So unless you really know what you are doing, my advice is to stay away from Marshal.GetHRForException.

    What if you really, really want to just get the HRESULT for a specific Exception? You should use Exception.HResult property.

Page 1 of 1 (2 items)