I'm seeing many people reporting that they are seeing strange P/invoke issues when they moved their code to VS 2012. Typically, they have P/invokes like this:
[DllImport("Win32Project2.dll", PreserveSig = true, CharSet = CharSet.Unicode)]
static extern int MyPInvoke(out string ret);
If you attach a native debugger (or enable native debugging), with the right symbols (public symbol would be just fine), you would see that it is actually crashing inside combase.dll!CoTaskMemFree:
> ntdll.dll!_RtlReportCriticalFailure@8() + 0x33 bytes ntdll.dll!_RtlpReportHeapFailure@4() + 0x21 bytes ntdll.dll!_RtlpLogHeapFailure@24() + 0xa2 bytes ntdll.dll!_RtlFreeHeap@12() + 0x2677b bytes combase.dll!CoTaskMemFree(void * pv) Line 475 C++
Problem like this are usually caused by a mismatch between the managed declaration and the native implementation. The usual suspect here is the 'out string' signature. In .NET world, having a out string in P/invoke means:
So, if MyPInvoke pass a LPWSTR that is not allocated from CoTaskMemAlloc, say from HeapAlloc, new, or a string literal, you are potentially asking CLR to do CoTaskMemFree on your pointer, which would result in undefined behavior.
Basically, this means the P/invoke was already broken before.
But why it didn't crash before and suddenly starts to crash now after upgrading?
This is exactly the kind of thing you could expect with undefined behavior, because that could change anytime. It turns out in Vista, Windows guys decided to make CoTaskMemFree a tiny bit better: by actually reporting errors. Before that, since it returns void, it would silently fail if a random pointer is passed to CoTaskMemFree (because CoTaskMemAlloc has some book keeping, so in 99.99% of the case it would know it is not theirs). In Vista, if the app is indeed compiled for Vista, which means its subsystem version would be 6.0 in the PE file, you would get the better behavior: you would crash!
How's VS 2012 involved in this puzzle? It actually pass /subsystemversion:6.0 to C# compiler (this is a new switch), which would enable this new crash-if-error behavior.
Fixing this issue is actually pretty straight-forward:
For more information, you can refer to an MSDN article that I wrote about 4 years ago here: http://msdn.microsoft.com/en-us/magazine/cc164193.aspx
Hi, thanks for sharing this.
I have one question though, does it affect code where instead of out string I pass StringBuilder ?
If you are passing 'out StringBuilder', the same rule applies. The native side of signature would be 'wchar_t **'
If you are passing 'StringBuilder', the native signature would be 'wchar_t *'. There are no memory ownership transfer in this case. CLR owns the wchar_t * buffer (it comes from pinned StringBuilder), and native code writes to that wchar_t* buffer.