Mason Bendixen's COM Interop Notes

CoSetProxyBlanket not supported from managed code

2007-04-19T01:52:00Z

COM objects are represented in managed code by runtime callable wrappers. I have written a reasonable summary of runtime callable wrappers (RCWs) here that you should read to understand the concepts in this entry.

CoSetProxyBlanket is a useful API to have control over COM security settings such as authentication and impersonation at the granularity of the interface pointer. It is an API whose functionality should have been exposed as part of the runtime callable wrapper abstraction. Even worse than not including this functionality directly, if people attempt to PInvoke to CoSetProxyBlanket, fxcop will flag this as an invalid call.

The truth is that not including it was an oversight (and a bad one at that). Because of the design of RCWs, PInvoking to CoSetProxyBlanket with an interface pointer from an RCW will not reliably work since there are many cases were we just fetch interface pointers on the fly using QueryInterface() so the effect of the call to CoSetProxyBlanket() will be lost. Additionally, RCWs are shared by all users in the app domain so even if your call hits one of the cached interfaced pointers on the RCW and the results are retained, it will affect everyone in the app domain using that object. For these reasons, we do not support calling CoSetProxyBlanket from managed code.

Unfortunately, people still need to call this API. The solution I recommend to all of these people is to write a small shim of native code that implements the interface whose security settings need to be modified. The native shim wraps the interface pointer but does the CoSetProxyBlanket call for the caller. From managed code, the user obtains an RCW to this shim and then makes the calls in the shim.

I realize that this is inconvenient, but the native shim should be trivial to write. ATL should make this something that can be done in under ten minutes.

I agree that this is a common scenario, and even though I can’t comment on specific features of future releases of the product, I will say that there is a high likelihood that this will be addressed in a future release of the product.

The mapping between interface pointers and runtime callable wrappers (RCWs)

2007-04-19T00:27:00Z

The term COM object is frequently thrown around, but people really access interfaces off of objects. The objects themselves are hidden. Managed is quite a bit different since interfaces are part of the system, but people can also access objects directly. When COM objects are accessed from managed, they are wrapped by a managed object called a runtime callable wrapper (RCW). The type of this runtime callable wrapper is either System.__ComObject or a strongly typed derived type as discussed here.

Instead of dealing with objects at an interface level, the CLR attempts to map back to the actual COM object being wrapped. We rely on a fundamental principle of COM that if you are given two interface pointers and QueryInterface() each of them explicitly for IID_IUnknown, they are required to give back an identical value if they in fact refer to the same object. This is basically the canonical IUnknown for a COM object and creates the basis of identity in COM.

In managed, we have a per app domain (please see any CLR reference for a definition of app domain) cache mapping canonical IUnknowns back to RCWs. When an IUnknown enters the system (through a marshal call, through activation, as a return parameter from a method call, etc.), we check the cache to see if an RCW already exists for the COM object. If a mapping exists, a reference to the existing RCW is returned. Otherwise a new RCW is created and a cache mapping is added.

Many people don’t understand that RCWs are shared and that anything that is done to the interface pointers wrapped by the RCW or RCW wide operations like System.Runtime.InteropServices.Marshal.ReleaseComObject() affect all users of the RCW. If people explicitly want a new RCW to be created even if a mapping already exists in the cache, they should call System.Runtime.InteropServices.Marshal.GetUniqueObjectForIUnknown(). However, they need to understand that RCWs are expensive so there is a good reason why we do this sharing.

Once an RCW is created, it can be cast to any interface that can be reached through QueryInterface() on the underlying COM object. Under the covers, we keep a small stash of interface pointers for each interface that has been accessed up to the size of the cache. Once the cache is full, any additional calls on new interfaces will result in a QueryInterface() on each method call. This is pretty important because some operations (like CoSetProxyBlanket) have affects on a given interface pointer and will not work with RCWs since we might generate a new interface pointer on the fly per call.

It is also important to remember that interface pointers have COM apartment / COM+ activity affinity and need the help of the Global Interface Table or CoMarshalInterThreadInterfaceInStream() to cross boundaries. RCWs shield users from this by doing all of the gymnastics to make this work behind the scenes for the user. This is both a good and bad thing. RCWs are most efficient when accessed from the apartment from which they were created; however, it is important to remember that the RCW might be getting pulled from the app domain wide cache (and the use case that caused the RCW to be created could be in a different apartment and cause unexpected performance sapping apartment transitions).

Unfortunately, even though RCWs are managed objects, most of the functionality for them is buried in the native implementation of the CLR virtual machine so there isn’t a lot of end user transparency into how they operate. If you have any questions about additional RCW or other Interop issues that you would like to see addressed, please don’t hesitate to email me.

ClassInterfaceType.None is my recommended option over AutoDispatch & AutoDual

2007-04-18T03:05:00Z

Advanced to intermediate COM programmers are well aware of the problems with dual interfaces. I won’t discuss that here, but people should read Don Box’s book Essential COM if they want an excellent explanation of this. There is a good discussion of avoiding ClassInterfaceType.AutoDual at:

http://msdn2.microsoft.com/en-us/library/ms182205(VS.80).aspx

I will go further and recommend that people don’t use the surface area of classes to ever generate interfaces (ie. always use ClassInterfaceType.None and never use ClassInterfaceType.AutoDispatch or ClassInterfaceType.AutoDual). COM is an interface based model. Interface design is very important and should always be split from the implementation.

I feel like the best way to define COM interfaces is in the language that was designed specifically for that task: IDL. I would recommend that people define their interfaces (even interfaces implemented in managed) in IDL, feed those interfaces through MIDL to generate a type library, import the interface into a managed Interop assembly with tlbimp.exe, and then implement that interface in their managed class to be exposed as a COM object. I realize that all of this sounds a little vague so I’ll go through a small example.

My sample interface is defined in a simple IDL file classinterface.idl:

import "oaidl.idl";

import "ocidl.idl";

[

uuid(b8728725-d116-46ad-a1fb-83348f341bf7),

version(1.0),

helpstring("classinterface")

]

library classinterface

{

importlib("stdole2.tlb");

[

uuid(46bcde68-5ffd-4423-81ce-09c3798af91d),

version(1.0),

oleautomation

]

interface IBlogDemo : IUnknown

{

HRESULT Add([in] long n1, [in] long n2, [out, retval] long * pResult);

}

};

I compile this into a type library with:

midl classinterface.idl /tlb .\bin\classinterface.tlb

I import the type library into an Interop assembly with:

tlbimp .\bin\classinterface.tlb /out:.\bin\interop.classinterface.dll

I create a small managed library in C# like so (main.cs):

using interop.classinterface;

using System;

using System.Runtime.InteropServices;

namespace main

{

[ComVisible(true)]

[Guid("b0541354-afb9-46bc-8273-5b276fa3812e")]

[ClassInterface(ClassInterfaceType.None)]

[ComDefaultInterface(typeof(IBlogDemo))]

public class BlogDemo : IBlogDemo

{

int IBlogDemo.Add(int n1, int n2)

{

return n1 + n2;

}

And I build this with:

csc /t:library /out:.\bin\main.dll /reference:.\bin\interop.classinterface.dll main.cs

The two main points in the C# file is that I say not to generate an interface based on the class with ClassInterface(ClassInterfaceType.None) and then I specify the class to use for the default interface using ComDefaultInterface.

There are several advantages to this approach. The main thing is that the user is conscious of interface design. IDL is better suited to defining interfaces and makes choices about the type of interface being created and how it is marshaled explicit. Using MIDL also means that the C++ programmer will get a real header file to consume instead of using tlbexp to output a type library from an assembly and then using nasty compiler extensions to import the type library into an auto-generated header.

A big win is that interfaces are immutable and it easier to uphold that principle with a clean split of the interface from the implementation. Auto-generated interfaces based on classes have a tendency to violate this and not version well. There are also features of the CLR (generics being a prime example) that are not supported through Interop. Using one of these features and then exposing an auto-generated class interface to COM will cause a runtime error. Many people got burned by this when they upgraded to the .NET Framework version 2.0 and one of their parent classes now inherited from a generic class which caused them to have strange runtime errors.

COM Interop is a complicated topic and there are unfortunately many scenarios where there are different ways to do exactly the same thing. When there are multiple approaches, I try to lay down a best practice that will have the highest odds of leading the programmer down the path to success. I believe that defining all interfaces in IDL is a great best practice that will result in the best possible results. I feel like COM was very successful because it split interface from implementation. We should continue that practice when doing COM Interop from managed code.

A single Interop assembly does not work for different architectures

2007-04-18T01:03:00Z

One of the exciting things about managed code is that it is quite a bit easier to write code that is portable across architectures. There is no question about the size of a short, int, or long integer. In fact, it is possible to write assemblies that run unaltered on 32 and 64 bit platforms. I say possible unless they interoperate with native code.

As an example, consider the following idl file:

import "oaidl.idl";

import "ocidl.idl";

[

uuid(4FBDEE5B-607B-45b1-8B8F-A5779ECB5766),

version(1.0),

helpstring("align")

]

library align

{

importlib("stdole2.tlb");

typedef struct Simple

{

int fourbytes;

double dForce64bit;

} Simple;

typedef struct Complex

{

int fourbytes;

void * pv1;

} Complex;

};

I built this IDL file with:

rmdir /q /s simple

mkdir simple

midl align.idl /zp8 /tlb .\simple\align.tlb

tlbimp .\simple\align.tlb /out:.\simple\interop.align.dll

ildasm /text .\simple\interop.align.dll > .\simple\align.txt

Note that I am explicitly stating /zp8 (8 byte packing), but it is the default for MIDL and the C compiler so it does not need to be specified. The result is a type library (built for x86 – the default) and an Interop assembly. Looking at the disassembly we see:

.class public sequential ansi sealed beforefieldinit interop.align.Simple

extends [mscorlib]System.ValueType

{

.pack 8

.size 0

.field public int32 fourbytes

.field public float64 dForce64bit

} // end of class interop.align.Simple

.class public sequential ansi sealed beforefieldinit interop.align.Complex

extends [mscorlib]System.ValueType

{

.pack 4 ß This will yield wrong results on 64 bit platforms

.size 0

.field public int32 fourbytes

.field public native int pv1

} // end of class interop.align.Complex

Note that the Complex struct gets converted to “.pack 4” even though the void* has been converted to “native int” because we have built on a 32bit platform. The /zp8 is somewhat hard to understand. It doesn’t mean “align everything on 8 byte boundaries.” It means “add what padding is needed to make eight byte or smaller types have natural alignment.”

The simplest way to think of packing is to think of a simple struct:

struct Basic

{

char c;

INT32 myint;

};

/zp1 would pack things on 1 byte boundaries so myint would directly follow c in memory. /zp2 would align on 2 byte boundaries so there would be one byte of padding after c before myint. /zp4 would aling on 4 byte boundaries so there would be three bytes of padding after c before myint. /zp8 would actually behave like /zp4 in this case since only three bytes of padding are needed to achieve natural alignment for a 32 bit integer. Note that /zp1 and /zp2 have the potential to produce unaligned results. /zp8 produces results that roughly make sense.

The Simple struct contains a four byte integer followed by an eight byte floating point type. With eight byte packing, four bytes of padding is needed to get the floating point value to start on an eight byte boundary. This causes the .pack 8 to appear in the managed assembly.

The Complex struct contains a four byte integer followed by a “native int.” The size of the native int is platform dependent and was determined at build time. In this case, we built for 32bit (the default) which meant the size was taken to be four bytes. Since things naturally align on four byte boundaries, .pack 4 is put into the managed assembly.

The problem is that C++ code (ignoring managed C++) is platform dependent and the code compiled on the C compiler for Complex will have four bytes of padding between the four byte integer and the “native int” / void* value. This will be one of the worst types of failures. X64 tolerates misaligned data access so there will not be some immediate fault when the data is accessed, but reads and writes will happen at the wrong address—silently! This is awful and one of the hardest types of issues to track down.

This is bad enough that I am going to look into to having tlbimp generated assemblies built on 32 bit platforms fail to load on 64 bit platforms as a potential future change (or at least an option).

The correct solution is to build separately with MIDL and tlbimp on all platforms. An x64 build would look like:

rmdir /q /s x64

mkdir x64

midl align.idl /zp8 /win64 /tlb .\x64\align.tlb

tlbimp .\x64\align.tlb /out:.\x64\interop.align.dll /machine:X64

ildasm /text .\x64\interop.align.dll > .\x64\align.txt

The disassembly of the resulting Interop assembly will look like:

.class public sequential ansi sealed beforefieldinit interop.align.Simple

extends [mscorlib]System.ValueType

{

.pack 8

.size 0

.field public int32 fourbytes

.field public float64 dForce64bit

} // end of class interop.align.Simple

.class public sequential ansi sealed beforefieldinit interop.align.Complex

extends [mscorlib]System.ValueType

{

.pack 8 ß This is what we expect on 64 bit platforms

.size 0

.field public int32 fourbytes

.field public native int pv1

} // end of class interop.align.Complex

Both structs are .pack 8 as expected and will match the output of the C++ compiler. Note that both Interop assemblies can coexist with the same name in the GAC and the loader will correctly bind to the right version for the executing platform.

I would like to thank Bill Evans for writing a detailed email to a customer that I scavenged for information. I’d like to thank my friend Kevin Frei for answering some C++ compiler questions about packing.

Non-Cocreatable Class Objects No Longer Wrapped in NDP 2.0 and Later

2007-04-12T22:14:00Z

I dealt with a customer attempting to automate Microsoft Outlook 2003 using the Outlook Primary Interoperability Assembly (Microsoft.Office.Interop.Outlook.dll) for the Microsoft Outlook 11.0 Object Library type library version 9.2.

The type library contains the following items:

…

[…noncreatable…]

coclass MailItem {

[default] interface _MailItem;

[source] dispinterface ItemEvents;

[default, source] dispinterface ItemEvents_10;

};

…

interface _MailItem : IDispatch {

…

};

…

dispinterface ItemEvents {

…

};

…

dispinterface ItemEvents_10 {

…

};

…

As you can see, the type library contains the noncreatable coclass MailItem which implements one interface and supports two eventing interfaces (with ItemEvents_10 being the default).

The primary interoperability assembly for the type library contains:

interface Microsoft.Office.Interop.Outlook._MailItem

interface Microsoft.Office.Interop.Outlook.ItemEvents_Event

interface Microsoft.Office.Interop.Outlook.ItemEvents_10_Event

interface Microsoft.Office.Interop.Outlook.MailItem

implements Microsoft.Office.Interop.Outlook._MailItem

implements Microsoft.Office.Interop.Outlook.ItemEvents_10_Event

class Microsoft.Office.Interop.Outlook.MailItemClass

implements Microsoft.Office.Interop.Outlook._MailItem

implements Microsoft.Office.Interop.Outlook.MailItem

implements Microsoft.Office.Interop.Outlook.ItemEvents_10_Event

implements Microsoft.Office.Interop.Outlook.ItemEvents_Event

As you can see, the MailItem interface implements just the default interface and the default event interface whereas the MailtItemClass implements every interface. Note that the customer can get to the non-default event interface from a MailItem interface by casting it to an ItemEvents_Event interface.

The customer has code like:

Outlook.Explorer oExp = …

Outlook.Selection oSel = oExp.Selection;

if (oSel[1] is Outlook.MailItem)

{

//Outlook.MailItem MI =

// (Outlook.MailItem) oSel[i]; ß This works

Outlook.MailItemClass mailItem =

(Outlook.MailItemClass)oSel[1]; ß This throws exception

}

When run, the previously listed code throws this exception:

Unhandled Exception: System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to class type 'Microsoft.Office.Interop.Outlook.MailItemClass'. COM components that enter the CLR and do not support IProvideClassInfo or that do not have any interop assembly registered will be wrapped in the __ComObject type. Instances of this type cannot be cast to any other class; however they can be cast to interfaces as long as the underlying COM component supports QueryInterface calls for the IID of the interface.

at Repro.Main()

The object returned from oExp.Selection is being bound to the type __ComObject instead of the type MailItemClass. Note that it can still be cast to all of the interfaces, but the cast to the class will fail. This code runs without the exception on pre-v2.0 versions of the CLR.

Explanation of Behavior

When a Runtime Callable Wrapper (RCW) is being created for a COM object, the runtime QueryInterfaces the COM object for IProvideClassInfo to determine if the interface is a coclass. If it is a coclass, the CLSID is fetched for the object and the runtime attempts to bind the wrapper object’s type to the typed class wrapper instead of a generic __ComObject.

The system has a hashtable mapping CLSID’s to the appropriate class wrapper types. The normal way that this table is populated is to look in the registry under:

HKEY_CLASSES_ROOT\CLSID\{CLSID}\InprocServer32

Assembly ß Strong name of assembly that contains class wrapper

Class ß Name of the wrapper class

The problem with this is that there are no registry entries for noncreatable coclasses (such as MailItem).

The reason that this used to work pre-v2.0 is because there used to be an optimization in this process that could lead to non-deterministic behavior.

Each Runtime Callable Wrapper (RCW) class that wraps a coclass provides information about the CLSID of the coclass that it wraps. When such a class was loaded in pre-v2.0 versions of the runtime, the CLSID to class mapping was added to the table if it was not already present.

This meant that non-cocreatable COM objects would map to __ComObject prior to loading the correct wrapper class (if the mapping didn’t exist in the registry), but COM objects of the same type would bind to the typed wrapper class after an action had caused the appropriate wrapper class to be loaded once.

Because of the non-determinism of this behavior, this optimization was removed in v2.0.

Solution to the problem

The correct solution is for the customer to not use the coclass wrapper class for noncreatable coclass’s. The customer can cast the interface that they have to any other interface (or event interface) to access to all of the functionality exposed by the coclass wrapper class.

The coclass wrapper class is a flattening of all of the methods of all of the interfaces (including eventing interfaces) implemented by the given COM object. Although simple to use, casting to interfaces gives the programmer access to all of the functionality exposed by the COM object.

I would go so far as to recommend everyone program against the actual interfaces instead of using the flattened typed class object. I think that trying to hide the interface based nature of COM was a bad idea. I feel like other issues caused by this flattening and hiding of interfaces could possibly introduce more breaking changes like this in future releases (note that I said possibly).

This post does bring up the whole topic of the __ComObject. People get very confused and think something is wrong when they see this for the type of a runtime callable wrapper in the debugger. I’ll try to tackle a more general discussion of RCWs and the nature of __ComObject in a future post.

Mismatch between object lifetime when bridging between COM and managed code

2007-04-12T02:00:00Z

From time to time, I see object hierarchies like:

interface IParent : IUnknown

{

    [propget] HRESULT Child([out, retval] IChild ** ppChild);

}

dispinterface IChildEvents

{

    // ...

}

interface IChild : IUnknown

{

}

coclass Parent

{

    [default] interface IParent;

}

[noncreatable]

coclass Child

{

    [default] interface IChild;

    [default, source] dispinterface IChildEvents;

}

This describes a simple parent object with a child that has connection point based events. The user imports the type library for this pseudo-IDL code using tlbimp.exe to generate an Interop assembly. In some C# code, they write:

IParent parent = // ...

parent.Child.myevent = MyHandler;

The intent seems clear to someone coming from a Jscript programming background, but they will get very undefined results.

Breaking this down, COM objects are bridged to managed code by runtime callable wrappers. For a given COM object in a given app domain, there should be a single runtime callable wrapper (with some caveats to be addressed in a future post). A runtime callable wrapper is a full blown managed object that “manages” a given COM object. When I say that it “manages” the underlying COM object, I mean that it maps to interfaces supported by the object, lifetime, eventing, and automatically performing cross apartment / context calls automatically so that the managed object appears to be apartment agnostic.

In the C# code snippet above, there are two runtime callables wrappers (RCWs from now on). The first one is referenced as long as the variable “parent” is in scope. The second one is created on the fly in response to “parent.Child.” This case is interesting in that this RCW is created on the fly, but it is never assigned to anything. From a garbage collector point of view, it is available to be collected as soon as the statement completes.

The problem is that the bridge between COM connection point based eventing and managed delegates resides on the RCW. In this case, the user has set up a delegate to subscribe for events. An implementation of the event sink mapping back to the delegate is created and registered using IConnectionPoint::Advise; however, the event sink is part of the RCW and will be destroyed as soon as the RCW is collected.

The problem here is that this will most likely work as expected in simple cases, but events will stop as soon as the RCW for the child object is collected. From my experience, this is the worst type of failure.

The correct solution is to do:

IParent parent = // ...

IChild child = parent.Child;

Child.myevent = MyHandler;

And to save “child” somewhere as long as the event subscription needs to remain valid.

This is somewhat annoying since you would look at the original code and expect it to work. Digging into how COM Interop works, I have realized that this is just an artifact of the design and not something that can be easily solved. It is also just one specific instance of the general mismatch between COM lifetime and managed lifetime.

COM lifetime is very explicit and deterministic: reference count drops to zero, object is released. Some people get caught up in this determinism and actually break the rules of COM. A good example is another parent child sample:

class Child

{

    // ...

    STDMETHODIMP_(ULONG, AddRef)() {return 2;}

    STDMETHODIMP_(ULONG, Release)() {return 1;}

    // ...

};

class Parent

{

    // ...

    Child m_child;

    STDMETHODIMP GetChild(IChild ** ppChild)

    {

        *ppChild = &m_child;

        return S_OK;

    }

    // ...

};

In this case, the child is allocated directly with the parent and doesn’t maintain its own lifetime. This is generally bad idea but usually works when called from a language like C++ from code that looks like:

IParent * pParent = //...

IChild * pChild = NULL;

hr = pParent->GetChild(&pChild);

if(SUCCEEDED(hr))

{

// ...

pChild->Release();

}

pParent->Release();

Note that it would be perfectly valid to fetch the child and then immediately release the pointer to the parent (which would cause this code to crash since the parent would release and take the child with it), but this is not common programming practice. When creating a technology that leverages something like COM, programming against the elegance of the COM spec isn’t too hard to get right (mostly...). Trying to support behavior that is relying on assumptions and convention instead of following the rules is hard to do. Note that we definitely try to recognize and support really common conventions, but there are definitely limits to what can be done.

In managed, the RCWs are released by garbage collection and there is no ordering. Each is just another managed object that is available for collection so the parent can easily be collected (causing a release under the covers) which will destroy the child and cause an AV when the child RCW is collected later.

Unlike the first eventing example, code like this is clearly playing fast and loose with COM, but people still get annoyed when code that works in their existing VB6 or MFC code base fails when called through managed code.

In general, bridging between two large, full-featured object models is a challenging problem. Given the size of the problem space I alternate between being impressed by how well COM Interop works and being surprised anything works at all. I’m going to follow up with a couple more object mismatches before branching off into other topics. I have my own hit list of things to cover, but if people have things they would like to see discussed please let me know. I’m the main development owner of the Interop space so I’d also love to hear your feedback on the product and suggestions for things to add or improve.