I am often asked if there is a bug in VS when using the object model from an out of process controller. People often find that, even though they tried to shut down VS that devenv.exe is still alive in the list of processes. Well, sit back, relax, grab a cup of <insert your favorite beverage here> and listen to a story of intrigue, suspense, COM-.NET interaction, reference counting, garbage collection, and even zombies!

 

Creating an object

Not many people know that VS can be created through COM. This begins the first problem. For side by side (known from now on as SxS – meaning that two versions of VS can be installed on the same computer) reasons, Visual Studio 2002 and Visual Studio 2003 have different ProgIDs, VisualStudio.DTE.7 and VisualStudio.DTE.7.1, respectively. This makes perfect sense since you want to be able to selectively create one instance or another. To use these ProgIDs in the C++ language, you would translate this ProgID into a CLSID, then call CoCreateInstance using the CLSID. If you are using .NET, then this gets a little tricky depending on the version of VS you are using. If you want to create a new instance of the 7.0 DTE you can use code such as this:

 

EnvDTE.DTE dteObject = new EnvDTE.DTE();

 

Simple, right? Not so if you want to create an instance of the 7.1 object model. This is because of SxS issues again, not only did the ProgID need to change, but also the CLSID needed to change (you cannot have one CLSID point to two different objects). For compatibility reasons, we did not change the metadata wrapper around the DTE.olb type library (nor did we change the typelib – maybe I will write a blog for the reasons of this someday), this meant that the old GUID is embedded into the metadata, and if you tried to use code like that above you will always get the 7.0 version. And if you don’t have 7.0 installed then you will get an exception thrown at you. To work around this, you need to use code such as the following:

 

System.Type t = System.Type.GetTypeFromProgID("VisualStudio.DTE.7.1");

EnvDTE.DTE dteObject = (EnvDTE.DTE)System.Activator.CreateInstance(t, true);

 

You can also grab any random version of VS with the version independent ProgID:

 

System.Type t = System.Type.GetTypeFromProgID("VisualStudio.DTE");

EnvDTE.DTE dteObject = (EnvDTE.DTE)System.Activator.CreateInstance(t, true);

 

The VS object model can be retrieved in other ways when out of process, for example, in the above code snippets, you can supply “Solution” wherever you see “DTE”. You can also go to the running object table (ROT) and find various forms of DTE and Solution there. Lastly, you can cause activation on the object model through a solution’s file name.

 

Using a created DTE object

Now that you have created a new instance of VS, you can use the DTE object like any other DTE object you are handed when your code is an Add-in, Wizard, or Macro. But because this DTE object was retrieved from an out of process application, a little bit of extra stuff needs to go on in the background. See, when a COM object is created in this way, something needs to control when the COM object, in this case, the devenv.exe process, should be closed and removed from memory. For VS, a formula is used to calculate when VS should shut down, and is known as the lifetime of the program. The formula used is:

 

Number of references on the DTE object + Number of references on the Solution object + Number of locks on the class factory + 1 if the DTE.UserControl property is set to true

 

Where: References on the DTE and Solution object is the number of calls to IUnknown::AddRef – the number of calls to IUnknown::Release. The number of locks on the class factory is the result of IClassFactory::LockServer(TRUE) - IClassFactory::LockServer(FALSE).

 

When the result of this equation becomes 0, Visual Studio knows that it is time to start its shutdown procedure and close. Now you do not have control over how and when calls to IClassFactory::LockServer are made unless you call the COM API CoLockObjectExternal (which you should not be doing). Calls to LockServer are usually handled by COM’s object activation methods (aka SCM). You do have control over DTE.UserControl and the AddRef/Release of COM objects in how you write your program.

 

You can set the DTE.UserControl property value to true, this means that the user is in control of VS. Suppose you wrote a program that would spawn off an instance of VS, but you wanted to leave that instance running so the user could interact with it, even after your program stopped using it. In this case you would want to set UserControl to true. UserControl could also become true in the case of you have the main window of VS visible, and the user opens a new solution through the user interface. If your program was in control of the lifetime of VS and the window suddenly disappeared while that user was doing work because you decided to shut down, then the user would think that is a VS bug and then I would get nasty calls in the middle of the night because of this perceived bug. Then I would be grumpy in the morning and get no work done. UserControl can not be set to false once it has been set to true except in one case. Suppose you create an instance of VS and show the main window. The user then opens a solution file though the UI (thus setting UserControl to true), but later (while your program is still using DTE) the user selects the File | Exit menu item. Since the user gave up control of VS, we will set UserControl back to false.

 

AddRef/Release

The calculation of AddRef and Release can be tricky to control if you are using a .NET programming language. This is because of differences in the memory management models of COM and .NET. If you were using VB6, you could easily set the variable to Nothing, in VC you could call Release on the interface pointer, or you could let the variable go out of scope and it will be released. [As a sidebar, if you are using ATL’s CComPtr, never, never, never release a pointer by calling the Release method (either on the COM interface or the CComPtr class). Set the variable to NULL using the = operator or let it go out of scope. I have seen way too many bugs caused by improper use of Release on CComPtr. Calling the interface’s Release will cause crashes, and CComPtr.Release is confusing to the reader of your code as to which Release is being called.] But a complex amount of code is run when using .NET to control an out of process COM object. As a perfect example, let’s examine this seemingly harmless bit of code taken from an Add-in:

 

public void OnConnection(...)

{

EnvDTE.Events events = applicationObject.Events;

EnvDTE.WindowEvents windowsEvents = (EnvDTE.WindowEvents)events.get_WindowEvents(null);

windowsEvents.WindowActivated += new _dispWindowEvents_WindowActivatedEventHandler(this.WindowActivated);

}

 

Seems quite simple, doesn’t it? Well, actually there is a bug hiding here, and many times per week a bug is reported about this problem. If you were to run this code the event handler would be called once, maybe twice, but eventually the event would stop being called. Why? Well, when you call off to events.get_WindowEvents a .NET object which wraps the COM WindowEvents object is created and put on the heap. The variable windowsEvents is then assigned to point to that object on the heap (notice, the variable is not the actual data on the heap, but points to it). The code finishes up by telling VS which function to call when the event occurs, and then the function returns. If windowsEvents were VB6 or an ATL CComPtr variable, the COM object would have its Release method called since the variable is going out of scope, and then the event handler would never be called. But when using .NET, this object is not immediately released, it is marked for a possible garbage collection and until a GC happens on that object it will stay in memory. So, even though your event may fire once, twice, or even three times, the object is doomed, and will eventually go out of memory when it is collected, causing what seems to be another false bug in VS (that I get calls for at 3am. Seriously people, please stop calling at that time! 3pm: good time, 3am: bad time). How can you fix this? Simply move the variable declaration outside the method to the function, and then in the On Disconnection method call the -= operator to remove the event.

 

Objects can also be created when you least expect them. Suppose you have code such as the following:

 

            DTE.Solution.Open(“C:\\foo.sln”)

 

In this code, the Solution property of the DTE object is called. Even though it may look harmless, calling this property will create a new object which wraps the Solution object, causing more overhead and creating objects that need to be GCed. If you are ever going to call a property over and over again, such as in the following code:

 

            DTE.Solution.Open(“C:\\foo.sln”)

            MsgBox(DTE.Solution.Name)

            MsgBox(DTE.Solution.Projects.Item(1).Name)

            DTE.Solution.Save()

 

Make sure you try to cut down the number of objects being generated with code like this:

 

            Dim sln as EnvDTE.Solution

            sln = DTE.Solution

            sln.Open(“C:\\foo.sln”)

            MsgBox(sln.Name)

            MsgBox(sln.Projects.Item(1).Name)

            sln.Save()

 

 

Staying in memory

So what do events have to do with the lifetime of VS? It demonstrates a common problem people have seen when they create an instance of VS out of process. Quite often (I have done it myself, so I know how painful it is to figure out what is going on) code is written in a .NET language where it expects that a reference on an object is removed when that variable goes out of scope. But, as was demonstrated with the event, this is not the case. Objects stay alive even though they have gone out of scope. Only when that object is garbage collected is it finally removed from memory, and the reference on the COM object is released. In the case of a DTE object, this makes it seem like there is a bug in VS because, even though all objects seemingly have been destroyed, they really are alive and a reference is kept on a DTE or Solution COM object causing VS to stay in memory. So people think “I can just call DTE.Quit, and everything should be fine”, but that assumption would be incorrect. DTE.Quit simulates the user clicking the File | Exit menu item though a PostMessage-esque way, which could keep VS around for a little longer than expected (don’t expect the process to close immediately, give it a little while). You also cannot rely on calling the method System.GC.Collect() because even though something is marked for collection, that does not guarantee that it will be collected. You could try System.Runtime.InteropServices.Marshal.ReleaseComObject, but this would be very dangerous. It is similar to using the strategy of calling Release on an interface pointer until it returns 0 (which is wrong in so many ways, I don’t have enough time to write about them).

 

A real bug

By now you should know that VS will not always shut down exactly when you expect it to. But there is a real bug here that is not that evident and can get you into trouble. Suppose you have a reference on two objects in the object model, DTE and (for sake of an example) EditPoint. Remembering the earlier formula, DTE contributes to the lifetime of VS, while EditPoint does not. Now suppose VS begins to shut down because the lifetime formula’s result was 0. What happens to the EditPoint object? Well, something really nasty happens here. EditPoint is still being referenced, and therefore the memory it occupies needs to be kept in a valid state so that the controlling program can call Release on it when it is no longer using that object. When the owner of an object (in this case, TextDocument owns an EditPoint), no longer maintains its control over the child object but someone owns a reference on it, we call the object zombied (it is still around for COM referencing rules, but it is really a dead object, a zombie). The zombie state does not apply to only the EditPoint object, but other objects in VS as well. However, when the lifetime formula is 0, the process is told to close (a PostQuitMessage is sent to the devenv.exe process), causing the memory to no longer be valid. The controlling program, thinking the object is still alive, even though it has been zombied and its memory destroyed, calls Release on the object. The result: either VS, the controlling program, or both crash. We have been looking at fixing these for the next version of Visual Studio (and in fact, a number of them have been fixed), but to protect yourself, for now you need to ensure that all references on objects other than DTE and Solution are released before starting to close down. DTE and Solution are safe because the method CoDisconnectObject is called on those two objects when VS is closing. This method severs the proxy-stub connection between the controller and the controlled and any future calls by the controller will return an RPC_* HRESULT or generate an exception in a managed language.

 

Other software

VS is not the only program affected by the difference in .NET and COM memory models. Korby Parnell and I were talking the other day about a problem with Visual Source Safe in how its object model works. There supposedly one object (and I cannot recall which one it was at the moment) in the VSS object model that can have only one instance of that object running at any given time. This object is retrieved by calling a property, which returns a new instance of this object. Now, let’s suppose you obtain a reference on this object using C# and the PIA wrappers for the tlb, causing an object to be created and returned. You use this object, then assume the object goes away because you have left the variable’s scope, set the variable to null, or caused a GC. That object may still be around, not referenced by any variable, but not destroyed yet. Since only one instance of the specific COM object may still be around and it is being referenced by a zombied .NET object, no more of the VSS object can be created.

 

The End

The end of this story is that you should not make assumptions about how memory works in .NET and COM interop. Even though you may be used to using a COM interface to an object, you need to remember that .NET garbage collection rules are in effect, they need to be obeyed, and take precedence over COM memory management rules. However, this does not mean that COM rules also need to be observed. .NET is working as it was designed to work, so are COM objects and VS (except in one specific case, a bug).