Misha Shneerson

Type Equivalence and Type Safety

What one should do on a plane back from the PDC? Of course! Write a new blog post! So, since I still have some topics to cover on the whole NOPIA shabang - here we go.

CLR’s 4.0 support for Type Equivalence allows you to define your own copy of an interface, stick a Guid on it and then you will be able to cast from any managed object implementing an interface with the same Guid. When calling a method on such interface CLR will find the equivalent interface in the inheritance chain of the managed object, verify you are not calling beyond the allowed vtable, verify that the signature at the same vtable offset does match and then call the corresponding method implementation.

This is all good and well and allows for loose type coupling, version resiliency and possibly some other goodness we have not found a name for yet but I have been repeatedly asked whether such arbitrary deviation from the fundamental .NET type system principals is type safe and whether FullTrust is required to make calls through equivalent type to work. Let’s address these concerns one by one.

I believe the mechanism is actually Type Safe . To try and illustrate my point I would make a fundamental assumption that calls through reflection are Type Safe. Most of you would probably agree. The way reflection mechanism works is it first verifies that parameters passed to the call can be cast to the parameter types specified by the method signature. If types do no match – reflection does not let the call to execute. Now compare this mechanism to the way calls for Type Equivalent interfaces are dispatched and try to find a difference :)

Now to the question of FullTrust. Let’s reuse the analogy with reflection once again and recall that reflection does not require FullTrust. Consequently, we can conclude that one is not needed for calls through Type Equivalent interface. The only exception to the rule is when the call contains Type Equivalent structures. Again, in theory, if we can prove that two struct types that are declared to be type equivalent are also structurally equivalent – we should not require FullTrust. The problem is that proving this might be more complicated than it seems to be (this is what I ‘ve been told) so this is too early to talk about what final bits will look like.

Face made for emails

Today I got a word that the video recordings I’ve made with Amanda Silver on the topics of Type Embedding and Type Equivalence did get published. My face is clearly should have been shaded out – but Amanda looks fantastic! Anyway, may be you will find this videos interesting. They are in the MP4 format so you need to have the corresponding player installed. Or see the below embedded player.

No PIAs: The Problems with Primary Interop Assemblies (PIA) - Part 1 of 4 (video)

No PIAs: Under The Hood - Part 2 of 4 (video)

No PIAs: Type Equivalence - Part 3 of 4 (video)

No PIAs: Versioning Object Models - Part 4 of 4 (video)

Also check out the entire library of other people from Microsoft that were recorded just before the PDC. To see the list of episodes click the multiple windows icon. 

Better eventing support in CLR 4.0 using NOPIA support

Events implementation in the Interop Assemblies does have its shortcomings. I enumerated the issues in “COM Interop: Handling events has side effects” post. The quick recap is that Interop Assemblies eventing support does create “ghost RCWs” that get in the way of deterministically managing the life-time of your COM objects.

But there is a hope! The Type Embedding work that we are doing for NOPIA feature eliminates the dependency on Interop Assemblies by embedding the partial local copies of Interop interfaces into the assembly that uses those interfaces.

This customized compiler behavior for Interop Types has been a great opportunity for us to fix the way COM events are handled. Since the implementation of the event sink is currently located in the Interop Assemblies in the pure IL – and the design decision behind types embedding is that we never copy IL – we needed to come up with a different mechanism.

So, we taught the compiler to recognize code patterns that subscribe to events for Interop Assembly types and replace this pattern with something else. To give you an example from my PDC Type Embedding demo – here is the code that subscribes to SheetSelectionChange event from Excel’s Applcation object.

xlapp.SheetSelectionChange += xlapp_SheetSelectionChange;

The code that is emitted by the compiler is very different though and it looks like this (I copied the below line from Reflector’s Disassembler window):

ComEventsHelper.Combine(xlapp, new Guid("00024413-0000-0000-C000-000000000046"), 0x616, new AppEvents_SheetSelectionChangeEventHandler(Program.xlapp_SheetSelectionChange));

You can probably figure out some of the parameters to ComEventsHelper.Combine API i.e. xlapp and the delegate are self-describing. But the reason for the existence of the Guid and the mysterious integer 0x616 may not be immediately apparent. So, what is going on here?

When compilers encounter code that subscribes to an event on an interface coming from Interop Assembly they are trying to analyze how this event is exposed by the COM object. In particular, they need to find the interface COM object would call on when raising the event. So they look in the PIA and end up finding the AppEvents interface which is declared like this:

[Guid("00024413-0000-0000-C000-000000000046")]
public interface AppEvents
{
    [DispId(2612)]
    void AfterCalculate();
    [DispId(1558)]
    void SheetSelectionChange(object Sh, Range Target);
    [DispId(1556)]
    void WindowActivate(Workbook Wb, Window Wn);

….
}

Look at the Guid on this interface – this is the same Guid passed to ComEventsHelper.Combine API.

Also, if you have no problem translating between decimal and hexadecimal you have already realized that 1558=0x616 (I personally used calc.exe to do translation).

So, Combine API attaches an event sink to the COM object’s connection point that handles invocation on the interface with specified GUID. It also tells the event sink that when a method with dispid==0x616 is called it should invoke the delegate that is passed as the last parameter.

Important thing about the event sink is:

1. It is implemented using unsafe managed code
2. It responds to QIs for the above GUID  using the new ICustomQueryInterface which allows managed objects to customize their IUnknown.QueryInterface behavior
3. It only handles late-bound invocation on the source interface. Generally, this should not be a big problem since almost all the COM applications use late-binding when raising events.
4. Subsequent calls to ComEventsHelper.Combine for the same COM object will re-use the existing sink – resulting that there always is only one interop transition when managed code registers multiple event handles
5. The event sink does not marshal parameters to the call if there is no user delegate register against a particular dispid. This solves the problem of the “ghost RCWs”
6. There is a corresponding ComEventHelpers.Remove API which allows to remove the event handler

So, effectively if you compile your code with NOPIAs – the problems I previously raised with the way events are handled by Interop Assemblies should go away!

I am also attaching the source of the simple demo that demonstrates Type Equivalence. You will need to download the Dev10 CTP here to run it - http://go.microsoft.com/fwlink/?LinkId=129231

Type Embedding support in C#

In my PDC session I did talk a lot about Type Embedding support in the compilers and that we call it NOPIA feature. The compiler changes have been implemented by Sam Ng and he is back to blogging. So I will keep an eye on what he has to say and so should you if you want to know his take on NOPIAs. Meanwhile, he is mentioning the list of features he has been working on for the past year in the first post after a long break.

Advances in .NET Type system: Type Equivalence Demo

I just finished my PDC session at Los Angeles where we introduced Type Embedding support in the compilers and Type Equivalence support in the CLR. Some people I talked to after the session were very enthusiastic about the whole thing and I did get comments like “this is the best thing in this conference”. This was very encouraging and exciting. Thanks!

On the other hand there were some points in the talk where people just assumed it is all about COM Interop. Couple of folks started leaving the room before I started showing the Type Equivalence being applied to entirely managed models. I’ve got to tell you that people leaving in the middle of your presentation makes for one of the worst feelings presenter could have (Jesse Kaplan actually shared with me his bad experiences during presentations and his seemed worse). As a result the rest of the presentation on type equivalence did go only south from there and not as many people as I would like to got it. Now I would like to rectify the situation and take some time to go through the Type Equivalence demo in this post.

The download of the demo sources is attached. To run it you will need to run Visual Studio 10 CTP – if you do not have one – get it here http://go.microsoft.com/fwlink/?LinkId=129231

Again, the goal of the demo is to show how you can achieve loose type coupling and version resiliency in a completely managed extensible application. I will show that by placing interfaces into a “compile-time only” programmability assembly and by linking the types from this assembly into your add-ins using new “Embed Interop Types” compilation option – you can achieve true version independence. This means that your add-in can run against any version of the host application – it will be both backward and forward compatible.

So, let’s get to it. Supposedly you already do have VS 10 CTP installed, you have downloaded the attached ConsoleApp solution and opened it in Visual Studio. And now you are asking yourself the very relevant question “what the heck is this thing in front of me?”.

Let’s start the discussion with ConsoleApp.Host project. It is actually a very simple application that does a very simple thing – it reads text from the console and sends lines one-by-one to its add-ins. So, here is the relevant code from Program.cs

static void Main(string[] args)
{
    Application app = new Application();
    LoadAddIns(app);

    string line = Console.ReadLine();
    while (line != "quit")
    {
        app.OnReadLine(line);
        line = Console.ReadLine();
    }

    UnloadAddIns();
}

Obviously, the Host application knows to do two things – load add-ins and quit. The real functionality though is not contained in ConsoleApp.Host project – it is the add-ins that do make this application interesting. The add-ins, however, do need an Object Model they could talk to.

So, let’s first talk about the OM exposed to the add-ins. It consists of interfaces and is located in Interfaces.cs file in ConsoleApp.Programmability project. The most important interface there is IApplication interface. This interface is handed to the add-in as part of the initial hand-shake which is happening when add-in is loaded and its implementation of IEntryPoint.Startup is called by the host (see the ConsoleApp.Host.Program.LoadAddIn method)

Here is how the IApplication interface looks like:

    /// <summary>
    /// This interface represents the top-level object in the host's OM
    /// It will be passed to the add-in during add-ins initialization
    /// </summary>
    [ComImport]
    [Guid("2885E362-5C29-4481-89DD-1E58F3770132")]
    public interface IApplication
    {
        int Version { get; }

        //
        // methods available in version 1
        //

        void WriteLine(string message);
        // TODO: we should have an event here instead of 
        // TODO: Register/Unregister pattern
        // TODO: but compiler currently errors out for events
        // public event ReadLineEventHandler ReadLine; 
        void RegisterReadLineCallback(IReadLineCallback callback);
        void UnregisterReadLineCallback(IReadLineCallback callback);

        //
        // methods available in version 2
        //

#if VERSION2
        ConsoleColor ForegroundColor { get; set; }
#endif
    }

There are a couple of things to note. Take a look at the toolbar and take a notice of the active configuration for your solution.

Your active configuration should be set to V1. If this is the case you will see ForegroundColor property appear grayed out since this functionality will only be available at Version 2 of the Host Applicaiton.

Other than that IApplication exposes quite simple functionality. In Version1 of the host application add-ins can ask the host application for its version number (IApplication.Version property) , write messages using IApplication.WriteLine method and also subscribe to read-line callbacks using RegisterReadLineCallback/UnregisterReadLineCallback methods.

We also are trying to enable locally embedding types from ConsoleApp.Programmability assembly exactly the same way types contained in Interop Assemblies are embedded for “No Primary Interop Assemblies” scenarios. To achieve this we need to make ConsoleApp.Programmability assembly looks like an Interop Assembly (remember that compilers only embed types from Interop Assemblies) by placing attributes in several key spots throughout the code in Interfaces.cs. In particular, each interface needs to be attributed with ComImport and Guid attributes. In addition ConsoleApp.Programmability assembly does have assembly level Guid and ImportedFromTypeLib attributes:

[assembly:Guid("35E86F71-1587-4453-A8AB-65A6EA5EAE42")]
[assembly:ImportedFromTypeLib("ConsoleApp.Programmability")]

Next thing is to actually take a look at the ConsoleApp.AddIn.Echo project. This project represents an add-in. ConsoleApp’s add-ins do need to implement IEntryPoint interface and they also need to reply to IReadLineCallback calls coming from the host. The implementation for the add-in is very minimal and looks this:

namespace ConsoleApp.AddIn.Echo
{
    public class AddIn : IEntryPoint, IReadLineCallback
   {
        IApplication app;

        public void Startup(IApplication app)
        {
            this.app = app;

            app.WriteLine("AddIn.Echo is starting up ... ");
            app.WriteLine("    Host object type - "+ app.GetType().AssemblyQualifiedName);
            app.WriteLine("    Entry point type - "+ typeof(IEntryPoint).AssemblyQualifiedName);

            app.RegisterReadLineCallback(this);
        }

        public void Shutdown()
        {
            app.WriteLine("AddIn is shutting down ... ");
        }

        public void ReadLine(IReadLineEventArgs args)
        {
            string message = args.Message;

            if (message == null)
                return;

            string echoCmd = "echo ";

            if (message.StartsWith(echoCmd))
            {
                app.WriteLine(message.Substring(echoCmd.Length));
            }
        }
    }
}

Probably no special explanation is required for this code – notice though that ReadLine callback recognizes lines starting with ‘echo ‘ and prints back the rest of the line.

Also notice, that the reference to ConsoleApp.Programmabilty assembly is marked with “Embed Interop Types”=”true” (aka the EIT switch)

So, now is the time to compile the solution. For Version1 configuration this will result in 3 projects being compiled – ConsoleApp.Programmability, ConsoleApp.Host and ConsoleApp.AddIn.Echo when both ConsoleApp.Host and CosoleApp.AddIn.Echo do reference programmability assembly using the EIT switch.

Now, if you run the application that was compiled you will notice that Echo add-in did get loaded and is running and functioning (just type ‘echo Hello PDC’ to see the echo command in action). Also, by attaching the debugger and checking out the Modules window, you can verify that ConsoleApp.Programmability assembly is not even loaded into the process! This means that host and add-in can talk to each other w/o sharing a common type!

The secret here is the new CLR 4.0 Type Equivalence support. During the compilation local copies of types contained in programmability assembly were embedded into both ConsoleApp.Host and ConsoleApp.AddIn.Echo assemblies and the host and the add-in are able to talk to each other using those local copies of IEntryPoint and IApplication interfaces. Since both copies do have the same Guid attribute – they are considered to be equivalent by the CLR. Casts and calls through equivalent interfaces are executed as if they are casts/calls on the interface the object derives from!

Next step in the demo is to change the solution configuration option to Version2.

Switching to Version2 configuration compiles the Host and Programmability projects with VERSION2 conditional compilation symbol defined. If you search through the source code for the occurrences of “#if VERSION2” directives throughout the solution you will see a number of places where v2 of the ConsoleApp.Host is different from the previous version.

In particular, Host’s OM now has new API – IApplication.ForegroundColor. Notice, that we have added this new property at the end of the existing interface. The placement at the end  is very important and is done in order to maintain IApplication’s vtable compatibility with IApplication interface from v1 (IMPORTANT: you can only add new methods at the end of the existing interface and never-never in the middle or at the beginning)

Another interesting place to look at is the implementation of IApplication.Version property. v1 implementation of of Application.Host returns value of 1 here, while v2 implementation returns value of 2. This is done to allow add-ins to perform “adaptive coding”. In particular, if they know the host is of a lower version than expected they could “disable” some of its functionality. e.g. here is the adaptive code from ConsoleApp.AddIn.Color add-in:

if (app.Version < 2)
{
    app.WriteLine("'color' command is not supported on this version of the ConsoleApp");
    return;
}

Now, compile Version2 of the host and notice that Color add-in is loaded and can execute various color commands.

Again, using the debugger you can easily verify that since Application.Programmability assembly has been linked in using the EIT flag – this assembly is actually not loaded into the memory.

So, next step is to try version resiliency in action. What I suggest you is to manually copy the ConsoleApp.AddIn.Echo.dll from Version1\addins folder to Version2\addins folder and run the Version2 of the host. You will notice that Echo add-in is loaded alongside the Color add-in and is functioning as expected. This shows that the v2 application is backward compatible and can load add-ins compiled against the old version of the OM.

Now, lets try to go in the other direction and now copy ConsoleApp.AddIn.Color.dll from Version2\addins folder to Version1\addins folder and start the v1 version of the host application. You will again notice that both Color and Echo add-ins were loaded, but when trying to execute the color command – we will get an error saying "'color' command is not supported on this version of the ConsoleApp”. This is adaptive coding recognize the color command being executed on a version of Host where it is not applicable.

Below is the screenshot that I took. It demonstrated that add-ins compiled against future version of host OM can still be executed on previous versions of the host application. This is known as forward compatibility and is possible in the VS 10 CTP version of the CLR because of the Type Equivalence support.

I hope you followed this demo through to this point. We have seen how Type Equivalence enables loose type coupling and version resiliency for managed applications. From here it should be pretty trivial to imply how easier the managed extensions deployment experience has become.

I also encourage you to use tools like ildasm, reflector and managed debugger to better understand how the whole thing does fit together. I also will be glad to hear feedback /answer question that are posted as comments to this post.

COM Interop: Handling events has side effects

From time to time Eric Carter shares with me some issues customers run into and wonders whether we can do something about it. The one I am going to talk about is quite interesting issue on its own AND I will talk about a solution we propose to this problem in my PDC session. So, if you like this stuff you should enjoy my PDC session as well.

So, here is the slightly edited thread.

From: Eric Carter
To: Misha Shneerson

I’m out in North Carolina with support [engineers – M.S.] and they showed me this issue—incredible.  Build a managed add-in for Excel and sync application events.  Now try to embed any excel workbook into Word.  Result—Word crashes.

http://blogs.msdn.com/vsofficedeveloper/archive/2008/04/11/Excel-OLE-Embedded-Error-With-Managed-Addin-Summary.aspx

From: Misha Shneerson
To: Eric Carter

I did not know about this particular issue but , yep, I am very well aware that the way events are implemented in PIAs is problematic. The main problem is caused by the fact that there is side-effect to using PIAs eventing support which I call “ghost RCWs” -  users do not see those RCW in their code but they are still created behind the scenes. To clarify what I mean - suppose Excel.Application exposes WorkbookOpen(Worbook) and WorkbookClose(Workbook) events (I am intentionally simplifying it here to only 2 imaginary evens off Excel’s Application). In my add-in code I only define an event handler for WorkbookOpen and in the handler itself I will call Marshal.RCO on the object to make sure there is no RCW hanging around. Sounds like I should not have outstanding RCWs wrapping the Workbook objects, right? Surprisingly, an RCW for the Workbook object will be re-created for me when WorkbookClose is fired even though I am not handling this event! What happens behind the scenes is that PIA’s eventing support creates an event sink which handles both WorkbookOpen and WorkbookClose but only WorkbookOpen calls the user’s delegate method. It only seems that WorkbookClose does nothing – but there is a side effect from the fact that this stub is defined at all – CLR still needs to marshal the parameters and that is how “ghost” RCWs are created.

Notice, that defining event handlers for both WorkbookOpen and WorkbookClose is not going to help the situation either because PIAs eventing support will create a separate sink object to handle WorkbookOpen (with side effect of marshalling COM object to RCW when WorkbookClose is called) and another sink object for WorkbookClose (with side effect of marshalling COM object to RCW when WorkbookOpen is called) .

….

For the rest of the thread I am describing how we are going to fix this. Fixing events for COM objects is relatively just a small part of the improvement CLR and Visual Studio are making to managed type system and COM interop. So, come and learn the bigger picture.

PDC Talk – Under the hood: Advances in .NET Type System

Advances in .NET Type System – sound like this is a PDC session one must come to. Here is the talks abstract:

Under the Hood: Advances in the .NET Type System

Enhancements to the type system in the next version of .NET Framework allow for loose type-coupling of components comprising your application. This talk is an in-depth examination of the changes in the Common Language Runtime and managed languages. See how these changes help to simplify versioning and deployment of components targeting either COM based and/or fully managed applications. For Office developers, learn how to eliminate the need to redistribute primary interop assemblies.

I will definitely be there – actually I am the speaker for this talk so I do not have much of a choice :) But the rest of you out there coming to PDC 2008 do have a lot of sessions to choose from. There are some very respected speakers that you just can not miss – the guys that are brand names of their own - Anders Hejlsberg (Future of C#), Scott Hanselman (Microsoft .NET Framework: Overview and Applications for Babies), Raymond Chen. But sometimes there are these seemingly little sessions where content just rocks.

The “Advances in .NET type system” is this one little gem that you can not overlook. Although I do not pretend to be the greatest speaker in the world – give me a good content and I will rock the house. Given I have been working for the past 9 months on improving the story I am going to tell – this will be very very very very very (how many more times should I say that) cool session. So, come on, register for this talk and you will be glad you came.

Targeting multiple versions of Office without PIAs

Having to rely on the correct version of PIAs to be deployed for implementing Office add-ins has become the inevitable reality for many Office devs. This reality is quite gruesome, I must add.

Andrew Whitechapel describes a technique allowing single add-in to target both Office 2003 and Office 2007 by importing interop definitions that are new in Office 2007 PIAs directly into your assembly. This allows you to use Office 2007 features w/o depending on Office 2007 PIAs. It is a good start to become PIA free.

Tell me your story. What has been your experience with PIAs and limitations that those impose on you?

Deploying your VSTO 2008 Add-In to All Users (Part III)

This post is both the continuation of part I and part II installments but it also addresses new product that has shipped since VSTO 2005 SE and this is VSTO 2008 (which is also known as VSTO 3.0 - see the complete matrix of VSTOs in this post by Andrew Whitechapel ).

Here is a quick recap of the last 2 episodes:
Traditionally Microsoft Office has allowed All Users installation of COM Add-Ins by the virtue of registering the add-in under HKLM\Software\Microsoft\Office\<Application>\AddIns registry key. There were few problems with managing HKLM add-ins - in particular non-admin users could not disable an add-in explicitly as well as crashing add-ins could not have been disabled automatically for such users. So, when managed add-ins were introduced in Office 2007 it was decided to break the cycle and disallow registering managed add-in under HKLM.

Today only those managed add-ins registered under HKCU registry hive are activated by Office 2007. This creates a different kind of problem - "how an administrator can deploy an Office add-in so that every user on the machine can use it?".

The part I post introduced an internal Office 2007 registry replication mechanism. The very sketchy part II post described, at a very high level, how to take advantage of the registry replication and modify the VSTO 2005 SE add-ins setup projects to be deployable to all users (The technique is a mixture of moving registry keys around in the deployment project itself coupled with creating custom actions similarly to those described in Darryn's VSTO 2005 SE deployment article). As a side note: I really should have invested more time into part II and make it a full walkthrough article. Now I am paying for my laziness since I need to answer a lot of "please clarify" comments on that post. But hey, on the flip side, trying to connect the sketchy pieces together and build the nice deployment package does build character - doesn't it?

If you are just learning this stuff - please understand there is no quick fix solution and before trying what I am going to describe below - please make sure to read all 4 articles I just mentioned: start with the deployment article (which by itself consists of 2 articles, then post I and post II).

VSTO 2008 significantly improves over VSTO2005SE deployment experience by introducing ClickOnce based deployment. The idea is that, as it is the case for ClickOnce deployment of Windows Forms project, you can now "publish" VSTO 2008 projects. This will create a setup.exe file you can give to your users to deploy your VSTO add-in.

The problem is that ClickOnce is also a per-user solution. Notice that there is no such thing as ClickOnce for All Users. And, if you are an administrator running a server and you want to deploy VSTO solution to all users on your server - you really need an MSI based setup project. This is, of course, not something we do anymore in VSTO 2008.

Let's see what can be done about it though. First of all, go and take a look into samples at http://code.msdn.microsoft.com/VSTO3MSI. These samples show how to create an MSI based deployment package for VSTO 3.0. The samples target single user - not All Users - but this is a very good start.

Key difference between VSTO 3.0 MSIs and VSTO 2.0 MSIs is that security model has changed. In VSTO 3.0 we no longer rely on CAS for security - both us and our users did have their share of problems with CAS so enough is enough. This was replaced with a concept of "Inclusion List" based security. Consequently, one of the things you will find in the site above is an InclusionListCustomActions - custom actions that call InclusionList APIs to pre-trust your solution.

So, now supposedly you have followed the VSTO3MSI samples and created an MSI based deployment package.

Next step for you is to follow general guidelines in the part II article and modify this solution to use Office's replication mechanism for All User deployment by moving registry entries from HKCU under HKLM\Software\Microsoft\Office\12.0\ User Settings\MyCompany.MyAddIn\Create registry key. Also, same post describes how to add a custom action to set up Count registry value the replication mechanism is based in and as well as handling the uninstallation of the add-in.

Next step is to remove the InclusionListCustomActions - these CAs are not good for All Users deployment.

But what do you need to do instead? Read on.

“Inclusion list” is just reg entries under HKEY_CURRENT_USER\Software\Microsoft\VSTO\Security\Inclusion – notice it is under HKCU not HKLM. The format of those registry entries is pretty self-describing – we use some random GUID to create a unique entry in the list, the entry contains the path to add-ins .vsto file and manifest's public key. Although "inclusion list" APIs create these registry entries for you but only under HKCU hive. That is not good for all users deployment. Instead your setup needs to install those registry keys into HKLM\Software\Microsoft\Office\12.0\ User Settings\MyCompany.MyAddIn\Create\Software\Microsoft\VSTO\Security\Inclusion and rely on Office’s replication mechanism to copy those into HKCU.

So, first go to HKEY_CURRENT_USER\Software\Microsoft\VSTO\Security\Inclusion and find the entry in the list that pre-trusts your solution. Export this entry into a .reg file. This file should look like this:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\VSTO\Security\Inclusion\30b8e0dd-8ff5-4828-a53b-4933424888fd]
"Url"="file:<SOMEPATH>/ExcelWorkbook1.vsto"
"PublicKey"="<RSAKeyValue><Modulus>nPAT3uo/l/ba+L74Am8cHuxNwe50oXwpVgdKyKQOjskBkZWMMb8vcFzN91NxMj3p7CehgQeGZNuuy64wmvwiFRRq20lKXca3Iv7dkWgED6rkG20EGp4je0E1LrxdwYQg5tj5OO0gn4+nXR201tBy2BuqV1hI1ydxCPNZ+jDX+Gk=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>"

Next, change the highlighted text to

HKEY_LOCAL_MACHINE\Software\Microsoft\Office\12.0\User Settings\<MyCompany.MyAddIn>\Create

Next, change <SOMEPATH>\    to     [TARGETDIR]  and save .reg file.

Your .reg file now should look something like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\Software\Microsoft\Office\12.0\User Settings\<MyCompany.MyAddIn>\Create\Software\Microsoft\VSTO\Security\Inclusion\30b8e0dd-8ff5-4828-a53b-4933424888fd]
"Url"="file:[TARGETDIR]ExcelWorkbook1.vsto"
"PublicKey"="<RSAKeyValue><Modulus>nPAT3uo/l/ba+L74Am8cHuxNwe50oXwpVgdKyKQOjskBkZWMMb8vcFzN91NxMj3p7CehgQeGZNuuy64wmvwiFRRq20lKXca3Iv7dkWgED6rkG20EGp4je0E1LrxdwYQg5tj5OO0gn4+nXR201tBy2BuqV1hI1ydxCPNZ+jDX+Gk=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>"

Go into your Visual Studio deployment project, open the Registry View, right click on "Registry on Target Machine" root node in the view and click "Import ..." to import the .reg file.

That's it - you should now have the functioning MSI based deployment package for VSTO 3.0 solution. Use it!

As usual, the post is not a complete walkthrgough and might be sketchy in parts - so if you have something to add or you have found a mistake in the post - please leave a comment.

Preserving transparency when rendering Office icons

Today I have a guest writer on my blog - Eric Faller.  Eric shows the correct way to render Office icons returned by GetImageMso API. I got involved into this by replying to this post in our forums but the explanation just would not fit into a regular forums post - so here we go.

-----------------------------------------------------------------------------------------

This is a follow-up to Andrew’s post about converting between the image formats used by Office and the .NET framework.  I’ll be talking about handling the alpha channel (transparency) of the images, mentioned at the end of that post and in the comments. I’d recommend reading that post first in order to get up to speed on the IPictureDisp interface and some of the other concepts we’ll be discussing.

I’d also recommend reading the RibbonX Image FAQ on Jensen Harris’ blog. It has a lengthy discussion about the different formats Office has used for image transparency in the past, as well as some common pitfalls when loading images into Office. In this post I’ll be talking about getting images out of Office, but many of the problems will be similar (DDB vs DIB, etc.).

Office 2007 introduces a new API for fetching icon images, the GetImageMso function on the CommandBars object. It takes the ID of a Ribbon control and returns its icon in IPictureDisp format. You can use one of the many methods discussed in Andrew’s previous post to convert these objects into .NET-friendly System.Drawing.Bitmap objects.

If you do, you might notice that the icons don’t look exactly correct when you draw them – the transparent edges show up white and shadow elements look black. For example here’s what the “Paste” icon looks like if drawn on a WinForm:

If you’re only using the smaller versions of the icons (16x16), drawing them on a white background, and don’t care too deeply about pixel-perfect visuals, you might be OK with this. Calling Bitmap.MakeTransparent on the icon will help get rid of the white border, but it’s still not quite perfect.

The bad news is that if you want to stick with purely .NET code, you’re stuck with this – that’s the best that your icon can look. The problem is that the alpha channel has already been lost during the conversion from IPictureDisp to System.Drawing.Bitmap.

The CLR and GDI+ internally call Win32 GDI functions during the conversion, and these functions are not alpha channel-aware. GDI itself was written long before alpha channels became popular, and as a result almost all of the standard Win32 GDI functions will ignore the alpha channel and appear to “throw it away” during various copy and conversion operations.  Alpha channel support was only added with the AlphaBlend function in Windows 98/2000 with the addition of MSIMG32.DLL.

The good news is that we can get a lot better transparency in our images if we’re willing to do a little native code interop and call AlphaBlend ourselves. It’s slightly complicated, so I’ll just show you the code and then explain it. Here’s a function that will convert an IPictureDisp object to a System.Drawing.Bitmap object, using the AlphaBlend function:

public static Bitmap ConvertWithAlphaBlend(IPictureDisp ipd)

{

    // get the info about the HBITMAP inside the IPictureDisp

    DIBSECTION dibsection = new DIBSECTION();

    GetObjectDIBSection((IntPtr)ipd.Handle, Marshal.SizeOf(dibsection), ref dibsection);

    int width = dibsection.dsBm.bmWidth;

    int height = dibsection.dsBm.bmHeight;

    // zero out the RGB values for all pixels with A == 0

    // (AlphaBlend expects them to all be zero)

    unsafe

    {

        RGBQUAD* pBits = (RGBQUAD*)(void*)dibsection.dsBm.bmBits;

        for (int x = 0; x < dibsection.dsBmih.biWidth; x++)

            for (int y = 0; y < dibsection.dsBmih.biHeight; y++)

            {

                int offset = y * dibsection.dsBmih.biWidth + x;

                if (pBits[offset].rgbReserved == 0)

                {

                    pBits[offset].rgbRed = 0;

                    pBits[offset].rgbGreen = 0;

                    pBits[offset].rgbBlue = 0;

                }

            }

    }

    // create the destination Bitmap object

    Bitmap bitmap = new Bitmap(width, height, PixelFormat.Format32bppArgb);

    // get the HDCs and select the HBITMAP

    Graphics graphics = Graphics.FromImage(bitmap);

    IntPtr hdcDest = graphics.GetHdc();

    IntPtr hdcSrc = CreateCompatibleDC(hdcDest);

    IntPtr hobjOriginal = SelectObject(hdcSrc, (IntPtr)ipd.Handle);

    // render the bitmap using AlphaBlend

    BLENDFUNCTION blendfunction = new BLENDFUNCTION(AC_SRC_OVER, 0, 0xFF, AC_SRC_ALPHA);

    AlphaBlend(hdcDest, 0, 0, width, height, hdcSrc, 0, 0, width, height, blendfunction);

    // clean up

    SelectObject(hdcSrc, hobjOriginal);

    DeleteDC(hdcSrc);

    graphics.ReleaseHdc(hdcDest);

    graphics.Dispose();

    return bitmap;

}

Except for the “unsafe” block, the code should be pretty straightforward if you’re a Win32 GDI programmer: we create a new blank 32-bit HDC from a Bitmap object, create a compatible HDC to select the IPictureDisp’s HBITMAP into, render it with AlphaBlend, and clean up.

Now we need to look at the pixel manipulations inside the “unsafe” block. If we leave that section out, this is what we would get:

This is better – the shadow inside if the icon doesn’t look as bad, but we still have the white border in the regions of the icon that are completely transparent.

The problem happens because of an ambiguity that occurs when a pixel is completely transparent. In this case the A (‘alpha’) component of the pixel is zero, but the R, G and B components of the pixel can be anything since they don’t show up. What actually happens with those values is dependent on the convention that you follow. Unfortunately, Office follows a different convention than the AlphaBlend function does. The AlphaBlend function expects the RGB values to all be zero if the A value is zero. Office leaves the R, G and B values all equal to 255, which creates the white color seen in the images above. It does this so that the transparent pixels don’t turn out black if the image is “compacted” by GDI+ or the CLR, leaving us with images that look like this by default, which is even worse than what we started with:

Fortunately we can convert between the two conventions for the completely transparent pixels by checking for zero A values and zeroing out the RGB values. It takes some unsafe code to do it, but it works. Here’s how it looks:

It looks a lot better, but if you look carefully, it’s still not perfect. The shadow has been “halftoned”: all of the alpha values have been rounded to either 0 or 255, making the shadow either completely transparent or completely black. We want a nice gray gradient shadow.  It looks like the problem happens in the Bitmap object, when converting to and from the HDC. If you skip the intermediate Bitmap object and use the above code to draw directly to a Graphics object on a window, then it will render properly. I’ve played around with the PixelFormat, CompositingMode, and other parameters to the Graphics and Bitmap objects, but haven’t been able to make it work.

It looks like we’ll have to give up on using AlphaBlend and go down to the lowest level: pixel-by-pixel copying.  Since we were already doing per-pixel processing in the previous function, the new one actually looks simpler:

public static Bitmap ConvertPixelByPixel(IPictureDisp ipd)

{

    // get the info about the HBITMAP inside the IPictureDisp

    DIBSECTION dibsection = new DIBSECTION();

    GetObjectDIBSection((IntPtr)ipd.Handle, Marshal.SizeOf(dibsection), ref dibsection);

    int width = dibsection.dsBm.bmWidth;

    int height = dibsection.dsBm.bmHeight;

    // create the destination Bitmap object

    Bitmap bitmap = new Bitmap(width, height, PixelFormat.Format32bppArgb);

    unsafe

    {

        // get a pointer to the raw bits

        RGBQUAD* pBits = (RGBQUAD*)(void*)dibsection.dsBm.bmBits;

        // copy each pixel manually

        for (int x = 0; x < dibsection.dsBmih.biWidth; x++)

            for (int y = 0; y < dibsection.dsBmih.biHeight; y++)

            {

                int offset = y * dibsection.dsBmih.biWidth + x;

                if (pBits[offset].rgbReserved != 0)

                {

                    bitmap.SetPixel(x, y, Color.FromArgb(pBits[offset].rgbReserved, pBits[offset].rgbRed, pBits[offset].rgbGreen, pBits[offset].rgbBlue));

                }

            }

    }

    return bitmap;

}

Here’s what the final pixel-perfect result looks like:

The final question you should have now is "where can I see the complete source code?". Easy. See attached DisplayIconAddIn.zip to get a shared add-in that demos this concept.

Deploying your VSTO Add-In to All Users (Part II)

In this post I am going to discuss how the observations made in the previous post can be incorporated into a setup project for Office 2007 Add-Ins. As with any setup we will need to address installation, un-installation and repair procedures.

• Upon the installation of Add-In we will write registry keys that are similar to what testpropagation_create.reg file from the previous post contained. This installation will contain the Create instruction telling Office propagation mechanism to copy the registry key into HKCU profile.
• During the uninstall, we will replace the Create instruction with Delete instruction which will cause Office to delete the corresponding registry key in HKCU hive. We will also bump up Count registry value to tell Office that the instruction needs to be carried out.
• During the repair we will need to increase Count value. This will cause Office to copy over the registry values from HKLM into HKCU.

Let’s first take care of the installation of the required registry keys. The easiest way to achieve this is by slightly modifying the default setup project that is created alongside the regular VSTO 2005 SE project. Let’s create a new Excel 2007 Add-In called “MyAddIn”. You will notice that in addition VSTO has also created a companion MyAddInSetup project. Let’s open the Registry view and clear out all the registry entries that have been created for you (i.e. completely clear the contents under HKEY_CURRENT_USER). Now we are going to import the registry information as presented below. To achieve this you will need to copy the below lines into a .reg file, then right-click on “Registry on Target Machine” node, select “Import …” and import the .reg file you have just created (notice that first you might want to adjust the highlighted strings for your particular situation).

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings\MyCompany.MyAddIn\Create\Software\Microsoft\Office\Excel\Addins\MyCompany.MyAddIn]

"Description"="MyAddIn -- an addin created with VSTO technology"
"Manifest"="[TARGETDIR]MyAddIn.dll.manifest"
"FriendlyName"="MyAddIn"
"LoadBehavior"=dword:00000003
"CommandLineSafe"=dword:00000001

After you complete this operation your Registry View should be very similar to this screenshot :  

By adding above entries into Registry View we ensure that corresponding keys will be removed when Add-In is uninstalled. I would also suggest to set Create's key “DeleteOnUninstall” property in the Properties Window to True to make sure it is always deleted on uninstall .

Notice that now we need to add a Count value under the MyCompany.MyAddIn key. Using the Registry View to add this value would not work because any registry keys and values added through this view will be removed on uninstall. On opposite, we need Count value not just stay when Add-In is uninstalled, but its value should be incremented.

To achieve the desired effect we will create a Custom Action that will add Count value or, in case it already exists, we will simply increment its value.

Similarly, Custom Action will create a "Delete" instruction when Add-In is uninstalled.

Below I am showing code that Darryn Lavery (who put tremendous amount of effort to design and validate all I am talking about right now) has already written for his future MSDN article on this (yeah, there will be an MSDN article with the full how-to instructions) and he kindly shared it with me and I shamelessly sharing it you, but credit sitll belongs with Darryn.

Now let's move close and see how the particular Custom Action code could look like. First, let's start defining a class RegisterOffice2007AddIn with a couple of private methods:

The code above contains helper method IncrementCounter that is responsible for correctly updating the Count registry value. GetApplicationPath helper method returns application-specific path Add-Ins registration key. Now, we are ready to move to the top-level function that will be called during Install and Repair:

In the above method, we first make sure the "Delete" instruction is gone and then we increment the Counter value. Notice that "Create" instruction is not explicitly installed by the Custom Action - this is handled by the installer automatically because of our previous work with the Register View.

And, finally, the function that will be called during uninstall:

This is pretty much it. Assuming you have already experimented with SetSecurity Custom Action you should be able to wrap this code into a Custom Action DLL, set CustomActionData properties as /addinName="MyCompany.MyAddIn" /application="Excel" (again choose the value that fit your particular case), and invoke the above methods on Install, Uninstall and Rollback. I will try to sum it all up in another post though.

Deploying your VSTO Add-In to All Users (Part I)

VSTO Add-Ins aka Managed Office Add-Ins have a major deficiency on the deployment side. Putting it simple, Microsoft has only provided guidance how to deploy these Add-Ins on per-user basis. Machine wide deployment has been our Achilles heel. In this post I will try to explain how this limitation can be worked around.

First, let's start with some background information.

Office 2007 has added built-in support for managed Add-Ins. Office applications use a Manifest registry value to differentiate between traditional COM Add-Ins and managed Add-Ins. This value can be found under HKCU\Software\Microsoft\Office\<App>\AddIns\<AddInName>.

Unlike traditional COM Add-Ins which can be deployed to all users on a machine by registering those under HKLM\Software\Microsoft\Office\<App>\AddIns, managed Add-Ins can only be registered in HKCU registry hive (those registered under HKLM will be ignored).

It is pretty easy to create a setup package that writes registry keys into HKCU for the user invoking the setup. But setup program duplicating same sets of registry keys for every user on this particular machine requires quite advanced skills in Win32 API. Slightly more advanced skill is required to create a setup that will write those registry entries into HKCU hives of future users of the machine (this is because "template hive" is stored in C:\Documents and Settings\Default User\ntuser.dat file and is not a part of live registry, see more details at Raymond Chen's post on the topic).

There is a solution though and it is reasonably easy one. The trick is to use an internal Office mechanism for propagating registry keys from HKLM to HKCU that is performed during startup of every Office application.

For Office 2007 the magic is in the keys located under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings. If you have an existing installation of Office 2007 you can examine the content of this registry key you will soon find out that it contains a seemingly random collection of sub-keys which all in turn have either Create or Delete sub-keys. The Create/Delete keys (which are essentially Copy/Delete instructions), in turn, have sub-keys that look like complete registry keys on their own!

What we are looking at is the HKLM-to-HKCU propagation mechanism that can be completely controlled from within the registry itself. Let's examine more closely how this mechanism is working. To start I would suggest the following exercise – copy and paste the below lines into a Notepad application, save the file as testpropagation_create.reg file and run this file to put the corresponding registry keys and values into your registry. Notice that you are adding registry keys to HKLM hive.

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings\TestPropagation] "Count"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings\TestPropagation\Create\Software\Microsoft\Office\TestKey]
"TestValue"="Test"

Now start Excel application. Examine the registry keys in HKCU hive e.g. you will find two interesting registry keys that appear under your HKCU hive:

• HKCU\Software\Microsoft\Office\TestKey registry key containing registry value TestValue
• You now also have HKCU\Software\Microsoft\Office\12.0\User Settings\TestPropagation registry key with Count value set to 1

Now, let's see how we can delete a registry key using similar mechanism. Below is the new registry script for you to run (copy&paste these lines into testpropagation_delete.reg)

Windows Registry Editor Version 5.00
[-HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings\TestPropagation\Create] [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings\TestPropagation]
"Count"=dword:00000002
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0\User Settings\TestPropagation\Delete\Software\Microsoft\Office\TestKey]

I highlighted the changes I made from the testpropagation_create.reg file. In this script we first remove the TestPropagation\Create registry key that we added earlier – notice the hyphen at the beginning of the line which indicate that this is a "delete" instruction. We also changed the Count value to 2 – in order for the instruction to execute we need to make sure that HKLM's Count value is different from HKCU's Count value. Next, Software\Microsoft\Office\TestKey is now placed under TestPropagation\Delete key – this to instructs Office to delete the registry key rather than create it.

After executing testpropagation_delete.reg file and starting Excel you will notice that:

• HKCU\Software\Microsoft\Office\TestKey registry key is now gone
• HKCU\Software\Microsoft\Office\12.0\User Settings\TestPropagation registry key has a Count value set to 2

Now, we have seen how Office's registry propagation mechanism works. It now becomes pretty clear how it is possible to take advantage of this behavior in your Add-In's setup. In my next post I will explain how we would go about building a custom action for VSTO 2005 SE Add-In setup package that will use this technique for machine-wide Add-In deployment.

VSTO 2005 SE runtime update is available

I have promised on several occasions that I will make a post on my blog as soon as an update to VSTO 2005 SE runtime becomes available. Unfortunately when this finally happened I was on vacation and could not immediately post this. My appologies for such bad timing.

Now it is the time to make amends. Please install the new runtime from here if you are still waiting for the fixes to couple of nagging problems like the bug when VSTO add-ins being disabled if Outlook is started in head-less mode (e.g. through automation).

UPDATE (08/31/2007):

As a result of the above upgrade the VSTO2005 SE Windows Installer deployment article has been also updated with the following changes.

• Vista
  Advice is given to help make your solution setup Vista User Access Control (UAC) compliant. This includes new bootstrapper packages for the VSTO 2005 SE runtime and VSTO 2005 language pack.
• Visual Studio 2008 Beta 2 Support
  Office 2003 projects in Visual Studio 2008 Beta 2 continue to use the VSTO 2005 SE deployment and security model. The article has been updated to include instructions on deploying Office 2003 solutions using  Visual Studio 2008.
  Notice that Office 2007 projects in Visual Studio 2008 will use ClickOnce as the deployment & security model.
• Updated detection logic for build 891 of the VSTO 2005 SE runtime
  The bootstrapper package for the 2005 SE runtime is updated to detect the latest version of the runtime. If earlier versions of the runtime are installed, the bootstrapper package will install the latest version.
  If your MSI has a launch condition to search for the latest version of the runtime , then you can update this launch condition to search for HKLM\Software\Microsoft\vsto runtime Setup\v2.0.50727 where Update  >= 2.

The deployment article is available in the usual place

Update for VSTO 2005 SE released

We have recently released an update for VSTO 2005 SE design time support on Vista. The KB article is available here and does not explicitly say what exactly was broken or what fix has been made – it simply states the symptoms.

On a Windows Vista-based computer, add-ins that are created by using Microsoft Visual Studio 2005 Tools for the Microsoft Office System Second Edition (VSTO 2005 SE) do not run in Microsoft Office 2003 Professional. This problem occurs when you try to debug one of these add-ins in Visual Studio 2005.

In KB article it is difficult to state that the issue in hand was quite interesting to deal with and reflects on certain aspects of deploying COM applications on Vista. So, let's start looking at the issue at hand and then I will generalize with more broad deployment insights.

First of all, Visual Studio writes certain registry keys when VSTO 2005 SE add-in is compiled. The goal of writing those registry keys is:

1. Add a registry entry that tells Office there is an add-in it needs to instantiate.
2. Register a COM component that Office will need to instantiate when initializing the add-in using standard CoCreateInstance API (this part is only true for Office 2003 since Office 2007 can natively recognize and instantiate VSTO add-ins).

Being good citizens we have decided that we do not want to require from our developers to run Visual Studio with administrative privileges hence those registry entries are all written under HKCU registry hive. Specifically the COM component related registry entries are written under HKCU\Software\Classes registry hive.

This registration model works quite well on Windows XP as well as on Windows Vista. The problems begin when the hosting process (e.g. Outlook, Excel, Word etc) is being run with elevated privileges or when User Access Control (UAC) is turned off. On Windows Vista this means that process receives High Integration Level. Junfeng Zhang points that on Vista COM is blind to HKCU\Software\Classes hive when IR is greater than MEDIUM. So, VSTO add-ins failed to run when host processes where executed in elevated mode.

Now, why would anyone run Excel in elevated mode while debugging it? The problem is that elevation happens unintentionally – the recommendation for running Visual Studio 2005 on Vista is to elevate it. Hence, any processes spawn by Visual Studio 2005 (which happens when you process F5 or CTRL+F5) and lack Vista activation manifests (and Office 2003 is old enough to have no idea what these manifests are) will be executed in elevated mode as well.

So, back to the update for VSTO 2005 SE this article started with. To address the problems above we have changed the location we are writing the COM-related registry keys to. During compilation we will attempt to put the registry keys under HKLM\Software\Classes and, if it fails due to insufficient privilege, will resort back to HKCU\Software\Classes. That's all this update is about.

Few words of caution, we did not actually make any modifications to the VSTO add-in satellite setup project. The way we still construct the setup project – it will always write COM-related registry keys under HKCU\Software\Classes. This may break the add-in on Vista for Office 2003 applications for the reasons described above. Hence you might want to modify what you get by default and move the COM-related registry keys under HKLM hive.

Additionally, I would caution those who use Windows Installer against putting their registry keys under HKEY_CLASSES_ROOT (HKCR) hive in general. This hive is a virtual one and is essentially a merged HKCU\Software\Classes and HKLM\Software\Classes hives. Windows Installer chooses which one of the two hive to use depending on the type of the installation – it will write into HKLM for machine-wide installations and into HKCU for per-user installs. And as we now know, the latter will not work well on Vista when running in High Integrity Level.

VSTO 2005 SE deployment paper is published

The original deployment paper targeted deployment aspects of customizations built using VSTO 2005 release but since VSTO 2005 SE came out many people wondered what are the changes in the deployment, specifically the questions where around deploying the VSTO2005SE runtime prerequisite.

I am glad to announce that today the updated article has been published and can be found at this link - http://go.microsoft.com/fwlink/?LinkID=57779.