I paused over my final mince pie today, and pondered the psychology of Office developers (as one does). Office development is a curious mix of different programming philosophies that has given rise to a number of interesting behavioral practices among developers. I picked the title for this post because one of the curious things is that VBA developers are in fact COM developers, though they might not know it.
Firstly, the Office system of applications – at least on the client – consists of a range of cohesive but different COM servers. COM is a well-understood system, which imposes a number of fairly strict constraints. Office apps mostly play by the COM rules – but sometimes don’t (more on this in a minute). Contrast that with the traditional Office developer tools – predominantly VBA. VBA exhibits a curious dichotomy: it is COM-aware, yet it hides most of the necessary COM plumbing so that the developer is largely oblivious to its COM aspects. You can also build Office-based solutions with languages at the other end of the spectrum (such as C and C++). In these languages, much more of the COM plumbing is exposed. It’s almost axiomatic that these languages force you to work at a lower level to get anything done, and also afford you much greater flexibility and power to do exactly what you want. With VBA, you’re stuck with whatever the VBA runtime decides to give you.
The idea is that you focus on the business requirements of your code, without worrying about the detailed underlying plumbing. The VBA developer knows what he/she wants to do, and writes just enough code to get it done. The VBA runtime takes the sometimes vague gestures passed to it by the developer and maps them to very specific COM plumbing. Think of the VBA runtime as a natty little Venus-to-Mars adapter.
For example, consider strong vs weak typing. Steve Ferg actually decomposes this into 3 pairs of contrasting categories. For a full discussion, see his excellent article comparing Python and Java here:
Every variable name is bound to a type at compile time
C#, VB, C++
A variable is bound only to an object not a type, so at runtime can be bound at different times to objects of different types.
Variables cannot be implicitly coerced to unrelated types, although they can be explicitly converted.
C#, VB, C++
Variables can be implicitly coerced to unrelated types (eg, from an int to a string).
Variable names must have explicit type declarations.
Variables do not need explicit type declarations.
So, VBA is a statically, weakly, implicitly typed language. What about COM? Well, if COM has a language it’s IDL. There is some COM implementation, of course, in a small set of RPC and OLE DLLs, but COM itself is primarily an interface-driven model. IDL of course is by definition an interface language (sorry, couldn’t resist). While IDL doesn’t deal with variables, it does deal with types – and IDL types are statically, strongly, explicitly typed. That is, apart from automation. Automation uses late binding – you can think of it as a dynamically typed implementation layer on top of COM (note, though, that it is not weakly nor implicitly typed). More on automation in a minute.
Now, there’s a long history of debate between the strong-typers and the weak-typers, so I won’t dwell on it here – just a little summary below:
The compiler helps you avoid or fix bugs early in the cycle.
Focusing on type declarations (syntactical correctness) makes you spend less time on the application’s semantic correctness.
The IDE, using the compiler to provide intellisense/autocomplete, helps you to avoid or fix bugs early in the cycle.
Relying on the compiler leads you into a false assumption that your code is correct.
Relying on the compiler allows you to write unit tests that focus on the application’s semantic correctness.
Without static compile-time type checking, you have to write unit tests that cover type correctness at runtime, which is both bad (more work) and good (forces you to write unit tests).
Without compiler support, each developer tends to invent his own wheel to workaround type limitations.
Errors that the compiler can find are usually simple errors that the developer can also find.
Allows you to overload methods on parameter type, where weak typing forces you to manually check parameter types within the method.
Lends itself to Test Driven Development, because it takes relatively little time to get the source code to compile before you can start running tests.
Makes it easy to navigate and search a codebase or object library.
Types are often version-specific, which makes modular systems difficult to maintain.
Reduces error handling when you build loosely-coupled systems.
Allows you to build loosely-coupled systems easily.
I’ll come back to the last two points in a minute. Before that, though, consider another curiosity in the evolution of Office development: the (sometimes extreme) resistance traditional Office developers put up when faced with the need to do work. Consider some of the limitations of the Office object models, which are catered for (and hidden) by the VBA runtime (and, optionally, by VB.NET), but cruelly exposed to C# developers:
· Passing ref parameters to Word (and the odd Outlook and Visio) methods.
· Optional parameters and positional parameters (all Office applications).
· Parameterized properties in Excel.
· Hidden accessor methods in Word.
All of these things are easier in VBA than they are in C#, but that is not an argument for weak typing, because they’re also easier in VB.NET which is strongly typed. So, if anything, it’s an argument for a runtime engine that will take care of all the fiddly details for you.
The most obvious place where languages like VBA are easier than languages like C# is in late binding. Which brings us back to automation. In VBA, if you want to launch Visio and open a Visio file, you can create a weakly-typed object and speculatively invoke its Open method, speculatively passing parameters of whatever types you think this method might take. What you’re saying is “I’m gambling that this object has an Open method, and I bet this method takes these kinds of parameters”. If you like, as an alternative, you can add a reference in your VBA project to the Visio typelib, and get the benefit of intellisense and autocomplete. In languages like C#, this is not optional – you must add a reference and you will only be allowed to call methods that are defined in the typelib, and only with parameter types defined in the typelib.
VB.NET supports late binding in a way that’s much the same as in VBA – largely transparent. However, this only works if you disable Option Strict (which, generally, you should not do) – otherwise, coding for late binding in VB.NET is the same as in C#. This requires you to use a technique that is essentially a subset of the reflection features of .NET.
One situation where Office forces you to use late binding is for Word and Excel built-in dialog boxes. Both Word and Excel offer a large number of internal dialog boxes, none of which are strongly typed. That is, there is a generic Excel::Dialog interface and a generic Word::Dialog interface that you must use for all built-in dialogs. Why is this? Well, the Excel and Word object models have evolved over many years, and implementing operations that are largely internal through late binding is a reasonable strategy for keeping changes to the object model to a minimum. COM and ‘changes to the OM’ are not good bedfellows (more later).
Let’s back up there a second. In Office development, VBA allows you to use late binding. It also allows you to bind using a typelib. Strongly typed languages force you to bind using a typelib. But, what is a typelib? It’s a representation of the definition of the interfaces in a given object model. Here’s the interesting thing: recall how you construct a COM server. Back in the mists of time, you probably used one of the ATL wizards, which generates some IDL and some C++ code. You mess with these a bit and then compile, which builds both a typelib from the IDL and a binary from the C++. What’s the connection between the typelib and the binary? Strictly speaking, none. The various project and project item wizards keep the IDL and C++ more or less in sync, and you can even embed your typelib in your DLL, but there’s no enforced connection. That is, you can easily publish a typelib that defines types (methods, properties, events) that bear absolutely no relation to what you’ve actually implemented in your COM server. So, when you religiously bind to a typelib, you’re taking it on faith that the typelib producer has correctly defined an interface that does actually map to the implementation in their COM server.
So what? People who build COM servers are not in the habit of deliberately mis-matching their typelibs, so isn’t the question academic? Well, think about versioning. COM rules state that an interface never versions. MAF rules state that a contract never versions. Office releases new versions every couple of years. How to reconcile these statements? When Office releases a new version of Excel, does it faithfully maintain the existing interfaces, and only expose new functionality through new interfaces?
The answer (as alluded to at the beginning of this post) is: Not completely. While Office does introduce new interfaces, it is also common to append additional parameters to existing methods on existing interfaces. These are all flagged as optional. So, the Excel 2002 version of Workbooks::Open takes 13 parameters, and the Excel 2003 version takes 15. The point being that your code may be built against one version of the typelib, but later find itself running against another version. Or to put it another way, you’re bound to a typelib that doesn’t completely match the COM server you’re running against. So how does it work?
It works (up to a point) because the additional parameters are always on the end of the parameter list, and are always optional. That means that an application that was built against Excel 2002 should continue to function correctly if it finds itself running against Excel 2003, because Excel 2003 doesn’t mandate the additional parameters. On the other hand, if you recompile your old solution with the new typelib (or its managed representation, the PIA), in VB you’ll be fine. In C#, it will fail to compile. Also, of course, if you build a new solution that uses the 2 additional parameters, and then attempt to run it against the old version of Excel, it will fail at runtime.
This brings me back to the points about version-specific types and loose coupling, which is a thorny issue that’s been taxing me for the whole of the current (Orcas) ship cycle…
Recall what actually happens when you build a managed customization for Office. At compile time, you bind statically and explicitly to strong types. You do this by binding to the Office typelib (or rather, the PIA). Then, at runtime, your managed code calls into the unmanaged Office COM code. How does it do this? All the low-level details are taken care of by the CLR’s auto-generated runtime callable wrappers (RCWs). These are very thin wrappers over the COM objects that take care of the necessary plumbing to ensure parameters are correctly marshaled back and forth and object lifetime is managed (or at least, manageable). The RCWs do not provide any functional wrapping – that is, as far as the functionality of the underlying COM objects are concerned, they are simple pass-through wrappers. With this framework in place, and barring any breaking changes between versions, the customization you built against one version of Office will continue to work fine when running against any later version.
VSTO complicates this picture somewhat, at least for document-level solutions. Firstly, by adding more complex programming artifacts – specifically by supporting a more .NET-centric view of Office development. This includes allowing the developer to place managed Windows Forms controls on the surface of a document (or on the document-level task pane), and by providing functional wrappers to some of the COM objects (notably, ListObject, Bookmark, XMLNode, and the Workbook, Worksheet and Document host item objects). So, now your code goes through another layer: a functional wrapper that sits on top of the RCW. This functional wrapper is not a straight pass-through (if it were, there wouldn’t be any point to it). The functional wrapper sits on top of the RCW purely in terms of call stack. In functional terms, it sits on top of the underlying COM object. That is, it adapts the behavior of the underlying COM object (or often, multiple objects) to present a particular functional interface to your usercode.
So, what you get is enhanced functionality – but what are you paying for this enhanced functionality? Part of the price is obviously complexity + performance. A less obvious part of the price is a certain versioning intransigence. Why is this so? After all, if you can write usercode that works in multiple versions of Office, why can’t VSTO write wrapper code that works in multiple versions of Office? The answer is that VSTO wrapper code relies on behavior that is not defined in the COM interfaces, and that Office never promises to maintain across versions. Plus, VSTO wrapper code sometimes merges the behavior of multiple objects in order to present a unified functional interface. For example, consider the events on the Microsoft.Office.Tools.ListObject: the native Microsoft.Office.Interop.Excel.ListObject object does not offer any events at all. The M.O.T.ListObject events include native M.O.I.E Workbook events, plus custom M.O.T events, plus System.ComponentModel.Component events, and so on.
So (continuing this example), because the native ListObject doesn’t expose any events of its own, the functional wrapper is forced to provide events that are not protected by the normal versioning (backward compatibility) promises made by Office. One of the reasons we released VSTO 2005 SE was because we had to ship an updated version of the VSTO ListObject wrapper to cope with the changes between the Office 2003 and 2007 versions of ListObject and related objects’ behavior. Without the VSTO 2005 SE runtime, your Office 2003 ListObject solutions would not run successfully in Office 2007. It was also important to us to ship this updated runtime without requiring any usercode to be re-compiled. To achieve this, we had to treat the old M.O.T.ListObject API as an immutable interface. Which brings us back to COM and MAF: we need to support an architecture where the user codes against an immutable interface (or “contract” in MAF-speak), not directly against any runtime objects that implement this interface. VSTO 2005 partly supports this by accident, VSTO Orcas is intended to support this on purpose by implementing MAF. MAF is a managed expression of COM. So, now we’ve come almost full circle – Office developers can now continue to behave as if they’re from Venus (ignoring all the underlying infrastructure), while the VSTO runtime takes over from the VBA runtime to handle the Venus-to-Mars mapping.
The question that’s really making me pause over my mince pie is: is this a good thing? We wrap stuff, and hide stuff, and jump through all kinds of hoops internally to present a unified version-resilient API to usercode. We treat the developer as if they like being from Venus, as if they can’t cope with the Martian landscape. I’m sure there’s some number of developers out there who like this model. I’m equally sure there’s some number who don’t. The new VSTO add-in model is an example of where we’re trying to achieve the optimum compromise. The new add-in object is an extremely minimal wrapper, unlike the very complex doc-level wrappers. It is a statically, strongly, explicitly typed model; also, we do just the bare minimum to get it correctly loaded and unloaded, and everything else is a straight pass-through to the underlying COM object, via the RCW. In this way, we cater for both extremes of the continuum: doc-level wrappers are for the Venusians who like a simple RAD experience and don’t care about runtime complexity; while add-ins are for the Martians who prefer to see bare metal without any intervening layers.