Customization, Automation, Addins and Life : MAF

TechEd 06: Chalk-talk today, MAF 101

TQuinn — Wed, 14 Jun 2006 16:58:00 GMT

I'm doing a "chalk-talk" today at TechEd -- at 3:45 in DEV (Blue area) Theatre 1 in the TLC. The listed subject is Designing Managed Add-ins for Reliability, Security and Versioning, the same subject of the talk Jim and I did at PDC. I've refined the...(read more)

Adapters

TQuinn — Wed, 17 May 2006 07:58:00 GMT

Geez, did I make you wait long enough!?! I have been remiss, hopefully I will have time to do this more regularly. Those of you with kids know how time can evaporate.

Anyway, I've promised a post on Adapters for months now, so here it is finally. As my colleague JackG said on my Proxies post, a picture is worth a thousand words, so here is a picture that pulls the whole story together.

So as you can see, I've drawn this picture so that the addin is calling into the host through the Proxy. Obviously (I hope its obvious anyway) there is at least one of these pipelines going the other way -- the host calling the addin through some contract. But in practice, the addin generally calls back to the host a lot more -- think Office addins. So there will be many proxies on the addin side, and many adapters sitting on top of many "real" host objects. So I drew the common case.

So the picture makes clear: the adapter implements the contract. But it must do more: it must also derive from MarshalByRefObject so that it can be remoted. In general this means also overriding MarshalByRefObject.InitializeLifetimeService to manage the leasing of the object. I'm not going to delve too deeply into that implementation here, I'm sure there are several other posts on MarshalByRefObject in the blogosphere. But suffice it to say, the way the leasing is implemented in the MarshalByRefObject plays directly into the implementation of IContract.AcquireLifetimeToken and IContract.RevokeLifetimeToken.

There is a very simple implementation to this: return null from IntializeLifetimeService and have Acquire\RevokeLifetimeService do nothing. Returning null from InitializeLifetimeService gives the object an infinite lease, meaning it will live as long as the AppDomain lives. In many cases this is entirely appropriate, if the object is going to live as long as the application is running, this is a simple and robust implementation. But, of course, transient objects should only live as long as necessary to avoid unnecessary working set. This means understanding and dealing with Sponsors and Leases in the .NET Remoting system.

The good news is that we provide a default implementation of this in VSTA in ContractAdapterBase. Almost all of the built-in Adapters derive from ContractAdapterBase (there are a very few that don't, and most of these are directly involved in the implementation of Acquire and RevokeLifetimeToken themselves). It provides a robust implementation of the leasing system that keeps objects alive as long as they need to be, but no longer. And this implementation can be tweaked by overriding the default sponsor timeouts.

But I seem to have digressed a bit. I could do an entire article on Sponsors and Leases, and may someday (I don't want to promise anything, now, given how long it has taken for me to do this one). Let's talk about adapters.

This first thing you may think when looking at the picture above is, "is the adapter really necessary? It appears I could just implement the Contract on 'Real' Host Object and be done." Well, appearances can be deceiving.

It is certainly possible to implement contracts on "real" objects and do away with adapters, and in fact may be done. But we have found that in practice an adapter is more common than not. For one thing, the "real" host object might already be implemented -- you are adding extensibility after the fact. Another thing is that, well, there can be only one base class -- no multiple inheritance in .NET. So if your "real" object needs to derive from something that doesn't derive from MarshalByRefObject you are going to need an adapter. And practically, it is easier to simply keep your "real" implementation separate from your extensibility code. It just sort of falls out, you'll see.

Of course we provide a ton of default adapters in the VSTA code base. Another reason to use adapters is because you can just use ours...

I mentioned ContractAdapterBase already. We also provide two different implementations of IRemoteObjectContract and friends -- one that "adapts" managed objects and one that "adapts" COM objects. These are quite useful for getting up and going quickly, but there is a performance overhead here since everything goes through late-binding. In many cases the performance hit is worth taking -- its not that big and you are done quickly. But I usually recommend doing a mix of custom contracts and adapters with the generic ones. Target critical performance scenarios with the custom contracts, but let the generic adapters and contracts take care of the rest.

Proxies

TQuinn — Mon, 16 Jan 2006 22:39:00 GMT

I'm sure that everyone's reaction to the contract concept is that it is very strict, that it can get complicated quickly, and that it is important to get it right and keep it right. In other words, you don't want just anyone partying in the contract space. Furthermore, the programming model presented by contracts is not very .NET friendly. For these reasons we have in the MAF model an abstraction that lives on top of the contracts called a "proxy."

Of course, proxy is one of those computer terms that has dozens of particular meanings. The general meaning of a "proxy" is "the authority to act for another." Computer proxies adhere to this meaning, and ours is no different. But the MAF proxy doesn't act on behalf of the contract, it acts through the contract on behalf of the underlying implementation. (We'll cover what the underlying implementation looks like in the next article: Adapters). The proxy presents a .NET Friendly programming model that represents or proxies the "real" implementation in the host application. As such, in many cases, its public surface probably looks suspiciously similar to that real class's public surface. The difference is that the implementation of the proxy is simply to call through a contract or contracts.

If you are familiar with .NET Remoting, you know that they have proxies too. In fact, these are used internally in MAF, as it is built on top of .NET Remoting. When a MAF proxy calls into a contract, assuming it is crossing a remoting boundary, it is really calling into a .NET Remoting Transparent Proxy for that contract. And that transparent proxy is backed, in .NET, by a "Real Proxy" that translates method calls on the Transparent Proxy into messages sent through the remoting channel. To differentiate the MAF Proxy from .NET Remoting's transparent proxy, I sometimes refer to the MAF Proxy as an "Opaque Proxy."

If MAF is using proxies already, internally, why do we need this extra layer of abstraction? The reason is in the Transparent Proxy itself. The Transparent Proxy is created in the remote appdomain with a particular System.Type instance that tells it what to emulate. If this System.Type is *not* a contract, then we are leaking types from the host. So while Transparent Proxy is not an instance of the type it is proxying for, it still *requires* an instance of that class's System.Type; in short it requires that the type be loaded into both AppDomains. As we know from from the discussions on Contracts, only Contract Types are allowed to span the boundaries.

So Opaque Proxies are their own types, isolated from the "real" types by contracts. They just happen to look like the real types. Transparent Proxies emulate the real instance by grabbing hold of its actaul System.Type. Opaque proxies, then, provide a level of abstraction on top of Transparent Proxies to keep us from leaking host types. The only Transparent Proxies in MAF are for contract types.

One must, then, create a Opaque Proxy class for each type one wishes to proxy and implement it to call through a contract. This can get daunting (and tedious) quickly. Fortunately, we are working on a tool -- proxygen -- that generates proxies given an existing assembly as input. This will be in the upcoming VSTA SDK.

Next, I will go into more detail on what contract implementations look like.

Are Arrays OK in Contracts?

TQuinn — Fri, 28 Oct 2005 08:30:00 GMT

I know its been a while, but I've been busy coding lately. That code will become public fairly soon, so I'll talk about it then. I'm still working on the promised Proxies and Adapters post, it will be coming, I promise. But in the meantime, the subject of my last post came up in practical terms at work the other day. One of the developers asked a seemingly innocent -- and presumably simple to answer -- question: Can I pass a byte array as a paramter in a contract, or do I have to use one of the collection contracts?

The answer, as it turns out is, unfortunately, "it depends." It depends on what you are doing with that array. I explained in the previous post that certain types can change behavior from reference to value depending on whether you pass them in AppDomain or cross-AppDomain. Any reference type that is *not* MarshalByRef but *is* serializable exhibits this idiosyncracy. And probably the most frequently used type in this category is System.Array.

So what it comes down to is this: are you OK with the recipient of the array getting a copy of the array, not a reference to the array: passing it cross-domain make Array into a value type. With byte array, in particular, this is most likely the intent. Byte arrays are typically used to move bags of bits around -- the recipient of the array would rarely change an element of the array and expect that change to be reflected in the caller. So for the particular case in question we decided that, yes, putting the byte array as a parameter in the contract *was* OK.

But you need to be careful. Many times, if not usually, if you are passing an array of objects, the opposite expectation would be true: in many scenarios I expect you to change the elements, or even fill them in. Of course in contracts object[] is strictly illegal, what you would consider passing instead is IContract[]. Even looking at that seems weird, don't you think? Instead, to get the desired behavior, you should pass the IArrayContract<> (or one of the other generic collection contracts) to get the desired behavior of a MarshalByRef array that can be altered by the callee and have the results show up in the original array.

So those are the two extremes: byte[] to IArrayContract<>. Everything in between, and in fact even these two, is a judgement call. Can you live with value type sematics with the array you are passing? Is that what the callee expects? Then its probably OK. But if you expect the callee to treat the array as a reference type, don't pass array.

Reference vs. Value: Types, Passing and Marshalling

TQuinn — Fri, 07 Oct 2005 03:48:00 GMT

The .NET Programming Model makes a clean distinction between Reference and Value on three different levels. These three levels interact directly, and it is important to understand the distinction. This article may be old hat to many of you, but it is confusing enough on a daily basis over here that I decided to write it all down so we’d have a place to look to keep it all straight. And I figured if we needed to keep it straight, then you might too. The main reason for writing this is getting to the Marshalling bit, but the other pieces are necessary background.

Understanding how different types Marshal (if they Marshal at all) is essential to getting Addin Models correct. Even if you are sure you understand the first two bits, I encourage you to skim over them and go to the end. Because an understanding of all of this is critical to understanding my next article upcoming in a week or so: Implementing Contracts : Proxies and Adapters.

Reference Types vs. Value Types

This is the first level, and probably the most well understood. In .NET all types are by default reference types unless they derive from System.ValueType. Of course, the various languages don’t allow you to derive directly from System.ValueType, there is some syntax to indicate that a type is specifically a value or reference type. In C#, of course, "class" is reference and "struct" is value. In VB .NET they are "Class" and, now, "Structure" (the keyword "Type" used to be used for struct-like things in VB, but Type means something else now…).

But what does that mean: what is the difference between a class and a struct? Those of you that know C++ (traditional, not managed) remember that in that language the only difference between class and struct was whether or not members were public by default (struct, yes, class, no). Both classes and structs are really value types: it was not the type that is "reference" but how the memory was allocated that determined it. If the object was declared on the stack, it was a value instance, if it was allocated on the heap the variable was a reference to the value.

In .NET, of course, the developer is freed from worrying about memory allocations. But the concept of "value" and "reference" live on in the types themselves. The differences end up being similar to C++ conceptually. There are three main differences in value types and reference types: how they are constructed/initialized, how assignment works, and what happens when you pass them into other functions. BTW, from now on I’m going to use C# in examples because I am more familiar with it.

First, let’s declare two simple types to use as examples:

public struct ValType
{
public int member;
}

public class RefType
{
private int member;

    public int Member
    {
        get { return member; }
        set { member = value; }
    }
}

Now, let’s see how these things are constructed and initialized. To use one of these things, you must declare a variable of the type, and the initialize it – this part is deliberately the same in C#. Well, the same set of steps, what happens is different.

public static void UseTypes()
{
ValType vt = new ValType();
RefType rt = new RefType();
}

Both types come with default parameterless constructors. But they do two different things. For ValType, you can think of the variable vt as the object itself (allocated "on the stack" conceptually), but in the case of the RefType, the variable is *not* the object, but a reference to the object (allocated "on the heap" conceptually).

The default constructor of a Value Type does not instantiate the object – it is already there. It simply initializes all of the fields to their default values – so "constructors" for value types are really more like initialization functions, or "intializers." You cannot use an instance of a value type before it is initialized, but you don’t have to call a constructor to do it. I could have written the code above as follows, and it would be just as valid – ValType would be usable:

public static void UseTypes()
{
    ValType vt;
    vt.member = 5;

    RefType rt = new RefType();
}

"Constructors" for value types are useful, though, if there are many fields that need initialization – one stop shopping. It is illegal to directly initialize members in a struct declaration like this:

public struct ValType
{
public int member = 0;
}

Note that it is also illegal to declare a parameterless constructor in a struct – if you want to initialize the fields to other than default values you either do it directly, outside of the declaration, or declare a parameterized constructor and do the initialization there. Of course if you make the fields private, with property accessors, then you *must* have a parameterized constructor to initialize the fields to other than default values, because calling a property is "using" the instance (it calls the underlying set_ method) and you cannot use the instance until it is initialized. Make sense?

With Reference Types, conversely, the constructor actually constructs the instance of the object – the memory is allocated, the constructor code is run, and a reference to that new memory is assigned back to the variable. So you cannot use a reference type without calling a constructor, as above. Otherwise you will get a NullReferenceException. The "constructor code" of the default, parameterless constructor just assigns default values (usually zero or null) to the members. But in reference types you *are* allowed to declare and implement a parameterless constructor and do whatever you want in it. Reference types also allow member intializers like this:

public class RefType
{
private int member = 5;

    public int Member
    {
        get { return member; }
        set { member = value; }
    }
}

The memory for "member" is initialized to 5 when the memory is allocated.

Assignment – the = operator – is different for Reference types and Value types as well. We have seen the operator used above for initialization, assignment of other instances is an expansion of the same thing:

public void ReassignTypes()
{

    ValType vt1 = new ValType();
    RefType rt1 = new RefType();

    ValType vt2 = new ValType();
    RefType rt2 = new RefType();

    Console.WriteLine("vt1.member = " + vt1.member);
    Console.WriteLine("ReferenceEquals(vt1, vt2) ? " + ReferenceEquals(vt1, vt2));

    vt2.member = 5;

    vt1 = vt2;

    Console.WriteLine("vt1.member = " + vt1.member);
    Console.WriteLine("ReferenceEquals(vt1, vt2) ? " + ReferenceEquals(vt1, vt2));

    Console.WriteLine("rt1.Member = " + rt1.Member);
    Console.WriteLine("ReferenceEquals(rt1, rt2) ? " + ReferenceEquals(rt1, rt2));

    rt2.Member = 5;

    rt1 = rt2;

    Console.WriteLine("rt1.Member = " + rt1.Member);
    Console.WriteLine("ReferenceEquals(rt1, rt2) ? " + ReferenceEquals(rt1, rt2));
}

This little program’s output is as follows:

vt1.member = 0
ReferenceEquals(vt1, vt2) ? False
vt1.member = 5
ReferenceEquals(vt1, vt2) ? False
rt1.Member = 0
ReferenceEquals(rt1, rt2) ? False
rt1.Member = 5
ReferenceEquals(rt1, rt2) ? True

ReferenceEquals is the way we determine identity in .NET. Of course in native programming, identity was determined by the physical address of the memory. But in .NET memory is managed by the Garbage Collection mechanism, and can be moved about, so addresses are not constant for a particular object. We use ReferenceEquals to tell us of two objects have the same identity. So this shows us something. For vt1 and vt2, the assignment copies the value from one instance to another. For rt1 and rt2 the assignment changes the reference held in the variable to the other chunk of memory. After the assignment the variables rt1 and rt2 reference the same instance, while vt1 and vt2 remain separate instances.

We can’t have a complete discussion of Value Types without mentioning "boxing and unboxing". The link is to the MSDN article on boxing and unboxing for further information. In short, boxing is what happens when you assign a value type to a variable of type System.Object. System.Object is, of course, a reference type, but it is also the base class of System.ValueType so the assignment is allowed. There is a special case, then, for this assignment. When you "box" a value type by assigning it to a variable of type System.Object, what happens is a new instance of the value type is created on the heap, its value is copied to this new instance, and a reference to that instance is assigned to the variable. "Unboxing" goes the other way, when you assign the object back to a variable of the type of the Value type. The value is copied from the referenced object back to the local variable.

The next difference between Reference Types and Value types opens up the next level. Reference Types and Value Types behave differently when they are passed into a function. But there are also two forms of passing arguments: Passing by Value and Passing by Reference. Both forms apply to both kinds of types.

Passing by Value vs. Passing by Reference

Both Value Types and Reference Types can be passed either by Value or by Reference. This is starting to get confusing already. By default – in other words, if you do nothing special – what you are doing is passing by value.

So let’s look at that first. When you pass a Value type by value, you are literally passing the value of the type. The value is copied into the argument. If the value of a member is altered within the call, it is not reflected in the passed in value. This is demonstrated here:

public static void PassValueType()
{
    ValType vt;
    vt.member = 5;

    Console.WriteLine("vt.member = " + vt.member);

    PassValueTypeByValue(vt);

    Console.WriteLine("vt.member = " + vt.member);
}

public static void PassValueTypeByValue(ValType vt)
{
    Console.WriteLine("vt.member = " + vt.member);

    vt.member = 1111;

    Console.WriteLine("vt.member = " + vt.member);
}

The output for this is as follows:

vt.member = 5
vt.member = 5
vt.member = 1111
vt.member = 5

So you can see vt.member is changed within the call, but the original passed-in vt is not changed. Value types passed by value are copied into their destination.

Reassigning the variable in the call produces the exact same result, because as we have seen, assignments to value types actually just copy the value from one instance to the other. So this code produces the exact same output:

public static void PassValueTypeByValue(ValType vt)
{

    Console.WriteLine("vt.member = " + vt.member);

    ValType vtx = new ValType();
    vtx.member = 1111;
    vt = vtx;

Console.WriteLine("vt.member = " + vt.member);
}

Why is this important? Because it is different than with reference types.

With Reference types, it is the reference that is passed by value rather that the value. So if I pass a reference type by value and change its state by altering one of its members, the value *is* reflected on the other side of the call.

public static void PassReferenceType()
{
    RefType rt = new RefType();
    rt.Member = 5;

    Console.WriteLine("rt.Member = " + rt.Member);

    PassReferenceTypeByValue(rt);

    Console.WriteLine("rt.Member = " + rt.Member);
}

public static void PassReferenceTypeByValue(RefType rt)
{

Console.WriteLine("rt.Member = " + rt.Member);

rt.Member = 1111;

Console.WriteLine("rt.Member = " + rt.Member);
}

The output of the above code, then, is this:

rt.Member = 5
rt.Member = 5
rt.Member = 1111
rt.Member = 1111

But, if I change the code to do a reassignment, I am actually trying to change the reference in the call, but the reference was passed by value, so the change is *not* reflected on the other side. In other words this code:

public static void PassReferenceTypeByValue(RefType rt)
{
    Console.WriteLine("rt.Member = " + rt.Member);

    RefType rtx = new RefType();
    rtx.Member = 1111;

    rt = rtx;

    Console.WriteLine("rt.Member = " + rt.Member);
}

Produces this result:

rt.Member = 5
rt.Member = 5
rt.Member = 1111
rt.Member = 5

This all makes sense if you think about it. Value types passed by value passes the value, Reference types passed by value passes the Reference. So for value types you can’t alter the value, and for reference types you can’t alter the reference – but you can alter the value. Reference types are one level of indirection removed from value types.

The .NET Programming model provides us with one more level of indirection for both Value Types and Reference Types, and that is, predictably enough, Passing by Reference. If you pass a Value type by reference, you are allowed to change the value, and if you pass a reference type by reference you are allowed to change the reference. So let’s consider the above examples by reference instead.

In C# the "ref" keyword indicates passing by reference. Both the parameter of the function and the argument passed to the function have to be decorated with "ref" for this to work. In other words the signature must specify passing by reference, and the code that passes the argument must match.

So passing a value type by reference looks like this:

public static void PassValueType()
{

    ValType vt;
    vt.member = 5;

    Console.WriteLine("vt.member = " + vt.member);

    PassValueTypeByReference(ref vt);

    Console.WriteLine("vt.member = " + vt.member);
}

public static void PassValueTypeByReference(ref ValType vt)
{
    Console.WriteLine("vt.member = " + vt.member);

    vt.member = 1111;

    Console.WriteLine("vt.member = " + vt.member);
}

Predictably, the output of passing a value type by reference looks the same as passing a reference type by value – we have elevated the value type to the same level of indirection that a reference type is by default.

vt.member = 5
vt.member = 5
vt.member = 1111
vt.member = 1111

But, as in the value type example above, reassigning the instance inside the call, produces this exact same output – it is not the same as trying to change the reference because, well, it is still a value type that is being assigned to, and that means copy the value, not the reference. So this code produces the same output as above:

public static void PassValueTypeByReference(ref ValType vt)
{
    Console.WriteLine("vt.member = " + vt.member);

    ValType vtx = new ValType();
    vtx.member = 1111;

    vt = vtx;

    Console.WriteLine("vt.member = " + vt.member);
}

Next, when passing Reference types by value, you can alter the value, as above. Again, we get the same output as above for this code:

public static void PassReferenceType()
{
    RefType rt = new RefType();
    rt.Member = 5;

   Console.WriteLine("rt.Member = " + rt.Member);

    PassReferenceTypeByReference(ref rt);

    Console.WriteLine("rt.Member = " + rt.Member);
}

public static void PassReferenceTypeByReference(ref RefType rt)
{
    Console.WriteLine("rt.Member = " + rt.Member);

    rt.Member = 1111;

    Console.WriteLine("rt.Member = " + rt.Member);
}

This is because it is really doing the same thing as we did when passing by value. It is changing the value of a reference. But now we can change the code to this and get the same output:

public static void PassReferenceTypeByReference(ref RefType rt)
{
    Console.WriteLine("rt.Member = " + rt.Member);

    RefType rtx = new RefType();
    rtx.Member = 1111;

    rt = rtx;

    Console.WriteLine("rt.Member = " + rt.Member);
}

So the output was now exactly the same for all four examples here of passing by reference. But it was for different reasons. With value types passed by reference, you can alter the state of the instance and reassign the instance, but because it is a value type you never reassign the reference. With reference types, you continue to be able to alter the value of the reference, plus now you can reassign the reference as well.

C# also provides another keyword that indicates passing by reference, the "out" keyword. The difference is that in either case – using a value or reference type – the argument passed in must be fully initialized in the called function. It may have already been initialized outside the call, or not. With "ref" calls, the object must be initialized before calling the function. So only the last example of each – the reassignment -- is valid for the "out" keyword. You’ll get a compile error otherwise. I’ll leave it as an exercise to the reader to play with "out" value and reference type parameters.

Marshalling by Value vs. Marshalling by Reference

Finally we get to the good part. Marshalling is important to the Managed Addin Framework (MAF) because we assume addins are isolated from each other and the host across a remoting boundary. They don’t have to be, but in order for them to be allowed to be, the system must assume the boundary is there. This places restrictions on the model, and these restrictions have to do with Marshalling.

Marshalling is what happens when you pass an object across a remoting boundary. For the addin model, we usually use separate AppDomains as the unit of isolation. We can also use separate processes. Marshalling between either AppDomains or processes is roughly the same, and those are what are considered here. Marshalling cross-machine using Web Services is a separate subject.

By default neither Value Types nor Reference Types can be Marshalled at all. In other words, neither ValType nor RefType as declared in the examples above could be passed across a remoting boundary. If you tried, you’d get a "SerializationException." I’ll explain what that means in a minute. This is done on purpose, of course. We want you to be explicit and know what you are doing when you Marshal something.

So how do you "try" to Marshal an object? You pass it into a function across a remoting boundary, or return it from a function through the boundary. I’m not going to go into all of the details of .NET remoting in this article – RealProxy, TransparentProxy and all of the infrastructure can be explored at the provided links. What I am going to cover is what it means to make a type Marshal by Value or Marshal by Reference.

One interesting thing here is that it is not possible to marshal a value type by reference. The best you can do is make a value type marshal by value. This is a good thing, it keeps the model symetrical. With Reference types it *is* possible to make them marshal either by value or reference. There is a good reason for this, and it has to do with how far the object is getting marshalled. For Addin models, that are only expected to go as far as another AppDomain or process, the best thing to do is to make your reference types marshal by reference – again to keep the model symetrical. Unfortunately, there are cases when this is out of our control.

So let’s look at Marshal By Value first. What does this mean? It is similar to passing by value: to Marshal by Value we copy the value of the object from one side of the boundary to a new instance on the other side. The way this is done is through a process called "serialization." I’m not going to go into too much detail on the process of serialization here, it is a book in itself. Use the link for more information.

A Marshal by Value object, then, is also said to be "Serializable." There are two ways to mark a type as "Serializable." The simplest way is to simply add the SerializableAttribute to its declaration:

[Serializable]
public struct ValType
{
public int member;
}

When applied, all fields, regardless of visibility, are serialized (copied), unless specifically excluded by applying the NotSerializedAttribute. Of course this attribute should be used with care – excluding fields could render the copy on the other side invalid.

What this means is that the transitive closure of all types held in members of a serializable type (and their members on down) must be Marshallable, either by value or by reference. If it is not true a Serialization Exception will result, meaning serialization failed an your program can't work. It is this transitive closure requirement that made us define the MAF contract layer as a closed system. More on that as we go along.

Versioning is an issue with Serialized types of course. If you try to serialize v1 of a type into an instance of v2 of a type (or vice versa) there could new, or missing, members and serialization may fail. Work was done in Whidbey to make serialization more version resilient. But of course with regard to the MAF this versioning stuff is moot: for this very reason, the only things allowed to cross a remoting boundary in MAF must be declared in the Contract layer, and contract assemblies do not version. Value types declared in contract assemblies *must* be serializable to work, but they can never version, so this restriction, along with the transitive closure restriction means that serialization won’t fail.

The other way to mark a type as Serializable is to implement the System.Serialization.ISerializable interface. This gives you fine grained control over the serialization process. The link above covers this in detail, too. Though not strictly necessary if you implement ISerializable, you should apply the SerializableAttribute to make it clear that this is a serializable type. ISerializable gives you a way to work around the versioning issue, but again for MAF you don’t need it. In fact, for MAF the contract-level serializable value types should be simple compositions of primitive types, other contract-level serializable value types, or contracts, and that’s it. They should contain no implementation. For this reason, implementing ISerializable should be unnecessary. See my previous entry Contracts and IContract for more information.

Marshal by Reference is, as you would expect, the ability to Marshal a reference to an object across the remoting boundary. The object is not copied, but a Proxy is created in the new domain that refers back to the original object. Calls into the proxy cross the boundary and are called on the original instance. Again, see the remoting articles linked above for more information on how proxies are created and used.

The way you mark a type as Marshal by Reference is completely different from Marshal by Value. To make an object Marshal by Reference you must derive it from System.MarshalByRefObject. Obviously the imposes a strict requirement on your object hierarchy – you have to know you are doing it. This is the reason value types cannot marshal by reference – you can’t change their derivation chain. So only reference types can marshal by reference.

Of course tranisitive closure comes into play here too. It is not as restrictive as the Serialization case, because we don't have to worry about private members. But the transitive closure of all types exposed in the public API of the Marshal by Reference type must also be marshalable, again either by reference or by value. Again, it is the tranisive closure requirement here that was a main driving force behind MAF.

But as noted reference types *can* marshal by value. You can just as easily apply the SerializableAttribute to a reference type as to a value type. In fact, you can apply the SerializableAttribute to a reference type the derives from System.MarshalByRefObject. I’ll cover this last case in a minute, there really is a valid reason for doing this, and some types in the .NET Framework are actually implemented this way.

I said above, for addin models you should make reference types marshal by reference to keep the model symetrical. This is important. Because, if you think about it, a reference type that marshals by *value* exhibits two different behaviors whether passed to a function within an AppDomain or passed to a function across a remoting boundary. Within an AppDomain the reference type is still a reference type, I can change its state and that state is reflected in the original object after the call. But if it is passed across a boundary to me, I only get a copy of the object and if I alter its state I do not alter the original object. It has in effect become a value type once it is marshalled. As the recipient of the object, I can’t know for sure which case its is have I – received a copy of it or not? – it just looks like a local reference type to me. So breaking the symetry is bad for addin models.

Of course with MAF the only types allowed to cross a boundary are defined within the contract layer. And the "reference types" of MAF are contracts. We have seen that contracts are actually just interfaces, they have no implementation, at the contract level. The transitive closure of types exposed by contracts must be other contracts, serializable value types defined within the contract layer or primitive types, meeting the restriction.

But of course, Contracts must be implemented somewhere. So, you can tell from this that Contracts must be implemented on reference types that derive from MarshalByRefObject. The actual type of the class that implements the contract is not visible across the domain (and really should have no other public surface besides the contract), only the contract is visible, but nonetheless the class must derive from MarshalByRefObject in order to Marshal by Reference.

To close out this section, let’s consider why one would make a reference type serializable – even a MarshalByRef type. The reason, as alluded to above, has to do with Web Services. One cannot marshal a reference cross-machine, objects must be stateless and serializable. It is not always practical to make types that one wants to expose for cross-machine serialization value types. One might want a type hierarchy that you cannot get with value types, for instance. And a Serializable MarshalByRef type is a special case of this. Here the type author wants the type to MarshalByRef when possible (MarshalByRef always wins when crossing AppDomain and Process boundaries) but to Serialize when going cross-machine. Of course this object must implemented with the utmost care. Another reason to mark a reference type as Serializable is to allow its state to be persisted with standard serialization. Again, see the serialization article linked above for more information. None of this pertains to MAF or addins, though.

As noted at the beginning of this very long article, my next article will be about implementing contracts and actually providing a real example of how this stuff works. All of this Value\Reference stuff is a prerequisite, we’re going to put it into play.

Contracts and IContract

TQuinn — Thu, 22 Sep 2005 23:53:00 GMT

In my post What is an Addin? I said that a "contract" is just that: "a previously decided upon method of communication with rules and limitations." Here is where we discuss the rules and limitations for .NET Addin contracts. In my post Addins in .NET I noted that we need to solve issues of versioning, unloadability and isolation for security. Our definition of a .NET Addin contract helps in all three areas.

The key tenet for solving the versioning issue is separation of "integration types" from "implementation types." In .NET, of course, the Type is the basic unit. For this article I assume you know the difference between Reference Types (class in C#), Value Types (struct in C#) and interfaces.

What we want to do is allow the implementations to version freely and independently of each other. In other words host v3 should be able to load addins that targeted host v1, v2 and v3. We call this "backward compatibility" in the host, meaning it is compatible with past versions. Ideally we also want host v1 to be able to load addins that targeted host v1, v2 and v3. We call this "forward compatibility" in the host, meaning it is compatible with *future* versions. Obviously this is harder to attain. Additionally, of course, a particular host version should be able load any addin version.

The way to attain this versioning, ironically, is to define integration types that *never* version. It is these types that become the "previously decided upon method of communication" or "contracts." These communication types are necessarily abstract -- they are separate from implementation types. We have chosen to use .NET interfaces as the kind of Type for Contracts. Now, technically, we could have used abstract classes, I'm sure you are thinking. However, abstract classes are *allowed* to have implementation, interfaces are not. But also using interfaces allows a particular implementation to implement multiple contracts. The absense of multiple inheritance in the .NET Type system makes abstract classes unworkable here. And finally, using interfaces divorces the type from any implied or explicit underlying remoting infrastructure. We'll go into why this is important below.

So, a contract is an interface, so far so good. The next thing one must realize is that contracts must be defined in separate assemblies from integration types -- because the assembly is the unit of versioning in .NET. Since contracts never version, they must be defined apart from things that *do* version. Which brings us to the next point: Contracts must define a closed system within themselves. The transitive closure of all types exposed by the system must be defined within the system. This must be true to make sure that all of the system maintains its lack of versioning as a unit. As soon as a versioning type breaks into the system, the entire system is compromised. As we will see the other problems require this restriction as well.

So what is legal? How may I define the signatures of the methods of my contracts? We have to start somewhere, and the primitive or intrinsic types defined in .NET are the basic building blocks. We usually refer to these as "strings and ints" but these can include any of the types with an explicit TypeCode value (other than TypeCode.Object) in System.TypeCode. These primitive types are basic enough that we can count on them never changing in future versions of .NET: a string will always be a string, an Int64 will always be an Int64.

Next, other Contracts are legal. Contracts, by definition, don't version, so we can use them. Contracts serve as the Reference Types of the contract type system. Any type that acts as an object with behavior should be passed as a contract.

And finally it *is* legal to define simple structs that are compositions of the primitive types. MAF does this as you will see. These structs *must* be marked as [Serializable] to make sure that they remote. And they must be defined *with* the contracts that they support, not just any serializable struct is OK, only structs defined *within* the contract system will work. Think of these as a way to conveniently pass chunks of data through contracts. You should *not* add behavior to these structs.

Now it could be argued, and indeed we do this in a limited way, that *any* enum, serializable struct or class or MarshalByRef class defined in mscorlib should be legal to be used. Why? Because the intrinisic types are all defined there -- if they are legal why not any type in mscorlib? Surely if the intrinsic types don't break the versioning, these others won't either.

While that line of argument may be true -- if you explore MAF you will see that we do use a few enums defined in mscorlib in our signatures -- it should be used only with EXTREME caution. And other considerations impose additional rules on what is legal, or more aptly illegal.

The first and most important illegal type is "Object." You may *never* have a parameter or return value typed as System.Object. Why? Hopefully this is obvious, but "object" represents "any type" or some type that may *not* be defined within the closed system. The system is no longer closed if Object is exposed.

But there is another reason Object is illegal that has to do with the other issues we need to solve: unloadability and isolation for security. As noted in Addins in .NET, both of these issues are solved with AppDomain isolation. In other words, we load the addins into separate AppDomains so that they can be unloaded -- with the appdomain -- and so that we can set a different Application Base directory to isolate the types as well as sandbox the addin, if desired, with a different, presumably more restricted, set of permissions.

So Object is illegal, along with System.Type, because Object and Type represent arbitrary types that may not be accounted for in the closed system. At best passing an arbitrary object or type across the domain boundary will simply fail -- the type won't be visible to the host AppDomain and fail to load. At worst it can be an elevation of privilege security hole. If a malicious addin can cause an arbitrary type to be loaded in a less restricted AppDomain, static constructors and initializers can run and run arbitrary, perhaps malicious, code.

There are a few other types specifically forbidden. MarshalByRefObject, obviously, for the same reason as Object. Any of the types in the System.Reflection namespace (with the exception of the enums) because they carry with them references to arbitrary types. And, as a catchall here, anything else that can cause any arbitrary type to be loaded is illegal.

Now, as soon as you begin to design any closed contract system of any complexity you realize you need something. You realize you need a base contract that you can use as the "object" of your system. It serves three purposes. The first purpose is to give you the "is a" semantic. You can easily test if an interface "is a" contract if it implements this base contract. The next purpose is to provide an "any contract" idea. You will find that you will have methods that may take or return one of several possible contracts. We've already said it can't be System.Object, so you need a type, this base contract, that can represent them all. And finally, you will find there are a set of basic operations that all contracts require. The base contract can be defined to provide those.

Of course, MAF provides such a base contract, which we call IContract. I noted in my earlier post that IContract provides the analog of both IUnknown and IMarshal from COM. IUnknown is, of course, *the* base "contract" from COM. Marshalling was optional in COM, but in MAF, while the act of marshalling is optional, the ability to marshal is not -- it is a requirement that if any piece of the system must be marshalled then the definition of the system must support marshalling. Of course contracts don't *implement* marshalling -- they don't implement anything by definition -- but IContract contains the definition of methods that help with lifetime and identity management required by marshalled objects. The fact that contracts are themselves interfaces allow the implementations to be marshallable (but the implementations must do work as well to make sure that they marshal).

So let's look at IContract. The current interface definition follows. It is unlikely to change at this point, but I make no promises.

    public interface IContract
    {
        IContract QueryContract(string contractIdentifier);
        int GetRemoteHashCode();
        bool RemoteEquals(IContract contract);
        string RemoteToString();
        int AcquireLifetimeToken();
        void RevokeLifetimeToken(int token);
    }

The first method is QueryContract. Gee, seems and awful lot like QueryInterface on IUnknown, no? It is there to serve the same purpose. The "contractIdentifier" parameter is just a string here. The convention in the implementation will be that this string contains the AssemblyQualifiedName of the contract being queried for (just use: typeof(IContract).AssemblyQualifiedName for IContract). This is as unique of an identifier as you can get in managed code, more verifiably unique that attaching an arbitrary GUID to an interface definition. Note that we are already using IContract as "any contract." So if I have some contract reference, I can call QueryContract with the AssemblyQualifiedName of any other contract to see if the object implementing the first contract implements the second. It is legal, then, to cast the returned IContract to the desired new contract. By convention, this method should simply return null if the desired contract is *not* implemented. We toyed with the idea of throwing a ContractNotFoundException or some such thing instead of returning null, but in practice this became too unwieldy with try\catches all over the place. And it is really *not* an exceptional situation anyway. Implementations should simply account for null being a possible return -- or else you risk your code throwing a NullReferenceException instead....

The next two methods help to establish and manage identity. The idea of GetRemoteHashCode is that it should always return the hash code of the underlying object to the remote caller. This ensures that the same remote object hashes the same every time it is used. But hash codes are not definitively unique -- one cannot simply compare hash codes to know if this IContract is implemented on the same object identity as that IContract. That is, in general, what RemoteEquals is for. By default Object.Equals maintains identity for reference types. It is overridden in a few cases to allow different object identities to compare as Equal -- identical strings are equal even if they are physically different strings, and delegates are equal if the are delegates for the same function. In general, though, this is the kind of behavior you would want remotely as well. So, in general, RemoteEquals should delegate to the Object.Equals of the underlying object and honor its idea of "Equal."

We debated whether RemoteToString was necessary or even desireable on the base interface. Its intention is to simply delegate to the underlying Object.ToString. It turned out to be useful enough in enough situations that we left it.

The final methods have to do with lifetime management, as their names indicate: AcquireLifetimeToken and RevokeLifetimeToken. The idea here is that if a client of a contract wants to guarantee that the object will stay alive for a certain amount of time, it should call AqcuireLifetimeToken. The client the hangs on to the returned token until it is done with the contract and then calls RevokeLifetimeToken, at which point the object may be eligble for garbage collection.

"Why is this even necessary?" you may ask. Well, garbage collection exists at the AppDomain level. As soon as an object it marshalled out of an AppDomain using .NET Remoting, its lifetime is subject to the .NET Remoting leasing system; .NET Remoting keeps a local reference to it to keep it alive until its "lease" runs out. Once the lease has expired, the object is given back to the GC of the owning AppDomain and it can go away. So remote clients of the object may get an exception when calling an object whose lease has expired. The default lease is 5 minutes. That's right, 5 actual minutes. The lease *is* renewed, though, every time a remote call comes through, so you actually have to let the object lie dormant for 5 consecutive minutes for the default lease to expire. However, this is likely a common thing to do in Addin scenarios: a host may load a bunch of addins, but have no reason to talk to them for hours, if not days, in many situations. Fortunately, the .NET remoting system is extensible. One can "sponsor" a lease, meaning the lease remains active as long as the sponsor (or a sponsor, a particular lease can have any number of sponsors) says to keep renewing it. It is also possible to give an object an infinite lease -- meaning it will never be GC'd, it will live as long as the AppDomain lives. Giving an object an infinite lease is the easiest to implement (though entirely inappropriate in many situations); one simply overrides MarshalByRefObject.InitializeLifetimeService to return null. Sponsors are more complicated to implement of course, and I will discuss how the VSTA implementation of MAF uses sponsors in a future article.

One might imagine that one possible implementation of AcquireLifetimeToken is to add a sponsor to the lease of the contract and of RevokeLifetimeToken is to revoke the same sponsor. So why not just build this in to the interface? Why is it abstracted? One simple reason is that the sponsorship model is not the *only* possible implementation, indeed there are occasions where an infinite lease is desireable. But the main reason is that we do not expect .NET Remoting to remain the underlying Remoting infrastructure for MAF for the forseeable future. As noted in other posts, WCF, nee Indigo, will eventually replace .NET Remoting at every level. The Contracts in general and IContract specifically cannot be tied directly to any Remoting infrastructure in general, nor .NET Remoting in particular. We think the Acquire\Revoke abstraction, though, is going to be flexible enough to support us going forward over different implementations.

So, hopefully, we see why we need contracts, what they are, and why IContract is itself useful. And when I discuss the MAF AddinLoader you will see that we actually require that contracts derive from IContract through a generic constraint. We do this for the "is a" reason above: we know a contract "is a" contract if it derives from IContract.

In future posts I will explore some of the other contracts in MAF, as well as discuss their specific implementations in VSTA.

MAF vs. COM

TQuinn — Fri, 16 Sep 2005 00:01:00 GMT

OK, I know I said my next post would be an in depth treatment of IContract, but I remembered I wanted to fininsh something I started with my earlied post Addins in .NET. I noted there that .NET currently was still analogous to the LoadLibrary\GetProcAddress era of Win32. In other words, it didn't have the analog of COM for .NET yet. Well, of course, MAF is the begininng of COM analog for .NET -- it provides the base infrastructure to begin to make addin models, and any dynamic component model, easier.

What MAF provides is the managed analogs of IUnknown, IDispatch, ITypeInfo, IMarshal, IEnumXXX and CoCreateInstance . IContract, which really will be covered in depth in my next post, is the managed analog of IUnknown. And in essence it covers IMarshal as well, because all contracts must be written to allow remoting -- we don't *require* that each addin be isolated or remoted, but the system has to allow it so that all addins *may* be isolated and remoted.

IRemoteObjectContract and IRemoteTypeContract (and friends) are the analogs of IDispatch and ITypeInfo and the COM type library stuff. It is important to note that *analog* is very operative here: the MAF system is quite different from the COM system due to the fact that managed code is quite different from native code, for one, and that the object-oriented nature of .NET is quite different from the procedural nature of old Win32. I will of course have extensive articles on all of these contracts in the weeks to come.

MAF also provides a set of collection contracts. Besides providing standard collection definitions, they also provide the notion of a MarshalByRef collection, something missing from the .NET framework right now -- and necessary for MAF as you will see. And analog is operative here, because the MAF collection contracts very closely resemble the .NET Collection paradigms as opposed to the IEnumXXX style of COM.

Finally, the AddinLoader provides the analog of CoCreateInstance. In other words it is the helper that wraps up all the complexity of securely activating the addins or dynamic components. Of course in MAF we do even more work that COM because we have AppDomains and the CLRs Code Access Security system to make MAF even more robust.

Stay tuned, more details than you may even want will be forthcoming.

What is MAF?

TQuinn — Thu, 15 Sep 2005 19:41:00 GMT

As I noted we also unveiled yesterday the Managed Addin Framework (MAF). MAF is necessary realization of the Addin Model architecture. It is included in two assemblies: System.Addin and System.Addin.Contract. These assemblies will ship first with the VSTA product, in the Office 12 timeframe, and then be rolled into the next version of WinFX. So MAF is an integral part of the platform.

So what is in there? There are two important pieces. System.Addin contains what we call the "Loader." The Loader is a fully trusted platform component that is necessary to activate addins and isolate them in their own domains. We solve a bunch of the AppDomain creation and isolation issues illustrated in my earlier post. I will go into great detail on the Loader in a future post.

System.Addin.Contract contains a set of "Contracts" for addin communication. We defined "contract" in our PDC talk yesterday. In COM, a COM interface was, in essence, a contract. However, .NET already has interfaces, so we needed a new, more constrained, term. We chose "contract" because the Indigo -- oops Windows Communication Foundation -- Service Oriented Architecture calls their communication interfaces "contracts". The Contract in WCF and the Contract for MAF are conceptually the same thing -- and in the future they will be exactly the same thing when WCF becomes the ubiquitous remoting infrastructure.

I will go into vastly more detail in my next post on the reasoning here, when I begin to describe the contents of System.Addin.Contract with IContract. But in brief: contracts are the types that are used to communicate between hosts and addins. A contract can *never* version. Once you publish it you can't change it -- you can always add *new* contracts, but old contracts must stay the same. This allows the pieces on either side (we call them proxies and adapters, I'll go into these later too) to version indepenedently of each other. Contracts are abstract -- no implementation. In practice they are interfaces, but interfaces with more strict rules. Contracts must define a closed system. The only types legal to expose as parameters in contracts are other contracts and primitive types like "string" and "int."

System.Addin.Contract defines a number of contracts to get you started. In fact, the contracts we supply may be all that you require. Next I will write an article on the base contract of all contracts: IContract.