Sam Ng's Blog : Dynamic

Channel 9: Dynamic in C#

samng — Tue, 03 Nov 2009 17:13:21 GMT

Not too long ago, I did another Channel 9 video on Dynamic in C# with another compiler dev, Chris Burrows. In this video, we discuss the making of dynamic, as well as some of the drawbacks, design decisions, and philosophies behind the feature. Enjoy!

link: http://channel9.msdn.com/posts/CharlieCalvert/CSharp-4-Dynamic-with-Chris-Burrows-and-Sam-Ng/

Dynamic in C# VII: Phantom Method Semantics

samng — Thu, 25 Dec 2008 06:44:35 GMT

By now, my hope is that you all have a well-rounded view of dynamic. We started this series by introducing dynamic and talking about the basics of the feature, and have just finished talking about some of the feature's limitations with the intent that giving both the good and the bad will help us gain a firm understanding of the topic.

So what more is there to talk about?

The thing that's been occupying my thoughts lately is the semantics around the phantom method. Recall from our previous discussion of the phantom method that it is the method which the compiler binds to when there is a dynamically typed argument with a static receiver. What exactly happens when the compiler determines that it needs to bind to this method? What checks will the compiler do? What checks should it do?

Static or dynamic?

The first question we need to ask ourselves is a somewhat philosophical one. Should the compiler treat binding against the phantom method as a dynamic operation with some static parts? Or a static operation with some dynamic parts? A quick example will allow us to consider both views:

public class C
{
    static void Main()
    {
        C c = new C();
        dynamic d = 10;

        c.Foo(d, 10);
        d.Foo(10, 10);
    }

    public void Foo<T, S>(T x, S y) { }
}

The first call is a statically known receiver with a dynamically typed argument. The second is a dynamically typed receiver with a statically known argument.

A 'dynamic' phantom

Lets consider the former position first. When the compiler binds the call to Foo against a dynamic receiver, it has no knowledge of the receiver's members. It therefore does not check for the existance of a Foo member, and does not perform checks like accessibility, arity, argument convertibility, or method type inference.

If the compiler were to consider binding against the phantom method as a dynamic operation with some static parts, then it should treat the first call in the same manner. This means that once the compiler encounters any method call with a dynamic argument, it should stop checking these things against the receiver, even though it knows the receiver's type at compile time and can therefore determine these things.

Seems counterintuitive doesn't it? If the compiler has the information, why wouldn't it use it to help give the user good diagnostic information at compile time? Ok, ok, I'm clearly biased and think that this is the wrong approach. :) Lets consider the second approach then.

A 'static' phantom

This position argues that the compiler use whatever static information it knows to help give diagnostics to the user wherever it can.

This means that it should perform name lookup on the receiver to make sure there is a method named Foo on class C.

It should check that the method Foo is accessible from the current location.

It should do an arity check to make sure that there is a Foo that takes two arguments.

It should do method type inference to determine as much information about the type parameters as it can. In our example, this means that the compiler will infer S to be type int, but not be able to infer T.

It should check that any non-dynamic arguments are convertible to their respective parameter types. In our example, this means verifying that the second argument is convertible to type int, since we inferred S to be int.

It should check the constraints of the method type parameters against the argument types.

Use the static information!

Though there may be some dynamic language guys that think we should be in the first camp, I'm of the opinion that C# is a static language, so lets stay in the latter camp and use as much static information as we can.

Moreover, we've already decided that for the method call off of a dynamic receiver, the compiler will encode the static types of the arguments so that the static types and not the runtime types will be used for overload resolution.

If we look on our little checklist above, all the items seem pretty straightforward - all but one (in my opinion). Method type inference. How should the type inference algorithm be altered to infer the most that it can, giving errors where it can guarantee that the code will never succeed, no matter what the runtime arguments?

Currently in our working compiler, we simply ignore type inference. That is, we skip type inference, and upon encountering a type parameter, we assume it is convertible at compile time. This can produce some unexpected behavior!

Consider the following example:

public class C
{
    static void Main()
    {
        C c = new C();
        dynamic d = 10;

        c.Foo(10, d);
    }

    public void Foo<T>(T t, int x) where T : class { }
}

One would expect that this produces a compile time error - the integer 10 given as the first argument will never satisfy the constraints to the type parameter T. However, we currently allow this call to compile successfully and fail at runtime!

Quite unexpected, yes?

Well, turns out we need to figure out a good way to have the type inference algorithm behave when it sees dynamically typed arguments in order for this to work.

A modification to the type inference algorithm

Here's where things get fun. Our current type inference algorithm has two results: pass or fail. We now need to introduce a third result: inconclusive.

Because I am the strongest believer that the current behavior is unacceptable, and that we need to make this change to type inference, it fell on me to come up with a reasonable proposal for the design team, and to see if I can usher it through.

So here goes!

I propose that we add the following behavior to the type inference algorithm. If the type of the argument is dynamic, then take all type parameters in the corresponding parameter's constructed type and mark them as inconclusive. No errors will be reported on inconclusive type parameters, and no constraint checks will be performed on them in the constraint checking phase.

Note that this proposal says nothing about types constructed over dynamic. For example, if we supplied an argument of type List<dynamic> to a parameter expecting an IEnumerable<T> where T : struct, then we would not mark T as inconclusive, and therefore report the error that the constraint is not satisfied.

A complex scenario

Lets consider a slightly more complex scenario.

public interface IAnimal { }
public interface IWatcher<in T> { }
public class Watcher<T> : IWatcher<T> { }

public class C
{
    static void Main(string[] args)
    {
        C c = new C();

        IWatcher<Giraffe> a = new Watcher<Giraffe>();
        IWatcher<Monkey> b = new Watcher<Monkey>();
        dynamic d1 = 10;
        dynamic d2 = new Watcher<Mammal>();
        IWatcher<dynamic> d3 = new Watcher<dynamic>();
        c.Bar(a, b, d1); // (1)
        c.Bar(a, b, d2); // (2)
        c.Bar(a, b, d3); // (3)
    }

    public void Bar<T>(IWatcher<T> w1, IWatcher<T> w3, IWatcher<T> w2) where T : IAnimal { }
}

public class Mammal : IAnimal { }
public class Giraffe : Mammal { }
public class Monkey : Mammal { }

In this example, the first two examples contain an argument that is typed dynamic. However, notice that we cannot simply ignore the dynamic argument and ignore the third parameter for method type inference.

If we were to do that, the type inference algorithm would first determine that the candidate set for T is {Giraffe, Monkey}. However, even though there is a common base class (ie Mammal), C#'s type inference algorithm requires that the base class be in the candidate set in order for a successful inference. Type inference would therefore fail at compile time.

In the first call, this is all fine and good - runtime type inference would also fail on the call. However, the second call will succeed at runtime! Because the runtime type of d2 is Watcher<Mammal>, Mammal is added to the candidate set. And because IWatcher is covariant on T, choosing T to be Mammal satisfies argument convertibility for each of the three arguments.

The third call will fail at compile time, because the candidate set for T is {Giraffe, Monkey, dynamic}, and T is not marked inconclusive. Type inference will infer T to be dynamic, since it is the common base class and IWatcher is covariant. However, constraint checking will fail, since dynamic is not an IAnimal.

Questions? Thoughts? Pick it apart!

This being my first real attempt at directly changing the spec and not just being a voice on the voting party, I'd greatly appreciate your thoughts and comments! Are there flaws in my proposed argument? Are there refinements that can help make the algorithm more robust? Are there clear scenarios that break down (ie we report an error at compile time, but the call can succeed at runtime)? Please send me your thoughts!

And as always, happy coding! And have a Merry Christmas!

Dynamic in C# VI: What dynamic does NOT do

samng — Mon, 15 Dec 2008 21:53:37 GMT

As I mentioned last time, there are a few gotchas that we'll need to look at in order to get a full understanding of the dynamic feature and its capabilities. Today we'll take a look at some of those limitations. As we go along, I'll try to shed some insights as to how the decision making process came about, and why we feel these calls are the right ones.

Mutating values using dynamic

Consider the following code:

static void Main()
{
    dynamic d = 10;
    d++;
}

What should happen here?

Intuitively, we'd expect d to contain the value 11. However, recall that under the covers, dynamic is really object. That means that the first line will generate a boxing conversion from int to object. The local variable d contains a boxed copy of the integral value specified. The second line, then, is really an application of the ++ operator on the local variable d. At runtime, calling operator ++ on d will result in an unbox of the object d into an integer, and a call to the operator ++ on the unboxed value. But this value is not copied back inside the box!

Turns out that what you expect to happen isn't really what would happen if we naively implemented this in the runtime binder. Good thing your trusty C# compiler team isn't naive!!

The solution to this little problem then, is essentially to pass things by ref to the runtime binder so that the runtime binder can write back into the value. This gets us half-way there - we still have the problem of boxed structs. Luckily for us, the architecture of the runtime binder is such that we return expression trees to the DLR (for more information on this, read my post that talks about this). There is an expression tree that performs an unbox and modifies the boxed value, and puts that value back into the box, allowing you to mutate the boxed values of these things.

Nested struct mutation

If we take our above example to the next level however, things start getting a bit trickier. Because of the nature of our architecture, dotted expressions get broken up into parts and bound in segments. So that means that for the expression A.B.C.D, the compiler will encode a site for A.B, use that result as the receiver for a second site for .C, and use the result of that as the receiver for the third site for .D.

That seems like a sensible architecture, doesn't it? Indeed, it is the same architecture that the compiler uses when it does it's binding. However, the runtime architecture has the limitation that it does not have the ability to return values by ref. (Well, this really isn't a limitation in the CLR, as they already have the support for this. This is more a limitation in the .NET languages as none of them provide the ability to have ref returns).

This means that if any of the dotted expressions were to bind to a value type, that value will be boxed (and hence a copy would be made), and further dots into it would be made on the copy of the value, and not the initial value as would be expected. Consider the following code:

public struct S
{
    public int i;
}

public class D
{
    public S s;
    public static void Main()
    {
        dynamic d = new D();
        d.s = default(S);
        d.s.i = 10;
        Console.WriteLine(d.s.i);
    }
}

We would intuitively expect the value '10' to be printed in the console. However, the value '0' is printed instead. We're currently working on determining the best way to fix this issue, and are also debating whether or not this is a critical enough of a scenario to fix.

The rule of thumb? Remember that dynamic is like object, and so boxing happens!

Base calls

There is a restriction in the CLR that prevents the compiler from generating non-virtual calls on virtual methods. This means that there is no way to call a base overload dynamically. This means that one cannot call a base call with any dynamically typed arguments, as it will trigger a dynamic binding.

The possible solution (which we have chosen not to implement) would be somewhat akin to the solution we performed for lambdas. Recall that if you had the lambda: x => base.M(x), the compiler will generate a private method that performs the call to the base access, and will have the lambda body call the generated method. The down side, however, is that for lambdas, we knew exactly which call the user was trying to make. In the dynamic scenario, we would be doing overload resolution at runtime, and so we would have to generate a base call stub for each possible overload. This solution is quite ugly, and since we currently lack an extremely compelling scenario, we have opted not to do this and simply give a compile time error when the user attempts to make a base call with any dynamic arguments.

Explicitly implemented interface methods

As one avid reader commented in one of my previous posts, explicitly implemented interfaces kinda get the shaft again here. Because interfaces are really compile time constructs, and have no runtime representation, explicitly implemented interface members get the short end of the stick at runtime. Consider the following:

interface IFoo
{
    void M();
}

class C : IFoo
{
    void IFoo.M() { }
}

Because of the way the compiler implements explicitly implemented interfaces, C.M gets its name removed (making it uncallable via a C pointer). Now this is fine at compile time, because the compiler can see when a receiver is known to be an IFoo pointer. However, at runtime, there is no notion of interfaces, and so there is no IFoo available for the runtime binder to use to dispatch methods. Combined with the fact that C.M's name has been removed, this makes the method entirely uncallable dynamically.

Accessibility

This last topic isn't really a limitation yet. We are still working on drawing the line between doing the pragmatic thing and doing the most consistent thing on this issue. The CTP implementation of dynamic currently performs accessibility checks only on the member that you are accessing. This means that the runtime binder checks to verify that any member you're trying to use is public.

Namely, we do not do accessibility checks on the type itself (ie if you really ought to be able to access the type of the object in your current context, or should it just look like an opaque object to you), and do not allow any non-public members to be used dynamically.

The down side of this scenario is that you could make a call with a static receiver to a private method that you know you can access from your context, but because a dynamic argument is given, the runtime binder will prevent you from calling the method. Below is an example:

public class C
{
    private void M(int x) { }

    static void Main()
    {
        dynamic d = 10;
        C c = new C();
        c.M(d);
    }
}

When the compiler encounters this expression at compile time, it will do the verification and know that C.M is accessible from the calling context, but because the argument is dynamic, the call gets resolved at runtime. Because of the public-only policy of the current binder, overload resolution will not bind to the method.

Conclusions?

As always, I love getting your feedback, whether positive or negative. But this post in particular I would love to get your thoughts on! The design is not set in stone, so any of your thoughts will definitely be personally brought to the design team by yours truly. Thanks for your comments in advance, and as always, happy coding!

Dynamic in C# V: Indexers, Operators, and More!

samng — Thu, 11 Dec 2008 21:30:40 GMT

Now that we're all experts in how dynamic invocations work for regular method calls, lets extrapolate from our previous discussion about phantom methods a bit and take a look at how those basic concepts apply to other dynamic operations.

Today we'll just go through a laundry list of each type of operation, and throw in a few caveats and gotchas (limitations really, but that's such a negative word) that come along with the whole package. As always, I'll try to give some insights as to why we made the decisions that we did, and if there are workarounds for certain scenarios, I'll definitely point them out.

So without further ado, lets hit the list!

Properties

Named properties take the form d.Foo, where d is some dynamic object and Foo is some name for some field or property that lives on the runtime type of d. When the compiler encounters this, it encodes the name "Foo" in the payload, and instructs the runtime binder to bind off of the runtime type of d.

Note however, that named properties are always used in context! You can do one of three things with these guys - access the value, assign a value to the member, or do both (compound operations, such as += etc). The compiler will thus encode the intent of the usage in the payload as well, so that the runtime will allow you to bind to a get-only property only if you're trying to access it, and will throw you an error if you're trying to assign to it.

The thing to note here is that the compiler will treat any named thing the same, and allow the runtime to differentiate between properties and fields.

The return type of these guys is dynamic at compile time.

Indexers

You can think of indexers in one of two ways - properties with arguments, or method calls with a set name. The latter is a much more useful way to think of these guys when we're dealing with dynamic. The reason is that just like method calls, even if the indexer itself can be statically bound, any dynamic arguments that don't directly map to dynamic can cause the phantom overload to come into play, and cause a late binding based on the static type of the receiver, and the dynamic types of the arguments.

They still do have some similarities to properties however - they're always used in context. As such, the compiler again will encode whether or not the user is accessing the value of the indexer, setting a value, or performing a compound operation into the payload for the binder to use.

The return type of these guys is also dynamic at compile time.

Conversions

Last time we mentioned that the although dynamic is not convertible to any other type, there are certain scenarios in which we allow it to be convertible. Assignments, condition expressions, and foreach iteration variables are a few examples.

These payloads are quite simple - because the compiler already knows the type that we're attempting to convert to (ie the type of the variable you're assigning to), it simply encodes the conversion type in the payload, indicating to the runtime binder that it should attempt all implicit (or explicit if its a cast) conversions from the runtime type of the argument to the destination type.

Note that user-defined conversions will be applied as well. We worked pretty hard to make sure that the runtime semantics will behave just like the compile time ones, so argument conversions for overload resolution and the like will all happen exactly as you'd expect.

These guys return the destination type at compile time. Note that these guys are the only guys who have a non-dynamic return type at compile time.

Operators

Operators are a bit of a strange beast. At first glance, it's hard to tell that anything dynamic is going on. However, a simple statement like d+1 still needs to be dispatched at runtime, because user-defined operators can come into play.

As such, any operation that has a dynamic argument will be dispatched at runtime. This includes all of the in place operators as well (+=, -= etc).

Note that the compiler will do the magic to figure out if you've got a member assignment (ie d.Foo += 10) or a variable assignment (ie d += 10), and figures out if it needs to pass d by ref to the call site so that it can be mutated. Note also that structs will get mutated as well! So if we were to do:

public struct S
{
    public int Foo;
}

public class C
{
    static void Main()
    {
        dynamic d = new S();
        d.Foo += 10;
    }
}

the result would be that d would point to a struct who's Foo member is set to 10.

Lastly, note that the compiler knows that if you're doing something like d.Foo += x, and at runtime d.Foo binds to a delegate type or an event type, then the correct combine/add call will be invoked for you.

Delegate invoke

The invocation syntax is very much like a method call. The only difference is that the name of the action is not explicitly stated. This means that just like calls, both of the following examples would end up causing runtime dispatches:

public class C
{
    static void Main()
    {
        MyDel  c = new MyDel();
        dynamic d = new MyDel();
        d();
        c(d);
    }
}

The first example causes a runtime dispatch of an invoke that takes no arguments. At runtime, the binder will check to verify that the recevier is indeed a delegate type, and then perform overload resolution to make sure the arguments supplied match the delegate signature.

The second example causes a runtime dispatch because the argument specified is dynamic. The compiler determines at compile time that we have an invoke of a delegate since c's type is a delegate, but the actual resolution must be done at runtime.

Okay, that's enough laundry listing for today. Next time we'll look at a few small caveats of things that aren't allowed in dynamic contexts. After that, I think we'll switch gears and start looking at some other VS 2010 features - named arguments, optional parameters, and more!

Dynamic in C# IV: The Phantom Method

samng — Mon, 10 Nov 2008 09:16:02 GMT

Yes, this does sound like a Star Wars movie, but no, I'm not a Star Wars geek that just likes to pull lines from my favorite movies (though I rather enjoyed Star Wars). This post will deal with what we've coined "the phantom method". It's the method that the static compiler will bind to during the initial binding phase when it recognizes that the invocation its trying to bind needs to be bound dynamically and cannot be resolved statically. It uses the rules that we talked about last time to determine what types to use at runtime.

Lets consider a simple example:

public class C
{
    public void Foo(int x)
    {
    }

    static void Main()
    {
        dynamic d = 10;
        C c = new C();
        c.Foo(d);
    }
}

When we try to bind the call to Foo, the compiler's overload resolution algorithm will construct the candidate set containing the sole candidate, C.Foo(int). At that point, we consider whether or not the arguments are convertible. But wait! We haven't talked about the convertibility of dynamic yet!

Lets take a quick segue into talking about the convertibility of the dynamic type.

Dynamic conversions

The quick and easy way to think about dynamic conversions is that everything is convertible to dynamic, and dynamic is not convertible to anything. "Wait a sec!", you say. "That doesn't make any sense!" And you're absolutely right, it doesn't make any sense - not until we talk about the special handling of dynamic in situations where you would expect convertibility.

In each of these special situations that you would expect some sort of conversion, the dynamic type signifies that the conversion is to be done dynamically, and the compiler generates all the surrounding DLR code that prompts a runtime conversion.

Lets let the local variable "c" denote some static typed local, and the variable "d" denote some dynamically typed expression. The special situations in question are the following:

Overload resolution - c.Foo(d)
Assignment conversion - C c = d
Conditionals - if (d)
Using clauses - using (dynamic d = ...)
Foreach - foreach (var c in d)

We'll look at overload resolution today and explore the concepts, and leave the remaining scenarios as excercises for the reader. :)

Overload resolution

Back to argument convertibility. Since dynamic is not convertible to anything else, our argument d is not convertible to int. However, since we've got a dynamically typed argument, we really want the overload resolution for this call to be bound dynamically. Enter the phantom method.

The phantom method is a method which is introduced into the candidate set that has the same number of parameters as the number of arguments given, and each of those parameters is typed dynamic.

When the phantom method is introduced into the candidate set, it is treated like any other overload. Recall that since dynamic is not convertible to any other type, but all types are convertible to dynamic. This means that though all (well, not really all, but we'll discuss that later) of the normal overloads will fail due to the dynamic arguments being present, the phantom method will succeed.

In our example, we have one argument which is typed dynamic. We also have two overloads: Foo(int) and Foo(dynamic). The first overload fails because dynamic is not convertible to int. The second, the phantom, succeeds and so we bind to it.

Once a call is bound to the phantom overload, the compiler knows to generate the correct DLR magic to signal dispatching the call at runtime.

Only one question remains: when does the phantom overload get introduced?

Introduction of the phantom overload

When the compiler performs overload resolution, it considers each overload in the initial candidate set. If the invocation has any dynamic arguments, then for each candidate in the initial set, the compiler checks to see if the phantom overload should be introduced. The phantom will be introduced if:

All of the non-dynamic arguments are convertible to their respective parameters.
At least one of the dynamic arguments is not convertible to its respective parameter.

Recall that earlier we had said that it would be possible for a call containing a dynamic argument to be dispatched statically instead of dynamically. This is explained by condition 2. If the overload in question contains dynamic parameters for each of the dynamic arguments, then the binding will be dispatched statically.

The following example will not yield a dynamic lookup, but will be bound statically:

public class C
{
    public void Foo(int x, dynamic y) { ... }

    static void Main()
    {
        C c = new C();
        dynamic d = 10;
        c.Foo(10, d);
    }
}

Once the compiler has gone through each of the overloads in the initial binding pass, if the phantom has not yet been introduced, then overload resolution will behave as it always has, despite the occurrence of a dynamic parameter.

How is the phantom dispatch different than dispatch from a dynamic receiver?

It is important to note that there is a subtle difference between dispatch signaled from the phantom method and dispatch signaled from a dynamic receiver.

With a dynamic receiver, the overloads that the runtime binder will consider are determined based on the runtime type of the receiver. However, with the phantom dispatch, the overloads will be determined based on the compile time type of the receiver.

This is because of intuition - one would expect that though the arguments are dynamic, the receiver is known at compile time, and so the candidate set that the call can dispatch to should be known at compile time as well.

More specifically, one would not expect some overload defined in some derived class (possibly not defined in one's own source!) to be called. This is precisely the consideration we took when designing the behavior of the dynamic dispatch.

So what's next?

Next time, we'll apply the rules and concepts that we talked about today to other invocations - operators, conversions, indexers, and properties.

Dynamic in C# III: A slight twist

samng — Thu, 06 Nov 2008 22:12:42 GMT

Last time we dealt with the basics of dynamic binding. This time, we'll add a small twist.

First, lets recall the example we were using last time:

static void Main(string[] args)
{
    dynamic d = 10;
    C c = new C();

    // (1) Dynamic receivers.
    d.Foo(); // Call.
    d.PropOrField = 10; // Property.
    d[10] = 10; // Indexer.

    // (2) Statically typed receivers (or static methods)
    //     with dynamic arguments.
    c.Foo(d); // Instance method call.
    C.StaticMethod(d); // Static method call.
    c.PropOrField = d; // Property.
    c[d] = 10; // Indexer.
    d++; // Think of this as op_increment(d).
    var x = d + 10; // Think of this as op_add(d, 10).
    int x = d; // Think of this as op_implicit(d).
    int y = (int)d; // Think of this as op_explicit(d).
}

Last time we dealt with the first set of invocations - those with dynamic receivers. This time, we'll deal with the second set - those with either static receivers, or no real apparent receiver.

What do you expect?

Lets take the simplest of this set of invocations and expand it out a bit. Suppose we have something like the following:

public class C
{
    public void Foo(decimal x) { ... }
    public void Foo(string x) { ... }
    static void Main(string[] args)
    {
        C c = new C();
        dynamic d = 10;
        c.Foo(d);
    }
}

First lets consider this from a purely intuitive standpoint. What would we expect to happen?

Since we know the type of our local variable 'c', intuitively, we know that one of the two overloads of Foo on C should be called. However, we also know that d is dynamically typed, so the compiler cannot determine the exact overload to be called until runtime. We would therefore expect the combination of these two to happen - the compiler will determine the candidate set at compile time, and determine which the call should resolve to at runtime. In this case, since d has the value 10 at the time of the call, we would expect the overload of Foo that takes a decimal to be called, since the value 10 is not convertible to type string.

What don't you expect?

Let's be a little more specific here, and expand our example to illustrate what we would NOT expect to have happen:

public class C
{
    public void Foo(decimal x) { ... }
    public void Foo(string x) { ... }
    static void Main(string[] args)
    {
        C c = new D();
        dynamic d = 10;
        c.Foo(d);
    }
}

public class D : C
{
    public void Foo(int x) { ... }
}

First of all, lets notice the subtle change in our source code, highlighted in yellow. We now creating an instance of the derived class D. This means that at runtime, the local variable c will be an instance of type D instead of C as in our previous example. Note also that D contains an overload of Foo that is a better match than all of the overloads on C - the value 10 is intrinsically typed int and so D.Foo is the best match.

However, note that although our example instantiates the local variable c within our code, it is very easy to imagine a method taking a parameter of type C and being given some other derived class at runtime. We do not expect this to change our candidate set used for overload resolution! Specifically (in compiler terminology), since the call to c.Foo can be bound statically to a method group, we expect the statically determined method group to be the one that is used. The dynamic argument should only serve to influence the resolution of the method group, not to influence the creation of the group itself.

What actually happens?

As I mentioned before, one of the design tenets that we've been trying to maintain is that dynamic binding behaves exactly as it would at static compile time, with the exception that the type used in place of the dynamic objects (arguments or receiver) is the runtime determined type instead of the compile time determined one. This means that for all arguments not statically typed dynamic, the compile time types will be used, regardless of their runtime types.

Applying this rule to our example means that at runtime, we should bind as if the type of the receiver c is C, and the type of the argument d is int. Using these types for overload resolution will yield C.Foo(decimal) as the result.

A slightly more complex example to (hopefully) drill home the point:

public class C
{
    public void Foo(object x, C c) { ... }
    static void Main(string[] args)
    {
        C c = new D();
        dynamic d = 10;
        c.Foo(d, c);
    }
}

public class D : C
{
    public void Foo(int x, D d) { ... }
}

Notice in this example that at runtime, c contains an instance of type D, and d contains the value 10. If we were to use the runtime types for everything involved in the binding at runtime, then the receiver would be type D, with the argument types being int and D respectively. This would yield D.foo(int, D) as the best result, but that's not at all what we would expect.

Because the only statically-known dynamically typed argument is the first argument d, it is the only one that has its runtime type used. The remainder of the arguments to the call (the receiver c, and the second argument c) have their static types used. As such, the only method considered is C.Foo(object, C), which is the method we'd expect to have resolved.

What's next?

Next time we'll look deeper at exactly what the compiler does in order to determine that we need a runtime dispatch for the given expression. After that, we'll apply the same principles we've discussed to other invocation types in our example (such as property-accessor-looking things and operators). By the end of it, we'll have turned this whole dynamic thing inside out, so stay tuned!

Dynamic in C# II: Basics

samng — Mon, 03 Nov 2008 02:53:32 GMT

Last time, we began to dive into dynamic binding in C# and what happens through the pipeline. This time, we'll take a simple scenario and pick apart the details of what happens under the covers, both during compile time and runtime.

We can break down what the compiler does into three parts: type and member declarations with dynamics (ie methods that return dynamic), binding and lookup, and emitting. We'll deal now with the binding aspects of dynamic.

Dynamic binding

Dynamic binding itself can be broken into two scenarios. Lets consider the following example.

static void Main(string[] args)
{
    dynamic d = 10;
    C c = new C();

    // (1) Dynamic receivers.
    d.Foo(); // Call.
    d.PropOrField = 10; // Property.
    d[10] = 10; // Indexer.

    // (2) Statically typed receivers (or static methods)
    //     with dynamic arguments.
    c.Foo(d); // Instance method call.
    C.StaticMethod(d); // Static method call.
    c.PropOrField = d; // Property.
    c[d] = 10; // Indexer.
    d++; // Think of this as op_increment(d).
    var x = d + 10; // Think of this as op_add(d, 10).
    int x = d; // Think of this as op_implicit(d).
    int y = (int)d; // Think of this as op_explicit(d).
}

Consider the first set of examples under (1). Each of these dynamic invocations happen off of the dynamically typed expression. It is clear where the dynamicity (yes, I like that word, even though it isn't one...) comes from, and where it goes.

The second set of examples under (2) are a little more complex. The use of dynamic is indirect in each of these. Because the argument to each operation is dynamic, they flow into the containing operation and make them dynamic as well. As such, the compiler does sort of a mix of dynamic binding and static binding - it will use the static type of the receiver to determine the set of members to overload on, but will use the runtime types of the arguments to perform overload resolution.

The first set of examples are much more straight forward to understand, so we'll use this set as our foundation for exploring the feature.

Dynamic receivers

When the compiler encounters an expression typed dynamic, it knows to treat the subsequent operation as a dynamic operation. Whether its an index get, index set, method call etc, the result type of the operation will be determined at runtime, and so at compile time, the result of the operation must also be dynamic.

The compiler transforms all dynamic operations into what we'll call a dynamic call site. This consists of creating a compiler generated static field on a generated static class that stores the DLR site instance for the invocation, and initializing it as necessary.

The DLR call site is a generic object that is generic on the delegate type of the call. More on how this delegate gets generated later. The type names may not be final yet, but currently the creation of the DLR call site takes a CallSiteBinder which is an object that knows how to perform the specific binding that is required for the call site. The DLR provides a set of standard actions that can be used to take advantage of the DLR's support for interop with dynamic objects (more on that in a later post).

The call site contains a field of type T that is an instance of the delegate type that the site is instantiated with. This delegate is used to contain the DLR caching mechanism which you can learn about on Jim Hugunin's blog. It stores the results of each bind and is used to invoke the resulting operation.

Once the call site has been created, the compiler then emits the code to invoke the delegate, passing it the arguments that the user passed to the call site.

What happens at runtime?

Once the compiler has created the DLR call site, it then invokes the delegate, which causes the DLR to do its magic with interop types, and its magic with caching. Assuming that we don't have a true IDynamicObject and we don't have a cache hit, the CallSiteBinder that we seeded the DLR site with will be invoked. C# has its own derived CallSiteBinders that will know how to perform the correct binding, and will return an expression tree which will be merged into the DLR call site's target delegate for caching.

The current caching mechanism simply checks exact type matches on the arguments. For example, suppose our call looks like the following:

arg0.M(arg1, arg2, ...);

And suppose our call has arg0.Type == C, and all the arguments passed to the call are of type int. The cache check would look like the following:

if (arg0.GetType() == typeof(C) &&
    arg1.GetType() == typeof(int) &&
    arg2.GetType() == typeof(int) && 
    ...
    )
{
    Merge CallSiteBinder's bind result here.
}
... // More cache checks
else
{
    Call the CallSiteBinder to bind, and update cache.
}

C# CallSiteBinder creation

The last thing we need to paint a full picture of dynamic binding is to understand what the C# CallSiteBinder implementation does.

In our example, we have 3 different types of dynamic operations. We have a call, a property access, and an indexer. Each of these operations have their own unique pieces to them, but still share much of the common functionality. As such, they all are initialized with a common C# runtime binder, and are used by the runtime binder as data objects that describe the action that needs to be bound. We'll call these objects the C# payloads.

A good way to think of the C# runtime binder is a mini compiler - it has many of the concepts you'd expect in a traditional compiler, such as a symbol table, and a type system, and much of the functionality as well, such as overload resolution and type substitution.

Lets use the simple example of d.Foo(1) for our consideration.

Once the runtime binder gets invoked, it is given the payload for the current call site, and the runtime arguments the site is being bound against. It takes the types of the all the arguments (including the receiver) and populates its symbol table with those types. It then unpacks the payload to find out the name of the operation it's trying to perform on the receiver (in this case, "Foo"), and uses reflection to load all members named "Foo" off of the runtime type of d, putting those members into its symbol table.

From there, we have enough information in the binder's internal system to do the binding that the action describes. At this point, we fork off and bind based on the payload's description.

One of the design choices we made was that the runtime binder should have the exact same semantics that the static compiler has. This includes reporting the same set of errors that the compiler would produce, and perform the same set of conversions (user-defined or otherwise).

As such, each payload is bound exactly as the static compiler would have. The result of the bind is an expression tree that represents the action to take if the binding was successful. Otherwise, a runtime binder exception is thrown. The resulting expression tree is then taken and merged into the call site's delegate to become part of the DLR cache mechanism, and is then invoked so that the result of the user's dynamic bind gets executed.

A slight limitation

As I mentioned, we tried to keep the philosophy of matching exactly what the static compiler would do. However, there are several scenarios that will not work in Visual Studio 2010 that we will hopefully get to in a future release.

Several to note are lambdas, extension methods, and method groups.

Because we currently do not have a way of representing the source of a lambda at runtime without a binding, dynamic invocations that contain lambdas produce a compile time error.

Also, because we don't currently have a way of passing in using clauses and scopes during runtime, extension method lookup will also not be available for this next release.

There is currently no way to represent a method group at runtime (ie there is no MethodGroup type), and so without introducing that concept into .NET, there is no good way for us to allow method groups to be represented dynamically. This means that you cannot do the following:

delegate void D();
public class C
{
    static void Main(string[] args)
    {
        dynamic d = 10;
        D del = d.Foo; // This would bind to a method group at runtime. 
    }
}

Because we cannot represent method groups at runtime, a runtime exception will be thrown if the runtime binder binds d.Foo to a method group.

Hopefully you have a better understanding about the details of what happens when C# performs a dynamic bind. Next time we'll take a look at the second set of scenarios we discussed today. We'll also introduce the Phantom Method, and describe what it does and see how it affects overload resolution.

Until then, happy coding!

Dynamic in C#

samng — Thu, 30 Oct 2008 00:59:42 GMT

The other day I was playing around with some office code, and I found myself writing a lot of code much like the following sample that Anders used at his PDC talk:

static void Main(string[] args)
{
    var xl = new Excel.Application();

    ((Excel.Range)xl.Cells[1, 1]).Value2 = "Process Name";
    ((Excel.Range)xl.Cells[1, 2]).Value2 = "Memory Usage";
}

As you can imagine, it very quickly became tiresome assigning the results of each call to a local variable, debugging and finding out what type it returns for my scenarios, making the cast to the strong type so that I can call certain methods on it, rinse, and repeat.

This pattern is common in dynamic APIs, and cause a lot of excess code to be written that essentially is used just to make the type system happy. Wouldn't it be nice if instead, we could write something like this?

static void Main(string[] args)
{
    var xl = new Excel.Application();

    xl.Cells[1, 1].Value2 = "Process Name";
    xl.Cells[1, 2].Value2 = "Memory Usage";
}

Well, in C# 4.0, we now allow you to write exactly that. One of the main features that we're working on in C# 4.0 is the dynamic late binding feature. This feature allows you to tell the compiler that the thing that I'm returning really ought to be treated like a dynamic type, and that any dispatch on it should be done dynamically. The runtime will then do the binding for you based on the runtime type of the object instead of the static compile time return type, and if the binding succeeds, then the code will succeed. This gives us exactly what we want.

So how does this feature work?

Firstly, we've introduced a dynamic type into the type system. This type indicates to the compiler that all operations based on the type should be bound dynamically and not at compile time. Secondly, we've created a C# runtime binder which does the late binding for you. Lastly, we've baked in the usage of the DLR and are making full use of their caching and dynamic dispatch capabilities, so that you can interop with dynamic objects (objects created from Iron Python for instance).

The dynamic type

In order to start using the dynamic binding, we've got to have some way to signify to the compiler that we want our object or expression to be bound dynamically. Enter the dynamic type.

The dynamic type is just a regular type that you can use in your code to denote local variables, fields, method return values etc. It tells the compiler that everything to do with that object or expression should be done dynamically. Consider the following example:

static void Main(string[] args)
{
    dynamic d = SomeInitializingStatement;

    d.Foo(1, 2, 3); // (1)
    d.Prop = 10; // (2)
    var x = d + 10; // (3)
    int y = d; // (4)
    string y = (string)d; // (5)
    Console.WriteLine(d); // (6)
}

In this example, each of the statements has some element of the dynamic type flowing through it, and is therefore dispatched dynamically. Lets consider each of them.

Since the receiver in this example is typed dynamic, the compiler will indicate to the runtime that it needs to bind some method named "Foo" on whatever the runtime type of d happens to be, with the arguments {1, 2, 3} applied to it.
This example also has a dynamic receiver, and so the compiler indicates to the runtime that it needs to bind a property-looking-thing (could be a field or property) named "Prop", and that it wants to set the value to 10.
In this example, the operator "+" becomes a dynamically bound operation because one of its arguments is dynamically typed. The runtime then does the normal operator overload resolution rules for "+", finding any user-defined operators named "+" on the runtime type of d, and considering that along with the regular predefined binary operators for int.
In this example, we have an implicit conversion from the runtime type of d to int. The compiler signifies to the runtime that it should consider all implicit conversions on int and on the runtime type of d, and determine if there is a conversion to int.
This example highlights an explicit conversion to string. The compiler encodes this cast and tells the runtime to consider explicit casts to string.
In this example, despite the fact that we're calling a statically known method at compile time, we have dynamic arguments. As such, we cannot perform overload resolution correctly at compile time, and so the dynamic-ness of d flows out to its containing call, and we end up dispatching Console.WriteLine dynamically as well.

There are several other scenarios that dynamic flows out to, but I've listed these to give you a general idea of what the dynamic type's implications are.

Now, we should note that the dynamic type is really just syntactic sugar to signify to the compiler that it should treat bindings dynamically. In metadata, dynamic is just object with an attribute signifying its dynamicity (if that's even a word... I don't think it is though!).

What happens at compile time?

For each dynamic operation, the compiler generates calls into the DLR, and takes advantage of its call sites. For more information about the DLR and what call sites are, my colleague Jim Hugunin gave an excellent talk at PDC about it - you can view his blog here, and watch his talk here.

The DLR call site takes a set of standard actions which indicate what type of dynamic action we want to take. The C# compiler emits a subclass of these standard actions, annotated with some C# specific details, and emits invocations of the call sites in place of the call that the user makes. For instance, this code sample gets translated into something like the following pseudocode:

// This code...
static void Main(string[] args)
{
    dynamic d = SomeInitializingStatement;
    d.Foo(1, 2, d);
}

// transforms into this code.
static void Main(string[] args)
{
    dynamic d = SomeInitializingStatement;
    _csharpCallAction = new CSharpCallAction("Foo");
    _dlrSite<T> = new Site<T>(_csharpCallAction); // Create the site. 
    _dlrSite.Target(1, 2, d); // Invoke the delegate. 
}

If you want more information on this, my colleague Chris Burrows has written an excellent blog on the dynamic type and what the compiler generates.

Note that the site creation pseudocode specifies a generic argument, T. This argument is a delegate type that represents the signature of the call. So in our example, our call takes 2 integer arguments and a dynamic argument, and has a dynamic receiver. T would then be a delegate that represents that.

Invoking that delegate invokes the C# runtime binder, which binds the expression based on the runtime types of the arguments and the receiver.

What happens at runtime?

When the DLR delegate gets invoked, it does a couple of cool things that I'll describe briefly. For more information, check out Jim Hugunin's blog.

The DLR checks a cache to see if the given action has already been bound against the current set of arguments. So in our example, we would do a type match based on 1, 2, and the runtime type of d. If we have a cache hit, then we return the cached result.
If we do not have a cache hit, then the DLR checks to see if the receiver is an IDynamicObject. These guys are essentially objects which know how to take care of their own binding, such as COM IDispatch objects, real dynamic objects such as Ruby or Python ones, or some .NET object that implements the IDynamicObject interface. If it is any of these, then the DLR calls off to the IDO and asks it to bind the action.

Note that the result of invoking the IDO to bind is an expression tree that represents the result of the binding.
If it is not an IDO, then the DLR calls into the language binder (in our case, the C# runtime binder) to bind the operation. The C# runtime binder will bind the action, and will return an expression tree representing the result of the bind.
Once step 2 or 3 have happened, the resulting expression tree is merged into the caching mechanism so that any subsequent calls can run against the cache instead of being rebound.

Jim gives a great description of points 1, 2, and 4, which deal with the DLR specifics, so I'm going to elaborate on what happens in step 3.

The C# runtime binder

The C# runtime binder uses Reflection to populate its internal symbol table to determine what to bind to. Each of the C# specific actions encodes the type of the binding, along with extra information that allows us to determine how to bind the action.

For example, if the argument is known at compile time to have a static type, then that type will be marked in the C# action, and will be used as the type of the argument during runtime binding. If it is known at compile time to be typed dynamic (ie it is a variable of type dynamic, or is an expression that returns dynamic), then the runtime binder will use reflection to determine its runtime type and use that type as the type of the argument.

The runtime binder populates its symbol table as needed. For instance, in our example, we were calling the method Foo. The runtime binder will load all members named Foo on the type of the receiver into the symbol table.

It then populates the necessary conversions for each of the argument types. Since we may need to coerce the arguments to types that match the method calls (using user-defined conversions as necessary), the binder loads those conversions into the symbol table as well.

It then performs overload resolution exactly like the static compiler does. That means that we get the exact same semantics as the static compiler. It also means that we get the same error semantics and messages - a failed binding at runtime results in an exception being thrown, which encapsulates the error message that you would have gotten at compile time.

It then takes the result of overload resolution and generates an expression tree that represents the result, and returns that back to the DLR.

A summary

So that's a brief summary of what the dynamic pipeline looks like. Of course, I've glossed over a lot of the details, but I'll be covering those details in my future posts. Until next time, some questions to ponder:

What happens when the receiver is known statically but the arguments are dynamic? What happens if the methods we're trying to bind against are private? What about operators - how does resolution work on them?

These, and more, I'll aim to address in my subsequent posts.

As always, happy coding!