Sam Ng's Blog : CLR

Optional Modifiers and Overload Resolution

samng — Wed, 05 Mar 2008 09:12:24 GMT

Optional Modifiers (or modopts) are CLR constructs that allow types to be annotated with optional information. This allows compiler writers to annotate their types with additional information that may not have a direct CLR representation. The managed C++ compiler for instance, uses modopts to represent const types.

The C# compiler does not use modopts for anything - rather, we generally use attributes to augment code. For instance, the difference between out and ref is simply an attribute applied to the parameter - out is just ref with an attribute tacked onto it. Since the CLR does not consider attributes on parameters or return types as part of the method signature, methods that differ only in out and ref look the same. This is why the C# language does not allow you to have two overloads which differ only in their out and ref-ness.

The same is not true of modopts, however. The CLR does allow you to differentiate methods by their modopts. The following type is a fragment from a perfectly legal and verifiable assembly:

.class public auto ansi beforefieldinit Modopt
       extends [mscorlib]System.Object
{
.method public hidebysig instance void
          Foo(int32 x) cil managed
{
    ...
} // end of method Modopt::Foo
.method public hidebysig instance void
          Foo(int32 modopt([mscorlib]IsConst) x) cil managed
{
    ...
} // end of method Modopt::Foo
...
} // end of class Modopt

Regardless of the fact that none of the Microsoft .NET languages allow creation of such assemblies (ie C++ does not allow you to distinguish methods only by their const-ness), the CLR allows this, and so compiler implementers that target the CLR must have some story when dealing with these assemblies.

The C# compiler does not understand any modopts. When we encounter assemblies which contain modopts, we simply ignore them and import them as if they were regular members.

When we import methods, we import each methods regardless of whether or not it has any modopts in it. We keep a note of the number of modopts in the method signature, but do not report any errors, even if there exist two imported methods which differ only in the modopts on their signature.

At overload resolution time, we consider each method whose name matches that which we're resolving, and add it to the candidate set. We then filter the candidate set based on arguments and conversions, applying the algorithm described in section 7.4.2 of the C# specification. At that point, if we're left with a candidate set in which all of the methods in the set are identical in signature with the exception of modopts, then we apply the rule that the method with the least number of modopts wins. If there is a tie between two or more methods with the minimal number of modopts, then the compiler reports an ambiguity error.

This has several ramifications.

Firstly, it allows methods with modopts not understood by the compiler to be called from the user's C# code. This has the arguable semantics of allowing the user to call a C++ function which expects a const argument with one which is not const, for instance. The philosophy of whether or not this is desirable can be argued, but by definition, these constructs are optional, so according to the definition, this behavior is perfectly acceptable.

Secondly, it allows importing assemblies which have the illegal-in-C# behavior of being overloaded only by modopts, and allows calling these methods from C# assemblies. Since the compiler doesn't understand the modopts, applying a simple deterministic heuristic seems as good behavior as one can expect.

"Why can't we do better than applying a heuristic?" you ask. Well, we could do better than that. We could note which modopt type is used for each parameter, and despite not understanding it, use it as part of the better-ness algorithm in overload resolution. That would allow the compiler to do something like the following:

In some assembly created from some other language:

modopt(MyModopt) Foo();
void Bar(modopt(MyModopt) int x) { ... }
void Bar(int x) { ... }

And then in C#, for the call Bar(Foo()) we could determine that the modopt-ed Bar should be called because the return type of Foo has the matching modopt. However, as my colleague Eric Lippert noted, "C# is not a glue language. If you want a glue-language, use VBA or something". Nicely put. :)

Generic Method Substitutions and Unification - Part two

samng — Tue, 04 Mar 2008 07:24:56 GMT

Last time we talked about generic method substitutions which resulted in types being declared with more than one method with identical constructed signatures. After thinking long and hard about this problem, we've come to a conclusion as to how to resolve this issue in the next release.

First, lets recap what we know about the problem.

We currently produce code which causes the CLR to throw TypeLoadExceptions at runtime.
Code that falls into this scenario can be easily fixed by a simple rename
It is a corner case - typical coding practices will not get you into this scenario
The code written is not CLS-compliant

The following is a general set of categories that can result in this scenario occurring:

// Same class
class Base<T>
{
    void Foo(T t);
    void Foo(int t);
}
 
class C : Base<int> { }
 
// Inherited from base class
class A
{
    void Foo(int i) { }
}
 
class B<T> : A
{
    void Foo(T t) { }
}
 
class C : B<int> { }
 
// Outer class
class A<T>
{
    private void Foo(T t) { }
    private void Foo(int i) { }
 
    class C : A<int> { }
}
 
// Outer class and its base class
class A<T>
{
    public void Foo(T t) { }
}
 
class B<T> : A<T>
{
    private void Foo(int i) { }
    class C : B<int> { }
}

Given the above points, we've decided to take the following action:

If the user specifies that the type is CLS-compliant (via the CLSCompliantAttribute attribute), the compiler will produce a warning indicating that the constructed type is not CLS-compliant
If the user does not specify CLS compliance, or specifies that the type is not CLS-compliant, then we will keep our current behavior, allow the type to be emitted, and produce an assembly that will throw a runtime exception.

This behavior will make it into the next release of the compiler. Until then, don't write code like this if you want it to work! Stay tuned for my investigations on the C# Compiler and modopts!

generic method substitutions and unification

samng — Wed, 13 Feb 2008 20:17:03 GMT

It's been a while since I've last written - my apologies. We've been hard at work figuring out what the next release of C# will look like, and I'm happy to say that I'm very excited about what we're working on. Great minds are at work figuring out things like integration of C# with the DLR, dynamic late binding, and better debugging experiences as we speak. Don't worry, I'll be sure to report to you about what they determine. :)

I'm always fascinated by discovering subtle details in the language that baffle me. Today I came across such a case. Consider the following code:

using System;

abstract class A<T>
{
    public abstract void Foo(T x);
}

abstract class B<T, S> : A<T>
{
    public virtual void Foo(S x){}
}

abstract class C<T, S> : B<T, S>
{
    public abstract override void Foo(T x);
}

class D : C<int, int>
{
    public override void Foo(int x) {}

    static void Main()
    {
        new D().Foo(1);
    }
}

What happens here?

Well, on the surface, this looks like it should be correct right? A's Foo and B's Foo are different, since they have different generic type parameters, and C clearly overrides A's Foo. But the weird part is D's Foo. Does it override A::Foo? Or B::Foo?

Well, it turns out that the specification outlines that when we look up members to override, we look from our current parent up - that is, when we find a method in one base class, we use that, regardless of other matches in higher ancestral classes. So since C is D's direct base class, D::Foo overrides C::Foo, which overrides the abstract A::Foo.

So we should be good here right? A::Foo has an implementation in D::Foo, and B::Foo has its own implementation as well. But we're not. We get a runtime type load exception, which complains that D has a method Foo with no body.

Well that's odd! I thought we just said that everything has an implementation!

So, being the inquisitive minded person that I am, (and of course, having a bug open to fix this issue), I went and poked around, and discovered that we could fix this problem by having the compiler emit the .override instruction, so that the CLR will know exactly which method we're overriding.

Well neat, that fixes that problem. Or so I thought!

Bug number two looks a little different, but very similar.

using System;

class A<T>
{
    protected virtual void Foo(T x){}
}

class B<T, S> : A<T>
{
    public virtual void Foo(S x) {}
}

class C<T, S> : B<T, S>
{
    protected override void Foo(T x) { }
}

class D : C<int, int>
{
    protected override void Foo(int x) {}
    
    static void Main()
    {
        new D();
    }
}

In this scenario, the compiler again allows us to compile the code without problems, and once again we get a runtime exception. This time, we get an exception saying that we're attempting to lower the access of B::Foo. Interesting. This looks like my first bug! So what do I do? I apply the fix to my first bug and see what happens.

No luck.

"But why not?", you ask? Well, after more digging (and finding the right contacts on the CLR team to chat with), we discovered that the initial fix was wrong! It turns out that by specifying the .override command, we tell the CLR to override A::Foo, but the CLR also finds B::Foo and tells D:Foo to override that as well.

At the end of it all, we discovered that the entire scenario is an error. Why? The CLI spec, chapter 11.9.9 "Inheritance and overriding" outlines:

Type definitions are invalid if, after substituting base class generic arguments, two methods result in the same name and signature (including return type). The following illustrates this point:

[Example:

.class B`1<T>

{ .method public virtual void V(!0 t) { … }

.method public virtual void V(string x) { … }

}

.class D extends B`1<string> { } // Invalid

Class D is invalid, because it will inherit from B<string> two methods with identical signatures:

void V(string)

At the end of it all, we discovered that we have a disconnect between the CLR and the C# compiler here. We're currently working on determining what C# spec changes we'll need to make to clarify this case, and from there we'll put a fix together.

So what did I learn today? Two things:

1) Our customer that found this bug (I only know them as nikov) is great at finding an exploit, and then massaging it in all possible ways to find other problems (which I might add, is phenomenal).

2) People should just not code like this cause its confusing and doesn't work :)