Fabulous Adventures In Coding : Static Methods

Color Color

Eric Lippert — Mon, 06 Jul 2009 16:29:00 GMT

Pop quiz: What does the following code do when compiled and run?

class C
{
    public static void M(string x)
    {
        System.Console.WriteLine("static M(string)");
    }
    public void M(object s)
    {
        System.Console.WriteLine("M(object)");
    }
}
class Program
{
    static void Main()
    {
        C c = new C();
        c.M("hello");
    }
}

(1) writes static M(string)
(2) writes M(object)
(3) uh, dude, this code doesn’t even compile much less run
(4) something else

Think about that for a bit and then try it and see if you were right.

In option (1), the compiler could decide that the best match is the static one and let you call it through the instance, ignoring the instance value. In option (2), the compiler could decide that the object version is the best match and ignore the static one, even though the argument match is better. But neither of those actually happens; the rules of C# say that the compiler should pick the best match based on the arguments, and then disallow static calls that are through an instance! The actual result here is therefore (3):

error CS0176: Member 'C.M(string)' cannot be accessed with an instance reference; qualify it with a type name instead

What is up with this craziness? If you believe that the rule “static methods should never be accessed through instances” is a good rule – and it seems reasonable – then why doesn’t overload resolution remove the static methods from the candidate set when called through an instance? Why does it even allow the “string” version to be an applicable candidate in the first place, if the compiler knows that it can never possibly work?

I agree that this seems really goofy at first.

To explain why this is not quite as dumb as it seems, consider the following variation on the problem. Class C stays the same.

class B
{
public C C = new C();
public void N()
{
C.M("hello");
}
}

What does a call to N on an instance of B do?

(1) writes static M(string)
(2) writes M(object)
(3) compilation error
(4) something else

A bit trickier now, isn’t it? Does C.M mean “call instance method M on the instance stored in this.C?” or does it mean “call static method M on type C”? Both are applicable!

Because of our goofy rule we do the right thing in this case. We first resolve the call based solely on the arguments and determine that the static method is the best match. Then we check to see if the “receiver” at the call site can be interpreted as being through the type. It can. So we make the static call and write “static M(string)”. If the instance version had been the best match then we would successfully call the instance method through the property.

So the reason that the compiler does not remove static methods when calling through an instance is because the compiler does not necessarily know that you are calling through an instance. Because there are situations where it is ambiguous whether you’re calling through an instance or a type, we defer deciding which you meant until the best method has been selected.

Now, one could make the argument that in our first case, the receiver cannot possibly be a type. We could have further complicated the language semantics by having three overload resolution rules, one for when the receiver is known to be a type, one for when the receiver is known to not be a type, and one for when the receiver might or might not be a type. But rather than complicate the language further, we made the pragmatic choice of simply deferring the staticness check until after the bestness check. That’s not perfect but it is good enough for most cases and it solves the common problem.

The common problem is the situation that arises when you have a property of a type with the same name, and then try to call either a static method on the type, or an instance method on the property. (This can also happen with locals and fields, but local and field names typically differ in case from the names of their types.) The canonical motivating example is a public property named Color of type Color, so we call this “the Color Color problem”.

In real situations the instance and static methods typically have different names, so in the typical scenario you never end up calling an instance method when you expect to call a static method, or vice versa. The case I presented today with a name collision between a static and an instance method is relatively rare.

If you’re interested in reading the exact rules for how the compiler deals with the Color Color problem, see section 7.5.4.1 of the C# 3.0 specification.

Why Doesn't C# Implement "Top Level" Methods?

Eric Lippert — Mon, 22 Jun 2009 18:39:00 GMT

C# requires that every method be in some class, even if it is a static method in a static class in the global namespace. Other languages allow "top level" functions. A recent stackoverflow post asks why that is.

I am asked "why doesn't C# implement feature X?" all the time. The answer is always the same: because no one ever designed, specified, implemented, tested, documented and shipped that feature. All six of those things are necessary to make a feature happen. All of them cost huge amounts of time, effort and money. Features are not cheap, and we try very hard to make sure that we are only shipping those features which give the best possible benefits to our users given our constrained time, effort and money budgets.

I understand that such a general answer probably does not address the specific question.

In this particular case, the clear user benefit was in the past not large enough to justify the complications to the language which would ensue. By restricting how different language entities nest inside each other we (1) restrict legal programs to be in a common, easily understood style, and (2) make it possible to define "identifier lookup" rules which are comprehensible, specifiable, implementable, testable and documentable.

By restricting method bodies to always be inside a struct or class, we make it easier to reason about the meaning of an unqualified identifier used in an invocation context; such a thing is always an invocable member of the current type (or a base type).

Now, JScript.NET has this feature. (And in fact, JScript.NET goes even further; you can have program statements "at the top level" too.) A reasonable question is "why is this feature good for JScript but bad for C#?"

First off, I reject the premise that the feature is "bad" for C#. The feature might well be good for C#, just not good enough compared to its costs (and to the opportunity cost of doing that feature instead of a more valuable feature.) The feature might become good enough for C# if its costs are lowered, or if the compelling benefit to customers becomes higher.

Second, the question assumes that the feature is good for JScript.NET. Why is it good for JScript.NET?

It's good for JScript.NET because JScript.NET was designed to be a "scripty" language as well as a "large-scale development" language. "JScript classic"'s original design as a scripting language requires that "a one-line program actually be one line". If your intention is to make a language that allows for rapid development of short, simple scripts by novice developers then you want to minimize the amount of "ritual incantations" that must happen in every program. In JScript you do not want to have to start with a bunch of using clauses and define a class and then put stuff in the class and have a Main routine and blah blah blah, all this ritual just to get Hello World running.

C# was designed to be a large-scale application development language geared towards pro devs from day one; it was never intended to be a scripting language. It's design therefore encourages enforcing the immediate organization of even small chunks of code into components. C# is a component-oriented language. We therefore want to encourage programming in a component-based style and discourage features that work against that style.

This is changing. "REPL" languages like F#, long popular in academia, are increasing in popularity in industry. There's a renewed interest in "scripty" application programmability via tools like Visual Studio Tools for Applications. These forces cause us to re-evaluate whether "a one line program is one line" is a sensible goal for hypothetical future versions of C#. Hitherto it has been an explicit non-goal of the language design.

(As always, whenever I discuss the hypothetical "next version of C#", keep in mind that we have not announced any next version, that it might never happen, and that it is utterly premature to think about feature sets or schedules. All speculation about future versions of unannounced products should be taken as "for entertainment purposes only" musings, not as promises about future offerings.)

We are therefore considering adding this feature to a hypothetical future version of C#, in order to better support "scripty" scenarios and REPL evaluation. When the existence of powerful new tools is predicated upon the existence of language features, that is points towards getting the language features done.

UPDATE: More thoughts on considerations motivating this potential change here.

Calling static methods on type parameters is illegal, part three

Eric Lippert — Thu, 21 Jun 2007 19:37:00 GMT

There were lots of good comments on my previous entries in this series. I want to address some of them, but first I want to wrap this up by considering how a small change to the scenario makes it plausible to choose a different option. Consider now the non-static, non-virtual instance method:

public class C { public void M() { /*whatever*/ } }
public class D : C { public new void M() { /*whatever*/ } }
public class E<T> where T : C { public static void N(T t) { t.M(); } }

I hope you agree that in a sensibly designed language exactly one of the following statements has to be true:

1) This is illegal.
2) E<T>.N calls C.M no matter what T is.
3) E<C>.N calls C.M but E<D>.N calls D.M .

As we've discussed, with static methods we chose the first option. But with instance methods, we choose the second! Our earlier objection to it -- that the user clearly meant for the more derived method to be called -- melts away. Why? Because as far as we are concerned, that might as well have been public static void N(C t) { t.M(); } which you would reasonably expect to always call the less derived method, since its not virtual.

Why not #3? Again, it has to do with static analysis. What it really comes down to is that in both the static and the instance cases, C.M and D.M are entirely different methods. The "new" calls that out; these are two different methods which just happen to share the same name. You can think of every method as having a "slot" in an object; in both the static and instance cases, we have defined two slots, not one. Had this been a virtual override then there would have been just one slot, and the contents of that slot would be determined at runtime. But in the non-virtual case there are two slots.

When the compiler generates the generic code it resolves all the slots at compilation time. The jitter does not change them. Indeed, the jitter does not know how! The jitter has no idea that D.M has anything to do with C.M ; again, they are completely different methods that just coincidentally share a name. They have different slots so they are different methods.

I hope that all makes sense. Now to answer a few selected questions from the comments:

First, a number of readers noted that there are languages (Delphi, Smalltalk) which have implemented the concept of "class" methods. That is, methods which are not associated with an instance (so they are neither instance nor virtual), but nevertheless, which method is called is determined at runtime (so they are not static.) How the .NET versions of these languages do the codegen, I do not know; my guess would be that they emit code that does late binding via reflection or some similar mechanism.

Second, this raises the question of whether C# ought to support some of the dynamic features which languages like JScript, Ruby, Python, etc, support. The VB team's motto as far as this question is concerned has always been early binding when possible, late binding when necessary. The C# team, by contrast, has always cleaved to the principle that we do early binding when possible, late binding when the user explicitly writes umpteen dozen of lines of ugly reflection code.

However, we do recognize both the power of dynamic language features, and the ugliness and pain of explicit reflection. We are considering ways to make late-bound code easier in C# without making it nigh-indistinguishable from early-bound code (as VB sometimes does.) These are very preliminary thoughts; we are still finishing up C# 3.0 here; we've hardly begun thinking about C# 4.0. But when we do, dynamic language features will be high on the list of things to think about. Anyone with bright ideas, by all means, send them my way.

Third, a number of readers noted that the word "static" has been consistently used in C# to mean "associated with a class", and not "determined at compile time". I agree; this is a good example of a poorly named abstraction. Language features should be named based on how the user is intended to use the feature, rather than on how we high-falutin' compiler designers classify different kinds of method calls. We allowed the "static" keyword to be used to mean "associated with a class". Why should "static" mean "associated with a class?" That makes no sense at all. It's really an accident. But it is one we are stuck with now. Part of the reason why I write this blog is to dig into some of these weird historical corners and the associated unfortunate artefacts of less-than-pure design choices.

And fourth, completely off topic, a reader asks us to expose the internals of the compiler's abstract syntax tree. Again, we are in the very, very early stages of designing C# 4.0, so I cannot comment on specific features or timetables. But I will say that many people have asked for this. I also note that in C# 2.0 and 3.0, we concentrated almost entirely on adding language features, so much so that we have fallen behind somewhat on the tools support for them.

Exposing the internals of the compiler so that third parties could more easily create analysis-and-rewriting tools would be a great way to combine the power of the C# semantic analysis engine with the power of community involvement. I encourage anyone who has ideas about what sort of tools they would like to develop, and what sort of API the compiler team could provide to make that easier, to drop me a note. We need all the real-world usage cases we can get in order to decide how to invest our time and energy in the next version.

Calling static methods on type parameters is illegal, part two

Eric Lippert — Mon, 18 Jun 2007 17:14:00 GMT

Last time I pointed out that static methods are always determined exactly at compile time, and used that fact to justify why static methods cannot be called on type parameter types. But aren’t the type arguments to generics actually determined at compile time?

On the caller side they are, sure. But on the callee side, the code emitted at compile time for a generic method is entirely generic. It is not until the jitter encounters the code at runtime that the substitution of type arguments for type parameters is done.

Consider a generic type:

public class C<T> { public string M() { return typeof(T).ToString(); } }

When you compile that, the compiler emits a generic class definition which says just that – that we have a class, it has a type parameter, and a method which calls ToString(). That is all the code which is emitted for this class at compile time.

Let me make sure that is clear. When you say

void N() { C<int> c = new C<int>(); string s = c.M(); //...

the compiler does not emit a copy of the class’s IL with int substituted for T.

Rather, what happens is when your method N is jitted, the jitter says hey, I need to jit up C<int>.M. At that point the jitter consumes the generic IL emitted for C<T>.M and creates brand new x86 (or whatever) code with int substituted for T.

Contrast this with C++ templates. C++ templates do not define generic types. Rather, C++ templates are basically a clever compile-time syntax for some complex search-and-replace macros. If you say C<int> in C++, then at compile time the C++ compiler textually substitutes int for T and emits code as if that was how you had written it in the first place.

If C# had templates instead of macros then a static method called on a template parameter really would be determined at compile time, because the entire constructed class would be resolved at compile time. In this sense templates are a more powerful mechanism than generics – you can do crazy things with templates because there is no type safety imposed upon the template as a whole. Rather, the type safety is only checked for every construction of the template actually in the program.

But C# generic types are not templates; they must be typesafe given any possible construction which satisfies the constraints, not just under the set that they are actually constructed from in a particular program.

Next time I'll consider what impact these design decisions have on non-virtual instance methods called from within generic types.

Calling static methods on type parameters is illegal, part one

Eric Lippert — Thu, 14 Jun 2007 23:24:00 GMT

A developer passed along a question from a customer to our internal mailing list for C# questions the other day. Consider the following:

public class C { public static void M() { /*whatever*/ } }
public class D : C { public new static void M() { /*whatever*/ } }
public class E<T> where T : C { public static void N() { T.M(); } }

That's illegal. We do not allow you to call a static method on a type parameter. The question is, why? We know that T must be a C, and C has a static method M, so why shouldn't this be legal?

I hope you agree that in a sensibly designed language exactly one of the following statements has to be true:

1) This is illegal.
2) E<T>.N() calls C.M() no matter what T is.
3) E<C>.N() calls C.M() but E<D>.N() calls D.M().

(If there is a fourth possible sensible behaviour which I have missed, I am happy to consider its merits.)

If we pick (2) then this is both potentially misleading and totally pointless. The user will reasonably expect D.M() to be called if T is a D. Why else would they go to the trouble of saying T.M() instead of C.M() if they mean "always call C.M()"?

If we pick (3) then we have violated the core design principle of static methods, the principle that gives them their name. Static methods are called “static” because it can always be determined exactly, at compile time, what method will be called. That is, the method can be resolved solely by static analysis of the code.

That leaves (1).

Related questions come up frequently, in various forms. Usually people phrase it by asking me why C# does not support “virtual static” methods. I am always at a loss to understand what they could possibly mean, since “virtual” and “static” are opposites! “virtual” means “determine the method to be called based on run time type information”, and “static” means “determine the method to be called solely based on compile time static analysis”.

(Then again, people occasionally accuse me of “dogmatic skepticism”. Since “dogmatic” and “skeptical” are opposites, I am never sure quite what they mean either. My conclusion: people sometimes say strange things.)

Really what people want I think is yet another kind of method, which would be none of static, instance or virtual. We could come up with a kind of method which behaved like our option (3) above. That is, a method associated with a type (like a static), which does not take a non-nullable “this” argument (unlike an instance or virtual), but one where the method called would depend on the constructed type of T (unlike a static, which must be determinable at compile time).

I’m not sure whether the CLR generic system supports codegenning such a beast, but other than that, I don’t see any in-principle reason why such a thing would be difficult to do. But we do not add language features just because we can; we add them when the compelling benefit outweighs the costs. No one has yet made the case to me that this kind of method would be really useful.

Next time I'll answer the follow-up question: why isn't this determined at compile time? That will take us into the subtle differences between templates and generic types. They are often confused.

Why are overloaded operators always static in C#?

Eric Lippert — Tue, 15 May 2007 00:35:00 GMT

A language design question was posted to the Microsoft internal C# discussion group this morning: "Why must overloaded operators be static in C#? In C++ an overloaded operator can be implemented by a static, instance or virtual method. Is there some reason for this constraint in C#?"

Before I get into the specifics, there is a larger point here worth delving into, or at least linking to. Raymond Chen immediately pointed out that the questioner had it backwards. The design of the C# language is not a subtractive process; though I take Bob Cringely's rather backhanded compliments from 2001 in the best possible way, C# is not Java/C++/whatever with the kludgy parts removed. Former C# team member Eric Gunnerson wrote a great article about how the process actually works.

Rather, the question we should be asking ourselves when faced with a potential language feature is "does the compelling benefit of the feature justify all the costs?" And costs are considerably more than just the mundane dollar costs of designing, developing, testing, documenting and maintaining a feature. There are more subtle costs, like, will this feature make it more difficult to change the type inferencing algorithm in the future? Does this lead us into a world where we will be unable to make changes without introducing backwards compatibility breaks? And so on.

In this specific case, the compelling benefit is small. If you want to have a virtual dispatched overloaded operator in C# you can build one out of static parts very easily. For example:

public class B {
public static B operator+(B b1, B b2) { return b1.Add(b2); }
protected virtual B Add(B b2) { // ...

And there you have it. So, the benefits are small. But the costs are large. C++-style instance operators are weird. For example, they break symmetry. If you define an operator+ that takes a C and an int, then c+2 is legal but 2+c is not, and that badly breaks our intuition about how the addition operator should behave.

Similarly, with virtual operators in C++, the left-hand argument is the thing which parameterizes the virtual dispatch. So again, we get this weird asymmetry between the right and left sides. Really what you want for most binary operators is double dispatch -- you want the operator to be virtualized on the types of both arguments, not just the left-hand one. But neither C# nor C++ supports double dispatch natively. (Many real-world problems would be solved if we had double dispatch; for one thing, the visitor pattern becomes trivial. My colleague Wes is fond of pointing out that most design patterns are in fact necessary only insofar as the language has failed to provide a needed feature natively.)

And finally, in C++ you can only define an overloaded operator on a non-pointer type. This means that when you see c+2 and rewrite it as c.operator+(2), you are guaranteed that c is not a null pointer because it is not a pointer! C# also makes a distinction between values and references, but it would be very strange if instance operator overloads were only definable on non-nullable value types, and it would also be strange if c+2 could throw a null reference exception.

These and other difficulties along with the ease of building your own single (or double!) virtual dispatch mechanisms out of static mechanisms, makes it easy to decide to not add instance or virtual operator overloading to C#.