Fabulous Adventures In Coding : Performance

String interning and String.Empty

Eric Lippert — Mon, 28 Sep 2009 16:23:00 GMT

Here's a curious program fragment:

object obj = "Int32";
string str1 = "Int32";
string str2 = typeof(int).Name;
Console.WriteLine(obj == str1); // true
Console.WriteLine(str1 == str2); // true
Console.WriteLine(obj == str2); // false !?

Surely if A equals B, and B equals C, then A equals C; that's the transitive property of equality. It appears to have been thoroughly violated here.

Well, first off, though the transitive property is desirable, this is just one of many situations in which equality is intransitive in C#. You shouldn't rely upon transitivity in general, though of course there are many specific cases where it is valid. As an exercise, you might want to see how many other intransitivities you can come up with. Post 'em in the comments; I'd love to see what obscure ones you can come up with. (Incidentally, one of the interview questions I got when applying for this team was to invent a performant algorithm for determining intransitivities in a simplified version of the 'better method' algorithm.)

Second, what's happening here is we're mixing two different kinds of equality that just happen to use the same operator syntax. We're mixing reference equality with value equality. Objects are compared by reference; in the first and third comparison we are testing if the two object references both refer to exactly the same object. In the second comparison we are checking to see if the two strings have the same content, regardless of whether they are the same object or not. In fact, the compiler warns you about this situation; this should produce a "possible unintended reference comparison" warning.

That might need a bit more explanation. In .NET you can have two strings that have identical content but are different objects. When you compare those strings as strings, they're equal, but when you compare them as objects, they're not.

That explains why the second comparison is true -- it's a value comparison -- and why the third comparison is false -- it's a reference comparison. But it doesn't explain why the first and third comparisons are inconsistent with each other.

This is the result of a small optimization. If you have two identical string literals in one compilation unit then the code we generate ensures that only one string object is created by the CLR for all instances of that literal within the assembly. This optimization is called "string interning".

String.Empty is not a constant, it's a read-only field in another assembly. Therefore it is not interned with the empty string in your assembly; those are two different objects.

This explains why the first comparison is true: the two literals in fact get turned into the same string object. And it explains why the third comparison is false: the literal and the computed value are turned into different objects.

Knowing that, you can now make an educated guess as to why we have this bizarre behaviour:

object obj = "";
string str1 = "";
string str2 = String.Empty;
Console.WriteLine(obj == str1); // true
Console.WriteLine(str1 == str2); // true
Console.WriteLine(obj == str2); // sometimes true, sometimes false?!

Some versions of the .NET runtime automatically intern the empty string at runtime, some do not!

But why, you might ask, do we not perform this interning optimization at runtime on every string? Why not aggressively turn all value-equal strings into reference-equal strings? Surely it is wasteful to have two identical strings around when you could have half as much memory.

The answer is that the TANSTAAFL Principle applies here, bigtime. That is, There Ain't No Such Thing As A Free Lunch. Interning has two positive effects: it decreases memory consumption and decreases time required to compare two strings. (Because if all strings are interned at runtime then all string comparisons can be cheap reference comparisons.) But those positive effects have a cost: allocating a new string now requires that you do a search of all string objects in memory to see if you have one that matches already. In our existing optimization, the cost is small; we can know at compile time what string literals are in a given assembly and which are identical. With the proposed optimization, that cost is imposed at runtime, and it could be a very large fraction of the time spent allocating strings.

In order to keep the time cost down, you'd have to build a hash table of all strings in memory. That means either computing the hashes frequently, which is itself expensive in time, or storing the hashes somewhere. If we do the latter then suddenly we are increasing the memory burden for strings that are not duplicated. That is, our optimization makes the normal scenario -- the vast majority of pairs of strings are not equal to each other -- take up more memory, so that a rare scenario saves on memory. That seems like a bad bargain; you usually want to optimize for the likely case.

There are also serious lifetime problems with interned strings. When can they be safely garbage collected? What if a new copy of the string is created while the old one is being collected on another thread? The safest thing to do is to make interned strings immortal, which looks like a memory leak. Memory leaks are bad for performance, particularly when the optimization you're doing is an attempt to save memory. TANSTAAFL!

In short, it is in the general case not worth it to intern all strings. However, it might be worth it in some specific cases. For example, if you were building a compiler in C#, odds are good that you are going to be producing a lot of strings that are the same at runtime. Our C# compiler is written in C++, in which we have written our own custom string interning layer so that we can do cheap reference comparisons on all strings in your program. Odds are good that "int" is going to appear tens, hundreds or thousands of times in a given program; it seems silly to allocate the same string over and over again. If you were writing a compiler in C#, or had some other application in which you felt that it was worth your while to ensure that thousands of identical strings do not consume lots of memory, you can force the runtime to intern your string with the String.Intern method.

Conversely, if you hate interning with an unreasoning passion, you can force the runtime to turn off all string interning in an assembly with the CompilationRelaxation attribute.

Anyway, to come back to the question of transitivity: object reference equality actually is transitive. It's also symmetric (A==B implies B==A) and reflexive (A==A), so it is an equivalence relation. Similarly, string value equality is transitive, symmetric and reflexive, since it uses a straight "character by character" ordinal comparison. But when you mix the two, then equality is no longer transitive. That's weird, but hopefully now understandable.

The Stack Is An Implementation Detail, Part One

Eric Lippert — Mon, 27 Apr 2009 16:15:00 GMT

" style="BORDER-TOP-WIDTH: 0px; DISPLAY: inline; BORDER-LEFT-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; MARGIN-LEFT: 0px; MARGIN-RIGHT: 0px; BORDER-RIGHT-WIDTH: 0px" height=240 alt="Stack" src="http://blogs.msdn.com/blogfiles/ericlippert/WindowsLiveWriter/TheStackIsAnImplementationDetail_C978/Stack_thumb_1.jpg" width=156 align=left border=0 mce_src="http://blogs.msdn.com/blogfiles/ericlippert/WindowsLiveWriter/TheStackIsAnImplementationDetail_C978/Stack_thumb_1.jpg">I blogged a while back about how “references” are often described as “addresses” when describing the semantics of the C# memory model. Though that’s arguably correct, it’s also arguably an implementation detail rather than an important eternal truth. Another memory-model implementation detail I often see presented as a fact is “value types are allocated on the stack”. I often see it because of course, that’s what our documentation says.

Almost every article I see that describes the difference between value types and reference types explains in (frequently incorrect) detail about what “the stack” is and how the major difference between value types and reference types is that value types go on the stack. I’m sure you can find dozens of examples by searching the web.

I find this characterization of a value type based on its implementation details rather than its observable characteristics to be both confusing and unfortunate. Surely the most relevant fact about value types is not the implementation detail of how they are allocated, but rather the by-design semantic meaning of “value type”, namely that they are always copied “by value”. If the relevant thing was their allocation details then we’d have called them “heap types” and “stack types”. But that’s not relevant most of the time. Most of the time the relevant thing is their copying and identity semantics.

I regret that the documentation does not focus on what is most relevant; by focusing on a largely irrelevant implementation detail, we enlarge the importance of that implementation detail and obscure the importance of what makes a value type semantically useful. I dearly wish that all those articles explaining what “the stack” is would instead spend time explaining what exactly “copied by value” means and how misunderstanding or misusing “copy by value” can cause bugs.

Of course, the simplistic statement I described is not even true. As the MSDN documentation correctly notes, value types are allocated on the stack sometimes. For example, the memory for an integer field in a class type is part of the class instance’s memory, which is allocated on the heap. A local variable is hoisted to be implemented as a field of a hidden class if the local is an outer variable used by an anonymous method(*) so again, the storage associated with that local variable will be on the heap if it is of value type.

But more generally, again we have an explanation that doesn’t actually explain anything. Leaving performance considerations aside, what possible difference does it make to the developer whether the CLR’s jitter happens to allocate memory for a particular local variable by adding some integer to the pointer that we call “the stack pointer” or adding the same integer to the pointer that we call “the top of the GC heap”? As long as the implementation maintains the semantics guaranteed by the specification, it can choose any strategy it likes for generating efficient code.

Heck, there’s no requirement that the operating system that the CLI is implemented on top of provide a per-thread one-meg array called “the stack”. That Windows typically does so, and that this one-meg array is an efficient place to store small amounts of short-lived data is great, but it’s not a requirement that an operating system provide such a structure, or that the jitter use it. The jitter could choose to put every local “on the heap” and live with the performance cost of doing so, as long as the value type semantics were maintained.

Even worse though is the frequently-seen characterization that value types are “small and fast” and reference types are “big and slow”. Indeed, value types that can be jitted to code that allocates off the stack are extremely fast to both allocate and deallocate. Large structures heap-allocated structures like arrays of value type are also pretty fast, particularly if you need them initialized to the default state of the value type. And there is some memory overhead to ref types. And there are some high-profile cases where value types give a big perf win. But in the vast majority of programs out there, local variable allocations and deallocations are not going to be the performance bottleneck.

Making the nano-optimization of making a type that really should be a ref type into a value type for a few nanoseconds of perf gain is probably not worth it. I would only be making that choice if profiling data showed that there was a large, real-world-customer-impacting performance problem directly mitigated by using value types. Absent such data, I’d always make the choice of value type vs reference type based on whether the type is semantically representing a value or semantically a reference to something.

UPDATE: Part two is here

*******

(*) Or in an iterator block.

Representation and Identity

Eric Lippert — Thu, 19 Mar 2009 16:46:00 GMT

I get a fair number of questions about the C# cast operator. The most frequent question I get is:

short sss = 123;
object ooo = sss;            // Box the short.
int iii = (int) sss;         // Perfectly legal.
int jjj = (int) (short) ooo; // Perfectly legal
int kkk = (int) ooo;         // Invalid cast exception?! Why?

Why? Because a boxed T can only be unboxed to T. (*) Once it is unboxed, it’s just a value that can be cast as usual, so the double cast works just fine.

Many people find this restriction grating; they expect to be able to cast a boxed thing to anything that the unboxed thing could have been cast to. There are ways to do that, as we’ll see, but there are good reasons why the cast operator does what it does.

To understand why this design works this way it’s necessary to first wrap your head around the contradiction that is the cast operator. There are two (¤) basic usages of the cast operator in C#:

My code has an expression of type B, but I happen to have more information than the compiler does. I claim to know for certain that at runtime, this object of type B will actually always be of derived type D. I will inform the compiler of this claim by inserting a cast to D on the expression. Since the compiler probably cannot verify my claim, the compiler might ensure its veracity by inserting a run-time check at the point where I make the claim. If my claim turns out to be inaccurate, the CLR will throw an exception.
I have an expression of some type T which I know for certain is not of type U. However, I have a well-known way of associating some or all values of T with an “equivalent” value of U. I will instruct the compiler to generate code that implements this operation by inserting a cast to U. (And if at runtime there turns out to be no equivalent value of U for the particular T I’ve got, again we throw an exception.)

The attentive reader will have noticed that these are opposites. A neat trick, to have an operator which means two contradictory things, don’t you think?

This dichotomy motivates yet another classification scheme for conversions (†). We can divide conversions into representation-preserving conversions (B to D) and representation-changing conversions (T to U). (‡) We can think of representation-preserving conversions on reference types as those conversions which preserve the identity of the object. When you cast a B to a D, you’re not doing anything to the existing object; you’re merely verifying that it is actually the type you say it is, and moving on. The identity of the object and the bits which represent the reference stay the same. But when you cast an int to a double, the resulting bits are very different.

All the built-in reference conversions are identity-preserving (£). Obviously trivial “conversions” such as converting from int to int are also representation-preserving conversions. All user-defined conversions (§) and non-trivial value type conversions (such as converting from int to double) are representation-changing conversions. Boxing and unboxing conversions are all representation-changing conversions.

The representation-preserving conversions that are known to never fail often result in no codegen at all (₪). If a representation-preserving conversion could fail then a castclass instruction is emitted, which does a runtime check and throws if the check fails.

But each representation-changing conversion is handled in its own special way. User-defined conversions are resolved using a special version of the overload resolution algorithm, and generated as a call to the appropriate static method. Boxing and unboxing conversions are generated as box and unbox instructions. All the other built-in conversions (int to double, and so on) are generated as custom sequences of instructions that do the right conversion.

So now that you know that, consider what the compiler would have to do to make this work the way some people expect:

int kkk = (int) ooo;

All that the compiler knows is that ooo is of type object. It could be anything. Suppose it is a boxed int – then the compiler should generate an unboxing instruction. Suppose it is a boxed short. Then the compiler should unbox the short and then generate the custom sequence of instructions that convert a short to an int. Suppose it is a boxed double – same thing, but different instructions. And so on, for all the built-in conversions that go to integer.

This would be a huge amount of code to generate, and it would be very slow. The code is of course so large that you would want to put it in its own method and just generate a call to it. Rather than do that by default, and always generate code that is slow, large and fragile, instead we’ve decided that unboxing can only unbox to the exact type. If you want to call the slow method that does all that goo, it’s available – you can always call Convert.ToInt32, which does all that analysis at runtime for you. We give you the choice between “fast and precise” or “slow and lax”, and the sensible default is the former. If you want the latter then call the method.

That’s just the built-in conversions. Let’s continue imagining what would have to happen if we wanted all possible conversions to int to just work out correctly at runtime, instead of just bailing out early if the boxed thing is not an int.

Suppose the object is a Foo where there is a user-defined conversion from Foo (or one of its base classes) to int (or a type implicitly convertible to int, like, say, Nullable<int>). Then the compiler would need to generate a call to that conversion method, just as it would if the type had been known at compile time, and then possibly also generate the conversion from the return type of the method to int.

Remember, there could be arbitrarily many such conversion methods on arbitrarily many types. The type Foo and its conversion method might not even be defined in the assembly currently being compiled or any assembly referenced. Therefore the compiler would have to generate code to interrogate Foo at runtime, do the overload resolution analysis, and then dynamically spit the code to do the call.

Which is exactly what the compiler does in C# 4.0 if the argument to the cast is of type “dynamic” instead of object. The compiler actually generates code which starts a mini version of the compiler up again at runtime, does all that analysis, and spits fresh code. This is not fast, but it is accurate, if that’s what you really need. (And the spit code is then cached so that the next time this call site is hit, it is much faster.)

I don’t think people really expect the compiler to start up again at runtime every time they cast an object to int; I think they just haven’t thought through carefully exactly how much analysis solving the problem would take. Rather a lot, it turns out.

*************

(*) Or Nullable<T>.

(¤) There are others that are not germane to this discussion. For example, a third usage is “Everyone knows that this D is also of base type B; I want the compiler to treat this expression of type D as a B for overload resolution purposes.” That would clearly be an identity-preserving conversion.

(†) There are many ways to classify conversions; we already divide conversions into implicit/explicit, built-in/user-defined, and so on. For the purposes of this discussion we’ll gloss over the details of those other classifications.

(‡) I’m glossing over here that certain conversions that the C# compiler thinks of as representation-changing are actually seen by the CLR verifier as representation-preserving. For example, the conversion from int to uint is seen by the CLR as representation-preserving because the 32 bits of a signed integer can be reinterpreted as an unsigned integer without changing the bits. These cases can be subtle and complex, and often have an impact on covariance-related issues; see next footnote.

I’m also ignoring conversions involving generic type parameters which are not known at compile time to be reference or value types. There are special rules for classifying those which would be major digressions to get into.

(£) This is why covariant and contravariant conversions of interface and delegate types require that all varying type arguments be of reference types. To ensure that a variant reference conversion is always identity-preserving, all of the conversions involving type arguments must also be identity-preserving. The easiest way to ensure that all the non-trivial conversions on type arguments are identity-preserving is to restrict them to be reference conversions.

(§) The rules of C# prohibit all user-defined conversions that could possibly be identity-preserving coercions. More generally, all user-defined conversions that could possibly be any "standard" conversion are illegal.

(₪) Again, I’m ignoring irksome generic issues here. There are situations where humans can prove mathematically that two generic type parameters must be identical at runtime, but the verifier is not smart enough to make those same deductions and requires the compiler to emit type checks.

Santalic tailfans, part two

Eric Lippert — Fri, 06 Feb 2009 21:19:31 GMT

As I have said before many times, there is only one sensible way to make a performant application. (As an aside: perfectly good word, performant, deal with it!) That is:

Set meaningful, measurable, customer-focused goals.
Write the code to be as clear and correct as possible.
Carefully measure your performance against your goals.
Did you meet your goal? Great! Don't waste any time on performance analysis. Spend your valuable time on features, documentation, bug fixing, robustness, security, whatever.
If you did not meet your goal, use tools to discover what the worst-performing fixable thing is, and fix it.

Iterate on that process, updating your goals if it proves necessary, until you either have something that meets your goals or you give up.

My explicit goal for my little dictionary search program was that it give results fast enough that I would not be sitting there impatiently waiting for more than a few seconds. That's a very customer-focused goal, me being the only customer of this program. With only a very small amount of tweaking my program met that goal right away, so why would I spend any more time on it? The program takes just under two seconds to report the results for a typical query, which is faster than I can read.

But suppose that we did want to improve the performance of this program for some reason. How? Well, let's go down the list. We have goals, we have very clear code, we can measure the performance easily enough. Suppose we didn't meet the goal. The last thing on the list is "use tools to discover what the slowest fixable thing is".

A commenter conjectured that the performance bottleneck of this program was in the disk I/O. As you can see, every time we do a query we re-read the two megabyte dictionary file dozens of times. This has the benefit of using very little memory; we never have more than one line of the dictionary in memory at a time, instead of the whole four megs it would take to store the dictionary (remember, the dictionary is in ASCII on disk but two-byte Unicode if in strings in memory, so the in-memory size will double.)

That's a conjecture -- a reasonable conjecture -- but nevertheless, it's just a guess. If I've learned one thing about performance analysis it's that my guesses about where the bottleneck is are often wrong. I'll tell you right now that yes, the disk-hitting performance is bad, but it is not the worst thing in this program, not by far. Where's the real performance bottleneck? Any guesses? Could you know without using tools?

Here's the result of a timing profile run of a seven-letter queries with one blank, repeated 20 times:

43%: Contains
21%: ReadLine
14%: Sort
7%: ToUpperInvariant
15%: everything else

Holy goodness! The call to the Contains extension method in the query to test whether the dictionary word is in the rack set array is almost half the cost of this program!

Which makes sense, once you stop to think about it. The "Contains" method is by its highly general nature necessarily very naive. When given an array, 99.9% of the time it has to look through the entire 26-item array because 99.9% of the time, the word is not actually going to match any of the possible racks. It cannot take advantage of any "early outs" like you could do if you were doing a linear search on a sorted list. And each time through the array it has to do a full-on string comparison; there's no fancy-pants checks in there that take advantage of string immutability or hash codes or any such thing.

We have a data structure that is designed to rapidly tell you whether a member is contained in a set. And even better, it already does the "distinct" logic. When I replace

var racks = (from rack in ReplaceQuestionMarks(originalRack)
select Canonicalize(rack)).Distinct().ToArray();

with

    var racks = new HashSet<string>(
                from rack in ReplaceQuestionMarks(originalRack)
                select Canonicalize(rack));

suddenly performance improves massively. The "Contains" drops down to 3% of the total cost, and of course, the total cost is now half of what it was before.

Another subtle point here: notice how when I changed the type of the variable "racks" from "array of string" to "set of string", I didn't have to redundantly change the type thanks to implicit typing of local variables. I want to emphasize the semantics here, not the storage mechanism. If I felt that communicating the storage mechanism was an important part of this code -- because it has such a strong effect on performance -- perhaps I would choose to emphasize the storage by eschewing the "var".

With this change, the program performance improves to about one second per query and the profile now looks like this:

39%: ReadLine
23%: Sort
11%: ToUpperInvariant
7%: the iterator block goo in FileLines
5%: ToArray (called by Join)
15%: everything else

Now the bottleneck is clearly the combination of repeated file line reading (48%) and the string canonicalization of every dictionary line over and over again (37%).

With this information, we now have data with which to make sensible investments of time and effort. We could cache portions of the dictionary in memory to avoid the repeated disk cost. Is the potential increase in speed worth the potentially massive increase in memory usage? We could be smart about it and, say, only cache the seven- and eight-letter words in memory.

We could also attack the canonicalization performance problem. Should we perhaps precompute an index for the dictionary where every string is already in canonical form? This in effect trades increased disk space, increased program complexity and increased redundancy for decreased time. Should we use a different canonicalization algorithm entirely?

All of these decisions are driven by the fact that I have already exceeded my performance goal, so the answer is "no". Good enough is, by definition, good enough. If I were using this algorithm to actually build a game AI, it would not be good enough anymore and I'd go with some more clever solution. But I'm not.

A nasality talisman for the sultana analyst

Eric Lippert — Thu, 05 Feb 2009 00:52:48 GMT

The other day my charming wife Leah and I were playing Scrabble Brand Crossword Game (a registered trademark of Hasbro and Mattel) as is our wont. I went first, drawing the Q and a bunch of vowels. Knowing that the Q is death to hold onto, I immediately opened with QI for 22 points. I silently thanked Miriam-Webster for adding QI, KI and ZA to the OSPD 4th edition.

My thankfulness was short-lived, as Leah thought for a few moments and then played ANALYST, for the fifty point "bingo" bonus, making QIS for good measure along the way. She then went on to thoroughly cream me. I am not very good at Scrabble.

We were wondering afterwards what all the possible seven and eight letter "bingo" words were that she could have made with the rack AALNST?. (A blank is conventionally written as "?".) I stared at it for a few moments and found SULTANA and SEALANT, but I was suspicious that there were a lot more. So I wrote a program to find out, which I shall share with you now.

(An interesting historical note: solving this problem efficiently was one of the questions I was given during my interviews when I first applied for full-time work at Microsoft; if you want to get a job here, you might need to know this!)

The core of the program is a method SearchDictionary which takes a rack string and returns a sequence containing every word in a dictionary file which can be formed using all the letters in that rack. In this case I wanted to know not just what all the words matching the given rack were, but what all the words matching the given rack that used an existing letter on the board were. In this case the only two letters on the board were Q and I, but let's assume that it could have been any letter on the board.

The main loop of my little console program looks like this:

public static void Main()
{
    while (true)
    {
        Console.Write("Enter rack (use '?' for blank): ");
        string rack = Console.ReadLine();
        if (rack == "")
            break;
        Console.WriteLine("{0} : {1}", rack, SearchDictionary(rack).Join());
        foreach (char c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ")
            Console.WriteLine("{0}+{1} : {2}", rack, c, SearchDictionary(rack + c).Join());
    }
}

Pretty straightforward so far. Of course, this is certainly not production quality code. This is a little utility that I hacked together for myself. A production-quality version would have a lot more error handling, for one thing.

The Join method is just a handy helper function that sticks a sequence of strings together:

private static string Join(this IEnumerable<string> strs)
{
return string.Join(" ", strs.ToArray());
}

The trick to efficiently searching for anagrams in a dictionary is to realize that all anagrams have the same letters, just in different order. If you "canonicalize" each word so that its letters are uppercase and in alphabetical order, then checking whether one word is an anagram of another is as simple as comparing their canonical forms:

private static string Canonicalize(string s)
{
    char[] chars = s.ToUpperInvariant().ToCharArray();
    Array.Sort(chars);
    return new string(chars);
}

I had a performance goal for this project; I wanted to be able to use the ~2MB 2006 Tournament Word List, searching for racks with up to two blanks, in reasonable human scale time, but not necessarily appearing to be instantaneous. My first naive implementation did not meet this goal so I made a few tweaks to the algorithm until it did, and then I stopped. (It is interesting to think about how this could be made much faster, but that's a subject for another day.)

private static IEnumerable<string> SearchDictionary(string originalRack)
{
const string dictionary = @"d:\twl06.txt";

    // Calculate all the possible distinct values for the rack.
    // As an optimization, stuff the resulting racks in an array so
    // that we do not recalculate them during the query.

var racks = (from rack in ReplaceQuestionMarks(originalRack)
select Canonicalize(rack)).Distinct().ToArray();

    // Check every line in the dictionary to see if it matches
    // any possible rack. As an optimization, do an early
    // out if the line length does not match the query length.

    return from line in FileLines(dictionary)
           where line.Length == originalRack.Length
           where racks.Contains(Canonicalize(line))
           select line;
}

LINQ queries are awesome. I love how the code reads like a description of what I'm trying to do, rather than how I'm doing it.

We need a way to turn a rack that might contain blanks into a sequence of the 26 or 26x26 possible racks without blanks. Here's a handy recursive method that does so:

private static IEnumerable<string> ReplaceQuestionMarks(string s)
{
    int index = s.IndexOf('?');
    if (index == -1)
    {
        yield return s;
        yield break;
    }
    foreach (char c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ")
    {
        string s2 = s.Substring(0, index) + c.ToString() + s.Substring(index + 1);
        foreach (string result in ReplaceQuestionMarks(s2))
            yield return result;
    }
}

And of course, the code to extract the lines from the dictionary is our old friend:

private static IEnumerable<string> FileLines(string filename)
{
    using (var sr = File.OpenText(filename))
    {
        while (true)
        {
            string line = sr.ReadLine();
            if (line == null)
                yield break;
            yield return line;
        }
    }
}

And there you go. Six very brief little methods that tell you that Leah could have made the seven letter bingos ANALYST CANTALS LATINAS PLATANS SALTANT SALTPAN SEALANT and SULTANA or by going through the "I", the eight letter bingos ALATIONS ANNALIST FANTAILS LANITALS NASALITY PLATINAS SANTALIC STAMINAL TAILFANS TALISMAN and VALIANTS.

Arrays considered somewhat harmful

Eric Lippert — Mon, 22 Sep 2008 21:04:00 GMT

I got a moral question from an author of programming language textbooks the other day requesting my opinions on whether or not beginner programmers should be taught how to use arrays.

Rather than actually answer that question, I gave him a long list of my opinions about arrays, how I use arrays, how we expect arrays to be used in the future, and so on. This gets a bit long, but like Pascal, I didn't have time to make it shorter.

Let me start by saying when you definitely should not use arrays, and then wax more philosophical about the future of modern programming and the role of the array in the coming world.

You probably should not return an array as the value of a public method or property, particularly when the information content of the array is logically immutable. Let me give you an example of where we got that horridly wrong in a very visible way in the framework. If you take a look at the documentation for System.Type, you'll find that just looking at the method descriptions gives one a sense of existential dread. One sees a whole lot of sentences like "Returns an array of Type objects that represent the constraints on the current generic type parameter." Almost every method on System.Type returns an array it seems.

Now think about how that must be implemented. When you call, say, GetConstructors() on typeof(string), the implementation cannot possibly do this, as sensible as it seems.

public class Type {
   private ConstructorInfo[] ctorInfos;
   public ConstructorInfo[] GetConstructors()
   {
     if (ctorInfos == null) ctorInfos = GoGetConstructorInfosFromMetadata();
     return ctorInfos;
   }

Why? Because now the caller can take that array and replace the contents of it with whatever they please. Returning an array means that you have to make a fresh copy of the array every time you return it. You get called a hundred times, you’d better make a hundred array instances, no matter how large they are. It’s a performance nightmare – particularly if, like me, you are considering using reflection to build a compiler. Do you have any idea how many times a second I try to get type information out of reflection? Not nearly as many times as I could; every time I do it’s another freakin’ array allocation!

The frameworks designers were not foolish people; unfortunately, we did not have generic types in .NET 1.0. clearly the sensible thing now for GetConstructors() to return is IList<ConstructorInfo>. You can build yourself a nice read-only collection object once, and then just pass out references to it as much as you want.

What is the root cause of this malaise? It is simple to state: The caller is requesting values. The callee fulfills the request by handing back variables.

An array is a collection of variables. The caller doesn’t want variables, but it’ll take them if that’s the only way to get the values. But in this case, as in most cases, neither the callee nor the caller wants those variables to ever vary. Why on earth is the callee passing back variables then? Variables vary. Therefore, a fresh, different variable must be passed back every time, so that if it does vary, nothing bad happens to anyone else who has requested the same values.

If you are writing such an API, wrap the array in a ReadOnlyCollection<T> and return an IEnumerable<T> or an IList<T> or something, but not an array. (And of course, do not simply cast the array to IEnumerable<T> and think you’re done! That is still passing out variables; the caller can simply cast back to array! Only pass out an array if it is wrapped up by a read-only object.)

That’s the situation at present. What are the implications of array characteristics for the future of programming and programming languages?

Parallelism Problems

The physics aspects of Moore’s so-called “Law” are failing, as they eventually must. Clock speeds have stopped increasing, transistor density has stopped increasing. The laws of thermodynamics and the Uncertainty Principle are seeing to that. But manufacturing costs per chip are still falling, which means that our only hope of Moore’s "Law" continuing to hold over the coming decades is to cram more and more processors into each box.

We’re going to need programming languages that allow mere mortals to write code that is parallelizable to multiple cores.

Side-effecting change is the enemy of parallelization. Parallelizing in a world with observable side effects means locks, and locks means choosing between implementing lock ordering and dealing with random crashes or deadlocks. Lock ordering requires global knowledge of the program. Programs are becoming increasingly complex, to the point where one person cannot reasonably and confidently have global knowledge. Indeed, we prefer programming languages to have the property that programs in them can be understood by understanding one part at a time, not having to swallow the whole thing in one gulp.

Therefore we tools providers need to create ways for people to program effectively without causing observable side effects.

Of all the sort of “basic” types, arrays most strongly work against this goal. An array’s whole purpose is to be a mass of mutable state. Mutable state is hard for both humans and compilers to reason about. It will be hard for us to write compilers in the future that generate performant multi-core programs if developers use a lot of arrays.

Now, one might reasonably point out that List<T> is a mass of mutable state too. But at least one could create a threadsafe list class, or an immutable list class, or a list class that has transactional integrity, or uses some form of isolation or whatever. We have an extensibility model for lists because lists are classes. We have no ability to make an “immutable array”. Arrays are what they are and they’re never going to change.

Conceptual Problems

We want C# to be a language in which one can draw a line between code that implements a mechanism and code that implements a policy.

The “C” programming language is all about mechanisms. It lays bare almost exactly what the processor is actually doing, providing only the thinnest abstraction over the memory model. And though we want you to be able to write programs like that in C#, most of the time people should be writing code in the “policy” realm. That is, code that emphasizes what the code is supposed to do, not how it does it.

Coding which is more declarative than imperative, coding which avoids side effects, coding which emphasizes algorithms and purposes over mechanisms, that kind of coding is the future in a world of parallelism. (And you’ll note that LINQ is designed to be declarative, strongly abstract away from mechanisms, and be free of side effects.)

Arrays work against all of these factors. Arrays demand imperative code, arrays are all about side effects, arrays make you write code which emphasizes how the code works, not what the code is doing or why it is doing it. Arrays make optimizing for things like “swapping two values” easy, but destroy the larger ability to optimize for parallelism.

Practical Problems

And finally, given that arrays are mutable by design, the way an array restricts that mutability is deeply weird. All the contents of the collection are mutable, but the size is fixed. What is up with that? Does that solve a problem anyone actually has?

For this reason alone I do almost no programming with arrays anymore. Arrays simply do not model any problem that I have at all well – I rarely need a collection which has the rather contradictory properties of being completely mutable, and at the same time, fixed in size. If I want to mutate a collection it is almost always to add something to it or remove something from it, not to change what value an index maps to.

We have a class or interface for everything I need. If I need a sequence I’ll use IEnumerable<T>, if I need a mapping from contiguous numbers to data I’ll use a List<T>, if I need a mapping across arbitrary data I’ll use a Dictionary<K,V>, if I need a set I’ll use a HashSet<T>. I simply don’t need arrays for anything, so I almost never use them. They don’t solve a problem I have better than the other tools at my disposal.

Pedagogic Problems

It is important that beginning programmers understand arrays; it is an important and widely used concept. But it is also important to me that they understand the weaknesses and shortcomings of arrays. In almost every case, there is a better tool to use than an array.

The difficulty is, pedagogically, that it is hard to discuss the merits of those tools without already having down concepts like classes, interfaces, generics, asymptotic performance, query expressions, and so on. It’s a hard problem for the writer and for the teacher. Fortunately, for me, it's not a problem that I personally have to solve.

Trivial Projections Are (Usually) Optimized Away

Eric Lippert — Mon, 12 May 2008 16:57:00 GMT

OK, computers aren't entirely dumb when it comes to LINQ. Here's an example of a place where we're a bit smarter.

Consider the following query:

IEnumerable<int> query = from n in number_array orderby n select n;

Does this get transformed by the compiler into

IEnumerable<int> query = number_array.OrderBy(n => n);

IEnumerable<int> query = number_array.OrderBy(n => n).Select( n => n );

This question caused tremendous debate amongst the members of the C# design and implementation teams. I am quite happy with the compromise we achieved.

The sections of the C# specification which cover this topic are sections 7.15.2.3, and 7.15.2.5:

***

A query expression of the form “from x in e select v” is translated into “( e ) . Select ( x => v )” except when v is the identifier x, the translation is simply “( e )”

A query expression of the form “from x in e select x” is translated into “( e ) . Select ( x => x )”.

A degenerate query expression is one that trivially selects the elements of the source. A later phase of the translation removes degenerate queries introduced by other translation steps by replacing them with their source. It is important however to ensure that the result of a query expression is never the source object itself, as that would reveal the type and identity of the source to the client of the query. Therefore this step protects degenerate queries written directly in source code by explicitly calling Select on the source. It is then up to the implementers of Select and other query operators to ensure that these methods never return the source object itself.

***

It might sound like the second statement contradicts the first. The first statement says that the “.Select(x=>v)” is omitted when “v” is “x”, and the second statement says that no, the “.Select(x=>x)” is in fact generated.

The spec is not particularly clear on this point, I agree, though the explanatory text about “degenerate query expressions” does help clear it up somewhat. Basically what we are saying here is that the optimization to remove degenerate selects is only performed on the result of an earlier query translation.

Some examples will help.

If you have from n in number_array orderby n select n then this is first translated into from n in (number_array).OrderBy(n=>n) select n and then, since the selector is the same as the range variable, this is translated into ((number_array).OrderBy(n=>n)).

We optimize away the Select(n=>n) because from n in (number_array).OrderBy(n=>n) select n is the intermediate result of an earlier query translation.

If you had from n in number_array select n then this would be considered a degenerate query expression requiring a Select. We cannot simply translate it into (number_array) because then you could assign that to a variable and have referential identity with the collection. The result of the query should not “leak” information about the collection out; it should just be an enumerable object of the appropriate type, and not have an identity relationship. Therefore, we take the performance hit and protect the identity of the collection by emitting a degenerate query translation (number_array).Select(n=>n).

This is important because the collection might be a mutable collection. One does not expect that the result of a query over that collection could be used to mutate the collection! The query results should be read-only.

We do not need to append the degenerate Select(n=>n) when you have an “orderby” in there because the OrderBy(n=>n) call already protects the identity of the original collection. This saves on the performance cost of the no-op Select.

I think we’ve made a reasonable compromise in the case of degenerate queries. If you are crazy enough to write “from x in xs select x” when you mean “(xs)”, then you’ll have to live with the decreased performance. But if you are introducing a degenerate select at the end of a meaningful query, then it will be optimized away.

Computers are dumb

Eric Lippert — Fri, 09 May 2008 18:20:00 GMT

A few short takes today, from questions I've received recently about LINQ in C# 3.0.

The first question was "in the following code, does it really check every single non-negative integer, or does it use the knowledge that once you're beyond ten, you can stop iterating?"

var smallNumbers = Enumerable.Range(0, int.MaxValue).Where(n => n < 10);
foreach (int i in smallNumbers)
Console.WriteLine(i);

The former. You asked LINQ to Objects to apply a predicate to a sequence and that's what it does. If the sequence has two billion elements in it, it runs the predicate two billion times.

You certainly could build your own range object which had a Where method that took an expression tree. Your Where method could then analyze the range and the expression tree in an attempt to build a new range object that iterated a smaller range. Analyzing simple predicates would be pretty easy; more complex predicates would have to be turned back into delegates.

Of course, if it hurts when you do that, don't do that. if you already know the range you want, just restrict it in the range constructor in the first place!

The second question was "which is the preferred coding style?" :

if (0 != customers.Count()) ...

if (customers.Any()) ...

On efficiency grounds alone the latter is far preferable. Suppose you have a box which may contain between zero and two billion pennies. You want to take some action if there are any pennies in the box, but you don't care how many there are, only whether there are zero or more. In that scenario, clearly it makes no sense whatsoever to count them all. The Count method doesn't know that all you care about is whether it's going to return zero or not. You asked it to count, so it counts.

(I am reminded of an old joke: on a bank teller's first day, he is asked to count a stack of twenties and verify that there are one hundred in the stack. He starts counting them from one pile into another: one, two, three, ... but when he gets to fifty, he stops and says to his boss "I'll stop here -- if it's right all the way up to fifty, odds are good it's right the whole way.")

Even leaving the obvious efficiency concern aside, the latter code reads better. It more clearly reflects the intention of the programmer.

The third question was "when iterating through a collection, I noticed that the former code is much faster than the latter code. Any insight as to why?"

myEventLogEntryCollection.CopyTo(myEntriesArray,0);
for(int i=0; i < myEntriesArray.Length, i++) {...}

foreach(EventLogEntry myEntry in myEventLogEntryCollection) {...}

This made no sense to me. Sure, there will be differences. The former code is using more memory because it makes a copy of the collection; initializing that memory is going to take significant time if the collection is large, but I could see how maybe the array access could be faster than the collection iterator in the loop. Perhaps the improvement in iteration is fast enough that for a large collection, the larger initialization cost was made up. But the user was reporting "much faster", not "slightly faster". I asked for more details.

What exactly were the timings? Three seconds vs sixteen seconds. Holy goodness! I had figured we were talking about microseconds of difference here, not actual humans sitting around watching the clock time. What on earth was the user doing? Accessing event logs from a remote machine over a slow network.

Aha! Now it is clear what is going on here. The iteration time is being dominated by the round trip to the remote machine; the former code makes one very expensive remote call that takes three seconds. The latter code makes hundreds of cheaper remote calls, possibly several per loop iteration, that take a fraction of a second each but which add up to more expense in the long run.

My "insights" were therefore:

You can sometimes trade increased memory use for decreased time.
The cost of fetching n items by making m requests might be dominated by m, not n.
If you don't actually describe the relevant features of the problem, it is impossible to get a correct analysis.

I blogged about these three questions not because I got them all in the same day, but because there is a common thread to the answers: computers are dumb. They do exactly what you tell them. If you want performance savings by eliminating unnecessary work through logical deductions about what work can be eliminated, you're going to have to be the one to do make those deductions!

Lambda Expressions vs. Anonymous Methods, Part Five

Eric Lippert — Wed, 28 Mar 2007 17:45:00 GMT

Last time I demonstrated that the compiler could have to do an exponential number of bindings in order to determine whether there was a unique best overload resolution for a function call that takes a lambda. Some of you may have wondered whether we simply were not being clever enough in the compiler. Perhaps there is some way to optimize this problem so that the unique solution is found in less than exponential time.

As it turns out, the question of whether there is a clever way to do this in C# 3.0 is equivalent to solving the most famous unsolved problem in computer science. That is a problem which has stumped generations of the finest minds in academia, and is widely believed to be unsolvable, so I don't feel particularly bad about not solving this one myself.

Consider the following slight revision of the set of overloads I sketched out last time:

class MainClass
{
    class T{}
    class F{}
    delegate void DT(T t);
    delegate void DF(F f);
    static void M(DT dt)
    {
        System.Console.WriteLine("true");
        dt(new T());
    }
    static void M(DF df)
    {
        System.Console.WriteLine("false");
        df(new F());
    }
    static T Or(T a1, T a2){return new T();}
    static T Or(T a1, F a2){return new T();}
    static T Or(F a1, T a2){return new T();}
    static F Or(F a1, F a2){return new F();}
    static T And(T a1, T a2){return new T();}
    static F And(T a1, F a2){return new F();}
    static F And(F a1, T a2){return new F();}
    static F And(F a1, F a2){return new F();}
    static F Not(T a){return new F();}
    static T Not(F a){return new T();}
    static void MustBeT(T t){}
    static void Main()
    {
        // Introduce enough variables and then encode any Boolean predicate:
        // eg, here we encode (!x3) & ((!x1) & ((x1 | x2) & (x2 | x3)))
        M(x1=>M(x2=>M(x3=>MustBeT(
          And(
            Not(x3),
            And(
              Not(x1),
              And(
                Or(x1, x2),
                Or(x2, x3))))))));
    }
}

This expression makes the compiler solve the Boolean satisfiability problem (aka "SAT"). If SAT has a unique solution then the program compiles and produces the unique solution. If it has more than one solution then compilation fails with an ambiguity error. If it has no solution then the program fails with the error that MustBeT has an F argument. But no matter how you slice it, the compiler must solve SAT.

Determining whether a given predicate has a set of valuations for its variables which make it true is an NP-complete problem; actually finding the set of valuations is an NP-hard problem. Therefore, overload resolution in C# 3.0 is at the very least NP-hard. There is no known polynomial-time algorithm for any NP-complete or NP-hard problem and it is widely believed that there is none to be found, though that conjecture has yet to be proven. So our dream of fast lambda type analysis in C# 3.0 is almost certainly doomed, at least as long as we have the current rules for overload resolution.

Does this really matter in practice? As I mentioned last time, hopefully not. Other languages also have this issue. When I ran an early draft of this post past him, Erik Meijer immeditely pointed out that the ML type inference system has similar worst cases, and the Haskell type inference system is even worse. Apparently in Haskell you can encode a Turing machine into the type system and make the compiler run it! In practice these sorts of problems do not arise in real-world code in any of these languages, so I am not too worried about it for C# 3.0.

Lambda Expressions vs. Anonymous Methods, Part Four

Eric Lippert — Tue, 27 Mar 2007 02:05:00 GMT

Hey all, sorry for the long time between posts; I have been crazy busy recruiting, interviewing, fixing bugs, making performance improvements and implementing last-minute changes to the language and expression tree library. The last few posts about lambda binding yielded many interesting comments which I hope to address over the next month or so.

Before that though -- back when I started this series of posts I noted that lambda expressions have an interesting property. Namely, the semantic analysis of the lambda body may depend upon what type the lambda is being converted to. This seemingly trivial fact has an important impact upon the compiler performance during analysis. At long last, let me demonstrate the impact this fact has.

Consider the following set of overloads:

int M(Func<int, int> f){ /* whatever ... */ }
string M(Func<string, string> f){ /* whatever ... */ }

Now suppose we have a call with a lambda argument:

M(x=>x.ToUpper());

During overload resolution we make a list of all the potential overloads and we see which are applicable. That is, which overloads have the property that all of the arguments are convertible to the types of their corresponding formal parameters. Then of all the applicable methods, we pick the best match if there happens to be more than one.

Therefore we must try this binding two ways. We try it as though it were

M((int x)=>x.ToUpper());

and

M((string x)=>x.ToUpper());

The former produces an error while binding the body because there is no ToUpper method on int. The latter binds the body correctly and the return type matches the expected return type, so this one succeeds. The first M is not applicable so it is discarded, the second one is applicable, so it is chosen, all is good in the world.

Or is it?

What if we did something crazy like this?

M(x1=>M(x2=>x1.ToLower() + x2.ToUpper());

OK, so now what do we do? We try to do overload resolution on the outer call. Again, we must try both int and string as candidates for x1. But then when we get to the outer body, we are faced with the same problem for x2.

We have to try all four possibilities for {x1, x2} (that is, {int, int}, {int, string}, {string, int} and {string, string}) in order to determine how many of them work, and of the ones that work, which one is the best. In this particular example "both are string" is the only one that works. (We could with a little more cleverness come up with examples where there were multiple possible matches that worked.)

So perhaps you see where this is going. If there are m overloads that are nested in this manner n deep, then the compiler must try mⁿ possibilities in order to determine which is the unique best one. If it's 2² = 4, that's not a big deal. If we triple each and go to 6⁶ = 46656, that's rather a lot of binding to do, and it takes up rather a lot of time and memory.

Does this seem like a contrived example? Unfortunately, it is not. In the original prototype of C# 3.0, nested from clauses in queries were translated into nested lambdas in exactly this manner, and most of the translation methods have at least two overloads, some have many more. We were easily getting into situations where simple queries could involve millions of body bindings. Fortunately we realized the problem in time; my colleague Wes came up with an alternative translation strategy which does not generate nested lambdas so aggressively and we therefore avoid this problem in the common case. You can still get into it yourself though; please do not.

Next time I'll create a truly contrived (and rather hilarious) example which demonstrates just how bad things can get and just how hard you can make the compiler work to do overload resolution on lambdas.

250% of what, exactly?

Eric Lippert — Thu, 01 Sep 2005 20:13:00 GMT

I just got a question this morning about how to take two collections of items and determine how many of those items had the same name. The user had written this straightforward but extremely slow VBScript algorithm:

For Each Frog In Frogs
For Each Toad In Toads
    If Frog.Name = Toad.Name Then
      SameName = SameName + 1
      Exit For
    End If
Next
Next

There were about 5000 frogs, 3000 toads and 1500 of them had the same name. Every one of the 3500 unsuccessful searches checked all 3000 toads, and the 1500 successful searches on average checked 1500 toads each. Each time through the inner loop does one loop iteration, two calls to the Name property, one string comparison. Add all those up and you get roughly 50 million function calls to determine this count.

This code has been somewhat simplified – the actual user code was doing more work inside the loop, including calls to WMI objects. The whole thing was taking 10+ minutes, which is actually pretty fast considering how much work was being done. Each individual function call was only taking a few microseconds, but fifty million calls adds up!

Now, we've been down this road before in this blog (here, here and here) and so of course I recommended building a faster lookup table rather than doing a full search through the collection every time.

Set FrogLookup = CreateObject("Scripting.Dictionary")
For Each Frog In Frogs
FrogLookup(Frog.Name) = Frog
Next
For Each Toad In Toads
If FrogLookup.Exists(Toad.Name) Then SameName = SameName + 1
Next

Which is much, much faster. That's only about 16 thousand function calls. Now, this is maybe not an apples-to-apples comparison of function calls, but we at least have good reason to believe that this is going to be several times faster.

And indeed it was. But I'm still not sure how much, which brings me to the actual subject of today's blog. The user reported that the new algorithm was "250% better". I hear results like this all the time, and I always have to ask to clarify what it means. You can't just say "n% better" without somehow also communicating what you're measuring.

(UPDATE: This reported result understandably confused some readers. Clearly the new loop here is thousands of times faster, not a mere 250% faster. As I said before, the sketch above is highly simplified code. The real-world code included many thousands of additional calls to WMI objects which were not eliminated by this optimization. Eliminating these 50 million function calls helped -- you should always eliminate the slowest thing first! But doing so also exposed a new "hot spot" that needed further optimization. However, the point of this article is not the benefits of using lookup tables, but rather that using unexplained percentages to report performance results is potentially misleading. The result above is merely illustrative. See the comments for more details.)

Allow me to illustrate. Suppose I have a web server that is serving up ten thousand pages every ten seconds. I make a performance improvement to the page generator so that it is now serving up fifteen thousand pages every ten seconds. I can sensibly say:

performance has improved by 50%, because we are now serving up 5000 more pages every ten seconds, and 5000 is 50% of the original 10000. In this world, any positive percentage is good.
performance is now 150% of original performance because 15000 is 150% of 10000. In this world, 0%-100% worse or the same, 100%+ is good.
We've gone from 1000 microseconds per page to 667 microseconds per page, saving 333 microseconds per page. 333 is 33% of 1000, so we've got a 33% performance improvement. In this world, 0% is bad, 100% is perfect, more than 100% is nonsensical.

If I can sensibly say that performance is better by 50%, 150% and 33%, all referring to exactly the same improvement, then I cannot actually be communicating any fact to my listeners! They need to know whether I'm measuring speed or time, and if speed, whether I'm comparing original speed to new speed or original speed to the difference.

So what is "250%" better? Clearly not raw timings. If this loop took 10 minutes to run before, 250% of 10 minutes is 25 minutes, but surely it is not running in -15 minutes now! I assume that the measurement is speed -- say, number of loops runnable per hour. If before it took ten minutes and therefore we could run this loop six times per hour, and now it is 250% better, 250% of 6 is 15, and this is "better", so we need to add. So that's 21 loops per hour, or about 3 minutes per loop. Had the user accidentally said "250% faster" but meant to say "250% of previous performance" then we'd conclude that 250% of 6 per hour is 15 per hour, so now we've got 4 minutes per loop.

And of course, this is assuming that the person reporting the improvement is actually trying to communicate facts to the audience. Be wary! Since 33%, 50% and 150% are all sensible ways to report this improvement, which do you think someone who wants to impress you is going to choose? There are opportunities here for what Kee Dewdney calls "percentage pumping" and other shyster tricks for making weak numbers look more impressive. Professor Dewdney's book on the subject, "200% of Nothing", is quite entertaining.

The moral of the story is: please do not report performance improvements in percentages. Report performance improvements in terms of raw numbers with units attached, such as "microseconds per call, before and after change", or "pages per second, before and after change". That gives the reader enough information to actually understand the result.

(And the other moral is, of course, lookup tables are your friends.)

Caching Dispatch Identifiers Is A Bad Idea

Eric Lippert — Thu, 25 Aug 2005 01:22:00 GMT

About two years ago I wrote a bit about when it was legal to cache a dispatch identifier so that you didn’t have to look it up a second time. I was reminded of this today because the scripting sustaining engineering team was working on a bug that involved a customer incorrectly caching a dispatch identifier. I’d like to take this opportunity to expand a bit on the rules I mentioned two years ago.

Suppose you've got two script engines that are totally identical – same script code added in the same order, similarly the same named items – running on two different threads. I’m sure you can see why that would be useful; think of an ASP server serving up two pages on two threads, for example. The customer needed to call a particular script function by invoking each script engine’s global dispatch. They were getting the dispatch identifier once, caching it and using it for calling both engines. Depending on what build of the script engine they were using, that either worked fine or died horribly.

Unfortunately it turns out that dying horribly is both legal and expected. A given dispatch identifier can only be cached in the context of a particular dispatch object. Two identical dispatch objects with identical functions are allowed to give different dispatch identifiers for a given method!

The script engines have changed their algorithm for how dispatch identifiers are generated several times over the years. In some builds of the script engines, dispatch identifiers are generated sequentially, so that the first function added is one, the second is two, and so on. In those builds, two identical engines that have had the same functions added in the same order will have identical dispatch identifiers for a given name. In other builds of the script engines, the dispatch identifiers are globally unique, so that two engines with two functions will always have different dispatch identifiers even if everything is identical otherwise.

Unfortunately the customer built and tested their code against the former kind of engine and then upgraded to a more recent version using the latter algorithm, which broke them. I regret the problem -- I'm the guy who screwed around with the algorithm -- but still, you've got to follow the rules. Calling GetIdsOfNames is almost always very fast compared to Invoke, so please don’t cache dispatch identifiers for performance reasons. We don’t do so in the script engines, as I mentioned two years ago, because the trouble you get into when you do it wrong is not worth the time you save.

While I’m on the subject, a quick word of advice for implementers too: If you are implementing IDispatch on an expando object it is very tempting to say "I’ll use a pointer to my function object as the dispatch identifier for that function". Just cast the pointer to int and you’ve guaranteed uniqueness and have an easy way to map back to the function when invoked, right? Please do not do this. First off, it’s not portable to 64 bit machines. Second, it’s dangerous – a hostile or broken caller can make you invoke pointers to garbage. Third, on processes with 3GB of user-accessible memory, pointers can be cast to negative integers and negative integers are illegal dispatch ids. Come up with some safer way to guarantee uniqueness of your dispatch ids, such as maintaining a global counter and lookup table.

Scripting Type Library Constant Injection Performance Characteristics, Part Three

Eric Lippert — Fri, 08 Jul 2005 00:20:00 GMT

We’ve got a system that meets all our needs for disambiguating hierarchical constants through partial or full qualification. (In the event of a collision between an enumerated type name and a value in a given library, the type name wins so that full qualification can work.) But what about the performance costs compared to declaring constants the old fashioned way?

There are two costs to consider: the per-use runtime cost, and the page startup runtime cost. Let's look at the per-use cost first. When you inject

typelib Cheese {
enum Blue {
Stilton = 1
Gorgonzola = 2
}
}

and then have a line of code that says

foo.bar = Stilton

in ASP, what happens? First VBScript searchs the top-level global named items for a match for Stilton -- we check Response, no, that's not Stilton. We check Server, Request, all the rest of the ASP objects, and do not find a match. Finally we check Cheese. None of these are Stilton.

Then we check the default dispatch objects. The ASP objects are not added with the "qualification is optional" flag, so we skip them. We do however ask Cheese if it has a Stilton property, and it does.

So far at runtime we’ve done one table search on the global nametable and one on the Cheese object. Had there been many type libraries injected we would have potentially had to search all of them as well looking for the property. (We cannot compute a fast lookup table for these "second level" names because the named items might be expando objects that change without warning.)

The hash tables are pretty fast, but since the hash tables are case-insensitive, the lookup also involves hashing and canonicalizing the string for every lookup.

And of course, so far we’ve just determined what property on what object corresponds to this name. We then have to do a property fetch on that name to get the value.

Declared constants, by contrast, are not looked up at all. When you say

Const Yellow = 3
Foo.bar Yellow

the compiler determines that Yellow is a constant and generates the same code as if you'd said Foo.bar 3. The lookup cost at runtime is zero for constants, compared to a potentially large number of hash table lookups plus an IDispatch Invoke call for the typelib case.

Now it should be clear why the runtime cost is much higher for type library constants compared to lexical constants. Now consider the startup cost.

When the script first runs, space has to be reserved for all the constants. Their names and values need to be stored. (We need to store the names in case someone uses Eval to evaluate an expression containing a constant; the compiler spun up by Eval will not know that the original compiler for the program had defined any constants.) This is no more expensive than any other local variable assignment with a literal on the right hand side. At startup time, every constant is just another global variable as far as the bytecode interpreter is concerned. (Of course at runtime, the interpreter ensures that attempts to write to these global "variables" fail! They are variables that don’t vary.)

That’s a pretty small startup cost. Not zero, by any means, but not very big.

Now consider the startup cost of initializing the type library objects. If you read in two type libraries with a dozen enumerations, each with say, five properties, then you end up constructing 26 objects, two with 72 properties and 24 with five properties. We have to parse the type libraries to do so, and we do so every time you serve the page.

(I added an optimization to this code which I do not remember if IIS uses or not. The objects we build are free-threaded because they are read-only hash tables. There is a special interface which allows you to parse the type libraries once and re-use the generated object tree in any script engine, thereby eliminating the need to regenerate it every time the script starts up. I think though that IIS does not use this feature.)

The startup cost is enormous for type libraries compared to constants. With constants, the compiler does the work once and the ASP engine caches and reuses the compiled state over and over again without doing a recompilation. With type libraries, every single time we grovel through the type libraries searching for enumerations. Those type libraries can be huge!

Type library groveling is a convenience feature and was not designed for speed. I have never measured the relative speeds, but I'm surprised by the claim that it is only two orders of magnitude slower. I would have expected more – using a constant at runtime is effectively zero cost. We could probably easily come up with scenarios where the throughput of simple pages was thousands of times slower.

Of course, realistically, you have to ask yourself not "what is faster?" but rather "what is fast enough?" and "what is the slowest thing?" Most pages do so many object accesses that these costs are a drop in the bucket. What are a few more object accesses here and there when you've already got thousands on a page? I do not want anyone to walk away from this with the thought that the secret to improving ASP page performance is to use constants instead of type libraries. That will certainly work if type library groveling is the slowest thing on your page. It probably isn’t, but on the other hand, it’s probably not a good idea to have one global page that includes every type library under the sun either.

Scripting Type Library Constant Injection Performance Characteristics, Part Two

Eric Lippert — Wed, 29 Jun 2005 21:02:00 GMT

Script developers can declare constants, variables, functions and classes at global scope by writing the appropriate lines of code. Script hosts (Internet Explorer, ASP, WSH, etc) however can only add objects to the script engine’s global scope. For practical purposes, it’s as though the host creates a new global variable which is assigned to a particular object and cannot be changed.

I suppose that we could have enabled hosts to add named constants pretty easily -- the host could pass in a name and a variant containing the value. We could even do host variables by having the host pass in the address of a variant containing the variable. There's no obvious way to do host-supplied functions other than having the host provide a named object with a default property that calls the function, and now we're back where we started with host-added objects. It's also kind of a pain in the rear to provide a hundred different function objects if you have a hundred functions to inject.

Fortunately we can effectively get host-provided constants, variables and functions even if hosts can only add named objects. How? By enabling the host to pass in flags to say "the members of this object may/must/cannot be accessed by qualifying with the object name".

That's how Internet Explorer does it. IE adds the window object to the script engine with the flag "you may access members of window without qualification". Therefore, both window.document.whatever() and document.whatever() work. ASP, by contrast, sets the flags indicating that members of the Response object must be accessed via Response.Write. Just plain Write doesn’t work.

Given this mechanism, the other mechanisms become unnecessary. If the host wants to add a named constant then the host just creates an object, sets the appropriate flags, and gives the object a property with a getter, no setter. From the scripter's point of view, it’s just like the constant was added to the global namespace.

Back to type library importing. We’ve got

typelib Cheese {
enum Blue{
Stilton = 1
Gorgonzola = 2
}
}

so we want to add two constants to the global namespace, plus two objects, so that we can say

foo.bar = Stilton
foo.bar = Blue.Stilton ' Blue must be an object
foo.bar = Cheese.Blue.Stilton ' Cheese must be an object

At this point we were faced with two choices. We could come up with some new mechanism whereby constants could be added to the script engine, or we could re-use the existing named object injection mechanism. The former wouldn’t have been that hard, but we chose the latter.

When you inject a type library into the engine, what we do is we build up an object with the following properties:

Cheese.Stilton
Cheese.Gorgonzola
Cheese.Blue

where Cheese.Blue is itself an object with the right properties. We then inject Cheese as a named item into the script engine and mark it as "may be qualified".

This solution has the (arguably) nice property that you can disambiguate with only the outer name as well. That is, you can say

foo.bar = Cheese.Stilton

That’s maybe a little weird. We could have instead built the object hierarchy by not putting the bottom-level values on the top-level object, and instead added the top and mid-level objects as named items instead. That is, create Cheese with only property Blue, and Blue with properties Stilton and Gorgonzola, and then add both Cheese and Blue as "may be qualified" named items. That solution presents additional problems though. Suppose we did that, and also added

typelib Food {
enum Cheese {
    Green = 1
    Blue = 2
    Yellow = 3
    White = 4
}
}

We’d then have two top-level items both named Cheese, both with a Blue property, and it would be ambiguous which was which – we’d need to invent a mechanism for resolving conflicts amongst global named items, which is not something we really wanted to do.

So far we’ve seen how type library injection works at a high level. Next time we’ll delve into some of the performance issues that arise.

Scripting Type Library Constant Injection Performance Characteristics, Part One

Eric Lippert — Wed, 22 Jun 2005 17:00:00 GMT

(Sorry about the title. I work for Microsoft; we like nouns.)

Over a year ago now a reader noted in a comment to this posting that defining named constants using Const in VBScript or var in JScript is way, way faster than importing a type library.

My empirical JScript testing showed that loading constants from a typelib is one or two orders of magnitude slower than preloading an engine with script which defines those same constants.

I responded

I wrote all the type library importation code -- you are correct, it is NOT FAST compared to simply defining a bunch of consts/vars. There are good reasons why. I'll blog about them at some point.

At long last I'll get around to explaining why. There are interesting design tradeoffs here, and as often happens in the scripting world, performance was traded off against ease-of-use. We're built for comfort, not speed. As I have said many times before, if you care about maximizing performance, using a late-bound unoptimized bytecode-interpreted dynamically-typed language is probably a bad choice.

Let's go through the design process first, and then get into the implementation details and what the performance costs are.

Let's make up a little syntax to describe the type libraries that I'm talking about. Note that this is not VBScript, I just need a concise way to describe the enumerated type contents of a type library. (String constants in modules work the same way and nothing else in the library is relevant, so without loss of generality I'll be considering only enumerated types in this series.) I'll make it a bracy language to emphasize that this is not VBScript.

typelib Video {
enum Color {
    Red = 1
    Green = 2
    Blue = 3
    Black = 4
    White = 5
}
}

typelib Food {
enum Cheese {
    Green = 1
    Blue = 2
    Yellow = 3
    White = 4
}
}

What does this VBScript program do if those type libraries have been added to the script engine namespace?

foo.bar = Blue

We've got a bit of a collision problem here. An obvious technique to solve this problem is to allow qualification:

foo.bar = Color.Blue ' OK, it's 3

Which works great until we also add this type library:

typelib Magic {
enum Color {
White = 1
Black = 2
}
}

Now it's not clear what foo.bar = Color.White means. We need two levels of disambiguation:

foo.bar = Video.Color.White ' five

Our major design decision here is to decide whether full qualification is a requirement all the time, or only when there is a collision. Or, somewhere in between; C# requires that enumerations be qualified by at least the type name but makes explicit full qualification optional via the "using" statement. C doesn't allow qualification at all -- you get a collision, that's your problem. (This is why you see so many enums in C that look like enum MagicColor { magicColorWhite, magicColorBlack }; in an attempt to avoid collisions.

We gave this a lot of thought and decided that the likely scenario was that collisions would be rare. Therefore users should not be forced to always qualify their enumerations at either the enumeration or library level. However, since collisions might occur, we should not act like C. In short, full, partial and no qualification should all be legal.

There are other design constraints that might not be immediately obvious. For example, full qualification must always work even in a world where type information is not known at compile time. Suppose you have the three type libraries above plus

typelib Cheese {
enum Blue{
Stilton = 1
Gorgonzola = 2
}
}

The code foo.bar = Cheese.Blue.Stilton must succeed -- we can't decide at runtime that this really means the nonsensical Food.Cheese.Blue.Stilton and raise an error. But we cannot do any analysis before the code runs either, because for all we know, someone used Execute to introduce a local object variable called Cheese, which would then win over either of those options. Furthermore, we might not even have the type libraries available when the compiler runs: the host can choose to add them after the script has been compiled. (Though that would be irksome, it is legal.) We can't make any assumptions at code generation time -- we have to assume that this is a series of object property calls like any other.

Next time we'll revisit how the script engines handle host-supplied objects, and see how that mechanism neatly solves our design problem. Then we'll see why this isn't fast compared to declaring the constant yourself.