Rico Mariani's Performance Tidbits : quiz

Performance Quiz #13 -- Linq to SQL compiled query cost -- solution

ricom — Mon, 14 Jan 2008 20:51:00 GMT

Well is there really a "solution" at all in general? This particular case I think I constrained enough that you can claim an answer but does it generalize? Let's look at what I got first, the raw results are pretty easy to understand.

The experiment I conducted was to run a fixed number of queries (5000 in this case) but to break them up so that the compiled query was reused a decreasing amount. The first run is the "best" 1 batch of 5000 selects all using the compiled query. Then 2 batches of 2500, and so on down to 5000 batches of 1. As a control I also run the uncompiled case at each step expecting of course that it makes no difference. Note the output indicates we selected a total of 25000 rows of data -- that is 5 per select as expected. Here are the raw results:

Testing 1 batches of 5000 selects
5000 selects uncompiled 9200.0ms 25000 records total 543.48 selects/sec
5000 selects compiled 5401.0ms 25000 records total 925.75 selects/sec

Testing 2 batches of 2500 selects
5000 selects uncompiled 9181.0ms 25000 records total 544.60 selects/sec
5000 selects compiled 5402.0ms 25000 records total 925.58 selects/sec

Testing 5 batches of 1000 selects
5000 selects uncompiled 9169.0ms 25000 records total 545.32 selects/sec
5000 selects compiled 5432.0ms 25000 records total 920.47 selects/sec

Testing 100 batches of 50 selects
5000 selects uncompiled 9184.0ms 25000 records total 544.43 selects/sec
5000 selects compiled 5511.0ms 25000 records total 907.28 selects/sec

Testing 1000 batches of 5 selects
5000 selects uncompiled 9166.0ms 25000 records total 545.49 selects/sec
5000 selects compiled 6526.0ms 25000 records total 766.17 selects/sec

Testing 2500 batches of 2 selects
5000 selects uncompiled 9165.0ms 25000 records total 545.55 selects/sec
5000 selects compiled 7892.0ms 25000 records total 633.55 selects/sec

Testing 5000 batches of 1 selects
5000 selects uncompiled 9157.0ms 25000 records total 546.03 selects/sec
5000 selects compiled 10825.0ms 25000 records total 461.89 selects/sec

And there you have it. Even at 2 uses the compiled query still wins but at 1 use it loses. In fact, the magic number for this particular query is about 1.5 average uses to break even. But why? And how might it change?

Well, as has been observed in the comments, Linq query compilation isn't like regular expression compilation. In fact compiling the query doesn't do anything that isn't going to have to happen anyway. In fact, actually creating the compiled query with Query.Compile hardly does anything at all, it's all deferred until the query is run just as it would have been had the query not been compiled. So what is the overhead? Why is it slower at all? And what's the point of it?

Well the main purpose of that compiled query object is to have an object, of the correct type, that also has the correct lifetime. The compiled query can live across DataContexts, in fact it could potentially live for the entire life of your program. And since it has no shared state in it, it's thread-safe and so forth. It exists to:

1) Give the Linq to SQL system a place to store the results of analyzing the query (i.e. the actual SQL plus the delegate that will be used to extract data from the result set)

2) Allow the user to specify the "variable parts" of the query. The most common case isn't that the query is exactly the same from run to run, usually it's "nearly" the same... That is it's the same except that perhaps the search string is different in the where clause, or the ID being fetched is different. The shape is the same. Creating a delegate with parameters allows you to specify which things are fixed and which things are variable.

Now there was some debate about how to make compiled queries durable, automatically caching them was considered, but this was something I was strongly against. Largely because of the object lifetime issues it would cause. First, you would have to do complicated matching of a created query against something that was already in the cache -- something I'd like to avoid. Secondly you have to decide where to store the cache, if you associate it with the DataContext then you get much less query re-use because you only get a benefit if you run the same query twice in the same data context. To get the most benefit you want to be able to re-use the query across DataContexts. But then, do you make the cache global? If you do you have threading issues accessing it, and you have the terrible problem that you don't know when is a good time to discard items from the cache. Ultimately this was my strongest point, at the Linq data level we do not know enough about the query patterns to choose a good caching policy, and, as I've written many times before, when it comes to caching good policy is crucial. In fact, analogously, we had to make changes in the regular expression caching system back in Whidbey precisely because we were seeing cases where our caching assumptions were resulting in catastrophically bad performance (Mid Life Crisis due to retained compiled regular expressions in our cache) -- I didn't want to make that mistake again.

So that's roughly how we end up at our final design. Any Linq to SQL user can choose how much or how little caching is done. They control the lifetime, they can choose an easy mechanism (e.g. stuff it in a static variable forever) or a complicated recycling method depending on their needs. Usually the simple choice is adequate. And they can easily choose which queries to compile and which to just run in the usual manner.

Let's get back to the overhead of compiled queries. Besides the one-time cost of creating the delegate there is also an little extra delegate indirection on each run of the query plus the more complicated thing we have to do: since the compiled query can span DataContexts we have to make sure that the DataContext we are being given in any particular execution of a compiled query is compatible with the DataContext that was provided when the query was compiled the first time.

Other than that the code path is basically the same, which means you come out ahead pretty quickly. This test case was, as usual, designed to magnify the typical overheads so we can observe them. The result set is a small number of rows, it is always the same rows, the database is local, and the query itself is a simple one. All the usual costs of doing a query have been minimized. In the wild you would expect the query to be more complicated, the database to be remote, the actual data returned to be larger and not always the same data. This of course both reduces the benefit of compilation in the first place but also, as a consolation prize, reduces the marginal overhead.

In short, if you expect to reuse the query at all, there is no performance related reason not to compile it.

Performance Quiz #13 -- Linq to SQL compiled queries cost

ricom — Fri, 11 Jan 2008 23:08:00 GMT

I've written a few articles about Linq now and you know I was a big fan of compiled queries in Linq but what do they cost? Or more specifically, how many times to you have to use a compiled query in order for the cost of compilation to pay for itself? With regular expressions for instance it's usually a mistake to compile a regular expression if you only intend to match it against a fairly small amount of text.

Lets do a specific experiment to get an idea. Using the ubiquitous Northwinds database and getting the same data over and over to control for the the cost of the database accesses (and magnify any Linq overheads) we run this query:

var q = (from o in nw.Orders
            select new {
                OrderID = o.OrderID,
                CustomerID = o.CustomerID,
                EmployeeID = o.EmployeeID,
                ShippedDate = o.ShippedDate
           }).Take(5);

and compare it against:

var fq = CompiledQuery.Compile
(
    (Northwinds nw) =>
            (from o in nw.Orders
            select new
                   {
                       OrderID = o.OrderID,
                       CustomerID = o.CustomerID,
                       EmployeeID = o.EmployeeID,
                       ShippedDate = o.ShippedDate
                   }).Take(5)
);

So now the quiz: How many times to I have to use the compiled version of the query in order for it to be cheaper to compile than it would have been to just use the original query directly?

Performance Quiz #12 -- The Cost of a Good Hash -- Solution

ricom — Fri, 26 Jan 2007 21:01:00 GMT

Well once again there have been many thoughtful replies to both the original question as well as the followup with hints.

Recall the original question asked a bit of trivia: could you name 5 implementations of GetHashCode in the framework that do things you might not expect to see in a hash function? It's a little bit of a vague question and also requires a good deal of arcane knowledge of the CLR. The most interesting thing about the discussion I think is just what people considered "unusual." But as always there was a method to my madness.

So without further ado here's my list of "unusual" hash functions, there are 53 of them.

**GetHashCode Methods that Allocate**
java.lang.reflect.Method.GetHashCode	4.2
com.ms.lang.MulticastDelegate.GetHashCode	4.2
com.ms.lang.Delegate.GetHashCode	4.2
System.Uri.GetHashCode	3.8
System.Security.Policy.HashMembershipCondition.GetHashCode	3.8
java.lang.reflect.Field.GetHashCode	3.5
com.ms.wfc.ui.Region.GetHashCode	3.5
com.ms.wfc.ui.Pen.GetHashCode	3.5
com.ms.wfc.ui.Font.GetHashCode	3.5
com.ms.wfc.ui.Color.GetHashCode	3.5
com.ms.wfc.ui.Brush.GetHashCode	3.5
System.Security.Policy.NetCodeGroup.GetHashCode	3.5
System.Security.Policy.FileCodeGroup.GetHashCode	3.5
System.Security.Policy.CodeGroup.GetHashCode	3.5
System.Security.PermissionSet.GetHashCode	3.3
System.Security.NamedPermissionSet.GetHashCode	3.3
System.Configuration.SettingElement.GetHashCode	3.1
System.Diagnostics.ListenerElement.GetHashCode	3.0
System.Configuration.ConfigurationElement.GetHashCode	3.0
java.math.BigDecimal.GetHashCode	2.8
System.Web.Configuration.TagPrefixInfo.GetHashCode	2.8
System.Web.Configuration.TransformerInfo.GetHashCode	2.4
System.Web.Configuration.TagMapInfo.GetHashCode	2.4
System.Web.Configuration.ProfileGroupSettings.GetHashCode	2.4
System.Web.Configuration.CustomError.GetHashCode	2.4
System.Web.Configuration.BuildProvider.GetHashCode	2.4
System.Web.Configuration.NamespaceInfo.GetHashCode	2.1
System.Security.Policy.StrongNameMembershipCondition.GetHashCode	2.0
System.Xml.Serialization.CaseInsensitiveKeyComparer.System.Collections.IEqualityComparer.GetHashCode(System.Object)	1.9
System.Security.Policy.PublisherMembershipCondition.GetHashCode	1.7
System.Xml.Schema.KeySequence.GetHashCode	1.6
System.Data.SqlTypes.SqlString.GetHashCode	1.6
System.Data.SqlTypes.SqlDecimal.GetHashCode	1.6
java.text.CollationKey.GetHashCode	1.5
System.Security.Policy.UrlMembershipCondition.GetHashCode	1.5
System.Security.Policy.SiteMembershipCondition.GetHashCode	1.5
System.Globalization.SortKey.GetHashCode	1.5
System.Net.Cookie.GetHashCode	1.3
System.Management.ManagementBaseObject.GetHashCode	1.3
java.math.BigInteger.GetHashCode	1.0
System.Windows.Forms.TableLayoutPanelCellPosition.GetHashCode	1.0
System.Windows.Forms.DataGridViewCellStyle.GetHashCode	1.0
System.Windows.Forms.DataGridViewAdvancedBorderStyle.GetHashCode	1.0
System.Windows.Forms.DataGridView+HitTestInfo.GetHashCode	1.0
System.Web.UI.ControlCachedVary.GetHashCode	1.0
System.Web.UI.AttributeCollection.GetHashCode	1.0
System.Web.FileSecurity+DaclComparer.System.Collections.IEqualityComparer.GetHashCode(System.Object)	1.0
System.Web.Configuration.RootProfilePropertySettingsCollection.GetHashCode	1.0
System.Web.Caching.CachedVary.GetHashCode	1.0
System.Security.Cryptography.X509Certificates.X509CertificateCollection.GetHashCode	1.0
System.Security.AccessControl.GenericAce.GetHashCode	1.0
System.Drawing.FontFamily.GetHashCode	1.0
System.Configuration.ConfigurationElementCollection.GetHashCode	1.0

And you can already see why I think they are unusual: They allocate. In my opinion any "normal" hash function should be able to do its job without creating any temporary objects. Given the frequency with which things can be hashed I think that's important. That doesn't mean that I'm going to go to Red Alert about any of the above but it does mean that we should think more carefully about creating big hash tables of the above.

OK, so, great, an interesting observation. What's the point?

Well, I while I believe that the particulars of this question are not profound, the notion that you can have a cost metric applied universally to the runtime -- even one that is as simple minded as the one I just proposed -- *is* profound.

Even though my "allocation complexity" metric may be feeble it is already useful and actually it is my hope that many people find it feeble and write about how their metric is much better and by the way here it is the much better metric in text format etc. etc. Imagine if we had the methods scored in a variety of ways all readily available and pluggable into intellisense. You could have scores relating to synchronization, i/o, memory, algorithmic complexity, even measured data where appropriate. Other engineering disciplines have abundant information about their raw materials but for those of us in the software business its often either guesswork or our own hard-won data.

Let's look at what you can do with even my feeble metric. In my very first quiz I asked (in part) whether it would be better to call Write three times on a stream or create a format string and call it once. Can we answer that question with the published metric?

Here are the relevant two lines from my costs file:

System.IO.TextWriter.Write(System.String) 0
System.IO.TextWriter.Write(System.String,System.Object[]) 1.5

That's not as good as the actual measurement but, wow, that's something now isn't it? Rough guidance to give you a clue! A great "tip" that the var-args formatting option has some underlying cost.

Even these rough costs are very powerful in terms of helping you write code. Suppose you're writing a new "Foo" and you want it to have cost that is roughly the same as existing "Foo". You could go and look at how the existing "Foo" methods score which gives you some idea what costs you can afford in your new "Foo".

Importantly you must observe Rico's Rule: if you are writing a new "Foo" method and you want that method to have a cost of X (on any metric) then you must not use any methods whose cost is known to be greater than X. Just that simple rule can save you from many costly mistakes.

Finally you have some way to answer the question: Is it even remotely reasonable to use a given method in a given context. Certainly this particular metric is imperfect, maybe even feeble, but it's something. And it will only get better if we all work on it.

Performance Quiz #12 -- The Cost of a Good Hash -- Some Help

ricom — Wed, 24 Jan 2007 22:30:00 GMT

I continue to be astounded by what my readers can come up with.

As usual I had a purpose for posing my last question and that purpose was to show that basically its hard to get a handle on what things cost in any kind of omnibus way. But imagine my surprise when Frank and Shuggy turned this into an interesting discussion of hashing techniques and combining hashes. To say nothing of some of the other regulars (you know who you are) that have insightful comments. Those comments are definately worth a read and thanks very much for presenting them!

But I don't know that anyone has answered my question quite squarely (except maybe Alois) and I think this makes my point that it is hard to answer this question.

But what if you were to download this file... I think you'd find it quite helpful.

What you see is a compilation of one cost metric for the entire framework. It's not the only cost metric by any means, you could imagine many others, but it is an interesting one. The cost is computed by doing a static analysis of all allocations made in the call tree of each method. It's the metric that I discussed in this article on performance signatures including the exception for methods that look like they do a one-time allocation. The cost is logarithmic, base 10, offset by 1, i.e.

if (allocation_count == 0)
cost = 0;
else
cost = 1+log(allocation_count)

And I'm only reporting one digit after the decimal because I want to give the idea that this is just a rough cost, it's not supposed to be super-precise it's just supposed to give you a rough idea of whether or not a method is very complex or not so much.

I'm trying to think of a good name for this metric, so far I'm thinking something like Allocation Complexity but I'm open to better ideas.

Anyway snoop the file then see if you can answer my question. For bonus points look at some other low level functions that might be interesting like say Compare, Equals, or GetEnumerator. It's very intersting to see that some methods that seem like they should be very low level are actually quite high level.

I think costs like this one can be very useful in making decisions about which framework features to use in which contexts.

Again this is only one kind of cost and its only approximate but I think it's interesting nonetheless.

Now would you like to try the original question again:

Q: Name five implementations of GetHashCode in the framework that do things that you wouldn't expect a hashing function to do.

Performance Quiz #12 -- The Cost of a Good Hash

ricom — Mon, 22 Jan 2007 22:38:52 GMT

This quiz is a little bit different than some of my others. Here is System.String.GetHashCode

public override unsafe int GetHashCode()
{
      fixed (char* text1 = ((char*) this))
      {
            char* chPtr1 = text1;
            int num1 = 0x15051505;
            int num2 = num1;
            int* numPtr1 = (int*) chPtr1;
            for (int num3 = this.Length; num3 > 0; num3 -= 4)
            {
                  num1 = (((num1 << 5) + num1) + (num1 >> 0x1b)) ^ numPtr1[0];
                  if (num3 <= 2)
                  {
                        break;
                  }
                  num2 = (((num2 << 5) + num2) + (num2 >> 0x1b)) ^ numPtr1[1];
                  numPtr1 += 2;
            }
            return (num1 + (num2 * 0x5d588b65));
      }
}

It's a fine little hash function that's does pretty much the sorts of things you'd expect a hash function to do.

So now the quiz. It's a bit of trivia but I'm going somewhere with it.

Can you tell me 5 implementations of GetHashCode in the framework that do things that you wouldn't expect a hashing function to do?

Performance Quiz #11: Ten Questions on Value-Based Programming : Solution

ricom — Fri, 08 Sep 2006 00:52:00 GMT

In my last quiz I asked a few questions about a few hypothetical classes that might appear in a value-rich context. I styled my example in the form of some graphics library classes but the idea is a general one. Many contexts can and should be rich in values to get nice data density and the resulting good performance that comes with it. I allude to some of these notions in my posting on real-time managed programming, so in a way this posting is sort of a continuation of that discussion.

Now as usual there is more than one way to do this properly so I will not pretend to be giving The Definitive Way To Write These Classes but my example does tend to show some important issues that are worth talking about. So without further ado, here's my thoughts on the matter.

Question #1: Point3d is a struct, not a class. Why?

The main reason goes right back to data density. There is a strong expectation that this class will be embedded in other classes such as Vertex and you can imagine many others. It is likely that there will be temporaries of this type, and importantly arrays of this type. Using a value type will make it so that walking through sets of related points does not require following multiple pointers and therefore data density is likely to be enhanced. And of course, there is storage overhead for classes -- the object header and the method table pointer -- these will be absent in packed arrays of structures.

Contrariwise the usual benefits of being a class are not especially valuable. Will we be subtyping? No. Sychronizing? No. Do we want point "identity" (e.g. the canonical instance of point (1,2,-1)? No.

This leads to a pretty clear decision for me.

What rules am I breaking? I just made a public mutable value type that is larger than 16 bytes. Three strikes and I have barely begun:)

Oh, I'm sorry about that last part, I didn't make it clear but I intended all of these to be public types. Krzysztof of course busted my chops about that. Thanks :)

On to...

Question #2: Point3d.x is a field, not a property. Why?

Common expected usage of the point primitive will involve things like translations, scaling, and other user defined operations which I can't possibly predict. The Point structure itself has the lovely property that there are no illegal values so I truly do not care how it is manipulated. The opportunity that I might have for adding any kind of side-effect to the point calculations is vanishingly small. Contrariwise the statement that I make by making the member a field is a powerful one. It's a field. I expect you to edit it. It will never be other than a field. Could I even, truly, afford even a modest overhead (a few clocks) for an operation so primitive as writing a single coordinate of a point? Even something that small would double the write costs.

The property setting code can certainly be no better than the field code, and in this case it's actually a tad worse (Alois discovered this in his comment). Furthermore, adding additional IL for the inliner to try to fold can only make the situation worse when these field operations are composed with other operations and we would like the larger operation inlined.

But for me, the main reason is to make the openness completely apparent. It's a strong promise and an important one.

I've broken another guideline now, there is no public property for this field.

Question #3: Vertex is a struct, not a class. Why? Same reason as #1?

Substantially yes, the same reasons as #1. Density, no need for identity, and substantial likelihood of composition.

Question #4: Vertex.location is a field, not a property. Why? Same reason as #2?

The reasons given in #2 all apply, but now there is an additional reason. Vertex is not leaf value-type, it has embedded members. We expect the vertex itself to be mutated and we would like to write things like Vertex.normal.dx = 0 or something. All manner of transformations might be possible. However the result of a property is not a true l-value so we cannot do something like Vertex.Normal.dx = 0. The chaining of properties gives very unexpected results.

Question #5: Quad has no methods. Why?

Importantly, Quad has no context and therefore it cannot hope to add any value other than as a bucket of 4 integers, for which we need no methods. Quad cannot have any context because we would not wish every Quad to know things like what Mesh it is a part of, the cost of such a pointer would be astonishing. Indeed a frugal graphics package might limit the Quad to using short integers for the vertex numbers if that were reasonable (though that would limit the mesh size signficantly). Lacking context we can't do things like range check the integers, check for consistency in the normals, or any other sort of thing we might like to do on a single face. We need to put those sorts of functions at the next higher level (the Mesh).

Since we can't do those sorts of things, lets be very open about the fact that this is nothing more than a 4 slot integer container, it's not "smart" and never will be. As a reward we get the benefits indicated in #1.

Question #6: MeshSection is a class with private members. Why?

Well now we've reached a level of implementation where the class actually has some idea what's going on and this class should probably be following all the regular sorts of rules you would expect. I'm showing some of the internals for illustration purposes, such as arrays of Vertex objects and Quad objects but naturally these should not be exposed. A MeshSection might wish to have some growable array form, or chunks, or some other dense representation that might evolve over time. The MeshSection can do validation on the items when they are passed in and provide high level semantic functions such as normal adjustment, tessalation, morphs, and other operations of that ilk.

Importantly, these representation choices, even the more flexible ones, are rich in represented data and extremely low in overhead. The number of pointers present is virtual nil so garbage collection costs even in the presence of massive peices of geometry would be virtually nil. Effectively the Quad objects are integer 'handles' to vertices as I discussed in my article on real-time managed programming. Yet I can still have a nice abstracted and maintainable interface.

Question #7: Why do I suggest that MeshSection methods accept arrays of Vertices, Quads, and the like rather than singletons?

Two significant reasons really. The first is that we might be dealing with MeshSections having millions of vertices and so it seems unwise to talk about operations that affect singleton primitives. While it's possible to imagine adding and connecting single vertices to a MeshSection it's as likely that we might want to merge signficant pieces of geometry. If this kind of operation requried millions of function calls we would have signficant overhead. So perhaps the most important reason of all is that chunky arrays as arguments offer a nice unit of work.

Secondly, the suggested prohibition against large value types (greater than say 16 bytes) is not without merit. But actually we give that advice not so much because large value types are inherently bad but because if used directly there can be an inordinate amount of pushing and popping of big objects on the stack as well as lost inlining opportunities. Using arrays neatly avoids both this problem as we merely pass around array references, and indeed the array could be reused.

This leads to an important observation, which is, wait for it...

It's not the size of the value type that matters. It's how you use it :)

Ahem. Moving swiftly along... :)

Question #8: No mention is made of synchronization here at all, is that just an oversight?

Certainly I would put no synchronization in any of the structs. The operations on those are so small that any sychronization primitive, even the cheapest, would be deadly at that level. But this is actually not the pivotal point. It is much more important to put synchronzation at a semantic level where the code has some understanding of what the unit of work is. That is, how big of an operation is intented to be atomic? What are the high level operations that need to be protected? The soonest you could imagine having this knowledge is at the MeshSection level in this program -- but even there, the code has no thread awareness per se -- it's just doing some vertex math. Lacking knowledge of how things might be scheduled it again becomes difficult to make any interesting synchronization choices. Since the contract for MeshSection is nice and chunky (see #7) you could imagine again leaving the synchronization to the caller, as we do in virtually all of our framework collection classes.

Ultimately, I don't believe these classes have anything of value to add in terms of synchronization assistance and so I decided not to include it. For more on this, see my article "Putting your synchronization at the correct level."

Question #9: How useful is the "foreach" construct likely to be when working with arrays of vertices or quads etc?

Not very. If you were to do foreach (Vertex v in SomeVertices) { whatever } you'd find that in the whatever block you can't modify the vertices because of course it's a value type. You'll likely be having custom iterators or using for loops to manage the array arguments to the MeshSection methods. Unfortunate but in my opinion it's the right thing to do.

Question #10: How many rules did I break? :)

I lost count. I think three big ones but more than once each I suppose.

Thank you very much for all the excellent responses!

P.S. The annotated design guidelines can be found here.

Performance Quiz #11: Ten Questions on Value-Based Programming

ricom — Thu, 31 Aug 2006 21:23:00 GMT

Some of you have probably heard one or more of my talks, or read the annotations I made in the Design Guidelines. If you have then you already know that I don't always agree with every suggested guideline. Especially not in every context. It's probably fair to say that one of the greatest areas of disagreement has to do with the handling of value types and the prevelance of properties vs. fields in the framework.

Generally, my feeling is that properties are highly overrated and fields terribly under-utilized. Did I mention that not everyone agrees with this position? :)

So suffice to say that the guidelines are still good guidelines but of course they are only that, and so you should know when it might be a good time to consider disregarding some of them. Today I wanted to write a motivational example for you that shows the kind of situation that might call for an approach that is more "old-school" if you like.

To try to make it more real, I've cooked it up in terms of some graphics library primitives that might happen in a real graphics library that was dealing with objects made up of 4 side polygons (quads) with some basic texturing. Naturally this example isn't complete but hopefully there is enough there that you could imagine what the other primitives might look like.

Below is the code written in the style that I would suggest, following are some key questions which are intended to suggest the origin of my thinking on some of the more important points.

    // typical 3d coordinate
    struct Point3d
    {
        public double x;
        public double y;
        public double z;
        // handy math methods
    }

    // projected cooridinates in the text space
    struct Point2d
    {
        public double u;
        public double v;
        // handy math methods
    }

    // useful for normals etc.
    struct Vector3d
    {
        public double dx;
        public double dy;
        public double dz;
        // handy vector math methods
    }

    // a single vertex, with location, and normal
    struct Vertex
    {
        public Point3d location;
        public Point2d uvmap;
        public Vector3d normal;
        // manipulation methods
    }

    struct Quad
    {
        // vertex indices within a mesh
        public int corner1;
        public int corner2;
        public int corner3;
        public int corner4;
    }

    // at last a reference type
    class MeshSection
    {
        private Vertex[] vertices;
        private Quad[] quads;
        private TextureMap texture; // defined elsewhere

        // assorted methods that accept arrays of vertices and quads
        // and insert them into the structure and things like that
    }

Question #1: Point3d is a struct, not a class. Why?

Question #2: Point3d.x is a field, not a property. Why?

Question #3: Vertex is a struct, not a class. Why? Same reason as #1?

Question #4: Vertex.location is a field, not a property. Why? Same reason as #2?

Question #5: Quad has no methods. Why?

Question #6: MeshSection is a class with private members. Why?

Question #7: Why do I suggest that MeshSection methods accept arrays of Vertices, Quads, and the like rather than singletons?

Question #8: No mention is made of synchronization here at all, is that just an oversight?

Question #9: How useful is the "foreach" construct likely to be when working with arrays of vertices or quads etc?

Question #10: How many rules did I break? :)

Full Disclosure:

Of course Brad and Krzysztof (the principle authors of the guidelines) both understand that the guidelines need to be broken sometimes. Sometimes we disagree when the bar for breaking them has been reached but I think that's actually a good thing because that kind of tension makes for good professional growth for everyone and better discourse for our customers.

Whenever you decide to go off the recommended path it's still good to be familiar with the guidelines in the area because they can give you some valuable information about what the consequences might be so that you can make an informed decision. Like in this parctiular case you might learn about some serialization issues you'll have to deal with since properties were not chosen.

Try to answer yourself before you peek at the comments, there's good thoughts down there. :)

Update: I posted my solution here.

Performance Quiz #10 -- Thread local storage -- Solution

ricom — Wed, 19 Jul 2006 02:25:00 GMT

I actually posted quiz #10 quite a while ago but a comment with the correct solution came in so quickly that I wasn't very motivated to post a followup. There are excellent links in the comments (thank you readers!) But now I'll have to make the quizzes harder :)

The problem was to see what overhead is associated with various methods of creating flexible thread local storage. I suggested two ways of having named storage.

I've posted a sample benchmark that expands on this and shows four different approaches (some less general than others).

On my machine I observed the following times:

Test1: Named Slot 7,991ms
Test2: Numbered Slot 4,136ms
Test3: Thread-local dictionary 2,006ms
Test4: Thread-local direct 704ms

So, what's going on? Well I looked into it with our profiler and got these results which show the extra costs pretty clearly. Have a look at all the helper functions under Test1 and Test2.

Exclusive

Inclusive

Function Name

0.39 %

89.92 %

Quiz10.Program.Main (string[])

0.78 %

53.07 %

Quiz10.Program.Test1 ()

0.95 %

25.19 %

\|	System.LocalDataStoreMgr.GetNamedDataSlot (string)

0.18 %

12.14 %

\| \|	JIT_MonReliableEnter (class Object ,bool )

5.76 %

8.06 %

\| \|	System.Collections.Hashtable.get_Item (object)

3.05 %

3.11 %

\| \|	@JIT_MonExitWorker@4

3.49 %

22.31 %

\|	NativeArrayMarshalerBase::NativeArrayMarshalerBase (class CleanupWorkList *)

0.43 %

5.97 %

\| \|	ThreadStore::LockDLSHash (void)

0.14 %

5.41 %

\| \|	CantAllocThreads::MarkThread (void)

0.04 %

2.80 %

\| \|	EEHashTableBase<int,class EEIntHashTableHelper,0>::FindItem (int)

0.77 %

2.19 %

\| \|	FrameWithCookie<class HelperMethodFrame_1OBJ>::FrameWithCookie<class HelperMethodFrame_1OBJ> (void ,struct LazyMachState ,unsigned int,class Object * *)

0.78 %

1.59 %

\|	System.Threading.Thread.get_LocalDataStoreManager ()

0.16 %

1.22 %

\|	ThreadNative::GetDomainLocalStore (void)

0.57 %

1.16 %

\|	System.LocalDataStore.GetData (class System.LocalDataStoreSlot)

0.66 %

26.72 %

Quiz10.Program.Test2 ()

3.73 %

21.79 %

\|	NativeArrayMarshalerBase::NativeArrayMarshalerBase (class CleanupWorkList *)

0.46 %

5.79 %

\| \|	ThreadStore::LockDLSHash (void)

0.18 %

5.13 %

\| \|	CantAllocThreads::MarkThread (void)

0.05 %

3.13 %

\| \|	EEHashTableBase<int,class EEIntHashTableHelper,0>::FindItem (int)

0.57 %

1.62 %

\| \|	FrameWithCookie<class HelperMethodFrame_1OBJ>::FrameWithCookie<class HelperMethodFrame_1OBJ> (void ,struct LazyMachState ,unsigned int,class Object * *)

0.11 %

1.19 %

\|	ThreadNative::GetDomainLocalStore (void)

0.44 %

1.08 %

\|	System.Threading.Thread.get_LocalDataStoreManager ()

0.53 %

1.05 %

\|	System.LocalDataStore.GetData (class System.LocalDataStoreSlot)

0.25 %

8.43 %

Quiz10.Program.Test3 ()

0.55 %

7.07 %

\|	System.Collections.Generic.Dictionary`2.get_Item (!0)

2.38 %

6.52 %

\|	System.Collections.Generic.Dictionary`2.FindEntry (!0)

0.20 %

1.30 %

Quiz10.Program.Test4 ()

The table above is showing all functions starting from Main with an inclusive cost >= 1% and a depth of no more than 3 -- so things are missing but it's good for discussion. Under Test1 there's a good deal of Locking and Marshalling... looks like there is a big oops here. The good news is that the contract is sound so hopefully this could be addressed. But really I'm not sure why I would even bother. The other approach, using [ThreadStatic] is much cleaner and much faster. I don't know why anyone would ever want to use the slots.

For my part rather than fix this I think I will ask that the relevant functions be deprecated -- the [ThreadStatic] approach seems better in every way. The slot methods hereby have my personal deprecation for what that's worth.

Performance Quiz #10 -- Thread local storage

ricom — Fri, 16 Jun 2006 22:55:00 GMT

It's time for another quiz!

A very short one this time. If I need some thread local storage and I might need several entries should I use:

Thread.GetData(slot) and Thread.SetData(slot, object)

or should I make my own static member like this

[ThreadStatic]
static Dictionary<String,Object> myItems = new Dictionary<String,Object>;

and get things through the dictionary?

Warning: spoilers below in the comments ... already :) Maybe my quizzes are too easy :)

Performance Quiz #9 -- extra innings due to dispatch stubs

ricom — Wed, 22 Mar 2006 03:37:00 GMT

Well you'll recall that in the Performance Quiz #9 solution there was a surprising result where Test7 was actually faster than Test5 even though they appear to be doing basically exactly the same work. So my new challenge to you was to see if you can explain it.

So, now this is your last chance to go and look for yourself before I give away the answer... so stop here if you want to work on the problem.

OK if you're still reading then you're just dying to hear what was going on. Well there's a lot of serendipity here. My colleague Vance had just written a blog about how to disassemble dispatch stubs. In what I can only categorize as a total unmitigated fluke (because believe it or not we didn't discuss this with each other at all) his blog which I linked to in the original has the answer right there on a silver platter.

As Ivan Stoev wrote in the solution comments:

"The difference is caused by different interface call stubs (as explained by Vance Morrison) in Test5 and Test7 since [the] foreach loop calls ([to] IEnumerator<ushort> MoveNext and Current) in SumSpecial are only on List<ushort> instance while [the] same calls in SumForeach are on both ushort[] (in Test4) and List<ushort> (in Test5). If we comment [out the] Test4 call, they would have similar speed (Test5 would be a little bit faster as expected).
Very interesting, thanks Rico and Vance!"

Which is right on the mark! Full points to Ivan!

SumSpecial has two call sites to the foreach methods, the first one always calls on an array instance so it can use the monomorphic stub (which is faster). The second one calls always on the List<T> instance so it too can use the monomorphic stub. SumForeach is handicapped because it's called first on an array and then on a List so the second time we decide to use a more flexible but slower stub to do the calling. That makes Test5 slower.

As Ivan writes you can see the difference go away by simply commenting out the call to Test4 in program.cs at which point all the call sites in question are monomorphic. Or equivalently you can create a duplicate of SumForeach called SumForeach2 and use that in Test5 and again all the call sites are monomorphic.

Remember, as Vance writes, stub selection is per call site.

And now that we have that little data we might want to rewrite program.cs to factor out stub selection in the reported times. But really that doesn't change the results a whole lot. The main thing that was going on was the array special helpers as discussed.

Chalk up another one for powerful secondary effects in the performance biz!

Performance Quiz #9 : IList<T> List and array speed -- solution

ricom — Sun, 12 Mar 2006 19:46:00 GMT

Last week I posted Performance Quiz #9 for discussion. Well the comments have been just awesome and many people went right ahead and measured the results. Some of you have discovered the key point in this particular test:

Arrays are magic.

In the test case as given there's a great deal of repeated work extracting the length of the array and accessing the items. This is because of the unusual property that arrays have -- they implement IList<T> for potetially more than one T even due to inheritance. In the interest of economy alone then it is worthwhile to consolidate the IList<T> implementations into some kind of secret helper but this has some consequences.

Luckily there are also alternatives -- the example that I provided is perhaps the worst performing choice. As one reader pointed out you could make it much better by extracting the count from the IList once and then using that in the loop. Note that the compiler cannot do this because IList<T>.Count is a virtual method and so we cannot assume anything about the implementation (it could sent a letter to grandma every time it is called and grammy wants her email!)

Here are the results from my program (source code attached, see link at the very bottom)

Test Case	Milliseconds
Test1: Array	54
Test2: List<>	8
Test3: ArrayWrapper	14
Test4: Array via foreach	9
Test5: List<> via foreach	11
Test6: Array via special	6
Test7: List<> via special	8

The first two rows are the problem as given and we can easily see that the array is performing much more slowly than the List. Let's see where the cost is:

Excl%	Incl%	Name
0.00	51.27	ArrayTest.Program.Test1(uint16[])
1.64	51.27	ArrayTest.Program.Test.Sum(System.Collections.Generic.IList`1)
2.38	25.64	System.SZArrayHelper.get_Item(int32)
5.47	23.26	JIT_IsInstanceOfArray(...)
4.72	7.26	ArrayIsInstanceOfNoGC(...)
2.63	10.02	SigTypeContext::InitTypeContext(...)
0.49	22.40	System.SZArrayHelper.get_Count()
5.36	21.91	JIT_IsInstanceOfArray(...)
4.30	6.99	ArrayIsInstanceOfNoGC(...)
2.40	9.07	SigTypeContext::InitTypeContext(...)

Well it pretty much leaps off the page, the cost is in those helpers and you can see they're doing a good bit of internal work to verify that we are talking about a simple array of a non-collectable type (ushort) and then more work happens to get to the real data. The tragedy here is that the work done to validate what we need to do far exceeds the actual job at hand -- which in both cases is just extracting one integer from a well known location. The price of abstraction.... Though I'm pretty sure there is room for improvement in there.

Let's look at some of the other alternatives. First how did the List<T> fare:

Excl%	Incl%	Name
0.00	8.62	ArrayTest.Program.Test2(System.Collections.Generic.List`1)
1.06	8.62	ArrayTest.Program.Test.Sum(System.Collections.Generic.IList`1)
5.82	5.82	CLRStubOrUnknownAddress
1.54	1.54	System.Collections.Generic.List`1.get_Item(int32)

In the list case there's still a good bit of dispatch code but you can see it's much cheaper. We're on the sweet path for regular interface dispatch. The actual count function calls are stil there if we were to look at lower percentages but I pruned anything with inclusive time below 1% in this example so it's not showing up. We're doing a lot better.

Now what about some of the other results? Well, now we're in the bonus marks zone and I don't want to drown you in perf results you can get yourself so I'll summarize a bit.

In Test 3 I made a generic wrapper class to hold the array. This adds a level of indirection but it avoids all the magic array processing. My wrapper is highly lame (it only implements count and get item) but you can see that it's doing fairly well performance wise. Much better than the raw array and it's a cheap object.

But the fun continues: What if we replace that for loop in the original test with a foreach? We can't get the ultra-special array foreach construct but we can get a nice standard foreach over an IList in both cases. Both Test4 and Test5 do introduce one allocation for the enumerator (which could be an issue if there were gazillions of these lists) but for our test case that's not even measurable (indeed I get no samples in the allocator).

And the result? It's pretty sweet: The array problem goes away entirely because we only have to do the array testing one time. After that we've got a nice bound up enumerator that knows all the details of the array -- no more redundant computations. The array works great.. but the list is a bit slower? Why would that be? Well there's some extra safety checks in the enumerator access code for List<T> to make sure the enumerator isn't being used on an List that is being modified for instance. Those checks slow down the list access a bit.

And the last two tests: What if we special case the array in the sum function; assuming that was a very common/important case. You can see that test6 gives the best result of all -- now there isn't any enumerator and we're on the most optimized code path of any -- foreach over a plain array.

But why is the List case faster in test7? Shouldn't it be pretty much the same as test5? After all there's an extra test and otherwise it's the same. Now there's a mystery... I actually ran a variety of control experiments, doing the tests in different order and with different array members to make sure this wasn't an anomolous result -- you can imagine with all these iterations cache effects would be important. But no matter what I did the code in test7 always went faster than the code in test5. Adding more iterations actually magnifies the effect.

The profile isn't especially helpful either -- both Test5 and Test7 have exactly the same call shape but Test7 spends less time in CLR stubs apparently. At the moment my best theory is that we happen to get better cache alignment in Test7 than Test5 -- less instructions split across cache lines or fewer unfortunate cache evictions because of happenstance.

Such is the life of the performance engineer -- we sometimes get bit by a secondary effect.

So shall we continue? What is happening to make Test7 faster than Test5?

And this seems like a great place to thank my colleague Vance Morrison for chiming in with the foreach and special case approaches when I first was bouncing this analysis around. Thanks a ton Vance :)

Performance Quiz #9 : IList<T> List and array speed

ricom — Fri, 10 Mar 2006 06:31:00 GMT

A short and sweet quiz with lots of juicy discussion possibilities:

           public int Sum(IList<ushort> indices)
           {
               int result = 0;
               for (int i = 0; i < indices.Count; i++)
                   result += indices[i];
               return result;
           }

Considering only the time it takes to do the Sum (i.e. assuming we had already set up the array/list) which gives better performance and why?

           // #1
           ushort[] tmp = new ushort[500000]; // this doesn't count
           Sum(tmp); // this is what we are timing

           // #2
           List<ushort> tmp = new List<ushort>(500000); // this doesn't count
           for (int i = 0; i < 500000; i++) tmp.Add(0); // this doesn't count
           Sum(tmp); // this is what we are timing

What say you gentle readers?

(my solution is now posted here)

Performance Quiz #8 -- The problems with parsing -- Part 5

ricom — Wed, 30 Nov 2005 22:32:00 GMT

And today we conclude our little saga with one final experiment, a little analysis and some comments. I hope you've enjoyed reading this series as much as I've enjoyed doing the experiments.

First, as promised I created yet another version of this little parser. This version is very similar to the last except now it's using our new lightweight code generation to actually make native code for each predicate. You can find parser_jitting.cs in my articles area like the others.

And how does this new parser combination fare? Surely generating native code is going to win over my lame little stack machine yes? And I've even stacked the deck by doing an admittedly unrealistic 1,000,000 iterations over the predicates for just one compilation.

The results are as follows:

test case	predicates	iterations	time
parser_compiling.cs	8	1,000,000	225ms
parser_jitting.cs	8	1,000,000	339ms
parser_original.cs	8	1,000,000	4,839ms

Wait a second, it's slower! What happened?

OK I admit it; once again I threw you a curve ball. What's going on here is that the predicates in the inital test case are fairly simple. There just isn't very much code. On the other hand the preamble for a call through a delegate is not totally trivial. Normally it doesn't matter much but here those functions are so small (maybe a half dozen instructions or so) that the function overhead is just killing us. So the delegates actually lose! Now for kicks I made a version that has the predicates hard coded, that one clocked in at a brisk 70ms (same test case), so that's a good baseline for the fastest this could possibly be. I used that version as a sanity check and as a template for how to create the IL in my jitting version.

But all is not lost. As has been pointed out to me these predicates are not especially complicated, maybe unrealistically so. So while strictly speaking the analysis for the benchmark as given is done at this point let's do a little more bonus work and see what we get.

Robert Pickering conveniently provided some more complicated predicates in his F# port of the parser, so I pasted those into my array making a total of 17 predicates, 9 of which are more complicated.

test case	predicates	iterations	time
parser_compiling.cs	8+9	1,000,000	1,378ms
parser_jitting.cs	8+9	1,000,000	716ms
parser_original.cs	8+9	1,000,000	not tested

Now we can see that if we can pay for the overhead of invoking a delegate then things go a lot better for the jitted version. But what about those compilation costs? Surely we shouldn't just disregard the cost of invoking the JIT. So again while it wasn't in my original formulation lets go on for even more bonus points and see what the cost of compliation is in a more realistic mix of compiling and evaluating. Here I added a LOT more predicates, duplicating the 9 above to get us above 2000 predicates. Then I turned the iteration count way down to 20 which I think is more like what I'll see in the application that motivated this whole problem. Finally I eliminated the test output and moved the timer start to the top of Main. And here are the results for the "more realistic mix."

test case	predicates	iterations	time
parser_compiling.cs	2258	20	27ms
parser_jitting.cs	2258	20	631ms
parser_original.cs	2258	20	213ms

I think it's very clear which one I should pick for my actual project at this point. There just aren't enough evaluations to pay for the cost of jitting. Even the original parser is holding up all right; with no compilation costs and fewer evaluations its problems are not being multiplied.

Now let me go back and answer my original questions directly.

Q1: What's wrong with this parser from a performance perspective?

Fundamentally the problem is that evaluation has not been seperated from parsing and that as a consequence each piece of input is being examined much more often than it needs to be.

Q2: What should we be doing to improve it?

Seperate the costs of parsing from the costs of evaluation. Reduce the cost of evaluation by using a better truth mechanism.

Q3: How big of a difference is it likely to make?

We can guess at this by thinking about how often we look at any given character of input. We scan it in getc, then we switch on it (hash and equality), then we do a lookup (hash+equality), then we copy it onto the heap, there is no per-iteration cost of compacting it off the heap as dead strings on the heap do not have to be moved (only live strings are moved). So I'm seeing memory traffic of at least 8 times the sum of all the parse strings (per iteration). In contrast if the strings were pre-tokenized then the memory traffic is more like 1/2 the sum of all the strings (per iteration). So I'd ballpark the gains available to be around a factor of 16 just on that basis. We ended up getting somewhat more as we did more than just eliminate the pressure on the memory system.

Q4: If you hadn't written all the code yet, what would you do to get a better idea what was going to matter and what wasn't?

Well I think I gave away the answer in Q3. I'd approach this my looking at how much work I do per character of input on average and go from there. That gives us good ballpark numbers without too much work.

Thank you very much for all the emails and comments!

Performance Quiz #8 -- The problems with parsing -- Part 4

ricom — Tue, 29 Nov 2005 22:48:00 GMT

In my last posting I made some recommendations about how to drastically improve the evaluation process based on the key observations that even the basic operations required to evaluate the truth of the facts were quite costly -- more so than all of the extra allocations.

I suggested a plan like this one:

preparse the predicates so that no string processing is required for evaluation

assign facts a number in sequence as they are encountered in the predicates

the preparsed predicate is a mix of fact numbers and operators probably in postfix order

when looking up facts simply access an array of truth bits indexed by fact number

as facts change simply change the array contents and then reevaluate predicates as desired

do the evaluation itself with a simple stack machine; no recursion in the evaluation

And as promised here's some code that does just that: parser_compling.cs

Now how do we fare with this code? Well the raw number I get on my machine is now 225ms to do those same predicate evaluations. That's down from 4.839s -- so it's over 21 times faster. Let's have a look at how the CPU usage is distributed now:

Module/Function	Exclusive %
ParserCompiling.exe	93.50
ParserCompiling.Program.EvaluatePredicate(int32[])	76.52
ParserCompiling.Program.Main(string[])	16.98
ParserCompiling.Program.CompilePredicate(string)	0.00
ParserCompiling.Program.CompileOptionalNot()	0.00
ParserCompiling.Program.GetToken()	0.00
ParserCompiling.Program.CompileAnd()	0.00
ParserCompiling.Program.CompileOr()	0.00
mscorwks.dll	3.15
ntdll.dll	1.47
mscorjit.dll	0.63
Unknown	0.42
SHLWAPI.dll	0.21
KERNEL32.dll	0.21
mscorlib.ni.dll	0.21
MSVCR80.dll	0.21
shell32.dll	0.00
mscoree.dll	0.00

Things have really improved at this point, the vast majority of the time (76.52%) is now being spent directly doing predicate evaluation -- that's just where we want the time to go. Our new implementation is also much more cache friendly as the number of facts increases because now we simply have an array of booleans (it could be even denser if it was an array of bits). Even if there were a few thousand facts we'd still have a tight representation -- the hash table approach tends to scatter things (deliberately) throughout a larger table. There are no redundant string comparisons, no hashing, it just goes voom :)

Now, what if we get out the big guns and start doing native code generation. Could we do even better? After all, this is managed code, we have the JIT at our disposal.

Tune in tomorrow :)

NOTE: When I originally posted this I reported the times for the debug build... I had to restate the results because the release build is approximately three times faster. While I was at it I re-verified that my previous numbers were on the release build and they were. Sorry about that...

Concluded in Performance Quiz #8 -- The problems with parsing -- Part 5

Performance Quiz #8 -- The problems with parsing -- Part 3

ricom — Mon, 28 Nov 2005 22:53:00 GMT

The thing about performance work is that it's very easy to be fooled into looking into the wrong areas. That's why you want your changes to be firmly grounded in data whenever they can be. Or if you're planning, you want to be thinking about what your customer experience needs to be, what the key metrics will be, and how your overall architecture will support those needs. Whatever it is that you're doing should be as grounded as you can make it for the stage your are in. After that you should be doing whatever you can to verify your assumptions, as often as you can with reasonable cost. The trick is to not go crazy applying too much effort doing all this because because that's just as bad.

Take this little parser example. You could conclude that all that string allocation in in GetToken() is the crux of the problems -- and you'd be partly correct -- but it's not really the full picture.

To try to see what was going on, one of the first things I did was try to eliminate the GC as a source of possible issues while leaving the rest as similar as possible. This actually illustrates a cool technique even though it turns out that it's not really the right approach. I wanted to eliminate all of those Substring calls while parsing -- since I expect we'll be parsing over and over again. The substring itself is just one example of looking at the input more than once -- a definate no-no in any high performance parser. But lets see where this leads us.

What I want to do is add a member that will hold the various fact and token strings that occur in the predicates once and for all. I'd like to have something like this:

private static System.Xml.NameTable ntTokens = new System.Xml.NameTable();

Now the reason I like this class so much is that it's one of few that offers you the opportunity to create strings without having to first allocate a string. You can call it starting with a character buffer and get a string back. That's just what we need... only the rest of the plumbing isn't quite right.

I want to make these changes:

<                return strInput.Substring(idx, ichInput - idx);
becomes
>                // returns an existing string if there is one
>                return ntTokens.Add(arrayInput, idx, ichInput - idx);

and

<                    return strInput.Substring(idx, 1);
becomes
>                    // returns an existing string if there is one
>                    return ntTokens.Add(arrayInput, idx, 1);

To do that I also have to change the input from a string to a char array

< private static string strInput = null;
becomes
> private static char[] arrayInput = null;

<        static bool EvaluatePredicate(string predicate)
<        {
<            strInput = predicate;
becomes
>       static bool EvaluatePredicate(char[] predicate)
>        {
>            arrayInput = predicate;

There's a couple more places where strInput has to be changed to arrayInput, like the getc() function, but these are very straight forward.

For my test case I wanted the predicates in character arrays now so I did this quick and dirty thing (in a real program you'd likely read the predicates from a file so you could just arrange for them to be in char arrays in the first place)

        struct CharArray
        {
            public char[] ptr;
        }

static CharArray[] predicateChars = new CharArray[predicates.Length];

and then later:

for (i = 0; i<len;i++)
predicateChars[i].ptr = predicates[i].ToCharArray();

            for (i = 0; i< len; i++)
                Console.WriteLine("{0} => {1}", predicates[i],
                      EvaluatePredicate(predicateChars[i].ptr));

And for the test loop itself simply call EvaluatePredicate using predicateChars.

OK that little experiment is about the smallest change that removes any need to create substrings while changing nothing else.

On my test system the original code was taking 4.839s to run; the new version with these improvements takes 4.680s. That's about a 3.3% savings.

Now I have to tell you in all honesty that if you had asked me to comment on that change off hand I think I would have told you that I'd expect a much greater impact than just 3.3% -- not that 3.3% is anything to sneeze at (see The Performance War -- Win it 5% at a time) but that isn't really the killer issue.

Lets go back and look at what the performance report shows and see if any of this makes sense.

Well the telling line is this one right here:

Exclusive% Inclusive% Function Name
0.01 4.62 WKS::GCHeap::Alloc(unsigned int,unsigned long)

The sum total of the allocations on that substring path, including all the garbage collections, was only 4.62%. It's no surprise then that we got less than a 4% improvement by only eliminating the garbage and doing nothing else. The collector was actually doing a pretty darn good job of cleaning up that trash on the cheap!

Even looking further at the string creation costs you'll see that InternalSubStringWithChecks has an inclusive time of about 22%. That means the bulk of the time creating these substrings had more to do with setting up for the array copy, moving the memory, checking the indexes for out of bounds, and so forth than it did to do with allocation. We can only attribute 4.62% of the time to actually allocating the memory. All of that checking and rechecking is costing us.

Let's look elsewhere: the getc() function is taking a total of 10.49% of the time -- that's more than the collector! It doesn't do anything but a couple of checks into an in memory array, but it's called an awful lot.

Looking at some other interesting operations, you can see that just hashing strings in this run, so that we can do fact lookups, is costing us 2.9% That's an astounding figure because you'll recall that ALL of our allocation activity was only 4.62%. Who would have expected that the allocation figure would be so low that just the string hashes could compete with it. And that's not all, if you look at the full comparison situation you'll see that checking for existing entries in the hashtable is accounting for 22% of the run time. Again, that whole operation is something we do merely to find out if any given fact is true or false.

Well, all of this really changed my mind about how to attack this problem. My plan looks like this:

preparse the predicates so that no string processing is required for evaluation
assign facts a number in sequence as they are encountered in the predicates
the preparsed predicate is a mix of fact numbers and operators probably in postfix order
when looking up facts simply access an array of truth bits indexed by fact number
as facts change simply change the array contents and then reevaluate predicates as desired
do the evaluation itself with a simple stack machine; no recursion in the evaluation

I'll blog some code that does this in the next part and we'll see how it does.

Any predictions based on what we've seen so far?

See the continuation in Performance Quiz #8 -- The problems with parsing -- Part 4