Autokinetic Effect - Erik Saltwell's Weblog

My brush with the dark side

eriksalt — Thu, 10 Nov 2005 20:52:00 GMT

Attention all MS Field personnel: Anyone who would like to come over to my office and slap me around, I will be available between 2:00 and 4:00...

I spent the first four years of my career here at Microsoft as part of Microsoft Consulting Services in NYC. One of my biggest pet peeves was that, when you interacted with the product teams, their answer for everything seemed to be

"Well, upgrade to <place-product-name-here> and everything will be better."

When customers come to you with issues, they clearly don't want to be up sold, they want their problem solved (duh).

Unfortunately, just a year after moving to Redmond, I almost found myself committing that same atrocity today. We have a customer who makes heavy use of xslt, and all the xslt work is eating up allot of cycles, and they asked us for some advice. Now, we have put a HUGE amount of work into increasing the performance of Whidbey, and you can see the results of that work in the product, so my first thought was 'Well, if you want to increase perf, why not move from your current engine to Whidbey?'

At this point I could hear Emperor Palpatine's voice in my head going 'Yes... Feel the upgrading flow through you. Come to the dark side...' I do think that there are benefits to upgrading here, especially for perf, but I was clearly focused on my product a little bit more then I was on the customer. So I wanted to offer an official apology to all my friends and peers in the field. I shall not be tempted again...

Erik gets a taste of the good stuff

eriksalt — Sun, 30 Oct 2005 22:32:00 GMT

(I know I promised to write some more about Linq and XLinq, which I will do, but I have had some other thoughts on my mind as well that need to get out before my head explodes, so I am going to be pushing out a few notes about other stuff before returning back to Linq in the near future).

Wow.

I had an amazing thing happen recently and I wanted to blog about it because it's had quite a powerful effect on me.

Recently, I had the opportunity to proctor a usability study for the XmlReader. If you have not heard of a usability study, I will explain. In a usability study you craft a few small tasks, such as:

Read in some xml and validate that the xml conforms to some schema.

You then get a bunch of developers together - we have a team at Microsoft who is responsible for finding developers, so most of the people taking the study are not MS employees - and you have them implement these tasks, all while you record everything they code as well as whatever they say. Then, after you run enough people through the study you collect the feedback in order to decide how to make improvements to your API. Of course it gets more complicated then this, you can head over to http://blogs.msdn.com/stevencl/default.aspx if you would like to know more, but that's the gist of it. A part of the study is that every time a subject comes in to participate, we have a team member sit in so that they can provide help if the person gets stuck, and that's where I come into the picture...

I recently had the opportunity to be one of these proctors, and for me it was like finding religion... Or perhaps like doing heroin (or I imagine that is what it would be like, anyhow)...

Well. whatever the metaphor, I definitely walked away wanting more. There were some many small things you can see when someone is using your API, like which pieces they blow right through and which pieces they stumble over, or how they interact with the API. These are all things that were amazing to me.

Ok, so maybe that means I didn't really grok our API in the first place, but I don't really believe that. The funny thing for me is that there are many ways in which having a huge, diverse customer base, like the customer base for Whidbey, can actually impede your ability to get the kind of feedback you would like to have, but (just to beat the drug metaphor to death) usability studies definitely feel like you're mainlining customer feedback.

Now, there are no silver bullets in software development, but for anyone who is interested in having an API which is easy to use, I would definitely recommend adding this tool to your toolkit (again, you can find more details at http://blogs.msdn.com/stevencl/default.aspx).

From IUnknown to IEnumerable

eriksalt — Wed, 28 Sep 2005 16:56:00 GMT

The core piece of Linq is a query language which can query many, many kinds of data. Of course, its the 'many' part which is difficult to get right. The tricky part is that, if you are going to query different kinds data, you have to specify something that all your data types have in common, and then build your query language on top of that commonality.

Take SQL & ODBC, which together form a technology that can be used to access many different kinds of data and provide a query language for that data using SQL. Over the years we have seen ODBC drivers that can query SqlServer, DB2, Oracle, and MSAccess. Of course ODBC drivers have provided access to more then just databases, there are ODBC drivers for Excel, text files and a host of other common data sources. What they all have in common is that this data is rectangular: it is made up of rows and columns. This makes sense because ODBC and SQL were designed for database access and databases are traditional stores of rectangular data, which is why using ODBC drivers to access non-rectangular data requires so much duct tape. This is not just about the SOURCE data being rectangular, in SQL the RESULTS are also primarily returned as rectangles. If you were thinking about generating a more general query language which can handle other types of data - like xml - you would need to be able to handle more then just this (not to mention the fact that you would want the creation of new data sources to be much easier then it was in ODBC).

For Linq, the commonality between the types of data you can query is that they all need to implement IEnumerable and, more frequently, its generic cousin IEnumerable<> (there is some simplification here for remote data sources, a simplification I hope to return to in a later post, but from a high level this is accurate). Once you start working with Linq for any real length of time, you will start to realise that IEnumerable<> is looking less like 'just another datatype' and more like a first class citizen of the language. Query expressions (i.e. the 'from ... select' syntax for querying in Linq) are designed to work over IEnumerable's as its data sources, and it returns IEnumerable's as the result.

Using IEnumerable as your common format is a good choice for a number of reasons, but the ones that impress me are:

It is familiar to developers.
It is easy to implement, especially using C# 2.0's "yield return" syntax. If your data is already contained in the .Net collection classes, then you don't even need to do that.
With the appearance of generics, IEnumerable<> can declare what type of object it is enumerating over. Anyone who remembers the collection interfaces in COM will remember how much a collection of IUnknown (or Object in .Net) can obfuscate your object model.
Because the enuerated type can be any .Net type, the returned data can have a variety of shapes, and is not restricted to being rectangular. Using XLinq, you can query and have that data returned as an xml fragment.
IEnumerable<> is a streaming interface over a collection of objects. What do I mean by streaming? Think of the difference between DataReader and DataSet, or between XmlReader and XmlDocument. The readers in these examples are both streaming interfaces, because they don't force you to allocate the whole collection at one time. With IEnumerable<>, you retrieve an IEnumerator on which you constantly call MoveNext() and Current, which means you don't have to allocate the whole collection of items being enumerated over at one time.

Let me give you an example; In DLink you can directly execure your own SQL statements (including stored procedures) by calling the ExecuteQuery method:

IEnumerable<Customer> customers = db.ExecuteQuery<Customer>("exec GetCustomersProc");

This is the general 'escape hatch' for using DLinq without the O/R mapping. The results that come back from this query get converted into Customer objects (one for each returned row) that are exposed through an IEmumerable<Customer> What is nice is that:

You get back strongly typed data.
You get back an IEnumerable, so you can use these results in other Linq querries.
Because you get back an IEnumerable, each Customer object can be retrieved when you call MoveNext(), so if you get back a billion rows, you don't have to allocate all billion Customer objects at one point in memory.

It will be interesting to know if in 3 years we don't see people talking about the pervasiveness of IEnumerable similar to how people used to talk about IUnknown. I wait for the first IENMRBL license plate with bated breath...

PDC and Linq, two great tastes...

eriksalt — Wed, 21 Sep 2005 16:44:00 GMT

I had the great pleasure of helping to represent one of Microsoft's new technologies at the PDC this year, its called LINQ; Language INtegrated Query. I hope to provide a number of posts about the details of LINQ and XLinq over the next few weeks, but I thought I should start with a simple introduction first, for everyone who was not able to make it to this event.

LINQ is a collection of new language features (which are planned to be included in future versions of C# and VB.Net) and new framework APIs that allow you to query and transform "data" from directly within the language. There are two interesting things to note about this sentence:

Notice the finger-quotes around the term "data" in the sentence above. The core of LINQ is a collection of new language features that allow you to query almost any kind of data (as long as it is exposed in a certain way) without ever having to leave the comfort off your favorite programming language (we will get to the details of what a data provider needs to do in order to work with LINQ later).
When Microsoft ships LINQ, it will also ship new API's for some kinds of data sources (the plan now is to ship APIs for XML and SqlServer, and LINQ already works with the current .Net collection classes) that can be queried using LINQ. That's why I was there, to represent the LINQ-enabled XML API.

In other words, LINQ is a syntax for querying data directly in the language , but it does not specify WHAT KIND of data is being queried (this statement is slightly inaccurate, because it is not technically a 'syntax' but rather a collection of language features which are implemented with different syntax for different languages, but we can gloss over that for the moment). Anyone can choose to implement their APIs in a way that allows LINQ language features to query that data. Now you may be thinking that this is already true with today's technology. After all, if you implement an OLE DB provider for your data, then you can query it using SQL. There are (at least) three major issues with this:

You effectively need a post-graduate degree in the OLE DB architecture before you would consider writing an OLE DB provider 'easy'. In LINQ, you expose you data as IEnumerable's, just like the existing collection classes. This makes exposing your data in a LINQ friendly way very easy.
In fact, if your data is already represented as an in-memory object model, there is a good chance your data already is in the correct format.
SQL is designed around providing access to 'rectangular' data (rows and columns) whereas some data (like XML) is better expressed in other ways. Since LINQ is designed to access many different types of data, it does not make assumptions about the shape or structure of your data.
Since your computer language's compiler knows nothing about SQL, you get no compile-time syntax checking. Since LINQ is a part of the programming language, you get a much better compile-time experience.

To put it another way:

If you've ever wanted to do a join between an xml file and data in your database, then LINQ is for you.
If you get tired every time a new data format comes out because that's another query language you will burn 3 weekends learning, then LINQ is for you.
If you wish that your programming language's compiler validated your queries for you, rather then having to run each query to make sure you got the syntax right, then LINQ is for you.

Ok, so LINQ is all about enhancing developer productivity when working with data. So what does a LINQ query look like? The following simple example shows us one way to query some xml data using LINQ and the new LINQ-Enabled XML API:

IEnumerable<XElement> transactions = from transaction in myXmlElement.Element("AllTransactions").Elements("Transaction") where (double)transaction.Attribute("amount") > 1000.0 select transaction;

This query returns all the 'Transaction' elements that are children of an "AllTransactions" element which is itself a child of myXmlElement and that have an 'amount' attribute whose value is greater then $1000.00. This could also be expressed with the following XPath:


      AllTransactions/Transaction[@amount>1000.00]

Now if you already understand XPath, then you are probably thinking 'Why would I move to a verbose syntax like LINQ when the XPath is so much more compressed?' Which is a fair statement, but it misses the point.  The point is that I can use the same syntax to query my database:

     IEnumerable<Individual> individuals = 
     from
         individual in myDatabase.Individuals
     where
         individual.Role=="Supreme Commander"
     select
         individual;
And you can use the same syntax to query collections of in-memory objects:
     IEnumerable<char> firstInitials = 
     from
         name in myNameList
     where
         name.Length<0
     select
         name[0];
Or you can do a join between xml and your database (sorry for the complicated example here):
     IEnumerable<XElement> transactions = 
     from 
         individual in myDatabase.Individuals,
         transaction in report.Element("AllTransactions).Elements("Transaction")
     where 
         individual.Name ==  (string)transaction.Attribute("name")
         &&
         individual.Role=="Supreme Commander"
         &&
        (double)transaction.Attribute("amount") > 1000.0
     select
         transaction;               
                    
The point is, LINQ is a general purpose language for querying and transforming data, so of course there will be cases where a domain-specific query language (like xslt) will be more powerful, more compact, or more expressive.  Our hope though, is that for those common cases where you don't need the extra power, this will be (as one PDC-goer put it) 'the last weird query language you'll ever have to learn.'

Please Recycle...

eriksalt — Mon, 15 Aug 2005 21:07:00 GMT

(My post today is on a topic some of you may already be familiar with - reusing NameTable across Xml messages. If so, please feel free to skip this entry. I posted it because, while I have seen other press on this issue, I thought it was important enough that I wanted to get more information out there.)

For anyone who has not met him yet, I would like to introduce you to an old friend, XmlNameTable. XmlNameTable is an interface used by XmlReader and XmlDocument when they want to store atomized names. For example, imagine that you parsed the following document:

<?xml version="1.0" encoding="utf-8" ?> <stable> <horse>Mr. Ed</horse> <horse>Seabicuit</horse> <horse>Man O' War</horse>  <horse>Quick Draw McGraw</horse> </stable>

Then you use the following code:

System.Xml.XmlReader reader = XmlReader.Create(…); List<string> elementNames=new List<string>(); reader.MoveToContent(); while(reader.Read()) { if(reader.NodeType==XmlNodeType.Element) elementName.Add(reader.Name); }

How many copies of the string "horse" do you expect to find contained in elementNames?

The answer is that the list will contain 10,000 or so references to only one string. This is because each time the parser comes upon a name, it checks to see if that name already exists in its XmlNameTable, and if so it uses that copy of the name. This not only decreases memory pressure, but it also allows you to do reference comparisons for names.

If you look at the XmlReaderSettings class, you will see that it contains a NameTable property that takes an implementation of XmlNameTable. This property exists so that you can pass in your own XmlNameTable when you create a new reader through the XmlReader.Create(…) API. Doing this allows you to pass in your own implementation of XmlNameTable, which is mildly useful in some odd scenarios. More importantly, this also allows you to re-use a name table across multiple documents, which can be an important performance optimization, particularly when parsing many small documents that have similar names.

For example, let's say that you had some xml that represented a purchase order:

<?xml version="1.0" encoding="utf-8" ?> <purchaseOrder poNumber="111"> <customer name="Mr. Ed"> <lineItems> <lineItem itemType="horseshoe" quantity="4" /> <lineItem itemType="saddle" quantity="1" /> </lineItems> </customer> </purchaseOrder>

If you have to process many of these purchase orders but use different name tables each time you process the message, you will continually have the overhead of allocating the names 'purchaseOrder', 'customer, name', 'lineItems', 'lineItem', 'itemType' and 'quantity' over and over. You will also have to allocate new entries in the underlying collection class used by the XmlNameTable. If instead you reuse your NameTable across readers, you can see quite a bit of savings.

In order to understand these differences, I ran a test. I created two xml documents. The first document was similar to the 'stable' sample above, where there were very few unique names, and some of the names ('horse' in this case) were repeated over and over. The second document was similar to the 'purchaseOrder' sample, where just about every name in the document was unique, although my document was somewhat larger then the sample shown above).

I parsed each of these documents 100,000 times. In half of these cases I reused the same name table. In the other half I recreated the name table each time I parsed the document.

Here are my results:

	Repeating Names (stable)	Unique Names
Re-use name table	2.596 seconds	1.308 seconds
Don’t re-use name table	2.696 seconds	1.771 seconds

As you can see, the performance improvement looks roughly like this:

Xml with many repeating names (and very few unique ones): 3.7% improvement.
Xml with many unique names (and very few repeating ones): 26.1% improvement.

This makes sense. If your document consists of the same names repeating over and over again, then you will not spend much of your time parsing names over and over. If, on the other hand, you have a small document, or one with many unique names, then you will spend a proportionately higher percentage of your time in name management. Make sure you measure your case, since your mileage will vary depending on the shape of your xml.

There are two things you need to be careful of when re-using your name table:

Watch out for multithreading: The default implementation is not thread-safe. If you are parsing on multiple threads using the same name table, you will either need to roll your own, thread-safe implementation or use one name table per thread (perhaps storing it in thread-local storage).
Watch out for name-bloat: If you are parsing untrusted data, a malicious hacker can send you a large number of different names, each of which will be added to the name table. Over time this can starve your application of memory. Even without hackers, you may see the size of your name table grow if your parse many xml messages with different names. For these reasons, you may want to occasionally clear the table out.

Priya on Unique Particle Attribution

eriksalt — Tue, 09 Aug 2005 20:56:00 GMT

Priya Lakshminarayanan, ruler of all things schema, has just posted an article on UPA errors (that's 'Unique Particle Attribution') that does a great job of presenting a very complicated schema issue very clearly. I would guess that even major schema geeks will learn a thing or two from the post. Go check it out here.

What the heck is OutputSettings

eriksalt — Wed, 27 Jul 2005 16:22:00 GMT

Oleg Tkachenko over at Signs in the Sand had a recent blog post about the OutputSettings property of XslCompiledTransform. He posted this example usage:

using (XmlReader src = XmlReader.Create("../../source.xml"))
{
    XslCompiledTransform xslt = new XslCompiledTransform();
    xslt.Load("../../style.xslt");
    XmlWriter result = XmlWriter.Create(Console.Out, xslt.OutputSettings);
    xslt.Transform(src, null, result, new XmlUrlResolver());
}

We have also received some feedback that this was somewhat confusing, so I figured I would explain a little bit more about what this 'OutputSettings' property is and why it exists.
When we first thought about creating the XslCompiledTransform class, we figured that we would use an XmlWriter to render the output of the transform. This made sense for a few reasons:

It provided for a good separation of responsibilities (disaggregation)
This separation follows the intent of the xslt design committee to separate serialization from transformation
It allowed users to pass in their own XmlWriter if they wanted more control over the results of the transform or what happened to those results

All of this was a great benefit to us in developing the transform engine, but, we ran into one sticky issue (well, one we care about for the purposes of this post, anyhow). The problem was that an xsl transform provides some ways within the xslt language to control how the results are written out. This meant that there were two ways to specify these output options, one in the xslt and one in the WriterSettings passed into an XmlWriter.
For example, if you write the following stylesheet:

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="no"/>
<xsl:template match="/">
<hello/>
</xsl:template>
</xsl:stylesheet>

You would expect to have your output formatted without indenting because you have specified indent="no". Of course, as you can imagine, indented formatting of xml also happens to be useful for cases where you are using an XmlWriter without any xslt involved. For this reason, you will find that the XmlWriterSettings class also has an Indent property. So, what would you expect to be the result from using the above xslt with the following code?

XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;
XmlWriter writer = XmlWriter.Create("out.xml", settings);
transform.Transform("test.xml", null, writer);

You will get the indenting specified in the WriterSettings (indent=true) rather then what is specified in the xslt (indent="no"). Of course, it is neither practical nor desirable to have the user specify the correct settings for each of their stylesheets, so instead the XslCompiledTransform exposes the OutputSettings property. When you call the Load method of the transform, it determines what the correct XmlWriterSettings should look like. In the case that you call one of the Transform() overloads which does not take an XmlWriter, this settings instance will be used to construct a writer on your behalf, but if you are passing in your own writer, then you can either use this instance directly (as you can see above from Oleg's code) or you can Clone() this instance and twiddle whatever particular settings you care about before constructing a writer.

Use of GetElementsByTagName considered harmful

eriksalt — Wed, 20 Jul 2005 04:58:00 GMT

As you may already be aware, there is an article on msdn about the great performance improvements we made in the V2 xml stack. This is a pretty big event for the team, since increasing performance was one of the big goals of this release.

While I posted this to give a shout-out to the team for the great perf work in they did for Whidbey, I want to take a minute to point out another interesting part of the article. Up at the top of this article you will find a this paragraph:

2. We also found that Sun is using the GetElementsByTagName method in C#, and this results in a mismatch of the C# and Java versions. Although both harnesses use XmlElement.GetElemensByTagName(), the C# implementation of this method keeps track of a live list of nodes which is negatively impacted by some of the DOM test scenarios that have edits. A better method for element selection in the C# harness is XmlElement.SelectNodes() which matches the Java harness in functionality.

When I first read this, it caused me some concern. Were we somehow ‘cheating’ by changing the nature of the test? Was there something wrong with GetElementsByTagName that I (and the various xml developers out there) should be aware of? I did a little digging, and wanted to share the results.

The W3C Level1 DOM spec has this to say about the results of GetElementsByTagName (I underlined the last sentence for emphasis):

The DOM also specifies a NodeList interface to handle ordered lists of Nodes, such as the children of a Node, or the elements returned by the Element.getElementsByTagName method, and also a NamedNodeMap interface to handle unordered sets of nodes referenced by their name attribute, such as the attributes of an Element. NodeLists and NamedNodeMaps in the DOM are "live", that is, changes to the underlying document structure are reflected in all relevant NodeLists and NamedNodeMaps.

In other words, GetElementsByTagName is, according to the spec, supposed to return a ‘live list’ where changes to the underlying DOM are reflected in the returned NodeList. The MS implementation of this function conforms to the spec, but the Sun/Java implementation does not.

Unfortunately, this conformance comes at a cost. In this case, the cost is a non-trivial hit to the speed and working set of code which uses GetElementsByTagName. Let me explain why.

In order to keep the NodeLists returned from GetElementsByTagName live, each list needs to ‘register’ somehow with the related element so that it can be notified when the children of that element change. When we first implemented this function, we thought about adding a series of events to XmlElement. NodeLists could register for these events and react appropriately. Unfortunately, this turns out to bloat the size of a DOM tree, and the performance costs caused by the related cache misses were unacceptable. Because of this we moved to a model where only XmlDocument exposed ‘OnChange’ events. When an item is added into the DOM, the XmlDocument.NodeInsterted event fires, each NodeList can determine if the new element should be added to its list or not, and then take the appropriate action.

This solution has the advantage that it adds no extra overhead to scenarios which do not call GetElementsByTagName, and the overhead is minimal even for scenarios which do use GetElementsByTagName, as long as those scenarios do not change the DOM (in this case the only overhead is the cost of registering for the event and a small increase in allocated memory for the created delegate).

The real problem shows up when you combine calls to GetElementsByTagName and edits to the DOM. Imagine that you had a moderate sized DOM (a few hundred elements) and for each element you called GetElementsByTagName, which results in a few hundred NodeLists registering for the XmlDocument.NodeInsterted. Now, after walking through the tree, you add a few elements. Each time you add an element, the NodeInserted event fires, and hundreds of delegate functions will get called. As you can imagine, this can cause quite a hit to the performance of your application.

The bottom line? If you are using GetElementsByTagName, especially in scenarios where you edit the DOM, you should investigate using SelectNodes instead. SelectNodes takes an XPath expression, but all you would need to do to get back the same elements as what is called by GetElementsByTagName would be to pass in “\\ElementName” [edit: that should read "//ElementName] to SelectNodes where you would have passed “ElementName” in to GetElementsByTagName. As always, you should measure how this changes your perf before switching over, especially in cases where you are not editing the DOM, since SelectNodes does a little more work to support the full syntax, but don’t be afraid to try. Even if your perf is equivalent in both cases, you may want to look at switching over, since we are very focused on improving XPath performance, so you may find your perf improve over time along with our XPath implementation (note: as always Microsoft’s plans are always subject to change, while we currently are focused on XPath performance, some future change may alter this in the future).