Marcelo's WebLog : XML

XslCompiledTransform and large template elements

marcelolr — Tue, 01 Dec 2009 22:00:00 GMT

When loading / compiling an XSLT document into an XslCompiledTransform instance, we've seen some customers hit an interesting gotcha.

In this case, a particular stylesheet had grown over time, through a series of, erm, clipboard operations, to about a 300KB file with a single template. Rather than factoring out common processing, we found it was repeated. While the functionality was correct, it took many minutes and a large amount of memory to compile. Why? Because a template translates (roughly) to a generated .NET method, which in this case turned out to be over 500KB in IL (normally a method will turn out to have less than 1KB). Clearly this is not a common case scenario, and the JIT was designed for a different point, but a mechanical translation can end up producing these kinds of problems.

So, remember - factoring is good for you, your platform, and your future program maintainers! Which probably includes you as well :)

Enjoy!

MSXML XHTML DTDs - making the web better

marcelolr — Mon, 30 Nov 2009 22:00:00 GMT

The fix to requests for XHTML DTD files from the W3C Web server has been recently released. Windows Update should offer the fix automatically, but you can download and install the fix manually from the following links for various MSXML versions:

http://support.microsoft.com/?kbid=973688 [MSXML4 SP2... if you haven't tried SP3 yet see below]
http://support.microsoft.com/?kbid=973685 [MSXML4 SP3]
http://support.microsoft.com/?kbid=973686 [MSXML6 Out-Of-Band, for XP SP2 and Win2K3]
http://support.microsoft.com/?kbid=973687 [MSXML3 and MSXML6 for all OSes where these components are in-band]

What exactly is this about? Well, there are a number of cases where this can happen, but one common scenario that we've seen people run into when writing AJAX web pages is to request a page of information from the server, load it into an XML document and extract some information or merge it into the existing page.

The problem with this approach is that when you load the document into an MSXML document "by hand" (ie, not through responseXML on the XHR object), DTD processing is enabled, and the DOCTYPE declaration directs MSXML to go look up the XHTML DTD so you can use entities like  .

Multiply this by each web browser accessing a popular site, and you can imagine why no-one is happy with this situation: web sites break when the resource is not available, the W3C servers get overloaded, and users of web sites lose functionality with odd scripting errors.

The fix caches the XHTML DTDs in MSXML - these resources haven't changed for years and will likely end up in different URLs if something new comes along. So now clients save at least one round-trip (possibly more) and always get the DTD support they need.

Try running this bit of script in a .js file to see the awesomeness in action.

If you don't have the update, you might get an error such as the following.

You have error The server did not understand the request, or the request was invalid.
Error processing resource 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'.

If you do have the update, you'll see the parsed XML. Run this with Fiddler open and you'll "see" the absence of network activity.

Enjoy!

MSXML4 SP3, now with a CAB

marcelolr — Fri, 02 Oct 2009 00:00:00 GMT

Yep, the redistributable CAB for MSXML4 SP3 is now available for download. Now you can deploy the latest / greatest version MSXML4 along with your app.

http://www.microsoft.com/DOWNLOADS/details.aspx?familyid=7F6C0CB4-7A5E-4790-A7CF-9E139E6819C0&displaylang=en

Remember, of course, that MSXML4 has been superseded by MSXML6, so you're better off upgrading; but if you can't because for example you don't own the component that has the dependency, the latest service pack contains the latest fixes.

Enjoy!

Problems when installing SQL Server 2005 on an upgraded Windows XP SP3 with MSXML6 SP2

marcelolr — Tue, 01 Sep 2009 00:00:00 GMT

As mentioned in the SQL Server Support Blog, you may find an issue installing SQL Server 2005 if you install MSXML Core Services 6.0 Service Pack 2 on Windows XP SP2, then upgrade to Windows XP SP3, then try the SQL install.

Check out that post for more details, along with the KB for the problem.

XSLT Timer Released on Code Gallery

marcelolr — Wed, 17 Jun 2009 00:00:00 GMT

Over this past weekend, the XSLT Timer tool has been released on Code Gallery.

This is a very simple tool, that will simply allow you to type / paste in a document and an XSLT stylesheet, and then tell you how long it takes on average to do the transform using the XslCompiledTransform class. Very simple, but handy when trying ad hoc variations on ideas and such.

You might want to look at the XML Resources page for links to other tools as well.

Enjoy!

update #1 - adding an actual link to the tool helps :)

XmlLite says: error C2146: syntax error : missing ';' before identifier 'IXmlReader'

marcelolr — Sat, 06 Jun 2009 00:00:00 GMT

This is something that I've been asked more than once, and so here goes an answer that might help if you run into this.

Let's say that you create a new project and borrow code or do something of the sort, and suddenly start getting an error message such as this one

xmllite.h(49) : error C2146: syntax error : missing ';' before identifier 'IXmlReader'

This typically points to a missing macro. You can fix this by adding this to your include headers:

#include "objbase.h"

If you're using Visual C++-style projects, this may already be included via stdafx.h including the Windows headers for you.

More group slicing in XSLT

marcelolr — Fri, 05 Jun 2009 00:00:00 GMT

To continue on my previous example on slicing (I made the term up, by the way), today we'll do something a little different.

Let's say that instead of just inserting the slice elements and a few items, we want to copy the whole tree n times (where n is the number of slices), each time with a different slice of items. That includes elements beyond the things we're currently slicing.

So let's say we have this document, similar to the previous one.

And we want to get this effect.

Here is a stylesheet that can accomplish this, using parameters to pass around the replacements nodes for each copy of the tree. As always, variations in the comments are welcome.

<xsl:stylesheet version='1.0' xmlns:xsl='http://www.w3.org/1999/XSL/Transform'>
<xsl:output method='xml' indent='yes'/>


<xsl:template match='@* | node()' mode='everything'>
<xsl:copy>
 <xsl:apply-templates select='@* | node()' mode='everything'/>
</xsl:copy>
</xsl:template>


<xsl:template match='@* | node()' mode='withreplacement'>
<xsl:param name='replacement' />
<xsl:copy>
 
 <xsl:apply-templates select='@* | node()[local-name() != "item"]' mode='withreplacement'>
 <xsl:with-param name='replacement' select='$replacement' />
 </xsl:apply-templates>

 
 <xsl:if test='item'>
 <xsl:apply-templates select='$replacement' mode='everything' />
 </xsl:if>
</xsl:copy>
</xsl:template>


<xsl:template match='item' mode='firstofslice'>
<xsl:apply-templates select='/group' mode='withreplacement'>
 <xsl:with-param
 name='replacement'
 select='. | following-sibling::item[1]'
 />
</xsl:apply-templates>
</xsl:template>


<xsl:template match='/'>
<root>
 
 <xsl:apply-templates select='group/item[position() mod 2 = 1]' mode='firstofslice' />
</root>
</xsl:template>
</xsl:stylesheet>

Enjoy!

LINQ to XSD on CodePlex

marcelolr — Thu, 04 Jun 2009 00:00:00 GMT

I'm very, very pleased about today's announcement on LINQ to XSD being available on CodePlex.

The functionality available should be esentially the same as what was released in the last Alpha Preview, so if you were already using this, it should be a smooth transition.

The community of developers interested in LINQ to XSD will now be able to include this in their applications, as it's being released under the Ms-PL license.

Enjoy!

Simple group slicing in XSLT

marcelolr — Wed, 03 Jun 2009 00:00:00 GMT

First, let me start with a disclaimer: I'm not an XSLT guru. I know folks that can truly do amazing things with it. Every now and then, though, I can help someone, and I thought I'd share a solution I came up with recently. I'll be happy to see alternate solutions posted in the comments.

So, let's say you have an XML document such as this one:

You'd like to create "slices" of the items in the group, such that you get them reparented in twos (could be three, four or whatever - we'll go with two items per slice). Any remainder try to fill up as much as possible of the last slice, so the transformed document should look like this.

My solution has three templates: one for processing each item (I do some simple content copying "by hand"), one to process each slice (which has a the current context the first item in the slice), and one to process the whole document. The trick is to use the mod function to pick the beginning item of each slice, and to use the following-sibling axis to grab the ones that come right after.

Here's the XSLT for the solution that produces the desired result.

<xsl:stylesheet
 version='1.0'
 xmlns:xsl='http://www.w3.org/1999/XSL/Transform'>
 <xsl:output method='xml' indent='yes'/>
 
 <!-- Template for a single item in a slice. -->
 <xsl:template match='item' mode='inslice'>
  <item><xsl:value-of select='text()' /></item>
 </xsl:template> 

 <!-- Template for each slice. -->
 <xsl:template match='item' mode='firstofslice'>
  <slice>
   <xsl:apply-templates select='.' mode='inslice' />
   <xsl:apply-templates select='following-sibling::item[1]' mode='inslice' />
   <!--
   "grow" the slice by adding more of these, but change the 'mod' below:
   <xsl:apply-templates select='following-sibling::item[2]' mode='inslice' />
   -->
  </slice>
 </xsl:template> 

 <xsl:template match='/'>
  <group>
   <!-- Grab every second item to create a slice. -->
   <xsl:apply-templates select='group/item[position() mod 2 = 1]' mode='firstofslice' />
  </group>
 </xsl:template> 

</xsl:stylesheet>

Enjoy!

XSLT Resources Posted

marcelolr — Fri, 29 May 2009 00:00:00 GMT

I've posted a page on XSLT Resources that I hope to add to over time - stay tuned.

OpenXML and XML Literals - great video

marcelolr — Wed, 22 Apr 2009 00:00:00 GMT

Beth Massi shows off how you can use XML literals in VB.NET to do some very, very powerful stuff in a very, very straightforward way.

http://www.dnrtv.com/default.aspx?showNum=138

Enjoy!

DTD resolution and network behavior

marcelolr — Tue, 21 Apr 2009 00:00:00 GMT

Following up from my last post, danieldsmith asked about a couple of additional details. I think I responded in the comments, but because (a) it's generally useful, and (b) I had some problems with my network connection (because I monkeying around with it), the reply may have been lost. So here we go again.

First, is caching implemented for the resolved DTDs? No, the default resolver doesn't do any caching - files will always be requested. So if you're loading documents with DTDs in a tight loop, you might get a pretty significant perf improvement here.

Second, what happens if there is no network connectivity? Well, the framework can't do validation without the DTD, and even if it could it would be unable to expand entities, so as you might guess, the problem isn't silently ignored - you'll get an exception instead.

Trying this on my machine at home, I got two slightly different exception - your mileage may vary. The first time I had Fiddler running, and I got a System.Net.WebException with a Message value "The remote server returned an error: (502) Bad Gateway.", and a Status value of ProtocolError. There was a Response assigned to it, which had a message of its own, "Fiddler - DNS Lookup Failed".

Then I shut down Fiddler and got another WebException, although this time the message was "The remote name could not be resolved: 'www.w3.org'", and the Response of the exception was null.

So now you know. Keep an eye on your network activity, and turn off features that you don't use to get perf / stability improvements without having to go and make changes to your application.

As a side note, I like to run Fiddler every now and then, just to see what kind of web traffic my machine is generating - it's awfully interesting to see the kinds of things that are going on sometimes.

Enjoy!

Faster XML - don't resolve DTDs if you don't need to

marcelolr — Sat, 18 Apr 2009 01:00:00 GMT

When loading an XML document through XmlDocument.Load or XDocument.Load, the default behavior when finding a DTD reference is to go resolve the URL, which typically means one or more web requests. Often, however, the DTD is there more as a marker of what the document contains than anything else, and you might not want to reference entities or do validation.

For example, let's say I have the following code. This will load an XML document from a string, then write it out to the console.

const string xmlText = 
@"<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>
<html lang='en' xml:lang='en' xmlns='http://www.w3.org/1999/xhtml'>
<head><title>Hello</title></head>
<body><h1>Hello from XHTML</h1></body>
</html>";

XmlDocument doc = new XmlDocument();
doc.LoadXml(xmlText);
Console.WriteLine(doc.InnerXml);
Console.ReadKey();

The problem here is that there is no reason for us to access the DTD for this particular application, and yet the following requests will go out.

http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent
http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent
http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent

The first solution, available on Silverlight, is to use the XmlPreloadedResolver. By default it will contain DTDs for XHTML and RSS, but you can add your own. In this case, you get the same level of functionality as if the real resources had been requests. For example, you can use entity references such as   to mean "non-breaking white space". If you assign an instance as your XmlResolver, you're good to go.

A second solution, which is also available on the full .NET Framework, is to simply set the XmlResolver property to null. In this case, no requests will be made at all, but you won't be able to use entity references nor have the document validated.

Note that this isn't just available on XmlDocument, but also through XmlReaderSettings, anytime you're creating an XmlReader (which can of course be used with both XmlDocument and XDocument, along with many other parts of the framework).

In the next release of the .NET Framework, there will be a few other ways in which you may save the resource requests, but these options are available to you today.

Enjoy!

XmlSchema and XmlSchemaSet thread safety

marcelolr — Tue, 17 Mar 2009 00:00:00 GMT

Here's a good word of warning: even if an object "feels" read-only because you're not calling code to modify it, if it's not documented as safe for use from multiple threads, then you shouldn't risk it.

In particular, I'd like to talk about XmlSchema and XmlSchemaSet today. Building these has a cost associated, and so it's nice to be able to build them once and then reuse them. But you have to be very careful in doing this. The docs say that all instance methods are not safe for multiple thread usage, but you don't really use them directly during validation, so it's hard to tell from the outside what's safe and what's not.

In a nutshell, the only thing you can do that is safe for concurrent usage is to use a validating reader. Here's the sample code to try this out (for some reason, this "breaks" more on 64-bit machines, but it's unsafe on all architectures).

First, a little helper to create an XmlSchema.

private XmlSchema CreateSchema()
{
  string schemaText = @"<?xml version='1.0'?>
<xs:schema id='play' targetNamespace='http://tempuri.org/play.xsd'
 elementFormDefault='qualified' xmlns='http://tempuri.org/play.xsd'
 xmlns:xs='http://www.w3.org/2001/XMLSchema'>
 <xs:element name='myShoeSize'>
  <xs:complexType>
   <xs:simpleContent>
    <xs:extension base='xs:decimal'>
     <xs:attribute name='sizing' type='xs:string' />
    </xs:extension>
   </xs:simpleContent>
  </xs:complexType>
  </xs:element>
</xs:schema>";

  using (StringReader reader = new StringReader(schemaText))
  {
    return XmlSchema.Read(reader, null);
  }
}

Next, a simple XmlSchemaSet.

private XmlSchemaSet CreateSchemaSet(XmlSchema schema)
{
  XmlSchemaSet set = new XmlSchemaSet();
  set.Add(schema);
  set.Compile();
  return set;
}

Finally, some validation:

private void ValidateDocument(XmlSchemaSet set)
{
  string doc = @"<myShoeSize xmlns='http://tempuri.org/play.xsd' sizing='123' />";
  XmlReaderSettings settings = new XmlReaderSettings();
  settings.Schemas = set;
  settings.ValidationEventHandler += new ValidationEventHandler(settings_ValidationEventHandler);
  using (StringReader reader = new StringReader(doc))
  using (XmlReader x = XmlReader.Create(reader, settings))
  {
    while (x.Read()) { }
  }
}

private int failCount;
void settings_ValidationEventHandler(object sender, ValidationEventArgs e)
{
  System.Threading.Interlocked.Increment(ref failCount);
}

Now, armed with these, I will show you some code that is thread-safe, but that a single line reorder would cause to break.

XmlSchema schema = CreateSchema();
Thread[] threads = new Thread[10];
XmlSchemaSet set = CreateSchemaSet(schema);
for (int i = 0; i < threads.Length; i++)
{
  threads[i] = new Thread((x) =>
    {
      for (int j = 0; j < 1000; j++)
      {
        // If the CreateSchemaSet were here
        // instead of outside this would break!
        //
        // Don't add the schema to the 
        // XmlSchemaSet from multiple threads!
        //
        // XmlSchemaSet set = CreateSchemaSet(schema);
        //
        ValidateDocument(set);
      }
    });
}

Array.ForEach(threads, (t) => t.Start());
Array.ForEach(threads, (t) => t.Join());
this.Text = "Failure count: " + failCount; 

The part before the thread creation is all thread-safe; the stuff inside the callback is happening on multiple threads at the same time. You can only use the set for validation here!

Enjoy!

Fast way to convert XmlDocument into XDocument

marcelolr — Sat, 14 Mar 2009 00:00:00 GMT

This question came up in the forums a little while ago, and I thought it might be interesting to do some comparisons.

I first came up with a few ways of turning an XmlDocument into an XDocument.

private static XDocument DocumentToXDocument(XmlDocument doc)
{
  return XDocument.Parse(doc.OuterXml);
}

private static XDocument DocumentToXDocumentNavigator(XmlDocument doc)
{
  return XDocument.Load(doc.CreateNavigator().ReadSubtree());
}

private static XDocument DocumentToXDocumentReader(XmlDocument doc)
{
  return XDocument.Load(new XmlNodeReader(doc));
}

Next I whipped up a function to time these with something quick and dirty. I make sure the past activity doesn't both much in terms of leaving garbage, and I warm up the action a bit (I also warm up the Stopwatch methods, just in case).

private static long Time(int count, Action action)
{
  GC.Collect();
  for (int i = 0; i < 3; i++)
  {
    action();
  }

  Stopwatch watch = new Stopwatch();
  watch.Start();
  watch.Stop();
  watch.Reset();
  watch.Start();

  for (int i = 0; i < count; i++)
  {
    action();
  }

  long result = watch.ElapsedMilliseconds;
  watch.Stop();
  return result;
}

And finally, all together:

StringBuilder sb = new StringBuilder();
sb.Append("<parent>");
for (int i = 0; i < 1000; i++)
{
  sb.Append(" <child>text</child>");
}
sb.Append("</parent>");

string text = sb.ToString();
XmlDocument doc = new XmlDocument();
doc.LoadXml(text);

long docToXDoc = Time(1000, () => DocumentToXDocument(doc));
long docToXDocNavigator = Time(1000, () => DocumentToXDocumentNavigator(doc));
long docToXDocReader = Time(1000, () => DocumentToXDocumentReader(doc)); 

Note that the actual numbers don't matter much, as this is my laptop running a bunch of things in the background, in the debugger and whatnot, but the relative values are interesting to see.

These are the values I got (they vary a bit each run, but not by much).

Using OuterXml: 1973 ms.
Using a navigator over the document: 1254 ms.
Using a reader over the document: 1154 ms.

Not surprisingly, avoiding the creation of a big string just to re-parse it is a big win - save the planet, use less CPU power!

Enjoy!