Eric White's Blog : LINQ

DescendantsAndSelfTrimmed LINQ to XML Axis Method

EricWhite — Wed, 19 Aug 2009 22:01:00 GMT

There are some circumstances where I need a variation on the DescendantsAndSelf axis method that allows me to specify that specific elements (and the descendants of those elements) are ‘trimmed’ from the returned collection. One of the things that's great about LINQ to XML is the ease with which we can create specialized axis methods when necessary.

Here is the scenario: Open XML contains paragraphs that contain runs. A run can contain a picture that contains a text box, which itself contains a paragraph. When I want to write a transform for paragraphs that are children elements of the w:body element, I want to disregard any paragraphs that are children of contained text boxes.

To make this clear, consider the following markup:

<body>

<r>

<t>First paragraph.</t>

</r>

<r>

<pict>

<shape>

<r>

<t>Text in a textbox.</t>

</r>

</textbox>

</shape>

</pict>

</r>

<r>

<t>Second paragraph.</t>

</r>

</body>

</document>

In this markup, I want to transform the elements highlighted in yellow, but while processing those elements, I want to completely disregard the element highlighted in green. If children paragraphs of the textbox element need to be transformed in some fashion, I would deal with them separately – perhaps in an embedded recursive transform.

To enable selecting descendant elements in this fashion, I’ve implemented four extension methods. These are variations on the Descendants and DescendantsAndSelf axis methods. The methods are:

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

XName trimName)

Returns a filtered collection of elements that contain this element, and all descendant elements of this element, in document order. Elements that have a matching XName are not included in the collection. Descendant elements of excluded elements are not included in the collection.

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

Func<XElement, bool> predicate)

Returns a filtered collection of elements that contain this element, and all descendant elements of this element, in document order. Elements that are selected by the predicate are not included in the collection. Descendant elements of excluded elements are not included in the collection.

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

XName trimName)

Returns a filtered collection of the descendant elements for this document or element, in document order. Elements that have a matching XName are not included in the collection. Descendant elements of excluded elements are not included in the collection.

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

Func<XElement, bool> predicate)

Returns a filtered collection of elements that contain this element, and all descendant elements of this element, in document order. Elements that are selected by the predicate are not included in the collection. Descendant elements of excluded elements are not included in the collection.

Here is the listing of these extension methods, as well as some example code to exercise them (also attached):

using System;

using System.Collections.Generic;

using System.Linq;

using System.Xml.Linq;

public static class LocalExtensions

{

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

XName trimName)

{

return DescendantsInternalTrimmed(element, e => e.Name == trimName, true);

}

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

Func<XElement, bool> predicate)

{

return DescendantsInternalTrimmed(element, predicate, true);

}

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

XName trimName)

{

return DescendantsInternalTrimmed(container, e => e.Name == trimName, false);

}

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

Func<XElement, bool> predicate)

{

return DescendantsInternalTrimmed(container, predicate, false);

}

private static IEnumerable<XElement> DescendantsInternalTrimmed(this XContainer container,

Func<XElement, bool> predicate, bool includeSelf)

{

if (includeSelf)

{

XElement element = container as XElement;

yield return element;

}

Stack<IEnumerator<XElement>> iteratorStack = new Stack<IEnumerator<XElement>>();

iteratorStack.Push(container.Elements().GetEnumerator());

while (iteratorStack.Count > 0)

{

while (iteratorStack.Peek().MoveNext())

{

XElement currentXElement = iteratorStack.Peek().Current;

if (predicate(currentXElement))

continue;

yield return currentXElement;

iteratorStack.Push(currentXElement.Elements().GetEnumerator());

}

iteratorStack.Pop();

}

class Program

{

static void Main(string[] args)

{

XElement rootElement = XElement.Parse(

@"<document>

<body>

<r>

<t>First paragraph.</t>

</r>

<r>

<pict>

<shape>

<r>

<t>Text in a textbox.</t>

</r>

</textbox>

</shape>

</pict>

</r>

<r>

<t>Second paragraph.</t>

</r>

</body>

</document>");

foreach (var item in rootElement.DescendantsAndSelfTrimmed("textbox"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

foreach (var item in rootElement

.DescendantsAndSelfTrimmed(e => e.Name == "p" && e.Ancestors("p").Any()))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

foreach (var item in rootElement.DescendantsTrimmed("r"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

foreach (var item in rootElement.DescendantsTrimmed(e => e.Name == "r"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

XDocument doc = XDocument.Parse(

@"<document>

<body>

<r>

<t>First paragraph.</t>

</r>

<r>

<pict>

<shape>

<r>

<t>Text in a textbox.</t>

</r>

</textbox>

</shape>

</pict>

</r>

<r>

<t>Second paragraph.</t>

</r>

</body>

</document>");

foreach (var item in doc.DescendantsTrimmed(e => e.Name == "r"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

}

Merging Comments from Multiple Open XML Documents into a Single Document

EricWhite — Tue, 28 Jul 2009 02:14:00 GMT

Microsoft Word 2007 allows you to lock a document, prohibiting users from making changes to content, while allowing them to add comments. If we have multiple documents that have the same content yet different comments, we can merge those comments into a single document. One possible use would be a specification review system. After the specification writer finishes a specification, she could send it to other members of her team for review. As each reviewer returns the specification, she could merge all comments into a single document, making it simpler to integrate those comments. (This example was inspired by an email thread with Sergey Solyanik, and his need for a comment merger for his very cool code (and soon-to-be spec) review system. Also, need to say thanks, Sergey did a code review, and the code is better for it. J) The code to do comment merging is available in a zip file named CommentMerger.zip in the downloads tab at www.codeplex.com/powertools.

Note: The CommentMerger class is part of the Power Tools for Open XML project. In the future, I’m going to build a new PowerShell cmdlet to do comment merging. PowerTools for Open XML is an open source project on CodePlex that makes it easy to create and modify Open XML documents using PowerShell scripts. It’s important to note that Power Tools for Open XML is not a supported Microsoft product and doesn’t necessarily represent future product direction. We think it will serve as inspiration for customers who need to create and modify Open XML documents programmatically. Power Tools for Open XML is published under the Microsoft public license (Ms-PL), which gives you wide latitude in how you use the code.

The CommentMerger.MergeComments method uses the Open XML SDK. The use of the comment merger class is pretty simple: You call the method passing two open WordprocessingDocument objects:

using (WordprocessingDocument destinationDocument =

WordprocessingDocument.Open("Test1a.docx", true))

using (WordprocessingDocument sourceDocument =

WordprocessingDocument.Open("Test1b.docx", false))

{

CommentMerger.MergeComments(destinationDocument, sourceDocument);

}

Upon return, the comments in the source document are merged into the destination document. To merge comments from multiple documents, you can call the function multiple times.

I wrote the CommentMerger.MergeComments method in the pure functional style. All methods are written without side-effects. After initializing, no variables are mutated. For a detailed explanation of this approach, see Recursive Approach to Pure Functional Transformations of XML.

The CommentMerger.MergeComments method is an example of a ‘Common-vocabulary document-centric transform’. For an overview of these types of transforms, see Document-Centric Transforms using LINQ to XML.

Before merging comments, the CommentMerger.MergeComments method validates that the two documents contain the same content. For more info, see Comparing Two Open XML Documents using the Zip Extension Method.

This code is based on the code I presented in Splitting Runs in Open XML Word Processing Document Paragraphs.

This code pre-atomizes XName objects. See A More Robust Approach for Handling XName Objects in LINQ to XML.

The code uses the Open XML SDK.

Recursive Approach to Pure Functional Transformations of XML

EricWhite — Mon, 20 Jul 2009 16:28:00 GMT

[Blog Map]

Writing pure functional transformations a in a recursive style enables us to put together interesting transformations in a very small amount of code. Using some specific techniques that allow us to write this code very concisely, this approach takes advantage of some perhaps obscure semantics of LINQ to XML. I’ve used this approach to write some interesting transformations – it has become my favorite way to write transformations of a certain variety. This post presents a short tutorial in writing these types of transformations.

Identity Transform

First, let’s take a look at the identity transform:

using System;

using System.Linq;

using System.Xml.Linq;

class Program

{

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(@"<Root Att1='1'><Child/></Root>");

XElement newRoot = (XElement)Transform(root);

Console.WriteLine(newRoot);

}

In this code, we are taking advantage of the semantics of the XElement constructor that allow us to pass collections of objects to the constructor. The following code creates a new XElement from an existing XElement:

new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

This code, of course, passes two objects to the constructor (in addition to the name) – a collection of attributes from the source element, and a collection of transformed child nodes from the source element. Note that even though we’re passing collections to the constructor, we’re only passing two objects. We’re passing the two variables that contain the iterators for the queries. Those iterators, of course, implement IEnumerable<T>. There are a number of LINQ to XML methods and constructors that can take content, and if we pass an object that implements IEnumerable of some T, LINQ to XML iterates the enumerable and adds each object in the collection to the newly constructed element. This is performed recursively. For detailed semantics of the constructors and methods, see this page in the LINQ to XML documentation.

This is one of the reasons that the signatures of LINQ to XML constructors and methods that take content are defined using object instead of XObject:

public XElement(XName name, params Object[] content);

Because the signature is defined using object, we can pass individual XElement, XAttribute and XNode objects to the constructor, and we can pass collections of them.

Cloning vs. Attaching

The identity transform also takes advantage of the cloning vs. attaching semantics of LINQ to XML. When we pass a newly constructed XElement to the XElement constructor, the new object does not have a parent so it is simply attached to the XElement that is being constructed. But the highlighted line below causes a node from an existing tree to be passed to the XElement constructor. In this case, the XElement constructor notices that the node is already part of an existing tree, and clones it and adds the cloned node to the XElement being constructed.

XElement element = node as XElement;

if (element != null)

{

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

To make this very clear, take a look at the following example:

XElement tree1 = XElement.Parse("<Root><Child1/></Root>");

XElement child1 = tree1.Element("Child1");

XElement child2 = new XElement("Child2");

XElement tree2 = new XElement("Root", child1, child2);

Console.WriteLine(tree2);

In this example, child1 already is part of tree1, so it is cloned and the newly cloned element is added to tree2. The child2 XElement object has no parent so it is simply attached to tree2.

For more information on cloning vs. attaching, see this page in the LINQ to XML documentation.

Note that when an XElement object is cloned by a LINQ to XML constructor or method, of course all attributes and descendant nodes are also cloned.

Removing Nodes from the Newly Cloned Tree

Sometimes when we clone a tree, we want to trim certain elements and attributes and nodes from the newly cloned tree. We can take advantage of the fact that it’s valid to pass null as one of the arguments to the XElement constructor. If one of the arguments to the XElement constructor is null, the XElement constructor simply ignores that argument:

XElement xml = new XElement("Root",

new XElement("Child1"),

null,

new XElement("Child3"));

Console.WriteLine(xml);

This produces the following output:

<Root>

</Root>

Let’s say that we want to trim all Child2 elements from the following tree:

<Root>

</Root>

We can code the transform like this:

using System;

using System.Linq;

using System.Xml.Linq;

class Program

{

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "Child2")

return null;

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(@"<Root><Child1/><Child2/><Child3/></Root>");

XElement newRoot = (XElement)Transform(root);

Console.WriteLine(newRoot);

}

These semantics apply regardless of whether null is passed directly to the XElement constructor, or whether null is passed as part of a collection to the XElement constructor.

Such element name comparisons (element.Name == "Child2") are very efficient if we pre-atomize element and attribute names. See this post for more information about atomization of element and attribute names.

Note: for simplicity of demonstration, in these examples, I didn’t bother to pre-atomize XNames, but pre-atomization is important if performance is important.

Replace an Element with another Element

A common operation in transforms is to replace some particular element with an entirely new element. Let’s say that we want to replace the Child2 element with a Child5 element that contains a child element named Text. We want to transform this:

<Root>

</Root>

To this:

<Root>

</Child5>

</Root>

We can code it like this:

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "Child2")

return new XElement("Child5",

new XElement("Text"));

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

Replace an Element with Multiple Elements

Sometimes we may want to replace a single element with multiple elements. We can take advantage of the fact that the behavior of the XElement constructor for collections of objects is, as mentioned, recursive. If one of the objects in the collection is itself a collection of objects then that child collection is itself iterated and all objects are added to the element being constructed.

The following example shows these semantics of the XElement constructor. The example first creates two lists of XElement objects. It then creates a list that contains the two lists. It passes the list of lists to the XElement constructor, and the resulting XElement contains all of the XElement objects in the two lists:

List<XElement> list1 = new List<XElement> {

new XElement("Child1"),

new XElement("Child2")

};

List<XElement> list2 = new List<XElement> {

new XElement("Child3"),

new XElement("Child4")

};

List<List<XElement>> listOfLists = new List<List<XElement>> { list1, list2 };

XElement newElement = new XElement("Root", listOfLists);

Console.WriteLine(newElement);

This produces the following output:

<Root>

</Root>

Let’s say that we want to replace the Child2 element with three Child5 elements. We want to transform this:

<Root>

</Root>

To this:

<Root>

</Root>

We can code it like this:

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "Child2")

return new List<XElement> {

new XElement("Child5"),

new XElement("Child5")

};

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

This example is coded a little bit artificially – more typically we would want to replace a particular element with a number of elements that are contained in a collection that is returned by some query. Let’s say that we wanted to effectively remove a certain element from the tree, and replace that element with its children elements. We want to transform this:

<Root>

<Child>

</Child>

</Root>

To this:

<Root>

</Root>

We can code it like this:

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "Child")

return element.Elements();

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(

@"<Root>

<Child>

</Child>

</Root>");

XElement newRoot = (XElement)Transform(root);

Console.WriteLine(newRoot);

}

This is why it’s important that the Transform function returns object instead of XNode or XObject. This is of course an exact parallel to the XElement constructor – the XElement constructor takes object as an argument so that you can pass either an XObject (or objects of derived classes) or a collection of XObject. The Transform function returns object so that you can return either an XObject or a collection of XObject. (There are other reasons that the XElement constructor takes object as a parameter, but these are the relevant reasons for this discussion.)

More Sophisticated Selection of Nodes to Transform

In the above examples, the code selected elements based on the element name. In XSLT, of course, we often write involved XPath expressions to select nodes to transform. Sometimes we want to select nodes for transformation where the node has specific ancestors. Let’s say that we want to transform a paragraph only if the paragraph is in a content control:

<body>

abc

<sdt>

def

</sdt>

</body>

We can code it like this:

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "p" && element.Parent.Name == "sdt")

return new XElement("p", "different content");

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(

@"<body>

abc

<sdt>

def

</sdt>

</body>");

XElement newRoot = (XElement)Transform(root);

Console.WriteLine(newRoot);

}

This will be efficient – if the node isn’t a paragraph node, then the Boolean expression will be short-circuited, and no extra work will be done. If the node is a paragraph node, then the code consists of following a pointer in the ancestor linked list, and another efficient XName comparison. I've written even more involved node selection code than this - select a paragraph to transform based on contents of the previous paragraph, for example. (Note that if you are selecting nodes in reverse document order, you may want to use an approach that gives good performance.)

Selectively Transforming Descendant Nodes

Sometimes we want to transform just a specific set of the descendant nodes, ignoring everything else. In the following query, the highlighted line serves the same purpose as xsl:apply-templates in an XSLT style sheet. This is the code that sends each of the descendants through the transform function. If we include a call to the Where extension method in that query, we can cause only a subset of the descendants to be sent through the recursive function.

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

Let’s say that we want to perform a transform on the following XML document:

<Root>

<Child>

</Child>

</Root>

We want to transform all of the ’A’ elements to ‘Apple’, the ‘B’ elements to ‘Banana’ and the ‘C’ elements to ‘Carot’, but only for all of the elements where Id equals ‘1’. We want the resulting XML file to look like this:

<Root>

<Child>

</Child>

</Root>

We can code the transform like this:

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "A")

return new XElement("Apple",

element.Attributes(),

element.Nodes().Select(e => Transform(e)));

if (element.Name == "B")

return new XElement("Banana",

element.Attributes(),

element.Nodes().Select(e => Transform(e)));

if (element.Name == "C")

return new XElement("Carot",

element.Attributes(),

element.Nodes().Select(e => Transform(e)));

if (element.Name == "Child")

return new XElement("Child",

element.Attributes(),

element.Elements()

.Where(e => e.Attribute("Id").Value == "1")

.Select(e => Transform(e)));

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

Only the XML elements that satisfy the predicate in the Where clause will be sent back through the transform function.

Notice that in this case, I used the Elements axis instead of the Nodes axis. Based on the needs of my transform, I can use any of a variety of axes to specify which nodes will be sent back through the recursive transform function.

Using Multiple Recursive Functions in the Same Transform

These types of transforms are always written in the pure style. In other words, no variables are ever mutated the while doing this transform. The transform always produces a completely new XML tree and never modifies the source tree. This gives us the freedom to call out to other recursive transforms that are also implemented in the pure style.

Let’s say that we’re transforming an Open XML document and we’ve written an interesting transform just for the paragraph element, we can write our transform for the main document part to call the paragraph transform function as appropriate. Because both transforms are written in a pure fashion, we can do this without worrying about introducing bugs associated with side effects in our code.

Following is the source document that we want to transform. This XML is ‘pseudo’ Open XML markup – namespaces are removed and the XML is simplified for the purposes of demonstration.

<body>

<r>

<t>This is a short sentence.</t>

</r>

<r>

<t>This is a different sentence.</t>

</r>

<sdt>

<r>

<t>This is a sentence in a content control.</t>

</r>

</sdt>

</body>

</document>

We want to produce the following document:

<body>

<r>

<t>THIS IS A SHORT SENTENCE.</t>

</r>

<r>

<t>THIS IS A DIFFERENT SENTENCE.</t>

</r>

<r>

<t>THIS IS A SENTENCE IN A CONTENT CONTROL.</t>

</r>

</body>

</document>

We could code the transformation like this:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Xml.Linq;

class Program

{

// This paragraph transform might do something very interesting - perhaps splitting

// runs in a new way or doing sophisticated modifications to styling. In this example,

// the code will transform the text node to upper case.

static object ParagraphTransform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "t")

return new XElement("t", ((string)element).ToUpper());

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => ParagraphTransform(n)));

}

return node;

}

// This transform might do something very interesting with content controls. In this

// case the transform will do something very simple - it removes the content control

// and promotes its children elements to the level of the content control.

static object Transform(XNode node)

{

XElement element = node as XElement;

if (element != null)

{

// call into the recursive transform for the paragraph

if (element.Name == "p")

return ParagraphTransform(element);

// remove the content control

if (element.Name == "sdt")

return element.Nodes().Select(n => Transform(n));

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n => Transform(n)));

}

return node;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(

@"<document>

<body>

<r>

<t>This is a short sentence.</t>

</r>

<r>

<t>This is a different sentence.</t>

</r>

<sdt>

<r>

<t>This is a sentence in a content control.</t>

</r>

</sdt>

</body>

</document>");

XElement newRoot = (XElement)Transform(root);

Console.WriteLine(newRoot);

}

Because this transformation was coded in this style, artifacts of Open XML such as custom XML elements and nested content controls are handled automatically without us taking any special care to handle them.

Sometimes it’s useful to write small utility functions for some of the functionality supplied by each recursive transform function. It may be useful to put each transform function and its associated utility functions in a static class.

Normalization of Empty Elements

There is one point regarding empty elements that is worth mentioning – empty elements <Tag></Tag> are normalized to use the empty-element tag <Tag />. This post contains a discussion on empty elements.

I had an unfortunate bike accident last week and broke my right arm, and I’m right handed! I used the voice recognition features of Windows 7 to write this post. These features are great! I don’t think that the process of writing this post was appreciably slower than usual.

Document-Centric Transforms using LINQ to XML

EricWhite — Thu, 09 Jul 2009 21:16:00 GMT

[Blog Map]

When thought of in a certain way, XML documents come in two flavors – data-centric and document-centric. Further, there are two types of document-centric documents. This post presents my thoughts about approaches to various types of document-centric transformations – data-centric to document-centric, document-centric to data-centric, and document-centric to document-centric. Then, I’ll tie my thoughts back to Open XML transformations.

Data-centric to data-centric is, of course, the scenario that LINQ to XML absolutely shines at. There’s been a lot written about this. This post won’t focus on these types of transformations, but instead will give my thoughts on the wrinkle that document-centric XML documents give to transformations.

First, I’ll define what I mean by data-centric and document-centric XML documents.

Data-Centric XML Document

A data-centric XML document contains regular repeating elements. Child elements of a given element might all have the same tag name, or they might not. Typically, child element order doesn’t matter. There are lots of examples of this – many types of transforms of a relational database to XML results in data-centric XML. RSS feeds are another.

Here’s a data-centric XML document:

</Customer>

</Customer>

</Customers>

Document-Centric XML Document

Document-centric XML documents have the characteristic that the child elements of a given element are much less bounded – you might have many child elements of a given name, or you might have none. You might have ‘recursion’ in the hierarchy – element A is a child of element B, which is itself a child of a different element A. A number of examples: Open XML word processing markup, XHTML, and XPS.

I further divide document-centric XML documents into two camps – those that contained mixed content, and those that don’t. Mixed content is a variety of XML where significant text nodes and elements are interspersed. Insignificant text nodes are the white space that provides indenting when formatting XML. Open XML word processing markup doesn’t contain mixed content, whereas XHTML does:

An Open XML paragraph that contains some bold text:

<w:p>

<w:r>

<w:t>abc</w:t>

</w:r>

<w:r>

<w:rPr>

<w:b/>

</w:rPr>

<w:t>def</w:t>

</w:r>

<w:r>

<w:t>ghi</w:t>

</w:r>

</w:p>

An XHTML document that contains significant text nodes interspersed with element start and end tags:

<html>

<body>

abcdefghi

</body>

</html>

Types of Transformations

If we’re going to divide the XML world into the categories of data-centric and document-centric, it follows that there are four types of transformations.

Data-Centric => Data Centric

There is a lot to say (and has been said) about these types of transformations. In the LINQ to XML documentation, I included a tutorial on pure functional transformations of XML. I also have a tutorial on my blog on composing queries in the pure functional style.

Data-Centric => Document Centric

These transforms are report writers for databases – take some subset of records, transform to XML, then transform that XML into another form – XPS, for instance. The transform may be based on another source document, the report definition. These types of transforms are straightforward to write in the pure functional style. Based on the simplicity or complexity of the report definition, this type of transform could be a few hundred lines of code or many thousands.

There are also many good examples of transforming data-centric XML to Open XML markup. We may want to transform a collection of records into a table in a word processing document, or into rows and cells in a worksheet.

Document-Centric => Data Centric

We write this type of transform when querying an Open XML document for some aspect of the markup. If we want to retrieve a collection of comments from a document, or if we want a collection of content controls, then we write a query that iterates over certain descendant elements, projecting a regular data structure – perhaps a collection of strings or anonymous types. The query that I develop in my functional programming tutorial is a document-centric => data-centric transform.

Another example is finding all hyper-links in an XHTML document. It is easy to write a LINQ query to retrieve a collection of links and transform the collection to a regular repeating data structure.

Document-Centric => Document-Centric

This is where it starts to get a little more involved. There are a variety of these types of transformations.

Common-vocabulary document-centric transform: Sometimes we want to transform a XML tree to a new tree in the same vocabulary. For example, we can transform an Open XML document into a new Open XML document with modified contents - comments are removed or revisions accepted. Another example - replace content controls with other markup based on the contents of the content controls.

Different-vocabulary document-centric transform: Sometimes we want to transform from one document-centric vocabulary to another one – Open XML => XHTML, or XHTML => Open XML. With this type of transform, the ease with which we write the transform is directly related to whether the two vocabularies have a similar structure. For instance, there is much that is parallel between Open XML and XHTML. There is a body element. The body contains paragraphs and tables. Tables contain rows, which contain cells. Tables can contain other tables in cells.

XSLT works well for these types of transformations – you write a pattern to match a node, and then supply the transformation for just that node. In the case of XSLT, you can indicate to the transform engine to ‘continue processing rules for child elements’, so that you can specify the transforms for those child elements in their own rules. If you are aware of Flat OPC, it is pretty easy to process Open XML documents using XSLT.

Some time ago, I write a post on an approach for using LINQ to XML annotations for doing this type of transform. In that post, I was proving out whether you could write document-centric transforms using LINQ to XML in a style similar to XSLT. It’s easy to read the code to specify the transform if you read LINQ code easily, but there are obvious problems with the approach, not the least of which is that annotating a tree in that fashion might have performance issues if you are working with too large of a tree.

Even though Open XML and XHTML have similar structures, there are places where the structures are not parallel, and in those cases, you still must jump through hoops. In XSLT, this often means generating intermediate trees to use in subsequent transforms. I’ve seen XSLT transforms where the first thing the developer did was to transform the tree to a new tree with new attributes on elements – the purpose of the attributes was to aid further transforms. If using the LINQ to XML approach that uses annotations, you must deal with the same issues – parts of the transformation are expressed as nice mappings between a pattern that matches nodes and the subsequent transform of those nodes, and parts of the transformation deals with abstractions that often must be explained in comments. It’s just more complicated to do these types of transformations.

Open XML Document-Centric Transforms

There are lots of examples of interesting common-vocabulary document-centric Open XML transforms. Removing comments and accepting revisions are two, but there are many others.

I've needed to write a number of these over the last couple of years. Because I know the size of documents that I potentially need to process (>2 million nodes), I rejected the annotations approach for simple transforms of Open XML documents. For performance reasons, it just wouldn’t fly.

I also rejected using XSLT – I really don’t want to step out into another language. XSLT is an attractive approach if you already have an XSLT transform written, or if you are particularly fluent in XSLT. You must deal with converting the OPC (Zip) file to the Flat OPC format, but this is easy. But when I’m writing little examples that show how to do something interesting in Open XML, XSLT isn’t appropriate.

So, for instance, for the code to accept tracked revisions, I opted for the tree-modification approach. This isn’t idea from a functional programming purist’s point of view, but it performs well in the real world. You have to be careful when coding, but no big deal.

Recursive Approach to Transforms

Lately, I’ve been writing more of these types of transforms in a recursive style. The gist of this technique is that you write a recursive function to clone a tree, and while cloning, you trim nodes, or transform nodes, or whatever.

This approach has good performance, and it is appealing in that when you are writing a more complicated recursive transform, you can write it in terms of other simpler recursive transforms. The code should be written with no side effects, and if so, transforms are easy to write and debug.

This approach has a draw-back. It’s somewhat harder to intuitively see the mapping between the pattern that matches a node, and the transform for that pattern. However, we don’t lose this entirely. For example, we may want to write a recursive transform of an XHTML document to Open XML. Here is the XHTML document:

<html>

<body>

abcdefghi

</body>

</html>

We want to transform it to this document:

<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" >

<w:body>

<w:p>

<w:r>

<w:t>abc</w:t>

</w:r>

<w:r>

<w:rPr>

<w:b/>

</w:rPr>

<w:t>def</w:t>

</w:r>

<w:r>

<w:t>ghi</w:t>

</w:r>

</w:p>

</w:body>

</w:document>

We can write the recursive transform like this:

using System;

using System.Linq;

using System.Xml.Linq;

class Program

{

static object XHtmlToOpenXml(XNode node)

{

XNamespace w = "http://schemas.openxmlformats.org/wordprocessingml/2006/main";

XElement element = node as XElement;

if (element != null)

{

if (element.Name == "html")

return new XElement(w + "document",

new XAttribute(XNamespace.Xmlns + "w", w.NamespaceName),

new XElement(w + "head", ""),

element.Elements().Select(e => XHtmlToOpenXml(e)));

if (element.Name == "body")

return new XElement(w + "body",

element.Elements().Select(e => XHtmlToOpenXml(e)));

if (element.Name == "p")

return new XElement(w + "p",

element.Nodes().Select(n => XHtmlToOpenXml(n)));

if (element.Name == "b")

return new XElement(w + "r",

new XElement(w + "rPr",

new XElement(w + "b")),

new XElement(w + "t",

element.Value));

}

XText t = node as XText;

if (t != null)

return new XElement(w + "r",

new XElement(w + "t", t.Value));

// ignore all other nodes

return null;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(

@"<html>

<body>

abcdefghi

</body>

</html>");

Console.WriteLine(XHtmlToOpenXml(root));

}

In the above transform, the code highlighted in yellow serves the same purpose as the XPath pattern to match in an XSLT template. The code highlighted in green is the “sequence constructor”. The expressions element.Nodes().Select(n => XHtmlToOpenXml(n)) and element.Elements().Select(e => XHtmlToOpenXml(e)) serves the same purpose as the xsl:apply-templates element in an XSLT template.

Key to understanding this transform is that selectively, we can cause every node/element to be passed to this method. But we can also trim descendant nodes if we like, sending only a subset back through this method.

I initially started talking about this approach in a post that described manually cloning XML trees. The code is short and easy to understand.

I used this approach for code to normalize an XML tree. It performs well. Of the approaches that I could have taken for coding the sample, it was by far the easiest.

I also used this approach for the code to split runs in paragraphs. Again, it was the easiest way for me to write the code.

For more information about writing these types of transformations, see Recursive Approach to Pure Functional Transformations of XML.

This certainly isn’t the last word. This is what has been on my mind lately, so wanted to blog it before I forgot about it.

I’m fascinated by XML document transformation, primarily because of the power it gives me. The ability to spin out an Open XML document in a couple hundred lines of code opens up a lot of interesting scenarios. Generating documents server-side in SharePoint or a web application allows us to use documents to make it easier for people to communicate. Document-centric transforms are key in these scenarios.

Empty Elements and Self-Closing Tags

EricWhite — Wed, 08 Jul 2009 02:06:00 GMT

Last January, I blogged about an approach to normalizing LINQ to XML trees. That post is based on another post, Manually Cloning LINQ to XML Trees. In those posts, my code to clone an element would clone a self-closing element (<Tag/>) as self-closing, and an empty element with a start and end tag (<Tag></Tag>) as an element with start and end tag.

But, in fact, this was not necessary – empty elements can be always serialized as self-closing elements – the XML specification states, “The representation of an empty element is either a start-tag immediately followed by an end-tag, or an empty-element tag."

Further, per the specification, “the empty-element tag SHOULD be used, and SHOULD only be used, for elements which are declared EMPTY”. This means that it’s always safe to serialize an empty element as a self-closing element, but sometimes it’s not correct to serialize an empty element with a start and end tag.

Originally, the code to clone an element looked like this:

static XElement CloneElement(XElement element)

{

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n =>

{

XElement e = n as XElement;

if (e != null)

return CloneElement(e);

return n;

}),

(!element.IsEmpty && !element.Nodes().OfType<XText>().Any()) ? "" : null

);

}

I’ve revised both of the above referenced posts to remove the code to exactly serialize empty elements as they were in the source document. The new code looks like this:

static XElement CloneElement(XElement element)

{

return new XElement(element.Name,

element.Attributes(),

element.Nodes().Select(n =>

{

XElement e = n as XElement;

if (e != null)

return CloneElement(e);

return n;

})

);

}

static void Main(string[] args)

{

XElement root = XElement.Parse("<Root></Root>");

Console.WriteLine("Original tree");

Console.WriteLine(root);

Console.WriteLine();

Console.WriteLine("Cloned tree");

XElement rootClone = CloneElement(root);

Console.WriteLine(rootClone);

}

The code is simpler and more correct. When you run this example, it produces:

Original tree

Cloned tree

Thanks to Sean Hederman for pointing this out.

Comparing Two Open XML Documents using the Zip Extension Method

EricWhite — Sun, 05 Jul 2009 19:09:00 GMT

[BlogMap]

(Note: this is a post on comparing two Open XML word processing documents. For a post on comparing two XML documents, see Equality Semantics of LINQ to XML Trees.)

Sometimes we want to compare two word processing documents to see if they contain the same content. I’m working on a blog post to merge comments from multiple Open XML documents into a single document. This is based on a feature in Word 2007 that allows you to lock a document and prevent changes to content, yet allows users to add comments to the document. However, we don’t want to attempt to merge comments if the documents don’t contain the same content.

One Approach to Comparing Two Documents for Equivalency

If two documents contain exactly the same content, they will have the same number of paragraphs, tables, content controls, and custom XML markup, and more, and these elements will occur in the same order, and have the same content. However, two paragraphs may contain the same content yet their XML representation may be very different if one has a comment and the other does not – the paragraph with the comment may have its runs split differently. I’ve written a previous post that examines run splitting in detail, and contains a method to report where the run splits are, and a method that splits runs based on a list of split locations.

The following markup shows a very simple paragraph. We can see the paragraph element, the run element, and the text element.

<w:p>

<w:r>

<w:t>abcdefghi</w:t>

</w:r>

</w:p>

If we select “def” in the above text, and add a comment, the markup changes to look like this:

<w:p>

<w:r>

<w:t>abc</w:t>

</w:r>

<w:commentRangeStart w:id="0"/>

<w:r>

<w:t>def</w:t>

</w:r>

<w:commentRangeEnd w:id="0"/>

<w:r>

<w:rPr>

<w:rStyle w:val="CommentReference"/>

</w:rPr>

<w:commentReference w:id="0"/>

</w:r>

<w:r>

<w:t>ghi</w:t>

</w:r>

</w:p>

We can write a query that returns a collection of a very specific subset of the elements in the XML document. This is the subset of elements that won’t change if the contents of the document don’t change. This query consists of all elements in the document except:

w:commentRangeStart and w:commentRangeEnd – these elements will be added when the user adds comments to a document. Most commonly, these elements occur under the paragraph element (to be trimmed, see below), but its valid for these elements to be children of the body elements, so we should trim them.
w:proofErr – this element is added automatically by Word when there are spelling or grammar errors, and has no effect on content. Word can (and will) add this element even though the document is locked for editing with the exception of being able to add comments. Therefore, we want to trim this element from the collection returned by a query that we’re going to use to determine document equivalency.
Finally, we want to eliminate all of the descendants of paragraphs from the query, as these elements can change quite a bit even if the contents of the document don’t change. Instead, we want to write a bit of code to determine whether two paragraphs are equivalent.

Here is the query that returns a collection of the elements that we’re interested in:

XDocument xDoc1 = doc1.MainDocumentPart.GetXDocument();

var doc1Elements = xDoc1

.Descendants()

.Where(e => e.Name != W.commentRangeStart &&

e.Name != W.commentRangeEnd &&

e.Name != W.proofErr &&

!e.Ancestors(W.p).Any());

We can query two word processing documents, and if the elements in the returned collection are not in the exact same order, then the documents are different. And if corresponding paragraphs contain the same content, per whatever algorithm that we define, then we can say that the documents contain the same content. In the example that I present in this post, I validate paragraph equivalency by checking actual textual content, disregarding formatting changes for runs within the paragraph. In my case, this is good enough, as the transformation that I wrote (and will present in an upcoming post) that moves comments from one document to another will work properly if the paragraphs have the same text.

The above query works just fine for documents that contain tables, content controls, and custom XML. The markup for tables, content controls, and custom XML contains paragraphs, and the paragraphs will be in the same order and have the same content if the documents are equivalent.

We could change this query easily enough to define document equality in just about any way we want. If we want to disregard bookmarks, it’s easy enough to remove them from the results of the query.

The above query is not the most efficient way to do this – more efficient would be to write an iterator that goes through the Descendants axis and trims appropriately. But queries show intent in a better way, and in my informal testing, the above query performs well enough as is for many scenarios.

Now that we’ve defined the query that will return the elements that won’t change if the document content doesn’t change, we can define another query that determines if two queries, evaluated on two Open XML documents, contain the same items, and in the same order. This is a job for the Zip extension method, coming with C# 4.0.

The Zip Extension Method

The Zip extension method processes two sequences, matching up each item in one sequence with a corresponding item in another sequence. While this method won’t be part of the framework until C# 4.0, a simple implementation that we can use with C# 3.0 is trivial:

public static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>(

this IEnumerable<TFirst> first,

IEnumerable<TSecond> second,

Func<TFirst, TSecond, TResult> func)

{

var ie1 = first.GetEnumerator();

var ie2 = second.GetEnumerator();

while (ie1.MoveNext() && ie2.MoveNext())

yield return func(ie1.Current, ie2.Current);

}

Note: Bart De Smet has a great explanation of the Zip extension method, as well as this example of the implementation of it. That post also has a good explanation of how iterators work, using IL to explain them. With regards to iterators, it’s also useful to read the section 8.14 in the C# 3.0 specification.

Using the Zip Extension Method

If we have one sequence that contains names, and another sequence that contains ages, and we know that the two sequences contain corresponding elements, we can project a new collection of anonymous objects:

string[] names = new[] { "Jim", "Bob", "Susan" };

int[] ages = new[] { 50, 35, 41 };

var q = names.Zip(ages, (name, age) => new

{

Name = name,

Age = age

});

foreach (var item in q)

Console.WriteLine(item);

When you run this example, you see:

{ Name = Jim, Age = 50 }

{ Name = Bob, Age = 35 }

{ Name = Susan, Age = 41 }

Notice that for the projection, we write a lambda expression that takes two arguments – each pair of corresponding items from the two source collections is passed as arguments to the lambda expression.

The following query uses the Zip extension method to project a collection of Booleans indicating if the element or paragraph is equivalent:

IEnumerable<bool> correspondingElementEquivalency = doc1Elements.Zip(doc2Elements, (e1, e2) =>

{

if (e1.Name != e2.Name)

return false;

// determine if two paragraphs contain the same content

if (e1.Name == W.p && (GetParagraphText(e1) != GetParagraphText(e2)))

return false;

return true;

});

GetParagraph text is defined as:

// return the text of a paragraph with revisions accepted

public static string GetParagraphText(XElement p)

{

return p.Descendants(W.r)

.Where(e => e.Parent.Name != W.del && e.Parent.Name != W.moveFrom)

.Descendants(W.t)

.Select(t => (string)t)

.StringConcatenate();

}

So then, we can use the Any extension method to determine if the documents are equivalent:

return ! correspondingElementEquivalency.Any(e => e != true);

This will be pretty efficient, as it uses lazy evaluation, and the Any extension method will terminate processing as soon as the code determines that the documents are different.

One Final Note

This code doesn’t process math markup – if the two documents contain a math formula, and one of the documents is commented, then this query will report that the documents differ. The structure and approach to take are exactly parallel to the approach that I take with comments in regular paragraphs. Extending this to math markup is another post.

The Code

Following is an example that compares two documents to determine if they have the same content. The code is quite short. Note this uses the Open XML SDK.

using System;

using System.Collections.Generic;

using System.IO;

using System.Linq;

using System.Text;

using System.Xml;

using System.Xml.Linq;

using DocumentFormat.OpenXml.Packaging;

public static class Extensions

{

public static XDocument GetXDocument(this OpenXmlPart part)

{

XDocument xdoc = part.Annotation<XDocument>();

if (xdoc != null)

return xdoc;

using (StreamReader streamReader = new StreamReader(part.GetStream()))

xdoc = XDocument.Load(XmlReader.Create(streamReader));

part.AddAnnotation(xdoc);

return xdoc;

}

public static string StringConcatenate(this IEnumerable<string> source)

{

StringBuilder sb = new StringBuilder();

foreach (string s in source)

sb.Append(s);

return sb.ToString();

}

public static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>(

this IEnumerable<TFirst> first,

IEnumerable<TSecond> second,

Func<TFirst, TSecond, TResult> func)

{

var ie1 = first.GetEnumerator();

var ie2 = second.GetEnumerator();

while (ie1.MoveNext() && ie2.MoveNext())

yield return func(ie1.Current, ie2.Current);

}

public static class W

{

public static XNamespace w =

"http://schemas.openxmlformats.org/wordprocessingml/2006/main";

public static XName p = w + "p";

public static XName r = w + "r";

public static XName t = w + "t";

public static XName commentRangeStart = w + "commentRangeStart";

public static XName commentRangeEnd = w + "commentRangeEnd";

public static XName proofErr = w + "proofErr";

public static XName del = w + "del";

public static XName moveFrom = w + "moveFrom";

}

class Program

{

// return the text of a paragraph with revisions accepted

public static string GetParagraphText(XElement p)

{

return p.Descendants(W.r)

.Where(e => e.Parent.Name != W.del && e.Parent.Name != W.moveFrom)

.Descendants(W.t)

.Select(t => (string)t)

.StringConcatenate();

}

// returns true if the documents contain the same content, otherwise false

private static bool CompareDocuments(WordprocessingDocument doc1,

WordprocessingDocument doc2)

{

XDocument xDoc1 = doc1.MainDocumentPart.GetXDocument();

XDocument xDoc2 = doc2.MainDocumentPart.GetXDocument();

var doc1Elements = xDoc1

.Descendants()

.Where(e => e.Name != W.commentRangeStart &&

e.Name != W.commentRangeEnd &&

e.Name != W.proofErr &&

!e.Ancestors(W.p).Any());

var doc2Elements = xDoc2

.Descendants()

.Where(e => e.Name != W.commentRangeStart &&

e.Name != W.commentRangeEnd &&

e.Name != W.proofErr &&

!e.Ancestors(W.p).Any());

IEnumerable<bool> correspondingElementEquivalency = doc1Elements

.Zip(doc2Elements, (e1, e2) =>

{

if (e1.Name != e2.Name)

return false;

// determine if two paragraphs contain the same content

if (e1.Name == W.p && (GetParagraphText(e1) != GetParagraphText(e2)))

return false;

return true;

});

return ! correspondingElementEquivalency.Any(e => e != true);

}

static void Main(string[] args)

{

using (WordprocessingDocument doc1 = WordprocessingDocument.Open("Test3a.docx", false))

using (WordprocessingDocument doc2 = WordprocessingDocument.Open("Test3b.docx", false))

{

bool same = CompareDocuments(doc1, doc2);

Console.WriteLine(same);

}

Querying LINQ to XML Nodes in Reverse Document Order with Better Performance

EricWhite — Wed, 24 Jun 2009 20:25:00 GMT

[Blog Map]

(Update June 25, 2009 - fixed bugs in event handlers associated with deleting last node and inserting node at beginning of list)

Occasionally I need to query LINQ to XML nodes in reverse document order. I’m currently writing some LINQ to XML queries over Open XML documents where I need to select paragraph nodes based on content in the immediately preceding paragraph. However, nodes in LINQ to XML are forward-linked only. We can see evidence of this in the XNode.NodesBeforeSelf and the XElement.ElementsBeforeSelf methods - these methods return collections of nodes in document order, not reverse document order. This was by design – LINQ to XML was designed to provide great performance for the vast majority of scenarios with the minimum memory footprint possible. The need to process nodes in reverse document order is rare, so the designers of LINQ to XML decided that it was more important to reduce memory footprint than to allow for good performance in the few scenarios that require processing in reverse document order, and of course it was a good decision. But the need does exist.

In my scenario (a functional transform that processes Open XML document revisions), it is possible that I would need to process 80,000 (or more) paragraphs. If we use the XNode.PreviousNode property, we won’t have acceptable performance. There is an easy work-around that provides us the ability to query in reverse document order in a way that performs well.

We define a new class, PreviousNodeAnnotation , that contains one public field, public XNode PreviousNode;.
We add instances of this class as annotations on nodes that we need to query in reverse document order.

In the following small example, I select nodes based on previous node value for a document size of 50,000. The slow version exhibits performance of O(n²). I limited the sample size in the example to 50,000 nodes. When I increased the doc size to 80,000 nodes (the size of one of my documents that I need to query), the execution time of the slow version exceeded my patience. In any case, it is clear that I can’t use XNode.PreviousNode for my scenario.

using System;

using System.Linq;

using System.Xml.Linq;

class PreviousNodeAnnotation

{

public XNode PreviousNode;

public PreviousNodeAnnotation(XNode prev) { PreviousNode = prev; }

}

class Program

{

static int DocumentSize = 50000;

static void SlowPreviousNodeAccess()

{

// create a tree with lots of nodes

XElement root = new XElement("Root",

Enumerable.Range(0, DocumentSize).Select(i => new XElement("Child", i)));

// query for all elements where the previous element has a value of 1000

DateTime start = DateTime.Now;

var q = root

.Elements()

.Where(e =>

{

XElement p = e.PreviousNode as XElement;

return (string)p == "1000";

});

var q2 = q.ToList(); // force iteration

TimeSpan duration = DateTime.Now - start;

Console.WriteLine(duration);

}

static void FastPreviousNodeAccess()

{

// create a tree with lots of nodes

XElement root = new XElement("Root",

Enumerable.Range(0, DocumentSize).Select(i => new XElement("Child", i)));

// initialize previous node annotations

XElement prev = null;

foreach (var item in root.Elements())

{

item.AddAnnotation(new PreviousNodeAnnotation(prev));

prev = item;

}

// query for all elements where the previous element has a value of 1000

DateTime start = DateTime.Now;

var q = root

.Elements()

.Where(e =>

{

XElement p = e

.Annotation<PreviousNodeAnnotation>()

.PreviousNode as XElement;

return (string)p == "1000";

});

var q2 = q.ToList(); // force iteration

TimeSpan duration = DateTime.Now - start;

Console.WriteLine(duration);

}

static void Main(string[] args)

{

FastPreviousNodeAccess();

SlowPreviousNodeAccess();

}

On my old slow laptop, the execution time of these two queries is .015 seconds and 30 seconds, respectively.

We can expand on this technique a bit by declaring three extension methods:

public static XNode PreviousNodeFast(this XNode node)

public static IEnumerable<XNode> NodesBeforeSelfFast(this XNode node)

public static IEnumerable<XElement> ElementsBeforeSelfFast(this XElement element)

It’s convenient to use these methods in queries.

In addition, while I’m partial to pure transforms with no side effects, we can declare two event handlers that keep the previous node annotations in sync when adding or deleting nodes. Here’s an example that includes these extension methods and event handlers:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Xml.Linq;

class PreviousNodeAnnotation

{

public XNode PreviousNode;

public PreviousNodeAnnotation(XNode prev) { PreviousNode = prev; }

}

public static class Extensions

{

public static XNode PreviousNodeFast(this XNode node)

{

return node.Annotation<PreviousNodeAnnotation>().PreviousNode;

}

public static IEnumerable<XNode> NodesBeforeSelfFast(this XNode node)

{

XNode currentNode = node;

while (true)

{

XNode prevNode = currentNode.PreviousNodeFast();

if (prevNode == null)

yield break;

else

yield return prevNode;

currentNode = prevNode;

}

public static IEnumerable<XElement> ElementsBeforeSelfFast(this XElement element)

{

return NodesBeforeSelfFast(element).OfType<XElement>();

}

class Program

{

static void ValidatePreviousNodes(XElement element)

{

XNode prev = null;

foreach (XNode node in element.Nodes())

{

if (node.PreviousNodeFast() != prev)

{

Console.WriteLine("ERROR: previous nodes are invalid");

Environment.Exit(0);

}

prev = node;

}

Console.WriteLine("Validated");

}

static void Main(string[] args)

{

// create a tree with lots of nodes

XElement root = new XElement("Root",

Enumerable.Range(0, 100).Select(i => new XElement("Child", i)));

// setup previous nodes after tree creation

XElement prev = null;

foreach (var item in root.Elements())

{

item.AddAnnotation(new PreviousNodeAnnotation(prev));

prev = item;

}

// add event handlers to take care of adding / deleting nodes

root.Changed += new EventHandler<XObjectChangeEventArgs>((o, e) =>

{

if (e.ObjectChange == XObjectChange.Add)

{

Console.WriteLine("Add");

XNode node = o as XNode;

// o could be an XAttribute, in which case it's not applicable

if (node != null)

{

node.AddAnnotation(new PreviousNodeAnnotation(node.PreviousNode));

if (node.NextNode != null)

{

node.NextNode.RemoveAnnotations<PreviousNodeAnnotation>();

node.NextNode.AddAnnotation(new PreviousNodeAnnotation(node));

}

});

root.Changing += new EventHandler<XObjectChangeEventArgs>((o, e) =>

{

if (e.ObjectChange == XObjectChange.Remove)

{

Console.WriteLine("Remove");

XNode node = o as XNode;

// o could be an XAttribute, in which case it's not applicable

if (node != null)

{

if (node.NextNode != null)

{

node.NextNode.RemoveAnnotations<PreviousNodeAnnotation>();

node.NextNode

.AddAnnotation(new PreviousNodeAnnotation(node.PreviousNode));

}

});

ValidatePreviousNodes(root);

root.Elements().ElementAt(3).AddAfterSelf(

new XElement("NewChild", 999)

);

ValidatePreviousNodes(root);

root.Nodes().ElementAt(2).Remove();

ValidatePreviousNodes(root);

root.Nodes().ElementAt(3).NodesBeforeSelfFast().Remove();

ValidatePreviousNodes(root);

root.AddFirst(

new XElement("ANode", 1)

);

ValidatePreviousNodes(root);

root.Add(

new XElement("ANode", 2)

);

ValidatePreviousNodes(root);

root.Nodes().Last().NodesBeforeSelfFast().Remove();

ValidatePreviousNodes(root);

root.Add(

new XElement("ANode", 2)

);

ValidatePreviousNodes(root);

root.Add(

new XElement("ANode", 2)

);

root.Add(

new XElement("ANode", 2)

);

root.Add(

new XElement("ANode", 2)

);

ValidatePreviousNodes(root);

root.Nodes().First().Remove();

ValidatePreviousNodes(root);

root.Add(

new XElement("ANode", 2)

);

ValidatePreviousNodes(root);

root.Nodes().Last().Remove();

root.Add(Enumerable.Range(0, 100).Select(i => new XElement("Child", i)));

XElement last = root.Elements().ElementAt(50);

foreach (var item in last.NodesBeforeSelfFast())

{

Console.WriteLine(item);

}

I would personally only define these extension methods in the module where I need good performance of reverse document order queries. It would be messy to have these extension methods in scope for modules that don't set up the annotations.

Code is attached.

Working with Optional Elements and Attributes in LINQ to XML Queries

EricWhite — Thu, 14 May 2009 07:02:00 GMT

Often XML schemas allow for optional elements and attributes. When you write queries on these elements or attributes, you may be tempted to write code that does lots of testing for null. There is a better way to do this, laid out in this post. I covered this idiom in a previous post, but the main purpose of that post wasn’t to explain this idiom. I’m speaking on using LINQ with Open XML tomorrow at TechEd 2009, and need a better example.

The following XML document is a simplified variation of markup that you can find in Open XML word processing documents:

<body>

<r>

<t>Text of first para.</t>

</r>

<pPr>

</pPr>

<r>

<t>Text of second para.</t>

</r>

</body>

</document>

The first paragraph doesn’t have a <pPr> element, whereas the second does. This is allowable in Open XML word processing documents. The first paragraph has the default style.

Our task is to write a query that returns the style name for each paragraph, but if the paragraph has no style name, then the paragraph has the default style. The code projects a collection of an anonymous type that contains the style name and the text. If the paragraph has the default style, the StyleName is set to null.

The approach where the code tests for null looks like this:

using System;

using System.Linq;

using System.Xml.Linq;

class Program

{

static string GetStyleName(XElement p)

{

XElement pPr = p.Element("pPr");

if (pPr != null)

{

XElement pStyle = pPr.Element("pStyle");

if (pStyle != null)

return (string)pStyle.Attribute("val");

}

return null;

}

static void Main(string[] args)

{

XElement root = XElement.Parse(

@"<document>

<body>

<t>Text of first para.</t>

<pPr>

</pPr>

<t>Text of second para.</t>

</body>

</document>");

var paragraphs = root

.Element("body")

.Elements("p")

.Select(p => new

{

StyleName = GetStyleName(p),

Text = (string)p.Element("t")

});

foreach (var item in paragraphs)

Console.WriteLine(item);

}

This works just fine, and yields the expected results:

{ StyleName = , Text = Text of first para. }

{ StyleName = Heading1, Text = Text of second para. }

Beyond making the code harder to read, this approach introduces two additional points of possible failure. If I had neglected to write the code to test for null, my code would throw an exception.

There is another way to write this query, which is to use the Elements and Attributes extension methods that operate on IEnumerable<XElement>.

using System;

using System.Linq;

using System.Xml.Linq;

class Program

{

static void Main(string[] args)

{

XElement root = XElement.Parse(

@"<document>

<body>

<t>Text of first para.</t>

<pPr>

</pPr>

<t>Text of second para.</t>

</body>

</document>");

var paragraphs = root

.Element("body")

.Elements("p")

.Select(p => new

{

StyleName = (string)p.Elements("pPr").Elements("pStyle")

.Attributes("val").FirstOrDefault(),

Text = (string)p.Element("t")

});

foreach (var item in paragraphs)

Console.WriteLine(item);

}

This also yields the same results, and doesn’t contain the two points of possible failure.

Here’s how this code works. In the snippet below, the highlighted code evaluates to a collection of XElement objects. Notice that I used the Elements method, not the Element method, even though I know that there could only be zero or one <pPr> elements. The highlighted code returns a collection of either zero or one items.

StyleName = (string)p.Elements("pPr").Elements("pStyle")

.Attributes("val").FirstOrDefault(),

The Elements extension method yields all child elements with the given name for each and every element in the source collection. In the snippet below, the highlighted code will return either one XElement object (if there was a <pPr> element), or an empty collection, if there wasn’t a <pPr> element:

StyleName = (string)p.Elements("pPr").Elements("pStyle")

.Attributes("val").FirstOrDefault(),

Next, the code ‘dots’ into the Attributes extension method. Again, the Attributes extension method is happy to take a collection of elements as source. If an empty collection is passed to the Attributes extension method, it also returns an empty collection:

StyleName = (string)p.Elements("pPr").Elements("pStyle")

.Attributes("val").FirstOrDefault(),

The FirstOrDefault extension method either returns the first element in a collection, or it returns the default value for the type of items in the collection. The default value for all reference types (which XAttribute and XElement are) is null. In this case, FirstOrDefault will either return the one “val” attribute, or it will return null.

Finally we cast this one value (either null or an XAttribute) to string. String is a nullable type, of course, and the explicit conversion operator (the cast operator for XAttribute or XElement) is happy to take null, and return null. If there is no <pPr> element, StyleName will be set to null. If there is a pPr element, a pStyle element, and a val attribute, then StyleName will be set to the value of the attribute.

This is where the other nullable CLR types (int?, bool?, double?, etc.) come in handy. If we want to get the value of an optional element or attribute, and we know that the value is an integer or double, or whatever, instead of casting to string, we can cast to any of the other nullable types. The same semantics apply.

Interestingly, this is also efficient. Here’s why:

The LINQ to XML axes use deferred execution.
FirstOrDefault starts the process of materialization, requesting the first item in the collection from the Attributes extension method.
It, in turn, requests the first item from the Elements(“pStyle”) call.
It, in turn, requests the first element from the XElement.Elements method, which finally yields up an XElement (or returns an empty collection if there isn’t one).
This one element is yielded up to the Elements(“pStyle”) extension method.
This one element is yielded up to the Attributes(“val”) extension method.
The one attribute is yielded up to the FirstOrDefault extension method, which due to its semantics, “short circuits” the query, and never requests another item from its source.

The net result of this is that this idiom is efficient, reduces points of possible failure, and is shorter. Once I started using this idiom regularly, it became quite natural. Once one of the LINQ architects described this idiom, I started using it, and mostly don't write code in the other style. So I’m curious – question for all you LINQ users out there, do you use this idiom?

LINQ / Open XML at TechEd 2009

EricWhite — Tue, 05 May 2009 01:44:00 GMT

I’ll be presenting a talk on LINQ and Open XML at TechEd 2009. The session is “OFC403 Developing Office Client Solutions Using LINQ and Open XML”. I’ll be presenting on Thursday, 5/14, at 4:30PM in Room 411. Here is the abstract for the talk:

In this session, learn how to leverage LINQ with version 1 and 2 of the Open XML SDK to create, modify, and query Open XML documents. Version 1 works well with LINQ to XML, and Version 2 provides a strongly-typed document object model (also LINQ friendly) that makes it easier to work with XML parts. Both approaches benefit from the conciseness of LINQ and the power of functional construction. In addition, we talk about and demonstrate tools that you can use to develop Open XML solutions faster and better. This session is very technical; knowledge of generics and LINQ are prerequisites to get the most from this session.

I’ve presented this talk before, at TechEd in Barcelona last fall, and at an internal Microsoft conference. In general, folks like this talk, and have given me high scores. However, in both previous presentations of this talk, one or two attendees criticize the talk because I spent some amount of time talking about LINQ and why you want to use it.

I feel conflicted about how to present the material in this talk. I always take an informal survey at the beginning of the talk, and 50% of the attendees haven’t really used LINQ, nor know too much about functional programming. I feel that without a little introduction to LINQ, these people might not appreciate the benefits that LINQ and functional programming offer to us. I certainly understand the sentiments of the LINQ / functional programming experts who attend – they basically want to see the tricks and patterns that I’ve developed for working with Open XML, and feel that the time spent on LINQ is wasted for them.

After a fair amount of reflection, I feel that more people are better served if I speak about LINQ for a few minutes before we dive into its application with Open XML, so I am again going to do the short intro to LINQ – I’ll limit this intro to 15 minutes, I promise! My goal in this intro is not to bring people up to speed on LINQ, but instead point out the basic principles that underlie it – specifically, how to craft pure functional transforms.

In the LINQ portion of the talk, as well as when I talk about using it with Open XML, I will be occasionally pointing out pages on my blog that contain more information. My goal here isn’t to increase traffic on my blog J, instead, it allows me to talk about concepts in as short amount of time possible, and attendees can refer to my blog to fill in the technical details. I want to cover as much information as possible, and this is an effective way, I believe.

Staffing the Office / SharePoint Booth

I’ll also be at the Office / SharePoint booths at the following times:

· Wednesday, May 13, 3:00 PM – 6:00 PM

· Thursday, May 14, 10:45 AM – 1:45 PM

· Friday, May 15, 10:00 AM - 1:15 PM

I really enjoy meeting customers and Microsoft employees who are using Office development technologies, including Open XML and SharePoint. Please feel free to come by the booth at the above times and introduce yourself! I’d love to hear about your development efforts, and am happy to help in any way that I can. I'll have a little time available for 1:1 consultations - if you'd like to chat in a side meeting about your development efforts, email me through the EMAIL link above.

The Use of Extension Methods to Manage Open XML Document Changes in PowerTools for Open XML

EricWhite — Tue, 28 Apr 2009 01:16:00 GMT

There is an interesting approach that we use in PowerTools for Open XML that makes it easy to write cmdlets that modify Open XML documents. This approach isn’t very complicated, but aspects of this approach need some explanation so that developers who are extending the PowerTools can understand what’s going on. This approach is based on the techniques detailed in Technical Improvements in the Open XML SDK and Using LINQ to XML Events and Annotations to Track if an XML Tree has Changed. This post explains the approach that we took in PowerTools in detail.

Note from Eric White: This is another guest post by Bob McClellan. I’ve met and worked with an awful lot of developers in my career, and Bob is one of the best developers that I’ve ever worked with. He’s an expert in more areas than is possible to list here, but a few relevant areas are C++, Windows development, C#, .NET, LINQ, Open XML, WPF, and SQL. His contributions to the Open XML conversation are well documented – he wrote this C code implementing the legacy hashing algorithm in word processing ML and the KParts proof-of-concept of an implementation of embedding linked objects from an Open XML document. He also wrote the code that enables document composability in v1.1 of PowerTools for Open XML. Bob, as you can see from the various guest posts on my blog, is also a good writer. Find out more about Bob here.

The Open XML SDK makes it very easy to create an XDocument object from a part by using a Stream. It does not, however, have any way to keep track of changes to that object. The PowerTools for Open XML uses a couple of simple extensions to the Open XML SDK classes to manage XDocument objects and changes to those documents.

Design Requirements

We wanted to meet certain requirements with these extensions. First, we didn’t want to add another class or new member variables and so on. We wanted the addition to be lightweight. Second, a part should only need to be read once for as long as the package is open. Third, changes should not be written out until we are ready to close the package (or when a logical group of changes are complete).

Synchronization Issues

In theory, it is a bad idea to represent the same data multiple times in a program. If the multiple values that should be the same become different for some reason, then the program will most certainly not work correctly. The best way to avoid that kind of problem is to store information just once and then there is nothing to synchronize. However, we often break this rule for various reasons and performance is one of them. When we are dealing with an Open XML package, we don’t want to have to keep reading and writing parts during a series of changes to that document. In particular, if the part is very large, many reads and writes could be time-consuming. As you will see below, the compromise is to keep the in-memory version of the document in an XDocument that is tightly coupled with the part so that it is unlikely that we will lose synchronization between the two.

Extension Methods

Extension methods are methods that are defined to appear as if they are member methods of an existing class. The limitation is that extension methods cannot add any member variables to the class, but there are ways around that limitation, as you will see below. The two extension methods that we will be examining are:

public static XDocument GetXDocument(this OpenXmlPart part)

public static void FlushParts(this OpenXmlPackage doc)

The first method will be used to get an XDocument for a particular part. The keyword “this” signifies that it is an extension method for the OpenXmlPart class. This method will look to see if an XDocument has already been created for that part. Otherwise, it creates the XDocument.

The second method is used to write out any changes to the XDocuments that have been created. It is an extension to the OpenXmlPackage class because changes to all parts will be written out when this method is called.

Caching XDocuments

As mentioned above, we want to make sure that an XDocument is only read once no matter how many times we might examine it or modify it while the package is open. An easy way to do this without creating new objects is by using Annotations on the part in the Open XML SDK. Annotations are simply objects that can be attached to other objects. We can avoid reading XDocuments more than once by attaching the XDocument to the part as an annotation. Here is the GetXDocument extension method:

public static XDocument GetXDocument(this OpenXmlPart part)

{

XDocument xdoc = part.Annotation<XDocument>();

if (xdoc != null)

return xdoc;

try

{

using (StreamReader sr = new StreamReader(part.GetStream()))

using (XmlReader xr = XmlReader.Create(sr))

{

xdoc = XDocument.Load(xr);

xdoc.Changed += ElementChanged;

xdoc.Changing += ElementChanged;

}

catch (XmlException)

{

xdoc = new XDocument();

xdoc.AddAnnotation(new ChangedSemaphore());

}

part.AddAnnotation(xdoc);

return xdoc;

}

The parts that deal with ElementChanged and the ChangedSemaphore will be explained in the next section. The first line calls a method that tries to retrieve an annotation that is an XDocument object. If there is no XDocument object for that part, the method returns null. The next line checks to see if the XDocument was returned. If so, it was read in a previous call and there’s nothing else we need to do except return that XDocument.

If there is no annotation for the XDocument, then we need to read it in. The process for reading an XDocument from a part is next. A StreamReader class is needed for XmlReader and then that can be used for the static Load method that creates the XDocument. If that process fails, then we can assume that the part doesn’t have any content yet, so we create an empty XDocument. In either case, once we have the new XDocument object, we then add an annotation with that object so that this part will not be read again.

Tracking Changes

There were a few lines of code in the GetXDocument method that are used to track changes to that XDocument. The basic approach to tracking changes is to use the ChangedSemaphore object to “tag” the XDocuments that have changed.

private class ChangedSemaphore { }

The class has no content; it is just used to identify which have changed. The other two lines of code from that method that are used to track changes are setting event handlers that will be called when the XDocument is changed by any other method calls. Here is the code for the event handler:

private static EventHandler<XObjectChangeEventArgs> ElementChanged = new
EventHandler<XObjectChangeEventArgs>(ElementChangedHandler);

private static void ElementChangedHandler(object sender,
XObjectChangeEventArgs e)

{

XDocument xDocument = ((XObject)sender).Document;

if (xDocument != null)

{

xDocument.Changing -= ElementChanged;

xDocument.Changed -= ElementChanged;

xDocument.AddAnnotation(new ChangedSemaphore());

}

This method is called when the XDocument is changed. It starts by removing itself as an event handler. Once we have detected a change, there is no reason to have the event handler called again upon subsequent changes. Next is the addition of the ChangedSemaphore as an annotation for the XDocument. As you will see in the next section, we will use that annotation to determine which parts need to be written. If you look back to the GetXDocument method, you will also see that the ChangedSemaphore object is added as an annotation for a new XDocument because we assume that a newly created empty XDocument will always be changed.

Writing Changes

The process of writing out the changes is handled by the FlushParts method and its helper method shown below:

public static void FlushParts(this OpenXmlPackage doc)

{

HashSet<OpenXmlPart> visited = new HashSet<OpenXmlPart>();

foreach (IdPartPair item in doc.Parts)

FlushPart(item.OpenXmlPart, visited);

}

private static void FlushPart(OpenXmlPart part, HashSet<OpenXmlPart> visited)

{

visited.Add(part);

XDocument xdoc = part.Annotation<XDocument>();

if (xdoc != null && xdoc.Annotation<ChangedSemaphore>() != null)

{

using (XmlWriter xw = XmlWriter.Create(part.GetStream(FileMode.Create, FileAccess.Write)))

{

xdoc.Save(xw);

}

xdoc.RemoveAnnotations<ChangedSemaphore>();

xdoc.Changing += ElementChanged;

xdoc.Changed += ElementChanged;

}

foreach (IdPartPair item in part.Parts)

if (!visited.Contains(item.OpenXmlPart))

FlushPart(item.OpenXmlPart, visited);

}

The FlushParts method calls its helper method for each part in the package. The FlushPart helper method checks the XDocument for that part to see if it has changed and then writes it, if it has. It then recursively calls all the related parts for that part. The HashSet collection is used to keep track of which parts have already been checked. It is needed because the parts of a package can be referenced from multiple parts. It is even possible that there could be “loops” of references that would cause the method to enter an infinite recursion. Instead, each part is added to the “visited” collection as it is checked and then that collection is checked to be sure the part is not processed a second time.

Writing the document is just a matter of getting the stream for the part and then using the XmlWriter object to write it out. Once it is written, the method also removes the ChangedSemaphore annotation and sets the event handlers for changes again. This is done because the package remains open after the FlushParts call. If additional changes are made, we want to be sure they are detected.

Summary

I hope this shows how a very small amount of carefully designed code can create very powerful functionality. Although we have a little bit of a compromise by storing a copy of the part in memory, that risk is reduced by the simplicity of the code that handles it. There is still a risk, though. If any part of the code using these methods makes direct calls to load, process and write a part, then they become out of sync and we don’t have any way to detect if that happened. It’s all or nothing with this approach, but as long as you always use GetXDocument to get a part and use FlushParts before you close the package, your code will work properly.

-Bob McClellan

Using DocumentBuilder with Content Controls for Document Assembly

EricWhite — Tue, 21 Apr 2009 15:43:00 GMT

DocumentBuilder is an example class that’s part of the PowerTools for Open XML project that enables you to assemble new documents from existing documents. One of the problems to solve when moving markup from one document to another is that of interrelated markup – markup in one paragraph often has dependencies with markup in other paragraphs, or other parts of the Open XML package. Document builder fixes up interrelated markup when assembling a new document from existing documents. This post shows how to use DocumentBuilder in concert with content controls to control the document assembly.

Zeyad Rajabi wrote a blog post on using content controls to control document assembly. His post uses the altChunk approach for document assembly. This post presents code that mirrors the code in his post, except that this code uses DocumentBuilder. I’ve covered altChunk also, in How to Use altChunk for Document Assembly.

The updated post Inserting / Deleting / Moving Paragraphs in Open XML Wordprocessing Documents documents interrelationships in paragraph markup in detail.

The post Move/Insert/Delete Paragraphs in Word Processing Documents using the Open XML SDK introduces the DocumentBuilder class.

See Comparison of altChunk to the DocumentBuilder Class for more information about both approaches to document assembly.

The gist of the approach is that you insert content controls in the ‘template’ document, setting the tag of each content control to the name of the document that you want inserted at the point of the content control. For example, in the following document, SolarOverview.docx will replace the content control in the assembled document:

Example Code

The example takes a ‘template’ document, solar-system.docx, and inserts eleven documents into it. As I mentioned, each inserted document replaces a content control. This example demonstrates one approach to coding document assembly using content controls and DocumentBuilder:

static void Main(string[] args)

{

using (WordprocessingDocument solarSystem =

WordprocessingDocument.Open("solar-system.docx", false))

{

XNamespace w =

"http://schemas.openxmlformats.org/wordprocessingml/2006/main";

// get children elements of the <w:body> element

var q1 = solarSystem

.MainDocumentPart

.GetXDocument()

.Root

.Element(w + "body")

.Elements();

// project collection of tuples containing element and type

var q2 = q1

.Select(

e =>

{

string keyForGroupAdjacent = ".NonContentControl";

if (e.Name == w + "sdt")

keyForGroupAdjacent = e.Element(w + "sdtPr")

.Element(w + "tag")

.Attribute(w + "val")

.Value;

if (e.Name == w + "sectPr")

keyForGroupAdjacent = null;

return new

{

Element = e,

KeyForGroupAdjacent = keyForGroupAdjacent

};

}

).Where(e => e.KeyForGroupAdjacent != null);

// group by type

var q3 = q2.GroupAdjacent(e => e.KeyForGroupAdjacent);

// validate existence of files referenced in content controls

foreach (var f in q3.Where(g => g.Key != ".NonContentControl"))

{

string filename = f.Key + ".docx";

FileInfo fi = new FileInfo(filename);

if (!fi.Exists)

{

Console.WriteLine("{0} doesn't exist.", filename);

Environment.Exit(0);

}

// project collection with opened WordProcessingDocument

var q4 = q3

.Select(g => new

{

Group = g,

Document = g.Key != ".NonContentControl" ?

WordprocessingDocument.Open(g.Key + ".docx", false) :

solarSystem

});

// project collection of OpenXml.PowerTools.Source

var sources = q4

.Select(

g =>

{

if (g.Group.Key == ".NonContentControl")

return new Source(

g.Document,

g.Group

.First()

.Element

.ElementsBeforeSelf()

.Count(),

g.Group

.Count(),

false);

else

return new Source(g.Document, false);

}

).ToList();

DocumentBuilder.BuildDocument(sources, "solar-system-new.docx");

// dispose of the opened WordprocessingDocument objects

foreach (var g in q4)

if (g.Group.Key != ".NonContentControl")

g.Document.Dispose();

}

How the Code Works

The code consists of chained queries that eventually build up a list of OpenXml.PowerTools.Source objects, which is what we pass to DocumentBuilder.BuildDocument to specify the sources for the document assembly.

When building up the list of document source objects, where the ‘template’ document contains paragraphs or tables, then we need to include a source object with the source document set to the ‘template’ document, and the source range set to the range of those paragraphs. Where the ‘template’ document contains a content control, then we need to include a source object with the source document set to the document being imported. We don’t need to set a range – we simply import the entire document.

In other words, we need to group together all paragraphs that don’t contain content controls, and we need to process separately all content controls. This is a job for the GroupAdjacent extension method. If we create a key such that all non content control paragraphs have the same key, and all content controls have a unique key, then we’ll end up with groups of paragraphs to import from the template document, and separate groups that contain one content control each. As I develop the query, I’ll show intermediate results so that you can see exactly what I mean.

The results of the first query is a collection of the child elements of the <w:body> element:

// get children elements of the <w:body> element

var q1 = solarSystem

.MainDocumentPart

.GetXDocument()

.Root

.Element(w + "body")

.Elements();

This is pretty simple – no need to show the output from this query.

Here is the second query:

// project collection of tuples containing element and type

var q2 = q1

.Select(

e =>

{

string keyForGroupAdjacent = ".NonContentControl";

if (e.Name == w + "sdt")

keyForGroupAdjacent = e.Element(w + "sdtPr")

.Element(w + "tag")

.Attribute(w + "val")

.Value;

if (e.Name == w + "sectPr")

keyForGroupAdjacent = null;

return new

{

Element = e,

KeyForGroupAdjacent = keyForGroupAdjacent

};

}

).Where(e => e.KeyForGroupAdjacent != null);

// temporary code to dump q2

foreach (var item in q2)

Console.WriteLine(item.KeyForGroupAdjacent);

Environment.Exit(0);

If the child element of the <w:body> element is a content control, then the KeyForGroupAdjacent member of the anonymous type is set to the tag value of the content control (highlighted in yellow above).

If the child element is not a content control, then KeyForGroupAdjacent is set to “.NonContentControl”, which is an invalid filename – no chance to conflict with the tag values of the content controls.

If the child element is a section marker (<w:sectPr>), then we want to ignore that child element. Setting the KeyForGroupAdjacent to null, and then filtering out those null items takes care of that.

When we dump out q2 to the console, we see:

.NonContentControl

SolarOverview

Sun

Mercury

Venus

Earth

.NonContentControl

Mars

Jupiter

Saturn

Uranus

Neptune

Pluto

Next, we use the GroupAdjacent extension method to group the .NonContentControls together:

// group by type

var q3 = q2.GroupAdjacent(e => e.KeyForGroupAdjacent);

// temporary code to dump q3

foreach (var g in q3)

Console.WriteLine("{0}: {1}", g.Key, g.Count());

Environment.Exit(0);

When we run this, we see:

.NonContentControl: 21

SolarOverview: 1

Sun: 1

Mercury: 1

Venus: 1

Earth: 1

.NonContentControl: 1

Mars: 1

Jupiter: 1

Saturn: 1

Uranus: 1

Neptune: 1

Pluto: 1

Next, the code validates that the .DOCX files referenced by the content controls exist:

// validate existence of files referenced in content controls

foreach (var f in q3.Where(g => g.Key != ".NonContentControl"))

{

string filename = f.Key + ".docx";

FileInfo fi = new FileInfo(filename);

if (!fi.Exists)

{

Console.WriteLine("{0} doesn't exist.", filename);

Environment.Exit(0);

}

Then, the code projects a collection of anonymous types that include the group, as well as the open WordprocessingDocument objects:

// project collection with opened WordProcessingDocument

var q4 = q3

.Select(g => new

{

Group = g,

Document = g.Key != ".NonContentControl" ?

WordprocessingDocument.Open(g.Key + ".docx", false) :

solarSystem

});

The observant will notice that opening these documents very definitely introduces state to this very not-pure query. We’ll need to close/dispose of those documents later. I’ve been fermenting an idea about wrappers around the Open XML SDK that give true functional composability to Open XML documents. This approach would eliminate this issue of classes that implement IDisposable. If when I open that bottle it hasn’t turned to vinegar, I’ll blog it.

Finally, we’re ready to project the list of OpenXml.PowerTools.Source objects:

// project collection of OpenXml.PowerTools.Source

var sources = q4

.Select(

g =>

{

if (g.Group.Key == ".NonContentControl")

return new Source(

g.Document,

g.Group

.First()

.Element

.ElementsBeforeSelf()

.Count(),

g.Group

.Count(),

false);

else

return new Source(g.Document, false);

}

).ToList();

Finally the code calls DocumentBuilder.BuildDocument and disposes of all of the opened WordprocessingDocument objects (except the ‘template’ document, which will be disposed when exiting scope of the using statement).

DocumentBuilder.BuildDocument(sources, "solar-system-new.docx");

// dispose of the opened WordprocessingDocument objects

foreach (var g in q4)

if (g.Group.Key != ".NonContentControl")

g.Document.Dispose();

The entire example, including the implementation of the GroupAdjacent extension method and the GetXDocument extension method follow. I’ve attached the source file and the sample documents to this post. This code works with version 1.1.1 of DocumentBuilder (and not prior versions). You can download DocProc.zip, which contains DocumentBuilder from http://www.CodePlex.com/PowerTools. It’s under the ‘Downloads’ tab.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.IO;

using System.Xml;

using System.Xml.Linq;

using DocumentFormat.OpenXml.Packaging;

using OpenXml.PowerTools;

public class GroupOfAdjacent<TSource, TKey> : IEnumerable<TSource>, IGrouping<TKey, TSource>

{

public TKey Key { get; set; }

private List<TSource> GroupList { get; set; }

System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()

{

return ((System.Collections.Generic.IEnumerable<TSource>)this).GetEnumerator();

}

System.Collections.Generic.IEnumerator<TSource> System.Collections.Generic.IEnumerable<TSource>.GetEnumerator()

{

foreach (var s in GroupList)

yield return s;

}

public GroupOfAdjacent(List<TSource> source, TKey key)

{

GroupList = source;

Key = key;

}

public static class LocalExtensions

{

public static XDocument GetXDocument(this OpenXmlPart part)

{

XDocument xdoc = part.Annotation<XDocument>();

if (xdoc != null)

return xdoc;

using (StreamReader streamReader =

new StreamReader(part.GetStream()))

xdoc = XDocument.Load(XmlReader.Create(streamReader));

part.AddAnnotation(xdoc);

return xdoc;

}

public static IEnumerable<IGrouping<TKey, TSource>> GroupAdjacent<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector)

{

TKey last = default(TKey);

bool haveLast = false;

List<TSource> list = new List<TSource>();

foreach (TSource s in source)

{

TKey k = keySelector(s);

if (haveLast)

{

if (!k.Equals(last))

{

yield return new GroupOfAdjacent<TSource, TKey>(list, last);

list = new List<TSource>();

list.Add(s);

last = k;

}

else

{

list.Add(s);

last = k;

}

else

{

list.Add(s);

last = k;

haveLast = true;

}

if (haveLast)

yield return new GroupOfAdjacent<TSource, TKey>(list, last);

}

class DocProc

{

static void Main(string[] args)

{

using (WordprocessingDocument solarSystem =

WordprocessingDocument.Open("solar-system.docx", false))

{

XNamespace w =

"http://schemas.openxmlformats.org/wordprocessingml/2006/main";

// get children elements of the <w:body> element

var q1 = solarSystem

.MainDocumentPart

.GetXDocument()

.Root

.Element(w + "body")

.Elements();

// project collection of tuples containing element and type

var q2 = q1

.Select(

e =>

{

string keyForGroupAdjacent = ".NonContentControl";

if (e.Name == w + "sdt")

keyForGroupAdjacent = e.Element(w + "sdtPr")

.Element(w + "tag")

.Attribute(w + "val")

.Value;

if (e.Name == w + "sectPr")

keyForGroupAdjacent = null;

return new

{

Element = e,

KeyForGroupAdjacent = keyForGroupAdjacent

};

}

).Where(e => e.KeyForGroupAdjacent != null);

// group by type

var q3 = q2.GroupAdjacent(e => e.KeyForGroupAdjacent);

// validate existence of files referenced in content controls

foreach (var f in q3.Where(g => g.Key != ".NonContentControl"))

{

string filename = f.Key + ".docx";

FileInfo fi = new FileInfo(filename);

if (!fi.Exists)

{

Console.WriteLine("{0} doesn't exist.", filename);

Environment.Exit(0);

}

// project collection with opened WordProcessingDocument

var q4 = q3

.Select(g => new

{

Group = g,

Document = g.Key != ".NonContentControl" ?

WordprocessingDocument.Open(g.Key + ".docx", false) :

solarSystem

});

// project collection of OpenXml.PowerTools.Source

var sources = q4

.Select(

g =>

{

if (g.Group.Key == ".NonContentControl")

return new Source(

g.Document,

g.Group

.First()

.Element

.ElementsBeforeSelf()

.Count(),

g.Group

.Count(),

false);

else

return new Source(g.Document, false);

}

).ToList();

DocumentBuilder.BuildDocument(sources, "solar-system-new.docx");

// dispose of the opened WordprocessingDocument objects

foreach (var g in q4)

if (g.Group.Key != ".NonContentControl")

g.Document.Dispose();

}

Why I Don’t Use the ForEach Extension Method

EricWhite — Wed, 08 Apr 2009 18:15:00 GMT

In my first version of my functional programming tutorial, I discussed a ForEach extension method, but I removed the topic from the second version. This extension method iterates through a source collection and performs an action on each item in the collection. ForEach isn’t included in the .NET base class library (BCL), but its implementation is pretty simple:

static class MyExtensions

{

static void ForEach<T>(this IEnumerable<T> source, Action<T> action)

{

foreach (T item in source)

action(item);

}

Perhaps the most common use of this extension method is to send output to the console:

static void Main(string[] args)

{

List<string> s = new List<string>() {

"abc",

"def",

"ghi"

};

s.ForEach(z => Console.WriteLine(z));

}

This extension method has been discussed in various blog posts. As my experience has grown in using LINQ and functional programming techniques, I’ve moved completely away from using it. This post explains why.

The common complaint about the ForEach extension method is that it encourages state mutation, which is contrary to the whole idea of stateless transformations. I’d go a step beyond, and say that the only possible use of the ForEach extension method is to perform actions that have side effects.

In pure functional programming, even simple outputting to the console is considered to be a side effect, the reason being that the compiler could in theory optimize your queries and perhaps parcel out portions of the query to different processors, and your nicely formatted report would become a jumbled mess. Currently, the C# compiler is deterministic – if you understand the laziness characteristics of your queries you can predict the order of execution. This possibly could change in the future, but this is really beside the point.

The reason to not use ForEach is that it blurs the boundary between pure functional code and state-full imperative code.

Over the last couple of years, I’ve used functional programming techniques in a variety of applications, including SharePoint features, ASP.NET applications, and Windows Forms applications. These frameworks are decidedly imperative in nature. However, I find it perfectly natural to combine pure functional code with traditional imperative code. But when I do so, I divide my code up so that the functional code is pure. If writing code for myself, I don’t bother to add comments in the code delineating the functional code from imperative code, but if writing code that others will maintain, I certainly comment on it. If I were to write production code that will be included in a product, I might separate the pure functional transformations into their own module, depending on the extent of the transformations.

In practice, I often write transforms to produce XML trees using LINQ to XML, even if my scenario doesn’t require XML. For example, when I wrote the code to automate testing of code that is embedded in documents, the pure functional transformation produces an XML tree that contains the ‘report’ of whether each unit test passed or failed. Then, code that is completely separate from the pure functional transform prints the report to the console. If I implemented that testing framework as a Windows Form application, I just as easily could have set text in controls in a form, or perhaps rows in a grid.

Alternatively, I write transformations that produce collections of strings, or collections of objects that have properties. In any case, I don’t combine pure functional code with imperative code within a single statement. I categorize outputting to the console as imperative code with side effects.

It could be possible to write an extension method that had the specific purpose of writing to the console, and that wouldn’t have such an ambiguous purpose as ForEach. However, it is easy instead to produce a collection of strings or whatever, completely separating the code that prints to the console. This separation gives me a warm and fuzzy feeling. J My personal opinion is that ForEach should not be included in the BCL.