Eric White's Blog : Functional Programming

Transforming Open XML Word-Processing Documents to Html (Post #3)

EricWhite — Sun, 25 Oct 2009 05:56:00 GMT

(Update Nov 11, 2009: This is the 3rd in a series of posts (#1, #2, #3, #4, #5, #6) on doing a transform of WordprocessingML to XHtml.)

Over the last couple of weeks, and over the next week, I've been designing and writing some code to convert Open XML word-processing documents to HTML (or Xhtml). My first post described in broad strokes my goals, my motivations for writing this code, and some details about the approach that I'm considering. My second post provided more detail about how I'll proceed, my first thoughts about my use of CSS, and specific limitations that I'll place on the conversion. I also presented my rational for not converting numbered/bulleted items to li elements. I also presented a skeleton of the conversion code. As I've been reading through the Open XML specification, more specifics about how I should proceed have become clear to me. In this post, I'm going to detail some of my conclusions.

First, here are some additional limitations that I'm going to apply to this conversion:

· I'm not going to attempt to convert documents that contain sub-documents. This almost certainly would not be one of the primary scenarios. The conversion will throw an exception if the document contains the w:subDoc element.

· There are a number of legacy elements that I might be able to ignore: w:dayLong, w:dayShort, w:monthShort, w:monthLong, w:yearLong, w:yearShort, w:pgNum. Conforming applications should not be writing out these elements. At some point in the near future, I'm going to write some code to crawl my collection of sample Open XML documents, and count how many documents contain these elements. This will help me decide whether to do the work to support these elements.

· I'm going to ignore the w:ruby (phonetic guide) element. This could be interesting, but I'll reserve this for a later version if it's important. If this is important to you, I'd be very appreciative if you'd let me know.

· For phase one of this project, I'm going to do only a rudimentary conversion of DrawingML, specifically to convert images in drawings. DrawingML contains very rich constructs. Doing all such transformations and generating appropriate images is in and of itself a complicated project. However, basic images are described in DrawingML, and we need to be able to generate web pages that contain appropriate references to basic images, so it's important to handle this aspect of DrawingML. I'll defer the high-fidelity conversion of all aspects of DrawingML to a later project.

· I'm going to ignore all w:object elements. Rendering w:object elements is certainly not a main-line scenario.

· For phase one, I'm not going to attempt to render MathML markup. This is interesting, but as with DrawingML, non-trivial. In the interest in getting something working in the next couple of weeks, I'm not going to include conversion of MathML in phase one.

· As I mentioned in last week's post, I'm not going to convert text separated by physical tabs in phase one. There is no clean way to approach this, so until the best approach is clear, I'm not going to convert them.

As I mentioned in the first post, I'm going to simplify the word-processing markup before transforming to HTML. Here are some of the ways that I'll simplify:

· I'll remove all rsid elements and attributes before doing the conversion to HTML.

· I'll remove all comments, end notes, and foot notes before doing the conversion.

· I'll coalesce superfluous runs – combine adjacent runs with identical formatting to a single run.

· And as I mentioned in the first post, I'll accept all tracked changes (tracked revisions) before doing the conversion.

Now that I've detailed what I won't convert, here is what I will convert:

· I'll convert all paragraphs and runs, including all text, formatted with the correct font, and with correct paragraph formatting, such as space before and after each paragraph. This includes honoring all style inheritance, as well as honoring all places where styles defer decisions on font and colors to themes.

· I'll convert all tables with a high degree of fidelity, including theme formatting, conditional formatting, and tables within tables. I believe I'll be able to correctly transform both horizontally and vertically merged cells.

· I'll convert all numbered/bulleted items to straight paragraphs, not li elements. I detailed my rational for this decision last week. I'm curious to see how this decision holds up in real-world situations.

· I'll convert all images, which are represented by DrawingML. This includes resizing, rotating, mirroring, and flipping images so that the resulting web page looks as close to the word-processing document as possible. By far, the most important of these is resizing.

· I'll render w:sectPr as a div element.

· I'll appropriately render the cr, noBreakHyphen, tab, and br elements.

There are two varieties of hyperlinks as defined in Open XML: hyperlinks described in field codes, and hyperlinks in the simplified version that uses an external reference. For simplicity, I'll convert all field code hyperlinks to the simplified version before doing the conversion to HTML. Other than hyperlink field codes, I'll remove all other field code markup, leaving the rendering markup. For example, markup for a typical field code looks like this:

<w:p>

<w:r>

<w:fldChar w:fldCharType="begin"/>

</w:r>

<w:r>

<w:instrText xml:space="preserve"> DATE </w:instrText>

</w:r>

<w:r>

<w:fldChar w:fldCharType="separate"/>

</w:r>

<w:r>

<w:rPr>

<w:noProof/>

</w:rPr>

<w:t>10/15/2009</w:t>

</w:r>

<w:r>

<w:fldChar w:fldCharType="end"/>

</w:r>

</w:p>

In the simplification process, I'll transform this markup to the following, which will be easy to render in HTML:

<w:p>

<w:r>

<w:t>10/15/2009</w:t>

</w:r>

</w:p>

Perhaps the most complicated thing to sort out is applying paragraph, table, numbering, and character styling per all the rules as laid out in the specification. For a single paragraph in a cell, there are properties that must be applied from a) document defaults, b) table formatting, including conditional table formatting, c) paragraph formatting, and d) run formatting. All of these properties must be applied in a very specific order. Further, fonts may be defined in themes, and if so, need to be rendered with those fonts as appropriate.

Section 2.7.2 of the Ecma 376 specification (17.7.2 of IS29500) has the following diagram indicating precedence of application of styles:

Here is a description of the process:

· First, the document defaults are applied to all runs and paragraphs in the document.

· Next, the table style properties are applied to each table in the document, following the conditional formatting inclusions and exclusions specified per table.

· Next, numbered item and paragraph properties are applied to each paragraph formatted with a numbering style.

· Next, paragraph and run properties are applied to each 1 paragraph as defined by the paragraph style.

· Next, run properties are applied to each run with a specific character style applied.

· Finally, we apply direct formatting (paragraph or run properties not from styles). If this direct formatting includes numbering, that numbering + the associated paragraph properties are applied.

I could write the code so that as I process each run, I go back through all locations where there are elements that might impact styling, and collect that information, coalesce that information, and then render the HTML. This might be the most efficient way to proceed in terms of processing time and memory consumption. However, it means that a specific rule for styling as dictated by the Open XML specification will be rendered in code that is scattered throughout the conversion code, which is not the easiest to debug.

Instead, I've decided on an approach where I process markup per the rules in section Ecma 2.7.2 (IS29500 17.7.2) of the specification, as well as the rules in 2.7.5 (IS29500 17.7.5), and add all styling information to every paragraph and run in the document. As I add this styling information, all add my own custom attributes that tell the order that this styling information must be applied. I'll also add an attribute that describes where that styling information came from. To make this clear, suppose I have a document that has the following document defaults that define the paragraph and run properties that should be applied to all paragraphs and runs in the document:

<w:docDefaults>

<w:rPrDefault>

<w:rPr>

<w:rFonts w:asciiTheme="minorHAnsi"

w:eastAsiaTheme="minorHAnsi"

w:hAnsiTheme="minorHAnsi"

w:cstheme="minorBidi"/>

<w:sz w:val="22"/>

<w:szCs w:val="22"/>

<w:lang w:val="en-US"

w:eastAsia="en-US"

w:bidi="ar-SA"/>

</w:rPr>

</w:rPrDefault>

<w:pPrDefault>

<w:pPr>

<w:spacing w:after="200"

w:line="276"

w:lineRule="auto"/>

</w:pPr>

</w:pPrDefault>

</w:docDefaults>

And suppose I have a paragraph with paragraph properties, that contains a run that has run properties:

<w:p>

<w:pPr>

<w:spacing w:after="0"/>

</w:pPr>

<w:r>

<w:t xml:space="preserve">This </w:t>

</w:r>

<w:r>

<w:rPr>

<w:rFonts w:ascii="Courier New"

w:hAnsi="Courier New"

w:cs="Courier New"/>

<w:b/>

</w:rPr>

<w:t>is</w:t>

</w:r>

<w:r>

<w:t xml:space="preserve"> a test.</w:t>

</w:r>

</w:p>

So before doing the final transform to HTML, I'll transform the paragraph and run to contain properties both from the document defaults, and properties that were directly applied:

<w:p>

<w:pPr

ptoxml:Order="0"

ptoxml:Source="document defaults: style part: /w:styles/w:docDefaults/w:pPrDefault/w:pPr">

<w:spacing w:after="200"

w:line="276"

w:lineRule="auto"/>

</w:pPr>

<w:pPr>

<w:spacing w:after="0"/>

</w:pPr>

<w:r>

<w:t xml:space="preserve">This </w:t>

</w:r>

<w:r>

<w:rPr

ptoxml:Order="0"

ptoxml:Source="document defaults: style part: /w:styles/w:docDefaults/w:rPrDefault/w:rPr">

<w:rFonts w:asciiTheme="minorHAnsi"

w:eastAsiaTheme="minorHAnsi"

w:hAnsiTheme="minorHAnsi"

w:cstheme="minorBidi"/>

<w:sz w:val="22"/>

<w:szCs w:val="22"/>

<w:lang w:val="en-US"

w:eastAsia="en-US"

w:bidi="ar-SA"/>

</w:rPr>

<w:rPr>

<w:rFonts w:ascii="Courier New"

w:hAnsi="Courier New"

w:cs="Courier New"/>

<w:b/>

</w:rPr>

<w:t>is</w:t>

</w:r>

<w:r>

<w:t xml:space="preserve"> a test.</w:t>

</w:r>

</w:p>

This means that I'll be creating an intermediate XML document that is larger than the original – memory use will be somewhat higher. But there are two important advantages of this approach. First, the code to deal with each area of styling markup will be localized. Second, the code will be significantly easier to debug.

For example, there will be only a few lines of code, located in one place, to deal with document default properties (Ecma 2.7.4, IS29500 17.7.4). This code will duplicate the default paragraph properties on every paragraph in the document, and will duplicate the default run properties on every run in the document. Then, I'll process the table styling properties in a similar fashion: I'll duplicate the paragraph and run properties, placing them on every paragraph and run in the table. Then I'll process paragraph and run properties, per the diagram above. As I add each of these paragraph and run properties to their respective locations, I'll assign an order of application to them. When it comes time to render the paragraphs and runs in HTML, I can simply roll up all paragraph and run properties per their assigned order, coalesce them, and then render the HTML. If I'm not getting the results I want, then debugging becomes a simple matter of looking at the paragraph and run properties before coalescing, and seeing where I went wrong in assembling the pile of formatting properties.

One key advantage of this approach is that the code and its algorithm will have a direct correlation to the text of the specification. The code should be relatively simple to read. One interesting point to note is that with this approach, it will take extra work to generate CSS classes for paragraph and character styles. It would be easiest to generate the CSS as in-line only, forgoing generation of classes. This is the approach that I'm going to take first. Then, I'll take a look at how much work it would be to generate CSS styles, and determine if the payoff is worth the cost. As I mentioned in my last post on this topic, my primary goal is to provide a reasonably accurate conversion for use in certain developer scenarios. It isn't a primary goal to provide perfect separation between content and presentation, which would be the primary advantage of generation of CSS styles.

My next work item is to write the code to propagate formatting properties per this approach. Following that, I'll write the code to roll up (coalesce) the formatting properties. And finally, I can write the code to do the transform to HTML (or XHtml).

Transforming Open XML Word-Processing Documents to XHtml (Post #2)

EricWhite — Wed, 21 Oct 2009 14:46:00 GMT

[Blog Map]

(Update Nov 11, 2009: This is the 2nd in a series of posts (#1, #2, #3, #4, #5, #6) on doing a transform of WordprocessingML to XHtml.)

Last week, I blogged about a small project that I'm embarking on: to make a reasonably accurate transform from Open XML word-processing markup to XHTML. I wrote about the approach that I'll be taking, and my initial thoughts about how to proceed. I've done a bit of research, and this week, I'll lay out more details about the approach that I'll take.

One small note about this series of blog posts – these are going to be much more ad-hoc than my usual posts. If I go down the wrong path, then you'll see this J. Also, I'm not going to spend too much time writing and re-writing the posts.

One of the key aspects of the approach that I'll take is to use the power of CSS:

I'll generate a CSS style for every block-level style that is used in the word-processing document. I'll try, as far as possible, to generate the appropriate CSS style that will render the word-processing document accurately. These styles will be applied to p and div elements.
I'll generate a CSS style for in-line styles. These styles will be applied to span elements.
I'll preface the generated style names with Ptoxml (PowerTools for Open XML), i.e. PtoxmlNormal, PtoxmlH1, etc. If this generated markup is embedded in other HTML, this will prevent class name collisions.
The classes for block-level and in-line styles will be generated in an internal style sheet.
If there is direct formatting applied to a paragraph or to a run within a paragraph, I'll generate the appropriate CSS as an in-line style. My goal here is not to generate a document where the content is perfectly separated from the presentation. Instead, my goal is to provide a conversion of a small chunk of word-processing markup to usable XHTML that can be used programmatically in a variety of contexts. Open XML word-processing markup has 'cascading' semantics – a paragraph can be of a specific style, and the user can override aspects of that style for a paragraph. This is a direct parallel to the semantics of CSS – a paragraph can be of a specific class, and can be overridden for a paragraph.

One key aspect of the approach that I'm going to take: I am not going to translate numbered/bulleted items from word-processing markup to li elements in the Xhtml. Instead, I'm going to generate paragraphs of a particular class, and format that class using CSS as appropriate, so that numbered items and bulleted lists are rendered properly. While numbered items that are formatted in a simple way translate to li elements in the Xhtml markup, the capabilities of numbered items in word-processing markup are rich (RICH!), and as soon as the markup uses more than the most rudimentary capabilities, the translation breaks down. This has been one of the biggest complaints about other projects that convert Open XML to html – that numbered items aren't translated properly. I could go down the road of translating rudimentary numbered items to li elements, and then translate the more rich variations into paragraphs, but this is messy. Instead, I believe that I'm going to discard using li elements altogether.

As I've researched how I'll implement this, I've decided on a few limitations:

Multi-column layout will be converted to single column layout. We're more concerned about accurately surfacing the content than exact representation.
Themes will be converted to straight CSS styles, both at the class level, and where over-ridden, at the in-line level. The abstraction of themes won't be carried over to the XHTML.
In cases where there is no direct correspondence between CSS styles and the specific representation in word-processing markup, I'll simplify the representation to whatever can be represented in CSS. For instance, there are a lot of varieties of underline styles in word-processing markup. All underline styles will be transformed to a simple underline in the generated CSS and XHTML.
Open XML word-processing markup has an abstraction of tabs – you can display the ruler above your document, below the ribbon, and insert tabs, then place tabs in your document. There is no corresponding abstraction in CSS/XHTML. This could be approximated using spaces, but at best, it will be inaccurate – text won't align properly vertically. My personal experience is that these days, people prefer tables for laying out text instead of tabs, and tables do translate properly. I think that for phase 1, I'll not attempt any sort of hacked conversion, but I've not yet decided on how to convert these. It could be neat to convert tabbed text to tables with invisible grids with merged cells, but I'm not sure how this would work practically. One problem is that numbered/bulleted items in word-processing markup makes use of physical tabs – if I don't have a way to render tabs, there will be a small loss in fidelity of placement of text of numbered items.

I'm sure that I'll discover other places where I will want to place limits on the transform.

The last thing I'll present in this post is the skeleton for the conversion. The following code will do a simplistic transform of simple Open XML documents to simple XHTML. I can then build and extend this code, handling more and more sophisticated varieties of markup. For a detailed explanation of how this type of transform works, see the post, Recursive Pure Functional Transforms of XML.

using System;

using System.Collections.Generic;

using System.IO;

using System.Linq;

using System.Text;

using System.Xml;

using System.Xml.Linq;

using DocumentFormat.OpenXml.Packaging;

namespace HtmlConverter

{

public static class Extensions

{

public static XDocument GetXDocument(this OpenXmlPart part)

{

XDocument partXDocument = part.Annotation<XDocument>();

if (partXDocument != null)

return partXDocument;

using (Stream partStream = part.GetStream())

using (XmlReader partXmlReader = XmlReader.Create(partStream))

partXDocument = XDocument.Load(partXmlReader);

part.AddAnnotation(partXDocument);

return partXDocument;

}

public static string StringConcatenate(this IEnumerable<string> source)

{

StringBuilder sb = new StringBuilder();

foreach (string s in source)

sb.Append(s);

return sb.ToString();

}

public static class W

{

public static XNamespace w =

"http://schemas.openxmlformats.org/wordprocessingml/2006/main";

public static XName body = w + "body";

public static XName document = w + "document";

public static XName p = w + "p";

public static XName pPr = w + "pPr";

public static XName r = w + "r";

public static XName rPr = w + "rPr";

public static XName t = w + "t";

public static XName tbl = w + "tbl";

public static XName tc = w + "tc";

public static XName tr = w + "tr";

public static XName txbxContent = w + "txbxContent";

public static XName val = w + "val";

public static XName pStyle = w + "pStyle";

public static XName b = w + "b";

}

public static class Xhtml

{

public static XNamespace xhtml = "http://www.w3.org/1999/xhtml";

public static XName html = xhtml + "html";

public static XName head = xhtml + "head";

public static XName title = xhtml + "title";

public static XName body = xhtml + "body";

public static XName p = xhtml + "p";

public static XName h1 = xhtml + "h1";

public static XName h2 = xhtml + "h2";

public static XName A = xhtml + "A";

public static XName href = "href";

public static XName b = xhtml + "b";

public static XName table = xhtml + "table";

public static XName border = "border";

public static XName tr = xhtml + "tr";

public static XName td = xhtml + "td";

}

public static class HtmlConverter

{

public static object ConvertToHtmlTransform(WordprocessingDocument wordDoc,

XNode node)

{

XElement element = node as XElement;

if (element != null)

{

if (element.Name == W.document)

return new XElement(Xhtml.html,

new XElement(Xhtml.head,

new XElement(Xhtml.title, "Test.docx")

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e))

);

// transform the w:body element to the XHTML h:body element

if (element.Name == W.body)

return new XElement(Xhtml.body,

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// transform every Heading1 styled paragraph to the XHTML h:h1 element

if (element.Name == W.p && (string)element

.Elements(W.pPr)

.Elements(W.pStyle)

.Attributes(W.val)

.FirstOrDefault() == "Heading1")

return new XElement(Xhtml.h1,

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// transform every Heading2 styled paragraph to the XHTML h:h2 element

if (element.Name == W.p && (string)element

.Elements(W.pPr)

.Elements(W.pStyle)

.Attributes(W.val)

.FirstOrDefault() == "Heading2")

return new XElement(Xhtml.h2,

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// transform w:p to h:p

if (element.Name == W.p)

return new XElement(Xhtml.p,

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// transform every text run that is styled as bold to the XHTML h:b element

if (element.Name == W.r &&

element.Elements(W.rPr).Elements(W.b).Any())

return new XElement(Xhtml.b,

element.Elements(W.t).Select(e => (string)e).StringConcatenate());

// transform every text run that is not styled as bold to a text node that

// contains the text of the paragraph.

if (element.Name == W.r &&

!element.Elements(W.rPr).Elements(W.b).Any())

return new XText(element.Elements(W.t)

.Select(e => (string)e).StringConcatenate());

// transform w:tbl to h:tbl

if (element.Name == W.tbl)

return new XElement(Xhtml.table,

new XAttribute(Xhtml.border, 1),

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// transform w:tr to h:tr

if (element.Name == W.tr)

return new XElement(Xhtml.tr,

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// transform w:tc to h:td

if (element.Name == W.tc)

return new XElement(Xhtml.td,

element.Elements().Select(e => ConvertToHtmlTransform(wordDoc, e)));

// the following removes any nodes that haven't been transformed

return null;

}

return null;

}

public static XElement ConvertToHtml(WordprocessingDocument wordDoc)

{

// TODO WE REALLY WANT TO DO THIS ON BLOCK LEVEL CONTENT, NOT JUST CHILD ELEMENTS

// OF THE BODY ELEMENT

return ConvertToHtml(wordDoc, wordDoc

.MainDocumentPart

.GetXDocument()

.Element(W.document)

.Element(W.body)

.Elements());

}

public static XElement ConvertToHtml(WordprocessingDocument wordDoc,

IEnumerable<XElement> blockLevelContent)

{

if (blockLevelContent == null)

throw new ArgumentException("blockLevelContent argument is null");

XElement firstBlockLevelElement = blockLevelContent.FirstOrDefault();

if (firstBlockLevelElement == null)

throw new ArgumentException("blockLevelContent sequence is empty");

XDocument doc = firstBlockLevelElement.Document;

XElement xhtml = (XElement)ConvertToHtmlTransform(wordDoc, doc.Root);

return xhtml;

}

class Program

{

static void Main(string[] args)

{

string fileName = "Test.docx";

FileInfo fi = new FileInfo(fileName);

string baseName = fi.Name.Substring(0, fi.Name.Length - 4);

string newFileName = baseName + "-Copy.docx";

File.Copy(fileName, newFileName);

using (WordprocessingDocument doc =

WordprocessingDocument.Open(newFileName, true))

{

XElement html = HtmlConverter.ConvertToHtml(doc);

html.Save("Test.html");

}

Transforming Open XML Word-Processing Documents to XHtml

EricWhite — Mon, 12 Oct 2009 05:01:00 GMT

[Blog Map]

(Update Nov 11, 2009: This is the 1st in a series of posts (#1, #2, #3, #4, #5, #6) on doing a transform of WordprocessingML to XHtml.)

Over the next couple of weeks, I'm going to spend some time writing some LINQ to XML code to transform pen XML word-processing documents to XHtml. Just for fun, as I go, I'm going to post my progress, posting the code, talking about the issues I come across, and in general, being transparent about this development process. I welcome your thoughts and opinions. And shortly, we'll have a useful chunk of code that we can use in a variety of cool scenarios.

This code will be part of the PowerTools for Open XML project, so will be released under the Ms-PL license, so I'll be posting zip files there.

A few example scenarios:

Convert word docs to Html, populate a SharePoint wiki.
Find some text in a word doc, grab the three paragraphs before and after, change the formatting of the Open XML text that we found, and convert the chunk to Html, then display the results, with the found text highlighted in some fashion.
Build a specialized Html converter for my blog, which puts a 'Copy Code' button above each code snippet.

So, as I start, here are some thoughts and ideas I have about this project:

I'll try to transform to XHTML that is validated against the schema, unless I run into blocking issues.

There already is code plus an XSLT style sheet that can convert Open XML word-processing docs to HTML. This is the CodePlex/OpenXmlViewer project. I have different goals from that project – that project aims for high fidelity (the resulting HTML looks as close as possible to the original word-processing document), and is (I think) primarily used as a browser plug-in. My goals – less effort spent on full fidelity, and more on making this easy and convenient for developers to modify and enhance for specific development efforts.

Also, I want to be able to convert a small selected chunk of a word-processing document, whereas the OpenXmlViewer project converts entire documents.

Finally, I want a chunk of code that is super-easy for a C# developer to customize and incorporate into another application.

I'm going to write this code as a pure functional transform that uses recursion. After a fairly long selection process, I've settled on this approach for a variety of reasons. I don't want to use XSLT, as I'm going to add extension points where developers can interject their own custom transformations for specific pieces. For example, a developer can provide a lambda expression (delegate) for images – the lambda gets an image as an argument – you can do what you want with the image – post it on a server, or whatever, and then return the link to the transform. This will give wide latitude in how you deal with images.

I've discarded the approach of using annotations for doing document-centric transforms, as it has performance issues when used for large documents. (Actually, I'm not completely sure about this, but I've had the sense of this as I've written various transforms on larger documents.) In contrast, pure functional recursive transforms are blindingly fast. The code that I wrote in the recursive style to accept revisions does no less than seven successive transformations, producing entirely new trees, and the code is fast. On a Dell D600, 2Ghz, single core, it processes extremely large documents (800 pages) in less than a second.

The disadvantage of using recursive LINQ transforms is that not too many developers are comfortable with this style of development. However, if I do this properly, developers won't need to plumb the depths of the transform, and instead can use it as a 'black box'. Besides, if I make this process transparent, maybe more developers will understand the power of this approach.

One key aspect of the approach I'll take: I'll accept all tracked revisions before doing the conversion. This will make my code much simpler to write. The resulting code will be more robust. I've been postponing writing the HTML converter until I had a revision accepter that I am satisfied with.

I'm also considering doing an initial transformation to the simplest word-processing markup possible. For example, if there are two adjacent runs that have the same formatting, I can combine them into a single run. I'll also discard superfluous markup, such as proofing errors. If I simplify the markup, then there are more possibilities for straight one-to-one conversions between the Open XML markup and HTML.

I think that it would be useful to preserve bookmarks and internal links, and construct the corresponding markup in HTML.

For this initial version, I'm going to discard comments. It could be interesting to build a conversion that surfaces comments, but this isn't one of the main scenarios. In most cases, we don't want comments placed into the resulting HTML, I think.

I probably will also discard footnotes and endnotes in the resulting HTML. These are interesting, but probably only to a small subset of developers. If there is a lot of demand for this, then I can enhance the code later to incorporate these conversions. But I'd have to decide how I would want them to be rendered in HTML, and this is a more complicated decision.

I want this code to have the highest fidelity that I can accomplish without jumping through too many hoops. Key goals – preservation of textual content – if there's text in the source document, the text shows up in the same place, with the same font, in the resulting HTML. Images should convert and show up in the same place, and if there is an easy way to make the text flow in the same fashion as the source document, the HTML will do so.

Because I'm writing this in the pure functional recursive style, you can almost prove that a) the code can't fail, because I'll make every effort to reduce 'points of possible failure', and b) the code can only produce valid XHtml. This adds robustness and reliability to applications that use this code.

Finally, I want to write this transform in the smallest amount of code possible. My off-the-cuff estimate is that the conversion should be 1000 lines of code or less. But we'll see how large the code becomes as I progress.

Anyway, on to more research and coding. I'll post the next update in about a week. This is going to be fun!

DescendantsAndSelfTrimmed LINQ to XML Axis Method

EricWhite — Wed, 19 Aug 2009 22:01:00 GMT

[Blog Map]

There are some circumstances where I need a variation on the DescendantsAndSelf axis method that allows me to specify that specific elements (and the descendants of those elements) are ‘trimmed’ from the returned collection. One of the things that's great about LINQ to XML is the ease with which we can create specialized axis methods when necessary.

Here is the scenario: Open XML contains paragraphs that contain runs. A run can contain a picture that contains a text box, which itself contains a paragraph. When I want to write a transform for paragraphs that are children elements of the w:body element, I want to disregard any paragraphs that are children of contained text boxes.

To make this clear, consider the following markup:

<body>

<r>

<t>First paragraph.</t>

</r>

<r>

<pict>

<shape>

<r>

<t>Text in a textbox.</t>

</r>

</textbox>

</shape>

</pict>

</r>

<r>

<t>Second paragraph.</t>

</r>

</body>

</document>

In this markup, I want to transform the elements highlighted in yellow, but while processing those elements, I want to completely disregard the element highlighted in green. If children paragraphs of the textbox element need to be transformed in some fashion, I would deal with them separately – perhaps in an embedded recursive transform.

To enable selecting descendant elements in this fashion, I’ve implemented four extension methods. These are variations on the Descendants and DescendantsAndSelf axis methods. The methods are:

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

XName trimName)

Returns a filtered collection of elements that contain this element, and all descendant elements of this element, in document order. Elements that have a matching XName are not included in the collection. Descendant elements of excluded elements are not included in the collection.

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

Func<XElement, bool> predicate)

Returns a filtered collection of elements that contain this element, and all descendant elements of this element, in document order. Elements that are selected by the predicate are not included in the collection. Descendant elements of excluded elements are not included in the collection.

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

XName trimName)

Returns a filtered collection of the descendant elements for this document or element, in document order. Elements that have a matching XName are not included in the collection. Descendant elements of excluded elements are not included in the collection.

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

Func<XElement, bool> predicate)

Returns a filtered collection of elements that contain this element, and all descendant elements of this element, in document order. Elements that are selected by the predicate are not included in the collection. Descendant elements of excluded elements are not included in the collection.

Here is the listing of these extension methods, as well as some example code to exercise them (also attached):

using System;

using System.Collections.Generic;

using System.Linq;

using System.Xml.Linq;

public static class LocalExtensions

{

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

XName trimName)

{

return DescendantsInternalTrimmed(element, e => e.Name == trimName, true);

}

public static IEnumerable<XElement> DescendantsAndSelfTrimmed(this XElement element,

Func<XElement, bool> predicate)

{

return DescendantsInternalTrimmed(element, predicate, true);

}

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

XName trimName)

{

return DescendantsInternalTrimmed(container, e => e.Name == trimName, false);

}

public static IEnumerable<XElement> DescendantsTrimmed(this XContainer container,

Func<XElement, bool> predicate)

{

return DescendantsInternalTrimmed(container, predicate, false);

}

private static IEnumerable<XElement> DescendantsInternalTrimmed(this XContainer container,

Func<XElement, bool> predicate, bool includeSelf)

{

if (includeSelf)

{

XElement element = container as XElement;

yield return element;

}

Stack<IEnumerator<XElement>> iteratorStack = new Stack<IEnumerator<XElement>>();

iteratorStack.Push(container.Elements().GetEnumerator());

while (iteratorStack.Count > 0)

{

while (iteratorStack.Peek().MoveNext())

{

XElement currentXElement = iteratorStack.Peek().Current;

if (predicate(currentXElement))

continue;

yield return currentXElement;

iteratorStack.Push(currentXElement.Elements().GetEnumerator());

}

iteratorStack.Pop();

}

class Program

{

static void Main(string[] args)

{

XElement rootElement = XElement.Parse(

@"<document>

<body>

<r>

<t>First paragraph.</t>

</r>

<r>

<pict>

<shape>

<r>

<t>Text in a textbox.</t>

</r>

</textbox>

</shape>

</pict>

</r>

<r>

<t>Second paragraph.</t>

</r>

</body>

</document>");

foreach (var item in rootElement.DescendantsAndSelfTrimmed("textbox"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

foreach (var item in rootElement

.DescendantsAndSelfTrimmed(e => e.Name == "p" && e.Ancestors("p").Any()))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

foreach (var item in rootElement.DescendantsTrimmed("r"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

foreach (var item in rootElement.DescendantsTrimmed(e => e.Name == "r"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

Console.WriteLine("==========");

XDocument doc = XDocument.Parse(

@"<document>

<body>

<r>

<t>First paragraph.</t>

</r>

<r>

<pict>

<shape>

<r>

<t>Text in a textbox.</t>

</r>

</textbox>

</shape>

</pict>

</r>

<r>

<t>Second paragraph.</t>

</r>

</body>

</document>");

foreach (var item in doc.DescendantsTrimmed(e => e.Name == "r"))

Console.WriteLine("{0}{1}", "".PadRight(item.Ancestors().Count() * 2),

item.Name);

}

Update to CommentMerger.MergeComments Method

EricWhite — Tue, 11 Aug 2009 13:58:00 GMT

Word 2007 has a feature where you can lock a document, preventing any changes to content, yet allowing the user to add comments. CommentMerger is a small method (~400 lines of code) that merges comments from two documents into a single document, provided the two contain the same content. This code is part of the PowerTools for Open XML project. In the future, I'm going to write a cmdlet that will use this code to merge comments.

I’ve updated the CommentMerger class (actually completely rewritten it). The new code (again, written in a functional style) is much more robust. It now merges comments on math formulas. The code (and example program to use it) is available in the ‘downloads’ tab at codeplex.com/powertools.

There have been two good side effects of this effort – I’ve learned more about the Open XML formats, and I’ve refined my approach for doing recursive pure functional transforms.

One of the biggest differences in the code is the approach that I took to selecting elements for transform. Previously, I was looking for paragraph nodes, then recursively processing the descendants. But this became more complicated in certain scenarios – paragraphs can contain runs which contain other content which can contain paragraphs. In the new version, I simply select the elements for transform based on their contents. For example, the following code selects a node for transform based on if the node contains any child w:commentRangeStart, w:commentRangeEnd, or runs (w:r) that contain w:commentReference elements:

// If fromElement contains comments, then do special clone that merges comments.

if (fromElement

.Elements()

.Where(e => e.Name == W.commentRangeStart ||

e.Name == W.commentRangeEnd ||

((e.Name == W.r || e.Name == M.r) &&

e.Elements(W.commentReference).Any()

))

.Any())

{

// Get new, cloned element with merged comments.

XElement newToElement = CloneElementWithComments(toElement, fromElement,

commentIdTransformDictionary);

return newToElement;

}

I used this approach throughout the transform. I had a question about whether this approach would perform well enough. I believe it does for two reasons – first, the Any extension method ‘short-circuits’ any positive matches, reducing the work to be done. Second, I think that CPU caching may be effective in these situations. In any case, the code seems to be more than fast enough.

Merging Comments from Multiple Open XML Documents into a Single Document

EricWhite — Tue, 28 Jul 2009 02:14:00 GMT

Microsoft Word 2007 allows you to lock a document, prohibiting users from making changes to content, while allowing them to add comments. If we have multiple documents that have the same content yet different comments, we can merge those comments into a single document. One possible use would be a specification review system. After the specification writer finishes a specification, she could send it to other members of her team for review. As each reviewer returns the specification, she could merge all comments into a single document, making it simpler to integrate those comments. (This example was inspired by an email thread with Sergey Solyanik, and his need for a comment merger for his very cool code (and soon-to-be spec) review system. Also, need to say thanks, Sergey did a code review, and the code is better for it. J) The code to do comment merging is available in a zip file named CommentMerger.zip in the downloads tab at www.codeplex.com/powertools.

Note: The CommentMerger class is part of the Power Tools for Open XML project. In the future, I’m going to build a new PowerShell cmdlet to do comment merging. PowerTools for Open XML is an open source project on CodePlex that makes it easy to create and modify Open XML documents using PowerShell scripts. It’s important to note that Power Tools for Open XML is not a supported Microsoft product and doesn’t necessarily represent future product direction. We think it will serve as inspiration for customers who need to create and modify Open XML documents programmatically. Power Tools for Open XML is published under the Microsoft public license (Ms-PL), which gives you wide latitude in how you use the code.

The CommentMerger.MergeComments method uses the Open XML SDK. The use of the comment merger class is pretty simple: You call the method passing two open WordprocessingDocument objects:

using (WordprocessingDocument destinationDocument =

WordprocessingDocument.Open("Test1a.docx", true))

using (WordprocessingDocument sourceDocument =

WordprocessingDocument.Open("Test1b.docx", false))

{

CommentMerger.MergeComments(destinationDocument, sourceDocument);

}

Upon return, the comments in the source document are merged into the destination document. To merge comments from multiple documents, you can call the function multiple times.

I wrote the CommentMerger.MergeComments method in the pure functional style. All methods are written without side-effects. After initializing, no variables are mutated. For a detailed explanation of this approach, see Recursive Approach to Pure Functional Transformations of XML.

The CommentMerger.MergeComments method is an example of a ‘Common-vocabulary document-centric transform’. For an overview of these types of transforms, see Document-Centric Transforms using LINQ to XML.

Before merging comments, the CommentMerger.MergeComments method validates that the two documents contain the same content. For more info, see Comparing Two Open XML Documents using the Zip Extension Method.

This code is based on the code I presented in Splitting Runs in Open XML Word Processing Document Paragraphs.

This code pre-atomizes XName objects. See A More Robust Approach for Handling XName Objects in LINQ to XML.

The code uses the Open XML SDK.