Antonio Zamora from
Staff DotNet has
posted something pretty cool - firing off a PowerShell script from C# to apply a consistent style to multiple Open XML documents. This is an interesting way to take advantage of the
PowerTools for Open XML.
The response to my previous blog post has been very interesting to me. And it has, to a very large extent, matched my own experience.
I have seen four basic scenarios where folks use LINQ:
-
Using LINQ to Objects (and LINQ to XML, which is really just LINQ to Objects), primarily using the extension methods, and the so-called ‘method syntax’. This really is a great scenario. When I really understood it, it blew my mind. I believe it leads to the possibility of “speed-reading” code – see below.
-
Using query expressions (query comprehensions to the FP crowd) to access objects and XML. I have mentioned in previous posts that I have naturally gravitated away from these. Why? They are not really necessary, and I find that I often have to go outside of them. I’ve thought of coding idioms to deal with this in a better way than parenthesizing query expressions, and ‘dotting’ into the extension methods I need to perform my desired transformation, but I think that ‘method syntax’ reads better. In any case, query expressions are syntactic sugar, and when you get really comfortable with lambda expression syntax, to me, it’s just clearer.
-
Using LINQ to SQL or LINQ to Entities to access a database. I haven’t used these much. I’ve been in XML land for quite a while. (I’m a database / accounting application / management information system developer from way back. When you live in the boondocks in Colorado, as I have much of my life, this is what pays the bills, so this is what you do. I’ve designed more databases and written more CRUD (Create, Read, Update, Delete) applications than I care to count.)
-
Using LINQ implemented using an IQueryable provider to something other than a SQL database. I’ve never personally used one, but feel it has huge potential. I think it depends on the quality of the implementation.
Speed Reading Code
This is one of the random benefits that I think you gain from using the FP style of code. One of the techniques that is possible, I believe, is that after becoming fully competent/clear/conversant with C# 3.0 syntax, is that it is possible to ‘speed read’ code. You can look at the code with an eye towards identifying the relevant transformations, and see large blocks of code as a single unit.
When I go back to code that I wrote two years ago, and can read my own code quickly, I appreciate it. Hey, when you are as old as me, the jokes about meeting new friends every day because you can’t remember them from yesterday are not as funny. :D
Anyway, the responses to the previous post mirror my own experience.
One thought I have about the responses – there is only a small percentage of negative responses. But I would bet that many who are not using LINQ haven’t left their comments. This is the expected outcome of an extremely biased poll like the one I posed. If you are not using LINQ, please comment!
In any case, I feel that there are enough folks who do get it, and who are using LINQ in its intended, most effective form, that the transition to a new FP main-stream world has begun. Slower than I’d hope, but probably not slower than I’d expect.
I had an interesting conversation with my nephew the other day. He is a very bright CS student working as a summer intern at a software company (not Microsoft). He is programming in C# using Visual Studio 2008. I asked him if developers at his company were using LINQ, and he said, "No, that the folks in charge had basically forbidden it, because no one understands it." Visual Studio 2008 has been out for over a year, but I know that a fair number of competent developers have not devoted the time and effort to learn about the style of programming that LINQ enables.
It IS worth it to spend the time to learn about LINQ. From comments I have received, I know that it takes only about eight to sixteen hours to work through my Functional Programming Tutorial. This is a far smaller effort than even a single CS course at a university. I know that I have saved that amount of time many times over in the last two years. I write programs faster, and they work more reliably. I have another blog post in mind that details exactly why programs written using LINQ have fewer bugs. I'll get to it pretty soon.
From the conversations I've had, I also know that there are a fair number who do get it. I receive comments from "typical" object-oriented developers who have read my Functional Programming tutorial, and have made the transition to the functional style of coding. While it is a bit of a transition to think about programming in terms of transformations instead of algorithms, it isn't a difficult concept. I think that this is analogous to the transition that we all made (if you are old, like me) to object oriented programming in the mid 80s.
It is possible to make the transition; I did so. And there is a high return on investment for the time spent. If you haven't done so yet, carve out a weekend, and work through the tutorial.
Some of the developers that I've spoken to have said that the functional programming course that they had to take when getting their CS degree was their least favorite course. They felt that the concepts were difficult to understand, and they had to write using a syntax that they felt was strange. Functional programming using C# 3.0 is a whole different animal from more "traditional" functional languages, such as Lisp, Haskell, Erlang, or Ocaml. It really is much easier using C#. The only really new syntax in C# is that of lambda expressions.
I'd like to take an informal poll about this. Do developers at your company understand and use LINQ? Do you? Can you see the benefit? Please leave a comment on this post, and let me (and others) know about the use of LINQ in your development efforts.
This post presents a custom application page in SharePoint that uses Open XML, the Open XML SDK and LINQ to XML to accept revisions, remove comments, and remove personal information from an Open XML word processing document.
The following 45 second screen-cast demonstrates the code presented in this post.
Video: Using Open XML with SharePoint
An approach that has interesting possibilities would be to create SharePoint workflows that query and modify Open XML documents. For example, you could write some code that would ensure that no documents in a SharePoint document library have comments, revisions, or personal information. I’ll present this code in the future.
The most interesting characteristic of this code is that until Open XML, it would be very difficult or impossible to implement reliably on a SharePoint server. If you wanted to implement this for binary office files, it would require using a library for accessing those binary documents, and that library may not be suitable for deployment on a SharePoint server. I have heard of people installing a copy of Office on their server. This has huge performance implications; in addition, it is a violation of the 2007 Office system license agreement. But the approach presented in this post is clean, performs well, and has no adverse licensing implications.
To make my development as easy as possible, I created a class, OpenXmlInfo, which has methods to query and modify the Open XML document. I developed the class using a console application. Then, when the class was coded and debugged, it was a simple matter to use it in the SharePoint custom application page. The class contains six static methods; three to query a document, and three to modify a document. Here are the signatures of the methods:
public static bool InspectForComments(WordprocessingDocument document)
public static bool InspectForRevisions(WordprocessingDocument doc)
public static bool InspectForPersonalInfo(WordprocessingDocument document)
public static void RemoveComments(WordprocessingDocument document)
public static void AcceptRevisions(WordprocessingDocument doc)
public static void RemovePersonalInfo(WordprocessingDocument document)
These methods are based on code that is presented in the following blog posts:
The custom application page (OpenXmlInspector.aspx) contains a few C# methods. They are pretty straightforward.
To open the document using the Open XML SDK, the code does the following:
-
Once the code has the document in an SPFile object, it uses the OpenBinary() method to get a byte array that contains the document. However, the Open XML SDK needs a stream to instantiate the document, so the code creates a MemoryStream from the byte array.
-
The code then creates the WordprocessingDocument from the stream.
The feature adds a menu item (Inspect Open XML Document) to the ECB menu. (The ECB menu is the drop down menu that you get for each document in a document library.)
To use this code, you will need to install the Open XML SDK.
For a good screen-cast on creating a feature in SharePoint, see Ted Pattison’s web casts. In addition, there are some great resources at http://www.microsoft.com/click/SharePointDeveloper/.
Note that this code is just a demonstration of using the Open XML SDK and LINQ to XML to query and modify documents; you would want to make modifications to this code before deploying it in a production environment. For instance, the code does not validate that the document is an Open XML document before querying it. I’m presenting it as an example of the type of development that you can do using these technologies.
Code is attached.
In the next series of blog posts, I’ll be exploring some interesting aspects of SharePoint development. In particular, I’ll be examining how to leverage the combination of Open XML, LINQ to XML, and SharePoint. The Open XML document formats enable new possibilities within SharePoint, and LINQ to XML brings the power, robustness, and succinctness of the functional approach to SharePoint development.
Each of these technologies (LINQ to XML, Open XML, and SharePoint) are important in their own right; when used in concert, the resulting power in the platform is greater than the sum of the parts.
What is SharePoint?
I realize that while many who read my blog will be aware of SharePoint, not all will, so a quick intro to SharePoint is in order.
SharePoint is a server product sold by Microsoft that allows you to create internet and intranet web sites. It is built on ASP.NET, and Microsoft SQL Server. There are six primary capabilities that SharePoint brings:
-
Collaboration: Allow teams to work together – collaborate on and publish documents, maintain task lists, implement workflows, and share information through the use of wikis and blogs.
-
Portals: Create a personal MySite portal to share information with others and personalize the user experience and content of an enterprise Web site based on the user’s profile.
-
-
-
-
Business Intelligence: Allow information workers to access critical business information, analyze and view data, and publish reports to make more informed decisions.
When someone asks me informally what SharePoint is about, I say it is about people working together.
It was only a few years ago when working with other people meant emailing documents around, and only a few years before that when it meant putting printed paper memos in physical mailboxes. I think that we have barely scratched the surface of what it means to work together.
Open XML and SharePoint
Much of the information in SharePoint is stored in documents, and until Open XML, those documents were binary office files. And this makes it harder to get at the information in those documents. You could find a library to work with those binary documents, but do you really want code like this running on your SharePoint server? What if you have gradual performance degradation? On a server, for all practical purposes, much of the information stored in binary office files is inaccessible.
But this changes with Open XML. We can access the contents of Open XML documents using high quality XML programming interfaces. XML programming API’s are the tools that servers are made of; we can now use those same tools to access and modify the information in our documents.
Within the SharePoint framework, there are three primary points where we can plug in functionality using Open XML:
-
In the user interface – we can access and modify Open XML documents when the user selects an item on a menu, clicks a button on a page or web part, etc.
-
In the storage architecture – we can access and modify Open XML documents when a document is checked in to or out of a document library. This can happen behind the scenes. We don’t need to bother the user. It just happens automatically and reliably.
-
In the business process layer – content that was previously locked in a black box can now be an integral part of a SharePoint workflow.
This blog post is about possibilities. Now that we can get at and modify the information in documents, what can we do? I’ve recently posted snippets of code that:
In the next blog post, I’m going to use these snippets of code from within SharePoint. I’ll present a small technology demonstration that will show the power and flexibility that is available to us, now that we can access and modify the contents of word processing documents, spreadsheets, and presentations.
(July 23, 2008: the technology demonstration of Open XML, the Open XML SDK, LINQ to XML, and SharePoint can be found here.)
For those who don't know, XmlLite is a very lightweight, fast pull parser that was introduced with the Vista SDK. It is a native parser, not part of the managed API. However, unlike MSXML, it's ok to use with .NET using COM interop. There have been questions on how to use it - specifically, where to get the header and lib files for it. In the future, there will be topics on MSDN that explain the information. However, it will be some time before the information is published to MSDN. Here is that info:
Installing XmlLite
Introduction
This topic provides information about installing XmlLite (Xmllite.dll) and the XmlLite development files, Xmllite.lib and Xmllite.h.
XmlLite Runtime File
The XmlLite runtime file, Xmllite.dll, is integrated into the following operating systems and products:
· Windows Server 2008
· Windows Vista
· Windows XP with Service Pack 3 or later.
· Windows Server 2003 with Service Pack 2 or later.
· Microsoft Internet Explorer 7 and later.
The XmlLite runtime (Xmllite.dll) is also available as a download from the XmlLite update page for the following operating systems:
· Windows Server 2003 (32-bit with Service Pack 1 or x64 Editions)
· Windows XP (32-bit with Service Pack 2 or x64 Editions)
XmlLite Development Files
To build applications with XmlLite, you must install the XmlLite development files, Xmllite.lib and Xmllite.h. These files are not included with the Xmllite.dll download; they are only provided in the Windows SDK for Vista. Note that you can install the SDK for Vista on Windows XP as well as on Vista. To install the SDK for Vista, see the SDK download page.
You do not have to install the entire Windows SDK to get Xmllite.lib and Xmllite.h. To get Xmllite.lib and Xmllite.h (as well as other .h and .lib files), select Header Files and the appropriate libraries for your computer (such as x86 Libraries) on the Installation Options page of the Windows SDK Setup Wizard.
See Also
XmlLite Samples
XmlLite Introduction
Here is something that is very, very cool! Pranav Wagh has built a much improved version of OpenXmlDiff, with a graphical user interface. And its free, already built (binaries available) on code.msdn.com/OpenXMLDiff.
When you start the program, you see a dialog box:
When you look at the HTML output, you see a beautiful window with deletions in red, additions in yellow, changes in green:
This program has all of the options I want or need: can select which types of changes to see, control type of output, and more.
This completely obviates my simple, crude OpenXmlDiff. Pranav's utility now has a permanent spot in my toolkit. I have it in my quick launch.
If you are a developer who works with Open XML, read his post, and download his utility. Nice job, Pranav!
Julien Chable written a concise
post on using PowerShell and
Power Tools for Open XML to secure and sign Open XML documents. He also has a short, useful intro to creating a PowerShell profile. One more thing, Julien has joined the Power Tools for Open XML v-team. Makes me happy. Welcome, Julien!
In the last three posts, in addition to the information regarding how we want to alter the markup in an Open XML document, I've made a few observations about how to write LINQ to XML code when modifying an XML tree in such a way that it becomes harder to introduce bugs. In addition, I've also mentioned a couple of other points that you can take into consideration - you can write code that performs better if you know a bit about how LINQ to XML operates under the covers.
In Using LINQ to XML to Accept Revisions, I mention a couple of approaches to modifying/transforming an XML tree, and tell why I picked the approach I did for the particular problem I'm solving.
In Using LINQ to XML to Remove Personal Information, I go through a couple of idioms in detail, showing how to write robust code to modify an XML tree.
In Using LINQ to XML to Remove Comments, I discuss one issue to take into consideration so that you can write queries that perform well.
While my primary motivation in writing these posts is to show how to modify the Open XML markup to acheive the desired goals, there is some good information for LINQ to XML developers. I felt that this might be lost behind the focus on Open XML markup, so I'm calling it out specifically here.
This post presents a snippet of code to remove comments from an Open XML Wordprocessing document.
Note: This post may be of interest to LINQ to XML developers, as it contains some information that helps you write queries that perform better. In the case of very large documents, the approach described below performs much better than other approaches.
The code is very simple: remove all w:commentRangeStart, w:commentRangeEnd, and w:commentReference elements in the main document part, and then remove the comment part.
The following is the code that removes the above mentioned elements.
// pre-atomize the XName objects so that they are not atomized for every item in the collection
XName commentRangeStart = w + "commentRangeStart";
XName commentRangeEnd = w + "commentRangeEnd";
XName commentReference = w + "commentReference";
mainDocumentXDoc.Descendants()
.Where(x => x.Name == commentRangeStart || x.Name == commentRangeEnd || x.Name == commentReference)
.Remove();
There are two other ways that you could write this code. A first attempt might be as follows:
mainDocumentXDoc.Descendants(w + "commentRangeStart").Remove();
mainDocumentXDoc.Descendants(w + "commentRangeEnd").Remove();
mainDocumentXDoc.Descendants(w + "commentReference").Remove();
Of course, this causes iteration of all of the descendants three times, not very desirable for large documents.
So, keeping this in mind, you might write it like this:
mainDocumentXDoc.Descendants()
.Where(x => x.Name == w + "commentRangeStart" ||
x.Name == w + "commentRangeEnd" ||
x.Name == w + "commentReference")
.Remove();
This causes iterations of the Descendants axis only once. However, there is a subtler performance issue here: the names (as expressed by w + "commentRangeStart", etc.) are atomized over and over again for every item in the Descendants axis. To make the code perform as well as possible, we pre-atomize the XName objects, then we use them in the call to the Where extension method:
XName commentRangeStart = w + "commentRangeStart";
XName commentRangeEnd = w + "commentRangeEnd";
XName commentReference = w + "commentReference";
mainDocumentXDoc.Descendants()
.Where(x => x.Name == commentRangeStart || x.Name == commentRangeEnd || x.Name == commentReference)
.Remove();
For more detailed information about atomization and LINQ to XML performance, see Performance of LINQ to XML.
The attached code also has a bool method that indicates whether the document contains comments.
Code is attached.
This post presents some code to remove personal information from an Open XML word processing document.
Note: this post contains interesting information for LINQ to XML developers even if you are not interested in removing personal information from a document. It demonstrates some easy ways to write robust code in LINQ to XML that behaves properly even if we don't know the original state of the document. For instance, it shows how to write very short, robust code to remove an element that doesn't fail if the element doesn't already exist.
One thing to note: The Ecma 376 Open XML standard states that what comprises personal information is application defined. To determine what I needed to do for the code in this post, I created a document normally, then created a copy and removed personal information, and then compared the two using OpenXmlDiff.
After creating a document with personal information, and using the code attached to this page to remove it, Word 2007 reports that no personal information exists in the document.
Here is what the code attached to this page does:
- In the extended file properties part (/docProps/app.xml), set the company name to the empty string.
- In the extended file properties part, set the TotalTime element to "0"
- In the core file properties part (/docProps/core.xml), set the dc:creator and cp:lastModifiedBy to the empty string.
- In the core file properties part, set cp:revision to "1"
- In the settings part (/word/settings.xml), add two empty elements, w:removePersonalInformation, w:removeDateAndTime. These elements indicate to an application that personal information has been removed, and should not be re-added. These elements should be added at the correct spot in the sequence, so that XSD validation will work properly.
This code uses some interesting but somewhat obscure LINQ to XML tricks to do this work in a robust fashion. For instance, the following code uses the Elements and Remove extension methods in such a way that if the Company element doesn't exist, it works just fine, and doesn't throw an exception:
extendedFilePropertiesXDoc.Elements(x + "Properties").Elements(x + "Company").Remove();
The way that this works - calling the first Elements(x + "Properties") method returns IEnumerable<XElement> that contains either 0 or 1 element in the collection. The next call into Elements uses the LINQ to XML Elements extension method, which returns a collection that contains all "Company" elements that are children of any elements of the source collection. Well, the first collection either contains 0 or 1 item in the collection, so this returns a new collection that contains either 0 or 1 "Company" elements. Then we dot into the Remove extension method, which removes all elements in the collection. The Remove extension method uses "snapshot semantics", which means that it materializes to a list, and then removes them. It is fine to call the Remove extension method with an empty collection, or a collection containing a single element.
So, by using this idiom, we can write code that removes the element if it exists, and know that the code will not fail if the element to be removed doesn't exist.
The following code uses a similar approach to set a variable to the TotalTime element. If the element doesn't exist, then the totalTime variable is set to null.
XElement totalTime = extendedFilePropertiesXDoc.Elements(x + "Properties").Elements(x + "TotalTime").FirstOrDefault();
if (totalTime != null)
totalTime.Value = "0";
The way that this works is that the following expression returns a collection that contains either 0 or 1 elements:
extendedFilePropertiesXDoc.Elements(x + "Properties").Elements(x + "TotalTime")
Then we "dot" into the FirstOrDefault extension method. If there is an element in the collection, it returns it (thereby converting the collection into a singleton). If there is no element in the collection, FirstOrDefault returns the default value for the items in the collection. XElement is a reference type, and default value of a reference type is null, so this idiom returns either the element of interest, or null if it doesn't exist.
The following code uses extension methods in a similar way - the Nodes call returns all child nodes of the relevant element. The call into the OfType<T> extension method then selects just for XText nodes. The result may contain a collection of text nodes that we want to set to the empty string, or it may contain an empty collection, in which case, the code will work just fine and not throw an exception.
foreach (var textNode in coreFilePropertiesXDoc.Elements(cp + "coreProperties")
.Elements(dc + "creator")
.Nodes()
.OfType<XText>())
textNode.Value = "";
Finally, the following code shows how to add elements at a specific position in a sequence, where the preceding elements may or may not exist:
// add the new elements in the right position. Add them after the following three elements
// (which may or may not exist in the xml document).
XElement settings = documentSettingsXDoc.Root;
XElement lastOfTop3 = settings.Elements()
.Where(e => e.Name == w + "writeProtection" ||
e.Name == w + "view" ||
e.Name == w + "zoom")
.InDocumentOrder()
.LastOrDefault();
if (lastOfTop3 == null)
{
// none of those three exist, so add as first children of the root element
settings.AddFirst(
settings.Elements(w + "removePersonalInformation").Any() ?
null :
new XElement(w + "removePersonalInformation"),
settings.Elements(w + "removeDateAndTime").Any() ?
null :
new XElement(w + "removeDateAndTime")
);
}
else
{
// one of those three exist, so add after the last one
lastOfTop3.AddAfterSelf(
settings.Elements(w + "removePersonalInformation").Any() ?
null :
new XElement(w + "removePersonalInformation"),
settings.Elements(w + "removeDateAndTime").Any() ?
null :
new XElement(w + "removeDateAndTime")
);
}
We need to add our new elements after any of the specified three elements, so we select for the three elements, sort them in document order, and then get the last one. If none of the three exist, then lastOfTop3 will be set to null:
XElement lastOfTop3 = settings.Elements()
.Where(e => e.Name == w + "writeProtection" ||
e.Name == w + "view" ||
e.Name == w + "zoom")
.InDocumentOrder()
.LastOrDefault();
And then, of course, we want to add our new elements only if they don't already exist, so we use the Elements axis method, and then use the Any extension method to tell us if the element exists or not:
lastOfTop3.AddAfterSelf(
settings.Elements(w + "removePersonalInformation").Any() ?
null :
new XElement(w + "removePersonalInformation"),
settings.Elements(w + "removeDateAndTime").Any() ?
null :
new XElement(w + "removeDateAndTime")
);
It is valid to pass null to AddAfterSelf (or any of the XElement methods that take content). If you pass null as one of the items in a params array, AddAfterSelf simply ignores that item. See this
topic for detailed information on this.
Using this approach, we add the elements in the right place, and don't add them if they already exist.
Some people don't reallize that by calling the Any extension method, we have created a very cheap operation. Any doesn't iterate through its entire source - it just iterates until it gets one item in the source collection, in which case it returns true, or it sees that the source collection is empty, in which case it returns false. So it is a very cheap operation.
Once you get your head around these idioms, you can write short and robust code to modify XML trees.
The code attached to this page also contains a bool function that tells you whether a document contains personal information.
Code is attached.
In this post, I present some code that uses the Open XML SDK and LINQ to XML to accept all revisions in an Open XML word processing document.
So, for example, if we have this document:
And we run this code, the document then looks like this:
The main part of the work when accepting revisions is to delete a bunch of nodes, however, there are other tasks to accomplish. In the attached code, you will see that in one case with inserted text, we need to move child nodes of the w:ins element up to the level of the w:ins element. There is also extra work when joining paragraphs, removing deleted rows in a table, etc.
This code is not written in the functional style. It is written in a procedural (imperative style) that reads the main document part into an XML tree, and then modifies the tree as appropriate to accept revisions. This could be accomplished using the approach documented in this post, however, the added complexity of using annotations isn't really necessary for this simple example. That approach is much more appropriate when doing something like converting a DOCX to XHTML. In addition, I want this code to execute as quickly as possible (although it's not necessary to spend a large amount of time to optimize, in my opinion). Using the annotations approach might slow down this code.
One note: this code does not accept tracked changes for content controls. For instance, if you create a content control, turn on track changes, then delete the control, the document will still show that revisions exist. Supporting this is a project for the future. In any case, this is not a main stream scenario.
As you can see, this code is not very long - it's pretty simple. The largest part of the code is comments documenting the changes that we want to make to the markup. I've run this code on over 1000 documents, and it worked properly.
I initially wrote this code for the Power Tools cmdlet to accept revisions - this code is basically the same as in the Power Tools cmdlet.
Code is attached.
Antonio Zamora at
Staff DotNet has written an interesting
post on using the PowerTools for Open XML to apply consistent styles to documents. It gets information from a template document (headers, footers, and theme) and sets this information on the target document. A document can have three types of headers (first, even, and odd), and three types of footers. This example script shows how to set all headers and footers for a document.
This is a fun, geeky post for those interested in functional programming.
Sometimes you have flat data where there is hierarchy expressed in the data, but the form of the data is flat. You may need to transform this data into a hierarchy, such as an XML tree.
For instance, you may have the following data:
Heading[] headings = new[] {
new Heading { Level = 1, Text = "Intro" },
new Heading { Level = 2, Text = "Hello" },
new Heading { Level = 2, Text = "Summary" },
new Heading { Level = 1, Text = "Technical Info" },
new Heading { Level = 2, Text = "Class API" },
new Heading { Level = 3, Text = "Class1" },
new Heading { Level = 3, Text = "Class2" },
new Heading { Level = 2, Text = "Interoperability" },
new Heading { Level = 3, Text = "Scenario 1" },
new Heading { Level = 3, Text = "Scenario 2" },
new Heading { Level = 1, Text = "In Conclusion" }
};
And you want to transform this data into the following XML:
<Root>
<Heading1 Name="Intro">
<Heading2 Name="Hello" />
<Heading2 Name="Summary" />
</Heading1>
<Heading1 Name="Technical Info">
<Heading2 Name="Class API">
<Heading3 Name="Class1" />
<Heading3 Name="Class2" />
</Heading2>
&nb