Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

Mike — Sat, 24 Jun 2006 13:48:51 GMT

May I point out a few objections to the article?

I think that you are deliberately omitting to mention automation even though it's still in the 2007 timeframe the best, most powerful, quickest and safest way to target the new xml file formats.

I think there is an ongoing confusion between the target file format and the programming language and SDKs used to target this new file format. That the new file format is "XML by default" does not mean you are required to use an XML programming stack to create/update it. Quite the opposite, if the XML file format serializes values resulting from engine calculations (Excel formulas, Word repaginations, ...), then I wonder how the ease and simplicity of an XML programming stack is going to do any good to update those values without an appropriate running instance of Excel, Word or Powerpoint doing the bulk of the work for you, just as it always did so far. And if you maintain all of that with the XML programming stack plus all your custom code, then you are rewriting Excel, Word and Powerpoint.

May be there is something obvious I have missed though. But if you take some of the code snippets mentioned in the article, then you have to say that some of these either put the document in an unknown state or a plain corrupt state. Why? because if you change the value somewhere, you need to also update all the places that were referencing it. So to me, telling everyone to use an XML programming stack to do any significant changes in an Office document is going to result in interesting discussions...

Not everything is bad though, there are scenarios where you can for instance replace a part in a file and be done with it, exemplifying a decoupling. What about documenting those scenarios? Here is one, change the chart part of an Excel document, by replacing 2D bars with 3D bars.

Automation is still the way to go. If you'd like to update a value in a document, it's one or a couple lines of automation code, not hundreds (see the code snippets). If you add this up in case you are really making big changes, or are generating a complex document, then you end up with perhaps ten of thousands lines of code, with all the consequences attached to it. Interesting...

There are a number of inaccuracies IMHO in the article. Let me point out a few ones,

- "The file is much more robust because it is broken up into different document parts. Should one part become damaged (for example, a part describing headers), the rest of the document remains intact and still opens successfully." : I don't think so. I think opening up the parts lead to an entire new class of document corruptions. Since what you have done is basically let anyone corrupt anything within the document. Don't forget that so far Office developers would either use automation (directly or through VSTO interop), or a third-party : developers would never touch the file themselves.

- "The file is easier to work with programmatically" : see the length of the code snippets, and compare that to automation code. Automation code is several orders of magnitude shorter, safer, intuitive, and so on. An Office developer using automation does not have to care about the underlying semantics, references and so on. It's not just "I like Automation better because it's my taste", it's really what it takes to make consistent changes to a file : in most cases, you need the engines to keep everything validated and consistent.

- "Custom XML is also easier to work with" : Custom XML is a feature of Office 2003, and was already stored in a separate stream.

- "The new Open XML Format, with a few minor exceptions, is written entirely in XML and is contained in a .zip file." : that's not quite true. The document implementation follows the specs, among which the open packaging conventions (whatever the "open" word means). And what this means is that the internals must adhere to a strict structure of parts, relations and relationships, where some of the parts may end up being XML. That's not quite the same than saying the document is mostly XML, since this would imply you are passing a real XML file (XML MIME content type) which could have had some binary data islands in some section(s). Also, there is at least one known case where OLE is involved as well (password-protected document).

- "Users can also use standard transformations to extract or repurpose the data." : in a number of cases, you need to understand the data you deal with. What if you are dealing with an Office document where one or more VBA macros are attached to some events triggered when a value gets updated? Again, if you don't make consistent changes throughout the entire document, maitaining indexes and references, updating the values and parts accordingly, then what are you doing exactly other than dirtying the document and leaving it to an unknown state. By unknown state, I mean a document that is not ready for printing for instance. Or a plain corruption by the way.

- "In addition, security concerns are drastically reduced because the information is stored in XML, which is essentially plain text. Thus, the data can pass through corporate firewalls without hindrance." : now I am confused, do you mean you are passing XML through the wire? Last time I checked, .docx/.xlsx/.pptx are binary file formats just like .doc/.xls/.ppt. It's only some internal parts of it which may end up being XML : if you are passing those documents over the wire, you are still left to the classic MIME content type exclusion. For instance, if you attach such document to an email, it may be blocked by default as per the enterprise policies.

In the end, I am not saying the XML programming stack being suggested (WINFX open packaging API, plus whatever XML parser/serializer) is not useful in 100% of cases. But I think the confusion needs to be cleared up sooner than later. I think that, at the moment, you are setting expections way too high.

re: Open XML File Formats: What is it, and how can I get started?

erikaehrli — Tue, 27 Jun 2006 06:39:52 GMT

Hi Mike,

Thanks for your comment, probably the longest comment I have seen since I started blogging! I have seen your comments on Brian Jones’ blog. I have seen that you have been following the news for a while…
http://blogs.msdn.com/brian_jones/archive/2006/06/02/613702.aspx

The purpose of this blog entry is to let developers know that there IS a technology that can help you to read/write/modify Office (Word, Excel, PowerPoint) files without automation. Also, the purpose of this blog entry is to point people to resources related with the Open XML File Formats and to let them know what the File Formats are.

My intentions with this blog entry were not to say that working with the File Formats is better than using automation. It’s hard to say that automation is better than the Open XML File Formats or the other way around. Any developer that has worked with automation and WordML/SpreadsheetML can tell you the right answer is: “It depends on what you are trying to accomplish.” This is because there are scenarios where using automation is not the best option. Developers should know what are the options on the menu and the pros/cons of each one in order to make the right decision when it comes to select a technology.

Why are people interested in not using automation? Not that they dislike it of course. I agree with you that it’s very easy to use and that it’s the bread and butter of all Office developers. However, there are scenarios when you know that automation is not the best option. Two examples I can quickly think about (and there are more, but I am doing my best to keep this answer short):

1. Server-side scenarios: If you have a Web application and you need to programmatically generate Word documents, or export data to an Excel Spreadsheet. I think the best option here would be to avoid the use of automation. It’s not a good idea to create an instance of a Word or Excel application on a Web server. Open XML File Formats would be a wise option to use for this specific scenario.
2. Massive document generation: If you need to generate a single document (client-side), automation might help. But try generating 1,000 documents using automation! You will see the performance is not going to be nice. Automation makes an intensive use of memory and even if you turn some settings off to optimize performance you are still not going to be happy with the time and resources needed to accomplish the mission. Now try serializing the 1,000 documents using XML (use an XMLTextWriter) and you will see it’s way much better. Also, if you use the XML File Formats, documents are more compact, so they are easier to move across the wire. Definitely an option that application developers and solution architects would consider.

About getting corrupted files, I don’t think that’s going to be the outcome. I think one of the best things of the Open XML File Formats is that it allows document to be loosely coupled. You have document parts that you can update without changing the entire document, and the result is not corruption. It’s just a great way to isolate document parts and update them easily. I have worked with the Office 2003 WordML and SpreadsheetML formats (you have everything on a single file); whereas the new File Formats isolate document parts and I can tell you it’s the dream of software engineers to work with loosely coupled applications.

"The new Open XML Format, WITH A FEW MINOR EXCEPTIONS, is written entirely in XML and is contained in a .zip file." : This is true, see, for OLE objects (embedded objects like Visio diagrams) or images you won’t have an XML serialization, but for everything else you have an XML document. Just do the exercise of saving a Word file as Open XML and you will see what I mean.

If I am setting high expectations for this technology, well, that’s because based in my experience working with Office, the current Office XML schemas, and automation, I find this technology to be AMAZING and I think it provides a great additional option for developers who for very specific reasons need to consider more options.

re: Open XML File Formats: What is it, and how can I get started?

Mike — Tue, 27 Jun 2006 09:44:21 GMT

Thanks for your answers.

"The purpose of this blog entry is to let developers know that there IS a technology that can help you to read/write/modify Office (Word, Excel, PowerPoint) files without automation." : I know you are evangelizing the technology, and don't expect you to bark on it.

"It’s hard to say that automation is better than the Open XML File Formats or the other way around." : Hmmm, this is one of the things I don't get. I think it's pretty obvious that automation is not only better, it's shorter and safer. Let me give a few examples. Let's say you'd like to delete a slide, or a worksheet. In automation code, this is really one line of code, period. How much with direct access to the internals of the file formats? And I have kept the best part of the argument : those code snippets above actually corrupt the document (this is a comment I made to Kevin Boske in his blog a few days ago).

If you are an Office developer, you can use automation to find your way out. And, you can rely on a third-party if you'd rather not have a running Office instance. There are tons out there that have managed to reverse engineer the file formats and expose it through an Automation-equivalent API without REQUIRING to deal with the guts of the internals (which is unfortunately what this new XML is about).

The only reason to use an XML programming stack, and dare to directly access the internals of the file format would be if there was a good reason to do so, for instance if relying on XML xpath, you end up with a very robust, short and elegant code to create/update a document. Can you give some examples of that? None of the code snippets above can beat their Automation's equivalent in robustness, conciseness and elegance. Also don't forget that the built-in Macro recorder has been providing Office developers a great way to discover Automation code snippets without having to actually learn the underlying object model. Tell me about it with the 4000+ page TC45 specs (ironically enough, those specs don't cover everything you need to know, far from that : an example is password-protected documents).

The core of the argument is how better are the XML schemas compared to a binary file format serialized as XML (in other words, binary bits with angle brackets). I have said in the previous comment that there could be a scenario where you could replace a chart part for another and be done with it. Even this positive view is actually oversimplying things a bit : if you use custom error bars in charts, then I believe there are references to other parts, and all of a sudden you can't do a simple part replacement anymore.

Where are the XML-based examples where Automation is much longer, less robust? Note that I am not saying "robust" as a reliable .exe, I am talking robust as something that keeps the document consistent, validated, ready for printing, ready for sharing, ...

"This is true, see, for OLE objects (embedded objects like Visio diagrams) or images you won’t have an XML serialization, but for everything else you have an XML document." : I think there is an over-statement here. The way everyone understands a sentence like this is as in Word 2003's XML, i.e. you have a single XML file, with some data islands in it. But the new Word 2007's XML (for instance), XML is scattered in pieces, so it's much harder to work with it, especially since you as a developer need to maintain the indexes and references yourself. I have also pointed out to Kevin that the WinFX packaging API exposes the guts of parts, not abstracting those enough to avoid mistakes.

And, dealing with XML, where is the validation API? How can you ensure that, after you have used one of the code snippets above, the resulting document is valid without adding all the clutter of a separate yet-to-be-mentioned XML validation stack combined with all of the required semantics of the internals of the file formats ?

All what this is calling for is for third-parties to abstract this away, because it's my belief (from my experience as an ISV) that it's too hard, time-consuming, dangerous and complicated to deal with directly. (third-parties provide a solution to the server-side scenarios you mention above : in server reliability and speed).

I think there is a disturbing confusion between XML as a serialization format (binary bits with angle brackets) and XML as a true programming model (strong decoupling, needs not much knowledge of core specs,...). We'll see how it goes.

OpenXML code snippets

Notes2Self.net — Tue, 27 Jun 2006 14:40:52 GMT

I just spotted a great post on the TechEd Bloggers feed from Erika Ehrli - "Open XML File Formats: What...

Interoperabilità tra OpenXML e Open Document Format

Blog Team TechNet Italia — Thu, 06 Jul 2006 12:22:55 GMT

E' stato fatto oggi un importante annuncio da Microsoft, che anche per noi è stato una bella sorpresa:...

Steps towards interoperability

Pranav ... Blogging — Sat, 08 Jul 2006 02:00:00 GMT

Yeah, I know long time no B(log). I was real busy those days, but don’t worry I will tell you what was...

Microsoft will bridge Open XML and ODF in 2007 Office System

Alain's Globosphere — Mon, 10 Jul 2006 12:21:06 GMT

Remember my last post about ODF?
Things go quite fast in our industry. Indeed, Microsoft has decided...

re: Open XML File Formats: What is it, and how can I get started?

whui1978@hotmail.com — Mon, 24 Jul 2006 06:47:09 GMT

Does somebody knows where I can get the Open XML file format detail reference?
I want to write a program to read and write Open Xml files directly.

re: Open XML File Formats: What is it, and how can I get started?

whui1978@hotmail.com — Mon, 24 Jul 2006 06:52:19 GMT

Infact i have got a PDF version of the reference:Office Open XML Document Interchange Specification /Ecma TC45 Working Draft 1.3 / Public Distribution May 2006.
But i want a .chm version of it.
If somebody knows where it is, please give me a link, thanks a lot!

Microsoft PacWest SharePoint Server Newsletter – July 2006

Owen Allen (MSFT PacWest) — Wed, 26 Jul 2006 09:05:33 GMT

Microsoft PacWest SharePoint Server Newsletter – July 2006

 Update on Download Availability of...

Word: Set Custom Property

Lars — Tue, 15 Aug 2006 11:45:40 GMT

As far as I have seen, it is not enough to change the custom property value in the docProps\Custom.xml file because the original value of the custom property inserted in the document is still there (in the word\document.xml file). I had to manually update all field codes in the document using Word (F9) before the new custom property value will be seen.

Comments?

re: Open XML File Formats: What is it, and how can I get started?

erikaehrli — Thu, 17 Aug 2006 02:08:27 GMT

whui1978, you can find the ECMA spec here:

http://www.ecma-international.org/activities/office%20open%20xml%20formats/tc45_fd_xml_docform.zip

I haven't seen a chm.

Lars, read my August blog entries, you will find samples on how to read/write Office XML File Formats. I am writing a new document and I change programatically the custom.xml file without changing anything else. I only replace the file and it works, not sure why you are experiencing this behavior.

re: can you please guide me the way of retrieving my file back to its initial format, ie once after i convert my power point file to open xml and if i need to pass this open xml file and retrieve back my original ppt,,, please hep

CASH — Mon, 24 Sep 2007 13:11:14 GMT

i ve to convert my ppt file to xml using c# , what i was able to do using system.io.packaging, now i ve to recollect the package and retrieve my original ppt back, please do hep out a way of doing it

re: Open XML File Formats: What is it, and how can I get started?

CASH — Mon, 24 Sep 2007 13:16:45 GMT

How can i retrieve my files back , once i save them as open xml format

Erika Ehrli : Open XML File Formats: What is it, and how can I get started?

Weddings — Sat, 07 Jun 2008 01:05:28 GMT

Some people have been following the news and are even ahead of most of us already buidling solutions to generate documents using the Open XML File Formats. Some other people are not familiar with this technology and want to learn more about this, so her

re: Open XML File Formats: What is it, and how can I get started?

slickshow — Fri, 22 Aug 2008 06:28:37 GMT

The examples for PresentationML show how to count slides and reorder and delete slides, etc., In other words, they work on existing slide deck. I am trying to append new pptx slides (one at a time) to create a new slide deck, and I dont see an example of how to programmatically (using OpenXML) append a pptx to another. If you have any code samples, could you please post them.

Thanks.

-ss

re: Open XML File Formats: What is it, and how can I get started?

John — Tue, 17 Feb 2009 20:22:45 GMT

Does anybody know whether it si possible to save a file as ".doc" i.e. word 2003 using Open XML. I could create docx file but not doc.

If yes, please let me know how.

Thanks & Regards

John

re: Open XML File Formats: What is it, and how can I get started?

allen — Tue, 24 Feb 2009 17:53:52 GMT

I am not able to get the header in the /docx file correctly. I am not able to find the structure of the word document by renaming it with .zip extension. I have an image in header in .docx and i want to find the name given for that in the structure. How to do this?

Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

OpenXML code snippets

Interoperabilit&#224; tra OpenXML e Open Document Format

Steps towards interoperability

Microsoft will bridge Open XML and ODF in 2007 Office System

re: Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

Microsoft PacWest SharePoint Server Newsletter – July 2006

Word: Set Custom Property

re: Open XML File Formats: What is it, and how can I get started?

re: can you please guide me the way of retrieving my file back to its initial format, ie once after i convert my power point file to open xml and if i need to pass this open xml file and retrieve back my original ppt,,, please hep

re: Open XML File Formats: What is it, and how can I get started?

Erika Ehrli : Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

re: Open XML File Formats: What is it, and how can I get started?

Interoperabilità tra OpenXML e Open Document Format