Interoperability of the Office Open XML formats

re: Interoperability of the Office Open XML formats

Stephan Jaensch — Thu, 21 Sep 2006 23:29:05 GMT

Brian Jones wrote:

<i>This is definitely not true, and I'm sorry if I haven't done a good enough job of explaining why we've been moving towards open formats for years now. The initial binary formats were designed back in a time when performance of reading and writing from a floppy disk were more important than shared documents via the internet.</i>

What a classic straw man argument setup: You refute his argument (no openness) by giving reasons why the past format was binary instead of text-based. The former (open, documented format) has nothing to do with the latter (binary/text format). Images are typically saved in a binary format, still there are many image formats which are considered open.

You could have opened the binary formats for a long, long time, but you didn't. You chose to keep them closed and closely guarded. If you were being honest, you would have just admitted that. Everybody knows it and you are moving to a new, open format anyway. Or is it the corporate doublethink culture at Microsoft which doesn't let you see that? Honest question, I would really like to see your viewpoint on that.

- Stephan

re: Interoperability of the Office Open XML formats

BrianJones — Fri, 22 Sep 2006 00:57:48 GMT

Stephan, I was actually tempted not to reply to your comment because you are focusing on a very small piece of the overall message I'm trying to get across. We've focused hard on making the binary formats a thing of the past. I can't just ignore your points though, so here it goes :-)

I have to admit that I disagree with you on your point about there being two seperate issues. From my point of view, moving to a text based format and being open really do go hand in hand (at least in this case). The move to a text based document format has made it actually manageable to both document and support interoperability. It would have been nearly impossible with the old formats to achieve this. If we really wanted to block interoperability we wouldn't have invested in RTF, HTML, and XML over the past 10 years.

The legacy binary formats were extremely complicated and even with the documentation (which we actually did provide to a number people who requested it), it was very difficult to work with. Do you know that the majority of corrupt documents we get from customers are corrupted by 3rd party applications? It's extremely difficult to work with the binary formats. They are essentially a dump of the internal memory structures of the applications and that made them pretty hard for others to use. There isn't anyone in Office who feels like our format allows us to win some type of competition though; it's the features and functionality that make Office so great.

There are plenty of competitors out there who support the Office binary formats. We've never tryied to block people from implementing them. We just weren't able to put resources into supporting them as an interoperability solution because they weren't designed that way. It would have been a huge support issue. But if you look at applications like Open Office, Gnumeric, etc. they do a great job working with the formats. In fact, when I use Open Office, it actually opens and saves spreadsheets in the XL binary format significantly faster than in the ODF format.

Now, I know that the views in the software industry haven't always been what they are now. It's true that in the past, some viewed file formats as a way to win some type of competition. But I don't think anyone has felt that way for quite some time (at least around here). File formats are just a way of persisting the data, and the easier those things are to work with (while still allowing for all the data to be preserved) the better.

So, if you are upset that the formats weren't documented sooner, I understand and I'm sorry. I think the work we are doing now is really going to help out there though, and to be honest many folks don't realize yet that we are doing this. We've been talking about it for over a year now, but I still see people speculating that the XML formats are just a bunch of undocumented binary goo, which couldn't be further from the truth.

-Brian

re: Interoperability of the Office Open XML formats

orlando — Fri, 22 Sep 2006 02:00:52 GMT

IMHO

most office software users _dont care_ about file formats.

I predict this:

after 4 years of Office 12 delivering, 95% of MS Office users ( any version ) will still be saving as .DOC , XLS, etc.

They don't care about this things, they just want to save his work and let other people see it ( mostly via e-mail attachment ).

If you want "openness" just document the legacy binary formats. Let this XML thing to specialized users and be honest when you talk about "interoperability" ( i.e.: release a Linux .doc, .xls, .ppt viewer ).

( sorry for the grammar/spelling/etc )

-orlando

re: Interoperability of the Office Open XML formats

Francis — Fri, 22 Sep 2006 03:03:49 GMT

"The legacy binary formats were extremely complicated and even with the documentation... it was very difficult to work with. Do you know that the majority of corrupt documents we get from customers are corrupted by 3rd party applications?"

Not just 3rd party applications! All versions of Office I have used (through 2003) periodically corrupt their own files.

Interestingly, this has not once happened to me when using the Open XML format with Office 2007 betas. If that's not a reason to move to simpler, open formats, I do not know what is! :-)

re: Interoperability of the Office Open XML formats

Patrick Schmid — Fri, 22 Sep 2006 05:27:43 GMT

orlando,

I highly doubt that this will be the case. The decision to have the Open XML formats be the default has already had significant impact. Despite Office 2007 still being in beta, there are already several messages a day asking how to open the 2007 formats in earlier Office versions, because users uninstalled 2007, or asking why someone can't open the 2007 file they emailed to the person.
I think that many, many users will (without making a conscious decision) end up using the Open XML formats with Office 2007. The only once making a conscious decision will be administrators for their companies and power users.

Patrick

re: Interoperability of the Office Open XML formats

Bob — Fri, 22 Sep 2006 05:46:42 GMT

Brian,
Speaking of interoperability, are there any plans to provide DOCX import and/or export converters for WordPad?

re: Interoperability of the Office Open XML formats

Juan R. — Fri, 22 Sep 2006 14:11:55 GMT

Interesting post by John. I read it several times and still are unable to fix he is really stating there.

Is John claiming that the previous movement to Open format from others was not caused by pressure from market and loosing the Office competition with Microsoft? Are those others non-profit organizations? Are those ‘non-profit organizations’ promoting open formats in other fields of interests to governments and users?

Microsoft uses their own format, whereas others reuse stuff from W3C. Well, is John claiming that standards becoming from the W3C are purely scientific oriented? If are not then why votes and core design options are only left to consortium members –i.e. folks from paying organizations and invited expertises-. Yes, you can do suggestions to W3C WG (I did many on the MathML list) but they can accept it or reject them in basis to their own criteria (often economical rather than purely technical).

About “to create your own format, which will be somewhat documented, but so complicated that nobody in the world at least for next 2 years will be able to open files the same way as your last...”

What does John mean? After 2 years, most of MathML tools are unable to open c-MathML format (none browser i know support it natively -Carlisle, XSLT client side to presentation code is not native support ;-) and even the most sophisticated (e.g. Mathematica 5.2) software got problems to process certain c-MathML files doing use of the most sophisticated features such as OpenMath extensions. Is John stating that MathML format was designed to be complex enough because market share purposes?

Does John knows that last XSLT 2 (rejected by Microsoft, was not?) is critized for being so complex that is being not implemented and even some folks doubt that XSLT can be completely implemented in practice because extreme complexity of the W3C spec?

It is true that ODF is imperfect (as Microsoft new format is also!), but I do not see the point here. Why cannot Brian critize ODF approach? Why cannot I applause some of design options of the new Office Math format against the p-MathML reused in ODF? And maybe more important, why W3C MathML people can state, in both active and passive ways, by both formal and informal channels, the limitations of previous approaches as TeX or ISO-12083 when defending themselves from not reusing both, but Brian cannot state here the limitations of ODF and rationale for not reusing MathML in next Office?

About conversion issues I predict many surprises to people using ODF. In some ODF blog (now I do not remember) I read about an experiment where a piece of Math generated from Mathematica was copied and pasted to ODF and errors generated.

If my proposal for a profile attribute is finally implemented in next MathML 3 the situation would improve in a future but still, sorry to say this, conversion will be not guaranteed by using W3C standard formats.

Juan R.

Center for CANONICAL |SCIENCE)

re: Interoperability of the Office Open XML formats

Anonymous — Fri, 22 Sep 2006 16:58:17 GMT

For those of you mentioning that the binary format was undocumented, I suspect you never tried searching the web. I've found the documentation for Word 97 and Word 6.0 by doing a search for these terms -

"word file format dxagaphalf"

Now I know no one would guess that last search term, that's the name of one of the properties in the format and I only put it in to ensure I'd get some good hits. And you'll get different pages if you mix and match some other search terms such as "documentation binary" etc. I even found documentation for Word 1.0 files once.

Yes, I realize no one has information on anything after Word 97 but there isn't that much that has changed, and a fair amount of it is more of the same, not that bad to reverse engineer. I've seen reasonable non-MS viewers that worked with just this information. Of course, they had their limitations.

BTW, the comment about why Word was binary originally? WordPerfect was binary, as well as AmiPro, it wasn't like MS was being malicious by being the only binary format out there.

The trick however is not just the format. How does one implement a full blown text processor? It is no easy task. Even if you can read the format, as we can now with Office 2007, it will take a long time for anyone to create something that shows a file just like Word, there are just too many options that interact with each other in unexpected ways.

Sometimes I get the sense that some people think this open format will magically allow them to easily create an application that will view Word files exactly like Word. I'm not saying it won't be much easier to implement a simple or even a pretty good viewer, for certain it is, but the format is really only the tip of the iceberg if you are looking for full-fidelity.

Brian, this isn't an attack on the format, nor its interoperability, that has all improved wonderfully (MindJet being an example). This is just making the point that the format, even when binary, wasn't always the main obstacle to having full-fidelity, non-MS viewers of Word files, and this isn't MS's fault either, it is the nature of such a complex text formatter.

I don't know how many applications support ODF, but I'd wager a guess that if you opened the same file in all of them, there will be many small, and in a complex document, several significant, differences between the display of the file.

re: Interoperability of the Office Open XML formats

A User — Fri, 22 Sep 2006 22:11:12 GMT

I think John makes a lot of correct points, but misses the forrest for the trees.

Recall how the interoperability issue played out in the days of the OS wars. Everybody's computers could interoperate when everybody used the same operating system. You could move information between applications reliably by using ascii TXT. To be interoperable was to be reductionist.

Something completely different is happening this time. reductionism and monoculture are at an end. Is MonoSoft looking out for its own interest? You betcha: to avoid being left behind.

re: Interoperability of the Office Open XML formats

Marbux — Sat, 23 Sep 2006 07:00:41 GMT

Brian, I am intrigued by your references to the binary formats in the past tense and repeated references to "moving" from the binary formats to the XML format.

Has something changed? My understanding is that the apps are too brittle for the surgery required to remove the binary formats and replace them with the XML formats, that the Ecma Office Open XML ("EOOX") format is part of a translation layer from and to the binary formats used for internal processing. Has a decision been made to perform the necessary surgery nonetheless?

And isn't the fact that EOOX is only part of a translation layer not used for internal processing the very reason that the ODF Translator can not set ODF as the default file format in Office? As I understand the situation, the Translator performs a transformation between EOOF and ODF.

And what is this about viewers for other operating systems? Has Microsoft decided to open source the Windows dependencies in EOOX such as OLE objects? Is it not a fact that EOOX implementation on non-Windows operating systems poses enormous development barriers for some document features?

I do not wish to rain on anyone's parade. But my evaluation of the situation is that the file formats used internally for Office are still not open and even EOOX is only partially open with closed binary code still incorporated for its full implementation.

In that regard, I note that the patent license for the necessary Microsoft Open Packaging Conventions for EOOX has now progressed beyond a draft stage and is now in final form. See http://www.microsoft.com/whdc/xps/pkgpatentlic.mspx

As I explained in my in-depth article soon after the Ecma standardization effort was announced, that license is absolutely incompatible with free and open source licensing requirements and specifically excludes licensing for "general word processing, spreadsheet or presentation features or functionality, operating system technology, programming interfaces, protocols, and the like." Moreover, the patent license prohibits sublicensing. Nothing seems to have changed in the those regards. See http://www.groklaw.net/article.php?story=20051129101457378&query=donnybrook#A3

I also note your statement on August 4th, ""I think any of you folks who've been frightened by some of the FUD that has been spread about the Ecma Office Open XML formats should take a look" at a linked "legal opinion" on EOOX intellectual property issues.
http://blogs.msdn.com/brian_jones/archive/2006/08/04/688932.aspx

I hope that your statement about FUD was not directed at my article. I bent over backward to approach the subject from a neutral standpoint. And the "legal opinion" Microsoft commissioned does not seem to address a solitary point made in my legal analysis. So far as I know Andy Updegrove and I are the only people who have actually dissected the relevant documents as a whole and published our analyses. My own article was couched as suggestions for Microsoft to improve and clarify its relevant legal documents. I believe that both of us took a principled approach to our discussion. So I hope that you might clarify whether you intended your remark to encompass my article.

Best regards,

Marbux

re: Interoperability of the Office Open XML formats

eshwar — Sat, 23 Sep 2006 08:11:41 GMT

first of all great blog. you are really doing a fine job of providing insights into the new office 12 xml based formats. i agree with you it was in microsoft's interests to move to the new format since if you would want to have developers building solutions on top of office you will have to move from the binary format since supporting that format would be a nightmare.

but you have to agree with the fact that governments and many large corporations played a huge part in your decision as well. atleast for the ecma standardization part wouldn''t have been done without pressure from some govts. but whatever it is, i think office is moving in the right direction. i hope to see many cool applications developed on top of these new formats. i hope to soon find some time to play with these new formats.

once again great blog and continue the good work.

re: Interoperability of the Office Open XML formats

davidacoder — Sat, 23 Sep 2006 13:27:24 GMT

"Has something changed? My understanding is that the apps are too brittle for the surgery required to remove the binary formats and replace them with the XML formats, that the Ecma Office Open XML ("EOOX") format is part of a translation layer from and to the binary formats used for internal processing. Has a decision been made to perform the necessary surgery nonetheless?"

Are you suggesting that Office should use EOOX as its internal data format at runtime in memory?!? If yes, what a nonsense request. I hardly know ANY app that uses its storage data format as the data format for its runtime data structures. You would get the slowest app you could possibly dream of. Almost all apps use different data structures from their storage format at runtime to get good performance. Just imagine what a slow dog you would get if everytime someone types a character in Word you need to actually parse and change your XML, if you use that as your data structure... But, once you accept that the runtime data structure just simply HAS to be different from the storage structure (and by the way, this is almost 100% true for OpenOffice as well), then of course the storage format (i.e. EOOX) is different from the binary representation of the document in-memory at runtime and you have a translation going on when you open or save the document.

You are not seriously requesting that any app that claims to support an open format HAS to use that format as its runtime in-memory strucutre, are you? :) Loading and saving in the open format should surely do.

re: Interoperability of the Office Open XML formats

davidacoder — Sat, 23 Sep 2006 13:38:17 GMT

"And what is this about viewers for other operating systems? Has Microsoft decided to open source the Windows dependencies in EOOX such as OLE objects? Is it not a fact that EOOX implementation on non-Windows operating systems poses enormous development barriers for some document features?"

There are a couple of points with regards to OLE. With OLE, essentially ANY third party app can embed data into a Word document. Word (or MS) does not control the format of this embedded stuff in any way. So you can ALWAYS end up with a Word document that has embedded OLE data (say from some third party graph editor, whatever) that is in a format no one but this third party app understands. And of course many of those apps will only exist on Windows. So, how could a viewer on Linux show a Word document with an embedded OLE part for which there is no software for Linux to decode that embedded stuff? Well, fairly simple. For every embedded OLE object Word also saves an image (I believe in WMF or EMF or something format, certainly something documented) of how this embedded object looked at save time. So, to just view the Word document, you DON't need to understand the binary format of the embedded object, you can just render the image that is saved along.

I think this is as ideal as you can get it. Things can be viewed on different platforms, and if there is an editor for the embedded format for Linux or another platform, it could possibly also be viewed.

The one thing Microsoft could not have possibly solved is to rule out these binary, non-standard format data islands coming from embedded OLE objects in the new EOOX format. They come from third party apps, not MS apps. MS can't force them to use any particular format. So, they had two choices: Ban embedded binary OLE data (and therefore break compatability with millions of old documents that just have them) or do what they did.

How would you have solved this problem differently?

re: Interoperability of the Office Open XML formats

hAl — Sat, 23 Sep 2006 13:49:58 GMT

@marbux
It would be more usefull to use OOXML or Ecma OOXML for the format as that is the termenologie used by Ecma itself.

Firstly I do not see why the internal processing of a format would be of interest to you. XML in it's essence is a horrible internal format as it is cluttered with tag's that have no use in the internal structure of a program. Only idiots use internal XML structures.

Secondly I found this in your writings:
"Such problems are compounded by the lack of definitions for the word "conform" and its variant "conforming." If read restrictively, those terms could reasonably be interpreted as prohibiting implementation of subsets and supersets of the Office Open XML specification."

However the Ecma OOXML draft clearly has a passage that states conformance to the format and those clealry allow subsets of the format if documented as such and also the draft discusses extensibility therefore allowing supersets as well.
I lack simular information in de OpenDocument specs so that it legally unclear when a document that is a subset or a superset can be called OpenDocument. This is potentially bothersome as OpenDocument seems an OASIS foundation trademark that might not be allowed to be used when you aren't conforming (allthough it is not clear when a format does). So if anything it seems rather unclear when a document can be called OpenDocument and rather a lot clearer when a document can be called OOXML.

re: Interoperability of the Office Open XML formats

Anonymous — Sat, 23 Sep 2006 17:06:04 GMT

@Marbux:

"In that regard, I note that the patent license for the necessary Microsoft Open Packaging Conventions for EOOX has now progressed beyond a draft stage and is now in final form. See http://www.microsoft.com/whdc/xps/pkgpatentlic.mspx"

I've seen this nonsense before. One of the classic FUD attacks being perpetrated by OpenOffice zealots is to state that while the OpenXML format itself *may be* open, developers would still have to separately license the patent for Open Packaging Conventions. The link for XPS is provided to give the FUD some veracity.

In my understanding the Open Packaging Conventions is an integral part of the OpenXML - (http://www.ecma-international.org/news/TC45_current_work/tc45-2006-335.pdf) and fully covered by Microsoft's Covenant Not to Sue.

More Marbux:

"the file formats used internally for Office are still not open and even EOOX is only partially open with *closed binary code* still incorporated for its full implementation."

Oh no! Is it you Gary Edwards???

re: Interoperability of the Office Open XML formats

Bryce Leo — Sun, 24 Sep 2006 17:55:43 GMT

I'll be honest, I just don't like Microsoft Office. I happen to use AbiWord, or the online ZohoWriter, because I just don't need that massive amount of *stuff* that's inside of word, and I don't need all of that powerpoint *stuff* either, i just use S5, and heck I don't even touch spreadsheets and if i want a database i'll use MySQL and a php front-end thank you.

You guys have a product that's designed for a specific type of user, and if you look at OpenOffice there's still tons more things that the current MS office versions can do that it can't. I couldn't imagine being able to squish all of the odd fucntionality and whatnot into the ODF format rather than one of your own formats. I don't really like that you have your own format, but I can see why. You've got a great tool too, but for the very small subset of features that I use I don't need it, just like I don't need Windows either. Hell I use PuppyLinux and it does all of mydevelopment (PHP, Python, Perl, C++) which is just fine for me. It's all about needs some people need all of that, and some people don't. It shouldn't make what you do any less impressive. It's a shame that it does. Keep up the great work.

re: Interoperability of the Office Open XML formats

John — Sun, 24 Sep 2006 18:12:09 GMT

Hello, all

it was a huge surprise that my post has appeared as a main article and i have to say big thank you to brian for carefully answering the points. And i admit that he really did explain thoroughly some things - again, great work and nice move from Microsoft and its employees, without irony.

Let me also state again - I am a network administrator of Active Directory domain, knowing and using MS Office since version 97. Currently running 2007 Beta TR, and hats off, its awesome. I also believe (after you explained that my concern of somewhat documentation is false) that the documentation for OpenXML will be really complete, not as in past version of Windows and DOS which contained undocumented features that only your software could use. But ...

Brian, your arguments are perfectly valid from your point of view as a developer and evangelist of the new formats. Surely, they are a huge step forward - I completely agree.

And, if

- Microsoft would not have near monopoly in Office applications
- the history of your behavior concerning "interoperability" was not so perfectly known all around the globe

I would not argue any further. But this is not the case. I am simply afraid, that the problems that I experience daily in my network, that the users are unable to work with their .DOC documents except when they created in the same version of Word. Between OpenOffice and Word this is even worse. And this happens with the simple documents just containing graphics, some wordart and formatting. They just do not look the same and my users are angry about it. That is the interoperability how Microsoft currently does it. And the result of it - when i want to read or edit DOC received in mail properly, I have to buy MS Word and MS Windows. Hmm - that is the best marketing you could have. Will this be better with the new formats??

Somebody here in the forum wrote, that the key of interoperability is making things simple. I accept you point, that making DOC, XLS, PPT documents working is not easy job, especially when all those documents floating around. But, how can i know, that the complexness of the documentation is driven solely by those real needs, or if it is driven also by need to make those formats so complex that nobody in the world will be able reproduce them as a whole?

Well, I am afraid, despite all the good effort beign put in the new formats, that the simple thing - making a document in Word or Excel and being able to work (that means - editing) with them with software other that MS Office will be as difficult as it is now. I bet that at the same time many companies will build solutions on top of new formats, which could be opened in MS Office. The thing is not about viewing, the key is the editing of documents and that is the main concern I have.

I could have argued much further. I hope that I made clear where are my main concerns. I am not nearly as experienced as Brian in this area, I simply have big problems with interoperability now and still I am in doubt that they will be solved.

Sincerely,

John

re: Interoperability of the Office Open XML formats

Gareth Horton — Sun, 24 Sep 2006 19:13:32 GMT

John,

We had no difficulty obtaining the full spec for the Excel BIFFx file formats from an MS Press book and our customers have exported and read in many millions of Excel files over the years. Admittedly there were various errors in that documentation, but you would be insane to say they were put in on purpose to obfuscate the format.

You figure the issues out pretty quickly (if you do QA that is!)

There are also other 3rd party native BIFF writers on the component side such as ExcelWriter etc, whose developers have gone through the same process.

Even the <;)>unmentionable</;)> OpenOffice does a pretty good job in reading and saving binary Excel.

MS also made it easy to read and write Excel files with DAO/ADO/OLEDB etc, so admittedly, not multi-platform solutions, but they were hardly trying to lock-in people to the Excel app itself.

So whatever some people and organizations come out with, you will ALWAYS at least be able to get all the underlying data out of any version of any Excel file, even if MS goes out of business next week, all it's source code (and escrowed stuff) and internal documentation is destroyed in a fire, etc etc.

We have already developed support for reading and writing OpenXML spreadsheets in the next version of our app and are awaiting the final nailing down of the spec and the documentation of the XLSB / XLSM formats, which MS have promised to deliver, as well as the documentation on encrypting XLSX.

It was a hell of a lot easier than the old Excel BIFF, that's for sure.

We are not a huge company and have bunches of other features to deliver for our main product, so the fact that we were able to do this in a couple of months (even creating our own Packaging IO engine in C++, rather than using the .NET ready rolled one) speaks well to the new formats. As Brian correctly points out, you don't have to support the ENTIRE spec for it to be valuable.

Brian: I hope you or David G will let me know if you get any corrupt / odd files that were generated by our products. I seriously doubt it, since we have not had a support issue on it since Excel had a smaller market share than Lotus;)

I would also like to thank Brian and David Gainer for their excellent blogs over the Office 2007 release cycle. There has never before been this level of quality information flowing about an unreleased version of Office in Microsoft's history.

Gareth

re: Interoperability of the Office Open XML formats

hAl — Mon, 25 Sep 2006 15:02:13 GMT

John, have you seen interoperability between applications using opendocument ???
That is currently a disasterarea as well.
If you edit an ODF wordprocessing file, created in OOo, in KOffice you are likely to loose parts of both text en formatting. Having the specs does not guarantee interoperability especially not with complex formats.
So the lack of interoperability not just due to lack of documentation (allthough that documentation certainly was not very good) but also due to a lack of communication between the different implementations about the formats.
And still today you have several parties that do not communicate with MS and OOo communication going mostyl trough weblogs like this...

Lost in translation

Office Rocker! — Tue, 26 Sep 2006 00:37:51 GMT

Chris Capossela (head of the Information Worker business group) was over in the UK today and I got to...

Lost in translation

Office Rocker! — Wed, 04 Oct 2006 13:26:01 GMT

Chris Capossela (head of the Information Worker business group) was over in the UK today and I got to