Brian Jones: Open XML Formats

Looking for some good reading on Open XML?

Wow, can't believe it's been over a month since my last post. I've been pretty busy getting back into the Office 14 project, the recently released Open XML APIs, as well as the Office 2007 SP work (where we'll be introducing the ODF support). I haven't had as much time to stay on top of the blogosphere, and also haven't been able to write much myself. For folks interested in following along with interoperability and file formats in Office, I would suggest subscribing to Doug Mahugh's blog. He's a member of Ecma TC45, and has also recently joined the OASIS ODF committee.

Today Doug has a post discussing the updated binary format documentation that was recently released.

I'll definitely aim to continue this blog, but most likely the frequency of posts will be fairly low (maybe once a month?). In the mean time though, here are a few great blogs:

• Doug Mahugh – Doug is helping to take over a lot of the file format and standards related work I was doing. Doug has been involved with the Open XML work for a couple years now and definitely knows his stuff. I'll continue to work closely with Doug and will definitely follow his blog closely.
• Gray Knowlton – While Doug will talk from the standards and development point of view, Gray is focused on the Office products. Gray's blog discusses the resources and opportunities available to folks interested in building their own Office business applications. Open XML is just one of the tools available to developers building Office solutions, and Gray plans to discuss them all.
• Eric White – Eric is on the developer evangelism team and his blog is really the ultimate place to go for hands-on info about building solutions on .NET that take advantage of Open XML. He already has a huge collection of great tools and code snippets. You should definitely check it out if you haven't already.

-Brian

More on yesterday’s ODF announcement

Gray Knowlton had a great post yesterday laying out more of the details on the "Save as ODF" functionality that will be built in natively for Office 2007 SP2: http://blogs.technet.com/gray_knowlton/archive/2008/05/21/microsoft-adds-save-as-odf-to-office-2007-service-pack-2.aspx

Even though ODF 1.0 is the official ISO version of the format, the decision was made to use version 1.1 which had some accessibility improvements. Hopefully future versions of ODF will be brought back to ISO so that we have a more current ISO version soon. You also may have noticed that we as Microsoft plan to start participating in the ODF standardization efforts, whether that happens in OASIS or ISO.

We hope to work closely with the community to help ensure we make the right design decisions in our ODF implementation. We will focus heavily on interoperability and observe how other products implement the standard, but we will also make sure that our priority one goal is to follow the standard itself. So if there are areas that aren't quite clear in the standard we'll look to other products like Open Office and how they do things. I've heard different opinions though on what to do if the standard says one thing and Open Office does something else. I think the right thing to do is to follow the standard, but I'd be curious to hear what other folks have to say. Doug Mahugh's blog is a great place to start conversations around these types of issues, and I know he'll be posting a lot more material around our design philosophies and areas where we hope to get feedback.

Gray and Doug both blogged yesterday with all the details:

• Gray Knowlton: Microsoft adds "Save as ODF" to Office 2007 Service Pack 2
• Doug Mahugh: Office support for document format standards

-Brian

ODF support in Office

I took a few weeks off recently, which is why my blog has been fairly inactive the past month. I wanted to take a quick moment to point out some big news that's been in the works for some time now. Today we announced a number of developments in the document formats arena. Doug Mahugh has all the information up on his blog: http://blogs.msdn.com/dmahugh/archive/2008/05/21/office-support-for-document-format-standards.aspx

The key bit of news that I think many folks will find surprising, but is actually fairly consistent with the direction we've been heading the past couple years is that we will provide updates to Office 2007 to include native ODF support. This will allow folks to read and write ODF files, and even set ODF as the default format if that's something they choose to do. Here's a screenshot from Doug's blog:

-Brian

ISO takes full control - SC34 now moving forward on maintenance and evolution of Open XML

I'm heading home from Norway in the morning, but wanted to give a quick update on the progress made over the past few days. I was attending the SC34 meetings in Oslo and there was a good amount of focus on document format standards (SC34, as designated by JTC1, is the maintenance body for Open XML). In the meetings, we had some great discussions around the next steps for ISO/IEC IS29500 (Open XML), as well as 26300 (ODF), and the topic of interoperability/harmonization. Today was the day that all the resolutions were accepted, and they are now public: http://www.itscj.ipsj.or.jp/sc34/open/1025.htm

I think Alex Brown covers it best, as he declared on his blog: ISO committee takes full control of OOXML

Three new working groups

SC34 was thinking about how to best handle the evolution of document format standards, and intends to create three new working groups to be created within SC34. The first working group would work on Open XML; the second would work on ODF; and the third would work on interoperability/harmonization between document format standards: http://www.itscj.ipsj.or.jp/sc34/open/1025.htm#Res4

I think that a number of folks have been interested in how the work will be handled now that Open XML has been transferred from Ecma to SC34, and I think you'll be pleased to see the public resolutions made at the meeting.

SC34 decided to start by first organizing the working group for Open XML, and in order to do so the following two ad hoc groups will be established:

Ad Hoc Group 1

This group will be led by Alex Brown, and the goal will be to create the official working group for the Open XML formats. There will be regular phone meetings over the next 6 months as well as a face to face meeting in the summer. It is expected that the structure of the working group they establish will also serve as a template for a working group to manage the ODF formats (should OASIS agree to pass maintenance on to ISO). Here's some background on the creation of Ad Hoc Group 1:

"SC 34 is the JTC 1 designated maintenance body for ISO/IEC 29500 (Office Open XML file formats).

The passage of ISO/IEC 29500 has instituted a new era of standards activity in SC 34 related to document formats. ISO/IEC 29500 does not represent an isolated phenomenon, since SC 34 is also responsible for ISO/IEC 26300 and for interoperability between these and other projects.

SC 34 envisages the creation of three distinct working groups that meet the needs of:

1. ISO/IEC 29500
2. ISO/IEC 26300
3. Work on interoperability/harmonization between document format standards and wishes to incorporate existing expertise on these standards.

For these reasons, SC 34 hereby establishes an ad hoc group pursuant to the JTC 1 Directives, clause 2.6.2, for investigating how the first of these groups may be set up most effectively."

Ad Hoc Group 2

This group will be led by Murata Makoto, and will try to quickly set up an infrastructure for collecting feedback on the Open XML standard. There has been a lot of progress made over the past year, and since there are still open issues that have not been fully addressed, they don't want to lose any of that information. The goal is also to use these same tools for allow for public comments to be submitted, and for the public to follow along with the comment resolution process. This work will then feed directly into the working group established by ad hoc group 1. Here's a bit more info on this group:

• "Definition of the task:
  • To define and put into operation a mechanism to compile a list of comments on ISO/IEC 29500 received from NBs, liaisons, and the general public.
  • To publish the on-going list as an open document on the SC 34 website.
• Time frame: The collection mechanism is to become operational within 90 days from the end of the April 2008 ISO/IEC JTC 1/SC 34 plenary. Once this is operational, collection will continue until a long-term maintenance process is operational.
• Membership: Open to ISO/IEC JTC 1/SC 34 P and O members, liaison organizations, and subgroup representatives.
• Convener: Makoto Murata (JP).
• Meeting arrangements: Work will be handled primarily by email, with optional telephone conference calls at dates and times to be announced."

Ecma's participation

Another important decision was that SC34 will seek to preserve the knowledge from TC45 members and will invite TC45 to fully participate in both Ad Hoc Group 1, Ad Hoc Group 2, and any future working group that is involved in the maintenance of IS29500:

"ISO/IEC DIS 29500 (Office Open XML file formats) has received the necessary number of votes for approval as an ISO/IEC International Standard. SC34 recognizes that Ecma TC45 members have in-depth knowledge, technical expertise on ISO/IEC 29500 and will seek to preserve and allow for inclusion of this existing body of technical expertise in SC34. SC34 therefore invite Ecma TC45 members to attend and fully participate in ISO/IEC JTC 1 SC 34 Ad Hoc Group 1, ISO/IEC JTC 1 SC 34 Ad Hoc Group 2 as well as in any future working group that will be dedicated to the maintenance of ISO/IEC 29500. SC34 intends to organize an efficient and timely process for maintaining and handling new work items to insure the evolution of the standard in following the JTC 1 Directives."

Interoperability/Harmonization

The DIN Delegate (DIN is the German national standards body) presented an update on the work that they have been doing around translation between the Open XML formats and ODF. I've discussed this a number of times before as being a key piece of the harmonization work.

DIN presented this to SC34 because they are going to propose a new work item within SC34 and are currently in the process of asking other countries to join them in this work. We were informed that AFNOR (France's national standards body) already told DIN that they will work together. It really seems to me that we are seeing some great movement and momentum on interop between formats led both by the thinking within SC34 of creating a new WG on interop and also clearly by this latest initiative of DIN and AFNOR.

Busy Week

As you can see it was a busy week, but we continue to make great progress. I said in my last post how much I was looking forward to working with the folks in SC34, and this meeting just reaffirmed those beliefs.

-Brian

Open XML Overwhelmingly Approved as an ISO / IEC standard (IS 29500): the end of the file formats war

I'm sure many folks have seen the news by now that Open XML has been approved as an ISO/IEC standard (IS 29500). Based on the numbers I've seen, looking at the P member countries there are now 24 who vote "yes", and only 8 vote "no". This puts the P member approval at 75% easily passing the 2/3 majority needed. Of the overall votes (both O and P members combined) 61 countries votes "yes" and only 10 voted "no" which puts the overall approval at 86% (so only 14% no). This puts us well below the minimum bar of no more than 25% voting "no". So on both criteria, Open XML now easily passes, which is a great indication of the general positive feelings amongst the national bodies of the progress made over the past 6 months.

Now that the voting over, it's time to move forward and start to work together in the ongoing development of these document format standards. There has been a lot of energy focused on the review period over the past year or two, and we need to use that same energy to move us forward. There is still a lot of work to do in order to make it even easier for developers to build solutions using these standardized technologies (new tools; test suites; labs; etc.). We also need to continue looking beyond traditional documents and identify the important innovations that will be necessary for the documents of the future. I may have been a bit premature last year when I declared the file format wars over. It was a couple days after we saw that Open Office was going to have Open XML support, and I thought at that point folks would start to move on to the more collaborative and mutually beneficial investments. Well, I was a bit premature I think, but now a year removed from my initial statements, I think we've reached the milestone that really will help put a lot of the tension to rest. Open XML has been approved as an ISO standard, and we can now switch our energy back to the technical work that will continue to drive things forward. As we move into the next stages I'm excited to see the energy and knowledge that will be brought to the table as we begin to innovate and move both Open XML and ODF forward as important internationally standardized file formats.

Large numbers of implementations already support Open XML

Open XML has already been developed on numerous platforms, by hundreds (if not thousands) of different implementers. The approval of Open XML as an ISO standard gives those implementers a stable platform on which to build their tools and solutions. We've already committed in Microsoft that we will work on updating our products so that they support the ISO version of Open XML, and I'm sure we'll see others make similar updates to their solutions.

Choice in file formats will always be important

I know you've heard me mention numerous times that choice in file formats is an important thing. Whether it's XHTML, PDF, ODF, UOF, DAISY, DocBook, NLM, RTF, .doc, or Open XML, folks have needs that drive the file format they choose. Last year we sponsored a translator project that gave people the ability to read and write ODF files from Microsoft Office. Last month we announced that we would update the Office product so that the ODF translators could natively plug into Office and give people the same options they get from the other file formats. People will be able to set ODF as the default format in Office if that's what they want by simply installing the translators and then changing their settings. There will be people that take this option, just as there may be others who decide to switch over to the old binary formats as their default for the time being. I believe the vast majority of folks will use Open XML as their default format, but ultimately that's just my opinion. What's important is that everyone has the ability to decide.

The future of documents, and the ongoing development of IS 29500

I have to admit that what I'm most excited about is that we can now start to move beyond the basic discussions of file formats as they relate to what are essentially digital typewriters, and start to move into the future of document content. The custom schema support in Open XML is really just the starting point of semantic documents, and it takes a small step in the new voyage we need to help convince the rest of the world to take. For far too long, we've focused simply on how to present document content. How it's formatted, where page breaks are, and what styles are used. We've only begun to scratch the surface though in terms of the actual semantics behind the documents people create. There are brilliant folks out there who've been doing a lot of thinking around the semantic web, and how to really tie together all the important information that affects our lives. The next challenge is to really identify how you get the average document author to write content that is semantically structured. Most folks don't yet see the advantage in structuring their documents, so it's important to find ways of providing immediate benefit to those that take the time (or use the right software). There are a number of experts in this area on SC34, so it's very fitting that many of the same people that have helped contribute to this area will also participate in the future developments of Open XML. In ISO it's called "maintenance" but I think that term sounds a bit limiting to folks. It's not "maintenance" in the way that you maintain your car so that it runs properly. Of course some of the work will be around corrections and general improvements, but a lot of the maintenance work will be innovative and forward thinking. We need to continue to move document formats forward, and I couldn't think of a better group to take on that responsibility.

-Brian

Open XML Resources for Developers

Doug has a great post today that helps get us back to what really matters in this whole file format discussion (at least if you're a developer). He links to a number of existing resources that can help folks who are just now getting started with Open XML development, as well as more advanced links for folks who've already started working with the formats: http://blogs.msdn.com/dmahugh/archive/2008/03/31/open-xml-resources-for-developers.aspx

Like many people, I thought we'd know the official outcome of the DIS 29500 process today, but it looks like we won't hear the official results until after ISO has had a chance to run them by the national bodies who participated in the review of the specification, which according to Reuters will be Wednesday.

While we wait, I've been thinking about how much attention this process has been getting, especially in recent months. Back when Ecma submitted the ECMA-376 standard to ISO at the beginning of 2007 (451 days ago, if my math is right), a relatively small number of people were following the discussion around document format standards. That group has expanded significantly, and there are now many people following the story of Open XML and DIS 29500.

Since some of those people may be developers who didn't see all of the Open XML content that has been made available in the past, I decided to pull together a list of links to various resources for Open XML developers. The list is included below. I'm sure I've left out a few good resources, so please let me know in the comments if you know of a useful Open XML developer resource that I've not included here.

-Brian

German standards body responds to "incorrect and misleading" reports

Via Stephen McGibbon: http://notes2self.net/archive/2008/03/31/din-responds-to-quot-incorrect-and-misleading-quot-reports.aspx

-Brian

Norway vote is now “yes” for Open XML

Norway has also decided to change their vote to "yes" for DIS 29500 (Open XML). http://www.standard.no/imaker.exe?id=18594

Norway had initially voted "no with comments" in September, so it's great to see that they now feel Open XML is ready to be an ISO standard.

[Update 4/1/08: Standards Norway (the folks actually responsible for making the decision) has an official response to some of the FUD: http://notes2self.net/archive/2008/04/01/standard-norge-responds-to-allegations.aspx]

-Brian

South Korea vote is now “yes” for Open XML

Oliver Bell just informed me that South Korea has decided to vote for approval of DIS 29500 (Open XML) as an ISO standard. This is a change from their original "no with comments" posisiton they took back in September, and is great news. South Korea was definitely influential in some of the bigger decisions made in the BRM, and it's great to see that they are now in favor of ISO approval.

Oliver has all the details: http://osrin.net/2008/03/28/south-korea-votes-approve-for-isoiec-dis29500/

-Brian

Denmark vote is now “yes” for Open XML

Jesper Lund Stocholm is reporting that Denmark has decided to change their initial vote of "no" in September to "yes" for ISO approval of DIS 29500 (Open XML). Here is a pointer to the press release: http://www.ds.dk/4225

As Stephen McGibbon put it: "Denmark is a ISO/IEC JTC1 P member and has changed from 'Nej med kommentarer' to 'Ja'."

[Update 3/28 @ 1:21pm] Jasper H. Bojsen has a blog post in English giving all the details: http://blog.hvorom.dk/post/2008/03/From-a-Triangle-to-a-Square---What-Happened-in-Denmark.aspx

-Brian

Finland vote is now “yes” for Open XML

Here's the official statement, although it's in Finnish: http://www.sfs.fi/ajankohtaista/tiedotteet/20080327195555.html

Finland had initially voted "abstain" back in September, but they have now decided to change their vote to "yes" in support of DIS 29500 (Open XML) as an ISO standard.

-Brian

Another example of Open XML as a submission format...

Just saw this on Pablo's blog:

Covering yet another set of scientific disciplines, last month the folks from ArXiv (the largest repository for Physics, also very popular for Math and Computer Science papers) posted the news that they now accept OpenXML files for submissions.  I am also looking forward to being able to help scientists and researchers that deposit OpenXML files into ArXiv.

-Brian

The OSP will apply to DIS 29500 (Open XML) going forward

Oliver has a post today discussing this, as it's come up a lot in the discussions he's been having: http://osrin.net/2008/03/27/the-osp-will-apply-to-future-versions-of-dis29500/

-Brian

Miguel de Icaza’s views on Open XML, and what the next steps should be

Interesting post from Miguel today: http://tirania.org/blog/archive/2008/Mar-26.html

I have been reading the OOXML storm in a teacup for more than a year now. Am looking forward to the approval of OOXML as an ISO standard and to be able to move the discussion back to the things that actually matter: free and open source software.

For a year, countless bytes have been wasted on what is now a very difficult plot to follow, specially for people that have not followed it since the start (or as Bill Maher said last week "Its like trying to make sense of a LOST episode". Note: am a Lost fan).

When we go back to what matters, we should be ready to ask IBM to open source its Lotus Notes software based on Open Office (as staunch supporters of open formats, and open source, I think its in their best interest to do so). There are nice components in Lotus Notes that would be nice to integrate into the upcoming OpenOffice 3.

-Brian

Video clip of the add-in for authoring Scientific Articles

If you want to get an idea of how the add-in works, Pablo has posted a short video:

 Pablo suggested skipping to about 4:40 into the video to see the actual solution. For more information on the add-in, check out the blog: http://blogs.msdn.com/exscientia/archive/2008/03/25/short-video-clip-on-the-add-in.aspx

-Brian