Doug Mahugh

WordCamp Victoria 2012

2012-01-16T20:01:10Z

Over the weekend, I attended my first WordPress event: WordCamp Victoria 2012. It was a great opportunity to learn more about WordPress and meet a bunch of people who are passionate about WordPress, blogging, travel, food, beer, sailing and other topics. (Hey, we didn’t just talk about work!) For more information about WordCamp Victoria this year, see the #WCV12 hashtag on Twittter.

If you’re wondering why I’d be at a WordPress event, the answer is that I’ve moved to a new role here at Microsoft on the Interoperability team, and in this role I’m working with a variety of open-source technologies that run on Windows Azure, including WordPress. If you didn’t know that WordPress runs on Windows Azure, check out this article on the InteroperabilityBridges website: “How to deploy WordPress using the Windows Azure SDK for PHP WordPress scaffold.” We’ve also published guidance on deploying Wordpress Multisite, and we’ll be continuing to work with the WordPress community to improve the Windows Azure deployment experience for them going forward.

My colleague Ben Lobaugh, the WordPress expert on our team, gave a presentation at WordCamp entitled “The Proof is in the Pudding – Yep it works with Microsoft.” You can find Ben’s slides here, and he covered several topics of interest to WordPress developers including WebMatrix and the BizSpark and WebsiteSpark programs that help developers and designers get free tools, support and visibility for building websites and other applications. I learned about WebMatrix’s support for WordPress from Ben last week, and used WebMatrix to design a new personal blog that I deployed to Mahugh.com during this trip to WordCamp. The experience was simple and straightforward – WebMatrix handles installing and configuring everything you need, so that you can just concentrate on the look and feel and content of your site.

It was fun to meet so many interesting people, and I’m looking forward to returning to WordCamp next year. Thanks to Paul Holmes for organizing and running a great event!

Here are a few photos from the event, including the spectacular ferry rides between Seattle and Victoria (by way of Bainbridge Island and Port Angeles) …

OASIS ODF 1.2 Approved

2011-10-01T20:47:00Z

As expected, OASIS members have approved ODF 1.2 as an OASIS Standard. Although the official announcement from OASIS hasn’t been posted yet, Rob Weir (chair of the OASIS OpenDocument Technical Committee) has a blog post with information about ODF 1.2 and those who have contributed to it.

As Rob notes, many people from around the world representing many organizations contributed to ODF 1.2. Here’s a Wordle of the attendees of every ODF Technical Committee (TC) call in the final year of ODF 1.2’s development:

Microsoft has supported ODF since Office 2007 Service Pack 2 and formally joined the ODF TC in early 2008. . I’ve attended many meetings myself, and my colleague Cherie Ekholm, Senior Standards Professional, has been active in the ODF TC as well as the OIC (ODF Interoperability and Conformance) TC since 2009.

We’ve also had some great support from others in broader Microsoft Office team who have made significant contributions to ODF 1.2:

Eric Patterson, a Senior Program Manager on the Excel team, has been a regular attendee and voting member in the ODF TC for more than three years, working with the Open Formula Subcommittee(SC) to finalize the specification of ODF spreadsheet formulas. Eric submitted 45 issues related to functions, referencing and syntax, worked with the SC to resolve those issues, provided detailed information about Microsoft Excel behavior in response to various requests, and worked with other implementers to improve formula interoperability.
David LeBlanc, Security Architect for Office, has worked with the ODF TC to ensure that XAdES digital signatures are specified in an interoperable manner. He helped ensure that the standard is consistent with the XMLDsig and XAdES standards and worked to establish an excluded path, so that implementers may maintain unsigned metadata in the archive without breaking the signature. He also helped tighten up the standard to avoid allowing overly broad transforms that can cause security problems.

Going forward, we’ll continue partnering with the ODF TC on the next version of ODF, informally referred to as ODF v-Next. Another colleague of mine, John Haug, Standards Professional, has been working with the Advanced Document Collaboration SC for the last nine months to help figure out how to address change tracking in ODF v-Next, and he has written a proposal for extending the existing ODF change tracking, including many specific examples of how to handle the use cases that the SC has defined for change tracking.

Even as work on ODF v-Next proceeds in OASIS, ODF 1.2 is expected to be submitted to ISO/IEC for consideration as a new ISO/IEC standard. This will provide the global standards community with an opportunity to review the standard out of OASIS, provide any comments and suggestions for improvement they may have, and ultimately decide whether to ratify it. ODF 1.2 may be modified by this work, but we have every reason to be optimistic that ODF 1.2 will be ratified by ISO/IEC at the end of the process.

At Microsoft, we are committed to ensuring that Office customers are able to choose the format that best meets their needs. As a result, we will continue to give them a range of format options, including ODF. We are also sensitive to our customers’ need for long term usability and minimal cost when it comes to using new versions of the formats we support. That is why we are active in the maintenance of ODF, PDF and OpenXML and tend to want to support stable versions of formats that have emerged from the standardization process.

We look forward to continuing our participation in the evolution of ODF. Congratulations to the entire ODF community on the approval of ODF 1.2 as an OASIS standard!

New DII website locations

2011-07-13T17:59:50Z

I’ve heard from a couple of people who have noticed that the Document Interoperability Initiative website (www.documentinteropinitiative.org) started redirecting to a new site recently (http://www.microsoft.com/openspecifications), so thought I’d provide some info to clarify where to find various types of information going forward. All of the same content related to document formats is still available, but we’ve consolidated things so that it all lives on Microsoft.com under the Open Specifications site and the Microsoft Office File Formats site. This structure will allow us to keep things well organized as more content is added and updated in the future, and that will make it easier to find the content that you need.

You can find general information about DII on the Open Specifications site, as well as a recap of past interoperability events. (We’ll also be posting some information about upcoming events soon, and I’ll cover that in a separate blog post.)

The implementer notes for Office’s implementation of Open XML and ODF used to be available in an interactive format on the DII site, but we heard feedback from implementers that they wanted them published in an all-in-one PDF that would be easier to work with offline. Those changes are all done, and the PDFs are available for free download on the Microsoft Office Protocols website. There you can find implementer notes for ECMA-376, ISO/IEC 29500, and ODF.

Finally, if you have technical questions about any of the content on these sites, you can get help from our support engineers and others from the implementer community over on the Interoperability Forums.

Microsoft Office Binary File Format Validator

2011-07-13T16:43:20Z

My colleagues on the Office Interoperability team have been hard at work on a tool that will help implementers of the Office binary formats identify and correct any issues in binary documents created or modified by their software. The Binary File Format Validator Beta is now publicly available here, and Michael Bowman has a blog post covering the details. In Michael's words:

BFFValidator runs as a command line without a dependency on the Office product. BFFValidator is targeted towards binary file format developers and generates prescriptive failure information when a given file does not conform to the Open Specifications. The tool will generate an XML log file that will correlate the issue with a specific section of documentation, the stream offset where the failure occurred, and other pertinent debug information. To learn more, please check out the BFFValidator reference file on the dowload page and stay tuned for a "How to guide" to be published on MSDN.

If you're working with the binary formats, this is a great tool for assuring the best possible interoperability with other implementations. Check it out, and if you have questions about the tool you can get those answered at the Microsoft Open Specification support alias, dochelp@microsoft.com.

libOPC version 0.0.1 released

2011-04-20T00:18:14Z

The first release of libOPC, a new API for Open XML development, was published on Codeplex last week. This API is the first open-source cross-platform API for developers working with Open Packaging Convention (OPC) packages as used by Open XML, XPS, and other formats. Full source code is available, and it’s written in portable C99, so can be used on all popular variants of Linux/Unix, Mac OS, Windows, Android, and many other platforms. The API uses other common cross-platform open-source APIs for some of the low-level details, including ZLIB for opening ZIP-compressed packages and libXML for parsing the XML streams from the parts in the package.

Historically, there have been two popular .NET APIs for Open XML development: System.IO.Packaging (which first appeared in .NET 3.0) and the Open XML SDK, released in early 2007. There’s also a COM-based native packaging API available for non-.NET Windows developers.

The libOPC API is roughly analogous to System.IO.Packaging, in that it’s focused on the details of OPC and MCE (parts 2 and 3 of IS 29500), but doesn’t provide higher-level abstractions for WordprocessingML, SpreadsheetML or PresentationML (as covered in parts 1 and 4 of IS 29500). I say “roughly” because libOPC doesn’t yet address some of the things that System.IO.Packaging handles (e.g., digital signatures) but does include some more advanced capabilities not available in System.IO.Packaging, such as the opc_generate functionality described below, which is essentially the same as the document reflector functionality of the Open XML SDK.

The key new feature in libOPC is its cross-platform capabilities. If you’re working on a non-Microsoft platform, or working with embedded systems that have limited OS support for XML and ZIP, you now have a very fast, simple API that you can use to implement Open XML read and write capabilities in your applications. And libOPC is designed from the ground up to be wrapper-friendly, for use from programming languages other than C.

The coordinator of the libOPC project is Florian Reuter, a well-known voice in the document formats community through his years of work with the OASIS ODF TC, Ecma TC45, OpenOffice.org and LibreOffice. Florian has worked with both ODF and Open XML, as a Sun employee, a Novell employee, and (now) an independent developer, and this gives him a well-rounded perspective on what sorts of tools can help developers to be more productive when working with XML-based standardized document formats. Microsoft is sponsoring Florian’s contributions to libOPC, and I can’t think of a better person to be leading this project.

The first release of libOPC is mostly about providing abstractions that make it easy for developers to work with OPC packages. There are several basic tasks that you need to do over and over when working with OPC packages, such as finding a relationship by type, determining the target of a relationship, retrieving the content of a part in a stream, and so on. The libOPC API provides simple methods for carrying out these tasks, and the developer doesn’t need to think about details such as how a package’s relationship tree is serialized into various XML parts.

There’s a video on the libOPC Codeplex site in which Florian goes over the basics of what’s available in version 0.0.1. One of the things he demonstrates is opc_generate – a tool that takes an Open XML document as input and generates a cross-platform C program that will re-create that document through use of libOPC. This is a very powerful tool for building customized document assembly systems, because the code it generates can provide a big head start for certain types of projects.

In addition to enhancing developer productivity, opc_generate can enable better collaboration between graphic designers and developers. A graphic designer can create a beautiful document without giving any thought to how to generate that document programmatically, and then a programmer can generate source code for that document instantly, then focus on customizing that code to attach it to data sources as appropriate. My colleague John Haug did an informal test of opc_generate by taking all of the documents in our internal SharePoint document library and generating libOPC code, then running the code to re-generate the original documents. They all worked great, including some quite complex documents.

On the documentation page, you’ll find information about how to use libopc on Linux, MacOSX, Windows, iOS, and Android, as well as demo videos showing libopc in action with WebKit and the iPhone. The WebKit demo uses the Open XML standard itself as an input, so it’s a great example of the raw performance that libOPC delivers.

If you’re familiar with the Open XML SDK, you may wonder whether libOPC will eventually include the sorts of higher-level functionality that the Open XML SDK provides. This is an interesting question, and I’m really looking forward to seeing how this project evolves going forward. Just as .NET developers started with the System.IO.Packaging API and then the Open XML SDK was built on top of it, we may see other higher-level APIs and tools built on top of libOPC. If you have thoughts in this area, or are interested in contributing, please get involved and join the libOPC project on Codeplex.

Release V2 of Open XML document generation system

2011-03-30T14:40:00Z

Eric White posted yesterday release V2 of his simple Open XML document generation system, a high-performance tool for generating customized documents from any data source that can be serialized as XML.

This tool is part of Eric’s ongoing series of posts on generating Open XML WordprocessingML documents. In his approach, you configure the document generation process by creating a template document, then inserting content controls in the template document, and then entering appropriate XPath expressions in the content controls. Those XPath expressions specify where to find the data that will populate the content controls for each custom-assembled document at runtime.

For an example of this approach in action, check out the two-minute screencast on Eric's blog post, which demonstrates generation of 3000 documents in less than 30 seconds. The source code for this system is less than 230 lines total, and provides a great starting point for those who want to do something more complex. For many common applications, however, this example already includes everything you need.

On a related note, Eric has recently started writing how-to articles for the OpenXMLDeveloper.org web site, so if you’re doing Open XML development you’ll want to follow those as well. Many of his articles are inspired by questions posted on the Forums section of that site, so be sure to post your Open XML development questions there, and if the answer merits a more comprehensive response he’ll write up an article illustrating the concepts involved.

XML Prague 2011

2011-03-29T10:15:07Z

I attended the XML Prague conference over the weekend, where I had the opportunity to see some great presentations, meet some interesting people, and drink plenty of Czech beer. The latter being a requirement for any trip to Prague, of course!

The conference was scheduled on the weekend between meetings of the W3C XSL working group last week and the ISO/IEC JTC 1/SC34 working group and plenary meetings that are taking place this week, which made it convenient for SC34 members to attend as part of the same trip to Prague. Today at the SC34 WG4 meeting we have several attendees who were also at XML Prague, including Jirka Kosek (Czech Republic), Mohamed Zergaoui (France), Murata Makoto (Japan), and my colleague Chris Rae and I, who are participating in SC34 as members of the Ecma TC45 delegation.

As you can see from the conference agenda, there were presentations on a wide variety of XML topics over the two days. As noted in the closing keynote (more on that below), there was a general theme of “XML in the browser” that cut through many sessions, as well as sessions on other topics such as EPUB, JSON, DITA, and others. The room was packed, with nearly every seat taken and extra chairs lining the aisle, for the opening session by Norm Walsh (above) that covered the history of HTML and XML and some of the use cases that have been considered by the W3C TAG HTML/XML Task Force. And the room stayed packed, all weekend long, with none of the drop-off in attendance many conferences experience.

After a full day of presentations on Saturday, we all walked up the hill behind Prague Castle to the Strahov Monastery brewery, for dinner, beers, and a demo competition. The view of Prague from the monastery was spectacular:

Sunday’s agenda was kicked off by Michael Kay’s “XML in the browser” presentation, which covered the history of XSLT and proposed how XSLT's declarative event-driven approach could be used to deliver the sorts of interactive functionality that has traditionally been the domain of JavaScript. Saxon CE, which allows you to run XSLT 2.0 in any web browser released in the last year, is now available in alpha release on the Saxonica web site.

There were many other great sessions, and you can see the videos when they’re posted on the web site, and/or download the conference proceedings to get all of the details. WG4 convenor Murata Makoto, wearing his EPUB hat (he is co-convenor of the SC34 ad-hoc group looking into EPUB standardization), provided an overview of EPUB and covered the current status of EPUB3.

Michael Sperberg-McQueen’s closing keynote was a masterly speech. He started with a story of Vaclav Havel characterizing the core of a crisis as being caused by cold impersonal machinery that was set against the aspirations of free-willed individuals. He then used this story as an allegorical backdrop for an exposition of the tension between the cold impersonal interests of the browser vendors and the original aspirations of many in the XML community, building up to an exhortation to put XML in the browser “whether they want it there or not.” I can’t describe it in a way that does it justice, but the video should be on the XML Prague web site soon, and it’s worth checking out.

Overall, XML Prague was a great learning experience for me, a 2-day crash course in the state of XML technologies and various creative ideas about how those technologies will evolve going forward. Thanks to Mohamed and Jirka for suggesting we attend!

Change Tracking in ODF

2011-03-16T19:19:11Z

When we implemented support for ODF in Office 2007, we decided to not support the limited change-tracking functionality included in that version of the ODF standard. I covered the thinking behind that decision in a blog post shortly after ODF 1.1 support was released in Office 2007 SP2.

I mentioned in that post that “we would welcome the opportunity to work with other TC members to improve ODF’s ability to handle tracked changes.” I reiterated that later the same year, when I spoke at OOoCon 2009 in Orvieto – in response to questions as to whether Microsoft would take an active role in helping to improve change tracking in ODF “v-Next.”

The time has come, thanks to the creation of the ADC SC (Advanced Document Collaboration sub-committee) of the OASIS ODF technical committee. The Statement of Purpose for the ADC SC emphasizes the importance of change tracking:

The initial and highest priority for the Subcommittee will be change tracking. Reliable and user-friendly revision management is critical for professional document workflows in corporate and public sector environments, and as such an important feature of Open Document Format.

Change tracking has indeed been the primary topic of discussion within the ADC SC, and per the “Roadmap for ADC SC work” on the ADC SC wiki, proposals may be submitted by SC members through the end of March.

Over the last couple of months, my colleague John Haug has been working closely with change tracking experts on the Word team to put together a proposal, and today we have submitted that proposal to the ADC SC. As stated in the Introduction to that proposal:

The basic idea behind this proposal is to preserve the existing ODF change tracking and extend it to support missing use cases. We believe this best preserves backward compatibility with existing implementations and makes best use of existing patterns and code. It also maintains the current model of ignoring changes for applications that do not support change tracking. A relatively small number of additions to the existing capability can enable a core set of previously unsupported use cases.

This proposal focuses on the need for robust change tracking in text documents, the primary user scenario based on customer feedback. In addition to supporting new use cases as described below, we propose expanding the prose that describes the current change tracking support to make its intended use and scope more explicit.

Our proposal is informed by our experience in building change tracking functionality over the course of several product releases, as well as our participation in the ADC SC discussions to date and our analysis of the existing ODF change tracking functionality when we implemented ODF support. We have identified the specific use cases that we feel need to be addressed, and for each of them we’ve provided examples of the ODF markup before and after the change, along with an explanation of our proposed approach. Of course additional use cases can also be addressed as the ADC SC’s work proceeds. And we look forward to digging in and understanding all other proposals submitted to the SC – the quality of the final work product will benefit from considering every submission carefully.

We have also included questions to the SC, and multiple options in some cases. As covered in the proposal itself, we see it as a starting point:

We believe that a robust discussion among implementers is necessary to create a good final design, we are open to changes in everything from design and applicable scope to element and attribute names and this document is provided as a vision for the general architecture and a seed for such a discussion. We expect that the mechanical details can be worked out in a straightforward manner once the subcommittee agrees on a broad architectural approach.

I’m excited to see the change tracking discussion taking place in the Advanced Document Collaboration SC, and like Rob Weir I hope that it will be every bit as successful as the Open Formula SC and Metadata SC have been in improving the ODF standard. It’s great to see the ODF community collaborating on addressing this critical area, and I look forward to some good discussions in the SC as the process moves forward.

Please get involved if you have an interest in this area, or expertise to contribute. Any OASIS member may participate in the SC, and if you’re not an OASIS member it’s easy to join.

Can progress be standardized?

2010-12-22T18:41:00Z

There’s a fundamental challenge that many standards maintenance groups and standards implementers deal with from time to time: how can standards evolve, and implementers innovate, while maintaining the sort of reliable and well-documented interoperability that standardization enables?

Standards maintenance processes add new functionality slowly and carefully, but the products that implement those standards may progress much more rapidly at times, to address new needs in the marketplace, new technologies, and ever-changing market dynamics. Is it possible to “loosely couple” a standard and its implementations in a way that allows implementers to deliver innovations while remaining interoperable and conformant to the standard?

I recently had the opportunity to work with a few of my Microsoft colleagues to put together a whitepaper (attached below) that touches on some of these issues. The paper was delivered at an interoperability event in Beijing earlier this month, and it provides an overview of MCE (Markup Compatibility and Extensibility, Part 3 of ISO/IEC 29500) and how it has been used to enable interoperability between various versions of the standard and its implementations.

There are several other strategies that have also been used to address this challenge in document formats:

A standard may allow foreign markup to appear inline. This gives implementers the freedom to innovate in any way they’d like, but it also means that documents can’t be validated against the published schemas in the standard. Removing the foreign markup is a necessary first step, and this can be difficult to do efficiently in some programming environments, or may be handled inconsistently across multiple implementations.
A standard may have defined extension points: places in the schema where an implementer can add non-standardized content to address new requirements, perhaps in the context of an xsd:any element in an XML schema. This approach makes it possible to validate the document against the schemas in the standard, but innovation can only occur in the places where extension points are defined.
A standard may not allow any extensions at all, and all innovation must occur through the standards maintenance process. This solves the validation problem, but is not always a realistic approach because of the slow pace of standards maintenance.

MCE is designed to address the limitations of some of these alternative approaches, and it has shown great promise for the types of issues that arise in the evolution of document formats. Each implementer can create their own MCE-based extensions, and documents containing those extensions can be gracefully handled by products that support varying sets of extension functionality (or none at all). MCE is not an unproven theory, but rather an approach informed by the things Office engineers learned from many years of dealing with these same sorts of issues in the binary file formats. We used MCE for all of the new innovations in Office 2010, and this has enabled interoperability between Office 2010 and other applications that don’t have that new functionality, as covered in the attached whitepaper. (For further information on Office’s use of MCE, see also the Microsoft Office File Formats Documentation on MSDN.)

As time goes on, of course, any new features used by more than one implementation should probably be integrated into the standard. The question of how and when to standardize these is one that WG4 have been discussing a lot this year. In October, the Japanese National Body submitted a proposal for a new multi-part standard containing standardized MCE-based extensions to IS 29500. Allowing standardized extensions inside MCE will permit implementers of the core IS 29500 standard to remain conformant while still allowing new features to be standardized.

Binary File Format Plugfest

2010-08-18T20:28:51Z

The Office interoperability team will host a Binary File Format Plugfest on October 19th and 20th in Redmond. The event will target a developer level audience. Microsoft subject matter experts from both the support organization and the product team will be onsite to answer questions about the Binary File Formats. This Plugfest will be a great opportunity for you to test your BFF implementations and receive immediate feedback and assistance from Microsoft. The event is free, and we will cover the doc, xls, ppt, and pst formats through a combination of presentations, 1:1 discussions, and workshops.

What to Expect:

Presentations on the Binary File Format Open Specifications (Word, Excel, PowerPoint, PST, Graphics, Document Cryptography etc.)
1:1 Sessions with product development team members
Access to Office's Binary File Format validation tool for Word, Excel, and PowerPoint
Interaction with Microsoft Support
Breakfast, lunch, and dinner will be provided on both October 19th and 20th

Space is limited, so register early. To learn more about the event or to register, please contact Michael Bowman.

Digital Signatures Workshop recap

2010-06-11T19:00:00Z

The latest DII workshop was focused on digital signatures, and took place here in Redmond on Tuesday of this week. This was the first event for which there has been a web participation option (via LiveMeeting), and we had 18 virtual attendees in addition to the 15 people who attended in person. The level of expertise in the room was very high, so it was a great learning experience for me, and I hope for others as well.

The presenters at this event represented a diverse range of perspectives, and included implementers, users, standards professionals, and policy makers. All of the presentations are attached below, and here’s a brief summary of each presentation …

John Marchioni of ARX was first up, with a presentation on the scalability and control benefits of a centralized approach to key management. ARX was one of the contributors to the development of the OASIS DSS (Digital Signature Services) standard, and their CoSign® product was the first implementation of DSS for centralized key management.

The final slide of John’s presentation simply said “What do you think?” but John didn’t have to wait nearly that long for feedback. Early in his presentation, a vigorous debate sprung up about the tradeoffs between centralized and distributed approaches to key management. Cindy Cullen (Safe BioPharma CTO), for example, expressed her concerns about the assurance level that can be obtained in a server-based approach, and others weighed in from the implementer, user and public-policy perspectives. It was a good example of the value of getting a group of experts together to discuss complex issues that don’t lend themselves to a one-size-fits-all solution.

Next up was Ernandes Lopes Bezerra, General Coordinator of Standards and Research at Brazil’s ITI (National Institute of Information Technology). Mr. Bezerra delivered a very comprehensive presentation on the technical, legal, and procedural details of the use of digital signatures in Brazil. It’s clear that ITI has done a great deal of work in this area, and the use of digital signatures is rising rapidly. As an example of that, the Brazilian PKI initiative issued over 126,000 digital certificates in March of this year, after averaging about 30,000 per month through the last few months of 2009.

Gerson Rolim was the next speaker, and he is both the Executive Director of the Brazilian e-Commerce Chamber and Brazil’s coordinator for the Digital Mercusol project. Digital Mercusol is a collaboration between the Mercusol block countries (Argentina, Brazil, Paraguay and Uruguay) and the European Union spanning a variety of technology areas including eID and digital signatures. Mr. Rolim explained how key investments in PKI (public key infrastructure), time stamp infrastructure and a regulatory framework are enabling cross-border e-commerce throughout the Mercusol block and with trading partners around the world.

Office’s support for digital signatures was the topic of the next presentation. Shelley Gu, Program Manager for Office’s Trustworthy Computing team, covered the types of digital signatures supported in Office 2010, and explained how Office supports RFC 3161 (Time-Stamp Protocol), RFC 2560 (Online Certificate Revocation Protocol), XML–Dsig, and XAdES. She then demonstrated Word’s UI and general approach to digital signatures, as well as the new XAdES support in Office 2010, using a simple rental agreement as an example.

Chris Brotsos, Program Manager on the InfoPath team, demonstrated how InfoPath supports EDRM (the Electronic Discovery Reference Model). EDRM defines a set of e-discovery operations and a process for applying those operations, and is used by e-discovery consumers and providers. After the demo, Chris and Shelley led a Q&A session on topics related to digital signature planning for the next version of Office. It was great to get this type of well-informed feedback from digital signature users as we begin planning for the next version of Office.

After a delicious lunch (which we couldn’t figure out how to share with the LiveMeeting participants, unfortunately), Dennis Hamilton (NuovoDoc) gave his presentation on Document Security Anti-Patterns. This was an entertaining talk on the dangers of inadvertently creating patterns of action, process or structure that appear to be beneficial but ultimately produce more bad consequences than beneficial results. Dennis started with simple generic examples and then went through several anti-patterns related to document security and digital signatures. He noted in particular the risks of adding complexity in the pursuit of security.

Cindy Cullen, CTO of Safe BioPharma, covered the use of digital signatures in healthcare and pharmaceuticals. Safe BioPharma is engaged in creating standards for these industries, and has a goal of creating a fully electronic global business environment that addresses the technical, legal, and regulatory requirements of working with the FDA and other entities such as the DEA and NIH. It was impressive to see the level of complexity that arises in these sorts of highly regulated activities; this slide is a good example:

The final speaker of the day was Microsoft’s David LeBlanc, the developer responsible for leading Office’s implementation of digital signatures. The bulk of his presentation was a review of the history and architecture of the XAdES standard, including explanations of the various levels within XAdES. He followed this up with some information about the work he has been participating in within the OASIS ODF TC on how XAdES and encryption will be specified in ODF 1.2.

Each of the sessions was recorded, and we’ll be posting this recordings over on the DII web site in the next few days. I’ll follow up with a link to them when they’re live.

In addition to the presentations, we ended the day with a roundtable discussion of various topics. This was a good free-flowing conversation that went beyond the time we had allotted, and some of the conversation continued over beers later. A few of the topics I found interesting included:

The suggestion that Office consider OASIS DSS
A continuation/expansion of the debate about the tradeoffs between centralized and distributed key management
Digital signature issues and options for mobile devices
Discussion of the UI challenges related to partial signing of documents
The need for audit frameworks to verify CFR Part 11 compliance
And a topic everyone agreed on: the most important emerging digital signature standard is XAdES

Thanks to all of the speakers and attendees for a very informative and interesting event.

Agenda for June 8 Digital Signatures Workshop

2010-05-26T20:10:42Z

I blogged recently about the upcoming digital signatures DII workshop, which will take place in Redmond on June 8. We’ve been working with a variety of volunteer speakers to finalize the details, and it’s shaping up to be an interesting event with perspectives from implementers, standards participants, and digital signature users from the private and public sectors.

Here’s the agenda:

Time	Topic	Presenter
8:45 – 9:15	breakfast in Building 20
9:15 – 9:45	Welcome, Logistics	Doug Mahugh, Microsoft
09:45 – 10:45	Public Policy on Digital Documents and Electronic Signatures in Brazil	Mr. Ernandes Lopes Bezerra, ITI
10:45 – 11:00	break
11:00 – 12:00	Office 2010 Digital Signatures	Shelley Gu and Chris Brotsos, Microsoft
12:00 – 13:00	lunch
13:00 – 13:30	Document Security Anti-Patterns	Dennis Hamilton
13:30 – 14:15	Digital Signatures in the Healthcare and Pharmaceutical Industries	Cindy Cullen, SAFE BioPharma
14:15 – 14:30	break
14:30 – 15:15	Presentation on Digital Signatures	David LeBlanc, Microsoft
15:15 – 16:30	Roundtable discussion

Based on feedback from various people who were interested in attending but couldn’t travel for the event, we’ve made arrangements for virtual participation for any of the sessions, or throughout the entire day. (Well, we haven’t figure out how to serve lunch virtually, but other than that …)

So if you’d like to attend, either in-person or virtually via LiveMeeting, please use Amruta Gulanikar’s contact form to register. Attendees who are registered for the LiveMeeting option will receive an email with all of the details prior to the event.

SharePoint Interoperability Plugfest

2010-04-28T21:37:50Z

Here’s another Plugfest event coming up here in Redmond to let everyone know about. It’s a SharePoint interop event, where developers can test interoperability, get answers directly from the SharePoint team, set up up 1-on-1 meetings, and generally do whatever’s need to improve interoperability between their applications and SharePoint.

My colleague Michael Bowman is running this event, and in his words:

The SharePoint interoperability team is planning their first file operations Plugfest on June 15^th and June 16^th 2010. This event will cover the file operations protocols that support file access, synchronization, and collaboration scenarios. Microsoft subject matter experts from both the support organization and the product team will be onsite to present and answer questions about SharePoint protocol interoperability. The event will include a combination of presentations, one on one discussions, and interoperability testing. The Plugfest is a great opportunity for you to test your implementations and receive immediate feedback from Microsoft or learn more about the file operation protocols.

The event is free but space is limited so be sure to register early. We will be taking feedback on the agenda up until May 20^th, but we suggest that you register as soon as possible. To receive the registration form and agenda survey or to learn more about the event, please contact Michael Bowman.

If you’re working on an application that uses FSSHTTP, WebDAV, FPSE or other file operation protocols to interoperate with SharePoint, contact Michael and take advantage of this free workshop.

Digital Signatures DII Workshop

2010-04-27T14:32:57Z

The next DII workshop will be on June 8, 2010 in Redmond. This event will focus on digital signatures and a new technical signing standard supported by Office 2010, called XAdES (XML Advanced Digital Electronic Signatures). XAdES consists of a set of extensions to the XML-DSig standard (supported by Office 2007) and gives users additional features, like the ability to timestamp signatures or store certificate and revocation information with the signature.

The goal of this event is to share information about Office’s support for XAdES and related standards and solicit feedback on how we can work together to improve interoperability in this space in the next version of Office.

Here’s a general outline of what the workshop will include:

Presentations from members of the Office product team that give an overview of digital signatures and how they are used in Office.
Presentation on XAdES support in Office.
Presentations from consumers of Office digital signatures and other experts on how they use digital signatures, what standards they prefer to work with, why they are important, etc.
Roundtable discussion time, so that we can hear your feedback.

The agenda will be fleshed out in more detail as the event draws nearer. Also, if you have a digital signatures or document format interoperability topic that you would like to present, please let us know. This is a community event, and we'd like to see many voices and many perspectives involved.

To register for this event, please send Amruta Gulanikar your name, email, and company/organization via this contact form, and we'll follow up with more details including hotel recommendations and related information.

Office’s Support for ISO/IEC 29500 Strict

2010-04-06T17:26:15Z

There has been some interest expressed lately regarding how soon Microsoft Office will offer full read/write support for the Strict conformance class of ISO/IEC 29500. I can certainly understand the interest in this topic from those involved in the standards process, as well as from our customers and other implementers. That’s why we’ve been looking into the issues and options for Strict support for quite some time. Many of you have observed our movement in this direction. Indeed, a member of WG 4 blogged about our progress toward Strict recently.

We generally don’t publicly discuss features this early in the product development lifecycle, but given the broad interest I’m going to share some of our thinking on Strict support here. We’re doing this to assure everyone involved that we understand – at all levels within Office – the importance of Strict support going forward. In short, we will support Strict no later than Office “15.” I’ll outline our general plans below, and ask you to stay tuned for more details as we get further into the Office 15 wave.

Conformance: Background and Jargon

Before I cover the topic at hand, I think it’s worthwhile to take a quick look at how conformance is defined in the standard itself. There are two versions of the Open XML standard that have been approved by standards bodies:

ECMA-376 was approved in 2006 by Ecma International. Ecma is a consortium standards body like OASIS, with members including implementers, vendors, public and private organizations, and individuals.
ISO/IEC 29500 was approved in 2008 by the member bodies of JTC 1, the joint technical committee of ISO and IEC responsible for development and maintenance of information technology standards. ISO and IEC are international standards bodies whose members are mostly countries. (To be precise, ISO/IEC members are the national standards organizations of various countries, such as ANSI in the US, BSI in the UK, or AFNOR in France.)

The ECMA-376 standard was submitted to JTC1 as a DIS (Draft International Standard) in 2007. Many countries (“member bodies”) participated in the standards process, and the version that was approved as ISO/IEC 29500 in 2008 included many changes that were suggested by the member bodies and then approved at the BRM (Ballot Resolution Meeting) in February 2008. For purposes of this post, the key changes to note are those in the conformance clause, which describes how to determine conformance to the standard.

In ECMA-376, two types of conformance were described in Section 2 of Part 1 of the standard: document conformance (Section 2.4) and application conformance (Section 2.5). These were just what you’d expect from their names: document conformance was about how to determine whether a document conforms to the standard, and application conformance was about how to determine whether an application conforms.

In ISO/IEC 29500, assessing conformance is more complicated because of several changes agreed to at the BRM that made conformance more granular than in ECMA-376.

The key change, which Alex Brown covers in his blog post, was the introduction of the concept of Strict and Transitional conformance classes. Transitional is intended to preserve the fidelity of existing binary documents being migrated to ISO/IEC 29500, and includes many legacy features for compatibility with existing documents. Strict is a subset of Transitional that does not include legacy features – this makes it theoretically easier for a new implementer to support (since it has a smaller technical footprint, so to speak), but also makes it less able to preserve the fidelity of existing documents.

Another conformance-related change at the BRM was the creation of separate conformance classes for word processing, spreadsheet and presentation documents and applications within both Strict and Transitional. So, for example, an application can be a conforming WML (WordprocessingML) Transitional application, or a document can be a conforming SML (SpreadsheetML) Strict document.

Yet another expansion of ECMA-376’s relatively simple approach to conformance was the addition of application descriptions, as covered in Section 2.6 of Part 1. An application may conform to either the Base Application Description (meaning that it supports at least one feature of its conformance class) or the Full Application Description (meaning that it supports every feature within its conformance class). That’s a pretty coarse distinction, but the standard anticipates refinement of application descriptions in Section 2.6.3, which states that “It is expected that additional application descriptions will be defined within the maintenance process for ISO/IEC 29500.” Indeed, SC 34/WG 4 (the working group tasked with maintenance of the standard) has discussed this concept just two weeks ago during the meetings in Stockholm, where Mohamed Zergaoui (representing France’s AFNOR) presented some thoughts on this topic. I expect that WG 4 will work to clarify and refine the conformance language of the standard going forward, and we look forward to participating in that process.

Office’s Approach to Open XML Conformance

Office 2007 was the first version of Office that supported the Open XML formats, with support for reading and writing of documents that conform to the ECMA-376 standard. To help improve interoperability between our implementation and others, we also published comprehensive implementer notes that transparently document the details of Office 2007’s implementation of ECMA-376.

After we shipped Office 2007, we got to work on the next version of Office, which was code-named “Office 14” but is now widely known as Office 2010, the version that we’ll be releasing very soon. For each new version of Office, we start with intensive research and planning to determine what new features will appear in the next release, and that process was ongoing during the DIS 29500 process. By the time of the BRM (in early 2008), we had our plans locked down and were working hard to deliver on Office 14, and meanwhile the standards community was working to make changes to the proposed DIS 29500 standard.

After approval and publication of final ISO/IEC 29500 text in 2008, the Word, Excel, PowerPoint and Graphics teams looked at how we could change our plans for Office 14 to accommodate the ISO/IEC version of the standard. As Shawn covered in a blog post one year after publication of the standard, we made the changes necessary to support ISO/IEC 29500 Transitional in Office 2010 .

The decision to start with Transitional was a relatively simple one at that time. Our primary consideration was simple: the needs of our customers. Our customers place a very high value on compatibility and interoperability, because they often need to allow people to collaborate across multiple versions of Office (due to varying upgrade schedules among trading partners, across supply chains, or between the departments of a large organization, for example). ISO/IEC 29500 Transitional is designed for high-fidelity interoperability with the binary formats and ECMA-376, so it’s the logical choice for these sorts of scenarios.

In addition to the work we did to move from ECMA-376 to Transitional, we also started doing the work to move toward Strict support as soon as the final text of ISO/IEC 29500 was locked down. For example, we invested resources in migration from VML to DrawingML for many features, we moved ink annotations to the new content part added at the BRM, and we added support for reading Strict files.

All of that work took has moved us much closer to full Strict support, and I’d like to state clearly and unequivocally at this time that we will support reading and writing of ISO/IEC 29500 Strict no later than the next major release of Office, code-named Office “15.”

I emphasized “and writing” there because we have already built read-only support for Strict into Office 2010, and Strict read-only support will also be available for Office 2007 SP2 through a downloadable filter. We’ve taken those steps to assure interoperability between Office 2007/2010 and other implementations of the Strict conformance class, including Office 15 in the future.

There’s one technical change that has come up during the maintenance process which I feel is worth pointing out, because of its large impact on the move to Strict support. There was a defect report submitted to WG 4 by the Swiss technical committee last year that proposed changing the namespaces of ISO/IEC 29500, so that implementers could have a simple and reliable mechanism for distinguishing ECMA-376 documents from ISO/IEC 29500 documents. WG 4 started discussing and debating various ways to address that proposal over a year ago, and last summer reached consensus on changing the Strict namespaces, but not the Transitional namespaces. This resulted in a large number of changes to the text of the standard – for those interested in a good overview of the magnitude of those changes, check out Orcmid’s latest blog post.

Implementers, including Microsoft Office, will need to think carefully about how to handle the namespace changes in a way that gives customers the best possible experience. This is yet another challenge in planning support for Strict, and something the product teams are currently looking into as we start planning for Office 15.

Maintenance of IS 29500

Another topic that Alex raised in his blog post was the ongoing maintenance activity in WG 4, including progress to date, prioritization of the work, and other considerations. I’d like to briefly respond to his thoughts here, while acknowledging that WG 4 itself is the proper place for in-depth discussion and planning of the maintenance process. Any person from any SC 34 member body can participate in WG 4, so if you have thoughts on maintenance of ISO/IEC 29500, I’d encourage you to get involved.

WG 4 has existed for about 18 months now, and we have worked through a very large number of defect reports in that time. Although I’ve not participated in other JTC 1 working groups before, I’ve heard that the pace of WG 4’s work, with conference calls of up to two hours every two weeks, ongoing email on the public WG 4 reflector, and face-to-face meetings every three months, has been exceptional. Over 340 defect reports have been submitted to date, and WG 4 has processed and closed 242 of those, with 36 others in “last call” status (meaning that a defined solution is pending final approval by WG 4), and less than 70 awaiting further consideration.

Japan, the UK, and Ecma have been the largest submitters of defect reports to date, and defect reports have also been submitted by Denmark, Switzerland, Czech Republic, Ireland, and others. The maintenance process is proceeding smoothly, and we’ve handled changes ranging from simple editorial corrections to major proposals such as the namespace change mentioned above. Through it all, I feel that the WG 4 team has really gelled, and we’ve established a productive results-oriented working style that is well-suited to both the participants and the work at hand. Could we improve the process in various ways? Of course we can, and we will. But I think it’s worth noting that WG 4 has been very productive to date, with the first batch corrigenda already prepared, reviewed and approved, the first set of amendments in the pipeline, and work already underway on the next sets of corrigenda and amendments.

I’d like to keep the discussion of WG 4 procedures within WG 4 itself, since those are the people who will ultimately be doing the work. But as I said above, if you have thoughts on how to tackle maintenance, please get involved. Contact your National Standards Body for information about how to participate from your country.

Validating IS 29500 Conformance

What’s the best way to assess conformance to a large complex document format standard? This is a question that challenges the best and brightest minds in all of the standards organizations responsible for such formats, including SC 34 as well as OASIS, Ecma, and others.

As Jesper Lund Stocholm recently noted in a blog post about his new validator project, schema validation is the easy part. It can be automated, and there’s no ambiguity regarding whether a specific XML instance is valid against a specific set of schemas. The bigger challenges come when you try to validate the semantic and syntactic constraints that are embodied in the normative text of the standard.

Many people are working on how to best tackle these challenges in the world of ISO/IEC 29500, including Jesper and Alex’s validator projects, as well as the work being done by Fraunhofer and others. Here at Microsoft, we’re excited to see so much talent being applied to this area, and we’re looking forward to working with fellow WG 4 members and others to assess conformance in a way that the community agrees is best.

This post is already quite long, so I’ll not go into a lot of detail here except to note that there are two main areas where we expect to see useful results soon that will help raise validation testing to a new level of rigor and repeatability:

Identification of the semantic constraints contained in the text of the standard, so that all validators can work against a known-complete set of such constraints. Fraunhofer has done some interesting work in this area, to extract potential semantic constraints from the normative text, and we’ll be working with them to find a way to provide those constraints to writers of ISO/IEC 29500 validators.
Availability of a community-driven document test library, which implementers can use to test interoperability across conformant implementations of the standard. Fraunhofer has started this work, and there is much more to be done. Microsoft has contributed to this activity, and we’ll be staying closely involved.

Regarding the specific details of conformance, Alex noted that in addition to conformance issues caused by bugs in implementations, there can be issues caused by contradictory provisions within the text of the standard. Such contradictions can and do occur within various standards (both ISO/IEC 26300 and ISO/IEC 29500 have at least one of them, for example), and one of the goals of standards maintenance is to identify and correct such errors. The common pattern for such contradictions is that some portion of the standard will state that implementers shall do X, but there is text elsewhere in the standard (often text that was added later) which states that implementers may do Y in the same situation.

A strict reading of the text would lead one to conclude that such errors make conformance impossible. As a practical matter, however, implementers need to do something – they need to make a judgment call regarding the most reasonable interpretation of the intent of the standard in these areas. In the case of our IS 29500 implementation, we have done exactly that, and we’ve documented such interpretations within our published ISO/IEC 29500 implementer notes, so that everyone can see how we’ve interpreted the standard.

Separately, such errors need to be corrected in the standard. We are also contributing to that work. As one recent example, I wrote up a defect report myself while WG 4 was in Stockholm, to address an internal inconsistency regarding relationship types that Alex’s Office-o-tron validator had identified. We will work with the community to proactive identify more of these sorts of errors and get them corrected, and as part of my job I’m thinking through how we can best do that going forward.

One other detail that Alex mentioned was the use of the phrase “new documents” in the conformance clause for ISO/IEC 29500 Transitional. He noted that this term is not defined in the standard, and we agree that the intent of that term needs to be clarified. Here’s relevant text from the conformance clause:

“The intent […] is to enable a transitional period during which existing binary documents being migrated to DIS 29500 can make use of legacy features to preserve their fidelity, while noting that new documents should not use them. […]

One thing to note there is the word should, which is a well-defined term. RFC 2119 covers the use of key words like should/shall/must/may within the normative text of standards, and here’s how should is defined:

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

In the case of Office 2010’s use of Transitional, we have decided to prioritize compatibility and interoperability with existing implementations, because we believe this is in the best interest of our customers. So although the conformance clause says that Transitional “should not” be used for new documents, we have decided that the needs of customers, combined with the realities of the current document format ecosystem (most existing implementations are Transitional, recent major changes to the Strict namespaces), make Transitional the right choice. We will continue to update our plans in response to feedback from customers, other implementers, and the standards community going forward.

Summary

In closing, here’s where we stand:

In Office 2010, we’re providing read/write for Transitional and read-only support for Strict.
We will include write support for Strict no later than the initial release of Office 15. (More details will be forthcoming after we complete our planning.)
We are committed to continuing to work closely with the community on validation techniques, and we are actively using the available ISO/IEC 29500 validators to improve the quality of our implementation.

We’ve learned a lot from the IS 29500 standards process, and we continue to learn from the open and respectful exchange of ideas within SC 34 and the broader standards community. None of us have all of the answers, and many of the challenges that we collectively face are complex, but I’m confident that we can work through them and find solutions that address the needs of customers, implementers, standards workers, and other stakeholders.

And once again, I’d like to reiterate that if you have opinions about ISO/IEC 29500 maintenance, please get involved. I’m humbled by the level of expertise that WG 4 members bring to the table, and also by the commitment of those who volunteer large amounts of their own time to work toward improving the standard. I know I speak for every member of WG 4 in saying that we’d love to have even more participants involved.

SC34 Meetings, Stockholm

2010-03-30T00:32:00Z

The SC34 plenary and working-group meetings took place in Stockholm last week, hosted by Swedish Standards Institute (SIS). Alex Brown has posted already a summary of some of the meetings that I didn’t participate in, and I’ll cover the here the activities and outcomes of the meetings I attended.

WG4: ISO/IEC 29500 (Office Open XML)

WG4 met all day Tuesday and Wednesday, with a brief closing session on Thursday morning. We had 30 participants representing 12 national bodies (Brazil, Canada, China, Czech Republic, Denmark, Germany, Finland, France, United Kingdom, Korea, Norway and the United States) and 3 liaison organizations (Ecma, W3C, and XML Guild).

The bulk of WG4’s time was devoted to processing defect reports. Going into the meeting, we had 133 defect reports outstanding, including 82 technical, 11 editorial, and 40 requests for information. During the meetings last week we processed 61 of these. That leaves 72 outstanding, and WG4 will continue work on those via the teleconferences that will resume on April 8.

Here’s a summary of WG4’s processing of defect reports in Stockholm, grouped by the date ranges when defect reports had been submitted:

Separate from the processing of specific defect reports, WG4 also discussed some general topics relation to Open XML maintenance:

WG4 convenor Murata Makoto covered the basic concepts behind kihon-hanmen, a grid-based approach that is used in CJK (China/Japan/Korea) page layout. Japan has submitted a proposal to add this concept to IS29500.
Gareth Horton (UK/Datawatch) and Chris Rae (Ecma/Microsoft) have been collaborating on defining a profile for the use of 8601 in ISO/IEC 29500, and Gareth provided WG4 with an overview of that work. This led to an agreement to request creation of a new amendment (covered below under the plenary).
We discussed at some length Japan’s proposal for adding the Office 2010 extensions, which are implemented through the extension mechanisms described in the standard, into IS29500 itself. Some national bodies are in favor of this concept, some are opposed, and others would like to consider the extensions individually to determine which are appropriate for standardization. Discussion will continue on this topic, including the question of whether such extensions should be incorporated into a new Part 5, or each extension should stand on its own as a separate Part.

WG6: ISO/IEC 26300 (OpenDocument Format)

WG6’s work was mostly focused on the errata for ODF 1.0 and the creation of an amendment to ISO/IEC 26300 in response to OASIS’s submission of ODF 1.1 to JTC1. There was a full room for the meeting, with over 30 attendees including several participating by telephone.

As Alex mentioned on his blog post, Dennis Hamilton provided a great deal of support via his phone participation on the messy issues of text production. One nice touch was that Dennis had made available a marked-up copy of ODF 1.0 that showed the errata items inline, to make it easier for WG6 members to review those changes in their proper context.

SC34 Plenary

The plenary meeting itself was fast-paced and businesslike, with the main order of business being the approval of various resolutions (as listed here). Resolutions 10, 11 and 12 may be of particular interest to readers of this blog:

Resolution 10 created a new amendment for Part 4 of IS29500, which will be used to create a ballot for removal of ISO 8601 dates from the Transitional conformance class. Although 8601 dates were added at the BRM, some WG4 members have expressed concern about the possible interoperability problems that could arise for ECMA-376 and IS29500 Transitional implementations that don’t understand 8601 syntax.
Resolution 11 announced a workshop that will be held in Seoul in May to cover “CJK Issues related to OOXML and ODF Extensions.” A report from this workshop will be presented to both WG4 and WG6, with the goal of enabling a coordinated approach to CJK issues across both standards.
Resolution 12 created a “project subdivision” the amendment to ISO/IEC 26300 that will bring it into alignment with OASIS ODF 1.1.

Finally, Ken Holman, the former SC34 Secretary and longtime SC34 participant from Canada, announced his retirement from standards work, and he will be replaced by Paul Cotton as chair of the Canadian SC34 mirror. Ken said that he plans to crash future SC34 parties, though, so we’ll still be seeing him from time to time.

It was a busy week, but we got a lot done to move things forward in both Open XML and ODF maintenance. Next stop for the SC34 meetings will be Helsinki in June.

Standards, Patents, and the OSP

2009-11-25T03:24:49Z

Arthur C. Clarke famously wrote that “any sufficiently advanced technology is indistinguishable from magic.” Another characteristic of advanced technologies is that they often contain intellectual property (IP) that may be covered by patents.

When such a technology becomes standardized, though, implementers need to feel comfortable that they have access to the technology on a non discriminatory basis. This is typically dealt with in one of three ways: either the technology owner agrees to license necessary patents on a RAND basis, the technology owner’s agreement to license necessary patents on a RANDz basis or the technology owner agrees not to assert necessary patents. The reason international standards exist is to enable and facilitate interoperability between multiple implementations, and it would surely limit adoption if implementers had to worry about whether they could obtain access to IP that was necessary to implement the standards.

Some examples of companies’ agreements not to assert necessary patents are IBM’s Interoperability Specifications Pledge, Sun’s OpenDocument Patent Statement, and Microsoft’s Open Specification Promise (OSP).

In the case of the OSP, we’ve published a list of covered specifications, and we’ve made a public commitment that we will not assert patents that we own (if any) which are needed to implement those specifications. The list include ISO/IEC standards such as ISO/IEC 29500, standards from consortia such as Ecma and OASIS, and also documentation that Microsoft has published for various protocols and formats that we use in our products. The list of covered specifications is constantly growing, and includes not only the Open XML spec itself (both ECMA-376 and ISO/IEC 29500), but also many closely related technologies including the Office binary formats, the Office 2010 extensions, and the published implementer notes.

It’s important to emphasize that the protection of the OSP applies to implementers regardless of whether they perform any other action. You don’t need to file a request, provide notice, sign an agreement, or do anything else. You can simply get to work implementing any of the covered specifications, and know that you have free global use of any Microsoft patent necessary to implement those specifications. You also don’t need to consider the details of any specific patents, because the OSP’s protection applies to patents which Microsoft currently owns, patents that may be issued to Microsoft in the future, and patents that Microsoft may acquire in the future.

I’ve summarized the OSP here because I’ve been asked a few questions about it lately, and I thought it would be good to provide a simple explanation in layman’s terms of what the OSP is all about and how it works. That said, I should add the obvious caveat that I am not a lawyer. So if you’re looking to dig into the details of how IP issues are addressed in standards development, I’m no expert. But that’s the point, really: you don’t need to be a legal expert, or do any kind of legal analysis, to know that you can freely implement Open XML (and many other standards) without worrying about whether a particular Microsoft patent applies or not.

Open XML support in new tools, apps, and custom solutions

2009-11-18T23:38:53Z

One of the more interesting aspects of my job is meeting people who are developing software that interoperates with Office through the various formats that we support. It’s exciting to hear their plans, work with them on the details of how our products can collaborate or share data, and then see the final solutions, whether they’re released in shrink-wrapped packages, available for download on the web, or delivered as a custom solution for a specific organization.

Three of the more innovative firms I’ve had the pleasure of meeting and working with over the last few years are Altova, Datawatch, and PSC. Here’s an overview of the latest work each of them has been doing with the Open XML formats.

Altova

Altova’s MissionKit is a powerful suite of XML tools for developers and XML architects. I know them as an Open XML implementer, and also as a tool vendor. We use XMLSpy, DiffDog, and other Altova tools for all kinds of things on our team, and we’ve even used them for Ecma TC45’s work on maintenance of the Open XML specification itself.

Altova first rolled out Open XML support in April 2007, when they added to XMLSpy the ability to create DOCX, XLSX, and PPTX documents from scratch, as well as the ability to edit Open XML documents with context-sensitive help, auto-completion, and all the other productivity-enhancing features of XMLSpy. The latest release of their product line, Version 2010, was rolled out last month with even more Open XML support options, as well as a host of other new features.

StyleVision 2010, the stylesheet designer in the Altova lineup, now offers an entirely new design paradigm for creating XML forms that can be saved in DOCX and many other formats. The key breakthrough in this release is that StyleVision is now a true electronic form design tool in every sense of the phrase. Users can precisely position form elements on a canvas and specify templates within configurable layout containers, and they can design the form first, then add data-source connections later. This provides many benefits, not least of which is that a graphic designer or similar person can create a great-looking form and then a developer or XML expert can add the necessary technical plumbing later. If you’ve spent much time around e-forms, you know that that approach works much better than the other way around. :-) StyleVision still offers free-flow (HTML-style) form creation as well as the new absolute-positioning functionality, for scenarios where a free-flowing form is more appropriate.

Datawatch

Datawatch’s flagship product Monarch has been helping people create reports since 1991, and has supported the Open XML formats since February 2007, just two months after the publication of ECMA-376. The latest release (Monarch Version 10.5) includes several changes related to their Open XML support.

Monarch is a report mining and analysis tool that allows users to non-programmatically retrieve data from various information sources (usually reports in PDF, HTML, text or other formats) and create consolidated reports from that data. It is both a consumer and a producer of the XLSX/XLSM formats, and it is used by hundreds of thousands of users to generate reports from data sources that would otherwise be very difficult to consolidate or work with.

Version 10.5 includes support for digital signatures in XLSX/XLSM documents, improved PDF import, and other features, but the really interesting feature from an Open XML point of view is Monarch Context, a free add-in for Excel. Monarch 10.5 has the ability to take advantage of the flexibility of OPC (the Open Packaging Convention) to store a full-fidelity XML representation of the original source report in the output XLSX document. It also includes metadata on each row of the spreadsheet that ties back to the source of that row’s data in the original report. With Monarch Context (see screenshot above), you can then navigate to a row in in the generated spreadsheet, click on Display Source, and see the source data for that row.

This is a powerful example of the creative possibilities that developers have when working with OPC. The generated report includes its own audit trail and source data, and you can even ask Monarch to sign the XLSX/XLSM file, to assure future viewers of the integrity of the report for compliance and auditing purposes.

PSC

PSC Group LLC is a Chicago-area professional services consulting firm, a Microsoft Gold Partner, an IBM partner, and active in a wide range of technology services and projects. I’ve gotten to know John Head, PSC’s Director of Enterprise Collaboration, fairly well through DII workshops and other activities, where he is a perennial force to be reckoned with. John’s a hands-on expert and is definitely not the kind of guy who ever leaves you wondering what he really thinks -- it’s great to get that kind of direct informed feedback when we participate in DII events.

One of PSC’s clients is RDI (Research Director, Inc.), a Maryland-based radio-research consulting firm that analyzes, interprets and presents Arbitron audience research data for over 200 radio stations. RDI came to PSC with a problem: they had been using the same custom system for 10 years, and their needs had grown so much that it was taking three weeks to build the presentations they needed to generate each quarter. As Marc Greenspan of RDI explains, “Taking three weeks to process the data was no longer an option. By the time we’d be done, the next set of data would be coming in, and the presentations would be close to useless to our customers. We needed a scalable, sustainable production system, and our motivation was literally business survival.”

PSC developed a custom Open XML solution for RDI that now delivers these same reports in just three days, enabling RDI to offer its analyses 13 times per year instead of 4 times per year, and dramatically reducing the cost of generating their analysis. John Head explains that “We were able to improve the application performance so drastically by not having to automate PowerPoint, because it’s not involved until the user actually opens the file. We couldn’t have done that without Open XML. We couldn’t support document generation on the server with binary formats. It was too hard and it didn’t always work. Open XML changed that.”

There’s a new case study on our web site about the system that John’s team has built for RDI. It’s a good example of how the world of automated document assembly is rapidly changing with the advent of standardized XML-based document formats. I had the opportunity to learn about this system early on, when the PSC team gave us a demo in Redmond last year, and it’s great to see it rolled out and in production.

Microsoft

And finally, for those who haven’t seen the news yet … we have a new Open XML implementation available, too. More on that one later. ;-)

DII workshop, Brussels

2009-11-17T12:17:24Z

Last week I participated in the DII workshop that took place in Brussels. Attendees included a variety of document-format experts from the ODF and Open XML worlds, including members of SC34 working groups, the OASIS ODF and OIC TCs, ODF and Open XML implementers, public-sector experts in interoperability and archiving, and others.

The roundtable discussions at this event covered some interesting issues, including various approaches to round-tripping content through different formats, extensibility strategies and technical considerations, and future possibilities for the Strict and Transitional conformance classes of ISO/IEC 29500.

The workshop presentations are now available for download here.

ODF plugfest and OOoCon, Orvieto

2009-11-07T21:42:00Z

I’ve spent the last week in the city of Orvieto, perched atop a hill in Umbria, Italy. Monday and Tuesday I particpated in the second ODF Plugfest, and then Wednesday through Friday I attended OOoCon, the annual OpenOffice.org conference. I gave a presentation on Wednesday about Office’s approach to interoperability with OpenOffice.org, which you can find on the OOoCon presentation page, and you can find the presentations from the plugfest, as well as the test scenarios we went through, on the plugfest web site.

It was great to see everyone I had met at the last plugfest, and I also had the opportunity to finally meet in person many people I’ve only known via email and the ODF TC calls, including Svante Schubert, Charles Schulz, Louis Suarez-Potts, Eike Rathke and others. Everyone was great, and made me feel very welcome.

I was planning to do some sightseeing in Rome this weekend, but there is a train strike that begins at 21:00 today (Saturday), so I’m going to stay right here in Orvieto until Monday, when I’ll fly to Brussels for meetings and preparations for the upcoming DII workshop on Thursday, November 12. If you’d like to see the photos I’ve taken in Orvieto this week, you can find them on Flickr, and I’ve also included thumbnails of a few favorites below.

And now, after a long day of photographing the sights of Orvieto, it’s time to get out and enjoy some local cuisine. Buon appetito!

DII workshop – Brussels, November 12

2009-10-14T05:38:00Z

The next DII (Document Interoperability Initiative) workshop will take place in Brussels on November 12. As always, the goal of the DII workshops is to share information with the developer community and solicit feedback on how we can work together to improve interoperability.

Much has changed since the last DII workshop in Brussels, when we discussed Office’s future plans for ODF support and the pending rollout of the implementer notes, as well as the need for a validator and document test library to improve ISO/IEC 29500 interoperability. Now there are two versions of Office that offer built-in ODF support (Office 2007 SP2 and the Office 2010 Technical Preview), implementer notes have been published for ODF, ECMA-376, and ISO/IEC 29500, and validator and test-library projects are underway.

This workshop will include presentations on a variety of document interoperability topics. I'll blog the details of the agenda after it's finalized, but I wanted to let everyone know the date so those who are interested can make plans to attend. In the meantime, here are a few of the presentations that are already being planned:

I'll be covering the Office 2010 extensions (as was covered by the Office program managers in last month's DII workshop in Redmond). I will also present the latest news on how we’re working to improve ODF interoperability between Office and other popular applications, and talk about our plans for the future.
Alex Brown will be covering present and future plans for the Office-o-tron validator project.
Klaus-Peter Eckert of Fraunhofer FOKUS will present the latest status of the document test library project and other work Fraunhofer is doing to improve interoperability.

Like all DII workshops, this event is open to anyone and everyone, and there is no cost to attend. If you're interested in attending, let us know by sending an email to diievents@microsoft.com, and we’ll follow up with information about the venue, agenda, and other details.

On a more personal note, I know that many people in the document formats community (myself included) are photo geeks, and cameras may have outnumbered laptops at some of the standards meetings and interop events I’ve attended. I’ve been to Brussels twice before, but never had time to snap more than a few photos on the way between meetings, like the snapshots below. This time, however, I’ll be in Brussels a day early and am planning to get out and take a bunch of pictures on the Armistice Day holiday, November 11. So if you’re coming to the workshop and would like to go on a photo outing the day before, let me know. The more the merrier.

See you in Brussels!

DII Workshop on Office 2010 Extensions

2009-09-24T06:11:04Z

If you’re using the Office 2010 Technical Preview and you’re the type of person who likes to look closely at the markup of documents that you’ve created (as most readers of this blog are), you may have noticed some new namespaces that Word, Excel, and PowerPoint used for new functionality such as sparklines in spreadsheets or presentation sections.

Last Friday, at a DII workshop in Redmond, program managers from the Office product groups explained how these new namespaces are used in Office 2010. This event was scheduled for the day after the SC34 Plenary which took place in nearby Bellevue, so that interested SC34 members could easily attend. In addition to the SC34 attendees, several US-based ISVs were also present, which led to a good roundtable discussion in the afternoon with a variety of perspectives represented.

Office 2010 takes advantage of two types of extensibility mechanisms in ISO/IEC 29500: extension lists and ACBs (alternate content blocks). These extension points, which are documented in the text of the standard (see Part 3), provide implementers with a standardized way to innovate and add new functionality while maintaining conformance to the standard itself. The core concept is that an implementer can provide more than one representation for an object (a shape on a slide, say), and then consumers of that document can render the version that they understand.

The workshop featured four presentations on how Office 2010 uses ACBs and extension lists. Word/Excel/PowerPoint program managers (Zeyad Rajabi, Chris Rae, and Ric Bretschneider, respectively) explained examples from each of those products, and Nick Chiang of the graphics team covered some general concepts about how graphics are handled in Office 2010. The presentations included demos of various new types of functionality in Office, followed by deep-dive explanations of the markup used to serialize this new functionality.

All of the presentations are now available for download here, and documentation of the specific extensions used by Office 2010 can be found in the Microsoft Office File Formats section of the Open Specifications Developer Center on MSDN::

[MS-DOCX]: Word Extensions to the Office Open XML File Format
[MS-XLSX]: Excel Extensions to the Office Open XML File Format
[MS-PPTX]: PowerPoint Extensions to the Office Open XML File Format
[MS-ODRAWXML]: Office Drawing Extensions to Office Open XML

After the presentations on Friday, we had a roundtable discussion of the topics covered during the day, and a variety of related topics. Here are some of the points of discussion that I found most interesting:

Standardizing implementer notes. Should there be a standardized approach for documenting the details of a specific implementation of a standard? If a standardized schema were used by all implementers, it would be possible to build tools to work with these notes and search them or mine the data to identify possible improvements to the standard and trends among implementations. There was consensus that this is a topic worth further discussion and consideration.
Should the Office 2010 extensions be standardized? I was interested to see the range of opinions on this question. Some people feel they should, others feel that they should not, and others felt that further investigation and study is needed to determine what subset of these extensions may be appropriate or worthwhile for de jure standardization.
Test suites and profiles. We discussed various approaches to creating document test suites and standards conformance profiles that would enable better interoperability. Participants in this discussion included members of the OASIS OIC TC, SC34 WG4 and SC34 WG5, as well as software developers who (as John Head put it) “don’t care about standards, we just want everyone to implement the same thing in the same way.”
Should all implementations have published implementer notes? This topic was raised by a participant who asked about implementer notes for Mac Office, which currently don’t exist. Some felt that such documentation is needed from all implementers, and one person even suggested that publication of implementer notes could be a requirement for claiming conformance to a standard. Others felt that such a requirement might be an obstacle to adoption of the standard, given the magnitude of the effort needed to create comprehensive documentation for an implementation.
Can some things be standardized in a way that allows for their re-use in both ODF and Open XML? One of the attendees suggested that slide transitions could be standardized independent of the underlying document format, to allow them to be re-used consistently in more than one format. This would certainly simplify some aspects of translation between formats, but as others pointed out, it may be more effective to focus on translation between the existing standards rather than modifying each of them to support a new approach, which could be technically and politically challenging.
Do standards slow the pace of innovation? I hadn’t expected this topic to come up, but it prompted an interesting discussion of whether forward progress in document formats is limited by the fact that the major alternatives are all published international standards that must adhere to well-defined processes for their maintenance and evolution. The conclusion on this topic seemed to be that this is not an issue (i.e., standards don’t restrict innovation), but more could be done to educate implementers and users about how innovation and standards can peacefully co-exist.
What’s the best way to handle schema validation for documents that use MCE? MCE (Markup Compatibility and Extensibility – Part 3 of ISO/IEC 29500) requires a consumer to pre-process the XML markup before doing schema validation. We discussed how implementers can best address this requirement. Some felt NVDL might be the best tool for the job, and others suggested that there is a need for more examples of how to work with MCE, and tools that support MCE, across all platforms. This is another area we agreed needs further discussion and consideration.

It’s great to have so many experts in one room, sharing their thoughts on these topics, and I learned a lot from the wide variety of opinions expressed. What do you think? Do you have a strong opinion on any of these topics? Let me know in the comments below.

SC34 WG4 Meeting, Bellevue

2009-09-16T02:33:13Z

WG4 has been meeting in Bellevue, Washington the last three days, as part of a full week of activities around the upcoming SC34 plenary here this Thursday. We wrapped up the meeting with a half-day this morning (Tuesday), after meeting all day on Sunday and Monday.

WG4 covered a variety of topics in this week’s meetings. I won’t get into great detail on those, because some of these topics are still open and WG4 deliberations will continue in future calls and meetings, but here are a few of the more interesting topics we covered:

The past, present and future of Strict and Transitional; WG4 is working toward consensus on the long-term intent of this distinction, to help guide future work on new functionality and other maintenance activities.
The W3C “Widgets 1.0: Packaging and Configuration” candidate recommendation, and how it compares/contrasts to OPC.
Possible use of Assembla for tracking defect reports and maintaining the IS29500 schemas.
Open defect reports on media types, fonts, and custom XML, discussed proposed resolutions, and closed most of the DRs we covered.
Issues around ISO 8601 dates, including how to define a profile of the 8601 standard for use in IS29500, and the question of whether ISO 8601 dates should be removed from Transitional.
Various options for addressing the unqualified attributes issue that has been raised by the Czech Republic.
How future reprints and revisions to the standard will be handled, and how the upcoming changes to the ISO/IEC Directives will affect that work.

In addition to the 20-25 people in the room each day, we had a few people call in to contribute and participate. Especially noteworthy, in my opinion, were the phone contributions from Japan’s Toshiya Suzuki (who was on the phone from roughly 2:00AM to 9:00AM his time) and Denmark’s Jesper Lund Stocholm, who juggled LiveMeeting, a phone call, and a 2-month-old baby with aplomb. Thanks for calling in, guys.

We also had two presentations from invited Microsoft experts. Zeyad Rajabi covered custom XML, and Jeff Chen covered Markup Compatibility and Extensibility. This gave WG4 members an opportunity to ask questions of subject-matter experts in these areas, to help inform future maintenance activities.

Other SC34 working groups are also meeting here this week prior to the Thursday plenary, including WG5 (ODF-OXML translation), WG3 (topic maps), and WG1 (validation). This afternoon and tomorrow, I’m attending AHG3, the ad-hoc group looking into ODF maintenance and how SC34 and OASIS can work together on maintenance going forward. I’m going to also try to spend some time with WG5 if I can (which is meeting at the same time as AHG3), and then Thursday is the SC34 plenary itself, and Friday we have a DII workshop over in Redmond on the Microsoft campus. Busy week.

I’ll cover some of these topics in more detail after decisions are made in WG4 and/or relevant resolutions are approved in the SC34 plenary.

DII Workshop: MCE Deep Dive, Redmond

2009-07-21T22:49:59Z

We’ll be hosting another DII workshop soon, and this one will be of special interest to those who want to understand the inner workings of MCE (Markup Compatibility and Extensibility) as defined in Part 3 of ISO/IEC 29500. We’ve used MCE for new functionality in documents created by Office 2010, and members of SC34 WG4 have expressed interest in understanding the details of our implementation. So we’re planning a workshop in Redmond on Friday, September 18, the day after the upcoming SC34 plenary in nearby Bellevue, to do a deep-dive review of how Office 2010 uses MCE.

For those who aren’t familiar with MCE, here’s how it is described in the Scope clause of the standard (ISO/IEC 29500-3):

This Part of ISO/IEC 29500 describes a set of conventions that are used by Office Open XML documents to clearly mark elements and attributes introduced by future versions or extensions of Office Open XML documents, while providing a method by which consumers can obtain a baseline version of the Office Open XML document (a version without extensions) for interoperability.

By using MCE for the new functionality in Office 2010, we can deliver innovations like sparklines in Excel 2010 or new slide transitions in PowerPoint 2010, while maintaining compatibility with the Open XML standard. At this one-day event, members of the Word, Excel, PowerPoint and graphics teams will demonstrate these new capabilities and show how these new capabilities are stored in MCE alternate content blocks and extension lists.

The date for this workshop will be Friday, September 18, and the location will be Microsoft’s Redmond campus. If you��re interested in attending, please contact my colleague Amruta Gulanikar via this form to get on the list. Amruta can also provide information about travel and hotel options.

I hope to see you there!

Open XML developers: where to get answers

2009-07-10T04:10:05Z

I recently returned from a long business trip, and while working through my email backlog I’ve come across several questions from developers who are working with the Open XML formats. I’ve responded to each of them with some tips on how to best get such questions answered, and I thought I’d summarize that information here for others who may find it useful.

Your first stop for most Open XML development questions should be the forums section of the Open XML Developer web site. You can post a question there and it will be seen by the people who manage that site, and also by the broader Open XML developer community. Over 4,000 comments have been posted to those forums, so you can also learn quite a bit by reviewing the existing threads. And the library section of the site has dozens of articles with Open XML code samples in many programming languages, including Java, C#, C++, PHP, Ruby, Python, and others.

Another great one-stop shop for Open XML development topics is the Open XML Developer Portal. There you can find a huge number of code samples, articles, whitepapers, how-to videos, and free downloads, as well as links to many other resources. This site is focused on the needs of developers who are working with Microsoft’s tools, including the Open XML SDK and the System.IO.Packaging API.

For developers working with the Open XML SDK, be sure to check out Erika Ehrli’s recent summary of “Getting Started Best Practices” for SDK developers. You can also find a rapidly growing collection of great in-depth blog posts about Open XML SDK development over on Brian Jones’s blog, where Zeyad Rajabi has been covering a wide variety of common scenarios for Open XML developers.

Speaking of Microsoft blogs for Open XML developers, any serious Open XML developer should also include Eric White’s blog in their RSS feeds. Eric is a leading proponent of the use of functional programming techniques based on LINQ to XML technology, and he covers the Open XML SDK and SharePoint topics as well as more general XML development concepts. Another Microsoft blogger to follow is Stephen Peront, who works with me on the Office Interoperability team and recently covered the File Format Converter API that we released in Office 2007 SP2.

If you’re looking for detailed information about Office’s implementation of Open XML, check out the implementer notes published on the DII web site. There you can find information about our support for ECMA-376, and soon you’ll find a similarly detailed set of notes for our implementation of ISO/IEC 29500 in Office 2010. This information can be very useful for maximizing interoperability with Office.

You can find related information about Office’s support for file formats (including other formats in addition to Open XML) at the Open Specifications Developer Center on MSDN. That site hosts thousands of pages of documentation for protocols and formats that are supported by Office, and the forums section of the site is a good place to get specific questions answered. The MSDN forums also include a very active forum for Open XML SDK developers.

One final detail worth mentioning is where to get the Open XML specification itself. You can download ECMA-376 from the Ecma International web site, and the home page of the Open XML Developer web site has a handy set of links for that purpose. And you can download the ISO/IEC 29500 specification from the “Freely Available Standards” page on the ISO web site.

As you can see, there are many good resources available for Open XML developers. If you have questions about how to do something specific, or are looking for general advice on working with Open XML, the links above are the best places to get help. Feel free to contact me as well, through this blog, if you’re looking for something you can’t find at these sites, and I’ll try to help you track down the best place to get the information you need.

OK, time to get back to working through that email backlog …