Welcome to MSDN Blogs Sign in | Join | Help

Interoperability @ Microsoft

Covering interoperability scenarios, the technologies enabling them and the community at large

News

Roadmap for Outlook Personal Folders (.pst) Documentation

By Paul Lorimer, Group Manager, Microsoft Office Interoperability

Data portability has become an increasing need for our customers and partners as more information is stored and shared in digital formats. One scenario that has come up recently is how to further improve platform-independent access to email, calendar, contacts, and other data generated by Microsoft Outlook.

On desktops, this data is stored in Outlook Personal Folders, in a format called a .pst file. Developers can already access the data stored in the .pst file, using Messaging API (MAPI) and the Outlook Object Model—a rich set of connections to all of the data stored by Outlook and Exchange Server—but only if Outlook is installed on the desktop.

In order to facilitate interoperability and enable customers and vendors to access the data in .pst files on a variety of platforms, we will be releasing documentation for the .pst file format. This will allow developers to read, create, and interoperate with the data in .pst files in server and client scenarios using the programming language and platform of their choice. The technical documentation will detail how the data is stored, along with guidance for accessing that data from other software applications. It also will highlight the structure of the .pst file, provide details like how to navigate the folder hierarchy, and explain how to access the individual data objects and properties.

This documentation is still in its early stages and work is ongoing. We are engaging directly with industry experts and interested customers to gather feedback on the quality of the technical documentation to ensure that it is clear and useful. When it is complete, it will be released under our Open Specification Promise, which will allow anyone to implement the .pst file format on any platform and in any tool, without concerns about patents, and without the need to contact Microsoft in any way.

Designing our high volume products to enable such data portability is a key commitment under our Interoperability Principles, which we announced in early 2008. We support this commitment through our product features, documented formats, and implementation of standards. The move to open up the portability of data in .pst files is another step in putting these principles in action.

Over the past year, Microsoft Office has taken several steps toward increased openness and document interoperability. We’re proud of the work we’ve done around document interoperability, offering customers a choice of file formats and embracing a comprehensive approach that includes transparency into our engineering methods, collaboration with industry stakeholders, and shared stewardship of industry standards.

We’re excited about the possibilities created for our customers and partners by this kind of effort, and we look forward to continued collaboration with the industry in our pursuit of improved interoperability with Microsoft Office. Stay tuned.

Paul Lorimer, Group Manager, Microsoft Office Interoperability.

Related Items:

Posted: Monday, October 26, 2009 9:02 AM by jccim

Comments

OneNote format? said:

Please make OneNote 2007 format open as well.

# October 26, 2009 11:11 AM

Leith Bade said:

Are there any plans to also document the .MSG file format? This is another useful format to have for importing into document management systems.

# October 26, 2009 12:02 PM

Alex Ott said:

Hi Paul

Is it possible to get early access to .pst format documentation? I'm working with many of MS formats, and we're planning to implement support for them all in our products.

P.S. My company has signed agreement with MS about support for different office formats

# October 26, 2009 12:50 PM

jccim said:

@Leitg Bade: the .MSG File Format Specification is actually already available at http://msdn.microsoft.com/en-us/library/cc463912.aspx  

# October 26, 2009 1:47 PM

maricela morales said:

how can I help. I can't wait to having alternatives in a central manner. My productivity could greatly improve with alternatives to traditional email. a next generation integrated email...maybe called eknowledge is desperatedly needed. EMAIL HAS TO GET SMART. IT IS HOLDING US BACK...I am tired of all the notifiations from social network sites requiring horrendous time for organization and filtering...AY CARAMBA! Help@

# October 26, 2009 3:06 PM

Jeff McKay said:

I'm an "interested customer" (and Microsoft Partner) - can I sign up to get an early look at the documentation?  Is there a way for me to be notified when it is available?

# October 26, 2009 3:44 PM

Jonathan F. said:

Please share this information with the Entourage group! To put it mildly, I was flabbergasted to discover that Entourage 2008 could NOT import contents directly from an Outlook 2003 PST file. I know that people get all hot and bothered discussing the merits of Mac OS vs. Windows, but for me, they're both just tools to run software applications. When Microsoft designs apps for the respective OS's, I expect that those apps should work nicely together. Instead, upon opening up Entourage, I felt beholden to some juvenile argument that was held between the Windows Office team and the Mac Office team. The above mentioned trajectory sounds much better, so please make sure it happens!

# October 26, 2009 4:07 PM

Russ Nemhauser said:

The PST file is one of the biggest problems with Outlook. Why would anyone else want to implement this on other platforms?

# October 26, 2009 4:29 PM

Darren Baker said:

Brilliant! Any timeframe on this?

# October 26, 2009 5:13 PM

@tj said:

What about the .OST file format too?

Is that documented somewhere as yet or will it be covered in the .PST format documentation set?

# October 26, 2009 6:57 PM

Helen said:

Interesting.

Any comments on whether the format is documented in the context of MAPI calls (requiring the implementation of underlying COM as well as MAPI APIs for use), or simply as a byte stream oriented structure?  The latter would be preferable from a multi-platform development standpoint.

# October 26, 2009 7:58 PM

robertb said:

Why not rather work on a format which removes the limitations inherent in the .pst format ( 2gb file size, 65535 items ) such as storing in a local MS SQL DB, or heaven forbid an open format such as sqlite?

# October 26, 2009 10:55 PM

mike said:

This is good news and it will open the way for people to create innovative software components and applications which access PST files.

Mike

# October 26, 2009 11:55 PM

Fredre said:

"This documentation is still in its early stages and work is ongoing."

Can you give a better time line for release like mid 2010 or 2011 or never ?

That would help

# October 27, 2009 1:18 AM

decatec said:

Interop is good, but please first:

-give us .NET Developers and MS partners easy .NET APIs rather than specs

-put formats in XML (e.g MSG, EML) like you did for the Windows Event Log since Vista

- Outlook (PST) storage could be extended/replaced by SQL Server (Filestream) or Sharepoint for better centralization/backup/retention/ .. you name it

# October 27, 2009 2:02 AM

Dave said:

Please release as soon as possible, ie beta documentation. Incomplete documentation is better than no documentation.

# October 27, 2009 2:55 AM

Jaspreet said:

Any timeline for the release of this documentation ?

Is there any way to join the beta testing  ?

# October 27, 2009 3:17 AM

Steve said:

What is the timeframe for releasing said documentation?

# October 27, 2009 7:58 AM

jccim said:

@Helen:

The specification will document the on-disk structure of .pst files. Developers will be able to develop multi-platform solutions that do not require MAPI or other runtime components provided by Outlook.

@tj:

The .pst and .ost files are created using the same underlying database engine, but there are some important differences in the file formats.

It is important to note that an .ost file is designed to be a local cache of data stored on Exchange servers, and the authoritative user data can be accessed from the Exchange servers through the documented Exchange protocols.

We will not be covering the .ost file format as we meet with industry experts, but people are welcome to provide us with feedback related to documenting the .ost file format.

@Darren Baker:

Thanks!  We plan to release the .pst specification to the general public in the first half of 2010.

# October 27, 2009 7:59 AM

Wolfgang Zerzawy (CleverITS) said:

That's a good idea to open the specification to the public, so we can develop 3rd party tools more easily.

We are looking forward to the release.

Best regards

Wolfgang Zerzawy

# October 27, 2009 8:06 AM

David said:

PST file format API already exist. Check PST .NET:

http://www.independentsoft.de

Regards,

David

# October 27, 2009 8:54 AM

Volker Dilg-Gruschinski said:

As the underlying database engine is both the same for pst and ost files, will that documentation allow 3rd party tools to access and read ost files as well ?

It's not the question how the usage is designed.

The general question is:

Would this documentation allow a hacker to read my confidential data from an ost file ?

Volker

# October 27, 2009 9:39 AM

Joshua Cranmer said:

Is there any plans to also release a specification for winmail.dat stuff?

# October 27, 2009 1:31 PM

Hai said:

Does this also benefit MAPI service providers that use the wrapped PST store?

Is this solely for the purpose of letting non-Outlook process to read/write information in a PST file?

How should the potential new applications handle scenarios of concurrent access to the same PST file from both Outlook process and the 3rd party application process? Or this is mainly for off-line manipulation to the PST file when Outlook is not running?

# October 27, 2009 1:37 PM

jccim said:

@OneNote question:

A draft of the OneNote file format specification has already been posted: http://msdn.microsoft.com/en-us/library/dd924743.aspx. It covers OneNote 2010.

# October 27, 2009 4:03 PM

jccim said:

Thanks everybody for your participation and interest. Here are a few more answers/comments from the team :)

@Robertb

The 2GB file size limit has been removed with the release of Outlook 2003. Please refer to this article http://support.microsoft.com/kb/830336 for more information. We are committed to investing in our storage technology to meet the increasing demands of our users.

@Hai

The .pst file format specification will cover wrapped .pst stores as wrapped pst stores are just a type of .pst files. It is important to note that MAPI and the Outlook Object Model provide a runtime service to serialize requests from multiple processes that are trying the access the same .pst file. If you want to implement a solution that directly accesses a .pst file without going through the runtime service provided by MAPI, you will need to make sure no other processes are using the same file.

@Volker Dilg-Gruschinski

In theory software developers can use information provided in the .pst file format specification to access ost files as well. Regarding the security concern, we don’t recommend users to rely on the rudimentary encoding in our file formats to protect confidential data. We recommend users to take advantage of advanced encryption features provided by the operation system or 3rd party tools to secure confidential data.

@David

We welcome developers to take advantage of the .pst file format specification to create tools that interoperate with Outlook. As of today we have not released a draft or final version of the .pst file format specification to the public.

@David

winmail.dat contains data encoded in TNEF format, which is documented in http://msdn.microsoft.com/en-us/library/cc425498.aspx

# October 28, 2009 1:22 AM

walo said:

you have to open all formats or adopt odf.

# October 28, 2009 5:50 AM

Roger said:

The format was hacked long ago. It's about time MS recognized it.

# October 29, 2009 8:04 AM

jccim said:

The best way to participate is to send an email message to pstinfo@microsoft.com .

Daniel Ko from the Outlook development team has just posted this information:

http://blogs.msdn.com/outlook/archive/2009/10/29/outlook-pst-file-format-and-interoperability.aspx

# October 29, 2009 1:09 PM

GS said:

  I am eager to see the documentation and how it can be used better than the MAPI?

# October 29, 2009 8:50 PM

Pim van der Eijk said:

I fear the documentation for this format will 5000 pages and just show that this is yet another very complex, binary format. A format so complex that no-one will be able to implement it correctly and robustly, except someone who was been working in the Office Division of Microsoft for the past ten years, and knows two dozen people who have been there too. Or am I too pessimistic?

It would be better for Office to move to some clean new format, like an relational data store format or a (ZIP'ed)  XML format so that people can use SQL or XPath to manipulate it.  And preferably standardize such a format in an open standards body. Is anything like that on the agenda?  

I regret the day I started to use Outlook, as my email of the past 15 years (> 10 GB) is locked in a proprietary, undocumented file format. It's amazing to me that PST has never been in the spotlight as much other Office file formats. Those formats, unlike PST, have been supported by other products (like OpenOffice) for years, so opening up PST should IMO have been done years before and with higher priority that developing Office Open XML.

But in any case, document PST is a good step!

Pim

# October 31, 2009 4:19 AM

Paul Diedrich said:

Thank you for providing the pst format.  We have instances where we must extract the contents of pst files for litigation purposes.  There have been some pst files that were damaged and the pst file fixing software provided with MSOutlook could not fix them.  In this case, we need to be able to extract as much information as possible from the files.  The format will allow us to interactively try to navigate to accessible parts of the file.  (In our case, the files being worked with are off-line and not being referenced by outlook, so access conflicts should not be a problem.)  Thanks again.

# November 3, 2009 10:25 AM

Carl Byington said:

It will be interesting to see the final documentation.

# November 6, 2009 5:15 PM

Russell Mangel said:

To: Pim van der Eijk

I would read all 5000 pages, nine times if necessary.

Consider the following:

The PST file format has been with us since the Microsoft Exchange Client was released with Exchange 4.0. since at least 1996. Every version of Outlook uses the PST file format.

I would be willing to bet there are far more PST files backing Outlook installations, than Outlook to Exchange Server (which uses Exchange Database). This makes the PST file *wildly* popular.

I have been waiting for this documentation for many years.

Kudos to Microsoft.

Russell Mangel

Las Vegas, NV

# December 4, 2009 7:40 AM

Angelo said:

This is a great strategic move.  I was considering how I need to long term archive email and the fact that pst is Not publicly documented, and that there are really no 3rd party tools, really made me question it as an option.

I may end up stayingwith pst if there was signifcant 3rd party development and integration.

Can't wait.  

# December 5, 2009 5:42 AM
Leave a Comment

(required) 

(required) 

(optional)

(required) 

  
Enter Code Here: Required

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Page view tracker