In honor of Outlook 2010 being released to manufacturing last week, Angela posted an article discussing the new Outlook 2010 PIA reference. It covers a wide variety of common tasks, from dealing with multiple Exchange accounts to the new conversation view. This is a great place to start if you’re writing Outlook solutions using managed code.
The March 2010 Release (build 22.214.171.1248) is live: http://mfcmapi.codeplex.com.
Two big changes in this release: a debug viewer and right click context menus in Hex Editor and other dialogs. The debug viewer is really nifty. MFCMAPI has had debug output available for years now. You can turn it on from Other/Options. However most people never find it there, and when they do find it, they’re unsure how to see the output. With the Debug Viewer (shortcut: Ctrl+D), you can see debug output directly inside MFCMAPI! Here’s some sample output (click for larger view):
Even though I tend to run MFCMAPI under the debugger anyway, being the developer, this new way to view debug output has already pinpointed the root cause of a couple bugs. Next time you’re chasing down a problem where you don’t understand what’s happening in MFCMAPI, try turning on the debug output and see if it sheds some light on the situation. And let me know if you see problems with the debug output itself.
Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:
[This is now documented here: http://msdn.microsoft.com/en-us/library/ff625288.aspx ]
One of the long standing requests for Outlook development is for documentation around the format for the nickname cache. A couple months ago, development asked if I’d be interested in hosting a preview of documentation for the NK2 file. Of course, I said yes. I didn’t write this doc, but I did assist in the tech review. Down the road, you can expect to see a version of this documentation show up in the MSDN. I’ll link back to it here when it does.
This documentation applies only to Outlook 2003 and 2007 (despite mostly saying only 2007 throughout the article). There are differences in the format used by Outlook 2010 and we do expect to document them, most likely when this information is incorporated into the MSDN. For now, if you use this to mess with Outlook 2010’s nickname cache, you’re on your own.
Finally, I’ve uploaded a PDF of this document here, which includes the parsing for a sample NK2 file.
As always, let me know if you find any problems with this documentation.
[Edited 5/25/2010 to incorporate minor updates based on user feedback]
This post explains how Microsoft® Office Outlook® 2007 interacts with the nickname cache file, also known as the “.nk2 file.” The .nk2 file is where Outlook 2007 persists the autocomplete list, which is the list of names that displays in the To, Cc, and Bcc edit boxes while composing an e-mail. This post also discusses the binary format of the file and the recommended ways for interacting with the .nk2 file.
This blog post should contain sufficient information to support reading and modifying the .nk2 file.
You can use any programming language to write your application because there are no dependencies on the Outlook object model or MAPI APIs.
When a user logs on to Outlook 2007, Outlook reads the user’s MAPI Profile. Later, when the autocomplete list is shown, the .nk2 file is loaded. The .nk2 file has the same name as the profile that was used to log on, typically outlook.nk2, in the default scenario.
This file can be found here:
Outlook 2007 interacts with the .nk2 file in two ways:
1. Loading the .nk2 file.
2. And, later, saving the .nk2 file.
Outlook 2007 loads the .nk2 file when any item with addressing functionality gets initialized. For example, e-mail addresses are used in a new mail, a mail reply, a contact item, a meeting request, etc. To load, Outlook 2007 reads all of the contents of the file as a binary stream into a structure in memory.
For autocomplete operations, Outlook interacts exclusively with this in-memory structure for the duration of the outlook.exe process lifetime. Outlook 2007 does not interact with the .nk2 file in any additional ways until it saves the in-memory structure back to disk on shutdown.
Outlook 2007 saves the .nk2 file on shutdown if the autocomplete list has changed in any way.
Here are the ways that the autocomplete list gets changed:
· A new nickname entry is added through resolving a name, picking a recipient from the address book dialog, or sending mail to a recipient that was not already in the list.
· An entry is modified by sending mail to an existing recipient in the list.
· An entry is removed by the user through the UI.
· Other minor scenarios not relevant to this blog post
Outlook 2007 does not save the .nk2 file on shutdown if the autocomplete list has not changed. The save involves writing the in-memory structure to the .nk2 file as a binary stream.
If the size of the .nk2 file contents shrink during an Outlook session (for example, the user deletes some entries), Outlook saves the new contents to the file, but the file itself will not shrink in size.
· Never partially modify the .nk2 file. The supported interaction is to 1) read the entire file into memory, 2) modify the memory structure, and 3) write out the entire file when the modifications are finished.
· We recommend locking the file from modification by other processes while you’re reading it and writing it using standard Windows file locking APIs (e.g. LockFile in C/C++ and FileStream.Lock in C#).
· Don’t interact with the .nk2 file while Outlook is running. If Outlook is running while you modify the file, Outlook will likely overwrite your changes when it shuts down.
· Do not write properties of type PT_MV_UNICODE and PR_MV_STRING8 into an .nk2 file to be consumed by Outlook 2003. These properties are only understood by Outlook 2007.
· Do not write properties of types that are not mentioned in this document.
In addition to knowing how Outlook interacts with the .nk2 file, you must also understand the binary file format of the .nk2 file.
The .nk2 file is a set of recipient property rows that are saved as a binary stream along with some bookkeeping metadata that is used only by Outlook 2007 and Outlook 2003.
The metadata is relevant to Outlook’s interactions with the .nk2 file sothird parties must preserve what is in each metadata block when saving a modified .nk2 file. In other words, third parties should modify only the row-set portion of the binary format and preserve what was already in the metadata blocks of the file.
When creating a new .nk2 file from scratch, use the metadata values from the binary example to populate the metadata in the new file.
The high-level layout of the file looks like this:
Metadata (12 bytes)
Number of rows n (4 bytes)
Number of properties p in row one (4 bytes)
Property 1’s property tag (4 bytes)
Property 1’s reserved data (4 bytes)
Property 1’s value union (8 bytes)
Property 1’s value data (0 or variable bytes)
… (property 2 through property P-1)
Property p’s property tag (4 bytes)
Property p’s reserved data (4 bytes)
Property p’s value union (8 bytes)
Property p’s value data (0 or variable bytes)
Number of properties q in row two (4 bytes)
… (row two’s properties)
… (row 3 through row n-1)
Number of properties r in row n (4 bytes)
… (row n’s properties)
Broadly speaking, this is the layout of the .nk2 file:
Number of Bytes
The row-set layout is as follows:
Number of rows
The number of rows identifies how many rows will come in the next part of the binary stream sequence.
Each row is of the following format:
Number of properties
The number of properties identifies how many properties will come in the next part of the binary stream sequence.
Each property is of the following format:
Property Value Union
0 or variable (depending on the prop tag)
The Property Value Union and the Value Data are to be interpreted based on the property tag in the first 4 bytes of the property block. This property tag is in the same format as a MAPI property tag. Bits 0 through 15 of the property tag are the property’s type. Bits 16 through 31 are the property’s identifier. The property type determines how the rest of the property should be read.
Some properties have no Value Data and only have data in the union. The following property types (which come from the Property Tag) should interpret the 8-byte Property Union data as follows:
Property Union Interpretation
Other properties have data in a Value Data block after the first 16 bytes that contain the Property Tag, the Reserved Data, and the Property Value Union. Unlike static values, the data stored in the 8 byte Property Value union is irrelevant on reading. When writing, make sure to fill these 8 bytes with something, but the content of the 8 bytes doesn’t matter. In dynamic values, the property tag’s type determines how to interpret the Value Data.
Number of bytes n
Bytes to be interpreted as an ANSI string (includes NULL terminator)
Bytes to be interpreted as a UNICODE string (includes NULL terminator)
Bytes to be interpreted as a GUID
Bytes to be interpreted as a byte array
Number of binary arrays X
A run of bytes containing X binary arrays. Each array should be interpreted like the PT_BINARY byte run described above.
Number of ANSI strings X
A run of bytes containing X ANSI strings. Each string should be interpreted like the PT_STRING8 byte run described above.
Number of UNICODE strings X
A run of bytes containing X UNICODE strings. Each string should be interpreted like the PT_UNICODE byte run described above.
As mentioned above, the binary blocks that represent properties have property tags that correspond to properties on address book recipients. For properties that aren’t listed here, you can look up the property description at http://msdn.microsoft.com/en-us/library/cc433490(EXCHG.80).aspx.
The properties below are the minimum set of properties necessary for a row to be valid. Therefore, new rows added to the .nk2 file must be populated with the properties below.
Description (see MSDN for more details)
PR_NICK_NAME_W (not transmitted on recipients, specific to .nk2 file only)
This property must be first in each recipient row. Functionally serves as a key identifier for the recipient row.
The address book entry identifier for the recipient.
The recipient’s display name.
The recipient’s e-mail address (e.g. firstname.lastname@example.org or /o=Contoso/OU=Foo/cn=Recipients/cn=johndoe).
The recipient’s address type (e.g. SMTP or EX).
The recipient’s MAPI search key.
The recipient’s SMTP address.
PR_DROPDOWN_DISPLAY_NAME_W (not transmitted on recipients, specific to .nk2 file only)
The display string that shows up in the autocomplete list.
PR_NICK_NAME_WEIGHT (not transmitted on recipients, specific to .nk2 file only)
The weight of this autocomplete entry. The weight is used to determine in what order autocomplete entries show up when matching the autocomplete list. Entries with higher weight will show before entries with lower weight. The entire autocomplete list is sorted by this property. The weight periodically decreases over time and increases when the user sends an e-mail to this recipient. See the description below for more information about this property.
The set of rows in the .nk2 file is sorted in descending order by the PR_NICK_NAME_WEIGHT property and the .nk2 file should always preserve this sorted characteristic. Therefore, any changes to a row’s weight should also ensure that the row’s position retains the sorted order of the entire set of rows. Any additions to the row-set should be inserted to the proper position to maintain the sorted order.
The minimum value of this weight is 0x1 and the maximum value is LONG_MAX. Any other values for the weight are considered invalid.
When Outlook 2007 sends a mail to or resolves a recipient, it will increase that recipient’s weight by 0x2000.
Angela has posted an article with the reg key you’ll need to figure out if Outlook is installed via Click-to-Run. I still haven’t figured out a good way to get MFCMAPI running against a C2R environment, but when I do, I’ll post it here. :)
From Austria came the report that the Unicode build of MFCMAPI wasn’t working right for non-English builds of Outlook. They had already debugged and narrowed the problem down to FGetComponentPath not working, even though it worked in the non-Unicode build of MFCMAPI. This was bizarre, since all the parameters should be the same in both builds, having been obtained from the registry, and there’s no flag passed to FGetComponentPath to indicate we’re in a Unicode build. So – how could we call the same function, on the same system, with the exact same parameters, and get different results?
I looked at what was different between the two scenarios. In the ANSI build of MFCMAPI, we request these strings using RegQueryValueExA, then pass the strings directly to FGetComponentPath. In the Unicode build, since FGetComponentPath only takes ANSI strings, after we request the strings using RegQueryValueExW, we convert the results from Unicode to ANSI using WideCharToMultiByte. I confirmed in the debugger that the strings in both cases were the same, yet we still got different results!
So I debugged FGetComponentPath itself and found, to my surprise, that the strings were indeed different. In the ANSI case, after the NULL terminator was another string, and FGetComponentPath was reading this string. In the Unicode build, we lose this extra data during our conversion because we stop at the NULL terminator!
The registry keys in question here are MSIComponentID, MSIApplicationLCID, and MSIOfficeLCID, which I last talked about here. When we look at them in the registry, we can see the problem – MSIApplicationLCID and MSIOfficeLCID are not REG_SZ after all. Instead, they are REG_MULTI_SZ, so these internal NULLs separating the strings are actually expected.
The fix here was to just make sure any reg keys I read for FGetComponentPath’s consumption were read using RegQueryValueExA. Then I don’t have to deal with converting the data, and FGetComponentPath gets everything in the format it expects. This fix will be in the next build of MFCMAPI.
I somehow missed when we posted this (here is Paul’s announcement), but the PST File Format Specification has been published. This should be everything you need to know to write parsers capable of reading and writing PST files without involving any pesky MAPI providers. Enjoy!
Angela posted an overview of the Outlook Social Connector yesterday. In addition to LinkedIn, Facebook, and Myspace providers, you’ll also be able to write your own. And the best part, which I don’t think we’ve announced before, is that we’ll be shipping the OSC as an add-in for Outlook 2007 and 2003, so you can get all that social goodness even if you’re not yet on Outlook 2010!
My team will be providing support for developers writing these providers. We’re very much looking forward to supporting this new platform.
The January 2010 Release (build 126.96.36.1997) is live: http://mfcmapi.codeplex.com.
I added some more Smart View parsers and fixed a few bugs, notably an issue with named props and some GDI leaks.
This came up in a case and the cause wasn’t immediately obvious. The customer used MAPI to open a number of mailboxes using OpenMsgStore. They found that if they tried to multithread their code, they’d open a few mailboxes, then they’d start getting MAPI_E_FAILONEPROVIDER. The odd thing was the same code running against Exchange 2007 did not experience this problem.
It turns out what they were running in to was Exchange 2010’s new client throttling feature. The purpose of this feature is to prevent a single client from causing a Denial of Service on the Exchange server by throwing commands at it and dominating it’s time. You may find that for certain applications, the default throttling policy isn’t appropriate. You can build a new policy using New-ThrottlingPolicy. For MAPI applications (whether they use Outlook or Exchange’s MAPI), the settings starting with RCA (RPC Client Access) are the ones you want to set, particularly RCAMaxConcurrency. Once you’ve created a policy, you can apply it to a user using Set-Mailbox.
An important note: In the policy, you’ll see a CPUStartPercent setting. This setting only applies to EWS, as confirmed by David Sterling in his comment to this post.
We recently had a customer who was copying messages around using CopyTo. They found that for certain messages, if the target message store was a PST, they’d get MAPI_E_NO_ACCESS (0x80070005). They saw this for the same messages every time, when other messages would copy to the same PST just fine.
When I debugged the failure, I found the root of the problem was that the source message had an attachment with the property 0x67100003 set on it. The PST provider does not allow this property to be set on a messages in its store, so any attempt to set it returns the error MAPI_E_NO_ACCESS. When they did the CopyTo of the message properties, because PR_MESSAGE_ATTACHMENTS was not excluded, one of the side effects was that MAPI looped through the the attachments and copied them, using calls to CopyTo at the attachment level. These CopyTo calls did not use an exclusion array at all, so every property on the source attachments were copied to the destination. When CopyTo reached the property 0x67100003, the PST provider returned an error, which caused the attachment level CopyTo to fail, which in turn caused the customer’s CopyTo call to fail.
By the way, 0x67100003 isn't the only property that causes this problem. In fact, most properties in the service provider defined non-transmittable range (0x6600-0x67FF) caused the same problem.
The fix we used was to add PR_MESSAGE_ATTACHMENTS to the exclusion list for CopyTo, then copy the attachments manually using OpenAttach, CreateAttach, and CopyTo, passing an exclusion array specifying that the non-transmittable properties should be excluded. Since this is the same logic CopyTo uses to copy attachments (without the exclusion array), doing the copy like this doesn’t have a performance hit.
It used to be that if you wanted to track a message through various Object Model events such as NewInspector and ItemSend, one way to do it was to grab PR_SEARCH_KEY. A customer recently discovered that this doesn’t always work after applying a recent hotfix. Before applying the hotfix, PR_SEARCH_KEY would remain unchanged during the various events. After applying the hotfix, if they were in cached mode they would see the search key change during the send event, but only if the user saved the message, or autosave kicked in, before submitting.
The hotfix in question is 973404. More specifically, the change that broke the customer’s code was to address the issue outlined in 974413. This issue is in how Outlook relates Read Receipts to the sent items using PR_SEARCH_KEY. In this issue, if a user created an item as a draft, perhaps to use as a template for commonly sent mail, then copied the message before sending, the draft item and the new item would have the same search key. This in turn broke the read receipt logic.
The fix we made was that whenever we copy a message in the Drafts folder, we don’t copy the search key, which allows the copy to get a new search key and allows the read receipt logic to function. Couple this with the user saving the message to Drafts while they’re working on it – under this new behavior the saved message now has a new search key. Some of the object model events the customer’s code would see would be for the message actually being submitted, and some were for the item in the draft folder. Before the read receipt fix, these different message objects would all have the same search key. But after the fix, the search keys can be different.
The fix here is not to rely on PR_SEARCH_KEY at all. In the customer’s scenario, they were using PR_SEARCH_KEY to ensure they didn’t “stamp” a message with their properties more than once during the submission process. However, since they were already stamping data on the message, it made sense to just look for their stamp before stamping again.
We just announced a new release of the MAPI Download that’s pretty much mandatory if you want to work with Exchange 2010. You’ll also need to get and apply Rollup 1 (RU1) for Exchange 2010.
Details about the MAPI Download:
The new MAPI Download fixes a number of problems MAPI had with Exchange 2010:
If I identify other issues fixed by the updated MAPI download, I’ll add them in here.
[Update: 12/14/09] Forgot to mention that you'll still need these tweaks for your profile, but you don't have to disable encryption anymore.[Update: 12/21/09] Added information about admin access to PF.
Angela’s put together a good set of guidelines for developing for Outlook 2010, particularly as it relates to whether your code is built for 32 bit or 64 bit (what we’re calling Bitness). To summarize, all MAPI or Outlook Object Model based apps and non-managed add-ins will need to be built in both 32 and 64 bit flavors to support both target environments, managed add-ins can be built as Any CPU, and VB 6 based add-ins and apps need to be scrapped and rewritten entirely. Read the full article to get the finer points, along with a good collection of links to more resources.
The November 2009 Release (build 188.8.131.526) is live: http://mfcmapi.codeplex.com.
The September 2009 release was downloaded over 18 thousand times, with nearly 1700 of them being the 64 bit build. This exceeded the July build by about 4000 downloads. Yay MFCMAPI!
Mostly bug fixes this time around. Win 7 finally made me dig into why Close All Windows wasn’t working, and I did a bit of work to make MFCMAPI a better multimon client.
From time to time, I get asked to explain where CDO publishes Free Busy information in the Public Folder store, so I figured I’d put this article together. The basic process CDO uses for updating FB information is outlined in the More Information section of KB 278954. We’re concerned with the part of the process summarized as “queued up to be sent to the site’s Free/Busy public folder.”
The first thing CDO does is to open the Public Folder store and read a couple properties off of it, PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID (0x66250102) and PR_SPLUS_FREE_BUSY_ENTRYID (0x66220102). These entry IDs will point at a couple folders in the Public Folder store. The first entry ID points the Free Busy folder for the user’s site, the folder CDO wants to look in for the Free Busy message. If CDO cannot open the folder specified by PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID, it turns to the second property, PR_SPLUS_FREE_BUSY_ENTRYID, which points at the “SCHEDULE+ FREE BUSY” folder, a child folder of the NON_IPM_SUBTREE folder. This property will point at the same folder for all users. In this case, CDO will use an algorithm similar to the one described in [MS-OXOPFFB] to find the correct site folder for the Free Busy message.
The point of failure here is the Entry ID specified in PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID. If this points at the wrong folder, CDO will publish Free Busy information to the wrong folder, and other clients won’t see the updated Free Busy. This property is not one that anybody sets. It’s actually computed on the server side. No amount of changes to a CDO based program can get it to publish Free Busy to the correct folder if the server is handing out the wrong site folder.
This is where server side troubleshooing articles such as KB 326637 and Exchange Public Folder Troubleshooting Resources come in handy. They walk you through the various things that could be wrong with site folders and how to fix them. If PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID is pointing at the wrong folder for your users you’ll need to use the steps in those articles to locate the problem and fix it.
Suppose you just want to determine if PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID is pointing at the folder you’d expect it to point to for a particular user. You can use MFCMAPI to see where this entry ID points to.
First, we check which folders we’re being pointed at:
Next, we determine which folders we should be looking at, following the algorithm from [MS-OXOPFFB]:
So, now you know what site folder the user has listed in PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID, and you know the name of the site folder they should be using. If these two are not the same folder, then you have site folder issues and need to get them addressed.
Hopefully, this should be enough to help folks who are having site folder issues to understand that CDO is just the innocent victim here when it publishes Free Busy to the wrong folder, and that server side configuration is the real villain.