SGriffin's MAPI Internals

November 2009 Release of MFCMAPI

The November 2009 Release (build 6.0.0.1016) is live: http://mfcmapi.codeplex.com.

The September 2009 release was downloaded over 18 thousand times, with nearly 1700 of them being the 64 bit build. This exceeded the July build by about 4000 downloads. Yay MFCMAPI!

Mostly bug fixes this time around. Win 7 finally made me dig into why Close All Windows wasn’t working, and I did a bit of work to make MFCMAPI a better multimon client.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

• Close All: Shutdown of contents table windows was swallowing the WM_CLOSE message for the other windows. Whoops!
• Multimon: New windows and context menus now show on the same screen as their parent window, as they should.
• Unicode Files: MFCMAPI can now handle files (like .msg or .eml files) that have Unicode characters in their names.
• Property Editor: It’s now possible to manually insert CR/LF characters directly in the hex stream of a property and write the exact data you intended, as long as you don’t touch the text editor. This is to work around a bug/feature of the rich edit control that made reproducing certain issues impossible.
• Smart View: Can now turn Smart View off if you’ve got a property that’s causing it to choke. Do continue to report these to me though!
• Version selection: Did some work to make it clearer when you’re running the wrong version of MFCMAPI (32/64) for the version of MAPI you have.
• Error Dialog Suppression: Identified a number of “error” dialogs that confused more than they helped and eliminated them. If there was an actual error it’s written to the debug output.
• Tons of minor issues – 22 total bugs tied to this release, and those are just the ones that were worth calling out.

Enjoy.

Where CDO Publishes Free Busy Information

From time to time, I get asked to explain where CDO publishes Free Busy information in the Public Folder store, so I figured I’d put this article together. The basic process CDO uses for updating FB information is outlined in the More Information section of KB 278954. We’re concerned with the part of the process summarized as “queued up to be sent to the site’s Free/Busy public folder.”

The first thing CDO does is to open the Public Folder store and read a couple properties off of it, PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID (0x66250102) and  PR_SPLUS_FREE_BUSY_ENTRYID (0x66220102). These entry IDs will point at a couple folders in the Public Folder store. The first entry ID points the Free Busy folder for the user’s site, the folder CDO wants to look in for the Free Busy message. If CDO cannot open the folder specified by PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID, it turns to the second property, PR_SPLUS_FREE_BUSY_ENTRYID, which points at the “SCHEDULE+ FREE BUSY” folder, a child folder of the NON_IPM_SUBTREE folder. This property will point at the same folder for all users. In this case, CDO will use an algorithm similar to the one described in [MS-OXOPFFB] to find the correct site folder for the Free Busy message.

The point of failure here is the Entry ID specified in PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID. If this points at the wrong folder, CDO will publish Free Busy information to the wrong folder, and other clients won’t see the updated Free Busy. This property is not one that anybody sets. It’s actually computed on the server side. No amount of changes to a CDO based program can get it to publish Free Busy to the correct folder if the server is handing out the wrong site folder.

This is where server side troubleshooing articles such as KB 326637 and Exchange Public Folder Troubleshooting Resources come in handy. They walk you through the various things that could be wrong with site folders and how to fix them. If PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID is pointing at the wrong folder for your users you’ll need to use the steps in those articles to locate the problem and fix it.

Suppose you just want to determine if PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID is pointing at the folder you’d expect it to point to for a particular user. You can use MFCMAPI to see where this entry ID points to.

First, we check which folders we’re being pointed at:

1. Open MFCMAPI on the machine where your CDO application runs.
2. Log on to a profile. You’ll get your best results if you can use the same profile the CDO application uses.
3. Select the Public Folder store in the list of stores.
4. The properties we want may or may not show in the default list of properties, so we’ll add them:
1. Click Property Pane/Modify Extra Properties
2. Click Add
3. In the Property Tag field, type in PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID. The rest of the form should fill out automatically.
4. Click OK, then click Add again.
5. In the Property Tag field, type in PR_SPLUS_FREE_BUSY_ENTRYID. The rest of the form should fill out automatically.
6. Click OK, then click OK again to close the Extra Properties window.
5. We’ve added the properties, so they should show in the property pane. You can sort by the Property Name column and look them up by name, or sort by the Tag column and look them up by number (0x66250102 for PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID and 0x66220102 for PR_SPLUS_FREE_BUSY_ENTRYID.
6. Right click on PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID and select Open as Entry ID or Object. In the dialog that comes up, hit OK.
7. In the resulting hierarchy view, you can now read the name of the site folder the server has directed you to.
8. Switch back to the main window and reselect the Public Folder store to refresh the property list.
9. Right click on PR_SPLUS_FREE_BUSY_ENTRYID and select Open as Entry ID or Object. In the dialog that comes up, hit OK.
10. This will display the hierarchy for the SCHEDULE+ FREE BUSY folder. Click the + icon to display the list of all the site folders in the Public Folder store.

Next, we determine which folders we should be looking at, following the algorithm from [MS-OXOPFFB]:

1. In the main window of MFCMAPI, select Session/Display Current User Properties. Hit OK.
2. In the Property Pane, locate PR_SEARCH_KEY (0x300B0102).
3. Double-click on the property to open it, and take note of the Text field. For me, this says “EX:/O=MICROSOFT/OU=NORTHAMERICA/CN=RECIPIENTS/CN=420275”
4. The appropriate site folder can be derived by truncating this string at the / before the first CN. So for me, my site folder should be “EX:/O=MICROSOFT/OU=NORTHAMERICA”.

So, now you know what site folder the user has listed in PR_FREE_BUSY_FOR_LOCAL_SITE_ENTRYID, and you know the name of the site folder they should be using. If these two are not the same folder, then you have site folder issues and need to get them addressed.

Hopefully, this should be enough to help folks who are having site folder issues to understand that CDO is just the innocent victim here when it publishes Free Busy to the wrong folder, and that server side configuration is the real villain.

Enjoy!

Strange Looking NDR

Not many people are aware of the feature, but MFCMAPI implements a MAPI Form Viewer. This means MFCMAPI can use MAPI to ask Outlook to display messages. You might have run into this if you ever double-click on a message in MFCMAPI. If you’re on a machine with Outlook installed, the message will open up like it does when you double-click it in Outlook. And if Outlook isn’t installed and you’re using Exchange’s MAPI, you get a scary dialog saying we couldn’t open the MAPI form manager.

This is one of those dark corners of MAPI into which few people tread, and most of those who do end up cribbing or parroting the Form Viewer implementation in MFCMAPI itself, meaning nearly every issue they report related to form viewers can be reproduced in MFCMAPI. Sometimes, this turns out to be a bug in my implementation, like an errant warning while clicking up and down through messages (fix coming very soon!). Other times, the problem is a bug in Outlook.

My latest customer’s issue turned out to be the latter case. They had noticed that when they tried to view NDR messages with their Form Viewer, the body of the message would be full of alien looking RTF text:

“{\rtf1\ansi\ansicpg1252\deff0\deflang1033{\fonttbl{\f0\fmodern\fcharset0 Consolas;}}
{\*\generator Riched20 12.0.6413.1000;}\viewkind4\uc1
\pard\fi-1440\li1440\tx1440\f0\fs21 Your message did not reach some or all of the intended recipients.\par
etc…”

What was really odd was that if you looked at the message in MFCMAPI, this text was no where to be found! And to top it off – if you opened the message from Outlook, you got the NDR message you expected.

So what was going on? It’s not surprising that Outlook builds the NDR text on the fly. It’s much easier to localize it that way. And the RTF does appear to be the underlying RTF we would generate for the NDR message. But why did we display it wrong, and why did Outlook display it right?

The key here is the LoadForm function. This function takes the message class of the form we want to load. Under the covers, LoadForm looks up the message class using ResolveMessageClass and gets the GUID for the associated class factory. The form manager then passes this GUID over to Outlook and asks for the class factory. This is where we have a problem. Outlook uses the same GUID for multiple form class factories, some of which have different attributes associated with them. The class factory we get has an attribute on it stating it should render using HTML, while the class factory we should have gotten did not have this attribute. So when it comes time to render the text, we use the wrong rendering method and get the ugly RTF text.

When Outlook opens the message, being the implementer of the forms, it gets to skip many of the MAPI steps. Instead of using the message class to look up a GUID, it uses the message class to look up the class factory directly, getting the right one in the process.

Based on the fact that this behavior has been unchanged for as long as we can trace back, and we’re only just now hearing of an issue with it, we chose not to fix this problem, but instead document it. So – here’s the documentation. Enjoy!

Announcing EWSEditor 1.5!

Matt’s very excited about the release of EWSEditor. This tool does for EWS what MFCMAPI does for MAPI. With this tool, Matt does what I did with MFCMAPI and what I recommend to anyone trying to learn a new API: the best way to learn an API is to write tools that use it.

As you work with EWSEditor, you may notice a few things which are similar to MFCMAPI. This is no coincidence. EWS and MAPI, both designed primarily to interact with an Exchange server, share much of the same object hierarchy, which in turn dictates that tools designed to explore these hierarchies will be similar. Also, since we sit near each other and act as each other’s ad-hoc PMs, we borrow each other’s features. For instance, EWSEditor implements much of the same Smart View parsing as MFCMAPI, using the same logic, recoded from C++ to CSharp.

Of course, there’s one similarity between EWSEditor and MFCMAPI that has nothing to do with the underlying APIs or fancy features and everything to do with the fact that Matt and I are programmers, not icon designers. Look familiar?

Enjoy!

Inside the PST

As Paul Thurrott notes, we just announced that we’re documenting the PST file structure. I haven’t decided yet if I’ll be writing a standalone parser/viewer/validator, but I’m sure I’ll be adding some interesting stuff to MFCMAPI. I’ll let you know when I know more.

Exchange MAPI Download Version 6.5.8131.0

Just had someone ask me what changed in this version compared to the last build. Most of the changes were tweaks to various timezones, such as the addition of a timezone for Yakutsk, my favorite Risk territory. Additionally, we fixed a problem with ITnef::AddProps. You’d only need to worry about the TNEF change if you’re actually calling into the TNEF API directly.

So Long System Attendant Mailbox

Dave has a post up about the fate of the System Attendant Mailbox in Exchange 2010. Since this mailbox isn’t there anymore in 2010, if you have a solution that depended on it, you’ll need to rework it. Dave gives a few suggestions, such as creating your own mailbox/account (the best option since you can then control your exact permission set) or using the System Mailbox.

Enjoy!

Outlook 2010 and Shutting Down Add-Ins

Ryan has a new post up about changes we’re making to Outlook 2010 to make shutdown faster. These changes are building on the previous changes we made introducing Fast Shutdown. Two key takeaways:

1. MAPI Providers now get Fast Shutdown by default. They may still opt out of Fast Shutdown by implementing IMAPIClientShutdown.
2. Outlook 2010 will not call the OnBeginShutdown or OnDisconnection of addins during shutdown.

Ryan gives plenty of best practices for rewriting your addins to deal with these changes (and improve the user experience in the process), including policies you can use to revert the changes if you’re an admin and you have add-ins that cannot be rewritten.

EnumAccounts Sample and Some New Account Management Props

A while back, Jason posted an Account Management API sample. I’ve threatened promised before that I would post an update to this sample, and I finally got around to it. You can download the updated sample here.

In addition to adding a –W command to call DisplayAccountList and show the Account Settings Wizard, I’ve also added a couple new properties that dev have decided to document:

#define PROP_ACCT_USER_EMAIL_ADDR PROP_TAG(PT_UNICODE, 0x000C)
#define PROP_MAPI_IDENTITY_ENTRYID PROP_TAG(PT_BINARY, 0x2002)

PROP_ACCT_USER_EMAIL_ADDR is simply the e-mail address for the account, and PROP_MAPI_IDENTITY_ENTRYID is the address book entry ID for the account, similar to what you’d expect from QueryIdentity. Neither property is expected on every account. For instance, in one profile here, an Exchange account has PROP_MAPI_IDENTITY_ENTRYID but not PROP_ACCT_USER_EMAIL_ADDR, while for an SMTP/POP3 account the situation is reversed.

Dev also asked me to call out an issue that had tripped some folks up. The function GetIdentity is supposed to return a string and the length of the string. The returned string should be NULL terminated and the length should include this NULL terminator in the count. We’ll be updating the MSDN to be more explicit about this, in addition to adding the two new properties.

Some more information about the updated sample: If you’ve been looking for a header for the Account Management API, you can grab and use AcctMgmt.h from this sample. It should have everything you need. Please let me know if it doesn’t. Also, intrepid explorers may want to look at the –I option I added to the sample. It iterates through every possible property tag and requests it, printing the results if it finds something.

Enjoy!

OpenStreamOnFile vs Unicode Files

Let’s take a look at the function OpenStreamOnFile, which is exported by MAPI. Suppose you want to use this function to open a stream on a file that has a Unicode file name. As has been noted before, this doesn’t work. Why is this? According to the documentation, and according to the header, maputil.h, this function takes two parameters, lpszFileName and lpszPrefix, which are of type LPCTSTR. However, it is impossible for an exported function to take a parameter of this type. Why? Because LPCTSTR is a macro that compiles to either LPCSTR or LPCWSTR depending on whether Unicode was defined in the project. But MAPI’s the same no matter how you compiled your project, so the export won’t change.

In truth, OpenStreamOnFile should have been declared as taking parameters of type LPCSTR. We’re working on an update to the MSDN and to the MAPI Headers to fix that. We’ll also be documenting a new function, which you can go ahead and use to open files with Unicode names: OpenStreamOnFileW. This is the Unicode version of OpenStreamOnFile which takes Unicode strings. Here’s the signature:

STDMETHODIMP OpenStreamOnFileW(

   LPALLOCATEBUFFER lpAllocateBuffer,

   LPFREEBUFFER lpFreeBuffer,

   ULONG ulFlags,

   __in LPCWSTR lpszFileName,

   __in_opt LPCWSTR lpszPrefix,

   LPSTREAM FAR * lppStream);


typedef HRESULT (STDMETHODCALLTYPE FAR * LPOPENSTREAMONFILEW) (

   LPALLOCATEBUFFER lpAllocateBuffer,

   LPFREEBUFFER lpFreeBuffer,

   ULONG ulFlags,

   __in LPCWSTR lpszFileName,

   __in_opt LPCWSTR lpszPrefix,

   LPSTREAM FAR * lppStream);

Other than the change to the strings, the function is identical to OpenStreamOnFile. Officially, we’re only documenting this function for Outlook 2010 and higher, but the function is also there in Outlook 2003 and 2007. The Exchange MAPI Download does not implement OpenStreamOnFileW at all.

The November build of MFCMAPI is being augmented to handle Unicode file names. Part of that rewrite is that MFCMAPI will use OpenStreamOnFileW if it’s available, and fall back to OpenStreamOnFile if it’s not.

Enjoy!

Detecting Sharepoint PSTs

We had a customer recently who was asking how to tell a PST had been configured as a Sharepoint PST, so they could exclude it from some processing they were doing. After some digging, I found the property that Outlook itself uses to determine a PST is a Sharepoint PST, and development gave me permission to document it:

#define PR_ASSOCIATED_SHARING_PROVIDER PROP_TAG(PT_CLSID, 0x0EA0)

To check if a PST is a Sharepoint PST, mount the PST using OpenMsgStore, then call GetProps on the message store object requesting this property. If it exists, you can assume the PST has been configured for Sharepoint. If it doesn’t exist, the PST has not been configured as a Sharepoint PST.

September 2009 Release of MFCMAPI

The September 2009 Release (build 6.0.0.1015) is live: http://mfcmapi.codeplex.com.

The July 2009 release was downloaded over 13 thousand times, with nearly 1700 of them being the 64 bit build. This was the most popular release yet – retiring MAPI Editor drove a lot of traffic over to codeplex.

I did a lot of work on the property and hex editors this time around. The property editor should be much better at editing large and complex streams you’ll find in PR_BODY or PR_RTF_COMPRESSED. And the Hex Editor is no longer modal! Now maybe Rick will stop bugging me about it.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

• Ctrl+H will now bring up a non-modal Hex Editor
• That “Sample version” text is now gone. There is only one version of MFCMAPI, and it’s what you get on codeplex
• Embedded NULLs in strings no longer break the property and stream editors
• All the editors (hex, property, stream) should be able to handle MUCH larger strings now
• Smart View: Added parsing for PR_ADDITIONAL_REN_ENTRYIDS, dispidPropDefStream
• Smart View: Added framework for Smart View parsing of multivalued properties, and added a bunch of previously unparsed properties.

Enjoy.

Exchange 2010 and the MAPI Download

Bunches of people have contacted me asking if the MAPI download works with Exchange 2010. My usual response has been that I know it works, but you have to tweak the profile. I didn’t know the specific tweaks ‘cause I hadn’t looked in to it.

Dave Vespa just blogged those tweaks. Note that they’re very similar to the modifications we use with Exchange 2007 to get referral working. Also note that the profile Dave builds is not referral enabled, but this doesn’t matter any more. With Exchange 2010, MAPI always talks to the Client Access Server for both mailbox and directory. The dreaded DSProxy that we were trying to avoid in 2007 and earlier has been replaced by the Address Book Service, which is pretty much what DSProxy should have been.

Enjoy!

Outlook 2010 MAPI Reference Updated

The Outlook 2010 MAPI Reference has been updated. I don’t have a list of everything that changed, so I’ll highlight one addition that’s pretty cool: the property definition stream. Long time readers might remember I mentioned this property before – it’s one of the properties you want to delete to kill a one-off form. It’s also the property that Outlook uses to store information about named properties created via the Object model and the user interface.

Now that the format of PidLidPropertyDefinitionStream is documented, you can create new user defined properties using MAPI and have them show up in the UI! The September release of MFCMAPI (which I’m furiously hammering on) will be able to parse this property.

As usual, the documentation is also available for download.

Enjoy!

Responding to Meetings with CDO

My colleague Edwin, writing for the Exchange Support Team Central Europe blog, posted an article on accepting meeting requests with CDO. The key takeaway from the post is this:

This means that when calling the respond method on meeting updates / requests, the sequence number must be read from the request and written to the response, otherwise Outlook will not be able to match the response to the original meeting.
(The proposed solution is to read the dispidApptSequence from the original message and write this to the meeting update response before sending.)

He also points out that this requirement is spelled out in the MAPI protocol documentation, here.

Since CDO has reach end-of life, we won’t be taking a change to CDO to have it do this, so you’ll need to do it yourself. Edwin gives sample code that demonstrates how.

Enjoy!