SGriffin's MAPI Internals

MFCMAPI and Old Versions of Visual Studio

Stephen Griffin — Thu, 09 Jul 2009 21:45:29 GMT

Ever since I went to the new Vista style icon for MFCMAPI, it’s not been possible to compile in Visual Studio 6. I’ve spent some time looking at this, and it appears the only clean way to get VS 6 working again is to remove compression from the icon, something I really don’t want to do. So I’m considering dropping support for Visual Studio 6 altogether. While I’m at it, it’s been a while since I tried to compile in VS 2003 or 2005, so I was thinking about dropping support for those two as well. The only compiler I’d test would be Visual Studio 2008 (and 2010 whenever it comes out). Note that other than the known break in VS 6 I’m not planning on actively breaking compilation for 2003 or 2005. I just wouldn’t test them, so over time, I probably would break them.

Before I do this – I wanted to get feedback. Would anybody care if MFCMAPI no longer compiled in Visual Studios 6, 2003, or 2005?

Exchange MAPI and the Form Manager

Stephen Griffin — Wed, 08 Jul 2009 15:27:13 GMT

Had a customer contact me recently asking why MAPIOpenFormMgr doesn’t work with the Exchange MAPI download. Actually, it wasn’t working in Exchange 2003 either. I’m not sure exactly when it last worked in Exchange’s implementation of MAPI. We might have to go all the way back to the Exchange Client to find a version of Exchange’s MAPI where it worked.

There was more to this story that I wanted to post. A few years ago, on a whim, I debugged why MAPIOpenFormMgr was broken. I found it was looking for a forms manager DLL that Exchange hadn’t shipped in a long long time. However, there was a mechanism for directing Exchange’s MAPI to use a different forms manager, and one of the DLLs Exchange shipped could act as a forms manager. Since this wasn’t likely supported, and no one was asking about it, I filed it away as an interesting tidbit.

Since I did get asked about it, my plan was to post the reg key that controls which forms manager is used with a warning that since it isn’t supported or tested, it’s quite possible it will break on you. However, when I went to try it out, I found that when you try it out with the Exchange MAPI download, you crash right away. This isn’t surprising considering no one’s run the code in 10+ years.

So there’s no point in posting the key, though someone who was really determined could figure it out pretty quickly, and with a little debugging even figure out which DLL to point at. I guess I’ll leave that as an exercise for the reader. The official answer here is that Exchange’s implementation of MAPI doesn’t include a functioning forms manager, so none of the forms management portions of MAPI, such as MAPIOpenFormMgr or MAPIOpenLocalFormContainer, are available.

June Cumulative Update for Office 2007

Stephen Griffin — Wed, 01 Jul 2009 14:03:15 GMT

Since I’ve got a few customers who were waiting on this update I figured I should announce it here. Yesterday we released the June Cumulative Update for Office 2007: http://support.microsoft.com/kb/972632. In there, you’ll find links to update packages for everything in Office 2007. We highlight two:

MAPI Profile Crash: Basically, SP2 caused MAPI to stumble over certain profiles because it didn’t like their names. This update fixes that.

The Big Rollup: This is the mother-load of fixes. In this one, you’ll find fixes for the following issues:

EMSMDB32 doesn’t handle PR_STORE_UNICODE_MASK properly
Custom forms with code can crash the preview pane
Outlook won’t display custom form icons when Windows is in High DPI mode
Categories created with the OOM aren’t preserved across restarts

BTW – if you’re looking for a central place to monitor for the latest Cumulative Update, flag this article: http://support.microsoft.com/kb/953878. It’s updated with every Cumulative Update release to point at the latest one.

Enjoy!

Outlook 2003 Also Rules

Stephen Griffin — Fri, 26 Jun 2009 17:15:58 GMT

A little follow-up to the Outlook Rules post, concerning getting rules to fire when you’ve got a wrapped PST store. I had noted in the original post that rules “just worked” in Outlook 2003. Turns out I was partially right. They can work in Outlook 2003, but you have to do something a little special that I had somehow overlooked in my original investigation. Here’s how I explained your should fire a notification for Outlook 2003:

In Outlook 2003, the way you solve this is to use the IMAPISupport object that was passed to you during logon. From there, you would call Notify, passing fnewNewMail and the details of the new message.

The tweak here is in the structure you pass to into the Notify call. Since we’re passing fnevNewMail, we use a NEWMAIL_NOTIFICATION structure, which contains a ulFlags member. The only flag currently documented for this structure is MAPI_UNICODE. To get notifications working in Outlook 2003, we need to OR in a new flag, MSGFLAG_OUTLOOK_NON_EMS_XP. This flag has the same effect the NON_EMS_XP_SAVE flag has for Outlook 2007, which is to let Outlook know this message did not originate from the Exchange transport, so it knows rules need to be processed client side. This flag is only required for Outlook 2003, though we’ve seen no side-effect to also passing it in Outlook 2007. Unlike Outlook 2007, Outlook 2003 does NOT use the ITEMPROC_FORCE flag.

Definition of the new flag:

MSGFLAG_OUTLOOK_NON_EMS_XP

0x00010000

A client can call IMAPISupport::Notify with a fnevNewMail notification, passing MSGFLAG_OUTLOOK_NON_EMS_XP in the ulFlags member of the NEWMAIL_NOTIFICATION structure to indicate that the message is being delivered to a PST and should be eligible for rules processing.This flag is only available if the message is being saved to a PST.

We will most likely not be adding this flag to the Outlook MAPI documentation since it only applies to Outlook 2003.

Named Properties in Exchange 2007 SP2 and Beyond

Stephen Griffin — Wed, 17 Jun 2009 16:57:12 GMT

With the recent Rollup 8 for Service Pack 1, the Exchange team previewed a change designed to reduce the impact of inbound SMTP messages on the named property map for a store. Jason talked about it here. The gist of the change is that before, random e-mails with random x-headers could exhaust the map, making it impossible for legitimate applications, like Outlook, to map new named properties. With Rollup 8, those x-headers will only be promoted if they’ve already been mapped, or if the mail was submitted by an authenticated user.

Jason’s latest post details where we’re taking this in SP2 and Exchange 2010. An x-header will only be promoted to a named property if the mapping already exists, either because it was mapped before SP2 was applied, or a MAPI based application requested the mapping. We know this breaks a few scenarios, but felt it was a reasonable tradeoff to preserve the stability of the Exchange server.

Jason has asked for feedback on this change, so please head over there and give it. Thanks!

So Long Mapi Editor…And Thanks For All The Fish

Stephen Griffin — Mon, 15 Jun 2009 13:44:36 GMT

I’ve touched on the history of MFCMAPI before. The rebranding of MFCMAPI as MAPI Editor and posting it on the Microsoft Download site back in 2006 served a purpose, one for which at the time there were few alternatives. However, since then, I have taken back control over publishing of the tool so I can get updates and fixes to my users faster (as opposed to “never”, which was the case with that download).

Since setting up the Codeplex site, one thing that’s been bugging me is that the “MAPI Editor” download was still fairly popular. No amount of evangelizing the new builds on the Codeplex site would prevent people from searching for MFCMAPI on Microsoft.com and coming up with what I view as an old and buggy build. So, last week, we “fixed the glitch”. If you visit the download page now, instead of MAPI Editor, you’ll find a link to Codeplex. And if you download the file that’s there (apparently – there has to be a file to have a page on the download site), it’s a PDF that also directs you to Codeplex.

The switch was flipped last Thursday, and my download stats already show a huge spike. So the redirection must be working! Hopefully now I’ll stop seeing problem reports on that old build.

Troubleshooting Outlook Crashes

Stephen Griffin — Thu, 11 Jun 2009 13:42:46 GMT

I recently reviewed an excellent article on troubleshooting Outlook crashes which was just published in the Knowledge Base. While I no longer work in Outlook product support directly, in my role as a Developer Support engineer, I see Outlook crashes all the time. This article encapsulates just about everything I or other engineers here would do to narrow down and define an issue before we attach a debugger. Even if you’re already familiar with most of the techniques given in the article, it’s still handy to keep around because it puts everything you need in one place. For instance, it includes:

Complete list of everything which changes in Safe Mode
An always current pointer to the latest updates for Office
Registry keys for disabling add-ins
Lists of temp files which are safe to delete
Command line parameters

If you support Outlook, I highly recommend keeping this article in your bookmarks.

Enjoy!

But I Really Need To Open 500 PSTs!

Stephen Griffin — Tue, 09 Jun 2009 14:18:00 GMT

Today we’re revisiting an issue I first encountered in Outlook 2003. We had a customer, a law firm, who kept all the mail associated with a particular case in a PST. Their users would pass these PSTs around, and depending on their case load, a single user might have over 100 PSTs open at the same time. This worked fine when they had Outlook 2000, but when they installed Outlook 2003 they found if they tried to open more than 20 or so PSTs, they started getting errors such as “Outlook failed to add the personal store to the session” and "Out of memory or system resources. Close some windows or programs and try again."

Although we don’t recommend opening so many PSTs at the same time, we investigated to find out what had changed, and whether we could help them out. In order to understand what had changed, it helps to know a bit about how the PST provider, and by proxy, Outlook, manages open PSTs.

There are two mechanisms involved in the PST provider’s management of PST files to consider. The first is the per-PST cache. Any time you’re dealing with random reads/writes to a file, maintaining an in memory cache is critical to performance. The PST provider uses an algorithm based on the file size and other internal details about the PST to determine how large of an in-memory cache to allocate. In Outlook 2002, we greatly increased the size of the caches we use for all types of PST files. Larger blocks of memory mean we consume more of the address space available to the process. In practical terms, this meant we could open about 20 average sized PSTs before we were no longer able to allocate a block of memory large enough to hold the cache for the next file. As a test, I tried to open brand new PSTs and was able to get 85 of them before we ran out of memory.

Based on our investigation, and the needs of this and other customers, we decided to go in and fine tune the cache sizes to allow more PSTs to be opened at once. This involved a tradeoff in performance, so we placed this change behind a registry key. The KB article 907651 documents the hotfix we generated for Outlook 2003, as well as the UseLegacyCacheSize registry key. Outlook 2007 includes the fine tuned cache as well, but the path to the registry key changes slightly. Whereas in Outlook 2003 the key is under “HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\PST”, in Outlook 2007 you’ll set it under “HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\PST”. (Incidentally, this fix was backported to Outlook XP too. There, the reg key will have a 10.0 in it.)

Now, once this registry key is set, if we try and stress test how many PSTs we can open, we’ll see we hit a practical limit of just under 300, but, unlike before, our address space isn’t exhausted. And where before, if your MAPI based process could only open 20 PSTs, you could always run another process and open 20 more, this new upper limit is spread across all MAPI processes on the box. This is due to the second mechanism the PST provider uses, shared memory.

As you might recall from my discussion of PST files in multiple profiles, the PST provider uses shared memory to track and coordinate access to PSTs across multiple processes. Each PST opened incurs a tax on this shared memory, and after enough PSTs are opened, there is no longer space in this block to track any more PSTs. As part of the work we did in introducing the UseLegacyCacheSize, we also increased the size of this shared memory block, which is what will allow you to reach nearly 300 open PSTs across multiple processes.

Now – just because you can open nearly 300 PSTs doesn’t mean you should. You’ll start to see noticeable performance issues when you get over 100 PSTs. In fact, if shouldn’t set UseLegacyCacheSize at all if you don’t need it. The smaller cache has a performance penalty. If you have an application which for some reason needs to open a large number of PSTs, I would consider designing it not to need all of them open at the same time. Only open the PST when you need it and unload it immediately after.

Accepting Appointments With CDO May Break Embedded Message Attachments

Stephen Griffin — Mon, 08 Jun 2009 19:24:00 GMT

This is an issue that has only been reproduced so far with Exchange 2003, but I wanted to document it here in case anyone else is running in to it. Suppose you have a CDO 1.21 based application which uses GetAssociatedAppointment to accept meeting requests. Suppose further that this application receives and processes a meeting request that contains an embedded message. The resulting appointment item created on the calendar will have an attachment on it, but this attachment will be unusable.

What’s going on: When you invoke CDO’s GetAssociatedAppointment call, if CDO doesn’t find a matching appointment on the calendar, it creates one. Part of this creation includes copying any attachments on the source meeting request over to the new appointment item. CDO does this by walking the attachment table on the source message, calling OpenAttach on each attachment. On the destination message, it calls CreateAttach, then uses CopyTo to copy from the source to the destination. No properties are excluded and no specific properties are requested. On the embedded messages, this call doesn’t return an error code, but looking with MFCMAPI, we see that PR_ATTACH_DATA_OBJ doesn’t get copied.

To understand this, I debugged the Store process on the Exchange 2003 server. This CopyTo operation is handled by a special function in Store that deals with copying attachments. This function has two main code paths, one called fast and one called slow, though it doesn’t look like one is necessarily faster than the other. The fast code path looks like it copies everything at once, while the slow code path reads all the properties from the source attachments and writes them one at a time on the destination. Oh – and PR_ATTACH_DATA_OBJ is specifically excluded from this copy. Since this property, which essentially is the embedded message, isn’t copied, the resulting attachment is no good as an embedded message. It’s unclear why the slow path excludes this property – it appears to have always worked this way, at least as far back as Exchange 5.5.

What’s also unclear is when we take this slow path. Given that this problem has only just now been observed, we must normally take the fast path. The specific test being made to determine which path to take is whether the column set of the destination attachment table is a subset of the column set of the source attachment table. It appears that usually, it is, and the fast path is chosen. But there are apparently circumstances, on Exchange 2003 at least, and perhaps depending on how the meeting request entered Exchange, under which the destination attachment table has columns not present in the source attachment table, and the slow path is taken, which doesn’t copy the embedded message portion of the attachment.

The good news here is that in Exchange 2010, these two paths have been merged. The winner in the merge was the slow path, but this path has been augmented specifically to handle the embedded message case. So this problem should be impossible to hit with Exchange 2010.

I didn’t get a chance to test workarounds for this issue, but someone who is encountering it might try checking the attachment table on the meeting request, and if embedded messages are found, delete them from the new appointment and recopy them manually.

[Edit] The customer who reported this to me wanted to point out that the problem seems to only happen when the embedded message comes from another Active Directory domain. The problem has not been observed in single domain scenarios.

The Fifth MAPI Multithreading Rule

Stephen Griffin — Fri, 22 May 2009 14:37:09 GMT

We had an issue recently where DDE broadcasts were being blocked on a system. The customer noticed that the problem happened after they called MAPIInitialize on their worker thread. Our investigation gives us a chance to add a 5th rule to the MAPI Multithreading Rules I wrote about a few years ago:

5. The first thread to initialize MAPI must pump messages.

To understand this, first we need to know a bit about DDE. Turns out, Raymond wrote about it a couple times already, so I don’t need to rehash that. The second thing to know is that MAPI uses a hidden window for idle processing. If you dig around a bit, you might even find the window, named “WMS Idle”. MAPI only needs one of these per process, so it’s created on the first thread to initialize MAPI. And as we know, any thread with a window on it must pump messages.

In the customer’s case, they were pumping on the main thread, but not on the MAPI worker thread. Once they inserted message pumping into the MAPI worker thread, DDE was no longer blocked. Alternatively, they could have called HrDispatchNotifications, which is a simple way to pump messages, or called MAPIInitialize on the main thread.

(Less Than) Portable Search Folders

Stephen Griffin — Thu, 21 May 2009 20:47:13 GMT

Let’s have fun with the object model and search folders:

Outlook running a profile in online mode.
Create this macro:
Sub TestAdvancedSearchComplete()

    Dim sch As Outlook.Search

    set sch = Application.AdvancedSearch("Inbox", "urn:schemas-microsoft-com:office:office#Keywords like 'Test'", True, "MySearchFolder")

    sch.Save ("MySearchFolder")

End Sub
Run the macro, observe the folder.
On the same machine, create a new profile in cached mode and start Outlook
Run macro, get “Run-time error '-2147219964 (80040604)': Cannot create folder.”

If we want to have more fun, instead of running the macro the second time, we can locate the folder “MySearchFolder” under the Search Folder node in Outlook. If we click on it, Outlook creates a search folder, but it doesn’t populate with the items we expect to find.

What’s going on here? To answer that, we have to know a bit about how Outlook’s persisted search folder feature works. When you create a search folder, either using the Object Model, or directly in Outlook, the folder isn’t the first thing created. The first thing created is the search folder definition message. This is a message with the message class “IPM.Microsoft.WunderBar.SFInfo” which lives in the associated contents of the Common Views folder. We can use MFCMAPI to take a look at this message:

And we can use MFCMAPI’s Smart View feature to parse the search folder definition stored in PR_WB_SF_DEFINITION:

From this, we find that the search folder definition contains the restriction that will form the basis of the search folder:

Restriction:
lpRes->rt = RES_CONTENT
lpRes->res.resContent.ulFuzzyLevel = FL_IGNORECASE | FL_FULLSTRING = 0x00010000
lpRes->res.resContent.ulPropTag = 0x8010101F (PT_MV_UNICODE)
lpRes->res.resContent.lpProp->ulPropTag = 0x8010001F (PT_UNICODE)
lpRes->res.resContent.lpProp->Value = Test
  Alt: cb: 8 lpb: 5400650073007400

Note that this restriction includes a property in the 0x8000 range, meaning it’s a named property. In fact, it’s the named property “Keywords” in the PS_PUBLIC_STRINGS namespace. Note also that there’s nothing in the restriction which tells me this. I only know it because I created the search folder and I know what the search folder is supposed to be looking for.

Now – what happens when we go to a different profile looking at the same mailbox? The search folder itself doesn’t get synched back to the server. Only this search folder definition message gets synced. When Outlook encounters this message, it places a dummy node under Search Folders and waits for the user to click on it. If the user does click on it, it builds the search folder using the information from PR_WB_SF_DEFINITION.

And that’s where the problem comes in: Named property mappings are store specific. A named property mapping that’s valid for one store will most likely be invalid for another store. When we switched to cached mode, we’re now working with the OST, not Exchange. Even if there happens to be an 0x8010101F property in this store, it’s not going to be the “Keywords” property. So when we clicked on the folder in Outlook, it created a search folder that searched for a bogus property.

What happened when we ran the macro is even more interesting: Outlook looked at our search string and built a search folder definition message with the appropriate PR_WB_SF_DEFINITION. However, it couldn’t save the message because it already existed, with a different definition!

Why don’t we have a problem when we create search folders directly in Outlook? The Outlook user interface severely restricts the kinds of restrictions you can set up. Because of that, Outlook is able to store the PR_WB_SF_DEFINITION in a different format (using SFST_FILTERSTREAM instead of SFST_MRES). That format is more flexible with named properties, but isn’t capable of storing the wide variety of restrictions that could be specified through the Outlook Object Model.

Workaround

We did look at taking a fix for this, but the limitations of SFST_FILTERSTREAM eliminated it as a possibility for a fix. And the fact that this problem only happens if the restriction contains a named property (IE it won’t repro on a search for subject or recipients) limited the scope of the problem.

However, for the case where the search folder is being created by a macro and we’re getting the “Cannot create folder” error, we found a neat workaround. We can use Outlook’s PropertyAccessor to delete the search folder definition message so it can be recreated. Here’s a sketch of the code:

Sub DeleteSFItem()
    Dim CommonViewsEIDBin As String
    Dim CommonViewsEIDString As String
    Dim CommonViewsFolder As Folder
    Dim ACTable As Table
    Dim oRow As Row
    Dim SFDefinitionEID As String
    Dim SFDefinitionItem As StorageItem
        
    CommonViewsEID = Session.DefaultStore.PropertyAccessor.GetProperty( _
        "http://schemas.microsoft.com/mapi/proptag/0x35E60102")
    CommonViewsEIDString = Session.DefaultStore.PropertyAccessor.BinaryToString( _
        CommonViewsEID)
    Set CommonViewsFolder = Session.GetFolderFromID(CommonViewsEIDString)
    Set ACTable = CommonViewsFolder.GetTable( _
        "[Subject] = 'MySearchFolder'", olHiddenItems)
    
    Set oRow = ACTable.GetNextRow()
    If (Not (oRow Is Nothing)) Then
        SFDefinitionEID = oRow("EntryID")
        Set SFDefinitionItem = Session.GetItemFromID(SFDefinitionEID)
        SFDefinitionItem.Delete
    End If
End Sub

For further reading on the inner workings of search folder definition messages, check out the Exchange Protocol Doc [MS-OSOSRCH].

May 2009 Release of MFCMAPI

Stephen Griffin — Wed, 20 May 2009 14:26:08 GMT

The May 2009 Release (build 6.0.0.1013) is live: http://mfcmapi.codeplex.com.

The coolest new feature has to be the addition of Smart View parsing to the Hex Editor. I actually had the idea to do this the morning I published the March 2009 release, but didn’t want to delay that release any longer. One side effect of exposing the parsing in the Hex Editor is I was able to do much better testing of the existing parsing engine and fix a bunch of memory leaks and crashes.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

Added ability to copy attachments from one message to another
Added Outlook version feature - uses MSI to retrieve installed version of Outlook
Added Smart View to hex editor
Fixed issue converting MAPI to MIME from an MSG file
Fixed parsing of fuzzy level flags
Updated MAPI headers
Removed extraneous pointer address output from restriction parsing
Added smart view parsing for Entry Lists, Properties, Restrictions, and Search Folder Definitions
Cleaned up memory leaks and crashes in smart view parsing

Enjoy.

Unaligned Data and the PST Provider Redux

Stephen Griffin — Fri, 08 May 2009 15:31:13 GMT

Just wanted to note that the April Cumulative Update for Outlook 2007 fixes the unaligned data issue with the PST provider. So passing a statically allocated NULL string as a PST password should no longer be a problem.

BTW – Outlook’s update isn’t listed in the overview article for the April Update. This is because the Outlook package was delayed for a few days. The overview article should eventually be updated to include Outlook.

Exchange Client Extension Dead In Outlook 2010

Stephen Griffin — Mon, 04 May 2009 23:53:00 GMT

Randy makes the official announcement over here. This shouldn't come as a shock considering we've been pushing Com Add-Ins for some time now and hinting that ECEs weren't long for this world. So - if your solution uses an Exchange Client Extension and you want it to work with Outlook 2010, you better get started rewriting it as a Com Add-In now. Have fun!

Office 2007 SP2 is Live!

Stephen Griffin — Tue, 28 Apr 2009 18:28:33 GMT

Get it here:

http://www.microsoft.com/downloads/details.aspx?FamilyId=B444BF18-79EA-46C6-8A81-9DB49B4AB6E5