A little follow-up to the Outlook Rules post, concerning getting rules to fire when you’ve got a wrapped PST store. I had noted in the original post that rules “just worked” in Outlook 2003. Turns out I was partially right. They can work in Outlook 2003, but you have to do something a little special that I had somehow overlooked in my original investigation. Here’s how I explained your should fire a notification for Outlook 2003:
In Outlook 2003, the way you solve this is to use the IMAPISupport object that was passed to you during logon. From there, you would call Notify, passing fnewNewMail and the details of the new message.
The tweak here is in the structure you pass to into the Notify call. Since we’re passing fnevNewMail, we use a NEWMAIL_NOTIFICATION structure, which contains a ulFlags member. The only flag currently documented for this structure is MAPI_UNICODE. To get notifications working in Outlook 2003, we need to OR in a new flag, MSGFLAG_OUTLOOK_NON_EMS_XP. This flag has the same effect the NON_EMS_XP_SAVE flag has for Outlook 2007, which is to let Outlook know this message did not originate from the Exchange transport, so it knows rules need to be processed client side. This flag is only required for Outlook 2003, though we’ve seen no side-effect to also passing it in Outlook 2007. Unlike Outlook 2007, Outlook 2003 does NOT use the ITEMPROC_FORCE flag.
Definition of the new flag:
We will most likely not be adding this flag to the Outlook MAPI documentation since it only applies to Outlook 2003.
With the recent Rollup 8 for Service Pack 1, the Exchange team previewed a change designed to reduce the impact of inbound SMTP messages on the named property map for a store. Jason talked about it here. The gist of the change is that before, random e-mails with random x-headers could exhaust the map, making it impossible for legitimate applications, like Outlook, to map new named properties. With Rollup 8, those x-headers will only be promoted if they’ve already been mapped, or if the mail was submitted by an authenticated user.
Jason’s latest post details where we’re taking this in SP2 and Exchange 2010. An x-header will only be promoted to a named property if the mapping already exists, either because it was mapped before SP2 was applied, or a MAPI based application requested the mapping. We know this breaks a few scenarios, but felt it was a reasonable tradeoff to preserve the stability of the Exchange server.
Jason has asked for feedback on this change, so please head over there and give it. Thanks!
I’ve touched on the history of MFCMAPI before. The rebranding of MFCMAPI as MAPI Editor and posting it on the Microsoft Download site back in 2006 served a purpose, one for which at the time there were few alternatives. However, since then, I have taken back control over publishing of the tool so I can get updates and fixes to my users faster (as opposed to “never”, which was the case with that download).
Since setting up the Codeplex site, one thing that’s been bugging me is that the “MAPI Editor” download was still fairly popular. No amount of evangelizing the new builds on the Codeplex site would prevent people from searching for MFCMAPI on Microsoft.com and coming up with what I view as an old and buggy build. So, last week, we “fixed the glitch”. If you visit the download page now, instead of MAPI Editor, you’ll find a link to Codeplex. And if you download the file that’s there (apparently – there has to be a file to have a page on the download site), it’s a PDF that also directs you to Codeplex.
The switch was flipped last Thursday, and my download stats already show a huge spike. So the redirection must be working! Hopefully now I’ll stop seeing problem reports on that old build.
I recently reviewed an excellent article on troubleshooting Outlook crashes which was just published in the Knowledge Base. While I no longer work in Outlook product support directly, in my role as a Developer Support engineer, I see Outlook crashes all the time. This article encapsulates just about everything I or other engineers here would do to narrow down and define an issue before we attach a debugger. Even if you’re already familiar with most of the techniques given in the article, it’s still handy to keep around because it puts everything you need in one place. For instance, it includes:
If you support Outlook, I highly recommend keeping this article in your bookmarks.
Enjoy!
Today we’re revisiting an issue I first encountered in Outlook 2003. We had a customer, a law firm, who kept all the mail associated with a particular case in a PST. Their users would pass these PSTs around, and depending on their case load, a single user might have over 100 PSTs open at the same time. This worked fine when they had Outlook 2000, but when they installed Outlook 2003 they found if they tried to open more than 20 or so PSTs, they started getting errors such as “Outlook failed to add the personal store to the session” and "Out of memory or system resources. Close some windows or programs and try again."
Although we don’t recommend opening so many PSTs at the same time, we investigated to find out what had changed, and whether we could help them out. In order to understand what had changed, it helps to know a bit about how the PST provider, and by proxy, Outlook, manages open PSTs.
There are two mechanisms involved in the PST provider’s management of PST files to consider. The first is the per-PST cache. Any time you’re dealing with random reads/writes to a file, maintaining an in memory cache is critical to performance. The PST provider uses an algorithm based on the file size and other internal details about the PST to determine how large of an in-memory cache to allocate. In Outlook 2002, we greatly increased the size of the caches we use for all types of PST files. Larger blocks of memory mean we consume more of the address space available to the process. In practical terms, this meant we could open about 20 average sized PSTs before we were no longer able to allocate a block of memory large enough to hold the cache for the next file. As a test, I tried to open brand new PSTs and was able to get 85 of them before we ran out of memory.
Based on our investigation, and the needs of this and other customers, we decided to go in and fine tune the cache sizes to allow more PSTs to be opened at once. This involved a tradeoff in performance, so we placed this change behind a registry key. The KB article 907651 documents the hotfix we generated for Outlook 2003, as well as the UseLegacyCacheSize registry key. Outlook 2007 includes the fine tuned cache as well, but the path to the registry key changes slightly. Whereas in Outlook 2003 the key is under “HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\PST”, in Outlook 2007 you’ll set it under “HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\PST”. (Incidentally, this fix was backported to Outlook XP too. There, the reg key will have a 10.0 in it.)
Now, once this registry key is set, if we try and stress test how many PSTs we can open, we’ll see we hit a practical limit of just under 300, but, unlike before, our address space isn’t exhausted. And where before, if your MAPI based process could only open 20 PSTs, you could always run another process and open 20 more, this new upper limit is spread across all MAPI processes on the box. This is due to the second mechanism the PST provider uses, shared memory.
As you might recall from my discussion of PST files in multiple profiles, the PST provider uses shared memory to track and coordinate access to PSTs across multiple processes. Each PST opened incurs a tax on this shared memory, and after enough PSTs are opened, there is no longer space in this block to track any more PSTs. As part of the work we did in introducing the UseLegacyCacheSize, we also increased the size of this shared memory block, which is what will allow you to reach nearly 300 open PSTs across multiple processes.
Now – just because you can open nearly 300 PSTs doesn’t mean you should. You’ll start to see noticeable performance issues when you get over 100 PSTs. In fact, if shouldn’t set UseLegacyCacheSize at all if you don’t need it. The smaller cache has a performance penalty. If you have an application which for some reason needs to open a large number of PSTs, I would consider designing it not to need all of them open at the same time. Only open the PST when you need it and unload it immediately after.
This is an issue that has only been reproduced so far with Exchange 2003, but I wanted to document it here in case anyone else is running in to it. Suppose you have a CDO 1.21 based application which uses GetAssociatedAppointment to accept meeting requests. Suppose further that this application receives and processes a meeting request that contains an embedded message. The resulting appointment item created on the calendar will have an attachment on it, but this attachment will be unusable.
What’s going on: When you invoke CDO’s GetAssociatedAppointment call, if CDO doesn’t find a matching appointment on the calendar, it creates one. Part of this creation includes copying any attachments on the source meeting request over to the new appointment item. CDO does this by walking the attachment table on the source message, calling OpenAttach on each attachment. On the destination message, it calls CreateAttach, then uses CopyTo to copy from the source to the destination. No properties are excluded and no specific properties are requested. On the embedded messages, this call doesn’t return an error code, but looking with MFCMAPI, we see that PR_ATTACH_DATA_OBJ doesn’t get copied.
To understand this, I debugged the Store process on the Exchange 2003 server. This CopyTo operation is handled by a special function in Store that deals with copying attachments. This function has two main code paths, one called fast and one called slow, though it doesn’t look like one is necessarily faster than the other. The fast code path looks like it copies everything at once, while the slow code path reads all the properties from the source attachments and writes them one at a time on the destination. Oh – and PR_ATTACH_DATA_OBJ is specifically excluded from this copy. Since this property, which essentially is the embedded message, isn’t copied, the resulting attachment is no good as an embedded message. It’s unclear why the slow path excludes this property – it appears to have always worked this way, at least as far back as Exchange 5.5.
What’s also unclear is when we take this slow path. Given that this problem has only just now been observed, we must normally take the fast path. The specific test being made to determine which path to take is whether the column set of the destination attachment table is a subset of the column set of the source attachment table. It appears that usually, it is, and the fast path is chosen. But there are apparently circumstances, on Exchange 2003 at least, and perhaps depending on how the meeting request entered Exchange, under which the destination attachment table has columns not present in the source attachment table, and the slow path is taken, which doesn’t copy the embedded message portion of the attachment.
The good news here is that in Exchange 2010, these two paths have been merged. The winner in the merge was the slow path, but this path has been augmented specifically to handle the embedded message case. So this problem should be impossible to hit with Exchange 2010.
I didn’t get a chance to test workarounds for this issue, but someone who is encountering it might try checking the attachment table on the meeting request, and if embedded messages are found, delete them from the new appointment and recopy them manually.
[Edit] The customer who reported this to me wanted to point out that the problem seems to only happen when the embedded message comes from another Active Directory domain. The problem has not been observed in single domain scenarios.