[This is now documented here: http://msdn.microsoft.com/en-us/library/ee201126(EXCHG.80).aspx ]
Those of you wrapping PSTs may see something interesting in the restrictions Outlook 2007 will try to set against your folders. There are a couple of new restriction types you won't have seen before. Outlook will only use these restriction types when it's working with stores it knows supports them (like a PST), so in general you can ignore them and just pass the restriction on to the underlying store. However, if you're trying to make a copy of the restriction structure for some later use, then you need to know what they look like in memory.
Here are the definitions of the two new restriction types:
#define RES_COUNT ((ULONG) 0x0000000B)
#define RES_ANNOTATION ((ULONG) 0x0000000C)
The layout in memory of the associated restrictions is the same as some existing types, which should make parsing them for the purposes of copying easier. RES_COUNT is identical to RES_NOT and RES_ANNOTATION is identical to RES_COMMENT, so to copy a restriction of type RES_COUNT, you can use the resNot member of the union, and for RES_ANNOTATION you can use the resComment member. As an example, we look at how we would fix up MFCMAPI's HrCopyRestrictionArray function. We can take advantage of our existing code by just adding the new cases to the switch. This works since the structures involved are identical:
LPSRestriction lpResSrc, // source restriction
The next update to MFCMAPI will have these changes (and more) in it.
Back when Outlook 2003 shipped, we were looking for ways to reduce the number of calls we made to the server. One optimization we found was to include the FQDN of the server in the entry ID for an Exchange store. In some situations, this eliminated the need to call back to the server to get a referral. However, this also had two side effects:
Now - technically, 2 shouldn't be a concern because we all know you're supposed to use CompareEntryIDs when comparing entry IDs (hence the name of the function). However, wouldn't it be nice if you could get the old style entry ID for scenarios where you're space constrained, or want to avoid the potentially expensive CompareEntryIDs call? Now you can. Whenever you want to ask for PR_STORE_ENTRYID, you can ask for PR_STORE_ENTRYID_EMSMDB_V1 instead. It's defined as follows:
#define PR_STORE_ENTRYID_EMSMDB_V1 PROP_TAG(PT_BINARY, 0x65f6)
This will return the "old style" entry IDs that we used to use back in Outlook XP. They're still valid now, though using them to open a mailbox may incur some additional RPCs if a referral is required. Also note that while you can memcmp two of these style entry IDs to see if they're equal, you should not interpret a difference to mean they point at different mailboxes. To determine that they're truly different you'd still need to use CompareEntryIDs.
Some other things to keep in mind: If you're in cached mode, to ask for PR_STORE_ENTRYID_EMSMDB_V1 you have to bypass the cache using MAPI_NO_CACHE in your GetProps call. Also, in case the property isn't available, your code should fall back to PR_STORE_ENTRYID. Finally, only Outlook 2003 and higher's version of MAPI supports this property.
I picked up a case a few weeks ago that had a simple question in it. "Are MAPI's profile APIs thread safe?" My answer, of course, was "As far as I know - why do you ask?". The reason they asked was because they had built a framework to encapsulate all of their MAPI operations (a very good idea) and they were seeing "issues" when they ran the profile configuration portion of that framework through their multithreaded test harness.
At first, I had a little trouble pinning down what they meant by "issues" - did they mean hangs? Crashes? Random error codes? Actually - it was all of the above. I got some dumps and traces from them and found 2-3 distinct problems with the way MAPI and some of the providers behave in an non thread safe manner during profile configuration/deletion. When I started running my own tests, I found a few more.
Now - I've always understood MAPI and the providers we ship to be thread safe. Sure, there are the occasional issues, but we usually can fix them. We go to a lot of trouble to make sure MAPI behaves correctly across multiple threads. But apparently all of this work, and all of our testing, was under the assumption that the profiles had already been created. I dug back through records of cases I had worked as well as bugs which have been filed against MAPI and found only one case where someone had configured profiles from multiple threads. And on examining that case, I saw that it was from the days before we routinely had multiple processors. Most of the threading issues I was seeing would be much harder to reproduce on a single processor.
I took the list of issues we found back to development to see what they thought we should do about this. The issues we found were in both Outlook and Exchange's implementation of MAPI (they predate the split), meaning we had a long road ahead of us if we started fixing the problems. And even if we assume we fixed every issue we found, we were certain to find that those issues were hiding other, nastier issues.
So we decided that the best course of action here would be to declare MAPI's profile API's not to be thread safe. All MAPI profile configuration should be serialized. This applies to both Exchange and Outlook's implementations of MAPI and all versions, including the MAPI download.
Here are the APIs you'll need to serialize as a MAPI client:
If you're just looking to use MFCMAPI to get in to the RSG, head on over to Jesper Bernle's blog where he gives step-by-step instructions. Otherwise, read on.
I promised I'd talk about accessing the Recovery Storage Group (RSG). The RSG is a neat trick we allow starting in Exchange 2003 where you can restore a copy of a database in your production environment without taking your user's mailboxes offline. This allows you to recover e-mails, archive, run discovery, etc. The concept is discussed in depth here: http://support.microsoft.com/kb/824126
Up until the February 2008 update to MFCMAPI, though, you couldn't use MFCMAPI to get into the RSG. In fact, a lot of people were convinced that MAPI couldn't be used to access the RSG at all! Of course, that can't be true, given that ExMerge, which is written entirely in MAPI, has no trouble getting in there. A customer asked Dave and I what the trick is, and that led to this article.
Let's look at what happens when you try to get into an RSG mailbox from MFCMAPI. Assuming you've read Jason's blog on the subject, you might try the following steps:
Sounds great. Doesn't work. When you open the RSG mailbox this way, you get the regular mailbox instead. Why? Because when you do this, MFCMAPI grabs the DN of the mailbox from the PR_EMAIL_ADDRESS column and passes that to CreateStoreEntryID. And PR_EMAIL_ADDRESS for the regular mailbox is the same as the PR_EMAIL_ADDRESS for the RSG mailbox, so we're always going to create the same entry ID.
So how does ExMerge it do it? Is it using secret magic? Yep! It goes through a very complicated procedure to build a MAPI profile that points to the RSG. However, all that work really results in an entry ID that has special flags set on it to tell Exchange that the mailbox we want to open isn't the regular mailbox but instead the RSG version of the mailbox.
I *could* go into the stuff that ExMerge does in building its profile, but it turns out getting into the RSG is much easier than that. Those flags set on the entry ID that ExMerge got can be set by hand using the ulFlags parameter of CreateStoreEntryID! And since using CreateStoreEntryID is always preferable to building a new profile, that's all I'll talk about here. :)
Here's the key flag, already listed in edkmdb.h:
#define OPENSTORE_RESTORE_DATABASE ((ULONG)0x00100000)
When you set this flag, it tells emsmdb32 and the Exchange store you're interested in getting the RSG version of the mailbox. Of course, life isn't ever this simple. You have to set some other flags too, most notably OPENSTORE_USE_ADMIN_PRIVILEGE, which tells the store you want to be treated as an administrator, assuming you have permissions. I have found the following cocktail to work the best:
OPENSTORE_USE_ADMIN_PRIVILEGE | OPENSTORE_TAKE_OWNERSHIP | OPENSTORE_RESTORE_DATABASE |
OPENSTORE_OVERRIDE_HOME_MDB | OPENSTORE_HOME_LOGON = 0x10001D
Now we see what the new feature I added to MFCMAPI allows you to do - instead of double-clicking on the mailbox in the table, we can right click and pick "Open with Flags", enter the flags we wish to use (0x10001D) and we're in like Flynn!
[Update: 2/18 - add note clarifying this is Exchange's MAPI only]
[Update: 6/20 - Added link to Jesper's excellent how to article that puts this information into practice]
The February 2008 Release (build 184.108.40.2064) is live: http://www.codeplex.com/MFCMAPI
Here's the change list - see the Issue Tracker on Codeplex for more details, or look at the code:
Hey - look what I found on one of my old machines. It's the documentation from the Exchange 5.5 SDK / EDK!
We're not supporting development against 5.5 anymore, and a rather large portion of what's in these docs only applies to Exchange 5.5, but there are some interesting tidbits still. Some highlights:
Note - these are old .CHM files. If you can't read them I can't really help you. I'm only providing them here as a historical curiosity.
BTW - for some reason, we've never pulled the 5.5 samples and libraries. You can still get those from here:
The January '08 refresh for the Outlook 2007 Auxiliary Reference is now live. In addition to a general scrub over most of the topics, there are some interesting new topics added, such as:
Timezone/rebasing docs: http://msdn2.microsoft.com/en-us/library/bb820976.aspx and http://msdn2.microsoft.com/en-us/library/cc160684.aspx
Protecting open PST files when you crash: http://msdn2.microsoft.com/en-us/library/cc160695.aspx
Rules processing in the wrapped PST: http://msdn2.microsoft.com/en-us/library/bb820947.aspx
Additionally, my update to the wrapped PST has been incorporated.
[This is now documented here: http://msdn.microsoft.com/en-us/library/ff960239.aspx ]
We just had a customer whose Wrapped PST based store wasn't getting rules to fire in Outlook 2007, when everything was working in 2003. In the course of figuring their issue out, we found a few flags that we should have documented before.
Suppose you built a message store based on the Wrapped PST provider. When your new messages are delivered to your backend database and you create matching messages in the PST, how do you inform Outlook that a new mail has arrived? In Outlook 2003, the way you solve this is to use the IMAPISupport object that was passed to you during logon. From there, you would call Notify, passing fnewNewMail and the details of the new message. When Outlook gets this new message notification, it passes the Entry ID over to the rules engine and everything is hunky dory.
Unfortunately, this doesn't work in Outlook 2007. You fire your notification, Outlook sees it, then Outlook decides not to run rules on the message. Why? Because when Outlook 2007 looked at PR_STORE_SUPPORT_MASK on the wrapped PST, it found the STORE_ITEMPROC flag was set. This means the wrapped PST is an "Item Proc" store, and handles rules differently. Without getting into too many details about "Item Proc", the general idea is that such stores will feed new items into a pipeline where rules, junk mail, and spam processing can happen before listening clients will get a notification. This concept was introduced in Outlook 2007 as a way to streamline the processing.
When an Item Proc store knows it has a new message, it calls a special callback function in Outlook to run it through the rules engine. In the PST, the concept of "newly delivered message" is handled through internal flags. These flags may be set or not set depending on how the message arrives in the PST. For instance, if the PST is being used as an OST and new mail is synchronized from an Exchange server, the flags will be set. Similarly, when the PST is being used as a back end POP3/IMAP, the flags will be set when a new mail is delivered.
For a wrapped PST, however, the way you set these flags is to pass ITEMPROC_FORCE in the SaveChanges call when the newly delivered message is committed to the store. This tells the PST to set its internal flags and mark the item as being eligible for the callback.
There's a twist though. If the mail being delivered to the PST came from Exchange, then rules would already have run on the server and the message has to be handled differently. And the default assumption is that the mail came from Exchange. So you have to indicate that Exchange was not the source of the mail by also passing the NON_EMS_XP_SAVE flag. When both flags are passed, the item will go through rules/junk/spam processing as soon as you call SaveChanges if Outlook is running, and shortly after Outlook starts if it wasn't.
Definitions of the new flags:
Whenever I find myself repeating the same message over and over again, I have to ask why I haven't blogged it yet. This is one of those cases. :)
I've seen quite a few issues over the years with MSG files. The issues range from "it takes too long to write properties" to "the properties on the MSG don't match what I see in the store" to "I get such and such error trying to copy this message to an MSG". The root cause of most of these issues is one of expectations. People are trying to use MSG files as an archival format, and that's not their intended purpose. If you really want to archive mail, you should develop your own format for persisting the data. You'll gain advantages in versatility, speed, and fidelity.
To understand why I make this recommendation, we first need to realize that not all messages can be copied over to the MSG format. This is noted at the end of http://support.microsoft.com/kb/171907. Since MAPI is transacted, the underlying MSG file has to be opened with STGM_TRANSACTED, meaning nothing is committed to disk until SaveChanges is called on the message. Couple that with the quirk in the MAPI specification that pretty much forces you to create a new transaction each time you add a recipient or attachment and you quickly run into the limit on open root storage files noted in http://support.microsoft.com/kb/163202. This OS imposed limit on open root storage objects isn't likely to ever change, as it's an artifact of the implementation. Likewise, the need for new transactions for each recipient and attachments also won't ever change. Neither the MSG format nor structured storage have seen active development in years. This limit is going to be hit whenever a message has a large number of recipients or attachments, or when there exist a deep level of embedded messages.
The next issue is speed. Writing a message to MSG can be quite slow. There is a huge performance penalty working with a structured storage file in STGM_TRANSACTED mode. And this penalty is multiplied by the number of open root storage objects. So not only do you run into a limit trying to add all those recipients and attachments, but each subsequent recipient and attachment is that much slower to add. For instance, I recently worked on an issue where the repro required that I have 5000 recipients on a message that I then copied over to MSG format. It took over an hour to write the file. And none of that delay was actually in the MAPI code - it was all at the COM level.
Next - not every MSG file you can write can be opened by Outlook. Over the years folks have tried various tricks to squeeze performance out of the code writing their MSG files. In many cases, they succeeded in writing the file faster, or allowing more recipients and attachments on the message. But the downside was they wrote a file that Outlook didn't know how to open! One variation of this issue surfaced with Outlook 2007. Given the performance problems working with MSG files, in Outlook 2007 we decided to check the number of recipients and attachments when opening the file. If either was over 2048, then we refused to open the file at all. The main reasoning for this was a number of corrupt MSG files that had surfaced in the wild with astronomical counts of recipients and attachments - on the order of millions. But a side effect was to block Outlook 2007 from opening some MSG files that Outlook 2003 could open. We've had some customers complain about this one and a fix is in the works. I'll report here when it's done. However, that fix will only cover this one variation of the problem. It won't fix the large number of other scenarios out there.
That covers the mechanics of reading and writing to MSG. Now we discuss fidelity. This isn't about whether the MSG format is out partying with the EML format, but rather how faithfully the MSG represents the source message. This is where MSG being a MAPI based format gets you in trouble. For instance, in archival scenarios, especially when the archive is used for legal discovery, properties such as PR_LAST_MODIFICATION_TIME and PR_LAST_MODIFIER_NAME are very important as they indicate who modified the message and when. But since MSG is itself a MAPI message and has such has to obey all the rules of MAPI, those properties will only reflect the time the MSG was written and the name of the account that wrote it, both of which aren't likely to match the original message. This problem can extend to the body properties as well: no matter how you do it, you're likely to end up converting the body from one format to another when storing it in the MSG file. And every conversion carries with it the possibility of a loss of data. Perhaps some line spacing is subtly changed, or font choices aren't preserved exactly. In some messages, these subtle textual differences could have huge semantic ramifications.
Fidelity also figures in when discussing Unicode. In a large organization, messages will be written in a variety of languages. The only way to preserve these messages into MSG format without converting half the characters to question marks or boxes is to use the Unicode format. Unfortunately, this format is only understood by Outlook 2003 and Outlook 2007. Exchange's MAPI doesn't understand this format at all. So if you're relying on MSG files to save out Unicode data, your solution is stuck using Outlook's implementation of MAPI for all processing of the archive. This severely hampers your ability to build a server based application.
So, we've got messages that cannot be copied to the archive, a painfully slow API, messages that cannot be opened once archived, and a format that's not capable of representing the actual message being archived. Clearly, these are not the attributes we want in an archive.
Fortunately, the workaround is simple: don't use MSG to archive messages. Instead, develop your own file format to preserve the important properties on a message. Here's one approach using the file system and XML files:
The only really hard part about this format is determining how to store each of the possible MAPI property types. However, when we look closely, we see there are only 13 types to consider, most of which can be represented as just a simple number or string. Even binary data is easy to store if it's first converted to hex. Multivalued properties, large binary and string properties, and named properties all add additional wrinkles, but are easily addressed. I figure a junior programmer could complete a reasonable first draft of the required code to both read and write a MAPI message to and from XML in an afternoon. In fact, most of the code for writing the XML format is already present in MFCMAPI - check out dumpstore.cpp.
Hopefully I've convinced most of you not to use the MSG file format for archiving. Some of you might not be convinced though. You might think you've got that one special case that requires you to use MSG. I don't believe such a case exists. I've anticipated a few of the common objections:
The final objection is my favorite: "But I've never had a problem with MSG files" - Bully for you! This article isn't addressed to you then. However, I had one customer who also made this claim when I found they were using MSG to archive messages. Not quite believing them though, I outlined each of the problems listed above. It turns out they had encountered or were encountering every single one of them. They just hadn't connected the problems back to their choice to use MSG to archive their data.
As promised, though a bit late, here are the change lists I put together for a couple of versions of MFCMAPI, along with a history lesson.
Versions 1-3 of MFCMAPI were essentially toys, built as I learned what could be done with MAPI. I can't even find them now - I wasn't very good at source control back then.
The oldest build of MFCMAPI I possess is 220.127.116.11, from 9/26/2001. At the time, it was for me just a collection of sample code that I would share with customers and other Dev Support engineers. Somewhere along the line, I found that Exchange and Outlook support folks wanted to use MFCMAPI as a tool. So I set up an internal site here at Microsoft where they could grab the latest stable build and use it. I didn't consider it ready for customers to use as a tool though, so I asked that I be kept in the loop any time it was sent out the door.
At some point (9/27/2002 to be exact) I bumped the major build number to 5.* to reflect the number of changes I had made recently. I also started work on the classic KB article: http://support.microsoft.com/kb/291794. It went live on 7/24/2003 with version 18.104.22.168. Minor build numbers were introduced along the way and I kept the article updated with builds and source. The last build I published in the KB was 22.214.171.1248 and it was built on 2/14/2005.
In August of 2004, a program manager for Exchange contacted me about having MFCMAPI officially replace MDBVU for Exchange 12, eventually known as Exchange 2007. Of course, I jumped at the chance. The experience of checking my code into the Exchange build tree was incredibly instructive. We ended up changing over half the code to get it up to the Exchange team's standards. We took our time getting the official Exchange build ready, so I kept up the KB releases for a while. Finally on 6/7/2006, we released the newly christened "MAPI Editor". Since this version was built out of the Exchange 2003 build tree, the version number was 06.05.7830.
As I stated at the time, my eventual goal was to get the source for this new build published. I also wanted to continually update the binary as I added features and bug fixes. But as we all know, when you hand control over to someone else - and their priorities aren't in sync with yours - what you want and what you get aren't always the same.
So - this past year, with the blessing of the Exchange team, I took control back. They can still publish an official build of "MAPI Editor" if they want, but I've put all the source and the releases for MFCMAPI up on CodePlex. The first release went live on 8/30/2007, and I've done three updates since then. Since I took over building the project, I resumed my old version numbering scheme, bumping the major build number to 6.
I've rambled enough - here are those diffs:
KB (126.96.36.1998) -> MAPI Editor Download (06.05.7830):
MAPI Editor Download (06.05.7830) -> Initial Codeplex population (188.8.131.520):
Ok - not exactly. But there are a few things you need to configure on your SQL box if you want SQLMail to follow the rules of MAPI multithreading.
Before we get too deep, a couple notes:
Avoiding crashes and hangs with SQLMail
Let's discuss that last one a bit, since there's a lot of SQLMail KB articles out there that mention running xp_startmail as part of a workaround. The folks that wrote those articles just aren't aware of what SQL does when xp_startmail is run and the effect it has on MAPI.
The thing to understand is that SQL uses a lot of threads. There's a thread pool that sits around ready to service various jobs. When SQL is processing a script, as it gets to various commands it will grab a thread from the thread pool, have it execute the command, then return it to the pool. So in the course of processing a single script, SQL might use a number of different threads. Suppose you have a script that executes xp_startmail, xp_sendmail and xp_stopmail. Each of these procedures could get handed to a different thread!
Why is this bad? Because xp_startmail is essentially a call to MAPInitialize and xp_stopmail is a call to MAPIUninitialize. So by using this commands, we will end up initializing MAPI on one thread and uninitializing it on a totally different thread. And this would be bad - very bad - crossing the streams bad.
Fortunately, xp_sendmail will call MAPIInitialize/MAPIUninitialize on its own, so we don't need xp_startmail or xp_stopmail at all!
Every SQLMail crash and hang I've debugged was resolved by implementing the above recommendations.
The December 2007 Release (build 184.108.40.2063) is live: http://www.codeplex.com/MFCMAPI
Yesterday, a new Developer Support engineer started here - Rick Hallihan, of OneManShouting fame. We snagged him away from the Home Server MVP program. (I hope we didn't make any enemies in doing so!) We're gonna teach him up good in MAPI and he'll in turn learn us all he knows about Home Server.
Speaking of Home Server, I don't usually spread links, but I thought this was funny:http://www.stayathomeserver.com/default.aspx
It looks like a parody, but I've been assured it's a real site run by the Home Server team. Not many videos up there yet, but the "Home Sweet Server" vid is great.
When I announced the Outlook 2007 Auxiliary Reference, I noted there were a few issues with the wrapped PST sample that I hadn't had time to fix. Well, they're fixed now. You can find the updated download here: http://www.microsoft.com/downloads/details.aspx?FamilyID=ee0af8fd-bbb7-44de-be4d-f33cb1b59563&displaylang=en
It should say Nov07 for version. If it says Oct07, then the new version hasn't propagated to all the servers yet. It should update soon.
BTW - while I was working on the last MFCMAPI update, I found a number of minor problems in the Auxiliary reference. They're not worth detailing here, but they've all been corrected internally and should be fixed when we do our next refresh. Let me know if you spot any problems in the reference and I'll do what I can to get them addressed.
The November 2007 Release (build 220.127.116.112) is live: http://www.codeplex.com/MFCMAPI
Here's the change list:
I also worked a bit on the front page of the Codeplex site as I had more than one person who went there to get the latest build and ended up with an old build from the MS Download site. It should be bit clearer where the latest build is now.
I haven't forgotten about that list of diffs from earlier builds I spoke about last time. I've just been busy with shipping the auxilliary reference and my transition from CPR back to Developer Support. Oh yeah - that reminds me - I don't work in CPR anymore. I'm still an Escalation Engineer, but a couple months ago I moved back to Developer Support to be closer to the dev type issues.