If you're just looking to use MFCMAPI to get in to the RSG, head on over to Jesper Bernle's blog where he gives step-by-step instructions. Otherwise, read on.
I promised I'd talk about accessing the Recovery Storage Group (RSG). The RSG is a neat trick we allow starting in Exchange 2003 where you can restore a copy of a database in your production environment without taking your user's mailboxes offline. This allows you to recover e-mails, archive, run discovery, etc. The concept is discussed in depth here: http://support.microsoft.com/kb/824126
Up until the February 2008 update to MFCMAPI, though, you couldn't use MFCMAPI to get into the RSG. In fact, a lot of people were convinced that MAPI couldn't be used to access the RSG at all! Of course, that can't be true, given that ExMerge, which is written entirely in MAPI, has no trouble getting in there. A customer asked Dave and I what the trick is, and that led to this article.
Let's look at what happens when you try to get into an RSG mailbox from MFCMAPI. Assuming you've read Jason's blog on the subject, you might try the following steps:
Sounds great. Doesn't work. When you open the RSG mailbox this way, you get the regular mailbox instead. Why? Because when you do this, MFCMAPI grabs the DN of the mailbox from the PR_EMAIL_ADDRESS column and passes that to CreateStoreEntryID. And PR_EMAIL_ADDRESS for the regular mailbox is the same as the PR_EMAIL_ADDRESS for the RSG mailbox, so we're always going to create the same entry ID.
So how does ExMerge it do it? Is it using secret magic? Yep! It goes through a very complicated procedure to build a MAPI profile that points to the RSG. However, all that work really results in an entry ID that has special flags set on it to tell Exchange that the mailbox we want to open isn't the regular mailbox but instead the RSG version of the mailbox.
I *could* go into the stuff that ExMerge does in building its profile, but it turns out getting into the RSG is much easier than that. Those flags set on the entry ID that ExMerge got can be set by hand using the ulFlags parameter of CreateStoreEntryID! And since using CreateStoreEntryID is always preferable to building a new profile, that's all I'll talk about here. :)
Here's the key flag, already listed in edkmdb.h:
#define OPENSTORE_RESTORE_DATABASE ((ULONG)0x00100000)
When you set this flag, it tells emsmdb32 and the Exchange store you're interested in getting the RSG version of the mailbox. Of course, life isn't ever this simple. You have to set some other flags too, most notably OPENSTORE_USE_ADMIN_PRIVILEGE, which tells the store you want to be treated as an administrator, assuming you have permissions. I have found the following cocktail to work the best:
OPENSTORE_USE_ADMIN_PRIVILEGE | OPENSTORE_TAKE_OWNERSHIP | OPENSTORE_RESTORE_DATABASE |
OPENSTORE_OVERRIDE_HOME_MDB | OPENSTORE_HOME_LOGON = 0x10001D
Now we see what the new feature I added to MFCMAPI allows you to do - instead of double-clicking on the mailbox in the table, we can right click and pick "Open with Flags", enter the flags we wish to use (0x10001D) and we're in like Flynn!
[Update: 2/18 - add note clarifying this is Exchange's MAPI only]
[Update: 6/20 - Added link to Jesper's excellent how to article that puts this information into practice]
The February 2008 Release (build 18.104.22.1684) is live: http://www.codeplex.com/MFCMAPI
Here's the change list - see the Issue Tracker on Codeplex for more details, or look at the code:
Hey - look what I found on one of my old machines. It's the documentation from the Exchange 5.5 SDK / EDK!
We're not supporting development against 5.5 anymore, and a rather large portion of what's in these docs only applies to Exchange 5.5, but there are some interesting tidbits still. Some highlights:
Note - these are old .CHM files. If you can't read them I can't really help you. I'm only providing them here as a historical curiosity.
BTW - for some reason, we've never pulled the 5.5 samples and libraries. You can still get those from here:
The January '08 refresh for the Outlook 2007 Auxiliary Reference is now live. In addition to a general scrub over most of the topics, there are some interesting new topics added, such as:
Timezone/rebasing docs: http://msdn2.microsoft.com/en-us/library/bb820976.aspx and http://msdn2.microsoft.com/en-us/library/cc160684.aspx
Protecting open PST files when you crash: http://msdn2.microsoft.com/en-us/library/cc160695.aspx
Rules processing in the wrapped PST: http://msdn2.microsoft.com/en-us/library/bb820947.aspx
Additionally, my update to the wrapped PST has been incorporated.
[This is now documented here: http://msdn.microsoft.com/en-us/library/ff960239.aspx ]
We just had a customer whose Wrapped PST based store wasn't getting rules to fire in Outlook 2007, when everything was working in 2003. In the course of figuring their issue out, we found a few flags that we should have documented before.
Suppose you built a message store based on the Wrapped PST provider. When your new messages are delivered to your backend database and you create matching messages in the PST, how do you inform Outlook that a new mail has arrived? In Outlook 2003, the way you solve this is to use the IMAPISupport object that was passed to you during logon. From there, you would call Notify, passing fnewNewMail and the details of the new message. When Outlook gets this new message notification, it passes the Entry ID over to the rules engine and everything is hunky dory.
Unfortunately, this doesn't work in Outlook 2007. You fire your notification, Outlook sees it, then Outlook decides not to run rules on the message. Why? Because when Outlook 2007 looked at PR_STORE_SUPPORT_MASK on the wrapped PST, it found the STORE_ITEMPROC flag was set. This means the wrapped PST is an "Item Proc" store, and handles rules differently. Without getting into too many details about "Item Proc", the general idea is that such stores will feed new items into a pipeline where rules, junk mail, and spam processing can happen before listening clients will get a notification. This concept was introduced in Outlook 2007 as a way to streamline the processing.
When an Item Proc store knows it has a new message, it calls a special callback function in Outlook to run it through the rules engine. In the PST, the concept of "newly delivered message" is handled through internal flags. These flags may be set or not set depending on how the message arrives in the PST. For instance, if the PST is being used as an OST and new mail is synchronized from an Exchange server, the flags will be set. Similarly, when the PST is being used as a back end POP3/IMAP, the flags will be set when a new mail is delivered.
For a wrapped PST, however, the way you set these flags is to pass ITEMPROC_FORCE in the SaveChanges call when the newly delivered message is committed to the store. This tells the PST to set its internal flags and mark the item as being eligible for the callback.
There's a twist though. If the mail being delivered to the PST came from Exchange, then rules would already have run on the server and the message has to be handled differently. And the default assumption is that the mail came from Exchange. So you have to indicate that Exchange was not the source of the mail by also passing the NON_EMS_XP_SAVE flag. When both flags are passed, the item will go through rules/junk/spam processing as soon as you call SaveChanges if Outlook is running, and shortly after Outlook starts if it wasn't.
Definitions of the new flags:
Whenever I find myself repeating the same message over and over again, I have to ask why I haven't blogged it yet. This is one of those cases. :)
I've seen quite a few issues over the years with MSG files. The issues range from "it takes too long to write properties" to "the properties on the MSG don't match what I see in the store" to "I get such and such error trying to copy this message to an MSG". The root cause of most of these issues is one of expectations. People are trying to use MSG files as an archival format, and that's not their intended purpose. If you really want to archive mail, you should develop your own format for persisting the data. You'll gain advantages in versatility, speed, and fidelity.
To understand why I make this recommendation, we first need to realize that not all messages can be copied over to the MSG format. This is noted at the end of http://support.microsoft.com/kb/171907. Since MAPI is transacted, the underlying MSG file has to be opened with STGM_TRANSACTED, meaning nothing is committed to disk until SaveChanges is called on the message. Couple that with the quirk in the MAPI specification that pretty much forces you to create a new transaction each time you add a recipient or attachment and you quickly run into the limit on open root storage files noted in http://support.microsoft.com/kb/163202. This OS imposed limit on open root storage objects isn't likely to ever change, as it's an artifact of the implementation. Likewise, the need for new transactions for each recipient and attachments also won't ever change. Neither the MSG format nor structured storage have seen active development in years. This limit is going to be hit whenever a message has a large number of recipients or attachments, or when there exist a deep level of embedded messages.
The next issue is speed. Writing a message to MSG can be quite slow. There is a huge performance penalty working with a structured storage file in STGM_TRANSACTED mode. And this penalty is multiplied by the number of open root storage objects. So not only do you run into a limit trying to add all those recipients and attachments, but each subsequent recipient and attachment is that much slower to add. For instance, I recently worked on an issue where the repro required that I have 5000 recipients on a message that I then copied over to MSG format. It took over an hour to write the file. And none of that delay was actually in the MAPI code - it was all at the COM level.
Next - not every MSG file you can write can be opened by Outlook. Over the years folks have tried various tricks to squeeze performance out of the code writing their MSG files. In many cases, they succeeded in writing the file faster, or allowing more recipients and attachments on the message. But the downside was they wrote a file that Outlook didn't know how to open! One variation of this issue surfaced with Outlook 2007. Given the performance problems working with MSG files, in Outlook 2007 we decided to check the number of recipients and attachments when opening the file. If either was over 2048, then we refused to open the file at all. The main reasoning for this was a number of corrupt MSG files that had surfaced in the wild with astronomical counts of recipients and attachments - on the order of millions. But a side effect was to block Outlook 2007 from opening some MSG files that Outlook 2003 could open. We've had some customers complain about this one and a fix is in the works. I'll report here when it's done. However, that fix will only cover this one variation of the problem. It won't fix the large number of other scenarios out there.
That covers the mechanics of reading and writing to MSG. Now we discuss fidelity. This isn't about whether the MSG format is out partying with the EML format, but rather how faithfully the MSG represents the source message. This is where MSG being a MAPI based format gets you in trouble. For instance, in archival scenarios, especially when the archive is used for legal discovery, properties such as PR_LAST_MODIFICATION_TIME and PR_LAST_MODIFIER_NAME are very important as they indicate who modified the message and when. But since MSG is itself a MAPI message and has such has to obey all the rules of MAPI, those properties will only reflect the time the MSG was written and the name of the account that wrote it, both of which aren't likely to match the original message. This problem can extend to the body properties as well: no matter how you do it, you're likely to end up converting the body from one format to another when storing it in the MSG file. And every conversion carries with it the possibility of a loss of data. Perhaps some line spacing is subtly changed, or font choices aren't preserved exactly. In some messages, these subtle textual differences could have huge semantic ramifications.
Fidelity also figures in when discussing Unicode. In a large organization, messages will be written in a variety of languages. The only way to preserve these messages into MSG format without converting half the characters to question marks or boxes is to use the Unicode format. Unfortunately, this format is only understood by Outlook 2003 and Outlook 2007. Exchange's MAPI doesn't understand this format at all. So if you're relying on MSG files to save out Unicode data, your solution is stuck using Outlook's implementation of MAPI for all processing of the archive. This severely hampers your ability to build a server based application.
So, we've got messages that cannot be copied to the archive, a painfully slow API, messages that cannot be opened once archived, and a format that's not capable of representing the actual message being archived. Clearly, these are not the attributes we want in an archive.
Fortunately, the workaround is simple: don't use MSG to archive messages. Instead, develop your own file format to preserve the important properties on a message. Here's one approach using the file system and XML files:
The only really hard part about this format is determining how to store each of the possible MAPI property types. However, when we look closely, we see there are only 13 types to consider, most of which can be represented as just a simple number or string. Even binary data is easy to store if it's first converted to hex. Multivalued properties, large binary and string properties, and named properties all add additional wrinkles, but are easily addressed. I figure a junior programmer could complete a reasonable first draft of the required code to both read and write a MAPI message to and from XML in an afternoon. In fact, most of the code for writing the XML format is already present in MFCMAPI - check out dumpstore.cpp.
Hopefully I've convinced most of you not to use the MSG file format for archiving. Some of you might not be convinced though. You might think you've got that one special case that requires you to use MSG. I don't believe such a case exists. I've anticipated a few of the common objections:
The final objection is my favorite: "But I've never had a problem with MSG files" - Bully for you! This article isn't addressed to you then. However, I had one customer who also made this claim when I found they were using MSG to archive messages. Not quite believing them though, I outlined each of the problems listed above. It turns out they had encountered or were encountering every single one of them. They just hadn't connected the problems back to their choice to use MSG to archive their data.
As promised, though a bit late, here are the change lists I put together for a couple of versions of MFCMAPI, along with a history lesson.
Versions 1-3 of MFCMAPI were essentially toys, built as I learned what could be done with MAPI. I can't even find them now - I wasn't very good at source control back then.
The oldest build of MFCMAPI I possess is 22.214.171.124, from 9/26/2001. At the time, it was for me just a collection of sample code that I would share with customers and other Dev Support engineers. Somewhere along the line, I found that Exchange and Outlook support folks wanted to use MFCMAPI as a tool. So I set up an internal site here at Microsoft where they could grab the latest stable build and use it. I didn't consider it ready for customers to use as a tool though, so I asked that I be kept in the loop any time it was sent out the door.
At some point (9/27/2002 to be exact) I bumped the major build number to 5.* to reflect the number of changes I had made recently. I also started work on the classic KB article: http://support.microsoft.com/kb/291794. It went live on 7/24/2003 with version 126.96.36.199. Minor build numbers were introduced along the way and I kept the article updated with builds and source. The last build I published in the KB was 188.8.131.528 and it was built on 2/14/2005.
In August of 2004, a program manager for Exchange contacted me about having MFCMAPI officially replace MDBVU for Exchange 12, eventually known as Exchange 2007. Of course, I jumped at the chance. The experience of checking my code into the Exchange build tree was incredibly instructive. We ended up changing over half the code to get it up to the Exchange team's standards. We took our time getting the official Exchange build ready, so I kept up the KB releases for a while. Finally on 6/7/2006, we released the newly christened "MAPI Editor". Since this version was built out of the Exchange 2003 build tree, the version number was 06.05.7830.
As I stated at the time, my eventual goal was to get the source for this new build published. I also wanted to continually update the binary as I added features and bug fixes. But as we all know, when you hand control over to someone else - and their priorities aren't in sync with yours - what you want and what you get aren't always the same.
So - this past year, with the blessing of the Exchange team, I took control back. They can still publish an official build of "MAPI Editor" if they want, but I've put all the source and the releases for MFCMAPI up on CodePlex. The first release went live on 8/30/2007, and I've done three updates since then. Since I took over building the project, I resumed my old version numbering scheme, bumping the major build number to 6.
I've rambled enough - here are those diffs:
KB (184.108.40.2068) -> MAPI Editor Download (06.05.7830):
MAPI Editor Download (06.05.7830) -> Initial Codeplex population (220.127.116.110):
Ok - not exactly. But there are a few things you need to configure on your SQL box if you want SQLMail to follow the rules of MAPI multithreading.
Before we get too deep, a couple notes:
Avoiding crashes and hangs with SQLMail
Let's discuss that last one a bit, since there's a lot of SQLMail KB articles out there that mention running xp_startmail as part of a workaround. The folks that wrote those articles just aren't aware of what SQL does when xp_startmail is run and the effect it has on MAPI.
The thing to understand is that SQL uses a lot of threads. There's a thread pool that sits around ready to service various jobs. When SQL is processing a script, as it gets to various commands it will grab a thread from the thread pool, have it execute the command, then return it to the pool. So in the course of processing a single script, SQL might use a number of different threads. Suppose you have a script that executes xp_startmail, xp_sendmail and xp_stopmail. Each of these procedures could get handed to a different thread!
Why is this bad? Because xp_startmail is essentially a call to MAPInitialize and xp_stopmail is a call to MAPIUninitialize. So by using this commands, we will end up initializing MAPI on one thread and uninitializing it on a totally different thread. And this would be bad - very bad - crossing the streams bad.
Fortunately, xp_sendmail will call MAPIInitialize/MAPIUninitialize on its own, so we don't need xp_startmail or xp_stopmail at all!
Every SQLMail crash and hang I've debugged was resolved by implementing the above recommendations.
The December 2007 Release (build 18.104.22.1683) is live: http://www.codeplex.com/MFCMAPI
Yesterday, a new Developer Support engineer started here - Rick Hallihan, of OneManShouting fame. We snagged him away from the Home Server MVP program. (I hope we didn't make any enemies in doing so!) We're gonna teach him up good in MAPI and he'll in turn learn us all he knows about Home Server.
Speaking of Home Server, I don't usually spread links, but I thought this was funny:http://www.stayathomeserver.com/default.aspx
It looks like a parody, but I've been assured it's a real site run by the Home Server team. Not many videos up there yet, but the "Home Sweet Server" vid is great.
When I announced the Outlook 2007 Auxiliary Reference, I noted there were a few issues with the wrapped PST sample that I hadn't had time to fix. Well, they're fixed now. You can find the updated download here: http://www.microsoft.com/downloads/details.aspx?FamilyID=ee0af8fd-bbb7-44de-be4d-f33cb1b59563&displaylang=en
It should say Nov07 for version. If it says Oct07, then the new version hasn't propagated to all the servers yet. It should update soon.
BTW - while I was working on the last MFCMAPI update, I found a number of minor problems in the Auxiliary reference. They're not worth detailing here, but they've all been corrected internally and should be fixed when we do our next refresh. Let me know if you spot any problems in the reference and I'll do what I can to get them addressed.
The November 2007 Release (build 22.214.171.1242) is live: http://www.codeplex.com/MFCMAPI
Here's the change list:
I also worked a bit on the front page of the Codeplex site as I had more than one person who went there to get the latest build and ended up with an old build from the MS Download site. It should be bit clearer where the latest build is now.
I haven't forgotten about that list of diffs from earlier builds I spoke about last time. I've just been busy with shipping the auxilliary reference and my transition from CPR back to Developer Support. Oh yeah - that reminds me - I don't work in CPR anymore. I'm still an Escalation Engineer, but a couple months ago I moved back to Developer Support to be closer to the dev type issues.
Excellent news! We just published the Microsoft Office Outlook 2007 Auxiliary Reference:
(This is a greatly enhanced version of the earlier Microsoft Office Outlook 2003 Integration API)
This reference was a huge collaborative effort, of which I was only a small part. Parts of this reference are based on my blog entries, but large sections of it were written by others. For instance, we have a new sample address book provider, written by Xiaoming Yin. And this reference wouldn't have happened at all without the tireless efforts of Angela Chu-Hatoun and Allison Bokone, who had the jobs of taking all this information from disparate sources and hammering on it until we had something we could publish. And then there were the PMs and developers who contributed, too many to name here. It was a race to the finish. The last topic added was actually written yesterday morning!
I wanted to point out a couple things that we know we didn't have time to finish:
http://peach.ease.lsoft.com/scripts/wa.exe?A2=ind0708&L=MAPI-L&P=R14899 http://peach.ease.lsoft.com/scripts/wa.exe?A2=ind0709&L=MAPI-L&D=0&P=13248 http://peach.ease.lsoft.com/scripts/wa.exe?A2=ind0709&L=MAPI-L&P=R1623 http://peach.ease.lsoft.com/scripts/wa.exe?A2=ind0709&L=MAPI-L&D=0&P=15085
I'll post when these issues are corrected.
As we did with the Integration API before, we'll be using my blog to post any errata. You can contact me directly or through comments if you discover any problems with the reference. Enjoy!
[This is now documented here: http://msdn2.microsoft.com/en-us/library/bb821131.aspx]
This is a topic from an upcoming refresh to the Outlook Integration API documentation. Since some folks had expressed interest in getting this topic now, development allowed me to blog it. Enjoy!
About Conflict Resolution for Custom Item Types
This topic describes how you can specify how conflicts are resolved for custom item types that you create in Microsoft Office Outlook.
Conflict Resolution for Standard Outlook Item Types
In Outlook, conflicts occur when two or more copies of the same item have been modified independently of each other. Outlook detects conflicts during synchronization. For example, you update a meeting request online on Outlook Web Access and then you update the same meeting request on Outlook when you work offline. When Outlook goes online again and synchronizes the data between the client computer and the server, it detects that there are two different copies of the same meeting request.
Versions of Outlook before Outlook 2003
In versions of Outlook before Outlook 2003, when Outlook synchronizes data, it does not consider properties specific to an item type. When Outlook detects conflicts, Outlook presents the conflicting copies of the item and the user decides which copy to keep.
Outlook 2003 and Outlook 2007
In Outlook 2003 and Outlook 2007, when Outlook synchronizes items that belong to a standard Outlook item type, it takes into consideration the properties that are specific to that item type to detect possible conflicts. Outlook tries to resolve conflicts and stores the resultant copy in the appropriate folder without requesting user intervention. In cases where Outlook considers that there is a possibility that the resultant copy may not contain all essential data, Outlook stores the conflicting copies in the Conflicts folder, under the Sync Issues folder. (Note that Sync Issues and its subfolders are hidden until you click Folder List on the Go menu.) In such cases, users can choose to go to the Conflicts folder to verify which items were in conflict and whether to use a copy in the Conflicts folder to replace the copy that Outlook decided to retain.
Conflict Resolution for Custom Item Types
Item Types and Message Classes
All items in Outlook are associated with a message class. For example, by default, a mail item is associated with the message class IPM.Note. The message class is primarily used to identify the form that should be used to display the item in Outlook. Outlook supports a list of message classes that are mapped to the types of items built in to Outlook. For more information about message classes, see Item Types and Message Classes.
Users can create custom item types, assign custom message classes to the custom item types, and have Outlook use a custom form to display the custom item types. For example, you may want Outlook to display a custom business contact form for your business contacts. To do that, you can create a custom message class IPM.Contact.Business, create a custom form for this message class, and assign business contacts with this message class.
Registering Conflict Resolution Scheme for Custom Item Types
When you create a custom item type, other than the custom message class and custom form, you should also consider how you would like Outlook to handle conflicts between copies of an item of this item type. By default, because custom item types may define custom fields in the custom form, and may have custom properties and custom code, Outlook employs a resolution scheme common to all items, does not consider properties that are specific to an item type, and presents conflicting copies for the user to make a decision. If you want Outlook to consider item-specific properties and attempt to resolve the conflict with minimal user intervention, you must specify that through a setting in the Windows registry. This can be achieved by either applying a Group Policy setting to the a local computer that sets the following registry key:
Or by directly modifying the following user registry key:
Setting the conflict resolution through Group Policy takes precedence over directly modifying the user registry key. The location of the key in the registry is dependent on the version of Outlook. You specify the name of the custom message class as a value under this key. Specify the type of the value as DWORD, and the data of the value as one of the following possible values, depending on the resolution scheme you choose:
If you specify one of the item-specific resolution schemes (data2 through 8), Outlook will try to resolve conflicts in item-specific fields (for example, Start and End fields of an appointment item) automatically without user intervention. If Outlook considers that the resolution may result in loss of essential data, Outlook will retain conflicting copies in the Conflicts folder and users can choose to go to the Conflicts folder to manually re-resolve these items and override the automatic resolution.
Using the same business contacts example above, if you want to specify the contact item-specific resolution scheme for the custom message class IPM.Contact.Business, you can add it as a DWORD value under [HKCU]\Software\Microsoft\Office\12.0\Outlook\Options\ConflictMsgCls, and specify 5 as the data.
In Outlook 2007 Service Pack 1, Outlook always uses a resolution scheme specific to appointment items for custom message classes that are based on the appointment message class (that is, any custom message class that is preceded by IPM.Appointment, for example, IPM.Appointment.Personal).
This is a question for those of you out there who still have Exchange Client Extensions.
Aside from budget concerns and legacy support, what blocks you from re-implementing your extension as an Outlook Add-In? Specifically, I'm looking for events and capabilites that would have to be added to the Outlook Object Model to allow you to migrate your code.
For example, ECEs have an OnRead event that happens before Outlook reads from a message. This allows the extension to modify the message first. The OOM's Read event happens after Outlook reads from the message, so it's too late. You would need some sort of BeforeRead event to match the ECE.
Can you come up with more? I've got a list I put together here, but I want to see what you folks have actually run up against, so I'm most interested in real world examples drawn from your own code. Thanks!
Welcome viewers from Patrick and Randy's blog! I've seen a few comments that indicate support for legacy clients is a major concern. We definitely understand that and it's on our list. But what we're looking for here is things that you're doing with ECEs right now that you cannot do with Add-Ins. To put it another way - if ECEs suddenly vanished and you had to implement your code in an Add-In, what missing features or capabilities would be showstoppers?