[This is now documented here: http://msdn.microsoft.com/en-us/library/bb820923.aspx]
[This information was also published as http://support.microsoft.com/kb/912239]
This one jumped ahead in the line because Oliver Seaman was asking me about detecting which items are headers. Next post should be about dealing with IMAP header messages.
Topic A property to identify items in header only state
dispidHeaderItem Both Cached Exchange mode and IMAP support the concept of header items. Messages in the Cached Exchange OST and in the IMAP PST can be in one of two states: a message in its entirety with the header and body, or a message with only its header downloaded.
This property can be used to determine the current state of a message.
Note: this property does not apply to remote transport headers, which can be distinguished by the message class “IPM.Remote”.
Definitions
#define dispidHeaderItem 0x8578 DEFINE_OLEGUID(PSETID_Common, MAKELONG(0x2000+(8),0x0006),0,0);
Usage This is a named property of type PT_LONG which will be present and non-zero on a message if the message is a header.
BOOL bIsHeader(LPMESSAGE lpMessage) { HRESULT hRes = S_OK; BOOL bRet = false; ULONG ulVal = 0; LPSPropValue lpPropVal = NULL; LPSPropTagArray lpNamedPropTag = NULL; MAPINAMEID NamedID = {0}; LPMAPINAMEID lpNamedID = NULL; NamedID.lpguid = (LPGUID) &PSETID_Common; NamedID.ulKind = MNID_ID; NamedID.Kind.lID = dispidHeaderItem; lpNamedID = &NamedID; hRes = lpMessage->GetIDsFromNames(1, &lpNamedID, NULL, &lpNamedPropTag); if (lpNamedPropTag && 1 == lpNamedPropTag->cValues) { lpNamedPropTag->aulPropTag[0] = CHANGE_PROP_TYPE(lpNamedPropTag->aulPropTag[0], PT_LONG); //Get the value of the property. hRes = lpMessage->GetProps(lpNamedPropTag, 0, &ulVal, &lpPropVal); if (lpPropVal && 1 == ulVal && PT_LONG == PROP_TYPE(lpPropVal->ulPropTag) && lpPropVal->Value.ul) { bRet = true; } } MAPIFreeBuffer(lpPropVal); MAPIFreeBuffer(lpNamedPropTag); return bRet; }
It has come to my attention that the site I was using to store my code samples, http://stephen_griffin.members.winisp.net poses a problem for some users due to the underscore. Apparently some proxy server have problems handling sites with underscores. Sounds like a bug in the proxy server to me, but still I want people to be able to get to the samples.
So I've moved them all over to http://stephengriffin.members.winisp.net. Note that I've just dropped the underscore from the name. I should have all my articles updated with the new url before the day is over.
There have been questions lately on the MAPI mailing list about how to get the connection state API working. Particularly the HrOpenOfflineObj and Advise calls. Since I had a partial sample already sitting here I figured I whip it into shape and publish it.
I'll refer most questions of implementation to the sample code itself. One thing does need to be called out though: This API only works in process. The only intended clients of this API are MAPI providers, COM Add-Ins, and Exchange Client Extensions. If your code is not running inside Outlook.exe, expect HrOpenOfflineObj to return MAPI_E_NOT_FOUND. This point is not made clear in the published documentation.
I've implemented this sample as a Com Add-In. When loaded, an "Offline State" menu will appear. Through this menu, you can enable/disable state monitoring, check the current state, and change the current state. A lot of the output for the sample is through OutputDebugString, so run under the debugger or use a debug monitor to watch it. It also writes a log file to c:\offstate.txt.
The sample should compile in VC6 or VS 2003. I haven't tested it with VS 2005 yet, but it ought to work there as well.
Here's the download link: http://stephengriffin.members.winisp.net/OfflineStateAddIn/OfflineStateAddIn.zip
Thanks to JasonJoh for writing some of the UI that I didn't feel like writing, and thanks to Shawn Walker for providing the impetus for fixing up and publishing the sample in the first place.
[11/11/05] Fixed sample URL
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb820960.aspx]
Using the Appropriate Interface to Manage Messages in an OST in Cached Exchange Mode
INTRODUCTION
The following article provides information when using Microsoft Collaboration Data Objects (CDO)and how to manage messages in an Offline Store (OST) while the client is in Cached Exchange mode.
MORE INFORMATION
When in cached Exchange mode, messages in the OST can be in one of two states:
Use the IID_IMessageRaw GUID when requesting an IMessage interface for a message in an OST while in cached Exchange mode. Using the IID_IMessage GUID to request an IMessage interface on a message that has only its header downloaded will force a synchronization that will attempt to download the entire message.
Using IID_IMessageRaw and IID_IMessage to request an IMessage interface will return interfaces that are identical in use. The only difference is that the IMessage interface requested with IID_IMessageRaw will return an email message as it exists in the OST, without forcing synchronization.
If OpenEntry returns the error code, MAPI_E_INTERFACE_NOT_SUPPORTED, then the message store does not support accessing the message in raw mode. In this case attempt OpenEntry again passing IID_IMessage.
Note: While cached mode is similar to remote mail headers as used in classic offline mode, the default behavior is different. Remote mail header messages are not downloaded when accessed. This GUID will not work with remote mail headers.
Definition of IID_IMESSAGE_RAW
#if !defined(INITGUID) || defined(USES_IID_IMessageRaw) DEFINE_OLEGUID(IID_IMessageRaw, 0x0002038A, 0, 0); #endif
Usage When calling OpenEntry, pass IID_IMessageRaw instead of IID_IMessage. The following code sample illustrates the usage:
HRESULT HrOpenRawMessage ( LPMDB lpMSB, ULONG cbEntryID, LPENTRYID lpEntryID, ULONG ulFlags, LPMESSAGE* lpMessage ) { ULONG ulObjType = NULL; HRESULT hRes = lpMDB->OpenEntry( cbEntryID, lpEntryID, IID_IMessageRaw, ulFlags, &ulObjType, (LPUNKNOWN*) lpMessage)); return hRes; }
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb905282.aspx, http://msdn.microsoft.com/en-us/library/bb820925.aspx, http://msdn.microsoft.com/en-us/library/bb821181.aspx]
This information was also published as http://support.microsoft.com/kb/912237.
Recently the Outlook development team has agreed to document some bits and pieces about Outlook that we've never documented before. I've been charged with getting this documentation out to the development community. After publishing the information here, my next step will be to take your feedback and submit this as an article in our Knowledge Base. This information may also show up in a future release of the MSDN.
The first I'll be covering - The Contact Address Book
Topic Relating Contact Address Book entries to Contact messages, reading e-mail addresses from Contact messages, and locating Contact photos on Contact messages
Contab Outlook 2003 ships with a Contact Address Book provider, contab32.dll. This provider presents information from a user’s Contacts folder in the form of an address book so that the contacts may be used to address e-mail messages. All Contacts which have an e-mail address or fax number will be represented in the address book, with a separate listing for each e-mail address or fax number. Since a Contact may have 3 e-mail addresses and 3 fax numbers, this means each Contact may be represented by up to 6 separate entries in the address book.
The information present on a Contact Address Book entry is a subset of the information present on the underlying Contact message. The following structure can be used to determine which Contact message a particular Contact Address Book entry is derived from.
Definition
#pragma pack(4) typedef struct _contab_entryid { BYTE misc1[4]; MAPIUID misc2; ULONG misc3; ULONG misc4; ULONG misc5; // EntryID of contact in store. ULONG cbeid; BYTE abeid[1]; } CONTAB_ENTRYID, *LPCONTAB_ENTRYID; #pragma pack()
Usage The CONTAB_ENTRYID structure defines the format of the entry IDs of IMailUser objects used in the Outlook Address Book.
Entry IDs of this type could be obtained from the PR_ENTRYID column of the Outlook Address Book's contents table or from the PR_ENTRYID property on an IMailUser object opened from the Outlook Address Book.
To open the underlying Contact message, cast the Contact Address Book entry ID to this structure and then use the cbeid and abeid members as the Contact’s entry ID. For example:
HRESULT HrOpenContact( LPMAPISESSION lpSession, ULONG cbEntryID, LPENTRYID lpEntryID, ULONG ulFlags, LPMESSAGE* lpContactMessage) { ULONG ulObjType = NULL; if (sizeof(CONTAB_ENTRYID) > cbEntryID) return MAPI_E_INVALID_PARAMETER; LPCONTAB_ENTRYID lpContabEID = (LPCONTAB_ENTRYID) lpEntryID; HRESULT hRes = lpSession->OpenEntry( lpContabEID->cbeid, (LPENTRYID) lpContabEID->abeid, NULL, ulFlags, &ulObjType, (LPUNKNOWN*) lpContactMessage); return hRes; }
Contact E-Mail Addresses The only way to access the e-mail addresses on a Contact message is through named properties on the message. These properties documented here are only supported as read-only properties. We cannot support solutions which write to these properties.
DEFINE_OLEGUID(PSETID_Address, MAKELONG(0x2000+(0x04),0x0006),0,0); #define dispidEmailAddrType 0x8082 #define dispidEmailEmailAddress 0x8083 #define dispidEmail2AddrType 0x8092 #define dispidEmail2EmailAddress 0x8093 #define dispidEmail3AddrType 0x80A2 #define dispidEmail3EmailAddress 0x80A3
Usage Use these to fill out the MAPINAMEID structure. PSETID_Adress is the lpGuid, MNID_ID is the ulKind, and the various dispids are the lID's. Then use GetIDsFromNames to get the current property IDs for these props. All of these properties are string properties. For example:
HRESULT HrGetEmail1(LPMESSAGE lpContact) { HRESULT hRes = S_OK; LPSPropTagArray lpNamedPropTags = NULL; MAPINAMEID NamedID = {0}; LPMAPINAMEID lpNamedID = &NamedID; NamedID.lpguid = (LPGUID)&PSETID_Address; NamedID.ulKind = MNID_ID; NamedID.Kind.lID = dispidEmailEmailAddress; hRes = lpContact->GetIDsFromNames( 1, &lpNamedID, NULL, &lpNamedPropTags); if (SUCCEEDED(hRes) && lpNamedPropTags) { SPropTagArray sPropTagArray; sPropTagArray.cValues = 1; sPropTagArray.aulPropTag[0] = CHANGE_PROP_TYPE(lpNamedPropTags->aulPropTag[0],PT_STRING8); LPSPropValue lpProps = NULL; ULONG cProps = 0; hRes = lpContact->GetProps( &sPropTagArray, NULL, &cProps, &lpProps); if (SUCCEEDED(hRes) && 1 == cProps && lpProps && PT_STRING8 == PROP_TYPE(lpProps[0].ulPropTag) && lpProps[0].Value.lpszA) { printf("Email address 1 = \"%s\"\n",lpProps[0].Value.lpszA); } MAPIFreeBuffer(lpProps); MAPIFreeBuffer(lpNamedPropTags); } return hRes; }
Contact Pictures Outlook supports a single picture for each contact. This picture is stored as an attachment on the contact’s underlying message. To determine which attachment is the picture, use the PR_ATTACHMENT_CONTACTPHOTO property. The property documented here is only supported as a read-only property. We cannot support solutions which write to this property.
#define PR_ATTACHMENT_CONTACTPHOTO PROP_TAG( PT_BOOLEAN, 0x7FFF)
Usage The PR_ATTACHMENT_CONTACTPHOTO property can be accessed as a column on the attachment table obtained through IMessage::GetAttachmentTable, or as a property on the IAttach object. If this value is true, then the attachment is the contact picture. The picture will be store as a JPEG formatted file. Use the normal MAPI methods to extract this attachment to a file.
[Update: 1/23/06 - added KB link]
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb821132.aspx]
[This sample is now part of the Outlook MAPI Code Samples!]
In the Outlook 2003 Integration API, we documented the Replication API. This is a nifty API for implementing replication between a wrapped PST and your own custom back end, which should be a great way to simplify implementation of a MAPI Message Store Provider without incurring all the pain involved in writing one from scratch. Unfortunately, we neglected to document how to create a wrapped PST. This documentation (which should eventually be part of the MSDN) corrects this oversight. Much thanks to Fergus Wilson of Meridio for his assistance in testing this sample.
Please post back any problems you encounter with this sample or the docs so I can correct them. Thanks!
The sample code can be downloaded here: http://stephengriffin.members.winisp.net/wrappst/wrappst.zip
Wrapping the PST provider in preparation for using the Replication API The Replication API can only be used in conjunction with a Wrapped PST provider. This document defines a Wrapped PST provider and gives instructions for constructing one.
Wrapped PST Provider: A wrapped PST provider is a custom MAPI message store provider which uses the PST provider as the back end for storing data. Most functions in a basic wrapped PST provider will proxy their arguments directly to the underlying PST provider. This allows the developer doing the wrapping to focus on the portions of the provider specific to their business needs without needing to invest time in writing a store provider from scratch.
Some functions and classes do need special logic in them to inform the PST provider that it has been wrapped so the Replication API can function. This document outlines what needs to be done in these functions.
The remaining code in a wrapped PST can be used to change default behavior, such as supporting additional properties not native to a PST by overriding GetProps, or allowing OpenEntry to do special processing on a message before handing it back to the client.
Constants: The following constants are used in this documentation. Note that some changes and clarifications have been made to the constant definitions given in the Replication API.
#define MDB_OST_LOGON_UNICODE ((ULONG) 0x00000800) #define MDB_OST_LOGON_ANSI ((ULONG) 0x00001000) // OffLineFileInfo typedef struct { ULONG ulVersion; // Structure version MAPIUID muidReserved; ULONG ulReserved; DWORD dwAlloc; // Number of primary source keys DWORD dwReserved; LTID ltidReserved1; LTID ltidReserved2; } OLFI; #define PR_OST_OLFI PROP_TAG(PT_BINARY, 0x7C07) #define OLFI_VERSION 0x01 // UID of an NST provider const MAPIUID g_muidProvPrvNST = { 0xE9, 0x2F, 0xEB, 0x75, 0x96, 0x50, 0x44, 0x86, 0x83, 0xB8, 0x7D, 0xE5, 0x22, 0xAA, 0x49, 0x48 }; // This is a clarification of the definitions of PXIHC and PXICC // mentioned in the Replication API DECLARE_MAPI_INTERFACE_PTR(IExchangeImportHierarchyChanges,PXIHC); DECLARE_MAPI_INTERFACE_PTR(IExchangeImportContentsChanges,PXICC); // It is essential that the FEID, MEID and SKEY structures // be wrapped in this pack pragma or they will be the wrong size #pragma pack(1) struct FEID { BYTE abFlags[4]; MAPIUID muid; WORD placeholder; LTID ltid; }; struct MEID { BYTE abFlags[4]; MAPIUID muid; WORD placeholder; LTID ltidFld; LTID ltidMsg; }; struct SKEY { GUID guid; BYTE globcnt[6]; }; #pragma pack()
MSProviderInit This is the entry point when your provider is being loaded by Outlook. Any message store provider needs to implement this function and expose it as an entry point in the store provider’s DLL. This is a well known function name which does not change.
Implementation:
ServiceEntry This is the entry point when your provider is being configured, such as through the Mail Control Panel. The prototype for this function is MSGSERVICEENTRY, but the name to be used is not mandated. Instead, it is advertised in the PR_SERVICE_ENTRY_NAME property of your profile. In this documentation, we use the name ServiceEntry. Be sure to expose the function as an entry point in the store provider’s DLL.
CMSProvider This is the wrapped implementation of IMSProvider. It is required to wrap this interface to provide some special processing needed to make the wrapped PST work.
Member functions not documented here can just pass their parameters into the underlying wrapped object. The IUnknown functions are implemented normally, making sure that the reference count on the underlying object is maintained.
The constructor should accept the LPMSPROVIDER object being wrapped.
The following functions need special handling: IMSProvider::Logon
IMSProvider::SpoolerLogon
CSupport This is the wrapped implementation of IMAPISupport. It is required to wrap this interface to provide some special processing needed to make the wrapped PST work.
The constructor should accept the LPMAPISUP object being wrapped and a LPPROFSECT object which will be used in OpenProfileSection.
The following functions needs special handling: IMAPISupport::OpenProfileSection
When pbNSTGlobalProfileSectionGuid is requested, return the profile section which has been cached in your implementation.
IMAPISupport::ModifyStatusRow
The PST provider will attempt to set a status row with PR_IDENTITY_DISPLAY, PR_IDENTITY_ENTRYID, and PR_IDENTITY_SEARCH_KEY. These values will be based on the user specified in PR_PROFILE_USER and assume the user has an EX address type. If this status row is allowed, then later calls to QueryIdentity can fail if no transport can handle this address type. This failure can in turn prevent creation of some items like contacts and appointments. Simply clearing these values will fix this problem. A more advanced fix would be to set a custom status row. Even more advanced would be to wrap the IMSLogon object returned in IMSProvider::Logon to implement a custom or wrapped status entry.
Using the Replication API The documentation of the Replication State Machine describes how to use the API to perform synchronization between an external store and the wrapped PST. See the functions DoSync, SyncDownloadFolders and SyncUploadFolders for sample code illustrating the API.
[10/6/05 - 10:39 AM] Clarified IMSProvider::Logon
[10/17/05 - 5:45PM] Added IMAPISupport::ModifyStatusRow, updated sample code
We just released the April 2005 update to the Outlook 2003 Integration API.
Download for the CHM is here:
http://www.microsoft.com/downloads/details.aspx?FamilyId=135F4D99-F480-4A81-AF8F-F6E4896611E2
The MSDN documentation should update some time in the next few days. When it does, you'll find it here:
http://msdn.microsoft.com/en-us/library/aa193231.aspx
Note that the date on the Welcome page will be updated from December 2004 to April 2005 when the updates go live.
Many thanks to Dmitry for some valuable feedback that went in to this update.
As always, let me know any questions/concerns you have with the documentation.
There is an article which you used to be able to find at http://support.microsoft.com/kb/259301. This article, which I helped to write, walked you through the mechanics of impersonating a user before using MAPI. I pulled this article recently and just wanted to discuss briefly why.
There are three main problems with doing impersonation and then using MAPI: registry access, support threads, and instance data.
The first problem is that MAPI wants to use HKEY_CURRENT_USER to access the registry for profiles. This works fine for any thread running as the same user that the process was run as. However, the predefined HKEY_CURRENT_USER handle, which MAPI uses, will always point to the registry hive of the process' user, not the thread's user. There is a trick (this article was pulled due to the problems described below) to substitute a different registry hive which works if you have full control of all code accessing the registry, but this is rarely true of most applications.
There's another trick to avoiding these registry problems: MAPI_TEMPORARY_PROFILES. This flag causes MAPI to use file based profiles instead of using the registry. However, support for this flag was removed in Outlook 2003, so this trick can only be used with Exchange's MAPI and older versions of Outlook.
Both Outlook's and Exchange's implementations of MAPI depend on several support threads which they will spin up in the course of normal operations. Outlook's MAPI uses support threads for it's implementation of Cancel RPC, and both use support threads to handle notification processing (even when the client has not specifically requested notifications). MAPI creates these new threads using CreateThread. Note that the documentation for CreateThread specifically states that this API should not be used from a thread running under impersonation.
If you're using Exchange's MAPI, then there is a way to switch to a different notification engine which doesn't suffer from as many of these problems. It's documented in XCLN: OWA Clients Receive a "Failed to Connect to the Microsoft Exchange Server" Error Message (323872). Note that the article mentions reg keys such as inetinfo and dllhost. These reg keys would need to be the name of your application in order to work.
MAPI stores “instance data” for each security context that calls MAPIInitialize. The information includes the heap handles, shared sections for interacting with other processes using the same security context, etc. Instance data lives in a structure that is keyed to a hash of the current security context’s SID. So, if you create a MAPI object under one security context and release it under another, MAPI either a) fails to find the instance data and crashes, or b) frees the object from the wrong heap and corrupts the heap. How this instance data is affected by impersonation was also involved in the Deleted Profile issue.
We've seen a number of crashes in code that uses impersonation and then uses MAPI. Many of these relate to heap allocations occurring with one user context and then the deallocations happening with another context. Both the Exchange and Outlook development teams are aware of these issues. However, especially due to the problem with CreateThread, we've not been able to fix all of them.
I had originally stated "Other problems we have seen, especially in code that manipulates profiles prior to logon is OpenMsgStore failing with MAPI_E_FAILONEPROVIDER (0x8004011D) and MAPI_E_NETWORK_ERROR (0x80040115)." After further investigation, it turns out the code that was seeing these problems was creating the profiles by editing MAPISVC.INF, and these file manipulations were not protected properly by a mutex. So one thread's edits of the file were overwritten by another thread's edits before the profile could be configured. Subsequent failures in MAPI were then due to the corrupted profiles.
If you're using impersonation in order to access multiple mailboxes, you may be doing too much work. You can use IExchangeManageStore::CreateStoreEntryID to log on to any mailbox for which you have the appropriate permissions. So for code running under an administrator account with the right permissions, you can use this API to access any mailbox without doing any impersonation.
If you must do impersonation, for example, to connect to different servers where it may not be possible to arrange a single account with the right permissions, then the recommendation is to use a stub program to impersonate, then call CreateProcessAsUser to launch your worker program which does the real MAPI work.
Getting MAPI to work with impersonation is very hard. If you have an application which uses MAPI with impersonation and you're not experiencing problems, congratulations. If you're planning on writing new MAPI code and you think you need to use impersonation, follow one of the above workarounds. I pulled the article (which did nothing more than walk through the mechanics of using LogonUser and leak at least two handles along the way) so as not to encourage new code using MAPI under impersonation.
If you absolutely must use impersonation with MAPI:
Thanks to Dana Birkby and others for reviewing this article.
[Edit - 4/22/05 - 9:30AM - Updated Observed Behavior section]
[Edit - 7/19/05 - 5:35PM - Noted that KB 199190 was pulled]
[This is now documented here: http://msdn2.microsoft.com/en-us/library/bb905128.aspx]
I've been working with Dmitry Streblechenko on getting a sample using the Connection State API. We've uncovered a number of doc bugs. All of them have been reported back to the content folks and should be corrected in the next refresh, but in the interest of unblocking folks who want to use this API, here they are:
On the Constants page:
MAPIOFFLINE_CALLBACK_TYPE_NOTIFY MAPIOFFLINE_NOTIFY_TYPE_STATECHANGE MAPIOFFLINE_NOTIFY_TYPE_STATECHANGE_DONE
Under Data Types
typedef enum { MAPIOFFLINE_CALLBACK_TYPE_NOTIFY = 0 } MAPIOFFLINE_CALLBACK_TYPE;
typedef enum { MAPIOFFLINE_NOTIFY_TYPE_STATECHANGE = 2, MAPIOFFLINE_NOTIFY_TYPE_STATECHANGE_DONE = 3 } MAPIOFFLINE_NOTIFY_TYPE;
Type of notification. Note that only notification on change of connection state is supported; the only supported values are: MAPIOFFLINE_NOTIFY_TYPE_STATECHANGE MAPIOFFLINE_NOTIFY_TYPE_STATECHANGE_DONE
On the HrOpenOfflineObj page
typedef HRESULT (STDMETHODCALLTYPE HROPENOFFLINEOBJ)( ULONG ulFlags, LPCWSTR pwszProfileName, const GUID* pGUID, const GUID* pInstance, IMAPIOfflineMgr** ppOffline );
Under Interfaces
Placeholder member This member is a placeholder and is not supported. GetCapabilities Gets the conditions for which callbacks are supported by a connection state object. Placeholder member This member is a placeholder and is not supported. Placeholder member This member is a placeholder and is not supported.
A bitmask of the following capabilities flags
void STDMETHODCALLTYPE Notify( const MAPIOFFLINE_NOTIFY *pNotifyInfo );
Outlook has decided to publish documentation for a number of previously internal only APIs which ship with Outlook 2003 SP1. Here's a link to the docs.
Due to a tight deadline, we were only able to document a small subset of these APIs. We are considering plans in place to expand these docs over time, but nothing is fixed in stone.
Note that some of the APIs being documented may be present in other versions of Outlook, but we're only providing support for their use in Outlook 2003 SP1. While it is likely that we will decide to support these APIs in future versions of Outlook, we will NOT provide support for earlier versions (including any version of Outlook 2000 and Outlook XP and Outlook 2003 RTM). There are both technical and resource related reasons for this. Some highlights of the new docs
Hit me with any questions you have about these APIs.
<2:25pm - fixed link>
[Update: This was superseded by the Outlook 2007 Auxiliary Reference]
A while back, I posted an article about the fixes Jason and I had to make to the MSLMS sample message store to get it to load under Outlook 2003. Since then, I've received numerous requests that I post the changes. I wasn't ignoring the requests - I've just been busy adding features to MFCMAPI (more on that later).
Before I posted the code, I wanted to scrub it once more for obvious defects. We have some tools here which can point out potential bugs in your code, like dereferencing a pointer without first checking that it's not NULL. I was amazed at the volume of errors these scripts found in the sample. Even though most of these errors were trivial to fix, the number of them made for slow goings.
Anyway, here's the updated code. Fair warning: There's no warrenty whatsoever implied in this code. USE THIS CODE AT YOUR OWN RISK!
Todo list:
I'll post updates as I make major changes and bug fixes. I may create a workspace for this on gotdotnet if I see enough interest in it. Until I do, send any bug reports to me.
[edit - crediting JasonJoh for the work he did on this]
I wrote these rules out while debugging a crash in another MS product:
I won't name the app, but it violated all 4 rules.
Known consequences of violating these rules:
This post was inspired by a case I worked recently. In this case, the customer was using the 5.5 Event Script service to autoaccept meeting requests. They weren't having any problems with their script or scalability. Their problem was that the service would run fine for days at a time, and then suddenly stop handling all incoming meeting requests. The error they would get in the event logs was this not especially helpful event 11:
Event ID: 11Source: MSExchangeESDescription: A fatal error (0x80004005)occurred in an IExchangeEventSink while processing message [Subject = '<subject>']
Examining the script logs was no help - they didn't say anything about the error. This case had me stumped for a while. The latest fixes for events.exe were no good. Turning on internal tracing didn't take me too far either.
Time to pull out the big gun
When I finally got a debugger attached to the events.exe process, I found that OpenMsgStore was returning MAPI_E_CALL_FAILED (==0x80004005). Debugging into that, I found that during handling of OpenMsgStore, MAPI has to go to the registry to retrieve some properties from the profile. MAPI's call to RegQueryValueEx was returning ERROR_KEY_DELETED (==1018). The profile we were using in the Event Script service had been deleted out from under us! Why?
The simplistic answer is that right after MAPILogonEx, events.exe calls DeleteProfile. To understand that, we need to digress.
What does DeleteProfile do?
Well, for starters, it doesn't delete the profile. At least not always. Here's what the MSDN has to say on this function:
The IProfAdmin::DeleteProfile method deletes a profile. If the profile to delete is in use when DeleteProfile is called, DeleteProfile returns S_OK but does not delete the profile immediately. Instead, DeleteProfile marks the profile for deletion and deletes it after it is no longer being used, when all of its active sessions have ended.
Let's break this down:
Check if the profile is in use
MAPI maintains a shared memory object (this is CreateFileMapping, not .Shared sections) for interprocess communication and synchronization. In that shared memory object, we keep a linked list of all profiles currently in use. When we log onto a profile with MAPILogonEx, its ref count is bumped up. When the client logs out, the ref count is dropped back down. So checking if a profile is in use is equivalent to looking in the shared memory object to see if the profile has a ref count.
Mark a profile for deletion
MAPI uses a special reg key to mark a profile for deletion. The key is
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Deleted Profiles
To mark a profile for deletion, we write the profile name as a subkey. When MAPI is asked to present a list of existing profiles, such as in the MAPILogonEx dialog or in response to GetProfileTable, profiles listed under Deleted Profiles are left out.
Delete a profile after it is no longer being used
This is the hard one. How does MAPI determine the exact instance a profile is no longer in use? During logoff is not sufficient, since we also need to account for cases where the application using MAPI has crashed. So we have to be cleverer than that. We delete profiles from the registry when:
The first two cases handle most profile deletions. The third case handles the abnormal situations - an application has crashed or leaked a session pointer. The third case is also where bugs in the implementation cause cross process problems.
Here come the bugs, part 1
The first bug is completely our fault. Suppose two different processes run as the same user. Suppose also that one of the processes uses impersonation to run under the credentials of another user. Note that by default, this does not load the impersonated user's registry hive. Doing that requires a different call. (See KB 259301 for example code that does this). So, both processes will use the same registry hive when accessing HKEY_CURRENT_USER.
The first process to start creates a profile for MAPI. It then calls DeleteProfile after logging on so as not to clutter up the registry with temporary profiles. The second process (which has done the impersonation) now comes in and calls MAPIInitialize. We have a problem with our shared memory now - each process got their own block of shared memory. Our scheme now breaks down. The second process checks its shared memory to see what profiles are in use, finds that no profiles are being used, and proceeds to delete every profile listed under "Deleted Profiles". The first process is now what we call hosed.
When this bug was first reported, we weren't sure how to fix it without doing a total rewrite of this feature. That is, until one of our devs had the brilliant insight that each shared memory section was essentially tracking the profiles in use by a particular NT account. So all we needed to do was reflect that in our use of the registry by appending a CRC of the SID under which MAPI is running to the "Deleted Profiles" string. Now all processes which are running MAPI under the same NT account (therefore accessing the same shared memory) are looking at the same place in the registry, and processes running MAPI under different NT accounts look in different places in the registry.
Here come the bugs, part 2
And yet, even after putting in the fix, we still saw profiles being deleted out from under us. This one isn't completely MAPI's fault. Suppose two different processes are running on the same machine. Both run as the same user and neither uses impersonation. However, one runs in Terminal Services and the other runs in the console.
Same dance as the first bug. When the second process goes to create a shared memory object, it does not get the same shared memory as the first one. We don't get the same shared memory because sessions started under Terminal Service use a different namespace than sessions started from the console. In general this is a good thing, but it blocks MAPI from doing what it's trying to do.
The fix from the first bug isn't effective here because the SID for both processes is the same. Fortunately, the fix here is even simpler - it's even spelled out in the MSDN: prepend "Global\\" to the name of the shared memory. Now all processes running as the same user will use the same shared memory regardless of the Terminal Service session.
Where was I?
This started as an Event Script problem, and digressed into a discussion of DeleteProfile. The Event Script service is built around MAPI. It creates a profile on the fly and calls DeleteProfile immediately after logging on. The customer used Terminal Services to manage their server, and to top it off, Outlook was installed on the machine. So every few days, they'd encounter Bug 1 or Bug 2 and their script was toast. I had them remove Outlook, sent them the patch, and they haven't seen the problem since.
Hey Steve, where's this wonderful patch you speak of?
Thought you'd never ask. Here it is by product:
[8/22/04 8:09PM Minor edit - clarified some text]
[8/25/04 11:04AM Clarified impersonation and added links]
I'm working on a followup to my memory management article and a writeup on a MAPI deleted profile bug I ran across recently, but this takes precedence.
Exchange and Outlook on the same machine is bad. In my last post I waved my hands about some scenarios which could lead to a crash. I got a dump today which illustrates one. The issue originally manifested as heap corruption in MFCMAPI while opening a message store. We enabled pageheap on the process to see what was corrupting the heap. Here's the stack we got:
0:000> kL ChildEBP RetAddr 0012f2bc 77fb44fb NTDLL!RtlpDphIsNormalHeapBlock+0x86 ... 0012f4d0 35525bda NTDLL!RtlFreeHeap+0x85 0012f4e0 35525b6e MSMAPI32!LH_ExtHeapFree+0x19 0012f50c 62a5248b MSMAPI32!MAPIFreeBuffer+0x64 0012f778 62a530dd emsmdb32!HrSetupOffline+0x144 0012f834 62cd1755 emsmdb32!RMSP_Logon+0x2c4 0012f894 62cd1478 mapi32!HrIntDoOneClientLogon+0x8a 0012f8b0 62cd0f25 mapi32!HrIntClientStoreLogon+0x47 0012f920 62cd1422 mapi32!HrCommonOpenStore+0x3c3 0012f9a0 004310b6 mapi32!SESSOBJ_OpenMsgStore+0x56 0012f9d0 004321c7 MFCMapi!CallOpenMsgStore+0x66
I've trimmed the stack a little to get to the heart of the issue. We're corrupting the heap when we call RtlFreeHeap. Notice that mapi32 and msmapi32 both appear on this stack. Here's the versions of both (again, output is trimmed):
0:000> lmvm mapi32 Image path: C:\WINNT\system32\mapi32.dll Timestamp: Sat Jun 12 00:17:32 2004 (40CA83DC) File version: 6.0.6603.0 ProductName: Microsoft Exchange 0:000> lmvm MSMAPI32 Image path: C:\Program Files\Common Files\System\Mapi\1033\MSMAPI32.DLL Timestamp: Tue Dec 16 20:39:51 2003 (3FDFB3E7) File version: 10.0.6515.0 ProductName: MAPI32
So what's the problem? We have two competing versions of MAPI loaded into the same process!
Emsmdb32.dll made a call to GetProps to get some properties on a profile. The resulting LPSPropTagArray was allocated using Exchange's MAPI32.dll. Later, Emsmdb32.dll calls MAPIFreeBuffer to clean up this memory. Somehow, Outlook's MSMAPI32.dll ended up handling this call. Since Outlook's MSMAPI32.dll doesn't know anything about the heaps created by Exchange's MAPI32.dll, we end up corrupting the heap during this free. Without pageheap enabled, this corruption is silent, and doesn't rear it's ugly head until later on when we try allocating some memory against the corrupted heap.
After removing Outlook from the box, this problem went away.