So - there's a memory leak in Outlook's implementation of MAPI. That shouldn't be too much of a shock. Just about any sizable application is bound to have a leak or two. What's interesting about this leak is that it is intentional.
A little history: We first shipped Simple MAPI back in MS Mail, circa 1992. Later on, when Extended MAPI was being developed for Windows 95, we chose to reimplement Simple MAPI as a wrapper around Extended MAPI. This reduced bug counts, simplified testing, etc. Everyone was happy. (Or so I've been told - I was still in school at the time!)
As we neared RTM however, our testers started finding Simple MAPI applications which had worked fine with MS Mail but now crashed. Investigation showed that these applications were accessing memory which had been freed. Why?
Let's digress a bit and look at a typical Simple MAPI function, MAPIResolveName:
ULONG FAR PASCAL MAPIResolveName( LHANDLE lhSession, ULONG ulUIParam, LPTSTR lpszName, FLAGS flFlags, ULONG ulReserved, lpMapiRecipDesc FAR * lppRecip )
There are two parameters of interest here: lhSession and lppRecip. lhSession can be used to pass in an existing session that was created with MAPILogon. Or it can be left NULL to let Simple MAPI create and use a temporary session for the duration of the call. lppRecip is used to pass back a structure - the resolved name. This structure is to be freed with MAPIFreeBuffer.
How would we implement this function in Extended MAPI? It would look something like this (pseudo-code):
if (!lhSession) { lhSession = InitializeMAPIAndLogonToProfile(); bTempSession = true; } DoWorkToResolveNames(lhSession,lppRecip); if (bTempSession) LogoffProfileAndUninitializeMAPI(); return;
So far so good. And as long as all the test harnesses for Simple MAPI pass a session in whenever they call MAPIResolveName, there's no problem. But what happens when we call MAPIResolveName without a session?
It might help here to comment that MAPI handles it's own memory allocations. That's why we have a special function, MAPIFreeBuffer, for freeing memory. When MAPI is initialized, it creates a heap to handle all of it's allocations. Similarly, when MAPI is uninitialized, it destroys this heap.
I'm sure the A students are bored with this lecture so I'll wrap it up now: In the original implementation, as soon as we clean up the temporary session and uninitialize MAPI, the heap from which lppRecip was allocated is destroyed. So any attempt to use it will result in a crash.
To fix this, we were faced with two choices. Either completely rethink how memory is handled in Extended MAPI, or leak the heap whenever we uninitialize MAPI. We chose the leak. The first version of Extended MAPI we shipped back in Windows 95 contained this leak.
Jump to today: That original version of Extended MAPI/Simple MAPI eventually forked to become the basis of both Outlook and Exchange's implementations of MAPI. While both implementations have had their changes over the years, the intentional leak has remained. And as long as Simple MAPI is implemented on top of Extended MAPI, we can't take the leak out. It will even be there in Outlook 2007.
Mitigation
If you're using Outlook's MAPI, or any version of Exchange's MAPI prior to Exchange 2003, the best you can do is minimize the number of times you uninitialize MAPI. If you're using Simple MAPI, this means do an explicit MAPILogon/MAPILogoff, and don't do them in a loop. If you're using Extended MAPI, initialize MAPI on your main thread before you do any MAPI work, and wait to uninitialize it until just before your process exits. Subsequent MAPIInitialize/MAPIUninitialize calls on other threads will act like Addref/Release on the MAPI subsystem so they won't incur the leak.
If you know you'll be using MAPI from Exchange 2003 or from the MAPI download, then you can benefit from the fact that we cut Simple MAPI there. This means we were finally able to put back the code to clean up the heap. It took us a couple times to get it right, but the upshot is that once the hotfix from KB 901014 is applied, you can set the CleanUpMAPIHeap key and no longer worry about this leak.
Incidentally, the reason we wrapped this fix in a registry key is because before we took this fix, any program which got memory from MAPI, uninitialized MAPI, and then used the memory would not appear to have a problem. But like Simple MAPI, as soon as we start cleaning up the heap, this pattern will cause crashes. So do test carefully before deploying this key!
Update – this is now part of the Outlook MAPI Code Samples.
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb905271.aspx, http://msdn.microsoft.com/en-us/library/bb821132.aspx]
Sam Khavari of Zimbra asked me if I had noticed that the Wrapped PST sample crashes when you turn on the preview pane. After a bit of wrangling over the repro, we found that it will crash if you preview one of the first handful of messages created in the store. Since I had been focused on the replication API, and my sample implementation of the replication API created many items in many different folders, every time I had tested it, those first few, dangerous items were in folders such as Drafts, which don't have a preview pane. So no, I hadn't noticed it, but since they pointed it out I've got to fix it. :)
A bit of debugging showed that Outlook had somehow gotten a hold of a folder object instead of a message and was trying to preview it, leading to much hilarity. Further debugging showed that the reason it had the folder object was because the CompareEntryIDs function was insisting that the entry ID for the folder and the entry ID for the message were trying to open referred to the same object. At this point I threw up my hands and went to the guys that wrote the PST to begin with.
They pointed out that my handling of the PR_OST_OLFI property in SetOLFIInOST was botched. They agreed to let me document more about the OLFI structure so we can handle this property correctly. Here goes:
Original SetOLFIInOST:
Let's review what we used to do in this function. We built an OLFI structure with dwAlloc set to 0x7FFFFFFF and everything else nulled out, then we write this to PR_OST_OLFI. We do this every time SetOLFInOST is called, regardless of what may have already been in the property. To understand why this almost worked and why it's such a bad idea, we have to discuss what the OLFI structure is for.
What is OLFI?:
The Offline File Info (OLFI) structure is used by the PST provider for allocating entry IDs in offline mode. Interestingly, it is not used by the PST when it is not being wrapped, so only a PST wrapper needs to worry about this. Here's the revised OLFI structure:
// OffLineFileInfo typedef struct { ULONG ulVersion; // Structure version MAPIUID muidReserved; ULONG ulReserved; DWORD dwAlloc; // Number of primary source keys DWORD dwNextAlloc; LTID ltidAlloc; LTID ltidNextAlloc; } OLFI, *POLFI;
Every time the PST needs an entry ID for a new object (message or folder) it needs two pieces of information from the wrapper, a GUID and an index. These are combined to make the ID. The OLFI structure is the mechanism for tracking GUIDs and indexes which have been handed out. The PST reserves entry IDs in blocks. To reserve a block of entry IDs, it consults the ltidAlloc structure to get the current GUID and index. It decrements dwAlloc by the number of entries being allocated and writes the next index back into ltidAlloc. It will continue to do this as long as dwAlloc is larger than the block size it's trying to reserve.
When dwAlloc is smaller than the block size, the PST copies ltidNextAlloc and dwNextAlloc over to ltidAlloc and dwAlloc respectively. It then nulls out ltidNextAlloc and dwNextAlloc. The wrapper is expected to periodically check ltidNextAlloc to see if it's NULL and if it is populate it with a new guid and reset dwNextAlloc.
If dwAlloc is still too small, we fail. Otherwise, we've updated dwAlloc and ltidAlloc, so we write the structure back to PR_OST_OLFI so we can check it next time.
What Almost Worked:
By initializing dwAlloc to 0x7fffffff, we guaranteed that many reservations of EntryIDs should work. We didn't set a guid, but that's not what killed us. Remember that we used to rewrite the property whenever SetOLFInOST is called? Turns out we're calling it from Logon, which can be called multiple times! So when the PST makes its second batch of reservations, it gets the same block as the first batch. The net effect is that some messages end up getting entry IDs that compare as equivalent to some of the default folders. As long as we never try to CompareEntryIDs two of these conflicting entry IDs we won't see a problem. But as soon as we do, we're toast.
Better SetOLFIInOST:
First, we need to read the current PR_OLFI_OST and see what's in it before we go writing anything. Blindly restamping dwAlloc is a recipe for failure. Second, we need to write guids into ltidAlloc and ltidAllocNext so our PST has a chance of working after dwAlloc is eventually exhausted. Finally, we need to periodically call SetOLFIInOST to ensure that we restamp ltidNextAlloc before it's too late.
New code:
I've shared the updated code out here:
http://stephengriffin.members.winisp.net/wrappst/wrappst.zip
Update: 10/05/06 - 10:44AM Sam let me know it's OK to give his name
Update – Get the latest MFCMAPI build and source (which should build with VS6) at http://codeplex.com/mfcmapi
I'm still working on getting the MAPI Editor source public. In the meantime, one of my colleagues informed me that the MFCMAPI source (http://support.microsoft.com/kb/291794/) doesn't build with Visual Studio 6 anymore.
This was quite surprising since I build it all the time with VS 6. So we compared our machines to figure out why.
First issue - if you just install VC 6 and try to build the source the first thing you see is this:
c:\mfcmapi\res\MFCMapi.rc2(53) : error RC2188: #error : "MFCMAPI requires that the February 2003 or later Microsoft SDK be installed for compilation" c:\mfcmapi\res\MFCMapi.rc2(54) : error RC2188: #error : "Go to the following URL" c:\mfcmapi\res\MFCMapi.rc2(55) : error RC2188: #error : " http://www.microsoft.com/msdownload/platformsdk/sdkupdate/" c:\mfcmapi\res\MFCMapi.rc2(56) : error RC2188: #error : "and follow the instructions to install the core SDK"
Ok - that's pretty straightforward. So you install that, but it still doesn't work. Oh yeah - I left something out of my instructions:
Fix 1
In Visual C++, under Tools/Options/Directories, you need to add the Platform SDK to the Include and Library paths. The new entries will need to come before the existing Visual Studio Entries. So, supposing the SDK was installed to c:\Program Files\Microsoft Platform SDK, to Include you would add:
c:\Program Files\Microsoft Platform SDK\Include c:\Program Files\Microsoft Platform SDK\Include\MFC c:\Program Files\Microsoft Platform SDK\Include\ATL
and to Library you would add:
c:\Program Files\Microsoft Platform SDK\Lib
So you do that and it mostly compiles. Well, except for this:
C:\mfcmapi\File.cpp(594) : error C2039: 'pwcsTemplateFile' : is not a member of 'tagSTGOPTIONS' C:\Program Files\Microsoft Platform SDK\Include\objbase.h(913) : see declaration of 'tagSTGOPTIONS'
Ok - this worked with the February 2003 SDK, but the SDK folks must have changed a header. No problem - we see that the line was redundant - we can get rid of it:
Fix 2
Cut this line from file.cpp:
myOpts.pwcsTemplateFile = 0;
Now we compile, but we still don't link! We get this:
Linking... uuid.lib(cguid_i.obj) : fatal error LNK1103: debugging information corrupt; recompile module Error executing link.exe.
If you do some searches on this error, you'll find a whole firestorm of debate on whether or not the SDK folks did the right thing with uuid.lib and discontinued support for VC 6. I won't give my opinion, but for the purposes of getting the project to build, here's one workaround:
Fix 3
Locate the platform SDK's copy of uuid.lib here:
c:\program files\microsoft platform SDK\Lib
c:\program files\Microsoft Visual Studio\VC98\Lib
and copy it into the above directory. Now everything should link.
If/when I get the MAPI Editor source published it will include a Visual Studio 2005 project, so hopefully this won't be a problem in the future.
Folks have been asking for some time now for a standalone installation package for MAPI and CDO. Well now we have one available: http://www.microsoft.com/downloads/details.aspx?FamilyID=e17e7f31-079a-43a9-bff2-0a110307611e&DisplayLang=en
Some details about this package:
Update – Get MFCMAPI source here: http://codeplex.com/mfcmapi
I've gotten some great feedback on the release of MAPI Editor yesterday. (Aside - I still can't get used to calling it that - it'll always be MFCMAPI to me.) I want to thank everyone who wrote me with congratulations.
There's one thing that I've been asked more than anything else. Will the source be made available?
Part of making this an official utility involved giving up direct ownership of the code to the product group. So while I'm still "the guy" when it comes to working on the code, they're the ones who will decide if we can release the source. I chose to simplify the process of getting this release by not asking that we include the source.
However, I do think the source *should* be available as a resource to the developer community. I think I will be able to make a great case for that here with the folks who have to give their blessing. My case will be even stronger if I can show them that there's a demand out there. So please keep the comments coming. Thanks!
BTW - I've made the offer elsewhere - until the source does become available, if you have any questions about how I implemented something, just ask and I'll try to help out.
Update – Get MFCMAPI here instead: http://codeplex.com/mfcmapi
I'm pleased to announce that Microsoft Exchange Server MAPI Editor is finally available: http://www.microsoft.com/downloads/details.aspx?FamilyID=55fdffd7-1878-4637-9808-1e21abb3ae37&DisplayLang=en
This is the "next version of MFCMAPI" that I've been working on for some time. Note that we changed the name to something long and literal in keeping with the rest of the Exchange tools. The binary is still MFCMAPI.exe, so I won't be confused if you continue to call it that. Official documentation will use the full name, or MAPI Editor for short.
So what's new in this version? For starters, the help banner at startup has been tweaked to be a bit more helpful. But if you don't want to see it anymore, you can uncheck the "Display at startup" box to dismiss it for good. Bring it back with F1 or Other/Help.
Once that's out of the way, you can log on and see the big visual change - Icons! They don't really do anything but I think their pretty so I'm compelled to point them out.
A few other changes worth calling out:
Property Pane: All known names for a property tag are given. Names which match the type are in the Property Name(s) column, while names that differ in type are given in Other Name(s). I've added over 6000 new property names in this release. PT_STRING8 and PT_UNICODE properties now show the binary (hex) values for the strings in the Value-Alternate View column. The property pane remembers the order of the columns if you've dragged them around. Named property guids which have been documented show their names Named property names which have been documented show their names The extra prop UI, used for adding specific properties for a GetProps call has been completely rewritten. Access it through Property Pane/Modify Extra Properties.
Contents Pane: PT_LONG properties for which we know flag values show the interpreted values. Column headers have tool tips with the tag number, type, and known property names.
File Output: All text based file output now uses XML. This includes the two main methods of dumping properties. I put in a lot of whitespace and linefeeds to make it easy to compare files. Property Pane/Save Property Pane Contents - will write out the properties in the Property Pane. This can be used to output properties for any kind of object. Save Message(s) to File - The Text File option will output all properties of the currently selected message(s). This includes the attachment and recipient tables and all of the body properties.
For fun - try selecting a folder and trying Save Folder Contents as Text Files. This will create an XML property dump for every message in the folder using the same routine as Save Message(s) to File.
Hex Editor: I know there are plenty of hex editors out there, but none of them ever did just what I needed. So Other/Hex Editor is a built in hex editor and base64 encoder/decoder.
That really only scratches the surface of the changes I've made. There are many more new commands sprinkled throughout the tool, so feel free to look around.
Oh, and definitely let me know about any bugs you find.
Since I posted some preview documentation for Outlook 2007 I've gotten some questions about support for the beta. We are handling beta support through the newsgroups. The newsgroup for Outlook issues is Microsoft.public.outlook. Our MVPs provide peer support for issues posted there and even file bugs when needed.
For more information, see http://www.microsoft.com/office/preview/community/community.mspx
I finally got around to testing the Wrapped PST sample with Outlook 2007 Beta 2. I found that I had hardcoded the path to mspst32.dll and it really didn't like that the path has changed in the beta. So I've gone in and hardcoded the new path in there as well. I'll leave the task of implementing a robust solution to anyone who wants to build a product out of this. :)
Anyway, that was the only change I had to make. Here's the updated code: http://stephengriffin.members.winisp.net/wrappst/wrappst.zip
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb820947.aspx]
There was a thread a while back in the microsoft.public.win32.programmer.messaging newsgroup about how Outlook 2003's new Unicode MSG files. The big question was how to create one. One poster even went as far as to reverse engineer the MSG format from scratch. Fortunately, there's an easier way, which I just got clearance to publish.
We start with the canonical sample for MSG files: http://support.microsoft.com/kb/171907
The key is the OpenIMsgOnIStg function. This function takes a ulFlags parameter. All you need to do to create a Unicode MSG file is to pass MAPI_UNICODE in this parameter. Once you do that, the resulting message will show STORE_UNICODE_OK in it's PR_STORE_SUPPORT_MASK and unicode properties will work.
This flag is supported in this function on Outlook 2003 and higher only.
Update: Support for these flags has been cut.
This is preliminary documentation for Outlook 2007 Beta 2. It does not apply to earlier versions of Outlook.
Full Text Search Query Support Outlook 2007 introduces full text queries and wordwheel filtering. Store providers wishing to support these queries will need to support new restriction flags and expose their support in their PR_STORE_SUPPORT_MASK.
New Store Support Mask Flags These flags are set by stores which support full text queries and wordwheel.
Definitions
#define STORE_FULLTEXT_QUERY_OK ((ULONG) 0x02000000)
#define STORE_FILTER_SEARCH_OK ((ULONG) 0x04000000)
Usage These flags can be retrieved by getting the property PR_STORE_SUPPORT_MASK from the store.
New Restriction Flags These flags are intended to be used for stores that have full text search support. The PST/OST provider will support these new flags in restrictions.
FL_PREFIX_ON_ANY_WORD: Will match prefix on words instead of the whole prop value. FL_PREFIX_ON_ANY_WORD can be combined with FL_SUBSTRING so the provider can default to substring matching if full text searching has not been enabled.
FL_PHRASE_MATCH: Phrase match means the words have to be exactly matched and the sequence matters. This is different than FL_FULLSTRING because it doesn't require the whole property value to be the same. One term matching another term in the property value is enough for a match even if there are more terms in the property. Word breaker characters will be ignored in the match. FL_PHRASE_MATCH can be combined with FL_SUBSTRING so the provider can default to substring matching if full text searching has not been enabled.
#define FL_PREFIX_ON_ANY_WORD 0x00000010 #define FL_PHRASE_MATCH 0x00000020
Usage These new flags are used in a similar fashion to existing restriction flags in building restrictions. These flags are only supported on stores which have specified STORE_FULLTEXT_QUERY_OK in their PR_STORE_SUPPORT_MASK. Using these flags in stores that do not support them may cause restrictions to fail.
[edit - correcting flag name]
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb821096.aspx, http://msdn.microsoft.com/en-us/library/bb821097.aspx, http://msdn.microsoft.com/en-us/library/bb821099.aspx, http://msdn.microsoft.com/en-us/library/bb821100.aspx]
Notification Based Indexing Support A store provider supports Notification Based Indexing if it doesn’t require incremental or full crawls and indexes all the items through notifications. Sometimes these providers are referred to as “pusher” stores, since such a store provider will push URLs to the indexer by calling appropriate indexer APIs. These URLs are built based on algorithms documented below. Each URL corresponds to one message, folder or attachment. The indexer will pass the URLs to the MAPI Protocol Handler (MAPI PH) which will parse it open to determine which message/folder/attachment it corresponds to. MAPI PH then opens that object through MAPI and gets the properties to be indexed. The text is returned to the indexer so that it can word-break it and save the terms in the catalog. Some properties like bodies or attachments may require the use of the filter host which is a different process. The filter host will basically use IFilters to crack the documents and get the text so that the indexer can then word-break the text.
New Store Support Mask Flag This flag is set by pusher stores.
Definition
#define STORE_PUSHER_OK ((ULONG) 0x00800000)
If the store provider sets this flag the MAPI PH won’t crawl the store and the store is responsible to push any changes through notifications to the indexer to have messages indexed.
Usage This flag can be retrieved by getting the property PR_STORE_SUPPORT_MASK from the store.
New Properties for the MAPI PH
#define PR_PROVIDER_ITEMID PROP_TAG(PT_BINARY, 0x0EA3) #define PR_PROVIDER_PARENT_ITEMID PROP_TAG(PT_BINARY, 0x0EA4)
These properties are used to identify an item or a folder by the store provider. They are retrieved when the store provider gets the search results from the search engine and are used to identify which items matched the query. Store providers can provide provider specific values for these properties but their values shouldn’t change between sessions otherwise they won’t know how to map the items when getting search results.
MAPI URLs Every time a folder, message or an attachment is to be indexed a unique Unicode URL has to be generated and sent to the indexer. This URL will later be used to identify which object to be indexed in the MAPI PH. Store providers that want to implement notification based indexing are responsible for generating these URLs.
MAPI URLs have the following format:
Mapi://SID/StoreDisplayName ($HashNumber)/StoreType/FolderNameA/…/FolderNameN/[EntryIDEncoded[?at=AttachIDEncoded:FileName]]
Parameters
SID Current user’s SID
HashNumber DWORD in hex calculated based on the store entry id and in some case the file path. This value will be stored in the registry and will be used later to identify the store in the MAPI PH. This number has to be calculated in a way that minimizes collisions between other stores. See below for the algorithm Outlook uses to calculate the hash.
StoreType Number that identifies the type of the store that contains the object to be indexed. Here are the possible values for pusher stores: 0 – Norma store (non public or delegate stores. This includes default stores, PSTs, etc.) 1 – Delegate store (used for delegate items cached locally) For stores which have been crawled by the indexer, this value will always be ‘X’.
FolderNameA/…/FolderNameN The path from the root of the IPM_SUBTREE to the folder or message. For instance, a message in the “Family” folder under “Inbox” will have Inbox/Family for this parameter.
EntryIDEncoded MAPI Entry ID for the item encoded as a Unicode string. See below for how it gets encoded. Note that when viewed as text, this encoded entry ID will appear as random Hangul characters or boxes depending on available fonts.
AttachIDEncoded Attachment ID encoded as a Unicode string.
FileName Attachment file name as it appears in the message.
Here are some examples of MAPI URLs for a folder, message and attachment respectively:
mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($be19928f)/2/Office mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Calendar/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾걤곂갠가 mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Inbox/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾간곷갦가/at=겅걋각가:somefile.txt
Algorithm to calculate the store hash
DWORD ComputeStoreHash(ULONG cbStoreEID, LPENTRYID pbStoreEID, LPCWSTR pwzFileName) { DWORD dwHash = 0; ULONG cdw = 0; DWORD* pdw = NULL; ULONG cb = 0; BYTE* pb = NULL; ULONG i = 0; // Get the Store Entry ID // pbStoreEID is a pointer to the Entry ID // cbStoreEID is the size in bytes of the Entry ID pdw = (DWORD*)pbStoreEID; cdw = cbStoreEID / sizeof(DWORD); for (i = 0; i < cdw; i++) { dwHash = (dwHash << 5) + dwHash + *pdw++; } pb = (BYTE *)pdw; cb = cbStoreEID % sizeof(DWORD); for (i = 0; i < cb; i++) { dwHash = (dwHash << 5) + dwHash + *pb++; } // You may want to also include the store file name in the hash calculation // Get store FileName // pwzFileName is a NULL terminated string with the path and filename of the store if (pwzFileName) { while (*pwzFileName) { dwHash = (dwHash << 5) + dwHash + *pwzFileName++; } } // dwHash now contains the hash to be used. It should be written in hex when building the URL. return dwHash; }// ComputeStoreHash
Algorithm to encode the Entry ID and the attachment ID The goal of this algorithm is to generate a compact representation of the Entry ID or attachment ID.
const WORD kwBaseOffset = 0xAC00; // Hangul char range (AC00-D7AF) LPWSTR EncodeID(ULONG cbEID, LPENTRYID rgbID) { ULONG i = 0; LPWSTR pwzDst = NULL; LPBYTE pbSrc = NULL; LPWSTR pwzIDEncoded = NULL; // rgbID is the item Entry ID or the attachment ID // cbID is the size in bytes of rgbID // Allocate memory for pwzIDEncoded pwzIDEncoded = new WCHAR[cbEID+1]; if (!pwzIDEncoded) return NULL; for ( i = 0, pbSrc = (LPBYTE)rgbID, pwzDst = pwzIDEncoded; i < cbEID; i++, pbSrc++, pwzDst++) { *pwzDst = (WCHAR) (*pbSrc + kwBaseOffset); } // Ensure NULL terminated *pwzDst = L'\0'; // pwzIDEncoded now contains the entry ID encoded. return pwzIDEncoded; }// EncodeID
New Notification to Notify the MAPI PH of the Process Pushing MAPI URLs to be Indexed A new MAPI notification type has been introduced to facilitate shutdown scenarios for pusher stores. These stores will have to persist what has to be pushed since it may not be able to index everything before a shutdown occurs. When a store provider is a pusher it should send the following notification so that the MAPI PH knows which process it should watch for a given store. This way if the process is shut down or crashes the MAPI PH will immediately detect that and close (stop indexing) the store it opened.
#define fnevIndexing ((ULONG) 0x00010000) /* Indexing notifications (used for FTE related communications) */ /* Shares EXTENDED_NOTIFICATION to pass structures below, */ /* but NOTIFICATION type will be fnevIndexing */ // Stores that are pusher enabled (PR_SUPPORT_MASK contains STORE_PUSHER_OK) // are required to send notifications regarding the process that is pushing. #define INDEXING_SEARCH_OWNER ((ULONG) 0x00000001) typedef struct _INDEX_SEARCH_PUSHER_PROCESS { DWORD dwPID; /* PID for process pushing */ } INDEX_SEARCH_PUSHER_PROCESS;
BLOB structure associated with each URL when indexing When pushing URLs to be indexed pusher stores should also create a blob with some information that will be used by the MAPI PH. This blob will be associated with each URL and will be sent when the URL is pushed to the indexer. The format of the blob is the following:
DWORD dwVersion DWORD dwFlags ULONG cbProfileName WCHAR wszProfileName ULONG cbProviderItemID WCHAR wszProviderItemID
Note that these values have to be written to the blob in the order shown above. The provider item ID should only be sent for folders to prevent opening extra folders to get this information.
dwVersion This is the version of the data being sent. Currently this value is 1.
dwFlags Reserved for future use. Currently this should be 0.
cbProfileName Size of the profile name in bytes. This information will be useful for the MAPI PH to know which profile to use when indexing the item.
wszProfileName Null terminated Unicode string with the profile name.
cbProviderItemID Size of the Provider Item ID in bytes
wszProviderItemID Null terminated Unicode string with the provider item ID that uniquely identifies the item in the store.
Named Properties The following named properties are indexed by the MAPI PH. These properties are documented as read-only. They should not be used to create or modify items. These GUIDs represent the name spaces of the named properties
const GUID PSETID_Appointment = {0x00062002, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}}; const GUID PSETID_Task = {0x00062003, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}}; const GUID PSETID_Address = {0x00062004, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}}; const GUID PSETID_Common = {0x00062008, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}}; const GUID PSETID_Log = {0x0006200A, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}}; const GUID PS_PUBLIC_STRINGS = {0x00020329, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}}; const GUID PS_INTERNET_HEADERS = {0x00020386, 0x0000, 0x0000, {0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46}};
MNID_ID properties:
// In PSETID_Address #define dispidWorkAddressStreet 0x8045 #define dispidWorkAddressCity 0x8046 #define dispidWorkAddressState 0x8047 #define dispidWorkAddressPostalCode 0x8048 #define dispidWorkAddressCountry 0x8049 #define dispidInstMsg 0x8062 #define dispidEmailDisplayName 0x8080 #define dispidEmailOriginalDisplayName 0x8084 // In PSETID_Task #define dispidTaskStartDate 0x8104 #define dispidTaskDueDate 0x8105 #define dispidTaskActualEffort 0x8110 #define dispidTaskEstimatedEffort 0x8111 #define dispidTaskFRecur 0x8126 // In PSETID_Appointment #define dispidLocation 0x8208 #define dispidApptStartWhole 0x820D #define dispidApptEndWhole 0x820E #define dispidApptDuration 0x8213 #define dispidRecurring 0x8223 #define dispidAllAttendeesString 0x8238 #define dispidToAttendeesString 0x823B #define dispidCCAttendeesString 0x823C // In PSETID_Common #define dispidReminderSet 0x8503 #define dispidSmartNoAttach 0x8514 #define dispidCommonStart 0x8516 #define dispidCommonEnd 0x8517 #define dispidRequest 0x8530 #define dispidCompanies 0x8539 #define dispidReminderNextTime 0x8560 // In PSETID_Log (also known as Journal) #define dispidLogType 0x8700 #define dispidLogStart 0x8706 #define dispidLogDuration 0x8707 #define dispidLogEnd 0x8708
MNID_STRING properties
// In PS_PUBLIC_STRINGS "Keywords" // In PS_INTERNET_HEADERS "return-path"
[4:12 - fixed some truncated prop vals]
[12:10 - 12/20/06 - fixed one-off bug per Robert's comment]
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb820936.aspx]
Blocked Attachments Microsoft Outlook includes a feature that blocks attachments that are considered unsafe. The attachments which are blocked can vary from client to client depending on how Outlook is configured and on policies the administrator may have applied. See http://support.microsoft.com/kb/829982 for more information on how this is configured.
Custom code can query to see if a particular attachment is considered blocked by Outlook using the IAttachmentSecurity interface. This interface exposes a function, IsAttachmentBlocked, which will analyze a file name and report if this attachment is considered blocked by Outlook and won’t be shown in the UI or indexed.
DEFINE_GUID(IID_IAttachmentSecurity, 0xB2533636, 0xC3F3, 0x416f, 0xBF, 0x04, 0xAE, 0xFE, 0x41, 0xAB, 0xAA, 0xE2); #define MAPI_IATTACHMENTSECURITY_METHODS(IPURE) \ MAPIMETHOD(IsAttachmentBlocked) \ (LPCWSTR pwszFileName, BOOL *pfBlocked) IPURE; DECLARE_MAPI_INTERFACE_(IAttachmentSecurity, IUnknown) { BEGIN_INTERFACE MAPI_IUNKNOWN_METHODS(PURE) MAPI_IATTACHMENTSECURITY_METHODS(PURE) };
Usage This interface can be obtained by calling QueryInterface on the MAPI session object, requesting IID_IAttachmentSecurity. IsAttachmentBlocked will return true in pfBlocked if the attachment is considered blocked by Outlook and won’t be shown in the UI or indexed
HRESULT IsAttachmentBlocked(LPMAPISESSION lpMAPISession, LPCWSTR pwszFileName, BOOL* pfBlocked) { if (!lpMAPISession || !pwszFileName || !pfBlocked) return MAPI_E_INVALID_PARAMETER; HRESULT hRes = S_OK; IAttachmentSecurity* lpAttachSec = NULL; BOOL bBlocked = false; hRes = lpMAPISession->QueryInterface(IID_IAttachmentSecurity,(void**)&lpAttachSec); if (SUCCEEDED(hRes) && lpAttachSec) { hRes = lpAttachSec->IsAttachmentBlocked(pwszFileName,&bBlocked); } if (lpAttachSec) lpAttachSec->Release(); *pfBlocked = bBlocked; return hRes; }// IsAttachmentBlocked
[This is now documented here: http://msdn2.microsoft.com/en-us/library/bb820932.aspx]
This should help dealing with scenarios in Outlook that cause forms to become one-offed.
Topic How to remove one-off form attributes from a message which has unexpectedly been “one-offed”.
One-Off Forms A message which has been “one-offed” is one where a custom form definition has been saved with the message. While there are legitimate scenarios for one-offing a form, typically this happens unexpectedly. The negative side effects and descriptions of how messages can become one-offed are discussed in the following Knowledge Base article: http://support.microsoft.com/kb/207896
Removing One-Off Properties A message which has been marked as a one-off can be restored to a regular message by deleting a set of named properties and removing some flags from another named property. This process can be done with Extended MAPI or CDO. The properties which must be deleted are all in the PSETID_Common namespace and are dispidFormStorage, dispidPageDirStream, dispidFormPropStream, and dispidScriptStream. The property which must be modified is also in the PSETID_Common namespace. The flags INSP_ONEOFFFLAGS must be removed from dispidCustomFlag.
In addition, the property dispidPropDefStream in the PSETID_Common namespace may also be removed. If this property is removed, then the flag INSP_PROPDEFINITION should be removed from dispidCustomFlag. A side effect of removing this property is that the Outlook Object Model and the Outlook User Interface will no longer be able to access user properties which have been set on the item. These properties and their values will still be accessible through MAPI. Note that if this property is not removed and the items is one-offed again, it is possible that the property can be overwritten with new data.
#define dispidFormStorage 0x850F #define dispidPageDirStream 0x8513 #define dispidFormPropStream 0x851B #define dispidPropDefStream 0x8540 #define dispidScriptStream 0x8541 #define dispidCustomFlag 0x8542 #define INSP_ONEOFFFLAGS 0xD000000 #define INSP_PROPDEFINITION 0x2000000 DEFINE_OLEGUID(PSETID_Common, MAKELONG(0x2000+(8),0x0006),0,0);
Usage The following sample code illustrates how to remove one-off attributes from a message using Extended MAPI:
ULONG aulOneOffIDs[] = {dispidFormStorage, dispidPageDirStream, dispidFormPropStream, dispidScriptStream, dispidPropDefStream, // dispidPropDefStream must remain next to last in list dispidCustomFlag}; // dispidCustomFlag must remain last in list #define ulNumOneOffIDs (sizeof(aulOneOffIDs)/sizeof(aulOneOffIDs[0])) HRESULT RemoveOneOff(LPMESSAGE lpMessage, BOOL bRemovePropDef) { if (!lpMessage) return MAPI_E_INVALID_PARAMETER; HRESULT hRes = S_OK; MAPINAMEID rgnmid[ulNumOneOffIDs]; LPMAPINAMEID rgpnmid[ulNumOneOffIDs]; LPSPropTagArray lpTags = NULL; ULONG i = 0; for (i = 0 ; i < ulNumOneOffIDs ; i++) { rgnmid[i].lpguid = (LPGUID)&PSETID_Common; rgnmid[i].ulKind = MNID_ID; rgnmid[i].Kind.lID = aulOneOffIDs[i]; rgpnmid[i] = &rgnmid[i]; } hRes = lpMessage->GetIDsFromNames( ulNumOneOffIDs, rgpnmid, 0, &lpTags); if (lpTags) { // The last prop is the flag value // we'll be updating, don't count it lpTags->cValues = ulNumOneOffIDs-1; // If we're not removing the prop def stream don't count it if (!bRemovePropDef) { lpTags->cValues = lpTags->cValues-1; } hRes = lpMessage->DeleteProps( lpTags, 0); if (SUCCEEDED(hRes)) { SPropTagArray pTag = {0}; ULONG cProp = 0; LPSPropValue lpCustomFlag = NULL; // Grab dispidCustomFlag, the last tag in the array pTag.cValues = 1; pTag.aulPropTag[0] = CHANGE_PROP_TYPE( lpTags->aulPropTag[ulNumOneOffIDs-1], PT_LONG); hRes = lpMessage->GetProps( &pTag, fMapiUnicode, &cProp, &lpCustomFlag); if (SUCCEEDED(hRes) && 1 == cProp && lpCustomFlag && PT_LONG == PROP_TYPE(lpCustomFlag->ulPropTag)) { // Clear the INSP_ONEOFFFLAGS bits so OL // doesn't look for the props we deleted lpCustomFlag->Value.l = lpCustomFlag->Value.l&~(INSP_ONEOFFFLAGS); if (bRemovePropDef) { lpCustomFlag->Value.l = lpCustomFlag->Value.l&~(INSP_PROPDEFINITION); } hRes = lpMessage->SetProps( 1, lpCustomFlag, NULL); } hRes = lpMessage->SaveChanges(KEEP_OPEN_READWRITE); } } MAPIFreeBuffer(lpTags); return hRes; }
The following sample code illustrates how to remove one-off attributes from a message using CDO 1.21:
Sub DeleteFormDefinitionWithCDO(MessageEID As String, bDeletePropDef As Boolean) Const strPSetCommonGUID = "0820060000000000C000000000000046" ' PSETID_Common Dim objSession As MAPI.Session Dim oFolderCDO As MAPI.Folder Dim oMessages As MAPI.Messages Dim oMessage As MAPI.Message Dim myFields As MAPI.Fields Set objSession = New MAPI.Session objSession.Logon , , True, False Set oMessage = objSession.GetMessage(MessageEID) Set myFields = oMessage.Fields On Error Resume Next myFields.Item("{" & strPSetCommonGUID & "}0x850F").Delete ' dispidFormStorage myFields.Item("{" & strPSetCommonGUID & "}0x8513").Delete ' dispidPageDirStream myFields.Item("{" & strPSetCommonGUID & "}0x851B").Delete ' dispidFormPropStream myFields.Item("{" & strPSetCommonGUID & "}0x8541").Delete ' dispidScriptStream ' Update dispidCustomFlag myFields.Item("{" & strPSetCommonGUID & "}0x8542") = _ myFields.Item("{" & strPSetCommonGUID & "}0x8542") And Not &HD000000 ' INSP_ONEOFFFLAGS If bDeletePropDef Then myFields.Item("{" & strPSetCommonGUID & "}0x8540").Delete ' dispidPropDefStream myFields.Item("{" & strPSetCommonGUID & "}0x8542") = _ myFields.Item("{" & strPSetCommonGUID & "}0x8542") And Not &H2000000 ' INSP_PROPDEFINITION End If oMessage.Update objSession.Logoff Set objSession = Nothing End Sub
[This is now documented here: http://msdn.microsoft.com/en-us/library/bb820953.aspx]
[This information also published as http://support.microsoft.com/kb/912238]
Just like Cached mode has a way of accessing items in the OST without triggering a download, our IMAP provider has a mechanism for accessing items in the PST without triggering the download. In an ideal world, they'd both use the same mechanism, but nobody's perfect.
Topic A mechanism to manage messages in an IMAP store.
IID_IProxyStoreObject In an IMAP profile, messages in the PST can be in one of two states
Query for this interface to access a function, UnwrapNoRef, which unwraps the IMAP store and allows access to the underlying PST. Without unwrapping, access of a message in the IMAP store can force a synchronization that will attempt to download the entire message. Using the unwrapped store allows access to the message in its current state without triggering this download.
The IMsgStore interface returned by UnwrapNoRef is identical in use to the IMsgStore that was unwrapped. The only difference is that the unwrapped store will return messages as they exist in the PST, without forcing synchronization.
If QueryInterface returns the error MAPI_E_INTERFACE_NOT_SUPPORTED, then the store was not wrapped.
#define DEFINE_PRXGUID(_name, _l) \ DEFINE_GUID(_name, (0x29F3AB10 + _l), 0x554D, 0x11D0, 0xA9, \ 0x7C, 0x00, 0xA0, 0xC9, 0x11, 0xF5, 0x0A) DEFINE_PRXGUID(IID_IProxyStoreObject, 0x00000000L); #define MAPI_IPROXYSTOREOBJECT_METHODS(IPURE) \ MAPIMETHOD(PlaceHolder1) () IPURE; \ MAPIMETHOD(UnwrapNoRef) (LPVOID *ppvObject) IPURE; \ MAPIMETHOD(PlaceHolder2) () IPURE; DECLARE_MAPI_INTERFACE_(IProxyStoreObject, IUnknown) { BEGIN_INTERFACE MAPI_IUNKNOWN_METHODS(PURE) MAPI_IPROXYSTOREOBJECT_METHODS(PURE) };
Usage Call QueryInterface on the source message store to obtain the IProxyStoreObject interface. Then call UnwrapNoRef to obtain the unwrapped store. Note that UnwrapNoRef does not addref the returned pointer.
HRESULT HrUnWrapMDB(LPMDB lpMDBIn, LPMDB* lppMDBOut) { HRESULT hRes = S_OK; IProxyStoreObject* lpProxyObj = NULL; LPMDB lpUnwrappedMDB = NULL; hRes = lpMDBIn->QueryInterface(IID_IProxyStoreObject,(void**)&lpProxyObj); if (SUCCEEDED(hRes) && lpProxyObj) { hRes = lpProxyObj->UnwrapNoRef((LPVOID*)&lpUnwrappedMDB); if (SUCCEEDED(hRes) && lpUnwrappedMDB) { // UnwrapNoRef doesn't addref, so we do it here: lpUnwrappedMDB->AddRef(); (*lppMDBOut) = lpUnwrappedMDB; } } if (lpProxyObj) lpProxyObj->Release(); return hRes; }