July, 2011

The July 2011 Release (build 6.0.0.1027) is live: http://mfcmapi.codeplex.com.

This round, the focus was on bulking up MrMAPI’s feature set. MrMAPI is one of the core data collection tools being used in a future update to our Software Diagnostics Platform for Outlook issues. This has driven several of the features I’ve added to MrMAPI to make it easier for the engineers writing the diagnostics to grab the bits of information they need so we can resolve issues faster.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

• MFCMAPI: Empty Folder can now delete one message at a time. Useful for those stubborn super large folders where the EmptyFolder API can’t complete.
• MrMAPI: Fid/Mid lookup can now be performed.
• MrMAPI: Open folders by name. Anywhere that takes a folder can now take a full path to the folder by name, or rooted on one of the “known” folders. The known folders list has been augmented to include the Sync Issues folders.
• MrMAPI: Restrict by subject and message class. In case you just want to output certain messages.
• MrMAPI: List folder contents and hierarchy tables.
• MrMAPI: Better handling of non-Exchange mailboxes.
• SmartView: I’ve augmented how PT_I8 is handled.

Enjoy.

Here’s the scenario the customer gave us: They use Outlook’s MAPI and a single MAPI profile connect to an admin mailbox, and then use CreateStoreEntryID and OpenMsgStore to open various work mailboxes to do their processing. All day they’re opening and closing these mailboxes. In some environments, after they run for some time, one of the OpenMsgStore calls will return MAPI_E_FAILONEPROVIDER. There were a number of variations of the issue, as we’ll discuss below, but what they all had in common was that they were rapidly logging in to mailboxes, at least 5 times over a 10 second span.

This was key to the issue. When we traced the process and debugged the reason for the failure, we saw the Exchange server was returning ecServerPaused when we got the error. Digging further back, we saw we were getting ecWrongServer on earlier, successful connections. If you read through the documentation on these errors, you’ll see this is a defense mechanism for Exchange. The ecWrongServer error, which never bubbles up to the caller, indicates that the mailbox requested is not on the server we’re talking to. This initiates a redirect conversation to find the right server. This negotiation is expensive, especially since the server conducting it shouldn’t have been contacted in the first place. So if the same client triggers too many redirects, Exchange returns ecServerPaused as a signal to the client that maybe it’s doing something wrong. You can find this error by calling GetLastError after OpenMsgStore fails. Outlook itself will use this information to trigger an update to the profile to ensure it’s talking to the right server.

So, armed with this information, we went looking at the customer’s code to see what they were doing wrong. They would connect to mailbox Homer on server Alpha, obtain the DNs for mailbox Bart and server Beta, and pass these in to CreateStoreEntryID. They would then use this entry ID, and still see the problem if they logged in rapidly. They did nothing wrong and still had a problem.

This is where things get complicated. First, let’s look at the output of CreateStoreEntryID (as parsed with MFCMAPI):

MAPI Message Store Entry ID:
abFlags = 0x00000000
Provider GUID = {10BBA138-E505-1A10-A1BB-08002B2A56C2} = muidStoreWrap
Version = 0x00 = MAPIMDB_VERSION
Flag = 0x00 = MAPIMDB_NORMAL
DLLFileName = EMSMDB.DLL
Wrapped Flags = 0x00000000
WrappedProviderUID = {20FA551B-66AA-CD11-9BC8-00AA002FC45A} = g_muidStorePrivate
WrappedType = 0x0000000C = OPENSTORE_HOME_LOGON | OPENSTORE_TAKE_OWNERSHIP
ServerShortname = Beta
MailboxDN = /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Bart

Note that the entry ID contains the server name “Beta”, but not the full DN, even though the full DN was passed in to CreateStoreEntryID. This is because the entry ID format generated by CreateStoreEntryID does not support using the full server DN. So only the short name is kept, and that’s all emsmdb32 has to work with. To talk to the server, we need the full DN, so we have to manufacture a DN using the information we have at hand. We do this by grabbing the DN of a known good server (in this example, Alpha) and replacing the server name. We then have a dilemma – this new DN may point to a real server, or it may be total garbage, for instance, if the OUs of the servers were different, this made up DN doesn’t exist. So instead of using this newly constructed DN to connect, we instead connect to the only server we know is functioning, using the server DN of the mailbox in the profile.

Of course, this is wrong and we get redirected, and if we repeat this dance several times we’ll hit ecServerPaused. We recognized the potential to do better by implementing a cache of recently used DNs. If the DN we constructed was on this list, we’d go ahead and try it. So the first time we connect to Beta, we redirect through Alpha, but subsequent connections will use Beta’s DN since it’s in the cache. This fix first showed up in Outlook 2003 and 2007 via 929307 and 937949 respectively. So I had the customer check that their customer’s Outlook was up to date. Sure enough, one of their clients wasn’t, and updating it fixed the problem. This didn’t fix the others though, so we debugged again.

The next site we looked at had an updated Outlook, but the the two servers were in different OUs. We’d build a (wrong) DN for Beta using Alpha’s DN. This DN wouldn’t be in our cache, so we’d connect to Alpha for a redirect. We’d then cache the (correct) DN for Beta. This same sequence of steps would repeat the next time we log in to Beta. As long as the two servers are in different OUs, we’ll always redirect. This customer was able to move mailboxes around and eliminate the problem.

We then encountered a site with a single OU and up to date Outlook where the problem still happened. A close look at a trace of the problem revealed that in this particular configuration, the customer’s code managed to log all the way out of MAPI in between mailbox logons. So the cache we went through all the trouble to maintain was torn down and rebuilt on every pass, making it totally ineffective. Keeping one mailbox open the entire time was enough to keep the cache alive.

We were able to get their code running in every environment except the ones where they couldn’t move their mailboxes and had to keep them in different OUs. For those, the only workaround they can use right now is to slow down a bit. Of course, things would be easier if there was a documented entry ID format that included the FQDN. Maybe one day there well be…

Update: I just documented the v2 Store Entry ID format.

I just got word that the Outlook 2010 MAPI Reference has been updated. There were a number of minor bug fixes. The most notable addition was the incorporation of information about the MAPIStubLIbrary into the article about linking MAPI.

Enjoy!

As I intimated before, Outlook’s MAPI supports a special kind of store entry ID which allows you to encode the full DN for the server and avoid redirects. Here’s the documentation for it.

An Exchange Store Entry ID encodes an Exchange server and mailbox for us in OpenMsgStore. The base version of this format is documented here:
http://msdn.microsoft.com/en-us/library/ee203516(EXCHG.80).aspx

If you follow the format documented there, you will build an internal entry identifier for the Exchange MAPI provider, emsmdb32.dll. This entry ID can be wrapped with WrapStoreEntryID to get an entry ID that you can pass to OpenMsgStore. If you do this, what you generate will be the same as the entry ID generated by CreateStoreEntryID. In Outlook 2003, we introduced support for a v2 entry ID format that builds on the original format. In protocol documentation format, it looks like this:

Store Object EntryIDs

A Store object EntryID identifies a mailbox Store object or a public folder Store object itself, rather than a message or Folder object residing in such a database. It is used in certain property values.

Flags (4 bytes): MUST be 0x00000000.

ProviderUID (16 bytes): MUST be %x38.A1.BB.10.05.E5.10.1A.A1.BB.08.00.2B.2A.56.C2.

Version (1 byte): MUST be zero.

Flag (1 byte): MUST be zero.

DLLFileName (14 bytes): MUST be set to the following value which represents "emsmdb.dll": %x45.4D.53.4D.44.42.2E.44.4C.4C.00.00.00.00.

WrappedFlags (4 bytes): MUST be 0x00000000.

WrappedProvider UID (16 bytes): MUST be one of the following values:

Store object type	ProviderUID value
Mailbox Store object	%x1B.55.FA.20.AA.66.11.CD.9B.C8.00.AA.00.2F.C4.5A
Public folder Store object	%x1C.83.02.10.AA.66.11.CD.9B.C8.00.AA.00.2F.C4.5A

WrappedType (4 bytes): MUST be %x0C.00.00.00 for a mailbox store, or %x06.00.00.00 for a public store.

ServerShortname (variable): A string of single-byte characters terminated by a single zero byte, indicating the shortname or NetBIOS name of the server.

MailboxDN (optional) (variable): A string of single-byte characters terminated by a single zero byte and representing the X500 DN of the mailbox, as specified in [MS-OXOAB]. This field is present only for mailbox databases.

V2 (optional) (variable): An EntryIDv2 structure giving DN and FQDN for the server.

EntryIDv2 Struct

Magic (4 bytes): MUST be 0xF32135D8.

Size (4 bytes): An unsigned 32 bit integer giving the total size of the EntryIDv2 structure, including the ServerDN and ServerFQDN.

Version (4 bytes): MUST be 0x00000001.

OffsetDN (4 bytes): The offset in the buffer of the ServerDN. Should be 0x00000000 if ServerDN is not present.

OffsetFQDN (4 bytes): The offset in the buffer of the ServerFQDN. Should be 0x00000000 if ServerFQDN is not present.

ServerDN (optional) (variable): A string of single-byte characters terminated by a single zero byte, indicating the DN of the server.

ServerFQDN (optional) (variable): A string of Unicode characters terminated by two zero bytes, indicating the FQDN of the server.

ReservedBlock (2 bytes): MUST be set to 0x0000.

Remarks

• This format is only understood by Outlook’s MAPI. It may cause errors if used with Exchange’s MAPI.
• If you are accustomed to building DNs to pass in to CreateStoreEntryID, take care not to append \cn=Microsoft Private MDB to the DN used in the v2 entry ID. That suffix is a hint used by CreateStoreEntryID and is not part of the entry ID format.
• If you build an entry ID using the above format, you will need to wrap it using WrapStoreEntryID.
• As a shortcut, you can use CreateStoreEntryID to build a v1 entry ID, then append the EntryIDv2 struct. If you build the entry ID in this manner, it will already be wrapped and WrapStoreEntryID will not be necessary.
• The ServerDN is an ANSI string, but ServerFQDN is a Unicode string. Be careful not to encode them incorrectly.
• For purposes of avoiding redirects, you can set ServerDN and leave out ServerFQDN. This will trigger a single unavoidable redirect the first time the entry ID is used, followed by direct connections on each subsequent use.
• Some C++ definitions that may be helpful in implementing this:

  #define MDB_STORE_EID_V2_VERSION (0x1)
  #define MDB_STORE_EID_V2_MAGIC (0xf32135d8)
  struct MDB_STORE_EID_V2
  {
  	ULONG	ulMagic; // MDB_STORE_EID_V2_MAGIC
  	ULONG	ulSize; // size of this struct plus the size of szServerDN and wszServerFQDN
  	ULONG	ulVersion; // MDB_STORE_EID_V2_VERSION
  	ULONG	ulOffsetDN; // offset past the beginning of the MDB_STORE_EID_V2 struct where szServerDN starts
  	ULONG	ulOffsetFQDN; // offset past the beginning of the MDB_STORE_EID_V2 struct where wszServerFQDN starts
  };

• If you’re using the above struct, you can compute the ulSize field as follows:

  ulSize = sizeof(MDB_STORE_EID_V2) + (lstrlenA(szServerDN)+1)*sizeof(char) + (lstrlenW(wszServerFQDN)+1)*sizeof(WCHAR);

Enjoy!

Update (8/15/2011)

I’ve recently discovered that some Public Folder entry IDs seen in Outlook 2010 do contain a mailbox DN. This interferes with the usual logic for parsing such an entry ID, since there is no definitive indicator whether the data following Server Shortname is a DN or a structure. A slight modification to the algorithm is necessary: If the DWORD following the ServerShortName is MDB_STORE_EID_V2_MAGIC, then there is no DN and we have a v2 structure. If it is anything else, we have a DN following the Server ShortName. There may or may not still be a v2 structure after the DN, which should be parsed in the usual fashion.