SGriffin's MAPI Internals

Inside the PST

Stephen Griffin — Mon, 26 Oct 2009 17:12:46 GMT

As Paul Thurrott notes, we just announced that we’re documenting the PST file structure. I haven’t decided yet if I’ll be writing a standalone parser/viewer/validator, but I’m sure I’ll be adding some interesting stuff to MFCMAPI. I’ll let you know when I know more.

Exchange MAPI Download Version 6.5.8131.0

Stephen Griffin — Tue, 20 Oct 2009 13:45:54 GMT

Just had someone ask me what changed in this version compared to the last build. Most of the changes were tweaks to various timezones, such as the addition of a timezone for Yakutsk, my favorite Risk territory. Additionally, we fixed a problem with ITnef::AddProps. You’d only need to worry about the TNEF change if you’re actually calling into the TNEF API directly.

So Long System Attendant Mailbox

Stephen Griffin — Fri, 09 Oct 2009 22:39:15 GMT

Dave has a post up about the fate of the System Attendant Mailbox in Exchange 2010. Since this mailbox isn’t there anymore in 2010, if you have a solution that depended on it, you’ll need to rework it. Dave gives a few suggestions, such as creating your own mailbox/account (the best option since you can then control your exact permission set) or using the System Mailbox.

Enjoy!

Outlook 2010 and Shutting Down Add-Ins

Stephen Griffin — Mon, 05 Oct 2009 14:07:00 GMT

Ryan has a new post up about changes we’re making to Outlook 2010 to make shutdown faster. These changes are building on the previous changes we made introducing Fast Shutdown. Two key takeaways:

MAPI Providers now get Fast Shutdown by default. They may still opt out of Fast Shutdown by implementing IMAPIClientShutdown.
Outlook 2010 will not call the OnBeginShutdown or OnDisconnection of addins during shutdown.

Ryan gives plenty of best practices for rewriting your addins to deal with these changes (and improve the user experience in the process), including policies you can use to revert the changes if you’re an admin and you have add-ins that cannot be rewritten.

EnumAccounts Sample and Some New Account Management Props

Stephen Griffin — Wed, 30 Sep 2009 16:57:07 GMT

A while back, Jason posted an Account Management API sample. I’ve ~~threatened~~ promised before that I would post an update to this sample, and I finally got around to it. You can download the updated sample here.

In addition to adding a –W command to call DisplayAccountList and show the Account Settings Wizard, I’ve also added a couple new properties that dev have decided to document:

#define PROP_ACCT_USER_EMAIL_ADDR PROP_TAG(PT_UNICODE, 0x000C)
#define PROP_MAPI_IDENTITY_ENTRYID PROP_TAG(PT_BINARY, 0x2002)

PROP_ACCT_USER_EMAIL_ADDR is simply the e-mail address for the account, and PROP_MAPI_IDENTITY_ENTRYID is the address book entry ID for the account, similar to what you’d expect from QueryIdentity. Neither property is expected on every account. For instance, in one profile here, an Exchange account has PROP_MAPI_IDENTITY_ENTRYID but not PROP_ACCT_USER_EMAIL_ADDR, while for an SMTP/POP3 account the situation is reversed.

Dev also asked me to call out an issue that had tripped some folks up. The function GetIdentity is supposed to return a string and the length of the string. The returned string should be NULL terminated and the length should include this NULL terminator in the count. We’ll be updating the MSDN to be more explicit about this, in addition to adding the two new properties.

Some more information about the updated sample: If you’ve been looking for a header for the Account Management API, you can grab and use AcctMgmt.h from this sample. It should have everything you need. Please let me know if it doesn’t. Also, intrepid explorers may want to look at the –I option I added to the sample. It iterates through every possible property tag and requests it, printing the results if it finds something.

Enjoy!

OpenStreamOnFile vs Unicode Files

Stephen Griffin — Tue, 29 Sep 2009 18:26:00 GMT

Let’s take a look at the function OpenStreamOnFile, which is exported by MAPI. Suppose you want to use this function to open a stream on a file that has a Unicode file name. As has been noted before, this doesn’t work. Why is this? According to the documentation, and according to the header, maputil.h, this function takes two parameters, lpszFileName and lpszPrefix, which are of type LPCTSTR. However, it is impossible for an exported function to take a parameter of this type. Why? Because LPCTSTR is a macro that compiles to either LPCSTR or LPCWSTR depending on whether Unicode was defined in the project. But MAPI’s the same no matter how you compiled your project, so the export won’t change.

In truth, OpenStreamOnFile should have been declared as taking parameters of type LPCSTR. We’re working on an update to the MSDN and to the MAPI Headers to fix that. We’ll also be documenting a new function, which you can go ahead and use to open files with Unicode names: OpenStreamOnFileW. This is the Unicode version of OpenStreamOnFile which takes Unicode strings. Here’s the signature:

STDMETHODIMP OpenStreamOnFileW(

   LPALLOCATEBUFFER lpAllocateBuffer,

   LPFREEBUFFER lpFreeBuffer,

   ULONG ulFlags,

   __in LPCWSTR lpszFileName,

   __in_opt LPCWSTR lpszPrefix,

   LPSTREAM FAR * lppStream);


typedef HRESULT (STDMETHODCALLTYPE FAR * LPOPENSTREAMONFILEW) (

   LPALLOCATEBUFFER lpAllocateBuffer,

   LPFREEBUFFER lpFreeBuffer,

   ULONG ulFlags,

   __in LPCWSTR lpszFileName,

   __in_opt LPCWSTR lpszPrefix,

   LPSTREAM FAR * lppStream);

Other than the change to the strings, the function is identical to OpenStreamOnFile. Officially, we’re only documenting this function for Outlook 2010 and higher, but the function is also there in Outlook 2003 and 2007. The Exchange MAPI Download does not implement OpenStreamOnFileW at all.

The November build of MFCMAPI is being augmented to handle Unicode file names. Part of that rewrite is that MFCMAPI will use OpenStreamOnFileW if it’s available, and fall back to OpenStreamOnFile if it’s not.

Enjoy!

Detecting Sharepoint PSTs

Stephen Griffin — Mon, 28 Sep 2009 19:58:05 GMT

We had a customer recently who was asking how to tell a PST had been configured as a Sharepoint PST, so they could exclude it from some processing they were doing. After some digging, I found the property that Outlook itself uses to determine a PST is a Sharepoint PST, and development gave me permission to document it:

#define PR_ASSOCIATED_SHARING_PROVIDER PROP_TAG(PT_CLSID, 0x0EA0)

To check if a PST is a Sharepoint PST, mount the PST using OpenMsgStore, then call GetProps on the message store object requesting this property. If it exists, you can assume the PST has been configured for Sharepoint. If it doesn’t exist, the PST has not been configured as a Sharepoint PST.

September 2009 Release of MFCMAPI

Stephen Griffin — Thu, 10 Sep 2009 23:48:12 GMT

The September 2009 Release (build 6.0.0.1015) is live: http://mfcmapi.codeplex.com.

The July 2009 release was downloaded over 13 thousand times, with nearly 1700 of them being the 64 bit build. This was the most popular release yet – retiring MAPI Editor drove a lot of traffic over to codeplex.

I did a lot of work on the property and hex editors this time around. The property editor should be much better at editing large and complex streams you’ll find in PR_BODY or PR_RTF_COMPRESSED. And the Hex Editor is no longer modal! Now maybe Rick will stop bugging me about it.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

Ctrl+H will now bring up a non-modal Hex Editor
That “Sample version” text is now gone. There is only one version of MFCMAPI, and it’s what you get on codeplex
Embedded NULLs in strings no longer break the property and stream editors
All the editors (hex, property, stream) should be able to handle MUCH larger strings now
Smart View: Added parsing for PR_ADDITIONAL_REN_ENTRYIDS, dispidPropDefStream
Smart View: Added framework for Smart View parsing of multivalued properties, and added a bunch of previously unparsed properties.

Enjoy.

Exchange 2010 and the MAPI Download

Stephen Griffin — Wed, 02 Sep 2009 21:06:17 GMT

Bunches of people have contacted me asking if the MAPI download works with Exchange 2010. My usual response has been that I know it works, but you have to tweak the profile. I didn’t know the specific tweaks ‘cause I hadn’t looked in to it.

Dave Vespa just blogged those tweaks. Note that they’re very similar to the modifications we use with Exchange 2007 to get referral working. Also note that the profile Dave builds is not referral enabled, but this doesn’t matter any more. With Exchange 2010, MAPI always talks to the Client Access Server for both mailbox and directory. The dreaded DSProxy that we were trying to avoid in 2007 and earlier has been replaced by the Address Book Service, which is pretty much what DSProxy should have been.

Enjoy!

Outlook 2010 MAPI Reference Updated

Stephen Griffin — Tue, 01 Sep 2009 00:01:16 GMT

The Outlook 2010 MAPI Reference has been updated. I don’t have a list of everything that changed, so I’ll highlight one addition that’s pretty cool: the property definition stream. Long time readers might remember I mentioned this property before – it’s one of the properties you want to delete to kill a one-off form. It’s also the property that Outlook uses to store information about named properties created via the Object model and the user interface.

Now that the format of PidLidPropertyDefinitionStream is documented, you can create new user defined properties using MAPI and have them show up in the UI! The September release of MFCMAPI (which I’m furiously hammering on) will be able to parse this property.

As usual, the documentation is also available for download.

Enjoy!

Responding to Meetings with CDO

Stephen Griffin — Fri, 21 Aug 2009 17:59:02 GMT

My colleague Edwin, writing for the Exchange Support Team Central Europe blog, posted an article on accepting meeting requests with CDO. The key takeaway from the post is this:

This means that when calling the respond method on meeting updates / requests, the sequence number must be read from the request and written to the response, otherwise Outlook will not be able to match the response to the original meeting.
(The proposed solution is to read the dispidApptSequence from the original message and write this to the meeting update response before sending.)

He also points out that this requirement is spelled out in the MAPI protocol documentation, here.

Since CDO has reach end-of life, we won’t be taking a change to CDO to have it do this, so you’ll need to do it yourself. Edwin gives sample code that demonstrates how.

Enjoy!

More Fun With Check Name

Stephen Griffin — Fri, 14 Aug 2009 00:49:18 GMT

I just closed case where the customer was trying to create a profile but Check Name was constantly prompting for credentials and then failing. This wasn’t the reconfig issue I mentioned before – they couldn’t create the profile in the first place. Even GCReconnect showed the same behavior – repeated prompts for credentials, then failure, with the error MAPI_E_USER_CANCEL. We took a network trace of the attempt to create the profile to see where the failure came from. The trace showed us attempting to set up NTLM authentication, sending the request to the Exchange server. Exchange passed the packet on the the GC, which promptly returned 0x721: RPC_S_SEC_PKG_ERROR.

This is a common error to get when attempting to use Kerberos authentication. Configuring Kerberos at times can be tricky, and if it’s not all set up right, this is the error you expect to get. One workaround usually given is to make sure NTLM is set up and fall back to NTLM when Kerberos doesn’t work. However, in this case, we were trying NTLM in the first place!

After some poking around, we took a look at the local policy settings on the three machines. Here’s what we found:

Client
Network security: Minimum session security for NTLM SSP based (including secure RPC) clients- Not defined
Network security: Minimum session security for NTLM SSP based (including secure RPC) servers- Not defined
Exchange
Network security: Minimum session security for NTLM SSP based (including secure RPC) clients- Require NTLMv2 session security, Require 128-bit encryption
Network security: Minimum session security for NTLM SSP based (including secure RPC) servers- Require NTLMv2 session security, Require 128-bit encryption
GC
Network security: Minimum session security for NTLM SSP based (including secure RPC) clients- Require NTLMv2 session security, Require 128-bit encryption
Network security: Minimum session security for NTLM SSP based (including secure RPC) servers- Require NTLMv2 session security, Require 128-bit encryption

Since the client was not configured to use NTLMv2, it tried to authenticate with NTLMv1. Since the server required NTLMv2, it rejected the request, with the appropriate error message: RPC_S_SEC_PKG_ERROR. Once the customer configured the policy on the client (it had just been overlooked during setup), they were able to complete profile configuration.

For more information on these policies, see here and here.

Auth and Check Names

Stephen Griffin — Wed, 12 Aug 2009 00:49:00 GMT

When I posted the Referral Madness article, there was an intriguing comment that I didn’t get a chance to investigate until it came up in a case. The commenter noted that when we used RPC_C_AUTHN_GSS_NEGOTIATE as our authentication mechanism, we could no longer use “Check Name” in the dialog brought up by ConfigureMsgService:

If we hit Check Name, we get an error that the name could not be resolved due to network problems:

To understand why we get this error, we look at one of the caveats I noted:

During profile configuration (ConfigureMsgService), the Exchange provider will do an NSPIBind to the Exchange server to lookup the mailbox name. This single bind and lookup is unavoidable.

In order to perform the name resolution involved in Check Name, the Exchange provider is going to go through DSProxy. In network terms, this means we’re sending the packets directly to the Exchange server. DSProxy picks up our packets and forwards them on, unchanged, to the GC. Consider how this affects authentication. When we use RPC_C_AUTHN_GSS_NEGOTIATE, in most environments this means we’re going to use Kerberos. Combining the two: the client thinks it needs to negotiate Kerberos authentication with the Exchange server. It begins the handshake, sending a packet with a ticket using the Service Principal Name (SPN) for the Exchange server. DSProxy grabs this packet and whips it on over to the GC. The GC tries to parse the ticket in the packet but can’t since the ticket wasn’t intended for the GC. The GC returns the Kerberos error KRB_AP_ERR_MODIFIED. This error bubbles up on the client side as MAPI_E_NETWORK_ERROR, resulting in the dialog.

This problem with proxying Kerberos packets was noted in a KB article a long time ago, which has one of the earliest references to profile settings to enable referral. In the KB article, they wanted to enable Kerberos authentication to get around a problem with authenticating local system accounts, but due to the problem discussed here, also had to enable referral.

Does this mean we’ve got a bug in DSProxy’s handling of Kerberos? Not really – DSProxy predates Windows’ adoption of Kerberos by a number of years. Kerberos authentication can by proxied, but the proxy must be aware it’s proxying Kerberos traffic – it’s not as simple as reading packets off one port and sending them out another. On the other hand, NTLM authentication, the default used by the Exchange provider when we don’t specify an authentication mechanism, can handle being passed to the GC by DSProxy. This would fall under the heading of design limitation.

So – the upshot:

Failure to perform Check Name in this scenario does not indicate a problem with the profile and does not indicate a problem with the network.
If you’ve configured a profile for referral and reconnect, using RPC_C_AUTHN_GSS_NEGOTIATE, you cannot use ConfigureMsgService to reconfigure the profile. If you really want to reconfigure the profile, build a new one instead.
If you want to be really clever, you could remove the auth setting before calling ConfigureMsgService, then add it back when you’re done.

Your Cache is No Good Here

Stephen Griffin — Fri, 07 Aug 2009 20:54:15 GMT

The first time we saw this, we wrote it off as an odd thing one customer did. But we’ve seen it a few times over the past couple years and it deserves some discussion. The issue is with Exchange Client Extensions (remember – they’re dead in 2010!): Customers would report that, when client extensions from certain companies are loaded, Outlook will behave strangely. The symptoms would usually be a crash or a hang. On debugging, we would find that even the hang was preceded by an access violation.

When we debugged this, what we saw on the stack was an Exchange Client Extension calling into an IExchExtCallback object. Nothing unusual there, that’s what a callback object is for. What was unusual though, was that when we look further up the stack, Outlook hadn’t called into into the extension! Here’s the usual sequence of operations:

ECE registers for various events
Outlook fires one of the events, calling into the ECE’s event handler.
One of the parameters of the event is an IExchExtCallback object, which the handler can use to obtain more information about the context of the event.
The event handler returns.
Outlook releases the IExchExtCallback object, which had been created specifically for this event.

Note that last bit – Outlook creates these callback objects individually for the events they’re handed to, and releases them when the event is done. Note also, that these objects only make sense in the context of an event. For instance, the GetObject method returns a message store and a MAPI object. During the context of an event, which message store and which object would be returned makes sense. But after the event is done, there’s no context to establish which store or which object should be referenced.

And this is why these extensions were throwing an access violation. They had cached a pointer to an IExchExtCallback object they had gotten during an earlier event and were using it later outside the context of the event. Since this object was not designed to handle this, it would crash. Whether Outlook would crash, hang, or otherwise behave oddly depended on whether or not an exception handler caught the AV.

Not only does caching these objects not make sense once you consider the design, it’s actually precluded in the documentation. On the page for IExchExtCallback:IUnknown we have the following note:

A pointer to this interface is passed as a parameter to many methods of the Microsoft Exchange extensibility API. This interface and any interfaces it might return are valid only for the time of the call to one of its methods and might not be retained when the extension object’s method returns.

So – even though extensions are going away in Outlook 2010, if you have an extension in market right now for Outlook 2007 or earlier, you might want to review your code and check if you’re caching this object. If you are, you’re causing crashes for your customers.

Outlook 2010 MAPI Samples

Stephen Griffin — Mon, 27 Jul 2009 17:03:19 GMT

I’ve updated the Outlook MAPI Samples and posted a new project up on Codeplex: Outlook 2010 MAPI Samples. We’ll get the MSDN updated to point at these samples soon. The goal of this update, of course, was to get these samples working with Outlook 2010, especially in 64 bit mode. Most of the changes were minor, swapping ULONG for ULONG_PTR, etc., but I did have to make a couple big changes:

None of the samples link to mapi32.lib anymore. I talked about this already. This was actually easier to do than I expected. I just copied my ImportProc files from MFCMAPI and tweaked them a bit.
The other major change was removing the Exchange Client Extension from Wrapped PST. The extension never had anything to do with the sample. It was just used as a convenient way to drive a test function to illustrate the replication API. I replaced the whole thing with an exported function and a simple VBA macro used to call the function. You’ll find the macro in the file TestMacro.txt

I’m sure there are bugs in these samples. I didn’t get to test every possible scenario. So feedback is welcome – see the Help/Feedback section of the main page.