Wiz/dumb : MAPI

ConfigureMsgService fails with MAPI_E_INVALID_PARAMETER (0x80070057)

Patrick Creehan — Fri, 10 Jul 2009 16:02:00 GMT

I recently helped a customer with an issue where they were calling ConfigureMsgService and that call was failing, returning an HRESULT of MAPI_E_INVALID_PARAMETER (0x80070057). After debugging it, we established that the reason that ConfigureMsgService was failing was that the PR_PROFILE_HOME_SERVER_ADDRS property was missing from the profile. Outlook seemed to work fine, logons worked, sending mail worked; it was just that ConfigureMsgService would fail. We tried recreating the profile, but still the property wasn’t being set on the profile.

It turns out that PR_PROFILE_HOME_SERVER_ADDRS gets its value from PR_EMS_AB_NETWORK_ADDRESS, which in turn, gets its value from the networkAddress attribute on the server object in Active Directory. That value was set correctly, but permissions to that object were not. After following the requirements laid out on technet for permissions to AD objects, we determined that the Authenticated Users group was missing the ACL for Read All Properties on the server object. Once we set that permission and recreated the profile, the property was set correctly on the profile and ConfigureMsgService started succeeding.

TNEF (Chapter 2): Old School

Patrick Creehan — Wed, 28 Jan 2009 02:09:40 GMT

As discussed in Chapter 1 of this captivating series, MAPI contains an interface to allow developers to create and read TNEF data. This interface is the ITnef interface. There are only a few methods in this interface and they are, for the most part, self explanatory. The entire process of creating a TNEF stream can be done in just a few steps:

Call OpenTnefStreamEx to get a TNEF stream to write into.

Make sure you pass in TNEF_ENCODE since you’ll be creating TNEF. If you were reading TNEF, you’d pass in TNEF_DECODE instead. The other flags to worry about here are TNEF_BEST_DATA, TNEF_COMPATIBILITY, and TNEF_PURE. All of these just signal to MAPI how you want the properties you add to the TNEF stream treated. They will either be all converted to the old-school attributes (TNEF_COMPATIBILITY) – you shouldn’t use this one; some will be converted to attributes but also written to the attMAPIProps section (TNEF_BEST_DATA); or they will all just be written to the MAPI props and none of them written to the attributes (TNEF_PURE).
Call EncodeRecips and pass in the Recipient table you get from a call to IMessage::GetRecipientTable on your message.
Call AddProps passing in an SPropTagArray of non-transmittable prop tags and use the TNEF_PROP_EXCLUDE flag.

There are essentially two schools of thought for building your TNEF: exclude the props you don’t want and let MAPI deal with the rest; or choose carefully which props you do want to include and add them each piecemeal. These are the reason for having the TNEF_PROP_EXCLUDE and TNEF_PROP_INCLUDE flags. One of them says here are the properties I don’t want you to encode (TNEF_PROP_EXCLUDE) and the other, TNEF_PROP_INCLUDE, says I want you to include all of these properties.

There’s another method, SetProps, which does just that, sets the value of a property in the TNEF stream to a value you supply. This allows you to modify the data of the message you are trying to encode, or add additional properties that weren’t on the original.

Back to AddProps for a moment. The flags that supports don’t stop with TNEF_PROP_INCLUDE and TNEF_PROP_EXCLUDE, There is also TNEF_PROP_ATTACHMENTS_ONLY which says “of the properties I’ve given you to work with, I only want you to include/exclude the ones that have to do with attachments. Contrast that with TNEF_PROP_MESSAGE_ONLY which says "”of the properties I’ve given you to work with, I only want you to include/exclude the ones that have to do with the message itself – not attachments.” Then there’s the CONTAINED flags: TNEF_PROP_CONTAINED and TNEF_PROP_CONTAINED_TNEF. TNEF_PROP_CONTAINED means that these properties are going on an attachment; and the TNEF_PROP_CONTAINED_TNEF means I have TNEF data I’m going to give you to put in an attachment – like if you already had a TNEF blob you wanted to include as an attachment, which I’ll demonstrate below.
Once you get all your properties added and included/excluded properly, you call Finish and you’re done. One thing that makes it a little complicated though, is that you have to keep alive all the pointers and streams, etc you’re using in your TNEF until Finish is called, because that’s when all the internal work is actually done. So when you call Finish, that’s when MAPI says, Oh, ok, let me go get that recipient table you gave me. If you’ve already released it, then the process fails.

So it’s pretty easy to do this. This is essentially the way that MFCMAPI demonstrates how to do it (look in File.cpp under SaveToTNEF). There are problems associated with doing it this way when it comes to Unicode properties and when having multiple embedded messages.

The more complicated way to do this to work around some of the issues described above is to add the properties you want explicitly, including each attachment.

The basic difference in the strategy is that instead of calling AddProps with TNEF_PROP_EXCLUDE, call it with TNEF_PROP_INCLUDE and give it the SPropTagArray you get from a call to GetPropList on the message. You’ll need to filter out the non-transmittable properties (such as custom props and things like the Store EntryID). Once you add all the message props, call GetAttachmentTable and loop through each attachment and do one of two things, if it’s not an embedded message, just add the attachment data with AddProps on PR_ATTACH_DATA_BIN; otherwise, you’re going to recurse over yourself and build a TNEF stream from the embedded message. When you call Finish on it, then you’ll add it to the parent TNEF stream by calling AddProps with PR_ATTACH_DATA_OBJ and using the TNEF_PROP_CONTAINED_TNEF flag and give it the stream for your TNEF blob. Once you unwind all the way, you’ll have your “master” TNEF stream. Essentially, you’ll follow the steps here: http://msdn.microsoft.com/en-us/library/cc839833.aspx

TNEF (Chapter 1): Basics

Patrick Creehan — Fri, 16 Jan 2009 19:17:36 GMT

I’ve worked quite a few cases recently regarding problems some folks have had either reading or composing TNEF content. I’ve learned quite a bit myself as a result, and I thought I’d share. I decided I would do a series of blog posts on the topic and hopefully save some of you the time I spent learning all this.

So, being the first post on the topic, I suppose now would be a good time for a review on just what TNEF is and how it’s structured.

TNEF stands for Transport-Neutral Encapsulation Format. If you use Outlook or any other MAPI client as your mail client, you may know that MAPI is a protocol for communication between the client and the mailbox server. MAPI defines a set of interfaces which the client can use to work with the data in the mailbox. The MAPI structure for the data is hierarchical with messages being contained in containers, which themselves can have a parent container, all the way up to the root of the store. MAPI also defines a set of properties understood by the client and, in some cases, the server. If all mail sent could stay on this one server and only go between clients on this one system, this would be all we need; but we know that’s not the case. A very large quantity of e-mail is sent over the internet to foreign systems every day. The vast majority of those use an industry-standard protocol called SMTP (Simple Mail Transfer Protocol) to send messages in an industry-standard format called MIME (Multipurpose Internet Mail Extensions). MIME is composed of body parts, which can in turn be composed of additional body parts themselves. MIME also allows you to add headers to each of the body parts which allow you to describe the content of that body part. So one body part may be a Word Document attachment, so the MIME headers on that body part would contain the MIME type such as application/doc and the transfer type, such as base64. The content of that body part would then contain a base64 encoding of the document. The headers for the root body part contain information such as the subject of the message, the sender and recipient information, etc.

As a message makes its way through transport from one person’s email client to another person’s, it encounters many “hops” (brief stops at SMTP servers in the routing path) which have the opportunity to modify the headers. They do this in order to track the path the message took or to flag it as SPAM, verify the sender address, etc. So there’s a chance the headers you specify when you send the message won’t be the same as when the message arrives at the destination. The headers also only support text values. One of the problems discovered early on about the MIME format was that it has no concept of “rich text.”

In early versions of Outlook, users wanted the ability to send and receive email that contained rich text bodies. Microsoft devised a plan to create an attachment to the messages it was sending that would have a certain content-type and would come to have a well-known name, “winmail.dat”. This attachment would contain an encapsulation of the MAPI properties that could represent this rich body that would work across any transport and be readable by any system that supported MIME.

The original structure just supported a very simple structure that was basically Name/size/value. These were called “attributes” and the names of these attributes are still prefixed by “att.” Many of the attribute names can be seen here: http://msdn.microsoft.com/en-us/library/cc765736.aspx. The most important of these attributes for the purposes of our discussion will be the attMAPIProps. This attribute contains a list of MAPI properties that the receiving system can set on the message once it has converted the other MIME parts into their MAPI format. Some of the TNEF attributes can be directly translated into MAPI properties as defined by the link earlier in this paragraph, but there is not a 1-1 mapping between TNEF attributes and MAPI properties – hence the attMAPIProps attribute. Attachments and recipient data can also be encoded into the TNEF structure, which we’ll examine more later.

MSDN documents the general structure of TNEF but it’s hard to understand. Last year, when Exchange decided to be among those systems that elected to publicly document their protocols, they created [MS-OXTNEF].pdf, which documents very clearly the structure of the TNEF data and how to parse it. Don’t get too nervous, though. I have parsed a 3MB TNEF blob manually myself, but in Exchange 2007, we provide managed code interfaces to allow you to read (or write) this data very easily. In subsequent posts, I’ll dive more into the structure and into the managed classes, as well as the legacy MAPI ITnef interfaces, and more into problems you may experience in developing TNEF-enabled applications.

New Drop of MAPI/CDO Download Released Today

Patrick Creehan — Sat, 20 Dec 2008 00:48:47 GMT

After a hiccup earlier this week in getting the new MAPI & CDO downloads published to microsoft.com, they released today.

MAPI / CDO

http://www.microsoft.com/downloads/details.aspx?familyid=E17E7F31-079A-43A9-BFF2-0A110307611E&displaylang=en

CDO 1.21

http://www.microsoft.com/downloads/details.aspx?familyid=2714320D-C997-4DE1-986F-24F081725D36&displaylang=en

MAPI Docs Moved

Patrick Creehan — Thu, 04 Dec 2008 18:34:00 GMT

So, the Exchange team decided they didn't want to maintain the MAPI documentation anymore since they don't ship MAPI anymore. So the Outlook team stepped up and took over the docs. As such, you can now find them under the Outlook branch in the MSDN left-nav.

http://msdn.microsoft.com/en-us/library/cc765775.aspx

Steve Griffin has been working with the Outlook team on prettying them up and updating them for a few months, and they're now better than ever! He even got a new icon for MFCMAPI out of the deal!

RPC - Really Problematic Conundrum

Patrick Creehan — Tue, 15 Jul 2008 17:49:52 GMT

I’ve had a couple cases recently where customers were having problems getting into deadlock with one of our modules, usually MAPI or Outlook. There’s one common thread that connects these cases: message pumps.

Your application has to make a remote call to a server object living in a different process or thread. Often times this is referred to as an “asynchronous” call. That term is somewhat misleading – suggesting your process could actually forget that it made the call and then be surprised when it comes back. In actuality, there’s always a listener constantly waiting for the call to return. What makes it seem asynchronous is that the waiting happens on a different thread. In order to wait for your call return, applications often implement logic called a “message pump.” This just processes incoming Windows messages or RPC calls and dispatches them to be processed; and it keeps processing messages until your call returns.

The problem with this is that there’s no easy way to tell what these incoming messages are or what they’ll do. This is what gets you into trouble.

Let’s say that a normal code path in one thread will go something like this:

Outlook does some work
Outlook locks a critical section (CS1)
Outlook calls into you
You do some work
You get a lock on a critical section (CS2)

There’s nothing wrong with this at all. Everything is fine at this point. Now lets suppose there’s a different thread that does something like this

You do some work
You get a lock on a critical section (CS2).
You make an RPC call or otherwise pump messages.

Now, up to this point there’s nothing wrong. Suppose, however, that one of the pending messages is from Outlook and the execution goes something like this:

…You make an RPC call…
The message pump dispatches the Outlook call
Outlook does some work
Outlook locks a critical section (CS1).

Again, nothing is inherently wrong with this execution path by itself. However, when you combine both threads, you have critical sections being locked in reverse order – classic deadlock.

Here’s a way to understand the problem: Let’s say you’re at home watching a Virginia Tech football game (There are other teams? Oh yeah, I just call them “losers.”) with a friend. You both get a hankering for a cold adult beverage. You go to the fridge, but there’s only one left. You decide to split it with your buddy, which is ok, since it’s a 40. Meanwhile, your other friends arrive with their own adult beverages and ask if anyone has an opener. Your buddy gets up and gets the bottle opener. You are thinking, “OK, I have the bottle, now all I need is the bottle opener, but my buddy has it so I’ll just wait until he’s done.” You both return to your respective recliners. Your buddy is thinking, “ok, I have the opener, I guess I’ll just start opening bottles with it until my glass gets filled. Surely one of the bottles will be mine.” As he’s opening your other friends’ bottles, the thought enters his mind – my friend [you] has a bottle he needs opened – I need to get his bottle to open it for him. Since both yours and your buddy’s rear ends are plastered to their respective recliners, you have now entered adult beverage deadlock.

Your buddy was keeping the bottle opener all to himself while waiting for his glass to be filled. Little did he understand that by holding the bottle opener to himself, he was preventing that from happening, since the bottle was already being held firmly in your grip. Had he relinquished control of the bottle opener, you could have easily picked it up and used it. He didn’t know that one of the bottles he had to open would be yours.

So how do we prevent this problem from occurring in the first place? The key is really that your buddy didn’t know what bottles he would be opening, while waiting for his glass to be filled. He didn’t know that you were planning to split the beverage with him. What he could have done is just told your friends that the bottle opener was in the drawer instead of holding onto the bottle opener while waiting for his glass to be filled. That way, when you needed it, it would be available and as long as everyone puts it back when they’re done with it, though you may need to wait for a short time, no one will go thirsty for long.

The moral of the story is that you shouldn’t make it a practice to hold onto a lock while making an RPC call or otherwise pumping messages because you don’t know what messages will arrive in your pump and what locks they will request.

Secondary moral of the story: always make sure your adult beverage supply is properly maintained.

Go Hokies!

MAPI_E_FAILONEPROVIDER (0x8004011d) after installing Windows Server 2003 SP2

Patrick Creehan — Sat, 12 Jan 2008 00:56:00 GMT

Sorry for the duplication, I'm just rebranding an earlier post of mine because many people have never heard of SNP and may discount the post due to the title. If you are getting MAPI errors because of RPC connectivity problems, and you have SP2 installed for Windows Server 2003, this is probably what's affecting you, especially if you have a Broadcom NIC.

http://blogs.msdn.com/pcreehan/archive/2007/10/16/ooh-snp.aspx

HELP: Outlook 2003 Doesn't Display my HTML Body

Patrick Creehan — Thu, 10 Jan 2008 18:46:00 GMT

If you've developed a custom message store and you can't get Outlook to display RTF or HTML content, check the length of your EntryIDs. If your RTF content is really just wrapped HTML content or you are providing PR_HTML, and for some reason Outlook 2003 won't display your HTML content, it may be the length of your EntryIDs.

Outlook 2003 leverages IE (mshtml) to display it's HTML content. It does this by implementing a protocol handler for the outbind:// protocol. Outbind's format leverages the EntryID of the message when creating the URL. You can probably see where I'm going here. If your entryID is longer than the allowable limit for a URL, which for IE7 is 0x200 (512) characters, then mshtml fails the load of the HTMLDocument. When that happens, Outlook says "oh, well, RTF/HTML didn't work, let's just load PR_BODY and live with that.

If you've verified that your EntryID is sufficiently short enough (maybe only 100 or so characters) then you may want to look at the values you're returning in the GetProps or QueryRows call.

This could change from version to version, but in general, you just return MAPI_E_NOT_FOUND for properties you don't support, MAPI_E_NOT_ENOUGH_MEMORY that you support but don't want to return in this GetProps call necessarily. If you return the size error for PR_BODY, PR_HTML, and PR_RTF_COMPRESSED then the RTFSync property will determine the preference on whether we use HTML or RTF. Of course, you should always be returning the value (or MAPI_E_NOT_FOUND) if asked for the property directly (as in OpenProperty or HrGetOneProp).

Folder Homepage no longer works in Outlook 2007 from MAPI store providers

Patrick Creehan — Wed, 19 Dec 2007 19:02:00 GMT

If you are developing or have developed a MAPI store provider for use in Outlook and are trying to support folder homepages in your store, you are probably familiar with the PR_FOLDER_WEBVIEWINFO property. This property is undocumented and not supported, but I'll assume for the sake of this article that you have a read/write store and you're allowing Outlook to set that property for you. :)

You may have noticed that folder homepages worked in previous versions of Outlook, but don't work in Outlook 2007. This is because we are turning off folder home pages (FHP) for non-default stores by default in Outlook 2007.

You cannot add a URL to the Address box on the Home Page tab in Outlook 2007 (923933)

Simply setting the registry key below will re-enable them:

\Software\Policies\Microsoft\Office\12.0\Outlook\Security
"NonDefaultStoreScript"=dword:00000001

Free/Busy Appointments Missing After Using CDO 1.21 to Access Calendar

Patrick Creehan — Fri, 14 Dec 2007 20:05:00 GMT

If you have written or use an application that uses CDO 1.21 (cdo.dll) that accesses your Calendar and you have users who complain that some recurring appointments are not appearing in Free/Busy in Outlook, this may be for you.

When a recurring appointment is created in Outlook, you may not realize that really this is just a singular message in the MAPI store. We commonly refer to this as the _Master_ appointment. Whenever a client application or API looks for appointments in your calendar, it will expand the Master appointment into several instances according to the recurrence pattern you set when you made it a recurring appointment. These instances of the recurring appointment are interpreted - they don't actually exist in the store.

When you make a change to one of these instances, such as changing the start time or the location, or if the instance is deleted, an _exception_ item is created. The exception item is a copy of many of the properties of the master appointment which is then attached to the master appointment. When the messaging infrastructure expands the master, this exception item is included in the expansion algorithm so that when expansion occurs, you see your changes or deletions appropriately.

The other change that occurs when you create an exception item is that the recurrence pattern is modified on the master appointment. The recurrence pattern is a binary MAPI property stored on the master appointment message. This binary property stores the details of the recurrence pattern as well as information regarding exception items. It is not supported to modify this property directly. You can find code available on the Internet to reverse engineer this property and modify it, but this is strictly unsupported by Microsoft.

What happens in cases where this property has been improperly modified or otherwise corrupted is that one or more messaging systems (Outlook, CDO, Exchange) are unable to properly expand the appointment.

This is the root cause of the issue described in the first paragraph. In one case I saw recently, the recurrence pattern had been improperly modified. Outlook and OWA clients could see the appointment just fine, with one exception. There were appointments missing in Free/Busy. We were able to determine that if Outlook was the last application to publish free/busy - that is, make a change to the calendar which caused free/busy information to be republished, then the recurring appointments would appear just fine. If a certain CDO 1.21 application was the last to access the calendar, then the appointments vanished from Free/Busy.

After doing some troubleshooting, I was able to determine that the Free/Busy symptom was just a symptom of a much larger problem. When I would get the Items collection from the calendar folder, that collection did not contain my problem appointments (not even the Master appointment). When CDO 1.21 would go to publish free/busy, it would get the collection of rows from MAPI (which contained the appointment) but would then attempt to expand the recurring instances. It detected a problem with the recurrence pattern (which Outlook didn't care about or check for) and would essentially ignore that appointment. That is why the appointment did not appear in Free/Busy - CDO was ignoring it altogether.

Moral of the story - respect the recurrence blob.

Ooh, SNP

Patrick Creehan — Tue, 16 Oct 2007 23:46:00 GMT

We've recently been seeing an elevated number of cases where networking issues are adversely affecting performance and causing other problems with messaging technologies. The common thread in many of them is that the problem only started to occur after Windows Server 2003 SP2 was installed. I recently learned that we shipped a new feature in SP2 and turned it on by default. This new feature is called Scalable Networking Pack, or SNP. It was actually originally shipped as a download for SP1, but was included in SP2 by default.

The Exchange team has posted an article on how SNP affects Exchange and also gives a good list of links and troubleshooting steps.

SNP was built to increase performance on the networking stack, and does for newer network adapters that support it. However, for older network adapters, it can cause significant delays which cause timeout problems and connection problems leading to various problems.

The basic troubleshooting steps at this point are

Read the Exchange team's post linked above and pages they reference so you understand the issue
Install the hotfix at http://support.microsoft.com/kb/936594
Upgrade your network adapter drivers to the most recent version
Disable the SNP features by setting the following registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]

EnableTCPChimney=dword:00000000

EnableTCPA=dword:00000000

EnableRSS=dword:00000000

UPDATE:

I've just had another case where SNP was the culprit. The customer had clustered Exchange 2003 boxes with recently applied SP2. They were using backup software which was using CDOEXM to get information about the environment. These calls were failing with 0x8000FFFF (E_UNEXPECTED) but after debugging we saw that RPC was getting RPC_S_SERVER_UNAVAILABLE (1722 0x6ba) errors. This prompted me to ask about SNP. Disabling SNP via the steps above resolved the issue.

UPDATE (1-8-07):

Yet again, another customer with clustered Exchange 2007 boxes. They had a MAPI application which was unable to OpenMsgStore on mailboxes that had migrated from Exchange 2003 (MAPI_E_FAILONEPROVIDER). Debugging the process I could see RPC_S_SERVER_UNAVAILABLE errors. Sure enough, they had SP2 installed. Disabling SNP resolved the issue. The reason nonmigrated mailboxes worked was that MAPI was just guessing the ServerDN since resolving against the DS didn't work. If the mailbox had never been migrated, this guess ended up being correct; for migrated mailboxes, it was wrong and the failure bubbled up.

New Blogger on the Team: DVESPA

Patrick Creehan — Thu, 22 Mar 2007 22:11:00 GMT

So my boy, Dave Vespa, has just posted his first real blog post: a good one, regarding using 32-bit MAPI on Exchange 2007 (64-bit).

http://blogs.msdn.com/dvespa/archive/2007/03/22/accessing-exchange-s-extended-mapi-from-a-64-bit-process.aspx

Dave is our resident COM / Interop (among other things) guy. He's had plenty of good ideas for blog posts in the past and he'll be a good one to follow.

S/MIME Magic

Patrick Creehan — Thu, 28 Sep 2006 04:18:00 GMT

I know it's been a while since I posted (for all my loyal fans!) but my wife and I had a baby last month, so we've been busy with him. Jacob Matthew is a great kid though, so I'm not apologizing, just giving you an explanation! On to business...

So recently I had a case where the customer was using the Outlook Object Model to parse messages in his Inbox. That's a pretty normal thing do I thought. His problem that was on certain S/MIME (Secure MIME) messages, the Outlook Object Model bombed. Upon investigation it turned out that it was when he was accessing the Body property that the exception was thrown. So like any other Developer Support Engineer in the Messaging and Collaboration team, I cracked that message open with our favorite tool, MFCMapi, written by my colleague, the renowned Stephen Griffin. Not only is the guy a MAPI genious, he also seems to remember every TV show or movie he has ever seen and who was in it.

Back to the story...

So there I was knee-deep in MFCMAPI and I'm looking at this message. I have some other S/MIME messages in my mailbox so I can compare them. So as I'm looking at the PR_ props of my problem message and I notice that I don't see the PR_HTML property...in a flutter of panic, I look for the PR_BODY property...nothing. Well, that's ok, often times messages get stuffed into the PR_RTF_COMPRESSED property. And that would be fine and good, except that that property was missing also. DUN DUN DUUUUNNNN...

So where is the message? I can see it in Outlook, but it doesn't appear to be in any properties in the MAPI message.

Then it hit me...this message carries a copy of itself in a hidden attachment called smime.p7m. This helps ensure that the message content does not change since whoever signed the message originally originally signed the message. I opened up the attachment and sure enough, the content of the message was in a property called PR_ATTACHMENT_DATA_OBJ. Well that clears up one mystery. This doesn't necessarily explain why the Outlook Object Model throws an exception.

Well, this story doesn't have a really happy ending. I'm not sure why the Body property fails...I know that it happens in source when the HTML stream is being copied into the Text stream, but that's all I know. What I do know is that the HTMLBody works. So if you find an exception gets thrown when you are trying to access the Body property on a message object, try hitting the HTMLBody property as a backup and strip out the HTML tags yourself using regular expressions or something like that.