As discussed in Chapter 1 of this captivating series, MAPI contains an interface to allow developers to create and read TNEF data. This interface is the ITnef interface. There are only a few methods in this interface and they are, for the most part, self explanatory. The entire process of creating a TNEF stream can be done in just a few steps:

1. Call OpenTnefStreamEx to get a TNEF stream to write into.
   
   Make sure you pass in TNEF_ENCODE since you’ll be creating TNEF. If you were reading TNEF, you’d pass in TNEF_DECODE instead. The other flags to worry about here are TNEF_BEST_DATA, TNEF_COMPATIBILITY, and TNEF_PURE. All of these just signal to MAPI how you want the properties you add to the TNEF stream treated. They will either be all converted to the old-school attributes (TNEF_COMPATIBILITY) – you shouldn’t use this one; some will be converted to attributes but also written to the attMAPIProps section (TNEF_BEST_DATA); or they will all just be written to the MAPI props and none of them written to the attributes (TNEF_PURE).
2. Call EncodeRecips and pass in the Recipient table you get from a call to IMessage::GetRecipientTable on your message.
3. Call AddProps passing in an SPropTagArray of non-transmittable prop tags and use the TNEF_PROP_EXCLUDE flag.
   
   There are essentially two schools of thought for building your TNEF: exclude the props you don’t want and let MAPI deal with the rest; or choose carefully which props you do want to include and add them each piecemeal. These are the reason for having the TNEF_PROP_EXCLUDE and TNEF_PROP_INCLUDE flags. One of them says here are the properties I don’t want you to encode (TNEF_PROP_EXCLUDE) and the other, TNEF_PROP_INCLUDE, says I want you to include all of these properties.
   
   There’s another method, SetProps, which does just that, sets the value of a property in the TNEF stream to a value you supply. This allows you to modify the data of the message you are trying to encode, or add additional properties that weren’t on the original.
   
   Back to AddProps for a moment. The flags that supports don’t stop with TNEF_PROP_INCLUDE and TNEF_PROP_EXCLUDE, There is also TNEF_PROP_ATTACHMENTS_ONLY which says “of the properties I’ve given you to work with, I only want you to include/exclude the ones that have to do with attachments. Contrast that with TNEF_PROP_MESSAGE_ONLY which says "”of the properties I’ve given you to work with, I only want you to include/exclude the ones that have to do with the message itself – not attachments.” Then there’s the CONTAINED flags: TNEF_PROP_CONTAINED and TNEF_PROP_CONTAINED_TNEF. TNEF_PROP_CONTAINED means that these properties are going on an attachment; and the TNEF_PROP_CONTAINED_TNEF means I have TNEF data I’m going to give you to put in an attachment – like if you already had a TNEF blob you wanted to include as an attachment, which I’ll demonstrate below.
4. Once you get all your properties added and included/excluded properly, you call Finish and you’re done. One thing that makes it a little complicated though, is that you have to keep alive all the pointers and streams, etc you’re using in your TNEF until Finish is called, because that’s when all the internal work is actually done. So when you call Finish, that’s when MAPI says, Oh, ok, let me go get that recipient table you gave me. If you’ve already released it, then the process fails.

So it’s pretty easy to do this. This is essentially the way that MFCMAPI demonstrates how to do it (look in File.cpp under SaveToTNEF). There are problems associated with doing it this way when it comes to Unicode properties and when having multiple embedded messages.

The more complicated way to do this to work around some of the issues described above is to add the properties you want explicitly, including each attachment.

The basic difference in the strategy is that instead of calling AddProps with TNEF_PROP_EXCLUDE, call it with TNEF_PROP_INCLUDE and give it the SPropTagArray you get from a call to GetPropList on the message. You’ll need to filter out the non-transmittable properties (such as custom props and things like the Store EntryID). Once you add all the message props, call GetAttachmentTable and loop through each attachment and do one of two things, if it’s not an embedded message, just add the attachment data with AddProps on PR_ATTACH_DATA_BIN; otherwise, you’re going to recurse over yourself and build a TNEF stream from the embedded message. When you call Finish on it, then you’ll add it to the parent TNEF stream by calling AddProps with PR_ATTACH_DATA_OBJ and using the TNEF_PROP_CONTAINED_TNEF flag and give it the stream for your TNEF blob. Once you unwind all the way, you’ll have your “master” TNEF stream. Essentially, you’ll follow the steps here: http://msdn.microsoft.com/en-us/library/cc839833.aspx

I’ve worked quite a few cases recently regarding problems some folks have had either reading or composing TNEF content. I’ve learned quite a bit myself as a result, and I thought I’d share. I decided I would do a series of blog posts on the topic and hopefully save some of you the time I spent learning all this.

So, being the first post on the topic, I suppose now would be a good time for a review on just what TNEF is and how it’s structured.

TNEF stands for Transport-Neutral Encapsulation Format. If you use Outlook or any other MAPI client as your mail client, you may know that MAPI is a protocol for communication between the client and the mailbox server. MAPI defines a set of interfaces which the client can use to work with the data in the mailbox. The MAPI structure for the data is hierarchical with messages being contained in containers, which themselves can have a parent container, all the way up to the root of the store. MAPI also defines a set of properties understood by the client and, in some cases, the server. If all mail sent could stay on this one server and only go between clients on this one system, this would be all we need; but we know that’s not the case. A very large quantity of e-mail is sent over the internet to foreign systems every day. The vast majority of those use an industry-standard protocol called SMTP (Simple Mail Transfer Protocol) to send messages in an industry-standard format called MIME (Multipurpose Internet Mail Extensions). MIME is composed of body parts, which can in turn be composed of additional body parts themselves. MIME also allows you to add headers to each of the body parts which allow you to describe the content of that body part. So one body part may be a Word Document attachment, so the MIME headers on that body part would contain the MIME type such as application/doc and the transfer type, such as base64. The content of that body part would then contain a base64 encoding of the document. The headers for the root body part contain information such as the subject of the message, the sender and recipient information, etc.

As a message makes its way through transport from one person’s email client to another person’s, it encounters many “hops” (brief stops at SMTP servers in the routing path) which have the opportunity to modify the headers. They do this in order to track the path the message took or to flag it as SPAM, verify the sender address, etc. So there’s a chance the headers you specify when you send the message won’t be the same as when the message arrives at the destination. The headers also only support text values. One of the problems discovered early on about the MIME format was that it has no concept of “rich text.”

In early versions of Outlook, users wanted the ability to send and receive email that contained rich text bodies. Microsoft devised a plan to create an attachment to the messages it was sending that would have a certain content-type and would come to have a well-known name, “winmail.dat”. This attachment would contain an encapsulation of the MAPI properties that could represent this rich body that would work across any transport and be readable by any system that supported MIME.

The original structure just supported a very simple structure that was basically Name/size/value. These were called “attributes” and the names of these attributes are still prefixed by “att.” Many of the attribute names can be seen here: http://msdn.microsoft.com/en-us/library/cc765736.aspx. The most important of these attributes for the purposes of our discussion will be the attMAPIProps. This attribute contains a list of MAPI properties that the receiving system can set on the message once it has converted the other MIME parts into their MAPI format. Some of the TNEF attributes can be directly translated into MAPI properties as defined by the link earlier in this paragraph, but there is not a 1-1 mapping between TNEF attributes and MAPI properties – hence the attMAPIProps attribute. Attachments and recipient data can also be encoded into the TNEF structure, which we’ll examine more later.

MSDN documents the general structure of TNEF but it’s hard to understand. Last year, when Exchange decided to be among those systems that elected to publicly document their protocols, they created [MS-OXTNEF].pdf, which documents very clearly the structure of the TNEF data and how to parse it. Don’t get too nervous, though. I have parsed a 3MB TNEF blob manually myself, but in Exchange 2007, we provide managed code interfaces to allow you to read (or write) this data very easily. In subsequent posts, I’ll dive more into the structure and into the managed classes, as well as the legacy MAPI ITnef interfaces, and more into problems you may experience in developing TNEF-enabled applications.

After a hiccup earlier this week in getting the new MAPI & CDO downloads published to microsoft.com, they released today.

MAPI / CDO

http://www.microsoft.com/downloads/details.aspx?familyid=E17E7F31-079A-43A9-BFF2-0A110307611E&displaylang=en

CDO 1.21

http://www.microsoft.com/downloads/details.aspx?familyid=2714320D-C997-4DE1-986F-24F081725D36&displaylang=en

So, the Exchange team decided they didn't want to maintain the MAPI documentation anymore since they don't ship MAPI anymore. So the Outlook team stepped up and took over the docs. As such, you can now find them under the Outlook branch in the MSDN left-nav.

http://msdn.microsoft.com/en-us/library/cc765775.aspx

Steve Griffin has been working with the Outlook team on prettying them up and updating them for a few months, and they're now better than ever! He even got a new icon for MFCMAPI out of the deal!

Several million of you have downloaded the Live Search Maps Add-in for Outlook which allows integration in Outlook with maps and has some cool functionality around extending your appointment blocks to account for automatically calculated travel time among other things.

We have received a large number of support cases that are caused either directly or indirectly because of this add-in. These include hangs, crashes, and leaks. There could be any number of different reasons for those, but to name one culprit, the add-in interops with CDO 1.21, which if you are a messaging developer following the blogs of anyone on our team, you will know that this is not supported.

http://support.microsoft.com/kb/266353

Worse than that, it is distributing CDO.dll, which is also not supported. http://support.microsoft.com/kb/171440

A customer of mine reported an issue that lead us to another discovery. The add-in installs a custom form to your Personal forms registry and changes the default form for the calendar to IPM.Appointment.Location. The form it publishes is based on the LEO.oft form that it installs in your Program Files. The OFT was clearly customized from an existing item in the designer’s mailbox. The form contains a value for PR_START_DATE and PR_END_DATE set to 4/6/2006. The problem with this is that Outlook doesn’t use PR_START_DATE and PR_END_DATE, but CDO 1.21 does and OWA will set them as well. Outlook uses custom named props to keep track of dates.  Exchange normally keeps these custom props and PR_START/END_DATE in sync, but only if it needs to because of a modification you did from OWA or something like that. Otherwise, the original values will stick.

If you have a CDO 1.21 application which filters based on a date, all the appointments you created with this form in your calendar will appear to be on 4/6/2006.

The long-term plan for what to do about all the problems in this add-in has not been determined at the time of writing of this blog, but it may result in the download being removed from microsoft.com. This won’t help you fix up any items that already exist in your calendar though – nor will it prevent users from using the add-in if they already have it downloaded and installed.

If you insist on continuing to use the add-in, then please at least use this updated form.

You will need to publish this form to your Personal Forms library over the original. If you uncomfortable with that, you can

1. Uninstall the add-in using Add/Remove Programs.
2. Remove the existing form from your Personal forms library,
3. Close Outlook and ensure no instances of Outlook.exe are running in Task Manager.
4. Reinstall the add-in,
5. Replace the LEO.oft in the Program Files folder (C:\Program Files\Live Search Maps for Outlook) with the one contained in the ZIP file linked above.
6. Open Outlook. (the add-in will publish your new form).

This should allow your CDO 1.21 code to execute normally on any newly created items. For existing items, you can use Outlook Object Model or CDO 1.21 (or Extended MAPI) code to loop through the appointments in your calendar and delete the PR_START_DATE and PR_END_DATE properties if they are on 4/6/2006 with a message class of IPM.Appointment.Location. Outlook should still be able to work with the appointment just fine without those properties.

Misha, a Senior Dev on the VSTO team just posted this blog describing why handling events in managed code can be problematic. This is not news to our team, but he provides a good explanation of why it’s problematic.

Misha Shneerson : COM Interop: Handling events has side effects

If any of what he says sounds familiar, it’s because our own Matt Stehle has been talking about this for a while.

The Exchange team is looking for some feedback on business scenarios that will be impacted by removing store sinks from the code. With Exchange 2007 and beyond, the new technology designed to replace store sinks is EWS Notifications and Transport Agents. However, with the removal of the synchronous store sinks, OnSyncSave and OnSyncDelete, there will be no means of synchronous notification of changes on the store. If you have a business scenario that makes use of synchronous store sinks, hop over to Matt’s blog post and give him some feedback on this.

We’ve had a lot of folks calling in recently about this one. The symptoms are that if you go to your calendar in Outlook and forward a meeting, the To field is prepopulated with all attendees of the meeting and the Subject field is not prefixed with “FW:.”  The common denominator of all victims of this problem is that they all had their meetings processed with CDO 1.21. This is common if you are a Blackberry user, but could happen if any application processes your appointment with CDO 1.21 before you’ve had a chance to touch it with Outlook.

The real cause of the problem is that CDO 1.21 doesn’t clear the MSGFLAG_UNSENT flag from the PR_MESSAGE_FLAGS property when it does a Respond. Because this flag is still set on the appointment item in the calendar, Outlook (rightfully) thinks the message has not yet been sent and so it treats it like a new message – essentially.

I’ve requested a fix for this issue and it is currently being worked on by our development team. I’ll update this post if/when the fix becomes available.

[UPDATE| The fix is now available and accessible here: http://support.microsoft.com/default.aspx?scid=kb;EN-US;957020.

If you wrote a Transport Agent for Exchange 2007 RTM which was accessing the raw MIME message which doesn’t seem to work after installing SP1, there’s a good reason for that. In SP1, we moved content conversion to after the OnRoutedMessage event. This leaves you in the predicament of not having access to raw MIME content in any Transport Agent even for messages bound for external recipients. The change was by design to enhance some pipeline tracing features.

While you don’t have access to raw MIME anymore, you can use the TnefReader class and the MailItem object you’re given to access message data and do your business logic there.

Your other option is to set up a Send Connector to route outbound mail through an SMTP server and create an OnArrival SMTP sink. You should be able to get raw MIME there.

We get calls about this all the time. You have a CDOSYS application that…what? No, I’m not talking about automobiles, why did you…oh, the title. Well, it’s true. Read on.

The Dreaded Transport Error

The problem we see more than any other is when a customer is trying to send a lot of e-mail using an SMTP API such as CDOEX, CDOSYS, or System.Web.Mail, they will start to get CDO_E_FAILED_TO_CONNECT errors (0x80040213) (-2147220973) which is “The transport could not connect to the server.”

The Resolution

We see it over and over again. This error is because you are sending such a high volume of email over one port, typically the default port (25), that you flood the port with traffic until your app can no longer talk to the server. If your application is sending such a large volume of mail so quickly that it generates this error, it’s time to install a local SMTP service (if one is not already installed) and start sending via port.

Setting the Send and Post Method

Sending or Posting using the Pickup Directory

How To Send HTML Formatted Mail Using CDO for Windows 2000 and the Local Pickup Directory

You can employ the same strategy in the article above to your System.Web.Mail emails:

How to send e-mail messages by using CDOSYS and the local SMTP pickup directory with VB.NET

How to use the Cdosys.dll library to send an e-mail message with attachments by using Visual C# (this also shows how to send using pickup)

Benefits of using the pickup folder

Performance

It is very easy to generate a text file on the file system. When your CDOSYS configuration is set to send via pickup, all CDOSYS is doing in your application is creating an .eml file in a specific directory. You’ve just removed the entire networking stack from your application. You can whip through email generation in a fraction of the time it takes to communicate over a TCP port.

Reliability

When you let the SMTP service do the sending of your emails, you are working on a far more stable architecture than for you to handle error conditions yourself in your application. IIS’s SMTP service makes it easy when there is a problem as well by just moving your .eml file to the BadMail folder. If the service runs into a problem where transport is too busy to take new connections, it just tries again later. Email is asynchronous by nature, so this shouldn’t be a big deal.

Resiliency

The SMTP service will just continually monitor the Pickup folder for any files in it and will try to send them. If there’s a power failure for example, or your machine crashes, when it reboots, the files are still there so the service will once again work on sending them.

When You Shouldn’t Use Pickup

If it is impossible for you to install a local SMTP service for whatever reason, sending via port may be your only option. For example, if you are writing an application to run on a desktop running Windows XP/Vista, it doesn’t make sense to install an SMTP service on a desktop platform. Additionally, if your application only sends an email occasionally, it will probably be fine to send using port instead of pickup – you’ll probably never run into this error. If you application does need to know immediately whether or not the email is going to succeed in each command in the protocol (MAIL FROM, RCPTTO, etc) then you may need to send via port as well. In those cases, if you will be sending a large volume of email very quickly, you should throttle your application to allow some time between each mail so that you don’t clog the pipes and end up with the dreaded transport error. This is, of course, assuming that you can’t install SMTP services locally.

Another reason you may not code your application to use a pickup folder is that you don’t know the environment your application will be deployed on. For example, you’ve written a web application that sends email notifications when some data has changed and you’re going to sell this application for companies to deploy in their own environments. You don’t know, nor can you count on an SMTP service running locally on the box. Your first instinct may be to code your application to send via port. Certainly, they’ll have an SMTP service somewhere in the organization and it’s easy for them to just type in a setting somewhere “mail.company.com” – folks are used to it by now. My plea to you is to account for both possibilities. Provide a mechanism for sending via pickup as well – you won’t regret it.

I had a case on this a few months ago, but thought more folks might run into this as they start moving to form regions. By default, if you use the VSTO template for creating a form region in Outlook, the item is leaked. This can show up in a number of different symptoms, but the one we saw was that when the item was created (then saved and closed), then modified in WebDav, then they tried to modify the item again in Outlook, the user would get the error:

"The item cannot be saved because it was changed by another user or in a another window."

Fortunately, the resolution is easy, you just need to add a call to Marshal.ReleaseComObject to the FormRegionInitializing event in the designer code.

The folks at CTS are asking us to post the following survey to our blogs. You only need to respond once, so if you see this on another blog you read, it's the same thing.

<survey_text>

Greetings Blog Readers,

My name is Ed Jolly, and I am a director in the Commercial Technical Support (CTS) organization at Microsoft. I am here to request a few minutes of your time.

We would like to learn more about blog readership through a brief survey. This is an opportunity for us to better understand what is valuable to you and what you would like to see in the future.

Below is a link that will take you to another website to complete the survey. Based on what we learn, we may request more feedback in future surveys like this.  When you open the survey, you will see a list of blogs that CTS engineers contribute to across many different products. We have not posted a listing of these blogs in the past, and I hope it helps you find other blogs that are helpful to you.

The blog survey is completely anonymous.

• Location: [Link removed since survey has expired]
• Availability:  Until August 22. You may receive a request to complete this survey through multiple RSS feeds. You need only to complete it one time.
• Length: The survey can be a maximum of 11 questions.
• Time:  Less than 5 minutes (but providing more information in the open text fields may take a minute or two extra, improving our ability to understand your needs in these blogs).

Thank you in advance for your time, participation and assistance.

Ed Jolly (edjolly@microsoft.com)

</survey text>

We’ve seen a bunch of folks asking recently if their Exchange 2007 CAS server can talk to their Exchange 2003 mailbox server. The answer is <gasp> it depends on what you mean.

It depends on what technology you are referring to and what you mean by “talk.” Are you referring to WebDAV calls? Then yes. Are you referring to OWA? Then sorta. Are you referring to Exchange Web Services (EWS), then no. There are really two things at play here: proxy and redirection. That is, can an Exchange 2007 CAS server serve as a proxy for calls to the backend mailbox server, or can it successfully redirect calls to the mailbox server where the call can be handled.

In Exchange 2003, HTTP calls were simply forwarded on from the front-end to the back-end. The back-end server would receive the HTTP request and handle it and then forward the response back up through the front-end. In Exchange 2007, however, the front-end CAS handles the HTTP request and speaks RPC to the back-end server. 

When a request is received by a CAS server, it knows how to figure out who should handle the request – whether it needs proxied to a different CAS, or can be sent directly to an Exchange 2003 or 2007 back-end. For example, when a request is received for the /exchange virtual directory on a mailbox on an Exchange 2003 server, the HTTP request is forwarded to the appropriate server. So in this way, OWA still works from your 2007 CAS – even though the client will see the OWA 2003 interface. The same is true for activesync clients requesting the \Microsoft-Server-ActiveSync vdir.

If an Exchange Web Services call hits an Exchange 2007 CAS and the CAS is able to determine that the mailbox is on an Exchange 2003 back-end the call will fail since proxying cannot be performed.

Here’s more on the topic:

Overview of Exchange Server 2007 CAS Proxying and Redirection

I’ve had a couple cases recently where customers were having problems getting into deadlock with one of our modules, usually MAPI or Outlook. There’s one common thread that connects these cases: message pumps.

Your application has to make a remote call to a server object living in a different process or thread. Often times this is referred to as an “asynchronous” call. That term is somewhat misleading – suggesting your process could actually forget that it made the call and then be surprised when it comes back. In actuality, there’s always a listener constantly waiting for the call to return. What makes it seem asynchronous is that the waiting happens on a different thread. In order to wait for your call return, applications often implement logic called a “message pump.” This just processes incoming Windows messages or RPC calls and dispatches them to be processed; and it keeps processing messages until your call returns.

The problem with this is that there’s no easy way to tell what these incoming messages are or what they’ll do. This is what gets you into trouble.

Let’s say that a normal code path in one thread will go something like this:

• Outlook does some work
• Outlook locks a critical section (CS1)
• Outlook calls into you
• You do some work
• You get a lock on a critical section (CS2)

There’s nothing wrong with this at all. Everything is fine at this point. Now lets suppose there’s a different thread that does something like this

• You do some work
• You get a lock on a critical section (CS2).
• You make an RPC call or otherwise pump messages.

Now, up to this point there’s nothing wrong. Suppose, however, that one of the pending messages is from Outlook and the execution goes something like this:

• …You make an RPC call…
• The message pump dispatches the Outlook call
• Outlook does some work
• Outlook locks a critical section (CS1).

Again, nothing is inherently wrong with this execution path by itself. However, when you combine both threads, you have critical sections being locked in reverse order – classic deadlock.

Here’s a way to understand the problem: Let’s say you’re at home watching a Virginia Tech football game (There are other teams? Oh yeah, I just call them “losers.”) with a friend. You both get a hankering for a cold adult beverage. You go to the fridge, but there’s only one left. You decide to split it with your buddy, which is ok, since it’s a 40. Meanwhile, your other friends arrive with their own adult beverages and ask if anyone has an opener. Your buddy gets up and gets the bottle opener.  You are thinking, “OK, I have the bottle, now all I need is the bottle opener, but my buddy has it so I’ll just wait until he’s done.” You both return to your respective recliners. Your buddy is thinking, “ok, I have the opener, I guess I’ll just start opening bottles with it until my glass gets filled. Surely one of the bottles will be mine.” As he’s opening your other friends’ bottles, the thought enters his mind – my friend [you] has a bottle he needs opened – I need to get his bottle to open it for him. Since both yours and your buddy’s rear ends are plastered to their respective recliners, you have now entered adult beverage deadlock.

Your buddy was keeping the bottle opener all to himself while waiting for his glass to be filled. Little did he understand that by holding the bottle opener to himself, he was preventing that from happening, since the bottle was already being held firmly in your grip. Had he relinquished control of the bottle opener, you could have easily picked it up and used it. He didn’t know that one of the bottles he had to open would be yours.

So how do we prevent this problem from occurring in the first place? The key is really that your buddy didn’t know what bottles he would be opening, while waiting for his glass to be filled. He didn’t know that you were planning to split the beverage with him. What he could have done is just told your friends that the bottle opener was in the drawer instead of holding onto the bottle opener while waiting for his glass to be filled. That way, when you needed it, it would be available and as long as everyone puts it back when they’re done with it, though you may need to wait for a short time, no one will go thirsty for long.

The moral of the story is that you shouldn’t make it a practice to hold onto a lock while making an RPC call or otherwise pumping messages because you don’t know what messages will arrive in your pump and what locks they will request.

Secondary moral of the story: always make sure your adult beverage supply is properly maintained.

Go Hokies!

Check this out:

Exchange Server 2007 Legacy API Property Mapping

This helps you compare WebDAV/CDOEX props to EWS properties. Also discussed are CDOEXM and WMI replacements and the look ahead.