Wiz/dumb : MIME

Fixing the Charset on MIME Messages When Using ADODB.Stream.ReadText

Patrick Creehan — Thu, 29 Oct 2009 20:00:00 GMT

We had an issue recently where the customer had written an application to process MIME messages that had body parts encoded with the koi8-r charset…or at least that’s what it said:

------_=_NextPart_001_01C9F067.D9742103
Content-Type: text/plain;
    charset="koi8-r"
Content-Transfer-Encoding: quoted-printable

If you use code similar to the following, the resulting characters returned were not correct.

private string ReadBodyPart()

{

    CDO.IDataSource oIDsrc;

    ADODB.Stream objBpStream = new ADODB.Stream();

    CDO.Message objCDOMsg = new CDO.Message();

    CDO.IBodyPart oIBodyPart;

    objBpStream.Open(vtMissing, 

        ADODB.ConnectModeEnum.adModeUnknown,

        ADODB.StreamOpenOptionsEnum.adOpenStreamUnspecified, 

        string.Empty, 

        string.Empty);

    objBpStream.LoadFromFile(EML_FILE_PATH);           

    oIDsrc = (CDO.IDataSource)objCDOMsg;

    oIDsrc.OpenObject(objBpStream, "_Stream");

    objBpStream.Close();

    oIBodyPart = (CDO.IBodyPart)objCDOMsg.TextBodyPart;

    objBpStream = (ADODB.Stream)oIBodyPart.GetDecodedContentStream();

    return objBpStream.ReadText(-1);

}

The problem is that koi8-r is the body name for both “windows-1251” (codepage 1251) and “koi8-r” (codepage 20866). I took a look at what the TextBody and HtmlBody properties were doing because they can return the characters correctly. Turns out, when the message is loaded, we make an API call to GetCharsetInfo and pass in the value defined in the header. This gives us back a structure that contains the uiCodePage used by TextBody and HtmlBody later.

Fortunately, the .net framework provides methods to do this as well. I simply leverage them to correct the charset I’m given.

public static string GetCharSetName(string bodyName)

{

    Encoding encoding = Encoding.GetEncoding(bodyName);

    if (encoding != null)

    {

        int iCP = encoding.WindowsCodePage;

        return Encoding.GetEncoding(iCP).WebName;

    }

    return bodyName;

}

Then I just insert a call to that function above my call to ReadText and we’re good.

private string ReadBodyPart()

{

    CDO.IDataSource oIDsrc;

    ADODB.Stream objBpStream = new ADODB.Stream();

    CDO.Message objCDOMsg = new CDO.Message();

    CDO.IBodyPart oIBodyPart;

    objBpStream.Open(vtMissing, 

        ADODB.ConnectModeEnum.adModeUnknown,

        ADODB.StreamOpenOptionsEnum.adOpenStreamUnspecified, 

        string.Empty, 

        string.Empty);

    objBpStream.LoadFromFile(EML_FILE_PATH);           

    oIDsrc = (CDO.IDataSource)objCDOMsg;

    oIDsrc.OpenObject(objBpStream, "_Stream");

    objBpStream.Close();

    oIBodyPart = (CDO.IBodyPart)objCDOMsg.TextBodyPart;

    objBpStream = (ADODB.Stream)oIBodyPart.GetDecodedContentStream();

    objBpStream.Charset = GetCharSetName(objBpStream.Charset);

    return objBpStream.ReadText(-1);

}

MimeContent vs Mime Content

Patrick Creehan — Wed, 16 Sep 2009 22:46:31 GMT

Recently, I helped a customer with an issue where they were wondering if it was essential to set the ToRecipients, CcRecipients, and BccRecipients if they were setting the same values in the MimeContent property when doing a CreateItem call in Exchange Web Services. While trying to test out the scenarios, I thought “maybe it depends on the order of the elements in your <t:Message>. I soon realized that the order of the properties in the <t:Message> node seemed to matter. As it turns out, MessageType inherits from ItemType and MimeContent is defined on the ItemType where the recipient properties are defined on the MessageType and both of these are xs:sequence types. This means that their elements must appear in a specific order. Any deviation from the prescribed sequence results in a schema validation error returned from EWS. That meant that the MimeContent property itself always had to appear before the recipient properties since MimeContent is the first element described in the ItemType.

Here’s the schema for the ItemType and you can see that MimeContent is first.

  <!-- Core contents of an item. -->
 <xs:complexType name="ItemType">
   <xs:sequence>
     <xs:element name="MimeContent" type="t:MimeContentType" minOccurs="0" />
     <xs:element name="ItemId" type="t:ItemIdType" minOccurs="0" />
     <xs:element name="ParentFolderId" type="t:FolderIdType" minOccurs="0"/>
     <xs:element name="ItemClass" type="t:ItemClassType" minOccurs="0" />
     <xs:element name="Subject" type="xs:string" minOccurs="0" />
     <xs:element name="Sensitivity" type="t:SensitivityChoicesType" minOccurs="0" />
     <xs:element name="Body" type="t:BodyType" minOccurs="0" />
     <xs:element name="Attachments" type="t:NonEmptyArrayOfAttachmentsType" minOccurs="0" />
     <xs:element name="DateTimeReceived" type="xs:dateTime" minOccurs="0" />
     <xs:element name="Size" type="xs:int" minOccurs="0" />
     <xs:element name="Categories" type="t:ArrayOfStringsType" minOccurs="0" />
     <xs:element name="Importance" type="t:ImportanceChoicesType" minOccurs="0" />
     <xs:element name="InReplyTo" type="xs:string" minOccurs="0" />
     <xs:element name="IsSubmitted" type="xs:boolean" minOccurs="0" />
     <xs:element name="IsDraft" type="xs:boolean" minOccurs="0" />
     <xs:element name="IsFromMe" type="xs:boolean" minOccurs="0" />
     <xs:element name="IsResend" type="xs:boolean" minOccurs="0" />
     <xs:element name="IsUnmodified" type="xs:boolean" minOccurs="0" />
     <xs:element name="InternetMessageHeaders" type="t:NonEmptyArrayOfInternetHeadersType" minOccurs="0" />
     <xs:element name="DateTimeSent" type="xs:dateTime" minOccurs="0" />
     <xs:element name="DateTimeCreated" type="xs:dateTime" minOccurs="0" />
     <xs:element name="ResponseObjects" type="t:NonEmptyArrayOfResponseObjectsType" minOccurs="0" />
     <xs:element name="ReminderDueBy" type="xs:dateTime" minOccurs="0" />
     <xs:element name="ReminderIsSet" type="xs:boolean" minOccurs="0" />
     <xs:element name="ReminderMinutesBeforeStart" type="t:ReminderMinutesBeforeStartType" minOccurs="0" />
     <xs:element name="DisplayCc" type="xs:string" minOccurs="0" />
     <xs:element name="DisplayTo" type="xs:string" minOccurs="0" />
     <xs:element name="HasAttachments" type="xs:boolean" minOccurs="0"/>
     <xs:element name="ExtendedProperty" type="t:ExtendedPropertyType" minOccurs="0" maxOccurs="unbounded" />
     <xs:element name="Culture" type="xs:language" minOccurs="0"/>
     <xs:element name="EffectiveRights" type="t:EffectiveRightsType" minOccurs="0" />
     <xs:element name="LastModifiedName" type="xs:string" minOccurs="0" />
     <xs:element name="LastModifiedTime" type="xs:dateTime" minOccurs="0" />
   </xs:sequence>
 </xs:complexType>

That’s followed by the MessageType schema that is an extension of ItemType. This is where the recipient properties appear.

  <!-- - - - - - - - - - - - - - - - - - - -->
 <!--   Message type: derived from item   -->
 <!-- - - - - - - - - - - - - - - - - - - -->
 <xs:complexType name="MessageType">
   <xs:complexContent>
     <xs:extension base="t:ItemType">
       <xs:sequence>
         <xs:element name="Sender" minOccurs="0" type="t:SingleRecipientType" />
         <xs:element name="ToRecipients" type="t:ArrayOfRecipientsType" minOccurs="0" />
         <xs:element name="CcRecipients" type="t:ArrayOfRecipientsType" minOccurs="0" />
         <xs:element name="BccRecipients" type="t:ArrayOfRecipientsType" minOccurs="0" />
         <xs:element name="IsReadReceiptRequested" type="xs:boolean" minOccurs="0" />
         <xs:element name="IsDeliveryReceiptRequested" type="xs:boolean" minOccurs="0" />
         <xs:element name="ConversationIndex" type="xs:base64Binary" minOccurs="0" />
         <xs:element name="ConversationTopic" type="xs:string" minOccurs="0" />
         <xs:element name="From" type="t:SingleRecipientType" minOccurs="0" />
         <xs:element name="InternetMessageId" type="xs:string" minOccurs="0" />
         <xs:element name="IsRead" type="xs:boolean" minOccurs="0" />
         <xs:element name="IsResponseRequested" type="xs:boolean" minOccurs="0" />
         <xs:element name="References" type="xs:string" minOccurs="0" />
         <xs:element name="ReplyTo" type="t:ArrayOfRecipientsType" minOccurs="0" />
         <xs:element name="ReceivedBy" type="t:SingleRecipientType" minOccurs="0" />
         <xs:element name="ReceivedRepresenting" type="t:SingleRecipientType" minOccurs="0" />
       </xs:sequence>
     </xs:extension>
   </xs:complexContent>
 </xs:complexType>

If you think about how the server will process this XML SOAP request, it will get the MimeContent first and then the first-class properties after that. So the server will build out its entire MIME message based on the content of the MimeContent property and then override those values with the first-class properties that follow. The same holds true for other properties like Body, Subject, etc.

After I figured all this out, I found this already clearly explained in Inside Microsoft Exchange Server 2007 Web Services explaining this same phenomenon when attempting to set the subject:

Well, during a CreateItem call, properties are processed on a first-come, first-served basis. You first create an empty message, set the contents from the MIME, and then set the explicit subject. This really isn’t a problem in practice although it has some interesting implications. The schema mandates a specific ordering of elements within item instances. In other words, trying to put the subject before the MimeContent property results in a schema validation error. Therefore, which property overwrites which is dictated by the ordering as defined in the schema. (page 176)

They later go on to say the only time you’ll have a conflict is when you try to also set the same properties using the extended properties, so avoid doing that. There’s no need to set both the first-class property and it’s equivalent extended property.

TNEF (Chapter 1): Basics

Patrick Creehan — Fri, 16 Jan 2009 19:17:36 GMT

I’ve worked quite a few cases recently regarding problems some folks have had either reading or composing TNEF content. I’ve learned quite a bit myself as a result, and I thought I’d share. I decided I would do a series of blog posts on the topic and hopefully save some of you the time I spent learning all this.

So, being the first post on the topic, I suppose now would be a good time for a review on just what TNEF is and how it’s structured.

TNEF stands for Transport-Neutral Encapsulation Format. If you use Outlook or any other MAPI client as your mail client, you may know that MAPI is a protocol for communication between the client and the mailbox server. MAPI defines a set of interfaces which the client can use to work with the data in the mailbox. The MAPI structure for the data is hierarchical with messages being contained in containers, which themselves can have a parent container, all the way up to the root of the store. MAPI also defines a set of properties understood by the client and, in some cases, the server. If all mail sent could stay on this one server and only go between clients on this one system, this would be all we need; but we know that’s not the case. A very large quantity of e-mail is sent over the internet to foreign systems every day. The vast majority of those use an industry-standard protocol called SMTP (Simple Mail Transfer Protocol) to send messages in an industry-standard format called MIME (Multipurpose Internet Mail Extensions). MIME is composed of body parts, which can in turn be composed of additional body parts themselves. MIME also allows you to add headers to each of the body parts which allow you to describe the content of that body part. So one body part may be a Word Document attachment, so the MIME headers on that body part would contain the MIME type such as application/doc and the transfer type, such as base64. The content of that body part would then contain a base64 encoding of the document. The headers for the root body part contain information such as the subject of the message, the sender and recipient information, etc.

As a message makes its way through transport from one person’s email client to another person’s, it encounters many “hops” (brief stops at SMTP servers in the routing path) which have the opportunity to modify the headers. They do this in order to track the path the message took or to flag it as SPAM, verify the sender address, etc. So there’s a chance the headers you specify when you send the message won’t be the same as when the message arrives at the destination. The headers also only support text values. One of the problems discovered early on about the MIME format was that it has no concept of “rich text.”

In early versions of Outlook, users wanted the ability to send and receive email that contained rich text bodies. Microsoft devised a plan to create an attachment to the messages it was sending that would have a certain content-type and would come to have a well-known name, “winmail.dat”. This attachment would contain an encapsulation of the MAPI properties that could represent this rich body that would work across any transport and be readable by any system that supported MIME.

The original structure just supported a very simple structure that was basically Name/size/value. These were called “attributes” and the names of these attributes are still prefixed by “att.” Many of the attribute names can be seen here: http://msdn.microsoft.com/en-us/library/cc765736.aspx. The most important of these attributes for the purposes of our discussion will be the attMAPIProps. This attribute contains a list of MAPI properties that the receiving system can set on the message once it has converted the other MIME parts into their MAPI format. Some of the TNEF attributes can be directly translated into MAPI properties as defined by the link earlier in this paragraph, but there is not a 1-1 mapping between TNEF attributes and MAPI properties – hence the attMAPIProps attribute. Attachments and recipient data can also be encoded into the TNEF structure, which we’ll examine more later.

MSDN documents the general structure of TNEF but it’s hard to understand. Last year, when Exchange decided to be among those systems that elected to publicly document their protocols, they created [MS-OXTNEF].pdf, which documents very clearly the structure of the TNEF data and how to parse it. Don’t get too nervous, though. I have parsed a 3MB TNEF blob manually myself, but in Exchange 2007, we provide managed code interfaces to allow you to read (or write) this data very easily. In subsequent posts, I’ll dive more into the structure and into the managed classes, as well as the legacy MAPI ITnef interfaces, and more into problems you may experience in developing TNEF-enabled applications.

S/MIME Magic

Patrick Creehan — Thu, 28 Sep 2006 04:18:00 GMT

I know it's been a while since I posted (for all my loyal fans!) but my wife and I had a baby last month, so we've been busy with him. Jacob Matthew is a great kid though, so I'm not apologizing, just giving you an explanation! On to business...

So recently I had a case where the customer was using the Outlook Object Model to parse messages in his Inbox. That's a pretty normal thing do I thought. His problem that was on certain S/MIME (Secure MIME) messages, the Outlook Object Model bombed. Upon investigation it turned out that it was when he was accessing the Body property that the exception was thrown. So like any other Developer Support Engineer in the Messaging and Collaboration team, I cracked that message open with our favorite tool, MFCMapi, written by my colleague, the renowned Stephen Griffin. Not only is the guy a MAPI genious, he also seems to remember every TV show or movie he has ever seen and who was in it.

Back to the story...

So there I was knee-deep in MFCMAPI and I'm looking at this message. I have some other S/MIME messages in my mailbox so I can compare them. So as I'm looking at the PR_ props of my problem message and I notice that I don't see the PR_HTML property...in a flutter of panic, I look for the PR_BODY property...nothing. Well, that's ok, often times messages get stuffed into the PR_RTF_COMPRESSED property. And that would be fine and good, except that that property was missing also. DUN DUN DUUUUNNNN...

So where is the message? I can see it in Outlook, but it doesn't appear to be in any properties in the MAPI message.

Then it hit me...this message carries a copy of itself in a hidden attachment called smime.p7m. This helps ensure that the message content does not change since whoever signed the message originally originally signed the message. I opened up the attachment and sure enough, the content of the message was in a property called PR_ATTACHMENT_DATA_OBJ. Well that clears up one mystery. This doesn't necessarily explain why the Outlook Object Model throws an exception.

Well, this story doesn't have a really happy ending. I'm not sure why the Body property fails...I know that it happens in source when the HTML stream is being copied into the Text stream, but that's all I know. What I do know is that the HTMLBody works. So if you find an exception gets thrown when you are trying to access the Body property on a message object, try hitting the HTMLBody property as a backup and strip out the HTML tags yourself using regular expressions or something like that.