mstehle: The CDOs and CDONTS of Messaging Development : Exchange Eventing

FYI: Exchange 2007 OnSyncSave Store Events and Plain Text Message Don’t Play Nice

mstehle — Fri, 06 Feb 2009 01:54:50 GMT

…Recently, I worked with a customer who was facing the problem described below. We requested a fix but ended up finding a workaround using a Transport Agent. At this time there is no plan to fix this issue in Exchange 2007…

Problem Description

In an OnSyncSave store event in Exchange 2007 every field's value of a plain text message which triggers the event returns the following exception in the first pre-commit event:

"Operation failed to complete and the status is unavailable. The field may be unavailable or the operation was not attempted."

When the OnSyncSave event fires again for this message after the message has been committed to the store the property values can be read.

NOTE: HTML and RTF messages do not have this problem - this only applies to Plain Text messages.

Workarounds

There are no straight forward workarounds, only other options which may or may not suit your application’s needs:

Use the OnSave asynchronous event sink. The trade off here is obvious but the OnSave event doesn’t have any problem reading properties from a message that triggers the event.
Use a Transport Agent. The main trade off here is that transport agents operate outside of the mailbox in the transport stack. They can’t react to events that happen within a mailbox such as changing an item in the calendar or moving a message from one folder to another. They only react to mail being delivered to or sent from a mailbox.
Use an Outlook Add-in. The big trade-off here is that changes in OWA are not handled, you can only react to events that the user triggers in Outlook.
Use rules. The main drawback is that they have a fixed set of functions but they don’t require any code to be written or bits to be deployed.

Again, depending on your solution one of these workarounds or combination of two might accomplish what you need. An example of combining two of these options would be to use a transport agent to tag messages as they are sent to a particular mailbox and then use a rule to key off this tag and move those messages to a specific folder on deliver.

HOWTO: Sample Transport Agent – Add Headers, Categories, MAPI Props, Even Uses a Fork!

mstehle — Wed, 14 Jan 2009 00:50:33 GMT

Welcome to my first in depth Exchange Transport Agent sample! If you are not familiar with Exchange Transport Agents this blog post is a great place to start. I created a sample Transport Agent for a customer a while ago that I recently added commenting to and cleaned up so it could serve as a general demo suitable for this blog. The focus of this sample is to illustrate a several techniques but no necessarily serve as a template or best practice for writing transport agents in general. The key techniques illustrated are:

Forking a message in the “OnSubmittedMessage” event to process a messages individually for each recipient.
Adding a custom header to a message.
Adding a Keywords (also known as Category) header to a message
Adding a custom named property to a MAPI message

The comments in the code speak for themselves so I won’t walk through it all. There are two very important notes regarding the Keyword header and adding MAPI properties that I want to emphasize though.

The Keywords Header is Removed By Default

If you are you trying to add the Keywords header in a transport agent or even just when sending MIME messages to Exchange 2007 keep in mind that this header is translated into Categories which are removed by default during content conversion. If you really need this functionality though, you can turn this off using the Set-TransportConfig cmdlet and setting ClearCategories to $false. Click here to read more.

If You Don’t Get TNEF, You Don’t Have TNEF

There is no API to convert MIME messages to TNEF within a transport agent. The TNEF body part must be a full representation of the message being submitted to Exchange. You can’t have a complex MIME message with a TNEF body part that simply adds a MAPI property or two. Messages that are submitted from outside the organization will mostly like not have TNEF, there is no mechanism to create a representative TNEF body part for these messages and then append properties.

One final note is that this sample logs information out to a text file on the system. Keep in mind that transport agents run in the context of the Exchange Transport Service which is typically Network Service. In this case, the Network Service account would need write permissions to whatever folder the log file is configured to write to.

Enjoy…

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using Microsoft.Exchange.Data.Transport;
using Microsoft.Exchange.Data.Transport.Routing;
using Microsoft.Exchange.Data.Common;
using Microsoft.Exchange.Data.Mime;
using Microsoft.Exchange.Data.ContentTypes.Tnef;

namespace MyRoutingAgent
{

    public sealed class MyRoutingAgentFactory : RoutingAgentFactory
    {
        public override RoutingAgent CreateAgent(SmtpServer server)
        {
            return new MyRoutingAgent();
        }
    }

    public class MyRoutingAgent : RoutingAgent
    {
        // Place this text in the subject to test the agent
        public const string TEST_TOKEN = "[TEST]";
        public const string LOG_FILE_PATH = @"C:\Users\Administrator\Desktop\MyRoutingAgent\MyRoutingAgent.log";

        public MyRoutingAgent()
        {
            this.OnRoutedMessage += new RoutedMessageEventHandler(MyRoutingAgent_OnRoutedMessage);
            this.OnSubmittedMessage += new SubmittedMessageEventHandler(MyRoutingAgent_OnSubmittedMessage);
            this.OnResolvedMessage += new ResolvedMessageEventHandler(MyRoutingAgent_OnResolvedMessage);
        }

        void MyRoutingAgent_OnSubmittedMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e)
        {
            // Don't process EVERY message...
            if (e.MailItem.Message.Subject.Contains(TEST_TOKEN))
            {
                // If we have multiple recipients, fork the message to process each recipient seperately...
                if (e.MailItem.Recipients.Count > 1)
                {
                    LogMessage("OnSubmittedMessage", e.MailItem, "Forking message with " + e.MailItem.Recipients.Count + " recipients.");

                    int count = 0;
                    while (e.MailItem.Recipients.Count > 1)
                    {
                        // Create an individual forked message for each recipient...
                        EnvelopeRecipient recip = e.MailItem.Recipients[e.MailItem.Recipients.Count - 1];
                        List<EnvelopeRecipient> recips = new List<EnvelopeRecipient>();
                        recips.Add(recip);
                        source.Fork(recips);
                        count++;
                    }

                    // This should always be one.
                    LogMessage("OnSubmittedMessage", e.MailItem, "Forked " + count + " messages.");
                }
                else
                {
                    LogMessage("OnSubmittedMessage", e.MailItem, "Single recipient, no forking needed.");
                }
            }
        }

        void MyRoutingAgent_OnResolvedMessage(ResolvedMessageEventSource source, QueuedMessageEventArgs e)
        {
            // Don't process EVERY message...
            if (e.MailItem.Message.Subject.Contains(TEST_TOKEN))
            {
                MimePart tnefPart = e.MailItem.Message.TnefPart;
                if (tnefPart != null)
                {
                    LogMessage("OnResolvedMessage", e.MailItem, "TNEF");
                }
                else
                {
                    LogMessage("OnResolvedMessage", e.MailItem, "No TNEF");
                }
            }
        }
        
        void MyRoutingAgent_OnRoutedMessage(RoutedMessageEventSource source, QueuedMessageEventArgs e)
        {
            // Don't process EVERY message...
            if (e.MailItem.Message.Subject.Contains(TEST_TOKEN))
            {
                if (e.MailItem.Recipients.Count == 1)
                {
                    LogMessage("OnRoutedMessage", e.MailItem, "Processing message sent to " + e.MailItem.Recipients[0].Address.LocalPart + ".");

                    // If we haven't already processed this message...
                    if (e.MailItem.Message.MimeDocument.RootPart.Headers.FindFirst("MyHeader") == null)
                    {
                        ProcessMailItem(e.MailItem);
                    }
                    else
                    {
                        LogMessage("OnRoutedMessage", e.MailItem, "Already processed this message, skipping.");
                    }
                }
                else
                {
                    LogMessage("OnRoutedMessage", e.MailItem, "Unexpected number of recipients, " + e.MailItem.Recipients.Count.ToString() + ".");
                }
            }
        }

        /// <summary>
        /// Add a custom header, keyword header, and MAPI
        /// named property into PS_PUBLIC_STRINGS if there
        /// is a TNEF body part.
        /// </summary>
        private void ProcessMailItem(MailItem item)
        {
            // Modify the subject of this message to ensure the
            // fork worked.  Append the first recipient name to the 
            // subject.  This should match the mailbox name the
            // message is delivered to.
            item.Message.Subject = item.Message.Subject + "| To: " + item.Recipients[0].Address.LocalPart;

            MimeDocument mdMimeDoc = item.Message.MimeDocument;
            HeaderList hlHeaderlist = mdMimeDoc.RootPart.Headers;
            MimeNode lhLasterHeader = hlHeaderlist.LastChild;

            // Add a custom header
            TextHeader nhNewHeader = new TextHeader("MyHeader", "MyHeaderValue");
            hlHeaderlist.InsertBefore(nhNewHeader, lhLasterHeader);

            // Add the Keywords header to set a Category in Outlook/OWA.
            // *** INFO ***
            // Keywords (also known as Categories) are stripped from
            // message by default.  This can be disabled using the 
            // "Set-TransportConfig –ClearCategories $false" cmdlet.
            // (http://technet.microsoft.com/en-us/library/bb124151.aspx)
            TextHeader nhNewKeywords = new TextHeader("Keywords", "1");
            hlHeaderlist.InsertBefore(nhNewKeywords, lhLasterHeader);

            MimePart tnefPart = item.Message.TnefPart;

            // Without a TNEF body part, we can't do this step.
            // There is no way to create a TNEF body part from
            // scratch if Exchange isn't giving us one.  Most
            // mail that comes from the internet won't have TNEF.
            if (tnefPart != null)
            {
                TnefReader reader = new TnefReader(tnefPart.GetContentReadStream());
                TnefWriter writer = new TnefWriter(
                    tnefPart.GetContentWriteStream(tnefPart.ContentTransferEncoding),
                    reader.AttachmentKey,
                    0,
                    TnefWriterFlags.NoStandardAttributes);

                while (reader.ReadNextAttribute())
                {
                    if (reader.AttributeTag == TnefAttributeTag.MapiProperties)
                    {
                        writer.StartAttribute(TnefAttributeTag.MapiProperties, TnefAttributeLevel.Message);
                        writer.WriteAllProperties(reader.PropertyReader);

                        int tag;
                        unchecked
                        {
                            // The first four bytes of the tag is must be at least 0x8000 
                            // when setting a named property, the second determines the type. 
                            // http://www.cdolive.com/cdo10.htm
                            tag = (int)0x8000001E;
                        }

                        Guid PS_PUBLIC_STRINGS = new Guid("00020329-0000-0000-C000-000000000046");

                        writer.StartProperty(
                            new TnefPropertyTag(tag),
                            PS_PUBLIC_STRINGS,
                            "MyProp");
                        writer.WritePropertyValue("Hello!");
                    }
                    else
                    {
                        writer.WriteAttribute(reader);
                    }
                }

                //  Close writer
                if (null != writer)
                {
                    writer.Close();
                }

                LogMessage("ProcessMailItem", item, "Added 'MyProp' to PS_PUBLIC_STRINGS.");
            }
            else
            {
                LogMessage("ProcessMailItem", item, "No TNEF, not adding property");
            }
        }

        private void LogMessage(string eventName, MailItem item, string message)
        {
            TextWriter tw = System.IO.File.AppendText(LOG_FILE_PATH);
            tw.WriteLine(DateTime.Now.Ticks + "\t" + eventName + " - " + item.Message.Subject + " - " + message);
            tw.Close();
        }
    }

}

TalkBack: Synchronous Event Processing In Your Custom Applications

mstehle — Tue, 30 Sep 2008 18:14:53 GMT

The Exchange Developer blog has a post updating the future vision of Exchange development. If you remember, there was a post on the Exchange Team blog back in September 2005 that gave us the first glimpse of this vision leading up to Exchange 2007. The vision has come into focus some more but the overall direction hasn't changed in the two and a half years since the first Exchange 12 roadmap post.

Here are the important notes from the roadmap…

Exchange Web Services are the API of choice in Exchange 2007 and beyond for mailbox and public folder access.
The web store APIs (WebDAV, CDOEX, and Store Events) which are de-emphasized in Exchange 2007 will be going away in the next release.
The MAPI and CDO 1.21 download will be aligned with the lifecycle of Exchange 2003, meaning they will be moving to extended support in the near future.
Since at this point store event sinks will be gone in the next version of Exchange, synchronous event processing with OnSyncSave and OnSyncDelete will not be possible.

We’d like to hear from you about the last point…

I am working with the product team to discuss any scenarios where EWS notifications and Transport Agents might not be sufficient to replace synchronous store event processing in a custom solution. The Exchange development team and my Exchange support team are aware of the potential challenges arising from changing the development interfaces available. Between Exchange Web Services notifications and Transport Agents we feel that we can fill a lot of the gaps left by removing synchronous store event processing but we want to hear from you. Are there scenarios that you currently require synchronous event processing for right now? Please email me or leave a comment describing your scenario. Getting your feedback will help us prepare documentation, workarounds, or solutions that can mitigate the challenges presented by this change.

Take Notice of Exchange Web Service Notifications

mstehle — Fri, 25 Jul 2008 00:11:50 GMT

The Exchange API-spotting blog, which is run by the Exchange MSDN content and Exchange API product folks, as a new post about the transition from store event sinks to Exchange Web Service notifications. The aim is to provide some more detail about notifications and try to directly answer some questions that have been posed by developers about how notifications can realistically replace store event sinks when they go away.

...If you have feedback for the product team about the transition or any scenarios that you are concerned about, please comment on their blog post. These posts are going up well before the next release of Exchange in order to keep you informed but also to hear what you have to say. The nice thing about Exchange 2007 is that you have EWS notifications already available to test with so while your store sinks might work now, you should begin thinking about and testing the migration of your sink solution to one based on EWS notifications. If you hit roadblocks now my team can work with you on Exchange 2007 and get answers before the upcoming release where Exchange store sinks are not a fallback option...

In fact, the Exchange MSDN content team has been very busy spreading the word about what the transition from the gang of de-emphasized APIs (CDO 1.21, CDOEX, WebDAV, etc.) to the new ones (EWS and Agents) will look like...

Migrating from Exchange 2007 Legacy APIs

Migrating to Exchange Web Services, Part 1: Messaging

Exchange 2007 Legacy API Property Mapping

Disclaimer: This post is about Disclaimers...

mstehle — Wed, 23 Jul 2008 21:31:40 GMT

I was reviewing a customer's case today and it reminded me of a topic I meant to blog about a while ago...

Ever since Exchange 2000/2003 customers have tried to apply the disclaimer SMTP sink sample to internal mail and found that the disclaimer text is not stamped on these messages. Of course in Exchange 2000/2003 this won't work. This is further described in the Exchange 2003 documentation on TechNet about The Advanced Queuing Engine...

"...the CDO interface does not support changing the content of store-submitted messages. This is a limitation that CDO_OnArrival event sinks share with all other event sinks. This limitation exists because Exchange converts a store-submitted message to a temporary SMTP version for the event sink to handle, and then discards the temporary version after the sink finishes processing...

...Because Exchange discards the temporary copy of a store-submitted message, you cannot use an event sink to add a disclaimer or other modifications to all outbound messages, unless you force all messages to be received through SMTP. To do this, you must install the event sink on a bridgehead server that is separate from the mailbox servers in your organization."

This is one of the better explanations about why disclaimers don't work for internal mail in Exchange 2000/2003 and provides more detail than the KB article. What you don't get from the KB article or the TechNet documentation is that in Exchange 2007 this all changes...

The entire transport process is redone to the point that not only can internal messages have disclaimers added to them but there is no scripting or coding involved in doing so. Exchange 2007 has built in transport rules that an administrator can turn on and configure as needed without writing code. The impact on Exchange development here is that the Exchange 2000/2003 experience leads many of us to assume that tasks like adding disclaimers, prepending text to the subject lines, adding headers, adding recipients, or redirecting messages requires code but in Exchange 2007 it does not.

...Of course not every solution fits within the fixed set of conditions and actions defined for these transport rules - in those cases a custom transport agent can be used to implement whatever business logic is needed. However, it is important to remember this functionality exists in Exchange 2007 before setting out on developing a custom solution as to not reinvent the wheel....

OUTBOX: Registering a VB Script Store Sink

mstehle — Fri, 07 Apr 2006 00:18:00 GMT

The following is an exerpt from an email describing the steps to setup a VBS event sink on Exchange...

Before registering the event sink you need to install and configure a COM+ component which will execute the VB Script. This COM+ ships with Exchange and is used to execute any VB Script event sinks on the server. You will need to download the Exchange Server SDK to get regevent.vbs and eshmts.vbs which are discussed below.

You only need to do the following steps one time on the server...

1. Register the Script Host Component...
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_exch2k_the_script_host_sink.asp

2. When you setup the Script Host in COM+ you will specify a domain account and password for the COM+ component to run as. This account will need permissions to the mailboxes or folders that the sink will be registered to as well as local permissions to write to the log file. You can give permissions to individual user accounts through Active Directory Users and Computers (ADUC). In Exchange System Manager (ESM) you can give permissions organizational levels to provide broader access. Also this scripts sends out email, for this script to be able to send mail as another account you need to grant that permission on the account in ADUC.

3. Setup the Script Host in COM+...
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_exch2k_the_eshmts_script.asp

After setting up the Script Host, you will then register the script on each folder or mailbox that you want it to run. I usually create a batch file to add and remove a script while testing. You need to run your batch file as a user with write permissions to the mailbox or folder you are registering the script on.

Here is the command I used in my batch file...

cscript "C:\Program Files\Exchange SDK\SDK\Support\OLEDB\Scripts\RegEvent.vbs" Add OnSave ExOleDB.ScriptEventSink.1 "file://./backofficestorage/mstehle03.extest.microsoft.com/mbx/mrmet/inbox/scriptevent" -file "C:\files\code\EventSinks\ScriptSinks\OnSave.vbs"

This will create a registration item at the given file path. Notice that my script is creating a registration item called "scriptevent" in mrmet's Inbox. You will need to change the VBS file path and mailbox URL according to your environment.

...After following these steps there is no requirement for getting the VBS file on the file system as it is stored in the registration item itself. If you do a GET on the registration item you will see your VBS code...

...Another note Exchange Explorer can setup these registrations through a wizard but for VBS sinks it doesn't add the script to the registration item, rather it creates a seperate IPM.Document that is the script file. This is annoying because it isn't hidden and will appear in the folder when viewed in Outlook or OWA. Obviously deleting this file will break the sink...

OUTBOX: Introduction to Exchange Store Event Sinks Part 2

mstehle — Fri, 20 Jan 2006 00:52:00 GMT

Coming from Exchange 5.5 Event Service Scripts to Exchange Store Event Sinks involves more than just understanding the new events available and the registration process. Exchange 5.5 event scripts are written in VBScript with CDO 1.21 as the API to interact with Exchange data. Store Event Sinks can be written in VBScript, VB6, managed (.NET) code, and C++ they use ExOLEDB with ADO and CDOEX to interact with Exchange data. This is greatly improves scalability and performance but requires a bit of retooling for the developer to be comfortable in their new surrounds. The following information discusses what you need to know about the web store how to interact with data using CDOEX and ExOLEDB.

Exchange Store

One important distinction to make between Exchange 5.5 and Exchange 200x is the storage architecture. This is an important consideration when looking at the new API's available in Exchange 2000 and 2003 because many of the new API's relate to what is called Web Store or stream database that is added to the Exchange storage system. Exchange store event sinks, WebDAV, CDOEX, and ExOLEDB with ADO all work against the web store primarily as opposed to CDO 1.21 and MAPI which work against the ESE database (which was part of Exchange 5.5 as well and is commonly referred to as the MAPI database). We will talk about ExOLEDB and CDOEX mainly because that is what you will use in an Exchange Store Event Sink.

Exchange Storage Architecture

http://www.microsoft.com/technet/prodtechnol/exchange/guides/E2k3TechRef/e82e8de0-803a-4c2b-b012-f73d6a677901.mspx

MSDN Exchange Store Reference

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/wss_references_wss.asp

CDOEX, ADO, Properties, and Exchange SQL

The web store uses native intermittent content such as MIME which allows access through ExOLEDB with ADO and WebDAV and the use of SQL-like queries to find information. This is quite helpful and a little more intuitive for the database programmer. Each item (folders and messages) in the store can be created, modified, and searched like a row in a database were item properties are columns.

Folders can be used like tables and rows, for example to query the subfolders of a given root folder you would select the URL from all items in the root folder where the item "isFolder" column is "true"...

Select "DAV:href" from scope('shallow traversal of <ROOT FOLDER URL>) Where "DAV:isfolder" = True

The following link provides a description of the SQL predicates supported against the web store.

Exchange Store SQL

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_exch2k_sql_web_storage_system_sql.asp

Add just like in a database when you write a query like the one of above you need to know two key pieces of data specific to your store, the table name and the available columns. The table name is pretty easy to identify as it is the folder you what to query from. The column names differ for each item type with some overlap. For example appointment items and contact items have most of the same properties of a basic message but they also have their own unique properties that relate to their functions.

The following link provides a listing of properties available on different items organized by namespace. You will use the full namespace and property name when pulling fields from the Fields collection of a CDOEX object or querying in WebDAV or ADO.

Properties by Namespace

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/wss_references_nsproperties.asp

The interesting thing about this paradigm is that it is often quite easy for a database developer to walk in and start using ExOLEDB with ADO and WebDAV to execute these kind of queries against Exchange. However, a developer coming from MAPI or CDO 1.21 is usually left looking for the objFolder.Folders collection. ExOLEDB with ADO and WebDAV provide a way to retrieve raw information from the server, they are not an object model. CDOEX provides some interfaces and CoClasses to provide a bit of an object model for interacting with the web store.

MSDN Reference for CDOEX CoClasses, Interfaces, etc.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_esdk_reference_cdoex.asp

In event sink development you will typically stay within the interfaces of the CDOEX object you open with the URL passed into the event interface. Especially when working with meetings and contacts note that CDOEX provides a supported way for working with these item types. ExOLEDB with ADO and much of WebDAV offers you raw access to the items in the web store, complex item types like appointments and contacts require CDOEX or special WebDAV commands to ensure that all the appropriate properties are being set to ensure that end user clients display and use the information appropriately.

It is important to understand the limitations of each API and where you should use which. CDOEX and ExOLEDB with ADO should only be used on an Exchange server, specifically they can only be used the Exchange server which hosts the mailboxes or public folders you want to access. WebDAV should be used to remotely access Exchange mailboxes and folders that may not reside on the server it is running on, it can also be executed on a non-Exchange server. All of these technologies are supported with managed (.NET) code, CDO 1.21 and MAPI are NOT supported with managed code.

813349 Support policy for Microsoft Exchange APIs with the .NET Framework applications
http://support.microsoft.com/default.aspx?scid=kb;EN-US;813349

Examples

Exchange Tasks

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_e2k3_tasks.asp

This is a great starting place when you are looking to see what you can do with Exchange API's. There are all kinds of great code samples here to allow you to do all kinds of stuff against the store. I have pulled a couple samples out that relate to some common tasks developers do inside event sinks.

Adding an Appointment to the Calendar

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_cdo_adding_an_appointment_directly_to_the_calendar.asp

Responding to a Meeting Request

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_cdo_responding_to_a_meeting_request.asp

Listing Inbox Contents Using ADO

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_cdosys_listing_inbox_contents_using_ado.asp