mstehle: The CDOs and CDONTS of Messaging Development : Exchange Web Services

Announcing EWSEditor 1.5!

mstehle — Mon, 09 Nov 2009 19:37:31 GMT

Over the past year I’ve been working with the new Exchange Web Services Managed API and I have to say, it’s a wonderful thing – especially compared to working with the Visual Studio generated proxy classes for Exchange Web Services. While working with Exchange Web Services (EWS) early in the Exchange 2007 release it became apparent to me that an editor tool in the spirit of MFCMAPI built on top of EWS might be useful. At the time I started working on such an editor and based it off the Visual Studio generated proxy classes. To my delight I was soon introduced to an API that the product team was working on to make my life (and that of any other EWS developer using .NET languages) much easier – the EWS Managed API.

I immediately started work on converting my editor code from proxy classes to this new API. This process was about 75% deleting code and 25% writing new EWS Managed API code. I was replacing hundreds of lines of proxy class code with half as much (or less) EWS Managed API code. Over the coming months I explored the features of the API and was able to quickly add new functionality to the editor because I could focus on UI code rather than wordy proxy class code.

Out of all of this work comes EWSEditor! This initial public release is called version 1.5 because it has been used internally for a while by my team. There are still features I’m working on and basic API coverage that will come together over time. It’s a work in progress but still very useful in its current state so I wanted to get it out to the public. Please feel free to send me suggestions, comments, bugs, etc. There is an Issue Tracker and Discussions page on the project site – feel free to use them.

From the project site, here are the goals of EWSEditor…

“1. Demonstrate the Exchange Web Services Managed API functionality and simplicity to developers through its source code.
2. Demonstrate the Exchange Web Services SOAP traffic used to perform actions initiated through an explorer user interface.
3. Assist non-developers in debugging and understanding Exchange stores by exploring items, folders, and their properties in depth. “

The plan right now is to do an update every two months (next one will be in January). Since this is the first public release there might be an interim release if some blocking issue come up. Available for download are release notes which I’ll add to with each release to track the changes over time and give an overview of what is new. Here are the notes from the 1.5 release…

“1.5 Release – 11/6/2009

The first public release of EWS Editor! The basic feature set is still be flushed out but it is still a very useful tool. There are three feature areas of EWS Editor: General UI, Support Tools, and API Coverage. The focus of this release was to establish the UI design and flow of the application, solidify some useful support tools, and illustrate the direction of the application.

There are some gaps in CRUD tasks across item and folder properties and items themselves. The major focus of upcoming releases will be to fill in these gaps before moving on to new features.”

If your interested, download the source or just the binary and try it out. Here are some the highlighted features from the release notes…

Exchange Service Profiles

Exchange Service Profiles describe not only the connection information for calls specified in the ExchangeService class but also the root folder to use in the tree view of EWS Editor. Multiple ExchangeService objects can be displayed at one time using unique configurations. This allows folders on multiple servers and server versions to be displayed at one time or displaying the same folders using different credentials.

Exchange Service Profiles can be saved to an XML file which contains the configuration information (sans specified passwords) so that it can be reloaded easily. If a profile requires credentials then a prompt will appear when loading the Services Profile to enter the credentials.

View properties of ExchangeServices, Folders, and Items

The PropertyDetailsGrid display first class properties for all nodes of the tree view starting with the root, ExchangeService node and all the way to the folder contents form, displaying first class properties for Items in a folder. Under the View menu the PropertySet used to retrieve item or folder data can be configured to customize which schema properties and extended properties are displayed in the PropertyDetailsGrid.

Extended Property Lookup

A number of known extended MAPI properties have been indexed in EWS Editor. Extended properties can be added to the PropertyDetailsGrid through a look up using a known constant name for the property such as PR_SUBJECT, PidTagSubject, etc. An Extended Property Lookup tool provides a way to quickly get the property information for a known constant.

Request and response logging

While EWS Editor is running, all requests and responses are logged to a file. The Chatter Log tool is used to view the history of requests and responses.

FYI: New Version of Exchange Remote Connectivity Analyzer Released (ExRCA) with new Exchange Web Service Tests!

mstehle — Tue, 20 Oct 2009 18:43:59 GMT

From the Exchange Team blog:

“Today we released an updated version of the Exchange Remote Connectivity Analyzer. For those of you not familiar with this site, it is a Web-based tool that helps you troubleshoot connectivity issues. The tool simulates several client logon and mail flow scenarios. When a test fails, many of the errors have troubleshooting tips to assist you in correcting the problem. For more information, see our previous blog post here.

…

Additional tests

Exchange Web Services - This allows you to perform connectivity testing for Exchange Web Services client such as Entourage. Developers can also use the Service Account Verification test to ensure things are configured and working properly for access with an alternate account or Exchange Impersonation.”

Brad Hughes came up with this awesome tool and I’ve been working with him to add these EWS tests to ExRCA. There are two basic tests that you can do at this time with Exchange Web Services: 1) Test the ability of an account to perform basic Exchange Web Services tasks like Synchronization, Notification, Availability, and OOF or 2) Test the ability of a service account to access a mailbox through delegate access or Exchange Impersonation. The first set of tests may be more useful to Exchange Administrators testing for Entourage client connectivity and as well as Availability and OOF which are used by Outlook too. The Service Account Access test is a quick way to test and diagnose issues with service accounts accessing other mailboxes.

Both of these tests should be run against a test mailbox with an empty Inbox and Calendar folder because they will attempt to create and delete test items. During the execution of the test ExRCA looks to ensure the Calendar and Inbox of the specified mailbox are empty before running.

EWS: Exchange 2007 RTM vs. SP1 Specifying MeetingTimeZone's TimeZoneName when creating an Appointment

mstehle — Fri, 04 Sep 2009 20:17:47 GMT

Prior to Exchange 2007 SP1 an article was published with some specific guidance related to handling time zones in calendar item tasks. Contrary to Best Practices for Using Exchange Web Services for Calendaring Tasks, specifying just a MeetingTimeZone TimeZoneName when creating a CalendarItem appears to work in Exchange 2007 SP1 and later. The best practices says not to submit the TimeZoneName in MeetingTimeZone because it will be ignored, though there is a hint that this might not apply to Exchange 2007 SP1:

“Do not supply the optional TimeZoneName element when you create calendar items. In the initial release version of Exchange 2007, the TimeZoneName element is ignored on incoming requests.”

However, take the following request which properly creates a CalendarItem in the Eastern time zone:

<t:CalendarItem>

<t:ItemClass>IPM.Appointment</t:ItemClass>

<t:Subject>Appointment in Eastern Time</t:Subject>

<t:Start>2009-06-26T09:00:00.000</t:Start>

<t:End>2009-06-26T09:30:00.000</t:End>

<t:MeetingTimeZone TimeZoneName="Eastern Standard Time">

</t:MeetingTimeZone>

</t:CalendarItem>

From the best practices, the following statement intends to confirm that TimeZoneName is not ignored in Exchange 2007 SP1 and can be used by itself to define a time zone:

“In Exchange Server 2007 Service Pack 1 (SP1), BaseOffset is an optional element, and supports MeetingTimeZones that specify the TimeZoneName only.”

This is quite true in fact, were in Exchange 2007 pre-SP1 specifying the offset elements was a requirement and TimeZoneName was ignored; in Exchange 2007 post-SP1 the TimeZoneName is actually the preferred method for declaring a time zone. Furthermore, offset and transition values should only be set when specifying a custom time zone. Otherwise, standard time zones should be specified using TimeZoneName which is the name of the time zone as it appears in the registry key name on the Exchange CAS server. Then the Exchange server will pull the transition and offset information from the registry for the time zone you specified. Here is an image showing where the registry keys are:

Bottom line, with Exchange 2007 SP1 don’t specify time zone offset and transition values for known time zones – use the TimeZoneName instead. This will help your application work seamlessly with any updates to known time zone definitions in the future.

EWS: UID not always the same for orphaned instances of the same meeting.

mstehle — Wed, 02 Sep 2009 23:37:00 GMT

Earlier I linked to a post by Henning Krauses which discusses searching for a meeting using the UID and GlobalObjectId properties. The point of UID is to provide a unique identifier for a meeting across the calendars each attendee to the same meeting. Henning gave some expert advise for using GlobalObjectId instead of UID in a FindItem request to locate instances of a recurring meeting in different user’s calendars. As noted in his article, in Exchange 2007 pre-SP2 the UID property is built from GlobalObjectId – he even shows how to convert from UID to GlobalObjectId.

Using GlobalObjectId poses a problem however…If you were to create a recurring appointment and invite one attendee for only one instance of that recurring appointment that attendee ends up with only the one exception instance of the parent recurring appointment in their calendar. This is called an “orphaned” appointment. This type of appointment’s GlobalObjectId has additional information in it related to the exception date (for more information look at PidLidGlobalObjectId in MS-OXOCAL). The bottom line here is that this orphaned instance in this one attendee’s calendar will have a GlobalObjectId, and therefore UID, which doesn’t match the related appointments in the other attendee’s and organizer’s calendar. Henning’s code won’t work in this specific scenario…

So what now? Is this a bug in Exchange? Is there another property Henning could use? The answer to both questions is, yes!

There is another property, CleanGlobalObjectId, which was defined just for this scenario. If you look at the documentation for this property, the remarks confirm that this property specifically addresses the issue described above:

“The format of this property is the same as that of LID_GLOBAL_OBJID ( PidLidGlobalObjectId). The value of this property must be equal to the value of LID_GLOBAL_OBJID, except the YH, YL, M, and D fields must be zero. All objects that refer to an Instance of a recurring series (including an orphan instance), as well as the recurring series itself, will have the same value for this property.”

So all Henning needs to do in his sample is change it to use the CleanGlobalObjectId instead of GlobalObjectId. Now, it would be logical to assume that Exchange should base the UID property off of the CleanGlobalObjectId too instead of GlobalObjectId. In fact, we’ve done just that. In Exchange 2007 SP2 UID is based on the CleanGlobalObjectId instead of GlobalObjectId. Because SP2 just came out and many customers may wait to install it because of the schema changes you’ll want to keep the workaround above in mind when working with GlobalObjectId and UID in Exchange Web Services.

FYI: DeleteItem behavior, Exchange 2010 SDKs, EWS performance and SyncFolderItems, and Named Property Mapping in Exchnage 2007+

mstehle — Mon, 13 Jul 2009 18:30:22 GMT

Fellow developer support engineer, Patrick Creehan, has any interesting post about the delta between the documentation of DeleteItem calls and the actual behavior involving including the ChangeKey in the ItemId of the item you want to delete. In theory the ChangeKey would be required for DeleteItem requests to ensure that you know you have evaluated the latest version of the item and decided to delete it. If you pass an ItemId with a stale ChangeKey, you should get a error indicating this isn’t the latest version of the item and you can retrieve it to make sure you still want to delete it. This is how an UpdateItem request works. As Patrick’s post indicates, DeleteItem doesn’t work the same way UpdateItem works – it just ignores the ChangeKey and does the delete no matter what. Patrick has a workaround to provide similar functionality in case you need it.

…

The Exchange API team announced, they’ve posted a beta SDK for Exchange 2010 on MSDN. In their post they highlight the most notable change in the structure of this SDK:

“…For earlier product releases, all Exchange developer documentation is included in a single SDK. For Exchange 2010, documentation for each technology area is published in a separate SDK, making it quicker and easier for you to find information that you need. The following SDKs are available online for Exchange 2010 Beta:

Exchange 2010 Beta Web Services SDK June 2009

Exchange 2010 Beta Transport Agents SDK June 2009

Exchange 2010 Beta Backup and Restore SDK June 2009…”

…

Henning Krause posted a couple interesting articles that go along with my earlier, performance themed EWS posts (part 1 & part 2). His first post is about processing items in a folder using the EWS Managed API, he walks through some of missteps I see customer make when they complain about how “slow” EWS is – as he points out it often comes down to choosing the right tool for the job and synchronization is often over looked by novice EWS developers who choose to loop through an entire folder looking for changes instead. His second post is more about limiting the requested PropertySet to increase the performance of your EWS code – something I have pointed out earlier with GetItem and FindItem calls but it is good to reinforce this concept in SyncFolderItem calls as well.

Speaking of GetItem vs. FindItem calls – Henning has a third post which shows you how to get the message body with a FindItem request even though technically you can’t request the “item:body” schema property in a FindItem’s PropertySet. His post points out an important concept which is that schema properties (or “first class properties” as they are sometimes called) are tied back to extended properties which are the “raw values”. Often times certain limitations are applied to the schema properties because they calculations can be expensive or unavailable at a certain time – however in some cases you can get around this by using and understanding the correlating extended properties.

…

Jason Nelson posted a couple items (round 1 and round 2) on the Exchange team blog about named property mapping detailng the history and problems with this mapping and upcoming changes in Exchange 2007 SP2 and Exchange 2010. Most notable from the “round 2” post is that in Exchange 2007 SP2 and Exchange 2010 “…no x-headers are ever promoted to individual properties if a client has not already requested (and mapped) them. Not even authenticated submissions can create new named property mappings.” This could pose a problem if your application relies on these headers to do processing, if you are using an Exchange API with property access like EWS, MAPI, CDO 1.21, WebDAV, or Outlook 2007+ object model then fear not, “…x-headers are still stored in PR_TRANSPORT_HEADERS and still accessible to MAPI clients but they are not individual properties.”

However, if your application uses IMAP the story is different…

“Exchange2007 does not contain an STM file - it doesn't save the original mime content. Messages are converted to MAPI. IMAP has the ability to search based on header values, but if these are not promoted, the result is that the message is not contained in the results. If a client has previously mapped t he header in question (a MAPI client) then messages delivered after this will have the x-header promoted as an individual property.”

FYI: Can’t use Exchange Impersonation with GetUserOofSettings, SetUserOofSettings, or GetUserAvailability

mstehle — Wed, 24 Jun 2009 23:45:16 GMT

I kept forgetting about this so maybe blogging it will help me remember. As this thread confirms – you can’t call GetUserOofSettings, SetUserOofSettings, or GetUserAvailability when using Exchange Impersonation. If you try to do this you’ll get an error that the Service Account is not the mailbox owner.

Exchange API Team Blog: Exchange Impersonation vs. Delegate Access…

mstehle — Tue, 16 Jun 2009 17:34:14 GMT

The Exchange API team has a new post to explaining the differences between using Exchange Impersonation vs. Delegate Access to access an Exchange mailbox using Exchange Web Services. I’ve seen first hand that there is a gap in understanding the difference between the two and when to use one versus the other. This post goes a long way to address some of the confusion.

An important note that some people miss is the differentiation between Windows and Exchange impersonation – they’re not the same thing and more importantly, they aren’t even related…

“Exchange Impersonation is different than Windows Impersonation. Windows Impersonation is an operating system concept that requires you to set Kerberos constrained delegation. Exchange Impersonation is a simpler authorization mechanism that is designed for use only within Exchange Web Services (EWS). For more information about Windows Impersonation, see Client Impersonation on MSDN.”

Conversely, the posts describes delegate access as…

“Delegate access is used in scenarios in which there needs to be a one-to-one relationship between users. One common application of delegate access is the sharing of calendars between users, such as when an admin manages an executive’s calendar, or a when handful of individuals working on a project need to coordinate calendars…

…Individual users can grant and remove delegate access to their own mailboxes through several mailbox clients, such as Microsoft Outlook, Outlook Web Access, or Exchange Web Services-based clients. A mailbox owner does not need administrator rights to grant another user delegate access to their mailbox.”

…To be clear, delegate access can be granted by an administrator on an entire mailbox (but not to specific folders within the mailbox) using the Add-MailboxPermission cmdlet but as the above statement points out individuals can also grant delegate access for other users to access specific folders in their mailbox or the entire mailbox.

A major difference between impersonation and delegation is that as far as Exchange Web Services and Outlook Web Access are concerned there is no way to assign a single delegate access to multiple mailboxes expect by establishing the delegate relationship for each individual mailbox. I’ve talked about this before, the Exchange team describes it this way…

“For delegate access, there is no option to set up a single delegate for multiple mailboxes. A relationship must be established for each user who needs to access a given mailbox…

…You must explicitly grant delegate access for any new users who are added.”

…This is a great post and a worthwhile read but from working with many customers to help them understand these concepts, I’d like to add some additional information and emphasize some key points…

Understanding the Requests

To emphasize a side point in the blog post there are three methods of mailbox to access a mailbox: Exchange Impersonation, Delegate Access, and Direct Logon. The article doesn’t provide examples of what the XML looks like in each type of access request using Exchange Web Services. For understanding, I’ve provided them below…

Direct Logon

<GetFolder>
<FolderShape>
    <t:BaseShape>AllProperties</t:BaseShape>
</FolderShape>
<FolderIds>
    <t:DistinguishedFolderId Id="inbox"/>
</FolderIds>
</GetFolder>

Delegate Access

<GetFolder>
<FolderShape>
    <t:BaseShape>AllProperties</t:BaseShape>
</FolderShape>
<FolderIds>
    <t:DistinguishedFolderId Id="inbox">
      <t:Mailbox>
        <t:EmailAddress>
          someoneelse@contoso.com
        </t:EmailAddress>
      </t:Mailbox>
    </t:DistinguishedFolderId>
</FolderIds>
</GetFolder>

Exchange Impersonation

<soap:Header>
<t:ExchangeImpersonation>
    <t:ConnectingSID>
      <t:PrimarySmtpAddress>
        someone@contoso.com
      </t:PrimarySmtpAddress>
    </t:ConnectingSID>
</t:ExchangeImpersonation>
</soap:Header>
<soap:Body>
<GetFolder>
    <FolderShape>
      <t:BaseShape>AllProperties</t:BaseShape>
    </FolderShape>
    <FolderIds>
      <t:DistinguishedFolderId Id="inbox"/>
    </FolderIds>
</GetFolder>
</soap:Body>

…The example above is for explicit delegate access. You can also access a mailbox using implicit delegate access by specifying a FolderId which links to another mailbox. All Ids returned by EWS embed their mailbox information in them so requesting them without explicitly specifying a mailbox will also work.

Putting It All Together

The terms used here come from the way Exchange Impersonation (also called Server-to-Server or S2S Authentication) is explained in the Inside Exchange Web Services book. The book describes three types of accounts that come into play when accessing a mailbox: the Service account, the Act As account, and the Mailbox account. Understanding how these accounts are affected by different mailbox access types is important. The Service account is the account used to generate the XML requests and authenticate to the virtual directory, the Act As account is the account which will be used to authorize actions taken against a mailbox, and the Mailbox account is the actual resource you are trying to access. In the grid below suppose you have an EWS application which runs as a user called ApplicationAccount, here is how the different access types affect each account context…

Access Type	Service Account	Act As Account	Mailbox Account
Exchange Impersonation	ApplicationAccount	the impersonated account as specified in the ExchangeImpersonation property of the request	any mailbox that the Act As account has mailbox rights to through Add-MailboxPermission or through permissions granted through Outlook, OWA, or EWS directly on a mailbox folder.
Delegate Access	ApplicationAccount	always ApplicationAccount	any account or mailbox folder that ApplicationAccount has mailbox rights to through Add-MailboxPermission or through permissions granted through Outlook, OWA, or EWS directly on a mailbox folder.
Direct Logon	ApplicationAccount	always ApplicationAccount	always ApplicationAccount

FYI: The Exchange Web Services Managed API is here!

mstehle — Mon, 20 Apr 2009 18:15:51 GMT

As you might have seen, we announced last week that the Exchange Web Services Managed API is now available as a beta download. I’ve been working with the API internally for about a year now and I must say, if you are writing .NET code that uses Exchange Web Services there’s no better way to do it. I’ve been working on an Exchange browser application, similar to MFCMAPI which started out leveraging the auto-generated proxy classes. Switching this project over to the EWS Managed API cut down hundreds of lines of code and improved the readability and organization of the code immensely. (Stay tuned for more information on this project…)

From David Claux’s introduction:

“The EWS Managed API is a fully object-oriented API that provides the experience developers have come to expect from Visual Studio and the Microsoft .NET Framework. Built on the EWS XML protocol, it provides an easy-to-learn, easy–to-use, and easy-to-maintain .NET interface to Exchange Web Services that both beginner and advanced developers will find to be an improvement over autogenerated proxies.

Although the EWS Managed API is a new API (in the sense that it is a new .NET assembly that developers have to explicitly reference in their applications), it is important to understand that we are not replacing the EWS protocol. The EWS Managed API is just a complement to EWS for .NET developers. This means that your prior investments are safe. Whether you are using raw XML (if you are a Javascript developer, for example) or autogenerated proxies (on Windows or other operating systems) to talk to EWS, your existing EWS applications will continue to work. The EWS protocol is still the future of Exchange development; all new features will continue to be added to the EWS protocol, and will also be exposed in the EWS Managed API.”

The API at it’s simplest is just a set of proxy classes for interacting with Exchange – they still generate the same Exchange Web Services SOAP XML that the auto-generated proxy classes do or that your code that doesn’t use proxy classes do. The difference is that this API is easy to use, understand, and has some additional business logic included. The release of this API doesn’t change the support or our recommendations towards use of Exchange Web Services in their raw form.

Here are Matt’s three rules for getting started:

Check out David Claux’s introduction for the API on MSDN as well as the “Working with…” section of the MSDN reference.
Don’t confuse the Exchange Web Services Managed Reference with the EWS Managed API Reference. The Exchange Web Services Managed Reference in the MSDN is reference of the auto-generated proxy classes.
For the sake of my sanity please don’t start calling this the EWS MAPI – it’s the EWS Managed API, the EWS API, the EWS Client API, The API, Managed EWS, anything you want that doesn’t involve acronyms shared with other Exchange APIs. (I guess I should be happy they didn’t call it EWS CDO, right?)

KB: XML schema validation errors when Exchange Web Service requests and responses have invalid XML characters

mstehle — Fri, 13 Feb 2009 02:23:21 GMT

…I recently submitted the following KB article for publication but wanted to get the content out. I will update this post when the KB article is published…

SYMPTOMS

Scenario 1

When you submit an Exchange Web Services request to Microsoft Exchange Server 2007 that contains invalid XML characters, you may get the following ErrorSchemaValidation exception response:

“The request failed schema validation: [character], hexadecimal value [value], is an invalid character.”

Scenario 2

When you submit an Exchange Web Services request to Microsoft Exchange Server 2007 that returns item properties in the response containing invalid XML characters using the Visual Studio auto-generated proxy classes, you may get the following InvalidOperationException exception:

”There is an error in XML document”

“[character] hexadecimal value [value], is an invalid character.”

CAUSE

It is possible for Exchange item properties to have values that contain characters outside the valid range in the XML specification which is defined here:

http://www.w3.org/TR/2000/WD-xml-2e-20000814#dt-character

When an Exchange Web Services client makes a request to retrieve that item property, the Exchange server encodes the property values to ensure the response is transmittable. If this response is received by the client and subsequently validated against the XML schema, it will fail validation because of the invalid XML characters.

Conversely, if an Exchange Web Services client tries to send these characters in a request, even if they are encoded, the Exchange server will try and validate the request against the XML schema which will fail.

RESOLUTION

Scenario 1

There is no way to successfully post an Exchange Web Service request that contains invalid XML characters. If you are updating a property’s value that contains invalid characters you must decide whether to replace or omit these characters.

Scenario 2

In order to read Exchange Web Service responses from Microsoft Exchange Server 2007 that contain invalid XML characters, you should skip XML validation. If you are using the Visual Studio auto-generated proxy classes for Exchange Web Services then you can create a special partial class which overrides the GetReaderForMessage method of the ExchangeServiceBinding class which turns XML validation off:

public partial class MyExchangeServiceBinding: ExchangeServiceBinding
{
    protected overrideXmlReader GetReaderForMessage(
        SoapClientMessage message,
        int bufferSize)
    {
        XmlReader retval = base.GetReaderForMessage(
            message,
            bufferSize);

        XmlTextReader xrt = retval asXmlTextReader;

        if(null!= xrt)
        {
            xrt.Normalization = false;
        }

        returnretval;
    }
}

Jason Henderson: A Brief History and Exciting Future for Exchange Development

mstehle — Fri, 12 Dec 2008 18:42:00 GMT

It took me forever to get around to watching this but this is just a great presentation about Exchange development’s future at PDC. He does a great job of explaining where we’ve been as well…

Learn about the Exchange Web Services Managed API and how Exchange is getting “Cloud Ready” at PDC’08

This is an absolute must watch for any Exchange developer!

FYI: Exchange API spotting - Exchange 2007 SP1 RU4 Released

mstehle — Wed, 08 Oct 2008 22:00:04 GMT

The Exchange Development Blog has a new post about the recently released Rollup 4 for Exchange 2007 SP 1. There are several key changes to Exchange Web Services in this release that are discussed in the post…

“…If you have written code against Exchange Web Services, we definitely recommend that you take a look at RU4. The following are the biggest changes that will be coming in the rollup:

- Item IDs are now returned after MoveItem/CopyItem calls.

- Unknown/unsupported item types are returned as Messages.

- Updating tasks can now have correct behavior for Start and Complete Date values…"

OUTBOX: Understanding and Fixing Slow Exchange Web Services Code (Part 2)

mstehle — Thu, 14 Aug 2008 00:09:00 GMT

In part 1 of this series I showed how to and the importance of doing the minimal number of requests, emphasizing the use of the IdOnly shape until we really needed more properties. I also illustrated the effect of batching requests and finding a balance between the number of items requested in a batch and the total time to retrieve a set of items...

Even after applying these techniques in the calendar folder performance can still suffer. Doing an AllProperties request for even a single item in the calendar can be exponentially slower than an AllProperties request for a single item in the inbox (or other "non-calendar" folder). Since Exchange's genesis the calendar folder has always been a special case when writing client applications - there is a ton of business logic and background processing that goes on to make everything work just right. Because of this, it might be tempting to just write off this performance hit to the "crazy 'ole calendar folder" but there is an answer and it lies in exactly what properties are requested on each item during an AllProperties request on a CalendarItem...

Understand the AllProperties BaseShape and Schema Properties

It turns out that this is a key concept in Exchange Web Services (read more here). There is no SELECT * or GetProps(NULL) in EWS, clients should know what properties they want to request and request explicitly. What about the AllProperties shape? Doesn't that return "all properties"? In fact it does not (not that the previously mentioned methods return absolutely every property on an item either) and that may be confusing to some but it is the way it is. What AllProperties returns is the full EWS schema for the given item type - in other words the property list returned will change based on the message class of the item requested. This decreases the value of this base shape, reducing it to more or less a lines-of-code-saver.

Many of the EWS schema properties do not correlate directly to a single column on the item in the Exchange database, instead they are calculated on the client access server based on the values of multiple columns. Some of these calculations, especially ones in the calendar folder, can be quite expensive. In the CalendarItem schema there are four properties whose names indicate they may take a significant amount of effort to calculate: AdjacentMeetings, AdjacentMeetingCount, ConflictingMeetings, and ConflictingMeetingCount. Calculating the values of these properties can be assumed to require knowing about all the other items in the calendar - it might be time consuming to do that for every item in the calendar. What would performance be like if the request was for AllProperties minus these four properties?..

The Solution

The first question is, "How do I request AllProperties minus AdjacentMeetings, AdjacentMeetingCount, ConflictingMeetings, and ConflictingMeetingCount?" The question, as stated, has no solution. However the reverse is possible, request IdOnly plus all the other CalendarItem schema properties by specifying the schema properties explicitly using their FieldURI adding them to the AdditionalProperties collection on the ItemShape. The code looks something like this...

GetItemType git = new GetItemType();
git.ItemIds = itemIds.ToArray();
git.ItemShape = new ItemResponseShapeType();
git.ItemShape.BaseShape = DefaultShapeNamesType.IdOnly;

List<BasePathToElementType> props = new List<BasePathToElementType>();

// Only add these properties if we are querying the calendar folder...
if (folderId.Id == DistinguishedFolderIdNameType.calendar)
{
    //props.Add(GetProp(UnindexedFieldURIType.calendarAdjacentMeetingCount));
    //props.Add(GetProp(UnindexedFieldURIType.calendarAdjacentMeetings));
    props.Add(GetProp(UnindexedFieldURIType.calendarAllowNewTimeProposal));
    props.Add(GetProp(UnindexedFieldURIType.calendarAppointmentReplyTime));

...

}

git.ItemShape.AdditionalProperties = props.ToArray();

It might seem tedious to request each property explicitly, however this approach can ensure the efficiency of EWS client applications by eliminating unused data from requests and responses. I'd consider this a best practice when requesting item properties. Let's look at my test results:

First, I created a simple application which calls GetItem with AllProperties against every appointment in a test account's calendar. I tested with individual requests for each item and batching items in the requests. Here are the results...

Items per request	Total # of Requests	Time (seconds) Test 1	Time (seconds) Test 2	Average Time (seconds)
1	1432	851	895	873
25	58	762	777	769.5
50	29	762	759	760.5

...And then I changed the application to call GetItem with IdOnly and added every schema property except AdjacentMeetings, AdjacentMeetingCount, ConflictingMeetings, and ConflictingMeetingCount...

Items per request	Total # of Requests	Time (seconds) Test 1	Time (seconds) Test 2	Average Time (seconds)
1	1432	104	105	104.5
25	58	15	15	15
50	29	13	13	13

...As you can see the results aren't even close, those few extra properties really cost you. In the end, it is best to request the properties needed explicitly. The expense of calculated properties and the reality of what AllProperties actually returns greatly devalues this BaseShape and for most purposes it should be avoided.

Updated 8/15/08 - I forgot to include the code above...

Updated 1/22/2009 – Fixed broken GetProps link.

Updated 9/25/2009 - Code should use BaseShape of IdOnly instead of AllProperties.

Take Notice of Exchange Web Service Notifications

mstehle — Fri, 25 Jul 2008 00:11:50 GMT

The Exchange API-spotting blog, which is run by the Exchange MSDN content and Exchange API product folks, as a new post about the transition from store event sinks to Exchange Web Service notifications. The aim is to provide some more detail about notifications and try to directly answer some questions that have been posed by developers about how notifications can realistically replace store event sinks when they go away.

...If you have feedback for the product team about the transition or any scenarios that you are concerned about, please comment on their blog post. These posts are going up well before the next release of Exchange in order to keep you informed but also to hear what you have to say. The nice thing about Exchange 2007 is that you have EWS notifications already available to test with so while your store sinks might work now, you should begin thinking about and testing the migration of your sink solution to one based on EWS notifications. If you hit roadblocks now my team can work with you on Exchange 2007 and get answers before the upcoming release where Exchange store sinks are not a fallback option...

In fact, the Exchange MSDN content team has been very busy spreading the word about what the transition from the gang of de-emphasized APIs (CDO 1.21, CDOEX, WebDAV, etc.) to the new ones (EWS and Agents) will look like...

Migrating from Exchange 2007 Legacy APIs

Migrating to Exchange Web Services, Part 1: Messaging

Exchange 2007 Legacy API Property Mapping

OUTBOX: Understanding and Fixing Slow Exchange Web Services Code (Part 1)

mstehle — Thu, 17 Jul 2008 21:39:00 GMT

Recently I was working with a customer who was concerned about Exchange Web Services performance. He was testing some EWS code whose purpose was to retrieve all properties of every item in a mailbox. The code was structured like this...

GetItem | AllProperties - Get the root folder of the mailbox (one call)
- FindFolder | AllProperties - Get the folder and the FolderIds of its sub folders (recursive, one call for each folder)
  - FindItem | AllProperties - Find each item in the folder (one call for each folder)
    - GetItem | AllProperties - Get the full set of properties on each item (one call for each item)

This first thing to take note of is that there is too much information being requested in the base shapes of these requests. Each of these requests is done requesting an AllProperties base shape. A very simple change to improve speed would be us to use IdOnly base shapes on the requests until making the GetItem calls on the items themselves at which point all properties are desired...

GetItem | IdOnly- Get the root folder of the mailbox (one call)
- FindFolder | IdOnly - Get the folder and the FolderIds of its sub folders (recursive, one call for each folder)
  - FindItem | IdOnly - Find each item in the folder (one call for each folder)
    - GetItem | AllProperties - Get the full set of properties on each item (one call for each item)

This code is still very chatty, it currently makes one FindFolder and one FindItem for each folder plus one GetItem for each item in the folders. That is a lot of requests and responses going back and forth. Fortunately, these calls can be restructured and combined to reduce chattiness. Notice that FindFolder has the option for a deep traversal in its shape, this makes it possible to retrieve all the FolderIds for all folders underneath the mailbox root in a single request and response. There is no deep traversal option in the base shape of a FindItem request and there is no ParentFolderIds property on the GetItem request but GetItem requests can request multiple ItemIds. So while there still needs to be one FindItem request per folder, the GetItem calls can be batched instead of doing one per item...

FindFolder | IdOnly, Deep - Get the FolderId for every folder under the mailbox root (one call)
- FindItem | IdOnly - Get the ItemIds for every item in the folder (one call per folder)
  - GetItem | AllProperties - Get the full set of properties on each item (approximately one call per folder)

This greatly reduces the number of requests made and in turn increases the performance of the code. Notice that for the GetItem call it says, "approximately one call per folder". Just because an ItemId array of 1500 items can be sent with a GetItem request doesn't mean it is the best idea. Certainly batching the GetItem calls help performance but this should be broken up into a "reasonable" amount of items per request - large requests can take minutes to process and return. The ideal threshold can be determined by taking into account the deployment environment, application requirements, and testing results for each application.

...Just to illustrate the point, here is some data from tests I did. In this test I'm simply calling GetItem|AllProperties on every item in a test account's inbox. The folder has about 1500 items in it of vary sizes, some of the items are pretty large. There is just one Exchange server with all roles installed on it. The time recorded here is the time it takes to get through all the GetItem requests and responses for the items in the inbox. "Items per Request" is the number of ItemIds included in each GetItem request. Remember, these results are specific to my environment and what I'm doing in my code, your mileage may vary...

Items per Request	Total # of Requests	Time (seconds) Test 1	Time (seconds) Test 2	Average Time	Time per request
1	1531	250	262	256	0.17 seconds
2	766	193	188	190.5	0.25 seconds
5	307	149	151	150	0.49 seconds
10	154	137	135	136	0.88 seconds
25	62	129	129	129	2.08 seconds
50	31	135	130	132.5	4.27 seconds
100	16	134	131	132.5	8.28 seconds
250	7	132	131	131.5	18.79 seconds
500	4	131	134	132.5	33.13 seconds
1000	2	133	134	133.5	66.75 seconds
2000	1	135	131	133	133 seconds

...This is part one of a two part series, click here to read part two.

KB: Exchange Web Services Does Not Honor Inherited Permissions from Server Level Objects

mstehle — Fri, 07 Sep 2007 16:52:00 GMT

The following KB article was just published based on a case I worked and bug that I filed against Exchange Web Services in Exchange 2007, have a look...

940846 Error message when an account tries to open a mailbox by using Outlook Web Access or Exchange Web Services in Exchange Server 2007: "You do not have permissions to open this mailbox"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;940846

Previously with other Exchange APIs and in earlier versions of Exchange Server if you set Receive-As rights at the server, storage group, organizational level, etc. for a service account then you could access any mailbox under those objects with your service account. This is *not* the case with Exchange Web Services or Outlook Web Access in Exchange Server 2007. In Exchange Server 2007, the underlying components shared by EWS and OWA do not aggregate the security descriptors of all the parent level objects of a mailbox to determine access - they only look at the mailbox security descriptor. To workaround this, as the KB article states, you have two options...

Use the Add-MailboxPermission cmdlet to add permissions directly to the mailbox you would like to access
For EWS, use Impersonation to access the mailbox

This is issue only affects the shared underlying components of OWA and EWS in Exchange 2007. It does not affect other Exchange 2007 APIs such as WebDAV, MAPI, and CDO.

...The customer I was working with was using their experience from Exchange 2000/2003 to test their service account configuration for their WebDAV application in Exchange 2007 using OWA. They had created their service account, applied permissions to a storage group using Add-ADPermission and were using OWA to verify that it was permissioned appropriately to allow access to all mailboxes in a storage group. They got hung up on the fact that OWA wouldn't allow access and never tested their WebDAV application. When they finally did a WebDAV test the service account accessed all mailboxes fine - even though they still could access the mailboxes with the service account using OWA. This is because while OWA and WebDAV were closing tied in Exchange 2000/2003, in Exchange 2007 they are not - Exchange Web Services and Outlook Web Access are more closely related now...