March, 2010

When studying the event log over a steaming hot cup of coffee, have you ever seen the following directory event (2914)?

Process w3wp.exe (UNKNOWN) (PID=15780). Deleted throttling policy was referenced.  Id: 'Org: SomeOrg…/Configuration, Id: CN=ThrottlingPolicy-DefaultMailboxPlan\0ADEL:4d73d344-d66f-4eed-85f9-b6c95dcd2a13,CN=Deleted Objects,CN=Configuration,DC=…,DC=com'.

This one does require some action. In the case of this event log, a throttling policy was deleted from Active Directory, but because *someone* did not use Remove-ThrottlingPolicy to do so, the links from the associated mailboxes did not get updated properly. The fix is relatively easy – you just need to reassign the affected mailboxes to a valid throttling policy. Now, I highlighted the “\0ADEL” wording in the event log for a reason. That is a very reliable way to determine whether we are dealing with a deleted object in Active Directory. We can use that to our advantage in the Exchange Management Shell.

[PS] D:\Windows\system32>get-mailbox | ? {$_.ThrottlingPolicy -ilike "*`nDEL*"} | fl Name, ThrottlingPolicy

Name             : JohnDoe
ThrottlingPolicy : SomeDomain/Configuration/Deleted Objects/SomePolicy
                   DEL:3fb98144-b317-413b-9a19-78003fd5a633

The key is in the “*`nDEL*” string.  The value “`n” is the Exchange Management Shell’s way of escaping the new-line sequence.

Now that we have discovered the mailbox that is having the issue, we can assign that mailbox to another policy. In my case, I just want JohnDoe to use the default policy, so I will just null out the ThrottlingPolicy parameter.

get-mailbox | ? {$_.ThrottlingPolicy -ilike "*`nDEL*"} | set-Mailbox –ThrottlingPolicy $null

Even with this problem present, the throttling policy framework was behaving properly and “falling back” to the default throttling policy governing John Doe. However, it is a good idea to clean up these issues in the directory if you encounter them and make your throttling policy association links explicit rather than relying on the fallback logic to do the “right thing”.

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

When you use the FindItems method or the FindAppointments method in Exchange Web Services (EWS), it’s important to realize that neither of these methods returns all properties for the items returned by the query. For example, FindItems will not return the Body property for a message item, and FindAppointments will not return the RequiredAttendees property for a calendar item. Therefore, if you are interested in additional properties that cannot be returned by FindItems or FindAppointments, you must explicitly load the additional properties to the items that are returned by the query.

Fortunately, the EWS Managed API provides an easy way to load additional properties for multiple items with a single call to EWS. The following code example shows you how to search the Inbox folder for items that match the specified search criteria, and then use the LoadPropertiesForItems method to load the Subject property and Body property for the items returned by FindItems.

// Specify the IdOnly response shape, and specify that
// FindItem results should be requested in batches of 25.
ItemView view = new ItemView(25);
view.PropertySet = new PropertySet(BasePropertySet.IdOnly);
view.Offset = 0;
view.OffsetBasePoint = OffsetBasePoint.Beginning;

// Specify search criteria.
SearchFilter searchFilter = new SearchFilter.ContainsSubstring(ItemSchema.Subject, "Message #");

// Declare findResults.
FindItemsResults<Item> findResults;

do
{
    // Call FindItems to search the Inbox folder.
    findResults = service.FindItems(WellKnownFolderName.Inbox, searchFilter, view);

    // Load additional properties for the current batch of items.
    service.LoadPropertiesForItems(findResults, new PropertySet(ItemSchema.Subject, ItemSchema.Body));

    // TODO: Process the current batch of items.

    // If more items are available, update the offset value so that the next
    // batch of items is returned by the next call to FindItems.
    if (findResults.NextPageOffset.HasValue)
    {
        view.Offset = findResults.NextPageOffset.Value;
    }
}
while (findResults.MoreAvailable);

Note: By specifying a page size of 25 when ItemView is instantiated, this example requests FindItems results (and subsequently calls LoadPropertiesForItems) for a maximum of 25 items at a time. Processing items in batches like this is helpful in that it reduces the likelihood that the server will be overloaded when the number of search results is large.

The following is the XML response that is returned by from FindItems in the above code example. A total of five messages that match the search criteria were found in the Inbox folder. Item identifiers and change keys have been shortened for readability.

<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <m:FindItemResponse
                xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages"
                xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
        <m:ResponseMessages>
            <m:FindItemResponseMessage ResponseClass="Success">
                <m:ResponseCode>NoError</m:ResponseCode>
                <m:RootFolder IndexedPagingOffset="5" TotalItemsInView="5" IncludesLastItemInRange="true">
                    <t:Items>
                        <t:Message>
                            <t:ItemId Id="AAT5QA=" ChangeKey="CQAAA" />
                        </t:Message>
                        <t:Message>
                            <t:ItemId Id="AAT5QB=" ChangeKey="CQAAB" />
                        </t:Message>
                        <t:Message>
                            <t:ItemId Id="AAT5QC=" ChangeKey="CQAAC" />
                        </t:Message>
                        <t:Message>
                            <t:ItemId Id="AAT5QD=" ChangeKey="CQAAD" />
                        </t:Message>
                        <t:Message>
                            <t:ItemId Id="AAT5QE=" ChangeKey="CQAAE" />                         
                        </t:Message>
                    </t:Items>
                </m:RootFolder>
            </m:FindItemResponseMessage>
        </m:ResponseMessages>
    </m:FindItemResponse>
</s:Body>

Note: In this example, the number of items that matched the specified search criteria was fewer than our specified paging size of 25; therefore, all items were returned by the first call to FindItems. If more than 25 items (the specified paging size) had matched the specified search criteria, FindItems (and LoadPropertiesForItems) would have executed multiple times — once for each batch of 25 items that matched the specified search criteria.

The following is the XML request that is generated by calling LoadPropertiesForItems in the above code example. As this XML request shows, LoadPropertiesForItems translates to a batch GetItem call under the covers — the Subject and Body property for all five items in findResults are requested in a single call GetItem call to EWS.

<m:GetItem>
    <m:ItemShape>
        <t:BaseShape>IdOnly</t:BaseShape>
        <t:AdditionalProperties>
            <t:FieldURI FieldURI="item:Subject" />
            <t:FieldURI FieldURI="item:Body" />
        </t:AdditionalProperties>
    </m:ItemShape>
    <m:ItemIds>
        <t:ItemId Id="AAT5QA=" ChangeKey="CQAAA" />
        <t:ItemId Id="AAT5QB=" ChangeKey="CQAAB" />
        <t:ItemId Id="AAT5QC=" ChangeKey="CQAAC" />
        <t:ItemId Id="AAT5QD=" ChangeKey="CQAAD" />
        <t:ItemId Id="AAT5QE=" ChangeKey="CQAAE" />
    </m:ItemIds>
</m:GetItem>

Now, you might assume that the same simple call to LoadPropertiesForItems can also be used to load properties for calendar items that are returned by FindAppointments. And it will work, with one small caveat: LoadPropertiesForItems takes an argument of type IEnumerable<Item>, whereas FindAppointments returns IEnumerable<Appointment> — so you must find a way to pass the results from FindAppointments to LoadPropertiesForItems as IEnumerable<Item>. Fortunately, this is easily accomplished by using Linq. The following code example shows you how search the Calendar folder for appointments that match the specified search criteria, and then use the LoadPropertiesForItems method to load the Subject property, the RequiredAttendees property, and the OptionalAttendees property for the calendar items returned by FindAppointments.

// Define search parameters and maximum number of items to return.
CalendarView calView = new CalendarView(DateTime.Now, DateTime.Now.AddDays(3), 25);

// Specify the IdOnly shape.
PropertySet props = new PropertySet(BasePropertySet.IdOnly);
calView.PropertySet = props;

// Call FindAppointments to search the Calendar folder.
FindItemsResults<Appointment> findResults = service.FindAppointments(WellKnownFolderName.Calendar, calView);

if (findResults.TotalCount > iMaxItemsReturned)
{
    string sMsg = "Total number of items that match search criteria exceeds specified maximum number of results to return.";
    sMsg += " To find all items that match the specified search criteria, either narrow the date range ";
    sMsg += "or increase the specified maximum number of items to return in CalendarView.";
    Exception e = new Exception(sMsg);
    throw e;
}
else
{
    // Call LoadPropertiesForItems to load the Subject, RequiredAttendees, and OptionalAttendees properties.
    service.LoadPropertiesForItems(from Item item in findResults select item,
                new PropertySet(BasePropertySet.IdOnly,
                        AppointmentSchema.Subject,
                        AppointmentSchema.RequiredAttendees,
                        AppointmentSchema.OptionalAttendees));
}

Note: To use Linq as shown in the above example, you must include the following directive in your program: using System.Linq;

Want to learn more about working with the EWS Managed API? Check out the code examples that are available in the Working with the EWS Managed API section of the Microsoft Exchange Web Services Managed API 1.0 SDK.

Have you ever seen event 2917 in your Client Access server’s event log? It might look something like this:

Process w3wp.exe (PID=1234). A budget charge was encountered that exceeded the limit of '2.768900000' minutes.  Budget Owner: 'SomeUser', Component: 'EWS, CostType: 'CAS'.

Sounds scary, doesn’t it? The wording is a bit odd (feel free to blame me) and can lead one to assume that there is some strange throttling limit of 2.7689 minutes (or some other equally non-integer-like number) lurking inside Exchange. There is no such limit – that value indicates when the condition was discovered, not what the limit is. This is really more of a warning message, but is completely un-actionable and will be removed from a future release of the product. 

All this event is saying is that the throttling infrastructure noted that a given operation has taken longer than expected, and it wanted you to know about it. This could be the result of an overloaded mailbox server, intense periods of garbage collection, a domain controller issue, and so on… It does not suggest that the “budget owner” has done anything wrong or nefarious nor does it suggest that the user in question was throttled. You can ignore this event log and look forward to the days when it will no longer darken the doorstep of your event log.

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

So, you decide that you want to edit a throttling policy. In the Exchange Management Shell, you run Set-ThrottlingPolicy and set some odd parameter to a new value. Then you make your EWS/Outlook Web App/etc. call and notice that the results have not taken effect. Bummer!

Slightly miffed, you open up the Exchange Management Shell again and call Get-ThrottlingPolicy, and lo, the policy was indeed changed. Why didn't Exchange pick up the policy??

Welcome to the world of caching. The throttling policy framework that Exchange uses works off a "frequency of use" cache. By default, when a policy is first accessed, the process that is using that policy will load it from Active Directory and stick it in a process-wide cache for a *minimum* of 5 minutes. If no one else in that process accesses that policy from the cache, it will expire after 5 minutes so that any future calls will pick up the new data from Active Directory. However, if that policy *is* used over and over and over again, it will extend the life of that policy in the cache up to 15 minutes. After 15 minutes, the policy will expire and an up-to-date version will be loaded.

The results:

1. You cannot expect policy updates to immediately be picked up by an Exchange process. No special keys, flags, calls, or whining will help.
2. If you can wait, the policy will automatically be picked up after 15 minutes, and possibly sooner - no need to cycle the process.
3. If you cannot wait for the 15 minutes to pass, you will need to cycle the process.
4. I have found that 15 minutes is the perfect amount of time to obtain a cup of coffee from the kitchen and to begin enjoying said beverage.

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

The throttling framework is intended to protect Exchange resources, so if it is going to "fail", it needs to do so in a safe and predictable way. Let's say that Ken Malcolmson is assigned to non-default throttling policy XYZ. Unfortunately, an Active Directory elf climbed into your domain controller and scrambled things around a bit. So, when Exchange Web Services (EWS) (or some other process) tries to load policy XYZ, it fails. Or does it? Actually, it fails along a fallback path.

If the non-default policy is corrupt or missing, it will first fall back to the default throttling policy for the organization in question. Aha! What if the default policy is corrupt? Well, then it falls back to a special policy defined in code called the "fallback policy". Given that this policy is embedded in the Exchange assemblies, there is little chance that such a read will fail.

What are the values of the fallback policy? They are the exact values that default policies are assigned when Exchange is first installed.

One interesting thing about the fallback policy is that it will also be applied to authenticated callers that don’t fall neatly into an organization. Computer accounts, cross-forest contacts, typical Active Directory users (with no mailboxes), etc… will all be given budgets based on the fallback policy, so Exchange is protected against such activity. The downside to this is that because the fallback policy is defined in code, there is no way to modify the policy values for any such accounts.

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

One of my favorite Exchange Web Services (EWS) methods is FindItem, primarily because it was the first Web method I wrote when I joined the EWS team long, long ago. Since then, it has undergone lots of optimizations, feature changes, and so on, to make it what it is today. One of the Exchange 2010 changes to FindItem (and FindFolder) is protective in nature – Exchange doesn’t want to drown a Client Access server just because someone decides to pull down the entire contents of their mailbox. This protective change is governed by the EWSFindCountLimit throttling policy parameter.

When a FindItem request comes in and the caller is authenticated, EWS obtains the caller’s budget. Every item (or folder in the case of FindFolder) that EWS processes for the current FindItem request is counted against the budget’s EWSFindCountLimit. As soon as the response is sent back to the caller, the find count charge for the current call is released. Why do we do this? Because items and folders that are waiting to be returned in a response take up memory on the Client Access server.

You will soon discover that budgets have a larger scope than a single request. If a caller makes two concurrent EWS FindItem requests that return 10 items each, the EWSFindCountLimit charge against that caller’s *single* budget is…20. When the first request returns, it will drop to 10 (because those items can now be garbage collected) and then when the second request returns it will drop to 0. It should be relatively obvious that we are trying to limit the caller’s memory footprint on the Client Access server.

What happens, though, when a caller reaches the limit? Well, that depends on whether the caller is being a “good EWS citizen”. And what makes an EWS citizen “good”? Well, in the context of EWSFindCountLimit, a good EWS citizen will do all their FindItem calls by using paging rather than by requesting the entire result set each time. Assume, for a moment, that the EWSFindCountLimit value is 1000 and a bad EWS citizen makes a FindItem call without paging. Let’s further say that the query will return 1001 items. EWS certainly can’t truncate the results, as the caller is expecting, to receive ALL the data back. And EWS can’t turn a non-paged FindItem call into a paged one, as the caller would not expect to check for such a response. So what does EWS do? If the call has a RequestServerVersion that is earlier than Exchange2010, you will receive a failure response with an error code of ErrorServerBusy. If your RequestServerVersion is Exchange2010, you will get back the following response:

<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <m:FindItemResponse xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
    <m:ResponseMessages>
      <m:FindItemResponseMessage ResponseClass="Error">
        <m:MessageText>You have exceeded the maximum number of objects that can be returned for the find operation. Use paging to reduce the result size and try your request again.</m:MessageText>
        <m:ResponseCode>ErrorExceededFindCountLimit</m:ResponseCode>
        <m:DescriptiveLinkKey>0</m:DescriptiveLinkKey>
        <m:MessageXml>
          <t:Value Name="PolicyLimit">1000</t:Value>
        </m:MessageXml>
      </m:FindItemResponseMessage>
    </m:ResponseMessages>
  </m:FindItemResponse>
</s:Body>

Very interesting. For kicks, let’s reduce the EWSFindCountLimit and cycle IIS to pick up the changes (ah yes, the joys of having your own test box to beat upon). Open the Exchange 2010 Management Console as an Exchange Admin and run the following (New-ThrottlingPolicy is only available to administrators, for obvious reasons):

New-ThrottlingPolicy –Name “FooPolicy” –EWSFindCountLimit 1
Set-Mailbox JohnDoe –ThrottlingPolicy FooPolicy
iisreset

Now, on my test box, I created 6 email messages in John Doe’s Drafts folder.  Let’s call FindItem again, but use indexed paging.  Here is my request:

<soap:Header>
    <t:RequestServerVersion Version="Exchange2010"/>
</soap:Header>
<soap:Body>
<FindItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
           xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" Traversal="Shallow">
    <ItemShape>
        <t:BaseShape>IdOnly</t:BaseShape>
        <t:AdditionalProperties>
            <t:FieldURI FieldURI="item:Subject"/>
        </t:AdditionalProperties>
    </ItemShape>
    <IndexedPageItemView BasePoint="Beginning" Offset="0" MaxEntriesReturned="10000"/>
    <ParentFolderIds>
           <t:DistinguishedFolderId Id="drafts"/>              
        </ParentFolderIds>
</FindItem>

And the response…

<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <m:FindItemResponse xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
    <m:ResponseMessages>
      <m:FindItemResponseMessage ResponseClass="Success">
        <m:ResponseCode>NoError</m:ResponseCode>
        <m:RootFolder IndexedPagingOffset="1" TotalItemsInView="6" IncludesLastItemInRange="false">
          <t:Items>
            <t:Message>
              <t:ItemId Id="AAMkA…"/>
              <t:Subject>Message5</t:Subject>
            </t:Message>
          </t:Items>
        </m:RootFolder>
      </m:FindItemResponseMessage>
    </m:ResponseMessages>
  </m:FindItemResponse>
</s:Body>

Whoa – wait a second! There were 6 messages to return, I asked for 10000, but it only returned 1 of them to me! What happened? Well, given that the caller’s EWSFindCountLimit is 1, EWS will NOT return more than 1 record to the user (at a time). However, because the caller was a good Exchange citizen and made a paged request, EWS can “fix up” the result because the user will (read: should) understand that EWS might not return ALL the items. The caller will instead look at the IncludesLastItemInRange attribute and, if it is false, make another FindItem request with the new Offset and continue until IncludesLastItemInRange returns true. If this behavior seems odd to you, consider that this is exactly how the various network streams in the Microsoft .NET framework behave – you continue reading from the stream until the stream reports that it has no more data to read.

So, the moral of the story is: Always use a paging mechanism when calling FindItem or FindFolder. Wisely, the Exchange Web Services Managed API forces you to do so by requiring an ItemView to be passed into FindItem:

service.FindItems(new FolderId(WellKnownFolderName.Drafts), new ItemView(10000));

Now that you know, there is no excuse for failing to use a page view :)

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

Throttling policies have this funny parameter called CPUStartPercent. This parameter defines the starting CPU use percentage at which Exchange will begin delaying inbound requests.  EWS, Exchange Active Sync, POP and IMAP honor the CPU Start Percent value in Microsoft Exchange Server 2010.

Three (no, five!) things to keep in mind:
1. The sampled CPU percent is across all CPUs. So at a CPUStartPercent of 75%, it (the process in question) would have to be using 75% of ALL CPUs for sleep throttling to kick in.
2. The sampling logic is an average over roughly a 10 second rolling window. This keeps us from responding to quick spikes in CPU.
3. This parameter is *per throttling policy*. This means that you could assign a different CPU start percent for Jane than you do for John. This is particularly useful if you have a service account whose work is more important than your ordinary user and where you just don't care if it taxes the system. Make wise decisions about this one, though, as a heavily taxed EWS will affect the user experience for someone using Outlook Web App (for example).
4. This delay is capped at 500 msec at a theoretical 100% CPU usage. However, this is *per* check. If you pass in a batch EWS request to get 100 items, it will check the CPU usage 100 times (once per item) for a maximum sleep of 50 sec... Keep that in mind :). Before you freak out about this, note that the operating system will do this anyway when it becomes overtaxed, just in a more non-deterministic fashion.
5. The delay time is linearly proportional to CPU usage. At CPUStartPercent, the delay is 0 (a thread yield) and increases linearly up to 500 msec at 100% CPU usage.

How can you determine if you are actually getting delayed due to sleep throttling? On the Exchange Client Access server, look at the MSExchange Throttling | Average Thread Sleep Time performance counter for the process you are interested in. It will give you an idea as to whether your requests are being affected by sleep throttling.

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

A few components (Outlook Web App, EAS, and EWS) log budget snapshot information to the IIS logs. EWS, notably, includes both a start and an end snapshot. Here is an example of an IIS log entry for EWS (lines are broken up to make it more readable):

2010-03-09 15:15:12 XX.XXX.XX.X POST /ews/Exchange.asmx ;RC:bfe35012-30fd-499e-90d6-552d0ca89fab;
Init>>Conn:0,AD:30000/30000/0%,CAS:54000/54000/0%,AB:30000/30000/0%,RPC:36000/36000/0%,FC:1000/0,Hash:14366112,Sub:20/0;
SoapAction=m:GetDelegate;Version=1;RpcC=14;RpcL=687;LdapC=0;LdapL=0;
End(17156.25ms)>>Conn:3,AD:30000/30000/0%,CAS:54000/36907/28%,AB:30000/30000/0%,RPC:36000/35985/0%,FC:1000/0,Hash:14366112,Sub:20/0;
443 Domain\User XX.XX.XXX.XX - 200 0 0 17468

I am not going to go into a lot of detail regarding all the information here at this point, but I do want to point out a few things.

The "Init" section indicates what the user’s throttling budget looked like when the call began, whereas the "End" section indicates what the user’s throttling budget looked like when the call ended.

You may be tempted to assume that the change in budget between Init and End was due only to this GetDelegate call. However, budgets take into account concurrent requests. So, if you have a number of client threads hitting Exchange for user X, a single budget actually tracks all this information. The end result is that each of these IIS logs is a snapshot that takes into consideration ALL the request threads that are currently accessing the budget.

So, for example, on this simple GetDelegate call, the initial connection count was zero and the ending connection count is 3. It would be erroneous to assume that GetDelegate created multiple connections. Instead, you would need to examine the IIS logs for the current user and see which ones overlap the current GetDelegate call. Then you can determine where those extra connections came from.

A few quick acronym helps…
RC – Request Correlation – used by Microsoft support. Enables them to correlate IIS log entries with server traces.
Conn – Connections/Concurrency (represents MaxConcurrency).
AD – Active Directory (represents PercentTimeInAD).
CAS – Client Access server (represents PercentTimeInCAS).
AB – Address Book (represents PercentTimeInAddressBook).
RPC – Remote Procedure Call (represents PercentTimeInMailboxRPC).
FC – Find Count Limit (how many records can be returned in a single paged FindItem/FindFolder call).
Sub – Number of active subscriptions (push or pull).
Hash – Hash code of the budget. You can ignore this.

You will also notice the following format in both the Init and End sections: X/Y/Z%

X - Total number of msec allowed per minute as defined by the active throttling policy.
Y – The remaining time in the budget for this policy part.
Z – The "effective percent" that represents how much of the policy the current budget has used.

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services