BradHugh's WebLog

Internet Explorer 8 impacts OWA Load Balancing Scenarios

2009-10-15T17:09:21Z

Recently we’ve had several issues reported regarding Internet Explorer 8 and Outlook Web Access load balancing scenarios.

Here are the scenarios we’ve identified so far:

DNS Round robin load balancing when DNS TTL expires
DNS Round robin load balancing when outbound HTTP Proxy servers are used
SSL Session-ID persistence with a hardware load balancer

The reason that each one of these scenarios occur is because of the new “LCIE” or “Tab Process” model in Internet Explorer 8. Essentially Internet Explorer 8 can create new “tab processes” for new browser tabs or browser windows that are opened. It doesn’t do this for every tab or window but uses an internal algorithm to determine whether or not it should spin up a new process.

For a detailed description of the new architecture, see the following two posts:
IE8 and Loosely-Coupled IE (LCIE)
Opening a New Tab may launch a New Process with Internet Explorer 8.0

Due to the new architecture certain things such as kept-alive connections and SSL sessions are not shared between Tab Processes. This plays an important role in the scenarios we have found.

In advance: Yes, I know it’s not called an “OWA Server” – I’m calling it that to make this work for Exchange 2003 or Exchange 2007.

The design of Forms Based Authentication in Outlook Web Access is another contributing factor. Forms based authentication works by the OWA server storing a cookie in the client browser for authentication. This cookie contains the encrypted credentials of the user that the OWA server can translate into basic authentication credentials for IIS with each new request. However this cookie is encrypted by a key that is maintained (and rotated) in memory only on the OWA server that set the cookie. For instance if a client logs into Server A and receives a cookie from Server A, it cannot use that same cookie to authenticate to Server B because Server B does not have they key to decrypt the credentials. If you do try to present a cookie from Server A to Server B, your browser will be redirected to the login page on Server B to provide valid authentication.

That explanation given, here is a brief explanation of the behavior you may see in each scenario.

DNS Round robin load balancing when DNS TTL expires

If you use DNS round robin load balancing for Outlook Web Access, you may experience confusing client behavior when the DNS time to live values expires on the record. The scenario usually manifests itself like this:

You are logged into OWA (with forms authentication) when the DNS TTL expires. If you continue to work within the main OWA window, you should be able to continue to work within the main OWA window, you should be able to continue to work and not be logged out. This is because IE uses it’s kept-alive connections that are already established to the original OWA server. However if you open a number of email messages in new windows (double-click mail messages), a new tab process for iexplore.exe will be created. This new tab process will have to create a new TCP session that, when resolved by Winsock, returns the IP address of a different OWA server in the DNS round robin. When this new window opens, you will be redirected to the forms authentication login page. If you provide your credentials to the “new” server, you will in-turn get logged out of the “main” OWA window that still has a persistent connection to the “old” server. And the next time you spin up a new tab process, the experience starts over again.

So needless to say, it’s a confusing user experience and the only way to recover is to close all browser windows, restart IE and log back in to OWA.

DNS Round robin load balancing when outbound HTTP Proxy servers are used

If you use DNS round robin load balancing for Outlook Web Access, you may experience being logged out randomly with Outlook Web Access. This is really due to the same behavior as above except instead of the name resolution happening on the client, it happens on the proxy server. Since proxy sessions are not maintained across tab processes, new outbound connections will also be established from the proxy server. Since many users may be accessing the same DNS name through the same proxy server, the TTL expiration may seem completely random.

SSL Session-ID persistence with a hardware load balancer

If you are using SSL session-id based persistence on your hardware load balancer, you may experience the same behavior as above except in a more severe form. In this situation the new connections that are established by the new tab process are routed to a different OWA server by the load balancer. This will happen every time a new tab process is spawned, not just at some arbitrary interval making this by far the worst of the experiences.

Workarounds and Solutions

We’ll start with the easy one: SSL Session-ID persistence. Because of issues (just like this) where the SSL session-id changes, SSL session-id based persistence is not (and has never been) a supported persistence method for load balancing Outlook Web Access. In fact, there aren’t really any Exchange Services for which we support or recommend SSL session-id based persistence (Edit: Obviously for services that don’t require persistence at all such as ActiveSync and certain Outlook Anywhere configurations, you can use SSL-ID based persistence to keep the SSL handshakes to a minimum). The supported persistence methods for Outlook Web Access are Client-IP based or Cookie-based persistence. In certain situations where many clients are coming from the same IP (maybe a proxy server), cookie-based persistence is a much better solution than client-ip based persistence to get the load spread evenly. Be aware that Outlook Web Access is the only Exchange web service that can use cookie-based persistence.

For DNS round-robin the answer is a bit more complicated. We’ve never really recommended DNS round robin for load balancing, but it technically is supported. Typically we just outline the caveats like the reliance on the client-side DNS resolver cache and the fact that it gives you zero redundancy in the case of a service failure. However with all this guidance, many folks still see this as the poor man’s load balancer and implement it anyway. If you are experiencing this issue, I strongly recommend you move to a real load balancing solution whether this is Windows NLB, the load balancing built-in to ISA Server 2006+, or a hardware-based load balancer. If you can’t or won’t, this is something you’ll have to live with for now. You can work around this issue by setting the TabProcGrowth registry value for the IE browser to 1 (It’s mentioned in the second of the earlier linked IE blogs). However, this isn’t feasible in most situations because you have little control over the client-base.

We are currently pursuing a design change request with the Internet Explorer team to see if we can make any of these experiences more consistent with the IE7 behavior. However, since changes in this area can be very risky, we may have to live with this new behavior. After all, it’s not a bug, it’s just an effect of the change in design for IE8.

Thanks to everyone for reading, and special thanks to John Towler for reporting this.

Autodiscover and Client-only Sites (Revisited)

2008-05-20T14:39:00Z

Previously I had posted a problem regarding Autodiscover site-scoping and client-only sites. For sites that do not have Exchange Servers in them, the corresponding client sites must be added to the SCP object so that Outlook clients will discover the closest Client Access Servers for Autodiscover.

I want to call out one issue I was informed about that one customer had run into (but most folks likely won't). There actually is an upper-limit to the maximum number of values (or actually overall size) of a multi-valued attribute in Active Directory. For these site values that limit typically exists somewhere between 800-900 entries.

The error returned is:

Set-ClientAccessServer : Active Directory operation failed on MICROSOF-D1CEB0.contoso.com. This error is not retriable. Additional information: The administrative limit for this request was exceeded. Active directory response: 00002024: SvcErr: DSID-02080490, problem 5008 (ADMIN_LIMIT_EXCEEDED)

In this case, the customer had multiple load-balanced CAS's in any given site. He simply split up the assignment and put 400 Sites in one SCP and 400 Sites in the other. It seems that this is likely the best workaround if you were to run into this.

On another note, I found out something really cool you could do with Powershell today and wanted to update my previous script post so that it is 100% Powershell. I was under the impression that you could not call Win32 API's from Powershell; while that is technically accurate, you can call Win32 API's from C# and you can call C# from Powershell so guess what you get? Dynamically compiled C# code to call the Win32 API embedded in and invoked from a Powershell script!

This is adapted from the following blog post: Powershell - PINVOKE or accessing WIN32 APIs

The main script is identical to the script I previously posted. However, this time instead of calling the command-line EXE, I call my .NET static method that I defined previously.

First: A Compile-CSharp function to do the on-the-fly compilation of my C# source code

We create a new instance of the CSharp code provider to do the compilation.

$cp=new-objectMicrosoft.CSharp.CSharpCodeProvider

We then set our parameters for the compilation operation.

$cpar=New-ObjectSystem.CodeDom.Compiler.CompilerParameters

$cpar.GenerateInMemory =$true

$cpar.GenerateExecutable =$false

$cpar.OutputAssembly ="custom"

Then we compile the source.

$cr=$cp.CompileAssemblyFromSource($cpar, $code)

Second: Defining the C# code and calling the compilation function

$siteCostCode=@'

<C# code here>

# Compile our C# code for doing the site cost calculations

Compile-CSharp$siteCostCode

Finally: Calling the managed static method defined in our C# Code

[int[]] $costArray=$null;

$siteCostStatus= [SiteCost.SiteCostLib]::QuerySitesByCost($clientsite.Name, $exchangeSites, [ref]$costArray);

NOTE: At this time the download is not available. I am trying to locate the script and will repost it if/when I can find it.

How NOT to Deploy Client Access Servers

2008-05-06T00:29:00Z

I have obviously been really busy since December since that is the last time I posted a blog entry. It's rather embarrassing, but I've been devoting a lot of my time to an Exchange Troubleshooting tool hopefully to be unveiled in the Exchange 14 timeframe - more to come later on this.

In these 5 months, I've had plenty of time to see customers do some "interesting" things with the Client Access Server deployments. Since many customers still ignore our prescriptive guidance, I'm just going to give you the gun to shoot yourself in the foot.

Here are some things you can do so you can ensure you have fun-filled days, every day of the week, on the phone with Microsoft Exchange Support.

Deploy multiple Client Access Servers in the same AD Site with different configurations

Here are some common things that lead customers to do this:

Want different functions for Client Access Servers. I.E. "This is my OWA / ActiveSync CAS, and this is my Autodiscover / EWS CAS", or "This is my Internal CAS, and this is my External CAS."
Want the Client Access role on the mailbox server as a "backup" in case the CAS goes down.
Have their "Test" or pre-production environment in the same AD-Site/Same Exchange Org as their Production environment.

Exchange cannot read your mind!

Don't assume that just because you have deemed CAS A as Internal and CAS B as External that Exchange will know the difference. I will admit there are (most times) certain configurations changes that can make configurations like this work, but they are extremely complicated and prone to error. All Client Access Servers in the same site should be configured the same way. I make mention of this on my previous blog post, CAS Load-Balancing Best Practices (Part 1). You will run into problems if you try this. There is no reason to try and split services as mentioned in reason 1 above. This was never tested and is not recommended. In general CAS Services scale well together and you should not try to isolate any particular service. For #2 and #3 above, just remember, that Exchange and Outlook can't read your mind. As soon as a server is deemed a CAS (in AD), it is fair game for Outlook clients, proxy targets from other sites, etc, etc. It is very difficult to understand and implement all the different factors that are considered when a client or a CAS in another site is choosing a CAS to connect to. This will cause you issues; so just don't do it.

Deploy your Client Access Servers in a DMZ or Perimeter network, but "pretend" it's not a DMZ

We've seen customers again and again try and skirt our support stance on this. Just in case you didn't know:

Planning for Client Access Servers: http://technet.microsoft.com/en-us/library/bb232184(EXCHG.80).aspx

Installation of a Client Access server in a perimeter network is not supported. The Client Access server must be a member of an Active Directory directory service domain, and the Client Access server machine account must be a member of the Exchange Servers Active Directory security group. This security group has read and write access to all Exchange servers within your organization. Communications between the Client Access server and the Mailbox servers within the organization occurs over RPC. It is because of these requirements that installing a Client Access server in a perimeter network is not supported.

Don't pretend your DMZ/Perimeter network isn't a DMZ/Perimeter network

We've had numerous customers who want to argue about what is and is not a Perimeter network. What was meant in the original documentation by "perimeter network" is any network that does not have unrestricted access to every Domain Controller and Exchange Server in the Organization. We've had customer who have called it a "pocket-DMZ" meaning that it's not their main DMZ where there web servers are. This DMZ sits off their internal network in a separate "pocket" where access to and from the internal network is restricted. This is still a Perimeter network and falls into the above support policy. If your CAS is NOT on your internal network, it's probably safe to assume that it's in a DMZ and likely not supported.

Not supported, means not supported

If you call into PSS and the support engineer you are working with finds that your Client Access Server is in a restricted or perimeter network, you are deemed unsupported. This does not mean that the PSS engineer hangs up the phone and says "too bad" right off the bat. What this does mean is that the Engineer will gather logs and attempt to better understand your issue. If at any point during troubleshooting, the PSS Engineer feels your issue may be caused by, or complicated by, the Client Access Server being in the perimeter, the engineer may request that troubleshooting be suspended until the Client Access Server in question be moved into the internal network. If the customer is not willing to do this, then the customer is at that point unsupported by Microsoft PSS.

It's not even a good idea...

Our guidance recommends that an ISA Server or another reverse proxy server be placed in the DMZ to handle requests for Internet Exchange Services such as OWA and ActiveSync. ISA 2004 and later allows ISA to operate in a Workgroup (non-domain) configuration and still pre-authenticate requests to Active Directory. This ensures that no unauthenticated traffic is ever passed to the Client Access Servers and also ensures that every URL and request is processed and scanned by ISA URL Filtering/Scanning/and IDS feature-set. This allows for a single port to be opened up (443) to each Client Access Server in the internal network (to allow ISA Web publishing) and is the most secure configuration by far.

If an Exchange 2007 Client Access Server is deployed in the perimeter, a plethora of ports must be opened (both random and static) to Exchange mailbox servers, Domain Controllers, and other infrastructure roles such as DNS. For an Exchange Mailbox Server, you must open Dynamic RPC ports 1024 and above, in addition to 135, 139, and 445 (standard Windows RPC and File sharing ports). These ports essentially make your back firewall in the perimeter scenario into swiss cheese and is counter-productive to a secure environment.

All of the Above

A picture is worth a thousand words. Names have been changed to protect the guilty.

This customer was doing all of the above. The CAS was in a DMZ, though according to the customer "It's not really a DMZ...Look, we even labeled it something different on the visio!" Additionally, they had Client Access Servers "designated" as "internal" or "external" in the same AD site.

Guess what? They ended up on the phone with us! (shocking I know).

And guess what the problem was? ActiveSync was attempting to proxy from the CAS in Data Center A's DMZ to the CAS in Data Center B's DMZ, but this traffic was not allowed. The customer intended for the proxy traffic to only go the the HUB/CAS combo servers in the Internal Network in data center B. But it didn't (obviously) and the firewall/routing caused the request to be blocked and fail.

This is an illustration of the simplest of all crazy issues you can and will hit if you deploy this way, but hey - maybe you're weird and just love problems or perhaps you just love calling into PSS. Either way - happy deployments!

I'm sure I'll be adding to this list as time goes on (or as I think about more things). Please spread the word, help our customers, and don't let them get into this situation. This goes out to the field, partners, MVP's, etc - It's your job to help our customers make the right call; together we can make a difference.

Autodiscover HTTP Redirect Method using ISA instead of a second web site

2007-12-18T23:12:00Z

Many customers that are hosting multiple SMTP domains need to use one of our Autodiscover "redirect" methods to deploy autodiscover instead of the pre-defined URL approach. When customers choose a "redirect" method they have two options. Below is a chart with both options and some Pro's and Con's of each.

Method

Pros

Cons

HTTP Redirect

Works in all scenarios

Works with all Outlook 2007 Client versions

Complex to implement

Complex to maintain

Requires 2 public IP's

DNS SRV Record

Very easy to implement

Requires only 1 Public IP Address

Some DNS Providers don't support DNS SRV records

Doesn't work in 100% of client scenarios (explained later)

Requires Outlook 2007 SP1 (or Post RTM hotfix)

The DNS SRV Record method was introduced by an Outlook 2007 Post-RTM hotfix and is also included in Outlook 2007 SP1.:
940881 A new feature is available that enables Outlook 2007 to use DNS Service Location (SRV) records to locate the Exchange Autodiscover service
http://support.microsoft.com/default.aspx?scid=kb;EN-US;940881

This method is great except for its few drawbacks. First, you have to make sure all of your clients have this fix or SP1. In some environments, control of the client is not possible so this may discount this option immediately. Another consideration with the SRV record method is whether or not *client* environments support the DNS SRV records. For instance, if your only internet access and/or name resolution is through a proxy (or ISA Firewall Client), you have no way of resolving SRV records.

With these drawbacks in mind, I wanted to bring light to the original method of redirecting autodiscover clients, the HTTP redirect method. The implementation of this method via IIS is discussed in by Jason Mayans at http://blogs.technet.com/jmayans/archive/2006/09/07/454716.aspx.

After reading that, you should have a good idea that load balancing and hosting this extra redirect IIS site can be a nightmare. Luckily, you can do it ALL at ISA! Well 04 and 06 at least, I don't care if ISA 2000 can do it or not. You still need an extra public IP, but the ISA redirection works great and if you have an array of ISA servers, it's automatically load balanced!

First, bind that new public IP to ISA. Then start by creating a new Web Publishing Rule.

The Rule action will actually be set to Deny (We'll configure the redirect later)

The rest of this stuff is mostly irrelevant since it's a Deny Rule. I chose:

"Publish a single Web site or load balancer"
"Use non-secured connections to connect to the published Web server or server farm"
"this.does.not.matter" for the "Internal Site Name"

Set the path to the actual Autodiscover.xml file: /autodiscover/autodiscover.xml

For the public name, you can set it to either the first of your hosted SMTP domains or just set the "Accept Requests for:" to "Any".

You'll want to create a new HTTP web listener, call it something like Autodiscover Redirect Listener.

Choose "Do not require SSL secured connections with clients" (very important as this redirect must be HTTP).

Select the new IP you bound to ISA for the Autodiscover Redirect listener.

Choose "No Authentication" as these requests will always be anonymous.

Ensure your new listener is selected in the wizard.

For the Auth delegation, you can choose "No delegation, and client cannot authenticate directly"

The rule cannot require authentication, so ensure "All Users" is selected.

Complete the rule.

You'll need to bring up properties on the rule and proceed to the "Action" tab. It should be set to deny. Put a check in the "Redirect HTTP requests to this Web Page" and enter the full HTTPS URL (matching your certificate), a which you have autodiscover published. Mine was https://mail.wingtiptoys.com/autodiscover/autodiscover.xml.

On the "Public Name" tab, you will need to add autodiscover.<each hosted smtp domain>

Finally in each hosted SMTP domain's public DNS zone, create an A record for Autodiscover and point it to this new IP on the redirect site.

You can test this by browsing to http://autodiscover.hosteddomain.com/autodiscover/autodiscover.xml in your browser.

It should redirect you to the URL you specified in the rule.

On launch of Outlook 2007, you should end up with a one-time dialog asking the user to allow this URL to configure their Outlook profile if you've done this right.

Handy way to restart downed Exchange Services (and only the ones you need)

2007-11-21T01:02:00Z

Here's a quick Powershell one-liner to start up all the necessary Exchange Services for any given role.

Test-ServiceHealth | Select-Object -Expand ServicesNotRunning | Start-Service

There is also the old standby:
Get-Service *Exchange* | Start-Service

But that one can start unnecessary services (set to manual) like the Exchange Pop3 and Imap4 service.

Just thought it'd make and interesting post because not too many people know about the Test-ServiceHealth cmdlet.

CAS Load-Balancing Best Practices (Part 2)

2007-10-29T12:57:00Z

CAS Load Balancing Best Practices (Part 2: Load balancing Intranet aka "Non-Internet facing" sites):

For Part 1 of this post, go here: CAS Load-Balancing Best Practices (Part 1)

You need to understand CAS->CAS Proxying before you try to understand load-balancing proxy CAS's. Please read the following technet documentation if you don't understand CAS->CAS Proxy before proceeding:

Understanding Proxying and Redirection

The first big question is whether or not load balancing is required in the second site. Originally it was thought that the CAS in the first site would do load balancing and redundancy for the individual CAS's at an application layer. We later found out that this is 100% true for OWA and Exchange Web Services Proxy but untrue for Activesync and Unified messaging proxy.

So, to get full load-balancing and redundancy for all services, you must deploy load balancing in the proxy sites as well as pictured below:

The trickiest part to this is not conifguring the NLB, it is knowing how to configure all of the URL parameters for the destination CAS.

Our documentation topic on technet "Understanding Proxying and Redirection" covers this in some detail. I'm leveraging our chart below for reference on how to set the individual parameters for the Virtual Directories.

Virtual directory settings for non-Internet-facing Client Access servers in an organization that uses NLB

Virtual directory	InternalURL setting
/OWA	https://computername/OWA
/OAB	https://NLBname/OAB
/UnifiedMessaging	https://NLBname/UnifiedMessaging
/Microsoft-Server-ActiveSync*	https://NLBname/Microsoft-Server-ActiveSync
/EWS**	https://NLBname/EWS

* NOTE: This does NOT work in RTM for ActiveSync due to some Keberos issues. You must set the InternalUrl to https://computername/Microsoft-Server-Activesync and you get no fault tolerance if a CAS in the seond site goes down. We corrected these Kerberos issues in SP1 so you should be able to set the InternalUrl to the NLBname at that time.

** NOTE: EWS does not use the InternalUrl for proxying core EWS traffic and instead uses the InternalNLBBypassUrl configured on the Virtual Directory (this should always be configured to the machine FQDN). Proxying for the Availability service will use the InternalUrl but affinity is not required.

Note in regard to Affinity settings
OWA and EWS are "stateful" applications. Because of this, they must maintain the connection with the same CAS throughout a users session. However, because OWA and EWS use application-level load-balancing and failover and do not route through your load-balancer in the remote site, Affinity on the load balancer should not be an issue. EAS and UnifiedMessaging are stateless and can be handled by any CAS on subsequent requests so for them, the load balancing affinity can be set to "None" in the proxy site to ensure even distribution of requests.

Have fun deploying your CAS Proxy + NLB Solutions!

New Updates to Autodiscover Whitepaper available

2007-10-15T15:54:00Z

Just wanted to let folks know that a new update to the Autodiscover Whitepaper is now available. This now contains tons more information regarding certificates, web service URL's, best practices, etc.

You can find the whitepaper here:
http://technet.microsoft.com/en-us/library/bb332063.aspx

You can find the Exchange team blog announcment here:
http://msexchangeteam.com/archive/2007/10/03/447176.aspx

The only thing it doesn't really cover right now is load-balancing, for that you can see my series on CAS load-balancing here: http://blogs.msdn.com/brad_hughes/archive/2007/09/10/cas-load-balancing-certificates-autodiscover-and-webservices.aspx

AutodiscoverSiteScope and Client-only sites

2007-09-11T16:53:00Z

The Exchange Autodiscover service is very smart when it hands out URLs for web services to clients. It will always hand out URLs that are in the same Active Directory site as the user's mailbox server. Unfortunately, when it comes to Outlook locating Autodiscover originally, the client just isn't that smart. The way a domain-joined client can find a local Autodiscover service is by utilizing the AutodiscoverSiteScope information that is part of the Service-Connection-Point (SCP) object in Active Directory.

This AutodiscoverSiteScope attribute must be populated manually using the Set-ClientAccessServer cmdlet.
Set-ClientAccessServer -identity <name of CAS> -AutodiscoverSiteScope Site1,Site2,Site3

At install, this attribute is populated with the AD Site of it's corresponding CAS. That way any clients that are in the same AD Site as the CAS will be able to use the local server to make Autodiscover Requests.

The problem occurs when you have AD Sites that only have Outlook clients (and not Exchange Servers). By default, they will just use a Random CAS for Autodiscover (or the first one comes back from AD in the LDAP query). To select a site for these clients to use, you can add the "Client site" to the AutodiscoverSiteScope Attribute on each CAS in the closest AD site. That way you conserve bandwidth and improve performance by keeping Autodiscover traffic relatively local.

For more information on configuring the AutodiscoverSiteScope attribute and it's uses, see: How to Configure the Autodiscover Service to Use Site Affinity.

I was recently presented with the situation of a customer who had over 2000 AD sites with Outlook clients but a relatively small number of sites with Exchange servers. They were faced with the potential of having to add thousands of AD Sites to the AutodiscoverSiteScope by hand. This got me to thinking that this could be automated with a small amount of effort.

Below is a sample of automating that process. This PowerShell script makes use of a command-line tool (CalcSiteCost.exe) that uses the NTDS API's DsQuerySitesByCost to calculate the cost between each client site and each Exchange Site.

The syntax used for CalcSiteCost.exe is: CalcSiteCost.exe fromSite toSite1 [toSite2 toSite3 ...]. The results are returned as simple integers (one per line) that are printed to the console. The PowerShell script parses the console output and casts each one to an integer and adds it to an int[].

Both the sample script and the sample source and binary for CalcSiteCost are available as an attachment to the post.

# Check for verbose param
if($Args -contains "-verbose" -or $Args -contains "-vb")
{
$VerbosePreference = "Continue";
}

# load required .NET Assemblies
$temp = [System.Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices")

# Get current forest info
$forest = [System.DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest();

# Build list of sites that have Exchange Servers
[string[]]$exchangeSites = @();

Get-ExchangeServer | Where {$_.serverrole -contains "ClientAccess"} | foreach { $exchangeSites += $_.Site.Name; }

# or you can manually set the Exchange sites if you prefer
# $exchangeSites = "Charlotte","Redmond"

# Loop through each Exchange Site
foreach($clientsite in $forest.sites)
{

$leastCostExchangeName = "";
$leastCostExchangeCost = 99999;

[int[]] $costArray = @();
.\CalcSiteCost.exe $clientsite.Name $exchangeSites | foreach { $costArray += [Int32]::Parse($_); }

# find the lowest cost and note the site
for($i=0; $i -lt $costArray.Length; $i++)
{
  if($costArray[$i] -lt $leastCostExchangeCost)
  {
   $leastCostExchangeCost = $costArray[$i];
   $leastCostExchangeName = $exchangeSites[$i];
  }
}

write-host ("Least cost Exchange Site for client site " + $clientsite.Name + " is " + $leastCostExchangeName + " Cost: " +$leastCostExchangeCost);

# Get the list of
$closestCASList = Get-ExchangeServer | Where {$_.serverrole -contains "ClientAccess"} | where {$_.site.Name -eq $leastCostExchangeName};

foreach($closeCas in $closestCASList)
{
  # Get existing attribute
  $clientsiteScopeList = (Get-ClientAccessServer -identity $closeCas.Name).AutoDiscoverSiteScope;

  # big special case for $null
  if($clientsiteScopeList -eq $null)
  {
   Set-ClientAccessServer -identity $closeCas.Name -AutoDiscoverSiteScope "";
   $clientsiteScopeList = (Get-ClientAccessServer -identity $closeCas.Name).AutoDiscoverSiteScope;
   $clientsiteScopeList.Clear();

  }
  # If It's not already in the list, add it.
  if($clientsiteScopeList -notcontains $clientsite.Name) { $clientsiteScopeList.Add($clientsite.Name); }

  # Set the updated list back to using Set-ClientAccessServer
  Set-ClientAccessServer -identity $closeCas.Name -AutodiscoverSiteScope $clientsiteScopeList
}

}

CAS Load-Balancing Best Practices (Part 1)

2007-09-10T17:05:00Z

There just isn't much in the way of How-to / Best Practices type of information regarding load Balancing Client Access Servers in Exchange 2007. Soon we should be releasing additional Technet content relating to the deployment of Autodiscover and Web Services in the form of an Autodiscover White Paper, but this content doesn't include any information regarding best practices around load balancing Autodiscover and Exchange Web Services. This information will be crucial for most medium to large companies migrating to Exchange 2007.

Below are my thoughts on some best practices when it comes to load balancing these Client Access Web Services. I will be posting a series of posts dedicated to this topic.

CAS Load Balancing Best Practices (Part 1: Internet-facing: NLB, Authentication, InternalUrls, ExternalURLs, Certificates):

Background:
AD Sites that contain Exchange 2007 servers should be thought of in two ways. Internet-facing sites and intranet (non-Internet facing sites).

Best Practice #1: ALL Client Access Servers within the same Internet-facing site should be:

In an NLB of some sort. Whether Windows NLB, ISA 06 NLB, or Hardware NLB.
Have ALL ExternalURL's exactly the same with the public name of the NLB.
Have ALL InternalURL's exactly the same with the private name of the NLB.
ALL be configured for the same types of authentication for each HTTP based service.

Why do all of them have to be in the NLB?

The real answer here is to eliminate confusion. Many Exchange 2007 Services (such as Autodiscover) choose Client Access Servers at random for clients to connect to. If they are all in the same site, they basically server the same function and thus should be in the same place. If you have some extenuating circumstance like needs to have 2 NLB's (one for internal OWA and one for external OWA), this guidance may not apply to you. However, you should adhere to the guidance below of having a different private name for each NLB of Client Access Servers.

What is this public name versus private name stuff?

Public Name: This should be the name that the outside world uses to access your Outlook Web Access, Exchange Activesync, and/or Outlook Anywhere services for mailboxes in this Internet Facing site. This would be a manually created DNS entry in your public DNS zone. For instance: https://usamail.contoso.com/EWS/Exchange.asmx might be my ExternalUrl for all my WebServicesVirtualDirectory objects in Contoso Corp's USA Site.

Private Name: This should be a manually created DNS name within your internal DNS zone. This will primarily be used for Exchange Web Services, so you may want to choose a name like ews.ad.contoso.com or websvcs.ad.contoso.com (Notice ad.contoso.com is within my internal DNS namespace). Likewise: https://ews.ad.contoso.com/EWS/Exchange.asmx might be the InternalUrl for all the WebServicesVirtualDirectory objects in Contoso Corp's USA Site. If you have two different NLB's within the same site for different purposes, you should have two different private names with the InternalUrl's on the corresponding CAS's modified accordingly.

What about certs?

The only real way to do this is to obtain a 3rd party certificate that supports Subject Alternative names. For a list of Certificate Authorities offering these type of certificates, check this KB article: http://support.microsoft.com/kb/929395/en-us. In most cases, you will only need 3 Subject Alternative Names on a certificate. I suggest:

mail.contoso.com (or whatever name you want to use for OWA, EAS, and RPC/HTTP. This should also be the "Issued to" or common name)
autodiscover.contoso.com (you will need one of these for every primary SMTP address domain in your organization).
ews.ad.contoso.com (one Subject Alternative Name that exists in your internal DNS namespace).

For more information on generating the certificate request see: Creating a Certificate or Certificate Request for TLS on Technet.

What do you mean by have the same authentication for every HTTP service?

For each Virtual Directory in IIS, you should have consistent authentication methods selected for every CAS in your NLB. If you don't, you'll run into goofy issues when a client that was talking to a certain CAS gets load-balanced to a different CAS. This doesn't mean that just because you use Basic Auth for OWA that you have to use Basic for Outlook Anywhere. It just means that if you use Basic on Outlook Anywhere on one CAS, you need to make sure that Basic is configured for every other CAS in the NLB. This should be common-sense.

I'll cover additional topics such as Client Affinity, Intranet CAS Deployments, etc in later posts.

For the next post in this series, see: CAS Load-Balancing Best Practices (Part 2)

Ah, leftovers from DST...

2007-09-06T17:23:00Z

Yesterday I got looped into a call where a customer that reported that meeting requests were ahead by an hour when he sent them to the Internet. Knowing this had to have something to do with Timezones and DST, I whipped together a quick tool to Display UTC time, Local time, and some other time zone information.

We can always verify what's in the Windows Time Zone Database using TZEdit but not information on the current time zone. You could also pull this from the registry HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\TimeZoneInformation if you know how to interpret the data, but this makes it much easier. But in addition to all this, I would've never figured out this issue without the UTC display from this tool.

In the customer's case, the UTC time showed he was an hour ahead of the rest of the world in his domain but because he had hacked up his timezone information to display the local time correctly, he had no idea his UTC time was wrong.

Here is some sample output:

UTC System Time: 9/5/2007 21:47
Local System Time: 9/5/2007 17:47
Operating in DAYLIGHT time
Time Zone Name: Eastern Daylight Time
Daylight Bias: -60 Minutes (or -1 Hours)
Standard: Occurs first Sunday of November at 2: 0
Daylight: Occurs second Sunday of March at 2: 0
Bias: 300 Minutes (or 5 Hours)

For anyone who's interested, here is the source of the main function. It demonstrates the use of the GetSystemTime, SystemTimeToTzSpecificLocalTime, and GetTimeZoneInformation functions.

int _tmain(int argc, _TCHAR* argv[])

{

SYSTEMTIME utctime;

// Apparently this function cannot fail - no need to check error case

GetSystemTime(&utctime);

wprintf(

L"UTC System Time: %i/%i/%i %i:%i\r\n",

utctime.wMonth,

utctime.wDay,

utctime.wYear,

utctime.wHour,

utctime.wMinute

);

SYSTEMTIME localtime;

BOOL fResult = SystemTimeToTzSpecificLocalTime(

NULL, // current timezone

&utctime, // input time

&localtime // output time

);

if(!fResult)

{

wprintf(L"SystemTimeToTzSpecificLocalTime failed.\r\n");

return -1;

}

wprintf(

L"Local System Time: %i/%i/%i %i:%i\r\n",

localtime.wMonth,

localtime.wDay,

localtime.wYear,

localtime.wHour,

localtime.wMinute

);

TIME_ZONE_INFORMATION tzInfo;

DWORD dwResult = GetTimeZoneInformation(&tzInfo);

switch(dwResult)

{

case 0:

wprintf(L"Operating in UNKNOWN time\r\n");

return -1;

case 1:

wprintf(L"Operating in STANDARD time\r\n");

wprintf(L"Time Zone Name: %s \r\n", tzInfo.StandardName);

wprintf(L"Standard Bias: %i Minutes (or %i Hours) \r\n", tzInfo.StandardBias, tzInfo.StandardBias / 60);

break;

case 2:

wprintf(L"Operating in DAYLIGHT time\r\n");

wprintf(L"Time Zone Name: %s \r\n", tzInfo.DaylightName);

wprintf(L"Daylight Bias: %i Minutes (or %i Hours) \r\n", tzInfo.DaylightBias, tzInfo.DaylightBias / 60);

break;

default:

return -1;

}

wprintf(L"Standard: Occurs %s %s of %s at %i:%2i\r\n",

GetOccurrenceString(tzInfo.StandardDate.wDay),

GetDayString(tzInfo.StandardDate.wDayOfWeek),

GetMonthString(tzInfo.StandardDate.wMonth),

tzInfo.StandardDate.wHour,

tzInfo.StandardDate.wMinute

);

wprintf(L"Daylight: Occurs %s %s of %s at %i:%2i\r\n",

GetOccurrenceString(tzInfo.DaylightDate.wDay),

GetDayString(tzInfo.DaylightDate.wDayOfWeek),

GetMonthString(tzInfo.DaylightDate.wMonth),

tzInfo.DaylightDate.wHour,

tzInfo.DaylightDate.wMinute

);

wprintf(L"Bias: %i Minutes (or %i Hours) \r\n", tzInfo.Bias, tzInfo.Bias / 60);

_getch();

return 0;

}