Clustering and High Availability

Updates for Windows Server 2008 Failover Cluster and Hyper-V

There is a new update available for download that enhances the management experience of Hyper-V with the Failover Cluster Management UI (CluAdmin.msc).  This update incorporates several changes into a single package. 

KB article 951308 (http://support.microsoft.com/kb/951308), Increased functionality and virtual machine control in the Windows Server 2008 Failover Cluster Management console for the Hyper-V role, provides detailed information about this fix and links to request the fix.

This package is also available on the http://microsoft.com/downloads website.  Search for 951308 and it will display the versions of the update for x64 and x86.  The update can also be installed on Vista SP1 if you have the Remote Server Administration Tools (RSAT) installed.

The fixes in this package include:

The following screenshot highlights a couple of these changes:

1.       The Virtual Machine Configuration resource shows as a sub item of the Virtual Machine resource that it’s associated with.

2.       Controls are the same as Hyper-V Manager, instead of offline/offline the controls for the Virtual Machine are Start/Turnoff/Shutdown/Save

3.       If you right click on the disk resource and select “properties”, the path to the disk can be copied.  This is very helpful if the path is a GUID.  This path is what should be used in hyper-V manger when specifying where the VM configuration and VHD’s are placed.

Thanks,

Steven Ekren

Senior Program Manager

Clustering & High Availability

Virtualization: Achieving High Availability for Hyper-V

Hi Cluster Fans,

One of the Clustering and High Availability team’s Senior Program Managers, Steven Ekren, recently published an article in TechNet Magazine on Virtualization: Achieving High Availability for Hyper-V (http://technet.microsoft.com/en-us/magazine/cc837977.aspx).  As one of Microsoft’s leading experts in HA for Hyper-V, Steven has provided some great information which both beginners and pros can appreciate.

Here’s an overview:

Server virtualization is poised to make a significant impact in enterprise IT departments, and Hyper-V with Windows Server 2008 can make it a reality. The consolidation of servers onto fewer physical machines has huge advantages in resource and cost savings, but two key factors need to be considered during the planning process. Users have increasing expectations regarding the availability of their software, including both line-of-business (LOB) applications and tools such as messaging and collaboration platforms. Furthermore, problems or failure on servers can have a significantly greater impact on operations. Windows Server 2008 and Hyper-V provide solutions that can be implemented to provide high availability (HA) of virtual machines (VMs) as well as to the workloads being hosted inside the VMs.

We hope you enjoy this article!

Thanks,

The Clustering and High Availability team

How to create the cluster.log in Windows Server 2008 Failover Clustering

Windows Server Failover Clustering logs information about cluster activities including normal operations like updates between nodes as well as errors and warnings related to problems that occurred on the cluster in a text file called cluster.log.  The information in the cluster.log is very valuable when trying to troubleshoot just about any problem encountered with a cluster.  Having been in Microsoft’s support team that helps customers resolve Failover Cluster problems, I can tell you that having this detailed logging makes the difference of being able to identify the cause of a problem at the first occurrence, instead of having to configure logging after the fact and wait for the problem to happen again. 

As valuable as it is, there is room for improvement for the Windows Server 2003 cluster.log.   The text-based file format means using a text editor/viewer to parse the log, which is not ideal when dealing with very large logs.

For those of you that don’t want all the gritty details, I’m going to first provide a quick outline of how to get the equivalent of the cluster.log from a Windows Server 2008 Failover Cluster and some background, and then get into some of the details.

CREATING THE CLUSTER.LOG:

From one of the nodes of the cluster, open a Command Prompt with Administrator rights.  The simplest command to create the log is to type “cluster log /g”.  A clusterl.log file will be generated and stored in the %windir%\Cluster\Reports directory on each node of the cluster.  Note that with all commands you can use either “cluster … ” or “cluster.exe …” as they have the same functionality.

Here are some cool new commands that can make this even easier:

·         /Copy:<directory> (example: /Copy:logs)  This command will take the cluster.log that is generated on each node, and copy it to a single directory.  This makes it incredibly easy to get all the logs for analysis.  One thing to note, the directory that you specify should be a subdirectory under the path which the command prompt is showing.  If you want to save the logs at c:\archive\logs, then you need to set the command prompt to c:\archive and then execute the “cluster log /g /copy:logs” command. 

·         /Span:<minutes> (example /Span:15).  This specifies the number of minutes to go back in time for the log collection.  For instance, you reproduce a problem and you then generate the cluster.log.  If you don’t use this switch, you will get up to several days of history.  Using this switch, you can limit the contents of the cluster.log to only include the last few minutes which you have specified.  So, what if you specified 15 minutes but it was really 20 minutes before?  No problem, generating the cluster.log does not remove any data from the servers.  You just run the command again specifying additional minutes for this /span option.

·         /Node:<node name> (example /Node:”node A”).  This command allows the specification of a specific node and the other nodes will not have a log generated.  If this option is not specified, all nodes in the cluster will have a cluster.log generated.  This is particularly useful if not all the cluster nodes are up, or some don’t have the cluster service started, which can cause a long delay with cluster log command execution because it will try to issue the command to those missing servers and will wait for a response when none will be forthcoming.

·         /Level:<0-5> (example /Level:4)  A note about another “cluster log” option that you may find information on in the help output for the command.  The /Level switch can be used to change the logging level being captured.  For Windows Server 2008, this has a default level of 3, which is the equivalent of what is captured by cluster.log in previous versions of Windows Server.  If you change this level to a higher number, more detailed information will be logged, but that means that the .etl file that is capturing the tracing will fill faster and there can be a small impact on system performance.  Setting this level lower than 3 will mean there is less tracing information and it may not be useful if analysis of a problem is needed.  For Windows Server 2008, 5 is the maximum effective level, although the command help notes that the level can be set between 0 and 10.  Any setting over 5 has the equivalent functionally as 5.   The level range was set to 10 to allow for further options if needed in the future.

THE GORY DETAILS:

Windows Server 2008 introduced new eventing and diagnostic channels and Failover Cluster moved to using ETW (Event Tracing for Windows).  You can see this new tracing exposed in the “Reliability and Performance Monitor” under “Data Collector Sets\Event Tracing Session\Failover Clustering” (shown below).

The logging is saved in files at %windir%\System32\winevt\logs\Clusterlog.etl

Each time the server is rebooted, a new log file will be used and a number used as an extension of the log name like ClusterLog.etl.001.  Up to 5 log files are kept, so after 5 reboots the older log files will start to be removed.  The default log file is 100 MB (for each .etl file), which can be changed using the command “Cluster log /size:<size in MB>” (example:  cluster log /size:120).  Although 100 MB may seem like a large log file, there is a significant amount of detail being saved for each entry due to this format change and 40 MB provides a reasonable amount of history.  To view the setting for the log file size setting, at a command prompt opened with Administrator privileges execute “cluster /prop”.  That command will list the properties for the cluster, including the “ClusterLogSize” and “ClusterLogLevel” property.

The .etl files themselves are not consumable by any viewer directly, but you can dump the contents into several different formats using tracerpt.exe (this TechNet article has the information on using tracerpt.exe: http://technet.microsoft.com/en-us/library/bb490959.aspx).  You can dump the contents to EVTX and view in Event Viewer, or .XML and manipulate the information in many ways.  For instance, you can apply a script that parses the file and provides formatting to a subset of the events.  Here is an example of how the display can look:

As you can see, the new logging mechanism provides more options and flexibility.  I personally love the /copy and /span options, they make targeting a specific time frame and getting the logs from all nodes much easier.

Thanks,

Steven Ekren

Senior Program Manager

Clustering & High Availability

Share Subdirectories, Where Hast Thou Gone?

You may have noticed that in Windows Server 2008 that the “Share Subdirectories” option for file shares on a Failover Cluster is no longer around…why would we do that?

First off, let’s take a stroll down memory lane and talk about the history of this feature and how it came to be…

Let’s timewarp back 10 years to the early clustering days.  In Failover Clustering (or MSCS in those days) there are cluster resources.  Resources control different things. In the case of File Shares, there was a “File Share resource” which controlled creating the SMB share for the cluster node it was being created on.  For each File Share, there needed to be a corresponding “File Share resource”…  a 1:1 mapping where each resource controlled one share.

This seemed fine, until someone wanted to create a huge, scaled up File Server with lots of shares (seemed reasonable).  We then hit a problem, the cluster Resource Monitor architecture only supported up to 900 resources in a single cluster in NT4.  So in Windows NT 4.0 Service Pack 4 we did some improvements to support up to 1674 resources in a cluster.  A great improvement, but still not enough to support the thousands of file shares that some customers needed.  So we introduced the “Share Subdirectories” feature as a way to work around the problem.  Basically, what it does is share one folder, and then all the subdirectories of that folder are automatically shared.

We limped along with this model, and it lasted for a solid decade.  When it came time for Windows Server 2008, we decided we wanted to improve this, and fix it the right way.  For starters, we re-wrote the Resource Monitor (ResMon) and replaced it with the super charged Resource Hosting Subsystem (RHS).  With that we resolved all the limitations around the numbers of resources a cluster could host in Windows Server 2008 (yes, no hard limitations now).  But before we declared victory, we wanted to put it to the test and do some serious scalability testing by creating tens of thousands of File Shares in a single cluster.  As we did this, we realized it became very painful to manage tens of thousands of resources… so we started to rethink the entire design.

We realized that in the context of manageability, performance, scalability, and usability, a 1:many relationship was much better than a 1:1. We replaced the legacy “File Share” resource, with the new “File Server” resource.  When we did this redesign, we looked long and hard at the “Share Subdirectories” option and what value it provided.  As I said before, it was originally a workaround to the limitations on the number of resources we could support in a cluster…that is now fixed  in Windows Server 2008.  In the new File Server model, we had a 1:many relationship.  So it turns out that all of the reasons the “Share Subdirectories” existed were solved in better ways which resulted in greater efficiencies within the cluster itself.  Note:  We also did a ton of performance optimizations in the new design, and MSIT has seen dramatic reductions in failover times… but that’s a story for another blog.

One central theme for Failover Clustering in Windows Server 2008 was to make clustering simple (“clustering for mere mortals”, as I like to call it), to remove all those “special” cluster considerations.  For example, to create a File Share on a server normally you would open Windows Explorer, right-click and select Share.  BUT WAIT!  If you are on a cluster, something so simple can’t work!  You have to go open a certain tool (CluAdmin.exe) and do a special process to create all these File Share resources.  And, if you miss one step in the long winded whitepaper, it probably won’t work.  At the end of the day, you couldn’t even trust a junior admin to do something as simple as setting up a file share on a cluster.  How silly…

We wanted to fix that. One of our major goals was to break down all those “special” considerations.  We wanted to achieve parity with the core Windows operating system.  The 'mantra' became - what you can do in a normal server… should be the same in a cluster, including the ways you do it.  We did work to totally integrate clustering. No longer was it necessary to manually create resources.  You could simply open Windows Explorer, use NET SHARE from the command line, programmatically call the NetShare() API’s, or use any 3^rd party tool you like.  The tool didn’t even have to be ‘cluster aware’, we handled all the logic under the covers.  The Share Subdirectories option is not available on stand-alone servers. Our goal is to have parity and no special cluster considerations so it didn’t make sense to make shares on clusters behave differently.

The Share Subdirectories option also wasn’t very flexible because you had to build your directory structure in a special way… as only subdirectories under a root were shared.  Once again, one more thing that required more special cluster considerations and planning.  File servers needed special planning and design to get the features to work.

There were also security concerns.  Share Subdirectories inherited the share level permissions of the root share, which meant that you couldn’t use share level permissions and had to depend on file system level permissions - not very flexible.  It could also be viewed as a security risk.  There was no other feature or option in Windows that doing something as simple as creating a folder would automatically share it and make its contents available on the network.  Sharing a folder requires higher level permissions on the system, then creating a folder.  This could allow unprivileged users to create shares on the network when you don’t want them to.  Yet again, more considerations on why dealing with clusters is hard, complex, and one has to be really careful.

With all of that in mind… we decided that Share Subdirectories had served its purpose, and it was time retire the functionality.  So are you left high and dry?  NOT AT ALL!  There are many ways to still accomplish exactly what you had for the last 10 years…let’s discuss.

The new File Server resource is a 1:many relationship, much like the File Share resource with Share Subdirectories was (but it’s better).  So let’s say you have a file server deployed today using the Share Subdirectories option to share out thousands of home directories for users.  The one and only consideration is getting the thousands of file shares created the first time.  Once they are created, they persist across failover. 

There are several options for enabling a large numbers of shares:

NET SHARE the Subdirectories

If you have the directory structure setup using Share Subdirectories where you want to share all the folders that are subdirectories, then run the single line syntax below and it will share all folders in a directory path.  Once they are created, no other actions are needed (you only need to run this single line syntax below ONCE).

How share a large number of child directories:

1.       Open cmd.exe and change to the directory where all sub directories are to be shared.

2.  Type the following:    for /d %d in (*.*) do Net Share %d=G:\%d

a.       Note:  Be sure to change G:\ to the appropriate drive and path to share.

'Migrate a Cluster' Wizard

Use the ‘Migrate a Cluster’ Wizard build right into Windows Server 2008 Failover Clusters, to migrate the file shares from a Windows Server 2003 cluster to a new Windows Server 2008 Failover Cluster.  The Migration wizard will handle creating the new File Server resource and all the shares.  Here’s a step-by-step guide with more information:

http://technet.microsoft.com/en-us/library/cc754481.aspx

File Share Migration Toolkit

Use the File Share Migration Toolkit.  This is a tool for migrating file shares, and is completely cluster aware.  Note:  You need the 1.1 version which is currently in Beta for Windows Server 2008 support.

http://www.microsoft.com/windowsserver2003/upgrading/nt4/tooldocs/msfsc.mspx

Once the shares are created, just do what you would normally do.  Open Windows Explorer, script with NET SHARE, etc…  Just use the exact same processes and procedures you use on stand-alone servers as there is absolutely no difference now in Windows Server 2008 Failover Clustering.

I hoped this sheds some light into why things were the way they were, why we made these decisions, and what your options are.  Share Subdirectories served us well, and the feature will be missed.  I know change is hard, but I sincerely believe that Windows Server 2008 Failover Clustering is the best highly available file server platform yet. 

Thanks!
Elden Christensen

Senior Program Manager Lead

Windows Clustering & High Availability

Microsoft Enterprise Server Products

Migration Options for Hardware with Failover Clustering in Windows Server 2008

Hi Cluster Fans,

Failover Clustering in Windows Server 2008 no longer supports the traditional “rolling upgrade” which you may be familiar with from Windows Server 2003.  This has given the impression that you will not be able to reuse your hardware.   That is not true – with clustering’s built-in ‘Migrate a Cluster Wizard’, you have the flexibility to reuse your hardware, reuse your storage, use new hardware or use new storage. 

To add some background, let’s first review the “rolling upgrade” process.

“Rolling Upgrade” from 2000 to 2003

Here’s an overview of the process which was used in the past to allow you to reuse your hardware:

·         You have 1 cluster with 2 nodes: Node1 (2000), Node2 (2000)

·         Move all your resources to Node2 (2000)

·         Pause Node1 in the cluster, then upgrade Node1 (now 2003)

·         Move all your resources to Node1 (2003)

·         Evict Node2 (2000) from the cluster then upgrade Node2 (now 2003)

·         Now all of your 2000 resource groups will be on your new Windows Server 2003 cluster: Node1 (2003), Node2 (2003)

If you want to use new hardware, just join those new nodes (2003) to the old cluster (2000), move the resources to the new nodes (2003), then evict the old nodes (2000).  Now all of your Windows 2000 Server resource groups will be on your new Windows Server 2003 cluster.  More information is available here: http://technet.microsoft.com/en-us/library/cc782024.aspx

Failover Clustering does NOT support this upgrade/migration path in Windows Server 2008 due to the significant architectural changes which breaks backwards compatibility (you cannot have a 2003 node and a 2008 node in the same cluster).  But don’t worry – our Migrate a Cluster Wizard will help you out!

Migrate a Cluster Wizard from 2003 to 2008

When you migrate from a Windows Server 2003 cluster to Windows Server 2008 cluster you should use the built-in Migrate a Cluster Wizard.  If you wish to reuse the same hardware (including storage hardware), each component must be logoed for Windows Server 2008 and pass the built-in “Validate a Configuration…” tests. 

If you want to use new hardware, here’s an overview of the migration process:

·         You have 2 clusters: OldCluster (2003), NewCluster (2008)

·         In the Windows Server 2008 Failover Clustering Management snap-in (CluAdmin.msc), connect to NewCluster (2008)

·         From NewCluster (2008), click ‘Migrate Services and Applications…’ to launch the Migrate a Cluster Wizard

o   Point to your Windows Server 2003 cluster, OldCluster (2003), and walk through the wizard

o   If you want to use new storage, select the new disk in Available Storage, otherwise it is assumed that you are reusing your storage

o   Complete the Migrate a Cluster Wizard

·         Now all of your 2003 resource groups will be on NewCluster (2008)

o   Each resource group will need to be onlined to start functioning

But what if you want to reuse your hardware?  Well although we do not support the traditional “rolling upgrade” you can perform a hybrid migration/upgrade, allowing you to reuse all your hardware, assuming each component is logoed for Windows Server 2008 and will pass the built-in “Validate a Configuration…” tests.

If you are reusing all your hardware, here’s an overview of the migration process:

·         You have 1 cluster (OldCluster, 2003) with 2 nodes: Node1 (2003) and Node2 (2003)

·         Failover every resource group to Node2 (2003)

·         Evict Node1 (2003) from OldCluster (2003)

·         Upgrade Node1 (2003) to Windows Server 2008 (Node1 is now 2008)

·         Create a 1-node cluster on Node1 (2008), which we will call NewCluster (2008)

·         From NewCluster (2008), click ‘Migrate Services and Applications…’ to launch the Migrate a Cluster Wizard

o   Point to your Windows Server 2003 cluster, OldCluster (2003), and walk through the wizard

o   If you want to use new storage, select the new disk in Available Storage, otherwise it is assumed that you are reusing your storage

o   Complete the Migrate a Cluster Wizard

·         Now all your 2003 resource groups will be on NewCluster (2008)

o   Each resource group will need to be onlined to start functioning

·         After you are sure that the resources are stable in 2008, destroy OldCluster (2003)

·         Upgrade Node2 (2003) to Windows Server 2008 (Node2 is now 2008)

·         Add Node2 (2008) to NewCluster (2008)

·         You now have a 2-node cluster, NewCluster (2008), running Windows Server 2008 on the same hardware

While this adds a few steps to the migration process, it gives you the flexibility to reuse your hardware.

FAQs:

·         Can I migrate to different architectures (ia64, x64, x86)?

o   Yes, this is fully supported

·         Can I directly upgrade a cluster node from 2003 to 2008?

o   No, you must first evict it from the cluster: http://support.microsoft.com/default.aspx?scid=kb;EN-US;935197

·         Do you have any more information on the Migrate a Cluster Wizard?

o   Here is a step-by-step guide for migration: http://technet.microsoft.com/en-us/library/cc754481.aspx

·         Can I migration from a 2008 cluster to another 2008 cluster?

o   No, this is not supported

·         Can I migrate a File Server from a single machine (2003) to a cluster (2008)?

o   Yes, however you must use the File Server Migration Toolkit (FSMT).  It does not support migration for other workloads.  

o   FSMT v1.0 works for NT4/Windows 2000 Server to Windows Server 2003, available here:  http://www.microsoft.com/windowsserver2003/upgrading/nt4/tooldocs/msfsc.mspx

o   FSMT v1.1, now in Beta, works for Windows Server 2003 to Windows Server 2008.  Contact FSMTBeta@microsoft.com for more information.

·         Can I migrate every highly-available workload?

o   No.  We support migration for DFS-Namespace (DFS-N), DHCP, File Server, Generic Application, Generic Script, Generic Service, and WINS.   We do NOT support migration for (MS)DTC, Exchange, Hyper-V, iSNS, MSMQ, NFS, Other Server, Print, or SQL.  However Exchange, Hyper-V, Print and SQL have their own migration processes which are cluster-aware and will allow you to migrate from a 2003 cluster to a 2008 cluster.

Best of luck with your migration to Windows Server 2008.

Thanks,

Symon Perriman

Program Manager

Clustering & HA

What happens when one of my Active Directory Objects gets Deleted?

Don't let it happen!

This goes beyond clusters.  If an identity is deleted, nothing using that identity will be able to log on or authenticate against it.  The service or application that you went through all the pains to make highly available on your cluster will no longer be available to clients.  Once an object is deleted, you have (by default) 180 days, in a Windows Server 2003 Domain, to recover the object from the deleted objects store (see http://support.microsoft.com/?kbid=840001 or Mark Russinovich's Active Directory tools: http://technet.microsoft.com/en-us/sysinternals/bb963906.aspx)

What if the object has been permanently deleted?  Can I recreate a new object with the same name and have clustering still work?  Unfortunately, no.  Network Names depend on the specific Active Directory objects, not their names.  This is especially true of the CNO.  So, be careful and don't delete your cluster computer objects.

The network names rotate their passwords according to domain and system policy, so you can use the last password set date in your Active Directory scripts to keep these objects around.

Thanks,
Matt Kurjanowicz
Software Development Engineer
Clustering & High Availability

Answer to Homework: Which identity needs access to a file share for the File Share Witness?

In my post “What happened to the Cluster Service Account?" I asked the following question:

Suppose you created a cluster named "SQLCLUSTER01" and this cluster hosts two applications, named "CORPSQL01" and "CORPDOCS."  Active Directory will contain three computer objects - SQLCLUSTER01, CORPSQL01, and CORPDOCS.  SQLCLUSTER01 be the owner of CORPSQL01 and CORPDOCS.

Taking this example one step further, suppose that this cluster is configured to have a File Share Witness.  To which identity would you need to grant permissions on the file share used as this witness?

The answer here is the computer object named SQLCLUSTER01$.  The CNO object name is SQLCLUSTER01 and this is the identity under which the File Share Witness operates when going over the network.  The $ at the end is to indicate that the name is a computer name instead of a user or group.

Thanks,
Matt Kurjanowicz
Software Development Engineer
Clustering & High Availability

What happened to the Cluster Service Account?

Before Windows Server 2008, the cluster required the use of a Cluster Service Account (CSA).  This was a domain user under whose credentials the cluster service, as well as cluster resources, ran.  The CSA presented some problems, the most obvious of which was requiring administrators to rotate this password every so often.

In Windows Server 2008, we removed this requirement.  To replace the CSA, we created the Cluster Name Object (CNO).  This is a Network Name resource that acts as the identity of the Cluster.  This CNO in turn owns all of the Virtual Computer Objects (VCO) in the cluster.  The VCOs are the computer names to which clients connect.  The cluster service and cluster resources, now impersonate the CNO or the proper VCO.

To give an example, suppose you created a cluster named "SQLCLUSTER01" and this cluster hosts two applications, named "CORPSQL01" and "CORPDOCS."  Active Directory will contain three computer objects - SQLCLUSTER01, CORPSQL01, and CORPDOCS.  SQLCLUSTER01 be the owner of CORPSQL01 and CORPDOCS.

Taking this example one step further, suppose that this cluster is configured to have a File Share Witness.  To which identity would you need to grant permissions on the file share used as this witness?

For more information about Active Directory with Failover Clustering, check out our TechNet guide on Configuring Accounts for Active Directory:   http://technet.microsoft.com/en-us/library/cc731002.aspx.

Thanks,
Matt Kurjanowicz
Software Development Engineer
Clustering & High Availability