Clustering and High-Availability

Failover Clustering Sessions @ TechEd 2013

Elden Christensen — Tue, 11 Jun 2013 17:35:00 GMT

If you were not able to make it to TechEd 2013 this year, you can still watch the sessions and learn about the new enhancements coming. Here’s links to the recorded cluster sessions at TechEd 2013:

Continuous Availability: Deploying and Managing Clusters Using Windows Server 2012 R2
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B305#fbid=WOoBzkT2vlt

Failover Cluster Networking Essentials
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B337#fbid=WOoBzkT2vlt

Upgrading Your Private Cloud with Windows Server 2012 R2
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B331#fbid=WOoBzkT2vlt

Application Availability Strategies for the Private Cloud
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B311#fbid=WOoBzkT2vlt

Storage and Availability Improvements in Windows Server 2012 R2
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B333#fbid=WOoBzkT2vlt

Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance
http://channel9.msdn.com/Events/TechEd/NorthAmerica/2013/MDC-B335#fbid=WOoBzkT2vlt

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

Windows Failover Cluster validation warning indicates your disks don't support the persistent reservations for Storage Spaces

Rob-MSFT — Fri, 24 May 2013 18:49:00 GMT

I have seen questions from customers who get a warning in the results of their failover cluster validation that indicates the storage doesn’t support persistent reservations for Storage Spaces. They want to know why they got the warning, what it means, and what should they do about it. First, here is the text you will see in the report from the failover cluster validation. It will be highlighted in Yellow and the test may have a Yellow triangle icon next to it:

Validate Storage Spaces Persistent Reservation

Validate that storage supports the SCSI-3 Persistent Reservation commands needed by Storage Spaces to support clustering.

Test Disk <number X> does not support SCSI-3 Persistent Reservations commands needed by clustered storage pools that use the Storage Spaces subsystem. Some storage devices require specific firmware versions or settings to function properly with failover clusters. Contact your storage administrator or storage vendor for help with configuring the storage to function properly with failover clusters that use Storage Spaces.

Question: Why did I get this warning?

Failover cluster requires a specific set of SCSI-3 persistent reservation commands to be implemented by the storage so that storage spaces can be properly managed as clustered disks. The commands that are specifically needed for Storage Spaces are tested, and if they are not implemented in the way that the cluster requires, this warning will be given.

Question: What does this mean and why is it a warning and not a failure?

Failover cluster has multiple tests that check how the storage implements SCSI-3 persistent reservations. This particular test for Storage Spaces is a warning instead of a failure because clustered disks that aren’t going to use Storage Spaces will work correctly if the other SCSI-3 persistent reservation tests pass.

Question: What should I do when I get this warning?

Check the disks that are identified in the warning message and verify whether you will ever want to use those disks with Storage Spaces.

If you want to use the disks with Storage Spaces on the cluster, then you should check your storage configuration and documentation to see if there are settings or firmware/driver versions required to support clustered storage spaces.

If you aren’t going to use Storage Spaces with this cluster and storage, and the other storage validation tests indicate the tests passed, then you can ignore this warning.

The following note is in the KB article that states the support policy for Windows Server 2012 failover clusters. The yellow yield sign mentioned is referring to a warning in the validation test results. http://support.microsoft.com/kb/2775067

Note The yellow yield sign indicates that the aspect of the proposed failover cluster that is being tested is not in alignment with Microsoft best practices. Investigate this aspect to make sure that the configuration of the cluster is acceptable for the environment of the cluster, for the requirements of the cluster, and for the roles that the cluster hosts.

Here are some links to more information regarding clustered storage spaces, cluster validation, and the support policies regarding the validation tests:

Blog: “How to Configure a Clustered Storage Space in Windows Server 2012” http://blogs.msdn.com/b/clustering/archive/2012/06/02/10314262.aspx

TechNet: Deploy Clustered Storage Spaces http://technet.microsoft.com/en-us/library/jj822937.aspx

TechNet: Validate Hardware for a Windows Server 2012 Failover Cluster http://technet.microsoft.com/en-us/library/jj134244.aspx

Microsoft Knowledge Base Article: The Microsoft support policy for Windows Server 2012 failover clusters http://support.microsoft.com/kb/2775067

Steven Ekren

Senior Program Manager

Windows Server Failover Clustering

Optimizing CSV Backup Performance

Elden Christensen — Tue, 07 May 2013 03:58:12 GMT

Cluster Shared Volumes (CSV) is a clustered file system available in Windows Server 2012 where all nodes in a Windows Server Failover Cluster can simultaneously access a common shared NTFS volume. CSV has a distributed backup infrastructure which enables backups to be taken from any node in the cluster. In this blog I will discuss some considerations with how backups work with CSV which can help optimize the performance of backups.

When a volume level backup is taken, the cluster service returns all the VMs hosted on the volume(s) to the requester (backup application), including VMs running on non-requester nodes. The requester can choose to pick only the VMs that are running on the node where the backup was initiated (this becomes a local node VM backup), or it can choose to include VMs that are running across different nodes (this becomes a distributed VM backup). The snapshot creation has some differences based on the type of snapshot configured:

Hardware snapshots – The snapshot will be created and surfaced on the node where the backup was invoked by the requestor, which need not be the case as the coordinator node. The backup will then be taken from the local snapshot.
Software snapshots – The underlying snapshot device will be created via volsnap.sys on the coordinator node, and a CSV snapshot volume will be surfaced on every node that points to this volsnap device. On non-coordinator nodes, the CSV snapshot device will access the volsnap snapshot over SMB. It is transparent to the requestor as the CSV snapshot volume appears like a local device, all access to the snapshot will be happening over the network unless the requester happens to be running on the coordinator node.

Considerations:

When taking a backup of a CSV volume, it can be done from any node. However, when using software snapshots the snapshot device will be created on the coordinator node and if the backup was initiated on a non-coordinator node the backup data will be accessed remotely. This means that the data for the backup will be streamed over the network from the coordinator node to the node where the backup was initiated. If you have maintenance window requirements that require shortening the overall backup time you may wish to optimize the performance of backups when using software snapshots in one of the following ways:

Initiate Backups on the Coordinator Node – When using software snapshots the snapshot device will always be created on the node which currently owns the cluster Physical Disk resource associated with the CSV volume. If the backup is conducted locally on the coordinator node, then the data access will be local and backup performance may be improved. This can be achieved by either initiating the backup application on the coordinator node or by moving the Physical Disk resource locally to the node before initiating the backup. CSV ownership can be moved seamlessly with no downtime.
Scale Intra-node Communication – If you wish to have the flexibility of invoking backups with software snapshots from any node, to achieve optimized performance of backups scale up the performance of intra-node communication. It is recommended to use a minimum of 10 GB Ethernet or InfiniBand. You may also wish to use aggregate network bandwidth with NIC Teaming or SMB Multi-channel to increase network performance between the nodes in the Failover Cluster.

Recommendations:

To achieve the highest levels of performance of backups on a Cluster Shared Volume, it is recommended to use Hardware snapshots over Software snapshots.
To achieve the highest levels of performance with Software snapshots on a Cluster Shared Volume, it is recommend either to initiate the backup locally on the CSV coordinator node or to scale up the bandwidth of intra-node communication.

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

MSMQ Errors in the Cluster.log

Elden Christensen — Fri, 05 Apr 2013 17:53:00 GMT

After using the Get-ClusterLog cmdlet to generate the Cluster.log, you may notice the following errors in the cluster log:

ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
WARN [RCM] Failed to load restype 'MSMQ': error 21.

ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.

Root Cause:

These events are logged because the MSMQ and MSMQ Triggers resource types are registered with the cluster service, but the MSMQ resource DLL cannot be loaded because the MSMQ feature is not installed. This is the default configuration when the Failover Clustering feature is installed.

Resolution:

These are benign events to a debug log and can be safely ignored. They have no impact on the functionality of the cluster, nor do they indicate a failure.
If you plan to make MSMQ highly available on this cluster, open Server Manager and install the “Message Queuing” feature on all nodes in the cluster. The above errors will no longer be logged.
If this is a non-MSMQ cluster, you can unregister the MSMQ and MSMQ resource type with the Cluster Service. The above errors will no longer be logged. This can be accomplished with the Remove-ClusterResourceType cmdlet. Open a PowerShell window and type the following:

PS C:\> Remove-ClusterResourceType MSMQ

PS C:\> Remove-ClusterResourceType MSMQTriggers

In summary... just ignore them, they are just noise. If they annoy you, then unregister the MSMQ resource types if you don't plan to use MSMQ.

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

Configuring How VMs Are Moved when a Node is Drained

Elden Christensen — Thu, 21 Mar 2013 17:50:00 GMT

Introducing some concepts:

Windows Server 2012 introduced the ability to drain a node, which is sometimes also referred to as Node Maintenance Mode. When you drain a node, the Windows Server Failover Cluster will Pause the node to prevent any new VMs from moving to that node, then it will automatically move virtual machines (VMs) and other workloads off of the node to other nodes in the cluster. I like to call the moves that aren’t initiated by a user or an external management tool (like System Center Virtual Machine Manager), “cluster initiated moves.”

Windows Server 2012 also introduced the concept of priorities for cluster roles. You can set the priority to High (3000), Medium (2000) or Low (1000). The failover cluster uses this priority setting for a number of things, such as the order in which VMs are started. It is also used to define how VMs should be moved when a node is drained.

Live migration moves VMs to other nodes without clients losing connection. However, it can use significant network bandwidth.

Quick migration involves putting a VM into saved state, moving it to another node, and then resuming from saved state. Quick migrations are usually faster than live migrations and use less network bandwidth. However, quick migration does cause clients to be disconnected during the move.

Virtual machines and cluster initiated moves

The default behavior for VMs is to live migrate the high and medium priority VMs, and quick migrate the low priority VMs. The logic is that additional time and resources will be spent to move important VMs with no downtime and that it is ok for non-critical VMs to have downtime during the move.

However this behavior is fully configurable, if you wish for non-critical VMs to also be live migrated you can change the default behavior so that low priority VMs are also live migrated during cluster initiated moves. Or, you can change it so that medium, or both medium and high priority VMs use quick migration.

The “Virtual Machine” resource type has a parameter (sometimes called a private property) that is named MoveTypeThreshold. Resource type parameters are settings that affect all cluster resources of that type. Changing this parameter value changes how all VMs on the cluster are moved during automatic moves.

The default value for the MoveTypeThreshold parameter is 2000, this means that medium priority and any higher priorities than medium will use live migration for cluster initiated moves (like node drain). If you set the value to 1000, this means that low priority and any priorities higher than low will use live migration. If you set it to 3000, only high priority VMs will be live migrated, and the medium and low priority VMs will be quick migrated. If you want to only use quick migration, set it to 3001 or higher.

Here is an example of using Windows PowerShell to get the setting and to set it:

PS C:\Windows\system32> Get-ClusterResourceType "Virtual Machine" | Get-ClusterParameter MoveTypeThreshold | fl *

ClusterObject : Virtual Machine
Name          : MoveTypeThreshold
IsReadOnly    : False
ParameterType : UInt32
Value         : 2000

PS C:\Windows\system32> Get-ClusterResourceType "Virtual Machine" | Set-ClusterParameter MoveTypeThreshold 1000

PS C:\Windows\system32> Get-ClusterResourceType "Virtual Machine" | Get-ClusterParameter MoveTypeThreshold | fl *

ClusterObject : Virtual Machine
Name          : MoveTypeThreshold
IsReadOnly    : False
ParameterType : UInt32
Value         : 1000

MoveTypeThreshold values and move types

MoveTypeThreshold Value	VM Priority
MoveTypeThreshold Value	Low	Medium	High
1000	Live	Live	Live
2000	Quick	Live	Live
3000	Quick	Quick	Live
3001	Quick	Quick	Quick

Changing the behavior for specific virtual machines

The MoveTypeThreshold parameter of the Virtual Machine resource type affects the behavior for all of the VMs in the failover cluster. If you wish to have different behavior for different VMs that can also be accomplished, for example you want all low priority VMs to be quick migrated but there is one VM that you want to be live migrated. Each Virtual Machine resource has a DefaultMoveType private property which by default it is set to a value of -1 (This shows as 4294967295 if you look at the value of the parameter in Windows PowerShell). The value of -1 tells the individual Virtual Machine resource that it has no unique setting and that it should inherit its settings from the Virtual Machine resource type.

Note: DefaultMoveType is a parameter of each virtual machine’s Virtual Machine resource. Each VM will have its own Virtual Machine resource.

DefaultMoveType parameter values and their behavior:

Value	Behavior
-1 (4294967295)	Use global setting (MoveTypeThreshold)
0	Turn off VM
1	Save VM (quick migration)
2	Shut down VM
3	Shut down VM
4	Live migrate VM

*Note: Value 2 and 3 have the same behavior.

Here is an example of using Windows PowerShell to get the setting and to set it:

Find the resource name:

PS C:\Windows\system32> Get-ClusterResource | ft Name,ResourceType

Name                                                        ResourceType
----                                                        ------------
Cluster Disk 1                                              Physical Disk
Cluster IP Address                                          IP Address
Cluster IP Address                                          IPv6 Address
Cluster Name                                                Network Name
Virtual Machine Configuration test1                         Virtual Machine Configuration
Virtual Machine test1                                       Virtual Machine

Get the resources’ private properties using the Get-ClusterParameter cmdlet:

PS C:\Windows\system32> Get-ClusterResource "Virtual Machine Test1" | Get-ClusterParameter | ft Name,Value

Name                                                        Value
----                                                        -----
VmID                                                        76138d6e-ff1d-45da-bce3-d6ddc46a9bae
OfflineAction                                               1
ShutdownAction                                              0
DefaultMoveType                                             4294967295
CheckHeartbeat                                              1
MigrationState                                              0
MigrationProgress                                           0
VmState                                                     3
MigrationFailureReason                                      0
StartMemory                                                 512
VirtualNumaCount                                           1

Set the private property using the Set-ClusterParameter cmdlet:

PS C:\Windows\system32> Get-ClusterResource "Virtual Machine Test1" | Set-ClusterParameter DefaultMoveType 1

Check that the private property was changed:

PS C:\Windows\system32> Get-ClusterResource "Virtual Machine Test1" | Get-ClusterParameter | ft Name,Value

Name                                                        Value
----                                                        -----
VmID                                                        76138d6e-ff1d-45da-bce3-d6ddc46a9bae
OfflineAction                                               1
ShutdownAction                                              0
DefaultMoveType                                             1
CheckHeartbeat                                              1
MigrationState                                              0
MigrationProgress                                           0
VmState                                                     3
MigrationFailureReason                                      0
StartMemory                                                 512
VirtualNumaCount                                            1

Understanding how Failover Clustering Recovers from Unresponsive Resources

Elden Christensen — Thu, 24 Jan 2013 17:54:00 GMT

In this blog I will discuss how Failover Clustering communicates with cluster resources, along with how clustering detects and recovers when something goes wrong. For the sake of simplicity I will use a Virtual Machine as an example throughout this blog, but the logic is generic and applies to all workloads.

When a Virtual Machine is clustered, there is a cluster “Virtual Machine” resource created which controls that VM. The “Virtual Machine” resource and its associated resource DLL communicates with the VMMS service and tells the VM when to start, when to stop, and it also does health checks to ensure the VM is ok.

Resources all run in a component of the Failover Clustering feature called the Resource Hosting Subsystem (RHS). These VM actions from the user map to entry point calls that RHS makes to resources, such as Online, Offline, IsAlive, and LooksAlive. You can find the full list of resource DLL entry-point functions here.

The most interesting in most cases where resources go unresponsive and you see clustering need to recover is with the LooksAlive and IsAlive which is a health check to the resource.

LooksAlive is a quick light lightweight check that happens every 5 seconds by default
IsAlive is a more verbose check that happens every 60 seconds by default.

Health check calls to the resource continue constantly while resources are online. If a resource returns a failure for the lightweight LooksAlive health check, RHS will then immediately do a more comprehensive health check and call IsAlive to see if the resource is really healthy. A resource is considered failed as the result of an IsAlive failure.

Think of it like this… Every 60 seconds RHS calls IsAlive and basically is asking the resource “Are you ok?”. And the resource then responds to RHS “Yes, I am doing fine.” This periodic health check goes on and on… Until, there can be a case where something happens to the resource and it doesn’t respond. Think of it like a dropped call on your cell phone, how long are you willing to sit there going “Hello? Hello? Hello?”… before you give up and call the person back? Basically resetting the connection…

Failover Clustering has this same concept. RHS will sit there waiting for the resource to respond to an IsAlive call, and eventually it will give up and need to take recovery action. By default RHS will wait for 5 minutes for the resource to respond to an entry point call to it. This is configurable with the resource DeadlockTimeout common property.

To modify the DeadlockTimeout property of an individual resource, you can use the following PowerShell cmdlet command:
(Get-ClusterResource “Resource Name”).DeadlockTimeout = 300000

Or if you want to modify the DeadlockTimeout for all resources of that type you can modify it at the resource type level with the following syntax (this example will be for all virtual machine resources):
(Get-ClusterResourceType “Virtual Machine”).DeadlockTimeout = 300000

Resources are expected to respond to an IsAlive or LooksAlive within a few hundred milliseconds, so waiting 5 minutes for a resource to respond is a really long time. Something pretty bad happened if a resource which normally responds in milliseconds, suddenly takes longer than 5 minutes. So it is generally recommended to stay with the default values.

If the resource doesn’t respond in 5 minutes, RHS decides that there must be something wrong with the resource and that it should take recovery action to get it back up and running. Remember that the resource has gone silent; RHS has no idea what is wrong with it. The only way to recover and get the resource back up and running is that the RHS process is terminated, then RHS restarts, which will then restart the resource, and everything is back up and running. You may also see the associated entries in the System event log:

Event ID 1230
Cluster resource ‘Resource Name’ (resource type ‘Resource Type Name’, DLL ‘DLL Name’) did not respond to a request in a timely fashion. Cluster health detection will attempt to automatically recover by terminating the Resource Hosting Subsystem (RHS) process running this resource.

Event ID 1146
The cluster Resource Hosting Subsystem (RHS) stopped unexpectedly. An attempt will be made to restart it. This is usually associated with recovery of a crashed or deadlocked resource.

This is the way clustering is designed to work… it is monitoring the health of the system, it detects something is wrong, and recovers. This is a good thing!

Impact of RHS Recovery

The Resource Hosting Subsystem (RHS) is the process which hosts resources, and for any given node if there are multiple resources currently online and being hosted by a node they may share a common RHS process. For example, if you had 5 clustered VMs running on the same node, all the resources associated with those VMs would all be running in the same RHS process.

There are some side effects from terminating the RHS process when a resource goes unresponsive. If there are multiple resources hosted on that node, they may be hosted in the same RHS process. That means when RHS terminates and restarts to recover an individual resource, all resources being hosted in that specific RHS process are also restarted. With Windows Server 2008 R2 if you have 5 VMs running on a node, all 5 VMs are going to get restarted.

If a resource becomes unresponsive and causes an RHS crash, the cluster service will deem that specific resource to be suspect and that it needs be isolated. Think of it as, one strike and you are out! The cluster service will automatically set the resource common property SeparateMonitor to mark that resource to run in its own dedicated RHS process, so that in the event that the resource becomes unresponsive again; it will not affect others. This setting is also configurable, you can either manually enable a resource to run in its own RHS process or you can disable a resource from running in its own RHS process as the result of having had an issue in the past which is now addressed.

To modify the SeparateMonitor property of an individual resource, you can use the following PowerShell cmdlet command:
(Get-ClusterResource “Resource Name”).SeparateMonitor = 0

The impact of running resources in their own dedicated RHS process is that each RHS process consumes a little more system resources. If you open Task Manager you will see a series of “Failover Cluster Resource Host Subsystem” processes running, each of which consuming a few MB of RAM.

In general clustering will self-manage misbehaving resources. Resources will be given a chance to play nicely with everyone else, and if they don’t they will be automatically isolated to minimize impact. So it is generally recommended to stay with the default values.

Improvements in Windows Server 2012

There are some feature enhancements in Windows Server 2012 to mitigate the impact of non-responsive resource recovery.

Resource Re-attach: When a resource goes unresponsive the RHS process will recycle just as before, but any healthy resources in a running state will have their resources re-attach to the new RHS process without having to be restarted. This means that impact from recovery is reduced, just 1 VM gets restarted and the other 4 are not impacted.
- The resource DLL must support resource re-attach to be compatible with this new feature. In-box resource types such as Virtual Machine and Physical Disk have been enhanced in Windows Server 2012 to take advantage of this new feature.
Isolation of Core resources: Resources are now segmented by default into multiple RHS processes to keep application resources deadlocks from impacting core cluster functionality
- All in-box resources (in ClusRes.dll) run in a dedicated Core RHS process
- All “Physical Disk” resources run in a dedicated Storage RHS process
- 3rd party resources run in a dedicated RHS process

Additionally resources can also be marked with the SeparateMonitor property to run in their own dedicated RHS process in Windows Server 2012, as they could in previous releases.

How to Troubleshoot RHS Recovery

Everything we have discussed in this blog to this point has describing the expected behavior of how Failover Clustering recovers when something goes wrong with a resource and it becomes unresponsive. Now the most important question… What do you do about it?

Troubleshooting steps overview:

Open Event Viewer and look for an Event ID 1230
Identify the date / time as well as the resource name and resource type
Generate the cluster.log with the Get-ClusterLog cmdlet
Go to the C:\Windows\Cluster\Reports folder and open the Cluster.log file
Using the time stamp from the Event ID 1230 find the point of the failure
1. Reminder: The event log is in local time and the cluster.log is in GMT. With Windows Server 2012 you can use the Get-ClusterLog –UseLocalTime to generate the Cluster.log in local time. This will make correlating with the event log easier.
Identify which entry point was being called to the resource. There will be a log entry something similar to:
ERR [RHS] RhsCall::DeadlockMonitor: Call ISALIVE timed out for resource 'ResourceName'.
INFO [RHS] Enabling RHS termination watchdog with timeout 1200000 and recovery action 3.
ERR [RHS] Resource ResourceName handling deadlock. Cleaning current operation and terminating RHS process.
Look up what that entry point for that resource type is attempting to do. For in-box resources you will find them documented here
KB914458 – Behavior of the LooksAlive and IsAlive functions for the resources that are included in the Windows server Clustering component of Windows Server 2003.
Now that you understand what entry point was being called to which resource and when, you need to investigate the underlying component. For example:

Physical Disk resource IsAlive will effectively attempt to enumerate the file system, so you should troubleshoot your storage subsystem in why I/O’s are not completing.

File Server LooksAlive will attempt to retrieve the properties of the SMB shares, so you should troubleshoot the Server service.

The key take-away is that RHS recovery is expected behavior for a resource that has become unresponsive. To address the root cause issue you need to dig in to which resource is failing and then by understanding what it was attempting to do, you can identify why it didn’t respond.

For additional information on troubleshooting resources that result in RHS recovery, see the blogs below. Microsoft support is also available to assist in advanced debugging to help you identify root cause.

Additional Resources

Resource Hosting Subsystem (RHS) In Windows Server 2008 Failover Clusters
http://blogs.technet.com/b/askcore/archive/2009/11/23/resource-hosting-subsystem-rhs-in-windows-server-2008-failover-clusters.aspx

What Is In A RHS Dump File Created By Windows Error Reporting
http://blogs.msdn.com/b/ntdebugging/archive/2011/05/30/what-is-in-a-rhs-dump-file-created-by-windows-error-reporting.aspx

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

How to Setup a Failover Cluster in a RODC Environment

Rob-MSFT — Thu, 13 Dec 2012 19:09:00 GMT

In Windows Server 2012, a Failover Cluster can be created in an environment that has access only to a Read Only Domain Controller (RODC) but not a Read Write Domain Controller (RWDC). This deployment model can be useful in a branch office with unreliable network connectivity or in a perimeter network (DMZ) where the cluster resides outside a firewall.

In a previous blog, we discussed how a cluster can be created in a restrictive active directory environment. In the blog, we explained the role of a Cluster Name Object (CNO) and Virtual Computer Object (VCO) in a Failover Cluster. With a Read Only Domain Controller, the Cluster Service is unable to create a CNO or VCO. Therefore, these computer objects will need to be pre-created on a RWDC and then replicated to the cluster RODC, before the cluster creation process is commenced. This blog provides the steps on how this can be done: 1) Using the graphical interface 2) Using Windows PowerShell^© These steps should be followed to first pre-create a CNO (computer object that has the same name as your cluster) and a VCO for each clustered service or application (computer object has the same name as the clustered service or application and is created in the same container as the CNO).

Steps to configure the CNO and VCO for a RODC Environment:

1. On an RWDC launch the Active Directory Users and Computers snap-in (type dsa.msc) or to configure using a script open a Windows PowerShell^© prompt in Administrator mode.

2. Right-click Computers or the organizational unit (OU) container in which computer accounts are created in your domain and create a new Computer object for your cluster CNO (Cluster Name) or VCO(Clustered Application or Service Name):

Using PowerShell:

To create the Computer object in the default Computers container:

new-adComputer -name “myclusterCNO” -dnshostname “testcluster.com”

To Create the Computer object in an alternate OU:

new-adComputer -name “myclusterCNO” -dnshostname “testcluster.com” -Path $OUDistinguishName -Enabled $true

3. For a CNO, give the user account that will be used to create the cluster, full control of the computer object created. For VCOs, ensure that you give the Cluster account (CNO) full permission to access the object. For instance for a cluster myclusterCNO in domain testcluster, the account testcluster\myclusterCNO should have permission to the VCO.

On the View menu ensure that Advanced Features is selected.
Right-click on the computer object created in step 2 and select Properties:

Select the Security tab and add the user account used for cluster creation.
Select the newly created user account and give it Full Control for the computer object:

Using PowerShell:

$objUser = New-Object System.Security.Principal.NTAccount(“domain\user”)

$objADAR = New-Object System.DirectoryServices.ActiveDirectoryAccessRule($objUser, “GenericAll”,"Allow")

$adName = get-AdComputer -Identity “myclusterCNO”

$targetObj = get-adobject -Identity $adName.DistinguishedName -properties *

$ntSecurityObj = $targetObj.nTSecurityDescriptor

$ntSecurityObj.AddAccessRule($objADAR)

Set-ADObject $adName –Replace @{ntSecurityDescriptor=$ntSecurityObj}

You can verify through the graphical interface that the permissions have now propagated for the user account:

4. Next modify the following attributes for the computer object:

Attribute Name	Value
Dnshostname	<FQDN>
sAMAccountName Must be less than 15 characters	<Cluster name>$
msds-supportedencryptiontypes	28
Service Principle Name	List which includes the following entries: Host/<computer object name> Host/<FQDN> MSClusterVirtualServer/<computer object Name> MSClusterVirtualServer/<FQDN> MSServerClusterMgmtAPI/<Computer Object Name> MSServerClusterMgmtAPI/<FQDN>, For CNO also add: MSServerCluster/<computer object Name> MSServerCluster/<FQDN>

You can modify the attributes by selecting the Attribute Editor tab on the computer object properties page:

Using PowerShell:

$adName = get-AdComputer -Identity “myclusterCNO”

$dn = $adName.DistinguishedName

set-adcomputer -Identity $dn -add @{'msds-supportedencryptiontypes'= 28}

set-adComputer -Identity $dn -ServicePrincipalName @{Add="Host/myclusterCNO", "Host/testcluster.com", "MSClusterVirtualServer/myclusterCNO", "MSClusterVirtualServer/testcluster.com", "MSServerClusterMgmtAPI/myclusterCNO", "MSServerClusterMgmtAPI/testcluster.com"}

For CNO also add:

set-adComputer -Identity $dn -ServicePrincipalNames @{Add=" MSServerCluster/myclusterCNO", "MSServerCluster/testcluster.com"}

5. Add the CNO or the VCO SAM account name to the Allow RODC Password Replication Group

Select the Domain Controller container from dsa.msc
Right-click on the Computer Object corresponding to the RODC

Select the Password Replication Policy tab in the property pane for the RODC Computer Object.
Add the CNO and VCO SAM account names(with $ at the end)> to the Allow RODC Password Replication Group:

Using PowerShell:

Add-ADDomainControllerPasswordReplicationPolicy -Identity “RODC” -AllowedList "testCluster$","vcoName$”

Supply the CNO and VCO SAM account name(with $ at the end) as arguments to the AllowedList parameter

6. Finally, replicate the CNO or VCO computer object created on the RWDC to the RODC:

repadmin /rodcpwdrepl <RODC server name> <RWDC server name> <distinguished name of the CNO or VCO without quotes e.g.: CN=myClusterCNO,CN=Computers,DC=testcluster,DC=com >

Now that you have the computer objects pre-staged and replicated to your RODC, you are ready to create a cluster in a RODC environment. In a previous blog we provided the steps to create a Failover Cluster.

Thanks!

Subhasish Bhattacharya

Program Manager

Clustering and High Availability

Microsoft Corporation

Tuning Failover Cluster Network Thresholds

Elden Christensen — Wed, 21 Nov 2012 23:46:00 GMT

Windows Server Failover Clustering is a high availability platform that is constantly monitoring the network connections and health of the nodes in a cluster. If a node is not reachable over the network, then recovery action is taken to recover and bring applications and services online on another node in the cluster.

Failover Clustering by default is configured to deliver the highest levels of availability, with the smallest amount of downtime. The default settings out of the box are optimized for failures where there is a complete loss of a server, what we will refer to in this blog as a ‘hard’ failure. These would be unrecoverable failure scenarios such as the failure of non-redundant hardware or an OS bug check. In these situations the server is lost and the goal is for Failover Clustering to very quickly detect the loss of the server and rapidly recover on another server in the cluster. To accomplish this fast recovery from hard failures, the default settings for cluster health monitoring are fairly aggressive. However, they are fully configurable to allow flexibility for a variety of scenarios.

These default settings deliver the best behavior for most customers, however as clusters are stretched from being inches to possibly miles apart the cluster may become exposed to more unreliable networking components between the nodes. Another factor is that the quality of commodity servers is constantly increasing, coupled with augmented resiliency through redundant components (such as dual power supplies, NIC teaming, and multi-path I/O), the number of non-redundant hardware failures may potentially be fairly rare. Because hard failures may be less frequent some customers may wish to tune the cluster for transient failures, where the cluster is more resilient to brief network failures between the nodes. By increasing the default failure thresholds you can decrease the sensitivity to brief network issues that last a short period of time.

Trade-offs

It is important to understand that there is no right answer here, and the optimized setting may vary by your specific business requirements and service level agreements.

Aggressive Monitoring – Provides the fastest failure detection and recovery of hard failures, which delivers the highest levels of availability. Cluster is less forgiving for transient failures, and may in some situations prematurely failover resources when there are transient network outages.
Relaxed Monitoring – Provides more forgiving failure detection which provides greater tolerance of brief transient network issues. These longer time-outs result in cluster recovery from hard failures being longer.

Think of it like your cell phone, when the other end goes silent how long are you willing to sit there going “Hello?... Hello?... Hello?” before you hang-up the phone and call the person back. When the other end goes silent, you don’t know when or even if they will come back.

The key question you need to ask yourself is: What is more important to you? To quickly recover when you pull out the power cord or to be tolerant to a network hiccup?

Settings

There are four primary settings that affect cluster heartbeating and health detection between nodes.

Delay – This defines the frequency at which cluster heartbeats are sent between nodes. The delay is the number of seconds before the next heartbeat is sent. Within the same cluster there can be different delays between nodes on the same subnet and between nodes which are on different subnets.
Threshold – This defines the number of heartbeats which are missed before the cluster takes recovery action. The threshold is a number of heartbeats. Within the same cluster there can be different thresholds between nodes on the same subnet and between nodes which are on different subnets.

It is important to understand that both the delay and threshold have a cumulative effect on the total health detection. For example setting CrossSubnetDelay to send a heartbeat every 2 seconds and setting the CrossSubnetThreshold to 10 heartbeats missed before taking recovery, means that the cluster can have a total network tolerance of 20 seconds before recovery action is taken. In general, continuing to send frequent heartbeats but having greater thresholds is the preferred method.

Windows Server 2012:

Parameter	Fast Failover (Default)	Relaxed	Maximum
SameSubnetDelay	1 second	1 second	2 seconds
SameSubnetThreshold	5 heartbeats	10 heartbeats	120 heartbeats
CrossSubnetDelay	1 second	1 seconds	4 seconds
CrossSubnetThreshold	5 heartbeats	20 heartbeats	120 heartbeats

Note: The “Relaxed” column above is a recommendation for customers looking to set their clusters to be more tolerant of transient failures. The recommended suggestions double the defaults to 10 second threshold for nodes on the same subnet and 20 second threshold for nodes on different subnets.

Disclaimer: When increasing the cluster thresholds, it is recommended to increase in moderation. It is important to understand that increasing resiliency to network hiccups comes at the cost of increased downtime when a hard failure occurs. In most customers’ minds, the definition of a server being down on the network is when it is no longer accessible to clients. Traditionally for TCP based applications this means the resiliency of the TCP reconnect window. While the cluster thresholds can be configured for durations of minutes, to achieve reasonable recovery times for clients it is generally not recommended to exceed the TCP reconnect timeouts. Evaluate your business needs to define what are the maximum values that are right for your deployments configuration.

It critical to recognize that cranking up the thresholds to high values does not fix nor resolve the transient network issue, it simply masks the problem by making health monitoring less sensitive. The #1 mistake made broadly by customers is the perception of not triggering cluster health detection means the issue is resolved (which is not true!). I like to think of it, that just because you choose not to go to the doctor it does not mean you are healthy. In other words, the lack of someone telling you that you have a problem does not mean the problem went away.

Configuration:

Cluster heartbeat configuration settings are considered advanced settings which are only exposed via PowerShell. These setting can be set while the cluster is up and running with no downtime and will take effect immediately with no need to reboot or restart the cluster.

To view the current heartbeat configuration values:
PS C:\Windows\system32> get-cluster | fl *subnet*

The setting can be modified with the following syntax:
PS C:\Windows\system32> (get-cluster).CrossSubnetDelay = 2000

Additional Considerations for Logging:

In Windows Server 2012 there is additional logging to the Cluster.log for heartbeat traffic when heartbeats are dropped. By default the RouteHistoryLength setting is set 10, which is two times the number of default thresholds. If you increase the SameSubnetThreshold or CrossSubnetThrehold values, it is recommended to increase the RouteHistoryLength value to be twice the value to ensure that if the time arises that you need to troubleshoot heartbeat packets being dropped that there is sufficient logging. This can be done with the following syntax:
PS C:\Windows\system32> (get-cluster).RouteHistoryLength = 20

For more information on troubleshooting issues with nodes being removed from cluster membership due to network communication issues, please see the following blog:
http://blogs.technet.com/b/askcore/archive/2012/02/08/having-a-problem-with-nodes-being-removed-from-active-failover-cluster-membership.aspx

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

BUILD 2012: Designing applications for highly-availability with Windows Server 2012 Failover Clustering

Elden Christensen — Mon, 05 Nov 2012 17:19:00 GMT

BUILD 2012 wrapped up last week. In case you were not able to make it, the sessions are now posted that you can watch. Here is a link to the cluster session.

Session: 3-051
Title: Designing applications for highly-availability with Windows Server 2012 Failover Clustering
Abstract: Learn how to leverage the power of the Failover Clustering features in Windows Server 2012 to deliver your application the highest levels of availability. This session will cover the options available to integrate your application to achieve failover, considerations when building on top of Microsoft’s clustered file system (CSVFS), and how to deliver increased availability when running your application on a private cloud by leveraging Guest Clustering and the new VM Monitoring feature.
Link: http://channel9.msdn.com/Events/Build/2012/3-051

Remember that BUILD is a developer conference, so this session is focused at ISV's, IHV's, OEM's, or any developer working on integrating with or leveraging Failover Clustering to achieve high availability with your applications and services.

Thanks!
Elden Christensen
Principal Program Manager Lead
Clustering & High-Availability
Microsoft

VM Monitoring in Windows Server 2012 – Frequently Asked Questions

Rob-MSFT — Tue, 30 Oct 2012 22:34:17 GMT

In a previous blog post I explained how VM Monitoring can be configured in Window Server 2012. In this blog I will answer the three most frequently asked questions related to the VM Monitoring feature.

1) Why can I no longer make my Print Server role Highly Available in Windows Server 2012? I was able to do this in Windows Server 2008 R2.

In contrast to previous versions of Windows Server, Windows Server 2012 defines a highly available print server as a Hyper-V virtual machine(VM) running on a node in a cluster. A single virtual machine with the Print Server role installed can then be migrated from one node in the Hyper-V cluster to the other using either manual or automatic methods.

In Windows Server 2012, the print spooler service is no longer a clustered resource and instead the entire virtual machine is migrated from one Hyper-V node to the other. This new model provides the same seamless user experience as previous versions of Windows but with the following added benefits:

Using Windows Server 2012 as the Hyper-V and failover clustering host allows access to the VM Monitoring feature. This allows greater flexibility and control over recovery actions.
Windows Server 2012 Print Servers can utilize the Live Migration and Quick Migration features of Hyper-V.
Windows Server 2012 Highly Available Print Servers are easier to deploy and have reduced complexity.
Print devices and drivers are deployed the same as on a physical machine which provides consistency for management. When deployed in a virtual machine, availability can be enhanced using the VM Monitoring feature.
Printer manufacturers will have a single driver so they can focus on higher quality and reduce the cost for creating drivers.

In a nutshell, using the VM Monitoring feature, the new print spooler HA model is able to streamline the deployment and management while providing higher availability for your users.

For additional information please refer to:

High Availability Printing Overview

Install and Configure High Availability Printing

Note:

The default VM Monitoring configuration will reboot or failover the virtual machine for every third Print Service failure in a 15 minute window. The duration of this virtual machine reboot is typically in the order of seconds. During this interval the print service will not be available. Later in this blog I will explain how this default recovery action can be customized.
During patching, the Print Service on the hosted virtual machine can be temporarily unavailable, if a reboot is required. The impact of this planned downtime can be mitigated by having an additional Print Server hosted on a Hyper-V node as a backup.

2) I want to configure VM Monitoring for a mission critical Virtual Machine. I do not want to take automated recovery actions such as rebooting my VM. I want to be notified when my VM encounters a critical condition so that my administrator can investigate the failure. How do I do this?

I have System Center Operations Manager deployed on my host (cluster node hosting the VM). How do I configure Operations Manager to work with VM Monitoring?

I want to customize the recovery action taken by the VM Monitoring. I don’t want to restart the VM or failover the VM on a failure. How do I do this?

If the “Enable automatic recovery for application health monitoring” option is deselected, the cluster service does not take any automatic recovery actions when a VM critical condition occurs. It does however log event ID 1250, to indicate that a critical condition occurred in your VM. To deselect this setting:

Using Failover Cluster Manager:

A) Select the VM that you want to configure this setting for

B) Click on the Resources tab

C) Right click on the Virtual Machine resource and select the Properties option.

D) Select the Settings tab and uncheck Enable automatic recovery for application health monitoring

a) Open a Windows PowerShell shell as an Administrator

b) Set the EmbeddedFailureAction property for the VM resource:

(Get-ClusterResource "*e test-VM").EmbeddedFailureAction = 1

Note: To re-enable automatic recovery actions on VM Critical failures this property should be set to 2 (default).

You can monitor Event 1250 to customize recovery action on VM Critical failures. Some options include:

A) Setting up a Cluster Scheduled task to carry out a desired sequence of actions on the occurrence of the VM event or service failure being monitored e.g.: Initiate a live migration on a VM network failure or send an email to a cluster administrator indicating the failure condition.

B) Configure System Center Operations Manager to take recovery actions when the event is triggered on the host.

C) Use a 3^rd party solution such as Symantec ApplicationHA^© for Hyper-V which provides advanced customization of recovery actions.

The administrator can investigate the VM in critical state as follows:

a) Log onto the VM

b) Launch Task Scheduler

c) Navigate to the Microsoft/FailoverClustering/VM Monitoring node

d) Examine when the last event or service failure occurred.

e) Once the failure has been examined and appropriate recovery actions taken, the VM can be removed from Critical State by running the following Windows PowerShell cmdlet as an Administrator on the guest:

Reset-ClusterVMMonitoredState

3) The Virtual Machine I want to monitor is not in the same domain as the cluster node it is hosted on. Can I configure VM Monitoring?

In this configuration VM Monitoring needs to be configured using Windows PowerShell by logging into the guest (virtual machine).

Pre-requisites:

Failover Clustering Admin Tools installed in the guest. This installs the FailoverClusters PowerShell module

Steps to configure in the guest using PowerShell:

a) Open a Windows PowerShell shell as an Administrator

b) Run the Add-ClusterVMMoniteredItem cmdlet inside the guest to configure monitoring

Example: Add-ClusterVMMonitoredItem –service spooler

Thanks!

Subhasish Bhattacharya

Program Manager

Clustering and High Availability

Microsoft