Clustering and High Availability

What is RHS and what does it do?

msclustm — Sat, 27 Jun 2009 01:08:00 GMT

In Window Server 2008 Failover Clustering, the team invested significant time into making clustering easier. In the Windows Server 2008 R2 release we have continued down that path, adding several troubleshooting enhancements.

One of the important aspects of troubleshooting a service outage is doing diligent postmortem analysis – to understand why you experienced the problem so that you can take corrective action to avoid seeing it again. A common problem can be due to third-party resource dlls which may not have had the detailed level of testing as the in-box dlls. In previous releases we offered the ability to isolate components into separate processes, and in 2008 R2 we have built in additional isolation logic, so that if a resource dll crashes, little else is affected, offering even higher availability to your mission-critical applications. The resource dll is a component is provided by the application being clustered and is a proxy between the application and the cluster. If the cluster wants to stop or start the application it will notify the resource dll, and resource dll will communicate this information to the application. The cluster does not load the resource dlls into the cluster service process, instead it loads them into the Resource Host Monitor (RHS.exe) process, which is recyclable.

Previously all resources used to run in a single RHS process by default. But this meant that if one resource crashes then the entire RHS process could fail and all resources hosted by this RHS will fail. We’ve improve our default behavior in 2008 R2 by separating our critical resources from our dlls in RHS. Now the Cluster Group (including the quorum resource) and Storage Group (including Available Storage and Clustering Shared Volumes) now all run in a single, isolated RHS process. The other resource dlls will run in one or more additional RHS processes.

There are two common reasons for seeing instability in a resource dll:

1. The resource dll itself may crash. In most cases this is caused by an access violation in the resource dll. In previous releases we took action to alert the admin of this event by reporting it to the Resource Control Manager (RCM) and exiting. RCM is a component inside the cluster service, which, upon receiving notification that the resource caused a crash, would mark this resource as “run in the separate monitor”. This offered higher availability to the cluster because this resource will be loaded in its own RHS process, and if it crashes again only that resource will be affected. In R2 we have enhanced this behavior by not only reporting the failure and isolating this resource, but additionally we report the access violation by generating a Windows Error Report (WER). WER will collect a dump file, create a problem report and will handle the report according to the policy applied on that computer which raises awareness of the issue to system administrator or Microsoft.

2. The resource dll might take too long to perform requested action, in some cases it might even deadlock. There is not effective way to detect if it is just taking long time or there is a deadlock. One way to solve this issue is to limit amount of time we are waiting for the resource to complete request, and if it does not complete in that time we would assume that the component handling this call is not in a healthy state. Some activities, such as online and offline, can take some time, so you may see the ‘pending’ state in the UI. If online is taking a long time, the resource might spawn a worker thread, and tell RHS that online call is pending, which can notify RHS that it requires more time. Once the resource comes online it will notify RHS. Offline is handled in the similar way. All other activities are simply limited by time and have to complete before RHS decides that it has timed out. In previous releases, when RHS decides that activity has timed out it will notify RCM and terminate the process. RCM will then isolate the resource in a separate RHS. In R2 we improved the logic based on whether the event is common or not. Many deadlocks are one-time events caused by a race condition, so it may not be appropriate to isolate that resource because of a single occurrence, as having too many individual RHS process can cause a slight performance impact. This will again create a WER and forward it to the appropriate destination.

In Windows Server 2008 R2 the reports can be found under Control Panel I System and Security Action Center Problem Reports. All RHS issues will be in the category “Failover Cluster Resource Host Subsystem”. The image below shows two issues. The first shows an Access Violation which is sent to WER as a Problem Report. Hooking up a debugger to the dump file would provide more details around what resource caused the problem.

The second item is generated when RHS has detected a call is taking too long. In this case, RHS explicitly calls WER to generate a problem report and provides additional information that allows the user to see details about which resource and call caused the issue without looking into the dump file. This example shows the case when the ONLINERESOURCE call to the resource “r1” of the type “FlexRes” took too long.

You can learn more about the benefits and configuration of Windows Error Reporting from the following resources:

· http://www.microsoft.com/whdc/maintain/WER/WERWorks.mspx

· http://msdn.microsoft.com/en-us/library/bb513638(VS.85).aspx

· http://msdn.microsoft.com/en-us/library/bb219076.aspx

· http://www.microsoft.com/whdc/winlogo/maintain/StartWER.mspx

Thanks,

Vladimir Petter

Senior Software Development Engineer

Clustering & High-Availability

Microsoft

PowerShell Remoting and the “Double-Hop” Problem

msclustm — Thu, 25 Jun 2009 06:16:00 GMT

The “Double-Hop” Problem

Most of the newly added PowerShell cmdlets (“command-lets”) in Windows Server 2008 R2 allow easy remote execution by including the -Cluster parameter to indicate the target cluster. However there are occasions when this capability cannot be used for different reasons, such as when a firewall is blocking the cluster management port, because a particular command (such as Add-ClusterDisk) does not have a -Cluster parameter, or when it has to be executed in a specific node.

For these cases PowerShell remoting can be a very useful feature which allows you run any PowerShell command from a remote machine. The feature can be easily enabled by running Enable-PSRemoting, or also by domain group policy (note that you will also need to enable the “Windows Remote Management” firewall exception). Then you can run Enter-PSSession <Target-Machine-Name> and you will be running cmdlets on the target machine.

Now you can remotely execute most cluster command like Add-ClusterDisk, but a problem arises when you try to execute command like Test-Cluster or New-Cluster. Those commands not only execute on the target machine, but they actually need to perform operations on every node of a cluster (like verifying those machines registry settings, etc). And that will fail because you are trying to make a remote operation from an environment which is already using a remote connection – this is known as the “double-hop” problem.

An Easy “Double-Hop” Solution

You can use CredSSP to delegate your credentials to the remote computer so every remote access from the remote machine will also work. To enable this, you will need to run (from an elevated command prompt) the following command on the client machine:

Enable-WSManCredSSP Client –DelegateComputer <FQDN of the server>

That would allow you to use CredSSP as a client of the specified server. Then, on the server machine(s), you need to run:

Enable-WSManCredSSP Server

That will allow incoming client CredSSP connections. Now you can use PowerShell remoting from the client by creating a session specifically thru CredSSP:

New-PSSession <FQDN of the Server> -Authentication CredSSP -Credential <User> | Enter-PSSession

You will then be asked for the corresponding password and then you can start the remote session. Now all those commands which call other nodes in your cluster will work.

Also instead of entering an interactive session with Enter-PSSession you can use the fan-out capabilities of PowerShell remoting for running the same command concurrently in several servers. For more information about this, see the PowerShell remoting links below.

CredSSP Details

By default, when a user remotely logs on to a machine the logon session created is of the “remote” type, which in opposition of an “interactive” logon session doesn’t contains the user credentials. Therefore this logon session allows operations on that particular machine but prevents opening a new remote session to another machine. This is done for security purposes as it could be dangerous grant those credentials to a different machine which can be potentially compromised.

CredSSP is a Security Service Provider that allows you to authenticate in a remote machine creating a logon session that contains the user credentials. It transfers the credentials securely over the network by first creating an encrypted session and then authenticating the machines by using the Negotiate protocol (which in turn will use Kerberos, NTLM, etc). The particular authentication settings could be configured as indicated in the links below. Therefore you should only enable CredSSP when you know that the target machine is not compromised as the plain text credentials will travel there.

Note that CredSSP is quite different of Kerberos delegation. Without going into too much detail, Kerberos delegation works in a different way (by transferring the TGT instead of the actual credentials), and it is established by domain group policy which selects the associated information for the user credentials from AD. Instead, CredSSP allow you to select a specific target with the specific credentials that you want to supply.

CredSSP can also be of great utility with other PowerShell environments or from applications that use SSPI. It was initially created for Terminal Services, an operation that always require an extra (initial) hop.

Resources

PowerShell Remoting:

http://technet.microsoft.com/en-us/library/dd347706.aspx

http://technet.microsoft.com/en-us/library/dd315359.aspx

CredSSP:

http://msdn.microsoft.com/en-us/library/cc540483.aspx

http://technet.microsoft.com/en-us/library/cc749211(WS.10).aspx

Thanks,

Greg Maeso

Software Development Engineer

Clustering & High-Availability

Microsoft

Rolling Upgrade to Windows Server 2008 SP2 Failover Clustering

msclustm — Fri, 12 Jun 2009 07:26:00 GMT

Hi Cluster Fans,

Windows Server 2008 Service Pack 2 (SP2) is now available for download on the web! This blog post will walk you through performing a “rolling upgrade” from Windows Server 2008 RTM to Windows Server 2008 SP2. A rolling upgrade allows you to progressively upgrade your cluster nodes while keeping your resources highly-available, however you may have some downtime while moving the resource groups between nodes. What happened to Service Pack 1? Windows Server 2008 RTM was actually SP1 as it was released around the same time as Vista SP1.

Note that when moving between Windows Server 2003, Windows Server 2008 and Windows Server 2008 R2 a migration is needed because mixed-mode clusters are not supported. A migration guide is at: http://technet.microsoft.com/en-us/library/cc754481.aspx.

In this guide I use a 2-node cluster so when I split the cluster into 2 logical groups, each group will have 1 node. If you have larger clusters, you can walk through this guide in as many iterative steps as you need to gradually upgrade all of you nodes to SP2 while ensuring that the remaining nodes can sustain the running workloads of the services and applications in your cluster and you maintain quorum.

1) Download Windows Server 2008 SP2 for your cluster nodes:

· x86: http://www.microsoft.com/downloads/details.aspx?FamilyID=a4dd31d5-f907-4406-9012-a5c3199ea2b3

· x64: http://www.microsoft.com/downloads/details.aspx?FamilyID=656c9d4a-55ec-4972-a0d7-b1a6fedf51a7

· ia64: http://www.microsoft.com/downloads/details.aspx?FamilyID=e890b3cf-972b-483f-a2ff-03f6aefac6f8

· ISO: http://www.microsoft.com/downloads/details.aspx?FamilyID=9f073285-b6ef-4297-85ce-f4463d06d6cb

2) Move all resources to the first node or subset of nodes (in my 2-node cluster I will move them to node 1). A best practice is to also pause the node(s) you are about to upgrade by right-clicking the node name and selecting 'Pause'. This will prevent resources from failing over to them (this has been skipped in this blog post).

3) Upgrade the other node or subset of nodes (those which are not hosting any HA resources)

4) Move resources to the second node or group of nodes (if it was paused, resume it first). It is best to be in this 'mixed version' state for as little time as possible. You may want to control the failover of your Core Cluster Group or Witness Disk which can be done using PowerShell or Cluster.exe, however this is not required as these resources will automatically failover during the upgrade.

5) Upgrade the first node or group of nodes (those which are not hosting any HA resources). You may want to pause the node(s) first.

6) Redistribute your resources across your nodes. Resume the node(s) first, if necessary.

All of your nodes are now running Windows Server 2008 SP2 and you have finished you rolling upgrade. A KB article for adding service packs to any version of clustering is available at: http://support.microsoft.com/default.aspx/kb/174799.

Thanks,

Symon Perriman
Program Manager
Clustering & High-Availability
Microsoft

Creating and Configuring a Generic Service Resource

msclustm — Tue, 09 Jun 2009 03:43:00 GMT

With Windows Server 2008 (and R2) Failover Clustering you can make almost anything highly-available. This does not only include programs and applications, but you can make any service running on the cluster highly available – even if it is custom or from a 3^rd party. These can be created, managed by and integrated with Failover Clustering using a generic container, with services using the Generic Service resource type. Generic Service resource type allows us to manage Windows Services as cluster resources. They are similar to the Generic Application resource types in that they also provide basic functionality. For example, when a service is online it means that the service is running. Additionally Service status (such as online, offline, failed etc) can be determined by querying the Services Control Manager (SCM) (more information).

How to Create/Configure a Generic Service Resource

In this section, we will cover how to create and configure a generic service resource using the following methods:

1. High Availability wizard

2. PowerShell cmdlets

3. Cluster.exe command-line tool (to be deprecated after Windows Server 2008 R2)

We will be using Application Experience service as an example. This service ships in Windows Server 2008 R2.

Checklist

Before creating a generic service resource, please review the checklist at below link.
You need to ensure that the service is installed on all clustered nodes.

http://technet.microsoft.com/en-us/library/cc758806.aspx

Cluster Configuration

I have a 2-node cluster running Windows Server 2008 R2. Cluster name is A1C1F4X64.

Name of my cluster nodes are:

1. A1C1F4X64N1

2. A1C1F4X64N2

Below is a partial screenshot of Failover Cluster Manager.

Create a Generic Service Resource Using High Availability Wizard

Now I will walk you through the High Availability wizard of Failover Cluster Manager to create a generic service resource. Below are the steps:

1. Launch Failover Cluster Manager from Start menu. Connect to the cluster.

2. Right click on Services and applications, and click on Configure a Service or Application…

3. High Availability Wizard appears. After reading the information presented on Before You Begin page, click Next.

4. On Select Service or Application page, select Generic Service and click Next.

5. Select Service page appears. This page lists all the services that are running on the clustered nodes. Select “Application Experience” and click Next.

6. On Client Access Point page, we will be providing input for the network name and IP addresses that clients will be using when accessing our highly available generic service.
I am choosing a network name (A1C1F4X64GSvc) and a static IP address suitable for my network configuration.

Click Next.

7. On Select Storage page, you have an option to choose a disk for the generic service. If the generic service needs a disk resource, you can select a disk.
In my example, Application Experience service is not going to use any disk. So, we are not making any selection.

Click Next.

8. On Replicate Registry Settings page, add the registry keys that the service will be using/updating. These are the keys that the service requires for it to function properly. Once added, the registry keys will be replicated on all cluster nodes so that the service will be functional on all nodes of the cluster.

In our example, Application Experience service does not require presence of any specific registry key to be functional. However, for the purpose of demonstrating an example, we will add two registry keys.

First click on the Add… button. This should bring the Registry Key dialog where you can input the registry key. After entering the registry key, click OK.

By following this step, we have added two registry keys as shown in the below screen.

Click Next.

9. On Confirmation page, verify that the information is correct. If you need to make modifications, you can use Previous button to go back to the previous pages in the wizard and modify the information.

Click Next.

10. The Wizard should successfully create the generic service resource and bring the resource online. A Summary page appears.

11. On Summary page, you can click on View Report… button. This will show you detailed report of what actions were taken to create the generic service resource.

The report will also be located under %SystemRoot%\Cluster\Reports directory for later viewing.

12. Locate the newly created item (A1C1F4X64GSvc) under Services and applications. This is the container (also known as group) for the generic service resource.

Above image shows that group A1C1F4X64GSvc is online on N2. The name of the generic service resource is “Application Experience”.

Create a Generic Service Resource Using PowerShell Cmdlets for Failover Clustering

I will now show you how to use PowerShell cmdlets to create a generic service resource. PowerShell is new to Failover Clustering in Windows Server 2008 R2 and will replace Cluster.exe in the next release (more information about WSFC PowerShell). The cmdlets that we need to achieve this are: Get-Cluster and Add-ClusterGenericServiceRole.

If we are running PowerShell on a machine that is not part of the cluster, we have to retrieve the cluster object first using the Get-Cluster cmdlet. Once we retrieve the cluster object, the object can then be passed to the Add-ClusterGenericServiceRole cmdlet. This is needed to specify the cluster where the resource will be created.

If we are running PowerShell on a clustered node, Get-Cluster cmdlet may be omitted and Add-ClusterGenericServiceRole can be used directly. In this case cluster object will be retrieved from the local node (i.e. the node where the command is executing).

We also need to identify the service name that we want to cluster. To do this, launch Services.msc (Service Control Manager), and launch the properties window for the service. Service name is highlighted in the below screenshot.

Here is an example command line to create the generic service resource (Note that A1C1F4X64 is our cluster name):

PS C:\Windows\system32> get-cluster A1C1F4X64 | Add-ClusterGenericServiceRole -ServiceName AeLookUpSvc -Name A1C1F4X64GSVC -StaticAddress 172.24.11.96

Report file location: C:\Users\wolfpack\AppData\Local\Temp\tmpF8BA.tmp.mht

Name OwnerNode State

---- --------- -----

A1C1F4X64GSVC a1c1f4x64n2 Online

To get further help, you can issue the below command on a PowerShell window:

Get-Help Add-ClusterGenericServiceRole

The following command will show you example usage:

Get-Help Add-ClusterGenericServiceRole -examples

For extended information, you can use:

Get-Help Add-ClusterGenericServiceRole -full

Create a Generic Service Resource Using Cluster.exe Command Line Tool

Cluster.exe is another command line tool that can be used to administer a cluster, however Windows Server 2008 R2 is the final release, so it is recommended to use PowerShell to create new scripts and utilities. Cluster.exe will coexist with PowerShell in Windows Server 2008 R2, however you will be able to see that PowerShell significantly simplifies the process.

Below is a sample script that creates and onlines a generic service resource.

REM Store cluster name in a variable

set ClusterName=A1C1F4X64

REM Create a group and online it

cluster %ClusterName% group GenSvcGroup /create

cluster %ClusterName% group GenSvcGroup /on

REM Create a IP address resource, set required properties, and online it

cluster %ClusterName% res GenSvcIP /create /group:GenSvcGroup /type:"IP Address"

cluster %ClusterName% res GenSvcIP /priv address=172.24.11.96 SubnetMask=255.255.255.0

cluster %ClusterName% res GenSvcIP /on

REM Create a network name, set properties, set dependency on the IP resource, and online it

cluster %ClusterName% res GenSvcNN /create /group:GenSvcGroup /type:"Network name"

cluster %ClusterName% res GenSvcNN /priv Name=%ClusterName%-NN

cluster %ClusterName% res GenSvcNN /setdep:"[GenSvcIP]"

cluster %ClusterName% res GenSvcNN /on

REM Create a generic service resource, set the properties, and online it

cluster %ClusterName% res GenSvcRes /create /group:GenSvcGroup /type:"Generic Service"

cluster %ClusterName% res GenSvcRes /prop RestartAction="0"

cluster %ClusterName% res GenSvcRes /priv ServiceName=AeLookUpSvc

cluster %ClusterName% res GenSvcRes /priv StartupParameters="-k netsvcs"

cluster %ClusterName% res GenSvcRes /on

The script does the following:

1. Creates and onlines a group called GenSvcGroup.

2. Creates an IP address resource called GenSvcIP in group GenSvcGroup. Sets Address and SubnetMask properties of the resource. Brings the IP resource online.

3. Creates a network name resource called GenSvcNN. Sets the Name property of the network name resource, sets the dependency of the network name to the IP resource, and brings the network name online.

4. Finally, it creates the generic service resource, sets the properties, and brings the resource online.

Resources

· Generic Service resource type - http://technet.microsoft.com/en-us/library/cc784978.aspx

· Checklist: Installing a Generic Service resource - http://technet.microsoft.com/en-us/library/cc758806.aspx

· Cluster-Unaware applications - http://msdn.microsoft.com/en-us/library/aa369166(VS.85).aspx

· Generic Service - http://msdn.microsoft.com/en-us/library/aa369600(VS.85).aspx

· Windows Clustering - http://msdn.microsoft.com/en-us/library/aa373130(VS.85).aspx

· Configuring and Creating a Generic Application resource: http://blogs.msdn.com/clustering/archive/2009/04/10/9542115.aspx

In the upcoming weeks I will be adding a post about creating and configuring Generic Scripts.

Regards,
Daud Howlader
Software Development Engineer in Test
Clustering & High Availability
Microsoft

Deploying IIS on a Failover Cluster

msclustm — Mon, 01 Jun 2009 05:20:00 GMT

Hi cluster fans,

Our general recommendation for achieving high-availability of Internet Information Services (IIS) servers is by using Network Load Balancing (NLB) since the servers tend to be hosts for stateless requests – a customer will visit a website, access content, then leave, and it doesn’t matter which server hosts the website. Additionally NLB can provide affinity between clients and cluster nodes so clients can stay connected to the same server for the duration of their session. Using NLB, the same data is stored on multiple servers, so if one becomes unavailable, the client is redirected to another server with the same information. You can learn more about NLB using IIS here: http://technet.microsoft.com/en-us/library/cc770558.aspx & http://learn.iis.net/page.aspx/213/network-load-balancing.

However there has been a demand for IIS high-availability using Failover Clustering, so persistent data can be saved on shared storage. Some use Failover Clustering to monitor the IIS Service (specifically the WWW Service) to ensure that it stays up and running, and can restart or failover if it crashes. In the past, it was still possible to use IIS in a Failover Cluster by using the Generic Script resource type, but custom scripts were required which were often not optimized for high-availability.

We have recently published a KB article which includes a detailed end-to-end guide and script for you to successfully deploy IIS on a Failover Cluster.

For more information, the guide and the script, please visit: http://support.microsoft.com/kb/970759

Thanks,
Symon Perriman
Program Manager
Clustering & HA
Microsoft

High-Availability Education from Hyper-V and Exchange

msclustm — Tue, 26 May 2009 17:43:00 GMT

Hi cluster fans,

Our partners on the Exchange and Hyper-V team have recently published some great resources to improve your high-availability knowledge.

Hyper-V: Free Training

Microsoft is offering some free training for Hyper-V which includes modules on HA and Failover Clustering.

Free eLearning: Exam 70-652: TS: Windows Server Virtualization, Configuring

This collection of five 2-hour courses helps you develop the skills necessary for implementing and managing Hyper-V in an IT environment, as well as, creating and managing virtual machines and hosts in a virtual environment.

Use promo code: 9350-Y2W6-3676

[Update] 6/9/09 – There is a new website needed to use this code. Please visit http://www.microsoft.com/learning/access and submit the promo code. It will ask you to sign-in then give you access to these courses. Thanks for your patience!

The courses within the collection include the following topics:

· An overview of the Hyper-V technology

· Creating a virtual environment

· Deploying systems in a virtual environment

· Configuring high availability in a virtual environment

· Administering a virtual environment with SCVMM

Details: https://www.microsoftelearning.com/eLearning/offerDetail.aspx?offerPriceId=228779

Exam Details: http://www.microsoft.com/learning/en/us/exams/70-652.mspx

Free eBook: Understanding Microsoft Virtualization Solutions

This guide will teach you about the benefits of the latest virtualization technologies and how to plan, implement, and manage virtual infrastructure solutions. The technologies covered include: Windows Server 2008 Hyper-V, System Center Virtual Machine Manager 2008, Microsoft Application Virtualization 4.5, Microsoft Enterprise Desktop Virtualization, and Microsoft Virtual Desktop Infrastructure.

Details: http://csna01.libredigital.com/?urmvs17u33

Exchange: Forefront and Webcasts

Guide: Installing Forefront on Clustered Exchange Servers

Clustered installations have become increasingly popular for Microsoft® Exchange Server 2007 which can be installed using both Cluster Continuous Replication (CCR) and Single Copy Cluster (SCC) configurations. Microsoft® Forefront Security™ for Exchange Server (FSE) can then be installed on Exchange mailbox servers in clustered systems. FSE supports volume mount points. The Forefront Security for Exchange Server (FSE) Cluster Installation Guide provides instructions for installing FSE in a clustered Exchange 2007 environment.

Details: http://technet.microsoft.com/en-us/library/bb892168.aspx

Webcasts: High Availability in Exchange Server 2010

Exchange Server 2010 includes a new unified platform for both high availability and site resiliency with significant changes at its core that integrate high availability deep in its architecture, making it even less costly and easier to deploy and maintain than previous versions of Exchange. Principal Technical Writer Scott Schnoll explores High Availability in Exchange 2010 in this two-part webcast.

Part 1 shows the concept of database mobility, and the features known as database availability groups (DAGs) and mailbox database copies.

Part 2 illustrates how to create a DAG and how to add Mailbox servers to a DAG.

Webcast Part 1: http://msexchangeteam.com/videos/9/drandha/entry451348.aspx

Webcast Part 2: http://msexchangeteam.com/videos/9/drandha/entry451399.aspx

Thanks,
Symon Perriman
Program Manager
Clustering & High Availability
Microsoft

PowerShell for Failover Clustering: Frequently Asked Questions

msclustm — Sat, 23 May 2009 04:29:00 GMT

Hi,

Today I’m going to talk a little more about PowerShell by answering some of the frequently asked questions I hear.

1) Why is PowerShell better?

2) How do I get started using PowerShell?

3) Where are the properties?

4) Where are the private properties?

5) How do I enable CSV?

1) Why is PowerShell better?

To get you excited I will talk about some cool stuff in our cmdlets. When you compare the functionality to that of Cluster.exe you will see that PowerShell is much better. All the pointers below could be topic in themselves, so for now I am not getting into details.

1) Failover Cluster Manager has been liked by many users, and the best part of cluster administrator is its simplicity of use, including wizards which walk you through many common operations.

However our cmdlets do not show any GUI or wizards, they are purely CLI and scriptable.

What is my point? Well we have tried to provide the Failover Cluster Manager type of experience in PowerShell. For now, I would leave this for you to discover what I am talking about :)

2) Running Validation. Using PowerShell cmdlets you can run the “Validate a Configuration…” wizard on a clusters or unclustered nodes. This even gives you the ability to run validation directly on Core installations.

3) We have many cmdlets to manage VMs, including live migration. Yes, I am talking about writing PowerShell ccripts to live migrate VMs. How cool is it to have a script which runs every few seconds or minute to monitor the cluster and its health, and based on some conditions re-balance the resources by live migrating VMs to different nodes.

4) If I am talking about VMs, I do need to mention CSV. I have already talked about how to enable CSV in PowerShell, but apart from that we do have a set of cmdlets to manage CSV disks.

5) Creating roles have been simplified through PowerShell cmdlets. For this release we have one line cmdlet to create roles (look at the Add-Cluster*role cmdlets).

6) We also have exposed a cmdlet to generate dependency report that can be saved. This allows you to render dependency reports directly on Core installations.

2) How do I get started using PowerShell?

I guess this might not be new to some of you, but I want to make sure that I have it in this article, so that you do not have to search for it. There are two ways to enable PowerShell.

1) In the Windows Server 2008 R2 Release Candidate (RC) build and later, there will be a ‘Windows PowerShell Modules’ link under your Administrative Tools which will automatically import all the PowerShell modules for features or roles which you have installed. It even runs it as an Administrator for you by default, so you don’t need to elevate the permissions every time.

You can then immediately start using Failover Clustering (or Network Load Balancing NLB) PowerShell cmdlets in the command prompt.

The RC build is available here: http://technet.microsoft.com/en-us/dd362341.aspx

2) To be able to use PowerShell cmdlet's you need to run PowerShell in Administrative Mode and run below cmdlet

Import-Module FailoverClusters

My suggestion would also be to create a default profile for yourself and have this line added, so that the module gets loaded every time you open a PowerShell Windows. In Windows Server 2008 R2, you do have the option to open a PowerShell Window and import all the modules on the machine.

Please note that modules are loaded in a session and will only remain while the session exists.

I have always found the output of the below command helpful. This makes a table of cmdlets with cluster cmdlet name and a synopsis/description. The description is pulled from the Help content of the cmdlet.

Feel free to modify this simple cmdlet to extract the information that you would need:

Get-Command -Module FailoverClusters | %{Get-Help $_.Name} | ft Name,Synopsis –Wrap

For Network Load Balancing, use NetworkLoadBalancingClusters

3) Where are the properties?

The cluster objects are: cluster node, cluster group, cluster resource, cluster network, cluster network interface, resource types and the cluster, itself. As I mentioned earlier our cmdlets are action oriented and you would be performing these actions on any of the above listed cluster objects. Most of the cmdlets expect to return a cluster object back.

These objects are the container of their own properties.

So for example, let say you want to see the property of a resource. Using cluster.exe you would type:

Cluster.exe res <Resource Name> /prop

To get the same data from Cluster PowerShell cmdlets you would

Get-ClusterResource <Resource Name> | Format-List *

Get-ClusterResource returns a ClusterResource object. This object contains all the pre-defined properties of the resource. But if you just type

Get-ClusterResource <Resource Name>

You would only see some selected information about the resource on the output screen like resource name, resource group, resource state and resource type. Also keep in mind the object that the cmdlet returns contains additional information which is hidden by default.

When you pipe the out of Get-ClusterResource to Format-List , the Format-List cmdlet takes the object and displays it as a list. The second parameter to Format-List cmdlet '*' will show all values, thus you get to see all the properties of the ClusterResource object as a table of Property Name and Value. You may also type ‘fl’ instead of ‘Format-List’.

So how do you modify the Value of these properties?

There are various ways to do this. You can either cache the object, access the property name and set new value for the property, or you can do the same thing on the fly, without caching it.

For example Let say you want to change the RestartDelay property of a resource. You would use:

$res = Get-ClusterResource <resource Name>

$res.RestartDelay=<new value>

And you are done.

You can present a counter argument that I can do the same with Cluster.exe and it is as simple as:

Cluster.exe res <resource name> /prop <propertyname>=<value>

I agree with you. But how easy is it to do this for all the resource on your cluster? The power of PowerShell helps us here. The below example would update a property of all the resource

Get-ClusterResource | For-Each { $_.<PropertyName>=<NewValue>}

What if you only wanted to change the value of property for specific resource type and not on all resource?

Yes, PowerShell does provide you with filtering options and you would be writing something like:

Get-ClusterResource | where-object { $_.ResourceType -ilike "<resourcetype" } | for-each { $_.<Property Name>=<value>}

Isn't that neat and simple? This is where you get the power of PowerShell with its rich language (both cmdlet’ing and scripting).

4) Where are the private properties?

Private properties? Well for now we would use cluster parameters to refer private properties. Let’s not get into the discussion of why they’re called cluster parameters and why not private properties. We’ll leave that for another blog :).

One of the unusual things about cluster parameters is that they are dynamic. The properties can be added or removed. Another challenge is with the data type of these properties and the way cluster keeps this information.

For now, let’s focus on Get-ClusterParameter and Set-ClusterParameter. Get-ClusterParameter is equivalent to /priv parameter in Cluster.exe

If you are interested in looking at the parameters of a resource, you use

Get-ClusterResource <resource name>| Get-ClusterParameter

The above cmdlet would list all the parameters of the given resource.

To Change the value of a parameter, enter

Get-ClusterResource <resource name> | Set-ClusterParameter -Name <parameter name> -Value <new value>

5) How do I enable CSV?

We do not have any cmdlets which enable Cluster Shared Volumes (CSV). Please do not misunderstand me, I am not saying that we cannot enable CSV feature from PowerShell.

Cluster objects have a property that controls enabling and disabling of CSV. In Failover Cluster Manager there is actually no way to disable CSV.

To enable CSV on a cluster using PowerShell cmdlets, use

$clus = Get-Cluster <cluster name>

$clus.EnableSharedVolume="Enabled"

The above cmdlet would not enable the CSV feature, but rather it should show you the terms and restrictions associated with using CSV. You will want to read these terms to better understand how and when CSV should be used.

For other CSV-related cmdlets, try

Get-Command -CommandType cmdlet *ClusterSharedVolume

Thanks,
Vikas Kumar
Software Development Engineer in Test
Clustering & High Availability

Microsoft

PowerShell for Failover Clustering: Getting Started

msclustm — Thu, 21 May 2009 04:39:00 GMT

Hi Cluster Fans,

Amongst the long list of features that is being added in Windows Server R2, Clustering team is also providing a set of PowerShell cmdlets (“command-lets”) to manage your Failover Cluster. Hopefully this is not news for most cluster administrators as we have publishing information on the blog and in our webcasts. We have written a guide for using PowerShell in the Beta release and will add additional information shortly for the RC release: PowerShell for Failover Clustering in Windows Server 2008 R2.

Cluster PowerShell cmdlets have come a long way from Beta to RC. There have been quite a few changes in some of the cmdlets, cmdlet names and parameter names. These changes have been made to improve stability, consistency among cmdlets and to ensure that they are self-discoverable.

Naming Convention

All Cluster cmdlets adhere to the naming convention guide line set by the PowerShell Team. Every cmdlet can be broke in two parts, a verb and a noun.

While designing the names of these cmdlets, we took action oriented approach. What I mean is, all our cmdlets corresponds to an action that can be performed on a cluster object. This should give administrators a jump start with using clustering cmdlets to reduce the learning curve.

Verbs

Below is a table that lists all the verbs used by cluster cmdlets with an explanation. The table does not detail the cmdlets. The Idea is to help familiarize yourself with the verbs and that should help a user discover the right cmdlets for the operation.

Verb	Description
Add	The verb is self explanatory. All the cmdlets that start with the verb Add would add the noun to the Cluster. For example, Add-ClusterDisk, would add Disks to the cluster, Add-ClusterGroup would add a group to the cluster.
Block	We have only one cmdlet that has the Block verb and that is Block-ClusterAccess. This cmdlet is used to manage user permissions and block would deny permissions on the cluster.
Clear	Cmdlets starting with Clear should help an administrator recover or re-claim a lost node or disk due to incomplete cleanup or information still lingering on them due to incomplete cleanup.
Get	This should be the simplest and easily discoverable. The Get-Cluster* cmdlets are to query either individual or all cluster object. For example, Get-ClusterNode is equivalent to Cluster.exe Node, Get-ClusterGroup is equivalent to Cluster.exe Group.
Grant	Cmdlets starting with Grant are among the set of cmdlets available to manage user permissions on Cluster. This can be paired up with Block.
Move	If you need to perform a Move operation on a cluster object, you need to be looking at cmdlets that start with Move to update the location of a cluster resource or group.
New	There is only one cmdlet that start with New and that is New-Cluster used to create a cluster.
Remove	Cmdlets starting with Remove should help you delete\destroy a cluster object that you do not need. For example, Remove-ClusterGroup, will delete a group. Remove-Cluster should destroy the cluster.
Repair	We have a cmdlet Repair-ClusterShareVolume that would enable us to Run ChkDisk and Defrag on a CSV Volume. This cmdlet needs to be run on a cluster node.
Resume	Any cluster object that can be suspended (paused) can be resumed using Resume-Cluster* cmdlets.
Set	This verb is not as obvious and a little harder to understand, but not really if you continue reading :). Set-Cluster* cmdlets were introduced to make some non-trivial tasks trivial, for example, setting values of cluster object parameter ( private properties), setting Group preferred owner list, or resource owner list.
Start	Cmdlets starting with Start are equivalent to Online or bringing a cluster resource up. For example, Start-ClusterResource, Start-ClusterGroup, Start-Cluster, Start-ClusterNode.
Stop	Stop is equivalent to Offline.
Suspend	After knowing about Resume, you can guess that Suspend is used to pause a resource.
Test	Some interesting and more common cmdlets start with Test verb. There are two cmdlets that start with Test, namely Test-Cluster (to run the “Validate a Configuration…” wizard) and Test-ClusterResourceFailover (to simulate a resource failure).
Update	Again two cmdlets start with Update. We have Update-ClusterIPResource (used to renew\release the DHCP IP Address leases) and Update-ClusterVirtualMachineConfiguration (used to refresh a VM’s information after a change)

I understand that there is change in terminology and it could take a while for us to get used to it, but it is a necessary design decision to align with the changes in technology.

I would also agree that some of you that the selection of verbs is somewhat inconsistent with our UI, but this is to align with the larger Windows Server goal of consistent naming conventions, which gives us a limited set of verbs to use.

Nouns

Now let’s move on to the nouns…

The interesting part of the cmdlets is the noun, the second part of the cmdlet. The first word is usually Cluster followed by a cluster object. For example, resource is a cluster object, so the appropriate noun would be ClusterResource, group is a cluster object, so it's ClusterGroup.

We do have a smaller set of cmdlets that have only one word (as noun) and that is cluster. Some examples are New-Cluster, Get-Cluster.

With some knowledge of verbs and nouns, you should have some guidance towards discovering the cluster cmdlet you need.

Discovering Parameters

One of the objectives of PowerShell is self-discovery. If you type '-' and press <tab> you should get the name of parameters which the cmdlet accepts. If you press <tab> again you should get the next parameters.

Another way to learn about the cmdlet is to view the built-in help topic. I know as of today there is no MSDN documentation available for the cmdlets, but do you really need it? Have you tried Get-Help on any of Cluster cmdlet? If not, please try it and send us any feedback.

This should not be new for any PowerShell user and don’t worry, I do not intend to talk about Get-Help and using <-><tab> in this section.

Some simple things first…

All our cmdlet's can be run from a cluster node or from a remote client. The cmdlets ship as part of admin pack and works with PowerShell V2.

For these cmdlet's to work against a cluster, we do need to provide cluster name. There are multiple ways to provide cluster information to a cmdlet.

1. Every cmdlet has a -Cluster parameter, where you can specify the name of the cluster, or an IP address of the cluster, or the name of a Network Name resource type hosted on the cluster. If I am confusing you, sorry, but for now just keep in mind the -Cluster parameter.

2. Most of our cmdlet do accept Cluster Object as piped in input. And a cluster object can be obtained by using Get-Cluster -Name <Cluster Name>. Get-Cluster is the only cmdlet that does not have a -Cluster parameter.

3. If you are executing a command on a cluster node, -Cluster need not be specified. The cmdlets are usually intelligent enough to detect that you are working on cluster node. But if you are managing multiple clusters from a cluster node, please use the -Cluster parameter to address appropriate cluster, if not specified, the cmdlet would assume that you are working with the local cluster.

4. The -Name parameter allows you to specify an object name representing the noun in the cmdlet. For example, Get-ClusterResource -Name <resource name>. If you provide a group name or any other name that is not a resource, the cmdlet would return an error.

5. Positional parameters! Positional parameters are parameter values that are not paired up with the parameter name. The Idea behind positional parameters is that a user does not always type the parameter name, if the values can be identified based on position. Most of our cmdlets are expected to work with positional parameters. In most cases the value for -Name is positional parameter, but there are some cases where the noun can be passed in as a piped in object and in that cases the position shifts.

For example, you could do the below to move a group to node n1

Get-Clustergroup <group name> | Move-ClusterGroup n1

In the example above <group name> is taking the advantage of positional parameter as I have not linked the value with any parameter, but then Move-ClusterGroup accepts -name <group name> -node <destination node>. In the example above, the cmdlet is getting the group object through piped input, thus it assumes that the parameter passed to it is a node name.

The Idea here is to keep things simple and make them work in most obvious cases.

Cluster.exe to PowerShell Comparison

To help you get started with the cmdlets, I built this table which shows some common cluster.exe command and their matching PowerShell cmdlets.

Cluster.Exe Command	PowerShell Command
Cluster.exe <Cluster Name> /prop	Get-Cluster <Cluster Name> \| fl *
Cluster.exe <Cluster Name> /priv	Get-Cluster <Cluster Name> \| Get-ClusterParameter
Cluster.exe <Cluster Name> group	Get-ClusterGroup -Cluster <Cluster Name>
Cluster.exe <cluster Name> node	Get-ClusterNode -Cluster <Cluster Name>
Cluster.exe <cluster name> res	Get-ClusterResource -Cluster <Cluster name>
Cluster.exe <Cluster name> net	Get-ClusterNetwork -Cluster <cluster name>
Cluster.exe <cluster name> netint	Get-ClusterNetworkInterface -Cluster <cluster name>
Cluster.exe <cluster name> group /move:<Node name>	Move-ClusterGroup -Name <group name> -Node <Node name> -Cluster <cluster name>
Cluster.exe log	Get-ClusterLog
Cluster.exe <cluster name> /create …	New-Cluster
Cluster.exe <cluster name> group /add	Add-ClusterGroup
Cluster.exe <cluster name> group <group name> /on	Start-ClusterGroup -Name <group name> -Cluster <cluster name>

That is all for now, please check back later for more PowerShell information.

Thanks,

Vikas Kumar

Software Development Engineer in Test

Clustering & High-Availability

Microsoft

Microsoft iSCSI Target Software available to the public!

msclustm — Thu, 14 May 2009 04:28:00 GMT

Hi cluster fans,

Many of you ask about the cheapest way to deploy your own cluster. We all know that the hardware requirements are very flexible, so long as the components have a Windows Server logo and the complete solutions passes Validate (support policy) … but what about storage?

Clustering does have a few storage requirements. The storage must be shared between the cluster nodes so that when a node fails and the other node picks up the workload, it will be able to access the same data which is on that shared disk. You may use Fibre Channel, Serial Attached SCSI (SAS) or iSCSI, and the device must support persistent reservations (hardware requirements). But networked storage can be expensive…until now!

Windows Storage Server 2008 (WSS 2008) with the Microsoft iSCSI Software Target 3.2 is now available to all MSDN and TechNet Plus subscribers. A version for evaluation (TechNet Plus or MSDN), demonstration (MSDN), development (MSDN) or test (MSDN) is now being provided to MSDN and TechNet Plus subscribers for the first time. Rather than using expensive shared storage you simply run Storage Server on another machine. All communication is done over regular Ethernet cables, so HBAs or an expensive SAN are not needed. With just any three computers (2 cluster nodes and 1 iSCSI Target / DC) and a hub you can have your own Failover Cluster! Now there are no excuses for not trying out clustering.

Full details and a FAQ are available at: http://blogs.technet.com/josebda/archive/2009/05/12/windows-storage-server-2008-with-the-microsoft-iscsi-software-target-3-2-available-to-msdn-and-technet-plus-subscribers.aspx

Thanks,

Symon Perriman

Program Manager

Clustering & High Availability

Microsoft