Clustering and High Availability

Failover Clustering Performance Counters – Part 4 – Command Line

2009-11-10T06:23:00Z

Hi Cluster Fans,

Most of you are familiar with the Performance Monitor that allows you to work with performance counters. Details have been described in the three previous posts in the series: Part 1, Part 2 and Part 3.

Using the command line, there are several functions which you may find useful. For information about how do this using PowerShell for Failover Clustering, please visit this earlier blog post: http://blogs.msdn.com/clustering/archive/2009/07/22/9844473.aspx.

Typeperf.exe allows you to enumerate, monitor and collect performance counters from the command line. The command bellow show how to enumerate all clustering performance counters

>typeperf.exe -q | findstr Cluster

\Cluster Shared Volumes(*)\Metadata IO Delta

\Cluster Shared Volumes(*)\Metadata IO

\Cluster Shared Volumes(*)\Redirected Read Bytes Delta

\Cluster Shared Volumes(*)\Redirected Read Bytes

\Cluster Shared Volumes(*)\Redirected Reads Delta

\Cluster Shared Volumes(*)\Redirected Reads

\Cluster Shared Volumes(*)\Redirected Write Bytes Delta

\Cluster Shared Volumes(*)\Redirected Write Bytes

\Cluster Shared Volumes(*)\Redirected Writes Delta

\Cluster Shared Volumes(*)\Redirected Writes

\Cluster Shared Volumes(*)\IO Read Bytes Delta

\Cluster Shared Volumes(*)\IO Read Bytes

\Cluster Shared Volumes(*)\IO Reads Delta

\Cluster Shared Volumes(*)\IO Reads

\Cluster Shared Volumes(*)\IO Write Bytes Delta

\Cluster Shared Volumes(*)\IO Write Bytes

\Cluster Shared Volumes(*)\IO Writes Delta

\Cluster Shared Volumes(*)\IO Writes

\Cluster Resource Control Manager\Groups Online

\Cluster Resource Control Manager\RHS Restarts

\Cluster Resource Control Manager\RHS Processes

\Cluster Global Update Manager Messages\Update Messages Delta

\Cluster Global Update Manager Messages\Update Messages

\Cluster Global Update Manager Messages\Database Update Messages Delta

\Cluster Global Update Manager Messages\Database Update Messages

\Cluster API Calls\Batch API Calls Delta

\Cluster API Calls\Network Interface API Calls Delta

\Cluster API Calls\Network API Calls Delta

\Cluster API Calls\Cluster API Calls Delta

\Cluster API Calls\Key API Calls Delta

\Cluster API Calls\Resource API Calls Delta

\Cluster API Calls\Group API Calls Delta

\Cluster API Calls\Node API Calls Delta

\Cluster API Calls\Notification API Calls Delta

\Cluster Checkpoint Manager\Crypto Checkpoints Restored Delta

\Cluster Checkpoint Manager\Crypto Checkpoints Restored

\Cluster Checkpoint Manager\Crypto Checkpoints Saved Delta

\Cluster Checkpoint Manager\Crypto Checkpoints Saved

\Cluster Checkpoint Manager\Registry Checkpoints Restored Delta

\Cluster Checkpoint Manager\Registry Checkpoints Restored

\Cluster Checkpoint Manager\Registry Checkpoints Saved Delta

\Cluster Checkpoint Manager\Registry Checkpoints Saved

\Cluster Network Messages(*)\Bytes Received Delta

\Cluster Network Messages(*)\Bytes Received

\Cluster Network Messages(*)\Bytes Sent Delta

\Cluster Network Messages(*)\Bytes Sent

\Cluster Network Messages(*)\Messages Received Delta

\Cluster Network Messages(*)\Messages Received

\Cluster Network Messages(*)\Messages Sent Delta

\Cluster Network Messages(*)\Messages Sent

\Cluster Network Reconnections(*)\Reconnect Count

\Cluster Network Reconnections(*)\Unacknowledged Message Queue Length Delta

\Cluster Network Reconnections(*)\Unacknowledged Message Queue Length

\Cluster Network Reconnections(*)\Normal Message Queue Length Delta

\Cluster Network Reconnections(*)\Normal Message Queue Length

\Cluster Network Reconnections(*)\Urgent Message Queue Length Delta

\Cluster Network Reconnections(*)\Urgent Message Queue Length

\Cluster Database\Flushes Delta

\Cluster Database\Flushes

\Cluster API Handles\Batch Handles Delta

\Cluster API Handles\Batch Handles

\Cluster API Handles\Network Interface Handles Delta

\Cluster API Handles\Network Interface Handles

\Cluster API Handles\Network Handles Delta

\Cluster API Handles\Network Handles

\Cluster API Handles\Cluster Handles Delta

\Cluster API Handles\Cluster Handles

\Cluster API Handles\Key Handles Delta

\Cluster API Handles\Key Handles

\Cluster API Handles\Resource Handles Delta

\Cluster API Handles\Resource Handles

\Cluster API Handles\Group Handles Delta

\Cluster API Handles\Group Handles

\Cluster API Handles\Node Handles Delta

\Cluster API Handles\Node Handles

\Cluster API Handles\Notification Handles Delta

\Cluster API Handles\Notification Handles

\Cluster Multicast Request-Response Messages\Messages Outstanding

\Cluster Multicast Request-Response Messages\Messages Sent Delta

\Cluster Multicast Request-Response Messages\Messages Sent

\Cluster Resources(*)\Resource Type Controls Delta

\Cluster Resources(*)\Resource Type Controls

\Cluster Resources(*)\Resource Controls Delta

\Cluster Resources(*)\Resource Controls

\Cluster Resources(*)\Resource Failure Deadlock

\Cluster Resources(*)\Resource Failure Access Violation

\Cluster Resources(*)\Resource Failure

\Cluster Resources(*)\Resources Online

The next command shows how you can use typeperf to watch counters from the command line.

>typeperf.exe "\Cluster API Calls\Resource API Calls Delta"

"(PDH-CSV 4.0)","\\VPCLUS01\Cluster API Calls\Resource API Calls Delta"

"06/03/2009 17:34:43.647","94.000000"

"06/03/2009 17:34:44.661","89.000000"

"06/03/2009 17:34:45.675","104.000000"

"06/03/2009 17:34:46.689","85.000000"

"06/03/2009 17:34:47.703","91.000000"

"06/03/2009 17:34:48.717","103.000000"

"06/03/2009 17:34:49.731","85.000000"

"06/03/2009 17:34:50.745","92.000000"

"06/03/2009 17:34:51.759","92.000000"

"06/03/2009 17:34:52.773","101.000000"

"06/03/2009 17:34:53.787","87.000000"

"06/03/2009 17:34:54.801","84.000000"

"06/03/2009 17:34:55.815","87.000000"

"06/03/2009 17:34:56.829","101.000000"

"06/03/2009 17:34:57.843","84.000000"

"06/03/2009 17:34:58.857","88.000000"

"06/03/2009 17:34:59.871","96.000000"

"06/03/2009 17:35:00.885","93.000000"

"06/03/2009 17:35:01.898","87.000000"

"06/03/2009 17:35:02.912","88.000000"

"06/03/2009 17:35:03.926","96.000000"

"06/03/2009 17:35:04.940","89.000000"

…

If you run “typeperf /?” you will find many other useful features you might like.

The other tool I’ve found handy is logman.exe. This is a very powerful utility and one of its features allows you to manipulate Data Collector Sets. You have probably already seen this feature in the Performance Monitor, but in perfmon you have to do everything manually. Logman allows you to create/start/stop/delete the sets from the command line. You can learn from the help a variety of things you can do with this tool, so instead of repeating the help here I want to share with you a batch file that will help you to manipulate the Data Collector Sets for the Failover Cluster counters. Enjoy it!

The script is available here: http://blogs.msdn.com/clustering/pages/9915526.aspx. Note that this script is a sample only and you should test this before using it in a production environment. This is not an officially supported script from Microsoft, so please use this at your own risk.

The Performance Monitor team has a post about the logman that can be used to manipulate perfmon logging sessions: http://blogs.technet.com/askperf/archive/2008/05/13/two-minute-drill-logman-exe.aspx.

To find more information about this script, you can run it with the /? parameter and see the following output:

> ClustPerf.cmd /?

clusperf.cmd [-c] [-stop] [-start] [-d] [-show] [-q] [-n {session name}] [-o {path to the log files}] [-f {counters filter}] [{list of nodes}]

-c - cerate new session.

aliases: create.

-start - starts session.

-stop - stops session.

-d - deletes session.

aliases: delete.

-show - prints session details.

aliases: q, query.

-q - same as show.

aliases: show, query.

-n {name} - Session name.

aliases: name.

default: WSFCCluster.

-o {path} - path to the trace files.

aliases: out.

default: c:\WSFCCluster.

-f {str} - filter for the counters we want to monitor.

aliases: flt, filter.

default: Cluster.

You can specify multiple commands and they will be executed in the following order

1.stop

2.dlete

3.create

4.start

5.show

Note that if you specify create then it already implies stop and delete.

Note that if you specify delete then it already implies stop.

You can provide list of nodes using clusnodes or myclusnodes environment variables

Thanks,
Vladimir Petter
Senior Software Development Engineer
Clustering & High-Availability
Microsoft

Microsoft Site Recovery Solutions Featuring Windows Server 2008 R2 Failover Clustering

2009-11-04T23:18:00Z

Hi cluster fans,

This week Microsoft is launching an end-to-end solution to help customers deploy integrated Site Recovery Solutions (http://blogs.technet.com/virtplanet/). This launch shows how Microsoft and partners like HP, EMC, HDS, NetApp, DataCore, DoubleTake and SteelEye are leveraging mature and proven technologies like clustering in combination with new virtualization technologies like Hyper-V to provide rich, high value solutions.

Main article: http://blogs.technet.com/virtplanet/

For more information, please check the following links:

· Cluster Resource Dlls: http://msdn.microsoft.com/en-us/library/aa372239(VS.85).aspx

· Cluster Resources & how to write them: http://msdn.microsoft.com/en-us/library/aa372152(VS.85).aspx

· Generic Application Installation: http://technet.microsoft.com/en-us/library/cc782179(WS.10).aspx & http://blogs.msdn.com/clustering/archive/2009/04/10/9542115.aspx

· Generic Script Installation: http://technet.microsoft.com/en-us/library/cc736970(WS.10).aspx & http://blogs.msdn.com/clustering/archive/2009/09/28/9900574.aspx

· Generic Service Installation: http://technet.microsoft.com/en-us/library/cc758806(WS.10).aspx & http://blogs.msdn.com/clustering/archive/2009/06/09/9712609.aspx

Thanks,

Jim Schwartz
Solutions Marketing Director

Virtualization

Microsoft

PowerShell for NLB: Part 1: Getting Started

2009-10-28T03:24:00Z

Hi NLB Fans,

NLB provides users with various methods to manage clusters. In Windows Server 2008, there are 3 ways to manage an NLB cluster:

1. Network Load balancing Manager GUI (nlbmgr.exe)

2. NLB command line tool (Nlb.exe)

3. NLB WMI Provider (root\MicrosoftNLB namespace)

In Windows Server 2008 R2, the NLB team has introduced a PowerShell interface for configuring, managing and debugging NLB. This awesome new feature makes it very easy to administer systems in an automated way.

In this blog post we will explore NLB's support for PowerShell. We will elaborate on the original post PowerShell for NLB, providing more details on naming mechanism, samples and CMDlet discovery.

This blog post contains the following sections:

· PowerShell Naming convention

· Exploring NLB CMDlets

o Using Get-Command

o Using command Auto-completion

o Using Argument auto completion

o Getting examples to use

Future blog posts in this series will discuss:

· NLB common scenarios

· Basics of Debugging NLB with PowerShell

NLB PowerShell follows the PowerShell CMDlet guidelines in naming and execution of the NLB CMDlets. Here we will explore the general naming conventions that will make it easy to further understand and explore NLB CMDlets.

PowerShell Naming Convention

A CMDlet is made up of two parts a Noun and a Verb. These two parts of speech are combined together with a hyphen in between. A NLB example would be:

PS > Get-NlbCluster

The ‘Get’ example above is split into 2 parts, the verb (Get) and the noun (NlbCluster), and these 2 words are separated by a hyphen. As rule of thumb, the verb defines the action to be performed on the noun. In the above example, we want to "Get" all instances of "NlbCluster".

To view all the NLB CMDlets, run PS > Get-Command –module NetworkLoadBalancingClusters

A list of all the NLB supported verbs can be seen below:

A list of all the NLB supported nouns can be seen below:

Exploring NLB CMDlets

PowerShell makes it quite easy to use CMDlets, even if you have no prior knowledge of the NLB CMDlets. PowerShell provides two main features that help with exploring/learning CMDlets.

Get-Command

You can use Get-Command to explore existing CMDlets that are available. This CMDlet, in conjunction with the knowledge of Verb-Noun pairing is a powerful way to getting to the CMDlet of interest.

Quick Syntax for this command

> Get-command -module NetworkLoadBalancingClusters [-Noun | -Verb <String>]

> Get-command <CommandFilter> -commandtype <commandtype>

Example usage

Let say we want to delete a node from the current cluster. We know our end goal, but don’t know how to achieve it via PowerShell. Using the above syntax we can try to reach our goal. So the action we want to perform is "delete", and the noun that we want to act on is "NLB Cluster Node".

First we try to find all commands that start with "delete" verb and are of type CMDlet, by running > Get-Command delete-* -commandType cmdlet, but do not find any results.

Instead of "delete" let’s try “Remove". Below we see that we found the CMDlet we are looking for.

We could have approached this in a different way. We could have searched for the noun "Node" and filtered further on the exact verb.

As we can see with the above examples, we can intuitively guess the Verb-Noun pair for the NLB operation we want to perform, and use the Get-Command CMDlet to get the exact CMDlet.

The list below shows the usage of Get-Command to list out all the supported NLB CMDlets:

Command Auto-Completion

Another way to find out what CMDlets exist is to use the command auto-completion key <TAB>.

PowerShell provides a feature where the arguments of a CMDlet autocomplete.

Examples

1. Open PowerShell window with the NLB modules loaded.

2. Type Add-NLBCluster<Tab>

This will automatically complete the above CMDlet, and display "Add-NLBClusterNode" on the screen.

This is another handy way to see what all CMDlets are supported. Another example would be:

Start-NLB<TAB> would display Start-NLBCluster

Hitting <TAB> again, would display Start-NLBClusterNode

Argument Auto-Completion

Now that we know how to find the CMDlet of interest, let’s see how we further use this information to formulate the exact command that we need to execute. PowerShell supports automatic expansion of the command arguments. Once you have typed in a CMDlet you can type a hyphen (-) and hit <TAB> key to automatically expand the available arguments for the given CMDlet.

Examples

1. Open a PowerShell Window with the NLB Module loaded

2. Enter > Get-NLBCluster-<TAB>

3. You will see that the “HostName” parameter will be auto-completed

4. Hit <TAB> again and you will see the text “InterfaceName” appear by the text prompt.

Using the <TAB> you can cycle through all the available arguments that the give CMDlet supports. If you went past an argument while hitting <TAB>, you can go back to it using the <SHIFT+TAB> key sequence.

This auto-completion can be further “filtered” by typing the first few characters of the argument you are interested in. For example, if I want to look for a parameter “InterfaceName”, you can try the following:

1. Open a PowerShell Window with the NLB Module loaded

2. Type “Get-NLBCluster -i“ <TAB>

3. This will directly show you all the CMDlets that begin with the letter “I”, in this case “InterfaceName”

Get-Help

As you may know from the ‘Help Documentation’ section of the earlier NLB blog post that the Get-Help CMDlet is incredibly powerful.

The final thing that I would like to bring up in this section is the use of the –example argument for the help. As the name suggests, you can quickly see the examples of a given CMDlet via the “-example” argument.

Example

Another awesome support option is the “-Online” option. This will launch the web browser with online content that is up-to-date with the latest information regarding the CMDlet (of course, this may not work if you are using a Server Core installation which cannot access Internet Explorer).

Example:

> Get-Help –Online New-NlbCluster

Rohan Mutagi & Gary Jackman
Clustering & High-Availability Test Team
Microsoft

Failover Clustering Performance Counters – Part 3 – Examples

2009-10-24T02:08:00Z

Hi cluster fans,

This third post in our series about Failover Clustering Performance Counters will give some practical examples of how to use this new Windows Server 2008 R2 feature to help troubleshoot your cluster.

In Part 1 of this blog series we discussed Performance Counters and their interaction with the Network, Multicast Request Reply, Global Update Manager and Database clustering components. In Part 2 we looked at monitoring some additional cluster components: the Checkpoint Manager, Resource Control Manager, Resource Types, APIs and Cluster Shared Volumes. Stay tuned for the fourth part of this series where we will discuss implementing Performance Counters using PowerShell.

Example 1: Cluster Handle Leaks

On my cluster I’ve observed that the Resource handles keep going up, which is an indication of a potential handle leak. I am sure that no external clients can be connected to the node, so it must be something running on the node. When I look at the ClusterAPI Performance Monitor I observe 500 resource handles:

All calls to the cluster API are coming from the ClusterAPI and in Windows Server 2008 R2 we have made tracing in this component available to all customers. To enable this tracing you need to run Event Viewer (or eventvwr from the command line). In the Event Viewer go to the View menu and check “Show Analytics and Debug Logs”.

On the tree navigate to the “Applications and Services Log\Microsoft\FailoverClustering-Client\Diagnostic”. Right click on the Diagnostic and click on Enable Log. You may see a notification reminding you when the data will be collected, which can be ignored.

Now wait for the handle to get increased again, right click on the Diagnostic and select Disable Log. Now on the right pane you can see list of the collected traces. Select an event, switch to the Details view and find information on what process this event came from.

Now since we know the Process ID we can go to the Task Manager and find this process. In this case it happens to be PowerShell.exe, which was running a script enumerating the resources from time to time.

If you are familiar with PowerShell you probably know that it is .NET based, and if you are familiar with .NET you probably know that it uses garbage collection to lazily collect freed memory. In this case, PowerShell was just taking a while to kick off garbage collection, but once garbage collection begins, all the opened handles no longer used by the process are collected and number of handles on the server decreases.

At the end it was not a handle leak, but hopefully it gives you some ideas on how you can approach this class of issues.

Thing will become harder if the client is running remotely. In this case you might first use NetMon (http://search.microsoft.com/Results.aspx?qsc0=0&q=netmon&mkt=en-US&FORM=QBME1&l=1) or Process Monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) to find where traffic to this node is coming from and then examine the clients to see if they run any apps that might cause the cluster handle leak.

Example 2: Overload of Cluster API Calls

On this particular 4 node cluster with 800 resource groups we have observed CPU utilization caused by the Cluster Service (clussvc.exe) at 95%. The puzzling part was that all the resources were offline and there was no known clients connected to this cluster, so we’re going to use performance counters to see if we can find out what component is causing the large CPU consumption.

Looking at the Cluster API calls we have observed that the cluster is getting hit with about 130 Resource API calls per second and about 70 Group API calls per second.

Looking at the Cluster Multicast Request Response (MRR) Messages we have observed that Messages Send Delta is around 90 MRR messages per second.

Examining the Cluster Global Update Manager Messages showed that there are no GUM updates going on, so most likely all the activity is coming from the API calls that are hitting resources not hosted on this node so the node forwards the request to the owning node using MRR.

Looking at the Cluster Network Messages confirms that there is lots of traffic passing between the nodes (see Bytes Received Delta and Bytes Sent Delta on the picture above).

This leaves two unanswered questions: What calls are being made, and who is the caller?

We will now run a Process Monitor. In the Process monitor we put a filter to show only events for the clussvc.exe registry and networking. In a minute we will stop collecting traces and look at the Network Summary and Registry Summary. The Network Summary shows us that there is no traffic besides the traffic between the nodes, so it has to be an application running on one of the nodes. Registry Summary demonstrates that something repeatedly opens group keys in the cluster database. So it looks like the caller is trying to enumerate all groups on the cluster.

We have started Event Viewer and collected ClusAPI logs as described in the previous example. This immediately pointed us to a process that was making most of API calls. By stopping this process we have confirmed that CPU consumption went down.

I hope you find this information helpful in troubleshooting issues using performance counters on your Windows Server 2008 R2 Failover Cluster.

Thanks,
Vladimir Petter
Senior Software Development Engineer
Clustering & High-Availability
Microsoft

Network Load Balancing in R2: Extended Affinity

2009-10-21T04:30:00Z

Hello!

I am Rohan Mutagi. My job at Microsoft is to do something that everyone likes: criticize others J, specifically, other people’s code. Yes, I am a tester and it’s my role to find bugs in Network Load Balancing (NLB). Over the next few months I will be blogging more about changes that NLB went through in Windows Server 2008 R2. In this blog, I will focus NLB Extended Affinity (TCP).

What is Extended Affinity?

To understand how NLB does load balancing, please refer to this TechNet article about various forms of affinity and their impact on load balancing decisions.

Extended Affinity is an extension to the Single and Network affinity. NLB does not rely on any network protocol’s state to make its load balancing decisions. As a result, NLB will work with a wide variety of protocols, documented and undocumented, stateless (HTTP/UDP etc.) and stateful (RDP/SSL etc). This makes NLB more flexible in deployment and easier to manage since we don’t have to configure the load balancer to work with every protocol that it needs to handle. However, some applications would benefit from being able to explicitly associate a connection to a server.

An example would be using IIS by online retailer using shopping carts. When a customer shops at their store, they save the intended purchases in a shopping cart which is stored on one of the nodes in the cluster. To keep the products in the shopping cart, the customer must stay connected to that same node. However configuration changes to the cluster (such as adding a new VIP or node) which cause cluster convergence may then directed customers to another cluster node, and they have lost the purchases saved in that shopping cart. Now the customer may become frustrated and the retailer may lose money.

Another instance could be with SSL where the SSL session can consist of multiple TCP Connections. In normal operations, if single affinity is used, NLB will guarantee that all connections coming from the same source IP will hit the same server. This would include multiple TCP connections of the same SSL session. However, configuration changes might cause NLB to accept different connections of the same SSL session by different servers during the convergence. As a result, the SSL session is broken.

With Extended Affinity, NLB provides the ability to associate a client connection to a NLB server over re-convergence. This association holds true until the timeout specified by admin for the given port rule expires without any new traffic on the same connection.

Scenario

1. We have a 2 node NLB cluster. (VIP: 2.2.2.2)

2. Web browser Client (1.1.1.1) connects via SSL to NLB VIP (2.2.2.2).

3. That particular connection is handled by IIS Server on NLB NODE1.

4. Client Requests a web page that involves filling a web form.

5. Client spends 20 minutes filling this form that would, once submitted, need to be stored on NODE1.

6. In the meantime, on the server, admin adds a new node (NODE3) to the NLB Cluster.

7. Now the connection (1.1.1.1 -> 2.2.2.2) is owned by NODE3

8. The client submits his web form.

Without Extended Affinity

9. Since the ownership of the connection (1.1.1.1 -> 2.2.2.2) has moved to NODE3, The server rejects the packet from the client.

10. The browser tries to re-establish the SSL connection and this time hits a new server

11. The new server will reject the “form data” that the browser provides since there is no authentication for this client on this node (NODE3). Thus the data that client filled-in is lost.

With Extended Affinity

9. The server notices that stickiness in enabled for that particular connection (1.1.1.1 -> 2.2.2.2) and will route the connection to the correct owner of the connection (NODE1) despite the configuration change that caused the connection ownership to move to the new node (NODE3).

10. The browser successfully communicates with the server and the transaction completes.

Applying Extended Affinity

The following sections detail how to use Extended Affinity in your Windows Server 2008 R2 NLB Cluster.

Using NLBManager

Extended Affinity can be modified by following the below steps:

1. Right click on the cluster and Select "Cluster Properties"

2. In the Cluster properties dialog box, Click on the "Port Rules" tab.

3. Choose the appropriate port rule and click Edit:

4. Select the appropriate affinity and set the "Timeout" value to required value. Click OK.

5. Now you should see the new “Timeout” to be the set amount (10 mins).

Using PowerShell

Using Powershell, you can set the timeout for the default port rule using the CMDlet Set-NLBClusterPortRule. For more information about using PowerShell with NLB, visit: http://blogs.msdn.com/clustering/archive/2008/12/26/9253786.aspx.

The below CMDlet will display all the port rules that are configured on a cluster on the current machine. The “Timeout” shows the currently configured “Extended Affinity” timeout. If this value is set to 0, that would mean that Extended Affinity is currently not enabled for the given port rule. The below example shows that the timeout for all the port rules is set to 0. This means that Extended Affinity is not enabled on any of the 3 port rules.

Now let’s enable Extended Affinity for the 2^nd port rule using PowerShell.

1. Get the required port role using the Get-NlbClusterPortRule command we used above, but this time lets add a filter to find a port rule that is configured on a port 443 and bound to cluster on network interface Test-4

2. Apply Extended Affinity to this port rule by using Set-NlbclusterPortRule to modify its timeout value.

Review:

Get-NlbClusterPortRule -Port <YourPortNumberHere> -InterfaceName <NetworkInterfaceName> | Set-NlbClusterPortRule –NewTimeout <NewTimeoutValueInMinutes>

That concludes the overview of the new Extended Affinity feature for NLB in Windows Server 2008 R2. Thanks for reading this blog post. If you have any questions feel free to contact is by clicking the ‘Email’ link on the upper right-corner of the page.

Thanks,

Rohan Mutagi
Software Development Engineer in Test
Clustering & High-Availability
Microsoft

PowerShell for Failover Clustering: Finding the Drive Letter

2009-10-16T18:41:00Z

Hi cluster scripting fans,

I often find myself looking for the drive letter(s) for a physical disk resource in a PowerShell console/script. After a bit of investigation, here is a way to do that using Cluster WMI through PowerShell:

$DiskResourceName = "Cluster Disk 2"

$DiskResource = gwmi MSCluster_Resource -Namespace root/mscluster | ?{ $_.Name -eq $DiskResourceName }

$Disk = gwmi -Namespace root/mscluster -Query "Associators of {$DiskResource} Where ResultClass=MSCluster_Disk"

$Partition = gwmi -Namespace root/mscluster -Query "Associators of {$Disk} Where ResultClass=MSCluster_DiskPartition"

After you run those CMDlets in your PowerShell window on a cluster node, the $Partition variable will contain a MSCluster_DiskPartition WMI object per volume on the physical disk resource you specified with $DiskResourceName.

Here is an example when I run this against a physical disk resource with one partition:

$Partition | select Path

Path

----

… and here with two partitions:

PS G:\Windows\system32> $Partition | select Path

Path

----

Enjoy!

Ahmed Bisht

Senior Program Manager

Clustering & High-Availability

Microsoft

Clustered Hyper-V Upgrades from Server 2008 to Server 2008 R2

2009-10-08T21:14:00Z

Hi Cluster Fans,

We’ve just published documentation describing the upgrade path for clustered VMs on 2008 Failover Clusters to 2008 R2 Failover Clusters. Hyper-V Upgrades: http://support.microsoft.com/kb/957256

This complements our other migration documentation to get you to Windows Server 2008 R2.

· General R2 Migration Guide: http://technet.microsoft.com/en-us/library/cc730990.aspx

· Migration from R2 RC to R2 RTM: http://blogs.msdn.com/clustering/archive/2009/09/02/9890272.aspx

o This has other steps and recommendations for many clustered resources.

· DHCP Migration from 2003 or 2008 to R2: http://technet.microsoft.com/en-us/library/ee460952(WS.10).aspx

Thanks,

Symon Perriman

Program Manager II

Clustering & High-Availability

Microsoft

Failover Clustering Performance Counters - Part 2

2009-10-02T07:32:00Z

Hi Cluster Fans,

In Part 1 of this blog series we discussed Performance Counters and their interaction with the Network, Multicast Request Reply, Global Update Manager and Database clustering components. This post will look at monitoring some additional cluster components: the Checkpoint Manager, Resource Control Manager, Resource Types, APIs and Cluster Shared Volumes.

Checkpoint Manager

Checkpoint Manager is a component that helps you to make sure that data of the clustered application is available on all the nodes. Failover Cluster supports two kinds of checkpoints.

Crypto Checkpoints allow you to keep your secret protected and available on all the nodes. The secret is used to create protected containers, generate keys in the containers, and encrypt data using the keys. If your application uses Crypto API and keeps secrets in the crypto container then you can associate a Crypto Checkpoint with your resource, providing it with information about the crypto provider and the container name. The cluster will export the keys from this container, export the container with data and will save all of this information to the cluster database. After offlining the resource on a node, the Checkpoint Manager will update the snapshot. Before onlining the resource on the other node, Checkpoint Manager will create/update the crypto keys and the container with the data on that node from the snapshot. Crypto Checkpoints performance counters allow you to monitor how often checkpoints save and restore operations are happening. The Delta counters tell you how many checkpoints have happen since the last sampling period, which is configurable through the properties of each performance counter. If sampling period is 1 second then it tells you how many checkpoints have happen during the last second.

The Registry Checkpoints have similar idea. Let’s say your application keeps some data in the HKEY_LOCAL_MACHINE registry hive and you do not want to or cannot move this data to the cluster database, but you still need the data to be available on all nodes in the cluster. In that case, you can associate a registry checkpoint with your resource and the cluster will make sure that before your resource is online on the other node the registry data is replicated to that node. For the registry checkpoint we have one additional feature. We monitor the registry key for changes, and if we are notified of a change we will make a new snapshot of the data from this registry key.

You can monitor the Registry checkpoints using the Registry Checkpoint Restored, Registry Checkpoint Restored Delta, Registry Checkpoint Saved and Registry Checkpoint Saved Delta performance counters. Remember that the checkpoint is saved every time registry has changed, but restoring will happen only before the resource is onlined on the other node.

Resource Control Manager

Resource Control Manager (RCM) is a component responsible for monitoring resource state and handling resource failures. This component also makes a decision about placing a resource in a separate Resource Host Monitor (RHS) if this resource is observed to be unstable and causing RHS crashes.

Groups Online – tells you how many groups are currently online on this node.

RHS Processes – tells you how many Resource Host Monitor processes are running on this node.

RHS Restarts – tells you how many Resource Host Monitor failures have happen on this node. A failure might be cause by one of the resources causing a crash or taking too long to perform an operation.

Resource Type

It would be great if we can expose information about every resource and/or group, but since we support thousands of them it is not practical to do this. However we do want to have some visibility into how resources behave. A sensible way to aggregate information about resources is to do that by resource type.

On the picture above each column represents a resource type. Some of the more interesting ones are described below:

There is one special entry _Total that is an aggregation of all resource types.

Resource Controls and Resource Controls Delta tell you how many resource controls the resources of the given type are handling on this node.

Resource Failure tells you how many times a resource of this type caused the Resource Host Monitor to get terminated due to a failure of a resource of this type.

If you see that RHS is getting restarted often, then looking at these counters can tell you what resource type is having issues.

Resource Type Controls and Resource Types Controls Delta tell you how many resource type controls the resource DLL of the given type is handling on this node.

Resources Online performance counter tells you how many resource of the given type are online on this node.

API

APIs are programmable functions in the cluster and many of the activities in the cluster are triggered by the external API calls. Out of all the clustering counters you might end up using this one the most.

The first broad category of counters is Cluster API Calls. These counters tell you what is the incoming API call rate to the cluster service. If you run cluadmin, cluster.exe or a PowerShell command, they all call into the API so you will see one of the counters going up.

All the counters in this group tell you how many calls of the given type have happened during the past sampling interval. If sampling interval is one second, then it will show the number of calls per second.

The second group of counters shows how many handles of the given type are opened. Please note that cluster handles have nothing to do with the operating system handles (kernel handles). These are just some internal structures for cluster to keep information what object they are referring to and how this object was opened. These handles still consume some amount of memory and if something is leaking them, then cluster service might eventually run out of memory. If you are writing your own resource dll then these counters are an excellent way to make sure that your resource does not leak any cluster handles. Delta versions of counters tell you the rate with which the new handles are getting opened.

Cluster Shared Volumes

Cluster Shared Volumes (CSV) is a new storage architecture in Windows Server 2008 R2 Failover Clustering which functions as a distributed-access file system optimized for Hyper-V Virtual Machines. We have added performance counters for this new technology.

This is the only counter set that is not exposed from the cluster service. It is exposed by our CSV kernel component. You can monitor each CSV volume separately with the 3 categories of performance counters.

“IO *” tells you how many read and write IO operations went directly to the disk from this node. If the disk is local then this is expected anyways, but if disk is mounted on another node then this tells you how much SMB traffic CSV has helped us to avoid.

“Redirected *” tells you how many read and write IO operations we had to send over SMB.

“Metadata *” tells you how many metadata operations was sent to the node that owns the disk. Please note that metadata is always sent to the disk owner.

Our next post in the series will walk you through some practical examples of how you can use these tools to troubleshoot your cluster.

Thanks,
Vladimir Petter
Senior Software Development Engineer
Clustering & High-Availability
Microsoft

Creating and Configuring a Generic Script Resource

2009-09-28T22:45:00Z

Hi,

In recent posts I discussed how to create and configure Generic Applications and Generic Services. In this post I will talk about creating the third of our generic containers, Generic Scripts.

What is Generic Script Resource Type?

The Generic Script resource type allows us to achieve high availability for applications that can be controlled through scriptable interfaces. If you have an application that can be started, stopped, and monitored through a script, you can run the application as a scripted resource. Note that you have to implement the Generic Script resource DLL entry points in the script file. A scripted resource is an instance of the Generic Script resource type under the control of a script. The scripts run in the Windows Scripting Host.

The Generic Script resource DLL implements a Resource Object for each instance of a generic script resource. Resource Objects have methods and properties that your script can use to accomplish several tasks such as identifying the resource, logging information about the resource, and manipulating private properties of the resource. For more information about the resource object, visit http://msdn.microsoft.com/en-us/library/aa372270(VS.85).aspx.

How to Create/Configure a Generic Script Resource

In this section, we will cover how to create and configure a generic script resource using the following methods:

1. High Availability Wizard

2. PowerShell CMDlets

3. Cluster.exe command-line tool (To be deprecated after Windows Server 2008 R2)

We will be using a VB script as an example. The example script can be found here: http://msdn.microsoft.com/en-us/library/aa372844(VS.85).aspx.

Checklist

Before creating a generic script resource, please review the following checklist:

http://technet.microsoft.com/en-us/library/cc736970.aspx

Cluster Configuration

I have a 2-node cluster running Windows Server 2008 R2. Cluster name is A1C1F4X64.

Name of my cluster nodes are:

1. A1C1F4X64N1

2. A1C1F4X64N2

Below is a partial screenshot of Failover Cluster Manager.

Create a Generic Script Resource using the High Availability Wizard

Now I will walk you through the High Availability wizard of Failover Cluster Manager to create a generic script resource. Below are the steps:

1. Launch Failover Cluster Manager from the Start Menu.

2. Connect to the cluster. Right click on Services and applications, and click on Configure a Service or Application…

3. The High Availability Wizard appears. After reading the information presented on Before You Begin page, click Next.

4. On Select Service or Application page, select Generic Script and click Next.

5. On Generic Script Info page, provide the path to the script file. Click Next.

6. On Client Access Point page, we will be providing input for the network name and IP addresses that clients will be using when accessing our highly available generic script.
I am choosing a network name (A1C1F4X64GScr) and a static IP address suitable for my network configuration.

Click Next.

7. On Select Storage page, you have an option to choose a disk for the generic script. If the generic script needs a disk resource, you can select a disk.

My example script writes data to a file, so the script needs access to a shared disk. I am choosing Cluster Disk 2 as my disk resource. After making the storage selection, click Next.

8. On the Confirmation page, verify that the information is correct. If you need to make modifications, you can use the Previous button to go back to the previous pages in the wizard and modify the information.

Click Next.

9. Wizard should successfully create the generic script resource and bring the resource online. The Summary page appears.

10. On Summary page, you can click on View Report… button. This will show you detailed report of what actions were taken to create the Generic Script resource.

The report can also be located under %SystemRoot%\Cluster\Reports directory for later viewing. Click Finish.

11. Locate the newly created item (A1C1F4X64GScr) under Services and applications. This is the container (also known as group) for the generic script resource.

The above image shows that group A1C1F4X64GScr is hosted on node A1C1F4X64N2. The name of the generic script resource is “GenericScriptTemplate Script”.

Create a Generic Script Resource using PowerShell CMDlets

I will now show you how to use PowerShell CMDlets to create a generic script resource. The CMDlets that we need to achieve this are: Get-Cluster and Add-ClusterGenericScriptRole.

If we are running PowerShell on a machine that is not part of the cluster, we have to retrieve the cluster object first using the Get-Cluster CMDlet. Once we retrieve the cluster object, the object can then be passed to Add-ClusterGenericScriptRole CMDlet. This is needed to specify the cluster where the resource will be created.

If we are running PowerShell on a clustered node, Get-Cluster CMDlet may be omitted and Add-ClusterGenericScriptRole can be used directly. In this case cluster object will be retrieved from the local node (i.e. the node where the command is executing):

PS C:\Windows\wttbin> Get-Cluster A1C1F4X64 | Add-ClusterGenericScriptRole -ScriptFilePath C:\windows\wttbin\GenericScriptTemplate.vbs -Name A1C1F4X64GSCR -StaticAddress 172.24.11.96

Report file location: C:\Users\Daud\AppData\Local\Temp\tmp2B58.tmp.mht

Name OwnerNode State

---- --------- -----

A1C1F4X64GSCR a1c1f4x64n2 Online

Create a Generic Script Resource Using Cluster.exe

Cluster.exe is another command line tool that can be used to administer a cluster. Note that this tool is in the phase of being deprecated. This tool will coexist with PowerShell in Windows Server 2008 R2. However, Cluster.exe will no longer be shipped after Windows Server 2008 R2.

Below is a sample script that creates and onlines a generic script resource.

REM Store cluster name in a variable

set ClusterName=A1C1F4X64

REM Create a group and online it

cluster %ClusterName% group GenScrGroup /create

cluster %ClusterName% group GenScrGroup /on

REM Create a IP address resource, set required properties, and online it

cluster %ClusterName% res GenScrIP /create /group:GenScrGroup /type:"IP Address"

cluster %ClusterName% res GenScrIP /priv address=172.24.11.96 SubnetMask=255.255.255.0

cluster %ClusterName% res GenScrIP /on

REM Create a network name, set properties, set dependency on the IP resource, and online it

cluster %ClusterName% res GenSvcNN /create /group:GenScrGroup /type:"Network name"

cluster %ClusterName% res GenSvcNN /priv Name=%ClusterName%-NN

cluster %ClusterName% res GenSvcNN /setdep:"[GenScrIP]"

cluster %ClusterName% res GenSvcNN /on

REM Create a generic script resource, set the properties, and online it

cluster %ClusterName% res GenScrRes /create /group:GenScrGroup /type:"Generic Script"

cluster %ClusterName% res GenScrRes /prop RestartAction="0"

cluster %ClusterName% res GenScrRes /priv ScriptFilepath="C:\windows\wttbin\GenericScriptTemplate.vbs"

cluster %ClusterName% res GenScrRes /on

The script does the following:

1. Creates and onlines a group called GenScrGroup.

2. Creates an IP address resource called GenScrIP in group GenScrGroup. Sets Address and SubnetMask properties of the resource. Brings the IP resource online.

3. Creates a network name resource called GenScrNN. Sets the Name property of the network name resource, sets the dependency of the network name to the IP resource, and brings the network name online.

4. Creates the generic application resource called GenScrRes, sets the properties, and brings the resource online.

Useful Links

· Generic Script Resource Type - http://technet.microsoft.com/en-us/library/cc737409.aspx

· Using The Generic Script Resource Type - http://msdn.microsoft.com/en-us/library/aa373089(VS.85).aspx

· Scripting Entry Points - http://msdn.microsoft.com/en-us/library/aa372846(VS.85).aspx

· Scripted Resource Example - http://msdn.microsoft.com/en-us/library/aa372844(VS.85).aspx

· Checklist: Installing a Generic Script resource - http://technet.microsoft.com/en-us/library/cc736970.aspx

· Resource Object - http://msdn.microsoft.com/en-us/library/aa372270(VS.85).aspx

· Windows Clustering - http://msdn.microsoft.com/en-us/library/aa373130(VS.85).aspx

· Generic Application blog: http://blogs.msdn.com/clustering/archive/2009/04/10/9542115.aspx

· Generic Service blog: http://blogs.msdn.com/clustering/archive/2009/06/09/9712609.aspx

Regards,
Daud Howlader
Software Development Engineer in Test
Clustering & High Availability
Microsoft

PowerShell for Failover Clustering: Let’s Rename a Few Things

2009-09-23T07:14:00Z

Hi Cluster fans,

In this blog post, I’ll walk you through how to rename a couple of objects in your failover clusters using the new PowerShell CMDlets we’ve introduced in Windows Server 2008 R2.

Trying to rename a cluster group in the GUI is easy, just right-click on the group, rename, enter the new name, and you’re done:

If you’re trying to do the same thing via PowerShell, you might guess Rename-ClusterGroup, right? Well, not really. We could’ve done it that way, but where would the fun be! J. No seriously, cluster objects have identifiers represented as GUIDs, and the name of a cluster object is nothing but an attribute:

PS G:\Windows\system32> Get-ClusterGroup

Name OwnerNode State

---- --------- -----

temp1 ahmedbc1-n1 Online

temp4 ahmedbc1-n1 Online

Cluster Group ahmedbc1-n1 Online

Available Storage ahmedbc1-n1 Offline

PS G:\Windows\system32> Get-ClusterGroup temp1 | fl Name,Id

Name : temp1

Id : cc5df8e6-0e99-4c94-8210-3e6bc9c490dd

So, instead of exposing a Rename CMDlet for each of our rename-able (is that even a word?) objects, we decided to make the Name attribute for each of those objects configurable. As a result, renaming objects in a failover cluster via PowerShell is as easy as setting the Name member on the .NET object returned by one of our Get CMDlets. For example, for a cluster resource group:

PS G:\Windows\system32> Get-ClusterGroup temp4 | %{ $_.Name = "temp2" }

Or, if you’re more comfortable with this format:

PS G:\Windows\system32> ( Get-ClusterGroup temp4 ).Name = "temp2"

Or, if you want to keep reference to the object (in case you want perform an operation on it later, for example), you can break this up into two steps:

PS G:\Windows\system32> $group = Get-ClusterGroup temp4

PS G:\Windows\system32> $group.Name = "temp2"

And, the group is renamed:

PS G:\Windows\system32> Get-ClusterGroup

Name OwnerNode State

---- --------- -----

temp1 ahmedbc1-n1 Online

temp2 ahmedbc1-n1 Online

Cluster Group ahmedbc1-n1 Online

Available Storage ahmedbc1-n1 Offline

Now, renaming a cluster resource is not any different:

PS G:\Windows\system32> Get-ClusterGroup temp2 | Get-ClusterResource

Name State Group ResourceType

---- ----- ----- ------------

IP Address 157.5... Online temp2 IP Address

IP Address 2001:... Online temp2 IPv6 Tunnel Address

IP Address 2001:... Online temp2 IPv6 Address

temp4 Online temp2 Network Name

PS G:\Windows\system32> Get-ClusterResource temp4 | %{ $_.Name = "newname" }

PS G:\Windows\system32> Get-ClusterGroup temp2 | Get-ClusterResource

Name State Group ResourceType

---- ----- ----- ------------

IP Address 157.5... Online temp2 IP Address

IP Address 2001:... Online temp2 IPv6 Tunnel Address

IP Address 2001:... Online temp2 IPv6 Address

newname Online temp2 Network Name

If you’re following so far, and if you’re double checking everything I say by looking at the Failover Cluster Manager UI, keep in mind that when the Failover Cluster Manager displays information, it doesn’t necessarily display the resource name on the list of resources, if there are other attributes in that cluster resource that are more important than the resource name. Network name resources like the one I used in my command above are a good example, as the DNS name of the network name resources are much more interesting in the GUI experience.

Just look at the properties for that resource to get the name of the resource:

One final rename we’ll do is renaming the cluster. In the GUI you can do this on the properties of the cluster:

In PowerShell, just like before with groups and resources:

PS G:\Windows\system32> Get-Cluster

Name

----

ahmedbc12

PS G:\Windows\system32> Get-Cluster | %{ $_.Name = "ahmedbc12new" }

PS G:\Windows\system32> Get-Cluster

Name

----

ahmedbc12new

And, notice how that automatically changes the cluster name resource in the cluster group:

PS G:\Windows\system32> Get-ClusterResource "Cluster Name" | Get-ClusterParamete

Object Name Value Type

------ ---- ----- ----

Cluster Name Name AHMEDBC12NEW String

Cluster Name DnsName ahmedbc12new String

...

Have fun!

Ahmed Bisht

Senior Program Manager

Clustering & High-Availability

Microsoft

PowerShell for Failover Clustering: Changing Values of a Cluster Object

2009-09-11T04:17:00Z

Hi,

This post will discuss changing values of cluster objects using PowerShell in Windows Server 2008 R2. For resources and other components in a cluster, every cluster object has properties (common property). Parameters (private properties) are optional and a cluster object may or may not have them.

Using Cluster.exe you can get these values by executing:

Cluster.exe res /priv /prop

With PowerShell CMDlets, all the common properties are attached with the cluster object that CMDlet returns:

Get-ClusterResource “Cluster Name” | format-list *

The above CMDlet would display all the properties of the ClusterResource Object returned by Get-ClusterResource CMDlet. The format-list CMDlet can be shortened to fl.

The same data can be viewed different with different fomating CMDlets provided by PowerShell. By default we have provided some formatting where the execution of CMDLet would only display certain properties that the user cares the most about.

For example, when you execute the below CMDlet, by default the Resource Name, State, Group and Resource Type, but then the resource object contains more properties (common properties).

Get-ClusterResource "Cluster Name"

Name State Group ResourceType

---- ----- ----- ------------

Cluster Name Online Cluster Group Network Name

To see a complete list of properties and its value you can use Format-List command (to get help on Format-List, use Get-Help).

Get-ClusterResource “Cluster Name” | Format-List *

Cluster : MyCluster

IsCoreResource : True

IsNetworkClassResource : False

IsStorageClassResource : False

OwnerNode : MyCluster-node2

ResourceType : Network Name

State : Online

OwnerGroup : Cluster Group

Name : Cluster Name

MaintenanceMode : False

MonitorProcessId : 908

Description :

SeparateMonitor : False

PersistentState : 1

LooksAlivePollInterval : 4294967295

IsAlivePollInterval : 4294967295

RestartAction : 2

RestartThreshold : 1

RestartDelay : 500

RestartPeriod : 900000

RetryPeriodOnFailure : 3600000

PendingTimeout : 180000

DeadlockTimeout : 300000

ResourceSpecificStatus :

Id : f6a82193-559c-4af3-9d59-27b99b59ca07

The same data can be viewed differently by using Format-Custom CMDlet (please see Get-Help Format-Custom for more information on Format-Custom). Basically all the objects of the properties are expanded. For example, Cluster Group is an object of type ClusterGroup and contains more information like the node owning the group, and the state of the group. So when a Format-Custom CMDlet is executed, it gets expanded and you can see the values of those objects too.

Get-ClusterResource “Cluster Name” | Format-Custom *

class ClusterResource

{ Cluster =

class Cluster

{ Name = MyCluster }

IsCoreResource = True

IsNetworkClassResource = False

IsStorageClassResource = False

OwnerNode =

class ClusterNode

{ Name = MyCluster-node2

State = Up }

ResourceType =

class ClusterResourceType

{ Name = Network Name

DisplayName = Network Name }

State = Online

OwnerGroup =

class ClusterGroup

{ Name = Cluster Group

OwnerNode =

class ClusterNode

{ Name = MyCluster-node2

State = Up }

State = Online }

Name = Cluster Name

MaintenanceMode = False

MonitorProcessId = 908

Description =

SeparateMonitor = False

PersistentState = 1

LooksAlivePollInterval = 4294967295

IsAlivePollInterval = 4294967295

RestartAction = 2

RestartThreshold = 1

RestartDelay = 500

RestartPeriod = 900000

RetryPeriodOnFailure = 3600000

PendingTimeout = 180000

DeadlockTimeout = 300000

ResourceSpecificStatus =

Id = f6a82193-559c-4af3-9d59-27b99b59ca07 }

Modifying Common Properties

To modify the value of any of the above property you need to cache the object, and set the value. For example, let say you want to change the name of the resource:

Get-ClusterResource “<resource name>”| % { $_.Name=”<new name>”}

Over here we are using foreach, where as we know the above CMDlet would only return one resource and piping the object to foreach object would result in processing on exactly one resource.

Another way would be:

$res = Get-ClusterResource “<resource name>”

$res.Name=”<new name>”

Another way of doing this is:

( Get-ClusterResource “<resource name>” ).Name = “new name”

Keep in mind that some of the properties of the object are read only and cannot be modified.

Obtaining the Parameters

To get parameters (Private Properites) of a resource we use Get-ClusterParameter CMDlet:

Get-ClusterResource "Cluster Name" | Get-ClusterParameter

Object Name Value Type

------ ---- ----- ----

Cluster Name Name MyCluster String

Cluster Name DnsName MyCluster String

Cluster Name RemapPipeNames 0 UInt32

Cluster Name HostRecordTTL 1200 UInt32

Cluster Name RegisterAllProvidersIP 0 UInt32

Cluster Name PublishPTRRecords 0 UInt32

Cluster Name TimerCallbackAdditionalThr... 5 UInt32

Cluster Name ResourceData {1, 0, 0, 0...} ByteArray

Cluster Name StatusNetBIOS 0 UInt32

Cluster Name StatusDNS 0 UInt32

Cluster Name StatusKerberos 0 UInt32

Cluster Name CreatingDC \\MyDc.contoso.com String

Cluster Name LastDNSUpdateTime 7/28/2009 4:38:59 PM DateTime

Cluster Name ObjectGUID 67513e33d76c6e4fab355771c4... String

The output of the above command can also be formatted in different ways by using any of the Format-* CMDlet from PowerShell.

For more information take a look at help of Get-ClusterParameter, by executing: Get-Help Get-ClusterParameter –Full.

Modifying Parameters

To modify a parameter we need to use Set-ClusterParameter.

For example if you want to change the value of DNSName for a “Cluster Name” resource:

Get-ClusterResource “Cluster Name” | Set-ClusterParameter –Name <parameter name> -Value <new Value>

You can create a new parameter or modify an existing one or delete an existing one using Set-ClusterParameter.

It is very uncommon to create a cluster object parameter (private property), but in case you have a need to create a new parameter, you need to pass the –Create switch to the CMDlet:

Get-ClusterResource “Cluster Name” | Set-ClusterParameter –Name <parameter name> -Value <new Value> -Create

To delete an existing one you need to pass –Delete switch:

Get-ClusterResource “Cluster Name” | Set-ClusterParameter –Name <parameter name> -Value <new Value> -Delete

Parameters are created dynamically and used by the specific resource(s) and the cluster service is not aware of it. These values are stored in the registry and can only support data types that are supported by the registry, so be precise when creating a parameter. The type of the parameter is determined by the type of the value passed.

Make sure that value passed to Set-ClusterParameter is verified and intended. For example, if you want to create a parameter foo with type int:

$val=[int]10

Get-ClusterParameter “resource name” | Set-ClusterParameter –Name “Foo” –Value $val –create

The trickiest resource I have seen so far is “IP Address” type resource. Changing parameters (private properties) of IP address resource is not trivial, especially in cases where you would like to change the IP address (static) or change the network or subnet mask.

All three parameters are linked to each other and every time you have to change one you need to modify all three:

$res = Get-ClusterResource <Resource Name>

$param1 = New-Object Microsoft.FailoverClusters.PowerShell.ClusterParameter –ArgumentList $res,”Network”,<Network Name, like Cluster Network 1>

$param2 = New-Object Microsoft.FailoverClusters.PowerShell.ClusterParameter –ArgumentList $res,”Address”,<Ip Address>

$param3 = New-Object Microsoft.FailoverClusters.PowerShell.ClusterParameter –ArgumentList $res,”SubNet Mask”,<subnet mask>

$params=$ param1,$ param2,$ param3

$params | Set-ClusterParameter

For more information run:

Get-Help Set-ClusterResource -Full

If you would have used either Get-ClusterParameter or Set-ClusterParameter, you would have noticed that they always require the object to be passed in as parameter. Hmm…that because these two CMDlets work with all the cluster objects, like Resources, Groups, Networks, NetworkInterfaces, ResourceType, etc. Using the name of the identity an object can be complicated, whereas passing the object through the pipeline works the best.

Thanks,

Vikas Kumar

Software Development Engineer in Test

Clustering & High-Availability

Microsoft

Failover Clustering Performance Counters - Part 1

2009-09-04T09:55:00Z

Hi Cluster Fans,

In Windows Server 2008 R2 we have added performance counters for Failover Clustering. Performance counters are like meters you have on some devices in your house. For instance, your electrical or water meter tracks what utilities you have consumed or your thermometer shows you the current temperature. They all have at least one thing in common – they show you aggregated information about the current state. However, they would not show you every event that contributes to that state. Some components in the cluster deal with lots of calls or traffic going through them and some buffer information in memory before it can get processed. We have added performance counters to several such components. This post will discuss each counter set and in future posts we will look at a couple of practical examples showing several issues resolved with the help of the performance counters.

For some information about using Performance Counters will PowerShell, check out this blog post: http://blogs.msdn.com/clustering/archive/2009/07/22/9844473.aspx

Following diagram gives you an overview of several cluster components that got instrumented and are relevant to these posts. Please note all the counters sets (except one) are in the cluster service. There are no counters in the Resource Host Monitor (RHS). One counter set that is outside the cluster service is in the Clustered Shared Volume component.

Network

The network is a subsystem that is responsible for the reliable communication between cluster nodes using TCP. All inter-node messages go over this component so you can see how much traffic cluster’s internal communication puts on your network.

On this picture you can see the set of the performance counters in the networking component. Hopefully names are self-explanatory, so let’s go over the meaning of the columns.

In this case we have a 2 node cluster (vpclus01 and vpclus22). The counters are collected on the node vpclus01. The column vpclus22 tells you how much traffic is going between vpclus01 and vpclus22. The column vpclus01 tells you how much traffic the node has sent to itself. Please note that this traffic does not go over the networking stack - this is a loopback implemented completely inside the cluster service. The _Network column tells how much traffic actually went through the networking stack. This is calculated as a sum of the how much traffic went to/from each node, and it does not count the loopback interface.

Cluster uses TCP to communicate between cluster nodes. In some cases the connection might get broken. If a TCP connection breaks, we will reestablish the connection and node will stay up. Reconnect Count tells how many times the TCP connection was broken and reestablished.

We cannot rely on TCP to tell us what messages made it across the wire so we had to implement our own acknowledgement mechanism.

The Normal Message Queue Length and the Urgent Message Queue Length tell you how many messages are sitting in the queue and waiting to be sent. Normally this number is 0, but if TCP connection breaks you might observe it is going up until TCP connection is reestablished and we can send all of them through.

The Normal Message Queue Length Delta and the Urgent Message Queue Length Delta are slightly misnamed. From the name you might expect that they would tell you how much queue length has changed, but actually they tell you incoming message rate to the queue.

Unacknowledged Message Queue and Unacknowledged Message Queue Delta tell how many messages we’ve sent, but have not yet received a returned acknowledgement. Similar to the other Delta counters in this group, Unacknowledged Message Queue Delta tells you not how much queue length has changed, but the incoming message rate.

Multicast Request Reply

Multicast Request Reply is a communication primitive that we have in the cluster. You can think of it as an RPC that allows you to send a call to multiple recipients and then get response from all of them. This component also has an internal queue. Messages Outstanding tells you the length of this queue. Usually it is close to 0. Messages Sent shows how many MRRs we have sent since the server has been running, and the Delta tells you what the change is since the last sampling interval.

There are many components in the cluster that are utilizing MRR.

One of the heavy MRR users is Cluster API. For instance when you run Get-ClusterResource using PowerShell, all the resources are displayed and the cluster has to communicate with the current resource owner to query the resource state. This communication is done using MRRs. If you have resources spread across cluster nodes and if you enumerate resources on one of cluster nodes you will observe MRR rate similar to what is displayed in the above picture.

Another MRR user is Global Update Manager. We will discuss this component next.

Global Update Manager

Global update manager is one of the components that allows a cluster to keep its state consistent across all its nodes. As you’ve probably already guessed from the name this component is responsible for making an update on all the nodes in the cluster and making sure that multiple updates are properly serialized.

Database Update Messages are making changes only to the cluster database. For instance this is happening when you change a property of a resource or a group.

Update Messages also might make changes to the database, but in some cases they just modify an unstable cluster state. You will see this counter changes when you move a group between nodes.

There are many components in the cluster that rely on the Global Update Manager.

Database

The database is the cluster component that provides persistent, consistent and replicated storage. You can find more information on how database works on MSDN and in the Global Update Manager chapter above.

You can find how many database updates are happening using the Global Update Manager counters (see above). The two counters on the picture bellow give you some visibility to how the transaction mechanism is functioning. When cluster writes data to the database, the data is first written into the transaction log, and only after that in the database. This way, if a node crashes we can replay the transaction log and make sure the database is consistent. When cluster writes data to the database it actually first just updates in memory cache and only after several seconds this cache will get flushed. By default we flush database every 20 seconds. But if there are too many database updates or if database updates are very large it might cause the transaction log to get overflowed. In this case cluster will flush the database and clean up the log before the 20 seconds have expired.

These performance counters help you to monitor how often database flushes are happening.

Stay tuned for Part 2 where we will discuss the Checkpoint Manager, Resource Control Manager, Resource Types, APIs and Cluster Shared Volumes.

Thanks,
Vladimir Petter
Senior Software Development Engineer
Clustering & High-Availability
Microsoft

Failover Clustering Upgrade Path from WS08 R2 RC to WS08 R2 RTM

2009-09-02T10:00:00Z

Hi cluster fans,

Many of you have been using the Release Candidate (RC) of Windows Server 2008 R2 and want to move to the Release to Manufacturing (RTM) build. This guide will walk you through the upgrade path which uses the “Migrate a Cluster Wizard” tool to move your resources to your RTM cluster. Details about the Migration Wizard are available here: http://go.microsoft.com/fwlink/?LinkID=142796.

Upgrade Paths for Resources

This guide applies to resources which are migrated using the “Migrate a Cluster Wizard” which include DFS-Namespace, DHCP, (MS)DTC, File Server, Generic Application, Generic Script, Generic Service, iSNS, MSMQ, NFS, Remote Desktop Connection Broker, WINS and 3^rd Party resources.

Below are notes and upgrade paths for some resources.

DFS-R

To migrate a DFS-R cluster, follow these guidelines:

1. Add the network name (VCO) of the new Windows Server 2008 R2 Failover Cluster to the replication group. Details available at: http://technet.microsoft.com/en-us/library/cc776309(WS.10).aspx

2. Wait until DFS-R finishes synchronizing the replicated data to that cluster

3. If you plan to decommission the old cluster, remove its network name from the replication group. If necessary, destroy the old cluster.

a. If you plan to use both clusters, give the File Server resource group a unique network name on the new cluster before adding it to the replication group.

For more information about DFS Replication in Windows Server 2008 R2 visit: http://technet.microsoft.com/en-us/library/cc737634(WS.10).aspx

(MS)DTC

Make sure you first drain the MSDTC transaction logs. Migrating a MSDTC resource to a new cluster does not carry the information stored in the transaction logs. This means, if you have any unresolved transactions, transaction state information about those transactions will be lost once you migrate the resource.

Follow these steps to check and resolve if you have any unresolved transactions.

1) Stop the application creating transactions on the DTC cluster resource that is being migrated, so that it’s possible to completely empty the DTC transaction log of all active, in-doubt, and failed-to-notify transactions.

2) Run dcomcnfg on one of the nodes in the Failover Cluster that hosts the DTC cluster resource.

3) Open the clustered DTC resource that is being migrated by navigating to Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Clustered DTCs.

4) Open the Transaction List for this DTC resource. If this list is empty, the transaction log is empty and this resource is ready to be migrated. If there are transactions listed, then either wait for them to be completed or manually resolve them by right clicking on them and selecting Forget, Commit, or Abort.

Additional Information:

· Transaction State Resolution Options: http://msdn.microsoft.com/en-us/library/ms681604(VS.85).aspx

· View Transaction Information: http://technet.microsoft.com/en-us/library/cc753009(WS.10).aspx

Exchange

To run Exchange on Windows Server 2008 R2, install a clean version of Exchange 2010 as there is no direct upgrade path. After Exchange 2010 has been installed, customer data can be migrated using the Move Mailbox tool to migrate mailboxes and Public Folder Replication to migrate public folders.

Hyper-V

To migration Hyper-V Virtual Machines (VMs), refer to the guidance in this KB: http://support.microsoft.com/kb/957256 which is regularly being updated.

Alternatively, you may use System Center Virtual Machine Manager’s (SCVMM) Quick Storage Migration feature: http://blogs.technet.com/virtualization/archive/2009/06/25/system-center-virtual-machine-manager-2008-r2-quick-storage-migration.aspx

MSMQ

When an MSMQ resource is included in the migration, it is important that you do NOT delete the source or the destination MSMQ resource until you are sure the migration is complete. Deleting either resource can remove the MSMQ service specific data from the shared storage, and cause the other resource to fail to come online. When you delete the source resource, or if you have to delete the newly migrated MSMQ resource or its group to perform another migration, please remove the disk resource from the group that is being deleted and take it offline first, then delete the MSMQ resource or the group. It is highly recommended that you backup the data associated with the MSMQ resource before migration. In case of the MSMQ service specific data being accidently deleted, you can restore the data from the backup. For more information about MSMQ backup and restore, please visit Backing up and restoring messages.

Print

To migrate your clustered printers, use either the UI or CLI version of PrintBRM.

· Print Migration Overview: http://technet.microsoft.com/en-us/library/cc722360.aspx

· Print Migration Guide: http://technet.microsoft.com/en-us/library/dd379488

SQL

To migrate your clustered SQL instances, use the SQL upgrade utility:

· SQL Server Supported Upgrade Paths: http://msdn.microsoft.com/en-us/library/ms143393.aspx

· SQL Server 2008 Failover Clustering Rolling Upgrade Guide: http://msdn.microsoft.com/en-us/library/ms191295.aspx

· SQL Server 2008 Failover Clustering whitepaper: http://sqlcat.com/whitepapers/archive/2009/07/08/sql-server-2008-failover-clustering.aspx

· SQL Server 2008: http://support.microsoft.com/kb/958734

· SQL Server 2008 Upgrade technical reference: http://msdn.microsoft.com/en-us/library/cc936623.aspx

· SQL Server 2005: http://msdn.microsoft.com/en-us/library/ms191295(SQL.90).aspx

3^rd Party / Other Server

For a clustered resource type which do not come inbox, the Migrate a Cluster Wizard will perform a “best effort” migration. If the resource type is not added to the new cluster is will not be migratable.

To attempt the “best effort” migration, you can first add this new resource type on the new cluster:

1) Right-click on the (new) Cluster Name

2) Select Properties

3) Click the ‘Resource Types’ tab

4) Select Add

5) Specify the Resource Dll path and file name, Resource Type Name, and Resource type display name

6) Click OK

7) Click Apply and exit the properties page

8) Run the Migrate a Cluster Wizard

Migrating from WS08 R2 RC to WS08 R2 RTM

Note : this guide assumes you are reusing the same hardware and storage. In this example I have a 2-node WS08 R2 RC Failover Cluster (Build 7100) running a highly-available DHCP Server, File Sever and Remote Desktop Connection Broker. I will be migrating to WS08 R2 RTM (Build 7600).

1. Move your running resources to a node or subset of nodes. The resources have already been moved to Node2 (SymonP-n2). Clients will still be connected to the services and applications.

2. Pause the node or subset of nodes which are not hosting the resources from the cluster, by right-clicking on the node name, then selecting Pause.

3. Evict the paused nodes by selecting More Actions…, then selecting Evict. In this example Node1 will be evicted.

4. Clean install Windows Server 2008 R2 RTM on the machines which have been evicted, an upgrade is not supported. Enable the Failover Clustering feature and any other features or roles.

5. Create a new cluster with this subset of nodes now running Windows Server 2008 R2 RTM.

6. Run the “Migrate a Cluster Wizard” and migrate your resources to the new cluster. Do not online these resources yet. For more information about cluster migration, review this guide: http://go.microsoft.com/fwlink/?LinkID=142796. The screenshot below shows my new RTM cluster which currently has 1 node.

a. Important: Read the pre- and post-migration reports to understand any additional steps or requirements to complete the migration.

7. Offline any remaining resources on the RC cluster. The services and applications will now be unavailable to clients.

8. Reattach the newly upgraded cluster nodes to the storage. The disks or CSV volumes may be mounted as a different path than what it was mounted as before the upgrade exercise. Add the storage to the Windows Server 2008 R2 RTM cluster if you are reusing storage.

9. Run “Validate a Cluster Wizard” once the storage is attached.

10. Online any other resources or applications which were migrated to the (new) RTM cluster. The services and applications will now be available to clients.

11. Delete the offline resources and destroy the Windows Server 2008 R2 RC cluster by right-clicking on the cluster name, selecting More Actions…, then selecting Destroy Cluster. Clean install Windows Server 2008 R2 RTM on these nodes, an upgrade is not supported. Enable the Failover Clustering feature and any other roles or features. Add these newly upgraded machines to the new Windows Server 2008 R2 RTM cluster, by clicking on the cluster name, then selecting Add Node…

You have now upgraded your Failover Cluster from Windows Server 2008 R2 RC to Windows Server 2008 RTM.

Thanks,

Symon Perriman
Program Manager
Clustering & High-Availability
Microsoft

Clustering and High Availability

Failover Clustering Performance Counters – Part 4 – Command Line

Microsoft Site Recovery Solutions Featuring Windows Server 2008 R2 Failover Clustering

PowerShell for NLB: Part 1: Getting Started

PowerShell Naming Convention

Exploring NLB CMDlets

Get-Command

Quick Syntax for this command

Example usage

Command Auto-Completion

Examples

Argument Auto-Completion

Examples

Get-Help

Example

Failover Clustering Performance Counters – Part 3 – Examples

Example 1: Cluster Handle Leaks

Example 2: Overload of Cluster API Calls

Network Load Balancing in R2: Extended Affinity

What is Extended Affinity?

Scenario

Without Extended Affinity

With Extended Affinity

Applying Extended Affinity

Using NLBManager

Using PowerShell

PowerShell for Failover Clustering: Finding the Drive Letter

Clustered Hyper-V Upgrades from Server 2008 to Server 2008 R2

Failover Clustering Performance Counters - Part 2

Checkpoint Manager

Resource Control Manager

Resource Type

API

Cluster Shared Volumes

Creating and Configuring a Generic Script Resource

What is Generic Script Resource Type?

How to Create/Configure a Generic Script Resource

Checklist

Cluster Configuration

Create a Generic Script Resource using the High Availability Wizard

Create a Generic Script Resource using PowerShell CMDlets

Create a Generic Script Resource Using Cluster.exe

Useful Links

PowerShell for Failover Clustering: Let’s Rename a Few Things

PowerShell for Failover Clustering: Changing Values of a Cluster Object

Modifying Common Properties

Obtaining the Parameters

Modifying Parameters

Failover Clustering Performance Counters - Part 1

Network

Multicast Request Reply

Global Update Manager

Database

Failover Clustering Upgrade Path from WS08 R2 RC to WS08 R2 RTM

Upgrade Paths for Resources

DFS-R

(MS)DTC

Exchange

Move Mailbox

Public Folder Replication

Coexistence with previous versions of Exchange

Hyper-V

MSMQ

Print

SQL

3rd Party / Other Server

Migrating from WS08 R2 RC to WS08 R2 RTM

3^rd Party / Other Server