Hi Clustering and Scripting fans,

 

If you’ve done any work with PowerShell v1.0 or v2.0, you have probably realized by now how rich the library of cmdlets is. There are a ton of cmdlets that allow you to automate almost any and everything across the system to use PowerShell.  Here is one more thing you can do with PowerShell and Failover Clustering.

 

As you know from previous posts, in Windows Server 2008 R2, we introduced PowerShell cmdlets for Failover Clustering:

·         http://blogs.msdn.com/clustering/archive/2008/12/20/9243367.aspx

·         http://blogs.msdn.com/clustering/archive/2009/05/21/9633316.aspx

·         http://blogs.msdn.com/clustering/archive/2009/05/23/9636665.aspx

 

Additionally, in Windows Server 2008 R2, we introduced performance counters for Failover Clustering. Through 11 performance counter sets (or objects) and 88 performance counters, you now have visibility inside the Failover Clusters you have deployed in your environment, the performance stats of different attributes of those clusters, and some trends that you can use to analyze different aspects of your clusters.

 

For the purposes of this post, I will focus on PowerShell (for information about using Performance Counters with the command line, check out this blog post: http://blogs.msdn.com/clustering/archive/2009/11/10/9919999.aspx).  But, of course, you can open “Performance Monitor” (perfmon.exe) and use this powerful GUI interface to see and use the cluster counters and the other system counters.  All the cluster counters are named “Cluster*” as you see below:

 

 1 

 

In PowerShell v2.0, many new cmdlets have also been introduced.  In the context of this article, the following new counters have been introduced to query and manipulate performance counters on a system locally or remotely:

PS G:\Windows\system32> Get-Command *counter* -CommandType cmdlet

CommandType     Name                            Definition

-----------     ----                            ----------

Cmdlet          Export-Counter                  Export-Counter [-Path] <Stri...

Cmdlet          Get-Counter                     Get-Counter [[-Counter] <Str...

Cmdlet          Import-Counter                  Import-Counter [-Path] <Stri...

 

You can get more information and examples of each of those by using the Get-Help cmdlets.  Or, just take the output of this command on the bus with you in the morning and enjoy the read:

PS G:\Windows\system32> Get-Command *counter* -CommandType cmdlet | %{ Get-Help

$_.Name -Full }

// Removed 55 pages worth of help content for those 3 cmdlets!!

 

I will focus on the Get-Counter cmdlet here, but I’m sure you can do fancy things with the Import/Export-Counter cmdlets as well.

 

To list the counter sets on your system:

PS G:\Windows\system32> Get-Counter -ListSet * | ft CounterSetName

 

And, to see the Failover Cluster ones only:

PS G:\Windows\system32> Get-Counter -ListSet * | ?{ $_.CounterSetName -like "Clu

ster*" } | ft CounterSetName

 

CounterSetName

--------------

Cluster Shared Volumes

Cluster Resource Control Manager

Cluster Global Update Manager Messages

Cluster API Calls

Cluster Checkpoint Manager

Cluster Network Messages

Cluster Network Reconnections

Cluster Database

Cluster API Handles

Cluster Multicast Request-Response Messages

Cluster Resources

 

Here are the Failover Cluster counters in all these counter sets:

PS G:\Windows\system32> Get-Counter -ListSet * | ?{ $_.CounterSetName -like "Clu

ster*" } | %{ $_.paths }

\Cluster Shared Volumes(*)\Metadata IO Delta

\Cluster Shared Volumes(*)\Metadata IO

\Cluster Shared Volumes(*)\Redirected Read Bytes Delta

\Cluster Shared Volumes(*)\Redirected Read Bytes

\Cluster Shared Volumes(*)\Redirected Reads Delta

\Cluster Shared Volumes(*)\Redirected Reads

\Cluster Shared Volumes(*)\Redirected Write Bytes Delta

\Cluster Shared Volumes(*)\Redirected Write Bytes

\Cluster Shared Volumes(*)\Redirected Writes Delta

\Cluster Shared Volumes(*)\Redirected Writes

\Cluster Shared Volumes(*)\IO Read Bytes Delta

\Cluster Shared Volumes(*)\IO Read Bytes

\Cluster Shared Volumes(*)\IO Reads Delta

\Cluster Shared Volumes(*)\IO Reads

\Cluster Shared Volumes(*)\IO Write Bytes Delta

\Cluster Shared Volumes(*)\IO Write Bytes

\Cluster Shared Volumes(*)\IO Writes Delta

\Cluster Shared Volumes(*)\IO Writes

\Cluster Resource Control Manager\Groups Online

\Cluster Resource Control Manager\RHS Restarts

\Cluster Resource Control Manager\RHS Processes

\Cluster Global Update Manager Messages\Update Messages Delta

\Cluster Global Update Manager Messages\Update Messages

\Cluster Global Update Manager Messages\Database Update Messages Delta

\Cluster Global Update Manager Messages\Database Update Messages

\Cluster API Calls\Batch API Calls Delta

\Cluster API Calls\Network Interface API Calls Delta

\Cluster API Calls\Network API Calls Delta

\Cluster API Calls\Cluster API Calls Delta

\Cluster API Calls\Key API Calls Delta

\Cluster API Calls\Resource API Calls Delta

\Cluster API Calls\Group API Calls Delta

\Cluster API Calls\Node API Calls Delta

\Cluster API Calls\Notification API Calls Delta

\Cluster Checkpoint Manager\Crypto Checkpoints Restored Delta

\Cluster Checkpoint Manager\Crypto Checkpoints Restored

\Cluster Checkpoint Manager\Crypto Checkpoints Saved Delta

\Cluster Checkpoint Manager\Crypto Checkpoints Saved

\Cluster Checkpoint Manager\Registry Checkpoints Restored Delta

\Cluster Checkpoint Manager\Registry Checkpoints Restored

\Cluster Checkpoint Manager\Registry Checkpoints Saved Delta

\Cluster Checkpoint Manager\Registry Checkpoints Saved

\Cluster Network Messages(*)\Bytes Received Delta

\Cluster Network Messages(*)\Bytes Received

\Cluster Network Messages(*)\Bytes Sent Delta

\Cluster Network Messages(*)\Bytes Sent

\Cluster Network Messages(*)\Messages Received Delta

\Cluster Network Messages(*)\Messages Received

\Cluster Network Messages(*)\Messages Sent Delta

\Cluster Network Messages(*)\Messages Sent

\Cluster Network Reconnections(*)\Reconnect Count

\Cluster Network Reconnections(*)\Unacknowledged Message Queue Length Delta

\Cluster Network Reconnections(*)\Unacknowledged Message Queue Length

\Cluster Network Reconnections(*)\Normal Message Queue Length Delta

\Cluster Network Reconnections(*)\Normal Message Queue Length

\Cluster Network Reconnections(*)\Urgent Message Queue Length Delta

\Cluster Network Reconnections(*)\Urgent Message Queue Length

\Cluster Database\Flushes Delta

\Cluster Database\Flushes

\Cluster API Handles\Batch Handles Delta

\Cluster API Handles\Batch Handles

\Cluster API Handles\Network Interface Handles Delta

\Cluster API Handles\Network Interface Handles

\Cluster API Handles\Network Handles Delta

\Cluster API Handles\Network Handles

\Cluster API Handles\Cluster Handles Delta

\Cluster API Handles\Cluster Handles

\Cluster API Handles\Key Handles Delta

\Cluster API Handles\Key Handles

\Cluster API Handles\Resource Handles Delta

\Cluster API Handles\Resource Handles

\Cluster API Handles\Group Handles Delta

\Cluster API Handles\Group Handles

\Cluster API Handles\Node Handles Delta

\Cluster API Handles\Node Handles

\Cluster API Handles\Notification Handles Delta

\Cluster API Handles\Notification Handles

\Cluster Multicast Request-Response Messages\Messages Outstanding

\Cluster Multicast Request-Response Messages\Messages Sent Delta

\Cluster Multicast Request-Response Messages\Messages Sent

\Cluster Resources(*)\Resource Type Controls Delta

\Cluster Resources(*)\Resource Type Controls

\Cluster Resources(*)\Resource Controls Delta

\Cluster Resources(*)\Resource Controls

\Cluster Resources(*)\Resource Failure Deadlock

\Cluster Resources(*)\Resource Failure Access Violation

\Cluster Resources(*)\Resource Failure

\Cluster Resources(*)\Resources Online

 

Hey!  Why count?  Let me do this for you:

PS G:\Windows\system32> ( Get-Counter -ListSet * | ?{ $_.CounterSetName

-like "Cluster*" } | %{ $_.paths } ).Count

88

 

Note that some of those counters have instances. Looking at the list of counters above, the counters that have instances are those that have “(*)” after the counter set name. For example:

\Cluster Resources(*)\Resources Online

 

This means that I can get the number of resources online on the current node for each instance (in this case, for each resource type):

PS G:\Windows\system32> Get-Counter -Counter "\Cluster Resources(*)\Resources Online"

 

Timestamp                 CounterSamples

---------                 --------------

7/2/2009 12:26:23 PM      \\ahmedbc1-n1\cluster resources(wins service)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(volume shadow copy service task)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(virtual machine configuration)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(virtual machine)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(print spooler)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(physical disk)\resources online :

                          2

 

                          \\ahmedbc1-n1\cluster resources(nfs share)\resourcesonline :

                          0

 

                          \\ahmedbc1-n1\cluster resources(network name)\resources online :

                          2

 

                          \\ahmedbc1-n1\cluster resources(message queue triggers)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(message queuing)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(isns cluster resource)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(ipv6 tunnel address)\resources online :

                          1

 

                          \\ahmedbc1-n1\cluster resources(ipv6 address)\resources online :

                          2

 

                          \\ahmedbc1-n1\cluster resources(ip address)\resources online :

                          2

 

                          \\ahmedbc1-n1\cluster resources(generic service)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(generic script)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(generic application)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(file share quorum witness)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(file server)\resources online :

                          1

 

                          \\ahmedbc1-n1\cluster resources(distributed transaction coordinator)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(distributed file system)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(dhcp service)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(dfs replicated folder)\resources online :

                          0

 

                          \\ahmedbc1-n1\cluster resources(_total)\resources online :

                          10

 

… or, I can get this info for a single instance, for example, the number of IP Address resources that are online on the current node:

PS G:\Windows\system32> Get-Counter -Counter "\Cluster Resources(IP Address)\Res

ources Online"

 

Timestamp                 CounterSamples

---------                 --------------

7/2/2009 12:35:32 PM      \\ahmedbc1-n1\cluster resources(ip address)\resources online :

                          2

 

… or, the total resources online on this node:

PS G:\Windows\system32> Get-Counter -Counter "\Cluster Resources(_Total)\Resources Online"

 

Timestamp                 CounterSamples

---------                 --------------

7/2/2009 12:36:01 PM      \\ahmedbc1-n1\cluster resources(_total)\resources online :

                          10

 

And, the fun begins when you can combine the Failover Cluster cmdlets and the other PowerShell cmdlets in the system.  For example, I can get the same info from all cluster nodes very easily, simply by combining Get-ClusterNode and Get-Counter cmdlets:

PS G:\Windows\system32> Get-ClusterNode | %{ Get-Counter -Counter "\\$($_.Name)\

Cluster Resources(_Total)\Resources Online" }

 

Timestamp                 CounterSamples

---------                 --------------

7/2/2009 12:53:46 PM      \\ahmedbc1-n1\cluster resources(_total)\resources online :

                          10

 

 

7/2/2009 12:53:47 PM      \\ahmedbc1-n2\cluster resources(_total)\resources online :

                          8

 

One cmdlet parameter that you might find useful as you use Get-Counter is the -Continuous parameter.  This gets a sample continuously, one every second, until you press ctrl-c.  If you want to change the default sample interval, use -SampleInterval parameter.  So, if I’m interested in monitoring the number of resource handles open (say, I have doubts that an application running on a cluster node is leaking handles for some reason):

PS G:\Windows\system32> Get-Counter -Counter "\Cluster API Handles\Resource Hand

les" -Continuous

 

Timestamp                 CounterSamples

---------                 --------------

7/2/2009 12:43:46 PM      \\ahmedbc1-n1\cluster api handles\resource handles :

                          54

 

 

7/2/2009 12:43:47 PM      \\ahmedbc1-n1\cluster api handles\resource handles :

                          54

 

 

7/2/2009 12:43:48 PM      \\ahmedbc1-n1\cluster api handles\resource handles :

                          60

// I pressed Ctrl-C

 

 I can also monitor other things with Failover Cluster performance counters including:

-          API calls made to the cluster by applications or a resource type.

-          CSV traffic patterns and details.

-          Number of resources that are online on a node as groups move between nodes.

-          Statistics about cluster resources per resource type.

-          Number and size of messages sent on the network between the cluster nodes.

 

Enjoy!

 

Ahmed Bisht

Senior Program Manager

Clustering & High-Availability

Microsoft