All topics fundamental : Hyper-V Performance Counters

Monitoring Hyper-V Performance

tvoellm — Thu, 23 Apr 2009 02:43:00 GMT

Now that Hyper-V has been in the market for over 9 months a common question that has come my way is “what should I monitor?” This question has been asked for a couple of reasons such as; How do I know if my machine is overloaded? How can I figure out what resources were used so I know what to bill? These questions usually amount to measuring the networking, storage, and CPU use. Another common question is where do you monitor Hyper-V? I’ll cover these questions and more.

Q: Where do I monitor Hyper-V?

Let’s start with the question of where to monitor. In order to answer this question you have understand a little of the Hyper-V architecture. Hyper-V has three main components – the virtstack, devices, and hypervisor. Windows Server 2008 is what boots the system and launches the virtstack and hypervisor. The virtstack responsible for handling emulated devices, managing VM’s, servicing I/O, and more. The hypervisor is responsible for scheduling Virtual Processors, managing interrupts, servicing timers, and controlling other chip level functions. It does not understand devices or I/O (ie there are no hypervisor drivers). The devices are part of the root and are also installed in guests as part of the Integration Services.

Since the root has a full view of the system and controls the VM’s it is also responsible for providing monitoring information via WMI and Performance Counters. You can see a full list of performance counters that are provided from the root @ http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx . See the WMI reference on MSDN @ http://msdn.microsoft.com/en-us/library/cc136992(VS.85).aspx . A common question is can you monitor a Virtual Machine from within the running guest? From within the guest there is only access to the performance counters provided by the OS that is running in the guest. All those should work as expected however keep in mind that utilization counters (like % Processor) are relative to the amount of virtual processor used. Not the physical processor. So it might appear the utilization counters are over reporting the actual physical resource usage. Rate counters (like network packets / sec) don’t have this problem. For more on guest counter skew see http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx

We could have provided some VM specific data inside the guest and might in the future. However our current thinking is not to do this. If you are running a Windows OS or have a tool that understands the WMI protocol you can use the remote management API’s to query the root (see msdn.microsoft.com for WMI documentation on how to do this). You can also do this from outside the physical server as well and tools like System Center Virtual Machine Manager.

Now that you know to monitor Hyper-V and all virtual machines from the root partition let’s look at what you should monitor.

Q: What should I monitor?

What you monitor on a regular basis really depends on what you are trying to do. I’ll cover the major resources and what to typically monitor. This will be a good starting point for you.

Here are the top level performance counters to monitor and I’ll go into more detail on each;

Overall health:

Hyper-V Virtual Machine Health Summary
Hyper-V Hypervisor

Processor:

Processor
Hyper-V Hypervisor Logical Processor
Hyper-V Hypervisor Root Virtual Processor
Hyper-V Hypervisor Virtual Processor

Memory:

Memory
Hyper-V Hypervisor Partition
Hyper-V Root Partition
Hyper-V VM Vid Partition

Networking:

Network Interface
Hyper-V Virtual Switch
Hyper-V Legacy Network Adapter
Hyper-V Virtual Network Adapter

Storage:

Physical Disk
Hyper-V Virtual Storage Device
Hyper-V Virtual IDE Controller

Now let’s go into each counter set and how to use them to monitor the system. As time allows I have been documenting each performance counter set on my blog.

You should take note of the bold italics because they represent the names of counters sets and counters.

Overall health:

There are really two counter sets I use to get an overall understanding of the system . The first is the “Hyper-V Virtual Machine Health Summary” which only has two counters; “Health Ok” and “Health Critical”. If anything is Critical it means some resource (most likely disk) has been exhausted or other unrecoverable error has occurred. If you server see “Health Critical” you should take action to figure out what has happened.

The second counter set is the “Hyper-V Hypervisor” counter set which is detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/09/hyper-v-performance-counters-part-two-of-many-hyper-v-hypervisor-counter-set.aspx . I like to use this counter set to understand how many Logical Processors the system recognizes (Logical Processor), the number of virtual machines running (“Partitions” – 1), and the total number of Virtual Processors (Virtual Processors). The logical processors (LP) are important because they are where all the work is done. They are a representation of the physical processor (core or CPU thread like HT and SMT). The virtual processors (VPs) tell you something about the guests and also if the system is overloaded. You should make sure the VP to LP ratio does not exceed eight to one because we don’t currently support beyond this limit. Hyper-V does not set a hard cap so you can exceed it. Just understand you are in largely untested configuration and might see guest failures beyond 8:1. Some other limits to be aware of is WS08 Hyper-V supports only 24 Logical Processors (LPs) and Windows Server 2008 R2 Hyper-V has a current stated limit of 32 LPs as of Windows Server 2008 R2 Beta. We will likely push this limit up.

The last counter in the “Hyper-V Hypervisor” counter set that I use is the “Total Pages”. This counter gives an indication of how much meta data memory the Hypervisor is using to manage the virtual machine. Unfortunately this counter does not capture all the overhead because another component called the Virtual Interface Driver (VID) also has overhead to manage partitions and in WS08 the “Hyper-V VID Partition” does not work . The good news is this counter set does work in Windows Server 2008 R2.

Processor:

Once you have an idea of the overall system capabilities and configuration though the “Hyper-V Hypervisor” counter set you will want to monitor the processors on the system. The most important counter set to monitor is the “Hyper-V Hypervisor Logical Processor”. This counter set allows you to determine how much of the physical processor are being used. The virtual processor counter sets only show a slice of the “Hyper-V Hypervisor Logical Processor”.

Hyper-V Hypervisor Logical Processor
Hyper-V Hypervisor Root Virtual Processor
Hyper-V Hypervisor Virtual Processor

The Hyper-V Hypervisor Logical Processor counter set is detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/09/hyper-v-performance-counters-part-three-of-many-hyper-v-logical-processors-counter-set.aspx. The most useful counters in this counter set are the following;

%Guest Run
%Hypervisor Run Time
%Idle Run Time
%Total Run Time

I generally only look at the _Totals. There is one logical processor that that carries more load than the rest and that is LP0. This LP is where all interrupts in the system are directed and if there is too much load you can see this LP hit 100% which likely means IO is a bottleneck in the system. There are some technologies in Windows Server 2008 R2 that help reduce the load for networking and those are – VMQ, Chimney and RSS. There is no RSS support in guest VM’s.

The “Hyper-V Hypervisor Root Virtual Processor” and “Hyper-V Hypervisor Virtual Processor” are just slices of the LP counter and can help you understand how much total CPU the root and guests are using on the system. These counters are detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/12/hyper-v-performance-counters-part-four-of-many-hyper-v-hypervisor-virtual-processor-and-hyper-v-hypervisor-root-virtual-processor-counter-set.aspx . There are real no limits one should expect for these counters however I generally expect to see the “% Hypervisor Time” be below 25%. Any higher and this could indicate you are not running with integration services installed. You should always make sure you have Integration Services installed for the best performance. See this link for me detail - http://blogs.msdn.com/tvoellm/archive/2008/01/02/hyper-v-integration-components-and-enlightenments.aspx .

You should also monitor the “Processor” counter set. This counter set is only for the root CPU and does suffer from skew as detailed here - http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx. Even with the skew this counter set is useful because it gives you an idea of how busy the root is. Remember the root is involved in all IO. This means that when the root CPU’s are saturated your whole system is likely saturated. In general you want to see the root CPU lower than 10% utilization and over 50% might indicate an issue.

Memory:

A common question I get is – “how much memory is a VM using?” There is no simple answer because different layers account for memory. For example the “Hyper-v [Root] Partition” counters determine how much memory the Hypervisor is managing and using on behalf of a VM which includes the guest address space but not all the memory in the worker process and VID partition. The “Hyper-V VM Vid Partition” counters account for the guest address space and any additional memory the VID needs to manage the VM. Nothing accounts for the memory in the Worker Process that is paired with the Hypervisor and VID Partition except the root processor memory counters but I know of no way to figure out what the pairing is. The root also uses memory to service IO on behalf of a VM and there is no accounting of this memory other than in the “Memory” counters which is not VM specific.

The following are the counter sets one should monitor in general;

Hyper-V Hypervisor Partition
Hyper-V Hypervisor Root Partition
Hyper-V VM Vid Partition
Memory

The Hyper-V Hypervisor [ROOT] Partition counters are interesting because they indicate in the “1G GPA Pages” and “2M GPA Pages” counters whether or not a VM is using large pages which improves overall VM performance. Large pages are only used on systems that have vTLB hardware support. See more on vTLB support here – http://blogs.msdn.com/tvoellm/archive/2009/04/06/why-does-my-desktop-box-slowdown-when-i-install-hyper-v.aspx . The partition counter set also indicates in the “Deposited Pages” counter how much memory the hypervisor is using for managing the VM. To figure out most of the memory used for the VM in Megabytes you can use the following formula; almost_total_vm_memory = “1G GPA Pages” * 1024 + (“2M GPA Pages” * 2) + ((“4K GPA Pages” + “Deposited pages”) / 256). The last counter that is interesting in the partition counter set is the “Virtual Processors” counter which lets you know how many Virtual Processors a VM is configured to use.

The Hyper-V VM Vid Partition counters have two interesting counters. The “Physical Pages Allocated” is the total number of guest pages and VID pages needed to manage the VM. The “Remote Physical Pages” let you know on NUMA based systems if a VM is spanning multiple nodes. You really want to avoid this whenever possible. You can require a VM to start of a particular node or nothing by using the API at http://blogs.msdn.com/tvoellm/archive/2008/09/28/Looking-for-that-last-once-of-performance_3F00_-Then-try-affinitizing-your-VM-to-a-NUMA-node-.aspx . Another way is to stop and restart the VM and if possible Hyper-V will allocate all memory on a single NUMA node.

The Memory counter set allows you to monitor how much memory is being consumed in the root. The root is responsible for managing all memory in Hyper-V. When a VM starts you will see the "Available Bytes" go down by at least the amount of memory given to the guest plus around another 16 - 64MB for guests meta data structures.

My recommendation is to monitor the following counters;

Available Bytes - This will give you an idea of how much memory is remaining for guests. There is a reserve of 256MBytes or 512MBytes that the root will always leave outside of guest memory. The exact amount varies but Hyper-V release. So if you find a time when a VM wont start it may be there are too few available bytes to satisfy the reserve.
Pages / Sec - This is a measure of memory pressure since it tracks hard faults. Those are page faults that require a disk access. Usually the cause for the number to spike is when there are two few available bytes on the system and processes are competing with each other for physical RAM.

Networking:

The network counters are useful for monitoring the overall networking performance on the system. The most important thing to generally monitor is the total throughput counters to make sure the NICs are not getting saturated. Once the NICs are saturated your overall system performance will be capped because no more web requests, remote storage requests, queries, etc can be received than what is currently being handled. The first counter set “Network Interface” gives the overall performance of physical device where as the other counter sets listed below represent the activity of the virtual switches and network adapters in the VM’s.

Network Interface
Hyper-V Virtual Switch
Hyper-V Legacy Network Adapter
Hyper-V Virtual Network Adapter

For the Network Interface the following are the top level counters to monitor;

Bytes Total / Sec
Offloaded Connections
Packets / Sec
Packets Outbound Errors
Packets Receive Errors

However if I’m going to monitor networking I generally look at all the counters. Mostly you want to make sure the network is not saturating and that the error counts are low. If the error counts grow rapidly you might have too much load on the system or some problem in end- to-end connectivity (including hardware).

The “Hyper-V Virtual Switch” counters are good to monitor because depending on how you have configured your network some or all of the traffic might only exist on the virtual switch. For example guest to guest packets don’t have to leave the machine to be routed and in fact on an internal network switch there is no physical adapter connected. In a future blog post I’ll detail what each counter means. The most useful ones are;

Bytes/Sec
Packets/Sec

The “Hyper-V Virtual Network Adapter” and the “Hyper-V Legacy Network Adapter” counter sets allow you to see how much ingress and egress a VM is doing. This counter sets are named with the friendly name of the VM plus the name of the network adapter followed by two GUIDs. The GUIDs are the internal id of the VM and adapter which is important when querying via WMI.

There are two counter sets because there are two types of virtual network card you can assign to a VM. If you assign a Legacy Network Adapter then the counter set you should use is the “Hyper-V Legacy Network Adapter”. In general you should not use the network adapter type because it is not enlightened, creates a lot of CPU load in the root, and is generally slower than the Network Adapter. The challenge is you need the Legacy Network Adapter to get a VM working before installing Integration Services. Once your VM is working with Integration Services you should use the Network Adapter and the “Hyper-V Virtual Network Adapter” counter set. Keep in mind Windows Server 2008 and Windows Server 2008 R2 both have integration Services pre-installed.

The “Hyper-V Legacy Network Adapter” counters to monitor are;

Bytes Dropped
Bytes Sent / Sec
Bytes Received / Sec

In the “Hyper-V Virtual Network Adapter” you should monitor;

Bytes / Sec
Packets / Sec

Storage:

The storage counters are useful for monitoring the overall disk performance on the system as well as for each VM. The first counter “Physical Disk set will give overall storage performance on the system. The next two are strictly for VM’s.

Physical Disk
Hyper-V Virtual Storage Device
Hyper-V Virtual IDE Controller

Inside the “Physical Disk” counter set I tend to monitor only three things. The first is “Current Disk Queue Length” which gives one an idea of how busy the drives are. The “Current Disk Queue Length” should be around two per drive. If you have a RAID 10 volume with 4+4 (total of 8 drives). Then a queue length of 16 is reasonable. A queue length of 32 might indicate this disk is saturated and is the bottleneck in the system. The next counter I monitor is the “Disk Bytes / Sec”. I generally expect to see about 10MB/sec per drive which is a fairly safe number for most drives. Some can do better and some worse. So for the RAID 10 4+4 a throughput of around 80MB/sec is reasonable for sequential workloads whereas 10MB/sec is not. For random workloads I look at the last counter “Disk Transfers / Sec” and expect to see about 100 IO’s per second (IOPs) per drive. Once again there are drives that do much better like 180 IOPs and some that do worse like laptop drives which are around 60 IOPS. For the RAID 10 4+4 around 800 IOPs for 8Kbytes reads and writes is reasonable. Generally writes are a bit lower.

There are two Hyper-V storage counter sets because of how storage works in Hyper-V. In Hyper-V we provide two virtual storage buses for VM’s. One is IDE and one is SCSI. The Virtual IDE counters show up in the “Hyper-V Virtual IDE Controller” counter set unless Integration Services are loaded and then you will see the activity for both virtual IDE and SCSI in the “Hyper-V Virtual Storage Device” counter set. If you don’t have integration services installed the only the “Hyper-V Virtual IDE Controller” will show the VM disk activity. If you want to read more on Hyper-V storage check out this link – http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx .

I tend to monitor the all the counters on both the “Hyper-V Virtual IDE Controller” and “Hyper-V Virtual Storage Device” because there is just the basics set.

For the “Hyper-V Virtual IDE Controller” there are;

Read Bytes / Sec
Write Bytes / Sec
Read Sectors / Sec
Write Sectors / Sec

And for “Hyper-V Virtual Storage Device” there are;

Error Count
Flush Count
Read Bytes / Sec
Write Bytes / Sec
Read Count
Write Count

The Error count should always be zero for the virtual storage device.

Conclusion:

There are many counters in Hyper-V which provide useful information and will help you understand what the system is doing. The table below summarizes the counters above in a perfmon like format. Note that you should not always collect all instances in a counter set. Some counter sets have too many counters to collect all instances. If you attempt to collect all counters for all instances you might see periods in the data that are empty. This means the system is not keeping up the amount of counters requested.

In the table below you will see (_Total) or (*). (_Total) means only the total should be collected. (*) means collect all counters. \* means collect all the counters in the counter set. \<name> means only collect that counter. This is the notation perfmon uses in its collection files.

\Hyper-V Virtual Machine Health Summary \*

\Hyper-V Hypervisor\*

\Processor(_Total)\*

\Hyper-V Hypervisor Logical Processor(*)\%Guest Run

\Hyper-V Hypervisor Logical Processor(*)\%Hypervisor Run Time

\Hyper-V Hypervisor Logical Processor(*)\%Idle Run Time

\Hyper-V Hypervisor Logical Processor(*)\%Total Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Guest Run

\Hyper-V Hypervisor Root Virtual Processor (*)\%Hypervisor Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Idle Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Total Run Time

\Hyper-V Hypervisor Virtual Processor (_Total)\*

\Memory\Pages / Sec

\Memory\Available Bytes

\Hyper-V Hypervisor Partition(*)\2G GPA Pages

\Hyper-V Hypervisor Partition(*)\Deposited Pages

\Hyper-V Hypervisor Partition(*)\Virtual Processors

\Hyper-V Hypervisor Root Partition(*)\*

\Hyper-V VM Vid Partition(*)\Physical Pages Allocated

\Hyper-V VM Vid Partition(*)\Remote Physical Pages

\Network Interface(*)\*

\Hyper-V Virtual Switch(*)\*

\Hyper-V Legacy Network Adapter(*)\*

\Hyper-V Virtual Network Adapter(*)\*

\Physical Disk(*)\Current Disk Queue Length

\Physical Disk(*)\Disk Bytes / sec

\Physical Disk(*)\Disk Transfers/sec

\Hyper-V Virtual Storage Device(*)\*

\Hyper-V Virtual IDE Controller(*)\*

Hyper-V Performance Counters - Part five of many - "Hyper-VM VM Vid Numa Node"

tvoellm — Mon, 29 Sep 2008 06:36:00 GMT

There are a couple of performance counter sets in Hyper-V that show memory usage. These counters tend to show memory used by one component of the system and the "Hyper-V VM Vid Numa Node" counter set is no different.

This counter set is intended to show memory used by the Virtual Infrastructure Driver (Vid). The good news is this counter set shows "guest physical memory". This memory includes the allocation you specify when creating the VM and some other support memory like video memory. It is the majority of the memory used by the Virtual Machine (VM).

Hyper-V VM Vid Numa Node shows memory on a per node basis. If you have an AMD machine you will likely see a Non-uniform Memory Access (NUMA) node for each processor. If you have an Intel based machine you will see anywhere from one to many nodes depending on the architecture.

The following are the performance counters in the counter set;

PageCount - Number of 4K pages that the node contains. To find the total physical memory on the system you would add PageCount from each NUMA node and multiply by 4Kbytes.
PagesInUse - Number of pages in use by the Vid on this node. Keep in mind this is most but now all memory supporting a VM.
ProcessorCount - Number of logical processors this node contains. Logical process are equal to the number of processor (sockets) * number of cores per processor * 2 if you have HyperThreading (HT) or now called Symmetric Multithreading (SMT)

Enjoy,

Anthony F. Voellm (aka Tony)

How to get Processor Utilization for Hyper-V via WMI

tvoellm — Mon, 14 Jul 2008 19:44:00 GMT

There are a number of groups building management software (OEMs, Microsoft, …) for Hyper-V which is cool to see. A common ask from these teams has been around reading and computing VM CPU usage.

The following is an example of how to compute Hyper-V guest processors usage. You can use the same formula for “% Total Run Time, “% Hypervisor Time” and “% Idle time”. The counters show up in the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object as “PercentGuestRunTime”, “PercentTotalRunTime”, “PercentHypervisorRunTime”, and “PercentIdleTime”.

To make the formula easier to read lets use:

G_N – Percent Guest Run Time (substitute other usage values here)

P_N – Timestamp_PerfTime

F_N – Frequency_PerfTime

LP = Number of logical processors (Get this from the “Hyper-V Hypervisor” counterset)

F₁ * (G₂ – G₁)

Utilization = -----------------------------------

100000 * LP * (P₂-P₁)

G₂ and P₂ are the second values read and G₁ and P₁ are the first values read.

To test the formula lets read the “Hyper-V Hypervisor Logical Process” counterset twice via the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object about 10 seconds apart with a single VM running at 100% Guest CPU. Since my test machine has two CPU’s (2 LP) this means we should see about 50% overall utilization.

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

C1TransitionsPersec = 409197889

C2TransitionsPersec = 0

C3TransitionsPersec = 0

Caption = null

ContextSwitchesPersec = 889911109

Description = null

Frequency_Object = 0

Frequency_PerfTime = 14318180

Frequency_Sys100NS = 10000000

HardwareInterruptsPersec = 92282462

InterProcessorInterruptsPersec = 8174254

InterProcessorInterruptsSentPersec = 8174254

MonitorTransitionCost = 16

Name = _Total

PercentC1Time = 4193635539355

PercentC2Time = 0

PercentC3Time = 0

PercentGuestRunTime = 314976793671

PercentHypervisorRunTime = 53745475789

PercentIdleTime = 8385447570540

PercentTotalRunTime = 368722269460

SchedulerInterruptsPersec = 384836664

TimerInterruptsPersec = 33425466

Timestamp_Object = 0

Timestamp_PerfTime = 6268633722843

Timestamp_Sys100NS = 4199325031975

TotalInterruptsPersec = 518718846

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

C1TransitionsPersec = 409201218

C2TransitionsPersec = 0

C3TransitionsPersec = 0

Caption = null

ContextSwitchesPersec = 889922035

Description = null

Frequency_Object = 0

Frequency_PerfTime = 14318180

Frequency_Sys100NS = 10000000

HardwareInterruptsPersec = 92283571

InterProcessorInterruptsPersec = 8174425

InterProcessorInterruptsSentPersec = 8174425

MonitorTransitionCost = 16

Name = _Total

PercentC1Time = 4193667417779

PercentC2Time = 0

PercentC3Time = 0

PercentGuestRunTime = 315044817737

PercentHypervisorRunTime = 53746578312

PercentIdleTime = 8385511363951

PercentTotalRunTime = 368791396049

SchedulerInterruptsPersec = 384840537

TimerInterruptsPersec = 33426627

Timestamp_Object = 0

Timestamp_PerfTime = 6268728855292

Timestamp_Sys100NS = 4199364353043

TotalInterruptsPersec = 518725160

Based the formula above and the data below we get = 51% which is spot on.

Enjoy,

Tony Voellm

Hyper-V Performance Counters – Part four of many – “Hyper-V Hypervisor Virtual Processor” and “Hyper-V Hypervisor Root Virtual Processor” counter set

tvoellm — Mon, 12 May 2008 07:31:00 GMT

The “Hyper-V Hypervisor Virtual Processor” and “Hyper-V Hypervisor Root Virtual Processor” counter sets have the same counters. The only difference between the two is the ““Hyper-V Hypervisor Root Virtual Processor” contains counters for only the Root Virtual Processors (VP’s) whereas “Hyper-V Hypervisor Virtual Processor” has counter for all other partitions.

The virtual processor counters are very useful because they help you understand how much guest VM’s are running and where they are running. Unfortunately these counters do suffer from a small amount of clock skew in WS08 Hyper-V but this only slightly reduces their usefulness. We hope to remove the clock skew in future releases. The skew shows up in that some” %” counters may exceed 100%. I’ve seen some go as much as 110% depending on the system load. The problem has to do with the fact this counter set uses the clock from the root rather than from the hypervisor as a basis of time. For more on clock skew see (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx).

Virtual Processors (VP) are the unit of execution for a partition and each partition contains one guest virtual machine (VM). For each VP there is a set of counters. Perfmon.exe will let you view the counters separately or as an average for all VP’s called “_Total”. VP counters are prefixed with the name of the partition like this “WS08 Guest 1:” followed by the VP id like this “Hv VP 0”. This makes it easy to identify which VP’s go with which partitions.

The VP counters have a lot of detail on what the virtual processors are doing so I have ordered them with the most useful counters at the top.

Hyper-V Hypervisor [Root] Virtual Processor counters

· %Guest Run Time – For guest VM’s this is the percentage of time the guest VP is running in non-hypervisor code on an LP or for the _Total the total across all guest VP’s. For the root this is the percentage of time the root VP is running in non-hypervisor code on an LP or for _Total the total across all root VP’s. If you sum the _Total for both the guest VP’s and root VP’s this will equal the % Guest Run Time _Total of the Logical Processor counter set.

· %Hypervisor Run Time – For guest VM’s this is the percentage of time the guest VP is running in hypervisor code on an LP or for the _Total the total across all guest VP’s. For the root this is the percentage of time the root VP is running in hypervisor code on an LP or for _Total the total across all root VP’s. If you sum the _Total for both the guest VP’s and root VP’s this will equal the % Hypervisor Run Time _Total of the Logical Processor counter set.

· %Total Run Time – This is just a sum of %Guest Run Time + % Hypervisor Runtime on a per VP basis. If you sum the %Total Run Time across the Root Virtual Processor and Virtual Processor counter sets it will equal the sum of %Total Run Time from all the Logical Processor counters.

· Total Intercepts/sec – Whenever a guest VP needs to exit is current mode of running for servicing in the hypervisor this is called an intercept. Some common causes of intercepts are resolving Guest Physical Address (GPA) to Server Physics Address (SPA) translations, privileged instructions like hlt / cupid / in / out, and the end of the VP’s scheduled time slice.

· Total Intercepts Cost – This is a relative measure of cost of intercepts. The cost can vary based on the types of intercepts and the machine architecture.

· Hypercalls/sec – Hypercalls are one form of enlightenment. Guest OS’s use the enlightenments to more efficiently use the system via the hypervisor. TLB flush is an example hypercall. If this value is zero and says zero this is an indication that Integration Components are not installed. New OS’s like WS08 can use hypercalls without enlightened drivers so it is only a prereq. not a guarantee of having Integration Components installed.

· Hypercalls Cost – This is a relative measure of cost of hypercalls. The cost can vary based on the types of calls and the machine architecture.

· HLT Instructions/sec – Number of CPU halts per second on the VP. A HLT will cause the hypervisor scheduler to de-schedule the current VP and move to the next VP in the runlist.

· HLT Instructions Cost - This is a relative measure of cost of halt. The cost can vary based on the machine architecture.

· IO Instructions/sec – Number of CPU in / out instructions executed per second. Many older or low bandwidth devices use “programmed I/O” via in / out instructions.

· IO Instructions Cost - This is a relative measure of cost of the in / out instructions. The cost can vary based on the machine architecture.

· Page Fault Intercepts/sec – Whenever guest code accesses a page not in the CPU TLB a page fault will occur. This counter is the number of Page Faults per second. This counter is closely correlated with the Large Page TLB Fills /sec Small Page TLB Fills / sec counters.

· Page Fault Intercepts Cost - This is a relative measure of cost of a page fault. The cost can vary based on the machine architecture.

· Large Page TLB Fills/sec – There are two types of TLB entries (and some three). Small TLB which generally means a 4K page and Large Page which generally means 2MB. There are fewer Large TLB entries on the order of 8 – 32. This counter is the number of Large Page TLB fills / second. A non-zero value indicates the guest OS is using large pages.

· Small Page TLB Fills/sec – There are two types of TLB entries (and some three). Small TLB which generally means a 4K page and Large Page which generally means 2MB. There are fewer Large TLB entries on the order of 64 – 1024+. This counter is the number of Small Page TLB fills / second.

· Emulated Instructions/sec – Some instructions require emulation to complete in the Hypervisor. One such example is APIC access. This counter is the number of emulated instruction completed per second.

· Emulated Instructions Cost - This is a relative measure of cost of emulation. The cost can vary based on the machine architecture.

· CPUID Instructions/sec – The CPUID instruction is used to retrieve information on the local CPU’s capabilities. This counter is the number of CPUID instructions calls per second. Typically CPUID is only called when the OS / Application first start so this value most likely will be 0 most of the time.

· CPUID Instructions Cost - This is a relative measure of cost of the CPUID instruction. The cost can vary based on the machine architecture.

· MSR Accesses/sec – Machine specific register instruction calls per second. There are many types of MSRs such as C-state config, Synthetic Interrupt (Synic) Timers, and control functions such as shutdown.

· MSR Accesses Cost - This is a relative measure of cost of the MSR instruction. The cost can vary based on the machine architecture.

· Control Register Accesses/sec – Number of CPU Control Register accesses per second. Control registers are used to set up address mapping, privilege mode, etc.

· Control Register Accesses Cost - This is a relative measure of cost of changing the control register. The cost can vary based on the machine architecture.

· MWAIT Instructions/sec – Number of MWAIT Instructions per second. MWAIT is the monitored wait instruction where the CPU waits for a memory location between a and b to change.

· MWAIT Instructions Cost - This is a relative measure of cost of the MWAIT instruction. The cost can vary based on the machine architecture.

The following counters (and some above) likely have limited usefulness to end users of Hyper-V outside of OS / driver developers so my plan to continue to document higher value counters in other counter sets.

Check back later as I plan to flush out these counters.

· Page Invalidations/sec

· Page Invalidations Cost

· Other Intercepts/sec

· Other Intercepts Cost

· External Interrupts/sec

· External Interrupts Cost

· Pending Interrupts/sec

· Pending Interrupts Cost

· Debug Register Accesses/sec

· Debug Register Accesses Cost

· Guest Page Table Maps/sec

· Reflected Guest Page Faults/sec

· APIC MMIO Accesses/sec

· IO Intercept Messages/sec

· Memory Intercept Messages/sec

· APIC EOI Accesses/sec

· Other Messages/sec

· Page Table Allocations/sec

· Logical Processor Migrations/sec

· Address Space Evictions/sec

· Address Space Switches/sec

· Address Domain Flushes/sec

· Address Space Flushes/sec

· Global GVA Range Flushes/sec

· Local Flushed GVA Ranges/sec

· Page Table Eviction/secs

· Page Table Reclamations/sec

· Page Table Resets/sec

· Page Table Validations/sec

· APIC TPR Accesses/sec

· Page Table Write Intercepts/sec

· Synthetic Interrupts/sec

· Virtual Interrupts/sec

· APIC IPIs Sent/sec

· APIC Self IPIs Sent/sec

· GPA Space Hypercalls/sec

· Logical Processor Hypercalls/sec

· Long Spin Wait Hypercalls/sec

· Other Hypercalls/sec

· Synthetic Interrupt Hypercalls/sec

· Virtual Interrupt Hypercalls/sec

· Virtual MMU Hypercalls/sec

· Virtual Processor Hypercalls/sec

· Total Messages/sec

Hyper-V Performance Counters – Part three of many – “Hyper-V Hypervisor Logical Processors” counter set

tvoellm — Fri, 09 May 2008 10:51:00 GMT

This is one of the most important counter set in all of Hyper-V. It is also one of the few counter sets that is not impacted by clock skew (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx)

Before digging into the counter set it is important to understand a Logical Processor (LP) is the number is a cores / HT that the hypervisor is managing. If you have a dual proc quad core without HT you will have 8 LP. If you also had HT you would have 16LP.

There are a set of counters for each LP in the system. Perfmon.exe will let you view the counters for each LP separately or an average for all LP called “_Total”.

Hyper-V Hypervisor Logical Processor counters

· %Guest Run Time – This is the percentage of time guest code is running on an LP or for the _Total the average percentage across all LP. For example if you have 2LP and one VM running CPU tests you might see the value be 95% for LP(0), 0% for LP(1) and 47.5% for the _Total. For this you can see you VM is running on LP(0).

· %Hypervisor Run Time – This is the percentage of time the Hypervisor is running on an LP or for _Total the average percentage across all LP. This is similar to % Kernel Run Time in the Processor counter set.

· %Idle Run Time – This is the percentage of time the LP is waiting for work for _Total the average percentage across all LP. This is similar to % Kernel Run Time in the Processor counter set.

· %Total Run Time – This is just a sum of %Guest Run Time + % Hypervisor Runtime. This counter can go over 100% just slightly (<0.5%). The problem has to do with how performance counters are computed. If you take the current time then value1 and later the end time and value2 this means value2 has the potential to increase between when end time was read and value2 is read. You would change it to get the start time then value1 and later value2 and end time. In this case the number would always be slightly less than 100.

· %C1 Time – C1 is a power saving mode in a CPU. This counter keeps track of how often the process is able to enter a power saving state when idle. So %C1 Time is the percentage of time the LP is in the C1 state and for _Total the average percentage across all LP. If you want to know more about C state and other power modes in windows check out - Processor Power Management in Windows Vista and Windows Server 2008

· %C2 Time – Similar to %C1 Time. C2 is a deeper power state than C1.

· %C3 Time – Similar to %C1 Time. C3 is a deeper power state than C2.

· C1 Transitions / Sec – The is the number of times the LP has entered the C1 state in one second or for _Total the number of C1 transitions across all LP.

· C2 Transitions / Sec – Similar to C1 Transitions / Sec. C2 is a deeper power state than C1.

· C3 Transitions / Sec – Similar to C1 Transitions / Sec. C3 is a deeper power state than C2.

· Hardware Interrupts / Sec – Number of hardware interrupts per second the LP is processing. _Total is the total for all LP. Hardware interrupts are delivered to the root VP’s corresponding the LP on which it was received. For example a network card will create and interrupt when a packet is received.

· Total interrupts / sec – Total number of interrupts of all kinds the LP is processing. For _Total this is the total number of interrupts happening on the system per second.

· Monitor Transition Cost – This is a measure of the cost to enter the Hypervisor via an Intercept on a Logical Processor (LP). For _Total it is the total cost across all processors. Intercepts are like User mode to Kernel Mode context switches except here is User/Kernel Mode to Virtual Machine Monitor (VMM) aka Hypervisor mode. The smaller this value the better. The only real use it has is to figure out the relative performance of processors.

· Context Switches / sec – These are the number of times a new Virtual Processor (VP) had been scheduled to a particular Logical Processor (LP). For _Total it is the total number of VP to LP switches. Ideal time context switches of around 1000 for a single guest running are not uncommon. This is due to the fact the VP will “Halt” and allow something else to run if it has no work to do.

· Scheduler Interrupts / sec – These interrupts are sent by the Hypervisor scheduler from one Logical Processor (LP) to another to reevaluate their runlist. The runlist is the list of Virtual Processors (VP) waiting to run on a given LP. This is also a “wake-up” mechanism for an LP that might be sitting idle in a lower power state. _Total is the total number of scheduler interrupts happen per second across all LPs.

· Inter-processor interrupts sent /sec – These interrupts are from one processor to another to get the processor to do memory coherency (like TLB, cache, …). High counts > 20ish per Logical Processor (LP) can indicate lots of guest pages modification (like page access). _Total is the total number of Inter-processor interrupts (IPIs) set per second.

· Inter-processor interrupts /sec – This counters is the total number of Inter-processor interrupts (IPI)received per second of a give Logical Processor (LP). _Total is the total number of IPI’s received by all LP.

· Timer interrupts / sec – There are a number of timers that the Hypervisor supports – APIC timer, PM Timer, … This is the number of times an LP is interrupted to service a timer interrupt. )

Hyper-V Performance Counters – Part two of many – “Hyper-V Hypervisor” counter set

tvoellm — Fri, 09 May 2008 09:58:00 GMT

The “Hyper-V Hypervisor” counter set is a good place to start when looking for overall system information and what is running under Hyper-V.

Hyper-V Hypervisor counter set counters:

· Logical Processors – These are the number of cores / HT that the hypervisor is managing. If you have a dual proc quad core without HT you will see this number set to 8. If you also had HT it would be set to 16. Today this value is fixed after boot and won’t change. In the future we may support hot add and remove of processors in which case this will be dynamic.

· Partitions – Each virtual machine on the system is run in a container called a partition. If you have no VM’s running this value will be set to 1 because the “host OS” called the “root” in Hyper-V is also running in a partition. So if you have 2 guest VM’s running this value will be 3. +1 for each guest and +1 for the root.

· Total Pages – The Hypervisor needs memory in order to keep track of Virtual Processors, Guest Virtual address to System Physical Address translation entries in the virtual TLB, etc. So the total pages keep track of the total amount of memory the Hypervisor is using for management or partitions. A page is 4KBytes. This is not the total amount used to support a guest. You would also need to get this by looking at the size of the worker process (vmwp.exe) and account for memory in vid. Since we don’t publish numbers on non-RTM releases I’ll have to wait to you the overhead value (future post). Total Pages can change based on what guests VM’s are running.

Here is an example of how the Hypervisor gets memory - A user want to start a VM so the vid makes a hypercall to the Hypervisor via winhv.sys to create a partition. In order to create VPs, vTLBs, … in the the Hypervisor needs memory so it makes a call to the root via winhv.sys. Winhv.sys then allocates memory from the root and makes a hypercall to deposit memory and then the whole process unwinds and the partition create completes.

You will see the term “Deposit” in other performance counters. Now you know what it means.

· Virtual Processors – All execution in the root and child partitions (where guest VM’s run) happens on Virtual Processors (aka VP’s). At a minimum you will see on VP for each Logical Processor (LP). These account for the root VP’s. You will then see +1 for each VP you have configured to a guest. So if you have an 8LP system with 1 guest running with 2VP’s the count here will be 10.

· Monitored Notifications – Monitored notifications are part of an interrupt coalescing technique Hyper-V uses to reduce virtualization overhead. For example when a guest has data to transmit over the network it could send an interrupt for each packet to the root VP that will actually do the I/O or it can send one interrupt to let the root know data is starting to flow. This counter is an indication of the number of “flows” of interrupts being set to the root and guests.

Hyper-V Performance Counters – Part one of many - The overview

tvoellm — Sun, 04 May 2008 08:05:00 GMT

[[ ***IF YOU HAVE SPECIFIC QUESTIONS ON PERFORMANCE COUNTERS YOU WANT ANSWERED PLEASE POST A REPLY WITH THE QUESTION***]]

In the next few posts I’ll be covering Hyper-V performance counters. There have been a lot of changes from Beta -> RC -> post RC.

In Beta the counters sets for Hyper-V were not grouped under a common namespace. This made them hard to find. In RC the counters are grouped under the “Hyper-V” namespace. That is each counter set starts with “Hyper-V.” For example “Hypervisor Logical Processors” are now “Hyper-V Hypervisor Logical Processors.” The prefix causes all the Hyper-V counters to be grouped together.

Post RC you will see some counter sets get renamed and a few like “Hyper-V VMMS Task Manager Summary" have been removed because they did not provide a lot of value. The most notable renaming was “Hyper-V Hypervisor …” rate counters where “/ sec” has been appended to to make it clear they are a rate. You will also see some counters have been removed.

The goal of the changes has been to simplify the counters and provide a very useful core that are easy to find. Improvements have also been made to make the counters more accurate.

The “Performance Counter” blog entries will focus on what you will see post RC but it should not be too hard to apply what is discussed to RC or Beta.

Hyper-V will RTM with 24 counter sets which are listed below by expected target audience;

Admin / Dev / Product Support (but not all counters in the counter set are for everyone)

· Hyper-V Hypervisor - Overall hypervisor & machine state

· Hyper-V Hypervisor Logical Processor - Details on what logical processors are doing

· Hyper-V Hypervisor Partition - Each guest runs in a partition. This has details on what the partition memory and processors

· Hyper-V Hypervisor Root Partition - Details on the root partition memory and processors.

· Hyper-V Hypervisor Root Virtual Processor - Details on what the root Virtual Processors are doing. There is one root VP for every Logical Processor (LP). Think of LP as cores on a processor.

· Hyper-V Hypervisor Virtual Processor - Details on guest VP’s

· Hyper-V Legacy Network Adapter - Emulated network controller statistics and rates of flows.

· Hyper-V Virtual Network Adapter - Virtualized network controller statistics. This is the faster networking device.

· Hyper-V Virtual Switch - A virtual switch can attached to one and only one physical NIC. Each Virtual / Legacy NIC plugs into a virtual switch. This counter set gives details on what the switch is doing and flows of sends / receives.

· Hyper-V Virtual Switch Port - Similar to the virtual switch stats but details just a single “port” aka attached virtual NIC.

· Hyper-V Virtual IDE Controller - Details on disk commands and rates.

· Hyper-V Virtual Storage Device - Details on virtual storage reads / writes like SCSI disks.

· Hyper-V Virtual Machine Health Summary - Toggle on Health Critical & Health OK

· Hyper-V Virtual Machine Summary - State information on the number of VM’s starting, stopping, resuming, … Gives good insight into what system wide control functions are running.

Developers / Product Support – ADVANCED

Think of VM below as Virtual Motherboard. This is not quite true but it will help you understand what much of the functionality is targeting.

· Hyper-V Task Manager Detail - Gives detail and timing on changes to virtual machines like import, export, …

· Hyper-V Virtual Machine Bus - Interrupt rates, … on VMBUS

· Hyper-V VM IO APIC - Details on the Virtual IO APIC

· Hyper-V VM Vid Driver - The VID control memory dedicated to VM’s. This give lots of details on the types of memory used, …

· Hyper-V VM Vid Message Queue - More internal details on the VID

· Hyper-V VM Vid Numa Node - Details on how the VID is managing NUMA memory

· Hyper-V VM Vid Partition - Similar to the “Hyper-V Hypervisor Partition States” but relative to the root’s view on the partitions.

Present but non-functioning counter sets – these make be removed in the future;

· Hyper-V VM Remoting - Tracks the number of pixels written to the virtual frame buffer, …

· Hyper-V VM Save, Snapshot, and Restore - Timing on how long save, restore, … take

· Hyper-V VM Worker Process Memory Management - Details on the worker processes view on memory. The worker process is where emulation for emulated devices happen, save / restore, …

The next post will cover the most useful set of counters to monitor.

[[ ***IF YOU HAVE SPECIFIC QUESTIONS ON PERFORMANCE COUNTERS YOU WANT ANSWERED PLEASE POST A REPLY WITH THE QUESTION***]]

- Tony Voellm