ALL TOPICS PERFORMANCE

BizTalk team releases best practice doc for running on WS08 Hyper-V

Below is a great link on how to get BizTalk 2006 working on top of Hyper-V.  It includes practical advice that applies to other servers and roles as well.

Check it out... http://msdn.microsoft.com/en-us/library/cc768518.aspx

   Tony

How to get Processor Utilization for Hyper-V via WMI

There are a number of groups building management software (OEMs, Microsoft, …) for Hyper-V which is cool to see.  A common ask from these teams has been around reading and computing VM CPU usage.

The following is an example of how to compute Hyper-V guest processors usage.  You can use the same formula for “% Total Run  Time, “% Hypervisor Time” and “% Idle time”.   The counters show up in the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object as “PercentGuestRunTime”, “PercentTotalRunTime”, “PercentHypervisorRunTime”, and “PercentIdleTime”.

To make the formula easier to read lets use:

                G_N – Percent Guest Run Time (substitute other usage values here)

                P_N – Timestamp_PerfTime

                F_N – Frequency_PerfTime

                LP = Number of logical processors (Get this from the “Hyper-V Hypervisor” counterset)

                         F₁ * (G₂ – G₁)

 Utilization =  -----------------------------------

                     100000 * LP * (P₂-P₁)

G₂ and P₂ are the second values read and G₁ and P₁ are the first values read.

To test the formula lets read the “Hyper-V Hypervisor Logical Process” counterset twice via the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object about 10 seconds apart with a single VM running at 100% Guest CPU.  Since my test machine has two CPU’s (2 LP) this means we should see about 50% overall utilization.

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

    C1TransitionsPersec = 409197889

    C2TransitionsPersec = 0

    C3TransitionsPersec = 0

    Caption = null

    ContextSwitchesPersec = 889911109

    Description = null

    Frequency_Object = 0

    Frequency_PerfTime = 14318180

    Frequency_Sys100NS = 10000000

    HardwareInterruptsPersec = 92282462

    InterProcessorInterruptsPersec = 8174254

    InterProcessorInterruptsSentPersec = 8174254

    MonitorTransitionCost = 16

    Name = _Total

    PercentC1Time = 4193635539355

    PercentC2Time = 0

    PercentC3Time = 0

    PercentGuestRunTime = 314976793671

    PercentHypervisorRunTime = 53745475789

    PercentIdleTime = 8385447570540

    PercentTotalRunTime = 368722269460

    SchedulerInterruptsPersec = 384836664

    TimerInterruptsPersec = 33425466

    Timestamp_Object = 0

    Timestamp_PerfTime = 6268633722843

    Timestamp_Sys100NS = 4199325031975

    TotalInterruptsPersec = 518718846

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

    C1TransitionsPersec = 409201218

    C2TransitionsPersec = 0

    C3TransitionsPersec = 0

    Caption = null

    ContextSwitchesPersec = 889922035

    Description = null

    Frequency_Object = 0

    Frequency_PerfTime = 14318180

    Frequency_Sys100NS = 10000000

    HardwareInterruptsPersec = 92283571

    InterProcessorInterruptsPersec = 8174425

    InterProcessorInterruptsSentPersec = 8174425

    MonitorTransitionCost = 16

    Name = _Total

    PercentC1Time = 4193667417779

    PercentC2Time = 0

    PercentC3Time = 0

    PercentGuestRunTime = 315044817737

    PercentHypervisorRunTime = 53746578312

    PercentIdleTime = 8385511363951

    PercentTotalRunTime = 368791396049

    SchedulerInterruptsPersec = 384840537

    TimerInterruptsPersec = 33426627

    Timestamp_Object = 0

    Timestamp_PerfTime = 6268728855292

    Timestamp_Sys100NS = 4199364353043

    TotalInterruptsPersec = 518725160

Based the formula above and the data below we get = 51% which is spot on.

  Enjoy,

    Tony Voellm

WS08 Hyper-V as RTMed!!!!

Wow its been a great ride helping get Hyper-V to perform the way it does.  So much so I've not posted in a while.  You will see some upcoming posts on performance counters, WMI perf interfaces, and real data (yes now with RTM I can post data).

In the meantime check out my co-workers post on where to get the final (aka RTM) WS08 Hyper-V bits...

http://blogs.technet.com/jhoward/archive/2008/06/26/hyper-v-rtm-announcement-available-today-from-the-microsoft-download-centre.aspx 

   Tony

Windows Server Performance 2008 Tuning guide now includes Hyper-V

Why can't I start my VM when there is plenty of free memory?

Here free memory is 9MB but really much of the "Cached" memory can be made free also.  Standby Pages, Modified Pages, and System File Cache make up the "Cached" memory.  Most (all but a couple 100MB) of the cached memory can be converted to "free",

Fix #1: To resolve the issue will need to contact Microsoft Product Support http://support.microsoft.com/contactus/cu_sc_more_master#tab1 and request the following hotfix - KB953585

Alternate Fix #1: The second way to avoid this bug is to not do any work in the root parition directly.  Create a "management" VM and do your work there.

Background #1:  The most common way of hitting this issue is by running large file copies in the root (aka host) parition.  Doing lots of file copies causes the System File Cache to bloat in size.  When Hyper-V goes to start it will flush the System File Cache if it needs memory.  The System File Cache pages (aka file pages from the copies) move to the Standy list and from there will get zeroed and freed.  The bug is WS08 prevents NUMA allocations from the Standyby list.

Issue #2: The second most common cause to the can't start the VM issue when it appears there is plenty of memory is due to configuring a NUMA machine to use fewer processors than the system actually has.  Typically this is done using "bcdedit /set numproc X". 

Fix #2: If you set NUMA machine in this mode for testing memory attached to the "hidden" processors will be inaccessible to Hyper-V.  Some machines allow you to reassign the memory (via the BIOS or front end controller) to the "active" processors.

  - Tony Voellm

Hyper-V Performance FAQ

Hyper-V Performance FAQ

Anthony F Voellm (aka Tony)

6/19/2008

http://blogs.msdn.com/tvoellm

Q: What is the recommend configuration for performance testing?

A: Here are some simple steps:

1.       Be sure to have the latest WS08 Hyper-V build – Hyper-V RC1 which is on Microsoft Downloads and Windows Update

2.       Next you need to make sure you are running a “Supported OS” with the latest SP. 

3.       Make sure the guest and root OS have integration components installed (http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx **and** http://blogs.msdn.com/tvoellm/archive/2008/01/02/hyper-v-integration-components-and-enlightenments.aspx )

4.       Make sure you are using the “Network Adapter” and not the “Legacy Network Adapter”.  The legacy adapter has a lot of emulation which causes lots of CPU overhead.

5.       Use passthrough disks attached to SCSI for the next performance.  Next best is Fixed VHD attached to SCSI.  To understand storage better see (http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx )

6.       Follow these tips for avoiding pitfalls http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx

Q: How do I monitor performance?

A: First you need to understand that the clocks in the root and guest Virtual Machines may not be accurate see (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx ).  Given an understanding of clocks you can see why we implemented the “Hyper-V Hypervisor Logical Processor” performance counters (access using perfmon) which are not skewed by clock effects.  There are other Hyper-V performance counters that are useful.  See the following for more details (http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx )

Q: Should I use passthrough or iSCSI attached to the guest for storage?

A: The decision depends on what features you need to expose to the guest.  In the passthrough case the drive will show up without knowledge of the underlying LUN. 

Educated guess: If you are looking for raw performance passthough will give you the best result.

Reason: When doing IO from the guest using passthough you traverse the guest storage stack + disk stack in the root.  When doing iSCSI you traverse the storage + networking stack in the guest + root networking stack

Q: Is there a simple way to disable the hypervisor to run some baseline tests on the native system.

A: Yes.  “bcdedit /set hypervisorlaunchtype off”  and reboot the server. You should also consider changing the protocols on the root network device to re-enable IP and turn off the “Microsoft Virtual Network Switch Protocol”.   See the following for more details (XXX – coming to my blog soon J).  For starters turning off the Hypervisor should be enough for testing native performance.

To turn it back on do “bcdedit /set hypervisorlaunchtype on” and reboot the server.

Q:  Is there any common terminology used to talk about virtual machine configurations?

A: Yes - Internal to Microsoft we typically use the following;

·         Native = System without the Hyper-V role. This means you have no virtual drivers, virtual switch, …

·         Root = you have Hyper-V role installed and but are not using a virtual switch for networking.

·         Guest = Guest Virtual Machine.

·         8p.child.2x1p or better 8p.child.2VMx1VP = A system with 8 logical processors / cores running 2 Virtual Machines (VM) each with 1 Virtual Processor (VP)

Q: Are there any services that should be stopped?

A: No If you are running Server Core which is the ideal root OS to use.  Regardless of running server core vs server you should close the Hyper-V Management Console because it has a noticeable impact on CPU.  If you want the details see http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx

Q: Is it ok run applications / processes in the root OS?

A: You should avoid running any Role / Feature or custom service in the root.  If you have services you want to run put them in a guest VM.  Running roles in the root can have a negative impact the guest VM’s.  This is due to how the Hypervisor scheduler handles the root virtual processors.

Q: Are there additional knobs for performance nuts?

A: We are trying to make Hyper-V knob-less.  However we are engineers and here are some tips.

1.       Remove the CDROM drive from the guest if you don’t need it

2.       Look into the Caps / Weights / Reserves in the CPU config.  You can use these to “balance” workloads.

3.       You can use the WMI interfaces to force a VM to a particular node (coming to my blog soon J).  You don’t guarantee node affinity for VP’s but we do for memory.  There is a good chance the VP’s will stay on the node because the scheduler is NUMA aware.

Q: Are their addition resources that are useful for understanding Hyper-V?

A: Yes – here is a list

•       http://blogs.technet.com/windowsserver/default.aspx

•       http://blogs.msdn.com/virtual_pc_guy/

•       http://blogs.technet.com/jhoward/

•       http://blogs.msdn.com/tvoellm

•       http://blogs.technet.com/winserverperformance/

Negative ping times in Windows VM's - whats up?

Just a quick blog post that might help you resolve an issue that some customers have seen running under Hyper-V VM's.  The issue is negative ping times on multi-processor guests.

If you see negative ping times in multiprocessor W2k3 guest OSes you might consider setting the /usepmtimer in the boot.ini file. 

The root issue comes about from the Win32 QueryPerformanceCounter function.  By default it uses a time source called the TSC.  This is a CPU time source that essentially counts CPU cycles.  The TSC for each (virtual) processor can be different so there is no guarantee that reading TSC on one processor has anything to do with reading TSC on another processor.  This means back to back reads of TSC on different VP's can actually go backwards.  Hyper-V guarantees that TSC will not go backwards on a single VP.

So here the problem with negative ping times is the time source is using QueryPerformanceCounter which is using TSC.  By using the /usepmtimer boot.ini flag you change the time source for QueryPerformanceCounter from TSC to the PM timer which is a global time source.

  - Tony Voellm

Hyper-V Performance Counters – Part four of many – “Hyper-V Hypervisor Virtual Processor” and “Hyper-V Hypervisor Root Virtual Processor” counter set

The “Hyper-V Hypervisor Virtual Processor” and “Hyper-V Hypervisor Root Virtual Processor” counter sets have the same counters.  The only difference between the two is the ““Hyper-V Hypervisor Root Virtual Processor” contains counters for only the Root Virtual Processors (VP’s) whereas “Hyper-V Hypervisor Virtual Processor”  has counter for all other partitions.

The virtual processor counters are very useful because they help you understand how much guest VM’s are running and where they are running.  Unfortunately these counters do suffer from a small amount of clock skew in WS08 Hyper-V but this only slightly reduces their usefulness.  We hope to remove the clock skew in future releases.  The skew shows up in that some” %” counters may exceed 100%.  I’ve seen some go as much as 110% depending on the system load.  The problem has to do with the fact this counter set uses the clock from the root rather than from the hypervisor as a basis of time. For more on clock skew see (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx). 

Virtual Processors (VP) are the unit of execution for a partition and each partition contains one guest virtual machine (VM).   For each VP there is a set of counters.  Perfmon.exe will let you view the counters separately or as an average for all VP’s called “_Total”.  VP counters are prefixed with the name of the partition like this “WS08 Guest 1:” followed by the VP id like this “Hv VP 0”.  This makes it easy to identify which VP’s go with which partitions.

The VP counters have a lot of detail on what the virtual processors are doing so I have ordered them with the most useful counters at the top.

Hyper-V Hypervisor [Root] Virtual Processor counters

·         %Guest Run Time – For guest VM’s this is the percentage of time the guest VP is running in non-hypervisor code on an LP or for the _Total the total across all guest VP’s.   For the root this is the percentage of time the root VP is running in non-hypervisor code on an LP or for _Total the total across all root VP’s.  If you sum the _Total for both the guest VP’s and root VP’s this will equal the % Guest Run Time _Total of the Logical Processor counter set.

·         %Hypervisor Run Time – For guest VM’s this is the percentage of time the guest VP is running in hypervisor code on an LP or for the _Total the total across all guest VP’s.   For the root this is the percentage of time the root VP is running in hypervisor code on an LP or for _Total the total across all root VP’s.  If you sum the _Total for both the guest VP’s and root VP’s this will equal the % Hypervisor Run Time _Total of the Logical Processor counter set.

·         %Total Run Time – This is just a sum of %Guest Run Time + % Hypervisor Runtime on a per VP basis.  If you add _Total from both the Root Virtual Processor and Virtual Processor counter sets it will equal  (% Total Run Time _Total - % Idle Time _Total) from the Logical Processor counters.

·         Total Intercepts/sec – Whenever a guest VP needs to exit is current mode of running for servicing in the hypervisor this is called an intercept.  Some common causes of intercepts are resolving Guest Physical Address (GPA) to Server Physics Address (SPA) translations, privileged instructions like hlt / cupid / in / out, and the end of the VP’s scheduled time slice.

·         Total Intercepts Cost – This is a relative measure of cost of intercepts.   The cost can vary based on the types of intercepts and the machine architecture.

·         Hypercalls/sec – Hypercalls are one form of enlightenment.  Guest OS’s use the enlightenments to more efficiently use the system via the hypervisor.   TLB flush is an example hypercall.  If this value is zero and says zero this is an indication that Integration Components are not installed.  New OS’s like WS08 can use hypercalls without enlightened drivers so it is only a prereq. not a guarantee of having Integration Components installed.

·         Hypercalls Cost – This is a relative measure of cost of hypercalls.   The cost can vary based on the types of calls and the machine architecture.

·         HLT Instructions/sec – Number of CPU halts per second on the VP.  A HLT will cause the hypervisor scheduler to de-schedule the current VP and move to the next VP in the runlist.

·         HLT Instructions Cost - This is a relative measure of cost of halt.   The cost can vary based on the machine architecture.

·         IO Instructions/sec – Number of CPU in / out instructions executed per second.  Many older or low bandwidth devices use “programmed I/O” via in / out instructions.

·         IO Instructions Cost - This is a relative measure of cost of the in / out instructions.   The cost can vary based on the machine architecture.

·         Page Fault Intercepts/sec – Whenever guest code accesses a page not in the CPU TLB a page fault will occur.  This counter is the number of Page Faults per second.  This counter is closely correlated with the Large Page TLB  Fills /sec  Small Page TLB Fills / sec counters.

·         Page Fault Intercepts Cost - This is a relative measure of cost of a page fault.   The cost can vary based on the machine architecture.

·         Large Page TLB Fills/sec – There are two types of TLB entries (and some three).  Small TLB which generally means a 4K page and Large Page which generally means 2MB.  There are fewer Large TLB entries on the order of 8 – 32.  This counter is the number of Large Page TLB fills / second.  A non-zero value indicates the guest OS is using large pages.

·         Small Page TLB Fills/sec – There are two types of TLB entries (and some three).  Small TLB which generally means a 4K page and Large Page which generally means 2MB.  There are fewer Large TLB entries on the order of 64 – 1024+.  This counter is the number of Small Page TLB fills / second. 

·         Emulated Instructions/sec – Some instructions require emulation to complete in the Hypervisor.  One such example is APIC access.  This counter is the number of emulated instruction completed per second.

·         Emulated Instructions Cost - This is a relative measure of cost of emulation.   The cost can vary based on the machine architecture.

·         CPUID Instructions/sec – The CPUID instruction is used to retrieve information on the local CPU’s capabilities.  This counter is the number of CPUID instructions calls per second.  Typically CPUID is only called when the OS / Application first start so this value most likely will be 0 most of the time.

·         CPUID Instructions Cost - This is a relative measure of cost of the CPUID instruction.   The cost can vary based on the machine architecture.

·         MSR Accesses/sec – Machine specific register instruction calls per second.  There are many types of MSRs such as C-state config, Synthetic Interrupt (Synic) Timers, and control functions such as shutdown.

·         MSR Accesses Cost  - This is a relative measure of cost of the MSR instruction.   The cost can vary based on the machine architecture.

·         Control Register Accesses/sec – Number of CPU Control Register accesses per second.  Control registers are used to set up address mapping, privilege mode, etc.

·         Control Register Accesses Cost - This is a relative measure of cost of changing the control register.   The cost can vary based on the machine architecture.

·         MWAIT Instructions/sec – Number of MWAIT Instructions per second.  MWAIT is the monitored wait instruction where the CPU waits for a memory location between a and b to change.

·         MWAIT Instructions Cost - This is a relative measure of cost of the MWAIT instruction.   The cost can vary based on the machine architecture.

The following counters (and some above) likely have limited usefulness to end users of Hyper-V outside of OS / driver developers so my plan to continue to document higher value counters in other counter sets.   

Check back later as I plan to flush out these counters.

·         Page Invalidations/sec

·         Page Invalidations Cost

·         Other Intercepts/sec

·         Other Intercepts Cost

·         External Interrupts/sec

·         External Interrupts Cost

·         Pending Interrupts/sec

·         Pending Interrupts Cost

·         Debug Register Accesses/sec

·         Debug Register Accesses Cost

·         Guest Page Table Maps/sec

·         Reflected Guest Page Faults/sec

·         APIC MMIO Accesses/sec

·         IO Intercept Messages/sec

·         Memory Intercept Messages/sec

·         APIC EOI Accesses/sec

·         Other Messages/sec

·         Page Table Allocations/sec

·         Logical Processor Migrations/sec

·         Address Space Evictions/sec

·         Address Space Switches/sec

·         Address Domain Flushes/sec

·         Address Space Flushes/sec

·         Global GVA Range Flushes/sec

·         Local Flushed GVA Ranges/sec

·         Page Table Eviction/secs

·         Page Table Reclamations/sec

·         Page Table Resets/sec

·         Page Table Validations/sec

·         APIC TPR Accesses/sec

·         Page Table Write Intercepts/sec

·         Synthetic Interrupts/sec

·         Virtual Interrupts/sec

·         APIC IPIs Sent/sec

·         APIC Self IPIs Sent/sec

·         GPA Space Hypercalls/sec

·         Logical Processor Hypercalls/sec

·         Long Spin Wait Hypercalls/sec

·         Other Hypercalls/sec

·         Synthetic Interrupt Hypercalls/sec

·         Virtual Interrupt Hypercalls/sec

·         Virtual MMU Hypercalls/sec

·         Virtual Processor Hypercalls/sec

·         Total Messages/sec

Hyper-V Performance Counters – Part three of many – “Hyper-V Hypervisor Logical Processors” counter set

This is one of the most important counter set in all of Hyper-V.  It is also one of the few counter sets that is not impacted by clock skew (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx)

Before digging into the counter set it is important to understand a Logical Processor (LP) is the number is a cores / HT that the hypervisor is managing.  If you have a dual proc quad core without HT you will have 8 LP.  If you also had HT you would have 16LP. 

There are a set of counters for each LP in the system.  Perfmon.exe will let you view the counters for each LP separately or an average for all LP called “_Total”.

Hyper-V Hypervisor Logical Processor counters

·         %Guest Run Time – This is the percentage of time guest code is running on an LP or for the _Total the average percentage across all LP.  For example if you have 2LP and one VM running CPU tests you might see the value be 95% for LP(0), 0% for LP(1) and 47.5% for the _Total.  For this you can see you VM is running on LP(0).

·         %Hypervisor Run Time – This is the percentage of time the Hypervisor is running on an LP or for _Total the average percentage across all LP.  This is similar to % Kernel Run Time in the Processor counter set.

·         %Idle Run Time – This is the percentage of time the LP is waiting for work for _Total the average percentage across all LP.  This is similar to % Kernel Run Time in the Processor counter set.

·         %Total Run Time – This is just a sum of %Guest Run Time + % Hypervisor Runtime.  This counter can go over 100% just slightly (<0.5%).  The problem has to do with how performance counters are computed.  If you take the current time then value1 and later the end time and value2 this means value2 has the potential to increase between when end time was read and value2 is read.   You would change it to get the start time then value1 and later value2 and end time.  In this case the number would always be slightly less than 100.

·         %C1 Time – C1 is a power saving mode in a CPU.  This counter keeps track of how often the process is able to enter a power saving state when idle.  So %C1 Time is the percentage of time the LP is in the C1 state and for _Total  the average percentage across all LP.  If you want to know more about C state and other power modes in windows check out - Processor Power Management in Windows Vista and Windows Server 2008

·         %C2 Time – Similar to %C1 Time.  C2 is a deeper power state than C1.

·         %C3 Time – Similar to %C1 Time.  C3 is a deeper power state than C2.

·         C1 Transitions / Sec – The is the number of times the LP has entered the C1 state in one second or for _Total the number of C1 transitions across all LP.

·         C2 Transitions / Sec – Similar to C1 Transitions / Sec.  C2 is a deeper power state than C1.