All topics fundamental

Note the new title....

Thanks to many of you for finding value in this blog.  I hope you have enjoyed the content and you'll see more Hyper-V in the near future.  For example I have a blog coming titled "Hyper-V Performance FAQ R2" coming soon.

I've changed the name of the blog to "All topics fundamentl" to reflect some of the expanded entries you will see around Reliability, Security and Privacy.  Together with performance these entries will help you have an even better Windows Experience.

 Enjoy.... Tony

What’s new in Windows Server 2008 R2 Hyper-V Performance and Scale?

When the first version of Hyper-V launched last year we worked really hard to make it a great performing product.  From what I’ve heard from all of you we did just that – delivered a great performing product.  Now in less than a year we have made great strides.  For example when we first shipped we only supported 16 Logical Processors and now we support 64!  We have also improved the Hypervisor, Networking, Storage, and even added Live Migration.

This blog post is designed to help you understand what has changed in terms of performance and scale of the product. 

A note on the releases

As you look at this blog entry please take special note to SP1, SP2 and R2. 

Here are the product names and the version of Hyper-V -

Windows Server 2008 RTM – This included only the Beta version of Hyper-V

Windows Server 2008 RTM + KB 950050 – This was the first RTM release of Hyper-V and shipped in June 2008.  We’ll call this WS08 + RTM Hyper-V in the tables below.

Windows Server 2008 SP2 – This is the bug fix release of Hyper-V plus improvements for performance and scale and was released in April 2009

Windows Server 2008 R2 – This is the latest release of Hyper-V and was announced in July 2009.

Comparison of Virtual Server and Windows Server 2008 SP1 Hyper-V

So lets start at the beginning of Microsoft’s virtualization products and that is Virtual Server.  Virtual Server was the first virtualization solution and worked with Windows Server 2003 [R2].  Virtual Server as a VMM solution and was replaced in Windows Server 2008 by Hyper-V a Hypervisor based solution.  You can see in terms of scale in Hyper-V we added Multi-VP support, 64 bit, and large memories.

Hypervisor improvements from Windows Server 2008 SP1 Hyper-V to Windows Server 2008 R2 Hyper-V

Once we moved in to the world of Hyper-V with Windows Server 2008 SP1 we had a mini-release with SP2 and now have R2.  In the tables below you will see how the performance and scale has evolved.

This table details the changes at the kernel and hypervisor level – Notice the changes in number of Logical Processors (LP), Virtual Processors (VP), and Virtual Machines (VM) counts along with hardware feature adoption.

Networking improvements from Windows Server 2008 SP1 Hyper-V  to Windows Server 2008 R2 Hyper-V

This table details changes in networking.  Notice the improved support for IPv6 and new hardware features like Virtual Machines Queues (VMQ) which helps with receive scaling and Chimney (TCP/IP offload into hardware) which helps with latency.

Storage improvements from Windows Server 2008 SP1 Hyper-V to Windows Server 2008 R2 Hyper-V

The following table shows changes in storage.  Some of the changes are performance related like hot add of storage.  This helps performance by reducing the time it takes to add new storage.  The big things to note are increases in IO Sizes passed from VM’s and huge improvements in Dynamic VHDs.

Storage improvements from Windows Server 2008 SP1 Hyper-V to Windows Server 2008 R2 Hyper-V

This last table shows some of the top level virtualization feature improvements effecting the whole system.

As you can see we made numerous improves in all the core areas while at the same time adding new functionality.  Hope you enjoy the new release!

  Enjoy,

  Anthony F. Voellm (aka Tony)

Useful IO profiles for simulating various workloads

A buddy of mine [Liang Yang] recently shared the following table with me and I thought you might also find it useful with trying to simulate various workloads.  In particular when testing and evaluating Hyper-V.

A tool like IOMeter [from www.iometer.org] can be programmed with the mixes of reads/wrtes and sequential/random IO's detailed below.

 Enjoy - Tony Voellm

Monitoring Hyper-V Performance

Now that Hyper-V has been in the market for over 9 months a common question that has come my way is “what should I monitor?” This question has been asked for a couple of reasons such as; How do I know if my machine is overloaded? How can I figure out what resources were used so I know what to bill? These questions usually amount to measuring the networking, storage, and CPU use. Another common question is where do you monitor Hyper-V? I’ll cover these questions and more.

Q: Where do I monitor Hyper-V?

Let’s start with the question of where to monitor. In order to answer this question you have understand a little of the Hyper-V architecture. Hyper-V has three main components – the virtstack, devices, and hypervisor. Windows Server 2008 is what boots the system and launches the virtstack and hypervisor. The virtstack responsible for handling emulated devices, managing VM’s, servicing I/O, and more. The hypervisor is responsible for scheduling Virtual Processors, managing interrupts, servicing timers, and controlling other chip level functions. It does not understand devices or I/O (ie there are no hypervisor drivers). The devices are part of the root and are also installed in guests as part of the Integration Services.

Since the root has a full view of the system and controls the VM’s it is also responsible for providing monitoring information via WMI and Performance Counters. You can see a full list of performance counters that are provided from the root @ http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx . See the WMI reference on MSDN @ http://msdn.microsoft.com/en-us/library/cc136992(VS.85).aspx . A common question is can you monitor a Virtual Machine from within the running guest? From within the guest there is only access to the performance counters provided by the OS that is running in the guest. All those should work as expected however keep in mind that utilization counters (like % Processor) are relative to the amount of virtual processor used. Not the physical processor. So it might appear the utilization counters are over reporting the actual physical resource usage. Rate counters (like network packets / sec) don’t have this problem. For more on guest counter skew see http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx

We could have provided some VM specific data inside the guest and might in the future. However our current thinking is not to do this. If you are running a Windows OS or have a tool that understands the WMI protocol you can use the remote management API’s to query the root (see msdn.microsoft.com for WMI documentation on how to do this). You can also do this from outside the physical server as well and tools like System Center Virtual Machine Manager.

Now that you know to monitor Hyper-V and all virtual machines from the root partition let’s look at what you should monitor.

Q: What should I monitor?

What you monitor on a regular basis really depends on what you are trying to do. I’ll cover the major resources and what to typically monitor. This will be a good starting point for you.

Here are the top level performance counters to monitor and I’ll go into more detail on each;

• Overall health:
• Hyper-V Virtual Machine Health Summary
• Hyper-V Hypervisor
• Processor:
• Processor
• Hyper-V Hypervisor Logical Processor
• Hyper-V Hypervisor Root Virtual Processor
• Hyper-V Hypervisor Virtual Processor
• Memory:
• Memory
• Hyper-V Hypervisor Partition
• Hyper-V Root Partition
• Hyper-V VM Vid Partition
• Networking:
• Network Interface
• Hyper-V Virtual Switch
• Hyper-V Legacy Network Adapter
• Hyper-V Virtual Network Adapter
• Storage:
• Physical Disk
• Hyper-V Virtual Storage Device
• Hyper-V Virtual IDE Controller

Now let’s go into each counter set and how to use them to monitor the system. As time allows I have been documenting each performance counter set on my blog.

You should take note of the bold italics because they represent the names of counters sets and counters.

Overall health:

There are really two counter sets I use to get an overall understanding of the system . The first is the “Hyper-V Virtual Machine Health Summary” which only has two counters; “Health Ok” and “Health Critical”. If anything is Critical it means some resource (most likely disk) has been exhausted or other unrecoverable error has occurred. If you server see “Health Critical” you should take action to figure out what has happened.

The second counter set is the “Hyper-V Hypervisor” counter set which is detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/09/hyper-v-performance-counters-part-two-of-many-hyper-v-hypervisor-counter-set.aspx . I like to use this counter set to understand how many Logical Processors the system recognizes (Logical Processor), the number of virtual machines running (“Partitions” – 1), and the total number of Virtual Processors (Virtual Processors). The logical processors (LP) are important because they are where all the work is done. They are a representation of the physical processor (core or CPU thread like HT and SMT). The virtual processors (VPs) tell you something about the guests and also if the system is overloaded. You should make sure the VP to LP ratio does not exceed eight to one because we don’t currently support beyond this limit. Hyper-V does not set a hard cap so you can exceed it. Just understand you are in largely untested configuration and might see guest failures beyond 8:1. Some other limits to be aware of is WS08 Hyper-V supports only 24 Logical Processors (LPs) and Windows Server 2008 R2 Hyper-V has a current stated limit of 32 LPs as of Windows Server 2008 R2 Beta. We will likely push this limit up.

The last counter in the “Hyper-V Hypervisor” counter set that I use is the “Total Pages”. This counter gives an indication of how much meta data memory the Hypervisor is using to manage the virtual machine. Unfortunately this counter does not capture all the overhead because another component called the Virtual Interface Driver (VID) also has overhead to manage partitions and in WS08 the “Hyper-V VID Partition” does not work . The good news is this counter set does work in Windows Server 2008 R2.

Processor:

Once you have an idea of the overall system capabilities and configuration though the “Hyper-V Hypervisor” counter set you will want to monitor the processors on the system. The most important counter set to monitor is the “Hyper-V Hypervisor Logical Processor”. This counter set allows you to determine how much of the physical processor are being used. The virtual processor counter sets only show a slice of the “Hyper-V Hypervisor Logical Processor”.

• Hyper-V Hypervisor Logical Processor
• Hyper-V Hypervisor Root Virtual Processor
• Hyper-V Hypervisor Virtual Processor

The Hyper-V Hypervisor Logical Processor counter set is detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/09/hyper-v-performance-counters-part-three-of-many-hyper-v-logical-processors-counter-set.aspx. The most useful counters in this counter set are the following;

• %Guest Run
• %Hypervisor Run Time
• %Idle Run Time
• %Total Run Time

I generally only look at the _Totals. There is one logical processor that that carries more load than the rest and that is LP0. This LP is where all interrupts in the system are directed and if there is too much load you can see this LP hit 100% which likely means IO is a bottleneck in the system. There are some technologies in Windows Server 2008 R2 that help reduce the load for networking and those are – VMQ, Chimney and RSS. There is no RSS support in guest VM’s.

The “Hyper-V Hypervisor Root Virtual Processor” and “Hyper-V Hypervisor Virtual Processor” are just slices of the LP counter and can help you understand how much total CPU the root and guests are using on the system. These counters are detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/12/hyper-v-performance-counters-part-four-of-many-hyper-v-hypervisor-virtual-processor-and-hyper-v-hypervisor-root-virtual-processor-counter-set.aspx . There are real no limits one should expect for these counters however I generally expect to see the “% Hypervisor Time” be below 25%. Any higher and this could indicate you are not running with integration services installed. You should always make sure you have Integration Services installed for the best performance. See this link for me detail - http://blogs.msdn.com/tvoellm/archive/2008/01/02/hyper-v-integration-components-and-enlightenments.aspx .

You should also monitor the “Processor” counter set. This counter set is only for the root CPU and does suffer from skew as detailed here - http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx. Even with the skew this counter set is useful because it gives you an idea of how busy the root is. Remember the root is involved in all IO. This means that when the root CPU’s are saturated your whole system is likely saturated. In general you want to see the root CPU lower than 10% utilization and over 50% might indicate an issue.

Memory:

A common question I get is – “how much memory is a VM using?” There is no simple answer because different layers account for memory. For example the “Hyper-v [Root] Partition” counters determine how much memory the Hypervisor is managing and using on behalf of a VM which includes the guest address space but not all the memory in the worker process and VID partition. The “Hyper-V VM Vid Partition” counters account for the guest address space and any additional memory the VID needs to manage the VM. Nothing accounts for the memory in the Worker Process that is paired with the Hypervisor and VID Partition except the root processor memory counters but I know of no way to figure out what the pairing is. The root also uses memory to service IO on behalf of a VM and there is no accounting of this memory other than in the “Memory” counters which is not VM specific.

The following are the counter sets one should monitor in general;

• Hyper-V Hypervisor Partition
• Hyper-V Hypervisor Root Partition
• Hyper-V VM Vid Partition
• Memory

The Hyper-V Hypervisor [ROOT] Partition counters are interesting because they indicate in the “1G GPA Pages” and “2M GPA Pages” counters whether or not a VM is using large pages which improves overall VM performance. Large pages are only used on systems that have vTLB hardware support. See more on vTLB support here – http://blogs.msdn.com/tvoellm/archive/2009/04/06/why-does-my-desktop-box-slowdown-when-i-install-hyper-v.aspx . The partition counter set also indicates in the “Deposited Pages” counter how much memory the hypervisor is using for managing the VM. To figure out most of the memory used for the VM in Megabytes you can use the following formula; almost_total_vm_memory = “1G GPA Pages” * 1024 + (“2M GPA Pages” * 2) + ((“4K GPA Pages” + “Deposited pages”) / 256). The last counter that is interesting in the partition counter set is the “Virtual Processors” counter which lets you know how many Virtual Processors a VM is configured to use.

The Hyper-V VM Vid Partition counters have two interesting counters. The “Physical Pages Allocated” is the total number of guest pages and VID pages needed to manage the VM. The “Remote Physical Pages” let you know on NUMA based systems if a VM is spanning multiple nodes. You really want to avoid this whenever possible. You can require a VM to start of a particular node or nothing by using the API at http://blogs.msdn.com/tvoellm/archive/2008/09/28/Looking-for-that-last-once-of-performance_3F00_-Then-try-affinitizing-your-VM-to-a-NUMA-node-.aspx . Another way is to stop and restart the VM and if possible Hyper-V will allocate all memory on a single NUMA node.

The Memory counter set allows you to monitor how much memory is being consumed in the root. The root is responsible for managing all memory in Hyper-V.  When a VM starts you will see the "Available Bytes" go down by at least the amount of memory given to the guest plus around another 16 - 64MB for guests meta data structures.

 My recommendation is to monitor the following counters;

• Available Bytes - This will give you an idea of how much memory is remaining for guests.  There is a reserve of 256MBytes or 512MBytes that the root will always leave outside of guest memory.  The exact amount varies but Hyper-V release. So if you find a time when a VM wont start it may be there are too few available bytes to satisfy the reserve.
• Pages / Sec - This is a measure of memory pressure since it tracks hard faults.  Those are page faults that require a disk access.  Usually the cause for the number to spike is when there are two few available bytes on the system and processes are competing with each other for physical RAM.

Networking:

The network counters are useful for monitoring the overall networking performance on the system. The most important thing to generally monitor is the total throughput counters to make sure the NICs are not getting saturated. Once the NICs are saturated your overall system performance will be capped because no more web requests, remote storage requests, queries, etc can be received than what is currently being handled. The first counter set “Network Interface” gives the overall performance of physical device where as the other counter sets listed below represent the activity of the virtual switches and network adapters in the VM’s.

• Network Interface
• Hyper-V Virtual Switch
• Hyper-V Legacy Network Adapter
• Hyper-V Virtual Network Adapter

For the Network Interface the following are the top level counters to monitor;

• Bytes Total / Sec
• Offloaded Connections
• Packets / Sec
• Packets Outbound Errors
• Packets Receive Errors

However if I’m going to monitor networking I generally look at all the counters. Mostly you want to make sure the network is not saturating and that the error counts are low. If the error counts grow rapidly you might have too much load on the system or some problem in end- to-end connectivity (including hardware).

The “Hyper-V Virtual Switch” counters are good to monitor because depending on how you have configured your network some or all of the traffic might only exist on the virtual switch. For example guest to guest packets don’t have to leave the machine to be routed and in fact on an internal network switch there is no physical adapter connected. In a future blog post I’ll detail what each counter means. The most useful ones are;

• Bytes/Sec
• Packets/Sec

The “Hyper-V Virtual Network Adapter” and the “Hyper-V Legacy Network Adapter” counter sets allow you to see how much ingress and egress a VM is doing. This counter sets are named with the friendly name of the VM plus the name of the network adapter followed by two GUIDs. The GUIDs are the internal id of the VM and adapter which is important when querying via WMI.

There are two counter sets because there are two types of virtual network card you can assign to a VM. If you assign a Legacy Network Adapter then the counter set you should use is the “Hyper-V Legacy Network Adapter”. In general you should not use the network adapter type because it is not enlightened, creates a lot of CPU load in the root, and is generally slower than the Network Adapter. The challenge is you need the Legacy Network Adapter to get a VM working before installing Integration Services. Once your VM is working with Integration Services you should use the Network Adapter and the “Hyper-V Virtual Network Adapter” counter set. Keep in mind Windows Server 2008 and Windows Server 2008 R2 both have integration Services pre-installed.

The “Hyper-V Legacy Network Adapter” counters to monitor are;

• Bytes Dropped
• Bytes Sent / Sec
• Bytes Received / Sec

In the “Hyper-V Virtual Network Adapter” you should monitor;

• Bytes / Sec
• Packets / Sec

Storage:

The storage counters are useful for monitoring the overall disk performance on the system as well as for each VM. The first counter “Physical Disk set will give overall storage performance on the system. The next two are strictly for VM’s.

• Physical Disk
• Hyper-V Virtual Storage Device
• Hyper-V Virtual IDE Controller

Inside the “Physical Disk” counter set I tend to monitor only three things. The first is “Current Disk Queue Length” which gives one an idea of how busy the drives are. The “Current Disk Queue Length” should be around two per drive. If you have a RAID 10 volume with 4+4 (total of 8 drives). Then a queue length of 16 is reasonable. A queue length of 32 might indicate this disk is saturated and is the bottleneck in the system. The next counter I monitor is the “Disk Bytes / Sec”. I generally expect to see about 10MB/sec per drive which is a fairly safe number for most drives. Some can do better and some worse. So for the RAID 10 4+4 a throughput of around 80MB/sec is reasonable for sequential workloads whereas 10MB/sec is not. For random workloads I look at the last counter “Disk Transfers / Sec” and expect to see about 100 IO’s per second (IOPs) per drive. Once again there are drives that do much better like 180 IOPs and some that do worse like laptop drives which are around 60 IOPS. For the RAID 10 4+4 around 800 IOPs for 8Kbytes reads and writes is reasonable. Generally writes are a bit lower.

There are two Hyper-V storage counter sets because of how storage works in Hyper-V. In Hyper-V we provide two virtual storage buses for VM’s. One is IDE and one is SCSI. The Virtual IDE counters show up in the “Hyper-V Virtual IDE Controller” counter set unless Integration Services are loaded and then you will see the activity for both virtual IDE and SCSI in the “Hyper-V Virtual Storage Device” counter set. If you don’t have integration services installed the only the “Hyper-V Virtual IDE Controller” will show the VM disk activity. If you want to read more on Hyper-V storage check out this link – http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx .

I tend to monitor the all the counters on both the “Hyper-V Virtual IDE Controller” and “Hyper-V Virtual Storage Device” because there is just the basics set.

For the “Hyper-V Virtual IDE Controller” there are;

• Read Bytes / Sec
• Write Bytes / Sec
• Read Sectors / Sec
• Write Sectors / Sec

And for “Hyper-V Virtual Storage Device” there are;

• Error Count
• Flush Count
• Read Bytes / Sec
• Write Bytes / Sec
• Read Count
• Write Count

The Error count should always be zero for the virtual storage device.

Conclusion:

There are many counters in Hyper-V which provide useful information and will help you understand what the system is doing. The table below summarizes the counters above in a perfmon like format. Note that you should not always collect all instances in a counter set. Some counter sets have too many counters to collect all instances. If you attempt to collect all counters for all instances you might see periods in the data that are empty. This means the system is not keeping up the amount of counters requested.

In the table below you will see (_Total) or (*). (_Total) means only the total should be collected. (*) means collect all counters. \* means collect all the counters in the counter set. \<name> means only collect that counter. This is the notation perfmon uses in its collection files.

\Hyper-V Virtual Machine Health Summary \*

\Hyper-V Hypervisor\*

\Processor(_Total)\*

\Hyper-V Hypervisor Logical Processor(*)\%Guest Run

\Hyper-V Hypervisor Logical Processor(*)\%Hypervisor Run Time

\Hyper-V Hypervisor Logical Processor(*)\%Idle Run Time

\Hyper-V Hypervisor Logical Processor(*)\%Total Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Guest Run

\Hyper-V Hypervisor Root Virtual Processor (*)\%Hypervisor Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Idle Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Total Run Time

\Hyper-V Hypervisor Virtual Processor (_Total)\*

\Memory\Pages / Sec

\Memory\Available Bytes

\Hyper-V Hypervisor Partition(*)\2G GPA Pages

\Hyper-V Hypervisor Partition(*)\Deposited Pages

\Hyper-V Hypervisor Partition(*)\Virtual Processors

\Hyper-V Hypervisor Root Partition(*)\*

\Hyper-V VM Vid Partition(*)\Physical Pages Allocated

\Hyper-V VM Vid Partition(*)\Remote Physical Pages

\Network Interface(*)\*

\Hyper-V Virtual Switch(*)\*

\Hyper-V Legacy Network Adapter(*)\*

\Hyper-V Virtual Network Adapter(*)\*

\Physical Disk(*)\Current Disk Queue Length

\Physical Disk(*)\Disk Bytes / sec

\Physical Disk(*)\Disk Transfers/sec

\Hyper-V Virtual Storage Device(*)\*

\Hyper-V Virtual IDE Controller(*)\*

Why does my desktop box slowdown when I install Hyper-V?

Hyper-V’s primary target is servers.  We love it when you run it on your desktop however you might encounter some challenges.  One such challenge is that once Hyper-V is installed on your desktop box you might find the overall system performance has dropped.

The most common problem users are experience is related to hi-end graphics cards and drivers.  In Windows Vista / Server 2008 we switched from our XP driver model (XDDM) to a new more performant model (WDDM).  The new drivers used non-cached and write-combined memory in order to allow graphics cards to pump lots of data around the system without impact on CPU caches.  This improves overall system performance.

The new model works great on non-virtualized hardware.  On virtual machines the model causes lots of root TLB flushing (Virtual to Physical address translation cache).  It is this constant flushing of the root’s TLB that is slowing down the system.

There are a couple of things you can do today;

• One is stop using your high end graphics card in your dev box (not likely).
• You can use VGA.sys as your graphics driver.  If you do this you will loose multi-monitor support and a few cool graphics features like glass. (somewhat likely)
• Look for an XP driver for your system.  Most XP drivers were not 64bit so this option is a bit limited because Hyper-V requires the root to be 64bit.  (most XP drivers are 32bit – bummer)
• The **Best Option** is use Win7 (currently in Beta) and upgrade the processors and/or system to one that supports Second Level Address Translation (SLAT) in hardware. 
  • AMD calls this feature “Rapid Virtualization Indexing” and is present on their Phenom and Opteron processors.  In engineering literature it is sometimes called Nested Page Tables (NPT)
  • Intel calls this feature Extended Page Tables (EPT) and is present on their Nehalem based processors.  The marketing name for the Nehalem is the Intel Core i7 for desktops and the Xeon 5500 series for servers.

For addition information on the issue see http://support.microsoft.com/kb/961661

Why does my Avg Disk Write / Sec counter keep climbing?

There have been a number of issues with time in W2k3 guests, the first of which was covered in http://blogs.msdn.com/tvoellm/archive/2008/06/05/negative-ping-times-in-windows-vm-s-whats-up.aspx .  The first issue involved negative ping times.

A recent issue has come up around performance counters reporting huge disk write times that did not correspond with how the Virtual Machines was actually behaving.  it took quite a while to repro the issue because it did not always happen.  It turned that that the key to reproducing the strange increasing of times in the performance counters was due to the Virtual Machine (VM) having two virtual processors.

In Hyper-V there is guarantee that on a given VP the TSC register will never go backwards.  The TSC is basically a CPU cycle counter used for precise timing.  New operating system like WS08 are aware of Hyper-V and make better timing calculation and use different sources of time.

So why does the TSC matter?  The problem is the performance counter can read the start value of an operation on one VP and the end value on a different VP.  Since the counter collection also includes collecting the time it might actually go negative causing for example the Avg Disk Write / Sec to continue to increase until the timing delta used actually rolls over.  You can see the behavior in the graph below;

The solution is the same as the problem with negative ping time and that is to use the “/usepmtimer” in the boot.ini file.  This causes the performance counters to use the PM timer rather than the TSC.  the PM timer is consistent across all processor albeit slower to read that TSC.  The TSC does not cause a switch into the Hypervisor where the PM timer does.

SQL Server 2008 on Windows Server 2008 Hyper-V Performance Guidance

Wow this is great day.  The SQL team has worked really hard in creating a performance guidance document with solid suggestions and performance numbers when running on WS08 Hyper-V.

They looked at OLAP and OLTP workloads.  Even if you are not running SQL it will give you and idea of how other database like workloads are likely to perform.

Check it out...
http://sqlcat.com/whitepapers/archive/2008/10/03/running-sql-server-2008-in-a-hyper-v-environment-best-practices-and-performance-recommendations.aspx

Hyper-V Performance Counters - Part five of many - "Hyper-VM VM Vid Numa Node"

There are a couple of performance counter sets in Hyper-V that show memory usage.  These counters tend to show memory used by one component of the system and the "Hyper-V VM Vid Numa Node" counter set is no different. 

This counter set is intended to show memory used by the Virtual Infrastructure Driver (Vid).  The good news is this counter set shows "guest physical memory".  This memory includes the allocation you specify when creating the VM and some other support memory like video memory.  It is the majority of the memory used by the Virtual Machine (VM).

Hyper-V VM Vid Numa Node shows memory on a per node basis.  If you have an AMD machine you will likely see a Non-uniform Memory Access (NUMA) node for each processor.  If you have an Intel based machine you will see anywhere from one to many nodes depending on the architecture.

The following are the performance counters in the counter set;

• PageCount                   - Number of 4K pages that the node contains.  To find the total physical memory on the system you would add PageCount from each NUMA node and multiply by 4Kbytes.
• PagesInUse                  - Number of pages in use by the Vid on this node.  Keep in mind this is most but now all memory supporting a VM.
• ProcessorCount           - Number of logical processors this node contains.  Logical process are equal to the number of processor (sockets) * number of cores per processor * 2 if you have HyperThreading (HT) or now called Symmetric Multithreading (SMT)

  Enjoy,

  Anthony F. Voellm (aka Tony)

Looking for that last ounce of performance? Then try affinitizing your VM to a NUMA node

There are not many performance knobs in Hyper-V which is by design.  We really seek out of the box performance.  However if you are looking for that last bit of performance from your Virtual Machines (VM’s) and have already made a good selection for networking and storage you might consider setting the Non-Uniform Access (NUMA) node.

On AMD and some Intel based architectures the machine might be defined as NUMA.  You can find out if the machine is NUMA by looking at the "Hyper-VM VM Vid Numa Node" performance counters.  This counter set will report a counter set instance for each NUMA node unless the machine is a single node. 

NUMA means each CPU has a different path to memory and those paths can have various lengths.  For example CPU 0 on Node 0 when accessing CPU M’s memory on Node X might take 10ns where as CPU 0 on Node 0 accessing CPU N’s memory on Node Y will take 20ns.  It’s this difference in memory access times that can impact overall VM performance.  The worst case would be for a VM’s Virtual Processor (VP) to be running on a node furthest from where the memory for the VM is allocated.

In order to improve performance you can place VM’s on different nodes.  In addition to placing the memory the Hypervisor scheduler will attempt to run the VM’s VP’s near where the memory where the VM is allocated.  This creates a dual affinity which can be very beneficial.

If you would like to try placing VM’s on different nodes to improve performance you can use the code sample below to place the VM.

Example 1 – List currently configured VM’s

            C:\> .\numa.ps1 /list

Example 2 – Set NUMA affinity to node 1.  Node numbering starts at node 0

            C:\> numa.ps1 /set tonyvm 1

Example 3 -  Clear NUMA affinity

            C:\> numa.ps1 /clear tonyvm

  Enjoy,

  Anthony F. Voellm (aka Tony)

Save the following powershell script into a file named numa.ps1.  WS08 has powershell as part of the release if you are running management from a remote machine that is XP / Vista you can download powershell here http://www.microsoft.com/windowsserver2003/technologies/management/powershell/default.mspx

################################################
# Developer: Anthony F. Voellm
#          : Taylor Brown
# Copyright (c) 2008 by Microsoft Corporation
# All rights reserved
#
# This is "demonstration" code and there are no
# warrantees expressed or implied
################################################

# This script will set the Virtual Machine to run
# on a specific NUMA node

# Check command line arguments

if (($args.length -lt 1) -or
    (($args[0] -ne "/list") -and
     ($args[0] -ne "/set") -and
     ($args[0] -ne "/clear")) -or
     (($args[0] -eq "/set") -and ($args.length -lt 3)) -or
     (($args[0] -eq "/clear") -and ($args.length -lt 2))) {
     Write-Host "numa.ps1 /list [<Hyper-V host>]"
     Write-Host "numa.ps1 /set <vm machine name> <required node> [<Hyper-V host>]"
     Write-Host "numa.ps1 /clear <vm machine name> [<Hyper-V host>]`n"
     Write-Host "Options:"
     Write-Host "`t/list - show configured VM's"
     Write-Host "`t/set <vm machine name> <required node> - set the NUMA node for the VM"
     Write-Host "`t/clear <vm machine name> - clear NUMA node seting for the VM"
     exit;
  }

# just display VM's
if ($args[0] -eq "/list") {
  if ($args.length -gt 1) {
    $HyperVHost = $args[1];
  }
  Get-WmiObject -Namespace 'root\virtualization' -Query "Select * From Msvm_ComputerSystem" | select ElementName
  exit;
}

# Set or clear

$HyperVHost = '.';
if ($args[0] -eq "/set") {
  if ($args.length -gt 3) {
    $HyperVHost = $args[3];
  }
  $VMName = $args[1];
  $RequiredNode = $args[2];
} elseif ($args[0] -eq "/clear") {
  if ($args.length -gt 2) {
    $HyperVHost = $args[2];
  }
  $VMName = $args[1];
}

  
#Main Script Body
$VMManagementService = Get-WmiObject -Namespace root\virtualization -Class Msvm_VirtualSystemManagementService -ComputerName $HyperVHost

$Query = "Select * From Msvm_ComputerSystem Where ElementName='" + $VMName + "'"
 

$SourceVm = Get-WmiObject -Namespace root\virtualization -Query $Query -ComputerName $HyperVHost

$VMSettingData = Get-WmiObject -Namespace root\virtualization -Query "Associators of {$SourceVm} Where ResultClass=Msvm_VirtualSystemSettingData AssocClass=Msvm_SettingsDefineState" -ComputerName $HyperVHost

if ($args[0] -eq "/set") {
  $VMSettingData.NumaNodesAreRequired = 1
  $VMSettingData.NumaNodeList = @($RequiredNode)
} else {
  $VMSettingData.NumaNodesAreRequired = 0
}

$VMManagementService.ModifyVirtualSystem($SourceVm, $VMSettingData.PSBase.GetText(1))

Hyper-V Storage Analysis

Hyper-V Perfies,

In a previous blog entry I explained the different types of storage choices you have with Hyper-V.  However we had nore released yet so no numbers could be published.  You can find the original post below...

http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx

Now that we have release I wanted to share some numbers from a mid-range storage device. 

For the storage performance results below we used;

•  A dual core 2GHz system
• 8 SAS disk system configured to run RAID0
• Off the shelf SCSI RAID controller
• We pre-populated the Virtual Hard Disks used with non-zero data to avoid testing an optimization we have were zero block are not actually written to Dynamic / Diff VHDs if they have never been written before. 
• IOMeter was used to generate the load and gather results.
• Root results were tested using VHD loopback.  This means we used the WMI interfaces in Windows Server 2008 to mount the VHD into the root without any virtual machine.  This was pretty close (<1% diff) to bare metal performance (no Hyper-V enabled).  Check out this link on how to do it http://blogs.msdn.com/virtual_pc_guy/archive/2008/02/01/mounting-a-virtual-hard-disk-with-hyper-v.aspx 
• SCSI and IDE results were collected from a Windows Server 2008 64bit one virtual processor guest.
• Only read performance was collected for fixed Root VHD + 1 Diff Disk.  This is why some parts of the graphs are label "NOT TESTED".  Reads have to traverse the differencing chain and writes go to the outer most disk so write look just like dynamic disk write performance and were not collected.

We chose this configuration because it was pretty low to mid range, would be very low cost, and therefore within reach of all customers running Hyper-V.  If you are running Hyper-V with big servers and storage containing 100's of disks you find it handles it very well with very little overhead.  That might be the subject of a future post.

 The following graphs show sequential I/O performance of 512Byte reads and writes.  We choose this to measure because it would approximate the behavior or loading applications, editing small files, and other types of workloads doing sequential I/O.  We saw big improvements with larger I/O's (>4x) but for many workloads those are not that likely to be encountered.

The following graphs show 8KByte random read and write performance.  We chose 8KByes reads and writes because many servers such as SQL Server 2008 use this sized I/O for most operations. 

From the graphs you can see that Emulated IDE (aka you are not running with Integration Components installed) is never a good choice.

 Passthough followed by fixed disks have the overall best performance.  Dynamic and differencing disks are good choices when you need flexibility.

   Enjoy,

   Tony Voellm

   Liang Yang

WS08 Hyper-V now supports 24LP

Hyper-V Perfies,

Now available to the general public is Windows Server 2008 (WS08) Hyper-V 24 Logical Processor (LP) Support.  This revises our RTM support limit of 16 Logical Processors.  We also increased the number of supported running virtual machines from 128 to 192.  This update also has the side benefit of improving workloads that start large numbers of processes by increasing the process address space cache limit from 192 to 384.  All in all this update will allow you to get even more mileage from WS08 Hyper-V.

You can find the needed update here http://www.microsoft.com/downloads/details.aspx?FamilyID=fe36823a-7e5a-4262-9bf5-d6b3ae3ad375&DisplayLang=en

Here is some Q & A to help you figure out how and when to install this update;

Q: Should everyone install this update?

A: No.  Install this update only on machine needing 24LP support or if you are running more than 192 processes in the root (no recommeded) / guest.  Keep in mind there are very few workloads that create this many processes and there are only two that come to mind.  Terminal Services and Hyper-V running > 192 virtual machines (VM).

Q: Does this mean I can now run 192 VM's on my 16 logical processor system?

A: No.  We only support an 8 virtual processor (VP) to 1 logical processor (LP) limit.

Q: What does 8 VP to 1 LP mean?

A: This mean is you have 8 logical processors you can run up to 64 virtual processors in any combinaiton.  For example 16 Windows Server 2008 (WS08) VM's with 4 VP each, or 64 Windows Server 2003 VM's with 1 VP each, or 8 WS08 4VP VM's with 32 W2k3 1VP VM's.  You get the picture.

Q: What is a logical processor aka LP?

A: A logical processor is what we call anything that shows up in Windows as a processor in task manager.  More precisely it is the "number processors * number of cores * 2 if you are running Hyper Threading (HT) or now called Symmetric Multi Threading (SMT)."  For example a dual proc quad core system with SMT enabled has 2 * 4 * 2 = 16 LP.  A quad proc dual core system has 4 * 2 = 8 LP.

Leading the 24LP effort was a blast and it is also why my blog has been pretty quiet recently.

  Enjoy,

  Tony Voellm

BizTalk team releases best practice doc for running on WS08 Hyper-V

Below is a great link on how to get BizTalk 2006 working on top of Hyper-V.  It includes practical advice that applies to other servers and roles as well.

Check it out... http://msdn.microsoft.com/en-us/library/cc768518.aspx

   Tony

How to get Processor Utilization for Hyper-V via WMI

There are a number of groups building management software (OEMs, Microsoft, …) for Hyper-V which is cool to see.  A common ask from these teams has been around reading and computing VM CPU usage.

The following is an example of how to compute Hyper-V guest processors usage.  You can use the same formula for “% Total Run  Time, “% Hypervisor Time” and “% Idle time”.   The counters show up in the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object as “PercentGuestRunTime”, “PercentTotalRunTime”, “PercentHypervisorRunTime”, and “PercentIdleTime”.

To make the formula easier to read lets use:

                G_N – Percent Guest Run Time (substitute other usage values here)

                P_N – Timestamp_PerfTime

                F_N – Frequency_PerfTime

                LP = Number of logical processors (Get this from the “Hyper-V Hypervisor” counterset)

                         F₁ * (G₂ – G₁)

 Utilization =  -----------------------------------

                     100000 * LP * (P₂-P₁)

G₂ and P₂ are the second values read and G₁ and P₁ are the first values read.

To test the formula lets read the “Hyper-V Hypervisor Logical Process” counterset twice via the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object about 10 seconds apart with a single VM running at 100% Guest CPU.  Since my test machine has two CPU’s (2 LP) this means we should see about 50% overall utilization.

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

    C1TransitionsPersec = 409197889

    C2TransitionsPersec = 0

    C3TransitionsPersec = 0

    Caption = null

    ContextSwitchesPersec = 889911109

    Description = null

    Frequency_Object = 0

    Frequency_PerfTime = 14318180

    Frequency_Sys100NS = 10000000

    HardwareInterruptsPersec = 92282462

    InterProcessorInterruptsPersec = 8174254

    InterProcessorInterruptsSentPersec = 8174254

    MonitorTransitionCost = 16

    Name = _Total

    PercentC1Time = 4193635539355

    PercentC2Time = 0

    PercentC3Time = 0

    PercentGuestRunTime = 314976793671

    PercentHypervisorRunTime = 53745475789

    PercentIdleTime = 8385447570540

    PercentTotalRunTime = 368722269460

    SchedulerInterruptsPersec = 384836664

    TimerInterruptsPersec = 33425466

    Timestamp_Object = 0

    Timestamp_PerfTime = 6268633722843

    Timestamp_Sys100NS = 4199325031975

    TotalInterruptsPersec = 518718846

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

    C1TransitionsPersec = 409201218

    C2TransitionsPersec = 0

    C3TransitionsPersec = 0

    Caption = null

    ContextSwitchesPersec = 889922035

    Description = null

    Frequency_Object = 0

    Frequency_PerfTime = 14318180

    Frequency_Sys100NS = 10000000

    HardwareInterruptsPersec = 92283571

    InterProcessorInterruptsPersec = 8174425

    InterProcessorInterruptsSentPersec = 8174425

    MonitorTransitionCost = 16

    Name = _Total

    PercentC1Time = 4193667417779

    PercentC2Time = 0

    PercentC3Time = 0

    PercentGuestRunTime = 315044817737

    PercentHypervisorRunTime = 53746578312

    PercentIdleTime = 8385511363951

    PercentTotalRunTime = 368791396049

    SchedulerInterruptsPersec = 384840537

    TimerInterruptsPersec = 33426627

    Timestamp_Object = 0

    Timestamp_PerfTime = 6268728855292

    Timestamp_Sys100NS = 4199364353043

    TotalInterruptsPersec = 518725160

Based the formula above and the data below we get = 51% which is spot on.

  Enjoy,

    Tony Voellm

WS08 Hyper-V as RTMed!!!!

Wow its been a great ride helping get Hyper-V to perform the way it does.  So much so I've not posted in a while.  You will see some upcoming posts on performance counters, WMI perf interfaces, and real data (yes now with RTM I can post data).

In the meantime check out my co-workers post on where to get the final (aka RTM) WS08 Hyper-V bits...

http://blogs.technet.com/jhoward/archive/2008/06/26/hyper-v-rtm-announcement-available-today-from-the-microsoft-download-centre.aspx 

   Tony

Windows Server Performance 2008 Tuning guide now includes Hyper-V