All topics fundamental : Hyper-V

Hyper-V Performance FAQ R2

tvoellm — Fri, 18 Dec 2009 22:35:00 GMT

Anthony F. Voellm (aka Tony)

12/17/2009

http://blogs.msdn.com/tvoellm

This FAQ is titled R2 since it updates the original FAQ as of the Windows Server 2008 R2 Hyper-V. Since Hyper-V originally released we have continued to improve it. We have new features in R2 like Live Migration, support for 64 processors and lots of improvements in networking and storage.

******************************************************************************

Contents

· Q: Is there a place that compares features and versions of Hyper-V?

· Q: What is the recommend configuration for performance testing?

· Q: How do I monitor performance?

· Q: Is there any “official” documentation for Hyper-V performance counters?

· Q: Are there any official virtualization benchmarks I can use to compare machines and virtualization solutions?

· Q: Is there any common terminology used to talk about virtual machine configurations?

· Q: How much memory should I reserve for the root?

· Q: Are there any services that should be stopped?

· Q: Is it ok run applications / processes in the root OS?

· Q: Is there a simple way to disable the hypervisor to run some baseline tests on the native system

· Q: Should I use passthrough disks or iSCSI attached to the guest for storage?

· Q: Are there ways to reduce overall networking overhead?

· Q: Are there additional knobs for performance nuts?

· Q: Are their additional resources that are useful for understanding Hyper-V?

******************************************************************************

Q: Is there a place that compares features and versions of Hyper-V?

A: http://blogs.msdn.com/tvoellm/archive/2009/08/05/what-s-new-in-windows-server-2008-r2-hyper-v-performance-and-scale.aspx

Q: What is the recommend configuration for performance testing?

A: Here are some simple steps:

1. Be sure to have the latest WS08 Hyper-V build – Windows Server 2008 R2 is the current verion.

2. Next you need to make sure you are running a “Supported OS” with the latest Service Packs installed on the guest OS. For the latest on which OS are supported and the number of virtual processors that can be used see http://support.microsoft.com/kb/954958

3. Make sure the guest and root OS have integration components installed (http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx **and** http://blogs.msdn.com/tvoellm/archive/2008/01/02/hyper-v-integration-components-and-enlightenments.aspx )

4. Make sure you are using the “Network Adapter” and not the “Legacy Network Adapter”. The legacy adapter uses emulation which creates additional CPU overhead.

5. Use pass-through disks attached to SCSI for the best performance. Next best is Fixed VHD attached to SCSI. To understand storage better see (http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx )

6. Follow these tips for avoiding pitfalls http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx

Q: How do I monitor performance?

A: First you need to understand that the clocks in the root and guest Virtual Machines may not be accurate See (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx ). Given an understanding of clocks you can see why we implemented the “Hyper-V Hypervisor Logical Processor” performance counters (access using perfmon) which are not skewed by clock effects. There are other Hyper-V performance counters that are useful. See the following for more details (http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx )

There is also a detailed posting on how to monitor Hyper-V performance at http://blogs.msdn.com/tvoellm/archive/2009/04/23/monitoring-hyper-v-performance.aspx

Q: Is there any “official” documentation for Hyper-V performance counters?

A: As of the writing of this blog there is no “official” documentation. However you can find documentation on my blog here http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx .

Q: Are there any official virtualization benchmarks I can use to compare machines and virtualization solutions?

A: There are no official and widely accepted benchmarks. That said, Intel put together a benchmark called vConsolidate and the SPEC group is working on SpecVirt.

Q: Is there any common terminology used to talk about virtual machine configurations?

A: Yes - Internal to Microsoft we typically use the following;

· Native = System without the Hyper-V role. This means you have no virtual drivers, virtual switch, …

· Root = Most people call this the host however we refer to the management partition of Hyper-V as the root because it is technically not a hosted VM solution. You can also think of this as the OS that boots Hyper-V and handles the management of VMs, devices, memory and other share VM resources.

· Guest = Guest Virtual Machine.

· 8p.child.2x1p or better 8p.child.2VMx1VP = A system with 8 logical processors / cores running 2 Virtual Machines (VM) each with 1 Virtual Processor (VP)

Q: How much memory should I reserve for the root?

A: There is a good discussion on how much memory to leave for root in the “Windows Server 2008 Tuning Guide.” You can find it here http://www.microsoft.com/whdc/system/sysperf/Perf_tun_srv.mspx. The relevant section from the guide is as follows:

Correct Memory Sizing

You should size VM memory as you typically do for server applications on a physical machine. You must size it to reasonably handle the expected load at ordinary and peak times because insufficient memory can significantly increase response times and CPU or I/O usage. In addition, the root partition must have sufficient memory (leave at least 512 MB available) to provide services such as I/O virtualization, snapshot, and management to support the child partitions.

A good standard for the memory overhead of each VM is 32 MB for the first 1 GB of virtual RAM plus another 8 MB for each additional GB of virtual RAM. This should be factored in the calculations of how many VMs to host on a physical server. The memory overhead varies depending on the actual load and amount of memory that is assigned to each VM.

Q: Are there any services that should be stopped?

A: Not if you are running Server Core which is the ideal root OS to use. Regardless of running server core vs full server you should close the Hyper-V Management Console because it has a noticeable impact on CPU. If you want the details see http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx

Q: Is it ok to run applications / processes in the root OS?

A: You should avoid running any Role / Feature or custom service in the root. If you have services you want to run put them in a guest VM. Running roles in the root can have a negative impact on the guest VM’s. This is due to how the Hypervisor scheduler handles the root virtual processors.

Note: Outside pure performance deploying applications in the root has licensing implications as in a virtualized environment the so called virtual use rights only allow to run virtualization itself (and its associated management like management agents etc) in the physical environment. More information can be found at:

· Windows Server 2008 R2 Licensing overview - look for Virtual Use Rights

· Windows Server 2008 Licensing Overview (Pdf)

Q: Is there a simple way to disable the hypervisor to run some baseline tests on the native system.

A: Yes. “bcdedit /set hypervisorlaunchtype off” and reboot the server. You should also consider changing the protocols on the root network device to re-enable TCP/IP and turn off the “Microsoft Virtual Network Switch Protocol”.

To turn it back on do “bcdedit /set hypervisorlaunchtype on” and reboot the server.

Q: Should I use passthrough disks or iSCSI attached to the guest for storage?

A: The decision depends on what features you need to expose to the guest. In the passthrough case the drive will show up without knowledge of the underlying LUN.

Educated guess: If you are looking for raw performance passthough will give you the best result.

Reason: When doing IO from the guest using passthough you traverse the guest storage stack + disk stack in the root. When doing iSCSI you traverse the storage + networking stack in the guest + root networking stack

Q: Are there ways to reduce overall networking overhead?

A: The way to reduce storage overhead (ie adding CPU use) is to use passthrough disks. There are also ways to reduce networking overheads by using two new features in R2. One feature is called Virtual Machine Queues (VMQ) and the other is called Chimney Offload. VMQ reduces overheads by reducing the cost of routing incoming packets, more optimized copy paths, and better interrupt distribution. Chimney offload is helpful for long running connections. It also improves overhead by reducing copy path costs. The challenge is that no networking devices I’ve seen support both of these features. Typically they support VMQ or Chimney offload.

If you have a device that supports one of these features the following article will help you configure it:

Networking Deployment Guide: Deploying High-Speed Networking Features

http://download.microsoft.com/download/8/E/D/8EDE21BC-0E3B-4E14-AAEA-9E2B03917A09/HSN_Deployment_Guide.doc

Q: Are there additional knobs for performance nuts?

A: We are trying to make Hyper-V knob-less. However we are engineers and here are some tips.

1. Remove the CDROM drive from the guest if you don’t need it

2. Look into the Caps / Weights / Reserves in the CPU config. You can use these to “balance” workloads.

3. You can use the WMI interfaces to force a VM to a particular node (http://blogs.msdn.com/tvoellm/archive/2008/09/28/Looking-for-that-last-once-of-performance_3F00_-Then-try-affinitizing-your-VM-to-a-NUMA-node-.aspx) Hyper-V does not guarantee node affinity for VP’s but it does for memory. There is a good chance the VP’s will stay on the node because the scheduler is NUMA aware.

Q: Are their additional resources that are useful for understanding Hyper-V?

A: Yes – here is a list

Useful IO profiles for simulating various workloads

tvoellm — Thu, 07 May 2009 07:21:00 GMT

A buddy of mine [Liang Yang] recently shared the following table with me and I thought you might also find it useful with trying to simulate various workloads. In particular when testing and evaluating Hyper-V.

A tool like IOMeter [from www.iometer.org] can be programmed with the mixes of reads/wrtes and sequential/random IO's detailed below.

Workload Category	I/O Size	Percentage of READ vs. WRITE	Percentage of RANDOM vs. SEQUENTIAL
Web File Server	4KB	95% RD vs. 5% WR	75% RAND vs. 25% SEQ
Web File Server	8KB	95% RD vs. 5% WR	75% RAND vs. 25% SEQ
Web File Server	64KB	95% RD vs. 5% WR	75% RAND vs. 25% SEQ
Decision Support System DB	1MB	READ	RANDOM
Media Streaming	64KB	98% RD vs. 2% WR	SEQUENTIAL
SQL Server Log	64KB	WRITE	SEQUENTIAL
OS Paging	64KB	90% RD vs. 10% WR	SEQUENTIAL
Web Server Log	8KB	WRITE	SEQUENTIAL
OLTP DB	8KB	70% RD vs. 30% WR	RANDOM
Exchange Server	4KB	67% RD vs. 33% WR	RANDOM
Workstation	8KB	80% RD vs. 20% WR	80% RAND vs. 20% SEQ
Video on Demand	512KB	READ	RANDOM

Enjoy - Tony Voellm

Monitoring Hyper-V Performance

tvoellm — Thu, 23 Apr 2009 02:43:00 GMT

Now that Hyper-V has been in the market for over 9 months a common question that has come my way is “what should I monitor?” This question has been asked for a couple of reasons such as; How do I know if my machine is overloaded? How can I figure out what resources were used so I know what to bill? These questions usually amount to measuring the networking, storage, and CPU use. Another common question is where do you monitor Hyper-V? I’ll cover these questions and more.

Q: Where do I monitor Hyper-V?

Let’s start with the question of where to monitor. In order to answer this question you have understand a little of the Hyper-V architecture. Hyper-V has three main components – the virtstack, devices, and hypervisor. Windows Server 2008 is what boots the system and launches the virtstack and hypervisor. The virtstack responsible for handling emulated devices, managing VM’s, servicing I/O, and more. The hypervisor is responsible for scheduling Virtual Processors, managing interrupts, servicing timers, and controlling other chip level functions. It does not understand devices or I/O (ie there are no hypervisor drivers). The devices are part of the root and are also installed in guests as part of the Integration Services.

Since the root has a full view of the system and controls the VM’s it is also responsible for providing monitoring information via WMI and Performance Counters. You can see a full list of performance counters that are provided from the root @ http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx . See the WMI reference on MSDN @ http://msdn.microsoft.com/en-us/library/cc136992(VS.85).aspx . A common question is can you monitor a Virtual Machine from within the running guest? From within the guest there is only access to the performance counters provided by the OS that is running in the guest. All those should work as expected however keep in mind that utilization counters (like % Processor) are relative to the amount of virtual processor used. Not the physical processor. So it might appear the utilization counters are over reporting the actual physical resource usage. Rate counters (like network packets / sec) don’t have this problem. For more on guest counter skew see http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx

We could have provided some VM specific data inside the guest and might in the future. However our current thinking is not to do this. If you are running a Windows OS or have a tool that understands the WMI protocol you can use the remote management API’s to query the root (see msdn.microsoft.com for WMI documentation on how to do this). You can also do this from outside the physical server as well and tools like System Center Virtual Machine Manager.

Now that you know to monitor Hyper-V and all virtual machines from the root partition let’s look at what you should monitor.

Q: What should I monitor?

What you monitor on a regular basis really depends on what you are trying to do. I’ll cover the major resources and what to typically monitor. This will be a good starting point for you.

Here are the top level performance counters to monitor and I’ll go into more detail on each;

Overall health:

Hyper-V Virtual Machine Health Summary
Hyper-V Hypervisor

Processor:

Processor
Hyper-V Hypervisor Logical Processor
Hyper-V Hypervisor Root Virtual Processor
Hyper-V Hypervisor Virtual Processor

Memory:

Memory
Hyper-V Hypervisor Partition
Hyper-V Root Partition
Hyper-V VM Vid Partition

Networking:

Network Interface
Hyper-V Virtual Switch
Hyper-V Legacy Network Adapter
Hyper-V Virtual Network Adapter

Storage:

Physical Disk
Hyper-V Virtual Storage Device
Hyper-V Virtual IDE Controller

Now let’s go into each counter set and how to use them to monitor the system. As time allows I have been documenting each performance counter set on my blog.

You should take note of the bold italics because they represent the names of counters sets and counters.

Overall health:

There are really two counter sets I use to get an overall understanding of the system . The first is the “Hyper-V Virtual Machine Health Summary” which only has two counters; “Health Ok” and “Health Critical”. If anything is Critical it means some resource (most likely disk) has been exhausted or other unrecoverable error has occurred. If you server see “Health Critical” you should take action to figure out what has happened.

The second counter set is the “Hyper-V Hypervisor” counter set which is detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/09/hyper-v-performance-counters-part-two-of-many-hyper-v-hypervisor-counter-set.aspx . I like to use this counter set to understand how many Logical Processors the system recognizes (Logical Processor), the number of virtual machines running (“Partitions” – 1), and the total number of Virtual Processors (Virtual Processors). The logical processors (LP) are important because they are where all the work is done. They are a representation of the physical processor (core or CPU thread like HT and SMT). The virtual processors (VPs) tell you something about the guests and also if the system is overloaded. You should make sure the VP to LP ratio does not exceed eight to one because we don’t currently support beyond this limit. Hyper-V does not set a hard cap so you can exceed it. Just understand you are in largely untested configuration and might see guest failures beyond 8:1. Some other limits to be aware of is WS08 Hyper-V supports only 24 Logical Processors (LPs) and Windows Server 2008 R2 Hyper-V has a current stated limit of 32 LPs as of Windows Server 2008 R2 Beta. We will likely push this limit up.

The last counter in the “Hyper-V Hypervisor” counter set that I use is the “Total Pages”. This counter gives an indication of how much meta data memory the Hypervisor is using to manage the virtual machine. Unfortunately this counter does not capture all the overhead because another component called the Virtual Interface Driver (VID) also has overhead to manage partitions and in WS08 the “Hyper-V VID Partition” does not work . The good news is this counter set does work in Windows Server 2008 R2.

Processor:

Once you have an idea of the overall system capabilities and configuration though the “Hyper-V Hypervisor” counter set you will want to monitor the processors on the system. The most important counter set to monitor is the “Hyper-V Hypervisor Logical Processor”. This counter set allows you to determine how much of the physical processor are being used. The virtual processor counter sets only show a slice of the “Hyper-V Hypervisor Logical Processor”.

Hyper-V Hypervisor Logical Processor
Hyper-V Hypervisor Root Virtual Processor
Hyper-V Hypervisor Virtual Processor

The Hyper-V Hypervisor Logical Processor counter set is detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/09/hyper-v-performance-counters-part-three-of-many-hyper-v-logical-processors-counter-set.aspx. The most useful counters in this counter set are the following;

%Guest Run
%Hypervisor Run Time
%Idle Run Time
%Total Run Time

I generally only look at the _Totals. There is one logical processor that that carries more load than the rest and that is LP0. This LP is where all interrupts in the system are directed and if there is too much load you can see this LP hit 100% which likely means IO is a bottleneck in the system. There are some technologies in Windows Server 2008 R2 that help reduce the load for networking and those are – VMQ, Chimney and RSS. There is no RSS support in guest VM’s.

The “Hyper-V Hypervisor Root Virtual Processor” and “Hyper-V Hypervisor Virtual Processor” are just slices of the LP counter and can help you understand how much total CPU the root and guests are using on the system. These counters are detailed here - http://blogs.msdn.com/tvoellm/archive/2008/05/12/hyper-v-performance-counters-part-four-of-many-hyper-v-hypervisor-virtual-processor-and-hyper-v-hypervisor-root-virtual-processor-counter-set.aspx . There are real no limits one should expect for these counters however I generally expect to see the “% Hypervisor Time” be below 25%. Any higher and this could indicate you are not running with integration services installed. You should always make sure you have Integration Services installed for the best performance. See this link for me detail - http://blogs.msdn.com/tvoellm/archive/2008/01/02/hyper-v-integration-components-and-enlightenments.aspx .

You should also monitor the “Processor” counter set. This counter set is only for the root CPU and does suffer from skew as detailed here - http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx. Even with the skew this counter set is useful because it gives you an idea of how busy the root is. Remember the root is involved in all IO. This means that when the root CPU’s are saturated your whole system is likely saturated. In general you want to see the root CPU lower than 10% utilization and over 50% might indicate an issue.

Memory:

A common question I get is – “how much memory is a VM using?” There is no simple answer because different layers account for memory. For example the “Hyper-v [Root] Partition” counters determine how much memory the Hypervisor is managing and using on behalf of a VM which includes the guest address space but not all the memory in the worker process and VID partition. The “Hyper-V VM Vid Partition” counters account for the guest address space and any additional memory the VID needs to manage the VM. Nothing accounts for the memory in the Worker Process that is paired with the Hypervisor and VID Partition except the root processor memory counters but I know of no way to figure out what the pairing is. The root also uses memory to service IO on behalf of a VM and there is no accounting of this memory other than in the “Memory” counters which is not VM specific.

The following are the counter sets one should monitor in general;

Hyper-V Hypervisor Partition
Hyper-V Hypervisor Root Partition
Hyper-V VM Vid Partition
Memory

The Hyper-V Hypervisor [ROOT] Partition counters are interesting because they indicate in the “1G GPA Pages” and “2M GPA Pages” counters whether or not a VM is using large pages which improves overall VM performance. Large pages are only used on systems that have vTLB hardware support. See more on vTLB support here – http://blogs.msdn.com/tvoellm/archive/2009/04/06/why-does-my-desktop-box-slowdown-when-i-install-hyper-v.aspx . The partition counter set also indicates in the “Deposited Pages” counter how much memory the hypervisor is using for managing the VM. To figure out most of the memory used for the VM in Megabytes you can use the following formula; almost_total_vm_memory = “1G GPA Pages” * 1024 + (“2M GPA Pages” * 2) + ((“4K GPA Pages” + “Deposited pages”) / 256). The last counter that is interesting in the partition counter set is the “Virtual Processors” counter which lets you know how many Virtual Processors a VM is configured to use.

The Hyper-V VM Vid Partition counters have two interesting counters. The “Physical Pages Allocated” is the total number of guest pages and VID pages needed to manage the VM. The “Remote Physical Pages” let you know on NUMA based systems if a VM is spanning multiple nodes. You really want to avoid this whenever possible. You can require a VM to start of a particular node or nothing by using the API at http://blogs.msdn.com/tvoellm/archive/2008/09/28/Looking-for-that-last-once-of-performance_3F00_-Then-try-affinitizing-your-VM-to-a-NUMA-node-.aspx . Another way is to stop and restart the VM and if possible Hyper-V will allocate all memory on a single NUMA node.

The Memory counter set allows you to monitor how much memory is being consumed in the root. The root is responsible for managing all memory in Hyper-V. When a VM starts you will see the "Available Bytes" go down by at least the amount of memory given to the guest plus around another 16 - 64MB for guests meta data structures.

My recommendation is to monitor the following counters;

Available Bytes - This will give you an idea of how much memory is remaining for guests. There is a reserve of 256MBytes or 512MBytes that the root will always leave outside of guest memory. The exact amount varies but Hyper-V release. So if you find a time when a VM wont start it may be there are too few available bytes to satisfy the reserve.
Pages / Sec - This is a measure of memory pressure since it tracks hard faults. Those are page faults that require a disk access. Usually the cause for the number to spike is when there are two few available bytes on the system and processes are competing with each other for physical RAM.

Networking:

The network counters are useful for monitoring the overall networking performance on the system. The most important thing to generally monitor is the total throughput counters to make sure the NICs are not getting saturated. Once the NICs are saturated your overall system performance will be capped because no more web requests, remote storage requests, queries, etc can be received than what is currently being handled. The first counter set “Network Interface” gives the overall performance of physical device where as the other counter sets listed below represent the activity of the virtual switches and network adapters in the VM’s.

Network Interface
Hyper-V Virtual Switch
Hyper-V Legacy Network Adapter
Hyper-V Virtual Network Adapter

For the Network Interface the following are the top level counters to monitor;

Bytes Total / Sec
Offloaded Connections
Packets / Sec
Packets Outbound Errors
Packets Receive Errors

However if I’m going to monitor networking I generally look at all the counters. Mostly you want to make sure the network is not saturating and that the error counts are low. If the error counts grow rapidly you might have too much load on the system or some problem in end- to-end connectivity (including hardware).

The “Hyper-V Virtual Switch” counters are good to monitor because depending on how you have configured your network some or all of the traffic might only exist on the virtual switch. For example guest to guest packets don’t have to leave the machine to be routed and in fact on an internal network switch there is no physical adapter connected. In a future blog post I’ll detail what each counter means. The most useful ones are;

Bytes/Sec
Packets/Sec

The “Hyper-V Virtual Network Adapter” and the “Hyper-V Legacy Network Adapter” counter sets allow you to see how much ingress and egress a VM is doing. This counter sets are named with the friendly name of the VM plus the name of the network adapter followed by two GUIDs. The GUIDs are the internal id of the VM and adapter which is important when querying via WMI.

There are two counter sets because there are two types of virtual network card you can assign to a VM. If you assign a Legacy Network Adapter then the counter set you should use is the “Hyper-V Legacy Network Adapter”. In general you should not use the network adapter type because it is not enlightened, creates a lot of CPU load in the root, and is generally slower than the Network Adapter. The challenge is you need the Legacy Network Adapter to get a VM working before installing Integration Services. Once your VM is working with Integration Services you should use the Network Adapter and the “Hyper-V Virtual Network Adapter” counter set. Keep in mind Windows Server 2008 and Windows Server 2008 R2 both have integration Services pre-installed.

The “Hyper-V Legacy Network Adapter” counters to monitor are;

Bytes Dropped
Bytes Sent / Sec
Bytes Received / Sec

In the “Hyper-V Virtual Network Adapter” you should monitor;

Bytes / Sec
Packets / Sec

Storage:

The storage counters are useful for monitoring the overall disk performance on the system as well as for each VM. The first counter “Physical Disk set will give overall storage performance on the system. The next two are strictly for VM’s.

Physical Disk
Hyper-V Virtual Storage Device
Hyper-V Virtual IDE Controller

Inside the “Physical Disk” counter set I tend to monitor only three things. The first is “Current Disk Queue Length” which gives one an idea of how busy the drives are. The “Current Disk Queue Length” should be around two per drive. If you have a RAID 10 volume with 4+4 (total of 8 drives). Then a queue length of 16 is reasonable. A queue length of 32 might indicate this disk is saturated and is the bottleneck in the system. The next counter I monitor is the “Disk Bytes / Sec”. I generally expect to see about 10MB/sec per drive which is a fairly safe number for most drives. Some can do better and some worse. So for the RAID 10 4+4 a throughput of around 80MB/sec is reasonable for sequential workloads whereas 10MB/sec is not. For random workloads I look at the last counter “Disk Transfers / Sec” and expect to see about 100 IO’s per second (IOPs) per drive. Once again there are drives that do much better like 180 IOPs and some that do worse like laptop drives which are around 60 IOPS. For the RAID 10 4+4 around 800 IOPs for 8Kbytes reads and writes is reasonable. Generally writes are a bit lower.

There are two Hyper-V storage counter sets because of how storage works in Hyper-V. In Hyper-V we provide two virtual storage buses for VM’s. One is IDE and one is SCSI. The Virtual IDE counters show up in the “Hyper-V Virtual IDE Controller” counter set unless Integration Services are loaded and then you will see the activity for both virtual IDE and SCSI in the “Hyper-V Virtual Storage Device” counter set. If you don’t have integration services installed the only the “Hyper-V Virtual IDE Controller” will show the VM disk activity. If you want to read more on Hyper-V storage check out this link – http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx .

I tend to monitor the all the counters on both the “Hyper-V Virtual IDE Controller” and “Hyper-V Virtual Storage Device” because there is just the basics set.

For the “Hyper-V Virtual IDE Controller” there are;

Read Bytes / Sec
Write Bytes / Sec
Read Sectors / Sec
Write Sectors / Sec

And for “Hyper-V Virtual Storage Device” there are;

Error Count
Flush Count
Read Bytes / Sec
Write Bytes / Sec
Read Count
Write Count

The Error count should always be zero for the virtual storage device.

Conclusion:

There are many counters in Hyper-V which provide useful information and will help you understand what the system is doing. The table below summarizes the counters above in a perfmon like format. Note that you should not always collect all instances in a counter set. Some counter sets have too many counters to collect all instances. If you attempt to collect all counters for all instances you might see periods in the data that are empty. This means the system is not keeping up the amount of counters requested.

In the table below you will see (_Total) or (*). (_Total) means only the total should be collected. (*) means collect all counters. \* means collect all the counters in the counter set. \<name> means only collect that counter. This is the notation perfmon uses in its collection files.

\Hyper-V Virtual Machine Health Summary \*

\Hyper-V Hypervisor\*

\Processor(_Total)\*

\Hyper-V Hypervisor Logical Processor(*)\%Guest Run

\Hyper-V Hypervisor Logical Processor(*)\%Hypervisor Run Time

\Hyper-V Hypervisor Logical Processor(*)\%Idle Run Time

\Hyper-V Hypervisor Logical Processor(*)\%Total Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Guest Run

\Hyper-V Hypervisor Root Virtual Processor (*)\%Hypervisor Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Idle Run Time

\Hyper-V Hypervisor Root Virtual Processor (*)\%Total Run Time

\Hyper-V Hypervisor Virtual Processor (_Total)\*

\Memory\Pages / Sec

\Memory\Available Bytes

\Hyper-V Hypervisor Partition(*)\2G GPA Pages

\Hyper-V Hypervisor Partition(*)\Deposited Pages

\Hyper-V Hypervisor Partition(*)\Virtual Processors

\Hyper-V Hypervisor Root Partition(*)\*

\Hyper-V VM Vid Partition(*)\Physical Pages Allocated

\Hyper-V VM Vid Partition(*)\Remote Physical Pages

\Network Interface(*)\*

\Hyper-V Virtual Switch(*)\*

\Hyper-V Legacy Network Adapter(*)\*

\Hyper-V Virtual Network Adapter(*)\*

\Physical Disk(*)\Current Disk Queue Length

\Physical Disk(*)\Disk Bytes / sec

\Physical Disk(*)\Disk Transfers/sec

\Hyper-V Virtual Storage Device(*)\*

\Hyper-V Virtual IDE Controller(*)\*

SQL Server 2008 on Windows Server 2008 Hyper-V Performance Guidance

tvoellm — Tue, 07 Oct 2008 03:30:00 GMT

Wow this is great day. The SQL team has worked really hard in creating a performance guidance document with solid suggestions and performance numbers when running on WS08 Hyper-V.

They looked at OLAP and OLTP workloads. Even if you are not running SQL it will give you and idea of how other database like workloads are likely to perform.

Check it out...
http://sqlcat.com/whitepapers/archive/2008/10/03/running-sql-server-2008-in-a-hyper-v-environment-best-practices-and-performance-recommendations.aspx

Hyper-V Performance Counters - Part five of many - "Hyper-VM VM Vid Numa Node"

tvoellm — Mon, 29 Sep 2008 06:36:00 GMT

There are a couple of performance counter sets in Hyper-V that show memory usage. These counters tend to show memory used by one component of the system and the "Hyper-V VM Vid Numa Node" counter set is no different.

This counter set is intended to show memory used by the Virtual Infrastructure Driver (Vid). The good news is this counter set shows "guest physical memory". This memory includes the allocation you specify when creating the VM and some other support memory like video memory. It is the majority of the memory used by the Virtual Machine (VM).

Hyper-V VM Vid Numa Node shows memory on a per node basis. If you have an AMD machine you will likely see a Non-uniform Memory Access (NUMA) node for each processor. If you have an Intel based machine you will see anywhere from one to many nodes depending on the architecture.

The following are the performance counters in the counter set;

PageCount - Number of 4K pages that the node contains. To find the total physical memory on the system you would add PageCount from each NUMA node and multiply by 4Kbytes.
PagesInUse - Number of pages in use by the Vid on this node. Keep in mind this is most but now all memory supporting a VM.
ProcessorCount - Number of logical processors this node contains. Logical process are equal to the number of processor (sockets) * number of cores per processor * 2 if you have HyperThreading (HT) or now called Symmetric Multithreading (SMT)

Enjoy,

Anthony F. Voellm (aka Tony)

Looking for that last ounce of performance? Then try affinitizing your VM to a NUMA node

tvoellm — Mon, 29 Sep 2008 00:22:00 GMT

There are not many performance knobs in Hyper-V which is by design. We really seek out of the box performance. However if you are looking for that last bit of performance from your Virtual Machines (VM’s) and have already made a good selection for networking and storage you might consider setting the Non-Uniform Access (NUMA) node.

On AMD and some Intel based architectures the machine might be defined as NUMA. You can find out if the machine is NUMA by looking at the "Hyper-VM VM Vid Numa Node" performance counters. This counter set will report a counter set instance for each NUMA node unless the machine is a single node.

NUMA means each CPU has a different path to memory and those paths can have various lengths. For example CPU 0 on Node 0 when accessing CPU M’s memory on Node X might take 10ns where as CPU 0 on Node 0 accessing CPU N’s memory on Node Y will take 20ns. It’s this difference in memory access times that can impact overall VM performance. The worst case would be for a VM’s Virtual Processor (VP) to be running on a node furthest from where the memory for the VM is allocated.

In order to improve performance you can place VM’s on different nodes. In addition to placing the memory the Hypervisor scheduler will attempt to run the VM’s VP’s near where the memory where the VM is allocated. This creates a dual affinity which can be very beneficial.

If you would like to try placing VM’s on different nodes to improve performance you can use the code sample below to place the VM.

Example 1 – List currently configured VM’s

C:\> .\numa.ps1 /list

Example 2 – Set NUMA affinity to node 1. Node numbering starts at node 0

C:\> numa.ps1 /set tonyvm 1

Example 3 - Clear NUMA affinity

C:\> numa.ps1 /clear tonyvm

Enjoy,

Anthony F. Voellm (aka Tony)

Save the following powershell script into a file named numa.ps1. WS08 has powershell as part of the release if you are running management from a remote machine that is XP / Vista you can download powershell here http://www.microsoft.com/windowsserver2003/technologies/management/powershell/default.mspx

################################################
# Developer: Anthony F. Voellm
# : Taylor Brown
# Copyright (c) 2008 by Microsoft Corporation
# All rights reserved
#
# This is "demonstration" code and there are no
# warrantees expressed or implied
################################################

# This script will set the Virtual Machine to run
# on a specific NUMA node

# Check command line arguments

if (($args.length -lt 1) -or
    (($args[0] -ne "/list") -and
     ($args[0] -ne "/set") -and
     ($args[0] -ne "/clear")) -or
     (($args[0] -eq "/set") -and ($args.length -lt 3)) -or
     (($args[0] -eq "/clear") -and ($args.length -lt 2))) {
     Write-Host "numa.ps1 /list [<Hyper-V host>]"
     Write-Host "numa.ps1 /set <vm machine name> <required node> [<Hyper-V host>]"
     Write-Host "numa.ps1 /clear <vm machine name> [<Hyper-V host>]`n"
     Write-Host "Options:"
    Write-Host "`t/list - show configured VM's"
     Write-Host "`t/set <vm machine name> <required node> - set the NUMA node for the VM"
     Write-Host "`t/clear <vm machine name> - clear NUMA node seting for the VM"
     exit;
}

# just display VM's
if ($args[0] -eq "/list") {
if ($args.length -gt 1) {
$HyperVHost = $args[1];
}
Get-WmiObject -Namespace 'root\virtualization' -Query "Select * From Msvm_ComputerSystem" | select ElementName
exit;
}

# Set or clear

$HyperVHost = '.';
if ($args[0] -eq "/set") {
if ($args.length -gt 3) {
$HyperVHost = $args[3];
}
$VMName = $args[1];
$RequiredNode = $args[2];
} elseif ($args[0] -eq "/clear") {
if ($args.length -gt 2) {
$HyperVHost = $args[2];
}
$VMName = $args[1];
}

#Main Script Body
$VMManagementService = Get-WmiObject -Namespace root\virtualization -Class Msvm_VirtualSystemManagementService -ComputerName $HyperVHost

$Query = "Select * From Msvm_ComputerSystem Where ElementName='" + $VMName + "'"

$SourceVm = Get-WmiObject -Namespace root\virtualization -Query $Query -ComputerName $HyperVHost

$VMSettingData = Get-WmiObject -Namespace root\virtualization -Query "Associators of {$SourceVm} Where ResultClass=Msvm_VirtualSystemSettingData AssocClass=Msvm_SettingsDefineState" -ComputerName $HyperVHost

if ($args[0] -eq "/set") {
$VMSettingData.NumaNodesAreRequired = 1
$VMSettingData.NumaNodeList = @($RequiredNode)
} else {
$VMSettingData.NumaNodesAreRequired = 0
}

$VMManagementService.ModifyVirtualSystem($SourceVm, $VMSettingData.PSBase.GetText(1))

Hyper-V Storage Analysis

tvoellm — Wed, 24 Sep 2008 15:59:00 GMT

Hyper-V Perfies,

In a previous blog entry I explained the different types of storage choices you have with Hyper-V. However we had nore released yet so no numbers could be published. You can find the original post below...

http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx

Now that we have release I wanted to share some numbers from a mid-range storage device.

For the storage performance results below we used;

A dual core 2GHz system
8 SAS disk system configured to run RAID0
Off the shelf SCSI RAID controller
We pre-populated the Virtual Hard Disks used with non-zero data to avoid testing an optimization we have were zero block are not actually written to Dynamic / Diff VHDs if they have never been written before.
IOMeter was used to generate the load and gather results.
Root results were tested using VHD loopback. This means we used the WMI interfaces in Windows Server 2008 to mount the VHD into the root without any virtual machine. This was pretty close (<1% diff) to bare metal performance (no Hyper-V enabled). Check out this link on how to do it http://blogs.msdn.com/virtual_pc_guy/archive/2008/02/01/mounting-a-virtual-hard-disk-with-hyper-v.aspx
SCSI and IDE results were collected from a Windows Server 2008 64bit one virtual processor guest.
Only read performance was collected for fixed Root VHD + 1 Diff Disk. This is why some parts of the graphs are label "NOT TESTED". Reads have to traverse the differencing chain and writes go to the outer most disk so write look just like dynamic disk write performance and were not collected.

We chose this configuration because it was pretty low to mid range, would be very low cost, and therefore within reach of all customers running Hyper-V. If you are running Hyper-V with big servers and storage containing 100's of disks you find it handles it very well with very little overhead. That might be the subject of a future post.

The following graphs show sequential I/O performance of 512Byte reads and writes. We choose this to measure because it would approximate the behavior or loading applications, editing small files, and other types of workloads doing sequential I/O. We saw big improvements with larger I/O's (>4x) but for many workloads those are not that likely to be encountered.

The following graphs show 8KByte random read and write performance. We chose 8KByes reads and writes because many servers such as SQL Server 2008 use this sized I/O for most operations.

From the graphs you can see that Emulated IDE (aka you are not running with Integration Components installed) is never a good choice.

Passthough followed by fixed disks have the overall best performance. Dynamic and differencing disks are good choices when you need flexibility.

Enjoy,

Tony Voellm

Liang Yang

WS08 Hyper-V now supports 24LP

tvoellm — Wed, 24 Sep 2008 15:29:00 GMT

Hyper-V Perfies,

Now available to the general public is Windows Server 2008 (WS08) Hyper-V 24 Logical Processor (LP) Support. This revises our RTM support limit of 16 Logical Processors. We also increased the number of supported running virtual machines from 128 to 192. This update also has the side benefit of improving workloads that start large numbers of processes by increasing the process address space cache limit from 192 to 384. All in all this update will allow you to get even more mileage from WS08 Hyper-V.

You can find the needed update here http://www.microsoft.com/downloads/details.aspx?FamilyID=fe36823a-7e5a-4262-9bf5-d6b3ae3ad375&DisplayLang=en

Here is some Q & A to help you figure out how and when to install this update;

Q: Should everyone install this update?

A: No. Install this update only on machine needing 24LP support or if you are running more than 192 processes in the root (no recommeded) / guest. Keep in mind there are very few workloads that create this many processes and there are only two that come to mind. Terminal Services and Hyper-V running > 192 virtual machines (VM).

Q: Does this mean I can now run 192 VM's on my 16 logical processor system?

A: No. We only support an 8 virtual processor (VP) to 1 logical processor (LP) limit.

Q: What does 8 VP to 1 LP mean?

A: This mean is you have 8 logical processors you can run up to 64 virtual processors in any combinaiton. For example 16 Windows Server 2008 (WS08) VM's with 4 VP each, or 64 Windows Server 2003 VM's with 1 VP each, or 8 WS08 4VP VM's with 32 W2k3 1VP VM's. You get the picture.

Q: What is a logical processor aka LP?

A: A logical processor is what we call anything that shows up in Windows as a processor in task manager. More precisely it is the "number processors * number of cores * 2 if you are running Hyper Threading (HT) or now called Symmetric Multi Threading (SMT)." For example a dual proc quad core system with SMT enabled has 2 * 4 * 2 = 16 LP. A quad proc dual core system has 4 * 2 = 8 LP.

Leading the 24LP effort was a blast and it is also why my blog has been pretty quiet recently.

Enjoy,

Tony Voellm

BizTalk team releases best practice doc for running on WS08 Hyper-V

tvoellm — Tue, 22 Jul 2008 22:18:00 GMT

Below is a great link on how to get BizTalk 2006 working on top of Hyper-V. It includes practical advice that applies to other servers and roles as well.

Check it out... http://msdn.microsoft.com/en-us/library/cc768518.aspx

Tony

How to get Processor Utilization for Hyper-V via WMI

tvoellm — Mon, 14 Jul 2008 19:44:00 GMT

There are a number of groups building management software (OEMs, Microsoft, …) for Hyper-V which is cool to see. A common ask from these teams has been around reading and computing VM CPU usage.

The following is an example of how to compute Hyper-V guest processors usage. You can use the same formula for “% Total Run Time, “% Hypervisor Time” and “% Idle time”. The counters show up in the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object as “PercentGuestRunTime”, “PercentTotalRunTime”, “PercentHypervisorRunTime”, and “PercentIdleTime”.

To make the formula easier to read lets use:

G_N – Percent Guest Run Time (substitute other usage values here)

P_N – Timestamp_PerfTime

F_N – Frequency_PerfTime

LP = Number of logical processors (Get this from the “Hyper-V Hypervisor” counterset)

F₁ * (G₂ – G₁)

Utilization = -----------------------------------

100000 * LP * (P₂-P₁)

G₂ and P₂ are the second values read and G₁ and P₁ are the first values read.

To test the formula lets read the “Hyper-V Hypervisor Logical Process” counterset twice via the Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor WMI object about 10 seconds apart with a single VM running at 100% Guest CPU. Since my test machine has two CPU’s (2 LP) this means we should see about 50% overall utilization.

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

C1TransitionsPersec = 409197889

C2TransitionsPersec = 0

C3TransitionsPersec = 0

Caption = null

ContextSwitchesPersec = 889911109

Description = null

Frequency_Object = 0

Frequency_PerfTime = 14318180

Frequency_Sys100NS = 10000000

HardwareInterruptsPersec = 92282462

InterProcessorInterruptsPersec = 8174254

InterProcessorInterruptsSentPersec = 8174254

MonitorTransitionCost = 16

Name = _Total

PercentC1Time = 4193635539355

PercentC2Time = 0

PercentC3Time = 0

PercentGuestRunTime = 314976793671

PercentHypervisorRunTime = 53745475789

PercentIdleTime = 8385447570540

PercentTotalRunTime = 368722269460

SchedulerInterruptsPersec = 384836664

TimerInterruptsPersec = 33425466

Timestamp_Object = 0

Timestamp_PerfTime = 6268633722843

Timestamp_Sys100NS = 4199325031975

TotalInterruptsPersec = 518718846

V:\backup>winrm enum wmi/root/cimv2/* -filter:"select * from Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor where name='_Total'"

Win32_PerfRawData_HvStats_HyperVHypervisorLogicalProcessor

C1TransitionsPersec = 409201218

C2TransitionsPersec = 0

C3TransitionsPersec = 0

Caption = null

ContextSwitchesPersec = 889922035

Description = null

Frequency_Object = 0

Frequency_PerfTime = 14318180

Frequency_Sys100NS = 10000000

HardwareInterruptsPersec = 92283571

InterProcessorInterruptsPersec = 8174425

InterProcessorInterruptsSentPersec = 8174425

MonitorTransitionCost = 16

Name = _Total

PercentC1Time = 4193667417779

PercentC2Time = 0

PercentC3Time = 0

PercentGuestRunTime = 315044817737

PercentHypervisorRunTime = 53746578312

PercentIdleTime = 8385511363951

PercentTotalRunTime = 368791396049

SchedulerInterruptsPersec = 384840537

TimerInterruptsPersec = 33426627

Timestamp_Object = 0

Timestamp_PerfTime = 6268728855292

Timestamp_Sys100NS = 4199364353043

TotalInterruptsPersec = 518725160

Based the formula above and the data below we get = 51% which is spot on.

Enjoy,

Tony Voellm

WS08 Hyper-V as RTMed!!!!

tvoellm — Sat, 28 Jun 2008 18:57:00 GMT

Wow its been a great ride helping get Hyper-V to perform the way it does. So much so I've not posted in a while. You will see some upcoming posts on performance counters, WMI perf interfaces, and real data (yes now with RTM I can post data).

In the meantime check out my co-workers post on where to get the final (aka RTM) WS08 Hyper-V bits...

http://blogs.technet.com/jhoward/archive/2008/06/26/hyper-v-rtm-announcement-available-today-from-the-microsoft-download-centre.aspx

Tony

Windows Server Performance 2008 Tuning guide now includes Hyper-V

tvoellm — Wed, 18 Jun 2008 21:58:00 GMT

The Windows Server Performance team has updated teh Window Server 2008 Tuning guide to include Hyper-V. The Hyper-V section provide a lot of recommendations you will find useful.

Check it out below...

http://blogs.technet.com/winserverperformance/archive/2008/06/17/power-and-hyper-v-are-now-part-of-the-windows-server-2008-tuning-guide.aspx

Why can't I start my VM when there is plenty of free memory?

tvoellm — Wed, 11 Jun 2008 12:46:00 GMT

Issue #1: A number of customers has reported than VM's fail to start when there is plently of memory. The error reported from the Hyper-V manager is displayed below. Here I tried to start a 4GB VM when there was 7+GB free. What's going on?

The most common cause has to do with a bug in Windows Server 2008 NUMA memory allocations. This means you'll only hit this bug on NUMA machines (all multi-proc AMD machines are NUMA + many high end Intel based servers). You can verify you have this bug by looking at the Task Manager reported "Cached" and "Free" memory.

Here free memory is 9MB but really much of the "Cached" memory can be made free also. Standby Pages, Modified Pages, and System File Cache make up the "Cached" memory. Most (all but a couple 100MB) of the cached memory can be converted to "free",

Fix #1: To resolve the issue will need to contact Microsoft Product Support http://support.microsoft.com/contactus/cu_sc_more_master#tab1 and request the following hotfix - KB953585

Alternate Fix #1: The second way to avoid this bug is to not do any work in the root parition directly. Create a "management" VM and do your work there.

Background #1: The most common way of hitting this issue is by running large file copies in the root (aka host) parition. Doing lots of file copies causes the System File Cache to bloat in size. When Hyper-V goes to start it will flush the System File Cache if it needs memory. The System File Cache pages (aka file pages from the copies) move to the Standy list and from there will get zeroed and freed. The bug is WS08 prevents NUMA allocations from the Standyby list.

Issue #2: The second most common cause to the can't start the VM issue when it appears there is plenty of memory is due to configuring a NUMA machine to use fewer processors than the system actually has. Typically this is done using "bcdedit /set numproc X".

Fix #2: If you set NUMA machine in this mode for testing memory attached to the "hidden" processors will be inaccessible to Hyper-V. Some machines allow you to reassign the memory (via the BIOS or front end controller) to the "active" processors.

- Tony Voellm

Hyper-V Performance FAQ

tvoellm — Fri, 06 Jun 2008 03:26:00 GMT

Hyper-V Performance FAQ

Anthony F Voellm (aka Tony)

6/19/2008

http://blogs.msdn.com/tvoellm

[**** NOTE THIS IS NOW OLD. ****]

[**** SEE THE NEW FAQ at ****]

[**** http://blogs.msdn.com/tvoellm/archive/2009/12/18/hyper-v-performance-faq-r2.aspx ****]

Q: What is the recommend configuration for performance testing?

A: Here are some simple steps:

1. Be sure to have the latest WS08 Hyper-V build – Hyper-V RC1 which is on Microsoft Downloads and Windows Update

2. Next you need to make sure you are running a “Supported OS” with the latest SP.

Operating System	Virtual Processor Limit
Windows Server 2008 64-bit	4
Windows Server 2003 32-bit	2
Windows Server 2008 32-bit	4
Windows Server 2003 64-bit	1
Windows Vista SP1 32-bit / 64bit	1
Windows XP SP3 32-bit / 64bit	1
Windows 2000 32bit	1
SUSE/RedHat LINUX	1

4. Make sure you are using the “Network Adapter” and not the “Legacy Network Adapter”. The legacy adapter has a lot of emulation which causes lots of CPU overhead.

5. Use passthrough disks attached to SCSI for the next performance. Next best is Fixed VHD attached to SCSI. To understand storage better see (http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx )

6. Follow these tips for avoiding pitfalls http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx

Q: How do I monitor performance?

A: First you need to understand that the clocks in the root and guest Virtual Machines may not be accurate see (http://blogs.msdn.com/tvoellm/archive/2008/03/20/hyper-v-clocks-lie.aspx ). Given an understanding of clocks you can see why we implemented the “Hyper-V Hypervisor Logical Processor” performance counters (access using perfmon) which are not skewed by clock effects. There are other Hyper-V performance counters that are useful. See the following for more details (http://blogs.msdn.com/tvoellm/archive/tags/Hyper-V+Performance+Counters/default.aspx )

Q: Should I use passthrough or iSCSI attached to the guest for storage?

A: The decision depends on what features you need to expose to the guest. In the passthrough case the drive will show up without knowledge of the underlying LUN.

Educated guess: If you are looking for raw performance passthough will give you the best result.

Q: Is there a simple way to disable the hypervisor to run some baseline tests on the native system.

A: Yes. “bcdedit /set hypervisorlaunchtype off” and reboot the server. You should also consider changing the protocols on the root network device to re-enable IP and turn off the “Microsoft Virtual Network Switch Protocol”. See the following for more details (XXX – coming to my blog soon J). For starters turning off the Hypervisor should be enough for testing native performance.

To turn it back on do “bcdedit /set hypervisorlaunchtype on” and reboot the server.

Q: Is there any common terminology used to talk about virtual machine configurations?

A: Yes - Internal to Microsoft we typically use the following;

· Native = System without the Hyper-V role. This means you have no virtual drivers, virtual switch, …

· Root = you have Hyper-V role installed and but are not using a virtual switch for networking.

· Guest = Guest Virtual Machine.

· 8p.child.2x1p or better 8p.child.2VMx1VP = A system with 8 logical processors / cores running 2 Virtual Machines (VM) each with 1 Virtual Processor (VP)

Q: Are there any services that should be stopped?

A: No If you are running Server Core which is the ideal root OS to use. Regardless of running server core vs server you should close the Hyper-V Management Console because it has a noticeable impact on CPU. If you want the details see http://blogs.msdn.com/tvoellm/archive/2008/04/19/hyper-v-how-to-make-sure-you-are-getting-the-best-performance-when-doing-performance-comparisons.aspx

Q: Is it ok run applications / processes in the root OS?

A: You should avoid running any Role / Feature or custom service in the root. If you have services you want to run put them in a guest VM. Running roles in the root can have a negative impact the guest VM’s. This is due to how the Hypervisor scheduler handles the root virtual processors.

Q: Are there additional knobs for performance nuts?

A: We are trying to make Hyper-V knob-less. However we are engineers and here are some tips.

1. Remove the CDROM drive from the guest if you don’t need it

2. Look into the Caps / Weights / Reserves in the CPU config. You can use these to “balance” workloads.

3. You can use the WMI interfaces to force a VM to a particular node (coming to my blog soon J). You don’t guarantee node affinity for VP’s but we do for memory. There is a good chance the VP’s will stay on the node because the scheduler is NUMA aware.

Q: Are their addition resources that are useful for understanding Hyper-V?

A: Yes – here is a list

• http://blogs.technet.com/windowsserver/default.aspx

• http://blogs.msdn.com/virtual_pc_guy/

• http://blogs.technet.com/jhoward/

• http://blogs.msdn.com/tvoellm

• http://blogs.technet.com/winserverperformance/

Negative ping times in Windows VM's - whats up?

tvoellm — Thu, 05 Jun 2008 23:40:00 GMT

Just a quick blog post that might help you resolve an issue that some customers have seen running under Hyper-V VM's. The issue is negative ping times on multi-processor guests.

If you see negative ping times in multiprocessor W2k3 guest OSes you might consider setting the /usepmtimer in the boot.ini file.

The root issue comes about from the Win32 QueryPerformanceCounter function. By default it uses a time source called the TSC. This is a CPU time source that essentially counts CPU cycles. The TSC for each (virtual) processor can be different so there is no guarantee that reading TSC on one processor has anything to do with reading TSC on another processor. This means back to back reads of TSC on different VP's can actually go backwards. Hyper-V guarantees that TSC will not go backwards on a single VP.

So here the problem with negative ping times is the time source is using QueryPerformanceCounter which is using TSC. By using the /usepmtimer boot.ini flag you change the time source for QueryPerformanceCounter from TSC to the PM timer which is a global time source.

- Tony Voellm