All topics fundamental : viridian

WS08 Hyper-V as RTMed!!!!

tvoellm — Sat, 28 Jun 2008 18:57:00 GMT

Wow its been a great ride helping get Hyper-V to perform the way it does. So much so I've not posted in a while. You will see some upcoming posts on performance counters, WMI perf interfaces, and real data (yes now with RTM I can post data).

In the meantime check out my co-workers post on where to get the final (aka RTM) WS08 Hyper-V bits...

http://blogs.technet.com/jhoward/archive/2008/06/26/hyper-v-rtm-announcement-available-today-from-the-microsoft-download-centre.aspx

Tony

Negative ping times in Windows VM's - whats up?

tvoellm — Thu, 05 Jun 2008 23:40:00 GMT

Just a quick blog post that might help you resolve an issue that some customers have seen running under Hyper-V VM's. The issue is negative ping times on multi-processor guests.

If you see negative ping times in multiprocessor W2k3 guest OSes you might consider setting the /usepmtimer in the boot.ini file.

The root issue comes about from the Win32 QueryPerformanceCounter function. By default it uses a time source called the TSC. This is a CPU time source that essentially counts CPU cycles. The TSC for each (virtual) processor can be different so there is no guarantee that reading TSC on one processor has anything to do with reading TSC on another processor. This means back to back reads of TSC on different VP's can actually go backwards. Hyper-V guarantees that TSC will not go backwards on a single VP.

So here the problem with negative ping times is the time source is using QueryPerformanceCounter which is using TSC. By using the /usepmtimer boot.ini flag you change the time source for QueryPerformanceCounter from TSC to the PM timer which is a global time source.

- Tony Voellm

Hyper-V Performance Counters – Part one of many - The overview

tvoellm — Sun, 04 May 2008 08:05:00 GMT

[[ ***IF YOU HAVE SPECIFIC QUESTIONS ON PERFORMANCE COUNTERS YOU WANT ANSWERED PLEASE POST A REPLY WITH THE QUESTION***]]

In the next few posts I’ll be covering Hyper-V performance counters. There have been a lot of changes from Beta -> RC -> post RC.

In Beta the counters sets for Hyper-V were not grouped under a common namespace. This made them hard to find. In RC the counters are grouped under the “Hyper-V” namespace. That is each counter set starts with “Hyper-V.” For example “Hypervisor Logical Processors” are now “Hyper-V Hypervisor Logical Processors.” The prefix causes all the Hyper-V counters to be grouped together.

Post RC you will see some counter sets get renamed and a few like “Hyper-V VMMS Task Manager Summary" have been removed because they did not provide a lot of value. The most notable renaming was “Hyper-V Hypervisor …” rate counters where “/ sec” has been appended to to make it clear they are a rate. You will also see some counters have been removed.

The goal of the changes has been to simplify the counters and provide a very useful core that are easy to find. Improvements have also been made to make the counters more accurate.

The “Performance Counter” blog entries will focus on what you will see post RC but it should not be too hard to apply what is discussed to RC or Beta.

Hyper-V will RTM with 24 counter sets which are listed below by expected target audience;

Admin / Dev / Product Support (but not all counters in the counter set are for everyone)

· Hyper-V Hypervisor - Overall hypervisor & machine state

· Hyper-V Hypervisor Logical Processor - Details on what logical processors are doing

· Hyper-V Hypervisor Partition - Each guest runs in a partition. This has details on what the partition memory and processors

· Hyper-V Hypervisor Root Partition - Details on the root partition memory and processors.

· Hyper-V Hypervisor Root Virtual Processor - Details on what the root Virtual Processors are doing. There is one root VP for every Logical Processor (LP). Think of LP as cores on a processor.

· Hyper-V Hypervisor Virtual Processor - Details on guest VP’s

· Hyper-V Legacy Network Adapter - Emulated network controller statistics and rates of flows.

· Hyper-V Virtual Network Adapter - Virtualized network controller statistics. This is the faster networking device.

· Hyper-V Virtual Switch - A virtual switch can attached to one and only one physical NIC. Each Virtual / Legacy NIC plugs into a virtual switch. This counter set gives details on what the switch is doing and flows of sends / receives.

· Hyper-V Virtual Switch Port - Similar to the virtual switch stats but details just a single “port” aka attached virtual NIC.

· Hyper-V Virtual IDE Controller - Details on disk commands and rates.

· Hyper-V Virtual Storage Device - Details on virtual storage reads / writes like SCSI disks.

· Hyper-V Virtual Machine Health Summary - Toggle on Health Critical & Health OK

· Hyper-V Virtual Machine Summary - State information on the number of VM’s starting, stopping, resuming, … Gives good insight into what system wide control functions are running.

Developers / Product Support – ADVANCED

Think of VM below as Virtual Motherboard. This is not quite true but it will help you understand what much of the functionality is targeting.

· Hyper-V Task Manager Detail - Gives detail and timing on changes to virtual machines like import, export, …

· Hyper-V Virtual Machine Bus - Interrupt rates, … on VMBUS

· Hyper-V VM IO APIC - Details on the Virtual IO APIC

· Hyper-V VM Vid Driver - The VID control memory dedicated to VM’s. This give lots of details on the types of memory used, …

· Hyper-V VM Vid Message Queue - More internal details on the VID

· Hyper-V VM Vid Numa Node - Details on how the VID is managing NUMA memory

· Hyper-V VM Vid Partition - Similar to the “Hyper-V Hypervisor Partition States” but relative to the root’s view on the partitions.

Present but non-functioning counter sets – these make be removed in the future;

· Hyper-V VM Remoting - Tracks the number of pixels written to the virtual frame buffer, …

· Hyper-V VM Save, Snapshot, and Restore - Timing on how long save, restore, … take

· Hyper-V VM Worker Process Memory Management - Details on the worker processes view on memory. The worker process is where emulation for emulated devices happen, save / restore, …

The next post will cover the most useful set of counters to monitor.

[[ ***IF YOU HAVE SPECIFIC QUESTIONS ON PERFORMANCE COUNTERS YOU WANT ANSWERED PLEASE POST A REPLY WITH THE QUESTION***]]

- Tony Voellm

Hyper-V: Clocks lie... which performance counters can you trust?

tvoellm — Thu, 20 Mar 2008 14:00:00 GMT

Many people have asked me how to do performance analysis using Hyper-V. The first thing I always tell them is clocks lie. Lets dig into this a little bit.

First some basic concepts. There are two types of partitions that the Hypervisor manages. The root partition which is special (manages physical devices, child partitions, … all except CPU and physical memory access which is the Hypervisors job) and the child partitions. The child partitions are where the “Guest” virtual machines run. The root partition has the “Root OS” or sometimes called “Host OS” (not technically correct in Hypervisor architectures).

Neither the Root nor the Guests VM’s control the PM Timer, APIC, … which means their concept of time in virtualized. Something else to be aware of is both the Root and Guest virtual processors share the physical processors in a Round Robin like fashion.

Now let’s take a very important Windows Performance Counter - % Processor Time. This performance counters details the total amount of CPU being used by a process, … (aka the CPU is not idling waiting for work). There are other % Processor Time counters for that detail process time on a CPU like Internet Explorer running.

Now for the lie. Check the picture below. On the guest I am running a CPU spinner program to make it 100% busy. You can see perfmon running in the guest shows 100% utilization and the root shows ~85% utilization using the Hyper-V Hypervisor Guest Run Time counters.

So which is correct? Actually they both are. The guest is using 100% of the CPU it has been given by the Hypervisor and each time the guest it running the CPU is busy running the spinner code. Thus the guest is using 100% or a Virtual CPU.

The Hyper-V Hypervisor Logical Processor counters are showing Logical (aka Physical) Processor utilization. This is what you are used to % Processor Time actually showing.

So if you are doing performance analysis and using performance counters be aware the counters in the Guest Virtual Machine “lie” so to speak. What you need to use are the Hyper-V Hypervisor Performance Counters in the Root to get Physical Processor usage.

Good overview of virtualization

tvoellm — Tue, 18 Mar 2008 20:06:00 GMT

Every now and then I come across articles on virtualization. This one caught my attention because of the amount of technical details. Detail is what I strive for in this blog and I have some good detailed posts on the way.

http://it.anandtech.com/IT/showdoc.aspx?i=3263

The coming Windows Hyper-V solution is Hypervisor based and has a mix of emulation and paravitualization. In fact we publish our paravirtualization calls in the Hypervisor TLFS that you can download here - http://www.microsoft.com/downloads/info.aspx?na=22&p=1&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=&u=%2fdownloads%2fdetails.aspx%3fFamilyID%3d91e2e518-c62c-4ff2-8e50-3a37ea4100f5%26DisplayLang%3den . This is currently the latest. If you want to search for updates just look for TLFS on http://www.microsoft.com/downloads

Hope you enjoy it.

Tony

Hyper-V: SCSI vs IDE - Do you really need an IDE and SCSI drive for best performance?

tvoellm — Thu, 03 Jan 2008 02:27:00 GMT

Wow... blogging is fun and can really help people understand a topic. Other times it can be a chore when the whole picture is not presented up front. This is just what I did in a previous post http://blogs.msdn.com/tvoellm/archive/2007/12/12/which-is-better-ide-or-scsi-windows-server-virtualization-08-code-name-viridian-controller-performance.aspx. In this post I talked about emulated IDE vs Sythetic SCSI which is really not a fair comparison. I also left out the IDE Filter driver that should have the performance of synthetic SCSI devices.

Here are some blog postings that help clarify a few points;

· Integration Components and Enlightenments

· Boot from SCSI in Virtual Server vs. Boot from IDE in Windows Server virtualization by John Howard

So on the question – Do you really need an IDE and SCSI drive for best performance?

The question / statement was posted in the following blog for which I really appreciate the point of view of the author http://www.virtualization.info/2007/12/hyper-v-will-not-boot-virtual-scsi.html.

Just like all things in performance - it depends.

First it is generally good practice to separate the “OS partition / disk” from “Data & Workload partitions / disks”. So in that respect yes two disks is a good idea. Once you have committed to having two disks my choice would be to use SCSI for the second drive. This is mostly following the premise of using synthetic devices whenever available.

If you are running workloads where you don’t mind sharing the drive the operating system a single drive will be just fine since the IDE filter driver (you get this by installing the Integration Components) is designed to give good performance. You might also want to use a single drive for portability.

If you decide to use two drives you can try both the IDE filter driver and Synthetic SCSI device to see which works best for your workloads. They should be virtually the same (no pun intended). Switching is easy since it just a matter of reconfiguring the Virtual Machine. The same VHD can be used (just not at the same time – besides it would be weird to dual port of a drive into a VM on IDE and SCSI).

Performance rule of thumb: Don’t start tweaking knobs until you know your goals and find there is a problem.

Keep in mind there are only 2 IDE controllers with 2 devices each. One device will be your boot drive and one might be your CDROM. If you follow the standard configuration (1 boot disk + 1 CDROM) and want more than two additional drives you will have to use SCSI.

[[ Since Hyper-V is not RTM'ed all numbers are subject to change as are the behaviors. ]]

Hyper-V: Integration Components and Enlightenments

tvoellm — Thu, 03 Jan 2008 02:09:00 GMT

In order to get the most from your VM’s it is good to understand a couple of concepts that can impact the performance and operation of your Virtual Machines. Here are some the concepts I’ll describe below;

· Integration Components

· Synthetic vs. Emulated Devices

· Operating System Enlightenments

Integration Components:

Integrations components (ICs) are sets of drivers and services that help your Virtual Machines have more consistent state and perform better by enabling the guest to use synthetic devices. Some ICs that come with Hyper-V are VMBUS (transport for Synthetic devices), Time Sync (used to keep VM clocks in sync with the root partition sometimes called the host), Video Driver, Network Driver, Storage Driver, …

You can tell you have these components installed by looking for VMBUS driver and “Virtual “ & “VMBus” in device names in the device manager. If a SCSI drive is attached you will also see the “StorVSC” driver. Since this is Beta we may update any or all of the naming.

Windows Server 2008 will come with the integration components pre-installed. For other operating systems you can use the “vmguest.iso” image in the %SYSTEMROOT%\system32 directory. A simpler way to install ICs is to use the vmconnect (console gui) and select Action->Insert Integration Services Setup Disk. You may need to kick off setup manually depending on how autorun is configured – just inserting the CD doesn’t guarantee the ICs are installed.

Synthetic vs. Emulated Devices:

Synthetic devices are new with Hyper-V and are designed to have the lowest overhead for devices. These devices package requests and forward them to the StorVSP driver in the root over VMBUS which then forwards them to the device after any needed processing. Emulated devices on the other hand emulate a real piece of hardware and emulation happens in the vmwp.exe process (one per VM).

Both types of devices are important and here are some reasons why. The most important reason for having emulated devices is to support operating systems that do not have “Integration Components” installed. The emulated devices can also be thought of a boot strapping devices since the preference should be to use the virtual devices (keep in mind the boot IDE drive uses a filter driver after boot for improved performance if ICs are installed). The Synthetic devices are important because they help reduce the CPU overhead when accessing a device.

To see what OS’s we’ve thought about for synthetic devices see the following blog http://blogs.technet.com/windowsserver/archive/2007/12/12/Yes_2C00_-Virginia_2C00_-there-is-a-Hyper_2D00_V-beta.aspx

Operating System Enlightenments:

Enlightenments are enhancements made to the operating system to help reduce the cost of certain operating system functions like memory management. One example of an OS enlightenment that comes with Hyper-V is TLB flushing. The TLB helps to protect memory and improve CPU hardware memory access performance by caching Virtual Address to Physical Address translations. Here is a nice deck on the Intel site with some slides on Guest Virtual Memory translations http://softwarecommunity.intel.com/isn/downloads/virtualization.pdf

The TLB enlightenment allows the guest OS to notify the Hyper-V hypervisor that a set of TLB entries need to be flushed rather than calling the x86 INVLPG instruction (a mov to CR3 causes a flush entire). This is really important for two reasons. The first is the TLB for the Virtual Machine is significantly larger than a real “TLB” because it helps performance and VM’s are not limited by physical die constraints like power and cooling that impact the size of real TLB’s. The second is each time the INVLPG instructions is called by the guest it will cause a Virtual Machine exit which is more expensive than a kernel context switch.

The enlightenments are accessed via Hypercalls to the hypervisor. If you want to know more about Hypercalls you can see the Beta documentation here - http://www.microsoft.com/downloads/details.aspx?FamilyID=91e2e518-c62c-4ff2-8e50-3a37ea4100f5&DisplayLang=en

Windows Server 2008 is fully enlightened which means it takes advantage of all possible enlightenments. Other operating systems have varying degree of support.

Summary:

The net on performance is to be sure you are running with Integration Components installed and are using Synthetic devices whenever available. You should also use the latest patches (like W2K3 SP2) & OS (Windows Server 2008) to take advantage of enlightenments.

[[ Since Hyper-V is not RTM'ed all numbers are subject to change as are the behaviors. ]]

Hyper-V: How to get the most from your virtualized disk performance

tvoellm — Wed, 12 Dec 2007 09:53:00 GMT

There are two types of disk controllers that Hyper-V supports: SCSI and IDE. There are two IDE controllers and four SCSI controllers available.

Each IDE controller can have two devices. You can not boot from a SCSI controller. This means an IDE disk will be required. The boot disk will be IDE controller 0 Device 0. If you want a CDROM it will consume an IDE device slot.

EACH SCSI controller can support up to 255 devices.

Both SCSI and IDE can support pass-through, Fixed, Dynamic, Sparse, and Delta drives (see http://blogs.msdn.com/tvoellm/archive/2007/10/13/what-windows-server-virtualization-aka-viridian-storage-is-best-for-you.aspx). The difference lies in how the controllers are actually implemented. The IDE controller is emulated where as the SCSI controller is synthetic. So what does this mean?

The IDE controller implements a well-known IDE controller and this means there is extra processing before the I/O is sent to the disk. This processing occurs in vmwp.exe (a user mode process that exists for each started VM. More on this in a later post). Once the IDE emulation is complete the I/O is sent into the Root Partition’s I/O Stack. I/O completion requires a trip back to vmwp.exe.

The SCSI controller is not emulated. The SCSI controller uses VMBUS (Virtual Machine BUS. More on this in a later post). The I/O's pass from the Child (aka Guest) Partition to the root over VMBUS and enter the I/O stack. You can already see one less process/machine context switch is required because vmwp.exe does not get invoked. Once and I/O completes its completion is sent over VMBUS.

There is a lot more to how both the IDE and SCSI controllers work however the descriptions below should help you to understand why SCSI controllers are the right choice for the best performance.

[[ Since Hyper-V is not RTM'ed all numbers are subject to change as are the behaviors. ]]

Windows Hyper-V Beta what's that ??? (Updated 12/13/2007)

tvoellm — Thu, 06 Dec 2007 21:21:00 GMT

Windows Hyper-V was previously called "Windows Server Virtualization" and shipped in the Windows Beta as a prerelease. Now Hyper-V Beta (different name same cool product) is shipping with the latest Windows RC bits. Get it here http://www.microsoft.com/downloads/details.aspx?FamilyId=8F22F69E-D1AF-49F0-8236-2B742B354919&displaylang=en). Also see the press release here http://blogs.technet.com/windowsserver/archive/2007/12/12/Yes_2C00_-Virginia_2C00_-there-is-a-Hyper_2D00_V-beta.aspx.

So what is Hyper-V? Simply put Hyper-V is Microsoft's next generation virtual machines product. It replaces Microsoft Virtual Server and has a whole new architecture along with lots of interesting enhancements. In the redesign from Virtual Server Hyper-V became a hypervisor based product rather than use ring compression (rings are isolation levels used by operating systems to prevent apps and the kernel from interacting without tight controls). The goal of moving to a hypervisor based solution is improved security and performance.

Some of the interesting enhancements to name just a few are;

X64 Support (both root and guests)
10 level deep snapshots
Pass-through drives
Supports Multiprocessors guests

As the person responsible for performance inside the Hyper-V product team I'm confident is saying you'll see the hypervisor investment is really paying off. As you use the product I'd love to get your feedback on Hyper-V performance. Please post a comment.

Which Windows Server Virtualization 08 (code named Viridian) storage is best for you?

tvoellm — Sat, 13 Oct 2007 06:10:00 GMT

If you have had a chance to play around with our prerelease bits you might be wondering what all the types of storage are. You might also be wondering how to decide which storage is right for your virtual machine (VM). One question I'll address in a future blog posting is storage performance. Its a little early to talk performance with prerelease bits but rest assured we are working hard to drive much better performance.

So what type of storage is there?

(dynamic / Fixed) Virtual Hard Drive (VHD) - This is a file stored in NTFS that looks like hard drive to the VM
Delta VHD - This stores all changes relative to a base VHD.
Pass-through drives - Pass-through drive are physical disks are exclusively assigned to the VM.

There is also another dimension to consider

Synthetic Devices - These are new with Viridian. These devices communicate over VMBUS (future post) with the root (sometimes called the host) partition. These devices have an advantage over emulated devices because they better leverage the existing Windows driver architecture.
Emulated Devices - These devices try to emulate well know devices like disk, video, etc. These are less performant than Synthetic Devices because of their emulation.

An overview of pros and cons…

Storage Type	Pros	Cons
Dynamic VHD	Don’t have to decide the size of the disk upfront	Not as performant as Fixed VHD
Fixed VHD	Best performance for file based hard drives	Fills up
Delta VHD	Great for deployment of base golden image	Read might have to access delta and fixed drive + more file overhead.
Pass-through	Attach a physical drive / LUN to a VM	Does not support Virtual Machine Snapshots + no easy as easy to move as a VHD. This is great for direct attached storage however. Tricky configuration. You have to take the drive offline in the root partition before it can be configured to a VM. Use Disk Manager (diskmgmt.msc), diskpart.exe, …

So which storage is best for you? You’ll like this answer – It depends J

We expect Fixed VHD and Pass-through to be the most common in deployment because of their predictable nature.

[[Since this is a CTP all these behaviors might change and there are no promises or warrantees it will continue to work this way]]

What you need to know about Windows Server Virtualization O8 (code named Viridian) Snapshot Performance

tvoellm — Fri, 28 Sep 2007 06:31:00 GMT

First let my say how happy I am to have real bits in your hands. The next generation virtualization product from Microsoft has just shipped with the Windows Server 2008 RC0 (not the final product but getting close). Check out this blog for some external to Microsoft perspective http://www.eweek.com/article2/0,1895,2189866,00.asp

What are virtual machine snapshots?

A snapshot refers to the process of saving a "frozen in time” image of a virtual machine (VM). This means the current contents of the VM memory, disk, ... are saved as they appear the instant you request the snapshot (well almost). This is a really cool feature and in a way is like transaction memory (different topic go "live" it -- hey I work for Microsoft J ) where you can go backwards in time to a snapshot and the machine is as it was.

What you need to know – Gotcha 1 – Slightly less performance

One gotcha with snapshots is when you click the snapshot button two things happen (there are also WMI interfaces if you want to use those). First the contents of memory are saved in a compressed format to disk and a "delta" drive is made on your existing Virtual Hard Disk (VHD). On a 1GB main memory VM this took me less than 2 - 6 seconds on a pretty snappy box with a single SATA drive. The gotcha is now all disk access essential hits two drives for reads. When a read happens first the VM looks into the AVHD file and then if not present into the VHD file. Writes just go to the AVHD file.

What you need to know – Gotcha 2 – Regaining performance but when?

Having an extra hard disk to check for a read is pretty interesting but what happens when the snapshots are removed? Ideally it would be nice to get rid of all the intermediate AVHD files (one for each snapshot on a path to the root VHD. Snapshots can happen on snapshots which creates a tree of points in time) to improve performance – no more multi-file lookups + accounting overhead. Well the process of getting rid of the intermediate AVHD is called merging and this is what happens. Each AVHD is merged into its parent. What's interesting is this merging is asynchronous and today there is no merge now button. Merging happens when the systems is idle to optimize for throughput. This means if you machine is forever loaded the merge may never happen and you are stuck in a sub-optimal mode.

Fortunately there is a way to make a merge happen immediately if not a little twisted. To get an immediate synchronous merge you can delete the VM and then recreate it with the left over VHD. The level over VHD has all the children from the VM merged in.

Keep in mind there is only one VM for a whole snapshot tree. The VM is targeted to a point in the tree and is that path to the root of the snapshots. This when a delete happens you have a well defined machine / hard disk.

What you need to know – Gotcha 3 – No snapshots on passthrough drives

Another cool feature of Viridian is the ability to have passthrough drives. This allows a virtual machine to use a physical disk rather than use a VHD. This should result in better performance. The gotcha is virtual machines using passthough drives don't have a VHD so there is not way to create an AVHD delta file (at least we dont today) which means no snapshots!

Hope these tips help and if you try this out let me know what you find.

Enjoy –Tony (Windows Viridian / Hypervisor Team)

[[Since this is a CTP all these behaviors might change and there are no promises or warrantees it will continue to work this way]]