-- Ben Armstrong, Virtualization Program Manager
Talking about core virtualization at Microsoft (Hyper-V, Virtual PC and Virtual Server).
Most people I know who have spent a lot of time with Hyper-V have had the experience of accidentally taking too much memory away from the parent partition. This happens when they start too many virtual machines – and all of a sudden the performance and responsiveness of the parent partition goes down significantly.
The response from people who hit this is usually to stop the last virtual machine that they started, to reduce its memory, and then start it up again.
This solution has worked in the past – but is no longer an option with dynamic memory. The reason why this will not work is because if you stop the last virtual machine – you just leave the memory available for your other virtual machines to use – getting you straight back where you started.
For this reason, with dynamic memory we have implemented a new “parent memory reserve” for Hyper-V. Here we attempt to calculate an appropriate amount memory to keep for the parent partition and ensure that virtual machines with dynamic memory enabled cannot eat into this reserved memory.
That said – this reserve is not perfect. Specifically – it only accounts for the memory requirements of Hyper-V in the parent partition. If you are running other workloads in the parent partition (contrary to our best practice guidance) then you may still see parent partition memory starvation.
To help mitigate this issue – we are also providing a new registry entry that lets you override our parent memory reserve with your own static memory reserve. This registry entry does not exist by default – but if you go to the HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization registry key and create a new DWORD entry with a name of memoryreserve – you can then set the value to the static amount of memory that you want to reserve for the parent.
You should be warned – if you set this value too low; virtual machines will be able to use too much memory and cause performance issues for you. Equally – the higher you set this the fewer virtual machines you can run.
That said – I have found this to be very useful on my laptop – where I use Hyper-V like a desktop operating system. By setting this value to 2048 (on my laptop with 8GB of RAM) I can run multiple virtual machines and know that they will not cause Outlook / Internet Explorer in the parent partition to be affected.
Cheers, Ben
Jeff has discussed this at some length over on the virtualization team blog, but as a general rule of thumb we believe that it is much better to have paging occur inside the guest operating system rather than at the virtualization layer (if paging is needed).
The simple reason for this is that the guest operating system has far better understanding of which are the best sections of memory to page out – where as all the virtualization layer can do is to guess at what should be paged out.
In my recent demonstrations of dynamic memory I came upon another interesting angle to consider. Here is a screenshot of task manager from a virtual machine that is running at –23% memory availability (i.e. it does not have enough memory available and is paging in the guest):
What is fascinating about this screenshot is that even though this virtual machine is significantly short of memory, the guest operating system is still keeping 112mb memory available / as file cache. The reason for this is that the copy of Windows inside the virtual machine knows that even though it is short of memory – it can provide the best experience for the user of the virtual machine by not using all the memory that it has and by keeping a little bit free to serve as a cache / be there for new applications.
It is exactly this sort of logic that gets lost when paging is done at the virtualization layer instead of inside the guest operating system.
With Windows Server 2008 R2 SP1 we have added a number of new performance counters for dynamic memory. These counters allow you to get some extra insight into what is actually happening with memory on your Hyper-V server. The counters are grouped in two categories. First is “Hyper-V Dynamic Memory VM”:
These counters allow you to access specific information about dynamic memory inside each of the virtual machines on your system. The second category is “Hyper-V Dynamic Memory Balancer”:
These counters allow you to view information about dynamic memory across the entire physical computer.
One thing to note is that unlike the user interface – which uses the concept of “memory availability” (which I discussed yesterday) – the performance counters instead talk about memory pressure. This is measuring the same thing as memory availability (i.e. how much memory does the virtual machine have compared to how much memory does it need) but the math is slightly different. A virtual machine with a pressure of 100 has exactly the amount of memory that it needs (this is equivalent to a memory availability of 0%). If the virtual machine pressure goes over 100 – it now has less memory than it needs (equivalent to a negative memory availability) and if the virtual machine pressure is under 100 then the virtual machine has more memory than it needs.
Virtual machine pressure is simply calculated by taking the amount of memory the virtual machine wants, dividing it by the amount of memory the virtual machine has and then multiplying the result by 100.
My favorite performance counter is the “Average Pressure” counter under the “Hyper-V Dynamic Memory Balancer” category. This gives you a very simple view of the overall memory allocation of your system:
As long as this number is under 100, you know that there is enough memory is your system to service your virtual machines. Ideally this value should be at 80 or lower. The closer this gets to 100, the closer you are to running out of memory. Once this number goes over 100 then you can pretty much guarantee that you have virtual machines that are paging in the guest operating system.
When you use dynamic memory with Hyper-V one of the obvious changes to the user interface is the addition of the “Memory Availability” column:
This is actually a really useful figure to have handy. What memory available represents is the ratio of how much memory a virtual machine has – compared to how much memory the virtual machine needs. For example:
If a virtual machine had 1000MB of memory – but only needed 900MB of memory we would report a memory available figure of 11%, because the virtual machine has 11% more memory than it actually needs.
Because the workload inside a virtual machine is constantly changing, and we are constantly moving memory around in response to these changes, you can expect the memory available figure for your virtual machines to keep on moving as the virtual machine is running as well. It is also possible for a virtual machine to report a negative figure for memory available:
What this means is that the virtual machine has less memory than it needs. Note that in this case the virtual machine is still running – but it is likely that the guest operating system is now needing to page heavily in order to make forward progress.
You should always keep an eye on the memory availability of your system. As a general rule of thumb you want to see this value at the level of your configured memory buffer (which is 20% by default). Once you see this value start to dip under 10% that is a good warning that you should not be starting new virtual machines. If this value starts to get close to 0% (or even into the negatives) you should really think about stopping some virtual machines or migrating them to another server.
I am a bit late in getting this post up (due to being on the road at TechEd Australia & TechEd New Zealand); but last week we released a hotfix rollup package.
This package includes the fixes for three of the five issues that I recently blogged about. The specific issues that are addressed are:
KB Number: KB975530 Title:Stop error message on an Intel Xeon 5500 series processor-based computer that is running Windows Server 2008 R2 and that has the Hyper-V role installed: "0x00000101 - CLOCK_WATCHDOG_TIMEOUT" Description:This is fairly simple. If you have a computer with an Intel Xeon 5500 series processor or an Intel Core-i series processor and you see this blue screen – install this hotfix. Extra detail: I blogged about this back in October. Some of you reported that this did not solve the problem for you. The hotfix has since been updated to address the problem on these systems that were missed with the original fix. Link: http://support.microsoft.com/kb/975530
KB Number: KB981791 Title:"STOP: 0x0000001a" error message on a computer that has an Intel Westmere processor together with the Hyper-V role installed on Windows Server 2008 SP2 or on Windows Server 2008 R2 Description: Once again, if you have this processor (Intel Westmere) and see this blue screen – get this hotfix. Link: http://support.microsoft.com/kb/981791
KB Number: KB974909 Title: The network connection of a running Hyper-V virtual machine is lost under heavy outgoing network traffic on a Windows Server 2008 R2-based computer Description:This affects all guest operating systems. The fix is applied to the hypervisor and while a reboot is required for installation, no updated need to be applied to the guest operating systems. Link: http://support.microsoft.com/kb/974909
You can get the new rollup package from here: http://support.microsoft.com/kb/2264080 – alternatively it will be appearing as an optional update under Windows Update.
I am sitting in an Airport as I write this post. I am currently on my way to Australia – where I will be speaking next week at TechEd Australia 2010. Then the following week I will be heading over to New Zealand for a repeat performance
If you are attending either of these shows – please come by and hear what I have to say – and pepper me with your virtualization questions.
Here is my speaking schedule for TechEd Australia:
Session Title: Hyper-V and Dynamic Memory in Depth Session Code: SVR302 Time: Wednesday, August 25th, 9:45am - 11:00am Location: Arena 1A
Session Title: Windows Server 2008 R2 Hyper-V Performance Analysis: How You Can Get the Most out of Hyper-V Session Code: SVR313 Time: Friday, August 27th, 8:15am – 9:30am Location: Central C
And for TechEd New Zealand:
Session Title: Hyper-V and Dynamic Memory in Depth Session Code: SVR302 Time: Monday, August 30th, 5:25pm – 6:25pm
Session Title: Windows Server 2008 R2 Hyper-V Performance Analysis: How You Can Get the Most out of Hyper-V Session Code: SVR313 Time: Wednesday, September 1st, 10:40am – 11:40am
I look forward to seeing you there!
For as long as I can remember – I have been talking to people about supported / optimal virtual processor to logical processor ratios with Hyper-V. Or – said with less technical jargon – how many virtual machines, with how many virtual processors, should you run on any given physical computer?
Today our documented guidance is that we support up to 8 virtual processors for each logical processor in the physical computer – and that for best performance under most reasonable circumstances you should aim for a ratio of 4 virtual processors per logical processor in the physical computer.
The problem with all of this? We do not provide any simple way to find out what your current ratio is.
When I realized this – I simultaneously realized that I did not know what the virtual processor to logical processor ratio was on any of my computers. As such – I decided that I needed to write a script to do this.
Now – in the past some of my readers have accused me of not being a “real PowerShell scripter”, and that I was “writing VBscript style PowerShell”. Today I would like to try and set the record straight on this accusation. Because as I started looking into this – I realized that this did not need to be a script at all. No! It can be done with one (nearly incomprehensible) line of PowerShell code. Which goes as follows:
write-host (@(gwmi -ns root\virtualization MSVM_Processor).count / (@(gwmi Win32_Processor) | measure -p NumberOfLogicalProcessors -sum).Sum) "virtual processor(s) per logical processor" -f yellow
When I ran this on my main Hyper-V server at home – I got the following result:
Neat! I am practically right on the sweet spot!
But what does this line of code do – and how does it work? Well – there are a couple of things to know:
I got asked this question a couple of weeks ago – and I did not know the answer. After a bit of investigation – I now know the answer – and thought I would take the time to share it with the world (if you do not know what Sysprep is – go and read this first). The short answer is:
Everything just works.
The long answer is as follows:
Prior to Window Server 2008 R2 (namely in Windows Server 2008) you could run into a couple of problems. In Windows Server 2008 R2 we have implemented a Sysprep provider to ensure that everything goes smoothly. This provider does work during two of the phases of Sysprep:
Generalize
The generalize phase of Sysprep is where an installed version of Windows is prepared to be duplicated – and all installation specific information is removed. During this phase Hyper-V makes a couple of key changes:
Specialize
The specialize phase is run after the system image has been deployed to a new computer. Hyper-V makes the following change as part of this phase:
After writing the script to find all virtual hard disks associated with a virtual machine – the next logical step was to answer some questions about these virtual hard disks. The two obvious questions that I could think of were:
So I modified yesterdays script to answer these questions:
# Prompt for the Hyper-V Server to use
$HyperVServer = Read-Host "Specify the Hyper-V Server to use (enter '.' for the local computer)"
# Prompt for the virtual machine to use
$VMName = Read-Host "Specify the name of the virtual machine"
# Get Storage management service
$ImgService = gwmi Msvm_ImageManagementService -namespace "root\virtualization" -computername $HyperVServer -locale MS_409
# Get the virtual machine object
$VM = gwmi MSVM_ComputerSystem -namespace "root\virtualization" -computername $HyperVServer -locale MS_409 | where {$_.ElementName -eq $VMName}
# Create an empty hashtable
$table = @{}
# Set size variables
$totalSize = 0
# Go over each of the virtual machines "system setting data" objects. There will be a system setting data
# for each snapshot associated with the virtual machine, and one for the active virtual machine.
foreach ($VSSD in @($VM.getRelated("Msvm_VirtualSystemSettingData")))
{
# Get all the VHDs associated with the current system setting data
$VHDs = [array]($VSSD.getRelated("Msvm_ResourceAllocationSettingData") | where {$_.ResourceType -eq 21} | where {$_.ResourceSubType -eq "Microsoft Virtual Hard Disk"})
# Only continue if the system setting data actually had virtual hard disks
if ($VHDs)
# Intialize index, and get the VHD and VHDPath for the first virtual hard disk
$index = 0
$VHD = $VHDs[$index]
$VHDPath = $VHD.Connection | select -first 1
# loop through all virtual hard disks
do
# Get the detailed information for the current virtual hard disk
$xml = [xml]($ImgService.GetVirtualHardDiskInfo($VHDPath)).info
# Only continue if the current virtual hard disk is not in the hashtable
if (!$table.ContainsKey($VHDPath))
# Store the file size for the virtual hard disk in the table - and add it to $totalSize
$table[$VHDPath] = [uint64]($xml.Instance.Property | ?{$_.Name -eq "fileSize"}).value
$totalSize = $totalSize + [uint64]($xml.Instance.Property | ?{$_.Name -eq "fileSize"}).value
# Check to see if the current virtual hard disk has a parent virtual hard disk
if (($xml.Instance.Property | ?{$_.Name -eq "ParentPath"}).value)
# if it does - make the parent the next virtual hard disk that we look at
$VHDPath = ($xml.Instance.Property | ?{$_.Name -eq "ParentPath"}).value
}
# If the current virtual hard disk does not have a parent - just move onto the next virtual
# hard disk in the system settings data object
else
# Increase the index - and make sure that we have not gone past the last virtual hard disk
$index = $index + 1
if ($index -ne $VHDs.count)
# Setup new values for next time through
$VHDPath = $VHD.Connection | select -first 1}
# If the current virtual hard disk is already in the hashtable - move onto the next virtual
# loop until we have covered all virtual hard disks
until ($index -eq $VHDs.count)
# Get the VSSD and VHD array for the currently active virtual machine
$activeVSSD = $VM.getRelated("Msvm_VirtualSystemSettingData") | ?{$_.SettingType -eq 3}
$activeVHDs = [array]($activeVSSD.getRelated("Msvm_ResourceAllocationSettingData") | where {$_.ResourceType -eq 21} | where {$_.ResourceSubType -eq "Microsoft Virtual Hard Disk"})
# Setup some blank varibles
$activeVMSize = 0
$activeVMCapacity = 0
# Calculate the total file size and internal capacity only for the active virtual machine
foreach ($VHD in $activeVHDs)
$activeVMSize = $activeVMSize + [uint64]($xml.Instance.Property | ?{$_.Name -eq "fileSize"}).value
$activeVMCapacity = $activeVMCapacity + [uint64]($xml.Instance.Property | ?{$_.Name -eq "MaxInternalSize"}).value
# Do some math and formatting to get from bytes to gigagytes
$totalPossibleSize = "{0:N2}" -f (($totalSize - $activeVMSize + $activeVMCapacity) / 1073741824)
$totalSize = "{0:N2}" -f ($totalSize / 1073741824)
# Display the results
write-host "Total disk space used by the virtual machine:" $totalSize "GB"
write-host "Most disk space that can be used by the virtual machine:" $totalPossibleSize "GB"
This script starts by just adding up the file size of all virtual hard disks associated with the virtual machine (this is the value that is stored in $totalSize). Then it takes this “total size” and removes the file size of the currently active disks – and replaces that with the maximum possible size of the currently active disks. This second value (stored in $totalPossibleSize) represents the largest amount of disk space that the virtual machine could use – in its current configuration.
This is something that sounds really simple to start with – but gets more complicated the more you think about it.
The question is: how do you identify all virtual hard disks that are associated with a given virtual machine?
Your first response might be “you open the virtual machine settings and have a look there!” But what if your virtual machine uses differencing disks? How do you identify all of the parent disks involved? How about if it has snapshots? What if those snapshots use a different set of virtual hard disks than the currently active virtual machine (this is a valid – though odd – configuration)?
To deal with all of these permutations and issues – I came up with the following PowerShell script:
foreach ($VSSD in $VM.getRelated("Msvm_VirtualSystemSettingData"))
# Add the virtual hard disk to the hashtable - including the type of the virtual hard disk
switch ([uint64]($xml.Instance.Property | ?{$_.Name -eq "Type"}).value)
2 {$table[$VHDPath] = "Fixed"}
3 {$table[$VHDPath] = "Dynamic"}
4 {$table[$VHDPath] = "Differencing"}
# Display the results in a nicely formated and sorted manner
$table.GetEnumerator() | Sort-Object Name | Format-Table -Autosize
This script will find every virtual hard disk in every snapshot associated with a virtual machine – as well as tracking down any parent disks of differencing disks. The neat trick in this script is that it uses a hash table so that it does not look at the same virtual hard disk twice (because the script goes through all snapshots of a virtual machine – you can end up looking at the same virtual hard disk over and over again if you do not do this).
At the end of the script it provides a list of all virtual hard disks that are associated with the virtual machine.