Hyper-V notes from the field

Quick or Live Migration may fail after you used Migrate Storage

2009-11-13T09:15:00Z

SCVMM R2 has a nice new feature to migrate the storage location of a VM. The move is done on one particular host from one drive to another.

The move can be done while the VM is offline, in safe state, or even online.

What I’ve seen recently, in a clustered environment, when a VM in safe state has its storage migrated it may fail to be moved to another node in the cluster.

It still runs fine on the host where it was migrated, but cannot be Quick or Live migrated to any other host.

10698 Virtual Maschine Cluster Configuration Error!

The error you are seeing in the event log is:

"'<Your VM Name> Configuration' failed to register the virtual machine with the virtual machine management service.

The Virtual Machine Management Service failed to register the configuration for the virtual machine '645D10B4-ABCD-47F4-3839-46BBFF48CC7A' at 'C:\ClusterStorage\Volume1\<Your VM Name>': The system cannot find the file specified. (0x80070002)"

The reason for this failure is that SCVMM changes the VM Guid during the migration, and the other cluster nodes are not aware of this change.

Obviously, restarting the nodes, or at least the cluster service would solve this issue.

A more easy solution for this issue is, to take both resources for this VM and temporarily assign it to “Run this resource in a separate Resource Monitor”. This will cause the other nodes to re-read the VM Guid

1. Start Failover Cluster Manager

2. Open Services and Applications and select the VM. Take the VM Offline

3. Open Properties on the Configuration, Advanced Polices. Check “Run this resource in a separate Resource Monitor”, Apply, uncheck, Apply

4. Do the same for the VM Resource

The root cause is currently investigated and I will post here when there is a better solution

Cheers

Robert

Installing SCVMM on a host with a name containing "-SCVMM-" fails with Error 257

2009-11-05T10:05:00Z

Well, the title says it all. If the host you're trying to install SCVMM on has a name that contains "-SCVMM-" in upper case letters, setup fails with Error 257.

Example: MUC-SCVMM-1

Resolution: Use a slightly different name, like "MUC-SCVMMR2-1" or "MUC-SCVmm-1",

The reason is that the uppercase string "-SCVMM-" is used internally for host cummincation by SCVMM.

Cheers

Robert

SCVMM 2008 Error (2927) A Hardware Management error has occurred trying to contact server

2009-10-23T15:08:00Z

I had a large customer where some Host occasionally where reported with an Error during Host refresh:

The Error in SCVMM was:

Error (2927)

A Hardware Management error has occurred trying to contact server <your host name comes here>.

(Unknown error (0x80338029))

As it turned out, during refresh we did not wait long enough for a wsman response. The operation that timed out was to enumerate the SAN LUNs.

In SCVMM 2008 this timeout is 120 seconds by default. Fortunately, we can increase this timeout to a maximum of 300 seconds.

To resolve this issue:

1. Use Registry Editor on the SCVMM 2008 Host and locate [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Server\Settings]

2. Add a DWORD Value named IndigoSendTimeout

3. Set this value to decimal 299

4. Restart the VMM service (net stop vmmservice & net start vmmservice)

Important note: SCVMM 2008 R2 already has a higher default value of 330 seconds. There should be no need to increase this value. If you modify this value for SCVMM 2008, be sure to remove it before you upgrade to SCVMM 2008 R2, otherwise you would run with a lower timeout then the default.

Cheers

Robert

BITS Compact Server cannot process the request for the following URL in the %1 URL group: %2. The request failed in step %3 with the following error: %4.

2009-10-20T13:45:00Z

Hi,

Windows Server 2008 R2 has a new feature component: Bits Compact Server! This is a lightweight Server for Background Intelligent Transfer Service.

SCVMM 2008 R2 makes use of it and enables this feature on the Hyper-V Hosts during Agent installation.

Recently, I had a customer where all Bits Transfers (creatíng, migration, etc) to one particular Host where failing.

ERROR CODE: 0x80072efe
ERROR CONTEXT: 0x00000005

The error in Eventlog is:

Following warning is logged in BITS analytic log:
Log Name:      Microsoft-Windows-Bits-CompactServer/Analytic
Source:        Microsoft-Windows-Bits-CompactServer
Date:          16.09.2009 0:49:22
Event ID:      50

As it turned out, removing the host (in this case all cluster nodes) from SCVMM, deleting the Certificates and adding the hosts back to SCVMM resolved the issue.

These steps are also described in here: http://blogs.technet.com/scvmm/archive/2009/07/20/vm-creation-may-fail-and-stall-with-copying-0kb-of-16gb.aspx

Cheers

Robert

After installing Hyper-V Integration Services on the next reboot the VM displays BSOD 0x0000007B

2009-10-07T12:46:00Z

Hi,

Recently, I had some customers with VMs they just P2V'ed, or even had them running already on Virtual Server or Hyper-V. Now, they installed the latest Integration Components that came with the R2 Release.

After the required reboot, the VM shows a Bluescreen 0x0000007B INACCESSIBLE_BOOT_DEVICE

During Debugging I found that the Storage Driver of the ICs require the Windows Driver Framework (WDF), which was not loaded in this case. So the Storage Driver fails to load.

Looking into the Registry of the VM, shows that the WDF Driver was already installed previously, but had the wrong Group relationship, so it is initialized too late.

So here’s how to fix this issue:

1. Boot the VM into LastKnownGood. Press F8 during boot.

2. Open the Registry and drill down to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Wdf01000

3. There is a Group Value that should have the Value WdfLoadGroup. In my cases it was wrongly set to base. Change this to WdfLoadGroup

4. Then remove the Integration Components from Control Panel/Software.

5. Reboot the VM (now without ICs)

6. Install the ICs once again

Cheers

Robert

My Recommended Fix List Post SP2

2009-07-17T11:39:00Z

While Service Pack 2 contains many fixes that I recommended earlier, I intend to post here some recent fixes that helped in newer situations:

1. Occasional crashes of Vmwp.exe http://support.microsoft.com/kb/972045/en-us especially because it contains (http://support.microsoft.com/kb/970089/en-us)

2. The volume GUID may unexpectedly change after a volume is extended on a Windows Server 2008 failover cluster node http://support.microsoft.com/default.aspx?scid=kb;EN-US;970529 , especially important when using SCVMM as well

3. A Hyper-V differencing disk that you create in Windows Server 2008 R2 cannot be used in Windows Server 2008. http://support.microsoft.com/kb/971677/en-us

...to be continued....

Cheers

Robert

SCVMM Service may take up lots of memory

2009-05-25T11:48:00Z

In a large SCVMM environment we noticed that the SCVMM Service (vmmservice.exe) allocates lots of memory (>4GB). You may notice that you are unable to create new console sessions or existing sessions lose their connection when all available memory is used. This is not a leak as the memory is freed over time.

The reason for this allocations was the large number of jobs that had run in the past. By default SCVMM keeps the last 90 days in the Database and the Console shows this in the Jobs pane. In the title area you can see the number of jobs in brackets. In the customer case we had almost 10.000 jobs.

The problem can be solved by setting a shorter history with the following registry key:

HKLM\Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\settings\sql
DWORD Value: TaskGC
Enter the history length in days. (e.g. 7)

SCVMM starts a maintenance procedure with this number of days every 20h. You may not see a complete reduction immediately, as the maintenance procedure limits itself in the number of objects it deletes in one run. So you may need to monitor this for some days.

Cheers

Robert

Availability of Hotifx Rollup package for System Center Virtual Machine Manager 2008

2009-04-17T14:13:00Z

Recently we published a Rollup for System Center Virtual Machine Manager 2008 on Microsoft Update.

http://support.microsoft.com/kb/961983/en-us

When your SCVMM Server is not directly connected to the Internet you may have the need to download the package from another machine

You can get the fix from http://catalog.update.microsoft.com and specifing Article 961983.

You receive a package containing a file vmmServer64update.msp

Transfer this file to the target SCVMM server and install it with the following command:

msiexec /p vmmServer64update.msp BOOTSTRAPPED=1

Cheers,
Robert

Virtual Machine "Disappears" from Hyper-V Manager or changes to a VM cannot be applied

2009-04-07T11:42:00Z

Recently I saw some strange errors:

- A complete VM was not listed in Hyper-V Manager anymore.

-or-

Making changes to a VM and clicking Apply caused error messages like: The requested operation cannot be performed on a file with a user-mapped section open. (0x800704C8)

When opening the machine configuration XML File in Notepad, I noticed that the file was not ended properly. The correct last line would be:

/configuration>

Instead, there where characters following like /configuration>> or, parts of the configuration where repeated and a 2nd /configuration> existed in the File.

So it appears that the Configuration file was shortened due to the changes, but the EOF was not properly set.

The resolution to this problem was to disable the Virusscanner on the XML File. We could recover the .XML file by deleting the extra information
It's a general recommendation anyway to exclude Hyper-V Processes (VMMS.EXE, VMWP.EXE) and VM Files (*.VHD, *.AVHD, *-XML ) from scanning. You should instead use an AV Scanner inside the VMs.

Cheers
Robert

During Quick Migration the Virtual Machine Configuration resource stays in "Offline Pending"

2009-03-25T15:28:00Z

I have seen this at a customer site. Whenver one VM was moved from one node to the other, it took hours.
During that the Virtual Machine Configuration resource in Failover Manager showed in "Offline Pending"

As it turned out, we found a self-referencing link in the Default Configuration folder. C:\ProgramData\Microsoft\Windows\Hyper-V

A dir in this folder showed a file that looked like this:

17.02.2009 11:56 <SYMLINK> 08CF336C-A430-4D3B-B47E-63AF9AE867E9.xml [C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\08CF336C-A430-4D3B-B47E-63AF9AE867E9.xml]

This is a SYMLINK pointing to itself.

Normally, it should look similar to:

07/29/2008 02:15 PM <SYMLINK> 0DB03AC6-FB95-4F83-A52A-31A04A5154B0.xml [V:\CONTOSO\W2KDC\CONTOSO W2KDC\Virtual Machines\0DB03AC6-FB95-4F83-A52A-31A04A5154B0.xml]

Resolution: Delete the self-referencing link(s) on all nodes,and restart the VMMS Service.

Please drop me a short note if you figure out how to get into this sitation.

Cheers
Robert

SCVMM P2V conversion fails during 'Make operating system virtualizable'

2009-03-17T18:30:00Z

You may run into the problem that a P2V conversion to a machine with an IBM SAN using an older version of SDDDSM fails. SDDDSM is the IBM MPIO Device Specific Module.

I have also seen this with SCVMM 2007 on Windows Server 2003 with Virtual Server 2005

Now, I had a similar issue with SCVMM 2008 on Windows Server 2008 with Hyper-V

Error (3101)
VMM failed to mount VHD file \\?\<SomeVolumeGuid\vhd> on the <hostname> server. A timeout occurred.
(Internal error code: 0x80990C1D)

Recommended Action
Check Device Manager/System Devices to make sure that Microsoft Virtual Server Storage Bus is installed and functional. If it is not, install VHDMount component of Virtual Server. Restart Virtual Disk Service, and then try the operation again.

This problem is fixed in SDDDSM version 2.4.0.0 and greater.

From <ftp://ftp.software.ibm.com/storage/subsystem/windows/2.4.0.0-9/readme_2400-9_081020.txt>

Defects fixed in 2.4.0.x

…

4336 Unmounting Virtual Devices in Microsoft virtual Server Fails.

…

After updating to a recent SDDDSM Version, P2V worked as expected.

Cheers

Robert

A Windows 2000 VM may perform badly and consume high CPU

2009-03-17T16:50:00Z

Just had a couple of issues where a Windows 2000 VM was apparently slow and consumed >80% CPU.

They had in common that they have been converted (P2V) from real hardware with multiple processors.

Now, in Hyper-V, they run with the one supported processor, but still with Multiprocessor HAL and Kernel.

After changing that to ACPI-Uniprocessor-PC, the VMs performed as expected.

To change HAL and Kernel, use Device-Manager, Computer, Update Driver, and select ACPI-Uniprocessor instead of ACPI-Multiprocessor.

(Kudos to André Roos)

Cheers

Robert

SCVMM error 3217, SP2.cab missing

2009-02-06T16:54:00Z

Today, an issue with System Center Virtual Machine Manager.

If you try to convert a physical machine with Windows Server 2003 x64 SP2 into a Virtual Machine using SCVMMs P2V feature, you may get the error 3217.

The detailed error indicates SP2.CAB would be missing. However, on your install media you cannot find SP2.cab.

If you run into this error, you propably don't have a US or JPN version of Windows Server 2003 x64 SP2, but a localized one. Localized version have only been released with SP2 integrated.

To Convert this kind of machines, follow this steps:

1. Copy the intelide.sy_ from your installation media. It's under \amd64\

2. Use Expand.exe to unpack it to intelide.sys and place it into „\Program Files\Microsoft System Center Virtual Machine Manager 2008\Patch Import“

3. From the SCVMM Console run Powershell

4. Inside Powershell execute Add-Patch

Now, the P2V Process should be successfull.

You would need to redo this for every language version you intend to convert.

Cheers

Robert

Comprehensive List of Hyper-V Updates

2009-02-02T13:58:00Z

I'm often getting asked for paritcular fixes required for Hyper-V . There is a comprehensive list published here:

http://technet.microsoft.com/en-us/library/dd430893.aspx

Note the links for other products like SCVMM and DPM.

Cheers

Robert

Howto manually add a VM Configuration to Hyper-V

2008-12-19T15:29:00Z

With Hyper-V you cannot simply „Add“ a configuration file to your Hyper-V machine as you could with Virtual Server. Mainly because of the use of Live Snapshots. The recommended way is to use Export and Import.

If you have a given VHD it is way easier to create a new VM and simply add the VHD to it. If you used snapshots, see my earlier blog on how to merge them to a new VHD.

However, recently my colleague Walter Weinfurter had a customer issue, where we only had the flat directory structure and needed to restore that. Here’s how we managed to get this running:

We start from a folder that contains the VM in v:\manualrecover

In this folder you usually have:

1. The machine VHD

2. The Virtual Machines Folder with

a: The <GUID>.xml File holding the machine Configuration

b: The <GUID> Folder

3. The Snapshots Folder with a <Snapshot GUID>.xml

Hyper-V uses a new Feature in Windows 2008 called Service SIDs. To access files from a VM, these files need to give this Service SID access permissions.

This SID is a combination of the Service SID ""NT VIRTUAL MACHINE" and the VM GUID

Example:

"NT VIRTUAL MACHINE\2F855D88-F990-47BA-95D6-0029BCD8C059"

Note, The GUIDS we used here are from our VM, You will need to adjust to the GUIDs used on your installation

=========================================================================

The first step to make this machine known to Hyper-V is to create a Symbolic link to the <GUID>.xml configuration files, in the following folder:

"%systemdrive%\programdata\Microsoft\Windows\Hyper-V\Virtual Machines"

We use the mklink command built into cmd.exe

C:\>mklink "%systemdrive%\programdata\Microsoft\Windows\Hyper-V\Virtual Machines\2F855D88-F990-47BA-95D6-0029BCD8C059.xml" "V:\manualrecover\Virtual Machines\2F855D88-F990-47BA-95D6-0029BCD8C059.xml"

The VM Name should now already appear in Hyper-V Manager

We need to add the Service SID to this Symbolic link so that Hyper-V is allowed to access it

C:\>icacls "%systemdrive%\programdata\Microsoft\Windows\Hyper-V\Virtual Machines\2F855D88-F990-47BA-95D6-0029BCD8C059.xml" /grant "NT VIRTUAL MACHINE\2F855D88-F990-47BA-95D6-0029BCD8C059":(F) /L

Note the /L parameter to indicate we work on a symbolic link

=========================================================================

We also need to give the Service SID access to all files of our VM

C:\>icacls v:\manualrecover\ /T /grant "NT VIRTUAL MACHINE\2F855D88-F990-47BA-95D6-0029BCD8C059":(F)

Note for simplicity we give Full Control, whereas the default is more granular

=========================================================================

If your machine had Snapshots, we also need to create another symbolic link

The snapshot itself has yet another guid.xml found in the snapshots folder.

C:\>mklink "%systemdrive%\ProgramData\Microsoft\Windows\Hyper-V\Snapshots\7DD74401-C2B4-4BD9-8079-3D48D8A78B32.xml" "V:\manualrecover\Snapshots\7DD74401-C2B4-4BD9-8079-3D48D8A78B32.xml"

Also give the Service SID access here too:

C:\>icacls "%systemdrive%\ProgramData\Microsoft\Windows\Hyper-V\Snapshots\7DD74401-C2B4-4BD9-8079-3D48D8A78B32.xml" /grant "NT VIRTUAL MACHINE\2F855D88-F990-47BA-95D6-0029BCD8C059":(F) /L

You will need to do the above for each individual snapshot!

=========================================================================

Before starting the VM, open the settings of the VM and assign the Network Adapters to the correct Switches, as those need to be created newly on the switch.

=========================================================================

NOTE:

This is not a supported way of adding a VM to Hyper-V. Use this just for disaster recovery, and once you are able to run the VM, backup your VM and recreate it from scratch.

If you restore to a different drive/ or folder, you may need to manually adjust the path to the VHD, and Snapshots make this far more complicated .

NOTE2: This procedure does not work on Windows Server 2008 R2.

Cheers

Robert