Random Musings of Jeremy Jameson : Virtualization

HVRemote and Remote Administration of Hyper-V from Windows 7

Jeremy Jameson — Thu, 08 Oct 2009 14:21:00 GMT

Last Sunday, I rebuilt my desktop (WOLVERINE) with the RTM build of Windows 7 Ultimate (x64). Previously, I'd been running the RC1 bits and I figured it was about time I got around to "upgrading" to the RTM version. [I say "upgrading" because -- at least it in my experience -- it is typically better in the long run to simply reformat the hard drive and start over when installing a newer version of the operating system. Obviously I don't always stick to this principle. For example, I've upgraded some of my "Production" VMs from Windows Server 2003 to Windows Server 2008. However, I've seen a few bugs in my time that only occur in upgraded environments and therefore try to avoid these whenever possible.]

In order to get back to managing my Hyper-V servers from my Windows 7 desktop, I installed the Remote Server Administration Tools for Windows 7 and turned on the corresponding Windows feature. However, I found that I still couldn't connect Hyper-V Manager to either of my Hyper-V servers. It was then that I remembered John Howard's blog series on Hyper-V Remote Management that describe the various hoops you have to jump through in order to get remote administration of Hyper-V working. However, this time when I went searching for John's blog posts, I stumbled across the Hyper-V Remote Management Configuration Utility (HVRemote).

Apparently -- quite some time ago -- John scripted the various configuration steps described in his blog series, thus making it much quicker and almost effortless.

Note that in my environment, both the client and server are in the same domain. Therefore, following the "10-second guide" for HVRemote, I created a firewall rule to allow Microsoft Management Console:

C:\NotBackedUp\Public\Toolbox\HVRemote>cscript hvremote.wsf /mmc:enable Microsoft (R) Windows Script Host Version 5.8 Copyright (C) Microsoft Corporation. All rights reserved. Hyper-V Remote Management Configuration & Checkup Utility John Howard, Hyper-V Team, Microsoft Corporation. http://blogs.technet.com/jhoward Version 0.7 7th August 2009 INFO: Computername is WOLVERINE INFO: Computer is in domain corp.technologytoolbox.com INFO: Current user is TECHTOOLBOX\jjameson-admin INFO: Assuming /mode:client as the Hyper-V role is not installed INFO: Build 7600.16385.amd64fre.win7_rtm.090713-1255 INFO: Detected Windows 7/Windows Server 2008 R2 OS INFO: Remote Server Administration Tools are installed INFO: Hyper-V Tools Windows feature is enabled INFO: No TCP rule was found INFO: No UDP rule was found INFO: MMC Firewall exception changes OK. INFO: Are running the latest version

However, when I attempted to connect to Hyper-V Manager to my Hyper-V servers, I encountered errors similar to the following:

Access denied. Unable to establish communication between 'ROGUE' and 'WOLVERINE'.

In addition to enabling the firewall rule, I found that I also needed to allow Anonymous Logon remote DCOM access:

C:\NotBackedUp\Public\Toolbox\HVRemote>cscript hvremote.wsf /mode:client /anondcom:grant Microsoft (R) Windows Script Host Version 5.8 Copyright (C) Microsoft Corporation. All rights reserved. Hyper-V Remote Management Configuration & Checkup Utility John Howard, Hyper-V Team, Microsoft Corporation. http://blogs.technet.com/jhoward Version 0.7 7th August 2009 INFO: Computername is WOLVERINE INFO: Computer is in domain corp.technologytoolbox.com INFO: Current user is TECHTOOLBOX\jjameson-admin INFO: Build 7600.16385.amd64fre.win7_rtm.090713-1255 INFO: Detected Windows 7/Windows Server 2008 R2 OS INFO: Remote Server Administration Tools are installed INFO: Hyper-V Tools Windows feature is enabled INFO: Obtaining current Machine Access Restriction... INFO: Examining security descriptor INFO Granted Remote DCOM Access to Anonymous Logon WARN: See documentation for security implications INFO: Are running the latest version

After both of these changes were made, I found that I could once again perform remote administration using Hyper-V Manager.

Kudos to John for simplifying the necessary configuration steps!

The "Jameson Datacenter"

Jeremy Jameson — Mon, 14 Sep 2009 15:53:00 GMT

Back in a post from February 2008, I first referred to the "Jameson Datacenter" while discussing one of the servers running in my basement. Since then I've referenced my home lab about a dozen times in different posts but never provided significant details about the various servers and corresponding configuration.

The purpose of this post is to share those details and explain some of the reasons behind the infrastructure, since I believe my configuration represents a suitable "development environment" for many small- and medium-size teams.

[Note that last weekend I finally got around to retrofitting my "new" house with CAT5e cable from the second story down to the basement. Consequently I was finally able to move the three servers out of my upstairs office and into the basement where they belong. As expected, my office now feels a good 5 degrees cooler during the day ;-)]

The following figure shows the physical architecture of the various computers and network devices that comprise the "Jameson Datacenter":

Figure 1: The "Jameson Datacenter" - physical architecture

See full-sized image.

Note that I'm not suggesting development teams require a Windows Media Center or an Xbox 360 (which doubles as a Media Center Extender) -- I merely included those for the sake of completeness. [Although, personally, I believe that if you provide your developers a relaxed environment where they can blow off some steam for an hour or so, you shouldn't be surprised if productivity actually increases dramatically, contrary to what you might predict ;-)]

If you've followed this blog for any significant length of time, you already know that I'm a big fan of virtualization. Notice that I dedicate two servers specifically to running Hyper-V virtual machines (VMs). The following figure illustrates the various servers (both logical and physical) that typically run 24x7 in my lab:

Figure 2: The "Jameson Datacenter" - logical architecture

See full-sized image.

I should also point out that these aren't true enterprise servers -- meaning that they don't have redundant power supplies, remote management cards, etc. These are simply "home built" servers that I put together from components typically purchased through newegg.

Also note that the "Jameson Datacenter" used to contain many more physical servers. In fact, COLOSSUS was originally a refurbished Dell PowerEdge 4300 server that I shelled out 2500 bucks ($2,500) for back in July, 2000. That's when I originally came up with the naming convention, because the original COLOSSUS -- with its three power supplies and six hot-swappable hard drives -- literally weighed over 100 pounds!

Less than a year later, I purchased another Dell 4300 from ebay (i.e. the original BEAST), a PowerEdge 2300 (the original XAVIER), as well as a PowerEdge 6100 and couple of 4200s. However, I got tired of the "limited upgrade" path for these servers (even though I only paid a couple hundred bucks for some of the used servers) -- and even more tired of the monthly electricity bill for powering that many servers. I subsequently started building my own.

It's pretty amazing what a few hundred dollars will get you on newegg these days. In fact, when I somehow managed to kill one of my servers a few months back, I replaced the motherboard, CPU, and memory for $260.89 (and that included an AMD quad core processor and 8 GB of RAM). Since Microsoft isn't paying for these servers -- well, I suppose I should say "isn't reimbursing me for these servers" since my paycheck comes from Microsoft -- I typically have to get approval from the wife before giving newegg my credit card number. But alas, I digress...

The following table provides more detail on the various servers:

Server Configurations
Server	Role(s)	Operating System	Domain
BANSHEE	E-mail server (POP3 and SMTP)	Windows Server 2003 Standard x64 Edition with Service Pack 2	TECHTOOLBOX
BEAST	Database server (SQL Server 2005)	Windows Server 2003 Enterprise x64 Edition with Service Pack 2	TECHTOOLBOX
COLOSSUS	Windows Server Update Services (WSUS)	Windows Server 2008 Standard x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX
CYCLOPS	Team Foundation Server (TFS) application tier; note that databases reside on BEAST	Windows Server 2003 Enterprise x86 Edition with Service Pack 2	TECHTOOLBOX
DAZZLER	TFS build server	Windows Server 2008 Standard x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX
DOGFOOD	Development server; SharePoint Server 2010; Visual Studio 2010	Windows Server 2008 Standard x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX
FAB-DC01	Domain controller for corp.fabrikam.com; e-mail server (POP3 and SMTP)	Windows Server 2003 Enterprise x86 Edition with Service Pack 2	FABRIKAM
FOOBAR2	Development server; Microsoft Office SharePoint Server (MOSS) 2007; Visual Studio 2008	Windows Server 2008 Standard x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX
ICEMAN	Hyper-V server	Windows Server 2008 Standard x64 Edition (core installation) with Service Pack 2	TECHTOOLBOX
JUBILEE	System Center Operations Manager 2007 R2; Root Management Server (RMS)	Windows Server 2008 Standard x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX
ROGUE	Hyper-V server	Windows Server 2008 Standard x64 Edition (core installation) with Service Pack 2	TECHTOOLBOX
XAVIER1	Domain controller for corp.technologytoolbox.com	Windows Server 2008 Enterprise x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX
XAVIER2	Domain controller for corp.technologytoolbox.com	Windows Server 2008 Enterprise x64 Edition (full installation) with Service Pack 2	TECHTOOLBOX

Note that I use two different Active Directory domains (really two different forests):

corp.fabrikam.com (FABRIKAM) - I use this domain for development and testing purposes. In other words, whenever I want to do something "experimental", that may or may not be a permanent change.
corp.technologytoolbox.com (TECHTOOLBOX) - I treat this as my "production" domain. In other words, I typically only make changes in this domain after I've tested them in my FABRIKAM development/test domain.

There is no trust relationship between these Active Directory domains. In other words, I can't login to one of the TECHTOOLBOX servers with my FABRIKAM\jjameson-admin account, nor can I login to any of the FABRIKAM servers with my TECHTOOLBOX\jjameson-admin account. In some enterprise organizations that I've consulted with, there actually is a trust relationship between the "development" and "production" forests -- in order to avoid forcing developers to manage multiple accounts.

Note that I utilize separate Domain Admin accounts (e.g. TECHTOOLBOX\jjameson-admin and TECHTOOLBOX\jjameson) in order to adhere to the principle of least privileges. In other words, when logging into my Windows 7 desktop (i.e. WOLVERINE) with my TECHTOOLBOX\jjameson account, I don't have Administrator rights on the machine. Whenever I need to do something with elevated privileges, I typically use either my Domain Admin account (for example to RDP to one of the servers or modifying some secured content on another server) or the local Administrator account (for example, when simply installing new software or making a local configuration change).

Also note that I didn't show any of the VMs that I run on either my desktop (i.e. WOLVERINE) or my Microsoft laptop (i.e. JJAMESON1). For example, I have yet another development VM (FOOBAR) that I occasionally fire up on my desktop; or, when working at a client site, I often fire up FAB-DC02 and FAB-FOOBAR on my laptop in order to develop or demonstrate something in MOSS 2007. Yes, it's true, FAB-DC01 and FAB-DC02 often complain about not being able to replicate (since there obviously is no connectivity between my laptop sitting in a customer location and the "Jameson Datacenter" running in my basement), but so far this hasn't seemed to cause any significant issues. Of course, I can't create any new objects (e.g. users and groups) on FAB-DC02 unless it can connect to FAB-DC01, since FAB-DC01 holds all of the Flexible Single Master Operations (FSMO) roles.

You might be wondering why several of the servers are still running Windows Server 2003 and not Windows Server 2008. There are actually several reasons for this -- which vary by server:

As I discovered shortly after Windows Server 2008 came out, the POP3 service is no longer included in the operating system. I guess Microsoft simply wanted to deprecate this service in order to eventually terminate the corresponding support obligations. This shouldn't be a big deal to most people, since I seriously doubt many enterprise organizations -- or even small businesses -- use the POP3 service in Windows Server 2003 for e-mail. However, I neither need nor want to use a full-blown instance of Microsoft Exchange simply for the purposes of, say, demonstrating various e-mail notifications from MOSS 2007. Thus, BANSHEE and FAB-DC01 will probably not be moved to Windows Server 2008 anytime soon. While there are certainly third-party POP3 alternatives out there, I really don't want to go learn how to install and configure them. Trust me, there are many more valuable ways that I can spend that time.
BEAST is still running Windows Server 2003 because, well, I simply haven't seen any need to upgrade it. This server has been humming along for years and as the old saying goes, "if it ain't broke, don't fix it." ;-)
CYCLOPS is still running Windows Server 2003 for essentially the same reason. Since I've been running Team Foundation Server on CYCLOPS since shortly after its release as part of Visual Studio 2005 Team System, I've never bothered to upgrade the operating system. Sure, I've since migrated the VM from Virtual Server to Hyper-V, and also upgraded to Team Foundation Server 2008, but I haven't seen any need to upgrade the base OS -- at least not yet.

It's also worth pointing out that while the three physical servers are running x64 versions of the operating system, I have a mixture of x86 and x64 VMs. Again, this can mostly be attributed to the history of the Jameson Datacenter (i.e. Virtual Server 2005 never did support x64 VMs). Note that I originally built out my development VMs (e.g. FOOBAR2) with x86, so that I could copy the VHD from the Hyper-V server to my laptop and subsequently run it under Virtual Server 2005 or Virtual PC. However, I discovered that while this should work in theory, I never had any success with it. I used to take VMs from my Hyper-V server and run them on my laptop back when it was running Windows Server 2008, but since my laptop has moved to Windows 7, that is no longer an option.

[I tried removing the Hyper-V Integration Services and configuring the VM to redetect the HAL before attempting to boot under Virtual PC, but the VM simply hung (repeatedly) early in the boot cycle. Thus, I simply decided to "punt" and concede that Hyper-V VMs will forever be Hyper-V VMs.]

The following figure provides more detail on the hardware configuration of the various servers:

Figure 2: The "Jameson Datacenter" - hardware configuration

See full-sized image.

Since BEAST is essentially my "production" SQL Server instance, I chose to configure all four disks in a RAID 10 (1+0) configuration. Ideally, I'd throw another four drives into that server -- or heck, even eight drives if the case would actually hold that many -- in order to separate my data I/O from my transaction log I/O, but, alas, the motherboard on BEAST only has four SATA II connectors. I suppose I might get faster throughput if I actually split the four disks into two RAID 1 arrays, but honestly, I doubt I really need it. If I planned on using my home lab for performance testing, this might be worthwhile, but since I rarely stress this SQL Server, I'm not going to bother. As you can see from the dual proc/2 GB configuration, one of these days I should probably throw another $260.89 at it (or perhaps even less) in order to replace the motherboard, CPU, and RAM in order to make BEAST look more like ROGUE.

The differences in disk configuration between ICEMAN and ROGUE reflect the trade-off I mentioned before. When I rebuilt ROGUE about six months ago (and also upgraded to 8 GB), I decided to split the previous RAID 10 configuration into two RAID 1 arrays. This allows me to place some VHDs on the C: drive and others on the D: drive -- thus dedicating different spindles to different workloads. I can tell you this definitely makes a difference because occasionally JUBILEE (my SCOM 2007 server that monitors all of the other servers) notifies me that the disk latency on one or more of the VMs running on ICEMAN exceeded the specified threshold. As I've stated before, in this day of incredibly fast CPUs, multiple cores, and gobs of RAM, you're most likely going to bottleneck on disk I/O first.

So, there you have it...the "Jameson Datacenter" in all its gory detail -- or at least enough detail to give sufficient context whenever I refer to one of the servers in other posts. Hopefully you've also picked up some good tips and recommendations for setting up a robust development environment for your organization.

Starting and Stopping Hyper-V VMs with Server Core

Jeremy Jameson — Thu, 13 Aug 2009 19:21:00 GMT

Last week before heading out to the airport for my SharePoint 2010 training, I powered down the "Jameson Datacenter" (i.e. the four computers running in my home office). Since I would be gone for almost 8 full days, it didn't make sense to waste the electricity just so my SQL Server could periodically send email messages to me about backups being completed successfully throughout the day. [Note that my wife uses only the Media Center in the family room and her laptop to fulfill all of her computing needs (primarily sending and receiving email, as well as surfing Facebook) and my daughter is too young to use any computer by herself.]

The day after I got back from Redmond, I powered up the servers, but then realized that this was the first time that I had explicitly shutdown both Hyper-V VMs that serve as domain controllers (i.e. XAVIER1 and XAVIER2, that I referred to in a previous post). Note that I had since moved one of the domain controller VMs to a different Hyper-V server. In other words, two of the four computers in my home office (ROGUE and ICEMAN) run Windows Server 2008 x64 Server Core, and each server hosts a DC so that I have some degree of fault tolerance.

When I attempted to start the VMs using Hyper-V Manager from my Windows 7 desktop, I encountered the following error shortly after seeing the "Connecting to Virtual Machine Management service..." message:

Cannot connect to the RPC service on computer 'ICEMAN'. Make sure your RPC service is running.

Well, the RPC service was definitely running, but since both DCs were powered down, Hyper-V Manager had no way of authenticating me in order to access the remote servers (note that I got the same error message when trying to access ROGUE).

I could login to both Hyper-V servers (with my domain account, thanks to cached credentials), but as I've noted before, I struggled with managing Hyper-V from the command line. I had seen some examples of how to do it using PowerShell, but never anything from a simple command console. Note that I never did install PowerShell on my Server Core boxes.

Fortunately, last Sunday I found Ben Armstrong's post for Starting a Hyper-V Virtual Machine which shows not only the PowerShell version that I had seen before, but also a WMI version (which can be used with plain old VBScript). While Ben's code certainly works, I ended up snarfing some similar sample code from MSDN to create a new ManageVM.vbs script (which seems more intuitive -- at least to me -- than using "StartVM.vbs" to stop a VM).

Now, whenever I am unable to manage my Hyper-V servers remotely, I can use something like the following to start one of the VMs:

\NotBackedUp\Public\Toolbox\Scripts\ManageVM.vbs XAVIER1 start

Using SysPrep'ed VHDs for New Hyper-V Virtual Machines

Jeremy Jameson — Thu, 13 Aug 2009 18:16:00 GMT

As noted in my previous post, I spent 7 days in "alpha" training last week for SharePoint 2010.

Consequently, one of my goals for this week was to update my "dogfood" VM with the CTP build of SharePoint Server 2010, so that I can continue building on the foundation of what I learned last week and dive deeper into the new features of the product.

This is the quintessential scenario for why I love virtual machines. I can quickly "spin up" a new VM and then proceed to load various products such as SQL Server 2008, SharePoint Server 2010, and Visual Studio 2010; and, since it's isolated in a VM, there's no risk with exploring early, pre-release builds of software.

Note that I don't start from scratch each time I need to create a new VM. Rather, I start from a SysPrep'ed image of the particular operating system that I want to use.

In this case, I copied my ws2008-std-x64.vhd (Windows Server 2008 Standard x64 with Service Pack 2) over to the folder for my DOGFOOD VM and renamed it -- yep, you guessed it -- dogfood.vhd.

I then proceeded to start the DOGFOOD VM using Hyper-V Manager. Unfortunately, I encountered the following error:

Virtual Machine Connection

The application encountered an error while attempting to change the state of 'dogfood'.

'dogfood' failed to start.

Microsoft Emulated IDE Controller (Instance ID {GUID}): Failed to power on with Error 'General access denied error'

IDE/ATAPI: Could not attach 'C:\NotBackedUp\VMs\dogfood\dogfood.vhd' to location 0/0 of IDE Controller. Error: 'General access denied error'

The file 'C:\NotBackedUp\VMs\dogfood\dogfood.vhd' does not have the required security settings. Error: 'General access denied error'

Using the cacls.exe utility, I examined the permissions specified for a VHD used by one of the other VMs that was already running on the same server. I found the following:

NT VIRTUAL MACHINE\48FEF7FD-04D0-4AFF-B6F F-4F0D159DC40E:(special access:)
- READ_CONTROL
- SYNCHRONIZE
- FILE_GENERIC_READ
- FILE_GENERIC_WRITE
- FILE_READ_DATA
- FILE_WRITE_DATA
- FILE_APPEND_DATA
- FILE_READ_EA
- FILE_WRITE_EA
- FILE_READ_ATTRIBUTES
- FILE_WRITE_ATTRIBUTES
NT AUTHORITY\SYSTEM:(ID)F
BUILTIN\Administrators:(ID)F
BUILTIN\Users:(ID)R

Note that the list of discrete permissions listed for the NT VIRTUAL MACHINE service account essentially equates to Read and Write, as observed by running the new icacls.exe tool introduced in Windows Server 2003 SP2, and also included in Windows Vista and Windows Server 2008.

Consequently, you can specify a command like the following to grant permissions on the VHD to the service account used by Hyper-V:

icacls dogfood.vhd /grant "NT VIRTUAL MACHINE\C60995EB-8B7D-46DB-BD1E-3638CD1AEC32":(R,W)

Note that the GUID varies with each VM, so you'll need to tweak this accordingly for your environment. To find the GUID to use, examine the Virtual Machines folder for the VM.

Also note that you may need to use the VM settings to remove the VHD from the IDE controller and subsequently re-add it (in other words, it looks like Hyper-V does not recognize that the file permissions have been updated in the background).

It takes about 10 minutes to copy the SysPrep'ed VHD, which is obviously much, much faster than the time it takes to install Windows Server 2008 from scratch (especially when you take patching into account). Overall, it takes me about 25-30 minutes to spin up a new Windows Server 2008 VM, which includes joining it to my Active Directory domain and getting all of the latest patches applied using Windows Server Update Services (which, in the "Jameson Datacenter" is enforced through Group Policy).

Why choose "Server Core" installation of Windows Server 2008?

Jeremy Jameson — Fri, 05 Jun 2009 03:12:00 GMT

If you ever find yourself looking for reasons or evidence why you should choose the "Server Core" installation option for Windows Server 2008, try searching for the following:

"Windows Server 2008 Server Core installation not affected" site:microsoft.com/technet/security

You will find page after page of results similar to the following:

Microsoft Security Bulletin MS08-054 – Critical
Windows Server 2008 server core installation not affected. The vulnerability addressed by this update does not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/Bulletin/MS08-054.mspx · Cached page

Microsoft Security Bulletin MS08-078 - Critical
Windows Server 2008 server core installation not affected. The vulnerabilities addressed by this update do not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/bulletin/ms08-078.mspx · Cached page

Microsoft Security Bulletin MS08-053 – Critical
Windows Server 2008 server core installation not affected. The vulnerability addressed by this update does not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/Bulletin/MS08-053.mspx · Cached page

Microsoft Security Bulletin MS08-024 - Critical: Cumulative Security ...
Windows Server 2008 server core installation not affected. The vulnerabilities addressed by these updates do not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/Bulletin/MS08-024.mspx · Cached page

Microsoft Security Bulletin MS08-052 – Critical
Windows Server 2008 Server Core installation not affected. The vulnerabilities addressed by this update do not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/bulletin/ms08-052.mspx · Cached page

...

Seeing all these results is refreshing when I think back on the challenges I had to overcome when building out my first Hyper-V server using the Server Core installation, such as configuring remote administration of Hyper-V, or whenever I need to view PerfMon data on a Server Core machine (which is trivial on a "Full" installation, but not quite so easy on a Server Core installation).

Thanks to Ana Paula Moreira Franco, a Senior Consultant with Microsoft Consulting Services in Brazil, for pointing out these search terms.

Update on Patching and Disk Space Usage

Jeremy Jameson — Wed, 03 Jun 2009 17:09:00 GMT

About a year ago, I wrote a post about saving huge amounts of disk space by slipstreaming service packs. Having just been through an ordeal installing Windows Server 2008 SP2, I thought it would be worthwhile to provide an update (since that original post refers to disk space usage with Windows Server 2003).

Note that since my original post, I have switched from using Virtual Server in favor of Hyper-V. Among other things, this allows me to run x64 virtual machines (VMs). Many months ago, I consolidated numerous physical machines onto a couple of "Server Core" machines running Hyper-V. In that time, I've also switched to running Windows Vista x64 on my primary desktop and Windows Server 2008 x64 on my laptop.

One of the things that I've noticed is that x64 versions of the operating system tend to use more disk space than their corresponding x86 equivalents. In particular, the "side-by-side" folder (WinSxS) is typically significantly larger on x64 installations. The storage differences are negligible on my physical machines, but on VMs I make a deliberate effort to "clamp down" the size of the VHDs. This can save me considerable time when copying VHDs from one server to another or from an internal hard drive to an external hard drive whenever I need to take one or more of them "on the road" with me.

Minimizing VHD sizes also allows me to cram more VMs onto my 100 GB external drive [I know, these days this isn't very big from a capacity perspective, but at least it's 7200 RPM (a must for running VMs) and it isn't nearly as bulky as my larger drive enclosure. It also doesn't require a separate power supply either.]

Here is a baseline of the disk space usage on a Windows Server 2008 Standard x64 VM:

Figure 1: Disk usage on Windows Server 2008 Standard x64 VM (baseline)

See full-sized image.

Notice that the total disk usage is about 7.5 GB and the Windows folder consumes a little over 7 GB. Also note that Windows Server 2008 included SP1 (i.e. Microsoft slipstreamed it into the initial installation in order to simplify the servicing model for both Windows Server 2008 and Windows Vista).

I then immediately installed Windows Server 2008 SP2 and captured the following:

Figure 2: Disk usage on Windows Server 2008 x64 VM (after installing SP2)

See full-sized image.

Observe that the Windows folder now consumes a little over 10 GB of storage. Ouch...3 GB for a service pack. That seems a little, um, irritating -- for VMs, anyway. Obviously for physical machines with 100+ GB hard drives, the additional space is trivial.

I then ran the Windows Component Clean tool (COMPCLN.exe) as described in my previous post, which reclaimed approximately 900 MB of space.

Figure 3: Disk usage on Windows Server 2008 x64 VM (after installing SP2 and running COMPCLN.exe)

See full-sized image.

Notice that the Windows folder now consumes about 8.5 GB of space (but the overall free space on the 20 GB VHD increased from roughly 9.7 GB to 10.6 GB). In other words, SP2 adds roughly 3 GB, but COMPCLN.exe trims this to a little over 2 GB.

Lastly, I want to point out the current disk space usage on COLOSSUS -- an x64 VM that I run WSUS (Windows Server Update Services) for managing patches and updates on the various machines in the "Jameson Datacenter." Note that this server only has WSUS (which requires IIS and SQL Server) but nothing else. Consequently, after installing Windows Server 2008 SP2 and running COMPCLN.exe, I was hoping it would have comparable disk space usage to that shown in Figure 3 (after deducting the space used by the WSUS database, of course).

Unfortunately, it isn't even close, as shown in the following figure.

Figure 4: Disk usage on a patched WSUS server (after installing SP2)

See full-sized image.

Notice that the Windows folder on COLOSSUS consumes almost 16.5 GB of space, of which roughly 10.5 GB is used by the WinSxS folder.

The lesson here is that you should expect some "bloat" in the Windows folder over time (largely due to the WinSxS folder), and while the Windows Component Clean tool (COMPCLN.exe) undeniably reclaims some hard drive space after installing SP2, it's definitely not the same as starting with a "fresh" SP2 install.

Some Gotchas with Remote Administration of Hyper-V

Jeremy Jameson — Thu, 28 Aug 2008 16:41:00 GMT

As I mentioned in my previous post, last month I built out a new virtual environment using Hyper-V on Server Core. Since you can't run MMC -- and therefore Hyper-V Manager -- on Server Core, you need to use remote administration to manage the VMs.

John Howard's blog series on Hyper-V Remote Management is by far the definitive source for getting Hyper-V up and running on Server Core. It provides an excellent step-by-step guide for enabling remote administration, opening various firewall ports, configuring DCOM permissions (if you don't want to use admin accounts), etc. If you haven't yet at least scanned John's posts, I highly recommend doing so before embarking on the Hyper-V on Server Core path.

There is one snag, however, that I want to point out with regards to John's scenarios.

The instructions in John’s blog posts work when:

both the Hyper-V server and the client are in WORKGROUP mode, or
when the client and server are members of the same domain or trusted domains, or
when the Hyper-V server is in WORKGROUP but the client is in a domain

[Note that I personally verified the second and third scenarios above while initially building out my Hyper-V server; I am trusting that the first scenario works based on John’s posts.]

However, if the client is a member of DOMAIN1 and the Hyper-V server is a member of DOMAIN2 – and there is no trust relationship between DOMAIN1 and DOMAIN2 – then Hyper-V Manager pukes with a message about not being able to connect to the RPC server. Also note that in this scenario Disk Management pukes as well with the infamous error message:

RPC server is unavailable

To picture this scenario, imagine you have a Hyper-V server joined to your internal domain, but now I come along and try to use Hyper-V Manager from my laptop which is joined to the internal Microsoft domain. It simply doesn't work -- and neither does Disk Management.

At this point, you might be thinking something like “Jeremy, it sounds like a firewall issue or you haven’t enabled Remote Volume Management.” However, immediately after receiving the "RPC server is unavailable" message on my laptop, I was able to connect the Disk Management console to the Hyper-V server just fine from a Windows Server 2003 member server in the same domain.

In my mind, that indicated the firewall and remote administration were configured correctly. After a little research, it appeared that I was hitting a known bug with WMI when the client and server are in different, untrusted domains.

To workaround this issue, I did two things.

First, I created a new Vista VM and joined it to my customer's internal domain. After installing the Hyper-V Remote Management Update for Windows Vista (KB952627), I was able to start, stop, and create VMs on the Hyper-V server. Excellent.

Second, since I didn't want to have to always fire up my Vista VM just to view or change the Hyper-V settings on the server, I installed the Hyper-V Update for Windows Server 2008 x64 Edition (KB950050) on one of the VMs running on the Hyper-V server. This obviously doesn't completely replace the need for a remote administration client due to the "Catch-22" scenario -- meaning, if the VM isn't running, you can't use Hyper-V Manager from the VM to start the VM ;-)

Someday soon, I'm hoping we will release a few command line tools for Hyper-V that allow you to perform some basic operations such as starting or stopping VMs. This would be great on Server Core -- and no, I don't want to install PowerShell in order to do this ;-) [In keeping with the spirit of Server Core, I want to install as little as possible on the host.]

Lastly, I want to point out one of the stumbling blocks that I encountered along the way. Before I actually created the Vista VM that I mentioned earlier for remote administration, I initially created a Windows Server 2008 x86 VM and installed the 32-bit version of KB950050 in order to use Hyper-V Manager to remotely administer the Hyper-V server.

According to the corresponding KB article:

Update for Windows Server 2008 (KB950050)
This 32-bit update package includes the release version of the following:

The Hyper-V Manager console

The Virtual Machine Connection tool for x86-based editions of Windows Server 2008

Based on my experience installing the Hyper-V Remote Management Update for Windows Vista (KB952627) and the note above from the KB article, after installing KB950050 on Windows Server 2008 I expected to be able to start MMC and add the Hyper-V Manager snap-in. However, it doesn't quite work that way.

Fortunately, I received a quick response to my inquiry from Alex Kibkalo, a fellow Architect with Microsoft Consulting Services in Russia.

To enable Hyper-V Manager after installing KB950050, you need to enable the corresponding feature:

Open Server Manager. (If Server Manager is not running, click Start, point to Administrative Tools, click Server Manager, and then, if prompted for permission to continue, click Continue.)
In Server Manager, under Features Summary, click Add Features.
In the Add Features Wizard, on the Select Features page, expand Remote Server Administration Tools, and then expand Remote Administration Tools
Click Hyper-V Tools, and then proceed through the rest of the wizard.

For more information on deploying Hyper-V, refer to the Hyper-V Planning and Deployment Guide.

Copy/Paste Gotchas with Server Core

Jeremy Jameson — Mon, 07 Jul 2008 21:11:00 GMT

I'm building out a new virtualized environment using Windows Server 2008 and Hyper-V. In order to maximize performance and follow recommended best practices, I am using Server Core as the host OS.

I have to admit, doing this much administration from the command line really brings back memories from my "old Unix days" (before I switched to the Microsoft platform). I'll also admit that I've had to do quite a bit of research in order to figure out which commands need to be run in order to get a "functional" server. Now please don't misunderstand me, I don't mind that Server Core doesn't really allow you to do much with its out-of-the-box configuration (after all, that's the whole point). It simply takes a little getting used to -- and, as is usually the case, the first time takes longer than you expected.

I'll summarize some great resources I've found for working with Server Core and Hyper-V in a separate post sometime soon, but for now I wanted to share this little gotcha to save you the few minutes of frustration I spent trying to figure out what I was doing wrong.

Since I needed to reconfigure the disks on the server, I opened the Disk Management MMC console on my Windows Vista laptop and connected to the server -- or, rather I should say that I tried to connect to the server. Instead of connecting, I was greeted with the following error:

Disk Management could not start Virtual Disk Service (VDS) on DMX-CORE1-MAINT. This can happen if the remote computer does not support VDS, or if a connection cannot be established because it was blocked by Windows Firewall.

A quick Windows Live Search for "Disk Management could not start Virtual Disk Service" led me straight to LaNae Wade's post on enabling remote administration of Server Core. I then copied the command line for enabling the firewall rules for the Disk Management MMC snap-in and pasted it into my RDP session to the server:

netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes

Unfortunately, the response wasn't exactly what I expected:

Group cannot be specified along with other identification conditions.

It turns out that the quotes around Remote Volume Management were "corrupted", meaning they were converted to the angled quotes that tend to break things in very bizarre ways. The really frustrating part is that the command prompt makes the angled quotes appear just like regular quotes.

Once I edited the command line to replace the quotation marks, the command completed with the expected results:

C:\>netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes

Updated 3 rule(s).
Ok.

Performance of Virtual Machines

Jeremy Jameson — Sun, 24 Jun 2007 16:52:00 GMT

Yesterday I had to rebuild our Test environment (TEST) to replace a VM (running on VMWare) with a physical server, due to very poor performance of the VM.

When I was copying files from the VM to a different server in order to deploy our solution, robocopy reported throughput of 13.418 MegaBytes/min. Yes, that number really is [MB/min]. It took almost 10 minutes to copy the 120 MB that comprises the entire build. Ouch.

Just to make sure that I was not completely losing it, I performed the copy again. The second time throughput doubled to 28.313 MB/min. Wow. Impressive. [That's sarcasm, in case you missed it.]

About a month ago, I pointed out that our document migration process appeared to be suffering from a network bottleneck since the CPU, memory, and disk metrics on the front-end Web servers and backend database server were all well below the performance threshold values.

[Side note: Months ago, we compromised on our infrastructure design where a single 100 Mbps network adapter on each of the front-end Web servers serves all traffic except for backups -- rather than having a separate "front-end" network (for Web requests) and backend network (for SQL, AD, and other "infrastructure" traffic). In other words, all network packets, regardless of whether they are HTTP requests from Web clients or TDP requests to read from or write to SQL Server, flow through a single NIC. Therefore I have long suspected that our first bottleneck would be the network.]

During subsequent testing, Windows Server 2003 Performance Advisor reported a high number of TCP retransmits during the document migration. In order to troubleshoot the issue with a network engineer, we used a simple robocopy operation to examine the network throughput and found that we weren't seeing anywhere near the maximum throughput on a 100 Mbps network. We discovered that the duplex setting on the network switch was not set correctly. After correcting the setting on the switch, throughput went up dramatically.

Therefore, on Friday when I observed the horrible throughput numbers when robocopying files in TEST, I immediately suspected that we had a network problem again. When we called the help desk to have someone look at the network utilization, they stated that the network was "fine" and suggested that the problem was due to VMWare -- not the network.

At this point, the VMWare support team was engaged to investigate the problem. They pointed out that the CPU usage on our VM was 13%, while memory was only 7%. In other words, they only looked at two of the four resources that ultimately bottleneck any computing operation.

Prior to the horrible throughput numbers I experienced on Friday when copying files, I had noted the performance of that particular VM has been an issue for a couple of months, and stated that it was either misconfigured or else the host VMWare server was overloaded.

I have seen many VM environments where organizations use a high-end server with 4 or 8 state-of-the-art processors and "gobs" of memory and subsequently assume that they can run 4 or more VMs on this one server without a problem. However, as noted earlier, CPU and memory represent only half of the resources that can ultimately become a bottleneck. In my experience, most VMs bottleneck first on disk I/O.

Don't spend all of your money on multi-core processors and RAM, then skimp on a single 300 GB drive (even though this is more than large enough for the VMs). If you do, I can almost guarantee that your VM users will quickly complain about performance. When you start looking into the issue, you'll find that the disk queue length has "gone through the roof." [I noted in a previous post that while I am at the customer site (i.e. when I can't use the server in my basement) I run my local Microsoft Office SharePoint Server (MOSS) 2007 development VM on an external 7200 RPM USB drive to achieve acceptable performance.]

Furthermore, let's suppose that you do spec your disk configuration like the server in my basement (4x 300 GB SATA2 drives in a RAID 10 configuration). You still need to keep in mind that, depending on the number and speed of the NICs on the host server, performance may bottleneck on the network before CPU and memory become an issue.

At this point we don't know why the robocopy operation reported extremely slow performance (and honestly I don't really care anymore since we have replaced that VM with a physical server). Honestly, I don't know much about this particular VMWare host, but clearly there are some fundamental problems with the configuration.

Nevertheless, this seemed like a good opportunity to share some important information for those running VMs.

Save HUGE Amounts of Disk Space by Slipstreaming Service Packs

Jeremy Jameson — Sat, 23 Jun 2007 15:12:00 GMT

This is a little embarrassing, but I captured numerous screenshots back in April while rebuilding my SharePoint development VM, but I never got around to writing a blog post to actually share this information with anyone. Well, it's long overdue, but here it is anyway.

One day while working on my SharePoint development VM, I started getting warnings about low disk space on the virtual hard disk (VHD). This struck me as rather odd when you consider that at the time I created the VHD, I selected the default size of 16 GB, and "all" that I had loaded on the VM was Windows Server 2003, SQL Server, Visual Studio 2005, Office 2007, and Microsoft Office SharePoint Server (MOSS) 2007. True, some of these installations are rather large, but 16 GBs? Come on now, I don't think so. Besides, I had been using this VM for almost 5 months and I always seemed to have plenty of free space before.

To figure out why, gradually over time, my VM managed to consume nearly all of the available disk space, I used a utility named wdu.exe (Windows Disk Usage, I believe) that some developer at Microsoft wrote in his or her spare time and threw up on Toolbox to share with others. [On a side note, http://toolbox/ is an intranet site within Microsoft where anyone is welcome to submit useful programs and utilities -- similar to CodePlex and other open source sites, but with a much longer history.]

The following screenshot shows the breakdown of the VHD content:

Figure 1: Disk usage on VM before rebuild

See full-sized image.

So, out of the 16 GB available, my NotBackedUp folder consumed a small fraction, the Program Files folder consumed a little under 6 GB, and the Windows folder accounted for a whopping 9 GB! Huh? Hey, wait a minute, what's going on here?! Surely there must be a mistake.

Looking at the results a little closer, you can see that in the Windows folder, the Installer, ServicePackFiles, and SoftwareDistribution folders consume roughly 5 GB of space. On a typical desktop computer with 100, 200, or even 300 GB hard drives, this doesn't seem all that bad. However, when working with VMs (often configured with 16 GB drives), this is huge! Especially when you consider that I currently have 21 VMs on my server at home (I typically only have a couple running at any given time).

Since this is a development VM that I can rebuild in a matter of hours, I went ahead and created a new VHD by copying my SysPrep'ed version of Windows Server 2003 (for which I had slipstreamed SP1 and subsequently installed various other patches from Windows Update several months before).

Immediately after booting up with this new "clean" VHD, I captured the following:

Figure 2: Disk usage on VM with Windows Server 2003 SP1 slipstreamed

See full-sized image.

Ahhhh...that looks much better. The Windows folder consumed a mere 1.3 GB of the 16 GB VHD. Excellent...or so I thought.

Like any responsible computing citizen, I then proceeded to use Windows Update to install all of the latest patches, which in the case of Windows Server 2003, included Service Pack 2 for the OS. So, after a lengthy download (actually it wasn't lengthy at all, since I have WSUS running on one of my servers in the basement), my new VM was up-to-date with all of the latest patches and ready for me to begin installing SQL Server, Visual Studio, etc.

Before beginning those installs however, I thought I would take capture the hard drive usage one more time. Here is what I saw:

Figure 3: Disk usage on VM after installing Windows Server 2003 SP2

See full-sized image.

You've got be to kidding! Just by installing Windows Server 2003 SP2 and a handful of other critical patches, the Windows folder grew from 1.3 GB to over 3 GB! That's ridiculous (again speaking from a VM perspective -- on a desktop with hundreds of gigabytes of available space, I would not quibble over the additional 1.7 GB needed by Windows).

So, I decided to "refresh" my SysPrep'ed image of Windows Server 2003 by rebuilding it from a slipstreamed Windows Server 2003 SP2 (I use nLite, by the way, which works great IMHO). Here are the results after booting up with the new VHD:

Figure 4: Disk usage on VM with Windows Server 2003 SP2 slipstreamed

See full-sized image.

Comparing this with the previous snapshot from the SP1 slipstreamed version (prior to updating with SP2), you can see that the size of the Windows folder is only slightly larger. Comparing this with the previous snapshot from the SP1 slipstreamed version after installing SP2 shows a remarkable difference. No longer do we need an additional 1.7 GB of "stuff" in the Windows folder just to ensure that we have all the latest patches.

Note that there are some caveats with starting from a slipstreamed version of the OS. The most notable is that you have to be sure to carry around a copy of the slipstreamed Windows Server 2003 SP2 CD. Otherwise, when you go to build a new VM (which takes about 15-20 minutes) and subsequently add specific features, such as IIS, you will get prompted for the Windows Server 2003 SP2 CD. At this point, if you try just using one of the "vanilla" DVDs from MSDN, it won't work.

On a related note, you have to be careful with VMs even after using a slipstreamed version of the OS. The Installer folder can still grow to be very large, especially if you happen to encounter any failed installations of Visual Studio 2005 Service Pack 1 -- but that is a topic for another post (perhaps after I have some pancakes with my daughter ;-)

Virtual Server Issues and Recommendations for MOSS Virtual Environments

Jeremy Jameson — Sat, 09 Jun 2007 12:42:00 GMT

One of the tasks that I completed this week was splitting our Development environment (DEV) into multiple VMs -- one SQL Server VM, one Microsoft Office SharePoint Server (MOSS) 2007 VM for the SSP, and another MOSS 2007 VM for the front-end Web server. Up to this point, DEV was comprised of two VMs: a domain controller and another single-server MOSS configuration (combined SQL Server and MOSS SSP/Web).

[As a side note -- I am not sure if I've mentioned this in a previous post or not -- I recommend that you always use a separate VM for the domain controller. I can't tell you how many times I've seen bugs that only surface when running on a DC. You're not going to use that configuration in PROD, so why use it in DEV? You can "clamp down" the VM running Active Directory to 96 MB if memory is really tight on your Virtual Server host (e.g. your laptop). I ran that configuration for a couple of years and performance is quite acceptable. About a year ago I increased the memory on the domain controller to 256MB since I also run SMTP and POP3 services on the DC so I can have basic e-mail functionality (this is my one and only exception to isolating the domain controller).]

To be honest, until this week I had not been using DEV all that much since I do all of my development on a different VM -- a single-server MOSS configuration that I run on one of my servers in the Jameson Data Center (a.k.a. my basement). I simply check in my code to source control after developing and testing it locally.

However, during our feature walkthroughs this week, the entire team experienced firsthand how bad the performance was in DEV. The CPU was pegged nearly all of the time. Apparently the dual core CPU in my home server -- that I built last year with $1,000 worth of components from Newegg.com -- is quite a bit faster than the 8-CPU server we use to run DEV. Since each VM can only use a single CPU, we all sat idly by waiting for the VM to respond to my mouse clicks. [By the way, the walkthrough wasn't the first hint that performance was really bad in DEV. I noticed that the last deployment to DEV required almost 3-1/2 hours to complete, whereas the same process typically takes a little over an hour on my local VM (deployment involves creating a couple of Web applications, creating and configuring an SSP, and building out roughly 180 site collections).]

In order to leverage the 8 processors on the DEV server, we needed to create separate VMs for SQL, MOSS Web, and MOSS SSP. We'll probably bottleneck on disk I/O after this change (due to the fact that the DEV server only has one drive we can use for VMs) but perf will certainly be better than it was before.

During the process of building out the new VMs, I encountered a couple of problems related to Virtual Server. In order to install SQL Server and MOSS 2007 on the new VMs, I downloaded the ISO images from MSDN to the Virtual Server host. However, when I attached the SQL Server ISO image to the new VM, I encountered the following error:

setup.exe - Bad Image

The application or DLL D:\SQL Server x86\Servers\Setup\setupex.dll is not a valid Windows image. Please check this against your installation disk.

Assuming that I had somehow downloaded a corrupt ISO image from MSDN, I went ahead and downloaded it again. However, even with the freshly downloaded ISO, I got the same error.

I then tried installing VS 2005 on the new SSP VM as well as the new front-end Web server VM (figuring I could deal with the SQL issue later before running the SharePoint Configuration Wizard). On both VMs, I got the following error when starting the setup:

Visual Studio 2005 Setup

An unknown error occured while copying files to your temporary folder. Setup will now exit.

The last thing that I could think of was to copy the files from the "CD" (i.e. the mounted ISO file) to the local VHD. I opened a command prompt and started robocopying files, but I soon got the following error:

Error Performing Inpage Operation

And, no, KB 141117 was not any help.

Ugh!

Since I had never encountered these problems before when building out numerous other VMs, I figured the problem had to be with the Virtual Server host itself (i.e. the 8-proc server that runs the VMs for DEV). Therefore, I resorted to robocopying the VMs locally to my laptop (actually an external USB drive connected to my laptop), where I was able to successfully install SQL Server, MOSS, and Visual Studio 2005). I subsequently copied them back to the server.

[Another side note: if you are developing MOSS solutions on a laptop, I highly recommend that you use an external 7200 RPM USB drive for the VHD files. A few months ago, I started running my MOSS 2007 development VM on my laptop -- I had previously been running it exclusively on my home server -- and I noticed a huge impact on my productivity. The disk churn was unbearable. I tried running it on both the internal hard drive as well as an external 5400 RPM drive. Performance in both of these cases was unacceptable. I sent a message to one of our internal SharePoint DLs inquiring if others were suffering a similar slow death that I was. I suspected that the answer was to get a faster drive, but I wanted to get some evidence before placing the order.

Several people indicated that they lugged around full-size external USB hard drive enclosures in order to get acceptable perfomance. I've done that before -- I am not doing it again. I ordered a Seagate ST910021U2-RK 100GB 7200 RPM External Hard Drive and I've been quite pleased with the performance ever since. Sure, my home server with 4 drives in a RAID10 configuration is still much faster, but it is much, much better than it was.]

Anyway, back to the Virtual Server issues...I could not understand why there would be problems mounting certain ISO images on VMs running on the DEV server (before robocopying the VMs to my laptop, I was able to mount my slipstreamed Windows Server 2003 SP2 ISO image in order to install IIS, so I knew the problem wasn't with all ISO images).

I had encountered some strange errors in this Virtual Server environment before. Most notably, when I would robocopy a non-trivial amount of content from the Virtual Server host to one of the VMs, it seemed like the network adapter would get "saturated" -- if I was connected via Terminal Services to the VM then my connection would disconnect, or if I was using the VMRC then I would get "The network path was not found" errors soon after the starting the copy.

I was starting to think that perhaps there was just something fundamentally wrong with the DEV server and it was time for a complete rebuild of the server. However, I figured that I would first try reinstalling Virtual Server.

Before I uninstalled Virtual Server, I made a note of the version that was previously running in DEV (somebody else had initially configured this server long ago). The Virtual Server Administration Website displayed: 1.1.465.0 SE.

I removed Virtual Server and then downloaded the latest version from microsoft.com (it is now a free download). The Virtual Server Administration Website now displays: 1.1.465.292 EE R2.

You know what's coming next...

All of the problems appear to be resolved! After upgrading to Virtual Server 2005 R2, I can mount the ISOs and start the installation without getting the errors I encountered before. It also appears that the network adapter "saturation" problem is fixed as well (I was able to robocopy Visual Studio 2005 SP1 -- which I definitely consider to be a "non-trivial" amount of content).

Bottom line: it appears that quite a few bugs were fixed in Virtual Server 2005 R2. I highly recommend that you verify that all of your VM environments are running the latest version. I have been running R2 on my laptop as well as in the Jameson Data Center for so long that I never encountered these problems before.