Random Musings of Jeremy Jameson : Virtual Server

Performance of Virtual Machines

Jeremy Jameson — Sun, 24 Jun 2007 13:52:00 GMT

Yesterday I had to rebuild our Test environment (TEST) to replace a VM (running on VMWare) with a physical server, due to very poor performance of the VM.

When I was copying files from the VM to a different server in order to deploy our solution, robocopy reported throughput of 13.418 MegaBytes/min. Yes, that number really is [MB/min]. It took almost 10 minutes to copy the 120 MB that comprises the entire build. Ouch.

Just to make sure that I was not completely losing it, I performed the copy again. The second time throughput doubled to 28.313 MB/min. Wow. Impressive. [That's sarcasm, in case you missed it.]

About a month ago, I pointed out that our document migration process appeared to be suffering from a network bottleneck since the CPU, memory, and disk metrics on the front-end Web servers and backend database server were all well below the performance threshold values.

[Side note: Months ago, we compromised on our infrastructure design where a single 100 Mbps network adapter on each of the front-end Web servers serves all traffic except for backups -- rather than having a separate "front-end" network (for Web requests) and backend network (for SQL, AD, and other "infrastructure" traffic). In other words, all network packets, regardless of whether they are HTTP requests from Web clients or TDP requests to read from or write to SQL Server, flow through a single NIC. Therefore I have long suspected that our first bottleneck would be the network.]

During subsequent testing, Windows Server 2003 Performance Advisor reported a high number of TCP retransmits during the document migration. In order to troubleshoot the issue with a network engineer, we used a simple robocopy operation to examine the network throughput and found that we weren't seeing anywhere near the maximum throughput on a 100 Mbps network. We discovered that the duplex setting on the network switch was not set correctly. After correcting the setting on the switch, throughput went up dramatically.

Therefore, on Friday when I observed the horrible throughput numbers when robocopying files in TEST, I immediately suspected that we had a network problem again. When we called the help desk to have someone look at the network utilization, they stated that the network was "fine" and suggested that the problem was due to VMWare -- not the network.

At this point, the VMWare support team was engaged to investigate the problem. They pointed out that the CPU usage on our VM was 13%, while memory was only 7%. In other words, they only looked at two of the four resources that ultimately bottleneck any computing operation.

Prior to the horrible throughput numbers I experienced on Friday when copying files, I had noted the performance of that particular VM has been an issue for a couple of months, and stated that it was either misconfigured or else the host VMWare server was overloaded.

I have seen many VM environments where organizations use a high-end server with 4 or 8 state-of-the-art processors and "gobs" of memory and subsequently assume that they can run 4 or more VMs on this one server without a problem. However, as noted earlier, CPU and memory represent only half of the resources that can ultimately become a bottleneck. In my experience, most VMs bottleneck first on disk I/O.

Don't spend all of your money on multi-core processors and RAM, then skimp on a single 300 GB drive (even though this is more than large enough for the VMs). If you do, I can almost guarantee that your VM users will quickly complain about performance. When you start looking into the issue, you'll find that the disk queue length has "gone through the roof." [I noted in a previous post that while I am at the customer site (i.e. when I can't use the server in my basement) I run my local Microsoft Office SharePoint Server (MOSS) 2007 development VM on an external 7200 RPM USB drive to achieve acceptable performance.]

Furthermore, let's suppose that you do spec your disk configuration like the server in my basement (4x 300 GB SATA2 drives in a RAID 10 configuration). You still need to keep in mind that, depending on the number and speed of the NICs on the host server, performance may bottleneck on the network before CPU and memory become an issue.

At this point we don't know why the robocopy operation reported extremely slow performance (and honestly I don't really care anymore since we have replaced that VM with a physical server). Honestly, I don't know much about this particular VMWare host, but clearly there are some fundamental problems with the configuration.

Nevertheless, this seemed like a good opportunity to share some important information for those running VMs.

Save HUGE Amounts of Disk Space by Slipstreaming Service Packs

Jeremy Jameson — Sat, 23 Jun 2007 12:12:00 GMT

This is a little embarrassing, but I captured numerous screenshots back in April while rebuilding my SharePoint development VM, but I never got around to writing a blog post to actually share this information with anyone. Well, it's long overdue, but here it is anyway.

One day while working on my SharePoint development VM, I started getting warnings about low disk space on the virtual hard disk (VHD). This struck me as rather odd when you consider that at the time I created the VHD, I selected the default size of 16 GB, and "all" that I had loaded on the VM was Windows Server 2003, SQL Server, Visual Studio 2005, Office 2007, and Microsoft Office SharePoint Server (MOSS) 2007. True, some of these installations are rather large, but 16 GBs? Come on now, I don't think so. Besides, I had been using this VM for almost 5 months and I always seemed to have plenty of free space before.

To figure out why, gradually over time, my VM managed to consume nearly all of the available disk space, I used a utility named wdu.exe (Windows Disk Usage, I believe) that some developer at Microsoft wrote in his or her spare time and threw up on Toolbox to share with others. [On a side note, http://toolbox/ is an intranet site within Microsoft where anyone is welcome to submit useful programs and utilities -- similar to CodePlex and other open source sites, but with a much longer history.]

The following screenshot shows the breakdown of the VHD content:

Figure 1: Disk usage on VM before rebuild

See full-sized image.

So, out of the 16 GB available, my NotBackedUp folder consumed a small fraction, the Program Files folder consumed a little under 6 GB, and the Windows folder accounted for a whopping 9 GB! Huh? Hey, wait a minute, what's going on here?! Surely there must be a mistake.

Looking at the results a little closer, you can see that in the Windows folder, the Installer, ServicePackFiles, and SoftwareDistribution folders consume roughly 5 GB of space. On a typical desktop computer with 100, 200, or even 300 GB hard drives, this doesn't seem all that bad. However, when working with VMs (often configured with 16 GB drives), this is huge! Especially when you consider that I currently have 21 VMs on my server at home (I typically only have a couple running at any given time).

Since this is a development VM that I can rebuild in a matter of hours, I went ahead and created a new VHD by copying my SysPrep'ed version of Windows Server 2003 (for which I had slipstreamed SP1 and subsequently installed various other patches from Windows Update several months before).

Immediately after booting up with this new "clean" VHD, I captured the following:

Figure 2: Disk usage on VM with Windows Server 2003 SP1 slipstreamed

See full-sized image.

Ahhhh...that looks much better. The Windows folder consumed a mere 1.3 GB of the 16 GB VHD. Excellent...or so I thought.

Like any responsible computing citizen, I then proceeded to use Windows Update to install all of the latest patches, which in the case of Windows Server 2003, included Service Pack 2 for the OS. So, after a lengthy download (actually it wasn't lengthy at all, since I have WSUS running on one of my servers in the basement), my new VM was up-to-date with all of the latest patches and ready for me to begin installing SQL Server, Visual Studio, etc.

Before beginning those installs however, I thought I would take capture the hard drive usage one more time. Here is what I saw:

Figure 3: Disk usage on VM after installing Windows Server 2003 SP2

See full-sized image.

You've got be to kidding! Just by installing Windows Server 2003 SP2 and a handful of other critical patches, the Windows folder grew from 1.3 GB to over 3 GB! That's ridiculous (again speaking from a VM perspective -- on a desktop with hundreds of gigabytes of available space, I would not quibble over the additional 1.7 GB needed by Windows).

So, I decided to "refresh" my SysPrep'ed image of Windows Server 2003 by rebuilding it from a slipstreamed Windows Server 2003 SP2 (I use nLite, by the way, which works great IMHO). Here are the results after booting up with the new VHD:

Figure 4: Disk usage on VM with Windows Server 2003 SP2 slipstreamed

See full-sized image.

Comparing this with the previous snapshot from the SP1 slipstreamed version (prior to updating with SP2), you can see that the size of the Windows folder is only slightly larger. Comparing this with the previous snapshot from the SP1 slipstreamed version after installing SP2 shows a remarkable difference. No longer do we need an additional 1.7 GB of "stuff" in the Windows folder just to ensure that we have all the latest patches.

Note that there are some caveats with starting from a slipstreamed version of the OS. The most notable is that you have to be sure to carry around a copy of the slipstreamed Windows Server 2003 SP2 CD. Otherwise, when you go to build a new VM (which takes about 15-20 minutes) and subsequently add specific features, such as IIS, you will get prompted for the Windows Server 2003 SP2 CD. At this point, if you try just using one of the "vanilla" DVDs from MSDN, it won't work.

On a related note, you have to be careful with VMs even after using a slipstreamed version of the OS. The Installer folder can still grow to be very large, especially if you happen to encounter any failed installations of Visual Studio 2005 Service Pack 1 -- but that is a topic for another post (perhaps after I have some pancakes with my daughter ;-)

Virtual Server Issues and Recommendations for MOSS Virtual Environments

Jeremy Jameson — Sat, 09 Jun 2007 09:42:00 GMT

One of the tasks that I completed this week was splitting our Development environment (DEV) into multiple VMs -- one SQL Server VM, one Microsoft Office SharePoint Server (MOSS) 2007 VM for the SSP, and another MOSS 2007 VM for the front-end Web server. Up to this point, DEV was comprised of two VMs: a domain controller and another single-server MOSS configuration (combined SQL Server and MOSS SSP/Web).

[As a side note -- I am not sure if I've mentioned this in a previous post or not -- I recommend that you always use a separate VM for the domain controller. I can't tell you how many times I've seen bugs that only surface when running on a DC. You're not going to use that configuration in PROD, so why use it in DEV? You can "clamp down" the VM running Active Directory to 96 MB if memory is really tight on your Virtual Server host (e.g. your laptop). I ran that configuration for a couple of years and performance is quite acceptable. About a year ago I increased the memory on the domain controller to 256MB since I also run SMTP and POP3 services on the DC so I can have basic e-mail functionality (this is my one and only exception to isolating the domain controller).]

To be honest, until this week I had not been using DEV all that much since I do all of my development on a different VM -- a single-server MOSS configuration that I run on one of my servers in the Jameson Data Center (a.k.a. my basement). I simply check in my code to source control after developing and testing it locally.

However, during our feature walkthroughs this week, the entire team experienced firsthand how bad the performance was in DEV. The CPU was pegged nearly all of the time. Apparently the dual core CPU in my home server -- that I built last year with $1,000 worth of components from Newegg.com -- is quite a bit faster than the 8-CPU server we use to run DEV. Since each VM can only use a single CPU, we all sat idly by waiting for the VM to respond to my mouse clicks. [By the way, the walkthrough wasn't the first hint that performance was really bad in DEV. I noticed that the last deployment to DEV required almost 3-1/2 hours to complete, whereas the same process typically takes a little over an hour on my local VM (deployment involves creating a couple of Web applications, creating and configuring an SSP, and building out roughly 180 site collections).]

In order to leverage the 8 processors on the DEV server, we needed to create separate VMs for SQL, MOSS Web, and MOSS SSP. We'll probably bottleneck on disk I/O after this change (due to the fact that the DEV server only has one drive we can use for VMs) but perf will certainly be better than it was before.

During the process of building out the new VMs, I encountered a couple of problems related to Virtual Server. In order to install SQL Server and MOSS 2007 on the new VMs, I downloaded the ISO images from MSDN to the Virtual Server host. However, when I attached the SQL Server ISO image to the new VM, I encountered the following error:

setup.exe - Bad Image

The application or DLL D:\SQL Server x86\Servers\Setup\setupex.dll is not a valid Windows image. Please check this against your installation disk.

Assuming that I had somehow downloaded a corrupt ISO image from MSDN, I went ahead and downloaded it again. However, even with the freshly downloaded ISO, I got the same error.

I then tried installing VS 2005 on the new SSP VM as well as the new front-end Web server VM (figuring I could deal with the SQL issue later before running the SharePoint Configuration Wizard). On both VMs, I got the following error when starting the setup:

Visual Studio 2005 Setup

An unknown error occured while copying files to your temporary folder. Setup will now exit.

The last thing that I could think of was to copy the files from the "CD" (i.e. the mounted ISO file) to the local VHD. I opened a command prompt and started robocopying files, but I soon got the following error:

Error Performing Inpage Operation

And, no, KB 141117 was not any help.

Ugh!

Since I had never encountered these problems before when building out numerous other VMs, I figured the problem had to be with the Virtual Server host itself (i.e. the 8-proc server that runs the VMs for DEV). Therefore, I resorted to robocopying the VMs locally to my laptop (actually an external USB drive connected to my laptop), where I was able to successfully install SQL Server, MOSS, and Visual Studio 2005). I subsequently copied them back to the server.

[Another side note: if you are developing MOSS solutions on a laptop, I highly recommend that you use an external 7200 RPM USB drive for the VHD files. A few months ago, I started running my MOSS 2007 development VM on my laptop -- I had previously been running it exclusively on my home server -- and I noticed a huge impact on my productivity. The disk churn was unbearable. I tried running it on both the internal hard drive as well as an external 5400 RPM drive. Performance in both of these cases was unacceptable. I sent a message to one of our internal SharePoint DLs inquiring if others were suffering a similar slow death that I was. I suspected that the answer was to get a faster drive, but I wanted to get some evidence before placing the order.

Several people indicated that they lugged around full-size external USB hard drive enclosures in order to get acceptable perfomance. I've done that before -- I am not doing it again. I ordered a Seagate ST910021U2-RK 100GB 7200 RPM External Hard Drive and I've been quite pleased with the performance ever since. Sure, my home server with 4 drives in a RAID10 configuration is still much faster, but it is much, much better than it was.]

Anyway, back to the Virtual Server issues...I could not understand why there would be problems mounting certain ISO images on VMs running on the DEV server (before robocopying the VMs to my laptop, I was able to mount my slipstreamed Windows Server 2003 SP2 ISO image in order to install IIS, so I knew the problem wasn't with all ISO images).

I had encountered some strange errors in this Virtual Server environment before. Most notably, when I would robocopy a non-trivial amount of content from the Virtual Server host to one of the VMs, it seemed like the network adapter would get "saturated" -- if I was connected via Terminal Services to the VM then my connection would disconnect, or if I was using the VMRC then I would get "The network path was not found" errors soon after the starting the copy.

I was starting to think that perhaps there was just something fundamentally wrong with the DEV server and it was time for a complete rebuild of the server. However, I figured that I would first try reinstalling Virtual Server.

Before I uninstalled Virtual Server, I made a note of the version that was previously running in DEV (somebody else had initially configured this server long ago). The Virtual Server Administration Website displayed: 1.1.465.0 SE.

I removed Virtual Server and then downloaded the latest version from microsoft.com (it is now a free download). The Virtual Server Administration Website now displays: 1.1.465.292 EE R2.

You know what's coming next...

All of the problems appear to be resolved! After upgrading to Virtual Server 2005 R2, I can mount the ISOs and start the installation without getting the errors I encountered before. It also appears that the network adapter "saturation" problem is fixed as well (I was able to robocopy Visual Studio 2005 SP1 -- which I definitely consider to be a "non-trivial" amount of content).

Bottom line: it appears that quite a few bugs were fixed in Virtual Server 2005 R2. I highly recommend that you verify that all of your VM environments are running the latest version. I have been running R2 on my laptop as well as in the Jameson Data Center for so long that I never encountered these problems before.