Random Musings of Jeremy Jameson : Windows Server

A Simple Backup Solution

Jeremy Jameson — Mon, 09 Nov 2009 13:44:00 GMT

As I've mentioned before, I don't spend much money or time maintaining the "Jameson Datacenter" (a.k.a. my home lab). However, that doesn't mean that I treat my infrastructure lightly.

In previous posts, I've covered many of the Group Policy objects that I use to minimize the maintenance effort associated with running more than a dozen servers (mostly virtual). In this post, I'll provide the details on how I backup these servers.

I should preface this by saying this is not meant to be an "enterprise-level" backup solution. Rather it is simply meant to provide a cheap (actually free) and easy solution to the problem of ensuring you can recover from data loss. Note that data loss rarely occurs through some sort of hardware failure or Act of God (as the insurance folks like to put it). Rather the majority of the time someone accidentally overwrites or deletes a file -- or, gasp, a complete folder hierarchy -- and you subsequently need to restore the data from a backup.

For as long as I can remember, Windows Server has included the NTBackup utility. I'm guessing from the name that this has been around since the days of Windows NT 3.1, but honestly I don't believe I even started running Windows NT until version 3.5. Or was it 3.51? I can't remember. Anyway, I certainly haven't been running NTBackup since then.

Here is the simple batch file that I use to perform scheduled backups:

@echo off

setlocal

set BACKUP_TYPE=normal

if ("%1") NEQ ("") set BACKUP_TYPE=%1

for /f "tokens=2-4 delims=/ " %%i in ('date /t') do set currentDate=%%k-%%i-%%j
for /f "tokens=1-2" %%i in ('time /t') do set currentTime=%%i %%j
set BACKUP_TIMESTAMP=%currentDate%-%currentTime:~0,2%-%currentTime:~3,2%-%currentTime:~6,2%

set BACKUP_FILE=D:\NotBackedUp\Backups\Backup-%BACKUP_TYPE%-%BACKUP_TIMESTAMP%.bkf

:: ----------------------------------------------------------------------------
call :LogMessage "Starting backup..."
call :LogMessage "BACKUP_TYPE: %BACKUP_TYPE%"
call :LogMessage "BACKUP_FILE: %BACKUP_FILE%"

C:\WINDOWS\system32\ntbackup.exe backup C:\BackedUp /n "Backup created %BACKUP_TIMESTAMP%" /m %BACKUP_TYPE% /j "Backup (%BACKUP_TYPE%)" /f "%BACKUP_FILE%"
if %ERRORLEVEL% neq 0 goto Errors

call :LogMessage "Successfully completed backup."

goto :eof

:: ----------------------------------------------------------------------------
::
:LogMessage

REM Strip leading and trailing quotes and then display message with timestamp
set MESSAGE=%1
set MESSAGE=%MESSAGE:~1,-1%

for /f "tokens=2-4 delims=/ " %%i in ('date /t') do set currentDate=%%k-%%i-%%j
for /f "tokens=1-2" %%i in ('time /t') do set currentTime=%%i %%j
echo %currentDate% %currentTime% - %MESSAGE%

goto :eof

:: ----------------------------------------------------------------------------
::
:Errors

echo Warning! One or more errors detected.

If you've seen any of my scripts before, then you'll quickly notice the typical LogMessage "function" that I use to write messages prefixed with a timestamp. For example here's the output from the log for this morning's backup:

2009-11-09 12:30 AM - Starting backup...
2009-11-09 12:30 AM - BACKUP_TYPE: differential
2009-11-09 12:30 AM - BACKUP_FILE: D:\NotBackedUp\Backups\Backup-differential-2009-11-09-12-30-AM.bkf
2009-11-09 12:31 AM - Successfully completed backup.

I use similar token parsing of the output from the date and time system commands to generate the name of the backup file (e.g. Backup-differential-2009-11-09-12-30-AM.bkf).

Also note that the type of backup (e.g. normal or differential) can be specified as a parameter when running the batch file. This is really powerful for scheduling different types of backups on various schedules.

Here are the scheduled backups on one of my servers (BEAST):

Scheduled Backups on BEAST
Name	Schedule
Daily Backup	At 12:00 PM every day
Differential Backup	At 12:30 AM every day
Full Backup	At 1:00 AM every Sun of every week

The Daily Backup task is configured as follows:

Run: C:\BackedUp\Backup.cmd daily >> Backup.log
Start in: C:\BackedUp
Run as: TECHTOOLBOX\svc-backup

Note that I specifically chose the middle of the day to perform daily backups so that I could potentially recover a file that was created in the morning but mistakenly deleted in the afternoon. I suppose I could schedule incremental backups throughout the day, but honestly, I haven't seen the need given my situation.

Also note that the service account that I use for backups (TECHTOOLBOX\svc-backup) is only a member of the Backup Operators group. It is not a member of the Administrators group.

Consequently there's a known issue with running batch files using scheduled tasks due to out-of-the-box security restrictions on cmd.exe:

"Access is denied" error message when you run a batch job on a Windows Server 2003-based computer

http://support.microsoft.com/kb/867466

Lastly, note that I am doing a simple disk-to-disk backup on my servers, so if there's a fire in the Jameson Datacenter (i.e. my basement) and I lose these servers completely then I'm going to be "hurtin' for certain." However should there ever be a fire in my basement (Heaven forbid), I'm going to be worried about a lot more than just restoring my data from backup. Note that I keep copies of the really important stuff (e.g. digital photos and home videos of my family) on DVDs at my parents' house.

I've read that there's a new backup tool in Windows Server 2008, so I suppose one of these days I'll need to get around to upgrading my backup solution ;-)

Configure IntelliMirror Using Group Policy

Jeremy Jameson — Wed, 21 Oct 2009 12:57:00 GMT

Yet another Group Policy object that I use in the "Jameson Datacenter" (a.k.a. my home lab) is one to automatically configure roaming profiles and redirect the Desktop and Documents folders to a server(a.k.a. "IntelliMirror").

Even though I don't have many users in my Active Directory domain -- it's not like I have eight kids, just one -- I still want to keep user data centrally managed on a server that I backup regularly. Besides, I find it really frustrating to have some items on your desktop on one computer, but a different set of desktop items on another computer (or VM).

To automatically configure this in the "Jameson Datacenter", I defined a Group Policy (named Default User Data and Settings Policy) with the following settings:

User Configuration
- Policies
  - Windows Settings
    - Folder Redirection
      - AppData(Roaming)
        
        Setting: Basic (Redirect everyone's folder to the same location)
        
        Path: \\beast\Users$\%USERNAME%\Application Data
        
        Options
        
        Grant user exclusive rights to AppData(Roaming): Enabled
        
        Move the contents of AppData(Roaming) to the new location: Enabled
        
        Also apply redirection policy to Windows 2000, Windows 2000 server, Windows XP, and Windows Server 2003 operating systems: Enabled
        
        Policy Removal Behavior: Leave contents
      - Desktop
        
        Setting: Basic (Redirect everyone's folder to the same location)
        
        Path: \\beast\Users$\%USERNAME%\Desktop
        
        Options
        
        Grant user exclusive rights to Desktop: Enabled
        
        Move the contents of Desktop to the new location: Enabled
        
        Also apply redirection policy to Windows 2000, Windows 2000 server, Windows XP, and Windows Server 2003 operating systems: Enabled
        
        Policy Removal Behavior: Leave contents
      - Documents
        
        Setting: Basic (Redirect everyone's folder to the same location)
        
        Path: \\beast\Users$\%USERNAME%\Documents
        
        Options
        
        Grant user exclusive rights to Documents: Enabled
        
        Move the contents of Documentsto the new location: Enabled
        
        Also apply redirection policy to Windows 2000, Windows 2000 server, Windows XP, and Windows Server 2003 operating systems: Enabled
        
        Policy Removal Behavior: Leave contents
      - Music
        
        Setting: Follow the Documents folder
      - Pictures
        
        Setting: Follow the Documents folder
      - Videos
        
        Setting: Follow the Documents folder

Note

Those of you that have a very keen eye (and also a photographic memory) might recall that in a previous post, I listed BEAST as a database server (it is currently running SQL Server 2005). Yes, it's true, I'm breaking one of my own cardinal sins by having a SQL Server also serve as a file server. I don't recommend doing this unless, like me, you are trying to go as cheap as possible -- and, even then, only for a lab environment like mine.

In order to allow users access to create their own folders on \\BEAST\Users$, I have configured the following permissions on C:\BackedUp\Users:

Domain Users
- Apply onto: This folder only
- Permissions
  - List Folder / Read Data
  - Create Folders / Append Data
CREATOR OWNER
- Apply onto: Subfolders and files only
- Permissions
  - Full Control

I also created a hidden share for the C:\BackedUp\Users folder and granted Full Control to Authenticated Users (since the NTFS permissions above ultimately determine the level of access for all users).

Thus when a new user logs in for the first time, a corresponding folder is created on the server and all of the user's files are stored on the server.

Eliminate MBSA Warnings Using Default Security Settings Policy

Jeremy Jameson — Wed, 21 Oct 2009 11:57:00 GMT

Another Group Policy object that I use in the "Jameson Datacenter" (a.k.a. my home lab) is one that I created a couple of years ago in order to eliminate various warnings from the Microsoft Baseline Security Advisor (MBSA).

To automatically change the default security settings in the "Jameson Datacenter", I defined a Group Policy (named Default Security Settings Policy) with the following settings:

Computer Configuration
- Policies
  - Windows Settings
    - Security Settings
      - Account Policies
        
        Password Policy
        
        Maximum password age: 60 days
        
        Minimum password age: 1 day
        
        Minimum password length: 8 characters
      - Local Policies
        
        Security Options
        
        Network security: LAN Manager authentication level: Send NTLMv2 response only. Refuse LM & NTLM
      - System Services
        
        TlntSvr
        
        Startup Mode: Disabled

I don't know about you, but I haven't used Telnet in almost fifteen years -- back when I used to work on Unix systems ;-)

This Group Policy is linked to the entire domain (i.e. corp.technologytoolbox.com).

Note that I still use the Default Domain Controllers Policy to configure the security settings on the domain controllers (and thus domain accounts). In other words, the settings noted above only affect local accounts (e.g. COLOSSUS\Administrator, not TECHTOOLBOX\jjameson).

Managing Group Membership via Group Policy - Part 2

Jeremy Jameson — Thu, 15 Oct 2009 13:04:00 GMT

In Part 1 of this post, I explained the Group Policy object (named Development - Restricted Groups Policy) that I use for enforcing group membership on a specific set of servers. As a follow-up to that post, I also want to cover an alternate method of managing group membership.

In the previous scenario -- i.e. ensuring that Development team leads always have administrative access to servers in their Development Integration Environment (DEV) -- we actually wanted to restrict the members of the local Administrators group on all servers in DEV. However, what if we need to address a slightly different scenario in which we want a specific user or group to always be a member of the local Administrators group -- in addition to other group members (that vary by server)?

For example, consider the fact that I use Systems Center Operations Manager (SCOM) in order to monitor the various physical and virtual servers in the the "Jameson Datacenter" (a.k.a. my home lab). One of the things I learned while deploying SCOM is that it is, um, challenging to deploy it in a least privilege configuration -- or at least for someone who primarily considers himself an AppDev (Application Development) flavor of Microsoft consultant.

At a bare minimum, your SCOM service account needs to be a member of the Performance Monitor Users group on each monitored server. Rather than forcing myself to configure this on all of my existing servers as well on new servers and VMs that I will undoubtedly add in the future, I decided to apply this change using Group Policy instead.

However, in this scenario, I don't want to restrict the members of the Performance Monitor Users group on each monitored server. Rather I simply want to ensure that the SCOM service account is a member of this group in addition to any other members.

To address this scenario, I created a startup script called EnsureLocalGroupMembership.cmd in the following folder:

file://corp.technologytoolbox.com/SysVol/corp.technologytoolbox.com/Policies/{GUID}/Machine/Scripts/Startup/OperationsManager

The contents of the script are actually quite trival:

net localgroup "Performance Monitor Users" TECHTOOLBOX\svc-mom-action /add

Note

Prior to deploying SCOM 2007 in the "Jameson Datacenter" I used its predecessor -- Microsoft Operations Manager (MOM) -- and thus had already created a service account named svc-mom-action.

To force this startup script to run on all monitored servers, I created a Group Policy object (named Default Operations Manager Policy) and linked it to the corresponding OU.

Here are the settings for the Group Policy:

Computer Configuration
- Policies
  - Windows Settings
    - Scripts (Startup/Shutdown)
      - Startup
        
        Name: OperationsManager\EnsureLocalGroupMembership.cmd

By linking this Group Policy to the appropriate OU (i.e. IT/Resources/Servers) the SCOM service account is ensured to be a member of the local Performance Monitor Users group on each monitored server. Voilà!

Enforcing Windows Update via Group Policy

Jeremy Jameson — Thu, 15 Oct 2009 12:15:00 GMT

Another Group Policy object that I use in the "Jameson Datacenter" (a.k.a. my home lab) is one to automatically configure Windows Update on all computers in the domain. This ensures that each server or workstation downloads updates from COLOSSUS (one of my VMs that is running Windows Server Update Services) rather than having each computer download, for example, a 577 MB service pack directly from the Internet. It also ensures that only the updates that I have approved through WSUS are applied.

To automatically configure Windows Update in the "Jameson Datacenter", I have defined a Group Policy (named Default Windows Update Policy) with the following settings:

Computer Configuration
- Policies
  - Administrative Templates
    - Windows Components
      - Windows Update
        
        Configure Automatic Updates
        
        Enabled
        
        Configure automatic updating: 4 -Auto download and schedule the install
        
        Scheduled install day: 0 - Every day
        
        Scheduled install time: 03:00
        
        Specify intranet Microsoft update service location
        
        Enabled
        
        Set the intranet update service for detecting updates: http://colossus
        
        Set the intranet statistics server: http://colossus

By linking this Group Policy to the entire domain (i.e. corp.technologytoolbox.com) Windows Update is automatically configured as soon as new computers are joined to the domain and rebooted.

This enables me to spin up new VMs with very little effort. More importantly, it takes less than a half hour to get a new Windows Server 2008 VM with all the latest patches (since I start from a SysPrep'ed VHD with Windows Server 2008 Service Pack 2).

Managing Group Membership via Group Policy - Part 1

Jeremy Jameson — Thu, 15 Oct 2009 11:45:00 GMT

In yesterday's post I covered one of the Group Policy objects that I use in the "Jameson Datacenter" (a.k.a. my home lab), specifically one that automatically enables Remote Desktop (Terminal Services) whenever I add a new server to my Active Directory domain. This post introduces a Group Policy object that enforces group membership on a specific set of servers.

To understand the value of this kind of Group Policy, consider a scenario where you have a Development organization that manages its own Development Integration Environment (DEV). Standard operating procedures state that certain individuals within the Development organization -- say, for example, the team leads -- are given full administrative access to the servers in this environment. These individuals are members of the Development Admins domain group. In order to avoid having to explicitly add this domain group to the local Administrators group on each server in DEV, you can instead manage the group membership through Group Policy. Thus, whenever the Development team "spins up" a new server for their environment, all of the Development team leads are granted administrative access as soon as the server is joined to the domain.

To address this scenario in the "Jameson Datacenter", I have defined a Group Policy (named Development - Restricted Groups Policy) with the following settings:

Computer Configuration
- Policies
  - Windows Settings
    - Security Settings
      - Restricted Groups
        
        Group Name: Administrators
        
        Members: TECHTOOLBOX\Development Admins, TECHTOOLBOX\Domain Admins

By linking this Group Policy to the appropriate OU (i.e. Development/Resources/Servers) the members of the local Administrators group are automatically configured as soon as I join a new DEV server to the domain and reboot.

See Part 2 of this post for an alternate method of managing group membership through Group Policy.

Enabling Remote Desktop via Group Policy

Jeremy Jameson — Wed, 14 Oct 2009 12:53:00 GMT

In a previous post, I provided some details on the "Jameson Datacenter" (a.k.a. my home lab). In a follow-up post, I also discussed the Active Directory domain structure and mentioned how I use the Group Policy feature of Active Directory to "effortlessly" configure new servers.

For example, I have defined a Group Policy (named Enable Terminal Services Policy) with the following settings:

Computer Configuration
- Policies
  - Windows Settings
    - Security Settings
      - Windows Firewall with Advanced Security
        
        Inbound Rules
        
        Remote Desktop (TCP-In)
        
        Enabled: Yes
        
        Action: Allow
  - Administrative Templates
    - Windows Components
      - Terminal Services
        
        Terminal Server
        
        Connections
        
        Allow users to connect remotely using Terminal Services: Enabled

By linking this Group Policy to the appropriate OUs (e.g. Development/Resources/Servers) I do not have to manually enable Remote Desktop connections on each new server (e.g. a new SharePoint development VM). Instead this is automatically configured as soon as I join a server to the domain and reboot.

I'll cover some of the other Group Policy objects in subsequent posts.

Why choose "Server Core" installation of Windows Server 2008?

Jeremy Jameson — Fri, 05 Jun 2009 03:12:00 GMT

If you ever find yourself looking for reasons or evidence why you should choose the "Server Core" installation option for Windows Server 2008, try searching for the following:

"Windows Server 2008 Server Core installation not affected" site:microsoft.com/technet/security

You will find page after page of results similar to the following:

Microsoft Security Bulletin MS08-054 – Critical
Windows Server 2008 server core installation not affected. The vulnerability addressed by this update does not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/Bulletin/MS08-054.mspx · Cached page

Microsoft Security Bulletin MS08-078 - Critical
Windows Server 2008 server core installation not affected. The vulnerabilities addressed by this update do not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/bulletin/ms08-078.mspx · Cached page

Microsoft Security Bulletin MS08-053 – Critical
Windows Server 2008 server core installation not affected. The vulnerability addressed by this update does not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/Bulletin/MS08-053.mspx · Cached page

Microsoft Security Bulletin MS08-024 - Critical: Cumulative Security ...
Windows Server 2008 server core installation not affected. The vulnerabilities addressed by these updates do not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/Bulletin/MS08-024.mspx · Cached page

Microsoft Security Bulletin MS08-052 – Critical
Windows Server 2008 Server Core installation not affected. The vulnerabilities addressed by this update do not affect supported editions of Windows Server 2008 if Windows Server ...
www.microsoft.com/technet/security/bulletin/ms08-052.mspx · Cached page

...

Seeing all these results is refreshing when I think back on the challenges I had to overcome when building out my first Hyper-V server using the Server Core installation, such as configuring remote administration of Hyper-V, or whenever I need to view PerfMon data on a Server Core machine (which is trivial on a "Full" installation, but not quite so easy on a Server Core installation).

Thanks to Ana Paula Moreira Franco, a Senior Consultant with Microsoft Consulting Services in Brazil, for pointing out these search terms.

Update on Patching and Disk Space Usage

Jeremy Jameson — Wed, 03 Jun 2009 17:09:00 GMT

About a year ago, I wrote a post about saving huge amounts of disk space by slipstreaming service packs. Having just been through an ordeal installing Windows Server 2008 SP2, I thought it would be worthwhile to provide an update (since that original post refers to disk space usage with Windows Server 2003).

Note that since my original post, I have switched from using Virtual Server in favor of Hyper-V. Among other things, this allows me to run x64 virtual machines (VMs). Many months ago, I consolidated numerous physical machines onto a couple of "Server Core" machines running Hyper-V. In that time, I've also switched to running Windows Vista x64 on my primary desktop and Windows Server 2008 x64 on my laptop.

One of the things that I've noticed is that x64 versions of the operating system tend to use more disk space than their corresponding x86 equivalents. In particular, the "side-by-side" folder (WinSxS) is typically significantly larger on x64 installations. The storage differences are negligible on my physical machines, but on VMs I make a deliberate effort to "clamp down" the size of the VHDs. This can save me considerable time when copying VHDs from one server to another or from an internal hard drive to an external hard drive whenever I need to take one or more of them "on the road" with me.

Minimizing VHD sizes also allows me to cram more VMs onto my 100 GB external drive [I know, these days this isn't very big from a capacity perspective, but at least it's 7200 RPM (a must for running VMs) and it isn't nearly as bulky as my larger drive enclosure. It also doesn't require a separate power supply either.]

Here is a baseline of the disk space usage on a Windows Server 2008 Standard x64 VM:

Figure 1: Disk usage on Windows Server 2008 Standard x64 VM (baseline)

See full-sized image.

Notice that the total disk usage is about 7.5 GB and the Windows folder consumes a little over 7 GB. Also note that Windows Server 2008 included SP1 (i.e. Microsoft slipstreamed it into the initial installation in order to simplify the servicing model for both Windows Server 2008 and Windows Vista).

I then immediately installed Windows Server 2008 SP2 and captured the following:

Figure 2: Disk usage on Windows Server 2008 x64 VM (after installing SP2)

See full-sized image.

Observe that the Windows folder now consumes a little over 10 GB of storage. Ouch...3 GB for a service pack. That seems a little, um, irritating -- for VMs, anyway. Obviously for physical machines with 100+ GB hard drives, the additional space is trivial.

I then ran the Windows Component Clean tool (COMPCLN.exe) as described in my previous post, which reclaimed approximately 900 MB of space.

Figure 3: Disk usage on Windows Server 2008 x64 VM (after installing SP2 and running COMPCLN.exe)

See full-sized image.

Notice that the Windows folder now consumes about 8.5 GB of space (but the overall free space on the 20 GB VHD increased from roughly 9.7 GB to 10.6 GB). In other words, SP2 adds roughly 3 GB, but COMPCLN.exe trims this to a little over 2 GB.

Lastly, I want to point out the current disk space usage on COLOSSUS -- an x64 VM that I run WSUS (Windows Server Update Services) for managing patches and updates on the various machines in the "Jameson Datacenter." Note that this server only has WSUS (which requires IIS and SQL Server) but nothing else. Consequently, after installing Windows Server 2008 SP2 and running COMPCLN.exe, I was hoping it would have comparable disk space usage to that shown in Figure 3 (after deducting the space used by the WSUS database, of course).

Unfortunately, it isn't even close, as shown in the following figure.

Figure 4: Disk usage on a patched WSUS server (after installing SP2)

See full-sized image.

Notice that the Windows folder on COLOSSUS consumes almost 16.5 GB of space, of which roughly 10.5 GB is used by the WinSxS folder.

The lesson here is that you should expect some "bloat" in the Windows folder over time (largely due to the WinSxS folder), and while the Windows Component Clean tool (COMPCLN.exe) undeniably reclaims some hard drive space after installing SP2, it's definitely not the same as starting with a "fresh" SP2 install.

Reclaiming Disk Space After Installing Service Pack 2

Jeremy Jameson — Tue, 02 Jun 2009 16:39:00 GMT

In yesterday's post, I noted the errors I encountered when trying to install Windows Server 2008 Service Pack 2 (SP2) due to "insufficient" disk space. I ended up having to expand numerous VHDs (one for each of my VMs running Windows Server 2008 x64) in order to have roughly 5 GB of free space to allow SP2 to install.

Note that SP2 certainly didn't use the 5 GB of free space, but apparently it insists on having that much for some "factor of safety" during the install.

Note that SP2 for Windows Server 2008 and Windows Vista include a new tool which helps recover hard disk space: the Windows Component Clean Tool (COMPCLN.exe). According to the Windows Server 2008 SP2 Deployment Guide this tool permanently removes the files that are archived after Windows Vista SP2 or Windows Server 2008 SP2 is applied. It also removes the files that were archived after Windows Vista SP1 was applied, if they are found on the system.

The deployment guide also states that "running this tool is optional" -- however I, personally, highly recommend it. Of course, you have to weigh this decision with the probability that you will need to uninstall SP2 -- which, in my case, is essentially "zilch."

On my x64 VMs, running COMPCLN.exe freed up roughly 800 MB of disk space. This might not seem like a lot -- given that many hard drives these days exceed 1 TB -- but it is very significant on VHDs that you are trying to keep as "lean" as possible.

Note that I'm still a little disappointed in the disk space requirements for SP2. Up until yesterday I had been running each of my domain controller VMs on 16 GB VHDs, and many of my other VMs on 20-22 GB VHDs. Note that on some of those VMs, disk space was certainly getting tight -- due to all of the patches that have been installed since I originally built them -- but they were functioning just fine until Windows Server 2008 SP2 came along. Although I have to say that on a 20 GB VHD, a WinSxS folder consuming 8-10 GB seems a little ridiculous.

I ended up having to expand the system VHD for my MOSS 2007 development VM to 25 GB -- which I really didn't want to do -- since it essentially gives the SharePoint Unified Logging System another 5 GB of free space to fill with useless log messages. [That is, of course, until I get the April 2009 Cumulative Update installed -- which supposedly fixes this issue.]

Errors Installing Windows Server 2008 SP2

Jeremy Jameson — Mon, 01 Jun 2009 17:14:00 GMT

Last week I approved Windows Server 2008 Service Pack 2 (SP2) and Windows Vista SP2 on my local WSUS (Windows Server Update Services) server. My expectation was that the various physical and virtual machines in the "Jameson Datacenter" would subsequently install the update around 3:00 AM the following morning.

Unfortunately when I examined the WSUS console this morning, I found a number of computers reporting errors. After selecting one of the failed computers, I discovered the following:

Windows Server 2008 Service Pack 2 Standalone x64-based Systems (KB948465) - English, French, German, Japanese, Spanish

Event reported at 6/1/2009 3:02 AM:

Installation Failure: Windows failed to install the following update with error 0x80070643: Windows Server 2008 Service Pack 2 Standalone x64-based Systems (KB948465) - English, French, German, Japanese, Spanish.

A quick search for 0x80070643 led to the following KB article:

You receive error code 0x80070643 or error code 0x643 when you use the Windows Update or Microsoft Update Web sites to install updates

http://support.microsoft.com/kb/958052

Unfortunately, this error code indicates "generic errors that basically state that an error was encountered by Windows Installer." Rather than immediately following the steps in the KB article to enable logging and try to reproduce the problem, I decided to take a quick look at the event logs and discovered the following:

Log Name: System
Source: Microsoft-Windows-Service Pack Installer
Date: 6/1/2009 3:02:00 AM
Event ID: 8
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: dazzler.corp.technologytoolbox.com
Description:
Service Pack installation failed with error code 0x800f0826.

Another quick search for 0x800f0826 suggested that the problem might be due to insufficient disk space. However, I checked the free space on DAZZLER and observed that it had 4.09 GB free. Surely, 4 GB of disk space is sufficient to install Windows Server 2008 SP2!

Then I came across the following:

Windows Server 2008 SP2 Deployment Guide

http://technet.microsoft.com/en-us/library/dd351467(WS.10).aspx

Here are the disk space requirements according to the deployment guide:

Disk Space Requirements for Windows Server 2008 SP2
Installation method	Approximate disk space requirements
Stand-alone installation	x86-based: 1.8 GB to 2.9 GB x64-based: 3.2 GB to 4.9 GB ia64-based: 2.9 GB to 3.2 GB
Windows Update	x86-based: 350 MB x64-based: 600 MB ia64-based: 2.25 GB
Integrated installation	x86-based: 9 GB x64-based: 12 GB ia64-based: 13 GB

Crikey! According to this table, DAZZLER needs up to 4.9 GB of free space in order to install the Service Pack (since it is an x64 VM). Wow!

This, quite honestly, seems absolutely absurd!

However, given that it is relatively easy to expand a VHD, I decided to just go ahead and add another 2 GB to the system drive on the VM and see if that eliminated the issue. After all, as long as SP2 doesn't actually use 4.9 GB of space, then the actual physical space consumed by the VHD should be significantly less than 5 GB.

Using Hyper-V Manager, I expanded the VHD from 20 GB to 22 GB and then started the VM up again. I then logged into the VM and used the Disk Management console to extend the volume to include the additional 2 GB of available storage.

Next, I kicked off the installation of Windows Server 2008 SP2 again. This time the installation completed without error. Woohoo!

Note that DAZZLER is a dedicated Team Foundation Server "build server" -- so I didn't expect that it would need lots of disk space. In fact, it didn't -- at least not until Windows Server 2008 SP2 came along.

Now all I have to do is add some more disk space to the other servers that are failing to install SP2.

Redirecting stderr to stdout

Jeremy Jameson — Fri, 27 Mar 2009 14:29:00 GMT

Yesterday I replied to an email from a teammate in which I incorrectly stated that you can't redirect stderr to stdout in DOS -- er, I mean a command window in Microsoft Windows.

I would have sworn the last time I tried something like the following in Windows Server 2003 (a couple of years ago), I got an error message:

"Redeploy Features.cmd" > tmp.log 2>&1

Fortunately another teammate on the thread, Prashant Nayak, experimented with this and confirmed that it actually does work. Thanks, Prashant, for setting the record straight!

Server Core Installation - "Accessing Windows in Notification period"

Jeremy Jameson — Wed, 05 Nov 2008 17:41:00 GMT

I had a rather rough start this morning.

When I attempted to boot up my primary workstation and login, I kept encountering a problem loading my roaming profile. I could login to Vista, but my desktop was blank and I kept getting prompted to enter my credentials to access the server (BEAST) where my user profile is stored. My first thought was that BEAST was either locked up or disconnected from the network (although I was really puzzled because the only times I can remember this server failing is when we experience a power outage). Plus, if the server was actually down I should have seen a message about not being able to load my profile -- not a dialog prompting for my credentials.

Using my KVM switch, I toggled over to the BEAST console and logged in. I noticed the login took considerably longer than usual, but using a couple of ping operations from the command-line, I verified the network was up and running.

I then tried to switch over to my other server running Hyper-V (ROGUE), which hosts my domain controller -- or actually, I should say domain controllers, since I recently built out a second DC when I virtualized my original domain controller (XAVIER). In other words, ROGUE is now running XAVIER1 and XAVIER2 -- two Windows Server 2008 VMs with the Active Directory Domain Services role enabled.

Unfortunately, I got "no love" from ROGUE when I tried to login. The screen was blank and no amount of CTRL+ALT+DELETE combinations could resuscitate it. I then checked the physical server to ensure the it hadn't somehow been turned off. Unfortunately, the "lights were on, but nobody was home", so I forced a hard reboot. (I really hate doing that -- especially on a Hyper-V server running 5 VMs. Oh well, desperate times call for...well, you know the rest.

After ROGUE rebooted, I noticed that I still had problems accessing it. When I was finally able to examine the event log, I discovered the following:

Log Name: Application
Source: Microsoft-Windows-Winlogon
Date: 11/5/2008 6:04:29 AM
Event ID: 4104
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: ROGUE.corp.technologytoolbox.com
Description:
Accessing Windows in Notification period.

Ugh...from this I determined that it must have been exactly 61 days this morning since I rebuilt ROGUE with Windows Server 2008 (Server Core) and virtualized a couple of other physical servers. Apparently, I neglected to perform a vital configuration step -- specifically, setting the product key.

For a Server Core installation, you need to use the Windows Software Licensing Management Tool (slmgr.vbs) to enter the product key:

slmgr.vbs -ipk <Product Key>

After changing the product key, you then need to activate Windows:

slmgr.vbs -ato

In my opinion, the fact that the Windows Server 2008 setup does not prompt for a product key is problematic. At least this is true for the MSDN version (which I use to run the "Jameson Datacenter" -- a.k.a. my home lab). I understand that many organizations use volume licensing (and volume activation), so I certainly can see why entering the product key at installation time should be optional. I would just prefer that it wasn't skipped altogether. Also, I know that I'm not the only one who has found it a little confusing to enter MSDN product keys for Windows Server 2008 after installation.

Anyway, enough ranting -- I need to get back to my "day job" ;-)

Some Gotchas with Remote Administration of Hyper-V

Jeremy Jameson — Thu, 28 Aug 2008 16:41:00 GMT

As I mentioned in my previous post, last month I built out a new virtual environment using Hyper-V on Server Core. Since you can't run MMC -- and therefore Hyper-V Manager -- on Server Core, you need to use remote administration to manage the VMs.

John Howard's blog series on Hyper-V Remote Management is by far the definitive source for getting Hyper-V up and running on Server Core. It provides an excellent step-by-step guide for enabling remote administration, opening various firewall ports, configuring DCOM permissions (if you don't want to use admin accounts), etc. If you haven't yet at least scanned John's posts, I highly recommend doing so before embarking on the Hyper-V on Server Core path.

There is one snag, however, that I want to point out with regards to John's scenarios.

The instructions in John’s blog posts work when:

both the Hyper-V server and the client are in WORKGROUP mode, or
when the client and server are members of the same domain or trusted domains, or
when the Hyper-V server is in WORKGROUP but the client is in a domain

[Note that I personally verified the second and third scenarios above while initially building out my Hyper-V server; I am trusting that the first scenario works based on John’s posts.]

However, if the client is a member of DOMAIN1 and the Hyper-V server is a member of DOMAIN2 – and there is no trust relationship between DOMAIN1 and DOMAIN2 – then Hyper-V Manager pukes with a message about not being able to connect to the RPC server. Also note that in this scenario Disk Management pukes as well with the infamous error message:

RPC server is unavailable

To picture this scenario, imagine you have a Hyper-V server joined to your internal domain, but now I come along and try to use Hyper-V Manager from my laptop which is joined to the internal Microsoft domain. It simply doesn't work -- and neither does Disk Management.

At this point, you might be thinking something like “Jeremy, it sounds like a firewall issue or you haven’t enabled Remote Volume Management.” However, immediately after receiving the "RPC server is unavailable" message on my laptop, I was able to connect the Disk Management console to the Hyper-V server just fine from a Windows Server 2003 member server in the same domain.

In my mind, that indicated the firewall and remote administration were configured correctly. After a little research, it appeared that I was hitting a known bug with WMI when the client and server are in different, untrusted domains.

To workaround this issue, I did two things.

First, I created a new Vista VM and joined it to my customer's internal domain. After installing the Hyper-V Remote Management Update for Windows Vista (KB952627), I was able to start, stop, and create VMs on the Hyper-V server. Excellent.

Second, since I didn't want to have to always fire up my Vista VM just to view or change the Hyper-V settings on the server, I installed the Hyper-V Update for Windows Server 2008 x64 Edition (KB950050) on one of the VMs running on the Hyper-V server. This obviously doesn't completely replace the need for a remote administration client due to the "Catch-22" scenario -- meaning, if the VM isn't running, you can't use Hyper-V Manager from the VM to start the VM ;-)

Someday soon, I'm hoping we will release a few command line tools for Hyper-V that allow you to perform some basic operations such as starting or stopping VMs. This would be great on Server Core -- and no, I don't want to install PowerShell in order to do this ;-) [In keeping with the spirit of Server Core, I want to install as little as possible on the host.]

Lastly, I want to point out one of the stumbling blocks that I encountered along the way. Before I actually created the Vista VM that I mentioned earlier for remote administration, I initially created a Windows Server 2008 x86 VM and installed the 32-bit version of KB950050 in order to use Hyper-V Manager to remotely administer the Hyper-V server.

According to the corresponding KB article:

Update for Windows Server 2008 (KB950050)
This 32-bit update package includes the release version of the following:

The Hyper-V Manager console

The Virtual Machine Connection tool for x86-based editions of Windows Server 2008

Based on my experience installing the Hyper-V Remote Management Update for Windows Vista (KB952627) and the note above from the KB article, after installing KB950050 on Windows Server 2008 I expected to be able to start MMC and add the Hyper-V Manager snap-in. However, it doesn't quite work that way.

Fortunately, I received a quick response to my inquiry from Alex Kibkalo, a fellow Architect with Microsoft Consulting Services in Russia.

To enable Hyper-V Manager after installing KB950050, you need to enable the corresponding feature:

Open Server Manager. (If Server Manager is not running, click Start, point to Administrative Tools, click Server Manager, and then, if prompted for permission to continue, click Continue.)
In Server Manager, under Features Summary, click Add Features.
In the Add Features Wizard, on the Select Features page, expand Remote Server Administration Tools, and then expand Remote Administration Tools
Click Hyper-V Tools, and then proceed through the rest of the wizard.

For more information on deploying Hyper-V, refer to the Hyper-V Planning and Deployment Guide.

Copy/Paste Gotchas with Server Core

Jeremy Jameson — Mon, 07 Jul 2008 21:11:00 GMT

I'm building out a new virtualized environment using Windows Server 2008 and Hyper-V. In order to maximize performance and follow recommended best practices, I am using Server Core as the host OS.

I have to admit, doing this much administration from the command line really brings back memories from my "old Unix days" (before I switched to the Microsoft platform). I'll also admit that I've had to do quite a bit of research in order to figure out which commands need to be run in order to get a "functional" server. Now please don't misunderstand me, I don't mind that Server Core doesn't really allow you to do much with its out-of-the-box configuration (after all, that's the whole point). It simply takes a little getting used to -- and, as is usually the case, the first time takes longer than you expected.

I'll summarize some great resources I've found for working with Server Core and Hyper-V in a separate post sometime soon, but for now I wanted to share this little gotcha to save you the few minutes of frustration I spent trying to figure out what I was doing wrong.

Since I needed to reconfigure the disks on the server, I opened the Disk Management MMC console on my Windows Vista laptop and connected to the server -- or, rather I should say that I tried to connect to the server. Instead of connecting, I was greeted with the following error:

Disk Management could not start Virtual Disk Service (VDS) on DMX-CORE1-MAINT. This can happen if the remote computer does not support VDS, or if a connection cannot be established because it was blocked by Windows Firewall.

A quick Windows Live Search for "Disk Management could not start Virtual Disk Service" led me straight to LaNae Wade's post on enabling remote administration of Server Core. I then copied the command line for enabling the firewall rules for the Disk Management MMC snap-in and pasted it into my RDP session to the server:

netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes

Unfortunately, the response wasn't exactly what I expected:

Group cannot be specified along with other identification conditions.

It turns out that the quotes around Remote Volume Management were "corrupted", meaning they were converted to the angled quotes that tend to break things in very bizarre ways. The really frustrating part is that the command prompt makes the angled quotes appear just like regular quotes.

Once I edited the command line to replace the quotation marks, the command completed with the expected results:

C:\>netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes

Updated 3 rule(s).
Ok.