So the VB.NET QA team went 'dark' for the past two weeks to work on finishing up our automation. This means no meetings, no email, no distractions. Just focus. As a result the team was insanely productive, myself included.
I would like to provide a peek at how we do install-time testing for the Bootstrapper. Traditional automated tests at Microsoft are written to be robust, so we can run them across a number of platforms and OS configurations. We have a wealth of infrastructure to help with that and for the most part things just work.
Testing the Bootstrapper however is a much more difficult venture. Normal tests interact with Visual Studio, performing a few tasks and verifying their results. The Bootstrapper however installs programs. Programs which usually cannot be easily be uninstalled. In order to do robust Bootstrapper testing we need to be able to make pretty invasive changes to the platform. (That is even before you mention running across everything from Windows98 to Win2K3 Server.)
To do this we use Virtual Server and do our end-to-end verification on a virtual machine. This way we can kung-foo the machine any way we need and just reset the virtual hard drive back to normal as if it never happened.
Using virtual machines for install-time testing isn't anything terribly new, but the main difficulty is integrating it with our existing test infastructure and lab setup. An example of this is our network security policy in place. In order to make sure that Microsoft machines aren't at risk to viruses and other exploits every machine is mandated to have anti-virus installed, and if a machine is found not to have AV software installed or isn't patched with the latest security updates that machine's network port will be shut down. Boom. Poof. No more network connectivity for you. (You can call the help desk, patch your machine, and get it set back up again though.)
Sure it is kind of annoying, but it is well understood that that is what it takes to keep the network safe and secure. However the 'no patch equals no port' policy still applies even to virtual machines. To bypass this, and therefore get test coverage for systems with older service packs such as XP Gold and XP SP1 we use the Microsoft Loopback Adapter.
By setting up the virtual machine's network a special way, the only machine the VM can see on the network is its host. So although it may be vulnerable to exploitation, no other machine can 'see' it. (It also doesn't have access to any 'main land' network resources anyways.)
To do this, do the following:
1.) Install the MS loopback adapter on the host machine2.) Set the host machines loopback network IP address to 184.108.40.206.) Set the VMs network adapter to point to the MS Loopback adapter in the Virtual Server console4.) On the VM's OS set the network address IP to 220.127.116.11
And after that it should automagically work. The VM should be able to see the Host machine on the network but not anything else.
It appears that this post has gotten long enough as-is so I will save other details about our Virtual Server testing for another day. Untill then I would highly reccomend seeing StarWars: Episode 3, definitely worth the wait. :)