We all know that hardware failures are not unheard of events, and - like most people - I try to cover for such eventualities. Many features of my network depend on a single server running a few Hyper-V machines, and so I have a cold-swap backup server all set up and configured to take over the tasks with only minimal work required. Except, when I turned it on to do the weekly directory sync and install this month's patches, it quietly keeled over and died.
Mind you, it's one of the pair I bought some seven or eight years ago. The other, which was the main server, died four years ago in much the same sad way so I guess this one hasn't done too badly. Unfortunately, like the first one, it seems to be a motherboard failure that will cost nearly as much to fix (even if I can get the bits) than a new server. So I've bitten the bullet, splashed the plastic, and am waiting with baited breath hoping that the remaining server hangs on until the new one arrives.
What made the situation worse was that, in my usual paranoid way, the backup server was also domain controller - even though I have a domain controller as the other Hyper-V host and one running as a VM on Hyper-V. Last time I had a domain controller fail (a Windows 2003 one) I spent many unhappy hours forcibly removing it, and the crud it leaves behind, from Active Directory before the remaining DCs could agree that the domain was valid. So I was dreading the task this time.
Amazingly, however, it was easy. I just followed the advice on MSDN for doing it with the GUI. In Active Directory Sites and Services on another domain controller, you delete the NTDS settings object for the failed server and then delete the server itself. It took only a few seconds and, after they had done a bit of sniffing round the network, the remaining two domain controller seemed to be happy. So far, everything is working as it should. Probably because many settings, such as the DHCP options and DNS settings, purposely omitted the backup server because it was offline most of the time. If it was the main DC that had failed, I'd have needed to update these.
However, now I face an interesting decision. The still surviving box runs Server 2008 R2. Do I install Server 2008 R2, Server 2012, or Server 2012 R2 on the new box? In theory it should be the latest and greatest, Server 2012 R2. However, somewhat unusually for me, I planned ahead by checking that Server 2008 R2 VMs could be painlessly imported onto Server 2012 R2 Hyper-V. It seems they can't. The latest version of Hyper-V uses a different schema for the info files, so you either have to copy the VM itself and set up all the options afterwards (such as network connections and other stuff), or use a conversion script.
The problem is that I suspect this is a one-way transaction. At the moment I stop and export each VM, and then copy the exported folder to the same directory structure on the back-up server so that - in case of emergency - I can simply import the exported image and run it. The servers had identical physical and virtual network setups, and this worked fine when I tested it (yes, I did test my backup strategy!). But it gets more complicated if I have Server 2008 R2 on one box and Server 2012 R2 on the other. And I probably won't be able to use the new box as the main host with the existing one as the backup because I can't export/copy the VMs that way.
So the choice seems to be to install either Server 2008 R2 to ensure compatibility (with Active Directory as well as Hyper-V), or Server 2012 not R2, which uses the same Hyper-V schema. With 2008 R2 I've maybe only got 4 or 5 years until support ends, though probably the old server will be dead before then. It seems like the second option is the best, but I wonder if I'll get continually nagged to upgrade to 2012 R2? In reality, I should probably bite the bullet, burn my bridges, cast fate to the winds, and upgrade the old box to 2012 R2 and just use that on both servers. Maybe it's a Star Wars spin-off: R2 2 DO ...