Time Synchronization in Hyper-V

Time Synchronization in Hyper-V

Rate This
  • Comments 43


There is a lot of confusion about how time synchronization works in Hyper-V – so I wanted to take the time to sit down and write up all the details. 

There are actually multiple problems that exist around keeping time inside of virtual machines – and Hyper-V tackles these problems in different ways.

Problem #1 – Running virtual machines lose track of time.

While all computers contain a hardware clock (called the RTC – or real-time clock) most operating systems do not rely on this clock.  Instead they read the time from this clock once (when they boot) and then they use their own internal routines to calculate how much time has passed.

The problem is that these internal routines make assumptions about how the underlying hardware behaves (how frequently interrupts are delivered, etc…) and these assumptions do not account for the fact that things are different inside a virtual machine.  The fact that multiple virtual machines need to be scheduled to run on the same physical hardware invariably results in minor differences in these underlying systems.  The net result of this is that time appears to drift inside of virtual machines.

UPDATE 11/22: One thing that you should be aware of here: the rate at which the time in a virtual machine drifts is affected by the total system load of the Hyper-V server.  More virtual machines doing more stuff means time drifts faster.

In order to deal with time drift in a virtual machine – you need to have some process that regularly gets the real time from a trusted source and updates the time in a virtual machine.

Hyper-V provides the time synchronization integration services to do this for you.  The way it does this is by getting time readings from the management operating system and sending them over to the guest operating system.  Once inside the guest operating system – these time readings are then delivered to the Windows time keeping infrastructure in the form of an Windows time provider (you can read more about this here: http://msdn.microsoft.com/en-us/library/bb608215.aspx).   These time samples are correctly adjusted for any time zone difference between the management operating system and the guest operating system.

Problem #2 – Saved virtual machines / snapshots have the wrong time when they are restored.

When we restore a virtual machines from a saved state or from a snapshot we put back together the memory and run state of the guest operating system to exactly match what it was when the saved state / snapshot was taken.  This includes the time calculated by the guest operating system.  So if the snapshot was taken one month ago – the time and date will report that it is still one month ago.

Interestingly enough, at this point in time we will be reporting the correct (with some caveats) time in the systems RTC.  But unfortunately the guest operating system has no idea that anything significant has happened – so it does not know to go and check the RTC and instead continues with its own internally calculated time.

To deal with this the Hyper-V time synchronization integration service detects whenever it has come back from a saved state or snapshot, and corrects the time.  It does this by issuing a time change request through the normal user mode interfaces provided by Windows.  The effect of this is that it looks just like the user sat down and changed the time manually.  This method also correctly adjusts for time zone differences between the management operating system and the guest operating system.

Problem #3 – There is no correct “RTC value” when a virtual machine is started

As I have mentioned – physical computers have a RTC that operating systems look at when they first boot to get the time.  This real-time clock is backed by a small battery (you have probably seen the battery yourself if you have every pulled apart a computer).  Unfortunately virtual machines do not have any “batteries”.  When a virtual machine is turned off there is no component that keeps track of time for it.  Instead – whenever you start a virtual machine we take the time from the management operating system and put this into the real-time clock of the virtual machine.

This is done without the use of the Hyper-V time synchronization integration servers (it happens long before the integration services have loaded). 

The downside of this approach is that this does not take into account any potential time zone differences between the management operating system and the guest operating system.  The reason for this is that “time zones” are a construct of the software that runs in a virtual machine – and is not communicated to the virtual hardware in any way.  So – in short – when we start a virtual machine there is no way for us to know what time zone the guest operating system believes it is in.

One partial mitigation we have for this issue is that when the Hyper-V time synchronization component loads for the first time – it does an initial user mode set of the time to ensure that the time gets corrected as quickly as possible (using the same technique as discussed in problem #2).

So now that you understand how this all works – let’s discuss some common issues and questions around virtual machines and time synchronization.

Question #1 – I have a virtual machine that is configured for a different time zone to the management operating system.  Should I disable the time synchronization component of Hyper-V?

No, no, no, no, no, no, no.  And I say again – no.  As I have mentioned above – all time synchronization that is done by the Hyper-V time synchronization integration service is time zone aware.  If you disable the Hyper-V time synchronization integration service you will disable all the time synchronization aspects of Hyper-V that are time zone aware – and only leave the initial RTC synchronization active – which is not time zone aware.

This means that your virtual machines will go from booting in the wrong time zone, and then being corrected as soon as the Hyper-V time synchronization integration service loads to booting in the wrong time zone and staying in the wrong time zone.

Question #2 – Is there any way that I can stop Hyper-V from putting the wrong time in the RTC at boot?

In short; no.  We need to put something in there – and that is the best thing that we have to work with.

Question #3 – Can’t you use UTC time in the RTC so that the correct time is established when the virtual machine boots?

UTC (which is the computer techy version of saying GMT) time would solve this problem nicely with only one problem.  Windows does not support UTC time in the BIOS (Linux does).  So while this would solve the problem for our Linux running user base – the fact of the matter is that most of our users run Windows – and this would not work for them.

Question #4 – What about if I am using a different time synchronization source (e.g. domain time or a remote time server)?

Hyper-V time synchronization was designed to “get along well” with other time synchronization sources.  You should not need to disable Hyper-V time synchronization in order to use a different time synchronization source – as long as it goes through the Windows time synchronization infrastructure.

In fact – if you are running a Domain Controller inside a virtual machine I would recommend that you leave Hyper-V time synchronization enabled but that you also setup an external time source.  You can do this by going to this KB article: http://support.microsoft.com/kb/816042 and following the steps outlined in the “Configuring the Windows Time service to use an external time source” section.

UPDATE 11/22: I should have mentioned: since virtual machines tend to lose time much faster than physical computer, you need to configure any external time source to be checked frequently.  Once every 15 minutes is a good place to start.

Question #5 – How can I check what time source is being used by Windows inside of a virtual machine?

This is easy to do.  Just open an administrative command prompt and run “w32tm /query /source”.  If you are synchronizing with a remote computer – its name should be listed.  If you are using the Hyper-V time synchronization integration service you should see the following output:

image

If you see this output:

image

It means that there is no time synchronization going on for this virtual machine.  This is a very bad thing – as time will drift inside of the virtual machine.

Question #6 – Wait a minute!  My virtual machine should be synchronizing to the domain (or an external server) – but when I run that command it tells me that the Hyper-V time synchronization provider is being used!  How do I fix this!

I do not know why this happens – but sometimes it happens.  The first thing that you should do is to check that your domain does have a correctly configured authoritative time source.  There have been a small number of times when I have seen this problem being caused by the lack of an authoritative time source.

Alternatively – you can “partially disable” Hyper-V time synchronization.  The reason why I say “partially disable” is that you do not want to turn off the aspects of Hyper-V time synchronization that fix the time after a virtual machine has booted for the first time, or after the virtual machine comes back from a saved state.  No other time synchronization source can address these scenarios elegantly.

Luckily – there is a way to leave this functionality intact but still ensure that the day to day time synchronization is conducted by an external time source.  The key thing trick here is that it is possible to disable the Hyper-V time synchronization provider in the Windows time synchronization infrastructure – while still leaving the service running and enabled under Hyper-V.

To do this you will need to log into the virtual machine, open an administrative command prompt and run the following commands:

reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t reg_dword /d 0

This command stops W32Time from using the Hyper-V time synchronization integration service for moment-to-moment synchronization.  Remember from earlier in this post that we do not go through the Windows time synchronization infrastructure to correct the time in the event of virtual machine boot / restore from saved state or snapshot.  So those operations are unaffected.

w32tm /config /syncfromflags:DOMHIER /update

This command tells Windows to go and look for the best time source in the domain hierarchy.  If you want to use an external time server instead you can use the commands found here: http://technet.microsoft.com/en-us/library/cc784553(WS.10).aspx

net stop w32time & net start w32time

w32tm /resync /force

These two commands just “kick the Windows time service” to make sure the settings changes take effect immediately.

w32tm /query /source

This final command should confirm that everything is working as expected.

When you run these commands you should see something like this:

image

Question #7 – I have a virtual machine that has gotten ahead of time, and it never gets corrected back to the correct time.  What is going on here?

As a general rule of thumb, when time drifts inside a virtual machine it runs slower than in the real world, and the time falls behind.  We will always detect and correct this.

However, in the past, we have had reports of software problems caused when the Hyper-V time synchronization integration service decides to adjust the time back – because it believes the virtual machine is ahead of time.  To deal with this (rare) issue – we put logic in our integration service that will not change the time if the virtual machine is more than 5 seconds ahead of the physical computer.

UPDATE 11/22: I was asked how having the virtual machine in a different time zone to the Hyper-V server would affect this.  The short answer is that it does not.  The 5 second check is done after we have done the necessary time zone translation.

Question #8 – When should I disable the Hyper-V time synchronization service (either in the virtual machine settings, or inside the guest operating system)?

Never.

There are definitely times when you will want to augment the functionality of the Hyper-V time integration services with a remote time source (be it a domain source or an external time server) but the only way to get the best experience around virtual machine boot / restore operations is to leave the Hyper-V time integration services enabled.

Hopefully you will all find this information useful Smile

Cheers,
Ben

Leave a Comment
  • Please add 5 and 6 and type the answer here:
  • Post
  • Dave Atkinson -

    You had some good questions - which I will be happy to answer if you can post them without linking to the site that advertises your software.  Sorry about deleting your comment - but I get a lot of spam attempts here and have a strict policy against any form of advertising in comments.

    Cheers,

    Ben

  • Sorry about that I do it from habit  :-)

    I love your article and I had a few questions. I just finished a P2V of my SBS 2003 DC and I was having a big problem with time drift. I tried turning off C2 states in the host BIOS, did all kinds of messing around with w32time and registry settings, and nothing was working. Time on my DC would drift considerably, affecting every machine on the network.

    If you have other VMs on the same machine that are Domain members, what is the best practice? If you "partially disabled" integration svcs time sync as in #6 above, and still had those machines point to the PDC emulator (in my case a single virtual DC), would you have to lower the time sync period so that it closely matches what integration svcs does? I can bring up the clock on these machines and watch the second hand stop for several seconds at a time - they start to drift very quickly. What time sync interval is safe for things like source control, Kerberos auth, etc.? I want to be sure when I check in my files and perform other Source Control-type operations that time stamps are all good and things make sense.

    Alternately, should you sync the host to an external time source and allow integration svcs time sync on all your VMs, ignoring Domain hierarchy altogether?

    Thanks for the great article,

    Dave

  • For question 7, does it mean if the VM is running ahead of the physical machine, we will not correct it by Hyper-V time synchronization service? So we have to configure a remote time source to correct it somehow?

  • with regards to the timezone of the clock, couldn't that be part of the VM's metadata? what timezone the clock should be on startup/restore? (and whether it should be adjusted for daylight savings)

  • Hi Ben

    I see alot of confusion over this particular subject, but I have seen a microsoft article about domain controllers as VM guests and they advise to disable the Time Sync service.

    Quoted here:

    "For virtual machines that are configured as domain controllers, disable time synchronization with the host through Integration Services. Instead, accept the default Windows Time service (W32time) domain hierarchy time synchronization.

    Host time synchronization makes it possible for guest operating systems to synchronize their system clocks with the system clock of the host operating system. Because domain controllers have their own time synchronization mechanism, host time synchronization must be disabled on virtual machines that are configured as domain controllers. If domain controllers synchronize time from their own source and also synchronize time from the host, the domain controller time can change frequently. Because many domain controller tasks are tied to the system time, a jump in the system time could cause lingering objects to be left in the directory and replication to be stopped. "

    Reference: technet.microsoft.com/.../dd348449%28WS.10%29.aspx

    I want to follow best practice but it seems there is soo much conflicting advice out there.

    kind regards

    Zoel

  • Hello Virtual PC Guy

    Thanks for sharing.

    Could you take a look at my problem I have with the Hyper-V time integration service?

    social.technet.microsoft.com/.../cd10a6f2-4274-4f17-af3d-75dc6004a92e

    When ever i reboot, the key :  HKLM\SYSTEM\CurrentControlSet\Services\w32time\TimeProviders\VMICTimeProvider

    Is incorrectly modifyed by the hyper-V integration components. Permissions and keys are missing.

    I can recreate this on every virtual machine that is win2k3 and restored from a System state backup.  

  • Ben, thanks for this excelent post, however in some place of the technet library when talk about best practices to virtualize domain controllers the recomendation is disable time sync feature in the integration components.  What is your opinion about this?

  • MoxyDave –

    No worries.  You are correct that you basically have two options: 1) get the virtual machines to sync remotely and partially disable the time synchronization or 2) make sure the time source for the parent is good – and sync to that.  The key thing is that *something* needs to be syncing with a remote server.

    You are also correct that when you configure a virtual machine to sync to a remote server – you should ratchet up the time sync interval because of the high tendency to drift.  I would recommend once every 15 minutes. (Note – I will update the post to include this).

    Shawn Xiao –

    Correct.

    Frymaster –

    This is something that we have looked into – but it is not straight forward – as time zone is something that is completely a construct of the software running inside the virtual machine.

    ZoelCardosa –

    Yes, I am quite aware of this guidance and am currently on an email discussion with the team that owns that article.  Unfortunately, their advice to turn off the Hyper-V time synchronization integration service and do nothing else is probably the worst thing you could do.  The (unwritten) assumption in this documentation is that after you disable the Hyper-V time synchronization integration service – you will then set up a reliable remote time synchronization source.

    Mobay –

    No, I have not heard of this.  Is this as case where you have backed up and restored the virtual machine?  Or did you backup a physical computer and restore it to a virtual machine?

    Germán Ruiz –

    As I mentioned to ZoelCardosa, I very strongly disagree with this guidance and am discussing it with the teams responsible for it.

    Cheers,

    Ben

  • Ben - I have to disagree with your answer to question #8 - "Never".  We found situations of multi-processor guests running on Hyper-V that simply would not keep correct time.  We found KB articles on BOOT.INI /usepmtimer settings, etc, but NONE of the MS documented solutions worked.  We disabled the Virtual Machine Integration Components (VMIC) for Time Synchronization via the Hyper-V Console  (because it is broken in the SCVMM Console) and now our "back level" guests like Windows XP keep perfect time based on domain synchronization.

  • Hi,

    Question #3 – Can’t you use UTC time in the RTC so that the correct time is established when the virtual machine boots?

    It is possible to set Windows to use UTC time in the RTC, using the RealTimeIsUniversal registry key, at least since Vista SP2 / Windows 7.

    See the following page:

    www.cl.cam.ac.uk/.../ut-rtc.html

    under 'Current support status in Windows' (post dated 2008-10-31)

  • Hi Ben,

    I have another unralated question about the Topic but related to Hyper-V

    When i started virtualizing i didnt pay much attention to which disk, was which clustered disk. And now i have a huge mess. I need to expand some disk but im not quite sure which is which.

    For Example:

    Disk 1- Clustered Disk 10 - Volume3

    Is there anyway to mix and match what Clustered disk is what "phisical" LUN?

  • Just a quick note on this, we had a dev server which was scheduled for a nightly reboot @ 12.30 am.  When daylight saving kicked in, it started rebooting continually for an hour from 12.30 to 1.30, as the task scheduler would start, (and the system time would be, eg, 11.35), THEN the time sync service woulds start, adjust the time from 11.35 to 12.35, then the task scheduler would see this change, and trigger all events due between 11.35 and 12.35, so the reboot got requeued, etc.  To make matters even more fun, there were tasks queued at 1 am, and these were also starting and stopping for half an hour as the system did it's nightly hour long death spiral.  To fix this, we changed the scheduler service so it was dependant on the time sync service. (KB 193888)

    It's probably worth noting that scheduled reboots will be affected by this time zone change, unless the scheduler is started after the time sync service.

  • Just a quick note on this, we had a dev server which was scheduled for a nightly reboot @ 12.30 am.  When daylight saving kicked in, it started rebooting continually for an hour from 12.30 to 1.30, as the task scheduler would start, (and the system time would be, eg, 11.35), THEN the time sync service woulds start, adjust the time from 11.35 to 12.35, then the task scheduler would see this change, and trigger all events due between 11.35 and 12.35, so the reboot got requeued, etc.  To make matters even more fun, there were tasks queued at 1 am, and these were also starting and stopping for half an hour as the system did it's nightly hour long death spiral.  To fix this, we changed the scheduler service so it was dependant on the time sync service. (KB 193888)

    It's probably worth noting that scheduled reboots will be affected by this time zone change, unless the scheduler is started after the time sync service.

  • I also vaguely remember certain VMs of mine (Windows Server 2003 I think) moaning with time related warnings and errors in the event log.  Disabling Time Synchronization for the guests stopped these from appearing.  BTW, the guests in question were domain members getting their time from local DCs so disabling the synchronization wasn't an issue.

    Just found a link which described my issue;

    www.mcbsys.com/.../w32time-errors-in-a-hyper-v-virtual-environment

  • I'm dealing with this issue for quite some time now in an enterprise environment and for sure i can tell, that the problem is solved for Windows Server 2003 VMs by reinstalling the HyperV Guest Services and disabling the HyperV Time Service.

    For 2008 R2 servers, however, this "trick" doesn't work as the services are more integrated into the core system. Overall i tried almost everything i could find in the internet and still searching for a solution for 2k8r2-servers. Even if there is the dc listed after the "w32tm /query /source" command the next day you just have more Event50 in the log. I also tried manipulating registry values like "SpecialPollInterval" but no effect.

    My last hope right now is "question 6" from this article.

    @Virtual PC Guy:

    Any Update regarding the Services-matter (keep them on or off)?

    My best guess right now is that if the host is really on its limit that it doesn't matter what you do - the warning will happen. Thats not a good sight, however.

Page 1 of 3 (43 items) 123