The Authoritative Restore Explained

Article
10/06/2011

(....well... in more detail)

This blog posting is for the purposes of explaining the mechanism behind the authoritative restore within Active Directory. While there are many documents out there explaining step-by-step instructions of how to perform authoritative restores and the logic behind it, I feel there’s some information missing from these documents and, hence, this blog posting.

To start off, let’s define some logistics.

This blog does not delve into the mechanics behind Active Directory Replication. It is assumed certain concepts such as USNs, high-watermark, or up-to-dateness vector are already understood. The following article explains these concepts in great detail.

How the Active Directory Replication Model Works
https://technet.microsoft.com/en-us/library/cc772726(WS.10).aspx

Although I don’t dive into how replication works I’d like to start with some definitions as I feel it is important to this blog.

Update Sequence Number (USN)
For every change that occurs on a domain controller (whether replicated inbound or an originating write) the update sequence number is incremented by one.

Highest Committed USN
This is held within RootDSE on every domain controller and is the highest USN committed to the database for that server. This is an attribute of RootDSE and is not replicated to the other domain controllers.

Originating Update
A domain controller that sets an attribute or builds an object performs an originating update of that attribute or object. The originating update can be seen within the metadata of the object and is used for interpreting which domain controller performed the change to an object.

High-Watermark
Each destination DC contains a high-watermark value of its interpretation of the upstream (or source) DC’s highest committed USN. If DC2’s high-watermark for DC1 is 3000 then DC1 can assume that DC2 already has knowledge of all changes less than or equal to USN 3000. The high-watermark is per replication link and knowledge a DC has of each direct replication partner. Although there may be hundreds of DCs within a domain, it’s possible DC2 only replicates with a handful of them and, therefore, would only have knowledge of the high-watermark of those DCs. The high-watermark prevents irrelevant objects from being considered by the source domain controller with respect to a single destination.

Up-to-Dateness Vector
Any DC that has ever made an originating update will have an entry in the up-to-dateness vector. This is per partition. We use the up-to-dateness vector to determine (rather filter out) attributes to objects that a downstream replication partner may have already received from a different source DC. Each DC has its own copy of the up-to-dateness vector for all domain controllers that replicate a specific partition.

Version
For the purposes of this blog, the term “version” refers to the version number of an attribute. As discussed within this blog, an authoritative restore of an object will increment the version number of every attribute by a specific value (discussed later).

DSA Object GUID
The Globally Unique Identifier of a domain controller.

DSA Invocation ID
The GUID of the Active Directory database (ntds.dit). The Invocation ID is used to uniquely identify a database and it changes during the restore process of Active Directory.

The Beginning

The environment, discussed in this blog, consists of only two domain controllers (DC1 and DC2) for a domain called contoso.com.

After creating the environment, repadmin /showrepl /v against each DC shows:

DC1

Default-First-Site-Name\DC1
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f
DSA invocationID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f

(Notice the DC GUID and Invocation ID GUID are the same for DC1. This is because it was the first domain controller promoted into the forest and, therefore, gets the same GUID for both.)

DC2

Default-First-Site-Name\DC2
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 5ad77be3-1e89-4bba-bd7c-c78a03860204
DSA invocationID: e7d0d6bb-056c-45b1-b752-1d6900e3b9e5

Out of the box, the up-to-dateness vector for the DC=contoso,DC=com partition on DC1 showed:

Caching GUIDs.

Default-First-Site-Name\DC1 @ USN 13190 @ Time 2011-10-04 21:11:39
Default-First-Site-Name\DC2 @ USN 12755 @ Time 2011-10-04 21:07:39

Out of the box, the up-to-dateness vector for the DC=contoso,DC=com partition on DC2 showed:

Caching GUIDs.

Default-First-Site-Name\DC1 @ USN 13185 @ Time 2011-10-04 21:10:14
Default-First-Site-Name\DC2 @ USN 12778 @ Time 2011-10-04 21:11:58

One thing to note is that it’s likely that interrogating the up-to-dateness vector on both domain controllers at the same time won’t show the same USN values. This is due to convergence within Active Directory. There’s always something going on. Active Directory replication is loosely consistent and we assume we’ll eventually converge.

First things first, let’s demote DC2. Why? This would be to illustrate a change in Invocation ID. Before the demotion, let’s look at the replication status of DC1:

Repadmin /showrepl DC1 /v shows:

Default-First-Site-Name\DC1
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f
DSA invocationID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f

==== INBOUND NEIGHBORS ======================================

DC=contoso,DC=com
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: 5ad77be3-1e89-4bba-bd7c-c78a03860204
        Address: 5ad77be3-1e89-4bba-bd7c-c78a03860204._msdcs.contoso.com
        DSA invocationID: e7d0d6bb-056c-45b1-b752-1d6900e3b9e5

(Notice the DSA invocation ID for the inbound partner DC2.)

After a demotion of DC2 and another DCPromo promotion of DC2, the repadmin /showrepl /v configuration of DC2 shows:

Default-First-Site-Name\DC2
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 8913255f-89c3-4954-ab3f-3148f7303320
DSA invocationID: 43c7f173-db0f-4ddb-aff0-ad50717729c9

Notice the DC GUID and Invocation ID both changed. This comes into play later.

The up-to-dateness vector for the DC=contoso,DC=com partition on DC1 now shows:

Caching GUIDs.

Default-First-Site-Name\DC2 @ USN 12399 @ Time 2011-10-04 21:26:04
Default-First-Site-Name\DC1 @ USN 13414 @ Time 2011-10-04 21:30:22
e7d0d6bb-056c-45b1-b752-1d6900e3b9e5 @ USN 12778 @ Time 2011-10-04 21:12:08

(Recall that the database GUID of DC2 used to be e7d0d6bb-056c-45b1-b752-1d6900e3b9e5. It is still in the up-to-dateness vector.)

The command repadmin /showrepl DC1 /v shows the following information about DC2:

Default-First-Site-Name\DC1
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f
DSA invocationID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f

==== INBOUND NEIGHBORS ======================================

DC=contoso,DC=com
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: 8913255f-89c3-4954-ab3f-3148f7303320
        Address: 8913255f-89c3-4954-ab3f-3148f7303320._msdcs.contoso.com
        DSA invocationID: 43c7f173-db0f-4ddb-aff0-ad50717729c9
        SYNC_ON_STARTUP DO_SCHEDULED_SYNCS WRITEABLE
        USNs: 12329/OU, 12329/PU
        Last attempt @ 2011-10-04 21:26:07 was successful.

(The above values for USNs represent DC1’s high-watermark knowledge of DC2 for the partition DC=contoso,DC=com.)

(The remainder of the repadmin /showrepl command has been snipped.)

So we just discussed that the up-to-dateness vector keeps records of domain controllers that have been demoted. This is to maintain history of DCs we’ve replicated with in the past. What if I were to restore a backup of DC2 at this point? We’ll cover that in a bit.

First, let’s talk about where this up-to-dateness vector is stored. The up-to-dateness vector is per partition and, therefore, is stored as attributes of a particular partition. In this case, the partition of interest is DC=contoso,DC=com. The attribute we query for is replUpToDateVector. This attribute is not a replicated attribute and looks different depending on which DC is interrogated.

To illustrate this I used LDP and searched for the replUpToDateVector attribute for each DC.

LDP connection to DC1:

***Searching...

ldap_search_s(ld, "DC=contoso,DC=com", 2, "(replUpToDateVector=*)", attrList, 0, &msg)
Getting 1 entries:
Dn: DC=contoso,DC=com
replUpToDateVector: dwVersion: 2, dwReserved1: 0, V2.cNumCursors: 2, V2.dwReserved2: 0, rgCursors:
{uuidDsa: 43c7f173-db0f-4ddb-aff0-ad50717729c9, usnHighPropUpdate: 12459, timeLastSyncSuccess: 10/04/2011 21:31:32},
{uuidDsa: e7d0d6bb-056c-45b1-b752-1d6900e3b9e5, usnHighPropUpdate: 12778, timeLastSyncSuccess: 10/04/2011 21:12:08},;

LDP connection to DC2:

***Searching...

ldap_search_s(ld, "DC=contoso,DC=com", 2, "(replUpToDateVector=*)", attrList, 0, &msg)
Getting 1 entries:
Dn: DC=contoso,DC=com
replUpToDateVector: dwVersion: 2, dwReserved1: 0, V2.cNumCursors: 2, V2.dwReserved2: 0, rgCursors:
{uuidDsa: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f, usnHighPropUpdate: 13435, timeLastSyncSuccess: 10/04/2011 21:40:17},
{uuidDsa: e7d0d6bb-056c-45b1-b752-1d6900e3b9e5, usnHighPropUpdate: 12778, timeLastSyncSuccess: 10/04/2011 21:12:08},;

Notice both domain controllers have knowledge of the original DC2 (uuidDsa e7d0d6bb….) as well as knowledge of the highest propagated update (12778) and the last time it synchronized (10/04/2011 at 9:12:08 p.m.).

Each DC also has knowledge in the up-to-dateness vector for all other domain controllers that replicate a partition. In the up-to-dateness vector for the partition DC=contoso,DC=com on DC1 you will see it does not hold knowledge of its own uuidDsa. The same is true for DC2.

Domain Controller Restore Process and the Invocation ID

So now that we’ve discussed the mechanics behind it, let’s discuss what happens when a domain controller is restored from backup.

As you may have read, when a restore operation of a domain controller is performed, we keep the DSA Object GUID the same but the DSA Invocation ID is recreated with a random GUID. This process informs downstream replication partners that a DC restore operation has been performed. When the backup was created, the highestCommittedUSN value was at a lower number than what every other domain controller remembered of DC1 before the restore process.

By modifying the Invocation ID it informs the downstream replication partners of this restore operation and we can get around USN rollback scenarios. It also instructs DC1’s replication partners to send all updates from highestCommittedUSN (x) to highestCommittedUSN (y) where X is the value at the time of the backup and Y is the value at the time DC1 went down to perform the restore.

So let’s look at this. I took a system state backup of DC1 and then restored the backup.

Repadmin /showrepl DC2 /v

Default-First-Site-Name\DC2
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 8913255f-89c3-4954-ab3f-3148f7303320
DSA invocationID: 43c7f173-db0f-4ddb-aff0-ad50717729c9

==== INBOUND NEIGHBORS ======================================

DC=contoso,DC=com
    Default-First-Site-Name\DC1 via RPC
        DSA object GUID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f
        Address: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f._msdcs.contoso.com
        DSA invocationID: 1f4b741a-6409-4eb6-996b-0f5f7417646a
        SYNC_ON_STARTUP DO_SCHEDULED_SYNCS WRITEABLE
        USNs: 24637/OU, 24637/PU
        Last attempt @ 2011-10-05 20:11:14 was successful.

(The rest of the repadmin /showrepl output has been snipped.)

Repadmin /showrepl DC1 /v

Default-First-Site-Name\DC1
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f
DSA invocationID: 1f4b741a-6409-4eb6-996b-0f5f7417646a

==== INBOUND NEIGHBORS ======================================

DC=contoso,DC=com
    Default-First-Site-Name\DC2 via RPC
        DSA object GUID: 8913255f-89c3-4954-ab3f-3148f7303320
        Address: 8913255f-89c3-4954-ab3f-3148f7303320._msdcs.contoso.com
        DSA invocationID: 43c7f173-db0f-4ddb-aff0-ad50717729c9
        SYNC_ON_STARTUP DO_SCHEDULED_SYNCS WRITEABLE
        USNs: 14407/OU, 14407/PU
        Last attempt @ 2011-10-05 20:11:20 was successful.

(The rest of the repadmin /showrepl output has been snipped.)

Looking at the /showrepl output from both domain controllers, you can see that the DSA InvocationID for DC1 has changed; however, the DSA Object GUID remains the same.

DSA object GUID: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f
DSA invocationID: 1f4b741a-6409-4eb6-996b-0f5f7417646a

Repadmin /showutdvec DC1 DC=contoso,DC=com /nocache

1f4b741a-6409-4eb6-996b-0f5f7417646a @ USN 24668 @ Time 2011-10-05 20:15:09
43c7f173-db0f-4ddb-aff0-ad50717729c9 @ USN 14452 @ Time 2011-10-05 20:11:16
9628bc8f-fa40-4c72-ac57-99ef34fe3f4f @ USN 13972 @ Time 2011-10-05 18:17:27
e7d0d6bb-056c-45b1-b752-1d6900e3b9e5 @ USN 12778 @ Time 2011-10-04 21:12:08

Repadmin /showutdvec DC2 DC=contoso,DC=com /nocache

1f4b741a-6409-4eb6-996b-0f5f7417646a @ USN 24663 @ Time 2011-10-05 20:14:04
43c7f173-db0f-4ddb-aff0-ad50717729c9 @ USN 14482 @ Time 2011-10-05 20:14:58
9628bc8f-fa40-4c72-ac57-99ef34fe3f4f @ USN 13972 @ Time 2011-10-05 18:17:27
e7d0d6bb-056c-45b1-b752-1d6900e3b9e5 @ USN 12778 @ Time 2011-10-04 21:12:08

Here’s the recap:

9628bc8f…. was the original DC1 that was just restored.
E7d0d6bb… was the original DC2 that was demoted at the beginning.
1f4b741a… is the new DC1.
43c7f173… is DC2.

Using LDP, I again performed a search on both domain controllers for the replUpToDateVector.

DC1

***Searching...

ldap_search_s(ld, "DC=contoso,DC=com", 2, "(replUpToDateVector=*)", attrList, 0, &msg)
Getting 1 entries:
Dn: DC=contoso,DC=com
replUpToDateVector: dwVersion: 2, dwReserved1: 0, V2.cNumCursors: 3, V2.dwReserved2: 0, rgCursors:
               {uuidDsa: 43c7f173-db0f-4ddb-aff0-ad50717729c9, usnHighPropUpdate: 14452, timeLastSyncSuccess: 10/05/2011 20:11:16},
               {uuidDsa: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f, usnHighPropUpdate: 13972, timeLastSyncSuccess: 10/05/2011 18:17:27},
               {uuidDsa: e7d0d6bb-056c-45b1-b752-1d6900e3b9e5, usnHighPropUpdate: 12778, timeLastSyncSuccess: 10/04/2011 21:12:08},;

DC2

***Searching...

ldap_search_s(ld, "DC=contoso,DC=com", 2, "(replUpToDateVector=*)", attrList, 0, &msg)
Getting 1 entries:
Dn: DC=contoso,DC=com
replUpToDateVector: dwVersion: 2, dwReserved1: 0, V2.cNumCursors: 3, V2.dwReserved2: 0, rgCursors:
               {uuidDsa: 1f4b741a-6409-4eb6-996b-0f5f7417646a, usnHighPropUpdate: 24679, timeLastSyncSuccess: 10/05/2011 20:17:41},
               {uuidDsa: 9628bc8f-fa40-4c72-ac57-99ef34fe3f4f, usnHighPropUpdate: 13972, timeLastSyncSuccess: 10/05/2011 18:17:27},
               {uuidDsa: e7d0d6bb-056c-45b1-b752-1d6900e3b9e5, usnHighPropUpdate: 12778, timeLastSyncSuccess: 10/04/2011 21:12:08},;

So, in a nutshell, that’s how the restore process works. There is an additional feature of NTBackup that adds some registry settings during a restore process of the system state. When performing a system state restore using NTBackup, a new key is created called Restore in Progress. This is located under:

HKLM\System\CurrentControlSet\Services\NTDS\Restore In Progress

Information about this key can be found in the following support article:

Description of the “Restore in Progress” Registry Key in Active Directory
https://support.microsoft.com/kb/814167

An important registry setting for the restore operation is called New Database GUID located in HKLM\System\CurrentControlSet\Services\NTDS\Parameters. This registry setting supplies the new DSA InvocationID to be used after the restore process is complete.

Authoritative Restore of Joe User

Next let’s create a user (Joe User) and then do some authoritative restores.

Created Joe User (sAMAccountName joeuser) on DC2.

Repadmin /showobjmeta DC2 “CN=joe user,CN=Users,DC=contoso,DC=com”

26 entries.

     Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute
     ======= =============== ========= ============= === =========
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 objectClass
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 cn
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 sn
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 givenName
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 instanceType
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 whenCreated
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 displayName
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 nTSecurityDescriptor

(The rest of the output has been snipped.)

From here I need to explain what happens when an object is authoritatively restored from a system state backup. There are many documents on the Internet stating that the version number of each attribute for an object will increase by 100,000 versions. Some state the version of each attribute will increase 100,000 versions total while others state the version will increase by 100,000 versions per day since the last backup was taken.

Neither of these statements are true.

Without creating a backup at all, using Windows Server 2008, I stopped the Active Directory Domain Services (NOT supported; always reboot into Directory Services Restore Mode) and performed these actions on DC1:

Net stop ntds
Changed the date ahead 2 days
Ntdsutil
Activate instance NTDS
Authoritative restore
Restore object “CN=Joe User,CN=Users,DC=contoso,DC=com”
Restored the object
Net start ntds

Repadmin /showobjmeta DC1 “CN=joe user,CN=users,DC=contoso,DC=com”

27 entries.

     Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute
======= =============== ========= ============= === =========
14612 Default-First-Site-Name\DC1 28675 2011-10-07 20:29:35 300001 objectClass
       14611 Default-First-Site-Name\DC2 14611 2011-10-07 20:30:23 2 cn
     14612 Default-First-Site-Name\DC1 28675 2011-10-07 20:29:35 300001 sn
     14612 Default-First-Site-Name\DC1 28675 2011-10-07 20:29:35 300001 givenName
     14612 Default-First-Site-Name\DC1 28675 2011-10-07 20:29:35 300001 instanceType
       14550 Default-First-Site-Name\DC2 14550 2011-10-05 20:26:16 1 whenCreated
     14612 Default-First-Site-Name\DC1 28675 2011-10-07 20:29:35 300001 displayName
       14612 Default-First-Site-Name\DC1 28675 2011-10-07 20:29:35 300000 isDeleted

(The rest of the output has been snipped.)

Notice that the version number has increased by 300,000. This is equal to “today + 2 days” or 3 days total.

Deleted Joe User
Changed the date back to the correct date
Created Joe User on DC2
Output of showobjmeta is similar to original Joe User object
Net stop ntds
Changed the date forward 11 days
Ntdsutil
Activate Instance NTDS
Authoritative restore
Restore object “CN=joe user,CN=Users,DC=contoso,DC=com”
Restored the object
Net start ntds

Repadmin /showobjmeta DC1 “CN=joe user,CN=Users,DC=contoso,DC=com”

27 entries.

     Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute
======= =============== ========= ============= === =========
14691 Default-First-Site-Name\DC1 32772 2011-10-18 20:33:30 1200001 objectClass
      14690 Default-First-Site-Name\DC2 14690 2011-10-05 20:34:37 2 cn
      14691 Default-First-Site-Name\DC1 32772 2011-10-18 20:33:30 1200001 sn
      14691 Default-First-Site-Name\DC1 32772 2011-10-18 20:33:30 1200001 givenName
     14691 Default-First-Site-Name\DC1 32772 2011-10-18 20:33:30 1200001 instanceType
      14647 Default-First-Site-Name\DC2 14647 2011-10-07 20:31:33 1 whenCreated
      14691 Default-First-Site-Name\DC1 32772 2011-10-18 20:33:30 1200001 displayName
     14691 Default-First-Site-Name\DC1 32772 2011-10-18 20:33:30 1200000 isDeleted

(The remainder of the output has been snipped.)

Notice the version number increase is now 1,200,000 (100,000 x 12 days…. “today plus 11 days”).

Here’s where I mention another “feature” of this process. The version number increase is capped at 60 days. I assume this is because the original Tombstone Lifetime was set to 60 days and backups can only be restored < TSL.

Deleted Joe User
Changed the date back to the correct date
Created Joe User on DC2
Output of showobjmeta is similar to original Joe User object
Net stop ntds
Changed the date forward to 2/1/2012
Ntdsutil
Activate Instance NTDS
Authoritative restore
Restore object “CN=joe user,CN=Users,DC=contoso,DC=com”
Restored the object
Net start ntds

Repadmin /showobjmeta DC1 “CN=joe user,CN=Users,DC=contoso,DC=com”

27 entries.

     Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute
======= =============== ========= ============= === =========
14824 Default-First-Site-Name\DC1 36869 2012-02-01 20:42:11 6000001 objectClass
     14823 Default-First-Site-Name\DC2 14823 2012-02-01 20:42:37 2 cn
      14824 Default-First-Site-Name\DC1 36869 2012-02-01 20:42:11 6000001 sn
      14824 Default-First-Site-Name\DC1 36869 2012-02-01 20:42:11 6000001 givenName
     14824 Default-First-Site-Name\DC1 36869 2012-02-01 20:42:11 6000001 instanceType
     14742 Default-First-Site-Name\DC2 14742 2011-10-05 20:37:28 1 whenCreated
     14824 Default-First-Site-Name\DC1 36869 2012-02-01 20:42:11 6000001 displayName
     14824 Default-First-Site-Name\DC1 36869 2012-02-01 20:42:11 6000000 isDeleted

(The rest of the output has been snipped.)

Notice that the version increase was capped at 60 days (or 6,000,000 version increases).

I hope this has helped clear up the understanding of the logic behind the authoritative restore, where we store knowledge of the Invocation ID for the database, and how DC restore process works.

One point to note is that the version increase default is 100,000 versions per day since the last time a change was committed to the database (not since the last backup). This version increase is a default value and can be manipulated using the verinc parameter during the authoritative restore.

Restore object “CN=Joe User,CN=Users,DC=contoso,DC=com” verinc 1000

This would increase the version number of every attribute for Joe User by 1,000 each day since the last time a change was committed to the database.

The Authoritative Restore Explained

Additional resources