Spat's WebLog (Steve Patrick)

When things go wrong...

Kerberos PAC Validation… what is it?

Kerberos PAC Validation… what is it?

  • Comments 12

 

 

Kerberos is well… Kerberos. And NTLM is NTLM right?  Right. Most of the time.   This post is from a recent hotfix I worked on where it was made painfully clear that this isn’t always true.

 

Here is a cool overview stolen from msdn

 

 

 

And the following table snipped:

 

Msv1_0.dll

The NTLM authentication protocol. This protocol authenticates clients that do not use Kerberos authentication.

Kerberos.dll

The Kerberos V5 authentication protocol. This protocol provides authentication using Kerberos protocol instead of plaintext, NTLM, or digest method.

 

Most people think that Kerberos.dll and msv1_0.dll  never really interact. However there is one very important interaction which slips by people until it bites them in the rear. PAC’s.

 

First , what is the PAC ?  PAC stands for Privilege Attribute Certificate

 

I won’t go into gory detail here but let’s say that the PAC contains various types of authorization data including groups that the user is a member of, rights the user has, and what policies apply to the user. When the client receives a ticket, the information contained in the PAC is used to generate the user’s access token.  See http://msdn2.microsoft.com/en-us/library/aa302203.aspx  for more info.

 

So how does this PAC thing work?

 

Well the Kerb client basically gets a ticket and then needs to do what is called PAC verification on the information ( to make sure it’s all cool to move ahead, after all if you read whats in it.. it’s pretty important ) We talk a little about it here http://support.microsoft.com/kb/906736

 

 

In order to do this we pass the information over and through the NTLM provider, msv1_0.dll and from there over the netlogon secure channel. Ahh it always seems to come back to it’s roots eh? J

 

You will see this in the netlogon logs as:

 

10/30 08:10:29 [LOGON] SamLogon: Generic logon of DOMAIN.COM\(null) from (null) Package:Kerberos Returns 0x0

 

 

Note the Kerberos package used over netlogon. Faskinating.

 

Anyway… how interesting, but lets apply it to some scenario.

 

The Perfect Storm

 

 

Let’s say you have assigned a number of  application to a machine via GPO’s.   You are concerned about licenses for these applications so when you remove machines you want the applications to be removed and checked the box you see below for “Uninstall this application when it falls out of the scope of management”

 

 

One day you find all the applications uninstalled on random machines ( one of them is the CIO ).  Every single application.  Uninstalled, even though you have not changed the policy or removed the machine from scope of the GPO.

 

You are a curious admin and prefer to keep your job. So.. you look at the userenv logs:

 

You see this:

USERENV(370.8fc) 16:13:11:240 ProcessGPOs: -----------------------

USERENV(370.8fc) 16:13:11:240 ProcessGPOs: Processing extension Software Installation

USERENV(370.8fc) 16:13:11:240 ReadStatus: Read Extension's Previous status successfully.

USERENV(370.8fc) 16:13:11:240 CheckForGPOsToRemove: GPO < Line of Business Applications-1> needs to be removed

USERENV(370.8fc) 16:13:11:240 CheckForGPOsToRemove: GPO <Line of Business Applications-2> needs to be removed

USERENV(370.8fc) 16:13:11:240 CheckForGPOsToRemove: GPO <Line of Business Applications> needs to be removed

USERENV(370.8fc) 16:13:11:250 CheckForGPOsToRemove: GPO <Very Important Apps> needs to be removed

 USERENV(370.8fc) 16:13:11:250 GetDeletedGPOList: Finished.

USERENV(370.8fc) 16:13:11:250 ProcessGPOList: Entering for extension Software Installation

USERENV(370.8fc) 16:13:11:250 MachinePolicyCallback: Setting status UI to Applying Software Installation policy...

USERENV(370.8fc) 16:13:11:300 LogExtSessionStatus: Successfully logged Extension Session data

USERENV(370.8fc) 16:13:11:301 MachinePolicyCallback: Setting status UI to Removing managed software Microsoft Project 2000…

 

 

Woah.

 

You open the event logs and look around that same time noted in the userenv logs..

 

Event Type:        Error

Event Source:    Kerberos

Event Category:                None

Event ID:              7

Date:                     11/6/1921

Time:                     4:13:01 PM

User:                     N/A

Computer:          MyClient

Description:

The kerberos subsystem encountered a PAC verification failure.  This indicates that the PAC from the client MyClient$ in realm DOMAIN.COM  had a PAC which failed to verify or was modified.  Contact your system administrator.

 

 

And

Event Type:       Error

Event Source:    NETLOGON

Event Category: None

Event ID:           5719

Date:                11/6/1921

Time:                4:13:04 PM

User:                N/A

Computer:         MyClient

Description:      No Domain Controller is available for domain DOMAIN due to the following:

                       The RPC server is unavailable.

                                       Make sure that the computer is connected to the network and try again.

 

And the corresponding netlogon log  error:

 

11/01 13:14:07 [LOGON] SamLogon: Generic logon of DOMAIN.COM\(null) from (null) Package:Kerberos Returns 0xC000005E

 

And in the apps log:

 

Source:        Application Management

Date:          11/6/9121  4:13:11 PM

Event ID:      303

Task Category: None

Level:         Information

Keywords:      Classic

User:          SYSTEM

Computer:      MyClient

Description:

The removal of the assignment of application Microsoft Project 2000 from policy LOB Applications succeeded.

 

 

 When systems encounter a Kerberos PAC validation error during log-on (perhaps due to transient network errors), it causes a machine to fall out of scope for all group policies, and all assigned applications go to an unmanaged state and to get uninstalled.

 

If you enable application management logging you will see something like this:

 

 

Software installation extension has been called for foreground synchronous policy refresh.

The following policies are to be removed, flags are 1.

    < Very Important AppsGPO > (unique identifier {4AE890C9-F0E7-4041-94A6-750896D93A3A})

        System volume path = \\domain.com\SysVol\domain.com\Policies\{4AE890C9-F0E7-4041-94A6-750896D93A3A}\Machine

        Active Directory path = LDAP://CN=Machine,CN={4AE890C9-F0E7-4041-94A6-750896D93A3A},CN=Policies,CN=System,DC=domain,DC=com

Enumerating the managed applications which are currently applied to this user.

The following 1 managed applications are currently applied to this user.

    Microsoft Project 2000 from policy < Very Important AppsGPO >  with state 511 and assign count 1.

Policy < Very Important AppsGPO >  has been removed.  All applications will be made unmanaged or removed.

No Active Directory path.

Calling Windows Installer to remove application Microsoft Project 2000  from policy < Very Important AppsGPO > .

Removing application Microsoft Project 2000  from the software installation database.

The removal of the assignment of application Microsoft Project 2000  from policy < Very Important AppsGPO >  succeeded.

 

 

So now the machine is toast.

 

Why? From the Event ID 7 we can see the PAC validation failed. These were machine scope GPO’s and the machine then failed to access the GPO. Whent it is denied access to the GPO, it thinks it has dropped out of scope and removes the apps. Quick – install this fix.  http://support.microsoft.com/?kbid=929624

 

Why did it fail the PAC validation in the first place? We never found out.   They had a complex network and perhaps had “hiccups” in the physical layers but in the end we thought – lets make this whole scenario more robust regardless of the *real* root cause. That’s hard for me to say sometimes, since we almost always want to get to true root cause.

 

Anyway…  have fun and be careful with your forks and knives.

 

Spatdsg

 

 

March 9, 2007

Added a few more notes:

Vista ( and apparently 2k3 SP2 _ has an option to not do PAC validation for services - ValidateKdcPacSignature ( i guess call PSS for more info on this )

Also - on XP and 2k3 we dont do PAC validation for ID's who are network service account , or who have the TCB privilege.

 

 

Leave a Comment
  • Please add 4 and 1 and type the answer here:
  • Post
  • Hi,

    We also had encountered the same problem on one of the machine, and searching for those events on eventid.net, we tried some solution listed there. One of them was to set MTU of the network right. After setting the MTU it solved the problem.

  • Ah good information. Thanks!

  • What is the response of the system if the PAC information is not provided in the ticket (NO_AUTH_DATA_REQUIRED)? Does this basically cripple a Win2003 server? Can users still authenticate and gain authorizations to access network resources, or do they just have a ticket to nowhere? I'm looking at removing the PAC as a means to enable Solaris 8 users to authenticate directly against AD2003 (where Solaris Kerberos only talks UDP), but I'm concerned about the effect on my Windows users.

  • Why are we doing a PAC verification for a computer account while doing a logon?

    I thought PAC verification is carried out only if a service is run as a user account and not with local system.

    Since the computer account would have the Tcbprivilege, why do we do a PAC validation?

  • I had been meaning to blog about this for a while, and recently was teaching a class when a friend of

  • Edgar's post on Open Specification blog: Understanding Microsoft Kerberos PAC Validation

  • I know this is an old topic and I hate to be a bit stroppy here, but as a Network guy who has just experienced this problem...

    You have a fault that causes all apps to be uninstalled if the network "fails" or experiences a "transient fault".  Hello - wireless roaming anyone? 3G access in trains? VPN's over the internet?

    Sorry, but how can this ever be interpreted as a network issue? Network connections come and go (sometimes for very good reasons - congested hubs / access lists/firewalls.), The application should simply acknowledge the failure to complete the authentication and move on - not trigger this type of event.

    Be fair, this is plainly an unfortunate oversight or poor coding.

  • Agreed! That's why we developed the hotfix in http://support.microsoft.com/?kbid=929624

  • I am experiencing the kerberos PAC validation issue with my database servers running MS SQL 2005. The SQl services account which is a domain account always failed to authenticate at the same  and then resume after all.

  • We are seeing the same issues. Also on a SQL Server 2005. Client workstations appear to be logging into the server but many are posting PAC Validation errors. Fortunataly the setting is off to unistall apps when sys falls out of scope.

    Looking into hot fix, thought this box was fully updated though.

  • In order to do this we pass the information over and through the NTLM provider, msv1_0.dll and from there over the netlogon secure channel. Ahh it always seems to come back to it’s roots eh? J

    ++++++++

    So does the above mean that PAC verification would fail in a wk8R2 forest/domain if i disable NTLM completely using NTLM blocker.

  • Interesting question - I don't think so since we are using msv1_0.dll for secure channel transport - not necessarily the NTLM authentication protocol.  But - I have not tested this. My bet? I think it would still work OK.

Page 1 of 1 (12 items)