All About Kerberos “The Three Headed Dog” with respect to IIS and Sql

All About Kerberos “The Three Headed Dog” with respect to IIS and Sql

Rate This
  • Comments 4

Kerberos Understanding and troubleshooting

Steps to be followed while dealing with Troubleshooting Kerberos related issues.

Prerequisites:

1) Kerberos can only work in intranet because client needs to interact with the Domain controller directly.

2) Kerberos can be used in internet only if you have a public facing active directory. But public facing active directory is not recommended due to security concerns.

3) Server , client and dc must be in the intranet.

Tools that come handy:

SETSPN (For adding, listing, deleting SPN entries for a domain)

KERBTRAY (on client machine)- For checking the Kerberos ticket used by the client to access a web server. It gives you information as to which ticket is being used by the client to access the IIS server, and whether the ticket is capable of delegation.

Deleg Config(server side tool) that adds a application under the side you are trying to browse which gives you information if the website is working for single hop and double hop

Klist(client side) to check which all tickets I am using.

Netmon

How Kerberos Works:

Overview in general:

Single domain Scenario

clip_image001

1) Client will contact kdc saying that it’s a user and it needs a ticket to get tickets (TGT)

2) KDC will give the TGT to the client by encrypting with the client machine password hash.

3) When client wants a service ticket, it will produce its TGT(for authorisation) and request TGS.

4) Based on auth the service ticket is given to the client.

5) Then the client passes its TGS to the network server or resource machine for authentication.

6) After authentication, normal client server response request process continues.

Multiple Domain Scenario

clip_image002

Both Child1 and Child2 are in the same forest with the same parent domain R2dom.local.

Administrator of the Child domain (CHILD1) login to a member server (CH1-Mem) in CHILD1 domain.

After login in the user tries to access \\r2dom-ch2-Mem1 . R2dom-ch2-Mem1 is a member server in Child2 domain.

--> I have used Network monitor to analyze and understand how Kerberos authentication would work, when accessing resource across domain.

Below you see that the Administrator is getting the required Kerberos tickets when accessing resources across domain.

10.2.1.2   = DC of CHILD1.R2DOM.LOCAL

10.1.1.2   = DC of R2DOM.LOCAL

10.3.1.1   = Dc of CHILD2.R2DOM.LOCAL

10.10.10.1   = CH1-Mem in CHILD1

10.10.10.1   10.2.1.2   KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

                                                                           --> KrbApReq: KRB_AP_REQ (14)

                                                            Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/CHILD1.R2DOM.LOCAL

 

** Here you see a Kerberos TGS request being sent to the local domain (CHILD1.R2DOM.LOCAL) DCs for a SPN cifs/r2dom-ch2-Mem1. Local domain TGT sent in the TGS request.

10.2.1.2   10.10.10.1   KerberosV5:TGS Response Cname: Administrator

                                             --> Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL

** Here the local domain DC returns a TGT of the Parent domain R2DOM.LOCAL. This is like a referral being sent to the client as the local domain does not have the right to issue a Kerberos Ticket for cifs/r2dom-ch2-Mem1.

10.10.10.1   10.1.1.2   KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL

--> KrbApReq: KRB_AP_REQ (14)

   Ticket: Realm: CHILD1.R2DOM.LOCAL, Sname: krbtgt/R2DOM.LOCAL

** Now the client machine send a TGS request to the Parent domain R2DOM.LOCAL, requesting for a TGT of another of its Child domain where the cifs/r2dom-ch2-Mem1 resides. When sending this request to the parent domain, the client uses the TGT of the Parent domain received in the earlier referral from local DC.

10.1.1.2   10.10.10.1   KerberosV5:TGS Response Cname: Administrator

                                             --> Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL

**Parent domain sends the TGT of the child2 domain to the client. This can also be taken as a referral to the CHILD2 domain.

10.10.10.1   10.3.1.1   KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

                                             -->KrbApReq: KRB_AP_REQ (14)

                                                            Ticket: Realm: R2DOM.LOCAL, Sname: krbtgt/CHILD2.R2DOM.LOCAL

** Eventually the Child sends a TGS request for the SPN cifs/r2dom-ch2-Mem1 to the DCs in domain CHILD2.R2DOM.LOCAL who is authorized to issue a ticket for the server (r2dom-ch2-Mem1) in its domain. This time the client uses the CHILD2 domains TGT to make request.

10.3.1.1   10.10.10.1   KerberosV5:TGS Response Cname: Administrator

                                             --> Ticket: Realm: CHILD2.R2DOM.LOCAL, Sname: cifs/r2dom-ch2-Mem1

** Finally the client gets the Kerberos Ticket for  cifs/r2dom-ch2-Mem1, which will help the Administrator user access the shares on r2dom-ch2-Mem1 in domain CHILD2.R2DOM.LOCAL.

The workstation the user 'Administrator" is using to access the resource across domain, also needs a similar ticket.

Tickets for r2dom-ch2-Mem1 is requested from local domain, who returns the error KDC_ERR_S_PRINCIPAL_UNKNOWN, as the r2dom-ch2-Mem1 computer is not a part of the local domain (CHILD1.R2DOM.LOCAL)

10.10.10.1   10.2.1.2   KerberosV5  KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: r2d-ch2-Mem1

10.2.1.2   10.10.10.1   KerberosV5  KerberosV5:KRB_ERROR  - KDC_ERR_S_PRINCIPAL_UNKNOWN (7)

Below you see that the client computer is getting the required Kerberos tickets when accessing resources across domain.

10.10.10.1   10.2.1.2   KerberosV5  KerberosV5:TGS Request Realm: CHILD1.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

10.2.1.2   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

10.10.10.1   10.1.1.2   KerberosV5  KerberosV5:TGS Request Realm: R2DOM.LOCAL Sname: krbtgt/CHILD2.R2DOM.LOCAL

10.1.1.2   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

10.10.10.1   10.3.1.1   KerberosV5  KerberosV5:TGS Request Realm: CHILD2.R2DOM.LOCAL Sname: cifs/r2dom-ch2-Mem1

10.3.1.1   10.10.10.1   KerberosV5  KerberosV5:TGS Response Cname: CH1-Mem$

Kerberos from IIS perspective , how it works:

1) Client sends a HELLO to the server

2) Server replies with a 401(sub status 2) Unauthorized telling the client that it requires to authenticate itself before accessing the page using www authenticate header (negotiate provider).

3) Client requests the Domain controller saying I am trying to access a particular website(say bing.com by specifying it as http/bing.com) from a particular server.

4) Domain controller looks through its SPN (service principal name) table for the particular application or site which will have the entries as below.

Service| Name of the service registered| registrar name (account name)

Eg: HTTP| bing.com| Chiranth

This means that bing.com is registered for the account Chiranth

Once it finds an entry in the table then DC will encrypt the token with the password hash of the account and send it to the client

5) The Client now sends the token to the server.

6) The server uses the account’s password hash to which the spn is registered(will use machine account if you have kernel mode authentication checked or will use application pool credentials if you have useapppoolcredentials as true in IIS (system.webServer/security/authentication/windowsAuthentication) to decrypt the token. If the server is able to successfully decrypt the token the requested page is given back.

NOTE : To register SPN use the command

setspn –a http/[fqdn server name] domain\account name

Specific Scenario when cx wants to register the host or machine spn to domain account, follow the below steps:

setspn –a http/<computer-name> <domain user account>
setspn –a http/<FQDN> <domain user account>

1) When you use kernel mode authentication we use Host SPN’s password for encryption and decryption of the token  by DC and server

2) So if you register an SPN for the account name check to useapppoolcredentials in the configuration editor of windows authentication.

Why Kerberos and not NTLM:

· Kerberos authentication offers the following advantages over NTLM authentication:

· Mutual authentication. When a client uses the Kerberos v5 protocol for authentication with a particular service on a particular server, Kerberos provides the client with an assurance that the service is not being impersonated by malicious code on the network.

· Delegation support. Servers that use Kerberos authentication to authenticate clients can impersonate those clients and use the client's security context to access network resources.

· Performance. Kerberos authentication offers improved performance over NTLM authentication.

· Simplified trust management. Networks with multiple domains no longer require a complex set of explicit, point-to-point trust relationships.

· Interoperability. Microsoft's implementation of the Kerberos protocol is based on standards-track specifications recommended to the Internet Engineering Task Force (IETF). As a result, the implementation of the protocol in Windows 2000 lays a foundation for interoperability with other networks where Kerberos version 5 is used for authentication.

Basic Troubleshooting steps:

What should we do when we see that we are getting 401 unauthorized or windows authentication failure.

Client Side

1) Check if Windows Integrated Authentication is enabled on the browser. Automatic logon with current username and password or, Automatic logon only in Intranet Zone.

2) Check if the URL is on Intranet Zone Sites.

Server side

3) Check if Basic Authentication is working. If it is not working, then it is not Kerberos issue.

4) If basic works, check for NTLM and if it doesn’t work, we should involve AD.

5) In a successful scenario when Kerberos works you will see a 401.2 and 200

clip_image001

6) If it is going over NTLM we will see 401.2 401.1 and 200 in fiddler.

clip_image001[7]

7) Also to check if the token is Kerberos or NTLM, the Kerberos ticket usually starts with Y and usually NTLM token starts with TLRM.

Kerberos Ticket eg:

Authorization: Negotiate YIIQiwYGKwYBBQUCoIIQfzCCEHugMDAuBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKw

YBBAGCNwICHgYKKwYBBAGCNwICCqKCEEUEghBBYIIQPQYJKoZIhvcSAQICAQB

ughAsMIIQKKADAgEFoQMCAQ6iBwMFACAAAACjgg6sYYIOqDCCDqSgAwIBBaEc

GxpGQVJFQVNULkNPUlAuTUlDUk9TT0ZULkNPTaIeMBygAwIBAqEVMBMbBEhUVFAb

C2NoaXJhbnRoOHIyo4IOXTCCDlmgAwIBEqEDAgEQooIOSwSCDkewzlFPUeEgCJxtUmS

7i1lNBiKYS9OCRPEHdyUytqKqs9SY

NTLM Token eg:

Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIIAAACUAZQBmgAAAA4ADgBYAAAADAAMAGYAAA

AQABAAcgAAAAAAAAAuAgAABYKIogYBsR0AAAAPnYnxQFaTOYGkUPDVgZ7Sck

YAQQBSAEUAQQBTAFQAYwBoAGkAcgBhAG0AVAAtAEMASABJAFIAQQBNA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAN6Op+WbvZB0Tchmb777NmIBAQ

AAAAAAAB5e5pMbts4B1nm7EWkuiDMAAAAAAgAOAEYAQQBSAEUAQQBTAFQA

AQAWAEMASABJAFIAQQBOAFQASAA4AFIAMgAEADQAZgBhAHIAZQBhAHM

AdAAuAGMAbwByAHAALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAAMA

TABjAGgAaQByAGEAbgB0AGgAOAByADIALgBmAGEAcgBlAGEAcwB0AC4AYwBv

AHIAcAAuAG0AaQBjAHIAbwBzAG8AZgB0AC4AYwBvAG0ABQAkAGMAbwByAHAA

LgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAAcACAAeXuaTG7bOAQYABAACA

AAACAAwADAAAAAAAAAAAQAAAAAgAACXZ723Po2/mq+xx6VA97FdnWz3JNKjqzWG

OYp01eYk9QoAEAAAAAAAAAAAAAAAAAAAAAAACQAgAEgAVABUAFAALwBjAGgAa

QByAGEAbgB0AGgAOAByADIAAAAAAAAAAAAAAAAA

8) To verify this we can copy this token value and decode it from base64 in fiddler tools option.

9) Create a test page and test with windows authentication(single hop).

10) SPN configuration

a. Check if Kernel mode Authentication is enabled, if yes, then spn must be registered for machine account

b. If Kernel mode is not enabled and app pool identity is domain account then SPN must be registered for the domain account

c. If Kernel mode authentication is enabled and useapppoolcredential is set to true(as described above) and app pool credential is domain account, then the SPN must be registered for domain account.

11) To check if u have duplicate spn’s u can try the commands.

To find duplicate spn’s on server

Use command prompt to execute this command

Setspn –x

Or you could run the below in DC

To find duplicate SPNs
Use command prompt to execute this command on dc:

ldifde -f <filename> -d "<dc=domain-netbiosname,dc=primary-domain>" -l serviceprincipalname -r "(serviceprincipalname=<serviceprincipalname-to-check-for-duplicates>)" -p subtree
e.g. if the domain name is test.abcd.com:
ldifde –f C:\log.txt -d "dc=test, dc=abcd, dc=com"-l serviceprincipalname –r "(serviceprinicpalname=http/test.abcd.com)" -p subtree

On the server side u can run

Setspn –l  acctname

Also you can run the below command to list the spn’s in the entire FOREST for the particular Hostname.

Setspn –F –Q service/hostname

Eg:

Setspn –F –Q */chiranth8r2.com

This will list all the service spn’s for chiranth8r2.com and also will tell you to which account it is registered.

Note : If everything looks fine and still single hop is failing. check which account is application pool running under. if it is running under application pool identity, change it to local system and give it a try. if test with local system succeeds, then there is a hotfix from Microsoft for it to work under applicationpoolidentity. Below is the hotfix link.
http://support.microsoft.com/kb/2545850

12) Create a virtual directory under the site pointing it to a shared location with a simple page(double hop delegation test). If the test fails follow step 13 or if it succeeds follow step 14.

13) (For double Hop) If your test page works check if the trusted for delegation setting is checked in active directory for the account to which machine is registered. If you are using kernel mode you have to trust the machine or host for delegation. To check if machine account is trusted for delegation got ‘Active directory users and computers’, right click on the machine name on go to the tab delegation and select the second option ‘Trust this computer for delegation to any service’ or the third option. If you are using app pool credentials trust the app pool account for delegation. Same above steps should be followed for app pool account in AD.

14) If trusted for delegation is already present and test with shared location succeeds, then there must be some issues with the SQL spn’s. Below are the steps we can follow.

15) How to Verify if SQL is good to except kerberos

1.First thing you would like to check is if your SQL is already capable to Kerberos or not. Simple way to verify this is to see if there is any existing connection using Kerberos. You could do this by running following command against your SQL instance and reviewing the result. If, even a single connection says Kerberos, You are good and don’t need to proceed further.

select auth_scheme from sys.dm_exec_connections

There could be combination of these protocols you would typically see as a result of above query.

NTLM, Kerberos, SQL

2. If the list does not show Kerberos, We would proceed further.  Now, we have to check if Service Principal Names (SPN) for your SQL instance are configured properly or not.

               In command below, I am presuming that your SQL box is called MySQLServer.

               You domain is called MyDomain.

               SQL Service account is SQLServiceAccount

Check the Service account of your SQL Service and run following command against setspn .

SetSPN -L MyDomain\SQLServiceAccount

If this is a standalone default SQL instance, It should return following two Service Principal Names (SPN).

MSSQLSvc/MySQLServer.MyDomain.com:1433

MSSQLSvc/MySQLServer:1433

If this is a named instance  listening on TCP port 4545, SPNs should look like this.

MSSQLSvc/MySQLServer.MyDomain.com:4545

MSSQLSvc/MySQLServer:4545

If this is a Clustered default SQL instance, It should return following two Service Principal Names (SPN).

MSSQLSvc/MySQLVirtualName.MyDomain.com:1433

MSSQLSvc/MySQLVirtualName.MyDomain.com

If this is a Clustered named SQL instance  listening on TCP port 4545, SPNs should look like this

MSSQLSvc/MySQLVirtualName.MyDomain.com:4545

MSSQLSvc/MySQLVirtualName.MyDomain.com 

3.If you see any of the required set of the SPN missing, You need to create it explicitly. For example you run following command.

SetSPN -L MyDomain\SQLServiceAccount

And it returned only one SPN .

MSSQLSvc/MySQLServer:1433

You have to create another required one using following command .

SetSPN -A MSSQLSvc/MySQLServer.MyDomain.com:1433 MyDomain\SQLServiceAccount

Please note, adding SPNs requires domain admin privileges. Once added successfully, You could confirm it using following command. SetSPN -L MyDomain\SQLServiceAccount it should return .

MSSQLSvc/MySQLServer.MyDomain.com:1433

MSSQLSvc/MySQLServer:1433

4.Once you have set of SPNs in place, your SQL Server is good to accept Kerberos authentication. You have to close your existing applications or connections and open them again to initiate fresh Kerberos connections.

5.In some cases, You would see your connections still using NTLM (see point 1). This would happen most probably because of similar SPNs existing under some other Service account, These would be considered as duplicate SPNs. This can happen if you have your SQL Service account being changed recently. As a result, There would be old SPNs still present under old account. We can find duplicate SPNs by running following command using SetSPN.

SetSPN -X 

In case there is same SPN registred under some other account, say OLDSQLServiceAccount, You would see it in output of SetSPN -X. You can then get rid of this duplicate SPN using following command.

SetSPN -D MSSQLSvc/MySQLServer.MyDomain.com:1433 MyDomain\OLDSQLServiceAccount

16) The dns entry for the url should be marked as an A record or else it will give u intermittent prompts for username and password(two types a record and c record).

17) Disable LoopBack Check if you are facing the problem only on the Server Side. (but able to browse the url from clients, as on server locally it works over ntlm)

18) You can also enable Kerberos event logging. On information about how to enable Kerberos event logging. see below or the kb http://support.microsoft.com/kb/262177

Start Registry Editor.

Add the following registry value:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters

Registry Value:

LogLevel

Value Type:

REG_DWORD

Value Data:

0x1

If the

Parameters

subkey does not exist, create it.


Note
Remove this registry value when it is no longer needed so that performance is not degraded on the computer. Also, you can remove this registry value to disable Kerberos event logging on a specific computer.

Quit Registry Editor. The setting will become effective immediately on Windows Server 2008, on Windows Vista, on Windows Server 2003, and on Windows XP. For Windows 2000, you must restart the computer.

You can find any Kerberos-related events in the system log.

19) You can check in deep by downloading delegconfig tool and put that application inside the website you are trying to browse.

20) If none of the above steps help, we can collect a netmon trace on client, server and AD simultaneously and see the communication. We can see if we are getting a service ticket(TGS) and also isolate if the issue lies in AD side, client side or server side.

SPN BLOGS and Kerberos Troubleshooting Blogs:

http://blogs.iis.net/brian-murphy-booth/archive/2007/03/09/the-biggest-mistake-serviceprincipalname-s.aspx

http://blogs.msdn.com/b/saurabh_singh/archive/2007/01/29/kerberos-troubleshooting-from-iis-perspective.aspx

http://blogs.msdn.com/b/saurabh_singh/archive/2008/11/16/who-knows-you-may-be-missing-these-points-for-kerberos-authentication-failures-for-web-applications.aspx

Leave a Comment
  • Please add 3 and 3 and type the answer here:
  • Post
  • Chiranth, excellent post! Keep up the good work.

    Jeff

  • thanks Jeff :)

  • Awesome Blog Chiru :)

  • Very useful blog Chiranth :)

Page 1 of 1 (4 items)