Darwin's Idle Times...

Replication troubleshooting...

2008-01-29T20:32:00Z

A customer has custom code that uses the Microsoft Provisioning Framework to do the following in their respective order:
1) Create a user
2) Create a Sharepoint site (When the Sharepoint site is created, roles are also created.)
3) Add the user created in 1) to a role defined for the Sharepoint Site created in 2)

When the code executes 3) the following error occurs:

MPF Response: System.Exception -

The customer needed assistance in successfully executing step 3)

The issue is found to be related to AD synchronisation. When the Secured Channel for each Sharepoint Server is the same as the preferred domain controller in the MPS requests, the issue does not reproduce. If you are experiencing what appears to be the same issue in your environment, you might want to confirm using the following command:-

netdom reset <target machine name> /Domain:<domain name> /Server:<preferred DC>

If you are experiecing the same issue, you may want to consider changing the domain topology so that servers naturally point to the appropriate DC
a. Create a site in AD
b. Associate the subnet with the Sharepoint Servers to the new Site
c. Move the preferred DC to the new site

Re-Introduction

2007-03-02T21:04:00Z

In case you are wondering where I have been or what I've been doing, I have left Syndey's sunny shores to live London on a kind of personal life experiences enrichment program :-). I'm just about settled in now in my new home, getting used to the colder weather, and getting into the swing of things in my new role. I am now an EMEA Solutions Escalation Engineer.

So what on earth is a Solutions Escalation Engineer? The official title of my team is the Solutions Support Team. We are a member of the Premier Field Engineering Organization (PFE) in EMEA. I fancy describing our specific goal being to minimize the chances of solutions related issues occurring in production and to improve the availability of the solution system.

The solutions that we work with can be classified as either one of two things. Microsoft Solutions or Custom Solutions. Microsoft Solutions include Hosted Messaging and Collaboration, Connected Services framework, Windows Based Hosting etc. (You can find the complete list here.) Custom solutions are those which may or may not have been developed by Microsoft consulting services and partners but meet our criteria of supportability.

As you can imagine, Solutions Support covers a broad range of technologies. We collaborate closely as team, and work alot with our other colleagues in the PFE organisation as well as Customer Services Support to achieve our objectives with customers.

My primary roles include providing support and delivering proactive services for HMC, CSF, and custom solutions. Proactive services activities include things like Supportability Reviews, Health Checks, and Compliancy Checks. Solutions Support also delivers workshops.

So where to from here? In addition to developer support related blogs, you will start to see ones related to Microsoft Solutions such as HMC and CSF. Plus, my learnings over the custom solutions that we support. I hope you find some of my banter useful in someway.

Authenticating via Kerberos against a web service using a C++ client...

2006-08-30T10:53:00Z

I just thought this information might assist in expediting your research into the matter.

When using the automatically generated proxy class template that is created when referencing a web service from a C++ client application, you may define the proxy as per the following example:-

Service::CServiceT<> svc;

The template uses the CSoapSocketClient class by default. This class only supports NTLM authentication with the web service.

If you require that the C++ client application to authenticate via Kerberos with the web service. Use the CSoapWininetClient as per the following example when declaring the proxy.

Service::CServiceT<CSoapWininetClient> svc;
CComBSTR str;
CoInitialize(NULL);
{
HRESULT res = svc.HelloWorld(&str.m_str);
this->MessageBox(str.m_str,0,0);
}
CoUninitialize();

The SecureSOAP Sample may assist you further with implementing a C++ client against a web service.

Delegation and LDAP

2006-03-29T15:09:00Z

A customer had a requirement to pull pieces of information about the browser user from AD in their web application. Accessible to the web application is the browser user's alias and the domain that the browser user belongs to.

The customer achieves this in their development environment by doing a serverless bind to AD and using a DirectorySearcher object to look up the entry corresponding to the user. The following is sample code which illustrates what I mean:

string ldapPath = "LDAP://" + userDomain;
DirectoryEntry rootEntry = new DirectoryEntry(ldapPath);
using (DirectorySearcher ds = new DirectorySearcher(rootEntry, "(samAccountName=" + userName + ")"))
{
            SearchResult result = ds.FindOne();
            if (result != null)
            {
                        ResultPropertyValueCollection resultValues = result.Properties["displayName"];
                        if (resultValues.Count > 0)
                        {
                                    Label1.Text = (string) resultValues[0];
                        }
                        else
                        {
                                    Label1.Text = "No display name";
                        }
            }
}

Upon migration of the application to the production environment, the application fails with a COM exception error code of 0x80072020. What could be the issue?

When the customer accesses the web page at the web server, the customer does not encounter the error. When the customer sets the authentication of the web application to Basic Authentication and browses the web page from a remote machine, the web page also worked as expected after entering correct credentials.

This evidence would suggest a Kerberos Delegation issue.

However, the web application also accesses SQL server. We obtained SQL Server Profiler traces of the access and we were able to determine that the access is made under the browser user's account. We were able to deduce from this that the issue is isolated to accessing AD via Kerberos Delegation.

The source of the issue is that the userDomain parameter supplied is the netbios name of the user's domain. What you should specify in the LDAP bind string is the Fully Qualified Domain Name (FQDN) of the domain. In other words, you should use an LDAP bind string of the form LDAP://<FQDN>. For example, LDAP://department.company.com. This is best practice and is essential for Kerberos Delegation to work against AD.

If you want to determine the FQDN of a domain programmatically, the following sample code may assist:

DirectoryEntry rootEntry = new DirectoryEntry("LDAP://" + userDomain + "/RootDSE");
string strRoot = rootEntry.Properties["defaultNamingContext"].Value.ToString();
string strFQDN = Regex.Replace(Regex.Replace(strRoot, @"DC=", ""),",",".");

RootDSE allows Anonymous binds and gives you a means of obtaining the appropriate designations for objects in AD that you can use in your LDAP bind strings. The following articles explain RootDSE and may assist you with your choice of a method to bind to AD:

Serverless Binding and RootDSE
Binding to Active Directory
Binding to the Global Catalog

If you are still encountering the symptoms that my customer experienced after changing the LDAP bind string to use the FQDN designation of the domain, seek the assistance of an expert in the domain or network infrastructure of your environment. This person is likely to be able to assist you with the correct FQDN to use and/or assist you in determining the FQDN dynamically.

Kerberos Authentication with IIS

2005-10-28T08:41:00Z

So you have decided to go with the delegation security model for your application. You have been developing all of the tiers of your application on your single development box where everything worked according to expectations. You have now separated all of the tiers into separate boxes for testing because this is the deployment scenario in production.

You have gone through the checklists in the Troubleshooting Kerberos Delegation article and made sure all the settings are correct. However, your multi-tier application is failing to flow the identity of the end user to the back end. Something is wrong, but where?

Multi-tier applications that use Kerberos Delegation can be difficult to troubleshoot because it takes a single incorrect configuration setting or problem somewhere along the chain to stop the identity flow. The most logical place to start IMHO is the communication between the client and the first middle tier. Before any Kerberos Delegation can occur, the client and first middle tier must be correctly configured for Kerberos Authentication.

A customer had IE as the client and a web application as the first middle-tier. To quickly learn if the client is authenticating via Kerberos, I asked the customer to create a "Negotiate Test" ASP page. The ASP code below is taken from "How To Use Kerberos with the ServerXMLHTTP Component in MSXML".

<HTML>
<HEAD>
<TITLE>Negotiate Test</TITLE>
</HEAD>

<BODY>
<%
Dim AuthMethod
Dim AuthType
Dim AuthLength
Dim AuthOther

' Get the authentication method being used
AuthMethod = Request.ServerVariables("AUTH_TYPE")

' Get the length of the HTTP_Authorization header (to determine Kerberos or NTLM)
AuthLength = Request.ServerVariables ("HTTP_Authorization")

' If some other authentication method (other than Negotiate) gets used, call it "Other"
If LTrim(RTrim(AuthMethod)) <> "Negotiate" Then AuthOtherMethod

' If Negotiate gets used, go straight to the subroutine to handle it
If LTrim(RTrim(AuthMethod)) = "Negotiate" Then AuthNegotiateMethod

Sub AuthOtherMethod()
            ' Since anonymous authentication will be blank, let's be sure we realize it's enabled to
            If LTrim(RTrim(AuthMethod)) = "" Then AuthMethod = "Anonymous"
            Response.Write "<table width=500>The user was logged in using the <B>" & AuthMethod & "</B> authentication method."
            Response.Write "<P>    If you were expecting a different method to be used,"
            Response.Write " please check the settings for the resource you are accessing. Remember, selecting"
            Response.Write " multiple authentication methods, or allowing anonymous access can result in a "
            Response.Write " different method being used.</table>"
End Sub

Sub AuthNegotiateMethod()
            ' Typically, NTLM will yield a 150 - 300 byte header, while Kerberos is more like 5000 bytes
            If LEN(AuthLength) > 1000 Then AuthType = "Kerberos"
            If LEN(AuthLength) < 1000 Then AuthType = "NTLM"
            Response.Write "<table width=500>The <B>Negotiate</B> method was used!<BR>"

            ' Tell us the Authentication method used to authenticate the user (and show a warning about the script)
            Response.Write "The user was logged on using <B>" & AuthType & "</B>."
            Response.Write "<P><font color=#800000><B>Please do not refresh this page</B></font>.<BR>"
            Response.Write "    If you do use refresh, <B>Kerberos</B> will always show up as <B>NTLM</B>."
            Response.Write " This is because the HTTP_Authorization header is being used to determine the authentication method used."
            Response.Write " Since the second request is technically unauthenticated, the length is zero. Please open a new browser"
            Response.Write " for any subsequent requests.</table>"
End Sub
%>
</BODY>
</HTML>

I asked the customer to situate this ASP page where his web application pages are physically located on his hard drive. If the ASP page is browsed, you can expect to see something like the following if there is Kerberos authentication between the client browser and IIS.

The Negotiate method was used! the user was logged on using Kerberos.

One thing that compounds the difficulty of troubleshooting Kerberos delegation is some form of caching. It is such that you don't know sometimes if a configuration change has fixed the problem. Because a change appears to not have had an impact on the Kerberos delegation problem, you may be tempted to reverse the change and/or move on to check/change something else.

When you are using IE to browse the "Negotiate Test" ASP, be sure to configure IE to check for newer versions of stored pages with every visit to the page. You can do that by using the following steps:

1) On the Tools menu, click Internet Options.
2) From the General tab, click Settings in the Temporary Internet Files panel.
3) In the Settings dialog box, click to select Every visit to the page in the Check for newer versions of stored pages check box.

Another thing to make sure about is that the browser sees the target web server as being in the Local Intranet Zone. This is easy to determine in IE by having a look at the bottom right hand corner of the IE Browser Window when you browse to "Negotiate Test" ASP. It should say "Local Intranet". If it doesn't say "Local Intranet" you can configure the browser to recognize the web server as a member of the Local Intranet Zone by using the steps in "Intranet site is identified as an Internet site when you use an FQDN or an IP address". Also, the Local Intranet zone within Internet Explorer needs to be set to "Automatic logon only in intranet zone". This setting can be changed within Internet Explorer by doing the following:

1) On the Tools menu, click Internet Options.
2) Click on the Security Tab.
3) Highlight Local Intranet and click on the Custom Level button.
4) Scroll to the bottom of the window and change the setting under the User Authentication, Logon section.

When the customer browsed to the "Negotiate Test" ASP, we quickly determined that NTLM authentication was occurring. The customer has gone through the magic checklists but he did not set up the web server machine. The machine has been used by other developers and administrators for various testing purposes so the settings on the Default Web Site may have been tampered with. We checked the NTAuthenticationProviders setting under the Default Web Site and found the setting to be just "NTLM". By default, the NTAuthenticationProviders key is not defined and IIS uses "Negotiate,NTLM". Having "Negotiate,NTLM" means that it would try Negotiate method first then NTLM second. For Kerberos Authentication, at least "Negotiate" has to be specified for the NTAuthenticationProviders.

Because we did not want to interfere with what the other developers or administrators were doing, we created the NTAuthenticationProviders key under the relevant virtual directory node in the IIS Metabase for the customer's web application. We then assigned this the value of "Negotiate". After this change, the "Negotiate Test" ASP page showed Kerberos authentication. When we tested the actual web application pages after this change, it was evident that the identity of the end user was being flowed to the back end.

You can set NTAuthenticationProviders for a specific virtual directory in the Default Web Site using something like the following command. Have a look also at "How to configure IIS to support both Kerberos and NTLM authentication" for another example and a corresponding output:

cscript adsutil.vbs set w3svc/1/Root/<virtual Directory Name>/NTAuthenticationProviders "Negotiate,NTLM"

To Delegate Or Not To...

2005-10-26T12:08:00Z

As part of my role at Microsoft, I sometimes assist customers in making design decisions with their applications. After understanding what the requirements are for the application, I seek relevant information or resources based on the requirements. Any relevant information I find, I will forward to the customer and outline the advantages and disadvantages of the options available.

One of the things I have been asked to assist on concern whether or not delegating the credentials of the end user to the backend resource in a multi-tier application is suitable. The following is a list of advantages and disadvantages that I think applies under most scenarios. Some of the items are quoted directly from MSDN articles and some are from Microsoft Consultants that I have worked with. I hope this will help you in making a decision as to whether or not Kerberos Delegation is appropriate security model for your application.

Delegation Model Advantages

The main advantages of the Delegation Model are the following:

The resource is always accessed as the user. More specific security is enabled at a resource level rather than at the task level. Resource level security is also useful if the resource is accessed from outside the application (such as through a report generator).
System level access auditing. Since all resources are accessed as the real user on all tiers of the system, you can enable system level access auditing on all tiers. This level of auditing can be very helpful when trying to meet new regulatory requirements for auditing and access reporting.

Delegation Model Disadvantages

Performance.
- No database connection pooling at the middle tier. This reduces scalability and performance.
Security.
- Misuse of delegation could make the network vulnerable to sophisticated attacks using Trojan horse programs that impersonate incoming clients and use their credentials to gain access to network resources.
  http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vsent7/html/vxconaspnetdelegation.asp.
- Direct user access available to resources. In some cases users may be able to directly access back-end resources using their normal credentials without going through the preferred application. For example, a user might use a SQL client to run commands directly against the database tier instead of through a business application.
Complexity
- The turnaround time for resolving issues can be lengthy. Assuming there is no problem on the domain/network, there is a checklist of necessary settings to verify for each tier in the architecture as well as Active Directory.
- Authorization may still need to be maintained on each of the middle tiers and introduce operational complexity.
- Not all tiers support the impersonation model for every scenario. A typical example is the back-end service that is hosted on a mainframe or other non-Microsoft server that does not support Kerberos protocol authentication.

Useful Reference:

The following article describes two architectures that you may employ, Delegation Architecture versus Trusted Subsystem, and lists the advantages and disadvantages of each.

Developing Identity-Aware ASP.NET Applications
http://www.microsoft.com/technet/security/topics/identity/idmanage/P3ASPD_1.mspx

Kerberos Delegation to SQL Server

2005-10-19T11:41:00Z

One of the most difficult issues to troubleshoot sometimes is a Kerberos Delegation issue to SQL Server.

A customer had a web application that is configured to access a remote back end SQL server under the security context of the end user. When users access the web application, the application displays an error similar to the following:

“Login failed for user '(null)'. Reason: Not associated with a trusted SQL Server connection.”

On the web server, when we browse the web application, it works as expected.

Fortunately, we were able to deduce very quickly that the middle tier is configured correctly for Kerberos Delegation. The customer had another SQL Server hosting the same database. When they pointed it to this server, it worked as expected. SQL Profiler showed that access is being made under the end user's account. So the issue appeared specific to Kerberos Delegation to the targeted SQL Server. What is it that is causing Kerberos Delegation to fail against this one SQL Server?

There are a couple of things to check out, quite a few actually! The following link provides a comprehensive checklist for the SQL Server backend as well as troubleshooting tips:

Troubleshooting Kerberos Delegation
http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/tkerbdel.mspx

I believe that the above article provides all the information you need to be able to troubleshoot and resolve most of your Kerberos Delegation problems on the Windows platform.

In the customer's environment, both the working and the failing SQL Server services are running under the same domain account. My immediate suspicion is an SPN problem. During setup, SQL Server can be configured to run under the Local System account. When SQL Server Services are started on a machine, it tries to register a corresponding SPN in AD against the account that it is running under. If SQL Server is running under local system, the SQL Server SPN would be registered against the machine name. If SQL Server is running under a domain account, provided that account has enough rights to AD, SQL Server will register the SPN against that account.

We verified that the correct SPNs are registered against the domain account that SQL Server services is running under on the problem machine. I then had the customer check the SPNs against the machine name and found SQL Server SPNs registered against the machine account:

MSSQLSvc/<Machine FQDN>:1433
MSSQLSvc/<Machine>:1433

Okay, so that is one configuration problem that we need to rectify. Using SETSPN, we deleted these entries but the problem persisted. What else could it be? Well, were not done with looking for duplicate SPN entries yet. There may be other accounts that the SPN may be registered to. We used the LDIFDE tool as per the article given to query AD for SPNs in an effort to weed out the problem. The following is the query that we used:

ldifde -f ldif.txt -j c:\ -d <your domain> -l serviceprincipalname -r (serviceprincipalname=MSSQL*)"

An example of <your domain> is "DC=microsoft,DC=com,DC=au" (including the quotes)

The resulting ldif.txt file showed that the SPN assigned to the account that SQL Server services are running under is also assigned to another domain account. We removed that SPN as well and after that change we had success!

Introduction

2005-10-19T03:53:00Z

I've been working for Microsoft now for six years as a Developer Support Engineer. For me that's a long time. It is the constant learning that has kept me here. I hope to share some of that learning by posting what I think are interesting customer problems and corresponding resolutions. Occasionally, I may write some meaningless banter.