DistributedWorld

AppFabric : Why is my local cache expiring too soon???

2013-02-14T10:37:00Z

Windows Server AppFabric 1.0 (or Microsoft AppFabric for Windows Server as 1.1 is now called) provides feature to enable local caching for enhanced performance. There can be a situation when AppFabric client could start deleting local cache much before expected (that is before configured Timeout, or Notification). One of the reasons of this could be if virtual bytes (total of virtual address space memory actually used by process) of client process touches 85% of RAM on machine.

To check if you are running into this situation, enable ‘Virtual Bytes Peak’ performance counter on AppFabric client process and graph it against total Physical Memory on client machine. If web application is your AppFabric client, the client process would be worker process (w3wp.exe) corresponding to its application pool.

WCF on TCP : Keep your connections alive

2012-06-15T08:15:46Z

I recently had a customer coming in with question and request on WCF connection pooling, which is very valid. I was almost convinced that answer to his question would be 'by design' but what worried me was if there is a solution for his predicament. To my pleasant surprise, mighty WCF team did think about the scenario and 'by design' there was a solution to his exact problem.

He is using net.tcp port sharing on the server for two different WCF services configured with net.tcp binding. The client code is such that proxy is created every time, service called and proxy closed. This sequence could be done for two services in any required combinations, lets say by calling Service 1() and Service2(). If Service1() is called repeatedly, TCP connection created in first call is pooled and reused. This is nice and as expected to save on connection establishment costs every time. But as soon as Service2() is called, TCP connection used for Service1() is reset. This sounds inappropriate. Seems, cost of port sharing is that pooled connection for Service1 has to be replaced with new connection to Service2. Very inefficient in an enterprise scenario where client makes random calls to Service1 and Service2 in volume as connection will be reset each time.

Whoever plays with TCP connection pooling is almost certain to stumble upon this: http://kennyw.com/work/indigo/173

Solution to my customer's predicament lies in this line taken from above link "Our connection pool is configurable through TcpConnectionPoolSettings/NamedPipeConnectionPoolSettings. These settings include a GroupName that we use for isolation"

You may think of it like this. TCP connection pool in WCF is identified with 'Port Number' and 'GroupName' (possibly more but only these are relevant for our purpose). If you omit 'GroupName', WCF provides a default. If you are not explicitly providing 'connectioPoolSettings', you are in effect omitting 'GroupName'. Since both the endpoints (for Service1 and Service2) use same port and have same 'GroupName' (Default), implies both endpoints will use same connection pool Id. When Service2 needs to be reached, its connection pool is already in place at client side but WCF infrastructure need to reset the connection to same port.

Use 'GroupName' to isolate connection pools for two endpoints and you can have connections alive when switching from Service1 to Service2 in there own respective pools.

This does mean that you will need to resort to custom binding but that’s an acceptable cost for such a huge benefit. A typical client config to achieve desired behaviour with net.tcp transport will look like this:

     <?xml version="1.0" encoding="utf-8" ?>

    <configuration>

      <system.serviceModel>

        <bindings>

          <customBinding>

            <binding name = "bindingA">

              <tcpTransport>

                <connectionPoolSettings groupName="connectionPoolA"/>

              </tcpTransport>

            </binding>

            <binding name ="bindingB">

              <tcpTransport>

                <connectionPoolSettings groupName="connectionPoolB" />

              </tcpTransport> 

          </customBinding>

        </bindings>

            </binding>

        <client>

          <endpoint address="net.tcp://&lt;hostname>/<APPLICATION-A>/Service.svc"

              binding="customBinding" bindingConfiguration="bindingA"

              contract="WCF1.IService" name="NetTcpBinding_IService_A">

          </endpoint>

          <endpoint address="net.tcp://&lt;hostname>/<APPLICATION-B>/Service.svc"

            binding="customBinding" bindingConfiguration="bindingB"

            contract="WCF2.IService" name="NetTcpBinding_IService_B">

          </endpoint>

        </client>

      </system.serviceModel>

    </configuration>

Identity delegation in WCF

2012-05-24T10:01:00Z

Further to my earlier post in an attempt to demystify Kerberos and IIS hosted WCF service (http://blogs.msdn.com/b/distributedworld/archive/2012/04/24/troubleshoot-kerberos-in-wcf.aspx) , I realized that successful delegation of client credentials to back end service/DB is also a common scenario and hence this short post. From WCF perspective, there are only three things to be done to successfully delegate client credentials :

1. First and foremost, client should allow itself to delegate. In WCF, this can be done by setting allowedImpersonationLevel="Delegation" on client side endpoint behavior

<clientCredentials>

       <windows allowedImpersonationLevel="Delegation"/>

</clientCredentials>

2. Second, service should be impersonating client. It could either be done in Declarative or Imperative manner per operation. You can get more details here : http://msdn.microsoft.com/en-us/library/ms730088.aspx

3. Third, and this is important and what often confuses users. The AD (Active Directory) object used to authenticate server/service should be trusted for Delegation. Essentially, what you need to figure out is if its 'Server' or 'Service' authentication, which is used in your scenario, which in turn is consequence of combination of 'useAppPoolCredentials', 'Kernel authentication' and 'Explicit service identity' (The table in my previous blog may come handy to find this out).

So for example, if your middle tier used Kernel authentication with useAppPoolCredentials="false", the middle tier server should be trusted for delegation (irrespective of Identity middle tier service is running as). On the other hand, if you did a 'Service' authentication with custom identity, that specific identity should be trusted for delegation (you will need to create a SPN for that identity in order to enable 'Delegation' tab in AD)

<Edit> I would like to add a 4th point here. Remember that backend WCF service's client is middle tier and this client will also want the service (backend) to authenticate itself. Same principles of Kerberos authentication apply here and easiest way around this is to provide backend WCF service identity a SPN (say "MyBackendSvc/MyBackendBox") and specify same in backend service and middle tier config

<endpoint..>

    <identity>

        <servicePrincipalName value="MyBackendSvc/MyBackendBox"/>

    </identity>

</endpoint>

Some useful tips:

All in previous post (like klist purge) are still useful : http://blogs.msdn.com/b/distributedworld/archive/2012/04/24/troubleshoot-kerberos-in-wcf.aspx
Following two properties to discover current process identity and caller identity come very handy while troubleshooting Identity issues.

// This is current process identity (will be same as caller if impersonating)

String user = System.Security.Principal.WindowsIdentity.GetCurrent().Name;

  

// This is caller's identity

WindowsIdentity callerWinIdentity = ServiceSecurityContext.Current.WindowsIdentity;

Kerberos and IIS hosted WCF : 'useAppPoolCredentials', 'Kernel mode authentication' and 'etc.'

2012-04-24T10:47:00Z

Having invested quite a deal of time in troubleshooting a WCF authentication issue this week, I thought of sharing the experience, as this can perhaps save some man hours in the future. Though I worked my way through multiple issues in course of this exercie, but at a higher level, this post can come handy if you are investigating an authentication failure of the type below of a WCF service hosted in IIS, with windows authentication enabled:

System.ServiceModel.Security.MessageSecurityException: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate, NTLM'

If you are trying to get your service being authenticated as a custom domain identity.

I also hope for this post to provide some directions to go around troubleshooting WCF authentication related issue in general as this is what I ended up doing in spirit.

Goal:

Aim of my exercise is to successfully use Integrated Windows Authentication (Kerberos and not NTLM) for the custom identity account under which the service is running. Why Kerberos and not NTLM?? In my case, I want to delegate client credentials from the middle tier WCF service to the backend Database and this is only possible with Kerberos (NTLM does not support delegation of credentials)

It’s worth mentioning that IIS 7 onwards, kernel mode authentication is enabled by default in IIS. Though Microsoft recommends this to remain enabled for reasons of performance and avoiding authentication related issues, there may be situations where someone really wants to authenticate against the custom identity and not the machine account and do it outside of Kernel mode (I was doing it to mimic IIS 6 days where kernel mode authentication was not an option). IIS 7 onwards, you can configure ‘Kernel-mode authentication’ under 'Advanced Settings' after enabling 'Windows Authentication' authentication type (Authentication feature of IIS, available at the server, site and application level)

Environment & Setup

I have a very simple WCF service configured with a single contract, basicHttpBinding and chose security mode 'TransportCredentialOnly' so as to avoid mixing it up with SSL and learning too much too soon. This is the relevant part of my service's web config :

      
      <system.serviceModel>

        <bindings>

          <basicHttpBinding>

            <binding name="binding1">

              <security mode="TransportCredentialOnly">

                <transport clientCredentialType="Windows"></transport>

              </security>

            </binding>

          </basicHttpBinding>

        </bindings>

        <services>

          <service name="MyService" behaviorConfiguration="Behavior">

            <endpoint name="endpoint1" binding="basicHttpBinding" bindingConfiguration="binding1" contract="MyContract">

              <identity>

                <servicePrincipalName value="MyService/MyMachine" />

              </identity>

            </endpoint>

          </service>

        </services>

    </system.serviceModel>

The service is hosted in IIS 7.5 (Windows 2008 R2) and running as identity 'EMEADS\SvcOwner'

If SPN in above config amazes you, with WCF client and service, you are free to venture out of SPN naming conventions (so forget about Service class, port etc.)

I find this MSDN magazine article quite useful to understand SPNs in general as well as wrt WCF : http://msdn.microsoft.com/en-us/magazine/cc163570.aspx

This is how I set the SPN for my service:

setspn -a MyService/MyMachine EMEADS\SvcOwner

Action time:

I disabled all other authentication schemes, keeping Windows Authentication only in IIS, right from the server level down through web site and up to my WCF application. I also disabled Kernel mode authentication at all levels under 'Advance Settings' for 'Windows Authentication'.

With SPN in place, I was all set to get pass the authentication when inevitable happened :

Unhandled Exception: System.ServiceModel.Security.MessageSecurityException: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate oWwwaqADCgEBomMEYWBfBgkqhkiG9xIBAgIDAH5QME6gAwIBBaEDAgEepBEYDzIwMTIwNDIwMDkzODU4WqUFAgMOUo6mAwIBKakOGwxFTUVBRFMuTE9DQUyqEzARoAMCAQGhCjAIGwZua3NuMiQ='. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized. ---> System.ComponentModel.Win32Exception: The target principal name is incorrect

Troubleshooting:

Win32Exception inner exception message often helps to indicate what is wrong here. "The target principal name is incorrect" suggest an issue with service Identity (SPN in this case). (For a quick experiment, I disabled the Active Directory account under which my client is running and though WCF MessageSecurityException remained same but inner Win32Exception message changed to "The logon attempt failed". Self-explanatory)

With the 'AllowNtlm’ property of WindowsClientCredentials set to ‘false’ (note: This property is now marked as obsolete-local machine policy should be used instead: http://msdn.microsoft.com/en-us/library/system.servicemodel.security.windowsclientcredential.allowntlm.aspx ) and 'Negotiate' provider enabled on WCF app, I knew I am troubleshooting Kerberos already. The first troubleshooting step was to enable Kerberos logging on my client box: http://support.microsoft.com/kb/262177

Reproducing the issue again, the following event was logged in the System event log:

-Log Name:      System
Source:        Microsoft-Windows-Security-Kerberos
Date:          20/04/2012 12:03:31
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      NKSN1.EMEADS.local
Description:
A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 11:3:31.0000 4/20/2012 Z
Error Code: 0x29 KRB_AP_ERR_MODIFIED
Extended Error:
Client Realm:
Client Name:
Server Realm: EMEADS.LOCAL
Server Name: nksn2$
Target Name:
Error Text:
File: 3
Line: 576
Error Data is in record data.

At this time, I noticed, even when my Kerberos logging was not enabled, there were these events in event log from earlier attempts of troubleshooting:

Kerberos event 4 :
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server nksn2$. The target name used was HTTP/nksn2.emeads.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the
server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (EMEADS.LOCAL) is different from the client domain (EMEADS.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

That was quite a wealth of info to give me directions.

setspn –X’ is the easiest way to trace duplicate SPNs ('duplicate SPN' is when the same SPN gets assigned to more than one accounts. 'Duplicate SPNs' is just one possible SPN related issue. Go through link in 'References' section below for complete SPN checklist for Kerberos). For some reason, I used an old school way to find which account(s) is (are) using this SPN: http://technet.microsoft.com/en-us/library/cc772897(v=WS.10).aspx

And I got this output from the LDP tool:

***Searching...
ldap_search_s(ld, "DC=EMEADS,DC=local", 2, "serviceprincipalname=MyService/MyMachine", attrList, 0, &msg)
Getting 2 entries:
Dn: CN=SvcOwner,CN=Users,DC=EMEADS,DC=local
canonicalName: EMEADS.local/Users/svcowner;
name: SvcOwner;
objectClass (4): top; person; organizationalPerson; user;

Dn: CN=iismanaged,CN=Managed Service Accounts,DC=EMEADS,DC=local
canonicalName: EMEADS.local/Managed Service Accounts/iismanaged;
name: iismanaged;
objectClass (4): top; person; organizationalPerson; user;

This confirms that duplicate SPNs are getting in my path. To start from a clean slate, I to deleted both the SPNs and created a new one, only for the custom identity the WCF service is running as (you need to be a domain administrator to run this):

Delete both SPNs:

setspn -d MyService/MyMachine emeads\iismanaged

setspn -d MyService/MyMachine emeads\svcowner

Create the SPN as I want it to be (for domain idenity under which the service is running)

setspn -a MyService/MyMachine emeads\svcowner

I thought I am close. Another run and again :

Unhandled Exception: System.ServiceModel.Security.MessageSecurityException: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate oWwwaqADCgEBomMEYW BfBgkqhkiG9xIBAgIDAH5QME6gAwIBBaEDAgEepBEYDzIwMTIwNDIwMDkzODU4WqUFAgMOUo6mAwIBKakOGwxFTUVBRFMuTE9DQUyqEzARoAMCAQGhCjAIGwZua3NuMiQ='. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized. ---> System.ComponentModel.Win32Exception: The target principal name is incorrect

Well, remember your client system caches SPNs for target identity, hence always run 'klist purge' after any SPN changes. Unfortunately, in my case, even 'klist purge' is of no consequence and I still run into the same exception. Perhaps because I was playing around with 'useAppPoolCredentials' setting in applicationHost.config: http://technet.microsoft.com/en-us/library/dd759186.aspx . I reverted everything to default on IIS (useAppPoolCredentials = false) and did an iisreset. The WCF client successfully negotiated credentials with service now and all is well.

Now is the time to demystify links between 'kernel mode authentication’, 'useAppPoolCredentials' and 'service SPN identity'.

After investing another great deal with various experiments and readings, these are the scenarios and respective implications:

Scenario 1. useAppPoolCredentials=false

1a. Kernel mode disabled. Client explicitly specifies service SPN Identity (Service identity can be of various types like Certificate, UPN etc) :

    <client>

      <endpoint....>

         <identity>

             <servicePrincipalName value="MyService/MyMachine">

         </identity>

      </endpoint>

    </client>

This succeeds with custom Identity credentials. But as already warned, this is non-performant as Kernel is not handling authentication

1b. Kernel mode enabled. Client explicitly specifies SPN Identity as above. As expected, authentication fails since the server is expecting host/http SPN.

1c. Kernel mode enabled. Client does not specify service SPN Identity. Authentication succeeds but with default host/http SPN. This is what we always recommend. Performant and (almost) no authetication issues.

Scenario 2. useAppPoolCredentials=true

2a. Kernel mode disabled. (With kernel mode disabled, useAppPoolCredentials is of no consequence as authentication is now onus of application host) Client explicitly specifies service SPN Identity. This succeeds with custom Identity credentials but non performant like 1a.

2b. Kernel mode enabled. Client explicitly specifies service SPN Identity. Authentication succeeds. Kernel authenticating against appPool identity. More performant and recommended for custom identity scenarios.

2c. Kernel mode enabled. Client does not specifies service SPN Identity. Authentication FAILS as client provided ticket for host/http SPN whereas Kernel mode authentication at server is configured to authenticate against appPool credentials.

Putting all of above in nice tabular form:

UseAppPoolCredentials	Kernel-mode Authentication	Client explicitly specify service SPN Identity	Server/Service Authentication Result
False (default)	Disabled	Yes	Succeeds (Service authenticated)
False (default)	Enabled	Yes	Fails (as Server authentication is attempted)
False (default)	Enabled	No	Succeeds (Server authenticated)
True	Disabled	Yes	Succeeds (Service authenticated)
True	Enabled	Yes	Succeeds (Service authenticated)
True	Enabled	No	Fails (as Server authentication is attempted)

The learning is that kernel mode authentication in IIS doesn't necessarily mean machine account. With 'useAppPoolCredentials=true', we are telling IIS to do kernel mode authentication with AppPool identity account (and not machine account). To be 100% sure, I verified network traffic using Netmon 3.4 and indeed kerberos ticket was issued for 'MyService/MyMachine' SPN and used successfully for service authentication with useAppPoolCredentials=true.

Some other tips:

- An easy way to rule out any SPN related issues is by running the service as ‘Network Services’ account where default host SPNs are used. That means, remove any ‘Identity’ related configs from client and service and test again.

- When Kernel-mode authentication is used and useAppPoolCredentials=’false’ (default), server ticket will be issued and decrypted with machine account . In this case, server Identity will be inferred from service URI and not required to be explicitly mentioned in client or service config. Refer : http://blogs.msdn.com/b/webtopics/archive/2009/01/19/service-principal-name-spn-checklist-for-kerberos-authentication-with-iis-7-0.aspx

-Remember to 'klist purge' after any SPN related change

-if duplicate SPNs are such a hassle, why are they even permitted. I hope to get an answer to this soon and update the post.

-For simple scenarios, your Authentication types should be coherent right from server to application level in IIS unless there are other considerations

-Remember to do iisreset after any change in applicationhost.config

How to force Kerberos with WCF:

Binding should specify clientCredentialType="Windows"

If your service is hosted in IIS 7.5, there is a new authentication provider 'Negotiate:Kerberos', which doesn't fall back to NTLM

How to force NTLM with WCF:

Binding should specify clientCredentialType="Ntlm"

WCF service host (IIS in most cases) should have NTLM as one of authentication provider in the list.

References:

http://blogs.msdn.com/b/asiatech/archive/2011/10/26/iis-7-kerberos-authentication-failure-krb-ap-err-modified.aspx

http://blogs.msdn.com/b/friis/archive/2009/12/31/things-to-check-when-kerberos-authentication-fails-using-iis-ie.aspx?wa=wsignin1.0