Todd's Blog : ASP.NET Performance

Logging 32bit ASP.Net Performance Counters on a Windows 2003 64bit OS

Toddca — Tue, 08 May 2007 21:25:00 GMT

Recently an issue came up where an admin was upgrading their Windows 2003 Server OS from 32bit to 64bit but they wanted to keep their ASP.NET application as is and running under WOW. The issue they hit was with monitoring performance counters. While using perfmon they can see the counter data they could not get these to log to a logfile. The solution here is to change the perfmon service to run under WOW64 (x86) if you want to log counters from a WOW64 process.

The following script sets this all up:

sc \\servername config sysmonlog binPath=%systemroot%\syswow64\smlogsvc.exe

Credit for the solution goes to Chris St.Amand of the Microsoft.com Debugging team.

An ASP.NET & SQL Server Performance Issue

Toddca — Tue, 15 Nov 2005 06:52:00 GMT

I have worked a few high profile performance related issues in the last six months which all resulted in the same root cause. Since most of my customers use SQL server and stored procedures to access data from their ASP.NET web sites I thought I would share with you my findings related to these cases and what you can do to prevent the issue from effecting you or your customers.

Typically with a web application if SQL server or some other external data source is causing a bottleneck these issues will come into Microsoft support as a web application/ASP.NET issue. One of the most common SQL slow downs is 'blocking'. Blocking is seen when locks are held to perform some action and other waiters or threads have to block or stop execution until that lock is released. For example a compiler lock is one that is held during the time a stored procedure is compiled. Holding locks for long periods of time or constantly needing to acquire a lock can severely effect throughput of both SQL and heavily dependent ASP.NET applications. With web applications most users use stored procedures as a method of interacting with the data layer. Not fully qualifying the stored procedure with the owner's name is one of the primary causes of compile locks contentions. This occurs when the following conditions are met:

- The user that runs the stored procedure is not the owner of the procedure.
- The stored procedure name is not fully qualified with the object owner's name.

For example, if user "dbo" owns object dbo.mystoredproc, and another user "Harry" runs this stored procedure with the command "exec mystoredproc," the initial cache lookup by object name fails because the object is not owner-qualified. (It is not yet known whether another stored procedure named Harry.mystoredproc exists, so SQL cannot be sure that the cached plan for dbo.mystoredproc is the right one to execute.) SQL Server then acquires an exclusive compile lock on the procedure and makes preparations to compile the procedure, including resolving the object name to an object ID. Before it compiles the plan, SQL Server uses this object ID to perform a more precise search of the procedure cache and is able to locate a previously compiled plan even without the owner qualification. For more information, see the following article: 263889 SQL blocking due to COMPILE locks ( http://support.microsoft.com/?id=263889 )

ASP.NET SQL Session State too hit this issue and needed to be corrected ( http://support.microsoft.com/?id=843400 ). Anyone running SQL Server Session State needs to install this fix. The fix involved changing our const strings that represented the various stored procs by prepending them with "dbo.". It was a very simple and non destructive change that many of our large customers have implemented with no side effect.

If you suspect you or your customer may be running into this issue or you want to determine if they are at risk and you have a dump of the process that is accessing the SQL server run the following command to dump out each commandString in each SQLCommand object.

Debugger Command: .foreach ( obj {!dumpheap -type System.Data.SqlClient.SqlCommand -short} ) {!duclean c+ poi( obj + 10) }

To run the command above you will need to load sos.dll and mdacexts.dll (not sure if this is public but you can substitute "!duclean" for "du" and it should work).

By analyzing the stored procedure names you can determine if you might be hitting this issue.

Here is a sample of what I found yesterday on one of my customers dump files:

0:000> .foreach ( obj {!dumpheap -type System.Data.SqlClient.SqlCommand -short} ) {!duclean c+ poi( obj + 10) }

BCC_GetSingle_SuperUsers
BCC_GetSingle_CurrentUserInput
BCC_GetSingle_Records
BCC_GetAll_Supplement
BCC_GetAllInvestigatorApprovers
BCC_GetAllCustomerApprovers
BCC_GetAllPrincipalInvestigators
BCC_GetAllProjectInitiators
BCC_GetAll_ResponsibleCostCenter
BCC_GetAll_ResponsibleCCGroupFromCC
BCC_GetAll_Customers
BCC_GetAll_CustomersWorkUnit
BCC_GetAllSysCreatedBy
BCC_GetUser_FilterCriteria
BCC_Add_FilterCriteria
BCC_Count_InvestigatorApprovers
BCC_Count_CustomerApprovers
BCC_Add_AttachedFiles

Once they prepend these stored procedure names (which you will find are typically implemented as const strings hopefully in a single or very few locations ) with "dbo." or another owner name the performance problem was no more.

IsInRole Is Slow

Toddca — Thu, 28 Apr 2005 19:23:00 GMT

Authentication to ASP.NET May be Slow

When using custom authentication modules and/or role based authentication in ASP.NET you may find that the authentication process is slow.

The IsInRole() method of WindowsPrincipal was implemented in 1.0 and 1.1 in such a way that depending on the circumstances you may hit a performance bottleneck. The first time you call IsInRole() each SID from the user's token is resolved back to a friendly name. These names are then cached internally in the WindowsPrincipal object for subsequent calls. The name passed into IsInRole() is then compared to each of the cached names to determine the return value (true/false).

The SID to name resolution process requires making a RPC call into LSASS on the local machine, from here the SID is either resolved (because it is owned by the local computer), pulled from SID cache, or forwarded to a higher authority DC. SID caching is only enabled on non DC machines and there is a max number of SID cache entries that can be stored and old entries are dumped to make room for new.

The poor performance can be the result of one or more factors such as 1) the user is a member of many groups, and/or 2) some of the groups that the user is a member are sourced from remote domains, that is, where local Domain Controllers (DC) are not available. Arguably #1 is a bit vague but I have seen customers that have users with 200-800 group memberships and have had just a horrible experience. The explosion in group membership can partly be attributed to the fact with Windows 2000 and Windows 2003 have the notion of nested groups. The Directory Services admin tools (Active Directory Users and Computers) shows the group membership a user is directly associated, however it does not walk the tree for nested groups. During authentication the tree is walked and the SIDs of each of these groups are added to the user's token.

Consider a user in Texas that is a member of 200 groups and 30 of which are sourced from a remote domain, that is, a domain where no local Domain Controller exists in his/her site. A call to IsInRole for this user would require about 200 RPC calls to one or more Domain Controllers and 30 of which that would be remote and may be slow to return their results. There is no batch SID to name resolution so each SID results in a call.

To mitigate these issues I have written a small class that derives from WindowsPrincipal and overrides IsInRole(). My implementation takes the role or name passed into the method and resolves this to a SID, caches the SID, and then does a call to the system API CheckTokenMembership() to check if the SID is in the user's token. Although the name to SID translation too hits a DC (maybe even a remote DC) the number of calls will be limited to the groups listed in the web.config files. The number is typically less than a dozen groups and once resolved to SIDs, no further calls to the DCs are made to satisfy the lookups. The call to CheckTokenMembership() is handled by Advapi32.dll all in process so not RPC calls are made here either.

Care should be taken when adding groups to the web.config files. Ensure they are not misspelled or have not been renamed and use the format domain\groupname to remove any ambiguities.

The results are that if you have 10 roles listed in your web.config there will be at most 10 calls to a DC for the life of the application/AppDomain. So 100 users configured as our user in Texas hitting the website with the below implementation will result in 10 calls to a DC whereas with the current implementation the same traffic to the website would result in up to 20,000 ( 100 * 200 ) calls to a DC. Of course if the users are members of many of the same groups and the website is not running on a DC the total count of DC calls will be reduced.

Note: Whidbey does not have this problem as the WindowsPrincipal has been totally reworked.

The FastPrincipal

namespace Utility
{
  public class FastPrincipal : System.Security.Principal.WindowsPrincipal
  {
    private const int ERROR_INSUFFICIENT_BUFFER = 122;  //from winerror.h
    private const int SecurityImpersonation = 2;        //SECURITY_IMPERSONATION_LEVEL enum from winnt.h
    private static SidCache s_sidCache = null;
    private static object s_cacheLock = new object();

    private class SidCache : System.Collections.Hashtable
    {
      public SidCache() : base()
      {  //ensure we clean up our native heap memory
        System.AppDomain.CurrentDomain.DomainUnload += new EventHandler( DomainUnload );
      }

      private void DomainUnload( Object sender, EventArgs e )
      {
        this.Clear();
      }

      public override void Clear()
      {
        foreach( Sid sid in base.Values )
        {
          sid.Dispose();
        }
        base.Clear();
      }
    }

    private class Sid : IDisposable
    {
      IntPtr _sidvalue;
      int _length;

      public Sid( IntPtr sid, int length )
      {
        _sidvalue = sid;
        _length = length;
      }

      public int Length { get { return _length; } }

      public IntPtr Value { get { return _sidvalue; } }

      public Sid Copy()
      {
        Sid newSid = new Sid( IntPtr.Zero, 0 );
        byte[] buffer = new byte[this.Length];
        newSid._sidvalue = Marshal.AllocHGlobal( this.Length );
        newSid._length = this.Length;
        Marshal.Copy( this.Value, buffer, 0, this.Length );
        Marshal.Copy( buffer, 0, newSid._sidvalue, this.Length );

        return newSid;
      }

      public void Dispose()
      {
        if( _sidvalue != IntPtr.Zero )
        {
          Marshal.FreeHGlobal( _sidvalue );
          _sidvalue = IntPtr.Zero;
        }
        GC.SuppressFinalize( this );
      }
    }//Sid

    public FastPrincipal( WindowsIdentity ntIdentity ): base( ntIdentity ){}


    public override bool IsInRole( string role )
    {
      Sid sid = null;
      bool ret = false;
      try
      {
        sid = GetSid( role );
        if( sid != null && sid.Value != IntPtr.Zero )
        {
          ret = IsSidInToken( ( ( WindowsIdentity )base.Identity ).Token, sid );
        }
      }
      catch {/*Don't allow exceptions to bubble back up*/}
      finally
      {
        if( sid != null )
        {
          sid.Dispose();
        }
      }

      return ret;
    }

    private bool IsSidInToken( IntPtr token, Sid sid )
    {
      IntPtr impersonationToken = IntPtr.Zero;
      bool inToken = false;

      try
      {
        if( DuplicateToken( token, SecurityImpersonation, ref impersonationToken ) )
        {
          CheckTokenMembership(impersonationToken, sid.Value, out inToken);
        }
      }
      finally
      {
        if( impersonationToken != IntPtr.Zero )
        {
          CloseHandle( impersonationToken );
        }
      }

      return inToken;
    }

    private Sid GetSid( string role )
    {
      Sid sid = null;

      sid = GetSidFromCache( role );
      if( sid == null )
      {
        sid = ResolveNameToSid( role );
        if( sid != null )
        {
          AddSidToCache( role, sid );
        }
      }
         return sid;
    }

    private Sid GetSidFromCache( string role )
    {
      Sid sid = null;

      if( s_sidCache != null )
      {
        lock( s_cacheLock )
        {
          Sid cachedsid = ( Sid ) s_sidCache[role.ToUpper()];
          if( cachedsid != null )
          {
            sid = cachedsid.Copy();
          }
        }
      }
      return sid;
    }

    private void AddSidToCache( string role, Sid sid )
    {
      if( s_sidCache == null )
      {
        lock( s_cacheLock )
        {
          if( s_sidCache == null )
          {
            s_sidCache = new SidCache();
          }
        }
      }

      lock( s_cacheLock )
      {
        if( !s_sidCache.Contains( role ) )
        {
          s_sidCache.Add( role.ToUpper(), sid.Copy() );
        }
      }
    }


    private Sid ResolveNameToSid( string name )
    {
      bool ret = false;
      Sid sid = null;
      IntPtr psid = IntPtr.Zero;
      IntPtr domain = IntPtr.Zero;
      int sidLength = 0;
      int domainLength = 0;
      int sidType = 0;

      try
      {
        ret = LookupAccountName(  null,
                                  name,
                                  psid,
                                  ref sidLength,
                                  domain,
                                  ref domainLength,
                                  out sidType );


        if( ret == false &&
            Marshal.GetLastWin32Error() == ERROR_INSUFFICIENT_BUFFER )
        {
          psid = Marshal.AllocHGlobal( sidLength );

          //LookupAccountName only works on Unicode systems so to ensure
          //we allocate a LPWSTR
          domain = Marshal.AllocHGlobal( domainLength * 2 );

          ret = LookupAccountName(  null,
                                    name,
                                    psid,
                                    ref sidLength,
                                    domain,
                                    ref domainLength,
                                    out sidType);



        }

        if( ret == true )
        {
          sid = new Sid( psid, sidLength );
        }

      }
      finally
      {
        if( domain != IntPtr.Zero )
        {
          Marshal.FreeHGlobal( domain );
        }
      }

      return sid;

    }

    #region /********IMPORTS***********/
    //only works on Unicode systems so we ar safe to go with an Auto CharSet
    [DllImport("advapi32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private extern static bool LookupAccountName(  string lpSystemName,
                                                   string lpAccountName,
                                                   IntPtr Sid, ref int cbSid,
                                                   IntPtr ReferencedDomainName, ref int cbReferencedDomainName,
                                                   out int peUse );

    [DllImport( "advapi32.dll" )]
    public extern static bool CheckTokenMembership(IntPtr TokenHandle, IntPtr SidToCheck, out bool IsMember);


    [DllImport( "advapi32.dll" )]
    public extern static bool DuplicateToken(   IntPtr ExistingTokenHandle,
                                                int SECURITY_IMPERSONATION_LEVEL,
                                                ref IntPtr DuplicateTokenHandle);

    [DllImport( "kernel32.dll" )]
    public extern static bool CloseHandle( IntPtr Handle );

    #endregion

  }//FastPrincipal
}

Example of how to use FastPrincipal in your web application.

private void Global_AuthorizeRequest(object sender, System.EventArgs e)
{
HttpApplication application = ( HttpApplication ) sender;

if( application.Context.User != null && //just to be safe
      application.Context.User is WindowsPrincipal && //only modify WindowsPrincipals
      //if not authenticated we don't have a token
      application.Context.User.Identity.IsAuthenticated == true )
{
    System.Threading.Thread.CurrentPrincipal =
    application.Context.User = new FastPrincipal( ( WindowsIdentity ) application.Context.User.Identity );
}
}