Mohd Abubakr's take on Web Application Performance : IIS

lsass.exe and w3wp.exe: Analyzing SSL performance in IIS 6.0

abubakrmohd — Tue, 30 Dec 2008 15:59:12 GMT

In several of the performance tuning projects I have done, the web pages with security settings such as SSL or Windows authentication are generally slow compared to other pages. In this post, I analyze how SSL performance can be monitored in IIS 6.0

As per the architecture of IIS 6.0, security related operations such as SSL, Windows Authentication, Kerberos, etc are executed by lsass.exe process, whereas all other operations are executed under a process called w3wp.exe. Whenever, a web page with SSL is requested, IIS transfers the control to lsass.exe process to encode or decode.

Architecture of IIS 6.0

Analyzing SSL performance:

For the web applications having SSL enabled, the performance of the SSL can be isolated from the performance of the normal web pages by monitoring lsass.exe process. One of the simple techniques to monitor the SSL performance is logging the Perfmon lsass.exe % processor time counter which is available under process counters. Below is the sample graph that shows the Overall CPU utilization observed on the web server and %CPU utilization contributed by the SSL.

In the above graph, the green curve represents the overall CPU obtained on the Web server and red curve represents the %CPU utilization of lsass.exe process. One can clearly see in the above graph that lsass.exe process is one of the major contributors for CPU. Monitor w3wp.exe process for pages with SSL disabled.

Note: Lsass.exe process is not used in the IIS 7.0. The architecture of IIS 7.0 is different and doesn’t utilize lsass.exe process for SSL authentication.

Application Pool Properties– Worker Process Idle Time Shut down setting

abubakrmohd — Mon, 13 Oct 2008 18:21:29 GMT

From my past experience I have learned that most of the project teams are not aware of App Pool properties of IIS. They tend to rely on the default values of App Pool. Even though the default values are chosen to give better performance in general, the values might not be suitable for your project.

By default, as per the property in App pool setting, the worker process gets shut down after remaining idle for 20 minutes. One of the reasons for this 20 minutes shut down duration is,

Case Study:

In one of my recent projects, the customer asked me to resolve slow page loading issue. I asked him for the production IIS logs in order to find out the slow running .aspx pages. I sorted the pages based on the time-taken field in IIS logs and I was surprised to notice that /sitehome.aspx page was consistently clocking over 120 seconds.

URL	Time (ms)
\sitehome.aspx	149312
\sitehome.aspx	149228
\sitehome.aspx	149218
\sitehome.aspx	149046
\sitehome.aspx	148640
\sitehome.aspx	142984
\sitehome.aspx	140093
\sitehome.aspx	139609
\sitehome.aspx	138562
\sitehome.aspx	130703

Woo hoo… there you go! There problem is that, this application is being used by different customers across the world. Even though per day, only 10 customers access this application, however all of them reside in different regions. Obviously, they all have different business hours. Each time a new user hits the website, the IIS worker process has already timed-out and it has to restart. That’s the reason, for every user the /sitehome.aspx page was consuming over 2 minutes of time. Starting a worker process all over again is a time-consuming process.

I recommended the customer to increase the idle timeout value to something reasonable, say 120 minutes. Bingo! The /sitehome.aspx page consumes less than 15 seconds now.

Some details about this setting:

In an environment, where multiple websites are hosted on a same web server (IIS), it is important to conserve resources. Each IIS worker process instance consumes server resources such as CPU, memory, etc. Hence, if a particular IIS worker process is remaining idle over duration, one can shut it down so that other worker process instances get more CPU/memory.

However, if only one website is hosted on the web server (IIS), small idle timeout value might not be suitable. In the above scenario, the customer had 200 users across the globe and around 10-15 unique users visit the web application per day. Considering that around 1 or 2 user access the website per hour, if the request timeout is set to 1 or 2 hours, none of the users are going to face a situation where they have to restart the whole IIS process.

Web server deadlocks - aspnet_isapi.dll

abubakrmohd — Sun, 21 Sep 2008 04:56:34 GMT

There are various reasons for a deadlock can occur on the web server. In this blog post, I would be pointing out the various ways to resolve aspnet_isapi.dll related deadlocks on web server. If the deadlock has occurred on the web server, you should notice a warning or error on Event Viewer.

In one of the Apps I was working on the following was the warning message noticed on Event viewer.

“ISAPI ‘C:\windows\Microsoft.Net\Framework\v2.0.050727\aspnet_isapi.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected’.

One of the reason, a dead lock can occur on the web server is due to usage of COM Interop cross-context infertace in .NET 1.1 or .NET 2.0. However, this was not the problem on the application I was working. In case, your application is facing deadlock due to COM Interop cross-context interface, this KB articles would be helpful in resolving them.

http://support.microsoft.com/kb/921217/

http://support.microsoft.com/kb/915808/

Finally, after doing a bit of research here and there, I found the correct KB article that defines my problem. Here is the cause of this deadlock.

This problem might occur because ASP.NET limits the number of worker threads and completion port threads that a call can use to execute requests. Typically, a call to a Web service uses one worker thread to execute the code that sends the request and one completion port thread to receive the callback from the Web service. However, if the request is redirected or requires authentication, the call may use as many as two worker and two completion port threads. Therefore, you can exhaust the managed ThreadPool when multiple Web service calls occur at the same time.
For example, suppose that the ThreadPool is limited to 10 worker threads, and all 10 worker threads are currently executing code that is waiting for a callback to execute. The callback can never execute because any work items that are queued to the ThreadPool are blocked until a thread becomes available.
Another potential source of contention is the maxconnection parameter that the System.Net namespace uses to limit the number of connections. Generally, this limit works as expected. However, if many applications try to make many requests to a single IP address at the same time, threads may have to wait for an available connection.

This excellent KB article “Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications” http://support.microsoft.com/kb/821268 clearly defines this problem and gives out solutions.

If the above solutions haven’t resolved your deadlock, than you can generate a dump about your deadlock, which might reveal inner characteristics of it.

How to generate a dump file when ASP.NET deadlocks in IIS 6.0

IIS 6.0 has a new feature that is named Orphan Worker Process. This feature lets you inspect a process that is scheduled to be recycled before the process is terminated. The Orphan Worker Process can be used to attach a debugger to the process and to generate a dump file for investigation.

OrphanWorkerProcess tells IIS not to shutdown the worker process. IIS will leave the process running but won't send the worker process any more requests to execute. This setting lets the process stay in memory so you can jump in with tools like a debugger and see where the deadlock is occurring.

For more details on this technique, read on the following KB article http://support.microsoft.com/kb/828222/

Effect of earth’s rotation on IIS logs

abubakrmohd — Sat, 23 Aug 2008 00:53:00 GMT

I was amazed when I observed “UTC” time tag associated with IIS logs. In this blog post, I would be explaining what UTC time is, how it is different from GMT, how it is been used in IIS logs and the difference it makes. Being physics enthusiastic, it was fascinating to know how geographical physics is rooted within our computer systems. I was under the belief that GMT (Greenwich Mean Time) is the standard accepted in internet and computer networks. However, that’s not true.

The accepted standard for Internet protocols used in many WWW applications is UTC but not GMT. UTC stands for Coordinated Universal Time. Even though, there isn’t much difference between GMT and UTC, however UTC is far more accurate compared to GMT. UTC time standard is based on atomic time units with leap seconds added at irregular time intervals. It has been calculated that Earth’s rotation is slowing down by margin of 2 milliseconds a day, hence increasing the day time to 86,400.002 seconds. 2 milliseconds overhead each day sums up to 1 second after around 18 months. Hence, in UTC standard, one extra leap second is added once 1 second overhead is accumulated. Certain research papers have indicated that there might be acceleration in earth’s rotation due to movements in earth’s crust. However, the margin of deceleration is higher than acceleration; hence the more and more leap seconds will get inserted in the coming years.

One of the internet standard that uses UTC time format for logging is W3C standard. (http://www.w3.org/TR/NOTE-datetime). W3C logging standard is being used in Internet Information Services (IIS). If you enable the IIS logs, then the time field you see is in UTC format. Occasionally, if you see 1 second leaps in it compared to your clock, then it’s due to UTC time format. However, these leaps happen once in 18 months and you should be really lucky to notice it.

Having read all that, if must be wondering, what’s so interesting in this? All I am trying to convey is that how much care we are taking making our data accurate. The technologists involved in making this happen should be applauded for their extra effort in perfecting the system.

Monitoring w3wp.exe Process (www worker process)

abubakrmohd — Sat, 19 Jul 2008 12:57:50 GMT

One of the important process that you need to monitor while doing Performance testing is w3wp.exe process. It is an IIS related process and runs whenever IIS is configured in Worker Process Isolation mode. By default, IIS runs in Worker process Isolation mode. Each site hosted on IIS Server has it's own app pool. This app pool will have an instance of w3wp.exe process running.

When you are doing Performance testing of a web application, make sure that you log the Perfmon counter for %CPU utilization for the concerned w3wp.exe process. This counter is available under Process Object. This counter would give the %CPU consumed by w3wp.exe process while you were doing Performance testing.

Consider a situation where you haven't logged the counter for w3wp.exe process and you only have overall %CPU utilization on the web server. How can you be sure that obtained %CPU utilization is caused by IIS process? It could be due something running in the background on the Web server or maybe caused by Anti-Virus running on the server. Once you have w3wp.exe %CPU counter, you can accurately say that out of X% CPU utilization on web server, Y% was caused by w3wp.exe process.

In case, you find the %CPU of w3wp.exe much much lesser than the overall %CPU utilization, it means that there was another process that caused the high %CPU utilization. Below is the sample Perfmon graph comparing the overall %CPU utilization (red) and %CPU utilization of w3wp.exe process (brown).