I spend a lot of my time analysing the performance of web sites and tuning the applications to make the sites run more efficiently and scale better. Over time I’ve pulled together a checklist of some of the more common performance issues that I see and how to resolve them, and I thought it was about time I shared them here.

Most of the issues I’ve identified are straightforward to fix (many are just configuration changes) and can give significant improvements to the scalability and the responsiveness of your web site. Some of them you may well already be aware of and I’m still amazed how many of the more obvious ones don’t get implemented as a matter of course, but then it keeps me in a job!

This post is broken down into three sections; the first is cold start improvements, or “why does my web site take so long to start up?”. This involves looking at what the IIS worker processes (w3wp.exe) are doing during initialisation, prior to completing the initial client request that caused them to launch.

The second section, which is generally more important, looks at the efficiency of processing requests once the server has “warmed up”, and is hopefully the state that the web site will spend most of its time in!

Finally the third section provides a general discussion around accessing SQL Server and web services from within web applications. These aren’t necessarily quick wins and may involve some changes to interfaces (or even the solutions architecture) to successfully implement, Finally, I’ll reveal the three golden rules for producing fast, scalable applications that I’ve derived from my investigations.

For each issue, I've included a brief description and references to where more information on the issue can be obtained and how to resolve it.

Cold start

• If the application deserializes anything from XML (and that includes web services…) make sure SGEN is run against all binaries involved in deseriaization and place the resulting DLLs in the Global Assembly Cache (GAC). This precompiles all the serialization objects used by the assemblies SGEN was run against and caches them in the resulting DLL. This can give huge time savings on the first deserialization (loading) of config files from disk and initial calls to web services.

http://msdn.microsoft.com/en-us/library/bk3w6240(VS.80).aspx

• If any IIS servers do not have outgoing access to the internet, turn off Certificate Revocation List (CRL) checking for Authenticode binaries by adding generatePublisherEvidence=”false” into machine.config. Otherwise every worker processes can hang for over 20 seconds during start-up while it times out trying to connect to the internet to obtain a CRL list.

http://blogs.msdn.com/amolravande/archive/2008/07/20/startup-performance-disable-the-generatepublisherevidence-property.aspx

http://msdn.microsoft.com/en-us/library/bb629393.aspx

• Consider using NGEN on all assemblies. However without careful use this doesn’t give much of a performance gain. This is because the base load addresses of all the binaries that are loaded by each process must be carefully set at build time to not overlap. If the binaries have to be rebased when they are loaded because of address clashes, almost all the performance gains of using NGEN will be lost.

http://msdn.microsoft.com/en-us/magazine/cc163610.aspx

Warm start

• Turn on IIS Compression (this is off by default on IIS6 and IIS7.x). Here are references for IIS6 and IIS7.x respectively:

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/d52ff289-94d3-4085-bc4e-24eb4f312e0e.mspx?mfr=true

http://technet.microsoft.com/en-us/library/cc771003(WS.10).aspx

• Turn on IIS content expiry (again this is off by default). See here:

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/0fc16fe7-be45-4033-a5aa-d7fda3c993ff.mspx?mfr=true

• Ensure ViewState is turned off wherever possible, and then only enable it for the controls that really need it. ViewState can be huge and is on by default. Turning off ViewState can radically reduce the size of pages and significantly improve page load times over slow links. Read a discussion on ViewState here:

http://msdn.microsoft.com/en-us/library/ms972976.aspx#viewstate_topic9

• Merge static content into as few files as possible (e.g. only have one .js file and .css file). It is faster to download one big file that several smaller ones over the internet. Ideally, merge .js and .css files into a single download as discussed here:

http://blogs.msdn.com/b/shivap/archive/2007/05/01/combine-css-with-js-and-make-it-into-a-single-download.aspx

• Enable ASP.NET output caching wherever possible. This will need to be looked at on a page by page basis, but it can yield huge gains for pages that contain mostly static content. See here:

http://support.microsoft.com/kb/323290.

• Enable web service output caching where possible. If there are any web services that always return the same results for any given set of input parameters this should be investigated. See here:

http://support.microsoft.com/kb/318299.

• If a web application makes web service calls, increase MaxConnections in machine.config to handle the level of concurrency the application needs to support. See "Threading Explained" here:

http://msdn.microsoft.com/en-us/library/ms998549.aspx#scalenetchapt06_topic8. However, don't take this as an absolute rule. For MaxConnections it is OK to go higher if the middle tier makes lots of long running web service calls.

• Change the IIS threading configuration in machine.config to support better scaling for long running requests. See "Threading Explained" here: 

http://msdn.microsoft.com/en-us/library/ms998549.aspx#scalenetchapt06_topic8

• If ASP.NET Ajax is being used, ensure <compilation debug=”false”/> is set in web.config. This avoids very costly parameter validation on both the client and server. See a full description of how to do this here:

http://msdn.microsoft.com/en-us/library/bb398817.aspx

• If  concurrency locks are needed on commonly accessed resources that are read mostly (e.g. configuration) make sure ReaderWriterLockSlim is used to protect them. Using a standard .NET lock on commonly accessed, read mostly resources can seriously impact scalability.

http://msdn.microsoft.com/en-us/library/system.threading.readerwriterlockslim.aspx

SQL Server and web services

• Run SQL Profiler against the solutions database whilst hitting all the key web pages. Identify all SQL operations that have high durations or CPU values and review them with an eye to optimising them. Also identify how many SQL operations are involved in the rendering of each page and see if any of them can be coalesced – aim for the goal of at most one SQL call to render any page. The new Tier Interaction Profiler (TIP) in Visual Studio 2010 is excellent for finding and measuring the SQL calls used to render individual pages:

http://blogs.msdn.com/b/habibh/archive/2009/06/30/walkthrough-using-the-tier-interaction-profiler-in-visual-studio-team-system-2010.aspx

• Look at any web services that are called, make sure they are called only once for any given set of inputs and cache any relevant data they return if that data needs to be used again. 
• If several web service calls to the same backend are needed to complete a request, can a new single web service be implemented to return all the data in one call? The fixed overheads for making web service calls can be very high which includes the cost of serializing\deserializing the request\response, establishing a TCP session to the server and actually sending the request\response over the wire. For lightweight requests the time for executing this overhead can easily exceed the time for executing the request itself! To at least partially mitigate this overhead, if WCF, IIS 7.x and private web services are being used, look at implementing endpoints using NetTcpBinding which avoids all the serialization and normally has a smaller footprint on the wire. The following article discusses how to setup a NetTcpBinding endpoint:

http://msdn.microsoft.com/en-us/library/ms752250.aspx

The golden rules for scalable, high performance applications

So here we are at my three core rules for writing fast, scalable applications. Beneath each rule I’ve included an example describing a typical scenario the rule applies to. You will find that at least one will apply to every scenario I list above, (try seeing if you can work which rule(s) apply in each case! I leave it as an exercise for the reader…). These rules apply to all applications, not just web apps. If they are kept in mind when designing and implementing any application, you’ll be a long way down the road to producing a fast, scalable application out of the starting blocks!

Never do anything more than once.

Cache everything that is likely to be used more than once. It is a cardinal sin to retrieve the same data from a SQL Server, web service or configuration file twice!

Don’t do anything you don’t need to.

Don’t be tempted to reuse an existing stored procedure or web service which returns large quantities of data when only a small subset of what it returns is actually needed – create a new stored procedure or web service which gets just the data required.

Get everything in one go.

Don’t use lots of fine grained web service or database calls when one chunky call can be used instead – the fixed overheads of making such calls are very high and repeated use should be avoided. Aim for the panacea of one web page executes at most one web service or SQL call.

  Written by Richard Florance