We were running some load testing on a SharePoint site that will be hosting a company’s portal page.  The page relies heavily on ContentQueryWebParts that have custom XSL files specified.  Before starting the load testing, we were expecting to see the backend databases getting hit pretty hard due to some business requirements that prevented them from caching the entire page.  Once testing started, we were actually seeing the the Web Front Ends showing pretty high CPU; averaging 40-55% CPU across multiple tests, with spikes up into the 90+% range.  The SQL box wasn’t even breathing hard yet.

To see what was going on in the process, we started grabbing memory dumps of the w3wp.exe process a minute apart.  We did this over a few different time windows to allow us to look for trends.  Analysis of the memory dumps showed that the majority of threads were doing XSLT Compilation.  I say majority, but it was around 95% of the threads doing work were doing the same thing.  You see DataFormWebPart on the stack as that is the class that ContentByQueryWebPart class inherits from.

System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet
System.Xml.Xsl.Xslt.XsltLoader.Load
System.Xml.Xsl.Xslt.Compiler.Compile
System.Xml.Xsl.XslCompiledTransform.CompileXsltToQil
System.Xml.Xsl.XslCompiledTransform.LoadInternal
System.Xml.Xsl.XslCompiledTransform.Load
Microsoft.SharePoint.WebPartPages.DataFormWebPart.GetXslCompiledTransform
Microsoft.SharePoint.WebPartPages.DataFormWebPart
Microsoft.SharePoint.WebPartPages.DataFormWebPart.PerformSelect
Microsoft.SharePoint.WebPartPages.DataFormWebPart.DataBind
Microsoft.SharePoint.WebPartPages.DataFormWebPart.EnsureDataBound
Microsoft.SharePoint.WebPartPages.DataFormWebPart.CreateChildControls
System.Web.UI.Control.EnsureChildControls

XSLT Compilation can definitely take up CPU, so I looked at the % Time in JIT perfmon counter.  Normally, you would expect this counter to spike up when the process loads, but once things get compiled, it should drop down to 0.  In this case, the counter was in the 70-80% range throughout the test indicating that compilation was definitely running heavily and killing performance.  I then went to look at the XSLT files that were being referenced, and discovered one of the files was checked-out.  This seemed odd, so I looked into how the DataFormWebPart determines when to cache the compiled XSLT.

I then found that we only cache under the following conditions:

• ItemXSL property is set on the web part
• CacheXslStorage is set to true.  The default is true, if it is not specified.
• XSLT File is in a Published state…e.g. not checked out, and not a draft version.  Dotted versions are drafts, so if you have a 1.2 version file, you need to publish a major version.

We checked the XSLT file in, published a major version, then re-ran the test.  Performance was much, much better.  Using the same load test and load pattern, the CPU on the web front end dropped to an average of 10-15% across multiple tests.