Alik Levin's : Performance

Architects UG: 16 Case Studies of ASP.NET Web Performance

alikl — Tue, 23 Jun 2009 22:38:44 GMT

Below is a slide deck for the talk I gave today on Architect UG. The presentation focused on 16 case studies of performance that is less than optimal. The session is focused on baking performance engineering into the development lifecycle.

Each case study has a link to the detailed walkthrough and relevant resources on how to improve performance.

Enjoy

Free Web Performance Tools From Microsoft, Google, Yahoo, And IBM

alikl — Thu, 11 Jun 2009 11:56:53 GMT

This post is a quick overview of free performance tools available from Microsoft, Yahoo, Google, and IBM. It also contains a pointers to related articles that go deeper regarding the best practices and how the tools can help in identifying compliance to the best practices.

Quick Resource Box:

Microsoft’s Fiddler (Performance Tuning with Fiddler)
Microsoft’s VRTA (12 Steps To Faster Web Pages With Visual Round Trip Analyzer)
Yahoo’s YSlow (rules for high performance web pages.)
IBM’s Page Detailer
Google’s Page Speed (Web Performance Best Practices)

Microsoft’s Fiddler

Fiddler is a free web performance tool, it is not really a property of Microsoft rather a side project by Eric Lawrence, a PM with Microsoft. I used Fiddler for both security testing and now for performance. I love it a lot. Must mention it requires Net Fx 2.0 as a prerequisite so it is limited to Windows OS. Recently Eric added support to Firefox – Fiddler Hook For Firefox, so the tool is great for both IE and FF. My related posts:

Microsoft’s VRTA

VRTA is a free web performance tool and it stands for Visual Round Trip Analyzer created by Microsoft’s Jim Pierson and used internally for sometime. It was made available for public use during last PDC 2008. Jim has written very detailed article about the tool and how it solves performance problems - 12 Steps To Faster Web Pages With Visual Round Trip Analyzer. VRTA installs and uses under the hood free Microsoft Network Monitor (Netmon) to capture and analyze network captures.

Yahoo’s YSlow

YSlow is a free performance analysis tool created by Steve Souders when he was with Yahoo. Steve created another good tool called Cuzilion. YSlowl comes with extremely good set of performance guidance that can be found here - rules for high performance web pages. YSlow requires Firebug as a prerequisite, meaning it is restricted to Firefox only.

IBM’s Page Detailer

Page Detailer is a free web performance tool from IBM. I was not able to identify any good articles that cover it – if you share with me please, or better off publish one. It does not have any prerequisites, consider it as an advantage.

Google’s Page Speed

Recently I stumbled on Page Speed from Google. It reminds me a Yahoo’s YSlow lot that makes me believe it comes from Steve Souders that works now for Google. It also requires Firebug as a prerequisite and works with FireFox only. It comes with nice guidance found here - Web Performance Best Practices. Must admit – I adore the concept of the tool although in most cases I cannot use it as I work for customers that IE is their target browser. Nevertheless the guidance is tool agnostic and I recommend bookmarking it for quick reference.

Distributed Architecture Drawbacks Revealed By Netmon(Bonus - TDS Parser Goes Public)

alikl — Fri, 13 Mar 2009 12:02:49 GMT

Distributed architecture can mercilessly backfire at you. In my case flexible architecture, elegant design patterns, and smart code led to abuse of the flexibility, resulting in very poor performance. Free Microsoft Network Monitor (Netmon) helped to identify the root cause of the

by striatic

performance hit. It showed that over-distributed-ness can cost you in terms of performance.

Customer Case Study

The customer complained about poor response times in his web application. The application's architecture was similar to the Web Application Archetype. Notice Services Agent box that connects your application to downstream services? Our assumption was that the services agents are too chatty causing the performance hit. Netmon only made it clear.

Analysis

We took captures on the application server where the Server Agents are to identify what other downstream servers are accessed. In no time we get very clear picture - the application server was accessing other three downstream resources:

There are two well known ports - 443 and 1433 - so we could safely assume there is communication over SSL/HTTPS and SQL Server. The other one - 1414 - turned out to be MQ.

Next step was to identify which one of the conversations is causing us troubles the most - either by chatty communication or by just long running transaction.

Looking at Time Delta column for HTTPS stateless communications we found nothing exciting regarding the latency:

For MQ communications we used magic ContainsBin(FrameData, 0,"StringToFindGoesHere") to identify XML messages going back and forth over MQ transport. For example, to find the beginning of the XML message we used the following filter:

Similar technique was used to correlate request and response XML messages.

To identify SQL communication we used shiny new TDS parser available for free download on Codeplex - SQL Parser in Latest CodePlex Package. I particularly like this one, it shows SQL Server communication without using SQL Server Profiler:

Conclusion

Lessons learned:

Distributed Architecture can mercilessly backfire at you by its distributed-ness. Use Distributed architecture to solve problems vs introducing new ones.
Elegant and flexible design can be abused resulting in poor performance. Review code to avoid the abuse.
Low level network monitoring tools such as Netmon can quickly reveal over-distributed-ness. Use it! It is free.

Related Materials

This post is made with PracticeThis.com plugin for Windows Live Writer

WinDBG Walkthrough - Dump Values Of DataSet or DataTable

alikl — Mon, 09 Mar 2009 09:44:02 GMT

This walkthrough is completely based on Johan's post WinDBG+SOS: Getting at the values in a DataTable. I have created this one to help me do the job in straightforward way next time I hit similar problem. Joan also offers few scripts for process automation - recommended.

by glennharper

Customer Case Study

The customer complained about potential memory leak. Following the procedure described in Identifying Memory Leak With Process Explorer And Windbg we realized that we are dealing with static variable that grows in unlimited way. This assumption is based on the fact that after running !gcroot on the leaking type we get the following:

HANDLE(Strong):23b1cd0:Root:0x12571730.....

After reviewing Tess' .NET Debugging Demos Lab 7: Memory Leak - Review we found the following which made us believe we are dealing with static variable:

DOMAIN(001CCE68):HANDLE(Strong) - Strong reference, Typically a static variable

To identify what this static variable is we needed to dump its values. The variable was a DataTable. I have not found a straightforward way of dumping contents of DataTable. This is the walkthrough that does the job.

Summary of steps

Step 1. Dump DataTables
Step 2. Dump DataTable
Step 3. Dump columnCollection
Step 4. Dump list object
Step 5. Dump raw memory - dd command
Step 6. Dump DataColumn
Step 7. Dump storage
Step 8. Dump values
Step 9. Bonus - automation

Step 1. Dump DataTables

The objective of this step is identify all DataTable object and pick the one of the interest

0:000> !dumpdatatables
DataTable       Rows    Columns    DataSet nextRowID ColumnCount
-----------------------------------------------------------------------------------------------
0x024dc948 0x024dcbc8 0x024dcdec 0x064f2400         1         2
0x025156b8 0x02515938 0x02515b5c 0x02515478         1         7
...

0x0e5b9ce4 0x0e5b9f64 0x0e5ba138 0x0e55a338       428         5
0x06510e54 0x065110d4 0x065112a8 0x064f2400     1,359         7
0x0e5778f0 0x0e577b70 0x0e577d44 0x0e55a338     1,359         7
0x0a55d270 0x0a55d4f0 0x0a55d6c4 0x06a62620 4,194,305        10
Total 61 DataTable objects

Step 2. Dump DataTable

The objective of this step is identifying the address of columnCollection of the DataTable.

0:000> !do 0x0a55d270
Name: System.Data.DataTable
...
0x176c1560 0x40003f2 0x18 CLASS instance 0x0a55d6c4 columnCollection
...

Step 3. Dump columnCollection

The objective of this step is identifying the address of list object of the columnCollection.

0:000> !do 0x0a55d6c4
Name: System.Data.DataColumnCollection
...
0x176c5ffc 0x4000377 0x8 CLASS instance 0x0a55d6f8 list
...

Step 4. Dump list object

The objective of this step is identifying the address of _items object.

0:000> !do 0x0a55d6f8
Name: System.Collections.ArrayList
...
0x79ba75ec 0x4000362 0x4 CLASS instance 0x0a55d710 _items
...

Step 5. Dump raw memory - dd command

The objective of this step is identifying addresses of DataColumn objects.

0:000> dd 0x0a55d710
0a55d710 01e5209c 00000010 79b92eec 0a55d854
0a55d720 0a55d8d4 0a55d954 0a55d9d4 0a55da54
0a55d730 0a55dad4 0a55db54 0a55dbd4 0a55dd74

Step 6. Dump DataColumn

The objective of this step is identifying the address of storage object.

!do 0a55d854
Name: System.Data.DataColumn
...
0x176c69e8 0x400036b 0x48 CLASS instance 0x0a55df40 storage
...

Step 7. Dump storage

The objective of this step is identifying address of value object.

0:000> !do 0x0a55df40
Name: System.Data.Common.Int32Storage
...
0x176ecc8c 0x4000729 0x10 CLASS instance 0x4dbd0030 values
...

Step 8. Dump values

The objective of this step is dumping the actual values, finally...

0:000> !do 0x4dbd0030
Name: System.Int32[]
...
Content: 8,388,608 items

Ouch, it is array.... !do -v will dump its values, but I am afraid it is not a good idea doing it for 8 million items here ;)

Step 9. Bonus - automation

The objective of this step is automate the process of dumping values (Step 8):

Create a text file named dumparray and save it in WinDBG directory. The contents of the file are:
- .foreach ( o { !do ${$arg1} -v -short }) { !do ${o} }
Run the following command in WinDBG to dump the values of the array
- $$>a< dumparray 0x4dbd0030

Related Materials

This post is made with PracticeThis.com plugin for Windows Live Writer

Identify And Fix Performance Bottlenecks In ASP.NET Web Applications - Case Studies, Problems, And Solutions

alikl — Sat, 07 Mar 2009 00:16:56 GMT

I thought it'd be useful to package the resources referenced in my previous post - Why My ASP.NET Application's Performance Is Less Than Optimal?
I packaged it in form of PDF file. Hope it'll be useful for you when dealing with performance bottlenecks in your ASP.NET web application.

Download "Identifying and Fixing Performance Bottlenecks" PDF file from my SkyDrive:

Feel free to distribute it. If you have suggestions for improvements or just a friendly constructive comment - feel absolutely free to submit one.

This post is made with PracticeThis.com plugin for Windows Live Writer

Why My ASP.NET Application's Performance Is Less Than Optimal?

alikl — Fri, 06 Mar 2009 11:01:39 GMT

From my recent engagements I collected few performance anti-patterns that make ASP.NET web application to perform the way that is less than optimal. Many related to architecture and design.
Below is the list of the anti-patterns and related materials on how to identify, analyze, and fix it.
Have fun - if you feel like sharing your own experiences - that would rock my world!

by striatic

Why My ASP.NET Application Slow?

Improper HTML Rendering
- ASP.NET Performance Sin - Serving Images Dynamically (Or Another Reason To Love Fiddler)
- Improve Web Application Performance By Reducing Number Of Http Requests - Fiddler To The Rescue
Large data volumes sent over the wire
- Identify ASP.NET, Web Services, And WCF Performance Issues By Examining IIS Logs
Complex algorithms, casting, serialization costs
- Best ASP.NET Performance Winner For Data Binding - Hands Up To Response.Write ()
Chatty database access
- Performance Sin - Chatty Database Access And Loops (Plus Another Free Performance Tool)
Session overuse – blows your RAM causing working process recycles

Identifying Memory Leak With Process Explorer And Windbg

Long running operations – locking on worker threads (threadpool)
Synchronous communications w/downstream resources (web services, databases)
- Improve ASP.NET 2.0 Performance With PageAsyncTask - Multithreading For The Masses
Locking files (notice the low CPU in the picture below - this is where the application is locked with teh following excetion)
- Exception message: The process cannot access the file 'c:\inetpub\wwwroot\webapp\my.log' because it is being used by another process.

Serialization
- Best ASP.NET Performance Winner For Data Binding - Hands Up To Response.Write ()
Dynamic compilation
- ASP.NET Performance: Dynamically Loaded Assemblies Cause Application Recycles (Problem and Solution)
ViewState
- How To Keep ASP.NET ViewState On The Server – Revised
- Improve ASP.NET Performance By Disabling ViewState And Setting Session As ReadOnly
Chatty resources access
- Performance Sin - Chatty Database Access And Loops (Plus Another Free Performance Tool)

This post is made with PracticeThis.com plugin for Windows Live Writer

Silverlight - Browser Cache And Performance Optimization

alikl — Fri, 27 Feb 2009 12:23:28 GMT

I was recently involved in the discussion about how Silverlight application is handled by the browser regarding browser cache. After quick research I have found two great articles by Dino Esposito that contain good explanation of the topic and also offers more caching solutions for improving the performance of pages that host Silverlight application

by emdot

Managing Dynamic Content Delivery In Silverlight, Part 1 – January 2009

"The XAP package that you get from the Web server has no special meaning to the browser. The browser, therefore, will cache it as it caches anything else it gets from a Web server, adhering to the request cache policies determined by the cache-control and "expires" HTTP header in the request or similar meta tags in the host HTML page.

A XAP package contains a manifest and one or more assemblies. One of the assemblies contains the entry point of the application; other assemblies are just referenced assemblies. The XAML for the user interface is stored in the resources of the entry point assembly. A XAP package is created by the Visual Studio 2008 extension for Silverlight 2 when you create and build the project."

Managing Dynamic Content Delivery In Silverlight, Part 2 – February 2009

"Caching the XAP package doesn't mean caching individual resources such as DLLs, XAML animations, or multimedia content. In the current implementation, resources are extracted from the XAP package every time they are used."

Related Materials

This post is made with PracticeThis.com plugin for Windows Live Writer

Use Process Explorer To Quickly Identify Performance Bottlenecks In ASP.NET Web Applications

alikl — Tue, 17 Feb 2009 21:18:35 GMT

Process Explorer is my best friend when I need to quickly identify potential performance bottlenecks for ASP.NET web applications. When I am called onsite I have it always with me.

It served me well recently so I thought it'd be proper to show my appreciation to the tool and to the creator, Mark Russinovich, by sharing how I use it. Every time I am called onsite to the customer's, first thing I ask to install ...ehm... copy the tool to the server in question. It usually shows hot spots right away and helps making assumptions and decisions.

by tacit requiem

CPU Usage History

This view helps to spot "unusual" activity for ASP.NET Web Application. When I see "too much" red color in this view I start asking question "Why Managed .Net Code performs privileged operations?" From my observations it mostly caused by excessive usage of threading.

Private Bytes Memory

This view has become my favorite recently as I was involved in few cases with memory leakage. I think the picture is self explanatory.

Identifying Memory Leak With Process Explorer And Windbg

I/O Bytes History

This view helps me spotting potential I/O bottlenecks.

This post is made with PracticeThis.com plugin for Windows Live Writer

Identifying Memory Leak With Process Explorer And Windbg

alikl — Sun, 15 Feb 2009 20:20:16 GMT

Your ASP.NET web application behaves strange. It gets stuck, it stop serving incoming requests, or gets recycled. You observe strange application pool recycles events in event log.
Chances you are dealing with memory leak in your ASP.NET web application. Lately I've been involved with few of such situations. Here is the method and the resources that helped me to extremely quickly identify the root cause of the memory leaks. I am super grateful to the resources published by Tess and Rico that discuss the process of memory leak identification in more detail:

by Sutto's Photography (was mns007)

Customer Case Study

The customer complained his application gets less responsive as more subsequent request come in. The customer reported the high CPU utilization. At some point the application gets stuck and the only way to get it back is..... iisreset. The customer felt there is a memory leak in the application.

Analysis - Summary of steps

Step 1 - Use Process Explorer To Confirm The Memory Leak
Step 2 - Take Memory Dump Using adplus.vbs
Step 3 - Use Windbg to identify the leaking Type
Step 4 - Use Windbg to identify to identify who's holding on the Type
Step 5 - Find the culprit in the source code

Step 1 - Use Process Explorer To Confirm The Memory Leak

I am using Process Explorer to quickly identify whether we are dealing with memory leak. In our case it showed we are. This is the memory consumption that Process Explorer reveals in very convincing way - you can clearly see how the memory consumption climbs up until the application hangs/restarts:

Next question is who's consuming the memory? Windbg to the rescue.

Step 2 - Take Memory Dump Using adplus.vbs

Use Windows Debugging Tools package to take a memory dump of the leaking managed process. Download the tools and extract on the server where the application runs. Open command prompt and change the directory to the folder that contains the tools, then type the following command:

adplus.vbs -quiet -hang -p <<PROCESSID>>

where <<PROCESSID>> is the process of w3wp.exe PID (process ID) that runs the leaking application. The result should be another folder inside of tools folder with very lengthy name. Open that folder- you should see a big size file with DMP extension. This is the memory dump you have just taken.

Step 3 - Use Windbg to identify the leaking Type

To analyze the dump file you will going to use Windbg.exe. The tool is located in the Debugging Tools for Windows folder, same folder that contains adplus.vbs that was used in the previous exercise. Open the tool, Windbg.exe. Press Ctrl+D, locate your DMP file and open it.

Now you need to load SOS.DLL extension to analyze managed memory dump. Type the following command according ro what .Net framework you are analyzing:

.load clr10\sos - for Net 1.X

.loadby sos mscorwks.dll - for Net 2.0 and up.

Type the following command to identify the Type that consumes most of the memory:

!dumpheap -stat

The result should be similar to the following:

.....
654088b4 92811 13736028 System.Data.DataColumn
000dec48 44 13900264 Free
654359c8 31704 32845344 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
7912d8f8 320769 47859900 System.Object[]
790fd8c4 827939 186834312 System.String

Total 3351270 objects

Most consuming is that System.String type that is responsible for almost 190 MB of allocated memory.

Step 4 - Use Windbg to identify to identify who's holding on the Type

Next question your should ask is who's holding on it so that it's not collected. For that purpose type the following command in Windbg but be prepared to hit Ctrl+Break to stop it as you do not want to list all 827939 strings, you are looking for a few to get its addresses:

!dumpheap -type System.String

The output should be similar to this:

057f9364 790fd8c4 4116
057fa4b8 790fd8c4 4116
057fb52c 790fd8c4 4116
057fc68c 790fd8c4 4116
057fd700 790fd8c4 4116
0583fef8 790fd8c4 4116

Next you want to start trying to identify who's holding on the objects. That can be done using the following command (I replaced the real Namespaces with XXXXXXXX to keep customer's privacy):

0:000> !gcroot 0583fef8
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
…
Scan Thread 27 OSTHread e40
DOMAIN(00102980):HANDLE(WeakLn):945e00:Root:19312e44(System.Web.Hosting.ISAPIAsyncCompletionCallback)->
190c2950(System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6)->
190c3138(System.Web.HttpContext)->
190c340c(System.Web.AspNetSynchronizationContext)->
0cbb80d0(ASP.global_asax)->
01b3a5b4(System.Web.HttpApplicationState)->
01b3a610(System.Collections.ArrayList)->
0a4e4c10(System.Object[])->
05846388(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
058331c8(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.XXXXXXXX)->
058333c0(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Statuses)->
058333cc(System.Object[])->
0583fe20(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Status)->
0583fee0(System.Object[])->
05840f0c(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Style)->
0583fef8(System.String)

Looks like the code using endlessly Application object sticking strings into it. Once there it's never collected until the end of life of the Application which caused only by recycle.

Step 5 - Find the culprit in the source code

Next is to look in the source code all the usages of assigning strings to Application object. After quick look up we identified the following code that is called recursively:

HttpApplicationState app = HttpContext.Current.Application;
for (int i=0; i < m_XXXXXX.Count; i++)
{
    string sProcessId = m_XXXXXX[i].SelectSingleNode("@XXXXXX").Value;
    app.Set (GetPrefix() + XXXXX, new XXXXXX(m_XXXXXX[i].OuterXml));
}

Conclusion

I have been using the following approach several times lately and it has proven to be very effective and efficient. It's simple too.

Happy memory leak analysis.

Related Materials

Avoid Manipulating Passwords In Memory - It Is Easy To Reveal

This post is made with PracticeThis.com plugin for Windows Live Writer

DebugView - Free Simple Tool To Quickly Identify Performance Problems

alikl — Tue, 11 Nov 2008 18:12:53 GMT

DebugView is well known free tool from Systinternals (Microsoft). Vast majority of developers is aware of its existence. Nevertheless, I've decided to write this post following my practice of journaling my technical accomplishments during engagements with the customers. Yesterday the tool helped us identify serious performance problem without installing profiler or even taking memory dumps.

Customer Case Study

The customer complained about "bad performance" of web pages. The application was pretty simple one - ASP.NET pages accessing SQL Server for simple queries. The SQL Server was installed on the same machine where ASP.NET app resided. The pages took up to ten seconds to render.

Analysis

After quick code review we found out that the code called function that traversed all server controls using recursion. This function was called several times during the page lifecycle. I presented the team with my assumption that this code is a potential culprit of the "bad performance". The opposite assumption I faced was "but there is only so few controls on the page, it cannot consume too much time."

We decided to add single line of code to the function in question:

System.Diagnostics.Trace.WriteLine ("Calling Recursive Function");

Then we fired up DebugView and ran the page.

The team was amazed by what DebugView presented interactively. "Calling Recursive Function" line kept piling up on and on. Each call took from 50 to 100 milliseconds but when multiplied to overall number of calls (~700) it gave us clear understanding what caused the performance hit.

Conclusion

Instrument your application properly - simple yet powerful technique that helps you avoid calling expensive consultants onsite.

Instrument your application and surprise your end users by high performing application.

Related Materials

This post is made with PracticeThis.com plugin for Windows Live Writer

SQL Server Performance Insights From The Pro

alikl — Thu, 06 Nov 2008 13:33:13 GMT

Jimmy May, my colleague and a good friend, started to blog recently. Jimmy is one of the most accomplished SQL Server Performance professionals. Point your RSS reader to Jimmy's RSS feed and start absorbing SQL Server Performance he readily shares.
Here are few recent pearls from Jimmy:

This post made with PracticeThis.com plugin for Windows Live Writer

ASP.NET Performance By Design: Takeaways From PDC

alikl — Mon, 03 Nov 2008 12:06:35 GMT

During PDC, there were 5 dedicated sessions for improving performance in .Net titled "Performance By Design". The presenters are Rico Mariani, Vance Morrison, and Mark Friedman. These guys live and breathe performance. Although I did not make to get to PDC, I was following after what's going on there. Fortunately, Vance published all slides on his blog. These are my takeaways from the slides he published.

Performance by Design Intro

By Rico Mariani

My favorite is the first slide that sets the expectations - Performance is about Culture:

Part 1 - Teaching Performance Culture
Part 2 - General Topics about Managed Code

I am so happy to see it! There is a perception about performance in the field - it is about tools. There is another perception - "we can fix performance after we build the app". I've been through few situations where such approach resulted in extra budget to "fix the performance issues", missed deadlines, and frustrated customers (and stakeholders). Rico says there are very few rules to follow:

Rule #1 - Measure
Rule #2 - Do your homework

Rico explains Performance Culture very simply but in powerful way:

Budget. An exercise to assess the value of a new feature and the cost you’d be willing to pay:
- Begin by thinking about how the customer thinks about performance
  - Responsiveness
  - Capacity
  - Throughput
  - Cost of Entry
- Identify the resource the customer views as critical to this system
- Choose the level of performance we want to deliver (do we need an “A+” or is a “D” good enough)
- Convert this into what resource usage needs to be to succeed
- Don’t think about the code, think about the customer
Plan. Validate your design against the budget, this is a risk assessment
- You can’t plan without a budget, so get one
- Use best practices to select candidate algorithms
- Understand their costs in terms of the critical resource
- Identify your dependencies and understand their costs
- Compare these projected costs against the budgets
- If you are close to budget you will need much greater detail in your plans
- Identify verification steps and places to abort if it goes badly
- Proceed when you are comfortable with the risk
Verify. Measure the final results, discard failures without remorse or penalty, don’t make us live with them
- The budget and the plan drive verification steps
- Performance that cannot be verified does not exist
- Don’t be afraid to cancel features that are not meeting their budgets – we expect to lose some bets
- Don’t inflict bad performance on the world

Bottom line - performance is about culture - tools only support it. Performance cost you either way - either you invest in it or not. Manage it as you'd manage any other risk.

My Related Posts

CPU Optimization for .NET Applications

By Vance Morrison

Vance supports Rico's Rule #1 - Measure. He provides several practical options:

Low Tech: System.Diagnostics.Stopwatch
Medium Tech: MeasureIt (Automates Stopwatch)
Medium Tech: Use ETW (Event Tracing for Windows)
Higher Tech: Sample Based Profiling.

Vance admits that ETW (Event Tracing for Windows) "...is not easy to use End-to-End:

We are working it
We will have more offerings in next release
It is a complete talk just by itself
If you need logging NOW you CAN use EventProvider, xperf
If you can wait a year, it will be significantly nicer.
If there is interest, we can have an ‘Open Space’ discussion"

My biggest takeaway from this session is that I know unforgivably too little about ETW - need to ramp up myself on it.

My Related Posts

ASP.NET Performance: High CPU Utilization Case Studies And Solutions

Memory Optimization for .NET Applications

By Vance Morrison

My favorite slide is #14:

"Fixing Memory Issues: Prevention!

Fixing Memory Issues is HARD
- Usually a DESIGN problem: Not Pay for Play
- Using every new feature in your app
  XML, LINQ, WPF, WCF, Serialization, Winforms, …
- Initialize all subsystems at startup
GC Memory Are your Data Structures
- Tend to be designed early
- Hard to change later
Thus it Pays to Think about Memory Early!"

What does it mean to think about memory early? The slide deck is packed with explanation about the measurement tools and the theory behind GC. I'd also expect to see few code samples - both patterns and anti-patterns. Mid-life crisis drill down would work for me.

My Related Posts

Parallelism for .NET Applications

By Vance Morrison

The only takeaway (beside good tips) is that Parallel Computing is getting into mainstream. From the field I see more and more demand for multithreaded work, I observe customers buying strong servers but do not utilize it to its capacity while asking to improve performance. I liked the structure of the session, especially "How .Net Can Help..." slides that offer practical tips and implementation suggestions to improve performance by parallelism.

My Related Post

ASP.NET Web Application Performance

By Mark Friedman

This is really huge slide deck - 110 slides... that covers tons of stuff.

I was looking for unusual stuff. I found it. Turns out Visual Round Trip Analyzer (VRTA) was released to the web. The tool was internal for some time and now it is available for the masses. Good news!

I also liked the slides about ETW for IIS. Especially the ETW Trace reporting tool , which is Excel ;). One statement made me feel alert - "Caching the same data in multiple places tends to be wasteful". Not the statement itself rather the relation to Velocity, which is MS distributed cache mechanism. Need to dig deeper. Overall, the slide deck is packed with very useful and practical recommendations spanning multiple technologies like ASP.NET, AJAX, and WCF.

My Related Posts

This template is made with PracticeThis.com plugin for Windows Live Writer

ASP.NET Performance: Fast AJAX, Faster AJAX

alikl — Wed, 29 Oct 2008 00:43:52 GMT

AJAX improves significantly both user experience and performance. It can be further improved by using down level capabilities that .Net framework offers. Specifically, consuming Web Services and WCF directly from client script. The best part is that ASP.NET AJAX comes with built in libraries - server and client - that make coding fun while significantly improving the web application's performance.

Customer Case Study

The customer was using hand crafted XmlHttp requests from client scripts requesting the data from ASPX pages. While the goal was achieved - the amount of information sent to the server was minimal and the user interface was responsive - the coding was not really fun. Also, since the requests were sent to regular ASPX pages the whole ASPX pipeline was executing unnecessarily utilizing CPU for nothing. The customer did not want to use Update Panel control. Although it boosts coding productivity, it also adds some burden on the network. Network utilization should have been kept to the minimum.

Analysis

After quick research I found two great resources that directed me to the solution that would satisfy both requirements:

Coding productivity.
Minimum of data in transit on the wire.

The first one is from Chris Hay - remix08 UK ASP.NET Front End Performance Slides and the other one is from Jeff Prosise - Power ASP.NET AJAX Programming. They both outline the usage of Script-Callable Web Services. There are three simple steps to follow:

Declare your Web Service as Script-Callable by adding class level attribute:

[System.Web.Script.Services.ScriptService]
public class AJAXCallableWebService : System.Web.Services.WebService
{
    [WebMethod]
    public string HelloWorld(string name)
    {
        return "Hello, " + name;
    }
}

Declare service reference inside the ScriptManager:

<asp:ScriptManager ID="ScriptManager1" runat="server">
    <Services>
        <asp:ServiceReference InlineScript="true" Path="~/AJAXCallableWebService.asmx" />
    </Services>
</asp:ScriptManager>

Call a Web Service from the client script:

<script type="text/javascript" language="javascript">
function callAjax()
{
    var text = document.getElementById("Text1").value
    
    AJAXCallsWebService.AJAXCallableWebService.HelloWorld(text,onSuccess);
}
function onSuccess(result)
{
    document.getElementById("result").innerText = result;
}
</script>

    <span id="result"></span>
    <input id="Button1" type="button" value="button" onclick="callAjax()" />
    <input id="Text1" type="text" />

Sample Visual Studio Solution

Grab the sample solution implemented using Visual Studio 2008 form my SkyDrive here:

Related Materials

This template is made with PracticeThis.com plugin for Windows Live Writer

ASP.NET Performance: Dynamically Loaded Assemblies Cause Application Recycles (Problem and Solution)

alikl — Wed, 08 Oct 2008 16:21:41 GMT

In my speak - dynamically loaded assemblies are those assemblies that were compiled during run time dynamically via CodeProvider like CSharpCodeProvider directly or by using types that use this class internally. Assemblies that are loaded dynamically using reflection via Load/LoadFrom method are out of the scope of this post.

Customer Case Study

The customer complained on periodic restarts/recycles of the ASP.NET application. We observed relative entries in the Event Log that showed up systematically on timely basis. We also observed memory utilization growth in Task Manager. After reaching 500 MB of memory utilization the application would recycle spawning another w3wp.exe to accept new incoming requests while draining the old w3wp.exe.

Analysis

After short discussion with the dev team about the design of the application we thought that the memory leak might be caused by improper usage of XmlSerializer that generate dynamic assemblies. Tess published fantastic walk through specifically dedicated to this case. Using either perfmon (.NET CLR Loading\Current Assemblies) or Process Explorer (see pic below)we observed unusual number (thousands) of loaded assemblies (notice Assemblies column). Also, notice the csc.exe in red - this is CSharp compiler that is invoked on each request:

We decided to take a memory dump to deeply investigate the case. Following are the steps that we took while analyzing the dump using WinDBG to identify the root cause:

Step 1 - Dumping memory heap to identify object allocated on heap

This is the fragment of the long list of objects. Our attention was caught by unusually large number of reflected assemblies.

!dumpheap –stat
9,019       216,456 System.Reflection.Assembly
112         4,032 System.Xml.Serialization.TempAssembly
104         5,408 System.Xml.Serialization.TypeDesc

After 3 minutes the number of dynamic assemblies is larger by more 350 assemblies (from subsequent dump):

!dumpheap –stat
9,379       225,096 System.Reflection.Assembly
114         4,104 System.Xml.Serialization.TempAssembly
102         5,304 System.Xml.Serialization.TypeDesc

Step 2 - Dumping appdomains to identify loaded assemblies

Another cross check to make sure we are dealing with tons of loaded assemblies.

!dumpdomain -stat
    Domain              Num Assemblies   Size Assemblies    Name
0x793f15d8                       1                       2,142,208        System Domain
0x793f2aa8                       56                    16,012,288      Shared Domain
0x000ab7d8                      2                       2,498,560        DefaultDomain
0x000d3368                  9,018                  55,447,040      /LM/W3SV......

Total 4 Domains, Total Size 76,100,096

Step 3 - Dumping all dynamic assemblies

How many of the assemblies are dynamic? (dda stands for dumpdynamicassemblies)

!dda

Domain: 0x793f15d8
-------------------
Domain: .
-------------------
Domain: DefaultDomain
-------------------
Domain: /LM/W3SVC/1/ROOT/......
-------------------
Assembly: 0x19058818 [RegexAssembly133_0] Dynamic Module: 0x16f4c220 loaded at: 0x0 Size: 0x0((null))
Assembly: 0x19058818 [RegexAssembly133_0] Dynamic Module: 0x190696a0 loaded at: 0x0 Size: 0x0((null))
Assembly: 0x19103ee8 [-0g5u8-v] Dynamic Module: 0x1920d6f8 loaded at: 0x19911000 Size: 0xc000((null))
Assembly: 0x190c9a40 [cvmmynwf] Dynamic Module: 0x190dc0d0 loaded at: 0x19a71000 Size: 0x4000((null))
Assembly: 0x1911bad8 [0ikhy_lx] Dynamic Module: 0x1911aa98 loaded at: 0x19f21000 Size: 0xc00((null))
.......
Assembly: 0x43199720 [nv1lvdiy] Dynamic Module: 0x431b3190 loaded at: 0x4cf61000 Size: 0xc00((null))
Assembly: 0x2d2bf008 [rk6dabem] Dynamic Module: 0x2d2bf258 loaded at: 0x4cf71000 Size: 0xc00((null))
--------------------------------------

Total 8,911 Dynamic Assemblies, Total size: 0x1d5b600(30,782,976) bytes.

Step 4 - Saving dynamic assembly to physical DLL

Save assemblies to the filesystem

!savemodule 0x2d2bc4c8 C: \0x1c344438.dll

To save all the assemblies to the file system use the following command:

!dda -save C:\MODULES

Step 5 - Using Reflector to reverse engineer the DLL:

Use Reflector to inspect the implementation/source code of the dynamic assemblies.

Step 6 - Using Reflector to find ExpressionEvaluator class

Try to locate the class in the static assemblies hopefully hitting the code that generates it:

Step 7 - Bingo! Each constructor for ExpressionEvaluator invokes compiler

ICodeCompiler compiler = new CSharpCodeProvider().CreateCompiler();
...
CompilerResults results = compiler.CompileAssemblyFromSource(options, builder.ToString());
...
this._Compiled = results.CompiledAssembly.CreateInstance("NavServices._ExpressionEvaluator");

This is actually the code that causes Process Explorer to show csc.exe under w3wp.exe (see red line in the first pic). And this is the code that caused number of loaded assemblies to grow. And this is the code that caused the application restarts.

Acknowledgements

During this investigation the following resources were used. Big THANK-YOU goes to:

Tess - .NET Memory Leak: XmlSerializing your way to a Memory Leak
Tom - Dynamic Assemblies and what to do about them
Johan - Getting started with windbg - part I
Cuko - OutOfMemory & CompiledAssembly

Related Materials

This template is made with PracticeThis.com plugin for Windows Live Writer

Improve ASP.NET 2.0 Performance With PageAsyncTask - Multithreading For The Masses

alikl — Thu, 02 Oct 2008 11:31:38 GMT

Multithreading is tough. This is what John Robbins says about it in his excellent book Debugging Microsoft .NET 2.0 Applications:

"Don't do it... Make sure there's no other way you can structure your program before you decide to incorporate multithreading into your application... you are easily adding a minimum of an extra month of development and testing to your schedule". I agree with it completely. But there are times multithreading is unavoidable. Especially when more and more Services popping up in the wild...

From Asynchronous Pages in ASP.NET 2.0

Customer Case Study

I am working with the customer to improve their ASP.NET 2.0 application performance. The application issues multiple requests to the backend middleware that is exposed as a web service. The application cannot get a hold on the actual proxies rather it is provided with the components that wrap the web services proxies. The application needs to issue concurrent requests. Serial request would result in serious latency which is unacceptable by end users.

Analysis

Since we cannot get hold on web services proxies we cannot utilize available asynchronous methods available with it: BeginMyMethod/EndMyMethod and MyMethodAsync/MyMethodCompleted. But we definitely can utilize another option available with ASP.NET 2.0 - registering asynchronous tasks using PageAsyncTask class.

The following information is based on the following materials:

PageAsyncTask Implementation Steps

This is the summary of the steps to implement and register PageAsyncTask:

Create a class that contains lengthy operation
Declare AsyncTaskDelegate
Add OnBegin, OnEnd, OnTimeout methods to the class
Mark the calling page as Async="true"
Register the PageAsyncTasks in the ASPX page and execute it

You can grab the Visual Studio 2008 project with the implementation from my SkyDrive here:

Related Materials

This template is made with PracticeThis.com plugin for Windows Live Writer