Alik Levin's

Clarity, Technology, and Solving Problems | PracticeThis.com 

October, 2008

  • Alik Levin's

    ASP.NET Performance: Dynamically Loaded Assemblies Cause Application Recycles (Problem and Solution)

    • 4 Comments
     Alik Levin    In my speak - dynamically loaded assemblies are those assemblies that were compiled during run time dynamically via CodeProvider like CSharpCodeProvider directly or by using types that use this class internally. Assemblies that are loaded dynamically using reflection via Load/LoadFrom method are out of the scope of this post.

    Customer Case Study

    The customer complained on periodic restarts/recycles of the ASP.NET application. We observed relative entries in the Event Log that showed up systematically on timely basis. We also observed memory utilization growth in Task Manager. After reaching 500 MB of memory utilization the application would recycle spawning another w3wp.exe to accept new incoming requests while draining the old w3wp.exe.

    Analysis

    After short discussion with the dev team about the design of the application we thought that the memory leak might be caused by improper usage of XmlSerializer that generate dynamic assemblies. Tess published fantastic walk through specifically dedicated to this case. Using either perfmon (.NET CLR Loading\Current Assemblies) or Process Explorer (see pic below)we observed unusual number (thousands) of loaded assemblies (notice Assemblies column). Also, notice the csc.exe in red - this is CSharp compiler that is invoked on each request:

    Process Explorer ASP.NET Performance Dynamically Loaded Assemblies

    We decided to take a memory dump to deeply investigate the case. Following are the steps that we took while analyzing the dump using WinDBG to identify the root cause:

    Step 1 - Dumping memory heap to identify object allocated on heap

    This is the fragment of the long list of objects. Our attention was caught by unusually large number of reflected assemblies.

    !dumpheap –stat
    9,019       216,456 System.Reflection.Assembly
    112         4,032 System.Xml.Serialization.TempAssembly
    104         5,408 System.Xml.Serialization.TypeDesc

    After 3 minutes the number of dynamic assemblies is larger by more 350 assemblies (from subsequent dump):

    !dumpheap –stat
    9,379       225,096 System.Reflection.Assembly
    114         4,104 System.Xml.Serialization.TempAssembly
    102         5,304 System.Xml.Serialization.TypeDesc

    Step 2 - Dumping appdomains to identify loaded assemblies

    Another cross check to make sure we are dealing with tons of loaded assemblies.

    !dumpdomain -stat 
        Domain              Num Assemblies   Size Assemblies    Name
    0x793f15d8                       1                       2,142,208        System Domain
    0x793f2aa8                       56                    16,012,288      Shared Domain
    0x000ab7d8                      2                       2,498,560        DefaultDomain
    0x000d3368                  9,018                  55,447,040      /LM/W3SV......

    Total 4 Domains, Total Size 76,100,096

    Step 3 - Dumping all dynamic assemblies

    How many of the assemblies are dynamic? (dda stands for dumpdynamicassemblies)

    !dda

    Domain: 0x793f15d8
    -------------------
    Domain: .
    -------------------
    Domain: DefaultDomain
    -------------------
    Domain: /LM/W3SVC/1/ROOT/......
    -------------------
    Assembly: 0x19058818 [RegexAssembly133_0] Dynamic Module: 0x16f4c220 loaded at: 0x0 Size: 0x0((null))
    Assembly: 0x19058818 [RegexAssembly133_0] Dynamic Module: 0x190696a0 loaded at: 0x0 Size: 0x0((null))
    Assembly: 0x19103ee8 [-0g5u8-v] Dynamic Module: 0x1920d6f8 loaded at: 0x19911000 Size: 0xc000((null))
    Assembly: 0x190c9a40 [cvmmynwf] Dynamic Module: 0x190dc0d0 loaded at: 0x19a71000 Size: 0x4000((null))
    Assembly: 0x1911bad8 [0ikhy_lx] Dynamic Module: 0x1911aa98 loaded at: 0x19f21000 Size: 0xc00((null))
    .......
    Assembly: 0x43199720 [nv1lvdiy] Dynamic Module: 0x431b3190 loaded at: 0x4cf61000 Size: 0xc00((null))
    Assembly: 0x2d2bf008 [rk6dabem] Dynamic Module: 0x2d2bf258 loaded at: 0x4cf71000 Size: 0xc00((null))
    --------------------------------------

    Total 8,911 Dynamic Assemblies, Total size: 0x1d5b600(30,782,976) bytes.

    Step 4 - Saving dynamic assembly to physical DLL

    Save assemblies to the filesystem

    !savemodule 0x2d2bc4c8 C: \0x1c344438.dll

    clip_image002

    To save all the assemblies to the file system use the following command:

    !dda -save C:\MODULES

    Step 5 - Using Reflector to reverse engineer the DLL:

    Use Reflector to inspect the implementation/source code of the dynamic assemblies.

    clip_image001

    Step 6 - Using Reflector to find  ExpressionEvaluator class

    Try to locate the class in the static assemblies hopefully hitting the code that generates it:

    image

    Step 7 - Bingo! Each constructor for ExpressionEvaluator invokes compiler

    ICodeCompiler compiler = new CSharpCodeProvider().CreateCompiler();
    ...
    CompilerResults results = compiler.CompileAssemblyFromSource(options, builder.ToString());
    ...
    this._Compiled = results.CompiledAssembly.CreateInstance("NavServices._ExpressionEvaluator");

    This is actually the code that causes Process Explorer to show csc.exe under w3wp.exe (see red line in the first pic). And this is the code that caused number of loaded assemblies to grow. And this is the code that caused the application restarts.

    Acknowledgements

    During this investigation the following resources were used. Big THANK-YOU goes to:

    Related Materials

    This template is made with PracticeThis.com plugin for Windows Live Writer

  • Alik Levin's

    Consultant Eaten By Sharks

    • 2 Comments
     Alik Levin    "If you swim with sharks, do not act like food" - I can crystal clearly remember that quote by Michio Kaku when he was giving a keynote during internal MS convention. It resonated with me a lot. As a consultant, I often find myself deep in the waters surrounded by sharks. Sometimes frightened to death. To "survive" I develop practices that distinguish me from "food" to keep me from being eaten.

    I found many practical advices when I was  reading a book by Harvey Mackay Swim with the Sharks Without Being Eaten Alive: Outsell, Outmanage, Outmotivate, and Outnegotiate Your Competition.

    This post is a collection of a few side notes that I took while reading the book. These advices, among few others,  help me stay alive and not eaten by sharks.

    Do You Believe In Luck?

    I do. For me luck is being the right guy in the right place at the right time. Sitting at home, doing nothing all the time won't get you no luck. Here is nice quote Harvey Mackay used in his book:

    "I am great believer in luck,"  said Stephen Leackock, the Canadian humorist, "and the harder I work, the luckier I get".

    Another one that I really love is from Thomas Alva Edison:

    "Opportunity is missed by most people because it is dressed in overalls and looks like work."

    Doing vs. Achieving

    This one keeps striking me time after time. Many of us still have hard times to distinguish between doing vs. achieving. Working vs. accomplishing. End result is what really matters. Here is what Harvey Mackay has to say about it:

    "One thing professional stock and commodity traders learn early is that they don't give away medals for courage in the marketplace. There is only one reward the marketplace has to offer: money. If you are not making any, bail out. Quickly."

    Well, you cannot argue with it these days...

    Hire The Best

    I was reading the book by 45 EFFECTIVE WAYS FOR HIRING SMART: How to Predict Winners and Losers in the Incredibly Expensive People-Reading Game by Pierre Mornell where he provides a very detailed process to hire the best talent. The whole book is dedicated to interviewing... After your complete all the procedures (many of which are very useful), how do you decided - "Hire" or "No Hire"? Harvey Mackay offers very simple yet powerful advice:

    "Ask yourself, How would you feel having this same person working for your competition instead of for you?"

    Competitive World

    Harvey Mackay writes:

    "Unless you have a unique product or service, or run state-owned bakery in the Soviet Union, competition is a fact of life. You must deal with it. The best way is to gather what knowledge you can and then act."

    My observations of state-owned bakeries only prove what Harvey Mackay writes (I was born in former USSR and lived there until I turned 20). My other observations resonate a lot with what he writes about gathering knowledge and acting. Focusing on high demand expertise pays off. The deeper your knowledge the harder it to push you down the food chain.

    "The grabbing hands grab all they can
    All for themselves - after all
    It's a competitive world
    Everything counts in large amounts" - Depeche Mode

    My Related Posts

    This template is made with PracticeThis.com plugin for Windows Live Writer

  • Alik Levin's

    ASP.NET Performance: Fast AJAX, Faster AJAX

    • 1 Comments
     Alik Levin    AJAX improves significantly both user experience and performance. It can be further improved by using down level capabilities that .Net framework offers. Specifically, consuming Web Services and WCF directly from client script. The best part is that ASP.NET AJAX comes with built in libraries - server and client - that make coding fun while significantly improving the web application's performance.

    Customer Case Study

    The customer was using hand crafted XmlHttp requests from client scripts requesting the data from ASPX pages. While the goal was achieved - the amount of information sent to the server was minimal and the user interface was responsive - the coding was not really fun. Also, since the requests were sent to regular ASPX pages the whole ASPX pipeline was executing unnecessarily utilizing CPU for nothing. The customer did not want to use Update Panel control. Although it boosts coding productivity, it also adds some burden on the network. Network utilization should have been kept to the minimum.

    Analysis

    After quick research I found two great resources that directed me to the solution that would satisfy both requirements:

    1. Coding productivity.
    2. Minimum of data in transit on the wire.

    The first one is from Chris Hay  - remix08 UK ASP.NET Front End Performance Slides and the other one is from Jeff Prosise - Power ASP.NET AJAX Programming. They both outline the usage of Script-Callable Web Services. There are three simple steps to follow:

    • Declare your Web Service as Script-Callable by adding class level attribute:
    [System.Web.Script.Services.ScriptService]
    public class AJAXCallableWebService : System.Web.Services.WebService
    {
        [WebMethod]
        public string HelloWorld(string name)
        {
            return "Hello, " + name;
        }
    }
    • Declare service reference inside the ScriptManager:
    <asp:ScriptManager ID="ScriptManager1" runat="server">
        <Services>
            <asp:ServiceReference InlineScript="true" Path="~/AJAXCallableWebService.asmx" />
        </Services>
    </asp:ScriptManager>
    • Call a Web Service from the client script:
    <script type="text/javascript" language="javascript">
    function callAjax()
    {
        var text = document.getElementById("Text1").value
        
        AJAXCallsWebService.AJAXCallableWebService.HelloWorld(text,onSuccess);
    }
    function onSuccess(result)
    {
        document.getElementById("result").innerText = result;
    }
    </script>
        <span id="result"></span>
        <input id="Button1" type="button" value="button" onclick="callAjax()" />
        <input id="Text1" type="text" />
    

    Sample Visual Studio Solution

    Grab the sample solution implemented using Visual Studio 2008 form my SkyDrive here:

    Related Materials

    submit to reddit

    This template is made with PracticeThis.com plugin for Windows Live Writer

Page 1 of 2 (6 items) 12