Alik Levin's

Clarity, Technology, and Solving Problems | PracticeThis.com 

February, 2009

  • Alik Levin's

    Use Process Explorer To Quickly Identify Performance Bottlenecks In ASP.NET Web Applications

    • 0 Comments
     Alik Levin    Process Explorer is my best friend when I need to quickly identify potential performance bottlenecks for ASP.NET web applications. When I am called onsite I have it always with me.

    It served me well recently so I thought it'd be proper to show my appreciation to the tool and to the creator, Mark Russinovich, by sharing how I use it. Every time I am called onsite to the customer's, first thing I ask to install ...ehm... copy the tool to the server in question. It usually shows hot spots right away and helps making assumptions and decisions.

    Performance Bottleneck
    by tacit requiem

    CPU Usage History

    This view helps to spot "unusual" activity for ASP.NET Web Application. When I see "too much" red color in this view I start asking question "Why Managed .Net Code performs privileged operations?" From my observations it mostly caused by excessive usage of threading.

    CPU Utilization

    Private Bytes Memory

    This view has become my favorite recently as I was involved in few cases with memory leakage. I think the picture is self explanatory.

    Memory Leak

    I/O Bytes History

    This view helps me spotting potential I/O bottlenecks.

    Network Utilization

     

    This post is made with PracticeThis.com plugin for Windows Live Writer

  • Alik Levin's

    What Do Solution Architect And Enterprise Architect Do?

    • 1 Comments
     Alik Levin    I am aspiring Architect and I am on my quest to become an MCA: Solutions certification (Microsoft Certified Architect).

    To start off I first need to have an idea what's expected from the Architect. I need to know what key characteristics are common to all effective architects.

    I need to create a role model, a vision so can better develop myself as an architect.

    This is what I discovered reading What Architect Job Roles Are Recognized By the Microsoft Certified Architect Program?

    image

    by Hamed Saber

    Enterprise architects

    • Communicate primarily with corporate business owners and staff architects.
    • Experts in the vertical industry in which they work.
    • Responsible for meeting regulatory and compliance requirements.
    • Set the overall vision and framework for the IT environment.
    • Main focus is on the IT environment from a strategic business perspective
    • Engage primarily in the envisioning stages of projects.

    Solutions architects

    • Communicate mainly with business owners within a company and with the technical staff that delivers the solution.
    • Affect the enterprise.
    • Design the solution to take advantage of the existing assets.
    • Integrate the assets into the existing environment.
    • Follow the enterprise architecture
    • Solve the business problems of the business owner or unit.
    • Take a project through envisioning and design
    • Consultative to the project manager
    • Problems are escalated to the solutions architect.
    • Create line of business-based solutions
    • Integrate with the existing infrastructure
    • Designs solutions supported by the operations group.
    • Do not manage the technical staff.
    • Demonstrate their skills as a technologist and persuade the staff regarding the validity and approach to the solution.
    • Gather business requirements.
    • Select the technologies that provide the best solution.
    • Identify the products available that will best fit the solution.
    • Focus include integration, workflow, and applications

    Related Materials

    This post is made with PracticeThis.com plugin for Windows Live Writer

  • Alik Levin's

    Identifying Memory Leak With Process Explorer And Windbg

    • 2 Comments

     

    Alik Levin    Your ASP.NET web application behaves strange. It gets stuck, it stop serving incoming requests, or gets recycled. You observe strange application pool recycles events in event log.
    Chances you are dealing with memory leak in your ASP.NET web application. Lately I've been involved with few of such situations. Here is the method and the resources that helped me to extremely quickly identify the root cause of the memory leaks. 

    Related Books 
    _________________________________

     

    I am super grateful to the resources published by Tess and Rico that discuss the process of memory leak identification in more detail:

    Customer Case Study

    The customer complained his application gets less responsive as more subsequent request come in. The customer reported the high CPU utilization. At some point the application gets stuck and the only way to get it back is..... iisreset. The customer felt there is a memory leak in the application.

    Analysis - Summary of steps

    • Step 1 - Use Process Explorer To Confirm The Memory Leak
    • Step 2 - Take Memory Dump Using adplus.vbs
    • Step 3 - Use Windbg to identify the leaking Type
    • Step 4 - Use Windbg to identify to identify who's holding on the Type
    • Step 5 - Find the culprit in the source code

    Step 1 - Use Process Explorer To Confirm The Memory Leak

    I am using Process Explorer to quickly identify whether we are dealing with memory leak. In our case it showed we are. This is the memory consumption that Process Explorer reveals in very convincing way - you can clearly see how the memory consumption climbs up until the application hangs/restarts:

    Porcess Explorer Memory Leak

    Next question is who's consuming the memory? Windbg to the rescue.

    Step 2 - Take Memory Dump Using adplus.vbs

    Use Windows Debugging Tools package to take a memory dump of the leaking managed process. Download the tools and extract on the server where the application runs. Open command prompt and change the directory to the folder that contains the tools, then type the following command:

    adplus.vbs -quiet -hang -p <<PROCESSID>>

    where <<PROCESSID>> is the process of w3wp.exe PID (process ID) that runs the leaking application. The result should be another folder inside of tools folder with very lengthy name. Open that folder- you should see a big size file with DMP extension. This is the memory dump you have just taken.

    Step 3 - Use Windbg to identify the leaking Type

    To analyze the dump file you will going to use Windbg.exe. The tool is located in the Debugging Tools for Windows folder, same folder that contains adplus.vbs that was used in the previous exercise. Open the tool, Windbg.exe. Press Ctrl+D, locate your DMP file and open it.

    Now you need to load SOS.DLL extension to analyze managed memory dump. Type the following command according ro what .Net framework you are analyzing:

    .load clr10\sos - for Net 1.X

    .loadby sos mscorwks.dll - for Net 2.0 and up.

    Type the following command to identify the Type that consumes most of the memory:

    !dumpheap -stat

    The result should be similar to the following:

    .....
    654088b4 92811 13736028 System.Data.DataColumn
    000dec48 44 13900264 Free
    654359c8 31704 32845344 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
    7912d8f8 320769 47859900 System.Object[]
    790fd8c4 827939 186834312 System.String

    Total 3351270 objects

    Most consuming is that System.String type that is responsible for almost 190 MB of allocated memory.

    Step 4 - Use Windbg to identify to identify who's holding on the Type

    Next question your should ask is who's holding on it so that it's not collected. For that purpose type the following command in Windbg but be prepared to hit Ctrl+Break to stop it as you do not want to list all 827939 strings, you are looking for a few to get its addresses:

    !dumpheap -type System.String

    The output should be similar to this:

    057f9364 790fd8c4 4116
    057fa4b8 790fd8c4 4116
    057fb52c 790fd8c4 4116
    057fc68c 790fd8c4 4116
    057fd700 790fd8c4 4116
    0583fef8 790fd8c4 4116

    Next you want to start trying to identify who's holding on the objects. That can be done using the following command (I replaced the real Namespaces with XXXXXXXX to keep customer's privacy):

    0:000> !gcroot 0583fef8
    Note: Roots found on stacks may be false positives. Run "!help gcroot" for
    more info.

    Scan Thread 27 OSTHread e40
    DOMAIN(00102980):HANDLE(WeakLn):945e00:Root:19312e44(System.Web.Hosting.ISAPIAsyncCompletionCallback)->
    190c2950(System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6)->
    190c3138(System.Web.HttpContext)->
    190c340c(System.Web.AspNetSynchronizationContext)->
    0cbb80d0(ASP.global_asax)->
    01b3a5b4(System.Web.HttpApplicationState)->
    01b3a610(System.Collections.ArrayList)->
    0a4e4c10(System.Object[])->
    05846388(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
    058331c8(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.XXXXXXXX)->
    058333c0(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Statuses)->
    058333cc(System.Object[])->
    0583fe20(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Status)->
    0583fee0(System.Object[])->
    05840f0c(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Style)->
    0583fef8(System.String)

    Looks like the code using endlessly Application object sticking strings into it. Once there it's never collected until the end of life of the Application which caused only by recycle.

    Step 5 - Find the culprit in the source code

    Next is to look in the source code all the usages of assigning strings to Application object. After quick look up we identified the following code that is called recursively:

    HttpApplicationState app = HttpContext.Current.Application;           
    for (int i=0; i < m_XXXXXX.Count; i++)
    {
        string sProcessId = m_XXXXXX[i].SelectSingleNode("@XXXXXX").Value;
        app.Set (GetPrefix() + XXXXX, new XXXXXX(m_XXXXXX[i].OuterXml));
    }

    Conclusion

    I have been using the following approach several times lately and it has proven to be very effective and efficient. It's simple too.

    Happy memory leak analysis.

    Related Materials

     

    This post is made with PracticeThis.com plugin for Windows Live Writer

Page 2 of 3 (8 items) 123