Alik Levin's

Clarity, Technology, and Solving Problems | PracticeThis.com 

  • Alik Levin's

    Stress Test ASP.NET Web Application With Free WCAT Tool

    • 12 Comments

     

    Building ASP.NET web applications? Plan to serve thousands of users? Would you like to see how your application would behave [misbehave] under stress?

    Use simple-to-use and freely available WCAT tool to generate the load and get detailed report for expected throughput (requests/sec) and other important performance-wise information.

    Summary of steps

    • Install WCAT
    • Create configuration files
    • Run the test
    • Examine results

    Next section describes each step in details. Note, this post is a how-to, a jump start - not the guidelines or best practices of how to use the tool.

    Install WCAT

    Download and install Internet Information Services (IIS) 6.0 Resource Kit Tools. For the purpose of our exercise there is no need to install all the tools included with the resources kit, only WCAT.

    Create configuration files

    There are three textual files one needs to create and configure (you can give any name and extension of your choice):

    • script.txt - this file defines the requests, in other words what pages to request and how to request it. Following is an example of simple script.txt file:

    NEW TRANSACTION
        classId = 1
        NEW REQUEST HTTP
            Verb = "GET"
            URL = "http://localhost/BankingShmanking/Default.aspx"

    • distribution.txt - defines weights among different requests. For example, if I need to generate request to page1.aspx twice as to page2.aspx, I will define it in this file. In case of loading only one page, the file is meaningless. Following is an example of simple distribution.txt file (1 refers to classId in script.txt file, and 50 is that 50% of the load should got this file which is meaningless since there is only one page to request, thus it will get the whole 100% load):

    1 50

    • config.txt - determines the duration of the test, number of clients that will generate the requests against the web application. Following is the example of simple config.txt file:

    Warmuptime 5s
    Duration 30s
    CooldownTime 5s
    NumClientMachines 1
    NumClientThreads 20

    Save the files in "C:\Program Files\IIS Resources\WCAT Controller" folder.

    Run the test

    To run the stress test open command prompt by opening Run window(Windows + R) type cmd and hit Enter. Change current directory to "C:\Program Files\IIS Resources\WCAT Controller>" and run the following command to test the page hosted on the localhost:

    wcctl -c config.txt -d distribution.txt -s script.txt -a localhost

    then open second command prompt, change current folder to "C:\Program Files\IIS Resources\WCAT Client" and run the following command to actually launch the virtual client's requests from local machine:

    wcclient.exe localhost

    Examine results

    The results are displayed interactively in the command line windows

     

    image

    The tool also generates log file that includes logged metrics - look for it in "C:\Program Files\IIS Resources\WCAT Controller" folder.

    WCAT tool is actively developed by IIS team and recently they released new version of the tool - WCAT 6.3, download it from here, free.

    Related Books

    Related materials

    My related posts

  • Alik Levin's

    Avoid Manipulating Passwords In Memory - It Is Easy To Reveal

    • 3 Comments

    Revealing clear text passwords in memory seems to be a trivial task. This post describes how to reveal clear text passwords and what countermeasures to apply.

    Summary of steps:

    • Install WinDbg
    • Attach to process or open dump file
    • Load SOS .Net extensions for WinDbg
    • Enumerate threads
    • Enumerate objects in thread
    • Dump object's values
    • Countermeasures and guidelines

    Install WinDbg

    Download and install WinDbg as described in How to install Windbg and get your first memory dump.

    Attach to process or open dump file

    WinDbg can analyze both running processes and memory dumps which conveniently can be taken offsite for further investigation. I've created simple console application that accepts user name and password pair as its parameters and stores in local variables in memory:

    static void Main(string[] args)
    {
        string userName = Console.ReadLine();
        string password = Console.ReadLine();

        Console.ReadLine();
    }

    Compile and run the application. I called it SecretsInMemory. This is how it looks when running:

    image

    Attach WinDbg to the running application by opening File->Attach to a Process:

    image  

    and press Ok.

    Alternatively, we can create dump file - for detailed how-to refer to How to install Windbg and get your first memory dump.

    To Investigate resulting dump file in WinDbg open File->Open Crash Dump

    Load SOS .Net extensions for WinDbg

    To analyze .Net assemblies we need to load .Net extensions by typing .load sos and hitting Enter:

    image

    Enumerate threads

    Run !threads command to enlist available threads:

    image

    and then choose specific thread - use left most column for thread identification as follows ~[thread number goes here]s:

    image

    Enumerate objects in thread

    Use !dso command to dump all objects in the thread:

    image

    Dump object's values

    Use !do <object address> to dump specific object's values. Object address is a second column in the list generated by !dso command, the column named "Object" - just copy and paste it:

    image

    The password is revealed either by attaching to the process or analyzing a crash file that was taken offsite.

    Countermeasures and guidelines

    As rule of thumb avoid using custom built identification and authentication mechanisms and leverage those that the infrastructure offers - preferably Windows Integrated authentication. In case where all options exhausted and there is no other way but accept end user credentials, refer to the following article - Using Credential Management in Windows XP and Windows Server 2003. Techniques described in the article allow to leverage built in mechanism of accepting credentials from end user in more secure manner. It also keeps common familiar look and feel across custom application and built in Windows mechanisms leaving less room for end user confusion.

    My related posts:

    Other resources:

  • Alik Levin's

    Identifying Memory Leak With Process Explorer And Windbg

    • 2 Comments

     

    Alik Levin    Your ASP.NET web application behaves strange. It gets stuck, it stop serving incoming requests, or gets recycled. You observe strange application pool recycles events in event log.
    Chances you are dealing with memory leak in your ASP.NET web application. Lately I've been involved with few of such situations. Here is the method and the resources that helped me to extremely quickly identify the root cause of the memory leaks. 

    Related Books 
    _________________________________

     

    I am super grateful to the resources published by Tess and Rico that discuss the process of memory leak identification in more detail:

    Customer Case Study

    The customer complained his application gets less responsive as more subsequent request come in. The customer reported the high CPU utilization. At some point the application gets stuck and the only way to get it back is..... iisreset. The customer felt there is a memory leak in the application.

    Analysis - Summary of steps

    • Step 1 - Use Process Explorer To Confirm The Memory Leak
    • Step 2 - Take Memory Dump Using adplus.vbs
    • Step 3 - Use Windbg to identify the leaking Type
    • Step 4 - Use Windbg to identify to identify who's holding on the Type
    • Step 5 - Find the culprit in the source code

    Step 1 - Use Process Explorer To Confirm The Memory Leak

    I am using Process Explorer to quickly identify whether we are dealing with memory leak. In our case it showed we are. This is the memory consumption that Process Explorer reveals in very convincing way - you can clearly see how the memory consumption climbs up until the application hangs/restarts:

    Porcess Explorer Memory Leak

    Next question is who's consuming the memory? Windbg to the rescue.

    Step 2 - Take Memory Dump Using adplus.vbs

    Use Windows Debugging Tools package to take a memory dump of the leaking managed process. Download the tools and extract on the server where the application runs. Open command prompt and change the directory to the folder that contains the tools, then type the following command:

    adplus.vbs -quiet -hang -p <<PROCESSID>>

    where <<PROCESSID>> is the process of w3wp.exe PID (process ID) that runs the leaking application. The result should be another folder inside of tools folder with very lengthy name. Open that folder- you should see a big size file with DMP extension. This is the memory dump you have just taken.

    Step 3 - Use Windbg to identify the leaking Type

    To analyze the dump file you will going to use Windbg.exe. The tool is located in the Debugging Tools for Windows folder, same folder that contains adplus.vbs that was used in the previous exercise. Open the tool, Windbg.exe. Press Ctrl+D, locate your DMP file and open it.

    Now you need to load SOS.DLL extension to analyze managed memory dump. Type the following command according ro what .Net framework you are analyzing:

    .load clr10\sos - for Net 1.X

    .loadby sos mscorwks.dll - for Net 2.0 and up.

    Type the following command to identify the Type that consumes most of the memory:

    !dumpheap -stat

    The result should be similar to the following:

    .....
    654088b4 92811 13736028 System.Data.DataColumn
    000dec48 44 13900264 Free
    654359c8 31704 32845344 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
    7912d8f8 320769 47859900 System.Object[]
    790fd8c4 827939 186834312 System.String

    Total 3351270 objects

    Most consuming is that System.String type that is responsible for almost 190 MB of allocated memory.

    Step 4 - Use Windbg to identify to identify who's holding on the Type

    Next question your should ask is who's holding on it so that it's not collected. For that purpose type the following command in Windbg but be prepared to hit Ctrl+Break to stop it as you do not want to list all 827939 strings, you are looking for a few to get its addresses:

    !dumpheap -type System.String

    The output should be similar to this:

    057f9364 790fd8c4 4116
    057fa4b8 790fd8c4 4116
    057fb52c 790fd8c4 4116
    057fc68c 790fd8c4 4116
    057fd700 790fd8c4 4116
    0583fef8 790fd8c4 4116

    Next you want to start trying to identify who's holding on the objects. That can be done using the following command (I replaced the real Namespaces with XXXXXXXX to keep customer's privacy):

    0:000> !gcroot 0583fef8
    Note: Roots found on stacks may be false positives. Run "!help gcroot" for
    more info.

    Scan Thread 27 OSTHread e40
    DOMAIN(00102980):HANDLE(WeakLn):945e00:Root:19312e44(System.Web.Hosting.ISAPIAsyncCompletionCallback)->
    190c2950(System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6)->
    190c3138(System.Web.HttpContext)->
    190c340c(System.Web.AspNetSynchronizationContext)->
    0cbb80d0(ASP.global_asax)->
    01b3a5b4(System.Web.HttpApplicationState)->
    01b3a610(System.Collections.ArrayList)->
    0a4e4c10(System.Object[])->
    05846388(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)->
    058331c8(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.XXXXXXXX)->
    058333c0(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Statuses)->
    058333cc(System.Object[])->
    0583fe20(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Status)->
    0583fee0(System.Object[])->
    05840f0c(XXXXXXXX.Web.UI.WebControls.XXXXXXXX.Style)->
    0583fef8(System.String)

    Looks like the code using endlessly Application object sticking strings into it. Once there it's never collected until the end of life of the Application which caused only by recycle.

    Step 5 - Find the culprit in the source code

    Next is to look in the source code all the usages of assigning strings to Application object. After quick look up we identified the following code that is called recursively:

    HttpApplicationState app = HttpContext.Current.Application;           
    for (int i=0; i < m_XXXXXX.Count; i++)
    {
        string sProcessId = m_XXXXXX[i].SelectSingleNode("@XXXXXX").Value;
        app.Set (GetPrefix() + XXXXX, new XXXXXX(m_XXXXXX[i].OuterXml));
    }

    Conclusion

    I have been using the following approach several times lately and it has proven to be very effective and efficient. It's simple too.

    Happy memory leak analysis.

    Related Materials

     

    This post is made with PracticeThis.com plugin for Windows Live Writer

Page 1 of 118 (354 items) 12345»