Alik Levin's

Clarity, Technology, and Solving Problems | PracticeThis.com 

March, 2009

  • Alik Levin's

    Distributed Architecture Drawbacks Revealed By Netmon(Bonus - TDS Parser Goes Public)

    • 2 Comments
     Alik Levin    Distributed architecture can mercilessly backfire at you.  In my case flexible architecture, elegant design patterns, and smart code led to abuse of the flexibility, resulting in very poor performance. Free Microsoft Network Monitor (Netmon) helped to identify the root cause of the Distributed Architecture Hits Bac
    by striatic 
    performance hit. It showed that over-distributed-ness can cost you in terms of performance.

    Customer Case Study

    The customer complained about poor response times in his web application. The application's architecture was similar to the Web Application Archetype. Notice Services Agent box that connects your application to downstream services? Our assumption was that the services agents are too chatty causing the performance hit. Netmon only made it clear.

    Analysis

    We took captures on the application server where the Server Agents are to identify what other downstream servers are accessed. In no time we get very clear picture - the application server was accessing other three downstream resources:

     netmon distributed architecture

    There are two well known ports - 443 and 1433 - so we could safely assume there is communication over SSL/HTTPS and SQL Server. The other one - 1414 - turned out to be MQ.

    Next step was to identify which one of the conversations is causing us troubles the most - either by chatty communication or by just long running transaction.

    Looking at Time Delta column for HTTPS stateless communications we found nothing exciting regarding the latency:

    HTTPS Communications network monitor

    For MQ communications we used magic ContainsBin(FrameData, 0,"StringToFindGoesHere") to identify XML messages going back and forth over MQ transport. For example, to find the beginning of the XML message we used the following filter:

    Netmon ContainsBin

    Similar technique was used to correlate request and response XML messages.

    To identify SQL communication we used shiny new TDS parser available for free download on Codeplex - SQL Parser in Latest CodePlex Package. I particularly like this one, it shows SQL Server communication without using SQL Server Profiler:

     netmon tds parser

    Conclusion

    Lessons learned:

    • Distributed Architecture can mercilessly backfire at you by its distributed-ness. Use Distributed architecture to solve problems vs introducing new ones.
    • Elegant and flexible design can be abused resulting in poor performance. Review code to avoid the abuse.
    • Low level network monitoring tools such as Netmon can quickly reveal over-distributed-ness. Use it! It is free.

    Related Materials

    This post is made with PracticeThis.com plugin for Windows Live Writer

  • Alik Levin's

    WinDBG Walkthrough - Dump Values Of DataSet or DataTable

    • 4 Comments
     Alik Levin    This walkthrough is completely based on Johan's post WinDBG+SOS: Getting at the values in a DataTable. I have created this one to help me do the job in straightforward way next time I hit similar problem. Joan also offers few scripts for process automation - recommended.

    image 
    by glennharper

    Customer Case Study

    The customer complained about potential memory leak. Following the procedure described in Identifying Memory Leak With Process Explorer And Windbg we realized that we are dealing with static variable that grows in unlimited way. This assumption is based on the fact that after running !gcroot on the leaking type we get the following:

    HANDLE(Strong):23b1cd0:Root:0x12571730.....

    After reviewing Tess' .NET Debugging Demos Lab 7: Memory Leak - Review we found the following which made us believe we are dealing with static variable:

    DOMAIN(001CCE68):HANDLE(Strong) - Strong reference, Typically a static variable

    To identify what this static variable is we needed to dump its values. The variable was a DataTable. I have not found a straightforward way of dumping contents of DataTable. This is the walkthrough that does the job.

    Summary of steps

    • Step 1. Dump DataTables
    • Step 2. Dump DataTable
    • Step 3. Dump columnCollection
    • Step 4. Dump list object
    • Step 5. Dump raw memory - dd command
    • Step 6. Dump DataColumn
    • Step 7. Dump storage
    • Step 8. Dump values
    • Step 9. Bonus - automation

    Step 1. Dump DataTables

    The objective of this step is identify all DataTable object and pick the one of the interest

    0:000> !dumpdatatables
    DataTable       Rows    Columns    DataSet nextRowID ColumnCount
    -----------------------------------------------------------------------------------------------
    0x024dc948 0x024dcbc8 0x024dcdec 0x064f2400         1         2
    0x025156b8 0x02515938 0x02515b5c 0x02515478         1         7
    ...

    0x0e5b9ce4 0x0e5b9f64 0x0e5ba138 0x0e55a338       428         5
    0x06510e54 0x065110d4 0x065112a8 0x064f2400     1,359         7
    0x0e5778f0 0x0e577b70 0x0e577d44 0x0e55a338     1,359         7
    0x0a55d270 0x0a55d4f0 0x0a55d6c4 0x06a62620 4,194,305        10
    Total 61 DataTable objects

    Step 2. Dump DataTable

    The objective of this step is identifying the address of columnCollection of the DataTable.

    0:000> !do 0x0a55d270
    Name: System.Data.DataTable
    ...
    0x176c1560 0x40003f2     0x18                CLASS   instance 0x0a55d6c4 columnCollection
    ...

    Step 3. Dump columnCollection

    The objective of this step is identifying  the address of list object of the columnCollection.

    0:000> !do 0x0a55d6c4
    Name: System.Data.DataColumnCollection
    ...
    0x176c5ffc 0x4000377      0x8                CLASS   instance 0x0a55d6f8 list
    ...

    Step 4. Dump list object

    The objective of this step is identifying the address of _items object.

    0:000> !do 0x0a55d6f8
    Name: System.Collections.ArrayList
    ...
    0x79ba75ec 0x4000362      0x4                CLASS   instance 0x0a55d710 _items
    ...

    Step 5. Dump raw memory - dd command

    The objective of this step is identifying addresses of DataColumn objects.

    0:000> dd 0x0a55d710
    0a55d710  01e5209c 00000010 79b92eec 0a55d854
    0a55d720  0a55d8d4 0a55d954 0a55d9d4 0a55da54
    0a55d730  0a55dad4 0a55db54 0a55dbd4 0a55dd74

    Step 6. Dump DataColumn

    The objective of this step is identifying the address of storage object.

    !do 0a55d854
    Name: System.Data.DataColumn
    ...
    0x176c69e8 0x400036b     0x48                CLASS   instance 0x0a55df40 storage
    ...

    Step 7. Dump storage

    The objective of this step is identifying address of value object.

    0:000> !do 0x0a55df40
    Name: System.Data.Common.Int32Storage
    ...
    0x176ecc8c 0x4000729     0x10                CLASS   instance 0x4dbd0030 values
    ...

    Step 8. Dump values

    The objective of this step is dumping the actual values, finally...

    0:000> !do 0x4dbd0030
    Name: System.Int32[]
    ...
    Content: 8,388,608 items

    Ouch, it is array.... !do -v will dump its values, but I am afraid it is not a good idea doing it for 8 million items here ;)

    Step 9. Bonus - automation

    The objective of this step is automate the process of dumping values (Step 8):

    • Create a text file named dumparray and save it in WinDBG directory. The contents of the file are:
      • .foreach ( o { !do ${$arg1} -v -short }) { !do ${o} }
    • Run the following command in WinDBG to dump the values of the array
      • $$>a< dumparray 0x4dbd0030

    Related Materials

    This post is made with PracticeThis.com plugin for Windows Live Writer

  • Alik Levin's

    Identify And Fix Performance Bottlenecks In ASP.NET Web Applications - Case Studies, Problems, And Solutions

    • 1 Comments
     Alik Levin    I thought it'd be useful to package the resources referenced in my previous post - Why My ASP.NET Application's Performance Is Less Than Optimal?
    I packaged it in form of PDF file. Hope it'll be useful for you when dealing with performance bottlenecks in your ASP.NET web application.

    Download "Identifying and Fixing Performance Bottlenecks" PDF file from my SkyDrive:

     

    Feel free to distribute it. If you have suggestions for improvements or just a friendly constructive comment - feel absolutely free to submit one.

    This post is made with PracticeThis.com plugin for Windows Live Writer

Page 2 of 3 (7 items) 123