ASP.NET Debugging : High Memory

POP QUIZ: What’s wrong with this code – part 3

Tom — Tue, 23 Sep 2008 13:00:00 GMT

Imagine you are a developer and your boss comes to you complaining that your piece of code has been deemed to be taking up too much memory and causing problems for the application. You take a look at your code and you see the following, assume that stream is defined above this and is correct:

     const String myConstStr = "CurrentFile";

    const String stylesheet = "..\\..\\XSLTFile1.xslt";

    int i = 0;

    for(i = 0; i < 1000; i++)

    {

        XslTransform xslt = new XslTransform();

        xslt.Load(stylesheet);

        //Load the XML data file.

        String filename = myConstStr + i.ToString() + ".xml";

        XPathDocument doc = new XPathDocument(filename);

        //Create an XmlTextWriter to write to the stream.         

        XmlTextWriter writer = new XmlTextWriter(stream);

        writer.Formatting = Formatting.Indented;

        //Transform the file.

        xslt.Transform(doc, null, writer);

        writer.close();

    }

So what is wrong with this snippet of code? (hint, there may be more then one thing that can be fixed).

High Memory part 6 – Fragmentation revisited

Tom — Wed, 17 Sep 2008 13:00:00 GMT

I have talked about a bunch of commands used to troubleshoot a managed memory problem in the past and given some situations of using them like:

There is another way that we could see a problem that involves fragmentation. Let’s consider the situation where we see the same situation we saw before, namely:

But we do see a bunch of files listed when we look in lm. So what should this tell us about the problem? Well, first off, since the files are showing up in lm, we know the problem isn’t the same as, Dynamic Assemblies and what to do about them. So next we would want to look and see specifically what the files are.

Situation 1

The first common problem that you will see is caused by either have debug=true, or batch=false is set. Batch is similar to debug, set in the same place and will tell the compiler to not create one DLL for the entire directory but rather create one DLL for each file we compile. Underneath, debug=true sets batch=false, so they do the same thing in this regard. If that is the problem, you will see a bunch of strangely named DLLs in the lm list. Names like:

  hbdp25u7.dll
  1gxdnryw.dll
  fzgq0g7y.dll
  fhlpqzqm.dll
  su0utu5-.dll
  vhaewyii.dll
  tdkuail7.dll

Turning debug off, or batch on will resolve the problem in this situation.

Situation 2

The second common problem that will cause this is if you have a common framework that you are using in multiple applications. One common way to do this is the copy the framework files into the bin directory of each application. This will cause us to create a different in-memory file for each one. So instead of just having 5 files that make up your framework loaded in memory, you will have 5 * (# of AppDomains) loaded.

You can identify this problem because in the list of files, you will see the same filename repeated, only some will have some numbers after them:

  MyFramework.dll
  MyFramework_50008012.dll
  MyFramework_50002008.dll
  MyFramework_40101008.dll

This is just showing the different versions of the file by showing you the other copies with their base address (load address).

The easiest solution to this type of problem is to place the framework files into the GAC. Then you will only have one copy of the files which all the Applications will share.

How do you test the performance of your web site?

Tom — Wed, 10 Sep 2008 13:00:00 GMT

I wanted to get a feel for what tools people are using to test the performance of their website. My questions are:

Do you check page-load times?
Do you dig in to see which files are taking the time
How do you track down a page that uses a lot of memory
How do you track down a high CPU consuming page

I don’t want to list any possible tools as I’d like to hear what people are using without me putting any suggestions out.

If I don’t see some being mentioned, I’ll be sure to bring it up so we have a holistic view of the tools.

Look forward to seeing what people use.

ASP.NET Tips: Finding what is taking up memory

Tom — Wed, 25 Jun 2008 13:00:00 GMT

There are some times that you want to figure out what is taking up your memory. There are a few ways that you can do this and some limitations put on this by .NET and ASP.NET.

Memory in a dump file

If you are looking at a dump file, the first thing you can do is find out how much memory is being taking up by the NT Heaps, Virtual Allocations, Images (EXE and DLL), this is done by using the command: !address –summary:

Here you can see the amount of data that each of these types holds on to. Another really useful output from this command is being able to see what the largest free region we have in the process. This is shown at the very bottom of this output. If that number is less then the size of a new memory allocation, we will fail with an Out Of Memory error.

Once you have this data, you can start digging deeper on some of these things. For example, you can run lm to see all the loaded modules in the process. Most of the time you will want to dig into the virtual allocations (RegionUsageIsVAD). This is where .NET allocates it’s heaps. Running !eeheap -gc will show the GC Heap Size so that we can see if all our virtual memory is .NET or something else.

Note: In a dump file, we cannot figure out the sizes of all individual pages. This is because these objects hold a reference to their parent object and we will end up getting the size of all the pages, not just the one. If you need that type of details, see the profilers below.

Memory in a live process

To look at memory in a live process, there are a number of different tools you can use depending on the situation. The most versatile and most common one to use is perfmon. There are again a few places to focus depending on the situation. You can look under the Process object and add the Private Bytes and Virtual Bytes to get a good overview of all memory. If you want to look at .NET, focus in the .NET CLR Memory object. You can use the # Total committed Bytes and # Total reserved Bytes to see how much memory is being used.

Other tools you can use to look at memory from a .NET perspective are:

You can find a bunch more profilers here.

ASP.NET and Performance

Tom — Mon, 23 Jun 2008 21:29:34 GMT

I wanted to talk about monitoring performance in ASP.NET for a bit. I have already posted a few postings that talk around this issue, namely:

But there are some other things to keep in mind. The best place to look for information on how to watch the performance of an ASP.NET Application is in the “Improving .NET Application Performance” guide on MSDN. Specifically Chapter 15 – Measuring .NET Application Performance. There are a lot of other very useful pieces of information in this. So I would suggest you review any Chapter that applies to your application.

If you want to get a description of all the counters, check out: Performance Counters for ASP.NET and Memory Performance Counters

As for how to capture the perfmon data (perfmon is the tool that we use to capture the performance information), take a look at my various posts around the different situations. This goes through what counters are important in each situation:

ASP.NET Tips- What to gather to troubleshoot

The main ones to keep an eye on are the memory counters (to watch if memory is staying under control or becoming unbounded) and the ones around the ThreadPool (Queuing, Request Execution Time, etc) to make sure things aren’t getting queued up or blocked in some manner.

ASP.NET Tips: What to gather to troubleshoot

Tom — Wed, 21 May 2008 18:38:59 GMT

So now that I am done with all the individual posts, I thought I would wrap them all together. So here are all of the posts that I created on gathering information when you are having a problem.

Please keep in mind that these are geared towards the type of information you would be sending to Microsoft Support. So if you are troubleshooting something yourself, you have have some better methods, such as Tracing or Logging.

So without further ado, here is the complete list:

Hope that these are useful to everyone and hope that you can put these to use. If you think there is a situation that I haven’t covered, please let me know and I will add it also.

ASP.NET Tips: What to gather to troubleshoot - part 6 - Recycle due to memoryLimit

Tom — Tue, 20 May 2008 19:03:04 GMT

Here is another situation that you may come across and what to do in order to gather information about it.

The following steps will configure DebugDiag to automatically capture a memory dump of the ASPNET_WP.exe process recycling when it exceeds the memoryLimit. This only applies to the ASPNET_WP.exe process, so if using w3wp.exe, this doesn't apply. The following event is logged when this occurs:

aspnet_wp.exe xxx was recycled because memory 
consumption exceeded 60% of available RAM

Before the issue occurs

A. Download DebugDiag:
====================

Install the Debug Diagnostic tool
The default installation location for the package is C:\Program Files\DebugDiag. It can be changed.
Make sure any other monitoring and debugging tools are not running.

B. Create a Performance Monitor log:
====================================

Expand Performance Logs and Alerts
Right Click on Counter Logs
Choose New Log Settings...
Enter a descriptive name
Note the log file location for later (or go to the Log Files tab and change the location)
Click the Add button
Click the All Counters and All Instances radio buttons
Select the following from the Performance Object dropdown, being sure to Add each one as you select it:
- Process
- Thread
- Processor
- Memory
- .NET CLR Data
- .NET CLR Exceptions
- .NET CLR Interop
- .NET CLR Jit
- .NET CLR Loading
- .NET CLR LocksAndThreads
- .NET CLR Memory
- .NET CLR Networking
- .NET CLR Remoting
- .NET CLR Security
- ASP.NET
- ASP.NET Applications
Click Close
Click OK

For the ASP.NET counters, select the version that you are wanting to monitor. Ex. For 1.1 framework, select ASP.NET v1.1.4322 and ASP.NET Applications v1.1.4322.

Note: For more information on Performance monitor, see KB 248345

C. Configure server for the High Memory Event
==================================

Add the following registry keys. These were added to the .NET Framework so that a breakpoint exception will be thrown when the process recycles due to the above event.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET
DWord: DebugOnHighMem
Value: 1

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET
DWord: UnderDebugger
Value: 0

C. Configure DebugDiag
===================

Configure DebugDiag to capture the memory dump when the BreakPoint Exception is thrown and when the process stops.

Open DebugDiag
On the Rules tab, click Add Rule
Select Crash and click Next
Select All IIS Processes and click Next
Click Advanced Exception Configuration
Click Add Exception
Select Breakpoint Exception, change Action Type to Full UserDump
Click OK
Click Save and Close
Click Advanced Breakpoint Configuration
Click Add Breakpoint
Type KERNEL32!ExitProcess and change Action Type to Full UserDump
Click OK
Click Save and Close
Click Next through the rest of the wizard

Inject LeakTrack.dll to capture native leak information:

Click the Processes tab
Right-click the ASPNET_WP.exe process, select Monitor for leaks

After the issue occurs

The memory dump will automatically get captured.

A. Stop the Performance Monitor log
===================================

In Performance Monitor:

Right click on your log that is now listed under Counter Logs
Choose Stop Log

ASP.NET Tips: What to gather to troubleshoot - part 5 - OutOfMemoryException

Tom — Mon, 19 May 2008 17:49:41 GMT

So we already talked about what to gather when you are experiencing high memory. But what if you are actually getting a System.OutOfMemoryException? Here is how you capture a dump at the time you are running out of memory.

The following steps will configure DebugDiag to automatically capture the memory dump when the System.OutOfMemoryException occurs.

Before the issue occurs

A. Download DebugDiag:
====================

Install the Debug Diagnostic tool
The default installation location for the package is C:\Program Files\DebugDiag. It can be changed.
Make sure any other monitoring and debugging tools are not running.

B. Create a Performance Monitor log:
====================================

Expand Performance Logs and Alerts
Right Click on Counter Logs
Choose New Log Settings...
Enter a descriptive name
Note the log file location for later (or go to the Log Files tab and change the location)
Click the Add button
Click the All Counters and All Instances radio buttons
Select the following from the Performance Object dropdown, being sure to Add each one as you select it:
- Process
- Thread
- Processor
- Memory
- .NET CLR Data
- .NET CLR Exceptions
- .NET CLR Interop
- .NET CLR Jit
- .NET CLR Loading
- .NET CLR LocksAndThreads
- .NET CLR Memory
- .NET CLR Networking
- .NET CLR Remoting
- .NET CLR Security
- ASP.NET
- ASP.NET Applications
Click Close
Click OK

For the ASP.NET counters, select the version that you are wanting to monitor. Ex. For 1.1 framework, select ASP.NET v1.1.4322 and ASP.NET Applications v1.1.4322.

Note: For more information on Performance monitor, see KB 248345

C. Configure server to capture OutOfMemoryException
===========================================

Add the following registry key. This was added to the .NET Framework so that a breakpoint exception will be thrown when an OutOfMemory condition occurs. This is documented in the following article:

820745 Failfast occurs when you experience an "out of memory" condition

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\
DWord: GCFailFastOnOOM
Value: 2

D. Configure DebugDiag
===================

Configure DebugDiag to capture the memory dump when the BreakPoint Exception is thrown and when the process stops.

Open DebugDiag
On the Rules tab, click Add Rule
Select Crash and click Next
Select All IIS Processes and click Next
Click Advanced Exception Configuration
Click Add Exception
Select Breakpoint Exception, change Action Type to Full UserDump
Click OK
Click Save and Close
Click Advanced Breakpoint Configuration
Click Add Breakpoint
Type KERNEL32!ExitProcess and change Action Type to Full UserDump
Click OK
Click Save and Close
Click Next through the rest of the wizard

Inject LeakTrack.dll to capture native leak information:

Click the Processes tab
Right-click the ASPNET_WP.exe process (or w3wp.exe if running in IIS 6), select Monitor for leaks

After the issue occurs

The memory dump will automatically get captured when the OutOfMemoryException is thrown.

A. Stop the Performance Monitor log
===================================

In Performance Monitor:

Right click on your log that is now listed under Counter Logs
Choose Stop Log

ASP.NET Tips: What to gather to troubleshoot – part 2 – High Memory

Tom — Mon, 05 May 2008 16:28:54 GMT

Identifying a Memory leak

A Memory Leak in an IIS process (INETINFO.EXE, DLLHOST.EXE, W3WP.EXE) occurs when Memory Usage in Task Manager continues to consume more than 50% of the physical RAM until running out of system memory or until the process stops functioning.

On heavily-used sites, a continuous growth in memory over the first 24 hours of use is normal. IIS caches much of its data and the TTL on the cache is 24 hours. Since IIS typically uses 50% of available physical RAM (per IIS Process; not the total of all IIS Processes), use this as a guide to identify a memory problem. The steps below will help troubleshoot a memory leak by gathering a memory dump of the leaking process.

Important Questions

How much RAM is on the machine?
How much RAM does the leaking IIS process take?
How long does it take to reproduce?

Troubleshooting Steps

1. Install the Debug Diagnostic tool

http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1286

The default install path is C:\Program Files\DebugDiag and can be changed during the installation.

2. Set up Performance Monitor Logging before the issue occurs:

a) Open DebugDiag (Start -> Programs –> Debug Diagnostics Tool 1.1)

b) Go to the Tools menu -> Options and Settings

c) Select the Performance Log tab

d) Click "Enable Performance Counter Data Logging"

e) Click OK

Note: The Data Sampling Interval and time to start monitoring is subjective to when the memory leak reproduces. Due to the log size, work with your Support Professional on a solution where we get the information needed while not overwhelming the server.

3. Create a Memory and Handle Leak Rule with the following steps:

a) Open DebugDiag (Start -> Programs -> Debug Diagnostics Tool 1.1)

b) Select "Memory and Handle Leak" and click Next

c) Select the leaking Process and click Next

d) In “Configure Tracking Duration”, a Warm-Up time can be specified but if the issue reproduces immediately, select “Start memory tracking immediately”. For “Tracking Time”, specify the time it will take to reproduce the issue. At the end of the specified time, a dump will be created. Make sure “Auto-create a crash rule” is checked.

e) Click Next

f) Click Finish for "Rule Name". The "Userdump Location" can be changed here.

Notice the Status is Active. The Userdump Count will increase each time a dump file is created.

4. Get the Data Manually if you are not sure when the leak will occur. In DebugDiag, go to the Processes tab, right-click the leaking process and select “Monitor For Leaks”. When the process gets up to 60%-80% of RAM, right-click the leaking process and select "Create Full Userdump”.

5. Stop PerfMon Logging after the dump has finished:

a) Open DebugDiag (Start -> Programs –> Debug Diagnostics Tool 1.1 )

b) Go to the Tools menu -> Options and Settings

c) Select the Performance Log tab

d) Click "Disable Performance Counter Data Logging"

e) Click OK

6. Analyze the Dump by selecting the "Advanced Analysis" tab and clicking "Add Data Files". When the .dmp is added, select the "Memory Pressure Analysis” script and click "Start Analysis". When finished, a report (.mht) will be created in C:\Program Files\DebugDiag\Reports and displayed in Internet Explorer with the results and recommendation. If using custom DLL’s, the Symbol path (Tools menu -> Options and Settings -> Symbol Search Path) to the custom PDB files can be added.

Understanding when to use a Finalizer in your .NET class

Tom — Sat, 26 Apr 2008 01:04:11 GMT

A common problem we see when moving to .NET all revolves around the finalizer. There are a few reasons that this happens:

Developers move from C/C++ to C# and are used to created classes with a constructor and destructor.
Developers don’t understand when they need to implement Dispose or Finalize and thus create them when they aren’t needed.

Finalization – what is it?

So what is Finalization and why is this such an important thing to understand? Well, there is a good description of it found here. Basically it is a way to make sure some code runs when an object gets cleaned up.

Why is it necessary?

If an object holds onto any unmanaged resources, such as file handles, sockets or database connections, the object is responsible for cleaning them up when it is destroyed. This only applies to unmanaged resources!

I would very strongly recommend that you read the documentation for the Finalize object very carefully before you use it. The things to watch for are:

If you use an unmanaged object and hold a reference to it, you must implement a Finalizer to allow that unmanaged object to be cleaned up. Otherwise you will leak unmanaged (heap) memory.
If you create a Finalize method (~Class in C#), even if it is empty, this will put the object on the finalize queue.
If you have a Finalize method, don’t try to clean up managed objects from it. That is why most Finalize methods call Dispose(false). So they don’t clean up any managed objects. This is because finalized can happen in any order and just cause you hold a reference to an object, doesn’t mean it hasn’t been cleaned up already.

Why is a Finalize method bad?

So now we know how we should use the finalizer, but why is it so important to do it correctly? So items 1 and 3 above are pretty explanatory as to why you would want to do that. So what is wrong with #2?

If an object has a finalizer, it will be placed in the FinalizationQueue and is subject so some additional clean-up. Once the object is no longer referenced on a thread or from a global reference, the next time the Garbage Collector (GC) runs, it will see this object is ready to be collected. But it can’t collect it yet. It has to let the Finalizer run first. So the GC will finish collecting, then the Finalizer will Finalize the object, and then another GC collection will occur.

This can have a huge affect on performance as you should remember that all managed threads will be stopped waiting on the GC and then the GC will be stopped waiting on the Finalizer thread.

There is a lot more data out there about Finalization so I encourage you to read about it as much as possible so that you use it in the best way possible.

ASP.NET Tips: How to identify a COM call that is blocking GC, causing a memory leak

Tom — Tue, 22 Apr 2008 17:42:01 GMT

Problem Description

So this is a customer situation we found were the customer was getting an System.OutOfMemoryException. We were able to reproduce this with a console application that just ran:

            For Iterator = 0 To 99999999999
                Regex.IsMatch(CONNECTION_STRING, "/S")
                'Create a SQLConnection object then get rid of it
                Connection.Open()
                Connection.Close()
            Next Iterator

It turns out that the SQLConnection object uses some COM objects internally. This objects need to get collected by the GC (Garbage Collector) but are unable to do so because the COM objects are created on this main STA thread and the finalizer thread cannot call back to the main thread due to this loop.

Debugging information

Looking at all the threads and look for the Finalizer thread

0:000> !threads
ThreadCount: 3
UnstartedThread: 0
BackgroundThread: 2
PendingThread: 0
DeadThread: 0
                                 PreEmptive   GC Alloc                     Lock     
       ID   ThreadOBJ      State     GC       Context           Domain     Count APT Exception
  0 0x7a0 0x0014d358   0x2006020 Enabled  0x00000000:0x00000000 0x00147b18     0 STA
  2 0xf9c 0x00158010      0xb220 Enabled  0x00000000:0x00000000 0x00147b18     0 MTA (Finalizer)
  6 0xe30 0x0019f460   0x1800220 Enabled  0x00000000:0x00000000 0x00147b18     0 MTA (Threadpool Worker)

So the Finalizer thread is thread 2. Looking at the callstack of that thread we see

0:002> kbnL100
 # ChildEBP RetAddr  Args to Child              
00 00bbf74c 77f43741 77e41817 000001e0 00000000 SharedUserData!SystemCallStub+0x4
01 00bbf750 77e41817 000001e0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
02 00bbf7c0 77e4168f 000001e0 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
03 00bbf7d0 771d3c39 000001e0 ffffffff 0016d53c kernel32!WaitForSingleObject+0xf
04 00bbf7ec 7726bef4 0018d7c0 00164538 0016d53c ole32!GetToSTA+0x6d
05 00bbf808 7726b087 00bbf8c4 0016d53c 001918f8 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xc8
06 00bbf8dc 771671c1 0016d53c 00bbf9c4 00bbf9b4 ole32!CRpcChannelBuffer::SendReceive2+0xb6
07 00bbf938 771ac725 0016d53c 00bbf9c4 00bbf9b4 ole32!CAptRpcChnl::SendReceive+0xab
08 00bbf98c 77ce6a4e 00000001 00bbf9c4 00bbf9b4 ole32!CCtxComChnl::SendReceive+0x8f
09 00bbf9a8 77ce6bdf 0018f9f4 00bbf9f0 06000169 rpcrt4!NdrProxySendReceive+0x41
0a 00bbfd78 77ce69cb 77164d98 771d1f38 00bbfdb0 rpcrt4!NdrClientCall2+0x1bc
0b 00bbfd98 77c64fcf 0000000c 00000008 00bbfe1c rpcrt4!ObjectStublessClient+0x89
0c 00bbfda8 771ac365 0018f9f4 00bbfdd0 00000000 rpcrt4!ObjectStubless+0xf
0d 00bbfe1c 772079fb 00000001 7921aa1d 00bbfea4 ole32!CObjectContext::InternalContextCallback+0x10b
0e 00bbfe5c 792190c6 00168018 7921aa1d 00bbfea4 ole32!CObjectContext::ContextCallback+0x6e
0f 00bbff08 79219404 7921aa54 00191058 00000000 mscorwks!CtxEntry::EnterContext+0xb7
10 00bbff30 792192ad 00000000 793e7e48 00000000 mscorwks!ComPlusApartmentCleanupGroup::CleanUpWrappers+0x63
11 00bbff68 7923d2c1 77e4b085 00000000 7923e791 mscorwks!ComPlusWrapperCleanupList::CleanUpWrappers+0x69
12 00bbff74 7923e791 791bbe2e 00000000 00000000 mscorwks!SyncBlockCache::CleanupSyncBlocks+0x85
13 00bbff78 791bbe2e 00000000 00000000 00000000 mscorwks!Thread::DoExtraWorkForFinalizer+0x23
14 00bbffb8 77e4a990 00000000 00000000 00000000 mscorwks!GCHeap::FinalizerThreadStart+0xbb
15 00bbffec 00000000 791ce2dc 00000000 00000000 kernel32!BaseThreadStart+0x34

In here, frame 0f shows us calling into a COM object to clean it up. And the frames 04 and 05 show us switching to an STA apartment to make the COM call. So now lets look at the process and thread that we are switching the COM call to

0:002> dd 18d7c0+8 l2
0018d7c8  00000f58 000007a0

So now lets see which thread 7a0 is

0:002> ~
#  0  Id: f58.7a0 Suspend: 1 Teb: 7ffde000 Unfrozen
   1  Id: f58.bf4 Suspend: 1 Teb: 7ffdd000 Unfrozen
.  2  Id: f58.f9c Suspend: 1 Teb: 7ffdc000 Unfrozen
   3  Id: f58.cb0 Suspend: 1 Teb: 7ffdb000 Unfrozen
   4  Id: f58.75c Suspend: 1 Teb: 7ffda000 Unfrozen
   5  Id: f58.698 Suspend: 1 Teb: 7ffd9000 Unfrozen
   6  Id: f58.e30 Suspend: 1 Teb: 7ffd8000 Unfrozen
   7  Id: f58.b38 Suspend: 1 Teb: 7ffd7000 Unfrozen

We can see that thread f58.7a0 is thread 0. This tells us that the finalizer thread is trying to clean up an object and it is a COM object. In order to clean up this STA object, we have to call back to it’s owner thread which is thread 0.

Looking at thread 0 we see

0:000> kbn100
 # ChildEBP RetAddr  Args to Child              
00 0012f2c8 77f4372d 77e41bfa 00000003 0012f318 SharedUserData!SystemCallStub+0x4
01 0012f2cc 77e41bfa 00000003 0012f318 00000001 ntdll!NtWaitForMultipleObjects+0xc
02 0012f374 77161e96 00000003 0012f4f0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
03 0012f39c 771d36eb 0012f4f0 00000003 0012f3bc ole32!CCliModalLoop::BlockFn+0x2d
04 0012f404 7921a9e4 00162e20 00003a98 00000003 ole32!CoWaitForMultipleHandles+0xc5
05 0012f424 7921a96f 00000000 00003a98 00000003 mscorwks!NT5WaitRoutine+0x51
06 0012f44c 792982dc 00000003 0012f4f0 00000000 mscorwks!MsgWaitHelper+0x62
07 0012f47c 7929883d 00000003 0012f4f0 00000000 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
08 0012f4d0 7933992b 00000003 0012f4f0 00000000 mscorwks!Thread::DoAppropriateWait+0x46
09 0012f534 00f80e8f 00000003 00000000 00000000 mscorwks!WaitHandleNative::CorWaitMultipleNative+0xff
WARNING: Frame IP not in any known module. Following frames may be wrong.
0a 0012f5a8 00febbbd 00000000 00000000 0504b1d4 0xf80e8f
0b 0012f5e8 00edff67 00000000 05041738 0012f6ec 0xfebbbd
0c 0012f62c 00edf44e 00000000 05046adc 0012f6ec 0xedff67
0d 0012f66c 00c51897 00000000 0012f6c8 0012f6ec 0xedf44e
0e 0012f6a8 791d7478 0012f7c0 791dcf66 0012f6fc 0xc51897
0f 0012f6b0 791dcf66 0012f6fc 00000000 0012f6d4 mscorwks!CallDescrWorker+0x30
10 0012f7c0 791dd2ca 008f507b 0015b9a8 0012f7e0 mscorwks!MethodDesc::CallDescr+0x1b6
11 0012f870 791dd336 008f507b 0015b9a8 110028f4 mscorwks!MethodDesc::CallDescr+0x43
12 0012f898 792363e9 0012f928 00000000 0014d358 mscorwks!MethodDesc::Call+0x97
13 0012f950 79238c4c 008f5080 00000001 00000000 mscorwks!ClassLoader::CanAccess+0x207
14 0012fa64 79238d49 0015b9a8 00000000 79041374 mscorwks!ClassLoader::ExecuteMainMethod+0x49d
15 0012fa7c 79238a36 00000000 0012fd70 00000000 mscorwks!Assembly::ExecuteMainMethod+0x21
16 0012fd60 791bcae7 00000001 0012fd90 0012ffe0 mscorwks!SystemDomain::ExecuteMainMethod+0x416
17 0012ffa0 791bc9ea 11000000 00000000 00000000 mscorwks!ExecuteEXE+0x1c0
18 0012ffb0 7917d0b8 00000000 791b0000 0012fff0 mscorwks!_CorExeMain+0x59
19 0012ffc0 77e4f38c 00000000 00000000 7ffdf000 mscoree!_CorExeMain+0x30
1a 0012fff0 00000000 7917d08c 00000000 78746341 kernel32!BaseProcessStart+0x23

Looking at the managed stack since frames 0a to 12 show what looks like managed code

0:000> !clrstack
Thread 0
ESP         EIP       
0x0012f568  0x7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] I4 System.Threading.WaitHandle.WaitMultiple(SZArray Class System.Threading.WaitHandle,I4,Boolean,Boolean)
0x0012f580  0x00fec071 [DEFAULT] I4 System.Threading.WaitHandle.WaitAny(SZArray Class System.Threading.WaitHandle,I4,Boolean)
  at [+0xc9] [+0x41]
0x0012f5b4  0x00febbbd [DEFAULT] [hasThis] Class System.Data.SqlClient.SqlInternalConnection System.Data.SqlClient.ConnectionPool.GetConnection(ByRef Boolean)
  at [+0xad] [+0x47]
0x0012f5f0  0x00edff67 [DEFAULT] Class System.Data.SqlClient.SqlInternalConnection System.Data.SqlClient.SqlConnectionPoolManager.GetPooledConnection(Class System.Data.SqlClient.SqlConnectionString,ByRef Boolean)
  at [+0x1a7] [+0xc3]
0x0012f634  0x00edf44e [DEFAULT] [hasThis] Void System.Data.SqlClient.SqlConnection.Open()
  at [+0xce] [+0x44]
0x0012f674  0x00c51897 [DEFAULT] Void ConsoleApplication1.Module1.Main()
  at [+0x9f] [+0x37]
0x0012f9b0  0x791d7478 [FRAME: GCFrame] 
0x0012fa94  0x791d7478 [FRAME: GCFrame]

Summary

So we can see that ConsoleApplication1.Module1.Main is calling into System.Data.SqlClient.SqlConnection.Open which is waiting for the connection to open. In a real world scenario you would see it doing different things in a loop but never pumping the message loop. In order for an STA component to receive calls the thread must pump messages (for more information see the Single-Threaded Apartments (STA) section of DCOM Architecture).

Note that any managed object could be using an STA COM object internally (in this case it was the Microsoft SqlConnection object) and there is no easy way to determine if it is doing so. This means that you should always pump messages on any thread that creates managed objects if you don’t know the internal workings of the managed object (any object you did not develop).

The following are some methods to determine if a managed object uses COM internally:

Using a debugger and setting a breakpoint on COM activation functions in ole32
Use regmon and watch for the process looking up CLSIDs
Digging through the IL and looking for COM calls

Solutions

The best solution is to pump messages on the STA thread. From MSDN we can see easy ways to do this, from the “Blocking Issues” section of and Unmanaged Threading in Microsoft Windows

WaitHandle.WaitOne, WaitAny, WaitAll, Monitor.Enter, Monitor.Block, Thread.Join, GC.WaitForPendingFinalizers, and so on are all responsive to Thread.Interrupt and to Thread.Abort. Also, if your thread is in a single-threaded apartment, all these managed blocking operations will correctly pump messages in your apartment while your thread is blocked
Mark the main thread as an MTA thread
If it is not easy to modify the main thread then you can create another thread that will handle the creation of the STA COM objects and will also pump messages.

Chat Question: What extension to use in what situation

Tom — Fri, 18 Apr 2008 18:34:13 GMT

So we have a bunch of debugger extensions that we need to use depending on the situation. So here are the extensions that can depend on various situations and when to use which. The first thing to decide is to load the right DLL for the Framework version and Process version:

Extension	Process Version	Framework Version
clr10\sos.dll	any 32-bit process	.NET 1.0/.NET 1.1
Framework\v2.0.50727\sos.dll	any 32-bit process	.NET 2.0/.NET 3.0/.NET 3.5
Framework64\v2.0.50727\sos.dll	any 64-bit process	.NET 2.0/.NET 3.0/.NET 3.5
IISInfo.dll	any 32-bit process	N/A – IIS extension

Keep in mind that you can run a 32-bit process on a 64-bit operating system. You would still use the 32-bit files to debug it.

If you are dealing with a crash, a good starting point is always to run !analyze -v. This will check for known issues and help to narrow down where the problem is for a crash. It isn’t perfect for .NET, but will do a great job getting you something to go on.

Once you have the right extension loaded, then you can start troubleshooting things.

Note: In the future, I hope to have a new extension that will run on top of SOS.dll and allow you to do some more advanced functions. This will give similar results to the clr10\sos.dll extension. But it will be able to run against all versions of the framework. I am still flushing things out on that, feel free to give me your comments here.

Keep checking the RECAP- ASP.NET Blog Chat to see other topics that Tess or I write about.

Chat Question: Memory Limits for 32-bit and 64-bit processes

Tom — Thu, 10 Apr 2008 18:00:24 GMT

During our recent blog chat, there were a number of topics that were asked about and I am going to expand on some of them. The first one is the memory limits for different processes.

This really depends on a few different things. The architecture of the process (32-bit or 64-bit) and also the architecture of the Operating System the process is running on. For 32-bit it also depends if you use the /3GB switch or not.

So I broke this up based on these things into the table below, this is the maximum amount of memory available for the process assuming you have that much RAM and Pagefile space.

	32-bit OS	64-bit OS
32-bit process	2 GB	4 GB
32-bit process with /3GB	3 GB	N/A
64-bit process	N/A	16 TB

These process numbers are contingent on how much RAM and disk space you have, so if you have 4 GB of RAM and 4 GB Pagefile, the total memory of all running processes can’t be greater then 8 GB.

Note: If you let Windows manage your Pagefile size, when you hit this limit, Windows will try to grow your Pagefile as long as there is disk space available.

For the amount a .NET application will use before we can expect to see out of memory, those numbers are:

	32-bit OS	64-bit OS
32-bit process	800-1200 MB	2800 MB
32-bit process with /3GB	1800 MB	N/A
64-bit process	N/A	2800 MB if using a 4 GB process or more if more RAM (around 70% of RAM + Pagefile)

Keep in mind that although a .NET process can grow this large, if the process is multiple GB in size, it can become very difficult for the Garage Collector to keep up with the memory as Generation 2 will become very large. I’ll talk about the generations more in an upcoming post.

Hopefully that will clear up how much memory each one uses.

Keep checking the RECAP- ASP.NET Blog Chat to see other topics that Tess or I write about.

GDI+ (GdiPlus) and ASP.NET

Tom — Wed, 19 Mar 2008 19:31:42 GMT

So we occasionally see customers that are either using System.Drawing themselves in their ASP.NET application or are using controls (like charting controls) that use it.

Under most circumstances, this works just fine. But there are times where it does not, and this is cautioned to users on the System.Drawing page on MSDN.

We recently had a case where GDI+ was causing memory to jump up very quickly. Looking at the heaps in DebugDiag showed us:

This shows the GdiPlus heap being very fragmented. And looking at the crt heap, where GDI will allocate things:

If you look at the full picture, you will see that there is an allocation size of 2,691,448 which is taking up 782.86 MB.

There is a very good description of this on:
Charting in ASP.NET (alas: DGI+ not supported in a service)

Note that there are also sites which talk about using System.Drawing on ASP.NET, for example:
http://www.geekpedia.com/tutorial123_Working-with-GDIplus-in-ASP.NET.html

While this code will work, I would caution that if you run into problems, don't be surprised if this is what is causing the problem, especially if it gets heavy use on your site.

High Memory part 5 - Fragmentation

Tom — Mon, 18 Feb 2008 19:27:00 GMT

So in our previous High Memory posts, we have talked about situations that are all causes of memory to appear high when looking at the process in Task Manager or Perfmon.

But in most situations, you aren't going to be monitoring these tools. So the way you will find out that you are high in memory is that the Web Server will crash and you will see that the exception given was System.OutOfMemoryException.

There are a number of problems that can cause this, as you can probably guess since this is part 5 of the High Memory blogs. So what if when you look at the process, it isn't taking up much memory, but is still giving you the OutOfMemoryException?

Problem 1

Let's take a look at a dump with this as the problem. So first we look at !address -summary as always, but don't see much memory in the Virtual Memory or Native Heaps:

But notice that the RegionUsageImage section is pretty high. This is the area where all of our files are. Like the DLLs and EXEs of the actual process. But if you look at the output from lm, you won't see that much data. So what is going on here? How can we be out of memory, with a lot of memory listed as being images, but list modules (lm) doesn't show that many?

The answer is something we have discussed in the past, Dynamic Assemblies and what to do about them. If we run the same command that is in the troubleshooting section of that post we can see:

And we can clearly see from this that we have a lot of dynamic assemblies. So now we just follow the same steps as in that previous post to see what they are and reduce the amount of them that we are using.

Problem 2

There is another type of Fragmentation that we could see. That we first mentioned in High Memory part 4. That is high memory in .NET, but when you look at the output of !dumpheap -stat you see something like:

From this we can see that the largest amount of memory is being taking up by System.Free (Free in the list). This is a special object that is used by .NET when an object is no longer being referenced, but we haven't been able to recapture the memory yet. So why haven't we been able to recapture the memory? Well the reason is because of the blocks that are printed below the report. These are the actual Free blocks that are in .NET and the object that is after them is also listed. Theses are typically caused by objects being pinned. There is a very good description of the situation here.

For this particular dump, the fix was to install the hotfix, 831138.