If broken it is, fix it you should

Kinect SDK for Windows – Hover Button / Hover Control

Tess1 — Tue, 16 Aug 2011 13:07:00 GMT

This is part of what will be a series of posts on Kinect SDK demos and howtos, go here to see the whole list.

In this post I will talk about how you can create a simple and versatile Hover Button / Hover Control for Kinect with a Metro look.
The control is largely based on the Kinect Buttons from Shai Raiten, so all cred goes to him, this is just a customization of his work.

Hover button UI

The HoverButton control can have either an image or text or both and can be resized to fit your needs. It features a Click event that will occurr after a hovering for 2 seconds.

Below are two examples of how you could use the HoverButton control…

In the first example I have used 10 HoverButtons to create a metro style UI for
getting into different demos and I have used both images and just buttons with text together so you can see the difference.

The center button is currently selected and the white overlay slowly covers the whole button… when the button is fully covered a Click event is fired.

In the second example I have used only images to show a completely different effect with the same hover button where the 3rd avatar is being selected.

Creating the HoverButton

Create a new WPF user control library (KinectControls) and add a new UserControl called HoverButton
Add Dependency Properties

The dependency properties will be used to bind xaml controls in the HoverButton so that we can customize the button from the using application

Add a dependency property for BackgroundColor (the color of the button) like this
	public Brush BackgroundColor{
		get{ return (Brush)this.GetValue(BackgroundColorProperty);}
		set{ this.SetValue(BackgroundColorProperty, value);}
	} 
	public static readonly DependencyProperty BackgroundColorProperty = DependencyProperty.Register(
	   "BackgroundColor", typeof(Brush), typeof(HoverButton), new PropertyMetadata(Brushes.Red));
Do the same for the following properties

Property Type Default value

HoverColor Brush Brushes.White

TextColor Brush Brushes.White

TextSize int 36

Image string “”

3. Create the XAML for the HoverButton and bind to the relevant controls

    <grid background="{Binding BackgroundColor}">
        <img stretch="Fill" source="{Binding Image}" />
        <textblock verticalalignment="Center" horizontalalignment="Center" fontfamily="Segoe UI" foreground="{Binding TextColor}" 
                fontsize="{Binding TextSize}" text="{Binding Text}" />
        <rectangle verticalalignment="Top" horizontalalignment="Left" height="0" rendertransformorigin="0,0" opacity="0.3" 
                width="{Binding Width}" fill="{Binding HoverColor}" name="Mask" />
    </grid>

4. Set up the data binding in the HoverButton constructor to bind the properties

    this.DataContext = this;

Now we are done with the properties, it is time to do the hover magic… for this we will use the Mask rectangle and animate it to grow when the cursor enters the button, and reverse the animation when the cursor exits

5. Add the following fields to the HoverButton class

        private Duration hoverDuration = new Duration(new TimeSpan(0, 0, 2));
        private Duration reverseDuration = new Duration(new TimeSpan(0, 0, 1));
        private DoubleAnimation maskAnimation;
        private bool isHovering = false;

the hoverDuration is the time it will take to do the full hover

6. Create two methods to start hovering and stop hovering that initiate and reverse the animation

        private void StartHovering()
        {
            double maxFillHeight = this.ActualHeight;

            if (!isHovering)
            {
                isHovering = true;
                maskAnimation = new DoubleAnimation(Mask.ActualHeight, maxFillHeight, hoverDuration);
                maskAnimation.Completed += new EventHandler(maskAnimation_Completed);
                Mask.BeginAnimation(Canvas.HeightProperty, maskAnimation);
            }
        }

        private void StopHovering()
        {
            if (isHovering)
            {
                isHovering = false;
                maskAnimation.Completed -= maskAnimation_Completed;
                maskAnimation = new DoubleAnimation(Mask.ActualHeight, 0, reverseDuration);
                Mask.BeginAnimation(Canvas.HeightProperty, maskAnimation);
            }
        }

7. Create a Click event that will fire when we have hovered for the hoverDuration

        public delegate void ClickHandler(object sender, EventArgs e);
        public event ClickHandler Click;

8. Finally create a maskAnimation_Completed eventhandler we will fire the Click event

        void maskAnimation_Completed(object sender, EventArgs e)
        {
            isHovering = false;
            if (Click != null)
                Click(this, e);
            Mask.BeginAnimation(Canvas.HeightProperty, null);
        }

Now the animation logic is complete, we just have to trigger the StartHovering and StopHovering methods, and we will do this by manually checking when a cursor enters and exits the button area… from user code we will have to call a function Check whenever the cursor moves…

9. Implement the Check and IsCursorInButton methods that check if the cursor (a framework element) is inside or outside the button

        public bool Check(FrameworkElement cursor)
        {
            if (IsCursorInButton(cursor))
            {
                this.StartHovering();
                return true;
            }
            else
            {
                this.StopHovering();
                return false;
            }
        }

        private bool IsCursorInButton(FrameworkElement cursor)
        {
            try
            {
                //Cursor midpoint location
                Point cursorTopLeft = cursor.PointToScreen(new Point());
                double cursorCenterX = cursorTopLeft.X + (cursor.ActualWidth / 2);
                double cursorCenterY = cursorTopLeft.Y + (cursor.ActualHeight / 2);

                //Button location
                Point buttonTopLeft = this.PointToScreen(new Point());
                double buttonLeft = buttonTopLeft.X;
                double buttonRight = buttonLeft + this.ActualWidth;
                double buttonTop = buttonTopLeft.Y;
                double buttonBottom = buttonTop + this.ActualHeight;

                if (cursorCenterX < buttonLeft || cursorCenterX > buttonRight)
                    return false;

                if (cursorCenterY < buttonTop || cursorCenterY > buttonBottom)
                    return false;

                return true;
            }
            catch
            {
                return false;
            }
        }

And with that we are done with the button…

As a helper I have also created a HandCursor control that is extremely simple with the following XAML

    <Ellipse Height="40" Width="40" Name="hand">
        <Ellipse.Fill>
            <ImageBrush ImageSource="/KinectControls;component/Images/hand.png" />
        </Ellipse.Fill>
    </Ellipse>

… and just one function (note: for this to work we need to include the Microsoft.Research.Kinect and Coding4Fun.Kinect.WPF dlls and add using statements for Microsoft.Research.Kinect.Nui and Coding4Fun.Kinect.WPF)

        public void SetPosition(Joint joint)
        {
            Joint scaledJoint = joint.ScaleTo(800, 600, 0.5f, 0.5f);
            Canvas.SetLeft(this, scaledJoint.Position.X);
            Canvas.SetTop(this, scaledJoint.Position.Y);
        }

Using the HoverButton

1. Add a reference to the KinectControls library in the application

2. Add the following XAML to the Window to add the HoverButton and HandCursor

    <Canvas Background="Black">
        <ctrl:HoverButton Name="btn1" BackgroundColor="Blue" Text="Test" HoverColor="White" Height="100" Width="200" Click="HoverButton_Click" Canvas.Top="100" 
                 Canvas.Left="100"/>
        <ctrl:HandCursor Name="hand" Canvas.Top="0" Canvas.Left="0"/>
    </Canvas>

3. Add a kinect Runtime to the window class

        Runtime kinect = null;

4. Initialize the kinect in window_loaded and uninitialize it when the window closes (this uses the RuntimeExtensions methods shown in the previous post

        private void Window_Loaded(object sender, RoutedEventArgs e)
        {
            kinect = new Runtime();
            kinect.Initialize(RuntimeOptions.UseSkeletalTracking);
            kinect.SetSmoothTransform();
            kinect.SkeletonFrameReady += new EventHandler(kinect_SkeletonFrameReady); this.Cursor = Cursors.None; } private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e) { if (kinect != null) kinect.Uninitialize(); kinect = null; }

5. And the last thing we need to do before we can try it out is to move the cursor in the SkeletonFrameReady event, as well as checking the button

        void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            SkeletonData data = e.SkeletonFrame.GetFirstTrackedSkeleton();
            Joint handJoint = data.Joints[JointID.HandRight];

            hand.SetPosition(handJoint);
            btn1.Check(hand);
        }

Voila, we are ready to rock!

Have fun,
Tess

Kinect SDK for Windows–Extensions Methods

Tess1 — Tue, 16 Aug 2011 13:05:00 GMT

This is part of what will be a series of posts on Kinect SDK demos and howtos, go here to see the whole list.

Every time I create a new Kinect SDK demo I find myself doing the same setup so I created a few extensions methods that I use in my projects to make life easier.
I figured I would share them here since I will be using them in my blog posts about Kinect in the future.

The extension methods are SetSmoothTransform, which sets the smoothing parameters on the skeleton to stabilize the skeleton movement a bit, and the second method (GetFirstTrackedSkeleton) simply grabs the first skeleton that is tracked, so that we can use it in the SkeletonFrameReady event if we only need data about one skeleton.

   public static class RuntimeExtensions{
        public static void SetSmoothTransform(this Runtime runtime)
        {
            runtime.SkeletonEngine.TransformSmooth = true;

            var parameters = new TransformSmoothParameters
            {
                Smoothing = 0.75f,
                Correction = 0.0f,
                Prediction = 0.0f,
                JitterRadius = 0.05f,
                MaxDeviationRadius = 0.04f
            };

            runtime.SkeletonEngine.SmoothParameters = parameters;
        }

        public static SkeletonData GetFirstTrackedSkeleton(this SkeletonFrame skeletonSet){
            SkeletonData data = (from s in skeletonSet.Skeletons
                                 where s.TrackingState == SkeletonTrackingState.Tracked
                                 select s).FirstOrDefault();
            return data;
        }
    }

I would also recommend another extension method from Greg Duncan called CreateLivePlayerRenderer which extracts just the player from the video, based on depth data… I will use this later in the series.

Enjoy,
Tess

Kinect SDK for Windows – demos

Tess1 — Tue, 16 Aug 2011 12:56:00 GMT

Lately I have been working quite a bit with the Kinect SDK for Windows, which you can install and use on your computer from our research department

Geting started

The only thing you need to get started is a computer running Windows, a Kinect device (must be the one with a power chord, but you can buy the power chord separately very cheep at most stores where they sell Kinect), and the Kinect SDK.

Once you have everything installed I suggest you go to Channel9 and watch the QuickStarts to familiarize yourself with the basics and finally move over to the Channel9 Kinect blog to get inspiration and look at all the cool projects people are making.

This series

In this series I will post a number of different controls, extensions and demo apps that I have created. Some are based on inspiration from the Kinect blog and other sources. The list of demos/howtos here will hopefully grow over time

1. RuntimeExtension methods for common stuff
2. Creating a Kinect Hover Button / Hover Control

Enjoy,
Tess

Entering geek territory or exiting geek territory? That’s the question…

Tess1 — Fri, 06 May 2011 14:12:00 GMT

After 12 years in support I decided to lift my head above the WinDbg window and take a look at what the rest of the world has to offer…

In fact, I decided not only to have a look but take a long long jump to the other end of the software spectrum, miles away from “My ASP.NET app hangs” to the world of “This new stuff that will be released in 2014 is soooo cool I’m gonna die”, and believe me, it’s a scary leap

For the next two months I’ll be working in a team at Microsoft called DPE (Developer Evangelism), looking at what’s next in Web, Phone, Kinect, HTML5, you name it, still doing what I love most though, which is making people excited about technology and talking about what I learn along the way.

Just to explain how far away from my “day job” this is, I’ll share what my first assigment was… my first task in this new role was to design a character for this site we are starting up http://www.migbi.se with all swedish content for people getting started with app and web development.

Several people have said, ooooh so you are going to join the geeks now Am I? I thought it couldn't get more geeky than spending your days in WinDbg, I guess I’ll just have to wait and see…

What does this mean for the blog?

To be honest I don’t know… It’s been so busy lately that I haven’t had time to take care of the blog as much as I should have, but I think it means that you will be seeing a lot more varied content here.

I am too inlove with WinDbg and debugging to stop it alltogether, and the plan is for this stint to be temporary, but it might mean that I’ll lean more towards showing stuff that I work on in my current role for a while… hope you don’t mind

Have a great weekend all,

Tess

Reader email: Need help troubleshooting perf/memory issues

Tess1 — Mon, 06 Dec 2010 09:43:00 GMT

I get several emails every day (through the blog) with urgent requests to help troubleshoot memory issues or memory related performance issues. Some just say “we have a memory issue, how do I troubleshoot it?” and others are far far more detailed with links to dumps and everything:)

I love getting emails, but since I have a day job too (troubleshooting such issues:)) I unfortunately have to ignore most such requests through the blog:( to have time to help the customers that contact me through the normal support channels, and therefore I would recommend that if you do have a pressing issue, that you create a support case on http://support.microsoft.com/ to make sure that you get a dedicated support engineer and that your issue gets the attention it deserves.

Having said all that, I wanted to point out one email I got today, both because I could see that the person writing it had put some effort into try troubleshooting the issue before contacting me, and because I think and hope that my answer might be able to give some starting points to other people in his situation.

Hi Tess,

We have a week to go for production and are stuck in one performance issue. Please guide me in resolving this.

We are running an ASP .Net (2.0) App on Windows Server 2003 (SP2), dual-processor, 4GB RAM. During stess-testing our app with 60 users, we see a drop in throughput from 17th minute to 22nd minute.
It rises and drops again around 40th minute.

I took 4 memory dumps for 8 mins, ran the perfmon counters. Here are the details:

        Dump1     Dump2     Dump3     Dump4
Size     1.16G     1.12G     1.04G     1.13G
GC       802MB     750MB     648MB     738MB
Loader   18MB     18MB     18MB     18MB Image 236MB     236MB     236MB     236MB
MEM_RESV     220MB     236MB     347MB     255MB
MEM_COMM     1.19GB    1.06GB    1021MB    1.08GB

Perfmon Counters (8 mins)
LOH:             307MB,
#Bytes in all heaps:     798MB,
%Time in GC:         49%(avg),
#Induced GC:        14 (avg).

So I can deduce the following:

1. We are stuck in GC for the 8 minutes.
2. Assembly leak is not the problem.
3. Am not sure if Finalizer thread is being blocked.

The kb reads as below:

0:017> kb
ChildEBP RetAddr     Args to Child
0e1afc74 7c827d0b     77e61d1e 00000334 00000000     ntdll!KiFastSystemCallRet
0e1afc78 77e61d1e     00000334 00000000 0e1afcbc     ntdll!NtWaitForSingleObject+0xc
0e1afce8 79e8c5f9     00000334 000007d0 00000000     kernel32!WaitForSingleObjectEx+0xac
0e1afd2c 79e8c52f     00000334 000007d0 00000000     mscorwks!PEImage::LoadImage+0x1af
0e1afd7c 79e8c54e     000007d0 00000000 00000000     mscorwks!CLREvent::WaitEx+0x117
0e1afd90 79fc2397     000007d0 00000000 00000000     mscorwks!CLREvent::Wait+0x17
0e1afdac 79fc24b4     000f2b28 0e1afeb0 000f3328     mscorwks!SVR::WaitForFinalizerEvent+0x4a
0e1afdc0 79e9845f     0e1afeb0 00000000 00000000     mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x49
0e1afdd4 79e983fb     0e1afeb0 0e1afe5c 79f7759b     mscorwks!Thread::DoADCallBack+0x32a
0e1afe68 79e98321     0e1afeb0 9b91703f 00000000     mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0e1afea4 79eef6cc     0e1afeb0 00000000 0010f770     mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0e1afecc 79eef6dd     79fc2469 00000008 0e1aff14     mscorwks!ManagedThreadBase_NoADTransition+0x32
0e1afedc 79ef12ab     79fc2469 9b91718f 00000000     mscorwks!ManagedThreadBase::FinalizerBase+0xd
0e1aff14 79f92015     00000000 00000000 89244470     mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
0e1affb8 77e64829     000f3328 00000000 00000000     mscorwks!Thread::intermediateThreadProc+0x49
0e1affec 00000000     79f91fcf 000f3328 00000000     kernel32!BaseThreadStart+0x34

4. !finalizequeue results
Heap 0
generation 0 has 158 finalizable objects (1d1d42c8->1d1d4540)
generation 1 has 269 finalizable objects (1d1d3e94->1d1d42c8)
generation 2 has 1961 finalizable objects (1d1d1ff0->1d1d3e94)
Ready for finalization 0 objects (1d1d4540->1d1d4540)
------------------------------
Heap 1
generation 0 has 830 finalizable objects (1d23d7c0->1d23e4b8)
generation 1 has 284 finalizable objects (1d23d350->1d23d7c0)
generation 2 has 1238 finalizable objects (1d23bff8->1d23d350)
Ready for finalization 0 objects (1d23e4b8->1d23e4b8)

How do I proceed from here to isolate the issue?

Thanks,

Looking at the above, these were the comments and suggestions for further troubleshooting that I sent him.

Hi,

I don’t agree that you are stuck in a GC for 8 minutes, you can check with # Gen 0/Gen 1/Gen 2 collections and will probably see them increasing. The reason I say that you are not stuck is because the % time in GC counter will only update on completion of a GC so if you were stuck it would not update.

I do think however that your % Time in GC looks really high and I think there might be two reasons for this

You are reaching the point where mem usage gets really close the OOM range (typically around 1 GB private bytes) so the process is probably working really hard to get rid of everything it can. # Induced GC here is not 0 which means that someone called GC.Collect, and unless it is your app that calls GC.Collect it is probably ASP.NET doing this to clean up at an OOM.

Each induced GC will be a full GC which can get pretty costly performance wise.

Recommendation: Look at general memory usage, i.e. look at !dumpheap –stat etc. and see what is using all this .net memory and try to find places where you can reduce the memory usage, or alternatively, if you need to use all this memory, then you would need to move to a 64-bit environment. (You might want to look at some of my posts on troubleshooting .net memory issues to narrow this down). A potential also, if you are running multiple sites here, would be to separate them into different app pools.

Your LOH looks pretty large (307 MB), if you are continuously allocating on the LOH you will also trigger a lot of full GC’s.

Recommendation: have a look at !dumpheap –min 85000 and see if there are any unexpected large objects, or if there are any that you can reduce in size or in number of objects. (See http://blogs.msdn.com/b/tess/archive/2006/06/22/asp-net-case-study-high-cpu-in-gc-large-objects-and-high-allocation-rates.aspx for more details)

Regarding the finalizer, the finalizer is not stuck, it is sitting in mscorwks!SVR::WaitForFinalizerEvent+0x4a which means that it is just idling, waiting for the GC to tell it to finalize.

I have a post with common asp.net threads and how they look in the “normal” idling state here http://blogs.msdn.com/b/tess/archive/2007/04/02/things-to-ignore-when-debugging-an-asp-net-hang-update-for-net-2-0.aspx that might be useful for further debugging.

You can further see that finalization is not a problem because !finalizequeue reports no objects to ready for finalization in either heap “Ready for finalization 0 objects”. Here you can find an example of what it would look like if the finalizer was actually stuck.

Hope that helps,
Tess

Fix for RestartWWWService=false in Visual Studio 2010

Tess1 — Tue, 09 Nov 2010 08:34:30 GMT

We recently published a fix for the following problem:

You create a deployment project for a web application in Visual Studio 2010, set the deployment property RestartWWWService=false and deploy to a Windows 2003 server.
http://msdn.microsoft.com/en-us/library/22f9106b.aspx

In this situation the w3wp.exe process will still be recycled even though the RestartWWWService is set to false.

The fix is available at http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=32285 and should only be installed on development machines if you experience this problem.

Have a good one,

Tess

Debug Diag 1.2 (Beta)

Tess1 — Thu, 30 Sep 2010 12:36:59 GMT

Are you looking for some good stuff to put in your family’s Christmas stockings this year? Maybe a new phone, a controller for your gaming console, why not a brand new debugging tool:)

Jokes aside, I get peppered with questions about Debug Diag and if there is a new version coming that will support Windows 7 / Windows 2008 and luckily some of my EE colleagues have been and are still working hard at making this happen.

While it is not available for public download yet at the MS download site, Beta 1 of Debug Diag has been released and is ready for use. The main new stuff in Beta 1 is that it now supports the aforementioned Windows 7 and 2008, and if you need it right now, you can send an email to dbgdiag (at) microsoft.com, or if you have a case open, you can ask the support engineer you are working with to send it to you.

It has a 32-bit and a 64-bit version (for 1.2, install the 64bit version on 64bit OSes and 32bit version on 32bit OSes, independently of the bitness of the target process).

I will write more when Beta2 is out but Beta2 looks very promising with some new features that will help our apps easier to debug.

Tess

Do you know ISA/Forefront and want to work in the cube next to mine (Sweden)?

Tess1 — Wed, 29 Sep 2010 09:43:19 GMT

If you live in Stockholm, know a lot about ISA/Foreftont/TMG/UAG and you’re just aching to spend the days debugging and troubleshooting interesting problems, we have a full-time Microsoft position open right now.

Not only that, but you would get to have your morning “fika” with some fine people like Graham, Spike and Johan :)
(ok, better stop now before this turns into an infomercial)

Anyways, if it tickles your fancy, you can read more about it on the ISA blog and send your CV to my manager Roffe - rolfhe(at)microsoft.com.

/Tess

Capturing memory dumps for 32-bit processes on an x64 machine

Tess1 — Wed, 29 Sep 2010 09:17:00 GMT

This is an issue that I often get questions around and we often have cases where we have to re-capture memory dumps because the memory dumps were captured the “wrong” way.

The short story is: If you are executing a 32-bit process on a 64-bit machine, (which is the default case for IIS on x64 machines for example) you need to capture the dump with a tool that allows you to create 32-bit dumps.

How do you know if your process is 32-bit?

If you are on a 64-bit machine, you can check task manager to see what architecture your process is using.

Processes with *32 are 32-bit and the rest are 64-bit so in the example above we can see that for example w3wp.exe is executing 32-bit code.

Why is it important to capture them with the right tools?

If you capture a dump with a tool that captures 64-bit dumps you will still get a memory dump, but you will get a memory dump of the syswow64 which means that a lot of extensions like sos and psscor2 won’t be able to read the data. Some things may still work but it is very limited and you might get errors or erroneous call stacks etc.

Typical stuff that happens when you try to read a 64-bit memory dump of a 32-bit process is:

1. You may see warnings like

  WARNING: wkscli overlaps srvcli
  ..............WARNING: wship6 overlaps dnsapi
  .WARNING: IPHLPAPI overlaps dsrole
  ...WARNING: FWPUCLNT overlaps rasadhlp
  WARNING: FWPUCLNT overlaps dnsapi
  .....WARNING: compstat overlaps iisres

2. The stacks show wow64cpu methods

0:000> kp 
   Child-SP          RetAddr           Call Site 
   00000000`000ce728 00000000`73a22bcd wow64cpu!CpupSyscallStub+0x9 
   00000000`000ce730 00000000`73a9d07e wow64cpu!Thunk0ArgReloadState+0x1a 
   00000000`000ce7f0 00000000`73a9c549 wow64!RunCpuSimulation+0xa 
   00000000`000ce840 00000000`76d684c8 wow64!Wow64LdrpInitialize+0x429 
   00000000`000ced90 00000000`76d67623 ntdll!LdrpInitializeProcess+0x17e2 
   00000000`000cf290 00000000`76d5308e ntdll! ?? ::FNODOBFM::`string'+0x2bea0 
   00000000`000cf300 00000000`00000000 ntdll!LdrInitializeThunk+0xe

3. You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process.

4. You get errors like the below when trying to run sos commands.
(Note: You can get these errors if you don’t have symbols properly set up as well, so this is not the only reason for getting these errors)

0:000> !eeheap -gc 
Failed to load data access DLL, 0x80004005 
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)             2) the file mscordacwks.dll that matches your version of clr.dll is                 in the version directory             3) or, if you are debugging a dump file, verify that the file                 mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.             4) you are debugging on the same architecture as the dump file.                 For example, an IA64 dump file must be debugged on an IA64                 machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

What tools should you use?

The best way to capture a process dump in this scenario is to use 32-bit versions of tools like Debug Diag, or adplus+cdb (32 bit debugging tools for windows). You can install the 32-bit versions of these tools on a 64-bit system. In the next version of Debug Diag you will be able to capture 32-bit dumps with the 64-bit version of Debug Diag.

I have mentioned before that you can capture memory dumps with task manager on Vista+, but if you do this on a 64-bit machine you will get a 64-bit dump. You can however use the 32-bit task manager, located in C:\Windows\SysWOW64\taskmgr.exe to get 32-bit dumps. If you want to verify that you are running the 32-bit version of task manager, you can check that taskmgr.exe is listed with *32 in task manager itself.

Edit: Sorry Mark and David, I of course didn't mean to leave out the nice and very lightweight procdump tool from SysInternals which will capture 32-bit dumps of 32-bit processes even on 64-bit Windows. Thanks for reminding me:)
For some reason I thought that I had written about this tool in an earlier post but a quick search reminded me that this is still in my blog-todo bin.

For all other dump capturing tools, I would suggest that you read the documentation to see if there is a 32-bit version (like for procmon where you can pass in the /run32 parameter) or if they will capture 32-bit dumps of 32-bit processes.

Is it important to read the dump with the right “bit” debugger?

Yes, to read 32-bit dumps you should use 32-bit windbg (either on a 32-bit or 64-bit OS), and for 64-bit dumps, the machine that you debug the dumps on, needs to be a 64-bit machine, running 64-bit windbg.

Remember that you should also use the 64-bit versions of sos, psscor2, sosex and any other extensions that you might use, if you are debugging a 64-bit dump.

Have a good one,
/Tess

Help! !clrstack is not showing any method names

Tess1 — Wed, 25 Aug 2010 11:50:00 GMT

From time to time I get questions about missing method names in the sos !clrstack output, like this:

0:019> !clrstack 
OS Thread Id: 0x5d0 (19) 
ESP      EIP 
0f31f340 7c80bef7 [HelperMethodFrame_1OBJ: 0f31f340] 
0f31f398 1449167a 
0f31f3c4 144915e6 
0f31f3f4 10d81b48 
0f31f4b4 793e25cf 
0f31f4bc 79366b3d 
0f31f4d4 793e2734 
0f31f4e8 793e26ac 
0f31f678 79e7c0e3 [GCFrame: 0f31f678]

Jonathan Dickinson from SourceCode, just emailed me and told me that he had written an article about this and how to resolve it. You can read the whole article here: http://jonathan.dickinsons.co.za/blog/2010/08/windbg-stack-fix/
Thanks Jonathan for writing that article.

Solution:

As mentioned in the article there are a couple of things you need to get rid of the problem.

1. The correct version of MSCORDACWKS.dll

You can get this in two ways…

a. from the Microsoft public symbol server by setting the symbol path to srv*c:\mycache*http://msdl.microsoft.com/download/symbols

b. by copying it from the machine you got the dump from, it is located in the Framework\<version> directory. The mscordacwks.dll needs to be copied to your symbol path, and to differentiate them if you work with multiple versions you can name it mscordacwks_x86_x86_2.0.50727.42.dll for example. On my machine, because I am working with so many different versions i have a c:\dac\x86 and a c:\dac\x64 directory where I store these.

2. A compatible version of sos.dll

sos.dll can be found in your framework\<version> directory and needs to match the architecture and major version of the framework used in the dump. If there is too big of a difference between the minor versions of the framework on your machine and the machine where you got the dump, you can try copying the sos.dll from the machine that the dump was taken from. Or you can use psscor2.dll.

Have a good one,
Tess

Getting full user mode dumps automatically when your process crashes

Tess1 — Mon, 23 Aug 2010 09:53:00 GMT

I recently got a question about if it is possible to generate full user mode dumps automatically every time the process crashes (without having debug diag or adplus installed and monitoring your system). This would be useful for example if you are setting up a site, and you want to make sure you get dumps if it ever crashes so that you don’t have to repro the issue at that point, just to get data.

It turns out that starting with Windows Server 2008 and Vista + SP1, Windows Error Reporting (WER) will let you do just that. It even worked on my Win 7 machine.

http://msdn.microsoft.com/en-us/library/bb787181(VS.85).aspx

The documentation here says that “Applications that do their own custom crash reporting, including .NET applications, are not supported by this feature”.

It’s a bit hard to tell what this actually means but I think this is added here due to ambiguity about what constitutes a crash. For example if you have an exception in an ASP.NET application that is handled by the ASP.NET error handler (i.e. showing you the yellow ASP.NET error page) that is not a “crash” since ASP.NET handles the error, and the process just continues. However for example a StackOverflow, or any unhandled .net exceptions on finalizers or non-request threads, that cause the process to “terminate unexpectedly” will be caught by WER and produce a dump if you configure it to do so.

Setting this up

I set this up on my machine (per the article) by adding a registry key specific to w3wp.exe to monitor crashes for my ASP.NET applications

HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\w3wp.exe

Under this key I added 3 values (types and descriptions can be found in the MSDN article)

Value Comment

DumpFolder c:\mycrashdumps This is the location where the dumps will be generated, so the user that the W3WP.exe process is running under needs access to write to this location.

DumpCount 10 Maximum # of dumps in the folder, depending on the size of the process 10 dumps can take up a lot of disk space, so monitor carefully.

DumpType 2 To debug .net crashes a full user mode dump is often necessary.

I also had to disable the JIT debugger for this to work, under

HKLM\Software\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug

By renaming the Debugger value to Debugger-unused, and now it is producing crash dumps for stackoverflows and other unhandled exceptions.

A word of caution

As always, any modifications to the registry should only be done by trained professionals:) and only after carefully backing up the registry first etc. etc. I take no responsibility for any issues caused by such modifications.

Also, keep in mind that as mentioned before, the dumps can get pretty big if you have processes that produce a lot of memory so be wary about the amount of disk space that this can consume.

Have a good one,
Tess

The SQL Server Support Team in Stockholm is hiring

Tess1 — Wed, 23 Jun 2010 10:45:46 GMT

If you are a contractor in Stockholm, skilled in SQL, have a look at this post:

http://blogs.msdn.com/b/grahamk/archive/2010/06/22/the-sql-server-support-team-in-stockholm-is-hiring.aspx

Have a good one,

Tess

.NET Memory Leak: XslCompiledTransform and “leaked” dynamic assemblies

Tess1 — Wed, 05 May 2010 14:39:19 GMT

I have written before about high memory usage caused by improper usage of XmlSerializer objects both in a case study and in a debugging lab. The problem there was that every time you create a new XmlSerializer object with a non-default constructor, you generate a new dynamic assembly that contains the definition and methods for the serializer. Since assemblies can’t be unloaded from a process unless the application domain they are loaded in is unloaded memory will keep increasing if you create new XmlSerializer objects until eventually you end up with a System.OutOfMemoryException.

In the case of the XmlSerializer dynamic assemblies, they are quite easy to spot as you can look at meta-data (see previous posts) and figure out that they are XmlSerializer generated assemblies, and you can even see what type is being serialized.

This time I’m going to walk through a similar issue where unfortunately the dynamic assembly generator is a bit more difficult to spot.

Symptoms:

Any issue where you are leaking dynamic assemblies will have very similar symptoms and they go something like this:

1. The perfmon counter Process/Private Bytes keeps increasing (Mem Usage in task manager will keep increasing too for that matter)

2. .NET CLR Memory/#Bytes in all heaps may increase but unless you also have a .net object “memory leak” #Bytes in all heaps should not follow the Private Bytes pattern as the dynamic assemblies are not stored on the GC heaps. For a typical assembly leak problem, #Bytes in all heaps will be relatively low compared to Private Bytes.

3. .NET CLR Loading/Current assemblies keeps increasing throughout the lifetime of the process. Normally the number of loaded assemblies should flatten out once all components in the process are loaded so if Current assemblies keeps increasing, that is a pretty sure sign that you have an assembly leak.

Debugging:

Once memory usage for the process is fairly high, you can capture a memory dump of the process with any of your favorite tools (debug diag, adplus, task manager, procdmp etc.). Using debug diag for example, you just right-click the w3wp.exe process in the processes view and select the “Create Full Userdump” option.

In my case, Private Bytes was around 790 MB, #Bytes in all heaps was 28 MB and I had 1031 Current assemblies

If you only have a memory dump but no perfmon counters, you can get this data from the dump as well if you open it up in windbg.
Note: for some commands in this article you will need to load psscor.dll in windbg.

0:026> !address -summary

...


-------------------- State SUMMARY --------------------------

    TotSize (      KB)   Pct(Tots)  Usage

   3081a000 (  794728) : 18.95%   : MEM_COMMIT      <- very similar to private bytes

   a5eaf000 ( 2718396) : 64.81%   : MEM_FREE

   29927000 (  681116) : 16.24%   : MEM_RESERVE 

...

0:026> !eeheap -gc

Number of GC Heaps: 2

------------------------------
...

GC Heap Size  0x1b5396c(28,653,932)                <- #Bytes in all heaps

For current assemblies, you can’t easily get the total number of assemblies, but !dumpdomain –stat will tell you how many assemblies are loaded in each domain. Note however that there may be some overlap here as some assemblies will be listed in multiple domains. Also, the size of the dynamic assemblies may or may not show up here in the size assemblies column.

The cool thing about this output though is that you can easily identify the domain that is leaking assemblies. In this case it is probably not all that surprising that it happens to be my XSLMemoryLeak demo application where I am reproducing the problem:)

0:026> !dumpdomain -stat

    Domain     Num Assemblies     Size Assemblies     Name

7a3bd058                  0                   0     System Domain

7a3bc9a8                 27          85,086,720     Shared Domain

0019f280                  9          48,486,400     DefaultDomain

001d3038              1,031          85,100,032     /LM/w3svc/1/ROOT/XSLMemoryLeak-1-129175353121633629

What are these assemblies?

Btw, if you don’t care for a deep dive in potentially non-interesting windbg / psscor gory details, feel free to move right to the conclusion at the bottom of the post:)

Time for a new function in psscor called !dumpdynamicassemblies (or !dda for those of us who don’t want to end up with COBOL fingers).

This function has two purposes

a. list all the dynamic assemblies in the process, or
b. save all dynamic assemblies to disc using !dda –save, so that you can open them up in reflector or similar

Running this on the dump I can see that there is a total of 1002 dynamic assemblies in the dump and that all of them are in the XSLMemoryLeak domain. Each assembly seems to have 2 modules associated with it so there is a total of 2004 modules associated with these assemblies.

0:026> !dda

Domain: DefaultDomain

------------------- 

Domain: /LM/w3svc/1/ROOT/XSLMemoryLeak-1-129175353121633629

------------------- 


Assembly: 0x10514528 [] Dynamic Module: 0x02af8740 loaded at: 0x00000000 Size: 0x00000000(0)

Assembly: 0x10514528 [] Dynamic Module: 0x02af8ac0 loaded at: 0x00000000 Size: 0x00000000(0) 




Assembly: 0x10589558 [] Dynamic Module: 0x106122c4 loaded at: 0x00000000 Size: 0x00000000(0)

Assembly: 0x10589558 [] Dynamic Module: 0x10612644 loaded at: 0x00000000 Size: 0x00000000(0)


...






Assembly: 0x3472c168 [] Dynamic Module: 0x53402e8c loaded at: 0x00000000 Size: 0x00000000(0)

Assembly: 0x3472c168 [] Dynamic Module: 0x5340320c loaded at: 0x00000000 Size: 0x00000000(0)



Assembly: 0x347728b8 [] Dynamic Module: 0x53403630 loaded at: 0x00000000 Size: 0x00000000(0)

Assembly: 0x347728b8 [] Dynamic Module: 0x534039b0 loaded at: 0x00000000 Size: 0x00000000(0) 



Assembly: 0x347b9010 [] Dynamic Module: 0x53403dd4 loaded at: 0x00000000 Size: 0x00000000(0)

Assembly: 0x347b9010 [] Dynamic Module: 0x53404154 loaded at: 0x00000000 Size: 0x00000000(0) 

--------------------------------------

Total 1,002 Dynamic Assemblies, Total size: 0x0(0) bytes.

=======================================

Unfortunately, because the reported size is 0, saving them out will do me no good as they don’t have any meta-data so I will have to find another way to figure out what they are.

I was working on this issue with my colleague Doug and after doing some random “checking around the dump to see if we can find anything even remotely interesting” we noticed that on the heap we had exactly the same amount of System.Reflection.Emit.ModuleBuilderData objects as the number of modules listed above.

0:026> !dumpheap -stat -type System.Reflection.Emit*

Loading the heap objects into our cache.

------------------------------

Heap 0

total 38,045 objects

------------------------------

Heap 1

total 40,553 objects

------------------------------

total 78,598 objects
 
Statistics:

        MT    Count    TotalSize       Change Class Name

0x79305120    2,004       64,128        2,004 System.Reflection.Emit.AssemblyBuilder

0x79318c9c    2,004       88,176        2,004 System.Reflection.Emit.ModuleBuilderData

From this we were able to get some very important data…

First we dumped out the ModuleBuilderData objects:

0:026> !dumpheap -type System.Reflection.Emit.ModuleBuilderData

Loading the heap objects into our cache.

------------------------------

   Address         MT     Size

...

07c8bc98 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData 

07c8c6ec 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData 

07cc04d8 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData 

07cc0f2c 79318c9c       44    0 System.Reflection.Emit.ModuleBuilderData

And took a closer look at one of them

0:026> !do 07cc0f2c 

Name: System.Reflection.Emit.ModuleBuilderData

...

Fields:

      MT    Field   Offset                 Type VT     Attr    Value Name

793308ec  4002612        4        System.String  0 instance 02dc67fc m_strModuleName

793308ec  4002613        8        System.String  0 instance 02dc67fc m_strFileName

...

79318b90  4002617       10 ...mit.ModuleBuilder  0 instance 07cc0c94 m_module

79332b38  4002618       20         System.Int32  1 instance        0 m_tkFile

...

The m_strFileName proved to be a crucial piece of information telling us that the module was called System.Xml.Xsl.CompiledQuery so now we know that they are generated when loading XSL Transforms

0:026> !do 02dc67fc 

Name: System.String

MethodTable: 793308ec

EEClass: 790ed64c

Size: 74(0x4a) bytes

GC Generation: 2

String: System.Xml.Xsl.CompiledQuery

...

But how can we prove that these ModuleBuilderData objects are really related to the dynamic assemblies?

For this we need to look closer at the ModuleBuilderData objects m_module field and dump out this ModuleBuilder and later also its m_internalModuleBuilder

0:026> !do 07cc0c94 

Name: System.Reflection.Emit.ModuleBuilder

...

Fields:

      MT    Field   Offset                 Type VT     Attr    Value Name

...

79318b90  400260e       2c ...mit.ModuleBuilder  0 instance 07cc0c5c m_internalModuleBuilder

79305120  400260f       30 ...t.AssemblyBuilder  0 instance 07cc0418 m_assemblyBuilder

...

0:026> !do 07cc0c5c 
Name: System.Reflection.Emit.ModuleBuilder

...

Fields:

      MT    Field   Offset                 Type VT     Attr    Value Name

...

793331b4  4000d21       14        System.IntPtr  1 instance        0 m__pRefClass

793331b4  4000d22       18        System.IntPtr  1 instance 1396719956 m__pData

793331b4  4000d23       1c        System.IntPtr  1 instance 669415152 m__pInternalSymWriter

793331b4  4000d24       20        System.IntPtr  1 instance        0 m__pGlobals

...

Don’t ask me how you come up with this and the next part:) This is the fruit of some serious desperation, trying to find anything that fits :)

We then see that the m__pData member variable is an IntPtr so we can turn this into hex by running

0:026> ?0n1396719956 

Evaluate expression: 1396719956 = 53404154

And voilà, this just happens to be the address of one of the dynamic assembly modules, so Houston, we have a match…

Assembly: 0x347b9010 [] Dynamic Module: 0x53403dd4 loaded at: 0x00000000 Size: 0x00000000(0) 

Assembly: 0x347b9010 [] Dynamic Module: 0x53404154 loaded at: 0x00000000 Size: 0x00000000(0)

Conclusion:

Ok, so all that crazy debugging to find out that the dynamic assemblies were related to XSL Transforms.

As it turns out, if you create a new XslCompiledTransform(true) where true means enable debugging, a new dynamic assembly will be generated when you load the stylesheet.

On the other hand if this is set to false, or if you use new XslCompiledTransform() this will not happen, so just remember to turn it off when you go into production, not only for this but to stop emitting debug info at that point.

Have a good one,
Tess

New Visual Studio 2010 and .NET 4 debugging content

Tess1 — Wed, 28 Apr 2010 12:41:00 GMT

It’s been a pretty busy few weeks at work coupled with some vacation so I’ve been lagging a bit with the blogging, so sorry for bombarding the blog with new posts now:)

As you all know, Visual Studio 2010 just released and there is a lot of new content out there and I thought I’d mention a few in the debugging/troubleshooting space.

ScottGu wrote a post on the VS 2010 debugger improvements as part of his VS 2010 and .NET 4 post series where he talks about news with Data Tips and Breakpoints.
The Visual Studio Profiler Team Blog has a lot of posts worth reading about profiling in VS 2010, including Silverlight Profiling and guidance for profiling.
Steve released a new version of SOSex.dll with features that work for both 2.0 and 4.0
John Robbins released 3 new really good webcasts (unfortunately the encoding is a bit poor but the content makes up for it) on channel 9 about Advanced .NET Debugging in VSTS 2010

There is probably plenty more out there but that should at least get you started.

My presentation from TechDays Sweden 2010 was also recorded (in Swedish).

Last minute, I was chatting with one of my friends Tiberiu Covaci who specializes in parallel programming. Since I was given the graveyard slot (last slot of the conference) we decided to have some fun with it and present together figuring people would be half-asleep anyways after the party from the day before:)

The first few minutes are meant as a joke and a bit of a nudge to those who like to do death-by-powerpoint and spend 30 mins presenting their company and their careers :) If you stay past that you’ll see a demo from start to finish of a performance problem (with profiling, dump debugging in VS 2010 and dump debugging in windbg).

If you want to know who is who in the video, I’m the girl who forgot to comb her hair before the presentation, and Tibi is the funny guy with the big Romanian family:)

Laters,

Tess

Debugging a classic ReaderWriterLock deadlock with SOSex.dll

Tess1 — Tue, 27 Apr 2010 11:07:01 GMT

I was helping out on an issue the other day where the process would stall if they added enough users in their load tests. Btw, serious kudos to them for making load tests, so much nicer to work with a problem in test rather than when it is getting critical on a production machine.

We gathered some memory dumps with debug diag of the asp.net process (w3wp.exe) and found that most of the threads were waiting in this type of callstack:

NOTE: I have changed a lot of function names and code snippets since it is not relevant for the post.

0:071> !clrstack 

OS Thread Id: 0x21ac (71) 

ESP       EIP     
1c8ceb88 7c8285ec [HelperMethodFrame_1OBJ: 1c8ceb88] System.Threading.ReaderWriterLock.AcquireReaderLockInternal(Int32) 

1c8cebe4 1c880d72 DataStore.get_SomeItem() 

...

So they were sitting in DataStore.get_SomeItem, waiting to acquire a ReaderLock for some resource. A ReaderWriterLock is used to synchronize access to resources, where you want to allow multiple readers at the same time, or one writer.

To be honest, I can’t say that I see it used all that often, and now there is also a ReaderWriterLockSlim that is similar to the ReaderLock but designed to avoid some of the more common deadlocks.

Sosex from Steve’s Techspot has some really nice commands for reader writer locks that both display information about the RWlocks as well as detects deadlocks between them.

First we have !rwlock… this is similar to !syncblk for locks/monitors and here we can get some basic information about the readerwriterlocks in the process.

0:174> !rwlock

Address    ReaderCount   WaitingReaderCount   WriterThread   WaitingWriterCount

---------------------------------------------------------------------------------------
…
0e9167d0             0                    0             50                    2

0e9167fc             2                   45              0                    1

We have two interesting locks here (0e9167d0, and 0e9167fc)

For 0e9167d0 (I’ll call this Lock1 in the future), thread 50 owns the WriterLock so no-one else can acquire neither a ReaderLock nor WriterLock for this, and we have 2 threads waiting to acquire the WriterLock on this.

For 0e9167fc (I’ll call this Lock2 in the future), we have two threads that own ReaderLocks on this, we have 1 thread waiting to acquire a WriterLock on it, and 45 other threads waiting to acquire a ReaderLock on it. The thread waiting to acquire the WriterLock is probably first in queue, but until the current readers give up their locks, the WriterLock can’t be acquired.

(I will get back to these thread numbers in a moment because it gets pretty tricky)

We can also dig into a particular lock by running !rwlock <lock address>

0:068> !rwlock 0e9167d0      (<- Lock1)

WriterThread:           50

WriterLevel:            1

WaitingWriterCount:     2

WriterEvent:            e54

WaitingWriterThreadIds: 14,47

ReaderCount:            0

CurrentReaderThreadIds: None

WaitingReaderCount:     0

ReaderEvent:            0

WaitingReaderThreadIds: None

0:174> !rwlock 0e9167fc       (<-Lock2)    
WriterThread:           0

WriterLevel:            0

WaitingWriterCount:     1

WriterEvent:            e00

WaitingWriterThreadIds: 50

ReaderCount:            2

CurrentReaderThreadIds: 14,47

WaitingReaderCount:     45

ReaderEvent:            e7c

WaitingReaderThreadIds: 54,57,58,61,62,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111,114,115,118,119,122,123,126,127,130,131,134,137,140,143,146,149

With this info, we can now gather that threads 14 and 47 own readerlocks for Lock2, but they won’t give them up until they can get WriterLocks for Lock1. Thread 50 on the other hand, has a WriterLock on Lock1 and wont give it up until it can get the WriterLock on Lock2, hence we have a deadlock.

The good thing here is that if you don’t care to do this investigation on your own, SOSEX has a function that will do it for you called !dlk (deadlock).

0:174> !dlk

Examining SyncBlocks...

Scanning for ReaderWriterLocks...

Scanning for lock holders on ReaderWriterLocks...

Scanning for threads waiting on SyncBlocks...

Scanning for threads waiting on ReaderWriterLocks...


Deadlock detected:

CLR thread 14 holds a Reader lock on ReaderWriterLock 0e9167fc

             is waiting for a Writer lock on ReaderWriterLock 0e9167d0

CLR thread 50 holds a Writer lock on ReaderWriterLock 0e9167d0

             is waiting for a Writer lock on ReaderWriterLock 0e9167fc

CLR Thread 14 is waiting at DataStore.InitializeDataIfNotInitialized()(+0x0 IL)(+0x0 Native)

CLR Thread 50 is waiting at System.Threading.ReaderWriterLock.UpgradeToWriterLock(Int32)(+0x0 IL)(+0x0 Native)
 

Deadlock detected:

CLR thread 47 holds a Reader lock on ReaderWriterLock 0e9167fc

             is waiting for a Writer lock on ReaderWriterLock 0e9167d0

CLR thread 50 holds a Writer lock on ReaderWriterLock 0e9167d0

             is waiting for a Writer lock on ReaderWriterLock 0e9167fc

CLR Thread 47 is waiting at DataStore.InitializeDataIfNotInitialized()(+0x0 IL)(+0x0 Native)

CLR Thread 50 is waiting at System.Threading.ReaderWriterLock.UpgradeToWriterLock(Int32)(+0x0 IL)(+0x0 Native) 

2 deadlocks detected.

Explanation of the Thread IDs coming from the SOSEX RWLock methods

So this is all nice and good, but what are these thread IDs? is it the logical thread ID (1st column in !threads), the CLR ID (2nd column in !threads) or the OSID (OS Thread ID)? Turns out it is neither… or rather it is actually the CLR ID in decimal form:) Steve mentioned he might change this later to be the hex value so stay tuned for new updates of SOSEX for that and other functionality.

0:054> !threads

ThreadCount: 154

UnstartedThread: 1

BackgroundThread: 103

PendingThread: 0

DeadThread: 1

Hosted Runtime: no

                                      PreEmptive   GC Alloc           Lock

       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception

  16    1 1474 0010c0a8   3808220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Worker)

...

  32    d 2520 1abd0948   200b020 Enabled  00000000:00000000 0013fa58     0 MTA

  11    e 1fcc 1abdc9e8   a80a220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Completion Port)

  33    f 1224 1abe5c10   380b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Worker)

...

  37   13 1cc8 1ac22618   380b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Worker)

  38   14 23e8 1ac229e8   200b020 Enabled  00000000:00000000 0013fa58     0 MTA

  39   15 2230 1ac22db8   380b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Worker)

...

  63   2d 23e0 1ab934c0   380b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Worker)

  64   2e 19ac 1ac3d190   200b020 Enabled  00000000:00000000 0013fa58     0 MTA

  65   2f 1644 1aba8f40   a80b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Completion Port)

  66   30 10f4 1abaaaa8   380b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Worker)

  67   31 121c 1ac47418   200b020 Enabled  00000000:00000000 0013fa58     0 MTA

  68   32 1b00 1abfdd70   a80b220 Enabled  00000000:00000000 0013fa58     1 MTA (Threadpool Completion Port)

  69   33 2198 1ac55358   200b220 Enabled  06e140e4:06e150b4 0013fa58     0 MTA

In our case, the threads we were interested in were 14 (0xe) = 11, 47 (0x2f) = 65 and 50 (0x32) = 68

If we take a look at these threads (11 and 68), the stacks look like this:

0:011> !clrstack

OS Thread Id: 0x1fcc (11)

ESP       EIP     
01c7f250 7c8285ec [HelperMethodFrame_1OBJ: 01c7f250] System.Threading.ReaderWriterLock.AcquireWriterLockInternal(Int32)

01c7f2ac 1c880e07 DataStore.InitializeDataIfNotInitialized()

01c7f2e4 1c880d78 DataStore.get_SomeItem()

... 


0:068> !clrstack

OS Thread Id: 0x1b00 (68)

ESP       EIP     
1c71ed0c 7c8285ec [GCFrame: 1c71ed0c] 

1c71ed28 7c8285ec [HelperMethodFrame_1OBJ: 1c71ed28] System.Threading.ReaderWriterLock.FCallUpgradeToWriterLock(System.Threading.LockCookie ByRef, Int32)

1c71ed88 792af05d System.Threading.ReaderWriterLock.UpgradeToWriterLock(Int32)

1c71edac 1c880e19 DataStore.InitializeDataIfNotInitialized()

1c71ede4 1c880d78 DataStore.get_SomeItem()

...

And the code for the get_SomeItem, and InitializeDataIfNotInitialized methods look something like this:

public static SomeItem get_SomeItem()
{
    ...
    DataLock.AcquireReaderLock(timeOut);
    try
    {
        InitializeDataIfNotInitialized();
        ...
    }
    finally
    {
        DataLock.ReleaseReaderLock();
    }
    ...
}

private static void InitializeDataIfNotInitialized()
{
    if (DataIsNotInitialized)
    {
        try
        {
            InitLock.AcquireWriterLock(timeOut);
            LockCookie lockCookie = DataLock.UpgradeToWriterLock(timeOut);
            try
            {
                ...
            }
            finally
            {
                DataLock.DowngradeFromWriterLock(ref lockCookie);
            }
        }
        finally
        {
            InitLock.ReleaseWriterLock();
        }
    }
}

So what happened here was that thread 11 and 68 both grabbed the DataLock ReaderLock in get_SomeItem.

Then 68 grabbed the WriterLock for InitLock in InitializeDataIfNotInitialized, and now it is trying to upgrade the DataLock to a WriterLock but can’t do so since thread 11 has the ReaderLock on it.

Thread 11 on the other hand now needs to grab the WriterLock on the InitLock but can’t do so because 68 owns it.

Conclusion

This is a pretty typical deadlock situation, and the solution is to re-design the initialization methods so that this can never happen, but hopefully the walkthrough has given some insight into how you can debug these types of issues with SOSEX if you do run into them.

Have a good one,

Tess

IIS and PAE

Tess1 — Tue, 27 Apr 2010 08:26:00 GMT

I recently got a question by one of my customers about PAE and IIS that I thought I’d share the answer to.

Their environment looked something like this:

32bit OS (Windows 2003)
IIS 6 with multiple application pools, where each app pool hosts a number of applications
~20 GB RAM

They were having problems with out of memory exceptions and also with paging under high load. They knew a lot about their application and memory usage in general, so they knew what they were using the memory for and why they got the OOMs, in this case, memory usage was relatively high per application (intentionally), but it remained stable so no leak. With this in mind, we discussed solutions to the OOMs like splitting up the applications further etc. to avoid this. Now their question was:

Is IIS 6 PAE aware, i.e. will we be using the full 20 GB of RAM or only 4 GB?

I didn’t really know much about this so I consulted with one of our IIS escalation engineers Andreas to get the scoop.

The answer (which Joe Stagner points out in this forum thread) is that IIS 6 is not PAE aware, and since they are running on W2k3 x86 which will not manage memory above 4GB they will effectively only use 4 GB, leaving 16 GB unused. (Unless of course they run other PAE aware apps on it, SQL server for example I believe is PAE aware).

Update: As per Dave's comments below the statement above seems to be incorrect, if you have multiple w3wp.exe processes they can have a total workingset bigger than 4GB (combined for all w3wp.exe processes) even on 32-bit if /PAE is on... So in retrospect, I suspect that the above statement is talking about RAM used for HTTP.SYS etc. This should teach me to not blog about things that I'm only hearing 2nd hand:) even still, it has been a good learning experience for me... and thanks to blogging about it, I now know more... Thanks Dave for commenting on it.

On the other hand, a solution that would solve both the RAM question and the OOM issue is to move to a 64bit OS (preferably 2008 R2 because of improvements in the kernel, memory management and IO management).

Since it is a 64bit OS it can take advantage of all of the RAM, and even if you run 32bit IIS, you can then address 4 GB of Virtual Memory rather than 2 GB which means that there is more room for memory usage before an OOM (in this case since memory usage was stable but relatively high, that is exactly what we want). In fact, as Bernard Cheah also mentions in the same thread, there is no reason to move to 64bit IIS unless you need to, i.e. unless you need to use even more memory than the 4GB of virtual bytes that the 32bit process can address.

Have a good one,
Tess

New debugger extension for .NET (PSSCOR2)

Tess1 — Tue, 30 Mar 2010 14:31:18 GMT

Tom just blogged about a new debugger extension called PSSCor2, which is a superset to the SOS.dll extension that ships with the .net framework.

PSSCor2.dll has been around for quite some time at Microsoft, and pretty much everyone debugging .net code with windbg within Microsoft is using this since it contains all the goodness of sos.dll plus a lot of special methods for asp.net and other technologies running on top of the .net framework. The news now is that after a lot of hard work by Tom and Jon Langdon in the CLR team, it is finally released publicly, YAY!!!

You can download it here and Tom promised to write some posts about the new commands so you might want to follow that… I will likely do the same later on as well…

Just to name two of the commands in there that I use a lot:

!dae (Dumps out all the exceptions on the .net heaps with callstacks)

!aspxpages (Dumps out all the asp.net pages that are currently executing or have been executing recently, along with the thread they are running on, what the timeout is and how long they have been running… excellent stuff)

Have a good one,

Tess

Videos of developer sessions at Oredev (including mine on .net debugging)

Tess1 — Tue, 09 Mar 2010 08:30:15 GMT

Just a quick post to share a link to last years videos for the Oredev developer conference

http://oredev.org/video

Here is my .net debugging presentation where I show off some windbg + new VS.NET features and apart from all the good developer content, you might also want to check out Scott Hanselmans session on Information Overload and managing the workflow with some tips and tricks on how he manages twitter/facebook/email etc.

Thank you to the organizers of Oredev for posting these,

Tess

Tool for generating DGML graphs showing why your object can’t be collected (VisualGCRoot)

Tess1 — Wed, 03 Mar 2010 13:57:50 GMT

Visual Studio.NET 2010 has a new feature that allows you to create nice directed graphs with a markup language called DGML (Directed Graph Markup Language).

Visualizing object graphs with DGML

Chris Lovett wrote a tool that takes GCRoot output and transforms it to a DGML document so that you can see the roots of an object in a more visual way, and finally Mohamed Mahmoud went one step further and made a debugger extension that does this for you directly.

The debugger extension is called VGCRoot and you can get the extension, and some instructions and demos for it here.

The VGCRoot extensions

If you are looking at a memory dump in windbg and try to figure out why your object is still sticking around, collection after collection, you can use the sos extension to run a command called !GCRoot which will give you a textual representation of how your object is rooted.

In other words, it will give you several “stacks” linking your object all the way up to one or more “roots”, where the roots may be threads, static objects, reference counted objects and until you break the chain between the root and your object your object can’t be garbage collected.

A typical root chain looks something like the one below taken from one of my memory investigation posts

0:000> !gcroot 00000000800dfb98 
... DOMAIN(000000000017E210):HANDLE(Strong):4b41308:Root: 00000000c0020990(System.Threading._TimerCallback)-> 
00000000c0020908(System.Threading.TimerCallback)-> 
00000000c001b530(System.Web.Caching.CacheExpires)-> 
00000000c001b568(System.Object[])-> 
00000000c001bef8(System.Web.Caching.ExpiresBucket)-> 
00000001008fbf40(System.Web.Caching.ExpiresPage[])-> 
00000001008fbff8(System.Web.Caching.ExpiresEntry[])-> 
000000010001ebd8(System.Web.Caching.CacheEntry)-> 
000000010001eb98(System.Web.SessionState.InProcSessionState)-> 
00000000800dd340(System.Web.SessionState.SessionStateItemCollection)-> 
00000000800dd3f0(System.Collections.Hashtable)-> 
00000000802fe7e0(System.Collections.Hashtable+bucket[])-> 
00000000800e09e8(System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry)-> 
00000000800dfb98(System.Xml.XmlDocument)

In this case we have an XmlDocument that can’t be garbage collected because it is stored in session scope so until it is removed from session scope (breaking the link between XmlDocument and SessionStateItemCollection) it can’t be garbage collected.

With !VGcRoot we can now get all the roots in a DGML file that we can open up in Visual Studio .NET 2010, making this much easier to digest.

You run it from windbg using !vgcroot /do /fields <objectaddress> [<DGML File path>] and the resulting graph (in this case for a Form object) looks like this.

Here we can see for example that the Form is linked by a UserPreferenceChangedEventHandler through the _target member variable, in other words, one reason that the Form can’t be garbage collected is because it is handling a UserPreferenceChangedEvent.

I will probably be showing this off more in context in future memory investigation posts but just wanted to show off the extension so that you can start playing around with it. Be sure to check out his post to see other stuff you can do with the graph once you have it generated.

Have a good one,
(and thanks to Mohamed for writing the extension)

Tess

New commands in SOS for .NET 4.0 Part 1

Tess1 — Mon, 01 Mar 2010 15:21:00 GMT

My friend and fellow debugger Brian at http://Kodehoved.dk recently wrote a couple of posts on news with sos for .NET framework 4.0 (in Danish)

New SOS Commands in .NET 4
More debugging news in CLR 4

Since Danish, although a beautiful language is probably foreign to most of you I figured I’d write a summary of the new commands in English and add some comments of my own.

Loading sos for .NET 4.0

As in 2.0 you will find sos.dll in the framework directory so you can load it in windbg or cdb using the full path

.load C:\Windows\Microsoft.NET\Framework\v4.0.30128\sos.dll

You can also use the short hand .loadby method but since the name of the core dll has changed in 4.0, you will no longer load it using .loadby sos mscorwks, instead you can now load it using

.loadby sos clr

SOS now takes advantage of windbg’s DML feature making it easier to debug

DML stands for Debugger Markup Language and allows commands to emit hyper links in the command output. For example, in the !threads command output there are DML links for the OSID column, the State column and the Domain column.

Clicking on the first link in the OSID column (below) for example would execute the command ~~[ff4]s which would set the context to that thread. Clicking on a link in the state column would show you the !ThreadState output so you can see exactly what the state 8220 means etc. This is very handy as you don’t have to remember all the various commands.

0:042> !threads

ThreadCount:      25

UnstartedThread:  0

BackgroundThread: 17
PendingThread:    0

DeadThread:       8

Hosted Runtime:   no

                                   PreEmptive   GC Alloc                Lock

       ID  OSID ThreadOBJ    State GC           Context       Domain   Count APT Exception

  11    1   ff4 012923d0      8220 Enabled  00000000:00000000 01289450     0 Ukn

  18    2  1dbc 0129a3a0      b220 Enabled  00000000:00000000 01289450     0 MTA (Finalizer)

  20    4  114c 012b1650   8008220 Enabled  00000000:00000000 01289450     0 MTA (Threadpool Completion Port)

…

In order to turn on DML in your debugger you can run the command .prefer_dml 1 or if you only want to enable it for a specific command you can run !threads /D for example.

DML is used in many places in sos… in !dumpheap –stat you get links for each type that executed !dumpheap –mt <methodtable> so that you can see all the objects of that type, and the !dumpobj output emits links so that you can easily click and look at the member variables in more detail.

New commands

To find a list of all the commands sos has to offer you can run !help in windbg (if sos is the last extension that was loaded) and to get help (including examples) for a specific command you can run !help <command>, for example !help DumpObj.

Understanding why an object is not garbaqe collected (!GCWhere, !FindRoots !HandleCLRN)

Previously we only had the !GCRoot command which was nice for strong references but didn’t work well when your object was not “really” rooted.

Running !GCRoot <object address> will tell us the root chain of any rooted object, in other words, if your object is not being collected because it is linked, directly or indirectly from a thread, a static object, a ref counted object etc. it will give you this link chain so that you can determine which links you need to break in order to make your object collectable.

Sometimes however you may be troubleshooting an issue where your object is not necessarily rooted but it still doesn’t get collected. The answer to why it is not collected is either that the generation your object is in has yet to be collected, or it is rooted by an object in a higher generation, and that it is waiting for this object to be collected.

If you are live debugging your application with Windbg you can now find out why your object is not collected by going through this sequence.

1. Find out which generation your object is in by running !GCWhere

0:030> !GCWhere 057cca48

Address  Gen Heap segment  begin    allocated size

057cca48   2      1     056f0000   056f0038   065c8a04    0x40(64)

!GCWhere tells you what generation your object currently belongs to, as well as some info about the segment in which it is allocated. In this case our object is in Gen 2 on Heap 1.

2. Turn on CLR GC Notifications

Next we run !HandleCLRN to tell the debugger to stop on GC CLR Notifications

3. Enable break on GC

!FindRoots –gen 2 tells the debugger to stop on the next Gen 2 collection, so we issue this command and hit g (for go) to allow the debugger to continue executing

0:030> !FindRoots -gen 2   
0:030> g

4. Find the roots for your object

When the garbage collection occurs the following is displayed in windbg and the debugger stops

(21d0.1a74): CLR notification exception - code e0444143 (first chance)
CLR notification: GC - Performing a gen 2 collection. Determined surviving objects...

Now we can run !FindRoots <object address> to determine what is holding on to the object. This might be a root graph similar to what you see with !gcroot if the object is strongly rooted.

It might also be something like:
0:016> !FindRoots 05e54f4c

Object 05e54f4c will survive this collection:

    gen(0x5e54f4c) = 2 > 1 = condemned generation.
… if you do findroots for an object in a higher generation than the one you stopped on… or something like
0:002> !findroots 06808094 

older generations::Root:  068012f8(AAA.Test+a)->

  06808094(AAA.Test+b)
… if your object is being held on to by an object in an older generation

There are plenty of other new commands as well but to avoid making this post all too long, I will continue with the other commands in my next post instead.

Until then,

Tess

It’s time for TechDays Sweden again…

Tess1 — Wed, 17 Feb 2010 10:24:47 GMT

March 23rd and 24th I will be presenting at TechDays Sweden in Örebro.

The agenda for my session is not set in stone yet but the plan is to talk about debugging .NET production type issues as usual (Performance, Memory, Crashes, Exceptions etc.) since that is what I know best.

I will be talking about them both from a developers perspective, using some of the nice features in Visual Studio .NET 2010 like dump debugging, profiling, and perhaps IntelliTrace as well as discuss options for debugging issues that occur in environments where you don’t have Visual Studio, concentrating on WinDbg/Debug Diag and other tools that lend themselves well for production environment debugging.

Hope to see you there,

Tess

System.Configuration.ConfigurationErrosException “Unrecognized element ‘setting’” and other unrecognized elements

Tess1 — Tue, 16 Feb 2010 14:47:41 GMT

If you have a .NET configuration file with multiple <setting> elements, you may get the following exception:

Unhandled Exception: System.Configuration.ConfigurationErrorsException: Unrecognized element 'setting'. (F:\MyApp\bin\Debug\MyApp.exe.config line 11)  at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult) 

   at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject) 

   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) 

   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) 

   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) 

   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey) 

   at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName) 

   at System.Configuration.ConfigurationManager.GetSection(String sectionName) 

   at System.Configuration.ClientSettingsStore.ReadSettings(String sectionName, Boolean isUserScoped) 

   at System.Configuration.LocalFileSettingsProvider.GetPropertyValues(SettingsContext context, SettingsPropertyCollection properties) 

   at System.Configuration.SettingsBase.GetPropertiesFromProvider(SettingsProvider provider) 

...

This happens if you have a configuration section with multiple <setting> elements like in this example

<setting name="Setting1" serializeAs="String"><value>Setting1</value></setting>
<setting name="Setting2" serializeAs="String"><value>Setting2</value></setting>

and you keep the whole <setting> element on one line.

To get rid of this error, split the <setting> element on multiple lines, or add any kind of whitespace after the </value> tag like this:

<setting name="Setting1" serializeAs="String">
<value>Setting1</value>
</setting>

Similar Unrecognized element errors will occur if you use an expanded <clear></clear> tag instead of <clear/>. In that case you may get an Unrecognized element ‘add’ for example if <add> is the first element that appears after the expanded <clear></clear> tag. The resolution here is to simply change <clear></clear> to its short form <clear/>

We have received a few reports about this issue

https://connect.microsoft.com/VisualStudio/feedback/details/508146/valid-rolemanager-app-config-setting-fails-to-parse

http://connect.microsoft.com/VisualStudio/feedback/details/352591/configurationerrorsexception-thrown-on-service-start-when-no-whitespace-between-value-and-setting

but because there are very simple workarounds to both issues, and thus the potential problems that can arise from changing the code are bigger than the gain from fixing them for the clients reporting the issue, there hasn’t been a fix for this as of yet.

If for some reason you can not use the workarounds given above you can either vote up the connect bug to give it higher priority for future releases or contact support to help find a solution that fits your needs.

Laters,

Tess

MSDN Radio (Swedish) - Tess Corner

Tess1 — Fri, 15 Jan 2010 12:04:48 GMT

MSDN Radio is a Swedish MSDN podcast that is Dag König puts together every other week. The show contains a number of different segments and is usually pretty fun to listen to.

Starting this week, and hopefully every other week or so during the spring, I will have a 5-15 min segment on the show called Tess Corner where I’ll be talking about current ASP.NET and other .net issues or something special I’m working on.

In the first episode posted here http://channel9.msdn.com/posts/buzzfrog/MSDN-Radio-11-Jan--Tess-Corner/ I talk about debugging .NET apps with IntelliTrace in Visual Studio.NET 2010.

Check out the other MSDN episodes on http://channel9.msdn.com/tags/MSDN+Radio/ (also in Swedish) and if you listen to the show and want to hear about something special, feel free to post me a comment.

Have a good one
Tess

Debugging Native memory leaks with Debug Diag 1.1

Tess1 — Thu, 14 Jan 2010 13:21:02 GMT

I often get questions about debugging native memory leaks. In other words, cases where you have high memory usage in your application but you can see that .net memory usage is not all that high.

Recently I was helping out on such a case and this post details both generally how you go about troubleshooting these issues as well as what troubleshooting steps we went through in this particular case.

Essentially you would go through these steps to troubleshoot a native memory leak:

1. Find out if your memory usage / leak is mostly .net memory or native memory
2. If it is native memory that is “leaking”, use a tool like debug diag 1.1. to track allocations and deallocations
3. Look at the main allocators and the stacks associated with the biggest chunks of allocations to determine who is making the allocations and why the memory is not released.

1. Find out if your memory usage / leak is mostly .net memory or native memory

There are various ways to find out if your memory usage is mostly .net or native.

Using perfmon to determine where your memory is going

The best way is probably to use performance monitor and collect a log over the lifetime of the process that is leaking with at least the following counters for the specific process (note that _total or _global counters are not very helpful here)

.NET CLR Memory/#Bytes in all heaps
Process/Private Bytes
Process/Virtual Bytes

Simplified, Virtual Bytes is the memory that the process reserves, Private Bytes is how much of that the memory that the process actually uses and #Bytes in all Heaps is how much of the Private Bytes goes to memory stored on the .NET GC heap.

This is not entirely accurate if you want to be a nitpicker about what Private Bytes really means, but it is accurate enough for the purposes of troubleshooting a memory issue.

You can create such a logfile by following these steps

1. Open perfmon (Performance under Administrative Tools)
2. Under Performance Logs and Alerts, right-click Counter Logs and choose New Log Settings, give it a name and click ok.
3. Click “Add Counters” and choose the counters above. The best way to make sure you get the instance counters, rather than _Total and _Global is to choose All instances
4. Once you have added the counters, fill in the name of an administrator in the Run As: box and fill out the password. (This is important if you are logging for w3wp.exe for example as some counters are only available when you run the log as an admin)
5. The counter log will now be running, collecting data (shows up as green in the counter logs view) but you can start and stop it at any time.

Once you have this performance log you can now compare these values and specifically compare the trend and amounts for Private Bytes and #Bytes in all heaps to see if your memory leak is native (non .net) or .net related.

If your memory growth/usage is mostly .net, have a look through some of the other posts on debugging memory issues to check into how to troubleshoot them.

If however # Bytes in all heaps is pretty moderate you should continue reading from step 2 and on.

Getting memory usage info from a dump file

If you got a memory dump of the process but “forgot” to create a perfmon log with the counters described above, you can find out from the memory dump where the memory is used.

Open the memory dump in Windbg (File/Open Crash Dump) and run the command !address –summary

Note: The output might differ with different versions of windbg

0:000> !address -summary 
-------------------- Usage SUMMARY --------------------------     
TotSize   (      KB)   Pct(Tots) Pct(Busy)   Usage    
1bf2f000  (  457916) : 21.84%    23.18%    : RegionUsageIsVAD     
772b000   (  122028) : 05.82%    00.00%    : RegionUsageFree     
5881000   (   90628) : 04.32%    04.59%    : RegionUsageImage     
22bc000   (   35568) : 01.70%    01.80%    : RegionUsageStack       
8b000     (     556) : 00.03%    00.03%    : RegionUsageTeb    
54dcb000  ( 1390380) : 66.30%    70.40%    : RegionUsageHeap           
0         (       0) : 00.00%    00.00%    : RegionUsagePageHeap        
1000      (       4) : 00.00%    00.00%    : RegionUsagePeb        
1000      (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs        
1000      (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock        
   Tot: 7fff0000 (2097088 KB) Busy: 788c5000 (1975060 KB) 
-------------------- Type SUMMARY --------------------------     
TotSize   (      KB)   Pct(Tots)  Usage     
772b000   (  122028) : 05.82%   : <free>     
5ee5000   (   97172) : 04.63%   : MEM_IMAGE      
5fb000    (    6124) : 00.29%   : MEM_MAPPED    
723e5000  ( 1871764) : 89.26%   : MEM_PRIVATE -------------------- State SUMMARY --------------------------

    TotSize (      KB)   Pct(Tots)  Usage

   5a1e4000 ( 1476496) : 70.41%   : MEM_COMMIT

    772b000 (  122028) : 05.82%   : MEM_FREE

   1e6e1000 (  498564) : 23.77%   : MEM_RESERVE 

Largest free region: Base 06b10000 - Size 02170000 (34240 KB)

From this we can deduce a couple of things:

MEM_COMMIT is the committed memory, this should be fairly close to Private Bytes. In this particular case this process was using almost 1.4 GB of memory

MEM_RESERVE + MEM_COMMIT is the total reserved bytes, so this is Virtual Bytes in Perfmon. Here we can see that the process has reserved about 500 MB more than it committed, i.e. 500 MB more than it actually used. This is fairly normal for a big process as for example it will reserve chunks of memory for the .NET GC heap that it might not use all of, and there is some overhead reservations for dlls etc. as well as other types of reservations.

From the Usage SUMMARY section we can see where the reservations were made. RegionUsageIsVAD for example is allocations made with VirtualAlloc (.NET GC Heap for example goes here). RegionUsageIsImage is memory used for dlls and executables. RegionUsageStack is stack space for the threads, and RegionUsageHeap is for allocations on the NT Heaps. In this particular case we can see that most of the reservations (66.3%) and the commits (70.4%) are done on the NT heaps. This is typical for a native leak. If it was high .NET memory usage we would see a lot of the memory used for RegionUsageIsVAD.

If you want to learn more about the !address –summary output, check out the help in windbg for the command or check the lab 3 review where I have described some of this in more detail.

We can also check out exactly how much memory is used for .net objects by loading up sos in windbg (.loadby sos mscorwks) and then run !eeheap –gc. I have truncated the output below to just show the important parts

0:000> !eeheap -gc
Number of GC Heaps: 8
------------------------------
…
------------------------------
GC Heap Size   0x84680(542,336)

In this process we were using an incredibly small amount of .NET object, ~540k, so we can say for sure here that the leak is native.

2. If it is native memory that is “leaking”, use a tool like debug diag 1.1. to track allocations and deallocations

The next step in the process is to collect data about the leak and for this we can use Debug Diag 1.1.

The download page gives the instructions needed, in order to start monitoring for leaks

To debug memory and handle usage, use one of the following:

1. Create a leak rule against the process in question. The leak monitoring feature will track memory allocations inside the process. Tracking is implemented by injecting a DLL (leaktrack.dll) into the specified process and monitoring memory allocations over time. When configuring a memory and handle leak rule, you can specify memory dump generation based on time or memory usage.
2. Using the “processes” view, right-click the process in question and select the “monitor for leaks” option. When the process has grown to the suspected problem size, manually dump the process by right-clicking on the same process in the processes view and choosing the “Create Full Userdump” option.

A few notes about this: Leak tracking will only record leaks that occur while leak tracking is turned on. I know this may sound obvious but I am mentioning this because we often get leak track dumps that span only over a small period of time and does not include the allocations, because people have started leaktracking when memory was already high.

The other thing that is important to note is that by default, leak tracking will start recording stacks 15 minutes after leak tracking begins. This is to allow for an app to do the initial allocations like build up the cache, start up the form etc. before starting to monitor, so that the tracking only contains the relevant leak info.

I’ve found that the best way to get all the info i need is to start up the application and let it initialize things, and then run leak tracking with the checkbox to record call stacks immediately checked. (This is under tools/options in debug diag)

When you start Debug Diag you will be greeted with the Select Rule Type and here you choose Memory and Handle Leak and follow the steps above.

You can choose to have debug diag automatically generate dumps when the process reaches a certain size, but the easiest is probably to just choose the Memory and Handle Leak rule, and click next next finish, and then right-click on the process in the processes tab once the leak has manifested itself to get a memory dump (as per the original instructions.

3. Look at the main allocators and the stacks associated with the biggest chunks of allocations to determine who is making the allocations and why the memory is not released.

Ok, so now we have the memory dump containing the information about the leak. The cool thing here is that Debug Diag has an analysis feature that analyzes the dump for you.

Under the Advanced Analysis tab in Debug Diag, select the MemoryAnalysis.asp script and click add data files to add your memory dump, then click on “Start Analysis”.

This will generate a report that will pop up in your browser showing you information about all the allocations etc.

The first part of the report is an analysis summary showing a list of the components that allocated the most memory (and didn’t release them). In this case rsaenh.dll allocated 703 MB so we probably don’t even have to worry about the advapi32.dll allocations.

If you happen to debug a dump where most of the memory is .net you will probably see mscorwks.dll as one of the top allocators. This doesn’t mean that mscorwks.dll is leaking, it just means that the GC allocated a lot of segments that it didn’t release because the .net objects are still in use. If that is the case you need to follow some of the other articles on how to debug .net memory leaks instead.

If you don’t see any big allocators in this list, you probably didn’t leak track for long enough. You can see how long tracking has been turned on, and how much has been allocated during that time by following the Leak Analysis Report a bit further down in the report output.

Going back to the issue at hand, we saw that most allocations were made by rsaenh!ContAlloc+17. Does that mean that this method leaks? Not neccessarily… we will have to look at the call stacks leading up to the allocations to see why it doesn’t release the memory.

If we follow the link for rsaenh!ContAlloc+17 we will see some statistics about the allocations

So from here we find that most of the problematic allocations are the 422 byte allocations since a total of 689 MB was allocated in 422 byte chunks.

From here we can then drop down and look at the call stacks for the 422 byte allocations and find

Call stack sample 4

Address           0x0e309518
Allocation Time   00:05:00 since tracking started
Allocation Size   422 Bytes

Function                 Source   Destination
rsaenh!ContAlloc+17               ntdll!RtlAllocateHeap
rsaenh!MakeNewKey+6c              rsaenh!ContAlloc
rsaenh!CPDeriveKey+3b1            rsaenh!MakeNewKey
advapi32!CryptDeriveKey+af
SomeCryptoComponent+8328          advapi32!CryptDeriveKey
SomeCryptoComponent+4ef5
oleaut32!DispCallFunc+16a
oleaut32!CTypeInfo2::Invoke+234   oleaut32!DispCallFunc
. . .

Note: I have obfuscated the name of the 3rd party component here.

This call stack tells us that a method in the component “SomeCryptoComponent.dll” is calling advapi32!CryptDeriveKey and this is what eventually leads to the allocations we are seeing. We can’t see the method in SomeCryptoComponent.dll because we don’t have the symbols (pdb files) for it, but if we did we could add them to the symbol path under tools/options and rerun the analysis to get a better stack.

Either way, whenever you call CryptDeriveKey, you need to pair it up with a CryptDestroyKey when you are done with the Key in order to release memory. For some reason this is not happening here and thus the memory used for the keys is left allocated and this is what is causing the memory leak here.

In this particular case, the vendor for the SomeCryptComponent.dll already had an update with a fix ready to go, but I wanted to write this case study to show how you can track down a native memory leak like this with debug diag.

Have a good one,

Tess

High CPU in .NET app using a static Generic.Dictionary

Tess1 — Mon, 21 Dec 2009 10:02:20 GMT

A couple of weeks ago I helped out on a high CPU issue in an ASP.NET application.

Problem description

Every so often they started seeing very slow response times and in some cases the app didn’t respond at all and at the same time the w3wp.exe process was sitting at very high CPU usage 80-90%. This started happening under high load, and to get the application to start responding again they needed to restart IIS.

Debugging the problem

They gathered a few memory dumps during the high CPU situation for us to review and when running the sos.dll command ~* e !clrstack (in windbg) to see what all the threads were doing we found that they were all stuck in callstacks similar to this one:

OS Thread Id: 0x27dc (124) 
  ESP       EIP     
2f77ed24 795b3c5c System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].FindEntry(Int32) 
2f77ed3c 795b3835 System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].ContainsKey(Int32) 
2f77ed40 209f1932 MyComponent.Settings.get_Current() 
... 
SOME STACK FRAMES REMOVED AS THEY ARE NOT IMPORTANT FOR THIS ISSUE 
... 
2f77f0a4 209f7545 ASP.MyApp_default_aspx.ProcessRequest(System.Web.HttpContext) 
2f77f0a8 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 
2f77f0dc 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 
2f77f11c 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 
2f77f16c 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 
2f77f188 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 
2f77f1bc 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 
2f77f1c8 65fcfa6d System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 
2f77f3d8 79f047fd [ContextTransitionFrame: 2f77f3d8] 
2f77f40c 79f047fd [GCFrame: 2f77f40c] 
2f77f568 79f047fd [ComMethodFrame: 2f77f568]

In other words, the method MyComponent.Settings.get_Current() was calling ContainsKey on a Generic.Dictionary object and for some reason it was getting stuck when trying to find the entry.

Looking at the MyComponent.Settings.get_Current() method, we found that the Generic.Dictionary it was calling ContainsKey on was a static dictionary and that all threads were working on the same dictionary.

The MSDN documentation about Generic.Dictionary has the following information about the thread safety of Dictionary objects

A Dictionary can support multiple readers concurrently, as long as the collection is not modified. Even so, enumerating through a collection is intrinsically not a thread-safe procedure. In the rare case where an enumeration contends with write accesses, the collection must be locked during the entire enumeration. To allow the collection to be accessed by multiple threads for reading and writing, you must implement your own synchronization.

What is happening here, and causing the high CPU is that the FindEntry method walks through the dictionary, trying to find the key. If multiple threads are doing this at the same time, especially if the dictionary is modified in the meantime you may end up in an infinite loop in FindEntry causing the high CPU behavior and the process may hang.

Resolution:

These type of timing issues with static collections are fairly common in ASP.NET apps with high load.

To resolve this timing issue you should take special care to synchronize (lock) around access to the dictionary if there is a possibility that you may have multiple writers working at the same time or if there is a possibility that you write while someone else is reading/enumerating through the same dictionary.

In general, I would recommend to always read the thread safety information carefully when using static collections as many of them require that you implement synchronization on concurrent read/write operations, to avoid this type of issue or issues with for example HashTables where you may get exceptions like InvalidOperationException “Load Factor Too High”.

Have a good one,

Tess

Property	Type	Default value
HoverColor	Brush	Brushes.White
TextColor	Brush	Brushes.White
TextSize	int	36
Image	string	“”

	Value	Comment
DumpFolder	c:\mycrashdumps	This is the location where the dumps will be generated, so the user that the W3WP.exe process is running under needs access to write to this location.
DumpCount	10	Maximum # of dumps in the folder, depending on the size of the process 10 dumps can take up a lot of disk space, so monitor carefully.
DumpType	2	To debug .net crashes a full user mode dump is often necessary.