ASP.NET Debugging : Debugging

Debugging .NET apps for .NET 4.0

Tom — Wed, 02 Sep 2009 19:45:35 GMT

With Visual Studio 2010 and .NET 4.0 getting ready to be shipped sometime in the next year, I wanted to see what were peoples ideas for how to debug applications that were written with them. Are you happy in production with what you are currently doing? Do you use SOS.dll to troubleshoot problems?

SOS

The main place I am really interested in is production debugging. Is SOS a sufficient tool to get the data that you need? If you had another version of SOS for .NET 4.0, would you be happy with the features it has or would you want something more?

Debugger

I understand that using a typical debugger can be very difficult for people. If you don’t know what you are looking at, it can be very difficult to understand what is a problem and what is normal. It is also very difficult to see typical problems. For example, if you know your program is crashing due to running out of memory, what do you do to see what is taking up memory? How do you tell if it is fragmentation? What types of fragmentation are there that you would need to look for?

I wanted to see what the interest is in a different way of looking at the data, something more along the lines of DebugDiag.

Opinions

I’d love to hear your thoughts. Let me know!

Logging modules for ASP.NET (MVC) and also for Windows Azure

Tom — Fri, 24 Apr 2009 17:07:20 GMT

I was just reading through Scott Hanselman’s post about ELMAH and this sounds like a great idea. Getting a easy to consume report of all of your exceptions is a wonderful thing, especially when you add in that you can get it as an RSS feed, an email or a web site. You can check out ELMAH or read through his post to get a lot of details on it.

The web site looks like this:

This also got me thinking about Windows Azure and some of the logging that has been done already for that. So I wanted to highlight one of the projects here and talk about what it does. It is called Azure Application Monitor.

This allows you to see how much time and memory your instance is using. After you integrate this into your application, you will get a report like:

It is easy to add this to your application.

Download the application
Add the following line to your code. If you are adding it to a WebRole, add it in the Page_PreRender() function. Or in the Start() of a WorkerRole, changing it to WorkerRole below:
```
Neudesic.Azure.AzureMonitor.Start("AppName", "WebRole");
```
Be sure you have the TableStorageEndpoint configuration setting configured.
Create a PerfCounterUpdateInterval configuration setting and set the value (the value is in ticks (1000 ticks = 1 second)
Test and deploy your application

To view the data, you need to configure the AzureMonitor so that it points to your TableStorageEndpoint

As for what all it tracks, here is the list:

Application Name (you supply this in your code)

Role Name (you supply this in your code)

Machine Name

Process ID

Thread count

Handle count

Total processor time

User processor time

Private memory size

Non-paged memory size

Paged memory size

Paged system memory size

Peak paged memory size

Peak virtual memory size

Peak working set

Page file usage

Peak page file usage

Non-paged Pool Usage

Local machine time

Process local start time

It is rather easy to add more stuff to this as the code is available from the projects location.

One a side-note, if you want to see what data you have in your Blobs, Queues and Tables, there is an updated version of Azure Storage Explorer available now. From that site:

The most current release of Azure Storage Explorer, version 2.0, has several improvements over the original version:

The UI is WPF-based and has Outlook-style navigation.
Multiple storage accounts are supported, and you can change your storage account details directly in the tool.
In addition to text and byte views of items, blob items that are pictures can be viewed as images.

Azure Storage Explorer 2.0 does not presently allow you to act on storage items. This capability is being considered for a future update.

Let me know what you think or if you use any other tools for ASP.NET or Windows Azure.

Any ASP.NET Debugging requests?

Tom — Wed, 22 Apr 2009 13:00:00 GMT

With us starting to look forward to .NET 4.0, I started to think about the debugging story for ASP.NET and wondered if you had any requests for what you would like to see.

Some of the things that are on my mind are ideas like:

Giving some kind of visual debugging experience to help people to troubleshoot ASP.NET problems more quickly.
Coming up with a better story for x64 dumps
Having some more automated commands to give information to users.

I am not saying that any of these will be coming, but they are all things that I am interested in seeing and will be investigating.

So what would you like to see for the future of the debugging experience?

ASP.NET Troubleshooting

Tom — Tue, 24 Mar 2009 19:31:03 GMT

So with how long ASP.NET has been out for now, I am really curious to know how people go about tracking down issues in their project.

I’d like to know things like:

How do you know there is a problem
What tools do you use to confirm that the problem really is a problem
How do you gather data about the problem
How do you determine if the problem is your code or something else (like the network, hard drive failure, etc)
If the problem is your code, how do you ensure your fix resolves the problem (how do you test the fix)

If there are any others, I’d also like to hear them.

Also, if you do anything in the Cloud now, I’d love to hear how you troubleshoot that. And it can be from any cloud service provider.

And have any of these things changed with some of the new ways to program that have come out. For example, AJAX, MVC, Silverlight, IE8, jQuery, etc.

What do you need to troubleshoot Azure?

Tom — Wed, 19 Nov 2008 14:00:00 GMT

Looking to the future with cloud computing, it is going to become increasingly important to have good information about what is happening with your site in order to properly maintain it.

Keeping development type of issues aside, what types of things do you think you will need in order to be able to properly troubleshoot a problem in your application once it is deployed to the cloud.

For example, when you publish your application, it will be similar to uploading today to a hoster in that:

No access to the server
Cannot get dumps of any processes
Cannot access perfmon logs
Cannot access the Event log

So given that type of situation:

What are the possible problem situations that may arise?
What types of information do you think you will require to find out what is going wrong with your application?
Would you try to pull your application onto your development box to try to repro some of the problems that way?

Or are there any other concerns or thoughts you have around this? I’d love to hear what you have to say.

Strange callstacks

Tom — Wed, 29 Oct 2008 13:00:00 GMT

How many times have you been troubleshooting a dump or application, you look at the callstack and you see something that just doesn’t quite look right. Chances are the problem is that you don’t have correct symbols.

For example, if you see something like this:

Vswebdesign!DllCanUnloadNow+0xb02a0
Vswebdesign!DllCanUnloadNow+0xb8ef4
Vswebdesign!DllCanUnloadNow+0xb2203
Vswebdesign!DllCanUnloadNow+0xb248f
Vswebdesign!DllCanUnloadNow+0x2c406
Vswebdesign!DllCanUnloadNow+105a89
...

That usually is pointing to this problem. You can see that the offsets for the functions are very large. Also, the callstack is the same function repeated over and over. And most applications probably don’t call DllCanUnloadNow.

If you get this problem, you can try to get the correct symbols, or just understand what you are seeing here and know that you aren’t seeing the correct functions. For most things that concern you, you should be able to get the correct symbols so that you can properly resolve your callstacks.

Videos are coming – suggested topics?

Tom — Mon, 13 Oct 2008 13:00:00 GMT

So I have started to create some videos showing various things that you can do. Are there specific things that everyone would like to see? I’d like to build a list of the things you want and then I can start knocking them out. I’m figuring most of them are going to be around debugging and probably around ASP.NET but feel free to suggest other things as well.

For example:

How to get sos to work when debugging on another machine.
What is a good way to know if a memory problem is managed or native?
How do I print out the column names of a DataTable?
How can I find out what is stored in the ASP.NET Cache?

These are simply examples. Please let me know what you would like and be as specific as possible.

Also, keep in mind that we have a bunch of development videos already up on http://www.asp.net/learn.

Tail calling in .NET

Tom — Thu, 02 Oct 2008 13:00:00 GMT

Thought I would give a little details on one type of optimization that it is possible to see and explain what it is and how it affects things so that if you come across it, you will understand what is happening.

So there is this concept of tail calling which is where the compiler will optimize code to save execution of instructions as well as saving some reads and writes of stack memory. This happens under certain circumstances when a function ends by calling another function (hence tail calling). For example, if we have the following:

     static public void Main()

    {

        Test();

    }

    static public void Test()

    {

        First();

        Second();

        Third();

    }

    static public void Third()

    {

    }

In this sample, it is possible for the compiler to make the call to Third() a tail call. So to better explain this, under normal situations we would see something like this on a callstack while we are executing in the function Second():

Second()
Test()
Main()

When Third() runs, it is possible for it to be a little different. Instead of creating a new allocation on the stack for Third, it can replace Test() with Third(). So then we would see:

Third()
Main()

When Third() returns, it will return directly to Main() and not have to go through Test() at all. This saves time and memory for the application overall.

To tail or not to tail

So how does the compiler decide if it should use this optimization? Well, first there is the discussion of which compiler we are referring to. There are two compilers at play with .NET. The first takes the source code and compiles it into IL. Then the JIT compiler will take the IL and create native code. The creation of IL can set up hints to try to help with the creation of tail calls, but it is ultimately the JIT compilers job to create them. There are a lot of rules that determine if it can do this but a quick summary of the rules that will make us NOT use this option are:

Note: This is subject to change and you shouldn’t base anything on this behavior. This is just for general knowledge.

Caller doesn’t return immediately after the call
Stack arguments between caller and callee are incompatible in a way that would require shifting things around in the caller’s frame before the callee could execute
Caller and callee return different types
We inline the call instead (inlining is way better than tail calling, and opens the door to many more optimizations)
Security gets in the way
The debugger/profiler/config turned off JIT optimizations

If you really want to see the full list, check this this post.

Who cares about tail calls, sounds like a trip to the zoo

Normally this kind of thing won’t matter to you unless you are trying to write a profiler, but there are times that you may see a callstack that doesn’t make any sense because one of the functions is missing. As someone that has done a lot of debugging, I have come across this from time to time and thought others may find it useful to know about.

ASP.NET Tip: How to avoid creating a GC Hole

Tom — Thu, 25 Sep 2008 13:00:00 GMT

There are only a few things that can make a .NET process crash. The most common one is an Unhandled Exception getting raised. Another way that is can happen is by creating a GC Hole.

What is a GC Hole

So first a little background on what I mean by a GC Hole. A GC Hole is any corruption that happens inside of the managed heaps. Under normal circumstances, this cannot happen as you don’t have pointers that reference objects in the heap so you can’t corrupt them. This corruption is generally seen by the GC (Garbage Collector) when it is trying to compact the heaps and release objects no longer referenced.

How does a GC Hole get created

So if we don’t have access to pointers (not counting managed C++ and that is a whole different conversation), how can this occur? Well the most common way is by making a native call (P/Invoke) that returns data. There are actually two things that can happen during this process to cause a GC Hole.

The first is a buffer overrun. You can imagine passing a byte array to a native function and having that array be 200 bytes. If the native function returns 400 bytes worth of data, it will write past the end of the object and corrupt the next object in the managed heap.

The second happens if the object being passed to a native function isn’t pinned first.

Pinning is a whole topic on it’s own. There are lots of problems that can happen from pinning object in memory, a great deal of information can be found here on the most common problem you can have while pinning.

Back to the GC Hole issue. If the following events occur, it will cause a GC Hole:

Allocate a byte array to be passed to a native call, do not pin it
Make the native call passing the byte array
While the native call is processing the request, a Garbage Collection occurs
Native call returns

So what is the problem with the GC running before the native call returns? Well, the GC has the ability to compact the heap (move object around so they are next to each other) and if it moves our byte array to a new location, the native call will still have the old location as to where to write it’s data to. So when it returns, it will write the result into memory where the byte array used to be, thus corrupting anything in that memory location currently.

For example, if we had this when making the native call:

_______________________________________
|             |             |                  |            |             |
|     int    |    int     |      Free     |   byte |   Free   |
|             |             |                  |            |             |
_______________________________________

Then after the GC runs, it will compact the ints and byte next to each other like:

_______________________________________
|             |             |            |                               |
|     int    |    int     |   byte |       Free                 |
|             |             |            |                               |
_______________________________________

Now assume objects get created inside that free block, we would then have something like:

_______________________________________
|             |             |           |           |           |         |
|     int    |    int     | byte | String | String | Free |
|             |             |          |            |           |         |
_______________________________________

Then when the native calls returns, it will write to the original location of the byte array and thus corrupt the two strings.

How to find a GC Hole and fix it

This is something I will address shortly in another blog post. We will look into troubleshooting a GC Hole and see what we can do to find the problem.

I hope that this post will help developers understand the areas where they need to be careful and help to keep everyone from creating them in the first place. For information on how to pin an object correctly, take a look at GCHandles, Boxing and Heap Corruption.

More great information can be found at, Asynchronous operations, pinning

Who uses SOSEX and what for?

Tom — Mon, 22 Sep 2008 13:00:00 GMT

Another thing I am curious to know is if anyone is using SOSEX and what advantages it gives you for debugging. I have talked to the author of it and I think it has a lot of useful commands, just wanted to get everyone’s take on it.

Let me know any problems you have with it also.

If you don’t know what SOSEX is, take a look at: Steve’s blog

High Memory part 6 – Fragmentation revisited

Tom — Wed, 17 Sep 2008 13:00:00 GMT

I have talked about a bunch of commands used to troubleshoot a managed memory problem in the past and given some situations of using them like:

There is another way that we could see a problem that involves fragmentation. Let’s consider the situation where we see the same situation we saw before, namely:

But we do see a bunch of files listed when we look in lm. So what should this tell us about the problem? Well, first off, since the files are showing up in lm, we know the problem isn’t the same as, Dynamic Assemblies and what to do about them. So next we would want to look and see specifically what the files are.

Situation 1

The first common problem that you will see is caused by either have debug=true, or batch=false is set. Batch is similar to debug, set in the same place and will tell the compiler to not create one DLL for the entire directory but rather create one DLL for each file we compile. Underneath, debug=true sets batch=false, so they do the same thing in this regard. If that is the problem, you will see a bunch of strangely named DLLs in the lm list. Names like:

  hbdp25u7.dll
  1gxdnryw.dll
  fzgq0g7y.dll
  fhlpqzqm.dll
  su0utu5-.dll
  vhaewyii.dll
  tdkuail7.dll

Turning debug off, or batch on will resolve the problem in this situation.

Situation 2

The second common problem that will cause this is if you have a common framework that you are using in multiple applications. One common way to do this is the copy the framework files into the bin directory of each application. This will cause us to create a different in-memory file for each one. So instead of just having 5 files that make up your framework loaded in memory, you will have 5 * (# of AppDomains) loaded.

You can identify this problem because in the list of files, you will see the same filename repeated, only some will have some numbers after them:

  MyFramework.dll
  MyFramework_50008012.dll
  MyFramework_50002008.dll
  MyFramework_40101008.dll

This is just showing the different versions of the file by showing you the other copies with their base address (load address).

The easiest solution to this type of problem is to place the framework files into the GAC. Then you will only have one copy of the files which all the Applications will share.

Debugger Extension update

Tom — Fri, 12 Sep 2008 13:00:09 GMT

The debugger extension that we were working on getting out with the debugger package has hit some roadblocks and it looks like it isn't going to be shipping with the debugger anytime soon.

We are looking to see if we can find an alternative method to get it out. But for the meantime, just stay tuned and I'll let you know when I find something out.

I know the additional functions will be a lot of help to everyone that uses it so hopefully we can get something to you soon. I'll keep this blog updated with the latest.

How do you test the performance of your web site?

Tom — Wed, 10 Sep 2008 13:00:00 GMT

I wanted to get a feel for what tools people are using to test the performance of their website. My questions are:

Do you check page-load times?
Do you dig in to see which files are taking the time
How do you track down a page that uses a lot of memory
How do you track down a high CPU consuming page

I don’t want to list any possible tools as I’d like to hear what people are using without me putting any suggestions out.

If I don’t see some being mentioned, I’ll be sure to bring it up so we have a holistic view of the tools.

Look forward to seeing what people use.

SOS Tip: Using help

Tom — Tue, 09 Sep 2008 13:32:00 GMT

This is just a quick little note to remind everyone that the !help command in sos can be very helpful. Not only does it list all of the possible commands, but if you run it and pass it the name of a command, it will print out a bunch of really useful information for that command.

For example, if you look at the help for !dumpheap: !help dumpheap

You will see, for example, the syntax that you can use to pass the list of objects returned from !dumpheap -mt to a .foreach command.

I use the help a lot to check on the different switches and also for reminders like with .foreach, when to use a ( and when to use a {. Those always seem to trip me up.

Another really useful command inside help is the FAQ. If you run !help faq, it will print out all sorts of things that are really useful to know.

If you haven't checked out the !help command much in sos, please do now and let me know what you think.

SOS Best Practice: Match the SOS version with the process being analyzed

Tom — Mon, 25 Aug 2008 13:00:00 GMT

I haven’t seen a whole of of issues around this as of yet, but now that 3.5 SP1 has released that may change. If you copy SOS.dll to another location and then load it out of that directory to troubleshoot dumps, you may start seeing problems. Namely, things like !clrstack not returning any valid names for functions and just printing out the numbers. For example:

0:006> !clrstack
OS Thread Id: 0x1154 (6)
ESP       EIP     
0154cfdc 7614f35f [HelperMethodFrame: 0154cfdc] 
0154d080 5fb7b8cd 
0154d094 5fb7b78b 
0154d0bc 5fd26907 
0154d0d0 5f95a87c 
0154d0e8 5f959458 
0154d108 5f9634f1 
0154d124 5f96342d 
0154d150 65525e55 
0154d17c 5e339ea3 
0154d1f8 5e33ce4d 
0154d37c 5e3400cc 
0154d380 5dd60d14 
0154d3b4 5e56def3 
0154d408 5dce876b 
0154d438 5dd12799 
0154d47c 5dd0df51 
0154d4b0 5dcea92a 
0154d714 6e871b4c [GCFrame: 0154d714] 
0154d730 6e871b4c [GCFrame: 0154d730] 
0154d8a4 6e871b4c [ContextTransitionFrame: 0154d8a4] 
0154d9b8 6e871b4c [GCFrame: 0154d9b8] 
0154d99c 6e871b4c [GCFrame: 0154d99c] 
0154ec1c 6e871b4c [CustomGCFrame: 0154ec1c] 
0154ec30 6e871b4c [CustomGCFrame: 0154ec30] 
0154ec44 6e871b4c [CustomGCFrame: 0154ec44] 
0154ec58 6e871b4c [CustomGCFrame: 0154ec58] 
0154ec6c 6e871b4c [CustomGCFrame: 0154ec6c] 
0154ec80 6e871b4c [CustomGCFrame: 0154ec80] 
0154ec94 6e871b4c [CustomGCFrame: 0154ec94] 
0154f060 6e871b4c [TPMethodFrame: 0154f060] 
0154f07c 5dcea109 
0154f0f0 5dce9d1f 
0154f128 5dce9bed 
0154f170 5dce9b3a 
0154f194 5e3bf136 
0154f3f8 6e968c4e [GCFrame: 0154f3f8] 
0154f554 6e968c4e [ComMethodFrame: 0154f554]

This is a sign that there may be something incorrect with SOS. We can clearly see this is a managed stack, but we aren’t getting any values.

The reason for this is that we don’t guarantee SOS to work with any version of the framework other then the version that was running. In other words, always match up the SOS version with the framework version. If you are troubleshooting on the same machine you captured the dump from, or a machine running the same version of the framework, you can easily do this by loading SOS using:

.loadby sos mscorwks

Please note that the SOS that ships with the debugger package is different from this. This version will work correctly with any 1.x version of the framework.