As you're probably well aware IIS7 relies heavily on a master configuration file named ApplicationHost.config. Now, instead of using a history file like IIS6 did, IIS7 will check this file every 2 minutes to see if it's changed. If the file has been updated, then a backup copy of the old .config will be stored in the inetpub\history-directory.

Configuration history

This is really cool and a very useful feature. There's only one little caveat in my opinion and that's the fact that by default IIS7 will only save up to 10 backups. All the other backups will be discarded. This means that if you're spending 20 minutes fiddling with the settings in ApplicationHost.config you will quite likely have overwritten all your old backups. Now, if something goes wrong at that point and you decide to spend another 20 minutes trying to fix this problem by further editing the ApplicationHost.config you will surely have overwritten the backups.

In a worst case scenario you might even have gone from Healthy Server to Unstable Server to Server Down and you now find that you aren't even able to restore it to the Unstable state.

My suggestion is to increase the default number of backups to 100, or why not 1000? You can then feel certain that any backups taken aren't overwritten when you try to undo the very same changes. Here's how you do it:

1. Go to %windir%\system32\inetsrv\config\
2. Open applicationHost.config in notepad
3. Locate the system.applicationHost section group
4. Add the following item to the section group:
   <configHistory maxHistories="100" period="00:02:00" />
   (Naturally the maxHistories value can be whichever value you prefer)
5. Save

To restore one of the previous configurations, simply go to inetpub\history\cfgHistory_[nnnnnnnnnn] and copy the config from there, or use appcmd. (See below)

Backup using appcmd

Now that we've done this there is one more thing I'd strongly recommend doing:

1. Open up a command-line with administrative priviliges
2. Go to %windir%\system32\inetsrv
3. Run "appcmd add backup "[Name of your backup]""

What you've done now is to create a manual backup of your current config. It's always good to have a decent (and persistent) restore point to return to.

If you wish to restore this backup you simply use the following syntax: "appcmd restore backup "[Name of your backup]"". You can also delete a backup using the "delete" keyword, or you can list all backups, including the ones generated by the Configuration history, like this: "appcmd list backup"

Okay, so now we have made sure we have some extensive automatic backups in place which might save us a lot of headaches later on.

Later! / Johan

Over the past months I’ve been holding a couple of workshops on PowerShell. It’s quite an interesting topic, but not something I originally thought would be suitable for this blog and its web-development theme. Still, having given this a bit more thought I decided that there are so many useful features and cmdlets that could be of great value when hosting a web site or two that I should reconsider.

This first post will be a brief introduction for those who’ve never touched PowerShell before. Later posts will probably be more hands-on and less theoretical. I’ll gladly receive any thoughts or comments you may have.

Installing

PowerShell can be downloaded from the Microsoft web site. Simply point your browser to http://www.microsoft.com/powershell and you’ll be set in no time. Apart from the executables you’ll find a lot of useful links, like the one to the PowerShell team blog and the IIS 7.0 PowerShell Provider.

Basic syntax

Okay, having installed PowerShell we’re now ready to try some basic commands.

Echoing text

Any object not used in an assignment or operation will be echoed to the screen. This means that if we simply type a string like this:

It will be displayed on the screen. (Without the quotes.) The same goes for numbers, so by typing an equation you’ll be given the result:

If you type in an array PowerShell will echo it one line at a time:

Basically you could say that any object will, unless told otherwise, be displayed on screen using its default formatting. The default formatting is defined by the object itself.

Calling executables

Running an executable is more or less as easy as in the regular command shell. At least if the .exe is in the PATH. For any executable not in the PATH you'll need to write the full path of the .exe for it to successfully execute. This may seem like a hassle, but this behavior is by design and was implemented in order to prevent you from accidentally executing an .exe.

Cmdlets

Cmdlets (pronounced "command-lets") are PowerShell-specific commands. They follow a strict naming standard of verb-noun, (for example "get-date" or "set-location").

Get-Help

This a command that you'll use quite a lot. Get-Help <cmdlet> will display some generic information about the cmdlet in question. Using various switches such as "-detailed", "-examples" or "-full" you can then get more detailed information on the cmdlet in question. For example:

This will show you the generic information on get-date as well as some sample code.

Set-Location

Set-Location is the PowerShell equivalent to the old MS-DOS command "CD". Some examples are:

Aliases

Actually you can write CD in PowerShell too. By default PowerShell will set up an alias for you that points to the Set-Location cmdlet. This means that CD is just another name for Set-Location. If you want you can create your own aliases using the New-Alias cmdlet:

This creates an alias named gh that points at Get-Help. If you type Get-Help gh PowerShell will be smart enough to show you the documentation for the Get-Help cmdlet.

Object orientation

One of the cool things about PowerShell is that it is object oriented. This means that all the cmdlets return objects, which can, in turn, be passed on to another cmdlet. To illustrate this, let’s look at the DIR-command.

Using DIR

You can use DIR in PowerShell, but it is actually an alias for the get-childitem cmdlet. How can you tell? Use the get-command cmdlet:

Since DIR in PowerShell is not the same thing as DIR in MS-DOS that means you can’t use all the switches you might be used to, for example DIR /w will not work.

So, what happens when you type DIR and press enter?

Well, first of all the get-childitem cmdlet is run. It looks in your present working directory, creates a collection of all the items there, and returns it. You can think of this collection as a big Excel Spreadsheet if you like.

Once the collection is returned to PowerShell it will be displayed on screen. This is because the default behavior for PowerShell when it comes to any object is to display it on screen. We saw this earlier with strings, integers and arrays.

The collection will be displayed using a default presentation. This does not mean you will see all the information the object contains. If I run Get-ChildItem (DIR) on a folder I’ll get the standard information. Like so:

Even though I get only text on screen the actual object returned is much more complex. I can demonstrate this by piping the object into another cmdlet.

Piping

A simple MS-DOS piping example would be type file.txt | more. Actually you can use more in PowerShell as well, but let’s look at another, more interesting cmdlet that we could pipe the object into; Format-List.

Format-List

By default the output of Get-ChildItem is formatted in a table, but let’s say we want it as a list instead? No problem. Simply use the following syntax:

We still have the default information, only presented in a different manner. Still, there is a lot more information we could get, by using the –property switch. By simply using the following syntax: dir | format-list –property * we get all the properties of each and every item in the directory. The output is quite extensive:

As long as something is returned it is possible to daisy-chain additional cmdlets onto the previous one. We could, for example, add | more to the command above to divide it into smaller chunks. We could use the Out-File cmdlet to save the result to a textfile, etc.

Summary

What have we covered so far?

• PowerShell is object-oriented
• By default any object will be displayed on screen, so if you want to know what 2+2 equals, simply type 2+2.
• You can call standard .exe applications, etc from PowerShell, but they either need to be in the PATH, or you need to write the full path to them.
• PowerShell-specific applications are called cmdlets (“Command-lets”)
• Most MS-DOS commands work, but they’re usually aliases to their PowerShell equivalent, so you might not be able to use all the switches you’re used to.
• You can use the | to daisy-chain cmdlets together, such as dir | format-list | more

We still haven’t covered things like functions, variables, creating .NET objects, etc. I also plan to write a few samples showing how to use the cool cmdlets to get information from computers on the network, looking for specific files, exporting to Excel, printing, etc. As always I’d be happy to receive feedback and suggestions on things to cover.

/ Johan

I got the following question in my Getting started with windbg – post and I thought it might be worth posting the replies in a separate article:

Hi Johan,

about those threads with an ID of XXXX, should they go away after certain amount of idle time like 2 minutes? I am trouble shooting an application while the application should sit idle (because of no stimulation), however it is still using quite some CPU time and and threadpool threads, where it should not. So I created an adplus dump when it is sitting idle, and I found that there are many threads with ID XXXX, and I created a dump file again after 15 minutes, again, it still have the exact same XXXX threads. All those threads are completion port threads, and I wonder why they did not get recycled?

Also, when doing a '~*e !clrstack', most of the worker threads and completion port threads are showing "Failed to start stack walk: 80004005". Is there a way to show the stack for those threads, because those are the threads I am interested in.

Thank you!

The GC will be triggered when you’re allocating memory. If your application is inactive, then so is the Garbage Collector. Even if you’re making a few, random allocations they still may not be enough to trigger a GC, so this is not at all unexpected.

The “Failed to start stack walk: 80004005”-error is displayed when the thread did contain a managed stack, but no longer does.

/ Johan

Due to the architecture of web services and web applications they can be quite slow to start. For example on my Windows 2003-box the initial localhost-call to a simple “Hello World!”-web service takes approximately 8 seconds, while the next request is more or less immediate.

Why is this?

This isn’t news, really. One of the first things we learned in the early beta stages of ASP.NET was that the first request would take a little extra time due to the Just In Time Compilation (JIT). This was a major change from the classic ASP architecture where everything was interpreted rather than compiled. Upon the first request the .aspx / .asmx file will be compiled (JIT’ed) into Microsoft Immediate Language (MSIL) and the resulting .dll will be moved to it’s proper location.

This compilation will occur every time the application pool starts causing a significantly longer response time on the first request compared to the following requests.

Run-time performance over startup performance

It all comes down to prioritizing run-time performance over a quick startup. Which, in my humble opinion, is a sound choice. There are, however, situations where you may feel that this is to your disadvantage. The other day I got a question from a customer who had a web service that was called quite infrequently and to him this meant that with every other request response times would be horrendous, simply because the application got recompiled.

For each case I get I usually do a quick search on the problem description. I don’t really expect to find the solution this way, since I can only assume that the customer has tried this as well. Instead it will likely present me with a list of troubleshooting steps that the customer has already tried as well as possible solutions that didn’t fit. (Off course I still need to verify this with the customer).

This time I didn’t really need to do this since I was pretty sure what the cause was. Still, I was curious to see how common this problem was. I honestly didn’t think this was a very common scenario, but my quick search on the web revealed the contrary. The few hits that I quickly browsed through suggested writing an application that would ping the web service in order to keep it alive. I must admit I find this quite inventive, but I really don’t think it’s the best approach. :)

What to do

So, what is to be done about this? Well, I’ve already mentioned that there are better solutions than pinging the web service. As I told my customer there are basically two good options:

1. Precompile the assemblies
2. Change the settings for automatic recycling on the worker process

Precompiling the assemblies

There is a nice little tool called Native Image Generator. (ngen.exe) It will compile the assembly into native images and installs them into the native image cache. The syntax is pretty straight-forward: ngen.exe install [assembly] The problem is that once you’ve changed the assembly you’d have to precompile it again making updates a bit tedious.

There is also a service-version of the ngen called the Native Image Service. More information about ngen can be found at http://msdn2.microsoft.com/sv-se/magazine/cc163808(en-us).aspx

Changing the recycling settings

The reason why my customer was experiencing this problem was because his application pool would time out and recycle itself due to the default settings in the IIS manager. By default the worker process will recycle after 20 minutes of inactivity, so if no one has pinged your application in that time the next request will cause the application to recompile, resulting in a longer response time. By tweaking this setting in the IIS manager to something more suitable to the current rate of requests, like 40 minutes or maybe even 60 he should also be able to find a nice balance. Actually, considering the low load it would probably be a good idea to turn this setting off completely. Here’s how:

• Open up IIS Manager
• Locate the Application Pool in question
• Right-click it and select “Properties”
• Go to the “Performance”-tab
• Change, or disable the “Idle timeout” setting

/ Johan

Yesterday an early Alpha of Windows 9 was released on the corporate network and I couldn’t resist installing it. I started up the installation, which went very smoothly by the way, filled in the necessary details, clicked “Finish" and left it running overnight. When I arrived this morning a friendly Power Shell-window was greeting me asking me to calibrate the voice control.

Working with Voice control

After less than 90 minutes of training I was able to open notepad and highlight a section of text which was very exciting.

I wasn’t the only one who’d leapt at the opportunity to run the Alpha, so there were quite a lot of us shouting commands at our computers which caused quite a ruckus at times. It was quite impossible to keep a normal conversation over the phone with people shouting “New Paragraph!” and “Dear Mom!” left and right. Still, that’s the price you pay for being an early adopter I guess.

The decision to use voice command (VC) only is definitively a brave move, but I think it’s the right way to go. People have been generally hesitant to use the VC features of Vista, but I think that’s mainly due to the lack of support. Forcing developers to think outside the box and adapt their software for this new means of control is definitively the way of the future. Just like Nintendo demands that developers adapt to their unorthodox controller for the Wii, I’m sure Microsoft will start a new era in computer software with this revolutionary step.

After only 90 minutes I was able to highlight text in Notepad.

Other cool features

Apart from the VC there were a lot of other cool features that I really enjoyed.

Xbox 360 emulator

The next Windows will ship with a complete Xbox 360 emulator. It is basically a tweaked version of Virtual PC that allows you to run console games on your PC. I fired it up and played a little Halo 3, but I must say I was a bit disappointed. The voice control needs additional working on. It quickly got tiresome yelling at Master Chief and my cries of “go forward”, “go forward”, “fire”, “turn around”, “left”, “left”, “fire” and “reload” made it quite hard for me to disguise to my boss what I was doing. One of my colleagues had just picked up a case regarding twelve crashed IIS-servers and an on-line banking application. He was on the phone trying to sort the situation out and started giving me dirty looks after a while. I also must add that the whole Xbox-live experience was somehow diminished. The usual chatter was sadly missing since we were all so busy shouting directions. Still, it’s only an alpha, so I’m certain these things will be addressed.

Boss key

Another nice addition that also seems to heir from the Home Entertainment Division is a boss key. It's really quite surprising that we had to reach the 9th installment of Windows before this was introduced. All I had to do was say "Hi boss" and the screen would immediately present a blue-screen. This is useful in more ways than one. Not only does it quickly hide your Halo 3 session from view, but it also gives you a reasonable explanation as to why your productivity has been somewhat lacking. It's a pure stroke of genius.

On one occasion it turned out that the alpha had blue-screened for real due to some conflict with my graphics driver. It took me almost 15 minutes to realize this after having spent several fruitless attempts trying to wake it up from boss mode. :)

Release date

From what I've heard the planned release date is April 1st 2009, exactly one year from now.

/ Johan

Here's a little scenario I came across the other day. I've forwarded the information to development, so it's pending further investigation. I still thought it would be a good idea to publish the scenario though.

Visual Studio 2008 + FTP = Possible trouble

It seems like there might be a bug in Visual Studio 2008 when it comes to projects that you access via FTP. I don't know how common it is to use FTP to access & deploy your projects. It would be really interesting to see some genuine statistics, but since FTP is highly insecure I guess it is very rarely used. This, I guess, is probably the reason why this little glitch slipped through in the first place.

File change notification works fine...

Consider the following scenario:

• You're working on a web project using FTP
• You've downloaded a file to your local cache and you're doing some changes.
• Another client changes the file
• You attempt to save the file, but the original file on the server has changed so it no longer matches the one in your local cache.

This causes a warning to pop up with the following information:

A more recent version of the file [File Name] has been saved to the web on '[Date]' . Do you want to replace the server file with your local file?

...overwriting the file doesn't

Clicking Yes you'd now expect Visual Studio to simply overwrite the file on the server. Unfortunately this is where the glitch rears it's ugly head. Instead of overwriting the file it gives you the following error message:

Cannot save the file [File Name] to the Webserver. The file [File Name] has been modified by (unknown) on [Date] [Time Zone].

Continuous attempts at saving will have no effect. The error pattern will repeat, displaying the two dialogue boxes over and over again.

Alternative solution

My customer originally fixed this by selecting everything in the editor, copying it to Notepad, canceling the operation in Visual Studio, reopening the connection, pasting the contents from Notepad into the document and saving. Fortunately there's an easier way to resolve it.

Simply click the Refresh-button in the Project Explorer window. Visual Studio will then ask you if you wish to update your cached version of the files you've edited. This may sound like you're going to perform a revert-to-save operation on the pages you've edited, undoing all the changes, but it's actually not the case. Visual Studio will simply refresh the files in the cache, not the ones in the editor. When Visual Studio has finished refreshing the cache it will notice the inconsistency between the files asking you if you want to update the file in the editor as well. At this point you should answer "No", unless you want all your changes undone.

My thoughts about this

Obviously this shouldn't happen. You shouldn't have to refresh the cache in order to overwrite the files. Still, as I mentioned before I believe there are two reasons why this problem managed to slip through the cracks.

• FTP is unsafe. Even the password is sent as clear text and can be intercepted by anyone. For this reason I would assume that FTP is quite rarely used.
• In order for the problem to occur you need to trigger a file change notification on the file in question. For normal scenarios this would mean that another client accesses the FTP-site at the same time and changes the file as you're working on it. This type of on-the fly editing without any form of source-control would be highly risky in a production environment and you'd most likely use some other means of connection while in development.

Visual Studio 2008 + FTP + Internet Explorer 6 = More trouble

Unfortunately the problem doesn't end there.

It turns out that if you are using Visual Studio 2008, are working on a web project using FTP and have IE6 on your machine, then IE will act as if a file change notification has occurred after 90 seconds wether this is true or not. This means that after 90 seconds all your attempts to save will trigger the behavior described above. The refresh cache-approach still works, but it will quickly become quite tedious.

According to the prerequisites for Visual Studio 2008, Internet Explorer 6 is a minimum requirement, so there is nothing documented on IE7 being a necessity in order to run Visual Studio 2008.

Alternative solution

So far I only know of one way to resolve this. Install IE7. You'll still encounter the first potential problem if you have more than one client working simultaneously on the same application, but as I've already mentioned, this sounds like a very unlikely scenario.

/ Johan

Problem:

A customer called in. They had a Web Service running on a single IIS6. Memory usage would slowly increase and not be released. As a workaround they'd currently set the application pool to recycle at 500 MB, causing a few failed requests upon each restart.

I thought I'd describe how I went about troubleshooting this scenario.

The first dump

The customer had already taken a memory dump using DebugDiag, so obviously I asked them to upload it. It was a huge dump. 1.64 GB! I opened it up in windbg and ran the following command:

!eeheap -gc

I haven't mentioned !eeheap before. It gives me a nice summary of what's on the managed heap(s).

So, as we can see the managed memory is pretty evenly distributed in the 4 managed heaps. If you look down at the bottom you'll see that total memory usage is at 88 MB. (This is a figure I would have gotten by running !dumpheap -stat as well. So in case you're wondering; that would have worked just as fine.) Anyway, if only ~90 MB is in the managed heap, then the remaining 1.55 GB must be on the native heap.

New dumps

Troubleshooting the native heap is a bit harder. The sos extension and windbg is a really nice team so working without them can be tough. Fortunately there is a nice feature included in DebugDiag that lets it perform an automatic analysis. This is usually the best way to get information from a native leak. Sometimes you may want to get a little additional information from windbg as well, but most of the time DebugDiag's auto analysis will get you a long way.

LeakTrack

DebugDiag comes with a handy little dll called LeakTrack. LeakTrack attaches to the process in question and monitors memory allocations and their related call stacks. Since the dll is injected into the process it can have a slight impact on performance, but if you want to troubleshoot the process you really have no other options.

There are two ways to start monitoring a process with LeakTrack:

1. Create a "Memory and Handle Leak"-rule using the wizard normally displayed upon start up.
2. Cancel the wizard. Go to the "Processes"-tab. Right-click the w3wp.exe you wish to monitor and select "Monitor for leaks".

LeakTrack will begin monitoring from the moment you inject it. Obviously, it won't be able to get any data for events that have already occurred, so injecting LeakTrack when memory usage is already at it's peak won't really do much good. Instead I usually recycle the process and attach LeakTrack as soon as possible.

This time I chose to let the customer create a "Memory and Handle Leak"-rule using the wizard. This rule then created dumps at certain intervals which the customer uploaded to me.

Analyzing the dumps

So the next step is to let DebugDiag analyze the dumps. Here's how you do it:

1. Run DebugDiag
2. Cancel the wizard
3. Go to the "Advanced Analysis"-tab
4. "Add Data Files" - point to the dump
5. Choose "Memory Pressure Analysis" in the "Available Analysis Scripts" list.
6. Make sure your dump is selected in the "Data Files" list.
7. "Start Analysis"

The usual suspects

Once DebugDiag has finished analyzing the dump you will get a new browser window with a report. The report I got showed the following right at the top:

So what does this mean? It means that LeakTrack has monitored the allocations made by a certain third party dll. It has also monitored if that memory has been properly released. All in all the .dll now has almost 150 MB of allocations that are unaccounted for. It would be safe to assume that this is our culprit. Further down in the report we have a link called "Leak analysis report". Clicking it will bring us to the following graphical representation:

As you can see the .dll in question is right at the top. But wait a minute! According to the graph mscorsvr has ~9 MB of outstanding allocations as well. Does this mean that mscorsvr is leaking too, though not as much?

The answer is no. As I mentioned before; LeakTrack will monitor all allocations and check that the memory is properly released as well. All we know is that mscorsvr has allocated 8,75 MB of memory that hasn't been released yet. There's nothing to suggest it won't release it eventually. The odds of the stray 148 MB eventually being released, on the other hand, are a lot slimmer in my opinion.

What causes the leak?

Since the dll responsible for the allocations is third party I can't find out exactly what causes the leak within the third party dll. What I can find out is what calls in the customer's code lead us down the leaking code path in the dll. LeakTrack monitors the callstacks related to memory allocations. Looking at the report I was able to find some nice callstacks that the customer should keep an eye on. It might be possible for them to tweak their code slightly in order to avoid the scenario.

Okay, now what?

Well, this is actually more or less where the road ends for me. Had the problem lied within a Microsoft component it would have been up to us to fix it. Had it been within the customer's code I would have been able to use the customers symbols to get a more detailed callstack and provide him with pointers on how to resolve it as well, but since this is a third party component the next step is to forward it to the third-party vendor.

/ Johan

I was really excited to hear that XNA 3.0 will support game development for the Zune. I've always been interested in XNA and have secretly dreamed of writing my own Xbox 360 game ever since the alpha. Sadly I just don't seem to find the spare time.

Key features

As I looked at the info released on the XNA Creators Club these are the features that caught my interest:

2D graphics

Well this was a no-brainer. I don't think anyone expected the Zune to have a full-featured 3D API available.

Support for all Zunes

This was really cool. XNA Game Studio will not only support the second generation Zunes (Zune 4/8 & Zune 80) but the first generation devices as well (Zune 30).

Access to music and Pictures

As long as the music isn't DRM protected it's available to use in the application.

Wireless support

This is probably the coolest. I'm really glad they got this included from the start.

I know this is a bit outside my normal scope. I just thought it was some pretty cool news and I didn't see them coming...

/ Johan

This morning I found the following in my inbox:

I had set my web servers running on IIS 6 to recycle if they hit 700 MB (Maximum used Memory).
Can I run a report to know how many times per day or week the W3WP got recycled.
Any suggestions please.

Thanks in anticipation

The quick answer to your question is to use performance monitor:

1. Start performance monitor
2. Add a new counter
3. Performance object: "ASP.NET v[n]"
4. Counter: "Worker Process Restarts"

So, there's the answer. But while we're on the subject I'd like to mention application restarts as well.

Applications vs. Application pools

Application restarts may lead you think that your entire application pool recycled, while in fact it was only one of the applications hosted by the application pool that restarted. If this is getting confusing, then please remember that there's a difference between application pools and application. An application pool will consist of one or several worker processes and may host one or several applications.

There are two easy ways of monitoring application restarts:

• Using performance monitor again
• Have ASP.NET write an event to the event log upon shutdown & startup. (Requires ASP.NET 2.0 or later)

Performance monitor

This is probably the easiest, though you won't get too much additional information other than the fact that an application restart occurred. Here's what you do:

1. Start performance monitor
2. Add a new counter
3. Performance object: "ASP.NET v[n]"
4. Counter: "Application Restarts"

Event log

By altering the root web.config you can get an event for every application shutdown and start up. This is a good way to get more detailed information on why the application shut down.

Open up the root web.config, (located in the %WinDir%\Microsoft.NET\Framework\v2.0.50727\CONFIG directory,) locate the healthMonitoring.rules subkey and add the following:

Now when the application exited for an application-specific reason you'll get an event like this:

Event code: 1002 
Event message: Application is shutting down. Reason: Configuration changed. 
Event time: 2/14/2008 10:00:41 AM 
Event time (UTC): 2/14/2008 9:00:41 AM 
Event ID: a1314c10a0c84222ae2d870d85308304 
Event sequence: 18 
Event occurrence: 1 
Event detail code: 50004 
 
Application information: 
    Application domain: /LM/w3svc/1/ROOT/Test-1-128474532435626182 
    Trust level: Full 
    Application Virtual Path: /Test
    Application Path: c:\inetpub\wwwroot\Test\ 
    Machine name: JOHAN

As you can see the application shut down because the configuration changed. Note that you won't get an event if you manually kill the entire application pool in IIS manager, or similar. One thing that you will get, however, is the following event each and every time the application starts up again:

Event code: 1001 
Event message: Application is starting. 
Event time: 2/14/2008 10:00:47 AM 
Event time (UTC): 2/14/2008 9:00:47 AM 
Event ID: 1f41fd3b17764330ac61804094b0abf0 
Event sequence: 1 
Event occurrence: 1 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/w3svc/1/ROOT/Test-1-128474532435626182 
    Trust level: Full 
    Application Virtual Path: /Test
    Application Path: c:\inetpub\wwwroot\Test\ 
    Machine name: JOHAN

So even if the application shut down for a reason that didn't generate an event, (IISReset, idle server, etc.) you'll at least see that for some reason it had to start up again.

/ Johan

Here's a little something I learned the other day.

If you go to the ASP.NET tab and change the ASP.NET version for an application pool this will not only reset the application pool, but the entire IIS. I was a bit surprised at first, but investigating the matter showed that there was a pretty solid architectural decision behind this.

As most "bugs" this turned out to be quite well documented once you know where to look. It's actually mentioned in the aspnet_regiis documentation.

The reason it is documented under aspnet_regiis is because this is also a rather good alternative if you wish to reconfigure an application pool, but not cause an IIS reset.

You do it by running the following two commands:

/ Johan

My colleague Tess Ferrandez just posted this excellent lab that provides you with a sample application hang. If you want to practice your skills in a controlled environment, then this is definitively a must.

/ Johan

As you may or may not have noticed I've started organizing all posts I've made regarding debugging training in a separate list on the left hand side of the page. If you're new to debugging and don't know where to begin, then at least you know where I'd recommend you to start.

I hope you'll find it useful.

Have a great weekend! / Johan

Prerequisites

This post will require some basic knowledge of windbg and the sos extension. For this I recommend looking at the following posts:

• Getting started with WinDbg - Part I
• Getting started with WinDbg - Part II

For more information on Exceptions in general and why they should be avoided I'd like to recommend this post:

• Exceptions

Introduction

I thought it was time to write another post on how to use windbg for troubleshooting. A lot of my time is spent locating exceptions in various web applications, so I thought this might be a good topic to cover. I've previously written a post specifically targeting OutOfMemoryExceptions, but I thought I should broaden the terms and make it a bit more general. There are two scenarios that are exceptionally common in my line of work:

1. Clients are reporting 2nd chance exceptions displayed on screen with the classic "Server Error"- page.
2. Performance is generally bad, and when we investigate it turns out that there are tons of exceptions being thrown every second.

In this post I'll cover how to investigate what exceptions have been thrown by an application, as well as how to use windbg and adplus to automatically gather specific information for us.

Where to start

Okay, so you have a web application that you've been monitoring and you believe it is throwing a lot of exceptions. You've taken a dump of the process and you're ready to begin the investigation. Where do you start?

!dumpallexceptions (!dae)

If your application is running under the .NET Framework 1.1 you can use the !dumpallexceptions-command (!dae) to get a list of all the exceptions still on the heap. Now, remember that an exception is a managed object, so they will eventually be garbage collected just like everything else. This means that when looking at the heap for exceptions you will only get the exceptions still in memory, not every exception thrown by the application since startup.

Anyway, if you run !dae you'll get a list of exceptions that looks like this:

As you can see the !dae-command lists all exception types found on the heap. If possible it also gives us a callstack for each exception. Please note, however, that this doesn't mean that the call stack is more or less the same for all exceptions. In the sample above you might have ~20 different callstacks leading to the 136 NullReferenceExceptions you see.

Unfortunately the !dae command is not available in version 2.0 of sos.dll. Still, it's quite easy to get (more or less) the same result by using the !dumpheap command. If we just type "!dumpheap -type Exception" we'll get a list of all objects with the string "Exception" in their class name. This is almost as good.

As you can see, this gives us almost the same information except for the callstacks.

Knowing what to ignore

When analyzing data it is always good to know how to filter the information.

The ever-present exceptions

There are a three exceptions that are created as soon as the worker process starts. This means that you will always see them on the heap even if they haven't been thrown at all.:

• System.ExecutionEngineException
• System.StackOverflowException
• System.OutOfMemoryException

So why are they created if we haven't thrown them? - Any guesses?

The answer is quite simple: If you run into a situation where you need to throw any of these exceptions you will probably be in a state where you can't create them. For example, you've run out of memory and are no longer able to allocate even the tiniest string. How would you then be able to allocate enough memory to create a new exception?

So, provided that there's still only one of each on the heap, you can most likely ignore these three exceptions. When it comes to ExecutionEngineExceptions and OutOfMemoryExceptions you will probably have a pretty good idea that this is what you're looking for, and finding a StackOverflowException isn't that hard. If you run !clrstack and find a callstack of 200+ lines you can be more or less certain that this is your problem.

System.Threading.ThreadAbortException

Usually when you see a ThreadAbortExceptions it is because you've called Response.Redirect.

Whenever you call Response.Redirect, this will also result in a call to Response.End. This will terminate the thread prematurely, resulting in a System.Threading.ThreadAbortException. See the callstack below for an example.

Obviously I'm not saying you should discard all System.Threading.ThreadAbortExceptions as irrelevant. Even if you have no reason to believe that ThreadAbortExceptions are a major concern it's always a good idea to investigate a few of them. Take a minute or two to confirm that there is an underlying call to Response.End caused by a Response.Redirect. Once you think that you have enough statistical data to imply that the ThreadAbortExceptions are caused by Redirects you can move on.

Examining the Exceptions

Okay, so say we want to look at the callstacks for the System.Data.SqlClient.SqlException, well first of all we need the address for it. As you might remember, this is easily obtained by using !dumpheap without the -stat option.

Now we have the address for the exception. In order to investigate the exception we could use !dumpobject, but there is another command I want to use first.

!printexception (!pe)

Running the !printexception command on the address of an exception will give us some neat information on the exception in question. Here's the result of running !printexception on the SqlException:

This is good stuff. The command was even able to generate a callstack for us. (This may not always be the case, since the callstack may very well have gone out of scope.)

!dumpobject (!do) still has its uses

I wouldn't say that !printexception is a complete replacement for !dumpobject when it comes to examining exceptions. !Printexception will fit the exception into a standard template, and since some exceptions may contain more data than others we sometimes want to use !dumpobject as well. The SqlException has a property called _errors that contains a System.Data.SqlClient.SqlErrorCollection that we might want to look at. This is not in the listing above, so we need to use !dumpobject to look at it.

There we have it. Now we can continue using !dumpobject to investigate it even further if we wish.

Inner exceptions

If we take a look at one of the HttpUnhandledExceptions we find that it has an inner exception. It is even nice enough to let us know how to find out more about it.

This means that the System.NullReferenceException mentioned lead to the HttpUnhandledException we're currently investigating. So if we want to find the root cause we'll need to investigate the inner exception as well.

Extra credit

If you've looked at my "Advanced commands"-post a while back you saw some examples of the .foreach command. This is a great command to use, for example if you want to see the callstack for all System.ArgumentNullExceptions. Instead of manually iterating through all the exceptions we can now dump them all at once, check their callstacks, etc.

Well this post was even longer than usual.

I hope you found it of value, and I'll gladly listen to any comments, feedback or wishes on future topics you might have.

/ Johan

Problem

Using the following VB-syntax in a DataRepeater will no longer work after applying .NET Framework 2.0 Service Pack 1 in a medium-trust environment.

Resolution

Use Eval-method, as recommended in all Microsoft documentation. Using explicit casting would work as well, but you'd have to re-write every page if you decide to change your data layer.

Repro

Create an ASP.NET page with a repeater using a simple OleDb database as it's DataSource.

If you run this application in full trust, you should encounter no problems.

Now, set the trust level to medium. (You'll have to modify the settings and give it access to OleDbConnections.) When you run it you'll get a Security Exception:

Cause and resolution

Visual Basic has always been strong on implicit casting. For example, this code is completely valid:

So, in the sample at the top, Container.DataItem is merely an Object. VB then casts this to a System.Data.Common.DbDataRecord object and we're able to retrieve the "data"-property. This is no longer working after Service Pack 1.

C, in all its variations wouldn't be caught dead accepting a syntax like String = Integer, so the code at the top would never work. Instead you'd have to explicitly cast the object to the appropriate type:

If you change the VB code so that it uses the VB equivalent: CType you will resolve the problem.

This has never been the recommended approach

So, is the resolution to use explicit casting? Well, not really. If you look at the C# example above you might notice a potential problem. If we change our data layer we would have to go through every single page and change the explicit casting. This is one of the reasons why we've always recommended using the Databinder.Eval method for VB and C# alike, so if you've followed the design guidelines your code should look like this:

Actually, as of Framework 2.0 it's enough to write Eval("data").

So, if you review the documentation on msdn for Framework 1.1, Framework 2.0, knowledge base articles, etc. you'll see that we consistently recommend using the Eval method.

/ Johan 

Did you know you can build your own advanced commands using for each, if, etc? The complete list of control tokens are:

• .if
• .else
• .elseif
• .foreach
• .for
• .while
• .do
• .break
• .continue
• .catch
• .leave
• .printf
• .block

Using these command tokes you can send quite advanced instructions to the debugger that not only will make your job a lot easier, but also impress your manager immensely. :)

.foreach

Let's begin with an easy example. Imagine you want to investigate all strings on the heap that are 6500 bytes or more. To list them you'd simply type !dumpheap -type System.String -min 6500. This will give you the following information:

So far, so good. The problem is that in order to investigate each string you'd have to run !dumpobject (!do) on every address. This might be acceptable now that we're only dealing with 6 strings, but what if it were 25, or 100? I don't know if you're aware of this, but if you pass the -short argument to !dumpheap it will give you the minimum information (just the addresses of the objects in question):

Now, let's use this information in a .foreach-clause: