Web server Administrators have been using the Debug Diagnostic 1.1 to troubleshoot crash\hang issues on IIS 5 and 6. The Debug Diagnostic tool needs no introduction. It is a great tool and it provides you a lot of flexibility in collecting crash dumps for any user mode process running on a server.  But as of now, the Debug Diagnostic tool is *not* compatible with Windows Server 2008 and it is recommended that you not use it on a Windows 2008 server because you may get unexpected results. The good news is that the Debug Diagnostic team is currently working on releasing a version that is compatible with Windows 2008, but what to do until that time?

The answer is to leverage the powerful Windows Error Reporting (WER) feature that comes with Windows 2008. You can use WER on a Windows 2008 server to collect a crash dump of any process without any other tools.

The good news is that Windows Error Reporting is enabled by default on every Windows 2008 server and takes dumps of all application crashes on the server. If you have experienced crashes of an IIS worker process, all you have to do is check if there is a dump generated for you already. To do that, please follow these steps.

  1. Click on Start-> Run and type WERCON
  2. This will open up the “Problems Reports and Solutions” dialog box and it will look like this.

    WerScreen1

  3. From here, just click on “View Problem History” and see if you can find any crashes for “IIS Worker Process”

    ViewProblemHistory

  4. You can select the particular IIS Worker Process crash entry and right click on it and choose “View Problem details”. This will open up a window like this which will give you an overall idea of the crash.

    ViewProblemDetails
  5. To get the dump files for this crash, just right click on “View a temporary copy of these files” and this will copy the relevant .hdmp , .mdmp and version.txt to your local temporary folder. These are the dump files which you can either analyze using WinDBG or send to Microsoft for further analysis.

Windows Error Reporting introduced a new feature (in Windows Vista and above) which lets you create a LocalDumps registry key under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting and then under that you can create a subkey with the name of the executable that you want to take dumps for. This approach had a few limitations as of now. It turned out that :

  1. This feature works only if HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Auto is set to 0. If Auto is set to 1, the control is given to the debugger specified in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger setting and the debugger decides what to do with the unhandled exception and WER is not given a chance to handle the exception.
  2. The .net framework version 3.5 and below do not recognize this registry key. This means that if your process has some managed code and if some unhandled exception happens while managed code execution (the exception may be native or a CLR exception), the LocalDumps method will not get you dumps for the process. The functionality inside the .net CLR to read the LocalDumps registry key is added in .net framework 4.0. Because of this limitation, the LocalDump feature becomes less useful for an IIS worker process because most of the time an IIS worker process is running ASP.NET applications or some managed code.

What if WER is not taking the dumps for the process of interest

  1. If you notice that dumps are not getting generated, you can check the general WER settings on your machine. These settings can be configured at HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting. The complete list is available at http://msdn.microsoft.com/en-us/library/bb513638(VS.85).aspx. I will just include the most relevant ones in regard to taking dumps :

    Disabled Should be 0. If this is set to 1, WER will be disabled completely
    ForceQueue Default value is 0 but you can set it to 1 to ensure that dumps are generated on every crash
    DisableQueue Should be 0
    DisableArchive Should be 0
    CorporateWERServer If you want to keep the data locally on the server and don't want the dumps to be moved to the WER server, you can set this setting to a non-existing server in your domain. This will ensure that the data is stored locally on the server.
    MaxQueueCount Default is 50, You can increase this to a higher number to increase the total number of dumps that will be saved to the ReportQueue(For more information on ReportQueue, refer to this article)

  2. Another important thing to be aware of while using Windows Error Reporting is that a lot of Windows Error Reporting settings can be configured via group policy. To look at the WER settings on your server, open up Local Group Policy Editor (GPEDIT.MSC) and you will be able to see these settings under Computer Configuration –> Administrative Templates –> Windows Components –> Windows Error Reporting. This is how it looks.

gp

The relevant WER settings defined in group policy will override the local settings. Most of these group policies don't apply to Windows Vista and above. The ones that do apply are

  1. Disable Windows Error Reporting

(Under Advanced Error Reporting Settings )

  1. Configure Report Archive
  2. Configure Corporate Windows Error Reporting
  3. List of applications to be excluded
  4. Configure Report Queue

There may be specific reasons why the WER settings are disabled on domain and it is recommended to get in touch with an Administrator before making any changes to the above policies. Explaining these group policy entries would result in a discussion outside of the scope of this post, so I recommend looking at the MSDN documentation for the above policy settings for more information on them.

Windows Error Reporting and ADPLUS

Administrators who are aware of Debug Diagnostic tool may also know about another tool called ADPLUS.VBS that ships with the Debugging Tools For Windows. Though ADPLUS.VBS tool has a -crash switch that lets you attach a debugger to a process in CRASH mode, it is not the best option to use while troubleshooting worker process crashes because ADPLUS doesn’t have the ability to auto-attach to a new process. You can only attach ADPLUS to a process which is already running on the machine. This becomes a big limitation for an IIS worker process because a worker process may recycle or shut-down based on what is configured in the application pool settings. Another problem with ADPLUS is that it starts a new window for cdb.exe and someone may accidently close this window which may end up killing the worker process itself.

You may have noticed that WER just lets you generate a crash dump for a process when it crashes and doesn’t provide you the flexibility that Debug Diagnostic provides you though (e.g. taking dump on specific exceptions, specific breakpoints, events etc.). If you want to dump your process on a particular exception on Windows 2008 and above, you still have to use the ADPLUS tool. The discussion how to use ADPLUS to troubleshoot crashes for a process is out of the scope of this article.

ProcDump

This is one of the latest additions in the Sysinternals suite of tools. ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use) and unhandled exception monitoring. It also can serve as a general process dump utility that you can embed in other scripts. I won’t spent too much time talking about PROCDUMP in this blog post because we already have another post which talks about this tool in detail - http://blogs.msdn.com/webtopics/archive/2009/08/08/using-procdump-exe-to-monitor-w3wp-exe-for-cpu-spikes.aspx

References