There are situations when we want to look closely at a process that is running in production environment – when a crash happens or when it just behaves badly (consumes too much memory, too much CPU, it hangs etc). There are multiple tools that can tell us what is wrong:
- Event log (Window Key + Run -> type Eventvwr). Here we can see warnings and errors related to our app. This will shed some light in some of the situations, but quite often there are no entries that can help.
- Performance monitor (Window Key + Run -> perfmon). We can look at the perf counters that are relevant to our application and the issue we are trying to investigate. In general, it’s a good idea to look at the perf counters to know what the processes are doing, but it won’t give us much insight on why apps crashed.
- Sysinternal tools. These contain file & disk utilities, process utilities, networking utilities and much more. You can read more about it on Mark's blog.
- Debuggers (Visual Studio, windbg etc). While developing and testing, the debugger is our best friend. In production however, debugging might not be just as easy. One way to investigate issues is to take dumps when something unexpected happens.
When the process is running under debugger – it has cdb or windbg attached, we can use the .dump command. Most of the times we want a mini-dump. For a managed process, we need to take a full minidump in order to be able to load sos and debug the managed structures. Use the command .dump /ma dumpName.dmp – it will add full memory data, handle data, unloaded module information, basic memory information, and thread time information to the mini dump. If you want to generate multiple dumps with unique names, a simple trick is to add /u switch, that appends the data, time and PID of the process to the dump name.
If the process is not running under debugger, we can:
- Attach the debugger – using <debugger> –p PID or <debugger> -pn ProcessName, where <debugger> is cdb or windbg, and then do what I mentioned above. For example, to attach the debugger to the worker process (and you have just one worker process running), use windbg –pn w3wp.exe. If you have multiple processes with the same name, you need to specify the PID of the process you want to attach (Eg windbg –p 1232). After attaching, take the dump and then run .detach to detach without killing the process. If you don’t want to keep running the process, use .q (ends the debugging session and closes the target application).
- Use adplus in hang mode. This will take a snapshot of the process right now (not waiting for a hang or anything), then detaches and leaves the process running: adplus -hang -pn processname.exe.
- If you want to take a dump later, when a crash will occur, use adplus in crash mode: adplus –crash -pn processname.exe. This attaches cdb.exe to the process in invasive mode, and waits for the crash – If it happens, it will take a full mini dump. It will also create mini dumps for access violation exceptions.
- Use DebugDiag. DebugDiag it’s really easy to use – it has a graphic interface and we can manually take dumps by right-clicking the process in the processes view and by selecting Create Full Userdump.
With DebugDiag, we can also create crash rules or hag rules (for IIS).
Take a look at this post with more details about DebugDiag and AdPlus (including some configuration files for AdPlus that let you control the behavior better).
There are some registry keys we can set to change the processes behavior:
- Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\
o GCFailFastOnOOM – This DWORD registry key causes failfast on out of memory conditions when it is set to a non-zero value:
§ 1: an event message is logged in the application event log.
§ 2: you experience a break in the debugger (app calls DebugBreak()).
§ 4: an event message is logged that includes the size of the allocation that failed and the number of bytes in use, like “Failed to allocate x bytes with y bytes in use. Process will be Terminated.”
For more information, see this post on how to debug ASP.NET Out of memory exceptions using DebugDiag and this registry key.
o DbgJITDebugLaunchSetting - controls whether a debugger should be started and attached to the process that throws an uncaught exception. See Enabling JIT-attach Debugging article on MSDN.
o DbgManagedDebugger – what debugger is used to debug managed code, if the previous setting shows that a debugger will be started
- HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug – We can specify the Auto value and Debugger key that tells what debugger to be started in case of failure – full path and command arguments. The default value is “drwtsn32 -p %ld -e %ld –g”, that will start Dr Watson. To start Windbg instead, set Auto = 1 and Debugger="C:\Program Files\Debugging Tools for Windows\windbg.exe" -p %ld -e %ld
Later on, I will write a post about Windows Error Reporting, and how we can use it to take dumps and create reports.