Exception and Debug Event, the feedback from OS
This section will firstly brief exception related tech, and then use example to demonstrate how to use exception to troubleshoot effectively.
Exception Brief
Exception is a mechanism to control code’s execution flow. In normal situation, the code executes consequently, like the following:
*p=11;
printf(“%d”,*p);
It should print 11. But how about if p points to an invalid memory address? Then the line to assign value to *p will trigger access violation exception, and the following line to print may not execute any more.
For applications, if the behavior does not follow the expectation, exception is likely a direct cause because this is the most obvious and common way that changes the execution flow. In most cases, troubleshooting problem is just the same meaning of troubleshooting exception.
In original Chinese version, I discuss how the OS plays an important role on exception handling and dispatching. Also I brief how different programming language leverages the SEH to support the exception handing mechanism. I will skip such introduction here because the following two articles cover them all:
A Crash Course on the Depths of Win32™ Structured Exception Handling
http://www.microsoft.com/msj/0197/Exception/Exception.aspx
RaiseException
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/raiseexception.asp
Case study, how to let C++ dump the callstack as the C#
For the application created by C# or Java, when exception occurs, they are able to dump the call stack where the exception comes from. However, for C++, we have to use debugger to get the callstack. Now the customer wants to achieve stack dump in C++. Any good idea?
My solution is to use SEH, due to that local variable’s destructor will be executed during stack unwind when exception occurs. The sample code worked fine in VC6+Win2k3 platform. However, when I retry the sample, the same code behaves strangely in VC2005 + Win2k3 SP1. If I compile in debug mode, it works fine. However, in release mode, the application quits silently. For the whole story, I saved in my MSN blog (English), please refer to:
SEH,DEP, Compiler,FS:[0] and PE format
http://eparg.spaces.msn.com/blog/cns!59BFC22C0E7E1A76!712.entry
Case study, Why Dr. Watson cannot save the dump file.
Problem Description:
The customer reports their VC application crashes randomly. To obtain detailed info, the customer registers Dr. Watson so that when exception occurs next time, we can get the dump file. However, when the problem reoccurs, Dr. Watson saves nothing.
Background Info:
In Chinese version, I provided brief info about what dump file is, and what info we can find in dump. Related info can be found at:
Description of the Dr. Watson for Windows (Drwtsn32.exe) Tool
http://support.microsoft.com/?id=308538
Specifying the Debugger for Unhandled User Mode Exceptions
http://support.microsoft.com/?id=121434
INFO: Choosing the Debugger That the System Will Spawn
http://support.microsoft.com/?id=103861
Generally speaking, by setting the AeDebug registry key, we can lunch the debugger when application crashes. If we choose Dr. Watson as the debugger, the default behavior is generating the dump file.
Problem Analysis:
Back to the case, the customer fails get the dump file, possible causes:
1. The Dr. Watson’s bug. It works abnormally.
2. The customer’s application does not crash, it just exits like calling ExitProcess.
To perform test against point 1, I provided the following sample code for testing:
int *p=0;
*p=0;
With above code, Dr. Watson captured the dump file successfully on the customer side. So Dr. Watson works fine. It seems that the crash exclaimed by the customer is not really caused by unhandled exception. Maybe the customer calls ExitProcess unexpectedly. Thus during information capturing, we should not limited in unhandled exception. What we need to check is how the process disappears, maybe normal quit, maybe unhandled exception.
One possible way to figure out is to run the application in windbg. However, manual operation is troublesome. It would be nice if there is some automatic way. Windows provides a registry, which allows an application starts under debugger. With this setting, when the specified process starts, OS starts the debugger firstly, and pass in the target process and command line to debugger, then debugger starts the target process to debug. This option is very useful especially when we cannot start the process manually, like Windows Service, which starts ahead the user logon:
How to debug Windows services
http://support.microsoft.com/?kbid=824344
Some malicious software uses this way to attach silent process. This method is also called IFEO (Image File Execution Option) hijacking in China.
In windbg folder, there is a script called adplus.vbs. We can use it to launch windbg to obtain the dump file. Here we will use the script:
How to use ADPlus to troubleshoot "hangs" and "crashes"
http://support.microsoft.com/kb/286350/EN-US/
Use adplus /? to obtain detailed info.
The Actions:
With above analysis, the detailed actions are:
1. In customer’s machine, create the key named by the problematic process under Image File Execution Options
2. Under the key, create a string value called Debugger.
3. Set the value to Debugger= C:\Debuggers\autodump.bat
4. Edit the C:\Debuggers\autodump.bat as the following:
cscript.exe C:\Debuggers\adplus.vbs -crash -o C:\dumps -quiet -sc %1
Based on above setting, when the application starts, the OS launches cscript.exe to execute the adplus.vbs script. The –sc switch in adplus.vbs specify the target process path, -crash means we will monitor for application’s quit, -o specifies the dump output folder, -quiet disables prompt. We can use notepad.exe as test to check if dump is generated when notepad.exe quits.
Based on above setting, when the problem reoccurs, we get two dump files in c:\dumps folder, called:
PID-0__Spawned0__1st_chance_Process_Shut_Down__full_178C_DateTime_0928.dmp
PID-0__Spawned0__2nd_chance_CPlusPlusEH__full_178C_2006-06-21_DateTime_0928.dmp
Pay attention to the second filename. The name indicates the 2nd chance C++ exception does happen. Open the dump in windbg, check the callstack, it shows that the customer throws some C++ exception in code, but forgets to capture that. By adding corresponding catch block, the issue gets fixed.
The solution is nice, but why Dr. Watson cannot get the dump?
The Dr. Watson’s behavior still confuses me. Since it is unhandled exception, why Dr.Watson cannot capture the dump file? Firstly I created two different applications to double verify the behavior of Dr. Watson:
int _tmain(int argc, _TCHAR* argv[])
{
throw 1;
return 0;
}
int _tmain(int argc, _TCHAR* argv[])
{
int *p=0;
*p=0;
return 0;
}
For the first one, Dr. Watson does not save the dump. For the second, Dr. Watson saves the dump. It looks like the behavior is related to the exception type.
Recall the detailed crash behavior for above two applications when the Auto key is set to 0 under AeDebug. On my side, the message boxes for crash are:
---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Error!
Program: d:\xiongli\today\exceptioninject\debug\exceptioninject.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------
---------------------------
exceptioninject.exe - Application Error
---------------------------
The instruction at "0x00411908" referenced memory at "0x00000000". The memory could not be "written".
Click on OK to terminate the program
Click on CANCEL to debug the program
---------------------------
OK Cancel
---------------------------
The behaviors are totally different! And the behavior is related to the compilation mode.
SetUnhandledExceptionFilter API is used to modify the default unhandled exception handler. Here, when C++ initialize the CRT, it passes CRT’s implementation (msvcrt!CxxUnhandledExceptionFilter). When unhandled exception occurs, the function checks the exception code. If it is a C++ exception, it shows up the first dialog, otherwise it bypass it to the default handler (ernel32!UnhandledExceptionFilter) provided by the OS. For the 1st situation, the callstack is:
USER32!MessageBoxA
MSVCR80D!__crtMessageBoxA
MSVCR80D!__crtMessageWindowA
MSVCR80D!_VCrtDbgReportA
MSVCR80D!_CrtDbgReportV
MSVCR80D!_CrtDbgReport
MSVCR80D!_NMSG_WRITE
MSVCR80D!abort
MSVCR80D!terminate
MSVCR80D!__CxxUnhandledExceptionFilter
kernel32!UnhandledExceptionFilter
MSVCR80D!_XcptFilter
For the second, it is
ntdll!KiFastSystemCallRet
ntdll!ZwRaiseHardError+0xc
kernel32!UnhandledExceptionFilter+0x4b4
release_crash!_XcptFilter+0x2e
release_crash!mainCRTStartup+0x1aa
release_crash!_except_handler3+0x61
ntdll!ExecuteHandler2+0x26
ntdll!ExecuteHandler+0x24
ntdll!KiUserExceptionDispatcher+0xe
release_crash!main+0x28
release_crash!mainCRTStartup+0x170
kernel32!BaseProcessStart+0x23
For detailed info, please refer to:
SetUnhandledExceptionFilter
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/setunhandledexceptionfilter.asp
UnhandledExceptionFilter
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/unhandledexceptionfilter.asp
Does above analysis help explain the Dr. Watson’s behavior? To be honest, I do not think so. I think it is due to Dr. Watons’s special handling on different exception types. The detailed research can be found at:
http://eparg.spaces.msn.com/blog/cns!59BFC22C0E7E1A76!1213.entry
Debug Event – communication between the OS and the debugger
Notification, also called Debug Event, it is a mechanism for OS to notify debugger when some thing happens. Similar as exception handing, OS dispatches the notification when some thing happens if the debugger is attached. Unlike exception, the notification can only be monitored by the debugger, not the target process. Also, there is no 1st chance and 2nd chance differences. In windbg’s help file, all the notifications are listed in the Controlling Exceptions and Events topic. Common notifications are DLL loading, unloading, thread creation and existing.
With exception and notification, we can capture the key for issue.
Case study, VB6’s version.
Customer’s VB6 application is not able to open data file created by Access 2003 in developer machine. It works fine for data created by Access 97. In other machines, both Access 2003 and 97 work fine.
They way to think is direct. Since it occurs in a specified machine, it means the issue is about the environment, not the code. Since it is about Access version, it should be related to the DAO’s version. By checking the modules loaded by the EXE, I found dao350.dll was loaded instead of dao360.dll. The next step is to figure out why dao350.dll gets loaded instead of dao360.
DAO is a COM component. It is likely created by COM API. A simple way is to trace the execution of the COM API like CoCreateInstanceEx with wt command, like I did in ShellExecute case. However, if we really try that, the wt command may execute for a whole day. It would be better if we can find a more workable way. Since we will trace until the library loading, why not set breakpoint at LoadLibrary to check how the dao350.dll gets loaded?
It is a very good way to set breakpoint on LoadLibrary because:
1. DLL loading is not necessary through LoadLibrary. Native API like ntdll!LdrLoadDLL may load the module directly.
2. If there are hundreds of DLLs to be loaded, breaking into LoadLibrary is troublesome, even if we can set conditional breakpoint to filder.
The better way is to leverage notification. During module load, OS sends notification to the debugger. In Windbg, we can use wide char to match and filter the DLL filename. It is easy to operate. Firstly, use “sxe ld:dao*.dll” command intercept the module load notification. When the filename is dao*.dll, the debugger breaks. (For windbg detailed usage, we will cover in next sections). The result in debugger is:
0:008> sxe ld:dao*.dll
ModLoad: 1b740000 1b7c8000 C:\Program Files\Common Files\Microsoft Shared\DAO\DAO360.DLL
eax=00000001 ebx=00000000 ecx=0013e301 edx=00000000 esi=7ffdf000 edi=20000000
eip=7c82ed54 esp=0013e300 ebp=0013e344 iopl=0 nv up ei pl zr na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c82ed54 c3 ret
ntdll!KiFastSystemCallRet
ntdll!NtMapViewOfSection
ntdll!LdrpMapViewOfDllSection
ntdll!LdrpMapDll
ntdll!LdrpLoadDll
ntdll!LdrLoadDll
0013e9c4 776ab4d0 0013ea40 00000000 00000008 kernel32!LoadLibraryExW
ole32!CClassCache::CDllPathEntry::LoadDll
ole32!CClassCache::CDllPathEntry::Create_rl
ole32!CClassCache::CClassEntry::CreateDllClassEntry_rl
ole32!CClassCache::GetClassObjectActivator
ole32!CClassCache::GetClassObject
ole32!CServerContextActivator::GetClassObject
ole32!ActivationPropertiesIn::DelegateGetClassObject
ole32!CApartmentActivator::GetClassObject
ole32!CProcessActivator::GCOCallback
ole32!CProcessActivator::AttemptActivation
ole32!CProcessActivator::ActivateByContext
ole32!CProcessActivator::GetClassObject
ole32!ActivationPropertiesIn::DelegateGetClassObject
ole32!CClientContextActivator::GetClassObject
ole32!ActivationPropertiesIn::DelegateGetClassObject
ole32!ICoGetClassObject
ole32!CComActivator::DoGetClassObject
ole32!CoGetClassObject
VB6!VBCoGetClassObject
VB6!_DBErrCreateDao36DBEngine
By checking the parameter of the LoadLibraryExW, it shows
0:000> du 0013ea40
0013ea40 "C:\Program Files\Common Files\Mi"
0013ea80 "crosoft Shared\DAO\DAO360.DLL"
With above information, we see:
1. DAO360 is not created by CoCreateInstanceEx. Instead it is created by CoGetClassObject. If we trace CoCreateInstanceEx, it wastes time.
2. COM invocation starts from VB6!_DBErrCreateDao36DBEngine function. We should check the function in detail.
With previous DLL hell’s lesson, here the first thing is to check VB6.EXE’s version since the function resides in VB6. Compared with normal condition, the workable module version is 6.00.9782, while the problematic one is 6.00.8176. By installation of VS6 SP6, the issue gets fixed.
Discussions:
(In Chinese version, I discussed how to analysis the dump even if the dump is not captured at the first place when exception happened. I have to skip here.)
Exit proactively for unhandled exception
In some situation, the developer exits the application proactively when unhandled exception occurs, instead of waiting for the OS to terminate it. COM+, ASP.NET use this kind of tech. A Chinese C2C software called taobao wangwang (also named ali wangwang) uses this kind of tech too. The benefits are:
1. We can define the UI for the crash.
2. We can save the unhandled exception info for postpone analysis.
3. To avoid the interference of the debugger, guarantee the immediate recycle, and try the necessary rescue operation like restarting the process.
It is easy to implement. One way is to use the __try and __except clause. The other way is to use SetUnhandledExceptionFilter API. For the study of taobao wangwang, please refer to:
http://eparg.spaces.msn.com/blog/cns!59BFC22C0E7E1A76!817.entry (Chinese)
Based on my analysis, taobao uses SetUnhandledExceptionFilter to capture unhandled exception, and use MiniDumpWriteDump API to capture the dump proactively.
With this tech, the debugger is hard to get the dump for crash directly. Some additional configuration and windbg command is necessary
How To Obtain a Userdump When COM+ Failfasts
http://support.microsoft.com/?id=287643
How to find the faulting stack in a process dump file that COM+ obtains
http://support.microsoft.com/?id=317317
How to troubleshoot UnhandledExceptionFilter
Based on MSDN, UnahandledExceptionFilter will be invoked only if the debugger is not attached. Thus we can use UnahandledExceptionFilter to bypass the trace of debugger, to protect some sensitive code. To avoid debugger’s check, there are two ways at least:
1. The target uses IsDebuggerPresent API to check if the debugger is attached. If so, it refuses to execute the sensitive code.
2. Put the sensitive code to a function, and register the function as UnHandledExceptionFilter. To execute the sensitive code, just trigger an exception manually. Due to the design of exception handling, it avoids the debugger’s trace.
For the first way is easy to by pass. Look at the implementation of IsDebuggerPresent:
:000> uf kernel32!IsDebuggerPresent
kernel32!IsDebuggerPresent:
281 77e64860 64a118000000 mov eax,fs:[00000018]
282 77e64866 8b4030 mov eax,[eax+0x30]
282 77e64869 0fb64002 movzx eax,byte ptr [eax+0x2]
283 77e6486d c3 ret
IsDebuggerPresent checks the flag in FS register. (FS:[18]]:30 saves PEB of current process). In debugger, we can change any of the register easily. Here we just need to change value of [[FS:[18]]:30]:2 to 0 to cheat IsDebuggerPresent to return false.
For the second way, changing [[FS:[18]]:30]:2 does not work because the judgment is based on the result of a kernel call. However, it does not mean impossible. Kwan Kyun Kim provides a way to cheat:
How to debug UnhandleExceptionHandler
http://eparg.spaces.msn.com/blog/cns!59BFC22C0E7E1A76!1208.entry
Next I will discuss memory, including Heap, Stack, and the lovely heap corruption and pageheap.