Rubato and Chord

Reiley's technical blog

July, 2011

  • Rubato and Chord

    A Debugging Approach to IFEO


    IFEO (Image File Execution Options) is a feature provided by the NT based operating system. It can be helpful when you are trying to debug at the very beginning of an application launch. A few people also taked about IFEO on MSDN Blogs:

    1. Image File Execution Options by Junfeng.
    2. Inside 'Image File Execution Options' debugging by Gregg.
    3. Beware the Image File Execution Options key by Raymond.
    4. IFEO and Managed-debugging by Mike.

    Debugger (REG_SZ)

    Let's begin with the following demo (I'm using 32bit Win7, you might see different things if you are using other version of Windows):

    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
    #define IfFalseRet(c) do{if(!(c)){return dwLastError = GetLastError();}}while(0)
    STARTUPINFO g_startupInfo = {sizeof(STARTUPINFO)};
    TCHAR g_szCommandLine[] = TEXT("notepad.exe");
    int WINAPI ExeEntry(void)
      DWORD dwLastError = ERROR_SUCCESS;
      PROCESS_INFORMATION processInformation;
      IfFalseRet(CreateProcess(NULL, g_szCommandLine, NULL, NULL, FALSE, 0,
                               NULL, NULL, &g_startupInfo, &processInformation));
      return dwLastError;

    To compile the source code, you may use Visual Studio Command Prompt (x86):

    cl.exe createprocess.c /D UNICODE /GS- /Od /Oy /link /ENTRY:ExeEntry /NODEFAULTLIB /OPT:ICF /OPT:REF /RELEASE /SUBSYSTEM:CONSOLE kernel32.lib

    Now with the Process Monitor opened, launch the demo, and you should be able to find the following record captured by Process Monitor:

    00:01:17.8231352 createprocess.exe 1320 RegOpenKey HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe NAME NOT FOUND Desired Access: Query Value, Enumerate Sub Keys

    Let's create an IFEO entry in the registry:

    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe]
    "Debugger"="\"C:\\Program Files\\Windows NT\\Accessories\\wordpad.exe\""

    After the IFEO entry got imported to the registry, launch the demo again and you will see notepad.exe was replaced by wordpad.exe, with the string "notepad.exe" passed in as an argument.

    00:24:32.3322232 createprocess.exe 4668 RegQueryValue HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\New Key #1\Debugger SUCCESS Type: REG_SZ, Length: 108, Data: "C:\Program Files\Windows NT\Accessories\wordpad.exe"

    00:24:32.3343642 createprocess.exe 4668 Process Create C:\Program Files\Windows NT\Accessories\wordpad.exe SUCCESS PID: 2724, Command line: "C:\Program Files\Windows NT\Accessories\wordpad.exe" notepad.exe

    Let's launch the demo application from WinDBG:

    windbg.exe createprocess.exe

    0:000> x *!*Create*Process*
    767007a2          kernel32!CreateProcessInternalW
    766b2082          kernel32!CreateProcessA
    766b204d          kernel32!CreateProcessW
    7670c89c          kernel32!CreateProcessInternalA
    77d756a8          ntdll!NtCreateProcessEx
    77d75698          ntdll!NtCreateProcess
    77d889fb          ntdll!RtlpCreateProcessRegistryInfo
    77d75698          ntdll!ZwCreateProcess
    77d756a8          ntdll!ZwCreateProcessEx
    77d56bdf          ntdll!RtlpCreateUserProcess
    77d56b71          ntdll!RtlCreateUserProcess
    77dd1072          ntdll!RtlCreateProcessReflection
    77d75778          ntdll!ZwCreateUserProcess
    77dd988a          ntdll!RtlCreateProcessParameters
    77d75778          ntdll!NtCreateUserProcess
    77d96ee9          ntdll!RtlCreateProcessParametersEx

    0:000> bp kernel32!CreateProcessInternalW

    0:000> g
    Breakpoint 0 hit

    0:000> k3
    ChildEBP RetAddr
    0018f7a8 7670c9bf kernel32!CreateProcessInternalW
    0018f888 766b20ae kernel32!CreateProcessInternalA+0x2f8
    0018f8c0 00d71068 kernel32!CreateProcessA+0x2c

    0:000> ddu esp L6
    0018f7ac  7670c9bf "???..?"
    0018f7b0  00000000
    0018f7b4  00000000
    0018f7b8  00383a08 "notepad.exe"
    0018f7bc  00000000
    0018f7c0  00000000

    Continue tracing, eventually we will reach this point:

    ChildEBP RetAddr 
    001af248 77d75784 ntdll!KiFastSystemCall
    001af24c 76700eff ntdll!NtCreateUserProcess+0xc
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xe75
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    0:000> bp 77d75784
    0:000> g
    Breakpoint 1 hit

    From Process Monitor we can trace the IFEO registry key access from kernel mode. On my Win7 machine, it showed three operations (RegOpenKey, RegQueryValue, RegQueryValue), this is because IFEO was moved into kernel mode since Longhorn. On Windows XP and Windows Server 2003, IFEO is a pure user mode action, which means you can easily bypass it without having to write a kernel mode driver. Another thing is, kernel mode IFEO doesn't have the bitness problem, as it would never read from Wow6432Node.

    Continue tracing until we have the following call stacks:

    ChildEBP RetAddr
    0020efe4 77d9ce6f ntdll!RtlQueryImageFileExecutionOptions
    0020f008 76704ab1 ntdll!LdrQueryImageFileExecutionOptions+0x1e
    0020f034 76702abb kernel32!BasepGetDisableLocalOverrideConfig+0x33
    0020f0ac 7670119b kernel32!BasepConstructSxsCreateProcessMessage+0x8e
    0020f728 7670c9bf kernel32!CreateProcessInternalW+0x16b1
    0020f808 766b20ae kernel32!CreateProcessInternalA+0x2f8
    0020f840 01371068 kernel32!CreateProcessA+0x2c

    ChildEBP RetAddr
    001af264 7672d529 kernel32!BuildSubSysCommandLine
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xf87
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    0:000> ddu esp L6
    001af268  7672d529 "??????"
    001af26c  00000003
    001af270  00234348 ""C:\Program Files\Windows NT\Accessories\wordpad.exe""
    001af274  00000000
    001af278  00233a08 "notepad.exe"
    001af27c  001af4d8 ".."

    Continue tracing:

    ChildEBP RetAddr 
    001af1ac 76701693 ntdll!RtlCreateProcessParametersEx
    001af258 76700e90 kernel32!BasepCreateProcessParameters+0x148
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xe06
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    0:000> dS poi(esp+8)
    00283cf8  "C:\Program Files\Windows NT\Acce"
    00283d38  "ssories\wordpad.exe"

    Continue tracing:

    ChildEBP RetAddr 
    001af24c 76700eff ntdll!NtCreateUserProcess
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xe75
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    Now the user mode IFEO logic looks clear to us, let's restart WinDBG and do some tweaking:

    0:000> .restart
    0:000> bp kernel32!BuildSubSysCommandLine; g; bc
    Breakpoint 0 hit

    0:000> ezu poi(esp+8) "calc.exe"

    0:000> g

    As you could see, Calculator is launched instead of WordPad.

    VerifierDlls (REG_SZ)

    VerifierDlls is a list of verifier provider DLLs used by the AppVerifier. The NT loader consumes VerifierDlls during user mode process initialization, which happens in ntdll!LdrpInitializeProcess. If VerifierDlls was found, the standard verifier provider verifier.dll, which comes with the Windows system, will be loaded first. After that, each verifier provider specified by VerifierDlls will be loaded one by one:

    ChildEBP RetAddr
    0020f37c 7719578b ntdll!AVrfpLoadAndInitializeProvider+0x117
    0020f3a0 77170bd9 ntdll!AVrfInitializeVerifier+0xd3
    0020f518 77156077 ntdll!LdrpInitializeProcess+0xe95
    0020f568 77153663 ntdll!_LdrpInitialize+0x78
    0020f578 00000000 ntdll!LdrInitializeThunk+0x10

    771950e1 push 1Ch
    771950e3 push offset ntdll! ?? ::FNODOBFM::`string'+0x762 (77140ed8)
    771950e8 call ntdll!_SEH_prolog4 (77142c0c)
    771950ed mov byte ptr [ebp-19h],0
    771950f1 mov esi,dword ptr [ebp+8]
    771950f4 test byte ptr [ntdll!AVrfpDebug (771ce380)],1
    771950fb je ntdll!AVrfpLoadAndInitializeProvider+0x2b (7719510c)
    771950fd push dword ptr [esi+0Ch]
    77195100 push offset ntdll! ?? ::FNODOBFM::`string' (7713a708)
    77195105 call ntdll!DbgPrint (7710f593)
    7719510a pop ecx
    7719510b pop ecx
    7719510c mov dword ptr [ebp-28h],offset ntdll!RtlpCsVerifyDoNotBreak+0x7 (771ce178)
    77195113 xor eax,eax
    77195115 mov word ptr [ebp-2Ch],ax
    77195119 mov eax,208h
    7719511e mov word ptr [ebp-2Ah],ax
    77195122 push offset SharedUserData+0x30 (7ffe0030)
    77195127 lea eax,[ebp-2Ch]
    7719512a push eax
    7719512b call ntdll!RtlAppendUnicodeToString (7714eed2)
    77195130 push offset ntdll!SlashSystem32SlashString (771551f0)
    77195135 lea eax,[ebp-2Ch]
    77195138 push eax
    77195139 call ntdll!RtlAppendUnicodeStringToString (771472be)
    7719513e lea ebx,[esi+10h]
    77195141 push ebx
    77195142 lea eax,[esi+8]
    77195145 push eax
    77195146 push 0
    77195148 push dword ptr [ebp-28h]
    7719514b call ntdll!LdrLoadDll (771522b8)
    77195150 test eax,eax
    77195152 jge ntdll!AVrfpLoadAndInitializeProvider+0x98 (77195179)
    77195154 push dword ptr [ebp-28h]
    77195157 push eax
    77195158 push dword ptr [esi+0Ch]
    7719515b mov eax,dword ptr [ntdll!PebLdr+0xc (771c788c)]
    77195160 ff7030 push dword ptr [eax+30h]
    77195163 push offset ntdll! ?? ::FNODOBFM::`string' (7713a6c6)
    77195168 call ntdll!DbgPrint (7710f593)
    7719516d add esp,14h
    77195170 mov byte ptr [ebp-19h],1
    77195174 jmp ntdll!AVrfpLoadAndInitializeProvider+0x21c (771952fd)
    77195179 and dword ptr [ebp-4],0
    7719517d push dword ptr [ebx]
    7719517f call ntdll!RtlImageNtHeader (7714fa29)
    77195184 mov edi,eax

    The AppVerifier package comes with a GUI frontend, together with a stack of verifier providers. A verifier provider is a special DLL which accepts DLL_PROCESS_VERIFIER. The WinNT.h header file in DDK 3790.1830 contains everything needed for implementing a verifier provider.

    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
    static RTL_VERIFIER_DLL_DESCRIPTOR aDlls[] = {{}};
      if(fdwReason == DLL_PROCESS_VERIFIER)
        *pVPD = &vpd;
      return TRUE;

    You can read A Debugging Approach to Application Verifier for more information.


    DisableExceptionChainValidation (REG_DWORD)

    SEHOP (Structured Exception Handling Overwrite Protection) is a feature introduced by Windows Vista SP1 to protect a certain kind of attack. However this feature would cause compatibility issues to some existing applications, so the purpose of this value is to workaround for legacy applications by disabling SEHOP feature.

    (to be continued...)

  • Rubato and Chord

    A Debugging Approach to OutputDebugString


    Using OutputDebugString is a common debugging technique for user mode debugging. It is easy but quite useful if you are debugging services or trouble shooting loader problem.

    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
    int __cdecl main()
      OutputDebugStringA("Hello, world! (first chance)\n");
      OutputDebugStringW(L"Hello, world! (second chance)\n");

    When you debug this demo application using Visual Studio, you would see the text messages from the Output window:

    And if you are running this demo without debugging it, you can actually use DbgView to see the outputs (the Microsoft Platform SDK also comes with a console application called Dbmon.Exe):

    Have you ever wondered how it works?

    Let's launch the demo application from WinDBG (I'm using 32bit Win7, you might see different things if you are using other version of Windows):

    windbg.exe outputdebugstring.exe

    0:000> x kernel*!OutputDebugString*
    760c1321          KERNELBASE!OutputDebugStringA
    760c1585          KERNELBASE!OutputDebugStringW
    766e6df2          kernel32!OutputDebugStringW
    766ef0f0          kernel32!OutputDebugStringA
    0:000> bp kernelbase!OutputDebugStringW
    0:000> bl
     0 e 760c1585     0001 (0001)  0:**** KERNELBASE!OutputDebugStringW
    0:000> g
    Hello, world! (first chance)
    Breakpoint 0 hit
    760c1585 8bff            mov     edi,edi
    0:000> pc
    760c15a2 ff15ec110a76    call    dword ptr [KERNELBASE+0x11ec (760a11ec)]
                   ds:0023:760a11ec={ntdll!RtlInitUnicodeStringEx (77d86f0a)}
    0:000> pc
    760c15b8 ff1500100a76    call    dword ptr [KERNELBASE+0x1000 (760a1000)]
             ds:0023:760a1000={ntdll!RtlUnicodeStringToAnsiString (77d89e8e)}
    0:000> pc
    760c15c7 e855fdffff      call    KERNELBASE!OutputDebugStringA (760c1321)

    If we continue tracing, eventually we will reach this point:

    ChildEBP RetAddr
    0015f610 77d762a4 ntdll!KiFastSystemCall
    0015f614 77d770fd ntdll!ZwRaiseException+0xc
    0015f8f8 760ad36f ntdll!RtlRaiseException+0x35
    0015f954 760c1375 KERNELBASE!RaiseException+0x58
    0015fbc0 01351021 KERNELBASE!OutputDebugStringA+0x286

    Dump the SEH chain:

    0:000> dt nt!_EXCEPTION_REGISTRATION_RECORD -r poi(fs:[0])
     +0x000 Next    : 0x0016fd58 _EXCEPTION_REGISTRATION_RECORD
      +0x000 Next    : 0x0016fdac _EXCEPTION_REGISTRATION_RECORD
       +0x000 Next    : 0xffffffff _EXCEPTION_REGISTRATION_RECORD
       +0x004 Handler : 0x77d4e0ed _EXCEPTION_DISPOSITION ntdll!_except_handler4+0
      +0x004 Handler : 0x00a72440 _EXCEPTION_DISPOSITION outputdebugstring!_except_handler4+0
     +0x004 Handler : 0x760c1425 _EXCEPTION_DISPOSITION KERNELBASE!_except_handler4+0

    0:000> !exchain
    0029fcb0: KERNELBASE!_except_handler4+0 (760c1425)
      CRT scope  0, filter: KERNELBASE!OutputDebugStringA+60 (760c13bc)
                    func:   KERNELBASE!OutputDebugStringA+74 (760c11d1)
    0029fd10: outputdebugstring!_except_handler4+0 (00e72440)
      CRT scope  0, filter: outputdebugstring!__tmainCRTStartup+1f4 (00e71584)
                    func:   outputdebugstring!__tmainCRTStartup+20f (00e7159f)
    0029fd64: ntdll!_except_handler4+0 (77d4e0ed)
      CRT scope  0, filter: ntdll!__RtlUserThreadStart+2e (77da7eeb)
                    func:   ntdll!__RtlUserThreadStart+63 (77da8260)

    An SEH was chained, and the control is returned to the OS kernel.

    0:000> u 760c11d1 L20
    760c11d1 8b65e8          mov     esp,dword ptr [ebp-18h]
    760c11d4 33ff            xor     edi,edi
    760c11d6 89bdd0fdffff mov dword ptr [ebp-230h],edi
    760c11dc 89bddcfdffff    mov     dword ptr [ebp-224h],edi
    760c11e2 89bdd4fdffff    mov     dword ptr [ebp-22Ch],edi
    760c11e8 89bde0fdffff    mov     dword ptr [ebp-220h],edi
    760c11ee e86d56feff      call    KERNELBASE!GetLastError (760a6860)
    760c11f3 8985c8fdffff    mov     dword ptr [ebp-238h],eax
    760c11f9 393d28490e76    cmp     dword ptr [KERNELBASE!BaseDataFileHandleTableLock+0x4 (760e4928)],edi
    760c11ff 0f8457040000    je      KERNELBASE!OutputDebugStringA+0xa4 (760c165c)
    760c1205 a128490e76      mov     eax,dword ptr [KERNELBASE!BaseDataFileHandleTableLock+0x4 (760e4928)]
    760c120a 3bc7            cmp     eax,edi
    760c120c 7441            je      KERNELBASE!OutputDebugStringA+0x166 (760c124f)
    760c120e 6810270000      push    2710h
    760c1213 50              push    eax
    760c1214 e83666feff      call    KERNELBASE!WaitForSingleObject (760a784f)
    760c1219 3bc7            cmp     eax,edi
    760c121b 0f8570040000    jne     KERNELBASE!OutputDebugStringA+0xf3 (760c1691)
    760c1221 6870120c76      push    offset KERNELBASE!`string' (760c1270)
    760c1226 57              push    edi
    760c1227 6a02            push    2
    760c1229 e8df7bfeff      call    KERNELBASE!OpenFileMappingW (760a8e0d)
    760c122e 8985d0fdffff    mov     dword ptr [ebp-230h],eax
    760c1234 3bc7            cmp     eax,edi
    760c1236 0f85dea00000    jne     KERNELBASE!OutputDebugStringA+0x111 (760cb31a)
    760c123c 39bde0fdffff    cmp     dword ptr [ebp-220h],edi
    760c1242 750b            jne     KERNELBASE!OutputDebugStringA+0x166 (760c124f)
    760c1244 ff3528490e76    push    dword ptr [KERNELBASE!BaseDataFileHandleTableLock+0x4 (760e4928)]
    760c124a e81204feff      call    KERNELBASE!ReleaseMutex (760a1661)
    760c124f c745fc01000000  mov     dword ptr [ebp-4],1
    760c1256 c745fc02000000  mov     dword ptr [ebp-4],2
    760c125d 8b85ccfdffff    mov     eax,dword ptr [ebp-234h]

    As we could see from the disassembly, the byte value on 0x760c11d1 is 0x8b, let's change it to INT3:

    0:000> ?? *(char*)(0x760c11d1) = 0xcc
    char 0n-52 ''

    Before we can continue, let's make sure WinDBG is installed as the Just-In-Time debugger by running the following command from an Elevated Command Prompt:

    windbg.exe -IS

    Now we are ready to continue:

    0:000> .detach
    NoTarget> q

    On the same time, Just-In-Time debugging is triggered and another WinDBG just pops up with the following information:

    (aa0.11cc): Break instruction exception - code 80000003 (!!! second chance !!!)
    760c11d1 cc              int     3

    Change the value on 0x760c11d1 back (or did you just forget the value?).

    0:000> ?? *(char*)(0x760c11d1) = 0x8b
    char 0n-117 ''


    760c11cb ff251c130a76    jmp     dword ptr [KERNELBASE!_imp__DbgPrint (760a131c)]
    760c11d1 8b65e8          mov     esp,dword ptr [ebp-18h] ss:0023:0022fbb8=0022f97c
    760c11d4 33ff            xor     edi,edi
    760c11d6 89bdd0fdffff    mov     dword ptr [ebp-230h],edi

    I'll leave the rest things for you to figure out. Hints:

    1. You may want to use dda esp to dump the stack.

      0:000> dda esp
      0022f97c  4464c9b1
      0022f980  00000000
      0022f984  0022fbdc ""
      0022f988  7ffd4000 ""
      0022f98c  0000001e
      0022f990  008f573c "Hello, world! (first chance)."
      0022f994  002c00c4 ".,,"
      0022f998  00000000

    2. You would eventually see something like CreateDBWinMutex.
    3. When you encountered with OpenEvent or similar function call, you can use da to check the string parameter.

      0:000> da 760cb478
      760cb478  "DBWIN_BUFFER_READY"

    Now we have finished the debugging approach to OutputDebugString, it's time to start writing a simple user mode dbgview:

    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
    #include <stdio.h>
    #define IfFalseRet(c) do{if(!(c)){return dwLastError = ::GetLastError();}}while(false)
    class CHandle
      CHandle(HANDLE h = NULL): m_h(h)
      void Release()
        m_h = NULL;
      operator bool() const
        return m_h != INVALID_HANDLE_VALUE && m_h != NULL;
      operator HANDLE() const
        return m_h;
      CHandle& operator= (const HANDLE& h)
        m_h = h;
        return *this;
      CHandle& operator= (CHandle& h)
        if(this != &h)
          HANDLE hSwap = m_h;
          m_h = h.m_h;
          h.m_h = hSwap;
        return *this;
      HANDLE m_h;
    #pragma pack(push, 1)
    struct CDBWinBuffer
      DWORD dwProcessId;
      BYTE  abData[4096 - sizeof(DWORD)];
    #pragma pack(pop)
    bool g_fContinue = true;
    BOOL CtrlHandler(DWORD fdwCtrlType)
      case CTRL_C_EVENT:
      case CTRL_CLOSE_EVENT:
        g_fContinue = false;
        return TRUE;
      return FALSE;
    int __cdecl main()
      DWORD dwLastError = ERROR_SUCCESS;
      IfFalseRet(SetConsoleCtrlHandler((PHANDLER_ROUTINE)(CtrlHandler), TRUE) == TRUE);
      CHandle hMutex = OpenMutex(MUTEX_ALL_ACCESS, FALSE, DBWIN_MUTEX);
        IfFalseRet(GetLastError() == ERROR_FILE_NOT_FOUND);
        IfFalseRet(hMutex = CreateMutex(NULL, FALSE, DBWIN_MUTEX));
      CHandle hEventBufferReady = OpenEvent(EVENT_MODIFY_STATE, FALSE, DBWIN_BUFFER_READY);
        IfFalseRet(GetLastError() == ERROR_FILE_NOT_FOUND);
        IfFalseRet(hEventBufferReady = CreateEvent(NULL, FALSE, TRUE, DBWIN_BUFFER_READY));
      CHandle hEventDataReady = OpenEvent(EVENT_MODIFY_STATE, FALSE, DBWIN_DATA_READY);
        IfFalseRet(GetLastError() == ERROR_FILE_NOT_FOUND);
        IfFalseRet(hEventDataReady = CreateEvent(NULL, FALSE, FALSE, DBWIN_DATA_READY));
      CHandle hFileMappingBuffer = OpenFileMapping(FILE_MAP_READ, FALSE, DBWIN_BUFFER);
        IfFalseRet(GetLastError() == ERROR_FILE_NOT_FOUND);
        IfFalseRet(hFileMappingBuffer = CreateFileMapping(
          0, sizeof(CDBWinBuffer), DBWIN_BUFFER));
      CDBWinBuffer* pDbgBuffer = (CDBWinBuffer*)(MapViewOfFile(
        hFileMappingBuffer, SECTION_MAP_READ, 0, 0, 0));
        if(WaitForSingleObject(hEventDataReady, 100) == WAIT_OBJECT_0)
          printf("%s", pDbgBuffer->abData);
      return dwLastError;


    1. You will need to run the sample from an Elevated Command Prompt.
    2. The debug buffer guarantees you that the string is NUL terminated (any string longer than 4091 will be truncated).
    3. To make the sample easier, I just pass pDbgBuffer->abData to printf without performing any decoding (I just treated them as ASCII string).
    4. DbgPrint is defined for each execution environment, for user mode this would be ntdll!DbgPrint. ntdll!DbgPrint would call into KdpTrap which would in turn output the bytes to the Debug Port. You may find it more convenient to use DbgPrint instead of OutputDebugString as DbgPrint supports string formatting, but be cautious with the potential breaking changes in NTDLL.


    1. As you might have discovered already, the underneath mechanism of OutputDebugString makes use of shared objects. What would happen if we forget to set the DBWIN_BUFFER_READY event?
    2. On 64bit Windows, the 32bit application runs under WoW64. Does the WoW64 own a separate set of OutputDebugString objects, or share with the 64bit environment?
    3. How would OutputDebugStringA|W be implemented using a same shared buffer?

    For the homework, you may find some hints from the WinDBG output, where I've already marked in color.

  • Rubato and Chord

    Debugging Tips for Multi-Threaded Application


    Multi-threaded applications are error-prone and hard to debug, so it's worth a dedicated topic. I will try to maintain a list of tips for debugging multi-threaded application using Visual Studio and WinDBG.

    Below is a trivial multi-threaded application which will be used as an example:

    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
    DWORD WINAPI ThreadProc(LPVOID lpParameter)
      return 0; // add breakpoint here
    } int __cdecl main() {   HANDLE hThread = CreateThread(NULL, 4096, ThreadProc, NULL, 0, NULL);   if(hThread)   {     WaitForSingleObject(hThread, INFINITE);     CloseHandle(hThread);   } }

    In the Visual Studio IDE, set a breakpoint at line 5 and Start Debugging (F5), you should be able to see the following Call Stack:

    Note that in the Call Stack window, frames from kernel32.dll and ntdll.dll were formatted as address, this is because we haven't load the symbols yet. In this demo, we will avoid using public symbols, instead, modify the debugger setting from menu [Debug] -> [Options and Settings...]:

    Check the "Load DLL exports" from the Options dialog window. Now launch the debugger again, and instead of using the Call Stack window, open the Parallel Stacks window from menu [Debug] -> [Windows] -> [Parallel Stacks]:

    One common task for multi-threaded debugging is to track thread creation, which is normally achieved by setting breakpoint at thread creation. We will take another approach by breaking at the very beginning. Ctrl + B and add a breakpoint at {,,kernel32.dll}BaseThreadInitThunk, then launch the debugger again to see what would happen:

    Another common task is to track thread (context) switch, this is extremely useful while debugging async applications. When we hit the breakpoint in ThreadProc, open the Threads window from [Debug] -> [Windows] -> [Threads]:

    Right click on the "Main Thread", and from the popup menu, choose [Switch to Thread]. Now the Call Stack window should look like:

    Double click on the top frame ntdll.dll!KiFastSystemCallRet, and set a breakpoint at the disassembly window (alternatively, you may set breakpoint by right clicking the top frame, and select [Breakpoint] -> [Insert Breakpoint] from the popup menu):

    Right click on the breakpoint (the red dot), from the popup menu choose [Filter...], and type in "ThreadId = 3436" where the 3436 is the main thread ID which shows in the Threads window (you will need to fill in the correct thread ID by checking the Threads window). Now if the main thread got activited, we will hit into this breakpoint, don't forget to clear it once hitted.


    1. In the popup menu of right clicking on a breakpoint, there are a number of items such like [Location...], [Condition...], [Hit Count...], [Filter...]. Can we use Hit Count in combination with Filter (ThreadId = xxx) to capture context switch?
    2. Would you try to figure out the execution order of determining whether to break or not, given the constraints of Location, Condition, Hit Count and Filter?
    3. Are these breakpoint constraints work well with Data Breakpoints?
    4. How to check the return value of a thread procedure?
  • Rubato and Chord

    Visual Studio Debugger or WinDBG?


    Microsoft has provided a number of debuggers. Visual Studio Debugger would be the most widely known one, while there is a less known tool set called Debugging Tools for Windows which is available for free. Even inside Microsoft, there were questions around why having so many debuggers, and which one to use.

    A quick answer is, use Visual Studio for everyday user mode software development and put Debugging Tools for Windows aside. Visual Studio has a good feature set, it supports various kinds of debugging target (native, managed, script, interop, Transact-SQL, HLSL, etc.), and has a lot of cool features such like Intellitrace, JMC (Just My Code) and EnC (Edit and Continue). Besides, Visual Studio has a very good user interface, it also hides the low level details that most people wouldn't care.

    Debugging Tools for Windows on the other hand, is a set of debuggers which includes WinDBG, a powerful debugger with a graphical interface and a console interface, as well as the console-based debuggers NTSD, CDB, and KD. These Windows debuggers (CDB, KD, NTSD, WinDBG) offer similar debugging capabilities which give you more details, and more control over the target. These debuggers only support native and managed target. Below are a few reasons for using the Windows debuggers instead of Visual Studio:

    1. Kernel debugging
      Visual Studio has no kernel debugging support.
    2. Small footprint
      The Windows debuggers do not require any install, and can run from a share folder over network. Also, CDB and NTSD are pure console applications, the consumption of memory and other resources are much lower than Visual Studio.
    3. System debugging
      NTSD has less demands on the operating system, which makes it ideal for debugging winlogon, services and subsystems.
    4. Non-invasive debugging
      The Windows debuggers can work in a non-invasive mode without attaching to the target. With non-invasive debugging, it's even possible to use the Visual Studio debugger and Windows debuggers cooperatively in a single debug session.
    5. Scripting
      All the Windows debuggers expose their functionality through text commands with many powerful syntax support (sorts like DEBUG command in the era of DOS), making it possible to automate debugging tasks.

    What debuggers do you use? Have you had problems while using the Visual Studio Debugger?

  • Rubato and Chord

    Data Breakpoints


    The Visual Studio debugger supports a kind of breakpoint called Data Breakpoint, sometimes it is also called watchpoint. Data breakpoint is architecture dependant, as it requires hardware support provided by CPU. For x86, this will be the DR (Debug Register).

    The following code demonstrates how to use the x86 debug register by implementing a very simple native debugger.

    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
    #include <stdio.h>
    HMODULE    g_hModule;                            /* linear address of exe   */
    LPVOID     g_pOEP;                               /* original entry point    */
    BYTE       g_bBreakPoint;
    BYTE       g_bINT3 = 0xcc;                       /* debug break instruction */
    BOOL       g_fDebugRegisterSupported = FALSE;    /* hardware DR supported   */
    int WINAPI ExeEntry()
        g_hModule = GetModuleHandle(NULL); /* HMODULE is always 4 bytes aligned */
        printf("HMODULE: %p\n", g_hModule /* insert a space here... */,
               *(INT32*)(g_hModule) /* read 4 bytes */);
        CONTEXT context;
        context.ContextFlags = CONTEXT_FULL | CONTEXT_DEBUG_REGISTERS;
        PROCESS_INFORMATION processInformation;
        STARTUPINFO startupInfo = {sizeof(STARTUPINFO)};
        if(CreateProcess(NULL, GetCommandLine(), NULL, NULL, FALSE, DEBUG_PROCESS,
                         NULL, NULL, &startupInfo, &processInformation))
          DWORD dwContinueDebugStatus = DBG_CONTINUE;
            DEBUG_EVENT debugEvent;
            WaitForDebugEvent(&debugEvent, INFINITE);
              g_pOEP = (LPVOID)(debugEvent.u.CreateProcessInfo.lpStartAddress);
              g_hModule = (HMODULE)(debugEvent.u.CreateProcessInfo.lpBaseOfImage);
              printf("CREATE_PROCESS_DEBUG_EVENT @%p OEP=%p\n", g_hModule, g_pOEP);
            case EXCEPTION_DEBUG_EVENT:
              printf("EXCEPTION_DEBUG_EVENT PID=%d TID=%d @%p\n",
                     debugEvent.dwProcessId, debugEvent.dwThreadId,
              GetThreadContext(processInformation.hThread, &context);
              case EXCEPTION_BREAKPOINT:
                  if(debugEvent.u.Exception.ExceptionRecord.ExceptionAddress == g_pOEP)
                    LPVOID IP = (LPVOID)(--context.Eip);
                    WriteProcessMemory(processInformation.hProcess, IP,
                                       &g_bBreakPoint, 1, NULL);
                    FlushInstructionCache(processInformation.hProcess, IP, 1);
                    context.Dr0 = (DWORD)(g_hModule);
                    context.Dr7 = 0x000f0101;
                    printf("\tbp $exentry\n");
                    ReadProcessMemory(processInformation.hProcess, g_pOEP,
                                      &g_bBreakPoint, 1, NULL);
                    WriteProcessMemory(processInformation.hProcess, g_pOEP,
                                      &g_bINT3, 1, NULL);
                    FlushInstructionCache(processInformation.hProcess, g_pOEP, 1);
              case EXCEPTION_SINGLE_STEP:
                printf("EXCEPTION_SINGLE_STEP DR6=%08X\n", context.Dr6);
                g_fDebugRegisterSupported = TRUE;
              context.Dr6 = 0;
              SetThreadContext(processInformation.hThread, &context);
            case EXIT_PROCESS_DEBUG_EVENT:
              dwContinueDebugStatus = 0;
            case LOAD_DLL_DEBUG_EVENT:
            ContinueDebugEvent(debugEvent.dwProcessId, debugEvent.dwThreadId,
        printf("Debug Register Test: %s\n", g_fDebugRegisterSupported ?
               "Hardware DR supported" : "Hardware DR not supported");
      return ERROR_SUCCESS;

    To compile the source code, you may use Visual Studio or either of the following compilers (x86 32bit):

    cl.exe watchpoint.cpp kernel32.lib msvcrt.lib /GS- /link /ENTRY:ExeEntry /NODEFAULTLIB /SUBSYSTEM:CONSOLE

    gcc.exe -fno-exceptions -fno-rtti -s -Os -o watchpoint watchpoint.cpp -Wl,--stack,65536

    A few things to mention:

    1. Data breakpoints respect the CPU working mode, which means the linear address is used if paging enabled, and physical address used if paging disabled.
    2. Each CPU core has its own set of debug registers. This wouldn't be a problem for user mode as the operating system maintains context switching, but it will be very different if you are implementing kernel mode driver which runs above dispatch level, or custom interrupt vector.
    3. Virtual machine sometimes does not implement DR, this is true for Virtual PC and VMware if you don't have hardware virtualization enabled.
    4. To make the sample easier, our trivial debugger assumes we only have one debugee and the debugee is single-threaded. For the real debugger, PID and TID should be used to get the correct handle (and should have it cached) in order to support multi-threaded debugee, as well as debugging multiple programs in a single debugger instance.
    5. A little modification is required in order to support 64bit:
                    if(debugEvent.u.Exception.ExceptionRecord.ExceptionAddress == g_pOEP)
      #if defined(_M_IX86)
                      LPVOID IP = (LPVOID)(--context.Eip);
      #elif defined(_M_X64)
                      LPVOID IP = (LPVOID)(--context.Rip);
                      WriteProcessMemory(processInformation.hProcess, IP,
                                         &g_bBreakPoint, 1, NULL);
                      FlushInstructionCache(processInformation.hProcess, IP, 1);
      #if defined(_M_IX86)
                      context.Dr0 = (DWORD)(g_hModule);
      #elif defined(_M_X64)
                      context.Dr0 = (DWORD64)(g_hModule);
                      context.Dr7 = 0x000f0101;
    6. ExeEntry is used as the entry point instead of the one provided by the C Runtime, which is not a good practice, as this might cause subtle CRT initialization problem. The main reason of doing this is that CRT initialization code reads from the module header just as we did, which would also trigger our watchpoint. To verify this:
      1. Change the following code by inserting a blank space:
            printf("HMODULE: %p\n", g_hModule /* insert a space here... */,
                   *(INT32*)(g_hModule) /* read 4 bytes */);
      2. After the change, your code would look like:
            printf("HMODULE: %p\n", g_hModule /* insert a space here... * /,
                   *(INT32*)(g_hModule) /* read 4 bytes */);
      3. Now change ExeEntry to int __cdecl main(), and recompile without the /ENTRY:ExeEntry flag.
      4. Run and see what happened, on my machine, this would look like:
        CREATE_PROCESS_DEBUG_EVENT @00250000 OEP=00251978
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @77DD04F6
                 bp $exentry EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @00251978
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @002518CA
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @0025104F
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @67D278FC
        Debug Register Test: Hardware DR supported


    1. What if multiple watchpoints (either Read, Write or Execute, for 1, 2 or 4 bytes) were added to a single address?
    2. What if the target address is not aligned?
    3. What if the data access triggers a GPF (General Protection Fault)?
    4. What would happen if a watchpoint was applied on an INT3?
    5. Why are we setting the watchpoint when the debugee reaches OEP, instead of setting it at the first DebugBreak?
    6. You might have noticed that we decreased EIP by 1 when we received the EXCEPTION_BREAKPOINT debug event, why? What would happen if the interrupt was caused by opcode CD 03 instead of CC?

    For the homework, you would want to either check the Intel x86 specification and do verification by playing around the code.

  • Rubato and Chord

    Hello, world...


    Yet another Hello World...

    Well, my name is Reiley Yang. I'm a developer working on the Visual Studio debugger. I would like to use this technical blog to share my understanding of debuggers. Previously I have also been working on the C++ libraries (ATL, CRT, MFC, STL) and compiler front-end, so I will probably also talk a bit about these as well.

Page 1 of 1 (6 items)