Rubato and Chord

Reiley's technical blog

  • Rubato and Chord

    Data Breakpoints

    • 2 Comments

    The Visual Studio debugger supports a kind of breakpoint called Data Breakpoint, sometimes it is also called watchpoint. Data breakpoint is architecture dependant, as it requires hardware support provided by CPU. For x86, this will be the DR (Debug Register).

    The following code demonstrates how to use the x86 debug register by implementing a very simple native debugger.


    #define WIN32_LEAN_AND_MEAN
    
    #include <Windows.h>
    #include <stdio.h>
    
    HMODULE    g_hModule;                            /* linear address of exe   */
    LPVOID     g_pOEP;                               /* original entry point    */
    BYTE       g_bBreakPoint;
    BYTE       g_bINT3 = 0xcc;                       /* debug break instruction */
    BOOL       g_fDebugRegisterSupported = FALSE;    /* hardware DR supported   */
    
    int WINAPI ExeEntry()
    {
      if(IsDebuggerPresent())
      {
        g_hModule = GetModuleHandle(NULL); /* HMODULE is always 4 bytes aligned */
        printf("HMODULE: %p\n", g_hModule /* insert a space here... */,
               *(INT32*)(g_hModule) /* read 4 bytes */);
      }
      else
      {
        CONTEXT context;
        context.ContextFlags = CONTEXT_FULL | CONTEXT_DEBUG_REGISTERS;
        PROCESS_INFORMATION processInformation;
        STARTUPINFO startupInfo = {sizeof(STARTUPINFO)};
        if(CreateProcess(NULL, GetCommandLine(), NULL, NULL, FALSE, DEBUG_PROCESS,
                         NULL, NULL, &startupInfo, &processInformation))
        {
          DWORD dwContinueDebugStatus = DBG_CONTINUE;
          while(dwContinueDebugStatus)
          {
            DEBUG_EVENT debugEvent;
            WaitForDebugEvent(&debugEvent, INFINITE);
            switch(debugEvent.dwDebugEventCode)
            {
            case CREATE_PROCESS_DEBUG_EVENT:
              g_pOEP = (LPVOID)(debugEvent.u.CreateProcessInfo.lpStartAddress);
              g_hModule = (HMODULE)(debugEvent.u.CreateProcessInfo.lpBaseOfImage);
              CloseHandle(debugEvent.u.CreateProcessInfo.hFile);
              printf("CREATE_PROCESS_DEBUG_EVENT @%p OEP=%p\n", g_hModule, g_pOEP);
              break;
            case EXCEPTION_DEBUG_EVENT:
              printf("EXCEPTION_DEBUG_EVENT PID=%d TID=%d @%p\n",
                     debugEvent.dwProcessId, debugEvent.dwThreadId,
                     debugEvent.u.Exception.ExceptionRecord.ExceptionAddress);
              GetThreadContext(processInformation.hThread, &context);
              switch(debugEvent.u.Exception.ExceptionRecord.ExceptionCode)
              {
              case EXCEPTION_BREAKPOINT:
                if(debugEvent.u.Exception.dwFirstChance)
                {
                  if(debugEvent.u.Exception.ExceptionRecord.ExceptionAddress == g_pOEP)
                  {
                    LPVOID IP = (LPVOID)(--context.Eip);
                    WriteProcessMemory(processInformation.hProcess, IP,
                                       &g_bBreakPoint, 1, NULL);
                    FlushInstructionCache(processInformation.hProcess, IP, 1);
                    context.Dr0 = (DWORD)(g_hModule);
                    context.Dr7 = 0x000f0101;
                  }
                  else
                  {
                    printf("\tbp $exentry\n");
                    ReadProcessMemory(processInformation.hProcess, g_pOEP,
                                      &g_bBreakPoint, 1, NULL);
                    WriteProcessMemory(processInformation.hProcess, g_pOEP,
                                      &g_bINT3, 1, NULL);
                    FlushInstructionCache(processInformation.hProcess, g_pOEP, 1);
                  }
                }
                break;
              case EXCEPTION_SINGLE_STEP:
                printf("EXCEPTION_SINGLE_STEP DR6=%08X\n", context.Dr6);
                g_fDebugRegisterSupported = TRUE;
                break;
              }
              context.Dr6 = 0;
              SetThreadContext(processInformation.hThread, &context);
              break;
            case EXIT_PROCESS_DEBUG_EVENT:
              dwContinueDebugStatus = 0;
              printf("EXIT_PROCESS_DEBUG_EVENT\n");
              break;
            case LOAD_DLL_DEBUG_EVENT:
              CloseHandle(debugEvent.u.LoadDll.hFile);
              break;
            }
            ContinueDebugEvent(debugEvent.dwProcessId, debugEvent.dwThreadId,
                               dwContinueDebugStatus);
          }
          CloseHandle(processInformation.hThread);
          CloseHandle(processInformation.hProcess);
        }
        printf("Debug Register Test: %s\n", g_fDebugRegisterSupported ?
               "Hardware DR supported" : "Hardware DR not supported");
      }
      return ERROR_SUCCESS;
    }
    

    To compile the source code, you may use Visual Studio or either of the following compilers (x86 32bit):

    cl.exe watchpoint.cpp kernel32.lib msvcrt.lib /GS- /link /ENTRY:ExeEntry /NODEFAULTLIB /SUBSYSTEM:CONSOLE

    gcc.exe -fno-exceptions -fno-rtti -s -Os -o watchpoint watchpoint.cpp -Wl,--stack,65536


    A few things to mention:

    1. Data breakpoints respect the CPU working mode, which means the linear address is used if paging enabled, and physical address used if paging disabled.
    2. Each CPU core has its own set of debug registers. This wouldn't be a problem for user mode as the operating system maintains context switching, but it will be very different if you are implementing kernel mode driver which runs above dispatch level, or custom interrupt vector.
    3. Virtual machine sometimes does not implement DR, this is true for Virtual PC and VMware if you don't have hardware virtualization enabled.
    4. To make the sample easier, our trivial debugger assumes we only have one debugee and the debugee is single-threaded. For the real debugger, PID and TID should be used to get the correct handle (and should have it cached) in order to support multi-threaded debugee, as well as debugging multiple programs in a single debugger instance.
    5. A little modification is required in order to support 64bit:
                    if(debugEvent.u.Exception.ExceptionRecord.ExceptionAddress == g_pOEP)
                    {
      #if defined(_M_IX86)
                      LPVOID IP = (LPVOID)(--context.Eip);
      #elif defined(_M_X64)
                      LPVOID IP = (LPVOID)(--context.Rip);
      #endif
                      WriteProcessMemory(processInformation.hProcess, IP,
                                         &g_bBreakPoint, 1, NULL);
                      FlushInstructionCache(processInformation.hProcess, IP, 1);
      #if defined(_M_IX86)
                      context.Dr0 = (DWORD)(g_hModule);
      #elif defined(_M_X64)
                      context.Dr0 = (DWORD64)(g_hModule);
      #endif
                      context.Dr7 = 0x000f0101;
                    }
      
    6. ExeEntry is used as the entry point instead of the one provided by the C Runtime, which is not a good practice, as this might cause subtle CRT initialization problem. The main reason of doing this is that CRT initialization code reads from the module header just as we did, which would also trigger our watchpoint. To verify this:
      1. Change the following code by inserting a blank space:
            printf("HMODULE: %p\n", g_hModule /* insert a space here... */,
                   *(INT32*)(g_hModule) /* read 4 bytes */);
      2. After the change, your code would look like:
            printf("HMODULE: %p\n", g_hModule /* insert a space here... * /,
                   *(INT32*)(g_hModule) /* read 4 bytes */);
      3. Now change ExeEntry to int __cdecl main(), and recompile without the /ENTRY:ExeEntry flag.
      4. Run and see what happened, on my machine, this would look like:
        CREATE_PROCESS_DEBUG_EVENT @00250000 OEP=00251978
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @77DD04F6
                 bp $exentry EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @00251978
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @002518CA
        EXCEPTION_SINGLE_STEP DR6=FFFF0FF1
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @0025104F
        EXCEPTION_SINGLE_STEP DR6=FFFF0FF1 HMODULE: 00250000
        EXCEPTION_DEBUG_EVENT PID=1560 TID=4852 @67D278FC
        EXCEPTION_SINGLE_STEP DR6=FFFF0FF1
        EXIT_PROCESS_DEBUG_EVENT
        Debug Register Test: Hardware DR supported

    Homework:

    1. What if multiple watchpoints (either Read, Write or Execute, for 1, 2 or 4 bytes) were added to a single address?
    2. What if the target address is not aligned?
    3. What if the data access triggers a GPF (General Protection Fault)?
    4. What would happen if a watchpoint was applied on an INT3?
    5. Why are we setting the watchpoint when the debugee reaches OEP, instead of setting it at the first DebugBreak?
    6. You might have noticed that we decreased EIP by 1 when we received the EXCEPTION_BREAKPOINT debug event, why? What would happen if the interrupt was caused by opcode CD 03 instead of CC?

    For the homework, you would want to either check the Intel x86 specification and do verification by playing around the code.

  • Rubato and Chord

    A Debugging Approach to IFEO

    • 2 Comments

    IFEO (Image File Execution Options) is a feature provided by the NT based operating system. It can be helpful when you are trying to debug at the very beginning of an application launch. A few people also taked about IFEO on MSDN Blogs:

    1. Image File Execution Options by Junfeng.
    2. Inside 'Image File Execution Options' debugging by Gregg.
    3. Beware the Image File Execution Options key by Raymond.
    4. IFEO and Managed-debugging by Mike.

    Debugger (REG_SZ)

    Let's begin with the following demo (I'm using 32bit Win7, you might see different things if you are using other version of Windows):


    #define WIN32_LEAN_AND_MEAN
    
    #include <Windows.h>
    
    #define IfFalseRet(c) do{if(!(c)){return dwLastError = GetLastError();}}while(0)
    
    STARTUPINFO g_startupInfo = {sizeof(STARTUPINFO)};
    TCHAR g_szCommandLine[] = TEXT("notepad.exe");
    
    int WINAPI ExeEntry(void)
    {
      DWORD dwLastError = ERROR_SUCCESS;
    
      PROCESS_INFORMATION processInformation;
      IfFalseRet(CreateProcess(NULL, g_szCommandLine, NULL, NULL, FALSE, 0,
                               NULL, NULL, &g_startupInfo, &processInformation));
      CloseHandle(processInformation.hThread);
      CloseHandle(processInformation.hProcess);
    
      return dwLastError;
    }
    

    To compile the source code, you may use Visual Studio Command Prompt (x86):

    cl.exe createprocess.c /D UNICODE /GS- /Od /Oy /link /ENTRY:ExeEntry /NODEFAULTLIB /OPT:ICF /OPT:REF /RELEASE /SUBSYSTEM:CONSOLE kernel32.lib

    Now with the Process Monitor opened, launch the demo, and you should be able to find the following record captured by Process Monitor:

    00:01:17.8231352 createprocess.exe 1320 RegOpenKey HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe NAME NOT FOUND Desired Access: Query Value, Enumerate Sub Keys

    Let's create an IFEO entry in the registry:


    Windows Registry Editor Version 5.00
     
    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe]
    "Debugger"="\"C:\\Program Files\\Windows NT\\Accessories\\wordpad.exe\""

    After the IFEO entry got imported to the registry, launch the demo again and you will see notepad.exe was replaced by wordpad.exe, with the string "notepad.exe" passed in as an argument.

    00:24:32.3322232 createprocess.exe 4668 RegQueryValue HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\New Key #1\Debugger SUCCESS Type: REG_SZ, Length: 108, Data: "C:\Program Files\Windows NT\Accessories\wordpad.exe"

    00:24:32.3343642 createprocess.exe 4668 Process Create C:\Program Files\Windows NT\Accessories\wordpad.exe SUCCESS PID: 2724, Command line: "C:\Program Files\Windows NT\Accessories\wordpad.exe" notepad.exe

    Let's launch the demo application from WinDBG:

    windbg.exe createprocess.exe

    0:000> x *!*Create*Process*
    767007a2          kernel32!CreateProcessInternalW
    766b2082          kernel32!CreateProcessA
    766b204d          kernel32!CreateProcessW
    7670c89c          kernel32!CreateProcessInternalA
    77d756a8          ntdll!NtCreateProcessEx
    77d75698          ntdll!NtCreateProcess
    77d889fb          ntdll!RtlpCreateProcessRegistryInfo
    77d75698          ntdll!ZwCreateProcess
    77d756a8          ntdll!ZwCreateProcessEx
    77d56bdf          ntdll!RtlpCreateUserProcess
    77d56b71          ntdll!RtlCreateUserProcess
    77dd1072          ntdll!RtlCreateProcessReflection
    77d75778          ntdll!ZwCreateUserProcess
    77dd988a          ntdll!RtlCreateProcessParameters
    77d75778          ntdll!NtCreateUserProcess
    77d96ee9          ntdll!RtlCreateProcessParametersEx

    0:000> bp kernel32!CreateProcessInternalW

    0:000> g
    Breakpoint 0 hit

    0:000> k3
    ChildEBP RetAddr
    0018f7a8 7670c9bf kernel32!CreateProcessInternalW
    0018f888 766b20ae kernel32!CreateProcessInternalA+0x2f8
    0018f8c0 00d71068 kernel32!CreateProcessA+0x2c

    0:000> ddu esp L6
    0018f7ac  7670c9bf "???..?"
    0018f7b0  00000000
    0018f7b4  00000000
    0018f7b8  00383a08 "notepad.exe"
    0018f7bc  00000000
    0018f7c0  00000000

    Continue tracing, eventually we will reach this point:

    ChildEBP RetAddr 
    001af248 77d75784 ntdll!KiFastSystemCall
    001af24c 76700eff ntdll!NtCreateUserProcess+0xc
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xe75
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    0:000> bp 77d75784
    0:000> g
    Breakpoint 1 hit

    From Process Monitor we can trace the IFEO registry key access from kernel mode. On my Win7 machine, it showed three operations (RegOpenKey, RegQueryValue, RegQueryValue), this is because IFEO was moved into kernel mode since Longhorn. On Windows XP and Windows Server 2003, IFEO is a pure user mode action, which means you can easily bypass it without having to write a kernel mode driver. Another thing is, kernel mode IFEO doesn't have the bitness problem, as it would never read from Wow6432Node.

    Continue tracing until we have the following call stacks:

    ChildEBP RetAddr
    0020efe4 77d9ce6f ntdll!RtlQueryImageFileExecutionOptions
    0020f008 76704ab1 ntdll!LdrQueryImageFileExecutionOptions+0x1e
    0020f034 76702abb kernel32!BasepGetDisableLocalOverrideConfig+0x33
    0020f0ac 7670119b kernel32!BasepConstructSxsCreateProcessMessage+0x8e
    0020f728 7670c9bf kernel32!CreateProcessInternalW+0x16b1
    0020f808 766b20ae kernel32!CreateProcessInternalA+0x2f8
    0020f840 01371068 kernel32!CreateProcessA+0x2c

    ChildEBP RetAddr
    001af264 7672d529 kernel32!BuildSubSysCommandLine
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xf87
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    0:000> ddu esp L6
    001af268  7672d529 "??????"
    001af26c  00000003
    001af270  00234348 ""C:\Program Files\Windows NT\Accessories\wordpad.exe""
    001af274  00000000
    001af278  00233a08 "notepad.exe"
    001af27c  001af4d8 ".."

    Continue tracing:

    ChildEBP RetAddr 
    001af1ac 76701693 ntdll!RtlCreateProcessParametersEx
    001af258 76700e90 kernel32!BasepCreateProcessParameters+0x148
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xe06
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    0:000> dS poi(esp+8)
    00283cf8  "C:\Program Files\Windows NT\Acce"
    00283d38  "ssories\wordpad.exe"

    Continue tracing:

    ChildEBP RetAddr 
    001af24c 76700eff ntdll!NtCreateUserProcess
    001af8a8 7670c9bf kernel32!CreateProcessInternalW+0xe75
    001af988 766b20ae kernel32!CreateProcessInternalA+0x2f8
    001af9c0 01371068 kernel32!CreateProcessA+0x2c

    Now the user mode IFEO logic looks clear to us, let's restart WinDBG and do some tweaking:

    0:000> .restart
    0:000> bp kernel32!BuildSubSysCommandLine; g; bc
    Breakpoint 0 hit

    0:000> ezu poi(esp+8) "calc.exe"

    0:000> g

    As you could see, Calculator is launched instead of WordPad.

    VerifierDlls (REG_SZ)

    VerifierDlls is a list of verifier provider DLLs used by the AppVerifier. The NT loader consumes VerifierDlls during user mode process initialization, which happens in ntdll!LdrpInitializeProcess. If VerifierDlls was found, the standard verifier provider verifier.dll, which comes with the Windows system, will be loaded first. After that, each verifier provider specified by VerifierDlls will be loaded one by one:

    ChildEBP RetAddr
    0020f37c 7719578b ntdll!AVrfpLoadAndInitializeProvider+0x117
    0020f3a0 77170bd9 ntdll!AVrfInitializeVerifier+0xd3
    0020f518 77156077 ntdll!LdrpInitializeProcess+0xe95
    0020f568 77153663 ntdll!_LdrpInitialize+0x78
    0020f578 00000000 ntdll!LdrInitializeThunk+0x10

    ntdll!AVrfpLoadAndInitializeProvider:
    771950e1 push 1Ch
    771950e3 push offset ntdll! ?? ::FNODOBFM::`string'+0x762 (77140ed8)
    771950e8 call ntdll!_SEH_prolog4 (77142c0c)
    771950ed mov byte ptr [ebp-19h],0
    771950f1 mov esi,dword ptr [ebp+8]
    771950f4 test byte ptr [ntdll!AVrfpDebug (771ce380)],1
    771950fb je ntdll!AVrfpLoadAndInitializeProvider+0x2b (7719510c)
    771950fd push dword ptr [esi+0Ch]
    77195100 push offset ntdll! ?? ::FNODOBFM::`string' (7713a708)
    77195105 call ntdll!DbgPrint (7710f593)
    7719510a pop ecx
    7719510b pop ecx
    7719510c mov dword ptr [ebp-28h],offset ntdll!RtlpCsVerifyDoNotBreak+0x7 (771ce178)
    77195113 xor eax,eax
    77195115 mov word ptr [ebp-2Ch],ax
    77195119 mov eax,208h
    7719511e mov word ptr [ebp-2Ah],ax
    77195122 push offset SharedUserData+0x30 (7ffe0030)
    77195127 lea eax,[ebp-2Ch]
    7719512a push eax
    7719512b call ntdll!RtlAppendUnicodeToString (7714eed2)
    77195130 push offset ntdll!SlashSystem32SlashString (771551f0)
    77195135 lea eax,[ebp-2Ch]
    77195138 push eax
    77195139 call ntdll!RtlAppendUnicodeStringToString (771472be)
    7719513e lea ebx,[esi+10h]
    77195141 push ebx
    77195142 lea eax,[esi+8]
    77195145 push eax
    77195146 push 0
    77195148 push dword ptr [ebp-28h]
    7719514b call ntdll!LdrLoadDll (771522b8)
    77195150 test eax,eax
    77195152 jge ntdll!AVrfpLoadAndInitializeProvider+0x98 (77195179)
    77195154 push dword ptr [ebp-28h]
    77195157 push eax
    77195158 push dword ptr [esi+0Ch]
    7719515b mov eax,dword ptr [ntdll!PebLdr+0xc (771c788c)]
    77195160 ff7030 push dword ptr [eax+30h]
    77195163 push offset ntdll! ?? ::FNODOBFM::`string' (7713a6c6)
    77195168 call ntdll!DbgPrint (7710f593)
    7719516d add esp,14h
    77195170 mov byte ptr [ebp-19h],1
    77195174 jmp ntdll!AVrfpLoadAndInitializeProvider+0x21c (771952fd)
    77195179 and dword ptr [ebp-4],0
    7719517d push dword ptr [ebx]
    7719517f call ntdll!RtlImageNtHeader (7714fa29)
    77195184 mov edi,eax

    The AppVerifier package comes with a GUI frontend, together with a stack of verifier providers. A verifier provider is a special DLL which accepts DLL_PROCESS_VERIFIER. The WinNT.h header file in DDK 3790.1830 contains everything needed for implementing a verifier provider.


    #define WIN32_LEAN_AND_MEAN
    #include <Windows.h>
     
    static RTL_VERIFIER_DLL_DESCRIPTOR aDlls[] = {{}};
    static RTL_VERIFIER_PROVIDER_DESCRIPTOR vpd = {sizeof(RTL_VERIFIER_PROVIDER_DESCRIPTOR), aDlls};
    
    BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD fdwReason, PRTL_VERIFIER_PROVIDER_DESCRIPTOR* pVPD)
    {
      UNREFERENCED_PARAMETER(hinstDLL);
      if(fdwReason == DLL_PROCESS_VERIFIER)
        *pVPD = &vpd;
      return TRUE;
    }
    

    You can read A Debugging Approach to Application Verifier for more information.

    DllNXOptions

    DisableExceptionChainValidation (REG_DWORD)

    SEHOP (Structured Exception Handling Overwrite Protection) is a feature introduced by Windows Vista SP1 to protect a certain kind of attack. However this feature would cause compatibility issues to some existing applications, so the purpose of this value is to workaround for legacy applications by disabling SEHOP feature.

    (to be continued...)

  • Rubato and Chord

    Use Windows Debuggers for Non-Debugging Tasks

    • 2 Comments

    Many people who has been using Emacs for decades were shocked when they heard that Emacs is actually a text editor instead of an operating system.

    - vi advocator

    Sharing a similar spirit as Emacs, Windows Debuggers are also super good at non-debugging tasks.

    Calculator

    The builtin expression evaluator of Windows Debuggers can be used as a handy calculator:

    0:000> ?? 1+2+3
    int 0n6
    
    0:000> .formats 0x00905a4d
    Evaluate expression:
      Hex:     00000000`00905a4d
      Decimal: 9460301
      Octal:   0000000000000044055115
      Binary:  00000000 00000000 00000000 00000000 00000000 10010000 01011010 01001101
      Chars:   ......ZM
      Time:    Mon Apr 20 19:51:41 1970
      Float:   low 1.32567e-038 high 0
      Double:  4.67401e-317

    Process Manager

    I try not to use the term Task Manager, since the name is already occupied by taskmgr.exe, and we have nothing to do with tasks (taskmgr also has nothing to do with tasks).

    0:000> .tlist -v windbg*

    Shell

    0:000> .shell tlist -t
    0:000> .shell dir
    0:000> !!dir

    Examine Export Table

    cdb.exe -y . -z ntdll.dll -c "x ntdll!*; q"

    Binary Editor

    The basic idea is to load a portion of file data into the address space of a debuggee, perform some inspection or modification, then write back to the file. WinDBG even has a Memory Window, which makes it a perfect GUI Hex Editor!

    To examine the file information such like name, permission and size, use the .shell command.

    0:000> .shell dir *.netmodule 
    
    10/23/2011  12:12 PM             2,048 bar.netmodule
    10/23/2011  12:12 PM             2,048 foo.netmodule
    
    .shell: Process exited
    Press ENTER to continue
    

    We can load the file into the debugee's user mode virtual address space, given that the pages are commited.

    0:000> .dvalloc 0n2048
    Allocated 1000 bytes starting at 00020000
    0:000> .readmem foo.netmodule 20000 L0n2048
    Reading 800 bytes.
    0:000> dt ntdll!_IMAGE_DOS_HEADER 20000
       +0x000 e_magic          : 0x5a4d
       +0x002 e_cblp           : 0x90
       +0x004 e_cp             : 3
       +0x006 e_crlc           : 0
       +0x008 e_cparhdr        : 4
       +0x00a e_minalloc       : 0
       +0x00c e_maxalloc       : 0xffff
       +0x00e e_ss             : 0
       +0x010 e_sp             : 0xb8
       +0x012 e_csum           : 0
       +0x014 e_ip             : 0
       +0x016 e_cs             : 0
       +0x018 e_lfarlc         : 0x40
       +0x01a e_ovno           : 0
       +0x01c e_res            : [4] 0
       +0x024 e_oemid          : 0
       +0x026 e_oeminfo        : 0
       +0x028 e_res2           : [10] 0
       +0x03c e_lfanew         : 0n128
    0:000> dt -r ntdll!_IMAGE_NT_HEADERS 20000+0n128
       +0x000 Signature        : 0x4550
       +0x004 FileHeader       : _IMAGE_FILE_HEADER
          +0x000 Machine          : 0x14c
          +0x002 NumberOfSections : 2
          +0x004 TimeDateStamp    : 0x4ea39447
          +0x008 PointerToSymbolTable : 0
          +0x00c NumberOfSymbols  : 0
          +0x010 SizeOfOptionalHeader : 0xe0
          +0x012 Characteristics  : 0x2102
       +0x018 OptionalHeader   : _IMAGE_OPTIONAL_HEADER
          +0x000 Magic            : 0x10b
          +0x002 MajorLinkerVersion : 0x8 ''
          +0x003 MinorLinkerVersion : 0 ''
          +0x004 SizeOfCode       : 0x400
          +0x008 SizeOfInitializedData : 0x200
          +0x00c SizeOfUninitializedData : 0
          +0x010 AddressOfEntryPoint : 0x22de
          +0x014 BaseOfCode       : 0x2000
          +0x018 BaseOfData       : 0x4000
          +0x01c ImageBase        : 0x400000
          +0x020 SectionAlignment : 0x2000
          +0x024 FileAlignment    : 0x200
          +0x028 MajorOperatingSystemVersion : 4
          +0x02a MinorOperatingSystemVersion : 0
          +0x02c MajorImageVersion : 0
          +0x02e MinorImageVersion : 0
          +0x030 MajorSubsystemVersion : 4
          +0x032 MinorSubsystemVersion : 0
          +0x034 Win32VersionValue : 0
          +0x038 SizeOfImage      : 0x6000
          +0x03c SizeOfHeaders    : 0x200
          +0x040 CheckSum         : 0
          +0x044 Subsystem        : 3
          +0x046 DllCharacteristics : 0x8540
          +0x048 SizeOfStackReserve : 0x100000
          +0x04c SizeOfStackCommit : 0x1000
          +0x050 SizeOfHeapReserve : 0x100000
          +0x054 SizeOfHeapCommit : 0x1000
          +0x058 LoaderFlags      : 0
          +0x05c NumberOfRvaAndSizes : 0x10
          +0x060 DataDirectory    : [16] _IMAGE_DATA_DIRECTORY
             +0x000 VirtualAddress   : 0
             +0x004 Size             : 0
    0:000> !dh 20000
    0:000> .writemem foo.netmodule 20000 L0n2048
    0:000> .dvfree 20000 0

    Be cautious while using .dvfree, the debugger simply backed this by VirtualFreeEx:

    kernel32!VirtualFreeEx
    dbgeng!LiveUserDebugServices::FreeVirtual+0x1e
    dbgeng!LiveUserTargetInfo::FreeVirtual+0x47
    dbgeng!DotDvFree+0xcb
    dbgeng!DotCommand+0x3f
    dbgeng!ProcessCommands+0x4e1
    dbgeng!ProcessCommandsAndCatch+0x49
    dbgeng!Execute+0x2b9
    dbgeng!DebugClient::ExecuteWide+0x6a
    windbg!ProcessCommand+0x156
    windbg!ProcessEngineCommands+0xb2
    windbg!EngineLoop+0x366
    kernel32!BaseThreadStart+0x37
    

    This means .dvfree can be used to free any block of virtual memory owned by the target process.

    When .dvfree is used without the /d option, it would use MEM_RELEASE instead of MEM_DECOMMIT (this can be verified with .dbgdbg), and you have to make sure BaseAddress is the value returned by !address and Size is always zero.

  • Rubato and Chord

    Undocumented WinDBG

    • 2 Comments

    Abstraction and encapsulation are good because they make it easier to build complex systems, however, there are times you have to peek inside the abstraction and demistify the encapsulation. This is especially true for debugging and performance tuning (I will not talk about reverse engineering this time). Familiar yourself with the right tools are very important, and one way to achieve this is to debug into these tools...

    Before getting started, I would like to wetting your appetite with a few questions:

    1. Are there undocumented commands in the Windows Debuggers?
    2. WinDBG supports child process debugging via .childdbg, does Visual Studio Debugger provide the same thing?
    3. Why do I need to put symbols on the target machine while remote debugging .NET applications in Visual Studio 2010?
    4. Why does the command foobarbaz not working as expected?

    Let's revisit the commands in WinDBG. There are three types of commands available:

    1. Debugger Commands
      These are the internal commands, examples are bp, g and pt. Internal commands are case insensitive in most cases (as and aS are exceptions), you can get a brief help of these commands by typing a single question mark (?).
    2. Meta Commands (a.k.a. Dot Commands)
      Meta commands always start with a dot, examples are .childdbg, .dvalloc and .dvfree. Meta commands are case insensitive, you can get a brief help of these commands using the meta command .help.
    3. Extension Commands
      Extension commands are provided by separate debugger extension modules, example are !dh and !sym. Debugging Tools for Windows has provided the SDK for implementing such extension DLLs, and it's fully documented (Writing New Debugger Extensions). Extension commands are case sensitive, some extensions (e.g. Son of Strike) has exported a same function under different names (e.g. SOS!CLRStack, SOS!ClrStack and SOS!clrstack).

    The documented way of working with extensions contains a few commands like .chain, .extmatch, .load, .loadby, .setdll.unload and .unloadall, an undocumented command .extcmds is also available.

    0:000> .extmatch /e dbghelp *
    !dbghelp.chksym
    !dbghelp.dh
    !dbghelp.homedir
    !dbghelp.lmi
    !dbghelp.stackdbg
    !dbghelp.sym
    

    However, once you understand how these extension DLLs work, you can dump the PE/COFF Export Directory to get a full list:

    0:000> .sympath .
    Symbol search path is: .
    Expanded Symbol search path is: .
    
    0:000> .reload /f dbghelp.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files\Debugging Tools for Windows (x64)\dbghelp.dll -
    
    0:000> x dbghelp!*
    00000000`68013dd0 dbghelp!ExtensionApiVersion (<no parameter info>)
    00000000`68013de0 dbghelp!WinDbgExtensionDllInit (<no parameter info>)
    00000000`68014090 dbghelp!fptr (<no parameter info>)
    00000000`68014130 dbghelp!vc7fpo (<no parameter info>)
    00000000`680141b0 dbghelp!stackdbg (<no parameter info>)
    00000000`680146e0 dbghelp!stack_force_ebp (<no parameter info>)
    00000000`680148c0 dbghelp!sym (<no parameter info>)
    00000000`68014a50 dbghelp!symsrv (<no parameter info>)
    00000000`68014bd0 dbghelp!lminfo (<no parameter info>)
    00000000`68015060 dbghelp!lmi (<no parameter info>)
    00000000`680164e0 dbghelp!itoldyouso (<no parameter info>)
    00000000`68016720 dbghelp!chksym (<no parameter info>)
    00000000`68016960 dbghelp!block (<no parameter info>)
    00000000`68016b50 dbghelp!omap (<no parameter info>)
    00000000`68016e10 dbghelp!homedir (<no parameter info>)
    00000000`68017030 dbghelp!srcfiles (<no parameter info>)
    00000000`68018a90 dbghelp!dh (<no parameter info>)
    
    0:000> !itoldyouso
    !IToldYouSo <module> [symbol]
    
    !IToldYouSo tests the validity of a module against a symbol file.
    The module can be specified by either its name or base address.
    If a symbol file is not specified, then the loaded symbol is tested.
    Otherwise, if a pdb or dbg symbol file path is specified, it is tested
    against the loaded module.
    

    Most of the meta commands are implemented in dbgeng.dll, a few are implemented inside specific debuggers (e.g. .browse and .wtitle are WinDBG specific meta commands). Some meta commands are only available for certain modes/targets/platforms, for example, .kdfiles is only available for live kernel debugging on x86-based and Itanium-based processors.

    Some meta commands might be undocumented (e.g. .aliascmds and .sxcmds) or documented separately (e.g. .perfer_dml is documented in dml.doc which can be found under the WinDBG installation folder, .jdinfo is only documented in .help), there is no centralized place to get all these commands.

    The following command would dump all the dot commands exported by dbgeng.dll (dbgeng!DotCommand is an exception, which is not a meta command):

    0:000> x dbgeng!Dot*
    00000000`517aba70 dbgeng!DotRestart = <no type information>
    00000000`51801c70 dbgeng!DotExtMatch = <no type information>
    00000000`517a9ac0 dbgeng!DotLogAppend = <no type information>
    00000000`517ae600 dbgeng!DotTTime = <no type information>
    00000000`517a3d70 dbgeng!DotEchoTimestamps = <no type information>
    00000000`517aacd0 dbgeng!DotPromptAllow = <no type information>
    00000000`517a53e0 dbgeng!DotEventStr = <no type information>
    00000000`517aa0b0 dbgeng!DotOCommand = <no type information>
    00000000`5179de00 dbgeng!DotPageIn = <no type information>
    00000000`517a09f0 dbgeng!DotCache = <no type information>
    00000000`517aaa80 dbgeng!DotPrintf = <no type information>
    00000000`517a60a0 dbgeng!DotFnRet = <no type information>
    00000000`5179d3c0 dbgeng!DotThread = <no type information>
    00000000`517a9a40 dbgeng!DotLocale = <no type information>
    00000000`517a46f0 dbgeng!DotEndSrv = <no type information>
    00000000`517a33b0 dbgeng!DotDumpOff = <no type information>
    00000000`517a5900 dbgeng!DotExr = <no type information>
    00000000`517a2880 dbgeng!DotCxr = <no type information>
    00000000`517a4f50 dbgeng!DotEvents = <no type information>
    00000000`5179fb30 dbgeng!DotApplyDbp = <no type information>
    00000000`517ae350 dbgeng!DotTrap = <no type information>
    00000000`517a98c0 dbgeng!DotKFrames = <no type information>
    00000000`517aca70 dbgeng!DotSleep = <no type information>
    00000000`5179f9e0 dbgeng!DotAllowImageMapping = <no type information>
    00000000`517a3e20 dbgeng!DotEcxr = <no type information>
    00000000`517a5e30 dbgeng!DotFixImports = <no type information>
    00000000`517aa3b0 dbgeng!DotOpenDump = <no type information>
    00000000`517a0500 dbgeng!DotBpSync = <no type information>
    00000000`5179d9b0 dbgeng!DotProcess = <no type information>
    00000000`517ac280 dbgeng!DotServers = <no type information>
    00000000`5179dd80 dbgeng!DotKernelKill = <no type information>
    00000000`517a2ea0 dbgeng!DotDmlStart = <no type information>
    00000000`517ab890 dbgeng!DotRecordBranches = <no type information>
    00000000`517a06c0 dbgeng!DotBugCheck = <no type information>
    00000000`517a2bd0 dbgeng!DotDebugSwWow = <no type information>
    00000000`517a1510 dbgeng!DotContext = <no type information>
    00000000`517a4740 dbgeng!DotEnumTag = <no type information>
    00000000`517a8b40 dbgeng!DotIf = <no type information>
    00000000`5179db50 dbgeng!DotFiber = <no type information>
    00000000`517ab810 dbgeng!DotReboot = <no type information>
    00000000`517a03a0 dbgeng!DotBpCmds = <no type information>
    00000000`5183b550 dbgeng!DotDmlFlow = <no type information>
    00000000`517ace20 dbgeng!DotSrcPath = <no type information>
    00000000`517a3380 dbgeng!DotDumpDebug = <no type information>
    00000000`517adf50 dbgeng!DotProcessInfo = <no type information>
    00000000`5179f950 dbgeng!DotAllowBpBaConvert = <no type information>
    00000000`517ab960 dbgeng!DotReCxr = <no type information>
    00000000`518568c0 dbgeng!DotPCmd = <no type information>
    00000000`517aba10 dbgeng!DotReload = <no type information>
    00000000`5179fe60 dbgeng!DotAttach = <no type information>
    00000000`517aa070 dbgeng!DotNoVersion = <no type information>
    00000000`518eb230 dbgeng!DotLines = <no type information>
    00000000`517a2dd0 dbgeng!DotDmlFile = <no type information>
    00000000`517a7740 dbgeng!DotFormats = <no type information>
    00000000`517accb0 dbgeng!DotSrcNoisy = <no type information>
    00000000`517a3170 dbgeng!DotDumpCab = <no type information>
    00000000`517a8e00 dbgeng!DotIgnoreWowKdContext = <no type information>
    00000000`517ad880 dbgeng!DotTList = <no type information>
    00000000`517ab3c0 dbgeng!DotReadMem = <no type information>
    00000000`517acb20 dbgeng!DotSrcFix = <no type information>
    00000000`517a5670 dbgeng!DotExPtr = <no type information>
    00000000`517a6540 dbgeng!DotForceBranchTrace = <no type information>
    00000000`517a3540 dbgeng!DotDumpPOff = <no type information>
    00000000`517aec40 dbgeng!DotWake = <no type information>
    00000000`517a0370 dbgeng!DotBlock = <no type information>
    00000000`517abcb0 dbgeng!DotSendFile = <no type information>
    00000000`517a62d0 dbgeng!DotFor = <no type information>
    00000000`517a3140 dbgeng!DotDrivers = <no type information>
    00000000`517a9b10 dbgeng!DotLogFile = <no type information>
    00000000`517a37d0 dbgeng!DotDvAlloc = <no type information>
    00000000`517ab0a0 dbgeng!DotPop = <no type information>
    00000000`517d9c20 dbgeng!DotDump = <no type information>
    00000000`517a7c30 dbgeng!DotFrame = <no type information>
    00000000`517af540 dbgeng!DotHelp = <no type information>
    00000000`517a9be0 dbgeng!DotLogOpen = <no type information>
    00000000`517af8b0 dbgeng!DotCommand = <no type information>
    00000000`517a9c00 dbgeng!DotNetSyms = <no type information>
    00000000`517ace80 dbgeng!DotStepFilter = <no type information>
    00000000`517a1660 dbgeng!DotContinue = <no type information>
    00000000`517a05d0 dbgeng!DotBreakin = <no type information>
    00000000`517a4170 dbgeng!DotEnableLongStatus = <no type information>
    00000000`517a6680 dbgeng!DotForceSystemInit = <no type information>
    00000000`517a5f90 dbgeng!DotFnEnt = <no type information>
    00000000`517a0e70 dbgeng!DotCatch = <no type information>
    00000000`517a9840 dbgeng!DotKdFiles = <no type information>
    00000000`517a0ea0 dbgeng!DotChain = <no type information>
    00000000`517a2380 dbgeng!DotCrash = <no type information>
    00000000`5183a420 dbgeng!DotAsm = <no type information>
    00000000`517ac900 dbgeng!DotShowSymFailures = <no type information>
    00000000`517ac1b0 dbgeng!DotServer = <no type information>
    00000000`517ac800 dbgeng!DotShowReadFailures = <no type information>
    00000000`517a6810 dbgeng!DotForEach = <no type information>
    00000000`517e9d60 dbgeng!DotEventLog = <no type information>
    00000000`517ad210 dbgeng!DotSymOpt = <no type information>
    00000000`517a6620 dbgeng!DotForceRadixOutput = <no type information>
    00000000`517a7160 dbgeng!DotFpo = <no type information>
    00000000`517a3fa0 dbgeng!DotEffMach = <no type information>
    00000000`517a7d80 dbgeng!DotFrameRel = <no type information>
    00000000`517ae430 dbgeng!DotTss = <no type information>
    00000000`51725a30 dbgeng!DotAliasCmds = <no type information>
    00000000`517a41d0 dbgeng!DotEnableUnicode = <no type information>
    00000000`517aa030 dbgeng!DotNoShell = <no type information>
    00000000`517aa540 dbgeng!DotOutMask = <no type information>
    00000000`517a9880 dbgeng!DotKdTrans = <no type information>
    00000000`517a3d20 dbgeng!DotEchoTime = <no type information>
    00000000`517aa800 dbgeng!DotPreferDml = <no type information>
    00000000`517a9a10 dbgeng!DotLeave = <no type information>
    00000000`517a7ec0 dbgeng!DotFrameEbpFix = <no type information>
    00000000`517a05a0 dbgeng!DotBreak = <no type information>
    00000000`517a3a30 dbgeng!DotDvFree = <no type information>
    00000000`517a54a0 dbgeng!DotExpr = <no type information>
    00000000`517a30b0 dbgeng!DotDbgDbg = <no type information>
    00000000`517a5930 dbgeng!DotExtCmds = <no type information>
    00000000`517aed10 dbgeng!DotWriteMem = <no type information>
    00000000`517ad040 dbgeng!DotSxCmds = <no type information>
    00000000`517a59e0 dbgeng!DotExtPath = <no type information>
    00000000`517aafc0 dbgeng!DotPush = <no type information>
    00000000`517ad0e0 dbgeng!DotSymFix = <no type information>
    00000000`517a2420 dbgeng!DotCreate = <no type information>
    00000000`517ac2b0 dbgeng!DotShell = <no type information>
    00000000`517a3010 dbgeng!DotDo = <no type information>
    00000000`517aec90 dbgeng!DotWhile = <no type information>
    00000000`517a9d40 dbgeng!DotNetUse = <no type information>
    00000000`517ae170 dbgeng!DotTimeZone = <no type information>
    00000000`517a0f90 dbgeng!DotChildDbg = <no type information>
    00000000`517a1130 dbgeng!DotClients = <no type information>
    00000000`517a4680 dbgeng!DotEndPSrv = <no type information>
    00000000`517a8910 dbgeng!DotHoldMem = <no type information>
    00000000`517a8ea0 dbgeng!DotImgScan = <no type information>
    00000000`517ad3e0 dbgeng!DotTListFromNtQuerySystemInformation = <no type information>
    00000000`517a4b40 dbgeng!DotEventCode = <no type information>
    00000000`517a3630 dbgeng!DotDumpAddRgn = <no type information>
    00000000`517a9790 dbgeng!DotJdInfo = <no type information>
    00000000`517ad330 dbgeng!DotSymPath = <no type information>
    00000000`517ade40 dbgeng!DotTime = <no type information>
    00000000`517a5430 dbgeng!DotExePath = <no type information>
    00000000`517a8d60 dbgeng!DotIgnoreMissingPages = <no type information>
    00000000`517aeaa0 dbgeng!DotTxtSym = <no type information>
    00000000`517a3c70 dbgeng!DotEchoCpuNum = <no type information>
    00000000`517a3c20 dbgeng!DotEcho = <no type information>
    00000000`517aa010 dbgeng!DotNoEngErr = <no type information>
    00000000`517aa7d0 dbgeng!DotPCache = <no type information>
    00000000`5179f8b0 dbgeng!DotAllowExecCmds = <no type information>
    00000000`517a1ab0 dbgeng!DotCorDll = <no type information>
    00000000`517a26e0 dbgeng!DotCreateDir = <no type information>
    00000000`517a1860 dbgeng!DotCopySym = <no type information>
    00000000`517a9af0 dbgeng!DotLogClose = <no type information>
    00000000`517a4230 dbgeng!DotTypeOpt = <no type information>
    00000000`517a0b70 dbgeng!DotCall = <no type information>
    00000000`517abc10 dbgeng!DotSecure = <no type information>
    00000000`517aa230 dbgeng!DotOFilter = <no type information>
    00000000`517a11f0 dbgeng!DotCloseHandle = <no type information>
    00000000`517ab180 dbgeng!DotQuitLock = <no type information>
    00000000`517a9990 dbgeng!DotLastEvent = <no type information>
    00000000`517a2220 dbgeng!DotCorStack = <no type information>
    

    Also, you might have noticed that .elif and .else donnot have their corresponding exports like .if does, they just look like meta commands (the document called them Command Tokens). To understand how the command line got parsed and executed (like a simplified version of compiler front end), I would recommend debugging the debugger by setting a breakpoint on the Win32 Beep function (kernel32!Beep or KERNELBASE!Beep) and use .beep, below is what I got on my Windows XP box:

    0:000> k
    kernel32!Beep
    windbg!DirectCommand+0x12a
    windbg!CmdExecuteCmd+0x90
    windbg!WinCommand::OnNotify+0x467
    windbg!WinBase::BaseProc+0xc91
    USER32!InternalCallWinProc+0x28
    USER32!UserCallWinProcCheckWow+0x150
    USER32!SendMessageWorker+0x4a5
    USER32!SendMessageW+0x7f
    MSFTEDIT!CW32System::SendMessage+0x3d
    MSFTEDIT!CTxtWinHost::TxNotify+0x97
    MSFTEDIT!RichEditWndProc+0x19b
    USER32!InternalCallWinProc+0x28
    USER32!UserCallWinProcCheckWow+0x150
    USER32!DispatchMessageWorker+0x306
    USER32!DispatchMessageW+0xf
    windbg!ProcessNonDlgMessage+0x2a2
    windbg!ProcessPendingMessages+0x64
    windbg!wmain+0x24a
    windbg!_initterm_e+0x163
    kernel32!BaseProcessStart+0x23
    
    0:000> k
    kernel32!Beep
    ntsd!UiCommand+0x287
    ntsd!MainLoop+0x48f
    ntsd!main+0x232
    ntsd!_initterm_e+0x163
    kernel32!BaseProcessStart+0x23
    
    0:000> k
    kernel32!Beep
    cdb!UiCommand+0x287
    cdb!MainLoop+0x48f
    cdb!main+0x232
    cdb!_initterm_e+0x163
    kernel32!BaseProcessStart+0x23
    
    0:000> k
    kernel32!Beep
    kd!UiCommand+0x287
    kd!MainLoop+0x48f
    kd!main+0x1ed
    kd!_initterm_e+0x163
    kernel32!BaseProcessStart+0x23
    

    In order to answer why Visual Studio lacks support for child process debugging, you need to know how the user mode native debugger works, especially how to write a debug event loop. I have an example in Data Breakpoints, and I've purposely left a note in the "A few things to mention" section #4. You can use these hints and debug into the Visual Studio Debugger.

    Regarding the symbol for remote debugging, you will need some background of metadata (ECMA-335 Partition 2) and PDB, also a bit of ICorDebug and ICLRData (mscordacwks!CLRDataCreateInstance). Launch procmon.exe and see who is consuming the symbol files.

    What if a WinDBG command fails to do the job, or is not working as expected? There is no magic, just grab a debugger and debug it, in fact the debugger team has presented you a gift called .dbgdbg to make things a little easier!

    0:000> uf dbgeng!DotReadMem
    dbgeng!DotReadMem:
    0213fc90 8bff            mov     edi,edi
    0213fc92 55              push    ebp
    0213fc93 8bec            mov     ebp,esp
    0213fc95 81ec64080000    sub     esp,864h
    0213fc9b a10c513302      mov     eax,dword ptr [dbgeng!__security_cookie (0233510c)]
    0213fca0 33c5            xor     eax,ebp
    0213fca2 8945fc          mov     dword ptr [ebp-4],eax
    0213fca5 833d4450360200  cmp     dword ptr [dbgeng!g_Process (02365044)],0
    0213fcac 750c            jne     dbgeng!DotReadMem+0x2a (0213fcba)
    
    dbgeng!DotReadMem+0x1e:
    0213fcae 6a00            push    0
    0213fcb0 6818100000      push    1018h
    0213fcb5 e8b6ba1400      call    dbgeng!ReportError (0228b770)
    
    dbgeng!DotReadMem+0x2a:
    0213fcba a1004d3302      mov     eax,dword ptr [dbgeng!g_SymOptions (02334d00)]
    0213fcbf 2500000400      and     eax,40000h
    0213fcc4 740c            je      dbgeng!DotReadMem+0x42 (0213fcd2)
    
    dbgeng!DotReadMem+0x36:
    0213fcc6 6a00            push    0
    0213fcc8 682a100000      push    102Ah
    0213fccd e89eba1400      call    dbgeng!ReportError (0228b770)
    
    dbgeng!DotReadMem+0x42:
    0213fcd2 c785c0f7ffff00000000 mov dword ptr [ebp-840h],0
    0213fcdc 8d8dbcf7ffff    lea     ecx,[ebp-844h]
    0213fce2 51              push    ecx
    0213fce3 6a0d            push    0Dh
    0213fce5 e876c41400      call    dbgeng!StringValue (0228c160)
    0213fcea 8985c4f7ffff    mov     dword ptr [ebp-83Ch],eax
    0213fcf0 8b154c1a3502    mov     edx,dword ptr [dbgeng!g_CurCmd (02351a4c)]
    0213fcf6 8995e4f7ffff    mov     dword ptr [ebp-81Ch],edx
    0213fcfc a14c1a3502      mov     eax,dword ptr [dbgeng!g_CurCmd (02351a4c)]
    0213fd01 668b8dbcf7ffff  mov     cx,word ptr [ebp-844h]
    0213fd08 668908          mov     word ptr [eax],cx
    0213fd0b c785e8f7ffff00000100 mov dword ptr [ebp-818h],10000h
    0213fd15 c785ecf7ffff00000000 mov dword ptr [ebp-814h],0
    0213fd1f 6800001000      push    100000h
    0213fd24 6a03            push    3
    0213fd26 6a01            push    1
    0213fd28 8d95e8f7ffff    lea     edx,[ebp-818h]
    0213fd2e 52              push    edx
    0213fd2f 8d85c8f7ffff    lea     eax,[ebp-838h]
    0213fd35 50              push    eax
    0213fd36 e8053b0400      call    dbgeng!GetRange (02183840)
    0213fd3b 33c9            xor     ecx,ecx
    0213fd3d 8b95e4f7ffff    mov     edx,dword ptr [ebp-81Ch]
    0213fd43 66890a          mov     word ptr [edx],cx
    0213fd46 6a00            push    0
    0213fd48 6880000000      push    80h
    0213fd4d 6a03            push    3
    0213fd4f 6a00            push    0
    0213fd51 6a00            push    0
    0213fd53 6800000080      push    80000000h
    0213fd58 8b85c4f7ffff    mov     eax,dword ptr [ebp-83Ch]
    0213fd5e 50              push    eax
    0213fd5f ff158c743302    call    dword ptr [dbgeng!kernel32_CreateFileW_Ptr (0233748c)]
    0213fd65 8985f4f7ffff    mov     dword ptr [ebp-80Ch],eax
    0213fd6b 83bdf4f7ffffff  cmp     dword ptr [ebp-80Ch],0FFFFFFFFh
    0213fd72 7512            jne     dbgeng!DotReadMem+0xf6 (0213fd86)
    

    As you can see, the .readmem command makes use of kernel32!CreateFileW with dwDesiredAccess = GENERIC_READ and dwShareMode = 0 (instead of FILE_SHARE_READ), that would explain why you got an error "Unable to open file" in some cases. A quick debugging showed CreateFileW failed and last error value is ERROR_SHARING_VIOLATION.

    The documented syntax for .readmem is .readmem FileName Range, while there is no notes for how big the Range value can go, I'll leave this as a homework for you, cheers.

  • Rubato and Chord

    Side Effects of Debugger

    • 2 Comments

    A target program might behave differently if it is being debugged, sometimes this can be very annoying. Also, these behavior deviations can be leveraged by anti-debugging.

    IsDebuggerPresent and CheckRemoteDebuggerPresent are well known APIs to tell if a program is attached by a debugger.

    0:000> uf KERNELBASE!IsDebuggerPresent KERNELBASE!IsDebuggerPresent:
    7512f41b 64a118000000    mov     eax,dword ptr fs:[00000018h]
    7512f421 8b4030          mov     eax,dword ptr [eax+30h]
    7512f424 0fb64002        movzx   eax,byte ptr [eax+2]
    7512f428 c3              ret

    CloseHandle would raise an exception under a debugger, as stated by MSDN:

    If the application is running under a debugger, the function will throw an exception if it receives either a handle value that is not valid or a pseudo-handle value.

    Windows heap manager would use debug heap (note: this has nothing to do with the CRT Debug Heap) if a program was launched from debugger:

    • The heap manager makes use of FLG_HEAP_ENABLE_TAIL_CHECK, FLG_HEAP_ENABLE_FREE_CHECK and FLG_HEAP_VALIDATE_PARAMETERS flags in PEB!NtGlobalFlags.
    • Low Fragmentation Heap might be disabled.
    • Heap functions might throw SEH, an article covering this can be found at debuginfo.com.
    • Debug heap can be turned off by setting the environment variable _NO_DEBUG_HEAP = 1.
    • Windows debuggers has a command line option -hd which specifies that the debug heap should not be used.

    OutputDebugString, we've have a dedicated topic on it.

    SetUnhandledExceptionFilter, a decent article can be found at debuginfo.com. A simple detouring is to intercept IsDebugPortPresent and return FALSE.

    NtSetInformationThread can be used to hide (detach) a thread from debugger.

    In addition, the target program can check its own integrity or the integrity of the system.

    • PEB and TEB, this is exactly what IsDebuggerPresent has used.
    • DebugPort, this is used by the kernel (EPROCESS). NtQueryInformationProcess from NTDLL can be used to retrieve this information.
    • INT3 and thread context, as we've already demonstrated here.
    • Environment variable, parent process, process startup information.
    • Image File Execution Options.
    • Call stack and register. If the debugger makes use of func-eval, conditional breakpoints with side effects, or caused some execution flow changes, it can be detected.

    A few things to mention:

    • You cannot attach a debugger to a program if the program is already attached by another debugger.
    • Attaching a debugger to a program can fail in many ways, such like loader lock, timeout and break-in thread creation failure. That is one reason why JIT debugging failed to work.
    • 64bit application cannot be debugged by a 32bit debugger, if you try to create a 64bit process from a 32bit process with debug creation flag, you always ended in failure. DebugActiveProcess would fail if a 32bit debugger tried to attach to a 64bit target.
    • Digital media application can take advantage of the windows kernel to protect itself from being debugged.
    • You should be cautious if you are debugging something that the debugger relies on (a GUI symbolic source level debugger relies on even more things), otherwise you would end up with deadlock or other strange behaviors.
    • Global Flags can affect the behavior of a program if running under a debugger (e.g. loader snaps).
    • CLR behaves very differently under a debugger (e.g. JIT compiler, GC).
  • Rubato and Chord

    Using Function Evaluation in WinDBG

    • 1 Comments

    People who develop debuggers would know in theory you cannot have a perfect disassembler (especially for x86) and stepper (especially for Step Over). People who develop commercial debuggers would know Function Evaluation (a.k.a. funceval) is a big challenge while implementing an Expression Evaluator. And people who develop the Visual Studio Debugger would face other difficulties - Interop Debugging, Edit & Continue.

    In this article, I'm not going to explain the bloody details of funceval, I just demonstrate how to use funceval in WinDBG and how powerful it is.

    Previously we mentioned the .call command in Microsoft Binary Technologies, at that time we were not able to invoke the function since we don't have private symbols - funceval requires private symbol since debugger needs to understand the calling convention, which is stripped out in public symbol.

    While I cannot use private symbols writting articles for this blog (private symbol is Microsoft privacy, also debugging without private symbol is much more fun), the way I'd take is to create a proxy DLL:


    #include <Windows.h>
    
    VOID WINAPI SetLastError(DWORD dwErrCode){}
    

    Now compile the code into a DLL, with PDB file generated:

    cl.exe funceval.cpp /D UNICODE /Fd /GS- /LD /Od /Zi /link /NOENTRY /NODEFAULTLIB /RELEASE /SUBSYSTEM:CONSOLE

    In order to use the proxy DLL, we will use the following approach:

    1. Launch a debug session.
    2. Allocate memory from debugee process.
    3. Inject the proxy DLL into the allocated memory (note that we skipped PE relocation).
    4. Load private symbol of the proxy DLL.
    5. Use WinDBG .call command to kick off funceval from proxy DLL.
    6. Change the IP register to the real address we want to execute.
    7. Start evaluating.

    Here is the automation script, enjoy!

    $$ cdb.exe -xe cpr -c "$$>a< .\funceval.txt" notepad.exe
    
    .echo [Launch Script]
    
    $$ change the following value to the size of funceval.dll
    r $t1 = 0n1536
    
    bp @$exentry; g
    
    .echo [Allocate Memory]
    .foreach ( token { .dvalloc @$t1 } ) {
    	aS alias token
    	.block {
    		.if ($spat("${alias}", "[0-9a-f]+")) {
    			r $t2 = 0x${alias}
    		}
    	}
    	ad /q alias
    }
    
    .printf "[Load Helper DLL(base address = %p, size = %p)]\n", @$t2, @$t1
    .readmem funceval.dll @$t2 (@$t1+@$t2-1)
    
    .block {
    	.sympath .
    }
    
    $$.symopt+ 0x40
    
    .reload /s /f funceval.dll=$t2
    
    $$.symopt- 0x40
    
    .echo [Function Evaluation]
    .call /s funceval!SetLastError kernel32!SetLastError(7777)
    
    g
    
    !gle
    
    .dvfree @$t2 0
    

    And here is the output from my machine:

    [Allocate Memory]
    [Load Helper DLL(base address = 00020000, size = 00000600)]
    Reading 600 bytes.
    [Function Evaluation]
    Thread is set up for call, 'g' will execute.
    WARNING: This can have serious side-effects,
    including deadlocks and corruption of the debuggee.
    LastErrorValue: (Win32) 0x1e61 (7777) - <Unable to get error code text>
    LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0
    Freed 0 bytes starting at 00020000
  • Rubato and Chord

    A Debugging Approach to Windows RT

    • 1 Comments

    Recently I got a Surface with Windows RT. Needless to mention, it's wonderful!

    I've figured out some quick facts about Windows RT by looking at the C:\Windows\system32\ntdll.dll from Windows RT:

    • A complete NT (instead of WINCE) kernel and almost a full stack of Windows operating system.
    • Almost the same PE/COFF structure as x86.
    • Using ARM's "non classic RISC style" Thumb-2 instruction set (pImageNtHeaders->FileHeader.Machine == IMAGE_FILE_MACHINE_ARMNT), which has great code density, and in turn gives smaller binary and less memory pressure.

    I've never had a chance to debug Thumb-2 code before, so I've listed the things I need to grasp:

    • Fundamental ARM architecture and Thumb-2 instructions.
    • ABI (Application Binary Interface), calling convention and exception handling mechanism.
    • Programming and debugging.

    Programming

    Although Visual Studio 2012 doesn't have an ARM version, it does included the x86 cross toolchain which allows targeting ARM architecture, which can be found from %ProgramFiles(x86)%\Microsoft Visual Studio 11.0\VC\bin\x86_arm\. By setting the correct environment variable (INCLUDE, LIB, LIBPATH, PATH) we can generate ARM module smoothly.

    int main()
    {
      return 0;
    }
    
    C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\x86_arm>cl.exe /link /NODEFAULTLIB /ENTRY:main test.cpp
    Microsoft (R) C/C++ Optimizing Compiler Version 17.00.51106.1 for ARM Copyright (C) Microsoft Corporation.  All rights reserved.
    
    test.cpp
    Microsoft (R) Incremental Linker Version 11.00.51106.1
    Copyright (C) Microsoft Corporation.  All rights reserved.
    
    /out:test.exe
    /machine:arm
    /NODEFAULTLIB
    /ENTRY:main
    test.obj
    

    By default Visual Studio doesn't allow generating native ARM binary, and the restriction was from a MSBuild property named WindowsSDKDesktopARMSupport, by setting this property to true I could target ARM native without an issue.

    Another problem is that Windows SDK doesn't have the ARM version of import libraries, which means we don't have files like gdi32.lib and shell32.lib. The solution would be creating one either by writing a DEF file, or creating a stub module. Since we can get the ARM version of DLLs from Windows RT, it is easy to dump the export directory and create DEF file automatically, as long as the DLLs we use is not exporting mangled name (otherwise I would prefer to use stub module approach).

    This is what I got while running my very first hello.exe on Windows RT :)

    For pure managed code programming, the .NET runtime in Surface RT comes with the standard C# compiler:

    class Hello
    {
      static void Main()
      {
      }
    }
    
    C:\Users\Reiley\Desktop>"%WINDIR%\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /debug+ /platform:anycpu hello.cs
    Microsoft (R) Visual C# Compiler version 4.0.30319.17929 
    for Microsoft (R) .NET Framework 4.5
    Copyright (C) Microsoft Corporation. All rights reserved.
    

    And this time I got something worse:

    After changing the compiler flag to target ARM instead of AnyCPU, it became better and I was again greeted with the "Windows cannot verify the digital signature ..." dialog.

    C:\Users\Reiley\Desktop>"%WINDIR%\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /debug+ /platform:arm hello.cs
    

    Debugging

    The x86 and amd64 version of WinDBG both support various architectures including ARM Thumb-2, which means you can open dump files from a PC. This is a good place to get started, and actually I copied notepad.exe from Surface to my PC and used cdb.exe -z notepad.exe to familiar myself with the ARM PE structure and disassembly.

    There is no ARM version of WinDBG available for public download. Also ntsd.exe is no longer shipped as part of the Windows since Vista.

    Visual Studio 2012 comes with a great debugger, together with a fantastic remote debugging agent. Jason Zander already explained how to setup remote debugging from his great article "What you need to know about developing for Windows on ARM". I'll just put a conclusion here:

    1. Visual Studio 2012 doesn't have ARM version, in fact only x86 version is available.
    2. Visual Studio 2012 supports remote kernel debugging, however there is no direct way to enable kernel debugging on Windows RT device.
    3. Visual Studio 2012 comes with an ARM version of remote debugging agent, which makes it possible to do user mode debugging on nearly all processes. To unleash the power, run remote debugging agent (msvsmon) as a service under an administrator account.
    4. User mode debugging on Windows RT is powerful enough that you can do whatever hack you like.

     

    (to be continued...)

  • Rubato and Chord

    Windows 8 and conhost.exe

    • 1 Comments

    While debugging a console application on Windows 8, I noticed the console application is trying to create a process in the very beginning:

    windbg.exe -xe ld:ntdll.dll -c "bm ntdll!*CreateProcess*; g; k" cmd.exe

    CommandLine: cmd.exe
    ModLoad: 000007ff`01d60000 000007ff`01f1e000   ntdll.dll
    ntdll!RtlUserThreadStart:
    000007ff`01d7c3d0 4883ec48        sub     rsp,48h
    Processing initial command 'bm ntdll!*CreateProcess*; g; k'
    0:000> bm ntdll!*CreateProcess*; g; k
      1: 000007ff`01d90f60 @!"ntdll!RtlCreateProcessParametersEx"
      2: 000007ff`01d63070 @!"ntdll!NtCreateProcessEx"
    breakpoint 2 redefined
      2: 000007ff`01d63070 @!"ntdll!ZwCreateProcessEx"
      3: 000007ff`01e1bf74 @!"ntdll!RtlCreateProcessReflection"
      4: 000007ff`01da8bb4 @!"ntdll!RtlpCreateProcessRegistryInfo"
      5: 000007ff`01e1ceac @!"ntdll!RtlCreateProcessParameters"
      6: 000007ff`01d63651 @!"ntdll!ZwCreateProcess"
    breakpoint 6 redefined
      6: 000007ff`01d63651 @!"ntdll!NtCreateProcess"
    Breakpoint 1 hit
    Child-SP          RetAddr           Call Site
    000000bf`8268e558 000007fe`feea02a4 ntdll!RtlCreateProcessParametersEx
    000000bf`8268e560 000007fe`feea00be KERNELBASE!ConsoleLaunchServerProcess+0x60
    000000bf`8268e5f0 000007fe`fee95d40 KERNELBASE!ConsoleAllocate+0xf6
    000000bf`8268e8c0 000007fe`fee7f6db KERNELBASE!ConsoleInitialize+0x1d1
    000000bf`8268e950 000007fe`fee7230d KERNELBASE!KernelBaseBaseDllInitialize+0x4dd
    000000bf`8268ec20 000007ff`01d6b9be KERNELBASE!KernelBaseDllInitialize+0xd
    000000bf`8268ec50 000007ff`01d8b3fc ntdll!LdrpCallInitRoutine+0x3e
    000000bf`8268eca0 000007ff`01d8a88b ntdll!LdrpInitializeNode+0x192
    000000bf`8268eda0 000007ff`01d8e74e ntdll!LdrpInitializeGraph+0x6f
    000000bf`8268ede0 000007ff`01d8c322 ntdll!LdrpInitializeGraph+0x8d
    000000bf`8268ee20 000007ff`01d8cc02 ntdll!LdrpPrepareModuleForExecution+0x1a5
    000000bf`8268ee70 000007ff`01d8337b ntdll!LdrpLoadDll+0x344
    000000bf`8268f0a0 000007ff`01d9264f ntdll!LdrLoadDll+0xa7
    000000bf`8268f120 000007ff`01d91826 ntdll!LdrpInitializeProcess+0x1664
    000000bf`8268f420 000007ff`01d7c1ae ntdll!_LdrpInitialize+0x1565e
    000000bf`8268f490 00000000`00000000 ntdll!LdrInitializeThunk+0xe

    0:000> dc rbx
    000000bf`8268e660  00000000 00000000 00000000 00000000  ................
    000000bf`8268e670  003f005c 005c003f 003a0043 0057005c  \.?.?.\.C.:.\.W.
    000000bf`8268e680  004e0049 004f0044 00530057 0073005c  I.N.D.O.W.S.\.s.
    000000bf`8268e690  00730079 00650074 0033006d 005c0032  y.s.t.e.m.3.2.\.
    000000bf`8268e6a0  006f0063 0068006e 0073006f 002e0074  c.o.n.h.o.s.t...
    000000bf`8268e6b0  00780065 00200065 00780030 00660066  e.x.e. .0.x.f.f.
    000000bf`8268e6c0  00660066 00660066 00660066 00000000  f.f.f.f.f.f.....
    000000bf`8268e6d0  8268e960 000000bf 00000008 00000000  `.h.............

    This means conhost.exe process on Windows 8 will be created by the console application itself, instead of the CSRSS. And conhost.exe would always have the native bitness (on Windows 8 64bit version, only 64bit version of conhost.exe is available).

    Now debug into conhost.exe using .childdbg, it's pretty clear that conhost.exe is in charge of drawing the console window, handling user inputs and communicate with the console application:

    0  Id: 124c.d34 Suspend: 1 Teb: 000007f6`3311b000 Unfrozen
    Child-SP          RetAddr           Call Site
    00000094`85aefb38 000007f6`33b91146 ntdll!NtWaitForSingleObject+0xa
    00000094`85aefb40 000007ff`00c9167e conhost!ConsoleIoThread+0xda
    00000094`85aefd80 000007ff`01d7c3f1 KERNEL32!BaseThreadInitThunk+0x1a
    00000094`85aefdb0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
    #  1  Id: 124c.1428 Suspend: 1 Teb: 000007f6`3311e000 Unfrozen
    Child-SP          RetAddr           Call Site
    00000094`85b6fd28 000007ff`0140171e conhost!ConsoleWindowProc
    00000094`85b6fd30 000007ff`014014d7 USER32!UserCallWinProcCheckWow+0x13a
    00000094`85b6fdf0 000007f6`33b92fcc USER32!DispatchMessageWorker+0x1a7
    00000094`85b6fe70 000007ff`00c9167e conhost!ConsoleInputThread+0xd2
    00000094`85b6fed0 000007ff`01d7c3f1 KERNEL32!BaseThreadInitThunk+0x1a
    00000094`85b6ff00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

    And cmd.exe itself doesn't draw the console window at all:

    0  Id: 1aec.5ec Suspend: 1 Teb: 000007f7`b280d000 Unfrozen
    Child-SP          RetAddr           Call Site
    00000008`50fff5b8 000007fe`fee8f17c ntdll!NtDeviceIoControlFile+0xa
    00000008`50fff5c0 000007fe`fef0bb29 KERNELBASE!ConsoleCallServerGeneric+0x118
    00000008`50fff710 000007fe`fef0b986 KERNELBASE!ReadConsoleInternal+0x131
    00000008`50fff850 000007f7`b3621025 KERNELBASE!ReadConsoleW+0x1a
    00000008`50fff890 000007f7`b362bd3e cmd!ReadBufFromConsole+0x111
    00000008`50fff960 000007f7`b3604aae cmd!_chkstk+0x3820
    00000008`50fffae0 000007f7`b36042e4 cmd!Lex+0x4be
    00000008`50fffb50 000007f7`b362d560 cmd!Parser+0x128
    00000008`50fffba0 000007f7`b361b721 cmd!_chkstk+0x5032
    00000008`50fffc00 000007ff`00c9167e cmd!mystrchr+0x27d
    00000008`50fffc40 000007ff`01d7c3f1 KERNEL32!BaseThreadInitThunk+0x1a
    00000008`50fffc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

    The interesting thing is that if we use Spy++, it would report that the console window is associated with the main thread of cmd.exe process! I believe this is a hack in the underlying implementation of GetWindowThreadProcessId for backward compatibility. Also, Spy++ cannot be used to inspect conhost.exe message loop.

    Due to the side effects of IFEO Debugger, console application would fail to start if IFEO Debugger is enabled for conhost.exe.

    The following call stack showed conhost.exe just created a normal window from Console Input Thread:

    USER32!CreateWindowExW
    conhost!CreateWindowsWindow+0x105
    conhost!InitWindowsSubsystem+0x69
    conhost!ConsoleInputThread+0x22
    KERNEL32!BaseThreadInitThunk+0x1a
    ntdll!RtlUserThreadStart+0x1d
    0:001> du @rdx
    000007f6`33b9d460  "ConsoleWindowClass"
    0:001> du @r8
    00000025`f7dc37c0  "C:\WINDOWS\SYSTEM32\cmd.exe"

    When cmd.exe exits, the Console I/O Thread would be notified:

    ntdll!NtTerminateProcess+0xa
    ntdll!RtlExitUserProcess+0xb6
    conhost!ConsoleIoThread+0xac4
    KERNEL32!BaseThreadInitThunk+0x1a
    ntdll!RtlUserThreadStart+0x1d

    (to be continued...)

     

  • Rubato and Chord

    MACRO Revisited

    • 1 Comments

    Macro is powerful, but few people understand how it works. In theory, syntax highlighting for C/C++ is impossible due to the presence of Preprocessing Directives FDIS N3290 16 [cpp]. Sometimes I do feel that C++ is a mixture of three languages instead of a single language, I have to keep in mind that there are several Phases of Translation FDIS N3290 2.2 [lex.phases] when I was coding.

    NULL

    It turns out that most people who have been using the Win32 API and C Runtime Library for years don't know NULL is complicated than it looks. It is defined by both Windows headers and C Runtime headers, and guarded by macro. The reason behind this is to make most people happy (e.g. C++ standard requires NULL to be 0, while Standard C does not).

    /* WinDef.h */
    #ifndef NULL
    #ifdef __cplusplus
    #define NULL  0
    #else
    #define NULL  ((void *)0)
    #endif
    #endif

    WIN32_LEAN_AND_MEAN

    You probably have noticed that I've used this macro extensively in my blogs, here goes the official voice:

    WIN32_LEAN_AND_MEAN excludes APIs such as Cryptography, DDE, RPC, Shell, and Windows Sockets.

    So, if you don't need these APIs, WIN32_LEAN_AND_MEAN would make the life of compiler easier, plus, Precompiled Header, Intellisense and other code analysis tools would also benifit from it.

    UNICODE and _UNICODE

    UNICODE is used by Windows header files to support generic Conventions for Function Prototypes and Generic Data Types.

    _UNICODE is used by the C Runtime (CRT) header files to support Generic-Text Mappings.

    The following interesting snippet is distilled from ATL headers:

    /* atldef.h */
    #ifdef  _UNICODE
    #ifndef UNICODE
    #define UNICODE         // UNICODE is used by Windows headers
    #endif
    #endif
     
    #ifdef  UNICODE
    #ifndef _UNICODE
    #define _UNICODE        // _UNICODE is used by C-runtime/MFC headers
    #endif
    #endif

    TEXT __TEXT and _T _TEXT __T

    The following snippet is distilled from WinNT.h, which can be found from DDK/WDK and SDK/PSDK:

    /* WinNT.h */
    #ifdef  UNICODE
    #define __TEXT(quote) L##quote
    #else
    #define __TEXT(quote) quote
    #endif
    #define TEXT(quote) __TEXT(quote)

    So the following code is correct:

    _tprintf(TEXT("%s") TEXT("\n"), TEXT(__FILE__));
    

    But this is wrong:

    _tprintf(TEXT("%s" "\n"), __TEXT(__FILE__));
    

    And if UNICODE is defined, it turns out that you can (evilly) use:

    class LOST
    {
    };
    TEXT(OST) lost;

    The following snippet was distilled from tchar.h, which is a part of CRT:

    /* tchar.h */
    #ifdef  _UNICODE
    #define __T(x)      x
    #else
    #define __T(x)      L ## x
    #endif
     
    #define _T(x)       __T(x)
    #define _TEXT(x)    __T(x)

    Conclusion:

    1. Use TEXT if you are using none of the ATL, CRT and MFC.
    2. Use _T if you are using the ATL, CRT and MFC.
    3. Use _TEXT instead of _T if you are not as lazy as me.
    4. Don't use __T and __TEXT unless you have a special reason.

    DEBUG _DEBUG and NDEBUG

    NDEBUG is a part of the C Language Standard, which controls the behavior of assert:

    /* assert.h */
    #ifdef NDEBUG
    #define assert(_Expression) ((void)0)
    #else
    ...

    _DEBUG is defined by the Microsoft C++ Compiler when you compile with /LDd, /MDd and /MTd. The runtime libraries such like ATL, CRT and MFC make use of this macro.

    DEBUG is defined in ATL:

    /* atldef.h */
    #ifdef _DEBUG
    #ifndef DEBUG
    #define DEBUG
    #endif
    #endif

    NTDDI_VERSION WINVER _WIN32_WINNT _WIN32_WINDOWS _WIN32_IE VER_PRODUCTVERSION_W

    WINVER has been existing since 16bit Windows, and is still in using. Note that Windows NT 4.0 and Windows 95 both have WINVER defined as 0x0400.

    _WIN32_WINDOWS is used by Windows 95/98/Me.

    _WIN32_WINNT is used by the whole NT family.

    NTDDI_VERSION was introduced by Windows 2000, as Win9x and NT evolved into a single operating system. Plus, NTDDI_VERSION contains more information and is able to distinguish service packs. The latest sdkddkver.h has all the information you would want to know.

    _WIN32_IE was introduced because Internet Explorer shares many components with the shell (a.k.a. Windows Explorer), installing a new version of Internet Explorer would eventually replace a number of system components and even change the APIs.

    VER_PRODUCTVERSION_W can be found in ntverp.h, which is used by the NT team to maintain the product build.

    Conclusion:

    1. Use NTDDI_VERSION whenever possible.
    2. Don't use WINVER unless you have special reason.
    3. Forget about _WIN32_WINDOWS unless you are still targeting Win9x or Win32s.
    4. Don't use VER_PRODUCTVERSION_W unless you are writing low level code such like drivers and debugger extensions.

    _X86_ _AMD64_ _IA64_ and _M_AMD64 _M_IX86 _M_IA64 _M_X64

    _M_AMD64, _M_IX86, _M_IA64 and _M_X64 are defined by the Microsoft C++ Compiler according to the target processor architecture. _M_AMD64 and _M_X64 are equivalent.

    _X86_, _AMD64_ and _IA64_ are defined by Windows.h (there is no _X64_ at all, because AMD invented x86-64).

    /* Windows.h */
    #if !defined(_X86_) && !defined(_IA64_) && !defined(_AMD64_) && defined(_M_IX86)
    #define _X86_
    #endif
     
    #if !defined(_X86_) && !defined(_IA64_) && !defined(_AMD64_) && defined(_M_AMD64)
    #define _AMD64_
    #endif

    _WIN32 _WIN64 WIN32 _WINDOWS

    If bitness matters, but we don't care about architecture, we can use _WIN32 and _WIN64 provided by the Microsoft C++ Compiler. This is useful while defining data types and function prototypes. Note that _WIN32 and _WIN64 are not mutual exclusive, as _WIN32 is always defined (unless you are using DDK and writing 16bit code).

    WIN32 is defined by Windows header file WinDef.h, and is not widely used in Windows header files (TAPI being a negative example).

    _WINDOWS is a legacy thing in the 16bit era, you should hardly see it in 21st century.

    /* WinDef.h */
    // Win32 defines _WIN32 automatically,
    // but Macintosh doesn't, so if we are using
    // Win32 Functions, we must do it here
     
    #ifdef _MAC
    #ifndef _WIN32
    #define _WIN32
    #endif
    #endif //_MAC
     
    #ifndef WIN32
    #define WIN32
    #endif 

    UNREFERENCED_PARAMETER

    This macro is defined in WinNT.h along with DBG_UNREFERENCED_PARAMETER and DBG_UNREFERENCED_LOCAL_VARIABLE.

    /* WinNT.h */
    //
    // Macros used to eliminate compiler warning generated when formal // parameters or local variables are not declared. // // Use DBG_UNREFERENCED_PARAMETER() when a parameter is not yet // referenced but will be once the module is completely developed. // // Use DBG_UNREFERENCED_LOCAL_VARIABLE() when a local variable is not yet // referenced but will be once the module is completely developed. // // Use UNREFERENCED_PARAMETER() if a parameter will never be referenced. // // DBG_UNREFERENCED_PARAMETER and DBG_UNREFERENCED_LOCAL_VARIABLE will // eventually be made into a null macro to help determine whether there // is unfinished work. //

    (to be continued...)

  • Rubato and Chord

    The Main Thread Problem

    • 1 Comments

    Every few months I heard people asking the same question:

    Given a process ID (or handle), how can I get its main thread ID (or handle)?

    Normally that would raise another question:

    What is the definition of a main thread?

    While the Windows operating system doesn't have a concept called main thread, and threads donnot have parent-child relationship at all.

    Let's reuse the sample code from Pop Quiz - Debug Event Loop and Timeslice Quota:

    1. The WinMain function would return right after it creates two worker threads.
    2. The two worker threads run into an endless loop.

    If we compiled the code using cl.exe test.cpp, the generated test.exe would return immediately after we run it. If we take a quick debug, the call stack would look like this:

    test!ILT+10(WinMain)
    test!__tmainCRTStartup+0x154
    kernel32!BaseThreadInitThunk+0xd
    ntdll!RtlUserThreadStart+0x1d

    That's because the compiler has made the decision that we need to use CRT initialization, although actually we are not using it either explicitly or implicitly. It is the CRT exit code which called ntdll!RtlExitUserProcess and terminated our worker threads, and this is by design.

    Now let's switch to the following command:

    cl test.cpp /link /NODEFAULTLIB /ENTRY:WinMain /SUBSYSTEM:CONSOLE kernel32.lib

    As you can see, test.exe would enter an endless loop.

    Now let's try to give some possible definitions of main thread:

    1. The thread in which the CRT startup and exit code runs. (what if we are not using CRT at all? what if we start without CRT, then load a DLL that triggered the CRT initialization?)
    2. The thread which pumps window message for the main window. (what if we are not a GUI application? what in turn is the definition of a main window, and can we have two main windows?)
    3. The thread which runs through the OEP (Original Entry Point, we mentioned that in Data Breakpoints). (what if OEP has been covered several times?)
    4. The thread in which DllMain function get called with DLL_PROCESS_ATTACH and lpReserved is not NULL.
    5. The oldest thread.

    It looks like option 4 and 5 have the most clean definition. If we use option 4 then we should stay with the facts that a process may not have a main thread, and that's why we would normally end up with option 5.

    Which one do you prefer and what is your own definition? Which option do you think the Visual Studio Debugger would use?

     

Page 1 of 4 (35 items) 1234