Rubato and Chord

Reiley's technical blog

November, 2011

  • Rubato and Chord

    x86 Segment Addressing Revisited

    • 0 Comments

    Memory segmentation was first introduced to x86 family with 8086, to make it possible to access 1MB physical memory under 16bit addressing mode.

    Real Mode

    Logical address points directly into physical memory location.

    Logical address consists of two parts: segment and offset.

    Physical address is calculated as Segment * 16 + Offset, and if the A20 line is not enabled, the physical address is wrapped around into 1MiB range.

    Protected Mode

    Two stages of address translation would be used to arrive at a physical address: logical-address translation and linear address space paging.

    Logical address consists of two parts: segment selector and offset.

    Segment selector is used to determine the base address as well as the access rights. Segment selelctors are maintained in a table known as GDT (Global descriptor-table) or LDT (Local descriptor-table), which is referenced by GDTR or LDTR register.

    When a thread is running in kernel mode, fs:[0] points to the PCR (Processor Control Region). Let's find it by manually translating the logical address:

    kd> r fs
    fs=00000030
    
    kd> .formats @fs
    Evaluate expression:
      Hex:     00000030
      Decimal: 48
      Octal:   00000000060
      Binary:  00000000 00000000 00000000 00110000
      Chars:   ...0
      Time:    Thu Jan 01 08:00:48 1970
      Float:   low 6.72623e-044 high 0
      Double:  2.37152e-322
    

    According to the x86 architecture specification, we have Index = 6 (0000000000110), Table Indicator = GDT (0) and RPL (Requested Privilege Level) = 0.

    kd> dd @gdtr
    8003f000  00000000 00000000 0000ffff 00cf9b00
    8003f010  0000ffff 00cf9300 0000ffff 00cffb00
    8003f020  0000ffff 00cff300 200020ab 80008b04
    8003f030  f0000001 ffc093df 00000fff 0040f300
    8003f040  0400ffff 0000f200 00000000 00000000
    8003f050  a0000068 80008954 a0680068 80008954
    8003f060  2f40ffff 00009302 80003fff 0000920b
    8003f070  700003ff ff0092ff 0000ffff 80009a40
    

    We can get the layout of segment descriptor from x86 architecture specification:

    Now we can verify the address translation by looking at the memory contents:

    kd> dg @fs
                                      P Si Gr Pr Lo
    Sel    Base     Limit     Type    l ze an es ng Flags
    ---- -------- -------- ---------- - -- -- -- -- --------
    0030 ffdff000 00001fff Data RW Ac 0 Bg Pg P  Nl 00000c93
    kd> dd fs:[0]
    0030:00000000  805495d0 80549df0 80547000 00000000
    0030:00000010  00000000 00000000 00000000 ffdff000
    0030:00000020  ffdff120 0000001c 00000000 00000000
    0030:00000030  ffff2050 80545bb8 8003f400 8003f000
    0030:00000040  80042000 00010001 00000001 00000064
    0030:00000050  00000000 00000000 00000000 00000000
    0030:00000060  00000000 00000000 00000000 00000000
    0030:00000070  00000000 00000000 00000000 00000000
    
    kd> dd ffdff000
    ffdff000  805495d0 80549df0 80547000 00000000
    ffdff010  00000000 00000000 00000000 ffdff000
    ffdff020  ffdff120 0000001c 00000000 00000000
    ffdff030  ffff2050 80545bb8 8003f400 8003f000
    ffdff040  80042000 00010001 00000001 00000064
    ffdff050  00000000 00000000 00000000 00000000
    ffdff060  00000000 00000000 00000000 00000000
    ffdff070  00000000 00000000 00000000 00000000

    It's time to dump the KPCR structure:

    kd> dt nt!_KPCR ffdff000
       +0x000 NtTib            : _NT_TIB
       +0x01c SelfPcr          : 0xffdff000 _KPCR
       +0x020 Prcb             : 0xffdff120 _KPRCB
       +0x024 Irql             : 0x1c ''
       +0x028 IRR              : 0
       +0x02c IrrActive        : 0
       +0x030 IDR              : 0xffff2050
       +0x034 KdVersionBlock   : 0x80545bb8 Void
       +0x038 IDT              : 0x8003f400 _KIDTENTRY
       +0x03c GDT              : 0x8003f000 _KGDTENTRY
       +0x040 TSS              : 0x80042000 _KTSS
       +0x044 MajorVersion     : 1
       +0x046 MinorVersion     : 1
       +0x048 SetMember        : 1
       +0x04c StallScaleFactor : 0x64
       +0x050 DebugActive      : 0 ''
       +0x051 Number           : 0 ''
       +0x052 Spare0           : 0 ''
       +0x053 SecondLevelCacheAssociativity : 0 ''
       +0x054 VdmAlert         : 0
       +0x058 KernelReserved   : [14] 0
       +0x090 SecondLevelCacheSize : 0
       +0x094 HalReserved      : [16] 0
       +0x0d4 InterruptMode    : 0
       +0x0d8 Spare1           : 0 ''
       +0x0dc KernelReserved2  : [17] 0
       +0x120 PrcbData         : _KPRCB
    
    kd> !pcr
    KPCR for Processor 0 at ffdff000:
        Major 1 Minor 1
    	NtTib.ExceptionList: 805495d0
    	    NtTib.StackBase: 80549df0
    	   NtTib.StackLimit: 80547000
    	 NtTib.SubSystemTib: 00000000
    	      NtTib.Version: 00000000
    	  NtTib.UserPointer: 00000000
    	      NtTib.SelfTib: 00000000
    
    	            SelfPcr: ffdff000
    	               Prcb: ffdff120
    	               Irql: 0000001c
    	                IRR: 00000000
    	                IDR: ffff2050
    	      InterruptMode: 00000000
    	                IDT: 8003f400
    	                GDT: 8003f000
    	                TSS: 80042000
    
    	      CurrentThread: 80552840
    	         NextThread: 00000000
    	         IdleThread: 80552840
    
    	          DpcQueue: 
    

     

  • Rubato and Chord

    The Main Thread Problem

    • 1 Comments

    Every few months I heard people asking the same question:

    Given a process ID (or handle), how can I get its main thread ID (or handle)?

    Normally that would raise another question:

    What is the definition of a main thread?

    While the Windows operating system doesn't have a concept called main thread, and threads donnot have parent-child relationship at all.

    Let's reuse the sample code from Pop Quiz - Debug Event Loop and Timeslice Quota:

    1. The WinMain function would return right after it creates two worker threads.
    2. The two worker threads run into an endless loop.

    If we compiled the code using cl.exe test.cpp, the generated test.exe would return immediately after we run it. If we take a quick debug, the call stack would look like this:

    test!ILT+10(WinMain)
    test!__tmainCRTStartup+0x154
    kernel32!BaseThreadInitThunk+0xd
    ntdll!RtlUserThreadStart+0x1d

    That's because the compiler has made the decision that we need to use CRT initialization, although actually we are not using it either explicitly or implicitly. It is the CRT exit code which called ntdll!RtlExitUserProcess and terminated our worker threads, and this is by design.

    Now let's switch to the following command:

    cl test.cpp /link /NODEFAULTLIB /ENTRY:WinMain /SUBSYSTEM:CONSOLE kernel32.lib

    As you can see, test.exe would enter an endless loop.

    Now let's try to give some possible definitions of main thread:

    1. The thread in which the CRT startup and exit code runs. (what if we are not using CRT at all? what if we start without CRT, then load a DLL that triggered the CRT initialization?)
    2. The thread which pumps window message for the main window. (what if we are not a GUI application? what in turn is the definition of a main window, and can we have two main windows?)
    3. The thread which runs through the OEP (Original Entry Point, we mentioned that in Data Breakpoints). (what if OEP has been covered several times?)
    4. The thread in which DllMain function get called with DLL_PROCESS_ATTACH and lpReserved is not NULL.
    5. The oldest thread.

    It looks like option 4 and 5 have the most clean definition. If we use option 4 then we should stay with the facts that a process may not have a main thread, and that's why we would normally end up with option 5.

    Which one do you prefer and what is your own definition? Which option do you think the Visual Studio Debugger would use?

     

  • Rubato and Chord

    What is Autos Window?

    • 0 Comments

    The developers in Microsoft have done a great job by bringing a great number of nice features, however, some of these features are poorly documented or even not documented at all.

    Autos Window in the Visual Studio Debugger is one of the best example of the gaps between implementation and documentation. I'm sure you have seen this window before, as it's shown by default while you are debugging, if not, you can always find it from the Debug menu:

    The MSDN document hasn't been updated since Visual Studio .NET 2003, which can be found at http://msdn.microsoft.com/en-us/library/aa290702.aspx:

    The Autos window displays variables used in the current statement and the previous statement. (For Visual Basic, it displays variables in the current statement and three statements on either side of the current statement.) 

    The document does leave us with some questions:

    1. What's the definition of variable? Is this the same concept as we are talking about the compiler frontends?
    2. What's the definition of used? What happens if I have a parameter that never got reference inside the function?
    3. What's the definition of statement? Is this the same concept as we are talking about parsing in compiler frontends?
    4. What's the meaning of previous statement? Is it based on the execution order (note that some statements donnot execute at all, such like plain variable definition) or just a lexical thing?
    5. How should I interpret "three statements on either side of the current statement"?

    Generally speaking, debuggers wouldn't care about source code, it knows nothing about the C++ preprocessing or the syntax (the only exception is the expression evaluator, which would be another topic). The magic of source debugging was brought by symbol files. The private PDB contains the path and checksum of source files, as well as the line number information (PDB actually supports line and column number, and this has been used in C# already), debugger just read from the symbols to get these information.

    Let's open Visual Studio 2010 and take a look at the following snippet in C:

    int x;
    int y;
    int z;
    int a;
    int b;
    int c;
    int main(int argc)
    {
      return b + c;
    }
    

    Set a breakpoint at the line of return statement, bring up the Autos Window:

    First, it looks like the ordering in the Autos Window is based on variable name, so we got another question "what would happen if there is a duplicated variable name?".

    Now let's switch to another snippet:

    int foo(int* x)
    {
      return *x;
    }
    
    int main(int argc)
    {
      return foo(&argc);
    }
    

    When we hit the breakpoint, the Autos Window would be:

    Step into the function foo and step out, now the Autos Window would look like:

    As you see, the return value is displayed as "foo returned" in the Autos Window, if you right click and select Add Watch, you will be welcomed with an error message CXX0013: Error: missing operator.

    Okay, I've given enough questions and hints, now it is time to try out by yourself, hopefully you could understand a bit more on what the Autos Window is and the way it works.

Page 1 of 1 (3 items)