• Ntdebugging Blog

    Great power. Great responsibility.


    When it comes to the registry, administrators are given great power to manually configure Windows to suit their needs, but even slight, seemingly innocuous changes to a particular key or value can have a drastic impact on basic operations of the system, even affecting its ability to boot properly.


    I recently had the pleasure of the debugging a black-screen system hang that occurred after applying security updates and rebooting.  After ruling out any “low-hanging fruit” such as deadlocks on executive resources, resource depletion, etc., I decided to survey how far along the boot had gotten.  In the output below we can tell that it’s fairly early in the boot process and that session zero is currently being setup.


    When a new session is created, the “Session Leader”(i.e. smss.exe instance not associated with a particular session) launches a new instance of smss.exe, who is then tasked with ensuring that the Windows subsystem gets setup properly, which includes loading and initializing win32k.sys and launching csrss.exe.


    3: kd> !process 0 0


    PROCESS 8e282840  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000

        DirBase: 00122000  ObjectTable: 97801e18  HandleCount: 554.

        Image: System


    PROCESS 94153ad8  SessionId: none  Cid: 0390    Peb: 7ffdf000  ParentCid: 0004

        DirBase: 03368020  ObjectTable: a13564f0  HandleCount:  19.

        Image: smss.exe


    PROCESS 92a55d90  SessionId: 0  Cid: 03c8    Peb: 7ffd9000  ParentCid: 0390

        DirBase: 03368040  ObjectTable: a95606e0  HandleCount:  10.

        Image: smss.exe


    PROCESS 92a56c48  SessionId: 0  Cid: 03d4    Peb: 7ffd9000  ParentCid: 03c8

        DirBase: 03368060  ObjectTable: a959ea28  HandleCount:  30.

        Image: csrss.exe


    So let’s dump out the threads for these session zero processes and see what they’re doing:


    1.  Notice how the Session Manager thread has been waiting for more than fifteen minutes for the Windows subsystem to load and initialize.


    3: kd> !process /s 0 0 0x17

    Searching processes with session id 0


    PROCESS 92a55d90  SessionId: 0  Cid: 03c8    Peb: 7ffd9000  ParentCid: 0390

        DirBase: 03368040  ObjectTable: a95606e0  HandleCount:  10.

        Image: smss.exe

        VadRoot 941e0578 Vads 8 Clone 0 Private 21. Modified 535. Locked 0.

        DeviceMap 97808b98

        Token                             a95767b8

        ElapsedTime                       00:15:32.714

        UserTime                          00:00:00.000

        KernelTime                        00:00:00.000

        QuotaPoolUsage[PagedPool]         6952

        QuotaPoolUsage[NonPagedPool]      384

        Working Set Sizes (now,min,max)  (125, 50, 345) (500KB, 200KB, 1380KB)

        PeakWorkingSetSize                125

        VirtualSize                       2 Mb

        PeakVirtualSize                   4 Mb

        PageFaultCount                    120

        MemoryPriority                    BACKGROUND

        BasePriority                      8

        CommitCharge                      29


            THREAD 92a5d030  Cid 03c8.03cc  Teb: 7ffdf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable

                941df930  SynchronizationEvent

                92a56c48  ProcessObject

            Not impersonating

            DeviceMap                 97808b98

            Owning Process            92a55d90       Image:         smss.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      1896           Ticks: 59778 (0:00:15:32.542)

            Context Switch Count      94             IdealProcessor: 0            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.046

            Win32 Start Address smss!NtProcessStartupW (0x4857d9a2)

            Stack Init 9dc89000 Current 9dc888c0 Base 9dc89000 Limit 9dc86000 Call 0

            Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            9dc888d8 81eb923a 92a5d030 97cb5120 92a5d0b8 nt!KiSwapContext+0x26

            9dc8891c 81eb4bca 92a5d030 00000000 00000002 nt!KiSwapThread+0x44f

            9dc88970 82040e83 00000002 9dc88aa8 00000001 nt!KeWaitForMultipleObjects+0x53d

            9dc88bfc 82040bf2 00000002 00000001 00000000 nt!ObpWaitForMultipleObjects+0x256

            9dc88d48 81e57c96 00000002 0008fb38 00000001 nt!NtWaitForMultipleObjects+0xcc

            9dc88d48 778d5d14 00000002 0008fb38 00000001 nt!KiSystemServicePostCall

            0008fac4 778d54a0 4857cc7e 00000002 0008fb38 ntdll!KiFastSystemCallRet

            0008fac8 4857cc7e 00000002 0008fb38 00000001 ntdll!NtWaitForMultipleObjects+0xc

            0008fb40 48579296 0008fb78 0008fb68 0008fbb0 smss!SmscpLoadSubSystem+0x9b

            0008fb80 4857ca8a 0008fbb0 00000000 00000000 smss!SmpExecuteCommand+0x8d

            0008fbc4 4857d0bc 00000000 00000000 00000000 smss!SmscpLoadSubSystemsForMuSession+0x182

            0008fbe8 4857b678 00000003 002417d8 00000000 smss!SmscMain+0xc2

            0008fc7c 4857d988 00000003 002417d8 002417e8 smss!wmain+0x50

            0008fcc0 77886885 00241898 779bde2d 00000000 smss!NtProcessStartupW_AfterSecurityCookieInitialized+0x221

            0008fd00 778b15d6 4857d9a2 7ffd9000 ffffffff ntdll!__RtlUserThreadStart+0x35

            0008fd18 00000000 4857d9a2 7ffd9000 00000000 ntdll!_RtlUserThreadStart+0x1b


    2.  Also, we can see that there’s a single active thread within the csrss.exe process, which is a red flag because we know that csrss.exe hosts the Desktop Thread and Raw Input Thread, among others.


    The user-mode portion of the Windows subsystem is implemented in csrss.exe and associated “ServerDlls” such as csrsrv.dll, winsrv.dll, basesrv.dll and, on Windows 7 and later, sxssrv.dll.  Also, csrss.exe hosts the Desktop thread and Raw Input thread, whose primary functions include handling inputs from the various input devices.


    PROCESS 92a56c48  SessionId: 0  Cid: 03d4    Peb: 7ffd9000  ParentCid: 03c8

        DirBase: 03368060  ObjectTable: a959ea28  HandleCount:  30.

        Image: csrss.exe

        VadRoot 9391c128 Vads 33 Clone 0 Private 193. Modified 60. Locked 0.

        DeviceMap 97808b98

        Token                             a9598b30

        ElapsedTime                       00:15:32.558

        UserTime                          00:00:00.000

        KernelTime                        00:00:03.182

        QuotaPoolUsage[PagedPool]         48312

        QuotaPoolUsage[NonPagedPool]      1584

        Working Set Sizes (now,min,max)  (582, 50, 345) (2328KB, 200KB, 1380KB)

        PeakWorkingSetSize                7285

        VirtualSize                       23 Mb

        PeakVirtualSize                   48 Mb

        PageFaultCount                    49628

        MemoryPriority                    BACKGROUND

        BasePriority                      13

        CommitCharge                      248


            THREAD 942c5590  Cid 03d4.03e4  Teb: 00000000 Win32Thread: 00000000 WAIT: (Executive) KernelMode Non-Alertable

                915e4078  NotificationEvent

            Not impersonating

            DeviceMap                 97808b98

            Owning Process            92a56c48       Image:         csrss.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      2516           Ticks: 59158 (0:00:15:22.870)

            Context Switch Count      1              IdealProcessor: 0            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address ati2mtag!IRQMGR_WorkerThreadRoutine (0xa1ccf340)

            Stack Init 9dcfd000 Current 9dcfcc30 Base 9dcfd000 Limit 9dcfa000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5

            ChildEBP RetAddr  Args to Child             

            9dcfcc48 81eb923a 942c5590 942c5618 00000000 nt!KiSwapContext+0x26

            9dcfcc8c 81e54f38 942c5590 00000000 942c5590 nt!KiSwapThread+0x44f

            9dcfcce4 a1d78724 915e4078 00000000 00000000 nt!KeWaitForSingleObject+0x492

            9dcfcd00 a1c13340 908be398 915e4070 00000000 VIDEOPRT!VideoPortWaitForSingleObject+0x53

            9dcfcd14 a1cce17f 908be398 915e4070 00000000 ati2mtag!IRQMgrMP_WaitForSingleObject+0x20

            9dcfcd6c a1ccf355 93f45000 93f45000 9dcfcdc0 ati2mtag!PassiveRing_WorkerThreadRoutine+0x6f

            9dcfcd7c 81fe301c 93f45000 ad8fc28d 00000000 ati2mtag!IRQMGR_WorkerThreadRoutine+0x15

            9dcfcdc0 81e4beee a1ccf340 93f45000 00000000 nt!PspSystemThreadStartup+0x9d

            00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16


    3.  Having seen this, we now know why the system is perpetually hung:  Csrss.exe is not running properly.  Because there is a video driver worker thread running, but the Desktop Thread and Raw Input Thread are not running, it appears that csrss has attempted to terminate.  The termination has not completed because of the video-driver worker thread performing a non-alertable wait.


    4.  Next, we need to check for any state in the dump that might tell us why csrss.exe attempted to terminate:


    3: kd> dt nt!eprocess 92a56c48 LastThreadExitStatus

       +0x184 LastThreadExitStatus : 0n-1073741619


    3: kd> !error 0n-1073741619

    Error code: (NTSTATUS) 0xc00000cd (3221225677) - The name limit for the local computer network adapter card was exceeded.


    After a quick search for STATUS_TOO_MANY_NAMES (0xc00000cd) through the source code, I was able to theorize that csrss.exe may have attempted the termination due to invalid command-line parameters.


    3: kd> vertarget

    Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (16 procs) Free x86 compatible

    Product: Server, suite: Enterprise TerminalServer SingleUserTS

    Built by: 6002.18881.x86fre.vistasp2_gdr.130707-1535

    Machine Name:

    Kernel base = 0x81e0d000 PsLoadedModuleList = 0x81f24c70

    Debug session time: Fri Oct 25 05:10:34.030 2013 (UTC - 5:00)

    System Uptime: 0 days 0:16:02.134


    3: kd> .process /p /r 92a56c48

    Implicit process is now 92a56c48

    Loading User Symbols



    3: kd> !peb

    PEB at 7ffd9000

    CommandLine:  'C:\Windows\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,1024 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16'


    Sure enough, there was additional command-line parameter that was not recognized on Vista/Windows Server 2008 SP2 (supported only on Windows 7 and later).  Once the invalid command-line parameter was removed, the server was able to boot normally again.


    So how did the invalid value get there?  It turns out that a logon script was setting the following registry value using an export from a Windows 7/Windows 2008 R2 machine where ServerDll=sxssrv,4 is a valid value.


    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems

    Name:  Windows

    Type:  REG_EXPAND_SZ


    Well, that concludes today’s segment, but in the timeless words of Uncle Ben remember “with great power comes great responsibility.”  As we just saw, this applies not only to those possessing a spider-sense, but also to Windows administrators.  J


    Until next time, happy debugging!

  • Ntdebugging Blog

    Debugging a Generation 2 Virtual Machine


    Hyper-V is based on the 440BX (PCI) chipset for emulation. The decision to use this chipset started years ago with Connectix Virtual PC.  The advantage of using an emulated chipset based on a popular motherboard like the 440BX, along with associated peripherals, is the compatibility with a large number of operating systems.


    Windows Server 2012 R2 introduced the Generation 2 Virtual Machine. It is a UEFI based design, removing emulated devices and replacing them with synthetic devices. Generation 2 VMs no longer support the following devices:

    • Legacy BIOS
    • COM Ports
    • Floppy Controller
    • DMA Controller
    • i8042 keyboard controller
    • PS/2 devices
    • Legacy NIC
    • IDE Controller
    • S3 video
    • PCI BUS
    • Programmable Interrupt Controller
    • Programmable Interrupt Timer
    • Super I/O Device


    After reading this list you might ask the question – how do I debug a Generation 2 VM?


    The COM port is not actually removed from a Generation 2 VM. The port is turned off by default and not present in the user interface. To enable it for debugging use the following steps.


    1.  Shutdown the VM.  You can verify the VM is off using the below PowerShell command.


    2.  Turn off secure boot using the following PowerShell Command.




    3.  Set a COM port path using the following PowerShell command where the path is equal the named pipe.




    4.  To confirm the COM port settings after making the change, use the following command.




    5.  Restart the Virtual Machine using the following command.

    Start-VM –Name VM2



    6.  Inside the guest VM, you can confirm that UEFI has been disabled with the following command. The results are False if UEFI was successfully disabled in step 2 above.




    7.  Enable Kernel Debugging using BCDEdit.

    BCDEdit /debug ON




    8.  Configure the debugger to connect to the pipe:



    9.  Connect the debugger and break in with Ctrl+Break:


Page 1 of 1 (2 items)