Great power. Great responsibility.

Great power. Great responsibility.

Rate This
  • Comments 1

When it comes to the registry, administrators are given great power to manually configure Windows to suit their needs, but even slight, seemingly innocuous changes to a particular key or value can have a drastic impact on basic operations of the system, even affecting its ability to boot properly.

 

I recently had the pleasure of the debugging a black-screen system hang that occurred after applying security updates and rebooting.  After ruling out any “low-hanging fruit” such as deadlocks on executive resources, resource depletion, etc., I decided to survey how far along the boot had gotten.  In the output below we can tell that it’s fairly early in the boot process and that session zero is currently being setup.

 

When a new session is created, the “Session Leader”(i.e. smss.exe instance not associated with a particular session) launches a new instance of smss.exe, who is then tasked with ensuring that the Windows subsystem gets setup properly, which includes loading and initializing win32k.sys and launching csrss.exe.

 

3: kd> !process 0 0

**** NT ACTIVE PROCESS DUMP ****

PROCESS 8e282840  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000

    DirBase: 00122000  ObjectTable: 97801e18  HandleCount: 554.

    Image: System

 

PROCESS 94153ad8  SessionId: none  Cid: 0390    Peb: 7ffdf000  ParentCid: 0004

    DirBase: 03368020  ObjectTable: a13564f0  HandleCount:  19.

    Image: smss.exe

 

PROCESS 92a55d90  SessionId: 0  Cid: 03c8    Peb: 7ffd9000  ParentCid: 0390

    DirBase: 03368040  ObjectTable: a95606e0  HandleCount:  10.

    Image: smss.exe

 

PROCESS 92a56c48  SessionId: 0  Cid: 03d4    Peb: 7ffd9000  ParentCid: 03c8

    DirBase: 03368060  ObjectTable: a959ea28  HandleCount:  30.

    Image: csrss.exe

 

So let’s dump out the threads for these session zero processes and see what they’re doing:

 

1.  Notice how the Session Manager thread has been waiting for more than fifteen minutes for the Windows subsystem to load and initialize.

 

3: kd> !process /s 0 0 0x17

Searching processes with session id 0

**** NT ACTIVE PROCESS DUMP ****

PROCESS 92a55d90  SessionId: 0  Cid: 03c8    Peb: 7ffd9000  ParentCid: 0390

    DirBase: 03368040  ObjectTable: a95606e0  HandleCount:  10.

    Image: smss.exe

    VadRoot 941e0578 Vads 8 Clone 0 Private 21. Modified 535. Locked 0.

    DeviceMap 97808b98

    Token                             a95767b8

    ElapsedTime                       00:15:32.714

    UserTime                          00:00:00.000

    KernelTime                        00:00:00.000

    QuotaPoolUsage[PagedPool]         6952

    QuotaPoolUsage[NonPagedPool]      384

    Working Set Sizes (now,min,max)  (125, 50, 345) (500KB, 200KB, 1380KB)

    PeakWorkingSetSize                125

    VirtualSize                       2 Mb

    PeakVirtualSize                   4 Mb

    PageFaultCount                    120

    MemoryPriority                    BACKGROUND

    BasePriority                      8

    CommitCharge                      29

 

        THREAD 92a5d030  Cid 03c8.03cc  Teb: 7ffdf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable

            941df930  SynchronizationEvent

            92a56c48  ProcessObject

        Not impersonating

        DeviceMap                 97808b98

        Owning Process            92a55d90       Image:         smss.exe

        Attached Process          N/A            Image:         N/A

        Wait Start TickCount      1896           Ticks: 59778 (0:00:15:32.542)

        Context Switch Count      94             IdealProcessor: 0            

        UserTime                  00:00:00.000

        KernelTime                00:00:00.046

        Win32 Start Address smss!NtProcessStartupW (0x4857d9a2)

        Stack Init 9dc89000 Current 9dc888c0 Base 9dc89000 Limit 9dc86000 Call 0

        Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5

        Kernel stack not resident.

        ChildEBP RetAddr  Args to Child             

        9dc888d8 81eb923a 92a5d030 97cb5120 92a5d0b8 nt!KiSwapContext+0x26

        9dc8891c 81eb4bca 92a5d030 00000000 00000002 nt!KiSwapThread+0x44f

        9dc88970 82040e83 00000002 9dc88aa8 00000001 nt!KeWaitForMultipleObjects+0x53d

        9dc88bfc 82040bf2 00000002 00000001 00000000 nt!ObpWaitForMultipleObjects+0x256

        9dc88d48 81e57c96 00000002 0008fb38 00000001 nt!NtWaitForMultipleObjects+0xcc

        9dc88d48 778d5d14 00000002 0008fb38 00000001 nt!KiSystemServicePostCall

        0008fac4 778d54a0 4857cc7e 00000002 0008fb38 ntdll!KiFastSystemCallRet

        0008fac8 4857cc7e 00000002 0008fb38 00000001 ntdll!NtWaitForMultipleObjects+0xc

        0008fb40 48579296 0008fb78 0008fb68 0008fbb0 smss!SmscpLoadSubSystem+0x9b

        0008fb80 4857ca8a 0008fbb0 00000000 00000000 smss!SmpExecuteCommand+0x8d

        0008fbc4 4857d0bc 00000000 00000000 00000000 smss!SmscpLoadSubSystemsForMuSession+0x182

        0008fbe8 4857b678 00000003 002417d8 00000000 smss!SmscMain+0xc2

        0008fc7c 4857d988 00000003 002417d8 002417e8 smss!wmain+0x50

        0008fcc0 77886885 00241898 779bde2d 00000000 smss!NtProcessStartupW_AfterSecurityCookieInitialized+0x221

        0008fd00 778b15d6 4857d9a2 7ffd9000 ffffffff ntdll!__RtlUserThreadStart+0x35

        0008fd18 00000000 4857d9a2 7ffd9000 00000000 ntdll!_RtlUserThreadStart+0x1b

 

2.  Also, we can see that there’s a single active thread within the csrss.exe process, which is a red flag because we know that csrss.exe hosts the Desktop Thread and Raw Input Thread, among others.

 

The user-mode portion of the Windows subsystem is implemented in csrss.exe and associated “ServerDlls” such as csrsrv.dll, winsrv.dll, basesrv.dll and, on Windows 7 and later, sxssrv.dll.  Also, csrss.exe hosts the Desktop thread and Raw Input thread, whose primary functions include handling inputs from the various input devices.

 

PROCESS 92a56c48  SessionId: 0  Cid: 03d4    Peb: 7ffd9000  ParentCid: 03c8

    DirBase: 03368060  ObjectTable: a959ea28  HandleCount:  30.

    Image: csrss.exe

    VadRoot 9391c128 Vads 33 Clone 0 Private 193. Modified 60. Locked 0.

    DeviceMap 97808b98

    Token                             a9598b30

    ElapsedTime                       00:15:32.558

    UserTime                          00:00:00.000

    KernelTime                        00:00:03.182

    QuotaPoolUsage[PagedPool]         48312

    QuotaPoolUsage[NonPagedPool]      1584

    Working Set Sizes (now,min,max)  (582, 50, 345) (2328KB, 200KB, 1380KB)

    PeakWorkingSetSize                7285

    VirtualSize                       23 Mb

    PeakVirtualSize                   48 Mb

    PageFaultCount                    49628

    MemoryPriority                    BACKGROUND

    BasePriority                      13

    CommitCharge                      248

 

        THREAD 942c5590  Cid 03d4.03e4  Teb: 00000000 Win32Thread: 00000000 WAIT: (Executive) KernelMode Non-Alertable

            915e4078  NotificationEvent

        Not impersonating

        DeviceMap                 97808b98

        Owning Process            92a56c48       Image:         csrss.exe

        Attached Process          N/A            Image:         N/A

        Wait Start TickCount      2516           Ticks: 59158 (0:00:15:22.870)

        Context Switch Count      1              IdealProcessor: 0            

        UserTime                  00:00:00.000

        KernelTime                00:00:00.000

        Win32 Start Address ati2mtag!IRQMGR_WorkerThreadRoutine (0xa1ccf340)

        Stack Init 9dcfd000 Current 9dcfcc30 Base 9dcfd000 Limit 9dcfa000 Call 0

        Priority 13 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5

        ChildEBP RetAddr  Args to Child             

        9dcfcc48 81eb923a 942c5590 942c5618 00000000 nt!KiSwapContext+0x26

        9dcfcc8c 81e54f38 942c5590 00000000 942c5590 nt!KiSwapThread+0x44f

        9dcfcce4 a1d78724 915e4078 00000000 00000000 nt!KeWaitForSingleObject+0x492

        9dcfcd00 a1c13340 908be398 915e4070 00000000 VIDEOPRT!VideoPortWaitForSingleObject+0x53

        9dcfcd14 a1cce17f 908be398 915e4070 00000000 ati2mtag!IRQMgrMP_WaitForSingleObject+0x20

        9dcfcd6c a1ccf355 93f45000 93f45000 9dcfcdc0 ati2mtag!PassiveRing_WorkerThreadRoutine+0x6f

        9dcfcd7c 81fe301c 93f45000 ad8fc28d 00000000 ati2mtag!IRQMGR_WorkerThreadRoutine+0x15

        9dcfcdc0 81e4beee a1ccf340 93f45000 00000000 nt!PspSystemThreadStartup+0x9d

        00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

 

3.  Having seen this, we now know why the system is perpetually hung:  Csrss.exe is not running properly.  Because there is a video driver worker thread running, but the Desktop Thread and Raw Input Thread are not running, it appears that csrss has attempted to terminate.  The termination has not completed because of the video-driver worker thread performing a non-alertable wait.

 

4.  Next, we need to check for any state in the dump that might tell us why csrss.exe attempted to terminate:

 

3: kd> dt nt!eprocess 92a56c48 LastThreadExitStatus

   +0x184 LastThreadExitStatus : 0n-1073741619

 

3: kd> !error 0n-1073741619

Error code: (NTSTATUS) 0xc00000cd (3221225677) - The name limit for the local computer network adapter card was exceeded.

 

After a quick search for STATUS_TOO_MANY_NAMES (0xc00000cd) through the source code, I was able to theorize that csrss.exe may have attempted the termination due to invalid command-line parameters.

 

3: kd> vertarget

Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (16 procs) Free x86 compatible

Product: Server, suite: Enterprise TerminalServer SingleUserTS

Built by: 6002.18881.x86fre.vistasp2_gdr.130707-1535

Machine Name:

Kernel base = 0x81e0d000 PsLoadedModuleList = 0x81f24c70

Debug session time: Fri Oct 25 05:10:34.030 2013 (UTC - 5:00)

System Uptime: 0 days 0:16:02.134

 

3: kd> .process /p /r 92a56c48

Implicit process is now 92a56c48

Loading User Symbols

.............

 

3: kd> !peb

PEB at 7ffd9000

CommandLine:  'C:\Windows\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,1024 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16'

 

Sure enough, there was additional command-line parameter that was not recognized on Vista/Windows Server 2008 SP2 (supported only on Windows 7 and later).  Once the invalid command-line parameter was removed, the server was able to boot normally again.

 

So how did the invalid value get there?  It turns out that a logon script was setting the following registry value using an export from a Windows 7/Windows 2008 R2 machine where ServerDll=sxssrv,4 is a valid value.

 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems

Name:  Windows

Type:  REG_EXPAND_SZ

 

Well, that concludes today’s segment, but in the timeless words of Uncle Ben remember “with great power comes great responsibility.”  As we just saw, this applies not only to those possessing a spider-sense, but also to Windows administrators.  J

 

Until next time, happy debugging!

Leave a Comment
  • Please add 3 and 4 and type the answer here:
  • Post
  • Thanks for sharing - again, awesome post. The error code 0xc00000cd and its description is very misleading in this scenario - I would spend hours in looking at registry settings related to network.

Page 1 of 1 (1 items)