Would you like to join the world’s best and most elite debuggers to enable the success of Microsoft solutions?
As a trusted advisor to our top customers you will be working with to the most experienced IT professionals and developers in the industry. You will influence our product teams in sustained engineering efforts to drive improvements in our products.
This role involves deep analysis of product source code and debugging to solve problems in multi-million dollar configurations and will give you an opportunity to stretch your critical thinking skills. During the course of debugging, you will uncover opportunities to improve the customer experience while influencing the current and future design of our products.
In addition to providing support to customers while being the primary interface to our sustained engineering teams, you will also have the opportunity to work with new technologies and unreleased software. Through our continuous investment in depth training and hands-on experience with tough customer challenges you will become the world’s best in this area. Expect to partner with many various roles at Microsoft launching a very successful career!
We have positions open at our sites in Charlotte, NC USA; and Issaquah, WA USA.
Learn more about what an Escalation Engineer does at:
Profile: Ron Stock, CTS Escalation Engineer - Microsoft Customer Service & Support - What is CSS?
Microsoft JobsBlog JobCast with Escalation Engineer Jeff Dailey
Microsoft JobsBlog JobCast with Escalation Engineer Scott Oseychik
When it comes to the registry, administrators are given great power to manually configure Windows to suit their needs, but even slight, seemingly innocuous changes to a particular key or value can have a drastic impact on basic operations of the system, even affecting its ability to boot properly.
I recently had the pleasure of the debugging a black-screen system hang that occurred after applying security updates and rebooting. After ruling out any “low-hanging fruit” such as deadlocks on executive resources, resource depletion, etc., I decided to survey how far along the boot had gotten. In the output below we can tell that it’s fairly early in the boot process and that session zero is currently being setup.
3: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8e282840 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 00122000 ObjectTable: 97801e18 HandleCount: 554.
PROCESS 94153ad8 SessionId: none Cid: 0390 Peb: 7ffdf000 ParentCid: 0004
DirBase: 03368020 ObjectTable: a13564f0 HandleCount: 19.
PROCESS 92a55d90 SessionId: 0 Cid: 03c8 Peb: 7ffd9000 ParentCid: 0390
DirBase: 03368040 ObjectTable: a95606e0 HandleCount: 10.
PROCESS 92a56c48 SessionId: 0 Cid: 03d4 Peb: 7ffd9000 ParentCid: 03c8
DirBase: 03368060 ObjectTable: a959ea28 HandleCount: 30.
So let’s dump out the threads for these session zero processes and see what they’re doing:
1. Notice how the Session Manager thread has been waiting for more than fifteen minutes for the Windows subsystem to load and initialize.
3: kd> !process /s 0 0 0x17
Searching processes with session id 0
VadRoot 941e0578 Vads 8 Clone 0 Private 21. Modified 535. Locked 0.
Working Set Sizes (now,min,max) (125, 50, 345) (500KB, 200KB, 1380KB)
VirtualSize 2 Mb
PeakVirtualSize 4 Mb
THREAD 92a5d030 Cid 03c8.03cc Teb: 7ffdf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
Owning Process 92a55d90 Image: smss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1896 Ticks: 59778 (0:00:15:32.542)
Context Switch Count 94 IdealProcessor: 0
Win32 Start Address smss!NtProcessStartupW (0x4857d9a2)
Stack Init 9dc89000 Current 9dc888c0 Base 9dc89000 Limit 9dc86000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
ChildEBP RetAddr Args to Child
9dc888d8 81eb923a 92a5d030 97cb5120 92a5d0b8 nt!KiSwapContext+0x26
9dc8891c 81eb4bca 92a5d030 00000000 00000002 nt!KiSwapThread+0x44f
9dc88970 82040e83 00000002 9dc88aa8 00000001 nt!KeWaitForMultipleObjects+0x53d
9dc88bfc 82040bf2 00000002 00000001 00000000 nt!ObpWaitForMultipleObjects+0x256
9dc88d48 81e57c96 00000002 0008fb38 00000001 nt!NtWaitForMultipleObjects+0xcc
9dc88d48 778d5d14 00000002 0008fb38 00000001 nt!KiSystemServicePostCall
0008fac4 778d54a0 4857cc7e 00000002 0008fb38 ntdll!KiFastSystemCallRet
0008fac8 4857cc7e 00000002 0008fb38 00000001 ntdll!NtWaitForMultipleObjects+0xc
0008fb40 48579296 0008fb78 0008fb68 0008fbb0 smss!SmscpLoadSubSystem+0x9b
0008fb80 4857ca8a 0008fbb0 00000000 00000000 smss!SmpExecuteCommand+0x8d
0008fbc4 4857d0bc 00000000 00000000 00000000 smss!SmscpLoadSubSystemsForMuSession+0x182
0008fbe8 4857b678 00000003 002417d8 00000000 smss!SmscMain+0xc2
0008fc7c 4857d988 00000003 002417d8 002417e8 smss!wmain+0x50
0008fcc0 77886885 00241898 779bde2d 00000000 smss!NtProcessStartupW_AfterSecurityCookieInitialized+0x221
0008fd00 778b15d6 4857d9a2 7ffd9000 ffffffff ntdll!__RtlUserThreadStart+0x35
0008fd18 00000000 4857d9a2 7ffd9000 00000000 ntdll!_RtlUserThreadStart+0x1b
2. Also, we can see that there’s a single active thread within the csrss.exe process, which is a red flag because we know that csrss.exe hosts the Desktop Thread and Raw Input Thread, among others.
VadRoot 9391c128 Vads 33 Clone 0 Private 193. Modified 60. Locked 0.
Working Set Sizes (now,min,max) (582, 50, 345) (2328KB, 200KB, 1380KB)
VirtualSize 23 Mb
PeakVirtualSize 48 Mb
THREAD 942c5590 Cid 03d4.03e4 Teb: 00000000 Win32Thread: 00000000 WAIT: (Executive) KernelMode Non-Alertable
Owning Process 92a56c48 Image: csrss.exe
Wait Start TickCount 2516 Ticks: 59158 (0:00:15:22.870)
Context Switch Count 1 IdealProcessor: 0
Win32 Start Address ati2mtag!IRQMGR_WorkerThreadRoutine (0xa1ccf340)
Stack Init 9dcfd000 Current 9dcfcc30 Base 9dcfd000 Limit 9dcfa000 Call 0
Priority 13 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
9dcfcc48 81eb923a 942c5590 942c5618 00000000 nt!KiSwapContext+0x26
9dcfcc8c 81e54f38 942c5590 00000000 942c5590 nt!KiSwapThread+0x44f
9dcfcce4 a1d78724 915e4078 00000000 00000000 nt!KeWaitForSingleObject+0x492
9dcfcd00 a1c13340 908be398 915e4070 00000000 VIDEOPRT!VideoPortWaitForSingleObject+0x53
9dcfcd14 a1cce17f 908be398 915e4070 00000000 ati2mtag!IRQMgrMP_WaitForSingleObject+0x20
9dcfcd6c a1ccf355 93f45000 93f45000 9dcfcdc0 ati2mtag!PassiveRing_WorkerThreadRoutine+0x6f
9dcfcd7c 81fe301c 93f45000 ad8fc28d 00000000 ati2mtag!IRQMGR_WorkerThreadRoutine+0x15
9dcfcdc0 81e4beee a1ccf340 93f45000 00000000 nt!PspSystemThreadStartup+0x9d
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
3. Having seen this, we now know why the system is perpetually hung: Csrss.exe is not running properly. Because there is a video driver worker thread running, but the Desktop Thread and Raw Input Thread are not running, it appears that csrss has attempted to terminate. The termination has not completed because of the video-driver worker thread performing a non-alertable wait.
4. Next, we need to check for any state in the dump that might tell us why csrss.exe attempted to terminate:
3: kd> dt nt!eprocess 92a56c48 LastThreadExitStatus
+0x184 LastThreadExitStatus : 0n-1073741619
3: kd> !error 0n-1073741619
Error code: (NTSTATUS) 0xc00000cd (3221225677) - The name limit for the local computer network adapter card was exceeded.
After a quick search for STATUS_TOO_MANY_NAMES (0xc00000cd) through the source code, I was able to theorize that csrss.exe may have attempted the termination due to invalid command-line parameters.
3: kd> vertarget
Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (16 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 6002.18881.x86fre.vistasp2_gdr.130707-1535
Kernel base = 0x81e0d000 PsLoadedModuleList = 0x81f24c70
Debug session time: Fri Oct 25 05:10:34.030 2013 (UTC - 5:00)
System Uptime: 0 days 0:16:02.134
3: kd> .process /p /r 92a56c48
Implicit process is now 92a56c48
Loading User Symbols
3: kd> !peb
PEB at 7ffd9000
CommandLine: 'C:\Windows\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,1024 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16'
Sure enough, there was additional command-line parameter that was not recognized on Vista/Windows Server 2008 SP2 (supported only on Windows 7 and later). Once the invalid command-line parameter was removed, the server was able to boot normally again.
So how did the invalid value get there? It turns out that a logon script was setting the following registry value using an export from a Windows 7/Windows 2008 R2 machine where ServerDll=sxssrv,4 is a valid value.
Well, that concludes today’s segment, but in the timeless words of Uncle Ben remember “with great power comes great responsibility.” As we just saw, this applies not only to those possessing a spider-sense, but also to Windows administrators. J
Until next time, happy debugging!
Hyper-V is based on the 440BX (PCI) chipset for emulation. The decision to use this chipset started years ago with Connectix Virtual PC. The advantage of using an emulated chipset based on a popular motherboard like the 440BX, along with associated peripherals, is the compatibility with a large number of operating systems.
Windows Server 2012 R2 introduced the Generation 2 Virtual Machine. It is a UEFI based design, removing emulated devices and replacing them with synthetic devices. Generation 2 VMs no longer support the following devices:
After reading this list you might ask the question – how do I debug a Generation 2 VM?
The COM port is not actually removed from a Generation 2 VM. The port is turned off by default and not present in the user interface. To enable it for debugging use the following steps.
1. Shutdown the VM. You can verify the VM is off using the below PowerShell command.
2. Turn off secure boot using the following PowerShell Command.
3. Set a COM port path using the following PowerShell command where the path is equal the named pipe.
4. To confirm the COM port settings after making the change, use the following command.
5. Restart the Virtual Machine using the following command.
Start-VM –Name VM2
6. Inside the guest VM, you can confirm that UEFI has been disabled with the following command. The results are False if UEFI was successfully disabled in step 2 above.
7. Enable Kernel Debugging using BCDEdit.
BCDEdit /debug ON
8. Configure the debugger to connect to the pipe:
9. Connect the debugger and break in with Ctrl+Break: