Hi, this is Anurag again. Here is a case study of an NDIS driver causing a problem due to double completion of a send packet.
A protocol driver allocates a NDIS packet and gives it to the miniport driver to be sent on the wire. A miniport driver is supposed to send or complete the packet, but miniport driver is supposed to complete the packet only once. The moment it tries to complete a packet twice (generally due to a bug in the miniport driver), the machine will bugcheck with a stack like the one shown below. The crash will happen either in a protocol driver or in the NDIS driver. The bugcheck would be the result of a trap on accessing an invalid memory in the packet.
Each time a packet gets completed the miniport stamps it as completed by putting a string “COM” just before the packet (somewhere in the packet stack). If a miniport driver tries to complete this packet again, the system would crash. The miniport driver which tried to complete the completed packet again is at fault and should be fixed.
When reviewing the bugcheck, !analyze –v will give you the stack.
1: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 00000008, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: f78521e0, address which referenced memory
Debugging Details:
------------------
READ_ADDRESS: 00000008
CURRENT_IRQL: 2
FAULTING_IP:
NDIS!ndisMSendCompleteX+71
f78521e0 8b7808 mov edi,dword ptr [eax+8]
DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0xD1
PROCESS_NAME: System
TRAP_FRAME: f78b2e18 -- (.trap 0xfffffffff78b2e18)
ErrCode = 00000000
eax=00000000 ebx=902bbab0 ecx=00000002 edx=fffffffe esi=8f3aabf8 edi=901bd440
eip=f78521e0 esp=f78b2e8c ebp=f78b2ea0 iopl=0 nv up ei ng nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282
NDIS!ndisMSendCompleteX+0x71:
f78521e0 8b7808 mov edi,dword ptr [eax+8] ds:0023:00000008=????????
Resetting default scope
LAST_CONTROL_TRANSFER: from f78521e0 to 8088c963
STACK_TEXT:
f78b2e18 f78521e0 badb0d00 fffffffe 00000000 nt!KiTrap0E+0x2a7
f78b2ea0 b9f47b5a 902bbab0 8f3aabf8 00000000 NDIS!ndisMSendCompleteX+0x71
f78b2ec0 f78521f9 8f411000 013aabf8 00000000 X_Miniport_Network_Teamming_Driver +0x7b5a
f78b2ee4 ba04a0c8 9029b5e8 8f3aabf8 00000000 NDIS!ndisMSendCompleteX+0x8a
f78b2f00 ba052744 904d8eb8 8f3aabf8 00000000 Y_Miniport_NIC_Driver +0x20c8
f78b2f3c ba050339 8f8c5000 00000000 f78b2f68 Y_Miniport_NIC_Driver +0xa744
f78b2f70 ba05065f 008c5000 8f8c5844 f78b2f8c Y_Miniport_NIC_Driver +0x8339
f78b2f80 ba0493f1 8f8c5000 f78b2f9c f783a0e8 Y_Miniport_NIC_Driver +0x865f
f78b2f8c f783a0e8 904d8eb8 8f8c58bc f78b2ff4 Y_Miniport_NIC_Driver +0x13f1
f78b2f9c 808320f0 8f8c58bc 8f8c5844 00000000 NDIS!ndisMDpcEx+0x1f
f78b2ff4 8088db27 b88fe454 00000000 00000000 nt!KiRetireDpcList+0xca
f78b2ff8 b88fe454 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x37
8088db27 00000000 0000000a 0083850f bb830000 0xb88fe454
1: kd> .trap 0xfffffffff78b2e18
Looking at the stack, it appears to be a NULL memory address being accessed that is causing problem. This could lead us to believe that this is a memory corruption problem. This is where we go off-track on the investigation.
The current NDIS packet status should be checked if you bugcheck on the NDIS stack and in NDIS!ndisMSendCompleteX.
Some basic debugging will display the NDIS_PACKET on the stack and in the ESI register (8f3aabf8).
Looking at the memory area before the packet shows the string “COM” which is the indication that it was marked as completed.
Here is how it looks.
For reference this is how a non-completed packet looks like
So for this issue the X_Miniport_Network_Teamming_Driver was at fault which tried to complete a completed packet. Fortunately the vendor for the driver was aware of the problem and was able to give the customer a fix.
For a 64 bit dump we do a “dc <Packet Address>-0x68” to look at the memory area before the packet. Below is an example.
BugCheck D1, {0, 2, 0, fffffade57895375}DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)An attempt was made to access a pageable (or completely invalid) address at aninterrupt request level (IRQL) that is too high. This is usuallycaused by drivers using improper addresses.If kernel debugger is available get stack backtrace.Arguments:Arg1: 0000000000000000, memory referencedArg2: 0000000000000002, IRQLArg3: 0000000000000000, value 0 = read operation, 1 = write operationArg4: fffffade57895375, address which referenced memory
0: kd> r
Last set context:
rax=0000000000000000 rbx=fffffade6f203000 rcx=0000000000000000
rdx=fffffade6f564740 rsi=fffffade6faa3548 rdi=0000000000000000
rip=fffffade57895375 rsp=fffff8000012b990 rbp=fffffade6f5647d0
r8=0000000000000000 r9=5bcdd13016000000 r10=5bcdd13016110009
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0010 ss=0018 ds=0002 es=0000 fs=0000 gs=0000 efl=00010202
tcpip!TCPSendComplete+0x124:
fffffade`57895375 488b1b mov rbx,qword ptr [rbx] ds:0002:fffffade`6f203000=fffffade6f52e010
0: kd> kvn
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr : Args to Child : Call Site
00 fffff800`0012b990 fffffade`578975a7 : 00000000`00000000 00000000`00000000 fffffade`6e8980b0 fffffade`6f564740 : tcpip!TCPSendComplete+0x124
01 fffff800`0012b9e0 fffffade`5789a89e : fffffade`6f1f1700 fffffade`00000000 00000000`00026200 fffffade`6e8980b0 : tcpip!IPSendComplete+0x477
02 fffff800`0012bab0 fffffade`5ae37237 : fffff800`0083f480 fffffade`6f887000 fffffade`6faab228 fffffade`6e8980b0 : tcpip!ARPSendComplete+0x14a
03 fffff800`0012baf0 fffffade`58443c15 : fffffade`6f887000 fffffade`6e8980b0 fffffade`6f887038 fffffade`6c31e3b0 : NDIS!ndisMSendCompleteX+0xda
04 fffff800`0012bb40 fffffade`5ae37237 : fffffade`6faa3010 00000000`00000002 fffffade`6fc16228 fffffade`6c31e3b0 : NDIS_IM_TEAM_DRIVER!po_SendDone+0x75
05 fffff800`0012bb80 fffffade`596f9a4c : fffffade`6faa3010 fffff800`0012bc50 00000000`00000000 00000000`00000000 : NDIS!ndisMSendCompleteX+0xda
06 fffff800`0012bbd0 fffffade`5af5e9cd : fffff800`0012bcb0 fffffade`6f52e010 fffffade`6f203000 00000000`00000006 : NDIS_MP_DRIVER!l2nd_indicate_tx+0xfc
07 fffff800`0012bc30 fffffade`5af64889 : fffffade`6f52e010 fffffade`6f5315c8 00000000`00000000 00000000`00000000 : NDIS_MP_DRIVER!um_bdrv_indicate_tx_done+0x121
08 fffff800`0012bc90 fffffade`5af64bab : fffffade`6f52e010 00000000`00000001 00000000`00000001 fffff800`011b2080 : NDIS_MP_DRIVER!service_tx_intr+0xa9
09 fffff800`0012bce0 fffffade`5af64d63 : fffffade`6f52e010 00000000`0005ffd4 00000000`00000000 0000023e`f26bf2bf : NDIS_MP_DRIVER!service_intr_mp+0xb3
0a fffff800`0012bd10 fffff800`010285a1 : 00000000`00000000 fffffade`5af64cb4 fffff800`0012bd68 00000000`00000018 : NDIS_MP_DRIVER!um_bdrv_dpc+0xaf
0b fffff800`0012bd40 fffff800`01067c10 : fffff800`011b0180 fffff800`011b0180 00000000`0005ffd4 fffff800`011b4500 : nt!KiRetireDpcList+0x150
0c fffff800`0012bdd0 fffff800`014141d1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x50
0d fffff800`0012be00 00000000`fffff800 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemStartup+0x1bf
Note that in this bugcheck we crash in the transport driver in the above stack. But the problem is that the packet (fffffade`6e8980b0) came from the NDIS driver NDIS_IM_TEAM_DRIVER.
0: kd> dc fffffade`6e8980b0 -0x68
fffffade`6e898048 00000000 00000000 4d4f4336 00000000 ........6COM....
fffffade`6e898058 00000000 00000000 00000000 00000000 ................
fffffade`6e898068 00000000 00000000 00000000 00000000 ................
fffffade`6e898078 00000000 00000000 00000000 00000000 ................
fffffade`6e898088 00000000 00000000 00000000 00000000 ................
fffffade`6e898098 00000000 00000000 00000000 00000000 ................
fffffade`6e8980a8 ffffffff ffffffff 6e898430 fffffade ........0..n....
fffffade`6e8980b8 6f564740 fffffade 6d9ee460 fffffade @GVo....`..m....
After reading this, I suspect you would have at least these two questions for me:
Q1: How did I find out that “COM” is use as an identifier for a completed packet?
A1: Well, mainly code review. This is just intended as a debug trick to verify that the packet is already completed.
Q2: Why did I use 0x38 (for 32 bit) or 0x68 (for 64 bit) as an offset to check the memory contents before the packet address?
A2: Again code review shows the field to be updated with string ‘COM” can seen going 0x38 bytes back from the packet start address.
NDIS double packet completion is a common issue that would crash the system. The above trick can certainly be used to isolate the culprit NDIS driver.
Today we’re posting links to some of our favorite debugging-related content on the web. Post your own favorites as a comment to share them with everyone!
Reverse Engineering and Debugging Blogs
DumpAnalysis
MetaSploit
Nynaeve
Mark Russinovich's Blog
Steve’s Techspot
John Robbins’ Blog
Uninformed.org
Windbg by Volker
CodeProject Debugging Tips
DebugInfo
Jigar Mehta's Blog
ReverseMode
Debug Reference
Debugging Commands (MSDN)
Debugger Reference (MSDN)
Common WinDbg Commands
Advanced Debugging Techniques (MSDN)
Other stuff
Debunking Common Windows Performance Tweaking Myths
Some disclaimers: Links are no particular order. Many of these links go to non-Microsoft sites, and our linking to them does not mean we endorse all the content on the sites.
Hi, my name Anurag Sarin, I am an escalation engineer in the Platforms Global Escalation Team. I would like to give some insight on NDIS.
NDIS Introduction
The Network Driver Interface Specification (NDIS) library abstracts the network hardware from network drivers. NDIS also specifies a standard interface between layered network drivers, thereby abstracting lower-level drivers that manage hardware from upper-level drivers, such as network transports. NDIS also maintains state information and parameters for network drivers, including pointers to functions, handles, and parameter blocks for linkage, and other system values.
Types of network drivers
ndiskd is a good extension for debugging NDIS drivers .The document Debugging NDIS Drivers has more information about ndiskd.
To get a list of NDIS protocols drivers in the system, the protocols option in ndiskd can be used.
This also gives a list of all NDIS open blocks on each protocol driver. Protocol open block is described in the later section.
A Protocol Driver is represented by a NDIS Protocol Block, which is shown as Protocol below:
1: kd> !ndiskd.protocols
Protocol 8ef57590: NDISUIO
Open 8ef61860 - Miniport: 902bbab0 X Network Team #1
Protocol 8f3aea50: TCPIP_WANARP
Open 8f3ae508 - Miniport: 90287ae8 WAN Miniport (IP)
Protocol 8f220008: TCPIP
Open 8f21d210 - Miniport: 902bbab0 X Network Team #1
Protocol 9029bbf8: NDPROXY
Open 901618e0 - Miniport: 9019f130 Direct Parallel
Open 90161c80 - Miniport: 9019f130 Direct Parallel
Open 9018bcd0 - Miniport: 901ba130 WAN Miniport (L2TP)
Open 90171490 - Miniport: 901ba130 WAN Miniport (L2TP)
Protocol 901e3008: RASPPPOE
Protocol 901bb008: NDISWAN
Open 90216b30 - Miniport: 9019f130 Direct Parallel
Open 9047e518 - Miniport: 901fdab0 WAN Miniport (PPTP)
Open 90198c20 - Miniport: 90276ab0 WAN Miniport (PPPOE)
Open 901989e0 - Miniport: 901ba130 WAN Miniport (L2TP)
Protocol 902de6e0: Y_TEAM
Open 9028ef10 - Miniport: 9029b5e8 P Gigabit Server Adapter #2
Open 90198b10 - Miniport: 9029eab0 Q Multifunction Gigabit Server Adapter #2
NDIS filter drivers are represented by Filter Driver Block (s) shown below.
0: kd> !ndiskd.filters
NDIS Driver verifier level: 0
NDIS Failed allocations : 0
Filter Driver Block: 97412e58
Filter: 97414c10 Z Network Connection-Native WiFi Filter Driver-0000
Miniport 85d160e8 Z Network Connection
Filter Driver Block: 8797bdb0
Filter: 97446c10 Z Network Connection - H Miniport-QoS Packet Scheduler-0000
Miniport 8610b0e8 Z Network Connection - H Miniport
Filter: 87b1b730 Y Network Adapter - H Miniport-QoS Packet Scheduler-0000
Miniport 8611c0e8 Y Network Adapter - H Miniport
Filter: 879c3008 G Network Connection - H Miniport-QoS Packet Scheduler-0000
Miniport 861150e8 G Network Connection - H Miniport
Filter: 879c0a50 WAN Miniport (IP) - H Miniport-QoS Packet Scheduler-0000
Miniport 861240e8 WAN Miniport (IP) - H Miniport
Filter: 879bb3f8 WAN Miniport (IPv6) - H Miniport-QoS Packet Scheduler-0000
Miniport 861250e8 WAN Miniport (IPv6) - H Miniport
Filter: 87981870 WAN Miniport (Network Monitor) - H Miniport-QoS Packet Scheduler-0000
Miniport 861260e8 WAN Miniport (Network Monitor) - H Miniport
Filter: 8797f518 Nortel IPSECSHM Adapter - H Miniport-QoS Packet Scheduler-0000
Miniport 861170e8 Nortel IPSECSHM Adapter - H Miniport
The “miniports” option lists all NDIS miniport drivers represented by a Miniport Driver Block (s).
kd> !ndiskd.miniports
Miniport Driver Block: 885915b8, Version 0.0
Miniport: 8863a0e8, NetLuidIndex: 1, IfIndex: 7, RAS Async Adapter
Miniport Driver Block: 88018010, Version 0.0
Miniport: 8828d488, NetLuidIndex: 1, IfIndex: 3, WAN Miniport (PPTP)
Miniport Driver Block: 87e535a8, Version 0.0
Miniport: 88150200, NetLuidIndex: 0, IfIndex: 4, WAN Miniport (PPPOE)
Miniport Driver Block: 87f63510, Version 0.0
Miniport: 880ac4b8, NetLuidIndex: 0, IfIndex: 5, WAN Miniport (IPv6)
Miniport: 880844c0, NetLuidIndex: 3, IfIndex: 6, WAN Miniport (IP)
Miniport Driver Block: 87f2ccb8, Version 0.0
Miniport: 88091488, NetLuidIndex: 0, IfIndex: 2, WAN Miniport (L2TP)
Miniport Driver Block: 8809de60, Version 10.1
Miniport: 883bf0e8, NetLuidIndex: 10, IfIndex: 15, MY PCI Fast Ethernet Adapter (Emulated) #4
Miniport: 883be0e8, NetLuidIndex: 5, IfIndex: 11, MY PCI Fast Ethernet Adapter (Emulated) #3
Miniport: 883bd0e8, NetLuidIndex: 6, IfIndex: 9, MY PCI Fast Ethernet Adapter (Emulated) #2
Miniport: 883bc0e8, NetLuidIndex: 4, IfIndex: 8, MY PCI Fast Ethernet Adapter (Emulated)
Miniport Driver Block: 87f77df0, Version 1.0
Miniport: 87f9c488, NetLuidIndex: 6, IfIndex: 16, isatap.{584AF5A9-63C2-44C5-970D-DB85057F2931}
Miniport: 87f75488, NetLuidIndex: 4, IfIndex: 14, isatap.fareast.corp.microsoft.com
Miniport: 87fc6488, NetLuidIndex: 3, IfIndex: 10, isatap.{CCDB4297-B958-4C2A-95A5-29150BD0A371}
The interfaces option lists all Network interfaces
kd> !ndiskd.interfaces
Interface block 87fb12a8
<Snip>
IfIndex: 20, IfType: 6
Inerface Guid: 5d0bd81a-47c7-11dc-a9d2-0003ff2b6bfa
Interface block 88101ab0
IfIndex: 21, IfType: 6
Inerface Guid: ed1c50d5-ff1f-11db-9b85-0003ff7133d2
Interface block 87f82ab0
IfIndex: 15, IfType: 6
Inerface Guid: f442c036-8bf5-43a6-91fd-6792ef752100
Interface block 881cd5d8
IfIndex: 22, IfType: 6
Inerface Guid: ed1c50d4-ff1f-11db-9b85-9459bf825974
Interface block 87f313f8
Interface guids correspond to each network interface. Some interfaces have their information in the in the registry .For Example on my machine Interface Guid: f442c036-8bf5-43a6-91fd-6792ef752100 corresponds to registry:-
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{F442C036-8BF5-43A6-91FD-6792EF752100}
NDIS Driver Stack
Basic Stack Configuration
The MSDN web page NDIS Driver Stack has basic documentation on the NDIS stack
For Miniport details, the miniport option can be used.
kd> !ndiskd.miniport 883bc0e8
Miniport 883bc0e8 : MY PCI Fast Ethernet Adapter (Emulated), v5.0
AdapterContext : 8809e000
Flags : 2c412008
BUS_MASTER, IGNORE_TOKEN_RING_ERRORS, RESOURCES_AVAILABLE
SUPPORTS_MEDIA_SENSE, DOES_NOT_DO_LOOPBACK, MEDIA_CONNECTED
PnPFlags : 80210000
RECEIVED_START, HARDWARE_DEVICE,
MiniportState : STATE_RUNNING
IfIndex : 8
Ndis5MiniportInNdis6Mode : 1
InternalResetCount : 0000
MiniportResetCount : 0000
References : 5
UserModeOpenReferences: 0
PnPDeviceState : PNP_DEVICE_STARTED
CurrentDevicePowerState : PowerDeviceD0
Bus PM capabilities
DeviceD1: 0
DeviceD2: 0
WakeFromD0: 0
WakeFromD1: 0
WakeFromD2: 0
WakeFromD3: 0
SystemState DeviceState
PowerSystemUnspecified PowerDeviceUnspecified
S0 D0
S1 PowerDeviceUnspecified
S2 PowerDeviceUnspecified
S3 PowerDeviceUnspecified
S4 D3
S5 D3
SystemWake: PowerSystemUnspecified
DeviceWake: PowerDeviceUnspecified
Current PnP and PM Settings: : 00000030
DISABLE_WAKE_UP, DISABLE_WAKE_ON_RECONNECT,
Translated Allocated Resources:
IO Port: 0000e480, Length: 80
Memory: febfc000, Length: 1000
Interrupt Level: 10, Vector: 3b
MediaType : 802.3
DeviceObject : 883bc030, PhysDO : 87ca1030 Next DO: 87ca1030
MapRegisters : 00000000
FirstPendingPkt: 00000000
DriverVerifyFlags : 00000000
Miniport Interrupt : 8809e008
Miniport version 5.0
Miniport Filter List:
Filter 88301c28: FilterDriver 88129300, FilterModuleContext 880d7230 MY PCI Fast Ethernet Adapter (Emulated)-QoS Packet Scheduler-0000
Miniport Open Block Queue:
88263278: Protocol 88300830 = RSPNDR, ProtocolBindingContext 87fb7820, v6.0
882446d0: Protocol 8835a300 = LLTDIO, ProtocolBindingContext 88323310, v6.0
880d2c58: Protocol 8831e2e0 = TCPIP, ProtocolBindingContext 88329008, v6.0
Current PnP capacities and Power Management Settings are shown by flags and can be one of these values in the Current PnP and PM Settings’ section.
NOT_STOPPABLE : The device is not stoppable i.e. ISA
NOT_REMOVEABLE : The device cannot be safely removed
NOT_SUSPENDABLE : The device cannot be safely suspended
DISABLE_PM : Disable all Power Management features
DISABLE_WAKE_UP : Disable device waking up the system .This is evident when the user disables Wake-On-LAN (WOL) feature on the miniport adaptor
DISABLE_WAKE_ON_RECONNECT: Disable device waking up the -system- due to a cable re-connect
Above , the miniport block 883bc0e8 represents MY PCI Fast Ethernet Adapter, i.e. the NIC driver on my machine. The NIC driver has a binding with filter driver MY PCI Fast Ethernet Adapter (Emulated)-QoS Packet Scheduler and protocols drivers RSPNDR, LLTDIO and TCPIP.
The stack with the debug output above would look somewhat like this.
To see what all miniport drivers the Protocol driver has bound to - the protocol option can be used.
kd> !protocol 8831e2e0
Protocol 8831e2e0 : TCPIP
RootDeviceName is \DEVICE\{B4982B71-0255-4D04-A585-4C339162A25D}
v6.0 RefCount 5
Open 880d2c58 - Miniport: 883bc0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated)
Open 881bbc58 - Miniport: 883bd0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated) #2
Open 881bcc58 - Miniport: 883be0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated) #3
Open 881c0c58 - Miniport: 883bf0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated) #4
BindAdapterHandlerEx 8e93da28, UnbindAdapterHandlerEx 8e9d7a87
PnPEventHandler 8e933b83, UnloadHandler 00000000
OpenAdapterCompleteEx 8e9d74ff, CloseAdapterCompleteEx 8e9d7722
SendNetBufferListsCompleteHandler 8e997067, ReceiveNetBufferListsHandler 8e98ff5f
StatusComplete 00000000, StatusHandler 8e942cad
AssociatedMiniDriver 00000000
Flags : 00000000
This also shows the various handler routines of the Protocol TCPIP Driver routines.
Un-assembling the routines would verify them further.
kd> u 8e93da28
tcpip!FlBindAdapter:
8e93da28 8bff mov edi,edi
8e93da2a 55 push ebp
8e93da2b 8bec mov ebp,esp
8e93da2d 83ec1c sub esp,1Ch
8e93da30 53 push ebx
8e93da31 56 push esi
8e93da32 57 push edi
8e93da33 6a06 push 6
kd> u 8e933b83
tcpip!Fl48PnpEvent:
8e933b83 8bff mov edi,edi
8e933b85 55 push ebp
8e933b86 8bec mov ebp,esp
8e933b88 837d0800 cmp dword ptr [ebp+8],0
8e933b8c 7406 je tcpip!Fl48PnpEvent+0x11 (8e933b94)
8e933b8e 5d pop ebp
8e933b8f e96c030000 jmp tcpip!FlPnpEvent (8e933f00)
8e933b94 8b450c mov eax,dword ptr [ebp+0Ch]
NDIS Open Block is a block that represents the binding between a Miniport Driver and a Protocol Driver. So there is one NDIS Open Block per binding between a protocol and a miniport.
kd> !ndiskd.opens
Open 885283c0
Miniport: 8863a0e8 - RAS Async Adapter
Protocol: 8803be48 -
Open 88263278
Miniport: 883bc0e8 - MY PCI Fast Ethernet Adapter (Emulated)
Protocol: 88300830 - RSPNDR
Open 88263628
Miniport: 883bd0e8 - MY PCI Fast Ethernet Adapter (Emulated) #2
Open 88124008
Miniport: 883be0e8 - MY PCI Fast Ethernet Adapter (Emulated) #3
Open 883276a0
Miniport: 883bf0e8 - MY PCI Fast Ethernet Adapter (Emulated) #4
Open 882446d0
Protocol: 8835a300 - LLTDIO
Open 882fa398
Open 882d5550
Open 882cf470
Open 88219968
Miniport: 880ac4b8 - WAN Miniport (IPv6)
Protocol: 880f04a0 - WANARPV6
Open 8816e008
Miniport: 880844c0 - WAN Miniport (IP)
Protocol: 882e5008 - WANARP
Open 881eec58
Protocol: 88383138 - TCPIP6
Open 881b6c58
Open 880d2c58
Protocol: 8831e2e0 - TCPIP
Open 881bbc58
Open 881c2c58
Open 881bcc58
Open 881c0c58
Open 8811a6a0
Miniport: 87fbb0e8 - isatap.{B4982B71-0255-4D04-A585-4C339162A25D}
Protocol: 882d8298 - TCPIP6TUNNEL
Open 8811a008
Miniport: 87fc6488 - isatap.{CCDB4297-B958-4C2A-95A5-29150BD0A371}
Open 881f8870
Miniport: 87f75488 - isatap.fareast.corp.microsoft.com
Open 882a4850
Miniport: 87f9c488 - isatap.{584AF5A9-63C2-44C5-970D-DB85057F2931}
Open 881ad960
Miniport: 8828d488 - WAN Miniport (PPTP)
Open 88018c58
Miniport: 88150200 - WAN Miniport (PPPOE)
Open 8831f818
Miniport: 88091488 - WAN Miniport (L2TP)
Protocol: 87f1e100 -
Open 8831fc10
Open 881425a8
Use mopen option to see the details of the NDIS Open Block.
kd> !mopen 881bbc58
Miniport Open Block 881bbc58
Protocol 8831e2e0 = TCPIP, ProtocolContext 88348008, v6.0
Miniport 883bd0e8 = MY PCI Fast Ethernet Adapter (Emulated) #2, v5.0
MiniportAdapterContext: 880a1000
Flags : 01000000
OPEN_USE_MULTICAST_LIST,
References : 1
The ‘References’ section above shows the number of outstanding Input Output Requests. This can be use full to investigate how many requests by a protocol driver are passed to the next lower driver which are currently outstanding.
Network Data
Network data consists of packets of data that are sent or received over the network. NDIS provides data structures to describe and organize such data. The primary NDIS 6.0 network data structures include the following:
· NET_BUFFER structures
· NET_BUFFER LIST structures
· NET_BUFFER_LIST_CONTEXT structures
For NDIS 5.x we have NDIS PACKETS in place of NET_BUFFER structure.
NDIS_PACKET Structure
NDIS packets (represented by a NDIS_PACKET structure) are allocated by a protocol driver, filled with data, and passed to the next lower NDIS driver so that the data can be sent on the network. Some lowest level NIC drivers allocate packets to hold received data and pass the packet up to interested higher-layer drivers. Sometimes, a protocol driver allocates a packet and passes it to a NIC driver with a request that the NIC driver copy received data into the provided packet. NDIS provides functions for allocating and manipulating the substructures that make up a packet. The following figure illustrates a structure of a packet.
Each NDIS Packet is basically a Packet Descriptor. Each Packet Descriptor has a series of Buffer Descriptors.
A packet is composed of the following:
The pkt option in ndiskd helps us to see the contents of the NDIS Packet. It has various verbose options:
Usage: pkt <pointer to packet> <verbosity>
<verbosity> can be between 1 to 5.
1-Packet Private
2-Packet Extension
3-Ndis Reference
4-Buffer List
5- Data in Packet List
1: kd> !ndiskd.pkt 0x8f3aabf8
NDIS_PACKET at 8f3aabf8
Packet.Private
PhysicalCount 00000001 Total Length 00000036
Head 8f3aa630 Tail 8a667d30
Pool 90331d20 Count 00000001
Flags 00000002 ValidCounts 01
NdisPacketFlags 00000000 NdisPacketOobOffset 006c
Private.Flags : 00000002
Private.NdisPacketFlags: 0
Above output indicates a typical Packet Descriptor.
Below is a description of the fields in above output.
PhysicalCount : Number of physical pages in packet.
TotalLength : Total amount of data in the packet in bytes.
Head : First buffer in the chain. If Head is NULL the chain is empty.
Tail : Last buffer in the chain.
Count : The number of Buffers in the chain.
ValidCounts : Represent a Boolean value on validity of the Counts.
Pool : NDIS Packet Pool address so we know where to free it back to.
To demonstrate what an NDIS packet looks like, a breakpoint was placed on routine NdisMSendComplete. The definition of NdisMSendComplete states that ‘PNDIS_PACKET Packet’ is the second parameter. So the address of NDIS Packet can be found at EBP+0xC position on the stack.
kd> kvn
# ChildEBP RetAddr Args to Child
00 818f1c44 8b8848c3 883be0e8 88727538 00000000 ndis!NdisMSendComplete+0x10 (FPO: [Non-Fpo])
01 818f1c7c 8b884b2f 8817d280 81827c97 00000000 dc21x4vm!ProcessTransmitDescRing+0x363 (FPO: [Non-Fpo])
02 818f1c9c 81718cb1 00010005 88166008 883be0e8 dc21x4vm!DC21X4HandleInterrupt+0xfb (FPO: [Non-Fpo])
03 818f1cc8 81682d1f 8816601c 88166008 00000000 ndis!ndisMDpc+0x16b (FPO: [Non-Fpo])
04 818f1ce8 818a93ae 8816601c 88166008 00000000 ndis!ndis5InterruptDpc+0x9c (FPO: [Non-Fpo])
05 818f1d50 818912ae 00000000 0000000e 00000000 nt!KiRetireDpcList+0x147
06 818f1d54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x46 (FPO: [0,0,0])
You can use verbosity 4 for looking at the Buffer Descriptors and a pointer to the next buffer descriptor, if any as shown below.
ndis.h available with Windows Driver Kit (WDK) or Windows Driver Device Kit (DDK) has definition for NdisPacketFlags
The verbosity 5 as most interesting - the NDIS packet data contents are displayed.
kd> !pkt poi(@ebp+c) 5
NDIS_PACKET at 88727538
MDL = 886f3ca0
StartVa ffffffff886f3000, ByteCount 0x36, ByteOffset 0xd06, NB MdlOffset 0x0
886f3d06: 00 13 5f 0b ef ca 00 15 5d 50 f3 34 08 00 45 00
886f3d16: 03 23 04 44 40 00 80 06 ce 2e 41 34 50 de 41 34
886f3d26: 52 1c c0 51 00 50 6d f9 e6 c2 9a 14 d4 99 50 18
886f3d36: 40 29 0e e5 00 00
MDL = 87f3aad0
StartVa ffffffff886f4000, ByteCount 0x2fb, ByteOffset 0x40, NB MdlOffset 0x0
886f4040: 47 45 54 20 68 74 74 70 3a 2f 2f 77 77 77 2e 6d
886f4050: 73 6e 2e 63 6f 6d 2f 61 6a 61 78 2f 48 2e 61 73
886f4060: 70 78 20 48 54 54 50 2f 31 2e 31 0d 0a 41 63 63
886f4070: 65 70 74 3a 20 2a 2f 2a 0d 0a 41 63 63 65 70 74
886f4080: 2d 4c 61 6e 67 75 61 67 65 3a 20 65 6e 2d 75 73
886f4090: 0d 0a 52 65 66 65 72 65 72 3a 20 68 74 74 70 3a
886f40a0: 2f 2f 77 77 77 2e 6d 73 6e 2e 63 6f 6d 2f 0d 0a
886f40b0: 55 41 2d 43 50 55 3a 20 78 38 36 0d 0a 41 63 63
886f40c0: 65 70 74 2d 45 6e 63 6f 64 69 6e 67 3a 20 67 7a
886f40d0: 69 70 2c 20 64 65 66 6c 61 74 65 0d 0a 55 73 65
886f40e0: 72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61
886f40f0: 2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 6c 65
886f4100: 3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 6e 64
886f4110: 6f 77 73 20 4e 54 20 36 2e 30 3b 20 53 4c 43 43
886f4120: 31 3b 20 2e 4e 45 54 20 43 4c 52 20 32 2e 30 2e
886f4130: 35 30 37 32 37 3b 20 2e 4e 45 54 20 43 4c 52 20
886f4140: 33 2e 30 2e 30 34 35 30 36 29 0d 0a 48 6f 73 74
886f4150: 3a 20 77 77 77 2e 6d 73 6e 2e 63 6f 6d 0d 0a 50
886f4160: 72 6f 78 79 2d 43 6f 6e 6e 65 63 74 69 6f 6e 3a
886f4170: 20 4b 65 65 70 2d 41 6c 69 76 65 0d 0a 43 6f 6f
886f4180: 6b 69 65 3a 20 4d 43 31 3d 56 3d 33 26 47 55 49
886f4190: 44 3d 31 61 31 61 65 64 38 31 38 36 34 30 34 32
886f41a0: 39 62 61 63 32 37 38 61 65 37 65 39 63 38 35 39
886f41b0: 64 39 3b 20 6d 68 3d 4d 53 46 54 3b 20 43 55 4c
886f41c0: 54 55 52 45 3d 45 4e 2d 55 53 3b 20 4d 55 49 44
886f41d0: 3d 32 43 38 33 35 42 36 32 34 35 42 46 34 46 30
886f41e0: 36 38 36 36 43 34 37 38 38 42 46 39 30 43 35 38
886f41f0: 32 3b 20 7a 69 70 3d 7a 3a 45 43 31 7c 6c 61 3a
886f4200: 35 31 2e 35 31 32 32 32 31 38 38 7c 6c 6f 3a 30
886f4210: 7c 63 3a 47 42 7c 68 72 3a 31 3b 20 46 6c 69 67
886f4220: 68 74 47 72 6f 75 70 49 64 3d 34 37 3b 20 46 6c
886f4230: 69 67 68 74 49 64 3d 42 61 73 65 50 61 67 65 3b
886f4240: 20 75 73 68 70 73 76 72 3d 4d 3a 35 7c 46 3a 35
886f4250: 7c 54 3a 35 7c 45 3a 35 7c 44 3a 62 6c 75 7c 57
886f4260: 3a 46 7c 50 3a 4e 7c 56 3a 30 3b 20 75 73 68 70
886f4270: 63 6c 69 3d 30 7c 48 2e 30 2e 31 7c 47 2e 30 2e
886f4280: 31 7c 5a 2e 30 2e 31 7c 52 2e 30 2e 31 2e 63 61
886f4290: 70 7c 43 2e 30 2e 31 2e 6c 67 3a 6e 65 77 79 6f
886f42a0: 72 6b 6e 79 7c 4c 2e 30 2e 31 2e 4c 4e 3a 57 4e
886f42b0: 42 43 3b 20 75 73 68 70 77 65 61 3d 77 63 3a 55
886f42c0: 53 4e 59 30 39 39 36 3b 20 75 73 68 70 70 72 3d
886f42d0: 43 3a 31 3a 30 38 30 37 32 31 7c 53 3a 31 3a 30
886f42e0: 38 30 38 30 36 3b 20 68 70 63 6c 69 3d 57 2e 48
886f42f0: 7c 4c 2e 7c 53 2e 7c 52 2e 7c 55 2e 4c 7c 43 2e
886f4300: 3b 20 68 70 73 76 72 3d 4d 3a 35 7c 46 3a 35 7c
886f4310: 54 3a 35 7c 45 3a 35 7c 44 3a 62 6c 75 7c 57 3a
886f4320: 46 3b 20 68 70 6f 6c 79 3d 4f 3a 31 7c 48 3a 31
886f4330: 3b 20 77 70 76 3d 30 0d 0a 0d 0a
Above output shows starting virtual address of each buffer, the buffer's byte offset into the page pointed to by the virtual address and the total number of bytes in the buffer
Looking at the contents of the memory buffer closely :-
kd> dc 886f4040 886f4330+0xc
886f4040 20544547 70747468 772f2f3a 6d2e7777 GET http://www.m
886f4050 632e6e73 612f6d6f 2f78616a 73612e48 sn.com/ajax/H.as
886f4060 48207870 2f505454 0d312e31 6363410a px HTTP/1.1..Acc
886f4070 3a747065 2a2f2a20 63410a0d 74706563 ept: */*..Accept
886f4080 6e614c2d 67617567 65203a65 73752d6e -Language: en-us
886f4090 65520a0d 65726566 68203a72 3a707474 ..Referer: http:
886f40a0 77772f2f 736d2e77 6f632e6e 0a0d2f6d //www.msn.com/..
886f40b0 432d4155 203a5550 0d363878 6363410a UA-CPU: x86..Acc
886f40c0 2d747065 6f636e45 676e6964 7a67203a ept-Encoding: gz
886f40d0 202c7069 6c666564 0d657461 6573550a ip, deflate..Use
886f40e0 67412d72 3a746e65 7a6f4d20 616c6c69 r-Agent: Mozilla
886f40f0 302e342f 6f632820 7461706d 656c6269 /4.0 (compatible
886f4100 534d203b 37204549 203b302e 646e6957 ; MSIE 7.0; Wind
886f4110 2073776f 3620544e 203b302e 43434c53 ows NT 6.0; SLCC
886f4120 2e203b31 2054454e 20524c43 2e302e32 1; .NET CLR 2.0.
886f4130 32373035 2e203b37 2054454e 20524c43 50727; .NET CLR
886f4140 2e302e33 30353430 0a0d2936 74736f48 3.0.04506)..Host
886f4150 7777203a 736d2e77 6f632e6e 500a0d6d : www.msn.com..P
886f4160 79786f72 6e6f432d 7463656e 3a6e6f69 roxy-Connection:
886f4170 65654b20 6c412d70 0d657669 6f6f430a Keep-Alive..Coo
886f4180 3a65696b 31434d20 333d563d 49554726 kie: MC1=V=3&GUI
886f4190 61313d44 64656131 36383138 32343034 D=1a1aed81864042
886f41a0 63616239 61383732 39653765 39353863 9bac278ae7e9c859
886f41b0 203b3964 4d3d686d 3b544653 4c554320 d9; mh=MSFT; CUL
886f41c0 45525554 2d4e453d 203b5355 4449554d TURE=EN-US; MUID
886f41d0 3843323d 36423533 42353432 30463446 =2C835B6245BF4F0
886f41e0 36363836 38373443 39464238 38354330 6866C4788BF90C58
886f41f0 7a203b32 7a3d7069 3143453a 3a616c7c 2; zip=z:EC1|la:
886f4200 352e3135 32323231 7c383831 303a6f6c 51.51222188|lo:0
886f4210 473a637c 72687c42 203b313a 67696c46 |c:GB|hr:1; Flig
886f4220 72477468 4970756f 37343d64 6c46203b htGroupId=47; Fl
886f4230 74686769 423d6449 50657361 3b656761 ightId=BasePage;
886f4240 68737520 72767370 353a4d3d 353a467c ushpsvr=M:5|F:5
886f4250 353a547c 353a457c 623a447c 577c756c |T:5|E:5|D:blu|W
886f4260 507c463a 567c4e3a 203b303a 70687375 :F|P:N|V:0; ushp
886f4270 3d696c63 2e487c30 7c312e30 2e302e47 cli=0|H.0.1|G.0.
886f4280 2e5a7c31 7c312e30 2e302e52 61632e31 1|Z.0.1|R.0.1.ca
886f4290 2e437c70 2e312e30 6e3a676c 6f797765 p|C.0.1.lg:newyo
886f42a0 796e6b72 302e4c7c 4c2e312e 4e573a4e rkny|L.0.1.LN:WN
886f42b0 203b4342 70687375 3d616577 553a6377 BC; ushpwea=wc:U
886f42c0 30594e53 3b363939 68737520 3d727070 SNY0996; ushppr=
886f42d0 3a313a43 37303830 537c3132 303a313a C:1:080721|S:1:0
886f42e0 30383038 68203b36 696c6370 482e573d 80806; hpcli=W.H
886f42f0 7c2e4c7c 527c2e53 2e557c2e 2e437c4c |L.|S.|R.|U.L|C.
886f4300 7068203b 3d727673 7c353a4d 7c353a46 ; hpsvr=M:5|F:5|
886f4310 7c353a54 7c353a45 6c623a44 3a577c75 T:5|E:5|D:blu|W:
886f4320 68203b46 796c6f70 313a4f3d 313a487c F; hpoly=O:1|H:1
886f4330 7077203b 0d303d76 2e0a0d0a 2e6e736d ; wpv=0.....msn.
So this packets contains HTTP traffic for MSN ! (Very true I had the msn site open while I was debugging this machineJ).
A list all NDIS packet pools can be displayed with the pktpools option, each pool would have set of NDIS packets.
kd> !pktpools
Pool Allocator BlocksAllocated BlockSize PktsPerBlock PacketLength
87faa268 8b88d1a1 0x6 0x1000 0x13 0xd0 dc21x4vm!AllocateAdapterMemory+16b
87f6f4e0 8b88d1a1 0x6 0x1000 0x13 0xd0 dc21x4vm!AllocateAdapterMemory+16b
87fbf2f0 8b88d1a1 0x6 0x1000 0x13 0xd0 dc21x4vm!AllocateAdapterMemory+16b
87f4eb40 8b88d1a1 0x6 0x1000 0x13 0xd0 dc21x4vm!AllocateAdapterMemory+16b
87e19620 8174d66c 0x1 0x1000 0x12 0xd8 ndis!DriverEntry+43d
87e19670 8174d65a 0x1 0x1000 0x13 0xd0 ndis!DriverEntry+42b
A list of the NDIS packets in a packet pool can be displayed with the findpacket option. The pool address 87e19670 was obtained from the pktpools output above.
kd> !findpacket p 87e19670
Searching Free block <0x88727000>
Packet at 0x88727538
0x88727538 is our http packet shown above.
Another variant of findpacket is used to find an NDIS packet with a Virtual address from the packet buffer. A random address 886f4110 was obtained from the packet buffer above with http contents.
kd> !findpacket v 886f4110
Searching Free block <0x881a3000>
Searching Used block <0x88185000>
Searching Used block <0x88187000>
Searching Used block <0x8818b000>
Searching Used block <0x8818f000>
Searching Used block <0x88193000>
Searching Free block <0x88176000>
Packet found
PhysicalCount 00000000 Total Length 00000000
Head 00000000 Tail 00000000
Pool 00000000 Count 00000000
Flags 00000000 ValidCounts 00
NdisPacketFlags 00000000 NdisPacketOobOffset 0000
Private.Flags : 00000082
DONT_LOOPBACK,
Private.NdisPacketFlags: 90
fPACKET_PENDING, fPACKET_CLEAR_ITEMS, fPACKET_ALLOCATED_BY_NDIS
I hope this gives the reader a better understanding of NDIS stacks.
So you have a dump from a hung server and you’re the first person on the scene. Your IT Manager is jumping up and down, the phone is ringing off the hook and people are hovering outside your cube. It’s game time and the pressure is on!!! Now what do you do?
Well take a deep breath, get a cup of coffee, and relax because I’m here to help you out! Let me share what we typically do on our first pass through a hung server kernel debug. This works for both live debugs and dumps. These are steps you can take and they will find problems!
Here’s something else to consider. If the server is mission critical you will probably want to get a dump vs. a live debug so you can get the server back up and running. This will take the pressure off because you can then do the debug offline, and if need be, send the dump to other people for review.
Before we get started let me state that the following data is completely fabricated and many of the process names and address in this output have been made up. Do not question odd offsets or alignments.
I’m also assuming that you know how to
1. Collect a kernel dump: http://support.microsoft.com/kb/244139
2. Set up the debugger: http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
3. Know how to use the symbol server: http://support.microsoft.com/kb/311503
0) Before I start these types of debugs I like to open a log file.
1: kd> .logopen H:\repro\hungserver.log
Opened log file 'H:\repro\hungserver.log'
1) !vm - Look for memory usage. Generally speaking you want to look at what the current pool or memory usage values are and compare them to the max available.
1: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 982890 ( 3931560 Kb)
Page File: \??\P:\pagefile.sys
Current: 3931560 Kb Free Space: 3742548 Kb
Minimum: 3931560 Kb Maximum: 4193280 Kb
Available Pages: 631300 ( 2525200 Kb)
ResAvail Pages: 888171 ( 3552684 Kb)
Locked IO Pages: 195 ( 780 Kb)
Free System PTEs: 202830 ( 811324 Kb) < THIS IS OK
Free NP PTEs: 32765 ( 131060 Kb) < THIS IS OK
Free Special NP: 0 ( 0 Kb)
Modified Pages: 241 ( 964 Kb)
Modified PF Pages: 241 ( 964 Kb)
NonPagedPool Usage: 11377 ( 45508 Kb) < THIS IS OK
NonPagedPool Max: 65536 ( 262144 Kb)
PagedPool 0 Usage: 6398 ( 25592 Kb)
PagedPool 1 Usage: 2201 ( 8804 Kb)
PagedPool 2 Usage: 2216 ( 8864 Kb)
PagedPool 3 Usage: 2179 ( 8716 Kb)
PagedPool 4 Usage: 2199 ( 8796 Kb)
PagedPool Usage: 15193 ( 60772 Kb) < THIS IS OK
PagedPool Maximum: 67584 ( 270336 Kb)
Shared Commit: 24569 ( 98276 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 12519 ( 50076 Kb)
PagedPool Commit: 15252 ( 61008 Kb)
Driver Commit: 2083 ( 8332 Kb)
Committed pages: 313611 ( 1254444 Kb) < THIS IS OK
Commit limit: 1925815 ( 7703260 Kb)
Check to see if any apps are using tons of memory. In this case I don’t see a problem.
Total Private: 239673 ( 958692 Kb)
36b0 EXCEL.EXE 10775 ( 43100 Kb) < THIS IS OK, etc
2ee8 myapploc.exe 10288 ( 41152 Kb)
097c MySSrv.exe 7497 ( 29988 Kb)
0418 MyFun32.exe 6277 ( 25108 Kb)
0474 svchost.exe 6164 ( 24656 Kb)
1be8 ABCDEFGH.EXE 4984 ( 19936 Kb)
0480 IEXPLORE.EXE 4924 ( 19696 Kb)
09c4 ANOTHER.exe 4768 ( 19072 Kb)
19a4 HMMINTER.exe 4207 ( 16828 Kb)
1b30 ohboya.EXE 4146 ( 16584 Kb)
4558 aprocess.EXE 4138 ( 16552 Kb)
30e8 another.exe 3691 ( 14764 Kb)
0924 aservicec.exe 3508 ( 14032 Kb)
0854 RRXXc.exe 3400 ( 13600 Kb)
3458 MYWIN.EXE 3389 ( 13556 Kb)
0d90 FunService.exe 3298 ( 13192 Kb)
1180 CustomAp.exe 3221 ( 12884 Kb)
06ac XYZvrver.exe 2769 ( 11076 Kb)
2cdc ABCDEFGH.exe 2591 ( 10364 Kb)
02f4 lsass.exe 2567 ( 10268 Kb)
21b4 IEXPLORE.EXE 2516 ( 10064 Kb)
3420 Process.exe 2450 ( 9800 Kb)
4cd4 XYZXY.EXE 2305 ( 9220 Kb)
4a30 lookup.EXE 2244 ( 8976 Kb)
4360 Process.exe 2201 ( 8804 Kb)
0564 spoolsv.exe 2166 ( 8664 Kb)
2e5c XYZXYZEXE 2076 ( 8304 Kb)
02bc winlogon.exe 1964 ( 7856 Kb)
4e48 winlogon.exe 1958 ( 7832 Kb)
42bc ABCDEFGH.exe 1943 ( 7772 Kb)
0eb8 svchost.exe 1922 ( 7688 Kb)
3b98 Process.exe 1919 ( 7676 Kb)
4c1c IEXPLORE.EXE 1864 ( 7456 Kb)
17b8 winlogon.exe 1852 ( 7408 Kb)
3124 winlogon.exe 1849 ( 7396 Kb)
14b8 winlogon.exe 1847 ( 7388 Kb)
32cc winlogon.exe 1843 ( 7372 Kb)
1f84 winlogon.exe 1843 ( 7372 Kb)
2ebc winlogon.exe 1842 ( 7368 Kb)
1548 winlogon.exe 1840 ( 7360 Kb)
21c4 PROCESS213.EXE 1833 ( 7332 Kb)
3b58 MYWIN.EXE 1817 ( 7268 Kb)
4b3c winlogon.exe 1816 ( 7264 Kb)
NOTE if you see high pool values you will want to issue a !poolused 2 and a !poolused 4 to dump out the pool usages so you can see what pool tags are consuming pool. (We will write a dedicated blog on this topic later.)
2) !sysptes - See if one of the lists is low (less than 10)
1: kd> !sysptes
All of these are ok
System PTE Information
Total System Ptes 224223
SysPtes list of size 1 has 225 free
SysPtes list of size 2 has 57 free
SysPtes list of size 4 has 136 free
SysPtes list of size 8 has 59 free
SysPtes list of size 16 has 95 free
starting PTE: c022b000
ending PTE: c03dff78
free blocks: 652 total free: 202831 largest free block: 191973
3) !defwrites - If throttling, the server is doing nothing other than writing to the disk.
1: kd> !defwrites
*** Cache Write Throttle Analysis ***
CcTotalDirtyPages: 187 ( 748 Kb)
CcDirtyPageThreshold: 130560 ( 522240 Kb)
MmAvailablePages: 631300 ( 2525200 Kb)
MmThrottleTop: 450 ( 1800 Kb)
MmThrottleBottom: 80 ( 320 Kb)
MmModifiedPageListHead.Total: 241 ( 964 Kb)
Write throttles not engaged < THIS IS OK. Good = NOT engaged.
4) !ready to see if we're holding stuff up
1: kd> !ready
Processor 0: No threads in READY state < THIS IS OK
Processor 1: No threads in READY state < THIS IS OK
If we had threads in a ready state you would want to investigate what those threads were and what is running on the processor.
5) !pcr x; kv on each processor - If they aren't idle then we could be doing DPCs
1: kd> !pcr 0 < Dump the processor control registers for CPU 0
KPCR for Processor 0 at ffdff000:
Major 1 Minor 1
NtTib.ExceptionList: ffffffff
NtTib.StackBase: 00000000
NtTib.StackLimit: 00000000
NtTib.SubSystemTib: 80042000
NtTib.Version: 012e7ace
NtTib.UserPointer: 00000001
NtTib.SelfTib: 00000000
SelfPcr: ffdff000
Prcb: ffdff120
Irql: 00000000
IRR: 00000000
IDR: ffffffff
InterruptMode: 00000000
IDT: 8003f400
GDT: 8003f000
TSS: 80042000
CurrentThread: 8056cd00
NextThread: 00000000
IdleThread: 8056cd00
DpcQueue: < NO DPCs: Not much to look at then
1: kd> !pcr 1 < Dump the processor control registers for CPU 1
KPCR for Processor 1 at f773f000:
NtTib.ExceptionList: f5ba1d30
NtTib.SubSystemTib: f773fef0
NtTib.Version: 0121925d
NtTib.UserPointer: 00000002
NtTib.SelfTib: 7ffda000
SelfPcr: f773f000
Prcb: f773f120
IDT: f77456e0
GDT: f77452e0
TSS: f773fef0
CurrentThread: 8963cb90
IdleThread: f7741fa0
6) !locks - Look for deadlocks and contention
The following output is of interest.
The thread ID with the <*> next to it means that he has exclusive access to the resource and that all the other threads are waiting on that thread to finish its work. Typically you would !thread that OWNER THREAD ID <*> (e.g., !thread 87bddda0) to see what that thread is doing. If you have two threads that have exclusive access to two different resources, and these threads are in each other’s exclusive waiters list, you have a deadlock. The following is an example of what a deadlock might look like. In this case you would want to !thread each owner and evaluate the logic of the code in each stack that allowed the threads to get into this state
1: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks......
Resource @ 0x8a50ee98 Shared 4 owning threads
Threads: 896856d0-01<*> 89686778-01<*> 896862d0-01<*> 89685da0-01<*>
KD: Scanning for held locks............................................................
Resource @ 0x896da1bc Exclusively owned
Threads: 896e3b20-01<*>
KD: Scanning for held locks..
Resource @ 0x81234567 Shared 1 owning threads
Contention Count = 15292
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 39
Threads: 87bddda0-01<*> 806d2020-01
Threads Waiting On Exclusive Access:
80ced020 80c036f8 80cdc7a0 80c438b0
80e6cda0 80f96987 8007fd60 8004dc10
80d7b020 80a2dd70 80b89620 80b58020
8036eda0 87abc123 80606da0 8056e890
802b3630 80cc7590 80d64020 80f7dda0
80129580 80b73da0 806d2578 80b505d8
KD: Scanning for held locks................
Resource @ 0x83245678 Exclusively owned
Contention Count = 4827
NumberOfExclusiveWaiters = 35
Threads: 87abc123-01<*>
803e6aa0 80876020 80240020 80f56588
808174f0 80bd6b28 80c3c448 8046d6c8
801e8da0 80356518 80b4c978 8069e020
80cb9020 87bddda0 80c65020 86daaac0
80379020 80fe4020
8) !process 0 0 - Search for drwtsn32. This would indicate that we have a process that has crashed and is in the process of being dumped. This could cause a server hang. Look at the PEB for drwtsn32 and get its command line to see what process is being dumped. You should be able to do this by getting its process id and doing a .process PROCESSID;.reload;!PEB
The following is how to extract a command line for any process, but it would work for Watson also.
1: kd> .process 89f31020
Implicit process is now 89f31020
1: kd> .reload
Loading Kernel Symbols
...........................................................................................................................................
Loading User Symbols
...............................
Loading unloaded module list
...............
1: kd> !peb
PEB at 7ffdf000
InheritedAddressSpace: No
ReadImageFileExecOptions: Yes
BeingDebugged: No
ImageBaseAddress: 01000000
Ldr 77fc23a0
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 00171ef8 . 00176c90
Ldr.InLoadOrderModuleList: 00171e90 . 00176c80
Ldr.InMemoryOrderModuleList: 00171e98 . 00176c88
Base TimeStamp Module
1000000 3e80245d Mar 24 05:41:49 2003 \??\P:\WINDOWS\system32\winlogon.exe
77f40000 3e802494 Mar 25 05:42:44 2003 P:\WINDOWS\system32\ntdll.dll
77e40000 44c60ec8 Jul 25 08:30:00 2006 P:\WINDOWS\system32\kernel32.dll
77ba0000 3e802496 Mar 25 05:42:46 2003 P:\WINDOWS\system32\msvcrt.dll
77da0000 3e802495 Mar 25 05:42:45 2003 P:\WINDOWS\system32\ADVAPI32.dll
77c50000 40566fc9 Mar 15 23:08:57 2004 P:\WINDOWS\system32\RPCRT4.dll
77d00000 45e7bafc Mar 02 00:49:48 2007 P:\WINDOWS\system32\USER32.dll
77c00000 45e7bafc Mar 02 00:49:48 2007 P:\WINDOWS\system32\GDI32.dll
75970000 3e8024a2 Mar 25 05:42:58 2003 P:\WINDOWS\system32\USERENV.dll
75810000 3e8024a3 Mar 25 05:42:59 2003 P:\WINDOWS\system32\NDdeApi.dll
761b0000 3e8024a0 Mar 25 05:42:56 2003 P:\WINDOWS\system32\CRYPT32.dll
SubSystemData: 00000000
ProcessHeap: 00070000
ProcessParameters: 00020000
WindowTitle: '< Name not readable >'
ImageFile: '\??\P:\WINDOWS\system32\winlogon.exe'
CommandLine: 'winlogon.exe' < HERE IS THE COMMAND LINE.. No args in this case
( output is truncated ... )
9) Look at the handle table size. If it’s over 10000 you may have trouble. If you do have a handle leak refer to TalkBackVideo Understanding handle leaks and How to use !htrace to find them
1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8a613270 SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
DirBase: 0acc0000 ObjectTable: e1001d10 HandleCount: 2510.
Image: System
PROCESS 8a294328 SessionId: none Cid: 0274 Peb: 7ffdf000 ParentCid: 0004
DirBase: ef1ac000 ObjectTable: e14ac1d0 HandleCount: 124.
Image: smss.exe
PROCESS 8a103424 SessionId: 0 Cid: 02a4 Peb: 7ffdf000 ParentCid: 0274
DirBase: ed804000 ObjectTable: e18caa68 HandleCount: 1171.
Image: csrss.exe
PROCESS 8a104343 SessionId: 0 Cid: 02bc Peb: 7ffdf000 ParentCid: 0274
DirBase: ed539000 ObjectTable: e18c67b0 HandleCount: 498.
Image: winlogon.exe
PROCESS 8a0f6634 SessionId: 0 Cid: 02e8 Peb: 7ffdf000 ParentCid: 02bc
DirBase: ece72000 ObjectTable: e1668e40 HandleCount: 568.
Image: services.exe
PROCESS 8a123423 SessionId: 0 Cid: 02f4 Peb: 7ffdf000 ParentCid: 02bc
DirBase: ecd7a000 ObjectTable: e16684a0 HandleCount: 30000. < This is bad
Image: lsass.exe
PROCESS 89f96453 SessionId: 0 Cid: 03e0 Peb: 7ffdf000 ParentCid: 02e8
DirBase: eb99c000 ObjectTable: e16bb570 HandleCount: 500.
Image: svchost.exe
PROCESS 8a0c6532 SessionId: 0 Cid: 042c Peb: 7ffdf000 ParentCid: 02e8
DirBase: eb6d7000 ObjectTable: e1731170 HandleCount: 156.
PROCESS 8a0a8d88 SessionId: 0 Cid: 0460 Peb: 7ffdf000 ParentCid: 02e8
DirBase: eb58f000 ObjectTable: e17372e8 HandleCount: 124.
PROCESS 89f77678 SessionId: 0 Cid: 0474 Peb: 7ffdf000 ParentCid: 02e8
DirBase: eb484000 ObjectTable: e17305b8 HandleCount: 1457.
9) !process 0 0 system - Check the worker threads in the system process (search for srv! to find server worker threads). What are these threads doing? These are the server service threads. Are they blocked on I/O or waiting for a resource?
10) 1: kd> !process 0 17 csrss.exe - Look for 16 LPC server threads.
What are they doing? Are they blocked?
11) !stacks 2, This will dump every call stack on the server. You may need to go through and evaluate every stack on the server. Look for critical sections, etc.
15) !qlocks This will allow you to check the stack of all the Queued spin locks on the machine. For further information on spinlocks refer to the Windows Internals book.
1: kd> !qlocks
Key: O = Owner, 1-n = Wait order, blank = not owned/waiting, C = Corrupt
Processor Number
Lock Name 0 1 << Nothing to worry about here.
KE - Dispatcher
MM - Expansion
MM - PFN
MM - System Space
CC - Vacb
CC - Master
EX - NonPagedPool
IO - Cancel
EX - WorkQueue
IO - Vpb
IO - Database
IO - Completion
NTFS - Struct
AFD - WorkQueue
CC - Bcb
MM - NonPagedPool
16) !process 0 17 winlogon.exe to look for hung LPC calls. If you find a LPC call calling out of winlogon you can follow the call with the !LPC debugger command. This will allow you to see what the thread is doing in the other process.
If you have further questions on any of these commands, please refer to the debugger.chm file in the Windows debugger tools install.
Good luck and happy debugging.
“This debugger is mine, there are many like it but this one is mine!” Jeff Dailey