• Ntdebugging Blog

    Remote kernel or user mode debugging of dumps or live systems



    GES (Global Escalation Services) is not only responsible for helping our external customers, but we spend a great deal of time collaborating with engineers and developers around the world at our support and development sites.  We often look at large dump files, but in some cases we perform a live debug to determine root cause of a server failure.  In the case of a memory dump, the files are usually very large, so copying the files over the network, even on the fastest WAN connection, can take a LONG time.

    The solution is remote debugging.    

    Here is how you do it!

    First of all it takes two to tango: a remote person and the expert who will help the remote person by debugging the process using the debugger installed on their machine.

    Let’s say you are the expert who is helping the remote person. Here’s how the process works:

    1.       The remote person opens a dump file, debugs a process, or kernel debugs a machine at some remote location using windbg.

    2.       The remote person decides, “I NEED HELP!”

    3.       The remote person simply types in .server tcp:port=9999 at the windbg prompt.



    Notice the following output.


    Server started.  Client can connect with any of these command lines

    0: <debugger> -remote tcp:Port=9999,Server=MyServerName



    1.       The remote person sends email or IM to the person they want help from with the connection string <debugger> -remote tcp:port=9999,Server=MyServerName

    2.       The expert runs WINDBG -remote tcp:port=9999,Server=MyServerName from the debuggers directory.

    3.       At this point the remote person should see the following message at the remote debugger site.


    EXPERTMACHINE\expert (tcp connected at Tue Mar 25 15:36:53 2008


    Once connected, the expert can issue any command to debug the dump or target machine remotely.   The great part is many people can connect to the remote debugger session if needed.  It’s a great collaboration tool, and something we use every day at Microsoft.   


    When remote debugging, I find it useful to save the debug session in a log file.  It’s as easy as typing .logopen C:\mydebuggersession.log in windbg before the remote debug session starts.  This gives everyone the opportunity to look at the debug session later if necessary.


    One last thing to keep in mind about remote debugging is security. I recommend using the .noshell command to prevent the execution of remote shell commands. Without the .noshell command, it is possible for people connected to your session to use the .shell (Command Shell) command to execute an application or a Microsoft MS-DOS command directly from the debugger.


    Thanks Jeff-



  • Ntdebugging Blog

    Windows Hotfixes and Updates - How do they work?


    Today I would like to talk about some of the work the Windows Serviceability (WinSE) team does regarding servicing Windows and releasing updates.

    The operating system is divided into multiple components. Each component can consist of one or more files, registry keys, configuration settings, etc.  WinSE releases updates based on components rather than the entire operating system. This reduces a lot of overhead with having to install updates to components that have not changed. Depending on the severity and applicability of the problem, there are different kinds of release mechanisms. Keep in mind, though, the actual fix still remains the same.

    1.       Updates and Security Updates

    These Updates are typically available on Windows Update. They frequently contain security fixes, and from time to time also contain reliability rollup packages. These updates are thoroughly tested and Microsoft highly recommends that you update your computer with these releases. In fact, most are automatically downloaded to your machine if you have Windows Update turned on. In most cases, Update releases are also available as standalone downloads from the download center.


    2.       Hotfixes

    When an individual customer reports a bug to Microsoft for a specific scenario, the WinSE team releases Hotfixes to address these problems. Hotfixes are not meant to be widely distributed and go through a limited amount of testing due to the customer's need for an urgent fix.  Hotfixes are developed in a separate environment than the regular Updates.  This allows Microsoft to release Updates that do not include the Hotfix files, thereby minimizing risk for the customer.

    Once the Hotfix is ready and packaged by WinSE, a KB article is written describing the problem, with instructions on how to obtain the Hotfix.  Microsoft recommends that only customers experiencing the particular problem install the Hotfix for that problem.

    Note: Hotfixes are also sometimes referred to as LDRs, or QFE's (Quick Fix Engineering). The term QFE is an old term that is mostly no longer used in reference to current versions of Windows.


    3.       SP  - Service Pack

    The service pack is a major update in the life of an OS. It contains a wide variety of fixes as well as all the GDR and LDR fixes that were released since the previous service pack was shipped. This is a thoroughly tested release and highly recommended by Microsoft for installation. This is usually available as a standalone release, and is then released through Windows Update as well.



    GDR vs. LDR branches

    Now that we have described the different kinds of updates, let's take a deeper look into how these fixes are built. When a new OS or service pack is released, 2 branches are created from the release code tree -a GDR (general distribution release) branch and a LDR (limited distribution release) branch. Hotfixes are built solely from the LDR branch, while Updates for broad release are built from the GDR branch.

    Service Packs are built from a third branch that contains all Updates , Hotfixes and additional fixes.  This way the new service pack is shipped with all the fixes from both branches.

    Note – Once the new service pack is shipped, the code branches from the previous release are still active and serviced as necessary.

    Installing a Hotfix

    By default, all components on Windows systems start on the GDR branch following each major release. When you install updates from Windows Update for a GDR component, it gets upgraded with the GDR version.

    When you install a specific Hotfix, the files and components in the Hotfix package are migrated to the LDR branch. At this point, that particular component is marked as a LDR component. If you install a newer Update over this component, the Windows servicing technology will automatically install the appropriate latest version from the LDR branch for you. This is possible because each Update package ships with both the GDR and LDR versions of the component.

    Once a component is marked as a LDR component, the only way to move back to the GDR branch is to uninstall all Hotfixes for that component, or move to the next available service pack.


    What would happen if a user installed a Hotfix, and then sometime later installed the next service pack? Well, in that case it depends on the Hotfix and when it was built.

    1.       If the Hotfix was built before the service pack, then the component will be moved to the GDR version contained in the service pack.

    2.       If the Hotfix was built after the service pack, the component will be migrated to the post-service pack version of the component, and will stay on the same branch that it was originally on.


    In order to make this work, these packages contain both the RTM GDR version, the RTM Hotfix branch, and the SP1 Hotfix and GDR version of each binary.


    All fixes built for Windows are cumulative in nature by branch, i.e. a new update will contain the new fix, as well as all the previous fixes for that branch. Referencing the chart above, installing fix #4 can get you fixes #2 and #4 on the GDR branch. If the component is on the LDR branch, then the user would get fixes #1-4.


    Finally, the servicing technology has to handle the case where you need the functionality of an older Hotfix (e.g. “Fix #1” in the diagram above) but you may already have installed “Fix #4” which might be a critical security update.  What happens is that when the GDR branch of a fix is installed, it also places a copy of the Hotfix version of the same fix on the system.  When you run the installer for Hotfix #1, it detects that a newer version of the file is already installed, but it also detects that it needs to migrate it to the Hotfix version of the binary that was previously stored on the system. The result is that you end up with the Hotfix binary for Fix #4, which has both the Hotfix you need plus the cumulative set of security fixes.


    Stay tuned for more, in the next blog entry, I will talk about the staging mechanism that Windows uses to install Updates and Hotfixes as well as the uninstall process. Also, I will talk about how to determine the branch a file is built from.


    - Omer 


    More Information

    Description of the standard terminology that is used to describe Microsoft software updates

    Description of the contents of Windows XP Service Pack 2 and Windows Server 2003 software update packages


  • Ntdebugging Blog

    How to Determine Which Resource is Causing the Cluster Resource Monitor to Crash – Possible Deadlock


    Hello, my name is John Marlin, and I am a Support Escalation Engineer on the Microsoft Platform Cluster Services Support team.  I wanted to talk about the Windows 2003 Cluster Resource Monitor and with what happens when it crashes. In this blog I’ll show you how to look under the hood to determine why it crashed.

    As a foundation for this article we need to understand the basics of Cluster Resource Monitor.  Below is taken from the Microsoft MSDN site describing the Cluster Resource Monitor.


    A Resource Monitor provides a communication, monitoring, and processing layer between the Cluster service and one or more resources. Resource Monitors have the following characteristics:

    ·         A Resource Monitor always runs in a process separate from the Cluster service. If a resource fails, the Resource Monitor isolates the Cluster service from the effects. If the Cluster service fails, the Resource Monitor allows its resources to shut down gracefully.

    ·         To work with a resource, a Resource Monitor loads the resource DLL responsible for that resource type into its process.

    ·         When the Cluster service requests an operation on a resource, the Resource Monitor routes the request to the appropriate entry point function of the resource DLL responsible for the resource. The Resource Monitor performs default processing for some resource operations.

    ·         A Resource Monitor stores synchronized state data, allowing the Cluster service and resource DLLs to operate asynchronously, checking and updating resource status as needed.

    ·         A Resource Monitor periodically checks the operational status of all of its resources. For more information on this process, see Resource Failure.


    By default, the Cluster service creates one Resource Monitor per node.


    As the MSDN information states, everything currently running on the node is in the one Resource Monitor.  If the Resource Monitor crashes, the system will dump the Resource Monitor Process to a file called RESRCMON.DMP, and create a new instance of the process.  Because it must create a new one, all resources in the monitor are gone and need to be restarted.  When this occurs, you would see the following entry in the Windows System Event Log. 


    Event ID:  1146

    Source:  ClusSvc

    Description:  The cluster resource monitor died unexpectedly, an attempt will be made to restart it


    After this, you could also see other resource failures (Event ID: 1069) as well as disk related events such as Lost Delayed Writes, etc.  You would see the disk related events because the disk(s) would be considered down and since there is data in the cache of the HBA, it has nowhere to write it.  Hence, lost delayed writes exist until the disk is brought back online.  For our examples here, we will ignore these disk related events as we will focus on why the Resource Monitor crashed.


    There are a couple reasons why a Resource Monitor would crash such as an Access Violation (0xC0000005) or a Deadlock (0xC0000194).  In a previous blog, we talked about the Access Violation (0xC0000005).  This blog will focus on deadlocks (0xC0000194) where a particular resource DLL was not responding properly.


    Along with the above System Event (Event ID: 1146) where the Resource Monitor died, you will see 0xc00000494 entries in the Cluster Log file. 


    NOTE:  The Cluster Log will convert times to Greenwich Mean Time (GMT), so you must ensure you do the proper GMT conversion of time to get to the location in the Cluster Log.


    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] Exception. Code = 0xc0000194, Address = 0x77E4BEE7

    00000a20.00002300::2008/02/06-20:10:25.762 ERR  [RM] Exception parameters: 0, d2fa2000, fffffff4, 7ffda000

    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] CallStack:

    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] Frame      Address

    00000a20.00002300::2008/02/06-20:10:25.762 INFO [RM] GenerateMemoryDump: Start memory dump to file C:\WINNT\Cluster\resrcmon.dmp


    Now that we see this entry in the log, we should take a look at the Resource Monitor dump to see what caused the failure.  The first thing to examine is the register states, specifically the ESP (stack pointer) value.


    0:017> r

    eax=00970000 ebx=000f0cd8 ecx=00000007 edx=7c8285ec esi=000f0cb0 edi=000f0d08

    eip=7c8285ec esp=0227ee10 ebp=0227ee20 iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246


    7c8285ec c3              ret


    Starting at the stack pointer address 0227ee10, we use the dds command to dump the raw stack.  We are looking for the value on the stack just below the routine resrcmon!GenerateMemoryDump.  It will take several iterations of the dds command to finally get to the value because the call was made much earlier in the stack.


    0:017> dds 0227ee10

    0227ee10  00700053 xpsp2res+0xc0053

    0227ee14  00630061

    0227ee18  005f0065

    *** pages removed ***

    0227f650  028a0c90

    0227f654  01ece948

    0227f658  01e94518

    0227f65c  0227f8ac                                           <<-- this address

    0227f660  0100e638 ResrcMon!GenerateMemoryDump+0x180

    0227f664  ffffffff

    0227f668  00001258

    0227f66c  00000018


    Now that we have our value, we will use the kv= command with the value 0227f8ac to dump out the stack contents.


    0:017> kv=0227f8ac


    ChildEBP RetAddr  Args to Child             

    0227eea0 7c826d2b 000801a8 0000066c 0227eff4 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    0227ef10 7c826c9b 7c833c4e ffffffff 0227ef5c ntdll!ZwClose+0xc (FPO: [1,0,0])

    0227efec 77e63f55 ffffffff 02a00000 00000008 ntdll!ZwAllocateVirtualMemory+0xc (FPO: [6,0,0])

    0227f8ac 0100e989 0227fb78 01003024 00000000 kernel32!UnmapViewOfFile+0x14 (FPO: [Non-Fpo])

    0227f8c4 01008b2c 0227fb78 01003024 0227fb78 ResrcMon!GenerateExceptionReport+0x7e (FPO: [Non-Fpo])

    0227f8d8 76348d17 0227fb78 0227fb78 0227f8f8 ResrcMon!RmpExceptionFilter+0x14 (FPO: [Non-Fpo])

    0227f8e8 7786d6d2 0227fb78 77ecb7c0 0227fb50 netshell!__CxxUnhandledExceptionFilter+0x4a (FPO: [Non-Fpo])

    0227f8f8 77e761b7 0227fb78 00000000 00000000 netman!__CxxUnhandledExceptionFilter+0x4a (FPO: [Non-Fpo])

    0227fb50 77e792a3 0227fb78 77e61ac1 0227fb80 kernel32!UnhandledExceptionFilter+0x12a (FPO: [Non-Fpo])

    0227fb58 77e61ac1 0227fb80 00000000 0227fb80 kernel32!BaseThreadStart+0x4a (FPO: [SEH])

    0227fb80 7c828752 0227ff3c 0227ffdc 0227fc5c kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7])

    0227fba4 7c828723 0227ff3c 0227ffdc 0227fc5c ntdll!ExecuteHandler2+0x26

    0227fc4c 7c82863c 0227e000 0227fc5c 00010007 ntdll!ExecuteHandler+0x24

    0227ff2c 77e4bee7 0227ff3c 000a7ca0 c0000194 ntdll!RtlRaiseException+0x3d

    0227ff8c 01007ddd c0000194 00000000 00000000 kernel32!RaiseException+0x53 (FPO: [Non-Fpo])

    0227ffb8 77e64829 000009ec 00000000 00000000 ResrcMon!RmpTimerThread+0xa8 (FPO: [Non-Fpo])

    0227ffec 00000000 01007d35 000a7ca0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])


    Based on the stack above, we have an exception at address 0x227fb78 which we will use to set the failing context.


    0:017> dc 0227fb78

    0227fb78  0227ff3c 0227fc5c 0227fba4 7c828752  <.'.\.'...'.R..|    <<-- Exception and Context Records

    0227fb88  0227ff3c 0227ffdc 0227fc5c 0227fc40  <.'...'.\.'.@.'.

    0227fb98  0227ffdc 7c828766 0227ffdc 0227fc4c  ..'.f..|..'.L.'.

    0227fba8  7c828723 0227ff3c 0227ffdc 0227fc5c  #..|<.'...'.\.'.

    0227fbb8  0227fc40 77e61a60 00000000 0227ff3c  @.'.`..w....<.'.

    0227fbc8  0227ffdc 7c8315c2 0227ff3c 0227ffdc  ..'....|<.'...'.

    0227fbd8  0227fc5c 0227fc40 77e61a60 00000102  \.'.@.'.`..w....

    0227fbe8  00000000 00000000 4f464e49 000a0d20  ........INFO ...


    The first DWORD is the Exception Record (0x0227ff3c) and the second is the Context Record (0x0227fc5c) that holds our true stack where the problem occurred.   Based on the .exr output, we see that this is a possible deadlock.


    0:017> .exr 0227ff3c

    ExceptionAddress: 77e4bee7 (kernel32!RaiseException+0x00000053)

       ExceptionCode: c0000194

      ExceptionFlags: 00000000

    NumberParameters: 0


    0:017> !error c0000194

    Error code: (NTSTATUS) 0xc0000194 (3221225876) - {EXCEPTION}  Possible deadlock condition.


    So let’s set the context to get the thread that caused the Resource Monitor to crash.


    0:017> .cxr 0227fc5c

    eax=0227ff3c ebx=00000000 ecx=77e61d43 edx=7c8285ec esi=00000000 edi=00000102

    eip=77e4bee7 esp=0227ff38 ebp=0227ff8c iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246


    77e4bee7 5e              pop     esi


    0:017> kv

      *** Stack trace for last set context - .thread/.cxr resets it

    ChildEBP RetAddr  Args to Child             

    0227ff8c 01007ddd c0000194 00000000 00000000 kernel32!RaiseException+0x53 (FPO: [Non-Fpo])

    0227ffb8 77e64829 000009ec 00000000 00000000 ResrcMon!RmpTimerThread+0xa8 (FPO: [Non-Fpo])     <<--

    0227ffec 00000000 01007d35 000a7ca0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])


    Based on the names of the functions in the stack below, we are in the timer thread.  As a side note, with the 0xC0000194 Resource Dumps, we tend to be either in the Timer Thread itself or the Event List that holds the current resource.  In this instance, we are in the timer thread. 


    Now let’s look at all the threads using the ~*kb command to find the Event List for this timer thread. I truncated the results showing thread 1 and thread 18.


    0:001> kv

    ChildEBP RetAddr  Args to Child             

    0095fc5c 7c827d0b 77e61d1e 00000958 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    0095fc60 77e61d1e 00000958 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])

    0095fcd0 77e61c8d 00000958 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [Non-Fpo])

    0095fce4 310079b8 00000958 ffffffff 010119e0 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])

    WARNING: Stack unwind information not available. Following frames may be wrong.

    0095fd04 31005464 00000958 000ab638 00000000 JohnHungApp!Startup+0x26de

    0095fe28 0100a385 00000002 0095fe58 01006548 JohnHungApp!Startup+0x18a

    0095fe34 01006548 000ab638 01c868a2 00088038 ResrcMon!Resmon_LooksAlive+0x14 (FPO: [Non-Fpo])

    0095fe58 01006728 00000000 00000102 00000000 ResrcMon!RmpPollBucket+0x156 (FPO: [Non-Fpo])

    0095fe78 010068c9 0008d908 00000000 00000000 ResrcMon!RmpPollList+0xa3 ß-- My Poll Event List

    0095ffb8 77e64829 00000102 00000000 00000000 ResrcMon!RmpPollerThread+0x133 ß-My Timer thread

    0095ffec 00000000 01006796 0008d908 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])


    0:018> kv

    ChildEBP RetAddr  Args to Child             

    022bfbd8 7c827d0b 77e61d1e 000003bc 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    022bfbdc 77e61d1e 000003bc 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])

    022bfc4c 77e61c8d 000003bc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [Non-Fpo])

    022bfc60 10032719 000003bc ffffffff 00000000 kernel32!WaitForSingleObject+0x12 (FPO: [Non-Fpo])

    WARNING: Stack unwind information not available. Following frames may be wrong.

    022bfc78 100fdb0e 000003bc ffffffff 100f7c6c JohnHungApp!VxWaitForEvent+0x22

    022bfd68 100c66d8 ffffffff 00000001 00000000 JohnHungApp!getAllChildNodes+0x8d172

    022bfde4 10012357 0026d838 01ece3f0 022bff44 JohnHungApp!getAllChildNodes+0x89914

    022bffb8 77e64829 0095fd14 00000000 00000000 JohnHungApp!Startup+0x24a1 ------ My Timer Thread

    022bffec 00000000 3100767d 0095fd14 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

    Threads 1 and 18 are of most interest as they show JohnHungApp Resource DLL waiting while the PollerThread performs a LooksAlive check.  So this is a likely cause as it did not respond within the expected 80 minute window, causing the crash.  The thread waits on the resource for 80 minutes.

    The next course of action is to review the Poll List to ensure that this is working with JohnHungApp, and to determine type of resource.  So let’s dump the DWORD values of my list (adding +0x114, which is the offset in the data structure that contains the string). Note: We cannot guarantee offsets in future versions of the component.


    0:017> dc 0008d908+0x114 l1

    0008da1c  0008d090                             ....                 <<-- My Resource


    0:017> dc 0008d090

    0008d090  63727352 00000001 0009ec20 000a7ca8  Rsrc.... ....|..

    0008d0a0  000a15f0 0008d158 0008d038 000a7070  ....X...8...pp..     <<-- My Resource strings

    0008d0b0  00001388 0000ea60 60180000 000a7b70  ....`......`p{..

    0008d0c0  00000000 00000000 00000000 00000000  ................

    0008d0d0  00000000 00000001 601c1d8d 601c3633  ...........`36.`

    0008d0e0  601c34cf 601c3549 601c35b5 601c21c5  .4.`I5.`.5.`.!.`

    0008d0f0  601c20e1 00000000 00000000 601c3712  . .`.........7.`

    0008d100  601c222f 00000003 00000006 0000000c  /".`............


    0:017> du 0x000a7070                         <<-- Resource Displayed in Cluster

    000a7070  "Johns Hung Resource"


    0:017> du 0x0008d038                         <<-- GUID in registry (HKLM\Cluster\Resources)

    0008d038  "0502cab5-3e1f-47d4-b490-e5301be7"

    0008d078  "2928"


    0:017> du 0x0008d158                         <<-- Resource Type

    0008d158  "Johns Hung App"


    0:017> du 0x000a15f0                         <<-- Specific DLL being Used

    000a15f0  "johnhungapp.dll"


    So this confirms that Johnhungapp.dll is the resource with the problem.  At this point we know the name and GUID of the resource.  Remember, we put an 80 minute time out check on the resource.   Sometimes you can get a little more information from the Cluster Log to possible narrow down the root cause.  So we are back to the original entry of where the Resource Monitor dumped.



    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] Exception. Code = 0xc0000194, Address = 0x77E4BEE7

    00000a20.00002300::2008/02/06-20:10:25.762 ERR  [RM] Exception parameters: 0, d2fa2000, fffffff4, 7ffda000

    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] Exception parameters: 0, d2fa2000, fffffff4, 7ffda000

    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] CallStack:

    00000a20.00002238::2008/02/06-20:10:25.762 ERR  [RM] Frame      Address

    00000a20.00002300::2008/02/06-20:10:25.762 INFO [RM] GenerateMemoryDump: Start memory dump to file C:\WINNT\Cluster\resrcmon.dmp


    If we traverse back 80 minutes, we see the following information in the log.


    00001d38.00000568::2008/02/06-18:50:10.179 WARN [FM] FmpHandleResourceTransition: Resource Name = 0502cab5-3e1f-47d4-b490-e5301be72928 [Johns Hung Resource] old state=2 new state=4

    00001d38.00000568::2008/02/06-18:50:10.179 INFO [FM] FmpPropagateResourceState: resource 0502cab5-3e1f-47d4-b490-e5301be72928 failed event.

    00001d38.00000568::2008/02/06-18:50:10.179 INFO [FM] FmpHandleResourceFailure: taking resource 0502cab5-3e1f-47d4-b490-e5301be72928 and dependents offline

    00001d38.00000568::2008/02/06-18:50:10.179 INFO [FM] TerminateResource: 0502cab5-3e1f-47d4-b490-e5301be72928 depends on dfc6b244-888c-43b4-ad3e-f1d26853c9a4. Terminating first

    00001d38.00000568::2008/02/06-18:50:10.622 INFO [FM] RestartResourceTree, Restart resource 0502cab5-3e1f-47d4-b490-e5301be72928

    00001d38.00000568::2008/02/06-18:50:10.959 INFO [FM] FmpPropagateResourceState: resource 0502cab5-3e1f-47d4-b490-e5301be72928 online event.

    00001d38.00000568::2008/02/06-18:50:10.959 INFO [FM] FmpOnlineWaitingTree, Start resource 0502cab5-3e1f-47d4-b490-e5301be72928

    00001d38.00000568::2008/02/06-18:50:10.959 INFO [FM] FmpRmOnlineResource: bringing resource 0502cab5-3e1f-47d4-b490-e5301be72928 (resid 1198584) online.

    00001d38.00000568::2008/02/06-18:50:10.959 INFO [CP] CppResourceNotify for resource Johns Hung Resource

    00001d38.00000568::2008/02/06-18:50:10.959 INFO [FM] FmpRmOnlineResource: called InterlockedIncrement on gdwQuoBlockingResources for resource 0502cab5-3e1f-47d4-b490-e5301be72928


    And nothing else other than Cluster issuing a reprieve (wait) every three minutes (default timeout value under the resource) until we dumped the resource monitor.  So now we know the resource actually failed and was attempting to come back online.  Because it was not coming back online, it caused the Resource Monitor to crash.  Since this is a third party resource, you would need to engage the programmer or vendor of the resource to see why it had the issue it did.



    Some Additional Information Regarding the Exception Address

    In some cases the steps to determine the exception record may not be necessary. The debugger sometimes provides the exception record information when the dump is first opened. Additionally it could show the stack that we found above by simply entering the .ecxr command...


    0:017> .ecxr

    eax=0227ff3c ebx=00000000 ecx=77e61d43 edx=7c8285ec esi=00000000 edi=00000102

    eip=77e4bee7 esp=0227ff38 ebp=0227ff8c iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246


    77e4bee7 5e              pop     esi


    0:017> kb

      *** Stack trace for last set context - .thread/.cxr resets it

    ChildEBP RetAddr  Args to Child             

    0227ff8c 01007ddd c0000194 00000000 00000000 kernel32!RaiseException+0x53

    0227ffb8 77e64829 000009ec 00000000 00000000 ResrcMon!RmpTimerThread+0xa8

    0227ffec 00000000 01007d35 000a7ca0 00000000 kernel32!BaseThreadStart+0x34


    You could also get to the same information using the above original steps (dds 0227ff38) but stopping at the resrcmon!RmpExceptionFilter (Resource Monitor handles the exception) which has the exception as the first parameter.


    0:017> dds 0227ee10

    0227ee10  00700053 xpsp2res+0xc0053

    0227ee14  00630061

    0227ee18  005f0065

    0227ee1c  00610043

    0227ee20  00610074

    *** pages removed ***

    0227f650  028a0c90

    0227f654  01ece948

    0227f658  01e94518

    0227f65c  0227f8ac                                        <<-- pointer to Exception address stack

    0227f660  0100e638 ResrcMon!GenerateMemoryDump+0x180

    0227f664  ffffffff

    0227f668  00001258

    0227f66c  00000018

    *** pages removed ***

    0227f874  0100c1fe ResrcMon!except_handler3

    0227f878  01005528 ResrcMon!`string'+0xc

    0227f87c  ffffffff

    0227f880  0100d27b ResrcMon!ClRtlLogPrint+0x499

    0227f884  0100e96c ResrcMon!GenerateExceptionReport+0x61

    0227f888  00000001

    *** pages removed ***

    0227f8b8  01003024 ResrcMon!`string'

    0227f8bc  00000000

    0227f8c0  7786d687 netman!__CxxUnhandledExceptionFilter

    0227f8c4  0227f8d8

    0227f8c8  01008b2c ResrcMon!RmpExceptionFilter+0x14       <<-- Frame 4 in kv= 0227f8ac above

    0227f8cc  0227fb78                                        <<-- Frame 3 in kv= 0227f8ac above

    0227f8d0  01003024 ResrcMon!`string'

  • Ntdebugging Blog

    Unlocking some puzzles requires building a better key... board


    Hi, this is Matt from the Windows Performance team.  Sometimes we are presented with problems that defy our usual troubleshooting and require a creative approach.  In a recent case, we needed a way to test the responsiveness of an application as text was typed into its fields.  Initially, we tested the program using a script that used the SendKeys method to time entry time.  Unfortunately, these tests aren’t completely realistic, since the script can be affected by the processor utilization on the system, and the script can’t create hardware interrupts like a keyboard does.  Realizing that only real keyboard input would be a valid test, and that the rate of typing needed to be reproduced exactly for each test, I set about building an automated keyboard. 


    First, I found an old PS/2 style keyboard that hadn’t been used in years and opened it up.  Luckily, it was old enough to use all through-hole components, which made it easier to modify.  The main component I cared about was the keyboard encoder, which was a COP943C.  A search online turned up a datasheet for the keyboard encoder with a sample circuit design that looked very similar to this keyboard.  The document shows there are a couple of steps to determining which pins need to be shorted to generate a particular key. 


    Each key has an ID number that is shown in figure one of the PDF (figure one below).  After finding the proper ID, a table is consulted to determine the row and column pins used to create that key code (figure two below).  Finally, those row and column numbers are translated into physical pins on the encoder using the schematic diagram (figure three below).  For example, the letter ‘a’ is number 31.  The matrix shows 31 is made with the L5 (column 6) pin and C6 (row 3) pin.  The pin out shows this to be physical pins 14 and 19.  When tested, shorting these pins creates an ‘a’.


    See the video:


    Figure 1: Key Codes


    Figure 2: Key Code Matrix


    Figure 3: Encoder Pin Out


    Now that we know how the keyboard circuit works, we need a method to generate key “presses.”  For this, I found a board I assembled a year or two ago using a PCB and components from moderndevice.com.  The board is an Arduino clone that is based on Atmel’s ATmega168 microcontroller.  One of the great things about using an Arduino is their IDE, which allows for C programming with a number of pre-defined functions to make development quick.  Also, the boot loader is already taken care of, which makes the work easier. 


    Wiring the board to the keyboard was straightforward.  Figure 4 shows how to control a relay with an Arduino, and triggering a keyboard is rather similar.  A resistor is placed between a digital out pin of the Arduino and the base pin of a transistor.  The collector then goes to one pin of the keyboard encoder needed to type the letter desired, and the emitter goes to the other pin.



    Figure 4: Arduino-controlled Relay[ii]

    In order to save on solder joints, I decided to chain together the transistors, which affected the key selection.  Additionally, because I wanted to leave the encoder in the original circuit and some of the pins were blocked by other components (resistors, capacitors), specific pins were selected.  Figure 5 shows the layout of the transistors.  These were soldered to a prototyping board with hook up wire to connect back to a breadboard with resistors and the Arduino and hook up wire soldered directly to the pins of the keyboard encoder.



    Figure 5: Transistor Layout


    These pins selected allowed characters a, s, z, space, and enter to be typed.  All that remained was to write some software to trigger the transistors.  The code first sets the digital pins to output and logic low, turns on a LED to show it is working, then waits 3 minutes to allow time for the PC to boot and application in question to be launched.  The LED then goes out for a five-second warning, and then the loop sequence begins.  The loop turns on the LED, types “as z” followed by enter, then turns off the LED and sleeps for 2.5 seconds before starting again.


    // Sample code to drive keyboard encoder


    // Matt Burrough


    // September, 2008



    int ledPin = 13;             // Use digital pin 13 for a status LED


    int sPin = 3;                // Connect pin 3 to the transistor connected to the s leads


    int aPin = 4;                // Pin 4 is for a


    int zPin = 5;                // Pin 5 is z


    int enterPin = 6;            // Pin 6 is enter


    int spacePin = 7;            // Pin 7 is space


    int holdKey = 30;            // Milliseconds to "hold" each key down


    int betweenKeys = 50;        // Milliseconds to wait between key presses



    void setup() {               // Initial setup code (runs at power-on)


      setupPin(ledPin);          // Set up each pin with function below












      digitalWrite(ledPin, HIGH);  // Turn on the LED to show the board is on


      delay(180000);               // Wait 3 minutes to allow time for PC to boot


      digitalWrite(ledPin, LOW);   // Turn off the LED


      delay(5000);                 // Wait 5 seconds





    void loop() {


      digitalWrite(ledPin, HIGH);  // Turn the LED on




      typeKey(aPin);               // Type keys












      digitalWrite(ledPin, LOW);    // Turn the LED off


      delay(2500);                  //Pause 2.5 seconds





    void setupPin(int pin) {        // Used to set up pins...


      pinMode(pin, OUTPUT);         // Set the digital pin as output


      digitalWrite(pin, LOW);       // Turn off the pin





    void typeKey(int pin) {         // Type a key...


      digitalWrite(pin, HIGH);      // "Press down" on a key


      delay(holdKey);               // Hold down the key


      digitalWrite(pin, LOW);       // "Release" the key


      delay(betweenKeys);           // Pause between keys





    Figure 6: Code Sample


    That’s how I made an automated keyboard.  I hope that you’ve found this post interesting; I’ll leave you with a photo of the finished product. 




  • Ntdebugging Blog

    NDIS Case Study 1 - NDIS Packet Double Completion


    Hi, this is Anurag again. Here is a case study of an NDIS driver causing a problem due to double completion of a send packet.


    A protocol driver allocates a NDIS packet and gives it to the miniport driver to be sent on the wire. A miniport driver is supposed to send or complete the packet, but miniport driver is supposed to complete the packet only once. The moment it tries to complete a packet twice (generally due to a bug in the miniport driver), the machine will bugcheck with a stack like the one shown below. The crash will happen either in a protocol driver or in the NDIS driver. The bugcheck would be the result of a trap on accessing an invalid memory in the packet.


    Each time a packet gets completed the miniport stamps it as completed by putting a string “COM” just before the packet (somewhere in the packet stack).  If a miniport driver tries to complete this packet again, the system would crash. The miniport driver which tried to complete the completed packet again is at fault and should be fixed.


    When reviewing the bugcheck, !analyze –v will give you the stack.


    1: kd> !analyze -v


    *                                                                             *

    *                        Bugcheck Analysis                                    *

    *                                                                             *




    An attempt was made to access a pageable (or completely invalid) address at an

    interrupt request level (IRQL) that is too high.  This is usually

    caused by drivers using improper addresses.

    If kernel debugger is available get stack backtrace.


    Arg1: 00000008, memory referenced

    Arg2: d0000002, IRQL

    Arg3: 00000000, value 0 = read operation, 1 = write operation

    Arg4: f78521e0, address which referenced memory


    Debugging Details:



    READ_ADDRESS:  00000008






    f78521e0 8b7808          mov     edi,dword ptr [eax+8]






    PROCESS_NAME:  System


    TRAP_FRAME:  f78b2e18 -- (.trap 0xfffffffff78b2e18)

    ErrCode = 00000000

    eax=00000000 ebx=902bbab0 ecx=00000002 edx=fffffffe esi=8f3aabf8 edi=901bd440

    eip=f78521e0 esp=f78b2e8c ebp=f78b2ea0 iopl=0         nv up ei ng nz na po nc

    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010282


    f78521e0 8b7808          mov     edi,dword ptr [eax+8] ds:0023:00000008=????????

    Resetting default scope


    LAST_CONTROL_TRANSFER:  from f78521e0 to 8088c963



    f78b2e18 f78521e0 badb0d00 fffffffe 00000000 nt!KiTrap0E+0x2a7

    f78b2ea0 b9f47b5a 902bbab0 8f3aabf8 00000000 NDIS!ndisMSendCompleteX+0x71

    f78b2ec0 f78521f9 8f411000 013aabf8 00000000 X_Miniport_Network_Teamming_Driver +0x7b5a

    f78b2ee4 ba04a0c8 9029b5e8 8f3aabf8 00000000 NDIS!ndisMSendCompleteX+0x8a

    f78b2f00 ba052744 904d8eb8 8f3aabf8 00000000 Y_Miniport_NIC_Driver +0x20c8

    f78b2f3c ba050339 8f8c5000 00000000 f78b2f68 Y_Miniport_NIC_Driver +0xa744

    f78b2f70 ba05065f 008c5000 8f8c5844 f78b2f8c Y_Miniport_NIC_Driver +0x8339

    f78b2f80 ba0493f1 8f8c5000 f78b2f9c f783a0e8 Y_Miniport_NIC_Driver +0x865f

    f78b2f8c f783a0e8 904d8eb8 8f8c58bc f78b2ff4 Y_Miniport_NIC_Driver +0x13f1

    f78b2f9c 808320f0 8f8c58bc 8f8c5844 00000000 NDIS!ndisMDpcEx+0x1f

    f78b2ff4 8088db27 b88fe454 00000000 00000000 nt!KiRetireDpcList+0xca

    f78b2ff8 b88fe454 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x37

    8088db27 00000000 0000000a 0083850f bb830000 0xb88fe454


    1: kd> .trap 0xfffffffff78b2e18

    ErrCode = 00000000

    eax=00000000 ebx=902bbab0 ecx=00000002 edx=fffffffe esi=8f3aabf8 edi=901bd440

    eip=f78521e0 esp=f78b2e8c ebp=f78b2ea0 iopl=0         nv up ei ng nz na po nc

    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010282


    f78521e0 8b7808          mov     edi,dword ptr [eax+8] ds:0023:00000008=????????


    Looking at the stack, it appears to be a NULL memory address being accessed that is causing problem.  This could lead us to believe that this is a memory corruption problem. This is where we go off-track on the investigation.


    The current NDIS packet status should be checked if you bugcheck on the NDIS stack and in NDIS!ndisMSendCompleteX.


    Some basic debugging will display the NDIS_PACKET on the stack and in the ESI register (8f3aabf8).


    Looking at the memory area before the packet shows the string “COM” which is the indication that it was marked as completed.


    Here is how it looks.

    For reference this is how a non-completed packet looks like

    So for this issue the X_Miniport_Network_Teamming_Driver was at fault which tried to complete a completed packet. Fortunately the vendor for the driver was aware of the problem and was able to give the customer a fix.


    For a 64 bit dump we do a “dc <Packet Address>-0x68” to look at the memory area before the packet. Below is an example.


    BugCheck D1, {0, 2, 0, fffffade57895375}

    An attempt was made to access a pageable (or completely invalid) address at an
    interrupt request level (IRQL) that is too high. This is usually
    caused by drivers using improper addresses.
    If kernel debugger is available get stack backtrace.
    Arg1: 0000000000000000, memory referenced
    Arg2: 0000000000000002, IRQL
    Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
    Arg4: fffffade57895375, address which referenced memory


    0: kd> r

    Last set context:

    rax=0000000000000000 rbx=fffffade6f203000 rcx=0000000000000000

    rdx=fffffade6f564740 rsi=fffffade6faa3548 rdi=0000000000000000

    rip=fffffade57895375 rsp=fffff8000012b990 rbp=fffffade6f5647d0

     r8=0000000000000000  r9=5bcdd13016000000 r10=5bcdd13016110009

    r11=0000000000000000 r12=0000000000000000 r13=0000000000000000

    r14=0000000000000000 r15=0000000000000000

    iopl=0         nv up ei pl nz na pe nc

    cs=0010  ss=0018  ds=0002  es=0000  fs=0000  gs=0000             efl=00010202


    fffffade`57895375 488b1b          mov     rbx,qword ptr [rbx] ds:0002:fffffade`6f203000=fffffade6f52e010


    0: kd> kvn

      *** Stack trace for last set context - .thread/.cxr resets it

     # Child-SP          RetAddr           : Args to Child                                                           : Call Site

    00 fffff800`0012b990 fffffade`578975a7 : 00000000`00000000 00000000`00000000 fffffade`6e8980b0 fffffade`6f564740 : tcpip!TCPSendComplete+0x124

    01 fffff800`0012b9e0 fffffade`5789a89e : fffffade`6f1f1700 fffffade`00000000 00000000`00026200 fffffade`6e8980b0 : tcpip!IPSendComplete+0x477

    02 fffff800`0012bab0 fffffade`5ae37237 : fffff800`0083f480 fffffade`6f887000 fffffade`6faab228 fffffade`6e8980b0 : tcpip!ARPSendComplete+0x14a

    03 fffff800`0012baf0 fffffade`58443c15 : fffffade`6f887000 fffffade`6e8980b0 fffffade`6f887038 fffffade`6c31e3b0 : NDIS!ndisMSendCompleteX+0xda

    04 fffff800`0012bb40 fffffade`5ae37237 : fffffade`6faa3010 00000000`00000002 fffffade`6fc16228 fffffade`6c31e3b0 : NDIS_IM_TEAM_DRIVER!po_SendDone+0x75

    05 fffff800`0012bb80 fffffade`596f9a4c : fffffade`6faa3010 fffff800`0012bc50 00000000`00000000 00000000`00000000 : NDIS!ndisMSendCompleteX+0xda

    06 fffff800`0012bbd0 fffffade`5af5e9cd : fffff800`0012bcb0 fffffade`6f52e010 fffffade`6f203000 00000000`00000006 : NDIS_MP_DRIVER!l2nd_indicate_tx+0xfc

    07 fffff800`0012bc30 fffffade`5af64889 : fffffade`6f52e010 fffffade`6f5315c8 00000000`00000000 00000000`00000000 : NDIS_MP_DRIVER!um_bdrv_indicate_tx_done+0x121

    08 fffff800`0012bc90 fffffade`5af64bab : fffffade`6f52e010 00000000`00000001 00000000`00000001 fffff800`011b2080 : NDIS_MP_DRIVER!service_tx_intr+0xa9

    09 fffff800`0012bce0 fffffade`5af64d63 : fffffade`6f52e010 00000000`0005ffd4 00000000`00000000 0000023e`f26bf2bf : NDIS_MP_DRIVER!service_intr_mp+0xb3

    0a fffff800`0012bd10 fffff800`010285a1 : 00000000`00000000 fffffade`5af64cb4 fffff800`0012bd68 00000000`00000018 : NDIS_MP_DRIVER!um_bdrv_dpc+0xaf

    0b fffff800`0012bd40 fffff800`01067c10 : fffff800`011b0180 fffff800`011b0180 00000000`0005ffd4 fffff800`011b4500 : nt!KiRetireDpcList+0x150

    0c fffff800`0012bdd0 fffff800`014141d1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x50

    0d fffff800`0012be00 00000000`fffff800 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemStartup+0x1bf


    Note that in this bugcheck we crash in the transport driver in the above stack. But the problem is that the packet (fffffade`6e8980b0) came from the NDIS driver NDIS_IM_TEAM_DRIVER.


    0: kd> dc fffffade`6e8980b0 -0x68

    fffffade`6e898048  00000000 00000000 4d4f4336 00000000  ........6COM....

    fffffade`6e898058  00000000 00000000 00000000 00000000  ................

    fffffade`6e898068  00000000 00000000 00000000 00000000  ................

    fffffade`6e898078  00000000 00000000 00000000 00000000  ................

    fffffade`6e898088  00000000 00000000 00000000 00000000  ................

    fffffade`6e898098  00000000 00000000 00000000 00000000  ................

    fffffade`6e8980a8  ffffffff ffffffff 6e898430 fffffade  ........0..n....

    fffffade`6e8980b8  6f564740 fffffade 6d9ee460 fffffade  @GVo....`..m....



    After reading this, I suspect you would have at least these two questions for me:


    Q1: How did I find out that “COM” is use as an identifier for a completed packet?

    A1: Well, mainly code review. This is just intended as a debug trick to verify that the packet is already completed.


    Q2: Why did I use 0x38 (for 32 bit) or 0x68 (for 64 bit) as an offset to check the memory contents before the packet address?

    A2: Again code review shows the field to be updated with string ‘COM” can seen going 0x38 bytes back from the packet start address.


    NDIS double packet completion is a common issue that would crash the system. The above trick can certainly be used to isolate the culprit NDIS driver.


  • Ntdebugging Blog

    Some of our favorite debugging-related links



    Today we’re posting links to some of our favorite debugging-related content on the webPost your own favorites as a comment to share them with everyone!



    Reverse Engineering and Debugging Blogs




    Mark Russinovich's Blog

    Steve’s Techspot

    John Robbins’ Blog


    Windbg by Volker

    CodeProject Debugging Tips


    Jigar Mehta's Blog




    Debug Reference

    Debugging Commands (MSDN)

    Debugger Reference (MSDN)

    Common WinDbg Commands

    Advanced Debugging Techniques (MSDN)



    Other stuff

    Debunking Common Windows Performance Tweaking Myths



    Some disclaimers: Links are no particular order.  Many of these links go to non-Microsoft sites, and our linking to them does not mean we endorse all the content on the sites.


  • Ntdebugging Blog

    NDIS - Part 1


    Hi, my name Anurag Sarin, I am an escalation engineer in the Platforms Global Escalation Team.  I would like to give some insight on NDIS.


    NDIS Introduction

    The Network Driver Interface Specification (NDIS) library abstracts the network hardware from network drivers. NDIS also specifies a standard interface between layered network drivers, thereby abstracting lower-level drivers that manage hardware from upper-level drivers, such as network transports. NDIS also maintains state information and parameters for network drivers, including pointers to functions, handles, and parameter blocks for linkage, and other system values.

    Types of network drivers

    • Miniport drivers
    • Intermediate drivers
    • Filter drivers
    • Protocol drivers

    ndiskd is a good extension for debugging NDIS drivers .The document Debugging NDIS Drivers  has more information  about ndiskd.


    To get a list of NDIS protocols drivers in the system, the protocols option in ndiskd can be used.

    This also gives a list of all NDIS open blocks on each protocol driver.  Protocol open block is described in the later section.


    A Protocol Driver is represented by a NDIS Protocol Block, which is shown as Protocol below:


    1: kd> !ndiskd.protocols

     Protocol 8ef57590: NDISUIO

        Open 8ef61860 - Miniport: 902bbab0 X Network Team #1


     Protocol 8f3aea50: TCPIP_WANARP

        Open 8f3ae508 - Miniport: 90287ae8 WAN Miniport (IP)


     Protocol 8f220008: TCPIP

        Open 8f21d210 - Miniport: 902bbab0 X Network Team #1


     Protocol 9029bbf8: NDPROXY

        Open 901618e0 - Miniport: 9019f130 Direct Parallel

        Open 90161c80 - Miniport: 9019f130 Direct Parallel

        Open 9018bcd0 - Miniport: 901ba130 WAN Miniport (L2TP)

        Open 90171490 - Miniport: 901ba130 WAN Miniport (L2TP)


     Protocol 901e3008: RASPPPOE


     Protocol 901bb008: NDISWAN

        Open 90216b30 - Miniport: 9019f130 Direct Parallel

        Open 9047e518 - Miniport: 901fdab0 WAN Miniport (PPTP)

        Open 90198c20 - Miniport: 90276ab0 WAN Miniport (PPPOE)

        Open 901989e0 - Miniport: 901ba130 WAN Miniport (L2TP)


     Protocol 902de6e0: Y_TEAM

        Open 9028ef10 - Miniport: 9029b5e8 P Gigabit Server Adapter #2

        Open 90198b10 - Miniport: 9029eab0 Q Multifunction Gigabit Server Adapter #2


    NDIS filter drivers are represented by Filter Driver Block (s) shown below.


    0: kd> !ndiskd.filters

    NDIS Driver verifier level: 0

    NDIS Failed allocations   : 0


    Filter Driver Block: 97412e58

      Filter: 97414c10 Z Network Connection-Native WiFi Filter Driver-0000

        Miniport 85d160e8   Z Network Connection


    Filter Driver Block: 8797bdb0

      Filter: 97446c10 Z Network Connection - H Miniport-QoS Packet Scheduler-0000

        Miniport 8610b0e8   Z Network Connection - H Miniport

      Filter: 87b1b730 Y Network Adapter - H Miniport-QoS Packet Scheduler-0000

        Miniport 8611c0e8   Y Network Adapter - H Miniport

      Filter: 879c3008 G Network Connection - H Miniport-QoS Packet Scheduler-0000

        Miniport 861150e8   G Network Connection - H Miniport

      Filter: 879c0a50 WAN Miniport (IP) - H Miniport-QoS Packet Scheduler-0000

        Miniport 861240e8   WAN Miniport (IP) - H Miniport

      Filter: 879bb3f8 WAN Miniport (IPv6) - H Miniport-QoS Packet Scheduler-0000

        Miniport 861250e8   WAN Miniport (IPv6) - H Miniport

      Filter: 87981870 WAN Miniport (Network Monitor) - H Miniport-QoS Packet Scheduler-0000

        Miniport 861260e8   WAN Miniport (Network Monitor) - H Miniport

      Filter: 8797f518 Nortel IPSECSHM Adapter - H Miniport-QoS Packet Scheduler-0000

        Miniport 861170e8   Nortel IPSECSHM Adapter - H Miniport


    The “miniports” option lists all NDIS miniport drivers represented by a Miniport Driver Block (s).


    kd> !ndiskd.miniports

    NDIS Driver verifier level: 0

    NDIS Failed allocations   : 0

    Miniport Driver Block: 885915b8, Version 0.0

      Miniport: 8863a0e8, NetLuidIndex: 1, IfIndex: 7, RAS Async Adapter

    Miniport Driver Block: 88018010, Version 0.0

      Miniport: 8828d488, NetLuidIndex: 1, IfIndex: 3, WAN Miniport (PPTP)

    Miniport Driver Block: 87e535a8, Version 0.0

      Miniport: 88150200, NetLuidIndex: 0, IfIndex: 4, WAN Miniport (PPPOE)

    Miniport Driver Block: 87f63510, Version 0.0

      Miniport: 880ac4b8, NetLuidIndex: 0, IfIndex: 5, WAN Miniport (IPv6)

      Miniport: 880844c0, NetLuidIndex: 3, IfIndex: 6, WAN Miniport (IP)

    Miniport Driver Block: 87f2ccb8, Version 0.0

      Miniport: 88091488, NetLuidIndex: 0, IfIndex: 2, WAN Miniport (L2TP)

    Miniport Driver Block: 8809de60, Version 10.1

      Miniport: 883bf0e8, NetLuidIndex: 10, IfIndex: 15, MY PCI Fast Ethernet Adapter (Emulated) #4

      Miniport: 883be0e8, NetLuidIndex: 5, IfIndex: 11, MY PCI Fast Ethernet Adapter (Emulated) #3

      Miniport: 883bd0e8, NetLuidIndex: 6, IfIndex: 9, MY PCI Fast Ethernet Adapter (Emulated) #2

      Miniport: 883bc0e8, NetLuidIndex: 4, IfIndex: 8, MY PCI Fast Ethernet Adapter (Emulated)

    Miniport Driver Block: 87f77df0, Version 1.0

      Miniport: 87f9c488, NetLuidIndex: 6, IfIndex: 16, isatap.{584AF5A9-63C2-44C5-970D-DB85057F2931}

      Miniport: 87f75488, NetLuidIndex: 4, IfIndex: 14, isatap.fareast.corp.microsoft.com

      Miniport: 87fc6488, NetLuidIndex: 3, IfIndex: 10, isatap.{CCDB4297-B958-4C2A-95A5-29150BD0A371}


    The interfaces option lists all Network interfaces


    kd> !ndiskd.interfaces

    Interface block 87fb12a8 



      IfIndex: 20, IfType: 6

    Inerface Guid: 5d0bd81a-47c7-11dc-a9d2-0003ff2b6bfa

    Interface block 88101ab0 


      IfIndex: 21, IfType: 6

    Inerface Guid: ed1c50d5-ff1f-11db-9b85-0003ff7133d2

    Interface block 87f82ab0 


      IfIndex: 15, IfType: 6

    Inerface Guid: f442c036-8bf5-43a6-91fd-6792ef752100

    Interface block 881cd5d8 


      IfIndex: 22, IfType: 6

    Inerface Guid: ed1c50d4-ff1f-11db-9b85-9459bf825974

    Interface block 87f313f8 




     Interface guids correspond to each network interface. Some interfaces have their information in the in the registry .For Example on my machine Interface Guid: f442c036-8bf5-43a6-91fd-6792ef752100 corresponds to registry:-




    NDIS Driver Stack


    Basic Stack Configuration


    The MSDN web page  NDIS Driver Stack  has  basic documentation on the NDIS stack


    For Miniport details, the miniport option can be used.


    kd> !ndiskd.miniport 883bc0e8


     Miniport 883bc0e8 : MY PCI Fast Ethernet Adapter (Emulated), v5.0


        AdapterContext : 8809e000

        Flags          : 2c412008



        PnPFlags       : 80210000

                         RECEIVED_START, HARDWARE_DEVICE,

        MiniportState        : STATE_RUNNING

        IfIndex                  : 8

        Ndis5MiniportInNdis6Mode : 1

        InternalResetCount    : 0000

        MiniportResetCount    : 0000

        References            : 5

        UserModeOpenReferences: 0

        PnPDeviceState        : PNP_DEVICE_STARTED

        CurrentDevicePowerState : PowerDeviceD0

        Bus PM capabilities

           DeviceD1:            0

           DeviceD2:            0

           WakeFromD0:          0

           WakeFromD1:          0

           WakeFromD2:          0

           WakeFromD3:          0


           SystemState          DeviceState

           PowerSystemUnspecified     PowerDeviceUnspecified

           S0                   D0

           S1                   PowerDeviceUnspecified

           S2                   PowerDeviceUnspecified

           S3                   PowerDeviceUnspecified

           S4                   D3

           S5                   D3

           SystemWake: PowerSystemUnspecified

           DeviceWake: PowerDeviceUnspecified

        Current PnP and PM Settings:          : 00000030


        Translated Allocated Resources:

            IO Port: 0000e480, Length: 80

            Memory: febfc000, Length: 1000

            Interrupt Level: 10, Vector: 3b

        MediaType      : 802.3

        DeviceObject   : 883bc030, PhysDO : 87ca1030  Next DO: 87ca1030

        MapRegisters   : 00000000

        FirstPendingPkt: 00000000

        DriverVerifyFlags  : 00000000

        Miniport Interrupt : 8809e008

        Miniport version 5.0

        Miniport Filter List:

     Filter  88301c28: FilterDriver  88129300, FilterModuleContext 880d7230  MY PCI Fast Ethernet Adapter (Emulated)-QoS Packet Scheduler-0000

        Miniport Open Block Queue:

          88263278: Protocol 88300830 = RSPNDR, ProtocolBindingContext 87fb7820, v6.0

          882446d0: Protocol 8835a300 = LLTDIO, ProtocolBindingContext 88323310, v6.0

          880d2c58: Protocol 8831e2e0 = TCPIP, ProtocolBindingContext 88329008, v6.0


    Current PnP capacities and Power Management Settings are shown by flags and can be one of these values in the Current PnP and PM Settings’ section.


    NOT_STOPPABLE       :   The device is not stoppable i.e. ISA

    NOT_REMOVEABLE  :    The device cannot be safely removed

    NOT_SUSPENDABLE :    The device cannot be safely suspended

    DISABLE_PM              :    Disable all Power Management features

    DISABLE_WAKE_UP  :    Disable device waking up the system .This is evident when the user disables Wake-On-LAN (WOL)  feature on the miniport adaptor

    DISABLE_WAKE_ON_RECONNECT: Disable device waking up the -system- due to a cable re-connect


    Above , the miniport block 883bc0e8 represents  MY  PCI Fast Ethernet Adapter, i.e. the NIC driver on my machine. The NIC driver has a binding with filter driver MY PCI Fast Ethernet Adapter (Emulated)-QoS Packet Scheduler and protocols drivers RSPNDR, LLTDIO and TCPIP.


                       The stack with the debug output above would look somewhat like this.

    To see what all miniport drivers the Protocol driver has bound to - the protocol option can be used.


    kd> !protocol 8831e2e0

     Protocol 8831e2e0 : TCPIP

     RootDeviceName is \DEVICE\{B4982B71-0255-4D04-A585-4C339162A25D}

     v6.0       RefCount 5


        Open 880d2c58 - Miniport: 883bc0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated)

        Open 881bbc58 - Miniport: 883bd0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated) #2

        Open 881bcc58 - Miniport: 883be0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated) #3

        Open 881c0c58 - Miniport: 883bf0e8 Intel 21140-Based PCI Fast Ethernet Adapter (Emulated) #4


     BindAdapterHandlerEx               8e93da28, UnbindAdapterHandlerEx       8e9d7a87

     PnPEventHandler                    8e933b83, UnloadHandler                00000000

     OpenAdapterCompleteEx              8e9d74ff, CloseAdapterCompleteEx       8e9d7722

     SendNetBufferListsCompleteHandler  8e997067, ReceiveNetBufferListsHandler  8e98ff5f

     StatusComplete                     00000000, StatusHandler                8e942cad

     AssociatedMiniDriver 00000000


          Flags          : 00000000


    This also shows the various handler routines of the Protocol TCPIP Driver routines.

    Un-assembling the routines would verify them further.


    kd> u 8e93da28


    8e93da28 8bff            mov     edi,edi

    8e93da2a 55              push    ebp

    8e93da2b 8bec            mov     ebp,esp

    8e93da2d 83ec1c          sub     esp,1Ch

    8e93da30 53              push    ebx

    8e93da31 56              push    esi

    8e93da32 57              push    edi

    8e93da33 6a06            push    6


    kd> u 8e933b83


    8e933b83 8bff            mov     edi,edi

    8e933b85 55              push    ebp

    8e933b86 8bec            mov     ebp,esp

    8e933b88 837d0800        cmp     dword ptr [ebp+8],0

    8e933b8c 7406            je      tcpip!Fl48PnpEvent+0x11 (8e933b94)

    8e933b8e 5d              pop     ebp

    8e933b8f e96c030000      jmp     tcpip!FlPnpEvent (8e933f00)

    8e933b94 8b450c          mov     eax,dword ptr [ebp+0Ch]


    NDIS Open Block is a block that represents the binding between a Miniport Driver and a Protocol Driver. So there is one NDIS Open Block per binding between a protocol and a miniport.

    kd> !ndiskd.opens

      Open 885283c0

        Miniport: 8863a0e8 - RAS Async Adapter

        Protocol: 8803be48 -


      Open 88263278

        Miniport: 883bc0e8 - MY PCI Fast Ethernet Adapter (Emulated)

        Protocol: 88300830 - RSPNDR


      Open 88263628

        Miniport: 883bd0e8 - MY PCI Fast Ethernet Adapter (Emulated) #2

        Protocol: 88300830 - RSPNDR


      Open 88124008

        Miniport: 883be0e8 - MY PCI Fast Ethernet Adapter (Emulated) #3

        Protocol: 88300830 - RSPNDR


      Open 883276a0

        Miniport: 883bf0e8 - MY PCI Fast Ethernet Adapter (Emulated) #4

        Protocol: 88300830 - RSPNDR


      Open 882446d0

        Miniport: 883bc0e8 - MY PCI Fast Ethernet Adapter (Emulated)

        Protocol: 8835a300 - LLTDIO


      Open 882fa398

        Miniport: 883bd0e8 - MY PCI Fast Ethernet Adapter (Emulated) #2

        Protocol: 8835a300 - LLTDIO


      Open 882d5550

        Miniport: 883be0e8 - MY PCI Fast Ethernet Adapter (Emulated) #3

        Protocol: 8835a300 - LLTDIO


      Open 882cf470

        Miniport: 883bf0e8 - MY PCI Fast Ethernet Adapter (Emulated) #4

        Protocol: 8835a300 - LLTDIO


      Open 88219968

        Miniport: 880ac4b8 - WAN Miniport (IPv6)

        Protocol: 880f04a0 - WANARPV6


      Open 8816e008

        Miniport: 880844c0 - WAN Miniport (IP)

        Protocol: 882e5008 - WANARP


      Open 881eec58

        Miniport: 883bd0e8 - MY PCI Fast Ethernet Adapter (Emulated) #2

        Protocol: 88383138 - TCPIP6


      Open 881b6c58

        Miniport: 883be0e8 - MY PCI Fast Ethernet Adapter (Emulated) #3

        Protocol: 88383138 - TCPIP6


      Open 880d2c58

        Miniport: 883bc0e8 - MY PCI Fast Ethernet Adapter (Emulated)

        Protocol: 8831e2e0 - TCPIP


      Open 881bbc58

        Miniport: 883bd0e8 - MY PCI Fast Ethernet Adapter (Emulated) #2

        Protocol: 8831e2e0 - TCPIP


      Open 881c2c58

        Miniport: 883bf0e8 - MY PCI Fast Ethernet Adapter (Emulated) #4

        Protocol: 88383138 - TCPIP6


      Open 881bcc58

        Miniport: 883be0e8 - MY PCI Fast Ethernet Adapter (Emulated) #3

        Protocol: 8831e2e0 - TCPIP


      Open 881c0c58

        Miniport: 883bf0e8 - MY PCI Fast Ethernet Adapter (Emulated) #4

        Protocol: 8831e2e0 - TCPIP


      Open 8811a6a0

        Miniport: 87fbb0e8 - isatap.{B4982B71-0255-4D04-A585-4C339162A25D}

        Protocol: 882d8298 - TCPIP6TUNNEL


      Open 8811a008

        Miniport: 87fc6488 - isatap.{CCDB4297-B958-4C2A-95A5-29150BD0A371}

        Protocol: 882d8298 - TCPIP6TUNNEL


      Open 881f8870

        Miniport: 87f75488 - isatap.fareast.corp.microsoft.com

        Protocol: 882d8298 - TCPIP6TUNNEL


      Open 882a4850

        Miniport: 87f9c488 - isatap.{584AF5A9-63C2-44C5-970D-DB85057F2931}

        Protocol: 882d8298 - TCPIP6TUNNEL


       Open 881ad960

        Miniport: 8828d488 - WAN Miniport (PPTP)

        Protocol: 8803be48 -


       Open 88018c58

        Miniport: 88150200 - WAN Miniport (PPPOE)

        Protocol: 8803be48 -


       Open 8831f818

        Miniport: 88091488 - WAN Miniport (L2TP)

        Protocol: 87f1e100 -


       Open 8831fc10

        Miniport: 88091488 - WAN Miniport (L2TP)

        Protocol: 87f1e100 -


       Open 881425a8

        Miniport: 88091488 - WAN Miniport (L2TP)

        Protocol: 8803be48 -


    Use mopen option to see the details of the NDIS Open Block.


    kd> !mopen 881bbc58

     Miniport Open Block 881bbc58

        Protocol 8831e2e0 = TCPIP, ProtocolContext 88348008, v6.0

        Miniport 883bd0e8 = MY PCI Fast Ethernet Adapter (Emulated) #2, v5.0


        MiniportAdapterContext: 880a1000

        Flags                 : 01000000


        References            : 1


    The ‘References’ section above shows the number of outstanding Input Output Requests. This can be use full to investigate how many requests by a protocol driver are passed to the next lower driver which are currently outstanding.

    Network Data

    Network data consists of packets of data that are sent or received over the network. NDIS provides data structures to describe and organize such data. The primary NDIS 6.0 network data structures include the following:

    ·         NET_BUFFER structures

    ·         NET_BUFFER LIST structures

    ·         NET_BUFFER_LIST_CONTEXT structures

    For NDIS 5.x we have NDIS PACKETS in place of NET_BUFFER structure.

    NDIS_PACKET Structure

    NDIS packets (represented by a NDIS_PACKET  structure) are allocated by a protocol driver, filled with data, and passed to the next lower NDIS driver so that the data can be sent on the network. Some lowest level NIC drivers allocate packets to hold received data and pass the packet up to interested higher-layer drivers. Sometimes, a protocol driver allocates a packet and passes it to a NIC driver with a request that the NIC driver copy received data into the provided packet. NDIS provides functions for allocating and manipulating the substructures that make up a packet. The following figure illustrates a structure of a packet.

    Each NDIS Packet is basically a Packet Descriptor. Each Packet Descriptor has a series of Buffer Descriptors.

    A packet is composed of the following:

    • A packet descriptor that contains private areas for the miniport driver and a protocol driver, a set of flags associated with the packet and whose meaning is defined by a cooperating miniport driver(s) and protocol driver(s), the number of physical pages that contain the packet, the total length of the packet, and a pointer to the first buffer descriptor that maps the first buffer in the packet.
    • A set of buffer descriptors. A buffer descriptor describes the starting virtual address of each buffer, the buffer's byte offset into the page pointed to by the virtual address, the total number of bytes in the buffer and a pointer to the next buffer descriptor, if any.
    • The virtual range, possibly spanning more than one page that makes up the buffer described by the buffer descriptor. These virtual pages map to physical memory.

    The pkt option in ndiskd helps us to see the contents of the NDIS Packet. It has various verbose options:

    Usage: pkt <pointer to packet> <verbosity>

    <verbosity>  can be between 1 to 5.

    1-Packet Private

    2-Packet Extension

    3-Ndis Reference

     4-Buffer List

    5- Data in Packet List


    1: kd> !ndiskd.pkt 0x8f3aabf8

    NDIS_PACKET at 8f3aabf8



      PhysicalCount       00000001  Total Length        00000036

      Head                8f3aa630  Tail                8a667d30

      Pool                90331d20  Count               00000001

      Flags               00000002  ValidCounts         01

      NdisPacketFlags     00000000  NdisPacketOobOffset 006c


          Private.Flags          : 00000002

          Private.NdisPacketFlags: 0


    Above output indicates a typical Packet Descriptor.


    Below is a description of the fields in above output.


    PhysicalCount :   Number of physical pages in packet.

    TotalLength       :   Total amount of data in the packet in bytes.

    Head                       :   First buffer in the chain. If Head is NULL the chain is empty.

    Tail                       :   Last buffer in the chain.

    Count                     :   The number of Buffers in the chain.

    ValidCounts       :   Represent a Boolean value on validity of the Counts.

    Pool                       :   NDIS Packet Pool address so we know where to free it back to.

    To demonstrate what an NDIS packet looks like, a breakpoint was placed on routine NdisMSendComplete. The definition of NdisMSendComplete states that ‘PNDIS_PACKET  Packet’ is the second parameter. So the address of NDIS Packet can be found  at EBP+0xC position on the stack.

     kd> kvn

     # ChildEBP RetAddr  Args to Child             

    00 818f1c44 8b8848c3 883be0e8 88727538 00000000 ndis!NdisMSendComplete+0x10 (FPO: [Non-Fpo])

    01 818f1c7c 8b884b2f 8817d280 81827c97 00000000 dc21x4vm!ProcessTransmitDescRing+0x363 (FPO: [Non-Fpo])

    02 818f1c9c 81718cb1 00010005 88166008 883be0e8 dc21x4vm!DC21X4HandleInterrupt+0xfb (FPO: [Non-Fpo])

    03 818f1cc8 81682d1f 8816601c 88166008 00000000 ndis!ndisMDpc+0x16b (FPO: [Non-Fpo])

    04 818f1ce8 818a93ae 8816601c 88166008 00000000 ndis!ndis5InterruptDpc+0x9c (FPO: [Non-Fpo])

    05 818f1d50 818912ae 00000000 0000000e 00000000 nt!KiRetireDpcList+0x147

    06 818f1d54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x46 (FPO: [0,0,0])


    You can use verbosity 4 for looking at the Buffer Descriptors and a pointer to the next buffer descriptor, if any as shown below.

    ndis.h available with Windows Driver Kit (WDK) or Windows Driver Device Kit (DDK) has definition for NdisPacketFlags

    The verbosity 5 as most interesting -  the NDIS packet data contents are displayed.



    kd> !pkt poi(@ebp+c) 5

    NDIS_PACKET at 88727538

           MDL = 886f3ca0

                  StartVa ffffffff886f3000, ByteCount 0x36, ByteOffset 0xd06, NB MdlOffset 0x0

           886f3d06:  00 13 5f 0b ef ca 00 15 5d 50 f3 34 08 00 45 00

           886f3d16:  03 23 04 44 40 00 80 06 ce 2e 41 34 50 de 41 34

           886f3d26:  52 1c c0 51 00 50 6d f9 e6 c2 9a 14 d4 99 50 18

           886f3d36:  40 29 0e e5 00 00

                  MDL = 87f3aad0

                  StartVa ffffffff886f4000, ByteCount 0x2fb, ByteOffset 0x40, NB MdlOffset 0x0

           886f4040:  47 45 54 20 68 74 74 70 3a 2f 2f 77 77 77 2e 6d

           886f4050:  73 6e 2e 63 6f 6d 2f 61 6a 61 78 2f 48 2e 61 73

           886f4060:  70 78 20 48 54 54 50 2f 31 2e 31 0d 0a 41 63 63

           886f4070:  65 70 74 3a 20 2a 2f 2a 0d 0a 41 63 63 65 70 74

           886f4080:  2d 4c 61 6e 67 75 61 67 65 3a 20 65 6e 2d 75 73

           886f4090:  0d 0a 52 65 66 65 72 65 72 3a 20 68 74 74 70 3a

           886f40a0:  2f 2f 77 77 77 2e 6d 73 6e 2e 63 6f 6d 2f 0d 0a

           886f40b0:  55 41 2d 43 50 55 3a 20 78 38 36 0d 0a 41 63 63

           886f40c0:  65 70 74 2d 45 6e 63 6f 64 69 6e 67 3a 20 67 7a

           886f40d0:  69 70 2c 20 64 65 66 6c 61 74 65 0d 0a 55 73 65

           886f40e0:  72 2d 41 67 65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61

           886f40f0:  2f 34 2e 30 20 28 63 6f 6d 70 61 74 69 62 6c 65

           886f4100:  3b 20 4d 53 49 45 20 37 2e 30 3b 20 57 69 6e 64

           886f4110:  6f 77 73 20 4e 54 20 36 2e 30 3b 20 53 4c 43 43

           886f4120:  31 3b 20 2e 4e 45 54 20 43 4c 52 20 32 2e 30 2e

           886f4130:  35 30 37 32 37 3b 20 2e 4e 45 54 20 43 4c 52 20

           886f4140:  33 2e 30 2e 30 34 35 30 36 29 0d 0a 48 6f 73 74

           886f4150:  3a 20 77 77 77 2e 6d 73 6e 2e 63 6f 6d 0d 0a 50

           886f4160:  72 6f 78 79 2d 43 6f 6e 6e 65 63 74 69 6f 6e 3a

           886f4170:  20 4b 65 65 70 2d 41 6c 69 76 65 0d 0a 43 6f 6f

           886f4180:  6b 69 65 3a 20 4d 43 31 3d 56 3d 33 26 47 55 49

           886f4190:  44 3d 31 61 31 61 65 64 38 31 38 36 34 30 34 32

           886f41a0:  39 62 61 63 32 37 38 61 65 37 65 39 63 38 35 39

           886f41b0:  64 39 3b 20 6d 68 3d 4d 53 46 54 3b 20 43 55 4c

           886f41c0:  54 55 52 45 3d 45 4e 2d 55 53 3b 20 4d 55 49 44

           886f41d0:  3d 32 43 38 33 35 42 36 32 34 35 42 46 34 46 30

           886f41e0:  36 38 36 36 43 34 37 38 38 42 46 39 30 43 35 38

           886f41f0:  32 3b 20 7a 69 70 3d 7a 3a 45 43 31 7c 6c 61 3a

           886f4200:  35 31 2e 35 31 32 32 32 31 38 38 7c 6c 6f 3a 30

           886f4210:  7c 63 3a 47 42 7c 68 72 3a 31 3b 20 46 6c 69 67

           886f4220:  68 74 47 72 6f 75 70 49 64 3d 34 37 3b 20 46 6c

           886f4230:  69 67 68 74 49 64 3d 42 61 73 65 50 61 67 65 3b

           886f4240:  20 75 73 68 70 73 76 72 3d 4d 3a 35 7c 46 3a 35

           886f4250:  7c 54 3a 35 7c 45 3a 35 7c 44 3a 62 6c 75 7c 57

           886f4260:  3a 46 7c 50 3a 4e 7c 56 3a 30 3b 20 75 73 68 70

           886f4270:  63 6c 69 3d 30 7c 48 2e 30 2e 31 7c 47 2e 30 2e

           886f4280:  31 7c 5a 2e 30 2e 31 7c 52 2e 30 2e 31 2e 63 61

           886f4290:  70 7c 43 2e 30 2e 31 2e 6c 67 3a 6e 65 77 79 6f

           886f42a0:  72 6b 6e 79 7c 4c 2e 30 2e 31 2e 4c 4e 3a 57 4e

           886f42b0:  42 43 3b 20 75 73 68 70 77 65 61 3d 77 63 3a 55

           886f42c0:  53 4e 59 30 39 39 36 3b 20 75 73 68 70 70 72 3d

           886f42d0:  43 3a 31 3a 30 38 30 37 32 31 7c 53 3a 31 3a 30

           886f42e0:  38 30 38 30 36 3b 20 68 70 63 6c 69 3d 57 2e 48

           886f42f0:  7c 4c 2e 7c 53 2e 7c 52 2e 7c 55 2e 4c 7c 43 2e

           886f4300:  3b 20 68 70 73 76 72 3d 4d 3a 35 7c 46 3a 35 7c

           886f4310:  54 3a 35 7c 45 3a 35 7c 44 3a 62 6c 75 7c 57 3a

           886f4320:  46 3b 20 68 70 6f 6c 79 3d 4f 3a 31 7c 48 3a 31

           886f4330:  3b 20 77 70 76 3d 30 0d 0a 0d 0a

    Above output shows starting virtual address of each buffer, the buffer's byte offset into the page pointed to by the virtual address and the total number of bytes in the buffer


    Looking at the contents of the memory buffer closely :-


    kd> dc 886f4040 886f4330+0xc

    886f4040  20544547 70747468 772f2f3a 6d2e7777  GET http://www.m

    886f4050  632e6e73 612f6d6f 2f78616a 73612e48  sn.com/ajax/H.as

    886f4060  48207870 2f505454 0d312e31 6363410a  px HTTP/1.1..Acc

    886f4070  3a747065 2a2f2a20 63410a0d 74706563  ept: */*..Accept

    886f4080  6e614c2d 67617567 65203a65 73752d6e  -Language: en-us

    886f4090  65520a0d 65726566 68203a72 3a707474  ..Referer: http:

    886f40a0  77772f2f 736d2e77 6f632e6e 0a0d2f6d  //www.msn.com/..

    886f40b0  432d4155 203a5550 0d363878 6363410a  UA-CPU: x86..Acc

    886f40c0  2d747065 6f636e45 676e6964 7a67203a  ept-Encoding: gz

    886f40d0  202c7069 6c666564 0d657461 6573550a  ip, deflate..Use

    886f40e0  67412d72 3a746e65 7a6f4d20 616c6c69  r-Agent: Mozilla

    886f40f0  302e342f 6f632820 7461706d 656c6269  /4.0 (compatible

    886f4100  534d203b 37204549 203b302e 646e6957  ; MSIE 7.0; Wind

    886f4110  2073776f 3620544e 203b302e 43434c53  ows NT 6.0; SLCC

    886f4120  2e203b31 2054454e 20524c43 2e302e32  1; .NET CLR 2.0.

    886f4130  32373035 2e203b37 2054454e 20524c43  50727; .NET CLR

    886f4140  2e302e33 30353430 0a0d2936 74736f48  3.0.04506)..Host

    886f4150  7777203a 736d2e77 6f632e6e 500a0d6d  : www.msn.com..P

    886f4160  79786f72 6e6f432d 7463656e 3a6e6f69  roxy-Connection:

    886f4170  65654b20 6c412d70 0d657669 6f6f430a   Keep-Alive..Coo

    886f4180  3a65696b 31434d20 333d563d 49554726  kie: MC1=V=3&GUI

    886f4190  61313d44 64656131 36383138 32343034  D=1a1aed81864042

    886f41a0  63616239 61383732 39653765 39353863  9bac278ae7e9c859

    886f41b0  203b3964 4d3d686d 3b544653 4c554320  d9; mh=MSFT; CUL

    886f41c0  45525554 2d4e453d 203b5355 4449554d  TURE=EN-US; MUID

    886f41d0  3843323d 36423533 42353432 30463446  =2C835B6245BF4F0

    886f41e0  36363836 38373443 39464238 38354330  6866C4788BF90C58

    886f41f0  7a203b32 7a3d7069 3143453a 3a616c7c  2; zip=z:EC1|la:

    886f4200  352e3135 32323231 7c383831 303a6f6c  51.51222188|lo:0

    886f4210  473a637c 72687c42 203b313a 67696c46  |c:GB|hr:1; Flig

    886f4220  72477468 4970756f 37343d64 6c46203b  htGroupId=47; Fl

    886f4230  74686769 423d6449 50657361 3b656761  ightId=BasePage;

    886f4240  68737520 72767370 353a4d3d 353a467c   ushpsvr=M:5|F:5

    886f4250  353a547c 353a457c 623a447c 577c756c  |T:5|E:5|D:blu|W

    886f4260  507c463a 567c4e3a 203b303a 70687375  :F|P:N|V:0; ushp

    886f4270  3d696c63 2e487c30 7c312e30 2e302e47  cli=0|H.0.1|G.0.

    886f4280  2e5a7c31 7c312e30 2e302e52 61632e31  1|Z.0.1|R.0.1.ca

    886f4290  2e437c70 2e312e30 6e3a676c 6f797765  p|C.0.1.lg:newyo

    886f42a0  796e6b72 302e4c7c 4c2e312e 4e573a4e  rkny|L.0.1.LN:WN

    886f42b0  203b4342 70687375 3d616577 553a6377  BC; ushpwea=wc:U

    886f42c0  30594e53 3b363939 68737520 3d727070  SNY0996; ushppr=

    886f42d0  3a313a43 37303830 537c3132 303a313a  C:1:080721|S:1:0

    886f42e0  30383038 68203b36 696c6370 482e573d  80806; hpcli=W.H

    886f42f0  7c2e4c7c 527c2e53 2e557c2e 2e437c4c  |L.|S.|R.|U.L|C.

    886f4300  7068203b 3d727673 7c353a4d 7c353a46  ; hpsvr=M:5|F:5|

    886f4310  7c353a54 7c353a45 6c623a44 3a577c75  T:5|E:5|D:blu|W:

    886f4320  68203b46 796c6f70 313a4f3d 313a487c  F; hpoly=O:1|H:1

    886f4330  7077203b 0d303d76 2e0a0d0a 2e6e736d  ; wpv=0.....msn.


    So this packets contains HTTP traffic for MSN ! (Very true I had the msn site open while I was debugging this machineJ).


     A  list all NDIS packet pools can be displayed with the pktpools option, each pool would have set of NDIS packets.


    kd> !pktpools

    Pool      Allocator  BlocksAllocated  BlockSize  PktsPerBlock  PacketLength

    87faa268  8b88d1a1   0x6         0x1000  0x13         0xd0   dc21x4vm!AllocateAdapterMemory+16b

    87f6f4e0  8b88d1a1   0x6         0x1000  0x13         0xd0   dc21x4vm!AllocateAdapterMemory+16b

    87fbf2f0  8b88d1a1   0x6         0x1000  0x13         0xd0   dc21x4vm!AllocateAdapterMemory+16b

    87f4eb40  8b88d1a1   0x6         0x1000  0x13         0xd0   dc21x4vm!AllocateAdapterMemory+16b

    87e19620  8174d66c   0x1         0x1000  0x12         0xd8   ndis!DriverEntry+43d

    87e19670  8174d65a   0x1         0x1000  0x13         0xd0   ndis!DriverEntry+42b


     A list of the NDIS packets in a packet pool can be displayed with the findpacket option.  The pool address 87e19670 was obtained from the pktpools output above.


    kd> !findpacket p 87e19670


    Searching Free block <0x88727000>

    Packet at 0x88727538


    0x88727538 is our http packet shown above.


    Another variant of findpacket is used  to find an NDIS packet with a Virtual address from the packet buffer. A random address 886f4110 was obtained from the packet buffer above with http contents.


    kd> !findpacket v 886f4110


    Searching Free block <0x881a3000>


    Searching Used block <0x88185000>


    Searching Used block <0x88187000>


    Searching Used block <0x8818b000>


    Searching Used block <0x8818f000>


    Searching Used block <0x88193000>


    Searching Free block <0x88176000>




    Searching Free block <0x88727000>


    Packet found

    Packet at 0x88727538



      PhysicalCount       00000000  Total Length        00000000

      Head                00000000  Tail                00000000

      Pool                00000000  Count               00000000

      Flags               00000000  ValidCounts         00

      NdisPacketFlags     00000000  NdisPacketOobOffset 0000


          Private.Flags          : 00000082


          Private.NdisPacketFlags: 90



    I hope this gives the reader a better understanding of NDIS stacks.



  • Ntdebugging Blog

    Red alert! My Server is hung - what do I do?


    So you have a dump from a hung server and you’re the first person on the scene. Your IT Manager is jumping up and down, the phone is ringing off the hook and people are hovering outside your cube.  It’s game time and the pressure is on!!!  Now what do you do? 


    Well take a deep breath, get a cup of coffee, and relax because I’m here to help you out!  Let me share what we typically do on our first pass through a hung server kernel debug.  This works for both live debugs and dumps. These are steps you can take and they will find problems!


    Here’s something else to consider.  If the server is mission critical you will probably want to get a dump vs. a live debug so you can get the server back up and running.  This will take the pressure off because you can then do the debug offline, and if need be, send the dump to other people for review.


    Before we get started let me state that the following data is completely fabricated and many of the process names and address in this output have been made up.  Do not question odd offsets or alignments.


    I’m also assuming that you know how to


    1.       Collect a kernel dump: http://support.microsoft.com/kb/244139


    2.       Set up the debugger: http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx


    3.       Know how to use the symbol server: http://support.microsoft.com/kb/311503



    0)      Before I start these types of debugs I like to open a log file.


    1: kd> .logopen H:\repro\hungserver.log

    Opened log file 'H:\repro\hungserver.log'



    1)      !vm - Look for memory usage.  Generally speaking you want to look at what the current pool or memory usage values are and compare them to the max available.



    1: kd> !vm



    *** Virtual Memory Usage ***

          Physical Memory:      982890 (   3931560 Kb)

          Page File: \??\P:\pagefile.sys

            Current:   3931560 Kb  Free Space:   3742548 Kb

            Minimum:   3931560 Kb  Maximum:      4193280 Kb

          Available Pages:      631300 (   2525200 Kb)

          ResAvail Pages:       888171 (   3552684 Kb)

          Locked IO Pages:         195 (       780 Kb)

          Free System PTEs:     202830 (    811324 Kb) < THIS IS OK

          Free NP PTEs:          32765 (    131060 Kb) < THIS IS OK

          Free Special NP:           0 (         0 Kb)

          Modified Pages:          241 (       964 Kb)

          Modified PF Pages:       241 (       964 Kb)

          NonPagedPool Usage:    11377 (     45508 Kb) < THIS IS OK

          NonPagedPool Max:      65536 (    262144 Kb) 

          PagedPool 0 Usage:      6398 (     25592 Kb)

          PagedPool 1 Usage:      2201 (      8804 Kb)

          PagedPool 2 Usage:      2216 (      8864 Kb)

          PagedPool 3 Usage:      2179 (      8716 Kb)

          PagedPool 4 Usage:      2199 (      8796 Kb)

          PagedPool Usage:       15193 (     60772 Kb) < THIS IS OK

          PagedPool Maximum:     67584 (    270336 Kb)

          Shared Commit:         24569 (     98276 Kb)

          Special Pool:              0 (         0 Kb)

          Shared Process:        12519 (     50076 Kb)

          PagedPool Commit:      15252 (     61008 Kb)

          Driver Commit:          2083 (      8332 Kb)

          Committed pages:      313611 (   1254444 Kb) < THIS IS OK

          Commit limit:        1925815 (   7703260 Kb)


    Check to see if any apps are using tons of memory.  In this case I don’t see a problem.


          Total Private:        239673 (    958692 Kb)

             36b0 EXCEL.EXE        10775 (     43100 Kb) < THIS IS OK, etc

             2ee8 myapploc.exe     10288 (     41152 Kb)

             097c MySSrv.exe        7497 (     29988 Kb)

             0418 MyFun32.exe       6277 (     25108 Kb)

             0474 svchost.exe       6164 (     24656 Kb)

             1be8 ABCDEFGH.EXE      4984 (     19936 Kb)

             0480 IEXPLORE.EXE      4924 (     19696 Kb)

             09c4 ANOTHER.exe       4768 (     19072 Kb)

             19a4 HMMINTER.exe      4207 (     16828 Kb)

             1b30 ohboya.EXE        4146 (     16584 Kb)

             4558 aprocess.EXE      4138 (     16552 Kb)

             30e8 another.exe       3691 (     14764 Kb)

             0924 aservicec.exe     3508 (     14032 Kb)

             0854 RRXXc.exe         3400 (     13600 Kb)

             3458 MYWIN.EXE         3389 (     13556 Kb)

             0d90 FunService.exe    3298 (     13192 Kb)

             1180 CustomAp.exe      3221 (     12884 Kb)

             06ac XYZvrver.exe      2769 (     11076 Kb)

             2cdc ABCDEFGH.exe      2591 (     10364 Kb)

             02f4 lsass.exe         2567 (     10268 Kb)

             21b4 IEXPLORE.EXE      2516 (     10064 Kb)

             3420 Process.exe       2450 (      9800 Kb)

             4cd4 XYZXY.EXE         2305 (      9220 Kb)

             4a30 lookup.EXE        2244 (      8976 Kb)

             4360 Process.exe       2201 (      8804 Kb)

             0564 spoolsv.exe       2166 (      8664 Kb)

             2e5c XYZXYZEXE         2076 (      8304 Kb)

             02bc winlogon.exe      1964 (      7856 Kb)

             4e48 winlogon.exe      1958 (      7832 Kb)

             42bc ABCDEFGH.exe      1943 (      7772 Kb)

             0eb8 svchost.exe       1922 (      7688 Kb)

             3b98 Process.exe       1919 (      7676 Kb)

             4c1c IEXPLORE.EXE      1864 (      7456 Kb)

             17b8 winlogon.exe      1852 (      7408 Kb)

             3124 winlogon.exe      1849 (      7396 Kb)

             14b8 winlogon.exe      1847 (      7388 Kb)

             32cc winlogon.exe      1843 (      7372 Kb)

             1f84 winlogon.exe      1843 (      7372 Kb)

             2ebc winlogon.exe      1842 (      7368 Kb)

             1548 winlogon.exe      1840 (      7360 Kb)

             21c4 PROCESS213.EXE    1833 (      7332 Kb)

             3b58 MYWIN.EXE         1817 (      7268 Kb)

             4b3c winlogon.exe      1816 (      7264 Kb)


    NOTE if you see high pool values you will want to issue a !poolused 2 and a !poolused 4 to dump out the pool usages so you can see what pool tags are consuming pool.  (We will write a dedicated blog on this topic later.)



    2) !sysptes - See if one of the lists is low (less than 10)



    1: kd> !sysptes


    All of these are ok


    System PTE Information

      Total System Ptes 224223

         SysPtes list of size 1 has 225 free

         SysPtes list of size 2 has 57 free

         SysPtes list of size 4 has 136 free

         SysPtes list of size 8 has 59 free

         SysPtes list of size 16 has 95 free


        starting PTE: c022b000

        ending PTE:   c03dff78


      free blocks: 652   total free: 202831    largest free block: 191973



    3) !defwrites - If throttling, the server is doing nothing other than writing to the disk.



    1: kd> !defwrites

    *** Cache Write Throttle Analysis ***


          CcTotalDirtyPages:                   187 (     748 Kb)

          CcDirtyPageThreshold:             130560 (  522240 Kb)

          MmAvailablePages:                 631300 ( 2525200 Kb)

          MmThrottleTop:                       450 (    1800 Kb)

          MmThrottleBottom:                     80 (     320 Kb)

          MmModifiedPageListHead.Total:        241 (     964 Kb)


    Write throttles not engaged  < THIS IS OK. Good = NOT engaged.



    4) !ready to see if we're holding stuff up



    1: kd> !ready

    Processor 0: No threads in READY state  < THIS IS OK

    Processor 1: No threads in READY state  < THIS IS OK


    If we had threads in a ready state you would want to investigate what those threads were and what is running on the processor.



    5) !pcr x; kv on each processor - If they aren't idle then we could be doing DPCs



    1: kd> !pcr 0  < Dump the processor control registers for CPU 0

    KPCR for Processor 0 at ffdff000:

        Major 1 Minor 1

          NtTib.ExceptionList: ffffffff

              NtTib.StackBase: 00000000

             NtTib.StackLimit: 00000000

           NtTib.SubSystemTib: 80042000

                NtTib.Version: 012e7ace

            NtTib.UserPointer: 00000001

                NtTib.SelfTib: 00000000


                      SelfPcr: ffdff000

                         Prcb: ffdff120

                         Irql: 00000000

                          IRR: 00000000

                          IDR: ffffffff

                InterruptMode: 00000000

                          IDT: 8003f400

                          GDT: 8003f000

                          TSS: 80042000


                CurrentThread: 8056cd00

                   NextThread: 00000000

                   IdleThread: 8056cd00


                    DpcQueue: < NO DPCs: Not much to look at then 


    1: kd> !pcr 1  < Dump the processor control registers for CPU 1

    KPCR for Processor 1 at f773f000:

        Major 1 Minor 1

          NtTib.ExceptionList: f5ba1d30

              NtTib.StackBase: 00000000

             NtTib.StackLimit: 00000000

           NtTib.SubSystemTib: f773fef0

                NtTib.Version: 0121925d

            NtTib.UserPointer: 00000002

                NtTib.SelfTib: 7ffda000


                      SelfPcr: f773f000

                         Prcb: f773f120

                         Irql: 00000000

                          IRR: 00000000

                          IDR: ffffffff

                InterruptMode: 00000000

                          IDT: f77456e0

                          GDT: f77452e0

                          TSS: f773fef0


                CurrentThread: 8963cb90

                   NextThread: 00000000

                   IdleThread: f7741fa0


                    DpcQueue: < NO DPCs: Not much to look at then


    6) !locks - Look for deadlocks and contention



    The following output is of interest.

    The thread ID with the <*> next to it means that he has exclusive access to the resource and that all the other threads are waiting on that thread to finish its work. Typically you would !thread that OWNER THREAD ID <*> (e.g., !thread 87bddda0) to see what that thread is doing. If you have two threads that have exclusive access to two different resources, and these threads are in each other’s exclusive waiters list, you have a deadlock.  The following is an example of what a deadlock might look like.  In this case you would want to !thread each owner and evaluate the logic of the code in each stack that allowed the threads to get into this state 


    1: kd> !locks


    KD: Scanning for held locks......


    Resource @ 0x8a50ee98    Shared 4 owning threads

         Threads: 896856d0-01<*> 89686778-01<*> 896862d0-01<*> 89685da0-01<*>

    KD: Scanning for held locks............................................................


    Resource @ 0x896da1bc    Exclusively owned

         Threads: 896e3b20-01<*>

    KD: Scanning for held locks..



    Resource @ 0x81234567    Shared 1 owning threads

        Contention Count = 15292

        NumberOfSharedWaiters = 1

        NumberOfExclusiveWaiters = 39

         Threads: 87bddda0-01<*> 806d2020-01 



         Threads Waiting On Exclusive Access:

                  80ced020       80c036f8       80cdc7a0       80c438b0      

                  80e6cda0       80f96987       8007fd60       8004dc10      

                  80d7b020       80a2dd70       80b89620       80b58020      

                  8036eda0       87abc123       80606da0       8056e890      

                  802b3630       80cc7590       80d64020       80f7dda0      

                  80129580       80b73da0       806d2578       80b505d8      



    KD: Scanning for held locks................


    Resource @ 0x83245678    Exclusively owned

        Contention Count = 4827

        NumberOfExclusiveWaiters = 35

         Threads: 87abc123-01<*>

         Threads Waiting On Exclusive Access:

                  803e6aa0       80876020       80240020       80f56588      

                  808174f0       80bd6b28       80c3c448       8046d6c8      

                  801e8da0       80356518       80b4c978       8069e020      

                  80cb9020       87bddda0       80c65020       86daaac0      

                  80379020       80fe4020      




    8) !process 0 0 - Search for drwtsn32.  This would indicate that we have a process that has crashed and is in the process of being dumped.  This could cause a server hang.  Look at the PEB for drwtsn32 and get its command line to see what process is being dumped.  You should be able to do this by getting its process id and doing a .process PROCESSID;.reload;!PEB


    The following is how to extract a command line for any process, but it would work for Watson also.


    1: kd> .process 89f31020 

    Implicit process is now 89f31020

    1: kd> .reload

    Loading Kernel Symbols


    Loading User Symbols


    Loading unloaded module list


    1: kd> !peb

    PEB at 7ffdf000

        InheritedAddressSpace:    No

        ReadImageFileExecOptions: Yes

        BeingDebugged:            No

        ImageBaseAddress:         01000000

        Ldr                       77fc23a0

        Ldr.Initialized:          Yes

        Ldr.InInitializationOrderModuleList: 00171ef8 . 00176c90

        Ldr.InLoadOrderModuleList:           00171e90 . 00176c80

        Ldr.InMemoryOrderModuleList:         00171e98 . 00176c88

                Base TimeStamp                     Module

             1000000 3e80245d Mar 24 05:41:49 2003 \??\P:\WINDOWS\system32\winlogon.exe

            77f40000 3e802494 Mar 25 05:42:44 2003 P:\WINDOWS\system32\ntdll.dll

            77e40000 44c60ec8 Jul 25 08:30:00 2006 P:\WINDOWS\system32\kernel32.dll

            77ba0000 3e802496 Mar 25 05:42:46 2003 P:\WINDOWS\system32\msvcrt.dll

            77da0000 3e802495 Mar 25 05:42:45 2003 P:\WINDOWS\system32\ADVAPI32.dll

            77c50000 40566fc9 Mar 15 23:08:57 2004 P:\WINDOWS\system32\RPCRT4.dll

            77d00000 45e7bafc Mar 02 00:49:48 2007 P:\WINDOWS\system32\USER32.dll

            77c00000 45e7bafc Mar 02 00:49:48 2007 P:\WINDOWS\system32\GDI32.dll

            75970000 3e8024a2 Mar 25 05:42:58 2003 P:\WINDOWS\system32\USERENV.dll

            75810000 3e8024a3 Mar 25 05:42:59 2003 P:\WINDOWS\system32\NDdeApi.dll

            761b0000 3e8024a0 Mar 25 05:42:56 2003 P:\WINDOWS\system32\CRYPT32.dll


        SubSystemData:     00000000

        ProcessHeap:       00070000

        ProcessParameters: 00020000

        WindowTitle:  '< Name not readable >'

        ImageFile:    '\??\P:\WINDOWS\system32\winlogon.exe'

        CommandLine:  'winlogon.exe' < HERE IS THE COMMAND LINE.. No args in this case



    ( output is truncated ... )


    9) Look at the handle table size.  If it’s over 10000 you may have trouble.  If you do have a handle leak refer to TalkBackVideo Understanding handle leaks and How to use !htrace to find them



    1: kd> !process 0 0



    PROCESS 8a613270  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000

        DirBase: 0acc0000  ObjectTable: e1001d10  HandleCount: 2510.

        Image: System


    PROCESS 8a294328  SessionId: none  Cid: 0274    Peb: 7ffdf000  ParentCid: 0004

        DirBase: ef1ac000  ObjectTable: e14ac1d0  HandleCount: 124.

        Image: smss.exe


    PROCESS 8a103424  SessionId: 0  Cid: 02a4    Peb: 7ffdf000  ParentCid: 0274

        DirBase: ed804000  ObjectTable: e18caa68  HandleCount: 1171.

        Image: csrss.exe


    PROCESS 8a104343  SessionId: 0  Cid: 02bc    Peb: 7ffdf000  ParentCid: 0274

        DirBase: ed539000  ObjectTable: e18c67b0  HandleCount: 498.

        Image: winlogon.exe


    PROCESS 8a0f6634  SessionId: 0  Cid: 02e8    Peb: 7ffdf000  ParentCid: 02bc

        DirBase: ece72000  ObjectTable: e1668e40  HandleCount: 568.

        Image: services.exe


    PROCESS 8a123423  SessionId: 0  Cid: 02f4    Peb: 7ffdf000  ParentCid: 02bc

        DirBase: ecd7a000  ObjectTable: e16684a0  HandleCount: 30000. < This is bad

        Image: lsass.exe


    PROCESS 89f96453  SessionId: 0  Cid: 03e0    Peb: 7ffdf000  ParentCid: 02e8

        DirBase: eb99c000  ObjectTable: e16bb570  HandleCount: 500.

        Image: svchost.exe


    PROCESS 8a0c6532  SessionId: 0  Cid: 042c    Peb: 7ffdf000  ParentCid: 02e8

        DirBase: eb6d7000  ObjectTable: e1731170  HandleCount: 156.

        Image: svchost.exe


    PROCESS 8a0a8d88  SessionId: 0  Cid: 0460    Peb: 7ffdf000  ParentCid: 02e8

        DirBase: eb58f000  ObjectTable: e17372e8  HandleCount: 124.

        Image: svchost.exe


    PROCESS 89f77678  SessionId: 0  Cid: 0474    Peb: 7ffdf000  ParentCid: 02e8

        DirBase: eb484000  ObjectTable: e17305b8  HandleCount: 1457.

        Image: svchost.exe


    9) !process 0 0 system - Check the worker threads in the system process (search for srv! to find server worker threads).  What are these threads doing?  These are the server service threads.  Are they blocked on I/O or waiting for a resource?


    10) 1: kd> !process 0 17 csrss.exe  - Look for 16 LPC server threads.

    What are they doing? Are they blocked?


    11) !stacks 2,  This will dump every call stack on the server.  You may need to go through and evaluate every stack on the server.  Look for critical sections, etc.


    15) !qlocks  This will allow you to check the stack of all the Queued spin locks on the machine.   For further information on spinlocks refer to the Windows Internals book.


    1: kd> !qlocks

    Key: O = Owner, 1-n = Wait order, blank = not owned/waiting, C = Corrupt


                           Processor Number

        Lock Name         0  1    << Nothing to worry about here.


    KE   - Dispatcher        

    MM   - Expansion         

    MM   - PFN               

    MM   - System Space      

    CC   - Vacb              

    CC   - Master            

    EX   - NonPagedPool      

    IO   - Cancel            

    EX   - WorkQueue         

    IO   - Vpb                

    IO   - Database          

    IO   - Completion        

    NTFS - Struct            

    AFD  - WorkQueue         

    CC   - Bcb               

    MM   - NonPagedPool     


    16) !process 0 17 winlogon.exe to look for hung LPC calls.  If you find a LPC call calling out of winlogon you can follow the call with the !LPC debugger command. This will allow you to see what the thread is doing in the other process.



    If you have further questions on any of these commands, please refer to the debugger.chm file in the Windows debugger tools install.


    Good luck and happy debugging.


    “This debugger is mine, there are many like it but this one is mine!” Jeff Dailey

  • Ntdebugging Blog

    Basics of Debugging Windows


    Hello, this is East again. This blog post is about a topic that we always skip over when discussing debugging; what and where are the tools for debugging. I will touch on the different types of debuggers, loading symbols and the basics of getting started with loading up a dump under your preferred debugger.

    Microsoft currently offers 4 types of debugging tools. With these tools you can remote debug another machine over firewire or serial cable (USB also but may not work consistently), as well as debug usermode processes and dump files.

    Command line debuggers:

    1 ) kd.exe: kernel debugger – Used to review Crash dumps created by a blue screen crash event or a stop error. (kd –z <location of dump> –y <location of symbols>)

    2 ) cdb.exe: User mode debugger for reviewing applications,  processes, and process dumps  (cdb  –z <location of dump> –y <location of symbols> )

    3 ) ntsd.exe: CDB and NTSD are virtually identical, except that NTSD spawns a new text window when it is started, whereas CDB inherits the Command Prompt window from which it was invoked.  When I refer to "CDB", it applies to both CDB and NTSD.

    Graphical User Interface Debugger:

    4) Windbg.exe is a GUI based debugger. It can debug the same things as KD & CDB using the same commands. Windbg gives you the ability to have multiple windows open simultaneously  to review source code or other selectable items under the view menu.

    I like using windbg for all of my user and kernel debugging, while  others I work with prefer kd for kernel debugging and cdb for user mode debugging.


    There are 32bit and 64bit debuggers available.

    NOTE: Some people use Visual Studio as well, but this blog post will not cover using Visual Studio as a debugger.


    You can review applications that already have started on your machine using CDB or Windbg. You can have the problematic application launch under the debugger as well:

    Cdb or Windbg

    -p <pid> specifies the decimal process ID to attach to ( use tlist or the task manger to obtain the PID)

    -psn <name> specifies the process to attach to by service name

    <application to launch> -y <symbol path>

    NOTE: windbg allows you to use menu options as well: select “Attach to a Process” on the File menu to debug a user-mode application that is currently running.


    What are dumps?

    Memory dumps are a record of what was in memory and the registers at the time of a crash. There are 3 types of memory dumps:

    NOTE: The type of dump that will be written upon bugcheck can be configured by right clicking my computer ->properties ->Advanced tab ->Settings, in the Write debugging section you will use the first drop down box to select what type of memory dump you want. (See KB307973)

    ·         Mini dump – is a subset of memory that is in use by the application creating the dump.

    A mini memory Dump file is written to %SystemRoot%\Minidump\Memory.dmp by default and is usually less than a 1mb in size.

    ·         Kernel only – This is used to review the machine’s kernel memory at the time of the crash.

    ·         Full/Complete – This is the largest kernel mode dump file. It contains all information from kernel and user mode address spaces that was in physical memory at the time of the dump (about the same size as the physical memory on the box).

    Kernel and Complete Memory Dumps are written to %SystemRoot%\Memory.dmp by default.

    Note: You can configure the server to crash using certain keystrokes . This would be useful when troubleshooting a hung server or a timing issue, KB244139 explains how to configure your server for a manual crash.

    You can also create dump files from an application or process, these are known as User-mode dumps.  Additional information can be found on these types dump in the Debugging Tools for Windows help file.


    How do I read a dump file?

    In order to make fast progress with a memory dump file, it is best to load symbol files. Symbol files contains data that the debugger uses to interpret the application or driver code. They may contain:

    -          Global variable names

    -          Function names

    Private Symbols would contain the above information and:

    -          Local variable names

    -          Source-line numbers

    -          Type information for variables, structures, etc.

     Microsoft currently has two ways you can access symbols for the Operating System:

    Service pack download site – You will need to create:

    -          Separate directories for Windows 2000 RTM, Windows 2000 SP1, Windows 2000 SP2, Windows XP RTM, etc.

    -          Separate directories for all of the above for free vs. checked build

    -          Separate directories for hotfix symbols


    Public symbol server – uses a symbol store, which is a collection of symbol files. The symbol server uses the time stamp & file size to match up symbols to the active binary.After getting your symbol files together, you will need a way to tell the debugger where they are located and set up some other options.

    To set the symbol path do one of the following:

    -          _NT_SYMBOL_PATH environment variable

    -          -y command line option

    -          .sympath (Set Symbol Path) debugger command

    -          WinDbg: File | Symbol File Path dialog, or CTRL+S

    To set the executable Image Path (needed for minidumps only), do one of the foolowing:

    -          -i command line option

    -          .exepath debugger command

    -          WinDbg: File | Image File Path dialog, or CTRL+i

    -          Source Path

    -          .srcpath WinDbg: File | Source File Path dialog, or CTRL+P

    If symbol errors appear when you begin, you can try the below commands to help narrow down some problems;

    !sym noisy — gives verbose symbol information


    .reload —  to reload all symbols


    Also using the srv* in your symbol path tells the debugger to load and save symbols being used out to a specific directory:

    srv*DownstreamStore*<symbol locations>


    NOTE: You must always use .reload after you change the symbol path or fix a symbol error — the debugger doesn’t automatically reload your symbols!


    Now that we are done with the overview, let’s configure our machine as a host computer to open memory a dump.  I will be using Microsoft Public Symbol servers and I want to store current symbols locally to my host machine.

    Using windbg I will set my current workspace symbols to: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols

    Click the menu option File ->Symbol File Path or Ctrl + S. This will bring up an empty box that will allow you to enter or browse to your symbol path.

    If using kd you want to set an environment variable (_NT_SYMBOL_PATH) under “my computer properties -> advanced tab” to always start with your symbols set to:  “srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols” or use this same path in your command line:

    Kd –z <path to dump.file> -y srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols


    NOTE: Windbg will append any workspace symbol path with the one set by the _NT_SYMBOL_PATH environment variable during loading of a memory dump.

    Ok, now we know what debugger we want to use and we know our symbol locations. Let’s open our first kernel memory dump , located on <drive letter> <path to dump file>

    Using windbg, I will load a dump file using menu options File ->Open crash Dump (ctrl + D) or drag the the dump file into the debugger; you can even  start windbg at the command prompt.  My command would look like this:

    Windbg  –z C:\training\case 7f\MEMORY051308.22.DMP

    I did not use the –y for symbol path, as it is set already in my default workspace or in my environment variable.

    When the debugger first loads a dump file it displays several lines of information before giving you a prompt to get started with your commands (by default):

    Microsoft (R) Windows Debugger Version 6.9.0003.113 X86  ß debugger version

    Copyright (c) Microsoft Corporation. All rights reserved. ß Copyright of the debugger creator

    Loading Dump File [C:\training\case 7f\MEMORY051308.22.DMP] ß location of the dump file loading

    Kernel Summary Dump File: Only kernel address space is available ß type of memory dump (mini, kernel, or full)

    Symbol search path is: srv*c:\pubsymbols*http://msdl.microsoft.com/download/symbols ß Symbol path for this debug session

    Executable search path is:  ß points to the directory the executable files are located. For most situations this is not needed. For other situations please check the debugger help file.


    The next 4 lines talk about The OS version, service packs and how many processors are on the box

    1 -Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible

    2 - Product: Server, suite: Enterprise TerminalServer SingleUserTS

    3 - Built by: 3790.srv03_sp2_gdr.070304-2240

    4 - Kernel base = 0x80800000 PsLoadedModuleList = 0x808a6ea8


    Next we would see when the machine crashed and how long it was up prior to this crash:

    Debug session time: Wed May 14 01:27:36.768 2008 (GMT-4)

    System Uptime: 0 days 16:32:51.921


    After completing the above process, the debugger starts loading the dump file and parsing through the loaded symbols. Here you may notice some warnings for some user space processes which are not included in the kernel dump. This is ok.

    WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000

    WARNING: Process directory table base BFF0A080 doesn't match CR3 007AF000

    Loading Kernel Symbols


    Loading User Symbols

    PEB is paged out (Peb.Ldr = 7ffdf00c).  Type ".hh dbgerr001" for details

    Loading unloaded module list


    *                                                                             *

    *                        Bugcheck Analysis                                    *

    *                                                                             *



    1- Use !analyze -v to get detailed debugging information.

    2 - BugCheck 7F, {8, f773ffe0, 0, 0}

    3 - *** ERROR: Module load completed but symbols could not be loaded for ql2300.sy

    The three things I want to point out from above are:

    1 - !analyze –v: This is the debugger command used to help analyze a dump file by reviewing information passed to KeBugCheck including specific parameters of that crash. It will analyze this information and provide a definition of the bugcheck, a stack showing all current function calls, and, when possible, the name of an offending driver or process that the debugger thinks is at fault.  Please review the debugger help file for additional information in this area.

    2 – The type of bugcheck that occurred on the machine.

    3 – An error telling you about symbols missing or not available to help diagnose a particular driver or application. This can lead to a misdiagnostis if you’re not careful.

    Once loading is completed you should be at a kd> prompt. This prompt shows you the current processor you are using (if the machine has more than one).

    For this dump we are at processor 3 on an 8 proc machine:

    3: kd>


    To view the current crash stack location you can use the "K" command. There are multiple forms of this command, each one dumping the basic plus additional information. As functions are executed and call other functions, a call stack is created in stack memory. Here are two common commands to view the stack:


    3: kd> k

    ChildEBP RetAddr

    00000000 baebf0ce nt!KiTrap08+0x75

    b3a4bffc baebf737 storport!RaCallMiniportInterrupt+0x2

    b3a4c008 8088d889 storport!RaidpAdapterInterruptRoutine+0x1d

    b3a4c008 80a59d8e nt!KiInterruptDispatch+0x49

    b3a4c09c 80a5c2fc hal!HalpGenerateInterrupt+0x1d2

    b3a4c0c0 80a5c44d hal!HalpLowerIrqlHardwareInterrupts+0x108

    b3a4c0d0 808256ed hal!KfLowerIrql+0x59



    3: kd> kb

    ChildEBP RetAddr  Args to Child

    00000000 baebf0ce 00000000 00000000 00000000 nt!KiTrap08+0x75

    b3a4bffc baebf737 97bedb88 b3a4c02c 8088d889 storport!RaCallMiniportInterrupt+0x2

    b3a4c008 8088d889 977b9e18 97bedad0 03010006 storport!RaidpAdapterInterruptRoutine+0x1d

    b3a4c008 80a59d8e 977b9e18 97bedad0 03010006 nt!KiInterruptDispatch+0x49

    b3a4c09c 80a5c2fc 97797004 97bedad0 00000102 hal!HalpGenerateInterrupt+0x1d2

    b3a4c0c0 80a5c44d 00000101 977b9e02 b3a4c0d8 hal!HalpLowerIrqlHardwareInterrupts+0x108

    b3a4c0d0 808256ed b3a4c0e8 baebf1c6 977b9bb0 hal!KfLowerIrql+0x59



    Either one can be used depending on how much information you want to see and can use.

    This completes the Basic of Debugging Windows, Part I. I will create a Part II using specific questions gathered from our readers.


    Miscellaneous information:

    To go further with this topic I would suggest starting with the debugger help file included with the Microsoft Debugging Tools. 

    ADPlus – An automated way to use the cdb.exe to capture/create a usermode dump when a process hangs or crashes. (more info - http://msdn.microsoft.com/en-us/library/cc265629.aspx or kb286350)

    Public Symbols for Microsoft Operating Systems:

    Microsoft Public Symbol server : srv * DownstreamStore * http://msdl.microsoft.com/download/symbols

    example: srv*c:\mysyms*http://msdl.microsoft.com/download/symbols

     Microsoft Symbol packages http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#d

    Use !Analyze-v to gather additional information about the bugcheck and a bucket-id for your dump file. The bucket-id can be submitted to Microsoft for review for similar crashes and resolutions. Try using the Microsoft Online Crash Analysis to submit your crash dump bucket-id for possible follow up from Microsoft or for Microsoft to look for trends: http://oca.microsoft.com/en/Welcome.aspx

    For concepts, tools and information about the system architecture:


    Windows Internal 4th edition (by Mark E. Russinovich & David A. Solomon) the whole book or Chapter 14 - Crash Dump Analysis

    Advanced Windows Debugging (by Mario Hewardt & Daniel Pravat )


  • Ntdebugging Blog

    Transcript of Windows NT Debugging Blog Live Chat


    For those of you that could not make the live chat on 8/13, here is the transcript of the chat session....


    Chat Topic: PGES-Windows NT Debugging Blog Live Chat
    Date: Wednesday, August 13, 2008

    Daniel (Moderator):
    Hello everyone-- thanks for coming to our chat on Platforms Global Escalation Services. The chat will officially get started at 1pm Eastern time. Only questions related to this topic will be addressed during this chat. Thanks!

    Daniel (Moderator):
    Hello everyone-- thanks for coming to our chat on Platforms Global Escalation Services. We'll get started in about 10 minutes.  You can start posting your questions now if you'd like and when the chat starts our Experts will begin answering them. Be sure to check the "Ask the Experts" box before you send your questions and please keep all questions on topic-- Thanks!

    Daniel (Moderator):

    Let's get started with our chat. Before we begin, though, I'd like to have our Experts introduce themselves and then they'll get started answering  your questions.

    Smoke [Windows Core] (Expert):
    Hi everyone, I'm an Escalation Engineer with the Window’s Core team.  I fix bugs for a living.

    Matthew [MSFT EE] (Expert):
    Hello, I am an Escalation Engineer with the Platforms Global Escalation Services (Windows Core) team.

    East - MSFT EE (Expert):
    I am East, an Escalation Engineer with the Microsoft Platforms Global Escalation Services. (Windows Core)

    Todd Webb - Msft (Expert):
    I am an Escalation Engineer with the Microsoft Platforms Global Escalation Services OEM hardware team...

    David (Expert):
    Hi, I'm an Escalation Engineer with Windows Core - reading code & debugging is my day-to-day.

    stheller (Expert):
    Hi, I'm a new Escalation Engineer with Platforms GES.

    Mr Ninja [MSFT EE] (Expert):
    Hi, I am an Escalation Engineer with Microsoft PGES.  I debug Windows for a living.

    Tate [MSFT EE] (Expert):
    Hi, I’m one of the EE’s on the Windows team.

    Jeff Dailey MSFT EE (Expert):
    Hi, my name is Jeff Dailey, I’m a Senior Escalation Engineer on the Microsoft Platforms Global Escalation Services team.

    Smoke [Windows Core] (Expert):
    Q: How can I track memory allocations through MmAllocateContiguousMemory?
    A: You could try poolhittag on MMCM or a breakpoint on MmAllocateContiguousMemory.  If you go with the break point, you can use a conditional breakpoint and dump the stack and anything else, then 'go' the system.  There will be a perf hit each time you break in.

    Tate [MSFT EE] (Expert):
    Q: For MmAllocatecontiguousMemory, will !poolused show the total amount used?
    A: !poolused 2 will show MmCm

    Matthew [MSFT EE] (Expert):
    Q: What's the best way to go about troubleshooting pool corruption dumps.
    A: Special Pool can be used to track down pool corruption problems.  http://msdn.microsoft.com/en-us/library/cc265889.aspx

    a-hstein (Expert):
    Greetings and sorry for the late message.  I am an intern in the GES group.

    Mr Ninja [MSFT EE] (Expert):
    Q: Could you explain the reasons why a memory dump analysis show an "illegal instruction" exception raised from a valid instruction?
    A: There are many reasons this could happen.  The instruction that was executed may not be what you see due to hardware problems such as a bit flip in the instruction when it was executed.  It is also possible for a hardware problem caused an exception to be raised on a valid instruction.  Sometimes software, or hardware, may trigger a jump to the middle of an instruction so that the instruction being executed is not what you think it is.I described a problem where we executed from the middle of an instruction in the blog http://blogs.msdn.com/ntdebugging/archive/2008/04/28/ntdebugging-puzzler-0x00000004-this-didn-t-puzzle-the-debug-ninja-how-about-you.aspx.

    Smoke [Windows Core] (Expert):
    Q: We use APC's to perform certain operations one of them is to have thread cleanup and exit.Is calling thread exit from an APC recommended ? This used to work fine, but with newer service packs we have threads exiting holding the heaplock!
    A: This sounds like a bad idea.  I would expect different ways that this could break (just like you have observed).

    David (Expert):
    Q: We use APC's to perform certain operations one of them is to have thread cleanup and exit.Is calling thread exit from an APC recommended ? This used to work fine, but with newer service packs we have threads exiting holding the heaplock!
    A: Part of the problem is that if ExitThread is called, any pending APCs on that thread's queue are lost.

    Matthew [MSFT EE] (Expert):
    Q: This question is in reference to special pool mentioned already. Is this article essentially the same as the MSDN reference?  http://support.microsoft.com/kb/188831/en-us
    A: The KB article documents enabling special pool via the registry, rather than verifier.  These are two different ways to accomplish the same thing.  Enabling it via the registry is sometimes preferred, since verifier enables additional checks beyond special pool.

    East - MSFT EE (Expert):
    Q: We use APC's to perform certain operations one of them is to have thread cleanup and exit.Is calling thread exit from an APC recommended ? This used to work fine, but with newer service packs we have threads exiting holding the heaplock!
    A: Would this help KB254956

     - If not we would need to follow-up with you for more information

    East - MSFT EE (Expert):
    Are there anything additional you want on the blog that we have not done?

    Jeff Dailey MSFT EE (Expert):
    Q: The final version of the Windows Internals Exam should be available before December 2008.  I’d like to thank all the community members that participated in the Beta.  Your feedback was very valuable.

    East - MSFT EE (Expert):
    Q: Are there anything additional you want on the blog that we have not done?

    Jeff Dailey MSFT EE (Expert):
    Q: When is the next Windows Internals exam scheduled? I would like to plan ahead.
    A: The final version of the Windows Internals Exam should be available before December 2008.  I’d like to thank all the community members that participated in the Beta.  Your feedback was very valuable.

    Matthew [MSFT EE] (Expert):
    Q: Will we get more puzzler on the blog?
    A: We’d like to do more puzzlers, but unfortunately they tend to take a lot of time, so I cannot say for sure when/if we’ll have more.

    Matthew [MSFT EE] (Expert):
    Q: How many of you in the audience are interested in more puzzlers on the ntdebugging blog?

    Smoke [Windows Core] (Expert):
    Q: Are you planning to write a book?
    A: Windows Internals is a great reference book that we all rely upon.  Additionally, you can check out: <http://www.amazon.com/Advanced-Debugging-Addison-Wesley-Microsoft-Technology/dp/0321374460>

    Tate [MSFT EE] (Expert):
    Q: As far as the blog is concerned I'm more a fan of the case studies type posts where you go through how you troubleshooted issues that you have enountered.
    A: So are we!!!

    Smoke [Windows Core] (Expert):
    Q: I'm very interested in puzzlers...
    A: Thanks for the feedback.  We will try to create some more in the future.

    Smoke [Windows Core] (Expert):
    Q: Debugging MPI apps - sometimes a crash happens on remote and the local smpd daemon will terminate the process being debugged. Using the debugger, is there a way to guard from TerminateProcess from the child? I guess that would break some security models.
    A: I'm not sure what MPI is, but this scenario sounds just like a service.  The service control manager will kill the service if it doesn't respond in a timely fashion.  With a service, there is a registry key to extend the timeout.  If such a mechanism isn't available for you, you should consider instrumentation/logging.

    East - MSFT EE (Expert):
    Q: I just skimmed over KB254956, we found APC to work. The issue here is that there are alertable waits in library modules like LSA/NDR/I_RPC calls where our APC fires which raises an user exception, gets handled and exits-thread exits holding the heap lock.
    A: We would need to discuss this further offline, how can I contact you?

    Matthew [MSFT EE] (Expert):
    Q: An award of puzzler like next edition of Windows Internals would definitely have my full attention. :)
    A: We'll consider it... thanks for the feedback!

    Jeff Dailey MSFT EE (Expert):
    Q: Have you ever found yourselves with an "unsolvable" case? :P
    A: No cases is unsolvable, nothing is truly random.  Some cases may take a very long time to resolve through multiple debugging passes, detailed code review, reverse engineering and multiple iterations of instrumentation.  In the end we find the problem.  

    Daniel (Moderator):

    Just a heads-up --we have about 15 minutes left in today's chat. Be sure to post your questions asap and our Experts will try to answer as many as possible before the chat ends. Thanks.

    Mr Ninja [MSFT EE] (Expert):
    Q: Tri-boot machine - XP, Server 2003 and Server 2000 with 2000 being the last one installed. After awhile, I got an error: "Windows 2000 could not start because the following file is missing or corrupt: \WINDOWS\SYSTEM32\CONFIG\SYSTEMd startup options for"..
    A: That is usually a known issue in Windows 2000 caused by the size of the system hive becoming too large.  We have several KB articles that describe this issue  KB269075, KB306038, KB323148, and KB277222 contain various resolutions you can try.  I have found that most often the steps in KB277222, using scrubber in a shutdown script, resolve this problem.  Starting with Windows 2003 we changed the boot architecture to prevent this problem, KB302594 describes this improvement.

    Tate [MSFT EE] (Expert):
    Q: Do you guys use USB debugging in Vista/2008? Why is that there is still one vendor that sells the debug dongle?
    A: Serial debugging works well enough most times.  Usually only if we have hardward that doesn't have a serial connection for some reason and only has USB or Firewire we try these alternates...

    East - MSFT EE (Expert):
    Q: I just skimmed over KB254956, we found APC to work. The issue here is that there are alertable waits in library modules like LSA/NDR/I_RPC calls where our APC fires which raises an user exception, gets handled and exits-thread exits holding the heap lock.
    A: On a better note it would be best to open a case with Microsoft Support - > <http://support.microsoft.com/> -> Need more help? -> Select a Product to start

    Jeff Dailey MSFT EE (Expert):
    Q: What companies are in attendance today?

    Graham (Expert):
    Q: There are lots of post mortum debuggers available, Dr Watson, NTSD, windbg, userdump, WER. Which ones do you usually recrommend your customers to use if you need to be sure to capture a dump from a crash?
    A: Userdump.exe is quite reliable for obtaining post-mortem dumps, and is easy to use.  It (along with ADPlus, which uses CDB) are good because they attach to the process and monitor exceptions, and can create dumps for times when a JIT debugger would not be able to create a thread in the process to obtain the dump.  Normally, I will set up drwtsn32 first, and if it cannot generate the dump, then I will go to userdump. 

    Smoke [Windows Core] (Expert):
    Q: How can I debug cases in which just I have the Minidump for CPU Hog? I tried !runaway and does not works
    A: The minidump alone may not be enough information.  You could try to look at the stacks and guess at what is using the CPU, but that require familiarity with the application.  You should capture a circular perfmon log with thread data.  Then get 3-5 dumps of the app.  From the perfmon log, you'll see what threads are active (and their activity profile).  From the dumps, you'll have a few snapshot of the process in motion.  Alternatively you could try a profiler like xperf.

    David (Expert):
    Q: Are there any free code coverage tools on Windows?
    A: This article describes how to obtain code coverage data:

    David (Expert):
    A: http://msdn.microsoft.com/en-us/library/ms182496.aspx

    stheller (Expert):
    http://www.microsoft.com/whdc/devtools/tools/prefast.mspx discusses the PREfast static source code analysis tool

    East - MSFT EE (Expert):
    Q: Are there any free code coverage tools on Windows?
    A: Please keep watching our blog site for the next chat - <http://blogs.msdn.com/ntdebugging>  or you can submit the question to the our blog site

    Daniel (Moderator):
    Well we're out of time for today's chat. Thank you very much to all of our guests who joined us today as well as to our Experts for answering so  many great questions. Have a great day!


Page 17 of 25 (241 items) «1516171819»