A Hole In My Head : WinDBG/KD Fun

MSDN link on how to set up a user or kernel debugger

doronh — Thu, 29 Jan 2009 22:15:00 GMT

This has got to be one of the top FAQs out there: how do I set up a kernel debugger? I just stumbled across a link on MSDN which gives instructions not only on how to set up a kernel debugger on all transports (serial, 1394, usb2), but also how to set up a user mode debugger or how to attach to a virtual machine. Pretty cool. This used to be an internal web page at Microsoft (and I think a topic in debugger.chm), it is great that it is now public

The MSDN topic is Starting the Debugger. Add it to your bookmarks and maybe we can create room in the FAQ for another question ;)

Debugger commands (.step_filter) that make my life easier

doronh — Thu, 17 Apr 2008 02:02:19 GMT

This is a pretty cool and somewhat obscure debugger command. It allows you to tell the debugger what functions to skip if you are using the trace command ('t'). I think of the trace command as the 'step into' command though, but that is just me. Let's say we have the following simple application:

#include <stdio.h>

struct Foo {
    Foo() : m_value(0) { }

    int Increment() { return ++m_value; }
    static void Print(int i) { printf("%d\n", i); }

    int m_value;
};

int _cdecl main(int argc, char *argv[])
{
    Foo f;
    Foo::Print(f.Increment());
    return 0;
}

If I were to run the program under the debugger and use the 't' command for each line, it would step into every function. I typically use 't' instead of 'p' because I usually want to step into a function at some point in time and I tend to press 'p' one too many times ;). Here is an example of the debugger session:

0:000> g test!main
>   13: {
0:000> t
>   14:     Foo f;
0:000> t
>    4:     Foo() : m_value(0) { }
0:000> t
>   15:     Foo::Print(f.Increment());
0:000> t
>    6:     int Increment() { return ++m_value; } [1]
0:000> t
>   15:     Foo::Print(f.Increment());
0:000> t
>    7:     static void Print(int i) { printf("%d\n", i); } [2]
0:000> gu
>   16:     return 0;
0:000> t
>   17: }
0:000> t
test!__mainCRTStartup+0x102:

Let's look at the statement Foo::Print(f.Increment()); When using the trace command, it will first step into Foo::Increment ([1]) before stepping into Foo::Print() ([2]). But let's say that I never want to step into Foo::Increment because I know that it is a simple function that I do not want to debug. I can tell the debugger to ignore trace commands into this function with the .step_filter command. The command takes a semi-colon delineated list of fully qualified symbol names (which can include wildcards so you can filter out entire modules) to ignore. Let's see the debugger session again with this command:

0:000> g test!main
>   13: {
0:000> .step_filter "test!Foo::Increment"
Filter out code symbols matching:
  test!Foo::Increment
0:000> t
>   14:     Foo f;
0:000> t
>    4:     Foo() : m_value(0) { }
0:000> t
>   15:     Foo::Print(f.Increment());
0:000> t
>    7:     static void Print(int i) { printf("%d\n", i); }
0:000> gu
>   16:     return 0;
0:000> t
>   17: }
0:000> t
test!__mainCRTStartup+0x102:

You will see now that when I trace into Foo::Print(f.Increment()); that the f.Increment() call is executed but not trace into (ignored is not the right word because it has run, I just didn't see it line by line) and I step directly into Foo::Print(). I think this is a pretty powerful debugger command, it can save you a lot of time if you are always accidentally stepping into the wrong function like I always do ;).

Getting 64 bit Vista to open my Inbox the way I want it to

doronh — Wed, 12 Mar 2008 02:00:05 GMT

A bit over a year ago I had to figure out why my Mail key started behaving differently on Vista vs XP and wrote about how I fixed it. Well, my dev box was so slow that I was able to employ enough sympathy that I got a new one. While my old box was a 32 bit machine, the new one came preinstalled with 64 bit Vista. Cool. I finally had a 64 bit box that was powerful enough to develop on (previously all I had a very noisy 64 bit test machine).

When I get a new machine that I will be using for dev work I do the following:

Blow away the preinstalled OS and install the latest OS release (beta or RTM) that I can do work on. In this case, I left the OS alone ;).
Enlist in windows sources
Install the latest WDK
Install Office Pro and Visio Pro (for the lovely WDF pnp/power/power policy state machines)
Apply the mail key registry change so Outlook works the way I want it to.

Well steps 1 through 4 went by flawlessly and I updated the registry to fix the mail key. All happy with myself I made the change and pressed the key. Nothing happened. The focus did not change nor did Outlook move to the default folder. Phooey. I double checked the changes in the registry and lo and behold I mistyped the value name and fixed it. Pressed the mail key. Still nothing happened. Rats.

So now my only recourse was to debug it ;). First searched for "RegisteredApp" in the source tree (via content indexing) to find the function which processed the WM_APPCOMMAND message. Once found, I traced it into the internal shlwapi function which looked up the appropriate locations in the registry and traced through it as well. Not the most enjoyable task either....first, I do not own the code so I am figuring out it intent while determining if it is behaving correctly...and second, it is optimized code so debugging with source was not an option. Thank god it was not IA64 assembly though. Phew. The one positive through this part of the debugging was that it was 100% reproducible, so when I got lost, I could just hit 'g' and press the mail key again to restart.

After looking at this for a half an hour or so, the logic was this

Query for the default mail client in HKCU
if (WIN64) { If that failed, 1) open the default mail client in the 32 bit HKLM software key and 2) query for the default mail client }
If that failed, query for the default mail client in the normal HKLM software key (which would be the 64 bit HKLM on 64 bit, the 32 bit HKLM on 32 bit).

Upon successfully querying for the default mail client, attempt to execute its open verb as registered in the key 64 bit HKLM\Software\Clients\Mail\<application>\shell\open\command. If this failed, attempt the same thing under the 32 bit version of the key.

The bug was that the result of the 2nd bullet (query on the 32 bit key on 64 bit machines) was being misinterpreted. The code treated the return value as if a RegQueryValueEx call was made, but the call was to RegOpenKeyEx. Because the return value was not being evaluated correct, step 2) was never performed (querying for the client on the new key) which led to an error when formatting the string that was used to query for the verb.

OK, great. Now at least I knew where the bug was, but I still wanted my mail key to work and I was definitely not going to run my own custom shlwapi.dll on my own machine. It is used by too many components for me to have confidence in making that kind of change. I needed a solution that was as minimal as possible and the least invasive. In the end I ended up attaching windbg and using the assembler (the 'a' command) to remove the incorrect checks. My initial removal was to nop the entire sequence...

0:004> u xxxxxxxxx`xxxxxx6c  (somewhere in the guts of shlwapi.dll)
$ the call whose result will be misinterpreted
xxxxxxxxx`xxxxxx6c ff153e08fcff    call    qword ptr [SHLWAPI!_imp_RegOpenKeyExW (000007fe`fee918b0)]

$ incorrect tests
xxxxxxxxx`xxxxxx72 85c0            test    eax,eax
xxxxxxxxx`xxxxxx74 8bf8            mov     edi,eax
xxxxxxxxx`xxxxxx76 7474            je      xxxxxxxxx`xxxxxxec
xxxxxxxxx`xxxxxx78 3bc3            cmp     eax,ebx
xxxxxxxxx`xxxxxx7a 742c            je      xxxxxxxxx`xxxxxxa8

$ call to query the value
xxxxxxxxx`xxxxxx7c 488b4c2438      mov     rcx,qword ptr [rsp+38h]
xxxxxxxxx`xxxxxx81 4c8d4c2430      lea     r9,[rsp+30h]
xxxxxxxxx`xxxxxx86 4c8d442440      lea     r8,[rsp+40h]
xxxxxxxxx`xxxxxx8b 33d2            xor     edx,edx
xxxxxxxxx`xxxxxx8d c7442430a0000000 mov     dword ptr [rsp+30h],0A0h
xxxxxxxxx`xxxxxx95 ff150d08fcff    call    qword ptr [SHLWAPI!_imp_RegQueryValueW (000007fe`fee918a8)]

$ now nop out the check, always do the query.  Stop @ xxxx7c since that is the start of the query call
0:004> a xxxxxxxxx`xxxxxx72 
xxxxxxxxx`xxxxxx72 nop
xxxxxxxxx`xxxxxx73 nop
xxxxxxxxx`xxxxxx74 nop
xxxxxxxxx`xxxxxx75 nop
xxxxxxxxx`xxxxxx76 nop
xxxxxxxxx`xxxxxx77 nop
xxxxxxxxx`xxxxxx78 nop
xxxxxxxxx`xxxxxx79 nop
xxxxxxxxx`xxxxxx7a nop
xxxxxxxxx`xxxxxx7b nop

$ and dump out the old code again to make sure the right thing was nop'ed
0:004> u xxxxxxxxx`xxxxxx6c 
xxxxxxxxx`xxxxxx6c ff153e08fcff    call    qword ptr [SHLWAPI!_imp_RegOpenKeyExW (000007fe`fee918b0)]
xxxxxxxxx`xxxxxx72 90              nop
xxxxxxxxx`xxxxxx73 90              nop
xxxxxxxxx`xxxxxx74 90              nop
xxxxxxxxx`xxxxxx75 90              nop
xxxxxxxxx`xxxxxx76 90              nop
xxxxxxxxx`xxxxxx77 90              nop
xxxxxxxxx`xxxxxx78 90              nop
xxxxxxxxx`xxxxxx79 90              nop
xxxxxxxxx`xxxxxx7a 90              nop
xxxxxxxxx`xxxxxx7b 90              nop
xxxxxxxxx`xxxxxx7c 488b4c2438      mov     rcx,qword ptr [rsp+38h]
xxxxxxxxx`xxxxxx81 4c8d4c2430      lea     r9,[rsp+30h]
xxxxxxxxx`xxxxxx86 4c8d442440      lea     r8,[rsp+40h]
xxxxxxxxx`xxxxxx8b 33d2            xor     edx,edx
xxxxxxxxx`xxxxxx8d c7442430a0000000 mov     dword ptr [rsp+30h],0A0h
xxxxxxxxx`xxxxxx95 ff150d08fcff    call    qword ptr [SHLWAPI!_imp_RegQueryValueW (000007fe`fee918a8)]

...but my final (and more refined) approach was the actual comparisons in the assembly. After the in place edits, my mail key now works again! The only issue is that I have to reapply every time explorer is started, I can live with that. I also filed a bug for the next release of Windows, so I will not have to make the edits in the future and neither will you ;).

A must have book for any Windows developer

doronh — Wed, 19 Dec 2007 21:21:00 GMT

I saw a book, Advanced Windows Debugging, in the Microsoft company store and quickly read through it. It looked pretty awesome in the level of detail and breadth that it covered. I ordered my own copy and I think it would be an invaluable resource for anyone who develops drivers or applications on Windows. I learn something new everytime I read it.

Changes to !poreqlist

doronh — Fri, 23 Mar 2007 21:47:00 GMT

I posted about !poaction and !poreqlist about a year ago. I tend to use these extensions whenever I am debugging a power related issue in the framework. A few months ago, I ran !poreqlist and got the following output

0: kd> !poreqlist
All active Power Irps from PoRequestPowerIrp
PopReqestedPowerIrpList
FieldOffset = 00000004

Which is not what I expected to see! It appeared to be broken :(. (See the aforementioned previous posting on what the output should have looked like). The output refered to PopRequestedPowerIrpList, so I went into the kernel power manager code to find the variable...but it was gone! Well, that would explain why the extension didn't work :). Just as a sanity check, I looked at the Server 2003 sources and did find this variable, so obviously a change was made for Vista (the related changes are unimportant for this posting).

What I really wanted to know is if I could still get the pending power IRP list in a Vista debug sesssion. After some digging around in the !poreqlist extension's source, I found what I was looking for. !poaction (which is implemented in the same file) now contains the pending IRP list as well as its original output. PopIrpList replaced PopRequestedPowerIrpList. This was executed while the machine was in S0:

0: kd> !poaction
PopAction: 818fc7d8
  State..........: 0 - Idle
  Updates........: 0 
  Action.........: None
  Lightest State.: Unspecified
  Flags..........: 10000003 QueryApps|UIAllowed
  Irp minor......: ??
  System State...: Unspecified
  Hiber Context..: 00000000

Allocated power irps (PopIrpList - 81909a60)
  IRP: 8452b5e8 (wait/wake), PDO: 83f509c8
  IRP: 845dfa98 (wait/wake), PDO: 83f4f4c0
  IRP: 845cd820 (wait/wake), PDO: 8460b6a8

Irp worker threads (PopIrpThreadList - 819094e0)
  THREAD: 83b6bad0 (static)
  THREAD: 83b6bd78 (static)

Threads with non-zero thread execution state attributes:

Debugger command (!list) that makes my life easier

doronh — Thu, 10 Aug 2006 18:06:00 GMT

Yesterday I introduced the dl command and demonstrated some of its limitations. Today I will talk about !list. Let's take yesterday's data structure, MY_DATA. What if the LIST_ENTRY is at the end of the structure or there is more data in your structure that fits into two pointer sized fields so that dl cannot display them? This is where the extension !list comes in very handy.

Getting all the parameters to this extension correct is a bit difficult and it took me quite a bit of experimentation to get it to work, so hopefully I will be able to make it easier to understand and you won't have to go through the same pain I did. First, let's rearrange the fields in MY_DATA so that the LIST_ENTRY is the last field, recompile, and rerun our test with the same breakpoint set on "return 0;" and run dl again to see the changes

    typedef struct _MY_DATA {
        LIST_ENTRY ListEntry;
        ULONG Data;
        ULONG Data2;
        LIST_ENTRY ListEntry;
    } MY_DATA, *PMY_DATA;

    0:000> dl head
    0013fbf0  004d4098 004d4138 00000005 0013fc40
    004d4098  004d40c0 0013fbf0 abababab abababab
    004d40c0  004d40e8 004d4098 abababab abababab
    004d40e8  004d4110 004d40c0 abababab abababab
    004d4110  004d4138 004d40e8 abababab abababab
    004d4138  0013fbf0 004d4110 abababab abababab

As you can see for all rows except for the first, that the data after the LIST_ENTRY is unitialized and the usefulness of the output is much less. So, let's run !list and see what we get

    0:000> !list "-t _LIST_ENTRY.Flink -x \"? @@(#CONTAINING_RECORD(@$extret, _MY_DATA, ListEntry));dt _MY_DATA @@(#CONTAINING_RECORD(@$extret, _MY_DATA, ListEntry))\" @@(head.Flink)"
    Evaluate expression: 5062800 = 004d4090
       +0x000 Data             : 1
       +0x004 Data2            : 0x101
       +0x008 ListEntry        : _LIST_ENTRY [ 0x4d40c0 - 0x13fbf0 ]

    [...]

    Evaluate expression: 5062960 = 004d4130
       +0x000 Data             : 5
       +0x004 Data2            : 0x105
       +0x008 ListEntry        : _LIST_ENTRY [ 0x13fbf0 - 0x4d4110 ]

So now we have all our structure, but there that has got to be one of the worst command lines I have seen in a long time! Let's take it apart and try to make some sense of it.

The entire command line is quoted which takes care of the beginning and ending "s
-t _LIST_ENTRY.Flink tells !list how to move from one entry to the next. !list will evaluate the expression to find the offset of the next list items pointer given the current list item pointer. My first impression was that it would be -t _MY_DATA.ListEntry.Flink, but !list requires an offset relative to the current pointer while _MY_DATA.ListEntry.Flink provides an absolute offset from the beginning of the structure.
-x indicates what command you want to run for each entry in the list. You can optionally specify the -e flag for !list to echo the command, but for this case the echo was completely useless. The -x command also needs to be quoted which is why it is encased in escaped (\") quotes. The command being executed is actually 2 commands (separated by the semicolon).
1. ? @@(#CONTAINING_RECORD(@$extret, _MY_DATA, ListEntry)) will dump the current _MY_DATA pointer value for the given entry. @$extret evaluates to the current item pointer in the list (a PLIST_ENTRY in this case). This value is not necessarily the same as the start of the structure so I use #CONTAINING_RECORD to adjust accordingly (just like it would in real live code). I evaluate the the pointer value because the next command, dt _MY_DATA @@(#CONTAINING_RECORD(@$extret, _MY_DATA, ListEntry)), does not print it out and I like to see what each entry's value is.
2. dt _MY_DATA @@(#CONTAINING_RECORD(@$extret, _MY_DATA, ListEntry)) will dump out the data structure for the given entry the same way it would work if you ran the dt command from the prompt
@@(head.Flink) tells !list where the first entry in the list is. This is not the list head, it is the address of the first entry. This is why I am evaluating getting the value of head.Flink and not &head

Debugger command (dl) that makes my life easier

doronh — Thu, 10 Aug 2006 01:34:00 GMT

The use of the LIST_ENTRY structure in WDM is quite pervasive. It is used for nearly all list keeping tasks. I have used it extensively in the past and KMDF uses it quite a bit as well. There are two debugger commands that help in viewing the contents of a list. I will talk about dl today and !list tomorrow.

Here is an example usage of LIST_ENTRY (ignoring the fact that it leaks memory):

    typedef struct _MY_DATA {
        LIST_ENTRY ListEntry;
        ULONG Data;
        ULONG Data2;
    } MY_DATA, *PMY_DATA;

    int __cdecl main()
    {
        LIST_ENTRY head;
        ULONG i;

        InitializeListHead(&head);

        for (i = 0; i < 5; i++) {
            PMY_DATA pData = (PMY_DATA) LocalAlloc(LPTR, sizeof(MY_DATA));
            if (pData == NULL) {
                return 1;
            }

            pData->Data = i + 1;
            pData->Data2 = 0x100 + i + 1;

            InsertTailList(&head, &pData->ListEntry);
        }

        return 0;
    }

Assuming we have set a breakpoint at "return 0;", you can use the dl command to dump the list contained in the local variable 'head'.

    0:000> dl head
    0024f7d0  00364090 00364130 00000005 0024f820
    00364090  003640b8 0024f7d0 00000001 00000101
    003640b8  003640e0 00364090 00000002 00000102
    003640e0  00364108 003640b8 00000003 00000103
    00364108  00364130 003640e0 00000004 00000104
    00364130  0024f7d0 00364108 00000005 00000105

Let's take the output apart and see what we are told. The first column is the address being dumped. The other four columns are the 4 pointer sized values that the first column points to. Of these four columns, the LIST_ENTRY is contained in the first (Flink) and second (Blink) columns. The remaining two columns are pure data and dependent on your data structure. So, let's annotate the previous output:

    [addr]    [Flink]  [Blink]  [Data]   [Data]
    0024f7d0  00364090 00364130 00000005 0024f820
    00364090  003640b8 0024f7d0 00000001 00000101
    [...]

The first row (0024f7d0 00364090 00364130 00000005 0024f820) is the list head, so the the final 2 column values (00000005 0024f820) should be ignored. The remaining rows are our structure MY_DATA; let's look at its layout in memory:

    0:000> dt _MY_DATA
       +0x000 ListEntry        : _LIST_ENTRY
       +0x008 Data             : Uint4B
       +0x00c Data2            : Uint4B

It is no coincidence that I put Data and Data2 after the ListEntry field in the structure. By placing these values after the ListEntry, the dl command dumped my data structure for me without any additional work. I chose a simple pattern for the Data and Data2 fields to further illustrate this concept. If we look at the first row (00364090 003640b8 0024f7d0 00000001 00000101), the MY_DATA at 0x00364090 has a Data value of 1 and a Data2 value of 0x101. You can also use the dlb command to dump the list backwards if want.

One word of caution though. The output for a 64 bit list is a little different then the 32 bit output. Each entry takes up two lines instead of one! Also, the layout trick that worked nicely for 32 bits does not work as nicely for 64.

    0:000> dl head
    00000000`001ff930  00000000`00383730 00000000`00383870
    00000000`001ff940  00000000`00000000 00000000`ff8213ac
    00000000`00383730  00000000`00383780 00000000`001ff930
    00000000`00383740  00000101`00000001 abababab`abababab
    00000000`00383780  00000000`003837d0 00000000`00383730
    00000000`00383790  00000102`00000002 abababab`abababab
    [...]

I annotated the output to clarify the data. For 64 bits I like to "fix" this issue by specifying how many pointers to dump (two). Unfortunately, to specify the number of pointers to dump you need to specify the maximum number of entries. I use a very large value (100) so that I do not accidentally clip the output

    0:000> dl head
    [addr]             [Flink]           [Blink]
    00000000`001ff930  00000000`00383730 00000000`00383870
    [addr continued]   [Data]            [Data]
    00000000`001ff940  00000000`00000000 00000000`ff8213ac

    [addr]             [Flink]           [Blink]
    00000000`00383730  00000000`00383780 00000000`001ff930
    [addr continued]   [Data]            [Data]
    00000000`00383740  00000101`00000001 abababab`abababab
    [...]

    0:000> dl head 100 2
    00000000`001ff930  00000000`00383730 00000000`00383870
    00000000`00383730  00000000`00383780 00000000`001ff930
    00000000`00383780  00000000`003837d0 00000000`00383730
    [...]

One final note, the dl command does not actually rely on the LIST_ENTRY structure, it can be used for nearly any list type as long as the pointer to the next item in the list can be retrieved dereferencing the current entry's address. Tomorrow's entry on !list will describe how to more robustly dump each entry in the list.

Viewing your KMDF log in a mini-dump (and other post mortem features)

doronh — Thu, 03 Aug 2006 02:03:00 GMT

Your KMDF driver log can also be available in a mini-dump under certain circumstances. If you have a full kernel dump or a full memory dump, the log will be always present (barring any memory corruption or problems writing out the dump file). KMDF will attempt to write your driver's log to the mini-dump if one of the following are true:

KMDF can determine if your driver is the cause of the bugcheck
You added a registry setting which told KMDF to always write your log

As of v1.5, KMDF can determine if your driver is the cause of a bugcheck if one of the bug check parameters contains a pointer which is within the loaded driver's memory range. KMDF could attempt to walk the stack to figure out if the client is on the stack, but KMDF plays it safe. Walking the stack may cause another fault, which could make it impossible to capture a dump (which is just one possible thing that could go wrong). When the following bug check codes occur, KMDF will attempt to determine if your driver was the cause of the bug check:

Code	Value
DRIVER_IRQL_NOT_LESS_OR_EQUAL	0xD1
IRQL_NOT_LESS_OR_EQUAL	0xA
KERNEL_APC_PENDING_DURING_EXIT	0x20
KERNEL_MODE_EXCEPTION_NOT_HANDLED	0x8E
KMODE_EXCEPTION_NOT_HANDLED	0x1E
PAGE_FAULT_IN_NONPAGED_AREA	0x50
SYSTEM_THREAD_EXCEPTION_NOT_HANDLED	0x7E

At first glance, to get your log out of the mini-dump you would think that you could use !wdflogdump to view the log. Unfortunately, you can't use this command because of the way that memory is stored in the mini-dump. Instead, you need to run the !wdfcrashdump command. All the other dump related commands (!wdfsearchpath, !wdftmffile) still work in the mini-dump environment.

If you have a kernel debugger attached, you can also save the log explicitly. The !wdflogsave [DriverName [FileName]] command will save the log for you in a WPP compatible format (which you can open in a trace viewing app like TraceView). If you do not specify [FileName], the log will be written to [DriverName].etl.

You can also force KMDF to always write your driver's log file to the mini-dump. To enable this feature, you must add the following registry value under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\<your driver>\Parameters\WDF. A value of zero (the default) turns the feature off, a non-zero value enables the feature.

    ForceLogsInMiniDump : REG_DWORD

Customizing the KMDF log for your driver

doronh — Wed, 02 Aug 2006 01:55:00 GMT

Yesterday I talked about the KMDF log. The KMDF log is a great tool to debug why a DDI call has failed or diagnose the cause of a bugcheck in your driver. You can customize different attributes of the log so that you can better debug your driver. The customizations available to you are:

The length of the log
The format of the output
Verbose output

When a customization requires you to add or modify a registry value, the value will always be under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\<your driver>\Parameters\WDF. WHDC has a great KMDF tip which describes all of these options as well.

Length of the log

Every KMDF driver has a log associated with it (you cannot turn it off). The default size of the log is one page (as defined by PAGE_SIZE; 4K on x86 and x64 systems, 8K on IA64). Since the log is measured by the size of a memory page it is not possible to say exactly how many entries fit in the log, but one page usually holds around 100 entries. You can increase the size of the log by specifying the number of pages it should be. The limit is currently 10 pages. You should be conservative in the number of pages you specify since these pages will not be available for general use by applications, other drivers, or the OS.

You specify the number of pages with the following value:

    LogPages : REG_DWORD

Format of the log

You can modify the format of the log, just like you can with a WPP log. The format specifiers are intentionally the same so that you can transition from one to the other. The DDK has a good listing of the specifiers here (the article is titled "Trace Message Prefix"). You can set the formatting in two ways. First, you can set the environment variable TRACE_FORMAT_PREFIX. If you use a constant format this is the best way to go since you just set it once and then leave it alone. Second, you can also use the !wdfsettraceprefix comand in wdfkd.dll.

The default formatting changed in KMDF v1.5. The big difference between the two is that for v1.5 the calling function is in the entry, while pre v1.5 the source file and line number are in the entry. The calling function is much more useful to an external developer then the file and line number (which is only available internally) which is why we made the change. (If you use the v1.5 wdfkd.ll with an earlier version of KMDF, you will get the v1.5 formatting.)

KMDF v1.5:          "%7!u!: %!FUNC! - "
KMDF v1.0 and v1.1: "%7!u!: %2!-20.20s! -- "

Verbose output

What you see in the log is not everything that KMDF can potentially log, only the most relevant entries are written to the log. Sometimes this is not enough information though. You can tell KMDF to log everything by turning on verbose logging, but this comes with a caveat. You will get a lot more information, but you may lose the entries you really need since the log might overflow with the verbose entries. I would recommend increasing the log size if you are going to enable verbose logging. When you enable the KMDF verifier on your driver, verbose logging is also enabled by default.

You specify verbose logging with the following registry value. A value of 0x0 (the default) turns verbose logging off, while a non zero value will enable verbose logging.

    VerboseOn : REG_DWORD

How can I view the KMDF log for my driver?

doronh — Tue, 01 Aug 2006 00:52:00 GMT

A lot of developers have a great experience with KMDF, but then they sometimes hit a wall. They add a chunk of code and the drivers start failing because something in the new code was not implemented correctly. The first question that comes to mind is "how do I figure out what went wrong?"

The framework team realized that this would be a very common occurrence so we made it a policy to log (using WPP) every error returned by the framework. WPP is cool technology, but if you don't have an application listening to the logger, the log entries are lost. Furthermore, if there is a crash, it is hard to get at the log itself. To solve these 2 issues, KMDF keeps a history of each driver's log. The history is about 100 entries long (you can specify a longer history using the registry).

Viewing the log

First you need to load the KMDF debugger extension, wdfkd.dll. The debugger packages that you download from WHDC contain a copy of this DLL, but they are usually out of date. The DLL is also shipped as a part of the KMDF distribution and the WDK. You should use distribution/WDK version of the DLL for now until things stabilize and both the debugger and WDK eventually ship with the same DLL. Here are the steps you should follow to view the log:

Load the correct symbols. Set your symbol path for the WDF binaries and make sure that your driver's symbols are loaded.
Load the correct DLL. This means using a fully qualified path name when running the !load command
Set the TMF search path. KMDF v1.5 changed the way you set the TMF search path, so you must issue a different command KMDF v1.0 and v1.1
Execute !wdflogdump

Before we proceed, there are a few generic substitutions that you will have to make when running these commands. Each substitution phrase is in <term>. It is up to you replace these phrases with the appropriate values on your installation.

Phrase	Meaning	Example
<arch>	Architecture of the target	x86, i386, amd64, or ia64
<build-type>	Type of build, either debug or retail	chk or fre
<DDK-root>	The root directory of your DDK/WDK install	C:\WinDDK
<DDK-build>	The build number of your DDK/WDK	3790.1830 (Server 2003 SP1), 2195 (XP), etc
<driver>	Your driver's name without the '.sys' extension	foo if you driver's image name is foo.sys

1) Load the correct symbols

You need to add the DDK/WDK symbol path to your debugger. For V1.0 and V1.1 DDK, the path is <DDK-root>\wdf\kmdf10\symbols\<arch><build-type>\wdf. As an example, the path would be C:\WinDDK\wdf\KMDF10\symbols\x86fre\wdf on my machine when debugging an x86 fre version of KMDF. Currently the beta versions of the WDK are not shipping with the symbols for KMDF, so you must use the symbols for your beta Vista install or use the v1.1 bits.

For V1.0 and V1.1 you will also need to make sure your driver's symbols are loaded since the debugger extension needs to resolve a symbol in your driver. V1.5 removes this requirement and you just need to have KMDF symbols setup properly.

2) Loading the correct wdfkd.dll

To load the correct version of wdfkd.dll you need to specify the fully qualified path to the DLL. For v1.0 and v1.1, it is <DDK-root>\wdf\kmdf10\bin\<arch>, where <arch> in this case is the machine type which is running the debugger (vs the machine being debugged). Here is how I load the extension on my machine:

0:000> !load c:\WinDDK\wdf\KMDF10\bin\x86\wdfkd.dll

To load the DLL from the WDK, the fully qualified path is different, it is <DDK-root><DDK-build>\bin\<arch>. Here is how I load the extension on my machine:

1: kd> !load c:\WinDDK\5461\bin\x86\wdfkd.dll

If you are having issues with which version is being loaded you have a couple of options. The first is to execute the .chain command in the debugger. This will list all loaded extensions. You can look through the list and see if there is more then one version of wdfkd.dll loaded. Second, you can copy the DDK/WDK version of the DLL over the debugger's version of the DLL. If you choose this option, you should make a backup copy of the original DLL.

3) Setting the TMF search path

To set the path for v1.0 , you use the !wdfsearchpath command. The TMF path is <DDK-root>\wdf\kmdf10\symbols\<arch>\<build-type>\wdf\tmf.

For v1.1 and v1.5 the !wdftmffile command should be used instead instead.

v1.1: the TMF file is located at <DDK-root>\wdf\kmdf10\symbols\<arch>\<build-type>\wdf\tmf\wdf010001.tmf.
v1:5: the TMF file is located at <DDK-root><DDK-build>\tools\tracing\<arch>\wdf01005.tmf.

Here are examples of all 3 variations, notice that I am using the fully qualified path name when executing the command and that the command echoes back the path.

V1.0
1: kd> !c:\WinDDK\wdf\KMDF10\bin\x86\wdfkd.wdfsearchpath c:\WinDDK\wdf\KMDF10\symbols\x86fre\wdf\tmf
wdftrace: searchpath is: c:\WinDDK\wdf\KMDF10\symbols\x86fre\wdf\tmf

V1.1
1: kd> !c:\WinDDK\wdf\KMDF10\bin\x86\wdfkd.wdftmffile c:\WinDDK\wdf\KMDF10\symbols\x86fre\wdf\tmf\wdf01001.tmf
Set TMF file name is : 'C:\WinDDK\wdf\KMDF10\symbols\x86fre\wdf\tmf\wdf01001.tmf
'

V1.5
1: kd> !c:\WinDDK\5461\bin\x86\wdfkd.wdftmffile C:\WinDDK\5461\tools\tracing\i386\wdf01005.tmf
Set TMF file name is : 'C:\WinDDK\5461\tools\tracing\i386\wdf01005.tmf'

4) Execute !wdflogdump

This is the same for all versions. The syntax for this command is !wdflogdump <driver>. Here is the output from the osrusbfx2 example using KMDF v1.1 (and again note the fully qualified path for the DDL):

1: kd> !c:\WinDDK\wdf\KMDF10\bin\x86\wdfkd.wdflogdump osrusbfx2

Trace searchpath is: c:\WinDDK\wdf\KMDF10\symbols\x86fre\wdf\tmf

Trace format prefix is: %7!u!: %!FUNC! -
Log at a400b000
Gather log: Please wait, this may take a moment (reading 4032 bytes).
% read so far ... 10, 20, 30, 40, 100
There are 47 log entries
--- start of log ---
1: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x5BFF06B8 !devobj 0xA400F128 entering PnP State WdfDevStatePnpInit from WdfDevStatePnpObjectCreated
2: FxPkgPnp::Dispatch - WDFDEVICE 0x5BFF06B8 !devobj 0xA400F128, IRP_MJ_PNP, 0x00000000(IRP_MN_START_DEVICE) IRP 0x87F7B5D0
[...]

Debugger commands (stack frame navigation) that makes my life easier

doronh — Fri, 21 Jul 2006 18:00:00 GMT

One thing that I have always found clunky is stack frame navigation in windbg/kd. Previously, I thought you had only a couple of options.

The first option, if you are using WinDBG, is that you can bring up the call stack window. I have found that this is not a great thing to do b/c WinDBG will try to update the call stack window while the target is running and not just when you are broken in (perhaps it does this only for module loads, I don't know). Of course, since this is WinDBG dependent, this does not work in kd. As the final straw, this requires moving my hands off the keyboard, losing valuable time navigating with the mouse ;).

The second option is to use the kn command. This will dump the stack with frame numbers, the output looks like this:

1: kd> kn
 # ChildEBP RetAddr
00 81e33c6c 81898d7c nt!RtlpBreakWithStatusInstruction
01 81e33c74 81898d2e nt!KdCheckForDebugBreak+0x22
02 81e33d20 8183ddd5 nt!KeUpdateRunTime+0x270
03 81e33d50 8187dba2 nt!PopIdleDefaultHandler+0x239
04 81e33d54 00000000 nt!KiIdleLoop+0xa

And then you can issue the .frame N command, where N is the frame number to navigate to that frame and start debugging

1: kd> .frame 3
03 81e33d50 8187dba2 nt!PopIdleDefaultHandler+0x239
1: kd> dv
[...]

My problem is that I didn't know how to do frame relative jumps, so every time I wanted to move to a new frame I ran kn again (since I forget the frame numbers), found the new frame and ran .frame N with the new frame number. Well today I saw 2 new ways how to navigate the frames.

The first way will work with the previous debugger releases. You use the pseudo register $frame. Pretty nice trick IMO.

1: kd> .frame @$frame-1
02 81e33d20 8183ddd5 nt!KeUpdateRunTime+0x270
1: kd> .frame @$frame+1
03 81e33d50 8187dba2 nt!PopIdleDefaultHandler+0x239

The second way to navigate frames requires the debugger package that was just released and is a little less cumbersome to use. There are 2 new debugger commands, .f+ and .f-. These are definitely easier to use in my book because I don't have to remember the register deref syntax.

1: kd> .f-
02 81e33d20 8183ddd5 nt!KeUpdateRunTime+0x270
1: kd> .f+
03 81e33d50 8187dba2 nt!PopIdleDefaultHandler+0x239

With these new commands, I think I get now get by without having to lift my hands off the keyboard ;).

New debugger package is now public

doronh — Thu, 20 Jul 2006 06:58:00 GMT

You can download the package from WHDC. It is hard for me to keep track of when they go public, we get internal drops more often and so it is hard for me to know when fixes see the (external) light of day. Enjoy.

Debugger commands (!error, .enable_long_status) that makes my life easier

doronh — Sat, 08 Jul 2006 00:29:00 GMT

One thing you learn very quickly when writing a driver is that NTSTATUS is used almost everywhere. The consistency is nice, especially compared to user mode where errors can be an HRESULT, LONG, or DWORD (yes they are all the same underlying type, but they have different meanings, particularly for success/failure checks). The problem with any error value that is returned is that you must figure out where it came from and what the value maps to in terms of description and error name.

The first simple tool I use is the .enable_long_status command. By default, all long integers are displayed in decimal, but for some error encoding systems, you want hex instead. In this example, status is an NTSTATUS which is set to STATUS_UNSUCCESSFUL. This works for HRESULT as well since all we are doing is changing the format of the number.

0:000> dt status
status = -1073741823
0:000> .enable_long_status 1
0:000> dt status
status = 0xC0000001

I found !error awhile ago and I just refered to it in a comment on another post, so I thought it was a good time to talk about it. !error takes an error value and returns the description for it. As a bonus, it works for win32 error codes as well as NTSTATUS values. Here is the description from the docs:

Syntax
!error Value [Flags]

Parameters:
Value  Specifies one of the following error codes:
         Win32
         Winsock
         NTSTATUS
         NetAPI

Flags If Flags is set to 1, then the error code is read as an NTSTATUS code.

And this is what the output looks like for various values:

Dump out the value as Win32
0:000>  !error 1 
Error code: (Win32) 0x1 (1) - Incorrect function.

Dump out the value as NTSTATUS
0:000> !error 1 1
Error code: (NTSTATUS) 0x1 - STATUS_WAIT_1

Dump out an NTSTATUS value that is not ambiguous with a Win32 value
0:000> !error c0000001
Error code: (NTSTATUS) 0xc0000001 (3221225473) - {Operation Failed}  The requested operation was unsuccessful.

There are others tools and commands that you can use to get error descriptions:

net helpmsg
WPP format specifiers, %!STATUS! for NTSTATUS. I am sure there are other specifiers for other types as well (I just don't know them offhand).
FormatMessage() in user mode
Visual Studio will do some automatic interpretation for you. IIRC, in a watch window you can apped ",err" or ",hr" to the variable name to get better error info.
...and I am sure there others...

Why does my COM port disappear when I enable the kernel debugger?

doronh — Thu, 08 Jun 2006 08:39:00 GMT

A lot of folks are told to connect a kernel debugger (over a serial calbe) to their systems if it is constantly blue screening or if there are suspected issues in the kernel or a loaded driver. Most of these folks do not have the skills to debug the issue themselves, they are just setting up their machine so that once the problem shows up, somebody else can debug it. One thing that happens is that the person whose machine is being debugged looks in device manager and sees one of two things...

The COM that they are using to debug the machine is yellow !'ed out and non functional.
The COM port does not appear in device manager at all.

...and then summarily freak out. Reactions range from wondering if the kernel debugger is working at all (how could it? the port it using appears to be non functional) to anger that the COM port that they wanted to use is now gone.

So what exactly is happening? First and foremost, in both situations the kernel has assumed ownership of the COM port's resources and is using those resources to directly manipulate the port. The kernel is not opening the port by name and using the serial driver (serial.sys) to control the hardware, it is direct manipulation. This has an important side affect...there must be real UART hardware for the kernel to access; a USB to serial adapter will not work on the target side of a kernel debug session because there are no real UART registers to touch! When debugging over a serial cable is enabled, the kernel export KdComPortInUse contains the starting address of the UART resources being used by the kernel debugger.

So why are there different visual results to enabling a kernel debugger? Well, the results depend on if it is an ACPI machine (and chances are, any machine built in the past 5 years is ACPI enabled by default). If ACPI is enabled, ACPI will look at the address in KdComPortInUse and see if a device in the ACPI table starts with that address. If a match is made, ACPI does not even enumerate the COM at all. This is why you don't see a !'ed out device. Looking at ACPI.sys's imports, we see that it does reference the export:

C:\WINDOWS\system32\drivers>link /dump /imports acpi.sys | findstr KdComPortInUse
                   38 KdComPortInUse

The !'ed out device occurs when ACPI is not enabled on the machine. In this case, the BIOS reported COM is enumerated, the serial driver is attached to the stack, and an IRP_MN_START_DEVICE is sent to the stack. In the process of starting the device, serial looks at the assigned resources to the device and if they match KdComPortInUse, it will fail the IRP_MN_START_DEVICE, resulting in a failed start which shows up as a !'ed device in device manager. Since the serial driver is a DDK/WDK sample, we can see what it does:

NTSTATUS
SerialInitController(
    IN PDEVICE_OBJECT PDevObj,
    IN PCONFIG_DATA PConfigData
    )
{
   [...]

#ifdef _WIN64
   BOOLEAN DebugPortInUse = FALSE;

   //
   // First check what type of AddressSpace this port is in. Then check
   // if the debugger is using this port. If it is set DebugPortInUse to TRUE.
   //
   if(PConfigData->AddressSpace == CM_RESOURCE_PORT_MEMORY) {
        PHYSICAL_ADDRESS  KdComPhysical;
        KdComPhysical = MmGetPhysicalAddress(*KdComPortInUse);
        if(KdComPhysical.LowPart == PConfigData->Controller.LowPart) DebugPortInUse = TRUE;
   } else {
        if ((*KdComPortInUse) == (ULongToPtr(PConfigData->Controller.LowPart))) DebugPortInUse = TRUE;
   }

   if (DebugPortInUse) {
      [...]
      return STATUS_INSUFFICIENT_RESOURCES;
   }
#else
   //
   // This compare is done using **untranslated** values since that is what
   // the kernel shoves in regardless of the architecture.
   //
   if ((*KdComPortInUse) == (ULongToPtr(PConfigData->Controller.LowPart))) {
      [...]
      return STATUS_INSUFFICIENT_RESOURCES;
   }
}

Previous command completion in WinDBG

doronh — Tue, 23 May 2006 02:24:00 GMT

One of the features I like about kd is that since you are using a console window, you get a lot of the console functionality for free. The 2 features that I really like are tab (err, F8) completion and the listing of command history (F7). On the other hand, windbg has a lot going for it (ignoring the inflamatory topic of docking windows :P ) like source browsing and easier stack frame navigation, but the one thing I miss the most in windbg is command completion. Sure you have command history like kd, but who likes to scroll through 10s of command to get the command you executed 5 minutes ago?

Well, windbg does have command completion! The problem is that it's hidden. The only reason I know about it is I read every line of the release notes for the debugger drop one week. As a lark, I tried searching debugger.chm for the phrace "command completion" and got 114 hits. Needless to say I didn't have patience to search all 114 to see if one of them actually documented this feature or not, so I leave it up to you to see if you can find a topic about this functionality in the help. Windbg's equivalent of kd's F8 is SHIFT+UP ARROW.

The functionality between kd's (really cmd.exe's) and windbg is nearly the same. They differ when there is no match for what has been typed. With kd, nothing occurs. On the other hand, with windbg, the text in the command window is selected. Furthermore, once you have reached the end of the history, windbg will not wrap back to the beginning (where kd will wrap). I have found the differences only matter when you are not paying attention to what you are typing :).

Here's a demo of the command at work. Please note that the output of !wdfrequest and !wdfmemory is from a new version of wdfkd (which will be in the WDK). Let's say I run these 3 commands:

kd> !WDFREQUEST 0x7df89888
   !IRP 0x82b6af20
   !WDFQUEUE 0x7e8fdb40
   State:  Pending, Allocated by WDF for incoming IRP
   System Buffer 0x817011a8, Length 0x24, !WDFMEMORY 0x7df89821
   IOCTL Output Buffer 0x817011a8, Length 0x10, !WDFMEMORY 0x7df89819

kd> !WDFMEMORY 0x7df89821
WDFMEMORY 0x7df89821:  Buffer 0x817011a8, Length 0x24 (36) bytes

kd> !WDFMEMORY 0x7df89819
WDFMEMORY 0x7df89819:  Buffer 0x817011a8, Length 0x10 (16) bytes

Now at this point at the prompt kd>, I type "!WDFR", so we have
kd> !WDFR

and if I know hold down the SHIFT key and then press the up arrow key once, 
the prompt will look like
kd> !WDFREQUEST 0x7df89888