driver writing != bus driving : windbg

KMDF Debugging Videos

iliast — Tue, 22 Sep 2009 22:29:00 GMT

A while ago, I wrote a blog post about our UMDF debugging videos, which were created by my teammate Abhishek.

Now, I'm really excited to announce that we've released KMDF debugging videos, which can be found at http://www.microsoft.com/whdc/devtools/debugging/kmdf.mspx. These videos were created by my teammate Kumar. They present 3 important aspects of KMDF debugging:

1) How to debug the KMDF log

2) How to get information about a driver and its objects

3) How to dump all the devices and queues

UMDF Debugging Videos

iliast — Wed, 10 Jun 2009 04:29:00 GMT

UPDATE: You can find my post about KMDF debugging videos at http://blogs.msdn.com/iliast/archive/2009/09/22/kmdf-debugging-videos.aspx

A while back I had written that we didn't have any videos, where we talk about UMDF (KMDF seems to get all the glory :P). Well, now we do!

My teammate, Abhishek, has created a series of tutorials that describe how to debug UMDF drivers. They can be found at http://www.microsoft.com/whdc/devtools/debugging/umdftraining.mspx. I've been through all the videos a few times and I think that they are very helpful and present some very nice tricks. I suggest that you install a UMDF driver and go through the same tasks that Abhishek is describing, in order to understand the tutorials better.

Feel free to give me any feedback (or any thank-you comments) that you want me to tell Abhishek :)

Viewing WDF Logs In Windbg

iliast — Wed, 10 Jun 2009 03:44:00 GMT

One feature that is really helpful in debugging WDF drivers is the log file that is created by the frameworks themselves. In the log files you can see many warnings and errors that are created by the framework (i.e. they come for free and the driver does not have to do anything). Did you ever have a problem trying to understand why a call to a WDF function fails or what the framework is doing under the hood? Then, continue reading:

In this post I'll explain how to look at the framework log files, while you're debugging a driver using windbg. I assume that you have already installed the WDK in the directory %winddk%.

So, let's start with UMDF:

You need to debug the wudfhost application that hosts your driver. This is described in an earlier post of mine.
In windbg execute the command "!wmitrace.searchpath %winddk%\tools\tracing\%arch%", e.g. "!wmitrace.searchpath c:\WinDDK\6001\tools\tracing\x86". The directory that you use should have files with names wdf01007.tmf, wdf01009.tmf, etc.
Execute the command "!wmitrace.strdump" and find the number that corresponds to "WUDF Trace". Let's say that this number is 0x11.
Execute the command "!wmitrace.logdump number_from_previous_step", e.g. "!wmitrace.logdump 0x11"
In order to control the verbosity of the output, you can use WdfVerifier, which can be found at %winddk%\tools\wdf\%arch%\wdfverifier.exe. Select the tab "User Mode Driver Settings" and change the tracing level. Also, enable the option "Send Log Output to Kernel Debugger". These options are global (i.e. they will be applied to all UMDF drivers)

For KMDF, things are easier:

Load wdfkd in windbg. This file is located at %winddk%\bin\%arch%. In order to load it execute "!load %winddk%\bin\%arch%\wdfkd.dll", e.g. "!load c:\WinDDK\6001\bin\x86\wdfkd.dll"
Execute "!wdftmffile %winddk%\tools\tracing\<arch>\wdf01009.tmf", e.g. "!wdftmffile c:\WinDDK\tools\tracing\x86\wdf01009.tmf". Make sure that the file wdf01009.tmf is in that directory. If you are debugging a KMDF 1.7 driver, then you need to use the file wdf01007.tmf, etc.
Execute "!wdflogdump my_driver" to see the log for your driver. For example, if you are debugging the echo driver, execute "!wdflogdump echo".
In order to control the verbosity, you can use WdfVerifier. Select the "Kernel Mode User Driver Settings" tab, select your driver in the left panel and then either select or de-select the option "Enable verbose logging". This option is per-driver, i.e. if you want to enable verbose logging for multiple drivers, then you need to select all of them in the left panel.

Channel9 Video on Debugging BSODs

iliast — Sun, 14 Sep 2008 22:33:00 GMT

I found a very interesting channel9 video on how to debug BSODs (Blue Screens of Death): http://channel9.msdn.com/posts/Dan/Daniel-Pearson-Debugging-a-Windows-Blue-Screen-of-Death/

Daniel Pearson explains why blue screens happen in Windows, how a user can find the cause and how Driver Verifier works. Also, in the end he talks a little bit about how to debug applications and he shows a neat trick with notepad :)

Crash Dump Analysis

iliast — Tue, 12 Dec 2006 05:43:00 GMT

I'm sure that many of you have had the unfortunate experience of watching the windows Blue Screen Of Death (BSOD) while working, and possibly have lost important data. A common reaction in this case is to blame Microsoft and continue working after the following reboot, as if nothing had happened. Another unfortunate experience is to see an application crash, while using it. In this case, there is a window that comes up, asking you, if you want to send the data to Microsoft for analysis (the same window also comes up after the BSOD). Many people might be afraid of the contents of the data that is sent to the network, so they select "No". The goal of this post is to help you understand what is going on in the background in each of the two cases and also to help clarify some misconceptions.

QUESTION 1: What causes all these reboots?

First of all, because of the architecture of the windows kernel in the NT/2000/XP/Vista series, an application cannot corrupt data that belongs to another application or to the kernel. This means that each application is totally isolated and cannot harm the system. The worst thing that can happen is that the application does something invalid and crashes without any further implications for the rest of the system. On the other hand, the windows kernel and the device drivers have unlimited access to the system. If the kernel or a driver misbehaves, then it can corrupt the whole system. The immediate result of this, is that the reason for all the blue screens lies either in the windows kernel or in the windows device drivers. That's why, whenever an application crashes, the system keeps working without a problem, whereas if there is a bug in the kernel or in a device driver, the whole system goes down.

Now that we've identified the possible causes of the crashes, it's time to go even further. According to the reports that were sent to Microsoft until April 2004 (from all those people, who pressed "Yes", when they were asked to send the data to Microsoft) the reasons for the crashes can be split as follows:

Third-party device drivers: 70%
Unknown, because of severe memory corruption: 15%
Hardware error: 10%
Microsoft code: 5%

This shows that Microsoft is not the one to blame. The main cause for these crashes is poorly written third-party (non-Microsoft) device drivers.

QUESTION 2: Why does the system crash, when there is a kernel-mode error?

So, from the above analysis it is obvious that the system can overcome an application crash, however a kernel-mode error causes it to reboot. Why can't the system continue, after finding a kernel-mode error? Actually, what happens is that while kernel-mode code (either a device driver or the windows kernel) is executing, some discrepancy is found. For example, a pointer might be pointing to an invalid address, a data structure might have invalid values, etc. Even though this problem was found, it's possible that the "bad" code that caused this problem might have corrupted more data. For example, it might be possible that basic kernel structures are corrupted. That's why the function KeBugCheckEx is called (inside the kernel, this type of forced crash is called "bugcheck"), in order to write logging information to the page file, paint the blue screen and show some information about the crash in the screen.

QUESTION 3: How do we configure, whether we want to send anything to Microsoft?

Even though, it's not well-known to most people, you can configure what will be sent to Microsoft. Somebody might want to report only the crashes that have to do with the operating system (after the BSOD). Somebody else might want to report only the crashes from the applications (either for all of them or only for some particular applications). In order to configure this, you need to go to

Control Panel | System | Advanced | Error Reporting

From there you'll be able to select which types of errors you want to report. Actually, even if you select a particular type of error to be reported, Windows will ask you again, after a program that belongs in that category crashes. So, by selecting a category, it doesn't mean that all crashes will be reported automatically.

QUESTION 4: Exactly what kind of data is sent to Microsoft?

Before answering this question, it is useful to understand what information is stored, when the system or an application crashes. Let's talk first about the files that are generated after a system crash. In order to set this, you need to go to

Control Panel | System | Advanced | Startup and Recovery -> Settings

In the "System Failure" part you'll be able to configure, if you want to write the crash to the system log, if you want to send an administrative alert and if you want to reboot after the crash. Also, you have the option of creating a memory dump and you can select the directory, in which you want to save it. There are 3 types of memory dumps:

Small memory dump or minidump (64kb for 32-bit systems, 128kb for 64-bit systems): It includes a minimum amount of information about the system before the crash, e.g. the bugcheck code, the loaded drivers, information for the current process and thread, etc.
Kernel memory dump: This includes all the kernel-mode memory that was in physical memory at the time of the crash. There is no default size for this dump, however it should be around 50-100MB for most "normal" systems (with <= 2GB of RAM).
Full memory dump: This includes all the physical memory. The size of the file will be the same as the physical memory of the system.

Here it's worth mentioning that at the time of the crash, the information is written to the pagefile. Therefore, the pagefile must be configured to be larger than the size of the dump. This might be a problem especially in the case of the Full memory dump. In order to set the size of the pagefile, you need to go to

Control Panel | System | Advanced | Performance Settings -> Advanced | Virtual Memory -> Change

After the system reboots, the information is copied from the page file to the file that was specified above. The reason that the information is not written directly to that file is that the kernel doesn't know the root of the problem at the time of the crash, so it's trying to use as fewer drivers as possible (the pagefile is already open and in use, so theoritically it's the safest destination).

On the other hand, in order to configure the data that is stored, when an application crashes, you need to open a command prompt (or go to Start | Run) and execute drwtsn32.exe. This application is called Dr. Watson and there you'll be able to configure the destination file for the dump, as well as the type of the dump. Here the only options are the minidump and the full memory dump. There is also an option to create an old-style NT-compatible full memory dump. In addition, in the textarea "Application Errors" you can look at the application crashes that had been logged in the system. You can select any of them and click "View", if you are interested in looking at exactly what the log file includes.

So, now it's time to answer the initial question: What exactly is sent to Microsoft? The answer is that regardless of the crash dump file that you have selected, the only thing that is sent to Microsoft is a minidump (both for kernel-mode and user-mode crashes). Of course, it would be impossible to send a 100MB kernel-mode dump or a 1GB full-memory dump, so that's why only the 64kb minidump is sent. Apart from the minidump, the information also includes an XML file with basic information about the version of the operating system and the loaded drivers, which you can look at, when you are prompted to select, if you want to send the data to Microsoft. There is no personal private information or anything like that. In fact, you can open the minidumps and check the included information. I'll also show a way of analyzing the minidumps and looking at the data that Microsoft has access to.

QUESTION 5: What does Microsoft do with this information?

When the minidump is received by Microsoft, it goes through some preprocessing and is stored in a server. If many minidumps that seem to have the same problem are received, then there is a team that analyzes them and finds the root of the problem. Afterwards, a webpage is created that shows exactly what the cause of the problem is. Most often it points to the driver causing the problem and gives a link to the manufacturer's webpage, so that a new version can be downloaded (if it exists). After the webpage is created, if somebody submits a dump that has the same problem, he is shown the corresponding webpage that will help him find the solution. Therefore, if somebody clicks "Yes", when asked to submit a crash dump he might either find the solution to the problem or help Microsoft find the solution and present it to the users in the future.

QUESTION 6: How can we analyze a crash dump?

Fortunately, there is a tool that can be used to analyze both the user-mode and the kernel-mode crash dumps: windbg. Microsoft has included the dump analysis algorithms in windbg, so in some basic cases it's easy to find the cause of the problem. Of course, there are many causes, in which there are many corrupted data structures and it's impossible to pinpoint the problem automatically. In that case, more advanced manual methods are used by the Online Crash Analysis (OCA) team in Microsoft.

In order to perform the analysis by yourselves (either because you are unable to submit the data or because there is no answer in Microsoft's website), you need to open windbg, go to "File" | "Open Crash Dump" and select the dump file that you want to analyze. As I wrote in my previous post you need to set the path to the symbol files and reload them. If this step is omitted, then it won't be possible to analyze the file.

The next step is to execute the command (this might take some time):

!analyze -v

At the top of the output you'll see the bugcheck code, it's description and some additional information (e.g. the address of the invalid memory that was accessed, whether it was a Read or a Write operation, etc). Further down you'll see the call stack of the dump under the title STACK_TEXT. This includes all the functions that were called, when the crash occurred. The function on the top is the most current one (it was called by the function below it, which was called by the function below it, etc), whereas the function at the bottom is the oldest function in the stack. The reason that the system crashed is because one of these functions did something invalid (e.g. passed or received an invalid argument that forced it to perform an invalid operation). Of course it's possible that the data was corrupted by another function that is not in the call stack. Fortunately, windbg has already done an automated analysis and points to the module that most probably caused the crash. You should look at the following fields:

SYMBOL_NAME: Exactly where the invalid operation was caused (module + function)
MODULE_NAME: The name of the module that caused the crash
IMAGE_NAME: The file, in which the problematic code resides

In order to find more information about the problematic code you can execute:

lm kv m MODULE_NAME*

for example, if MODULE_NAME is problematic_driver, you should execute:

lm kv m problematic_driver*

lm stands for "list modules", k stands for "kernel modules", v stands for verbose and m stands for "match".

Another option is to find the problematic file name, from the IMAGE_NAME tag, search it in the hard drive and either look at its properties, in order to identify its manufacturer or search it in the internet. Afterwards, you might need to update the buggy driver.

Of course, it's possible that windbg's automatic analysis was not able to pinpoint the faulting driver. The reason for that might be that the call stack was corrupted or that some important code or data structure was overwritten or that there was a memory leak, etc. In that case, you might want to proceed into some manual solutions, in order to detect the problem. From this point there is no automated way to proceed, so I can just provide a few useful commands that might help you find more information about your system.

First you can execute

!process 0 0

!process

The first command prints information about all the running processes. this way you might be able to find a suspicious process. The second command shows information about the current process. If you execute

!process <process_address>

then you'll see more information about the particular process. You can find the addresses of the processes from the "!process 0 0" command. The process information includes information about its threads. You can find more information about each thread by executing

!thread <thread_address>

and if the thread belongs to a driver with pending IRPs, then you can find more information about them by executing

!irp <irp_address>

Also, as I wrote above, in order to see all the loaded modules you can try

lm kv

In order to find more information about the used memory (and possibly detect memory leaks), you can execute:

!vm and !memusage

Finally, it's possible that the system hangs and does not crash. In order to debug it, you need to force a crash. The only way to do that is to go to the registry key HKLM\System\CurrentControlSet\Services\i8042prt\Parameters\CrashOnCtrlScroll and set it to 1. This works only for PS2 keyboards (not for USB). When the system hangs, you can keep the right control key pressed and press the scroll lock key twice. This will cause a crash, which you will be able to debug using windbg.

A useful command in that case is:

!locks (prints the locks, which are currently held, provided that there is at least one additional thread waiting on them).

Another tool that can help you, if !analyze cannot find the root case is the Driver Verifier. This tool enables additional system checks, that will make it possible for the system to crash immediately, when a driver does something invalid, without allowing it to corrupt more data. This way, the crash dump will point directly to the driver. In order to execute it, you need to open a command prompt with administrative privileges (or from Start | Run) and execute "verifier.exe".

There are some small differences between the User Interface in Windows 2000, Windows XP and Vista, so I'll explain the Windows XP interface. What you'll see is a window and some tasks that you're called to select. You need to select the task "Create custom settings (for code developers)", and then "Select individual settings from a full list". After that you'll see a screen with the following options:

Special pool: This option forces the memory allocation routines to operate on a special pool. For example, if a driver wants to allocate 100kb, then he is given a pointer that points to 100kb before the end of a free page. The rest of the page is marked with a specific signature. Also, the pages that are before and after this particular page are marked invalid. So, if a driver tries to write something after the end of the allocated space, there will be a page fault and the Driver Verifier will crash the system immediately. If the driver tries to write before the beginning of the allocated space, then after he frees the memory, the Driver Verifier will check the signature of the page, find that it's invalid and crash the system. The crash dump that will be generated from this crash will point directly to the faulting driver.
Pool tracking: Each space allocation is marked with a special tag that is different for each driver. When the driver is unloaded, the Driver Verifier will check for the corresponding tags and if it finds any, then this means that the driver has a memory leak, so the system will crash.
Force IRQL checking: Whenever a driver goes to IRQL at DPC/dispatch level or above, the Driver Verifier will cause all the pageable memory to be paged out to disk. So, if the driver tries to access this memory, then the system will crash with a bugcheck code equal to IRQL_NOT_LESS_OR_EQUAL
I/O verification: All the IRPs are allocated from a special pool. If any of them is completed with an invalid I/O status, then the system crashes.
Enhanced I/O verification (used to be "I/O Verification level 2" in Windows 2000): This includes even deeper tests for the IRPs. The I/O manager checks if the drivers complete asynchronous IRPs complete correctly, if they manage the device stack locations correctly and if they delete the device objects only once.
DMA checking: The I/O manager makes sure that all the drivers configure the DMA operations correctly, otherwise it crashes the system.
Deadlock detection: This option enables deadlock detection. When a deadlock is detected, the system crashes and you can use the !deadlock command from windbg to find more information about what is causing it.
Low Resources simulation: 7 minutes after the boot completes (so all the drivers have been loaded), the I/O manager starts failing random memory allocations. This way, if a driver doesn't check the status of a memory allocation, the system will crash.
Disk Integrity Verification (only in Windows Server 2003): Windows keeps checksums of written data, so after each read it checks, if the data is still valid. If there is a discrepancy, the system crashes.
SCSI Verification (included automatically, when a SCSI miniport driver is monitored): It includes some additional SCSI-related checks.

You should select all of the existing options, apart from the "Low Resources Simulation" (because it includes very heavy operations and the system will be really slow). In the next screen, you need to select the drivers that you want to debug. You should start by selecting drivers that you think that are suspicious (either because windbg pointed to them or because the crashes started after you installed them, etc). Reboot and run the system for some time and observe its behaviour. If the system continues crashing, but not because of the drivers that you specified (i.e. the crash dump files are still vague and don't pinpoint to a specific driver), the next step is to select all the unsigned drivers. If you don't see any result, then the next step is to start enabling driver verifier for bigger groups of drivers, until you find the buggy one.

Another useful tool for debugging memory leaks is poolmon, which is part of the Windows Support Tools and can be found here (for Windows XP). This tool displays information about memory allocations (both from paged pool and from non-paged pool), as well as discrepancies between allocations and deallocations. You just execute the tool from a command prompt (there is no User Interface) and select the types of memory pool that you want to look at. If the amount of allocated memory increases constantly, this means that there is a possible memory leak. You can find an overview here and a more detailed explanation here.

ADDITIONAL RESOURCES:

This tutorial provides an introductory overview of the same concepts.
Mark Russinovich presented this video in TechEd 2006. The corresponding slides are here.
OSR has a more advanced article on the next steps that you can follow, in order to analyze more difficult crash dumps.

Windbg Tutorials

iliast — Mon, 11 Dec 2006 10:27:00 GMT

The debugger is always a very helpful tool for a developer. In this post I'll present windbg. This tool works both as a user-mode debugger (in order to debug user applications) and as a kernel-mode debugger (in order to debug windows drivers). It's not as fancy and heavy as the Visual Studio debugger, however it has the advantage that it's really small and extensible using debugger extension dlls. Besides, since SoftIce has been discontinued, currently windbg is the only windows kernel-mode debugger available (The Visual Studio debugger is only a user-mode debugger. Also, kd is just a non-graphical frontend to the same core functions that windbg supplies).

STEP 1: Installation

The first step for somebody interested in windows debugging is to download windgb from this link. The default installation path is "C:\program files\Debugging Tools for Windows". In order to use windbg you need to execute the file C:\program files\Debugging Tools for Windows\windbg.exe

STEP 2: Setting the symbol file path

After executing windgb, you need to set the path to the symbol files. This path points to the directories with the pdb files of your drivers. You can have different directories by seperating them with a semicolon (;). You also need to point to the corresponding pdb files of all the windows components, if you want the call stacks that you'll see to include the functions from the components that are developed by Microsoft. The problem in this case is that the windows pdb files change between service packs, hotfixes, etc. Fortunately, Microsoft has configured a symbol server, which can be used to download the needed files on-demand. This means that you just set the symbol path to the symbol server and windbg downloads only the pdb files that it needs. As it is mentioned here, in order to do this, you need to add an entry to the windbg source path that's equal to:

SRV*DownstreamStore*http://msdl.microsoft.com/download/symbols

In the above line, http://msdl.microsoft.com/download/symbols is the symbol server (so you don't need to modify this), and DownstreamStore is the path, where you want the pdb files to be downloaded. This needs to be substituted by a local directory, e.g. c:\symbols, so the complete entry would be

SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

Finally, as I said above, if you have additional pdb files for the drivers that you are developing in directories c:\drivers1, c:\drivers1\misc and d:\drivers2, the complete symbol path would be:

SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols;c:\drivers1;c:\drivers1\misc;d:\drivers2

In order to set this line, you need to open windbg, go to "File", then click at "Symbol File Path" and paste the line in the textarea. Also, as it can be seen from the above example, the directories in the path aren't recursive, so if you have pdb files both in c:\drivers1 and c:\drivers1\misc, then you need to include both of them, since the format doesn't imply the latter.

If you want to observe the process of symbol matching, you can enable verbose symbol matching by executing the command

!sym noisy

Afterwards, you can force windbg to reload the symbols by typing

.reload /f

and look at the output, in order to identify possible problems.

STEP 3: Setting the source file path

After setting the symbol file path, you might want to set the source file path, which points to your driver source files. This way you'll be able to do source level debugging instead of assembly-level debugging. In order to do that, you need to go to "File" and then "Source File Path".

STEP 4: Find a good command reference list

The next step is to find a good list of windbg commands. Actually, windbg supports a huge list of commands and there are many lists that you'll find in the internet. I am using the one that I found in this website. Until you get familiar with the list, it might be helpful to print it and keep it next to you while debugging. Fortunately, after playing for a while with windbg, you'll be able to remember the most common ones and learn where to find information about the rest.

STEP 5: Configuring the working environment

One important feature for debugging is the usage of breakpoints, which stop the normal flow of execution. So, if you set a breakpoint, while debugging an application, then the application will stop executing, when it hits it. This way you'll be able to debug. However, if you wanted to set a breakpoint in the windows kernel and the kernel hit the breakpoint, then the whole system would freeze, because kernel controls (among other stuff) process management, thread switching, etc. Therefore, even though it's possible to perform restricted kernel debugging (no breakpoints and some other limitations) using windbg with only one computer (by going to "File", "Kernel Debug", "Local"), the true power of kernel debugging can be seen, if you have 2 computer.

So, let's say that you have "Computer1", which is the machine that has the drivers that you want to debug, and "Computer2", which is the computer that will be running windbg. First of all, you need to connect these two computers using a serial cable or a USB cable or Firewire (IEEE 1394). After that, you need to go to Computer2, start windbg, go to "File", "Kernel Debug" and select COM or USB or 1394 from the top tab according to the connection that you have chosen. You need to fill the corresponding tabs (e.g. for COM you need to specify the baud rate, which would most probably be 115200 and the COM port, which would be COM1 or COM2). After that you press "ok" and windbg will wait, until you configure "Computer1" to go into debug mode.

The configuration for "Computer1" depends on its Operating System. So, if you are running any Windows version prior to Windows Vista, you need to open the file "C:\boot.ini" with an editor (it's a hidden system file, so you might need to configure your system to show the system files). The file might look something like this:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect

What you need to do is either add "/debug /debugport=COM1 /baudrate=115200" after the end of the last line, in order to make it equal to

"multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect /debug /debugport=COM1 /baudrate=115200"

(but you need to keep in might that this way your system will always start in debug mode) or create a second entry by modifying boot.ini to look like:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional Debug Mode" /fastdetect /debug /debugport=COM1 /baudrate=115200

(so, when you boot the system, you'll see a menu and you'll be able to select whether you want to start in debug mode or not). All the above applies for serial debugging (which is the most popular method).

On the other hand, if you are using Windows Vista (or later), you have to use bcdedit. You need to open a command prompt as an Administrator (using runas) and execute bcdedit. The format of the command is:

bcdedit /dbgsettings DebugType [debugport:Port] [baudrate:Baud]

So, for serial debugging (using COM1 at 115200bps) it is:

bcdedit /dbgsettings serial debugport:1 baudrate:115200

For 1394 (using channel 32) it is:

bcdedit /dbgsettings 1394 CHANNEL:32

For USB (using "debugging" as the target name) it is:

bcdedit /dbgsettings USB targetname:debugging

After that, you need to enable debugging by typing:

bcdedit /debug on

In order to disable it later, you can type:

bcdedit /debug off

If you want more information about bcdedit's debug options, you can look at the following OSR article.

After that you need to restart the computer and if everything was configured correctly, you will see Computer1 stop during boot, and Computer2's windbg will print a message saying that the connection was established.

STEP 6: Becoming familiar with windbg

After that, it's time to start playing with windbg. Since there are many online tutorials that cover this part, I won't try to reinvent the wheel by writing an additional one, but I would like to provide some pointers. Keep in mind that most of the tutorials in the internet don't require a second computer, so you can just use local kernel debugging, in case you don't have a second computer.

I think that one of the best tutorial to start with (both because it is very clear to understand and because it explains some terms in more depth than others) is the one that is supplied with windbg. It is located in the directory, where you installed windbg, and it is called kernel_debugging_tutorial.doc. This file explains both fundamental and more advanced debugging techniques by examining the sample driver IoCtl, which can be found at the WDK. If you don't have the WDK, then you can use any other driver, for which you have the source code, since the concepts are similar. This tutorial needs two computers, however even if you have only one, it would be useful to go through it and look at the commands and the different methods.

A couple more introductory tutorials are the codeproject Windbg tutorial and Mike Taulty's tutorial. Both include a list of common commands, but the second one also goes step-by-step through a debug session (windbg is used as a user-mode debugger). Another list of common windbg commads can be found here,

After that, you can proceed to some more advanced debugging concepts by looking at the debuging tutorials in codeproject and most specifically:

Debug Tutorial Part 1: Beginning Debugging Using CDB and NTSD
Debug Tutorial Part 2: The Stack
Debug Tutorial Part 3: The Heap
Debug Tutorial Part 4: Writing WINDBG Extensions
Debug Tutorial Part 5: Handle Leaks
Debug Tutorial Part 6: Navigating The Kernel Debugger
Debug Tutorial Part 7: Locks and Synchronization Objects

A very good list of tutorials, which includes most of the above ones and some additional ones can be found here. In addition, for those, who prefer watching videos than reading tutorials, Mike Taulty has a list of debugging videos here. Both these blogs provide useful debugging information.

In addition, the 2 blogs below cover more advanced debugging topics:

Ken Johnson's blog: http://www.nynaeve.net
Oleg Starodumov's blog: http://www.debuginfo.com

Another very useful source of information is windbg's help file. Its filename is debugger.chm and is installed in the same directory as windbg's executable. It is an easy-to-read file, even though it might seem overwhelming in the beginning.

Finally, for additional information, you can look at OSR's windbg list (you need to register for free) and at the microsoft.public.windbg newsgroup.