driver writing != bus driving : Windows

Driver Developer Conference - DDC 2008

iliast — Fri, 26 Sep 2008 08:26:00 GMT

Next week (9/29 - 10/1) we'll have the Microsoft Windows Driver Developer Conference (DDC) in Redmond. We're expecting several driver developers. I'm excited about this, not only because it's the first time that I'll be participating in a Microsoft conference, but also because I'll be giving a talk! Indeed, Bob Kjelgaard and I are giving a talk titled "Packaging and Deploying KMDF and UMDF drivers". Actually, a better name for the talk would be "Everything technical that you ever wanted to know about the WDF coinstallers and were afraid to ask" (yes, I actually requested to have this title, however it didn't go through :) ). I think that right now the information that's externally available for the coinstallers is pretty limited. Actually, apart from the fact that "they are coinstallers and you need to have them with all your WDF drivers" and a few posts that Bob and I have written, I don't think that anything else is known about them. In our talk we'll try to show lots of light into this part of WDF.

The last several days we've been practising our talk and I'm pretty happy about it. There is enough technical information in the talk to explain all the common questions that people have and also there will be enough time in the end to answer all sorts of questions that might arise from the presentation. Also, we'll both be available to answer questions in the "Ask The Experts" session, in the panel discussion for WDF and of course we'll be talking to customers and partners in the corridors and in the halls, in order to answer the questions that they might have. I've been teasing Bob by telling him that I'll forward all the hard questions to him, however I've been really happy to work with Bob for such a long time. He's definately a really technical person and he's explaining all these advanced techniques of testing different parts of the framework that blow me away :)

Apart from our talk, there will be a series of presentations for WDF. I've actually sat through all the dry runs of the talks and I think that we have excellent material to present (yes, I know that it's kind of weird to praise the team that I'm part of, but I really believe it). I won't try to write down all the WDF presentations, however you can look at the agenda and the descriptions.

So, for those of you, who are coming to DDC and are interested in the WDF coinstallers, please come with all your questions and we'll be happy to answer them as best as we can!

Code Quality: Windows vs Linux vs FreeBSD vs Solaris

iliast — Sat, 17 May 2008 06:06:00 GMT

Diomidis Spinellis has written a good paper for the “30^th International Conference on Software Engineering” (ICSE ’08) that looks at the code quality of the source codes of Windows (WRK – Research Kernel based on Windows Server 2003), Linux, Solaris and FreeBSD. Diomidis has analyzed the source codes of these 4 kernels and uses some code metrics, in order to measure the quality of each kernel in each area.

For those, who don’t like reading lengthy texts, a summary of the results is at the end (section 5: summary and discussion). Each operating system has its own strengths and weaknesses, so there is no clear winner.

The paper can be found at http://www.spinellis.gr/pubs/conf/2008-ICSE-4kernel/html/Spi08b.pdf and all the queries for the code analysis can be found at http://www.spinellis.gr/sw/4kernel/.

Finally, Diomidis has a very lengthy list of classic reads at his website.

Developing Windows Drivers With Visual Studio

iliast — Wed, 30 Jan 2008 01:44:00 GMT

Today morning I received an email from Patrick with a picture of Visual Studio with Intellisense on a WDF driver. Ok, I have to admit that in the beginning I thought that Patrick was using Photoshop! He's a guy, who just doesn't like GUIs in the first place! He can just go on and on about the advantages of using kd instead of windbg! He even refuses to look at the source code (even if it's available), since "he can look at the assembly"! I have to admit that until now I thought that he was doing "copy con driver.c" to write drivers... and then use edit.com from a full-screen command prompt to do additional editing. Anyway, it seems that I was wrong.

So, in order to be sure that Patrick hadn't used Photoshop (or just opened the screenshot with a hex editor and manually edited it, in order to add Intellisense!), I went to his office. Over there, indeed he showed me that it's really easy to Intellisense and help integration to Visual Studio (yay!). That's nice stuff! My next question was, if it's possible to load wudfext.dll (the UMDF windbg extension) from Visual Studio and use the UMDF extensions from the VS debugger, however it's not possible (even though there's some info on how to load windbg extensions at http://www.osronline.com/showThread.cfm?link=66160), because not all windbg interfaces are implemented by Visual Studio.

Anyway, Patrick has written a post saying that indeed he's been using Visual Studio for ever, in order to develop Windows Drivers. If more people ask him tips on how to make VS the best editor for Windows Drivers, he promised that he'll share a small part of his infinite wisdom :) So, feel free to ask him and/or send him an email!

DISCLAIMER: Visual Studio is not officially supported by Microsoft for driver development. The only currently supported build environment is the command window that comes with the WDK. Any opinions expressed in this article don't affect Microsoft's policy on this issue.

Impressions from Seattle Code Camp v3.0

iliast — Mon, 28 Jan 2008 07:30:00 GMT

During this weekend I went to the Seattle Code Camp v3.0. The talks were mostly oriented towards .NET,so it was a good opportunity for me to get a better understanding of all those buzzwords that are unknown to the world of drivers. I also had the opportunity to talk to many interesting people and listen to their ideas about .NET and software development in general.

I started my first day at the code camp by going to Jason Haley's talk, which focused on the .NET Reflector (nothing to do with the UMDF Reflector :) ), which is the most commonly used .NET dissassembler. Jason covered both the reflector, as well as some plug-ins. I had seen the Reflector in use before, however it was a nice overview. I've been reading Jason's link-blog very often and fortunately we had the time during the break to discuss a little bit more.

After his talk, it was time for me to understand what Silverlight and WPF (Windows Presentation Foundation) are. I've heard lots of good stuff about them, however I had no idea what they actually do (apart from the fact that they have to do with the way that applications look). So, after looking at some cool demos in Kelly White's talk, Adam Kinney explained all the theory behind it (what Silverlight is, what are the differences between v1.0 and 2.0, what are the differences between Silverlight, WPF and Flash etc). Adam also went through the development of a cool app that can be shown both in a web browser (using Silverlight) and in the Vista sidebar (using WPF).

Then I decided to move to a bit more familiar space by going to Wayne Berry's talk on hashing. Wayne showed some practical aspects of how hashing is implemented in applications (as opposed to the mathematical theory behind it), as well as potential pitfalls that developers face. Jim McKeeth also had a very interesting talk on implementing cryptography using the Win32 CryptoAPI, however I had to miss it, because I wanted to go to see the xUnit.NET talk from Brad Wilson and James Newkirk. The xUnit.NET talk presented some really cool ideas on unit testing, however I hope that Brad will upload his Cryptography presentation to the web, since he did the same with his "Advanced Downloads" presentation. Now that I'm thinking about it, the Cryptography talk was the only non-.NET talk of the event and I didn't go to it!

Anyway, the next morning, it was time to understand WCF (Windows Communication Foundation). Robert Green gave an excellent presentation. He explained what WCF is and he created a service that could be reached both through TCP and through HTTP (depending on the client being local or remote) and showed how easy interoperability has become with the use of WCF.

The last talk that I went to was Charles Sterlings' talk on how Visual Studio 2008 Team System's features that are related to Application Lifecycle Management. Ok, first of all, I'd like to say that Charles is an excellent presenter. He had only demos and no slides (!) His whole talk was interactive and he actually asked us (the viewers) to do the demos for him (!) I also got a T-shirt with the Visual Studio logo :) Anyway, during his talk I couldn't prevent myself from (mentally) comparing the tools that are being used by application-developers to the ones that are being used by driver developers. Charles was showing all these cool features that VS2008 has and I just had to think that in order to compile a driver I need to go to a command prompt and type bcz (ok I know that I can use batch files, in order to call bcz from VS, however this is not the main point)... Also, I'm using an editor without Intellisense (and that's even more important than the previous issue)... No help-file integration... I was feeling like a prehistoric person in the middle of modern New York. I know that Doron has also blogged about this feeling. When I told a couple of .NET developers about the tools in driver-land they couln't believe their ears. Anyway, hopefully sometime in the future, driver developers might reach a critical mass and things might change... Until then...

Unfortunately I had to leave before the end of the event, otherwise I would like to see more stuff about WCF and Ajax. However, the event was very well organized (kudos to everybody, who helped organize it). I understood several buzzworks (Silverlight, Moonlight, WPF, WCF, etc). All the talks were really interesting and I'm sure that all the visitors enjoyed it. I know that I'll definately go to Seattle Code Camp v4.0 :)

My List of Top 5 Windows Books

iliast — Thu, 24 Jan 2008 03:15:00 GMT

I've been reading quite a bit for the last couple of months and I compiled my list of the 5 Windows development books that I want to complete:

1) Windows Internals (finished): The classic book by Mark Russinovich and David Solomon is now in its 5th edition. I've read the 4th one and I think that it's a must-read for every windows developer.

2) Developing Drivers with the Windows Driver Foundation (finished): I just finished the book and I think that its the best book currently available to windows driver developers, especially for beginners. I'll write a review shortly, however I think that it's definately the most complete and at the same time easier to read book on windows drivers.

3) Advanced Windows Debugging (not started): After John Robbins decided not to write a debugging book for Win32 programming, I was wondering which book would take its place. I've read lots of good reviews about this book and I have it in my bookcase, so I'm eager to read it. Just with a first glance it seems to be exactly what I was searching for.

4) Windows via C/C++ (not started): This is the latest version for "Programming Applications for Microsoft Windows". It covers application development in Windows. I've taken a class by Jeffrey Richer and I know that he's definately both a good developer and a good teacher, so I'm waiting to read it.

5) Programming the Windows Driver Model (half-read): This is the most tough-to-read and most advanced book in windows driver development. It covers many aspects of windows in depth, however I don't feel that I'm ready to read it yet. I've tried a couple of times in the past and I failed miserably. I think that really understanding this book means that your level is definately above intermediate.

Of course, there are other books, like Subverting the Windows Kernel, however these are my top choices. I'd love to hear your comments, as well as your top choices.

How Driver Installation Works

iliast — Sat, 06 Oct 2007 23:59:00 GMT

The last few months I've been working on the WDF 1.7 (UMDF+KMDF) coinstallers (that's one of the reasons that I've been silent for quite some time).

Through this process I managed to learn a lot of things about how driver installation works and what is required by the driver developer. Unfortunately, this area is often a black box for driver developers, since their job ends, after the driver is up and running (and hopefully tested :) ). However, it's possible that you might want the driver installation to do something "more", e.g. change the icon of the device in the device manager or add a page in the device manager that shows the driver capabilities, etc. This is when a coinstaller or a class installer is needed.

Let's start from the beginning though. If you have a driver, then how do you install it? The easy way is OSR's Driver Loader. Just point to your sys file, click "Register Service" (i.e. create the appropriate settings in the registry at HKLM\System\CurrentControlSet\Services\<Driver Name>) and "Start Service" (i.e. call the Service Control Manager APIs that load the driver according to the registry settings). If you want to remove the driver, just click on "Stop Service" (stops the driver from running) and "Unregister Service" (deletes the registry settings). Easy, right? You don't need an inf file or anything more apart from your driver. However, this program supports non-pnp (legacy) drivers only.

For WDF drivers we provide a coinstaller (one for UMDF and one for KMDF). The main tasks of the coinstaller is to upgrade the framework (e.g. from 1.5 to 1.7), parse the inf file and configure the driver correctly. In the past, we've had some issues with the previous versions of the coinstallers, as shown by Bob Kjelgaard (part 1, part 2, part 3), however we're trying our best to tackle them. By the way, if you have any problems with the installation of a WDF driver, please look at the Doron's post for information on how to debug it and at Bob's post for information about how you can ask more help from Microsoft.

Microsoft has lots of useful information on Driver Installation at http://www.microsoft.com/whdc/driver/install/default.mspx, however I'd also like to point specifically to:

Eugene Lin and Jason Cobb's channel9 video: Introductory overview on how the user-mode Plug-and-Play manager works and how the installation works from the point that the device is connected to the machine, until the corresponding driver is selected and the device is running
Jim Cavalaris' presentation on coinstallers and class installers: Excellent presentation that shows how the class installers and the coinstaller cooperate during driver installation. Also look at the resources, which are provided at the last page.
Debugging Device Installation on Windows Vista: Everything that you wanted to know (and even more than that!) about how to debug problems during device installation (and were afraid to ask)
Device Installation Design Guide: Microsoft's official documentation on everything that has to do with device installation

What's New in Windows Vista?

iliast — Mon, 26 Mar 2007 07:52:00 GMT

One of the most popular questions that I've been hearing lately is "what's new in Windows Vista?". Therefore, I could not prevent myself from writing a post with links that provide information about this issue:

Let's start with the changes in the kernel. Mark Russinovich covers this topic very well:

In user-mode, Arstechnica provides a deep overview (this is the first post of a three-part review)
Security-related topics:

Michael Howard presents the big picture about Windows Vista's security here and focuses on ASLR (here and here)
Windows Vista Security Enchancements discusses about the changes in more depth
Channel9 videos about security in Vista (part 1, part 2) and User Account Control (UAC)

Channel9 has videos that cover many windows components that are new in windows vista
Wikipedia provides a quick overview of all the new additions in Windows Vista

UPDATE:

If you are interested in a detailed comparison between the different Windows Vista versions (Home, Business, Enterprise, etc), then you can look here (4th post).

Tips On How To Analyze Strange Crash Dumps And Uninstall Hidden Drivers

iliast — Fri, 12 Jan 2007 05:13:00 GMT

Recently, a friend of mine had the following problem: his computer crashed exactly 2 hours after booting into windows. As usual, I opened windbg and executed !analyze -v in the minidumps, however I didn't get any useful information:

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses. If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 00000081, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: 865d7aa8, address which referenced memory

Debugging Details:
------------------
READ_ADDRESS: 00000081
CURRENT_IRQL: 2
FAULTING_IP: +ffffffff865d7aa8
865d7aa8 0000 add [eax],al

CUSTOMER_CRASH_COUNT: 2
DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0xD1
LAST_CONTROL_TRANSFER: from 80544f5f to 865d7aa8

STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
f78aafcc 80544f5f 865941a8 86581000 00000000 0x865d7aa8
f78aaff4 80544acb a7554d44 00000000 00000000 nt!KiRetireDpcList+0x61
f78aaff8 a7554d44 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2b
80544acb 00000000 00000009 0081850f bb830000 0xa7554d44

STACK_COMMAND: kb
FOLLOWUP_IP: nt!KiRetireDpcList+61 80544f5f 837c240c00 cmp dword ptr [esp+0xc],0x0
FAULTING_SOURCE_CODE:
SYMBOL_STACK_INDEX: 1
FOLLOWUP_NAME: MachineOwner
SYMBOL_NAME: nt!KiRetireDpcList+61
MODULE_NAME: nt
IMAGE_NAME: ntkrpamp.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 4356d823
FAILURE_BUCKET_ID: 0xD1_nt!KiRetireDpcList+61
BUCKET_ID: 0xD1_nt!KiRetireDpcList+61
Followup: MachineOwner

Unfortunately, windbg doesn't have any other information from the call stack, so he can only point to the windows kernel. This is a common behavior that can be seen, when a driver really messes things up. You should be alert that, when this happens, you might have to do some more advanced digging or some more "trial and error" (like I did). I'm sure that the problematic driver could have been found using driver verifier or some more advanced techniques, but here I would like to show a more quick-and-dirty solution.

Since the problem happened exactly 2 hours after he booted into windows, this could not be a hardware problem (since a hardware problem would occur more randomly). Also, from the bugcheck code it is obvious that it is a driver's fault. As a first step, I executed

lm and lm kv m specific_driver*

to find all the drivers that were loaded into the system and also to find specific information about some "interesting" ones. I saw that no driver was loaded at an address close to 0x865d6668.

The next step was to try and isolate drivers that might seem more suspicious than others. I found that an easy way to look at the drivers running on a system is driverview. This tool shows approximately the same information like windbg (driver name, corresponding filename, description, company name, etc), but also has a nice GUI. So, after finding some "interesting" cases, the next step was to uninstall some drivers. Of course, before that I tried to enable driver verifier on different driver categories, however this took quite some time and I opted for an easier solution :)

The problem here was the fact that by default not all drivers are viewable from the control panel. In order to show all drivers (even the hidden ones), you need to do the following:

Open a command prompt (e.g. go to Start/Run and type cmd)
At the command prompt, type "set devmgr_show_nonpresent_devices=1" (without the quotes)
Type "devmgmt.msc" (without the quotes) and press enter. The Device Manager comes up.
Go to "View" -> "Show Hidden Devices" and you'll see all the drivers that are currently running on your computer, as well as the drivers that are installed, but not running (e.g. because the corresponding device is not currently connected to the computer).
After removing the drivers (by right-clicking on the corresponding device and clicking "uninstall") that you are sure that you don't need (don't remove all the unused devices just because they are greyed out! Make sure to remove only the ones that you don't need), you can close the device manager.
Keep in mind that, until you close the above command prompt, every instance of device manager that you launch from there will be able to show all the hidden devices.
Also, if you want to avoid having to set the variable any time that you want to look at the hidden device drivers, you can go to Control Panel -> System -> Advanced -> Environment Variables and set it there.

Of course, if you want to have a tool that allows you to remove drivers easily, you can download the Driver Manager, which shows the list of the running drivers and allows you to disable them or remove them.

So, in my case, after removing different sets of suspicious drivers, the culprit was found and removed from the system, so everything is now back to normal :)

Crash Dump Analysis

iliast — Tue, 12 Dec 2006 05:43:00 GMT

I'm sure that many of you have had the unfortunate experience of watching the windows Blue Screen Of Death (BSOD) while working, and possibly have lost important data. A common reaction in this case is to blame Microsoft and continue working after the following reboot, as if nothing had happened. Another unfortunate experience is to see an application crash, while using it. In this case, there is a window that comes up, asking you, if you want to send the data to Microsoft for analysis (the same window also comes up after the BSOD). Many people might be afraid of the contents of the data that is sent to the network, so they select "No". The goal of this post is to help you understand what is going on in the background in each of the two cases and also to help clarify some misconceptions.

QUESTION 1: What causes all these reboots?

First of all, because of the architecture of the windows kernel in the NT/2000/XP/Vista series, an application cannot corrupt data that belongs to another application or to the kernel. This means that each application is totally isolated and cannot harm the system. The worst thing that can happen is that the application does something invalid and crashes without any further implications for the rest of the system. On the other hand, the windows kernel and the device drivers have unlimited access to the system. If the kernel or a driver misbehaves, then it can corrupt the whole system. The immediate result of this, is that the reason for all the blue screens lies either in the windows kernel or in the windows device drivers. That's why, whenever an application crashes, the system keeps working without a problem, whereas if there is a bug in the kernel or in a device driver, the whole system goes down.

Now that we've identified the possible causes of the crashes, it's time to go even further. According to the reports that were sent to Microsoft until April 2004 (from all those people, who pressed "Yes", when they were asked to send the data to Microsoft) the reasons for the crashes can be split as follows:

Third-party device drivers: 70%
Unknown, because of severe memory corruption: 15%
Hardware error: 10%
Microsoft code: 5%

This shows that Microsoft is not the one to blame. The main cause for these crashes is poorly written third-party (non-Microsoft) device drivers.

QUESTION 2: Why does the system crash, when there is a kernel-mode error?

So, from the above analysis it is obvious that the system can overcome an application crash, however a kernel-mode error causes it to reboot. Why can't the system continue, after finding a kernel-mode error? Actually, what happens is that while kernel-mode code (either a device driver or the windows kernel) is executing, some discrepancy is found. For example, a pointer might be pointing to an invalid address, a data structure might have invalid values, etc. Even though this problem was found, it's possible that the "bad" code that caused this problem might have corrupted more data. For example, it might be possible that basic kernel structures are corrupted. That's why the function KeBugCheckEx is called (inside the kernel, this type of forced crash is called "bugcheck"), in order to write logging information to the page file, paint the blue screen and show some information about the crash in the screen.

QUESTION 3: How do we configure, whether we want to send anything to Microsoft?

Even though, it's not well-known to most people, you can configure what will be sent to Microsoft. Somebody might want to report only the crashes that have to do with the operating system (after the BSOD). Somebody else might want to report only the crashes from the applications (either for all of them or only for some particular applications). In order to configure this, you need to go to

Control Panel | System | Advanced | Error Reporting

From there you'll be able to select which types of errors you want to report. Actually, even if you select a particular type of error to be reported, Windows will ask you again, after a program that belongs in that category crashes. So, by selecting a category, it doesn't mean that all crashes will be reported automatically.

QUESTION 4: Exactly what kind of data is sent to Microsoft?

Before answering this question, it is useful to understand what information is stored, when the system or an application crashes. Let's talk first about the files that are generated after a system crash. In order to set this, you need to go to

Control Panel | System | Advanced | Startup and Recovery -> Settings

In the "System Failure" part you'll be able to configure, if you want to write the crash to the system log, if you want to send an administrative alert and if you want to reboot after the crash. Also, you have the option of creating a memory dump and you can select the directory, in which you want to save it. There are 3 types of memory dumps:

Small memory dump or minidump (64kb for 32-bit systems, 128kb for 64-bit systems): It includes a minimum amount of information about the system before the crash, e.g. the bugcheck code, the loaded drivers, information for the current process and thread, etc.
Kernel memory dump: This includes all the kernel-mode memory that was in physical memory at the time of the crash. There is no default size for this dump, however it should be around 50-100MB for most "normal" systems (with <= 2GB of RAM).
Full memory dump: This includes all the physical memory. The size of the file will be the same as the physical memory of the system.

Here it's worth mentioning that at the time of the crash, the information is written to the pagefile. Therefore, the pagefile must be configured to be larger than the size of the dump. This might be a problem especially in the case of the Full memory dump. In order to set the size of the pagefile, you need to go to

Control Panel | System | Advanced | Performance Settings -> Advanced | Virtual Memory -> Change

After the system reboots, the information is copied from the page file to the file that was specified above. The reason that the information is not written directly to that file is that the kernel doesn't know the root of the problem at the time of the crash, so it's trying to use as fewer drivers as possible (the pagefile is already open and in use, so theoritically it's the safest destination).

On the other hand, in order to configure the data that is stored, when an application crashes, you need to open a command prompt (or go to Start | Run) and execute drwtsn32.exe. This application is called Dr. Watson and there you'll be able to configure the destination file for the dump, as well as the type of the dump. Here the only options are the minidump and the full memory dump. There is also an option to create an old-style NT-compatible full memory dump. In addition, in the textarea "Application Errors" you can look at the application crashes that had been logged in the system. You can select any of them and click "View", if you are interested in looking at exactly what the log file includes.

So, now it's time to answer the initial question: What exactly is sent to Microsoft? The answer is that regardless of the crash dump file that you have selected, the only thing that is sent to Microsoft is a minidump (both for kernel-mode and user-mode crashes). Of course, it would be impossible to send a 100MB kernel-mode dump or a 1GB full-memory dump, so that's why only the 64kb minidump is sent. Apart from the minidump, the information also includes an XML file with basic information about the version of the operating system and the loaded drivers, which you can look at, when you are prompted to select, if you want to send the data to Microsoft. There is no personal private information or anything like that. In fact, you can open the minidumps and check the included information. I'll also show a way of analyzing the minidumps and looking at the data that Microsoft has access to.

QUESTION 5: What does Microsoft do with this information?

When the minidump is received by Microsoft, it goes through some preprocessing and is stored in a server. If many minidumps that seem to have the same problem are received, then there is a team that analyzes them and finds the root of the problem. Afterwards, a webpage is created that shows exactly what the cause of the problem is. Most often it points to the driver causing the problem and gives a link to the manufacturer's webpage, so that a new version can be downloaded (if it exists). After the webpage is created, if somebody submits a dump that has the same problem, he is shown the corresponding webpage that will help him find the solution. Therefore, if somebody clicks "Yes", when asked to submit a crash dump he might either find the solution to the problem or help Microsoft find the solution and present it to the users in the future.

QUESTION 6: How can we analyze a crash dump?

Fortunately, there is a tool that can be used to analyze both the user-mode and the kernel-mode crash dumps: windbg. Microsoft has included the dump analysis algorithms in windbg, so in some basic cases it's easy to find the cause of the problem. Of course, there are many causes, in which there are many corrupted data structures and it's impossible to pinpoint the problem automatically. In that case, more advanced manual methods are used by the Online Crash Analysis (OCA) team in Microsoft.

In order to perform the analysis by yourselves (either because you are unable to submit the data or because there is no answer in Microsoft's website), you need to open windbg, go to "File" | "Open Crash Dump" and select the dump file that you want to analyze. As I wrote in my previous post you need to set the path to the symbol files and reload them. If this step is omitted, then it won't be possible to analyze the file.

The next step is to execute the command (this might take some time):

!analyze -v

At the top of the output you'll see the bugcheck code, it's description and some additional information (e.g. the address of the invalid memory that was accessed, whether it was a Read or a Write operation, etc). Further down you'll see the call stack of the dump under the title STACK_TEXT. This includes all the functions that were called, when the crash occurred. The function on the top is the most current one (it was called by the function below it, which was called by the function below it, etc), whereas the function at the bottom is the oldest function in the stack. The reason that the system crashed is because one of these functions did something invalid (e.g. passed or received an invalid argument that forced it to perform an invalid operation). Of course it's possible that the data was corrupted by another function that is not in the call stack. Fortunately, windbg has already done an automated analysis and points to the module that most probably caused the crash. You should look at the following fields:

SYMBOL_NAME: Exactly where the invalid operation was caused (module + function)
MODULE_NAME: The name of the module that caused the crash
IMAGE_NAME: The file, in which the problematic code resides

In order to find more information about the problematic code you can execute:

lm kv m MODULE_NAME*

for example, if MODULE_NAME is problematic_driver, you should execute:

lm kv m problematic_driver*

lm stands for "list modules", k stands for "kernel modules", v stands for verbose and m stands for "match".

Another option is to find the problematic file name, from the IMAGE_NAME tag, search it in the hard drive and either look at its properties, in order to identify its manufacturer or search it in the internet. Afterwards, you might need to update the buggy driver.

Of course, it's possible that windbg's automatic analysis was not able to pinpoint the faulting driver. The reason for that might be that the call stack was corrupted or that some important code or data structure was overwritten or that there was a memory leak, etc. In that case, you might want to proceed into some manual solutions, in order to detect the problem. From this point there is no automated way to proceed, so I can just provide a few useful commands that might help you find more information about your system.

First you can execute

!process 0 0

!process

The first command prints information about all the running processes. this way you might be able to find a suspicious process. The second command shows information about the current process. If you execute

!process <process_address>

then you'll see more information about the particular process. You can find the addresses of the processes from the "!process 0 0" command. The process information includes information about its threads. You can find more information about each thread by executing

!thread <thread_address>

and if the thread belongs to a driver with pending IRPs, then you can find more information about them by executing

!irp <irp_address>

Also, as I wrote above, in order to see all the loaded modules you can try

lm kv

In order to find more information about the used memory (and possibly detect memory leaks), you can execute:

!vm and !memusage

Finally, it's possible that the system hangs and does not crash. In order to debug it, you need to force a crash. The only way to do that is to go to the registry key HKLM\System\CurrentControlSet\Services\i8042prt\Parameters\CrashOnCtrlScroll and set it to 1. This works only for PS2 keyboards (not for USB). When the system hangs, you can keep the right control key pressed and press the scroll lock key twice. This will cause a crash, which you will be able to debug using windbg.

A useful command in that case is:

!locks (prints the locks, which are currently held, provided that there is at least one additional thread waiting on them).

Another tool that can help you, if !analyze cannot find the root case is the Driver Verifier. This tool enables additional system checks, that will make it possible for the system to crash immediately, when a driver does something invalid, without allowing it to corrupt more data. This way, the crash dump will point directly to the driver. In order to execute it, you need to open a command prompt with administrative privileges (or from Start | Run) and execute "verifier.exe".

There are some small differences between the User Interface in Windows 2000, Windows XP and Vista, so I'll explain the Windows XP interface. What you'll see is a window and some tasks that you're called to select. You need to select the task "Create custom settings (for code developers)", and then "Select individual settings from a full list". After that you'll see a screen with the following options:

Special pool: This option forces the memory allocation routines to operate on a special pool. For example, if a driver wants to allocate 100kb, then he is given a pointer that points to 100kb before the end of a free page. The rest of the page is marked with a specific signature. Also, the pages that are before and after this particular page are marked invalid. So, if a driver tries to write something after the end of the allocated space, there will be a page fault and the Driver Verifier will crash the system immediately. If the driver tries to write before the beginning of the allocated space, then after he frees the memory, the Driver Verifier will check the signature of the page, find that it's invalid and crash the system. The crash dump that will be generated from this crash will point directly to the faulting driver.
Pool tracking: Each space allocation is marked with a special tag that is different for each driver. When the driver is unloaded, the Driver Verifier will check for the corresponding tags and if it finds any, then this means that the driver has a memory leak, so the system will crash.
Force IRQL checking: Whenever a driver goes to IRQL at DPC/dispatch level or above, the Driver Verifier will cause all the pageable memory to be paged out to disk. So, if the driver tries to access this memory, then the system will crash with a bugcheck code equal to IRQL_NOT_LESS_OR_EQUAL
I/O verification: All the IRPs are allocated from a special pool. If any of them is completed with an invalid I/O status, then the system crashes.
Enhanced I/O verification (used to be "I/O Verification level 2" in Windows 2000): This includes even deeper tests for the IRPs. The I/O manager checks if the drivers complete asynchronous IRPs complete correctly, if they manage the device stack locations correctly and if they delete the device objects only once.
DMA checking: The I/O manager makes sure that all the drivers configure the DMA operations correctly, otherwise it crashes the system.
Deadlock detection: This option enables deadlock detection. When a deadlock is detected, the system crashes and you can use the !deadlock command from windbg to find more information about what is causing it.
Low Resources simulation: 7 minutes after the boot completes (so all the drivers have been loaded), the I/O manager starts failing random memory allocations. This way, if a driver doesn't check the status of a memory allocation, the system will crash.
Disk Integrity Verification (only in Windows Server 2003): Windows keeps checksums of written data, so after each read it checks, if the data is still valid. If there is a discrepancy, the system crashes.
SCSI Verification (included automatically, when a SCSI miniport driver is monitored): It includes some additional SCSI-related checks.

You should select all of the existing options, apart from the "Low Resources Simulation" (because it includes very heavy operations and the system will be really slow). In the next screen, you need to select the drivers that you want to debug. You should start by selecting drivers that you think that are suspicious (either because windbg pointed to them or because the crashes started after you installed them, etc). Reboot and run the system for some time and observe its behaviour. If the system continues crashing, but not because of the drivers that you specified (i.e. the crash dump files are still vague and don't pinpoint to a specific driver), the next step is to select all the unsigned drivers. If you don't see any result, then the next step is to start enabling driver verifier for bigger groups of drivers, until you find the buggy one.

Another useful tool for debugging memory leaks is poolmon, which is part of the Windows Support Tools and can be found here (for Windows XP). This tool displays information about memory allocations (both from paged pool and from non-paged pool), as well as discrepancies between allocations and deallocations. You just execute the tool from a command prompt (there is no User Interface) and select the types of memory pool that you want to look at. If the amount of allocated memory increases constantly, this means that there is a possible memory leak. You can find an overview here and a more detailed explanation here.

ADDITIONAL RESOURCES:

This tutorial provides an introductory overview of the same concepts.
Mark Russinovich presented this video in TechEd 2006. The corresponding slides are here.
OSR has a more advanced article on the next steps that you can follow, in order to analyze more difficult crash dumps.

Windows Vista Has Shipped!

iliast — Thu, 09 Nov 2006 06:29:00 GMT

Finally, it is true! Windows Vista is ready and has been shipped to the manufacturer! So, this means that everything is locked, the CDs will shortly be printed and put into boxes together with the corresponding manuals. Finally, in 30th January everybody will be able to buy them from the stores either as standalone product or preinstalled in computers.

So, how is Vista? From my point of view, I think that it's much more secure than Windows XP and has some very interesting new features. I definately agree with Michael Howard that after all this multi-year effort, it'll be interesting to see how the attacks against the operating system evolve. I'm sure that it'll be tough to break it :) Definately, the bar has been raised. Also, from the end-user's perspective there is a new User Interface, which might take some time to get used to, but is proves to be user-friendly. Also, there are many technologies that allow faster booting, faster application loading, easier application development, greater stability etc. From the system developer's perspective, there are many changes, because of WDF, UMDF and KMDF. Hopefully these frameworks will make it much easier for driver developers to develop, debug and maintain drivers, and so the overall system will be more stable.

So, here it is! Enjoy :)

Windows Device Drivers Book Reviews

iliast — Thu, 26 Oct 2006 03:18:00 GMT

A quick search in the web reveals that the number of the books that are related to windows device drivers can be counted with the fingers of one hand. Even worse, most of the books are either too old (published before or around windows 2000) and/or not easily readable. Another problem is that the Windows Driver Model (WDM) is becoming more complex as time passes, so the newer books are relatively more complex to read than the older ones.

Based on all of this and after looking at different book reviews, I decided to read a few books that make it easier for a beginner to get an insight on driver development.

So, I think that the best book to start with regarding driver development is Windows NT Device Driver Development (OSR Classic Reprints) (original version was published in 1997) by Peter G. Viscarola and W. Anthony Mason. This book is divided into 3 parts. The first part (chapters 1-8) talk about Windows internals fundamental concepts, like HAL, scheduling, virtual memory, registry, etc. The second part (chapters 9-20) is the bulk of the book and talks about the development of device drivers. Actually, this can be further subdivided into chapters 8-10, which talk about the I/O manager, 11-17, which is the core part of the book (DriverEntry, Dispatch routines, Interrupt Service Routines, Deffered Procedure Calls, Programmed I/O and DMA), and chapters 18-20, which talk about building and debugging a driver. Finally, chapters 21-24 talk about alternate NT driver architectures, like File system drivers, SCSI drivers, Video miniport drivers and NDIS (network) miniport drivers. I think that there are both advantages and disadvantages in this book. I'll start with the advantages:

It's very easy to read. It starts with very fundamental stuff, builds on top of them and explains the concepts in an easy-to-understand way. This is really to hard to find in the rest of the books. Possibly the fact that both writers teach driver development courses make it easier for them to understand what questions and problems a beginner driver developer might have, so they try to answer them in front.
It focuses on explaining the concepts and doesn't present pages and pages of un-understandable code. It just presents the functions, explains how they work and shows a cumulative example that shows the use of a few functions bundled together.
It's a short book (the first 20 chapters take less than 550 pages) and has small chapters. I think that this fine grain analysis helps the reader. I like the fact that I can read a few small chapters within a day and then pickup the book the next day (or a few days later) without breaking my reading in the middle of the chapter and trying to remember what the first part of the chapter was saying.

Of course, the book has also some disadvantages:

It doesn't cover plug-n-play, power management and other WDM concepts. This is normal, since the book covers NT drivers and not WDM drivers.I haven't found many outdated parts of the book (i.e. almost everything that is written in the book applies even for WDM drivers in Windows XP), apart from the fact that the DriverEntry chapter (chapter 13) talks a lot about manually finding the resources (memory+I/O) that the driver should be using, whereas for WDM drivers this is done automatically. It also doesn't cover Physical Device Objects (PDOs), Function Device Objects (FDOs) and Filter Device Objects (FiDOs), since they didn't exist at that time.
The examples are small and after understanding the concepts, it's nice to be able to look at some code that will explain them.

So, in order to solve the disadvantages of the first book, I think that the solution is to read The Windows 2000 Device Driver Book: A Guide for Programmers (2nd Edition) (published in 2000) by Art Baker and Jerry Lozano. The strengths of this book are exactly those above:

The book covers Windows 2000 drivers, so it covers Power Management, Plug-n-Play, Windows Management Instrumentation (WMI) and many WDM concepts that were not covered by the first book.
The book is full of examples. The writer also provides the source code in the accompanying cd-rom, so it's easy to build them and learn from the source.
Again, the book is easy to read and short. I like a lot compact books that stick to the topic and don't provide stuff that distract the reader. This book is one of those. The whole book is around 400 pages, so it can be finished quite quickly.

So, after finishing both these books and playing with the source code (and with the source code that can be found in the WDK or elsewhere in the net), I think that the next step is to read Programming the Microsoft Windows Driver Model, Second Edition (published in 2002) by Walter Oney. Again, this book has both advantages and disadvantages. This time, I'll start with the disadvantages :) :

This book is definately NOT for beginners. I consider it hard to understand and it considers way too many things as known.
I don't like the structure of the book very well. For example, even though the author mentions that the first 7 chapters are fundamentals and the rest go deeper, he puts the I/O control operations after the plug-n-play or the power management. Of course, this is a personal view, however I was overwhelmed, when I managed to reach the 6th chapter. The first 3 chapters are very good, however I think that after that the difficulty grows exponentially, at least for a beginner.
The material presented in the book is very dense, so a lot of stuff is presented within a small number of pages and then you look at a big code listing that should explain everything. As I said above, I prefer a slower approach that goes step by step and builds on top of fundamental concepts.
I don't think that the book gives a a clear high-level view of what a device driver is composed of (I'll explain this below). I had the feeling that the author explains one thing, then the next and the next, but it's hard to understand how everything fits together and how all of these components are related.

On the other hand, this book has some advantages that cannot be covered by any other existing book:

It is the newest book, so it provides the best picture about the current version of the Windows Driver Model (WDM).
It's the only book that covers Windows XP, so it's the only choice for somebody, who wants to take advantage of this platform.
I think that each section is described more deeply than from the rest of the books.

Finally, I think that such a review would be incomplete, if I didn't refer to Microsoft Windows Internals, Fourth Edition: Microsoft Windows Server(TM) 2003, Windows XP, and Windows 2000 (Pro-Developer) (published in December 2004) by Mark E. Russinovich and David A. Solomon. This book doesn' t talk directly about writing device drivers, but it talks about windows internals in general. Some of them are covered in the above mentioned books, however this book is the newest of all and provides the most in-depth analysis. Also, chapter 9 is devoted specifically to the I/O manager and to windows device drivers, so it's definately worth to read. I especially like one particular section in chapter 9 that is titled "Structure of a Driver" and gives a high-level overview of all the components of a driver. That's exactly an important thing that is missing from Walter Oney's book and is not clearly explained in the rest of the books. So, according to the book, a windows device driver is composed of:

An initialization routine: This function is named DriverEntry (for WDM drivers) and is called by the I/O manager only once, when the driver is initially loaded. It is used by the driver to initialize its internal (global) structures.
An add-on device routine: This function is named AddDevice and is called by the I/O manager, whenever a new device, for which the driver is responsible, is detected. It is used by the driver to initialize the device object (i.e. the internal structure) that corresponds to the newly-found device.
A set of dispatch routines: These are the functions that are called by an application (or another driver) to communicate with the device. They include functions like read, write, open, close, control, etc. When called to perform an I/O operation, the I/O manager creates an IRP (I/O Request Packet), which is a structure that descibes the request, and sends it to the driver.
An Interrupt Service Routine (ISR): One of the ways that an external device can communicate with the CPU is through interrupts (the other way is polling, but let's forget about it for now). So, whenever the device wants to communicate with the CPU, it sends an interrupt to the CPU and the CPU executes some code that needs to serve the device (e.g. find if the device finished some calculation or received a packet, etc and find what to do afterwards). The code that is executed is called the Interrupt Service Routine (ISR). Therefore, the driver that controls the device needs to implement this ISR.
Deffered Procedure Call (DPC): Whenever an ISR is executing, the IRQL (IRQ Level) of the CPU is high, so all the interrupts that have lower IRQL than the current interrupt cannot be served. In order to solve this problem, drivers perform only very basic calculation in their ISR and for the rest of the processing they schedule a DPC. This function executes in a low IRQL (which is named DISPATCH_LEVEL), so that the rest of the interrupts can be serviced. Therefore, the DPC can be considered as just an extension to the ISR (they serve the same purpose, but they execute at different IRQL). Of course, a driver can execute DPCs not only from an ISR, but from any other function, however this is the main use of a DPC.
A start I/O routine: Some drivers wants to process the incoming I/O requests serially, i.e. one at a time. Therefore, they ask the I/O manager to queue the incoming IRPs. This function is used to initiate the next data transfer to or from the device.
One or more I/O completion routines: Sometimes a driver, which receives an I/O request does some processing and passes the IRP to the next driver in the stack, without finalizing the IRP processing (finalizing means that the I/O operation was completed successfully, failed or cancelled). The I/O completion routine informs the driver that another driver in the stack has marked the I/O as "finalized", so the current driver's I/O completion routine can be called, in order to perform some cleanup operations.
A cancel I/O routine: This function is called, when the I/O manager wants to cancel the processing of an IRP. This function mostly releases any resources that were acquired during the processing of the IRP and it also marks the IRP as "cancelled". Not all IRPs can be cancelled, though.
An unload routine: The I/O manager calls this routine, when it wants to unload the driver from memory. The drives releases any resources that it might have acquired.
A system shutdown notification routine: The I/O manager calls this routine, when the system is about to shutdown.
Error-logging routines: Whenever an error occurs, the driver uses these routines to log the error and notify the I/O manager.

Here I would like to note that all the above books are for the Windows Driver Model. As I've mentioned many times before, Microsoft is changing the driver model for Windows Vista. However, there aren't any books for the Windows Driver Foundation (WDF - the new windows model) yet. The only public announcement for such a book is the Introduction to the Windows Driver Foundation - Kernel Mode Driver Framework by Peter Viscarola, Tony Mason, Mark Cariddi, Brenda Ryan, Scott Noone, and OSR.

Hardware and Driver Developer Blogs + Driver tips

iliast — Sun, 08 Oct 2006 05:10:00 GMT

Thanks, to craigrow, I saw that Microsoft has created a list for hardware and driver developer blogs. I'm still not there yet, but I hope that this will change :)

Another interesting link are the driver tips, which include papers about driver synchronization, memory management, driver design, debugging, etc.

A Comparison of the Linux and Windows Device Driver Architectures

iliast — Wed, 04 Oct 2006 02:00:00 GMT

While searching for something irrelevant, I found a very interesting paper, called "A Comparison of the Linux and Windows Device Driver Architectures" by Melekam Tsegaye and Richard Foss. It was published at the ACM Operating Systems Review, Volume 38, Number 2, 2004. You can find a link to the paper here.

The paper compares the device driver architectures between linux and windows. It starts by providing an extensive analysis to both architectures and then presenting a detailed comparison. The paper covers the 2.4 linux kernel and the Windows Driver Model (WDM) from Windows XP, so it can be considered that it is quite up-to-date (at least from the Windows perspective). Also, in the same page, you'll find Melekam Tsegaye's MSc Thesis, which covers the same contents as the paper, but it also presents the design and the implementation of an IEEE-1394 driver for both windows and linux. Both drivers can be found here, while the presentation of the thesis can be found here.

Becoming familiar with the windows internals

iliast — Fri, 29 Sep 2006 01:31:00 GMT

When I started learning about windows drivers, my first thought was to search for "introduction to windows drivers" or something equivalent in my favorite search engine. This led to a few links for tutorials on how to create simple drivers. Right now I think that this is a very bad approach. Even, if you understand how to write a small driver, it will be difficult to proceed. I believe that the best first step is to become familiar with the windows internals. A driver developer needs to be familiar with lots of operating system-dependent stuff, e.g. the synchronization primitives, the way that the scheduling works, the I/O manager, etc.

Currently, the definitive bible for windows internals is a book called "Microsoft Windows Internals, Fourth Edition: Microsoft Windows Server(TM) 2003, Windows XP, and Windows 2000" by Mark Russinovitz and David Solomon. This book provides the most in-depth coverage of the windows internals and is used as a base for every driver teaching course. It covers a wide area of topics, e.g. processes, I/O manager, file systems, networking, Online Crash Analysis, etc. I believe that this book is a MUST-read.

Based on the book, a lot of additional material has been created that can help somebody understand the fundamental windows concepts:

The authors of the book have created a series of 6 dvds with videos that present the most important part of the topics that are covered by the book. Additional information about the dvds can be found at http://www.solsem.com/vid_internals.html. In the same link you can find information about the seminars that are taught by David Solomon on windows internals.
Microsoft offers the Windows Operating System Internals Curriculum Resource Kit (CRK), which is a freely available collection of instructor resources to supplement operating system (OS) lectures and assignments with Windows kernel illustrations. The CRK provides PowerPoint presentation slides, experiments, lab descriptions, sample quizzes and assignments, in order to introduce case studies from the Windows kernel into operating system courses. The CRK, and all the components of the Windows Academic Program, are for academic, non-commercial use only.
At http://www.i.u-tokyo.ac.jp/edu/training/ss/lecture/new-documents/Lectures/ you can find an extensive presentation that covers all the parts of the windows kernel.
Microsoft also offers the Windows Research Kernel to faculty members, instructors or other people, who are teaching or conducting research in a highly accredited institute of higher education. The WRK includes most of the kernel sources from the latest released version of Windows, which supports the x64 architecture on the desktop.
Apart from David Solomon, there are also other companies, which offer seminars on windows internals, e.g. OSR and Azius.