driver writing != bus driving

Code Quality: Windows vs Linux vs FreeBSD vs Solaris

Diomidis Spinellis has written a good paper for the “30^th International Conference on Software Engineering” (ICSE ’08) that looks at the code quality of the source codes of Windows (WRK – Research Kernel based on Windows Server 2003), Linux, Solaris and FreeBSD. Diomidis has analyzed the source codes of these 4 kernels and uses some code metrics, in order to measure the quality of each kernel in each area.

For those, who don’t like reading lengthy texts, a summary of the results is at the end (section 5: summary and discussion). Each operating system has its own strengths and weaknesses, so there is no clear winner.

The paper can be found at http://www.spinellis.gr/pubs/conf/2008-ICSE-4kernel/html/Spi08b.pdf and all the queries for the code analysis can be found at http://www.spinellis.gr/sw/4kernel/.

Finally, Diomidis has a very lengthy list of classic reads at his website.
 

When Is WDF 1.7 RTM Coming Out?

Lately, I've been hearing the same question over and over: Where is the WDF 1.7 RTM? Many driver developers have gone to the onConnect website and there they see the following notice:

·**URGENT** - WDK WS08 - WDF ALERT

·         WDF Users:  Two patches from Windows Update (KB 938371 & KB 933607) interfere with installation of the Kernel Mode Driver Framework and the User Mode Driver Framework on Windows Vista.   Testing of the fixes is in progress.  Our current estimate to complete these fixes is the end of March.  Until then, we ask that you not release device drivers using either KMDF or UMDF version 1.7.  

For further information, contact WDFINFO@microsoft.com.

Sincerely,
KMDF Team

 

Of course, the end of March has passed and right now we're in April. So, what has happened? Let's start from the beginning.

First of all, I'll say that we had tested all WDF binaries and coinstallers and we were ready to ship with the 6001.18000 RTM WDK (i.e. the Windows Server 2008 WDK). Everybody was happy in the team, since we were on time and we were eagerly working on the new WDF version. Finally, the WDK went out and we had a mini-party :)

A few days after the release of the WDK I received a couple of emails saying that there is a problem with the KMDF 1.7 coinstaller: it cannot update a system, where KMDF 1.5 was already installed. Of course, this was a scenario that we had already tested (many times and using multiple configurations and test scenarios) and it seemed weird at first. However, the log files that I got were not lying: the files were not being updated. In the beginning, it was hard to believe that such a fundamental scenario was not working and it was even harder to convince other people in my team that there was a potential problem. Anyway, after some tests we found out that even in our machines (which we were using to test the KMDF 1.7 coinstallers), the coinstallers were not working!

So, while brainstorming with the WDF test lead, Shefali, she told me that the KMDF 1.5 binaries were part of the KB938371, which is a prerequisite for Vista SP1. Up to that moment I had no idea about the existence of this package. Anyway, indeed we verified that this package was creating a versioning conflict with the coinstaller and it was preventing the coinstaller from working correctly. All this happened within a couple of days, after we received the initial emails that were informing us about the installation failure. Immediately we informed people about workarounds on this issue. You can find the initial announcement that I made in OSR's ntdev mailing list here. The workaround for testing purposes (which can also be found in the above link) is to extract the files from the coinstallers and manually place them to their directory.

After that, we had to take the hit, remove the 6001.18000 WDK from the onConnect website and post the 6001.18001 WDK, which is exactly the same as the previous one, however it doesn't have the WDF coinstallers. Around that time, while searching for the best solution, we found (again, while brainstorming with Shefali) that there was an issue with the package of the UMDF coinstaller: it would not upgrade a small number of Windows Vista users. So, we created new coinstallers and started testing them. The code is EXACTLY the same as the one coinstaller that we had released in the 6001.18000 WDK, but we had to modify the update packages that are inside the coinstaller. Of course, right now our testing scenarios are checking everything, since we want to make sure that we're controlling all the unknown factors and there are no more suprises in the corner for us. Fortunately, testing is going well and we haven't found any issues. AFAIK, we're almost done testing the coinstallers and the only thing that remains is to find a way to re-release them. Our PM, Jeff, is currently working on this issue.

So, when are is WDF 1.7 coming out? We're doing our best. Most probably, if everything goes well, sometime in April. The whole issue has been as disturbing for you as it has for us and we want to overcome it asap.

Why Do We Need WDF Coinstallers?

Lately, I've been hearing lots of questions like "why do we need this bloated KMDF/UMDF coinstaller?" or "why don't you use an msi" or "why doesn't a WDM driver always need a coinstaller?", etc both in some mailing lists and in emails. So, I'd like to throw some light into this issue.

First of all, I'd like to say that the coinstaller is actually doing 2 very important tasks:

• Update the framework of the computer (i.e. that files that are on the disk), if necessary (i.e. if the coinstaller supports KMDF 1.7 and the computer has KMDF 1.5, then there will be an update, however if the coinstaller supports KMDF 1.5 and the computer has KMDF 1.7, then there is no reason for update)
• Configure the device node, after the driver files are copied to the disk

 
QUESTION 1: Why are the coinstallers so big? 

If you take a look at the WDF coinstallers, their size for x86 architectures are ~1mb. This gives a false impression that they are bloated. However, more than 99% of the size is attributed to the "update" part (1st bullet) and only a few kb are responsible for the "configure" part (2nd bullet). Let's see why this happens.

KMDF 1.7 currently supports all OSs from Windows 2k to Windows Vista SP1 (both client and client and server). UMDF 1.7 currently supports all OSs from Windows XP to Windows Vista SP1 (both client and client and server). This means that if you write your driver with UMDF/KMDF 1.7, then the same binary will work for any OS in that range. Let me repeat that: we ensure both source-code compatibility, as well as binary compatibility for all the above-mentioned operating systems! Currently, you CANNOT do this with any other driver model for Windows. If you write a WDM driver that works in Vista and takes advantage of the latest WDM features, this driver won't work in 2k. However, if you write a WDF 1.7 driver, then it will.

In order to do this, we have to update the operating system that you're installing the driver at. This is done by the coinstaller. The coinstaller has an embedded Windows Update packages inside it that is actually executed in the beginning of the installation, if it's needed. This package includes the latest version of the WDF files that correspond to the current coinstaller. So,  in the case of the KMDF 1.7 coinstaller (wdfcoinstaller01007.dll), we update wdfldr.sys and wdf01000.sys, which are located at %windir%\system32\drivers. The UMDF 1.7 coinstaller (WUDFUpdate_01007.dll) updates wudfhost.exe, wudfplatform.dll, wudfsvc.dll, wudfx.dll and wudfcoinstaller.dll (located at %windir%\system32), wudfpf.sys, wudfrd.sys (located at %windir%\system32\drivers). Currently, each coinstaller has 2 Windows Update packages: 1 for pre-Vista operating systems (e.g. 2000, XP, 2003, etc) and 1 for Vista (RTM, SP1). This happens, because the windows update technology changed in Vista. These 2 packages combined are around 99% of the coinstaller size. The other 1% is just the code for the configuration of the device and is pretty minimal (a few kb only).

So, if we didn't update the system, then our coinstallers would be really small. However, in that case it would not be possible for driver developers to write drivers that are source code compatible and binary compatible back to Windows 2000. WDM doesn't have full backwards binary compatibility.

QUESTION 2: How can you verify that what I'm saying is correct?

This is easy. Let's take a look at the KMDF 1.7 x86 coinstaller. Its size is 1098kb. Now, let's look at the KMDF files. wdf01000.sys is 492kb and wdfldr.sys is 35kb. Their sum is 492+35=527kb. I already said that we have 2 update packages in each coinstaller, so this size is multiplied by 2 and becomes 527*2=1054kb. Of course, each update package has additional files that are needed by Windows Update and everything is compressed, however you get the general idea. The same thing applies for the UMDF coinstaller, too.

However, if you want to delve deep into the coinstaller's contents, then you just need to disassemble the coinstaller and look at the files inside it. Bob has already written a post about how to do it here, but I'll repeat the steps for completeness:

• Drag-and-drop the coinstallers into Visual Studio or use File/Open to load the binary. You can also use any other resource extractor.
  
• By default, you'll see the coinstallers' resources. Look at the RCDATA resources. The KMDF coinstaller has a resource called WDFCAB_RESOURCE and the UMDF coinstaller has 2 resources: WUDF_UPDATE_VISTA_RTM (Vista update package) and WUDF_UPDATE_XP-SRV03 (XP and 2003 update package).
• Right click on the resource name, select export and find a directory to extract the resources.
• For KMDF, the filename needs to have a .cab extension. For KMDF, the pre-Vista update file needs to have a .exe extension, while the Vista one needs to have a .msu extension.
  
• If you uncompress the .cab KMDF file somewhere, you'll see a file called Microsoft Kernel-Mode Driver Framework Install-v1.7-Win2k-WinXP-Win2k3.exe (pre-Vista update package) and a file called (Microsoft Kernel-Mode Driver Framework Install-v1.7-Vista.msu).
• You can open the exe and the msus using a program that zips/unzips compressed files (like WinZIP, WinRAR, etc), or you can execute the KMDF exe with the  "/extract:<directory_to_extract_files> /quiet" arguments and extract the files.
  

 After that you can look at the file sizes, extract them from the coinstaller and see how big the coinstallers really are.

QUESTION 3: Why not use an msi?

As you know, there are 2 ways to install a device and its driver:

• Hardware-first: Plug-in the device, point the "add-new hardware wizard" to the inf and install the driver
• Software-first: Install the driver, then plug-in the device
  

In KMDF and UMDF we want to provide support for both ways and we don't want to change the model that windows drivers have been using so far. This means that we need an inf to install the driver. If we provided just an msi, which you'd have to install before pluging-in your device, this means that we're breaking the hardware-first installation. We don't want to do that, that's why we have the coinstaller So, the inf calls the coinstaller, which includes a windows update package. This package updates the WDF binaries that are on disk. From the user's perspective, it doesn't have any difference, if the coinstaller is internally using an msi, a windows update package or any other technology. So, just by replacing the update technology and using an msi, wouldn't give any benefit. On the contrary, we'd have to start development and testing from scratch and find solutions to problems like how it's possible to overcome Secure File Protection in Windows Vista in such a way that we don't break support for Windows 2000. At the same time, we also need to keep our binary small. That's why we're not changing the current architecture (at least until we find something better).

QUESTION 4: Why not use a different update mechanism?

Some alternatives here would be to put the Windows Update package in Windows Update (e.g. the same way that it works with applications that depend on DirectX and .NET). For example, we could have a small coinstaller that checks the UMDF/KMDF version that's installed in the system and then asks the user to download the package. The problem here is that this solution works well with APPLICATIONS, but not with DRIVERS. It's acceptable, that if you try to install a .NET 3.5 application, and you have .NET 2.0 installed in the system, then this case you'll have to connect to the internet, download .NET 3.5, install it and run the application. However, what happens, if you have KMDF 1.5 in the system and your brand new keyboard/mouse need KMDF 1.7? How do you control the computer then? Or how do you connect to the network, if the device that needs KMDF 1.7 is your network card? What if you have no internet connection at all and you want to use your brand-new cool gadget that is supported by a UMDF version that's newer than the one already installed in the system? So, this solution cannot work.

 

Summarizing, we need WDF coinstallers, because:

• We want both to update the framework (in all supported operating systems) and configure the device
• We need something that can be called from the inf, so that we don't break the driver installation methods and patterns
• We need to install DRIVERS and not APPLICATIONS
  

Debugging User-Mode Processes Using a Kernel-Mode Debugger

In this post I'll try to clarify some small details, that are related to debugging a user-mode process (focusing on a UMDF driver) using a kernel-mode debugger. So, the setup is that we have a test computer, where the UMDF echo driver is running and another computer, where windbg is running and we're using it as a kernel-mode debugger. A first thing to do in our case would be to see what modules are loaded:

kd> lm
start    end        module name
81800000 81b95000   nt         (export symbols)       ntkrnlmp.exe

Unloaded modules:
85dac000 85db4000   drmkaud.sys
85ce8000 85cf5000   crashdmp.sys
82a05000 82a10000   dump_ataport.sys
85d94000 85d9c000   dump_atapi.sys
85c27000 85c38000   dump_dumpfve.sys
8d618000 8d632000   serial.sys
88020000 88029000   kbdhid.sys
88e52000 88e65000   i8042prt.sys
88e63000 88e78000   WUDFRd.sys
880aa000 880b0000   nothing.sys

Ok, that's interesting. Just with a first a look it seems that only the kernel is loaded and that there are a few unloaded modules. This view is deceiving, though. Let's reload symbols and try again:

kd> .reload /f
Connected to Windows Vista 6000 x86 compatible target, ptr64 FALSE
Loading Kernel Symbols
........................
Loading User Symbols

Loading unloaded module list
..........
kd> lm
start    end        module name
81800000 81b95000   nt         (pdb symbols)          c:\Debuggers\sym\ntkrnlmp.pdb\E556D3F077BB42BB83B132247BE9C4942\ntkrnlmp.pdb
81b95000 81bc9000   hal        (pdb symbols)          c:\Debuggers\sym\halmacpi.pdb\AE84FF5D9CEE4D64927E629F756036841\halmacpi.pdb
82004000 82012000   PCIIDEX    (pdb symbols)          c:\Debuggers\sym\pciidex.pdb\0A98C6B81AB842C483351BCA042A9B1A1\pciidex.pdb
82012000 82019000   intelide   (pdb symbols)          c:\Debuggers\sym\intelide.pdb\BFCA935B0A6B47C2AA4B9F25100409F11\intelide.pdb
82019000 82029000   mountmgr   (pdb symbols)          c:\Debuggers\sym\mountmgr.pdb\6F08CCFAE97F4F139853B1769DAB0CF31\mountmgr.pdb
82029000 82038000   volmgr     (pdb symbols)          c:\Debuggers\sym\volmgr.pdb\3C43C06A961143719A6DF9F0B2A9699C1\volmgr.pdb
82038000 8205d000   pci        (pdb symbols)          c:\Debuggers\sym\pci.pdb\A5E895C861984D7393087EB0459E7FE01\pci.pdb
... [output has been truncated] ...

94b78000 94b8c680   WUDFRd     (pdb symbols)          c:\Debuggers\sym\WUDFRd.pdb\D92A3D77AEBE4FFE8EE42628096819371\WUDFRd.pdb

Ok, this seems more promising. The symbols have been loaded and we now see many more modules. We also see that the reflector (WUDFRd) is loaded. However, where is our UMDF driver? Now it's time to go back to some theory. The UMDF driver is actually a user-mode process. When you break into a kernel-mode debugger, you just break into an arbitrary process. Which one? The !process command will show us.

 kd> !process
PROCESS 818eff00  SessionId: none  Cid: 0000    Peb: 00000000  ParentCid: 0000
    DirBase: 00122000  ObjectTable: 85800178  HandleCount: 593.
    Image: Idle
    VadRoot 00000000 Vads 0 Clone 0 Private 0. Modified 215. Locked 0.
    DeviceMap 00000000
    Token                             858037d8
    ElapsedTime                       00:00:00.000
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (4, 50, 450) (16KB, 200KB, 1800KB)
    PeakWorkingSetSize                0
    VirtualSize                       0 Mb
    PeakVirtualSize                   0 Mb
    PageFaultCount                    0
    MemoryPriority                    BACKGROUND
    BasePriority                      0
    CommitCharge                      0

        THREAD 818efac0  Cid 0000.0000  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0

So, I'm debugging the idle thread. Now it's time to switch to the echo driver. In order to do that, we need to find all the process and search for wudfhost.exe.

kd> !process 0 0 WUDFHost.exe
PROCESS 85709738  SessionId: 0  Cid: 0624    Peb: 7ffde000  ParentCid: 03d8
    DirBase: 1ba3d000  ObjectTable: 93a5cd40  HandleCount: 161.
    Image: WUDFHost.exe
 

Ok, now that I've found the process that I need to connect to, I need to switch to it. However, it's not enough just to do a

.process 85709738

This would just tell windbg to change just the debugger context, i.e. show what's going on in that process (e.g. the stack), however remain in the context of the idle process. If we put breakpoints, then they will be set in the idle process and NOT in the process that we want to debug! We need to do a context switch to that process.

So, one way to do this is:

kd> .process /i 85709738
You need to continue execution (press 'g' <enter>) for the context to be switched. When the debugger breaks in again, you will be in the new process context.
kd> g
Break instruction exception - code 80000003 (first chance)
nt!RtlpBreakWithStatusInstruction:
818356e8 cc              int     3

However, as my friend Patrick (the debugging guru) told me: with the /i switch you have to think about the possible traps of releasing the system for the context switch to happen. After you press "g", you have to wait for the windows scheduler to do the context switch. And while the process is being brought in to context, you also have to think you’re releasing the system.  On a single hardware thread execution machine, switching in to the process context won’t cause any issues (hopefully), but what about when you get in to an SMP / Multicore environment?  The context switch happens and the flagged process is locked, so nothing is scheduled from that process, but what about all those other threads in queue?

So, the best option here is to do

kd> .process /r /p 85709738

In this case, the debugger does the context switch immediately (as opposed to the windows scheduler) and there is no waiting time. 

Anyway, now we're in the correct context. Let's see what's loaded:

kd> lm
start    end        module name
00f60000 00f86000   WUDFHost   (pdb symbols)          c:\Debuggers\sym\WUDFHost.pdb\AEC9577097EE46CAAE1D12CDA87064E61\WUDFHost.pdb
73c80000 73c87000   WUDFEchoDriver   (export symbols)       WUDFEchoDriver.dll
73d00000 73d50000   WUDFx      (pdb symbols)          c:\Debuggers\sym\WUDFx.pdb\A416B8E3114F4EA4BE1EAC879505D2771\WUDFx.pdb
74460000 74491000   WUDFPlatform   (pdb symbols)          c:\Debuggers\sym\WUDFPlatform.pdb\2BA7585C1C874EBBB9B082D649DFD80F1\WUDFPlatform.pdb

[... snip ...]
8270d000 8271f000   WUDFPf     (pdb symbols)          c:\Debuggers\sym\WUDFPf.pdb\98F0D4EDE42944CF8C87B11DD7985EC41\WUDFPf.pdb
94b78000 94b8c680   WUDFRd     (pdb symbols)          c:\Debuggers\sym\WUDFRd.pdb\D92A3D77AEBE4FFE8EE42628096819371\WUDFRd.pdb

Ok, now I can see my echo driver (WUDFEchoDriver.dll). If the symbols are not loaded, then a .reload /f would fix everything. Of course, if there are many WudfHost.exe processes that are running (i.e. you are using many UMDF drivers), then you might have to search for your driver (i.e. WUDFEchoDriver.dll) in the list of loaded modules and if it's not there, then you'll have to switch to another instance of WudfHost.exe.

Now it's time to do some debugging. The UMDF extension is called wudfext. Let's try and see, if it's loaded:

kd> !wudfext.help
The call to LoadLibrary(wudfext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

No it's not... Ok, let's see what's loaded:

kd> .chain
Extension DLL search Path:
    c:\Debuggers\WINXP;c:\Debuggers\winext;c:\Debuggers\winext\arcade;c:\Debuggers\pri;c:\Debuggers;c:\Debuggers\winext\arcade;
Extension DLL chain:
    c:\Debuggers\winext\wdfkd: image 6.0.6001.16470, API 1.0.0, built Sun Feb 25 04:47:28 2007
        [path: c:\Debuggers\winext\wdfkd.dll]
    dbghelp: image 6.7.0004.0, API 6.0.6, built Thu Mar 15 10:39:24 2007
        [path: c:\Debuggers\dbghelp.dll]
    ext: image 6.7.0004.0, API 1.0.0, built Thu Mar 15 10:39:14 2007
        [path: c:\Debuggers\winext\ext.dll]
    exts: image 6.7.0004.0, API 1.0.0, built Thu Mar 15 10:38:51 2007
        [path: c:\Debuggers\WINXP\exts.dll]
    kext: image 6.7.0004.0, API 1.0.0, built Thu Mar 15 10:38:58 2007
        [path: c:\Debuggers\winext\kext.dll]
    kdexts: image 6.0.6001.16470, API 1.0.0, built Thu Mar 15 10:58:28 2007
        [path: c:\Debuggers\WINXP\kdexts.dll]

 I know that my extension is in the WDK at, C:\WinDDK\6001\bin\x86\wudfext.dll. Let's load it:

kd> !load C:\WinDDK\6001\bin\x86\wudfext.dll
kd> .chain
Extension DLL search Path:
    c:\Debuggers\WINXP;c:\Debuggers\winext;c:\Debuggers\winext\arcade;c:\Debuggers\pri;c:\Debuggers;c:\Debuggers\winext\arcade
Extension DLL chain:
    C:\WinDDK\6001\bin\x86\wudfext.dll: image 6.0.6001.17138, API 1.0.0, built Thu Jan 17 23:39:16 2008
        [path: C:\WinDDK\6001\bin\x86\wudfext.dll]
    c:\Debuggers\winext\wdfkd: image 6.0.6001.16470, API 1.0.0, built Sun Feb 25 04:47:28 2007
        [path: c:\Debuggers\winext\wdfkd.dll]
    dbghelp: image 6.7.0004.0, API 6.0.6, built Thu Mar 15 10:39:24 2007
        [path: c:\Debuggers\dbghelp.dll]
    ext: image 6.7.0004.0, API 1.0.0, built Thu Mar 15 10:39:14 2007
        [path: c:\Debuggers\winext\ext.dll]
    exts: image 6.7.0004.0, API 1.0.0, built Thu Mar 15 10:38:51 2007
        [path: c:\Debuggers\WINXP\exts.dll]
    kext: image 6.7.0004.0, API 1.0.0, built Thu Mar 15 10:38:58 2007
        [path: c:\Debuggers\winext\kext.dll]
    kdexts: image 6.0.6001.16470, API 1.0.0, built Thu Mar 15 10:58:28 2007
        [path: c:\Debuggers\WINXP\kdexts.dll]

Ok, it seems to be loaded. Let's try and use it:

kd> !wudfext.help
The call to LoadLibrary(wudfext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

It's not working! Why's that? Notice that .chain mentions wudfext.dll with the full path, whereas everything else just uses the filename without the extension. Let's try the full name:

kd> !C:\WinDDK\6001\bin\x86\wudfext.dll.help
Help for host process debug extension:
  umdevstacks [flags]                      - This dumps all the device stacks in the host process.
  umdevstack <address> [flags]             - This dumps detailed info on each device stack.
  umirps                                   - This dumps UM Irps in the host process.
[... the rst of the output is truncated ...]

Ok, so it's loaded indeed. If I try and use an extension directly, then it works:

kd> !umirps
Number of pending IRPS: 0x0
####  CWudfIrp          Current Type                         UniqueId          KernelIrp
----  ----------------  -----------------------------------  ----------------  ---------

However, if there are extensions with the same name (e.g. !help) in other files, this is a problem. Let's make it easier to use:

kd> !unload C:\WinDDK\6001\bin\x86\wudfext.dll
Unloading C:\WinDDK\6001\bin\x86\wudfext.dll extension DLL
kd> .extpath+ C:\WinDDK\6001\bin\x86
Extension search path is: c:\Debuggers\WINXP;c:\Debuggers\winext;c:\Debuggers\winext\arcade;c:\Debuggers\pri;c:\Debuggers;c:\Debuggers\winext\arcade;C:\WinDDK\6001\bin\x86
kd> !load wudfext

kd> .chain
Extension DLL search Path:
c:\Debuggers\WINXP;c:\Debuggers\winext;c:\Debuggers\winext\arcade;c:\Debuggers\pri;c:\Debuggers;c:\Debuggers\winext\arcade;C:\WinDDK\6001\bin\x86
Extension DLL chain:
    wudfext: image 6.0.6001, API 1.0.0, built Thu Jan 17 23:39:16 2008
        [path: C:\WinDDK\6001\bin\x86\wudfext.dll]

kd> !wudfext.help
Help for host process debug extension:
  umdevstacks [flags]                      - This dumps all the device stacks in the host process.
  umdevstack <address> [flags]             - This dumps detailed info on each device stack.
  umirps                                   - This dumps UM Irps in the host process.
[... the rest of the output is truncated ...]

Ok, so now it works. As a final note, just remember that the UMDF extension should be from the same WDK as the UMDF framework, e.g. if you're using UMDF 1.7 RC1, then wudfext should be 1.7 RC1. The driver can be compiled with any framework version (as long as the major version of UMDF on the machine is the same as the one that the driver was compiled with and the minor version of the running UMDF is greater or equal to the driver's). So, the UMDF extension doesn't have to be the same as the version of the framework that the driver was compiled with.

Developing Windows Drivers With Visual Studio

Today morning I received an email from Patrick with a picture of Visual Studio with Intellisense on a WDF driver. Ok, I have to admit that in the beginning I thought that Patrick was using Photoshop! He's a guy, who just doesn't like GUIs in the first place! He can just go on and on about the advantages of using kd instead of windbg! He even refuses to look at the source code (even if it's available), since "he can look at the assembly"! I have to admit that until now I thought that he was doing "copy con driver.c" to write drivers... and then use edit.com from a full-screen command prompt to do additional editing. Anyway, it seems that I was wrong.

So, in order to be sure that Patrick hadn't used Photoshop (or just opened the screenshot with a hex editor and manually edited it, in order to add Intellisense!), I went to his office. Over there, indeed he showed me that it's really easy to Intellisense and help integration to Visual Studio (yay!). That's nice stuff! My next question was, if it's possible to load wudfext.dll (the UMDF windbg extension) from Visual Studio and use the UMDF extensions from the VS debugger, however it's not possible (even though there's some info on how to load windbg extensions at http://www.osronline.com/showThread.cfm?link=66160), because not all windbg interfaces are implemented by Visual Studio.

Anyway, Patrick has written a post saying that indeed he's been using Visual Studio for ever, in order to develop Windows Drivers. If more people ask him tips on how to make VS the best editor for Windows Drivers, he promised that he'll share a small part of his infinite wisdom :) So, feel free to ask him and/or send him an email!

 

DISCLAIMER: Visual Studio is not officially supported by Microsoft for driver development. The only currently supported build environment is the command window that comes with the WDK. Any opinions expressed in this article don't affect Microsoft's policy on this issue.

Impressions from Seattle Code Camp v3.0

During this weekend I went to the Seattle Code Camp v3.0. The talks were mostly oriented towards .NET,so it was a good opportunity for me to get a better understanding of all those buzzwords that are unknown to the world of drivers. I also had the opportunity to talk to many interesting people and listen to their ideas about .NET and software development in general.

I started my first day at the code camp by going to Jason Haley's talk, which focused on the .NET Reflector (nothing to do with the UMDF Reflector :) ), which is the most commonly used .NET dissassembler. Jason covered both the reflector, as well as some plug-ins. I had seen the Reflector in use before, however it was a nice overview. I've been reading Jason's link-blog very often and fortunately we had the time during the break to discuss a little bit more.

After his talk, it was time for me to understand what Silverlight and WPF (Windows Presentation Foundation) are. I've heard lots of good stuff about them, however I had no idea what they actually do (apart from the fact that they have to do with the way that applications look). So, after looking at some cool demos in Kelly White's talk, Adam Kinney explained all the theory behind it (what Silverlight is, what are the differences between v1.0 and 2.0, what are the differences between Silverlight, WPF and Flash etc). Adam also went through the development of a cool app that can be shown both in a web browser (using Silverlight) and in the Vista sidebar (using WPF).

Then I decided to move to a bit more familiar space by going to Wayne Berry's talk on hashing. Wayne showed some practical aspects of how hashing is implemented in applications (as opposed to the mathematical theory behind it), as well as potential pitfalls that developers face. Jim McKeeth also had a very interesting talk on implementing cryptography using the Win32 CryptoAPI, however I had to miss it, because I wanted to go to see the xUnit.NET talk from Brad Wilson and James Newkirk. The xUnit.NET talk presented some really cool ideas on unit testing, however I hope that Brad will upload his Cryptography presentation to the web, since he did the same with his "Advanced Downloads" presentation. Now that I'm thinking about it, the Cryptography talk was the only non-.NET talk of the event and I didn't go to it!

Anyway, the next morning, it was time to understand WCF (Windows Communication Foundation).  Robert Green gave an excellent presentation. He explained what WCF is and he created a service that could be reached both through TCP and through HTTP (depending on the client being local or remote) and showed how easy interoperability has become with the use of WCF.

The last talk that I went to was Charles Sterlings' talk on how Visual Studio 2008 Team System's features that are related to Application Lifecycle Management. Ok, first of all, I'd like to say that Charles is an excellent presenter. He had only demos and no slides (!) His whole talk was interactive and he actually asked us (the viewers) to do the demos for him (!) I also got a T-shirt with the Visual Studio logo :) Anyway, during his talk I couldn't prevent myself from (mentally) comparing the tools that are being used by application-developers to the ones that are being used by driver developers. Charles was showing all these cool features that VS2008 has and I just had to think that in order to compile a driver I need to go to a command prompt and type bcz (ok I know that I can use batch files, in order to call bcz from VS, however this is not the main point)... Also, I'm using an editor without Intellisense (and that's even more important than the previous issue)... No help-file integration... I was feeling like a prehistoric person in the middle of modern New York. I know that Doron has also blogged about this feeling. When I told a couple of .NET developers about the tools in driver-land they couln't believe their ears. Anyway, hopefully sometime in the future, driver developers might reach a critical mass and things might change... Until then...

Unfortunately I had to leave before the end of the event, otherwise I would like to see more stuff about WCF and Ajax. However, the event was very well organized (kudos to everybody, who helped organize it). I understood several buzzworks (Silverlight, Moonlight, WPF, WCF, etc). All the talks were really interesting and I'm sure that all the visitors enjoyed it. I know that I'll definately go to Seattle Code Camp v4.0 :)

My List of Top 5 Windows Books

I've been reading quite a bit for the last couple of months and I compiled my list of the 5 Windows development books that I want to complete:

1) Windows Internals (finished): The classic book by Mark Russinovich and David Solomon is now in its 5th edition. I've read the 4th one and I think that it's a must-read for every windows developer.

2) Developing Drivers with the Windows Driver Foundation (finished): I just finished the book and I think that its the best book currently available to windows driver developers, especially for beginners. I'll write a review shortly, however I think that it's definately the most complete and at the same time easier to read book on windows drivers.

3) Advanced Windows Debugging (not started): After John Robbins decided not to write a debugging book for Win32 programming, I was wondering which book would take its place. I've read lots of good reviews about this book and I have it in my bookcase, so I'm eager to read it. Just with a first glance it seems to be exactly what I was searching for.

4) Windows via C/C++ (not started): This is the latest version for "Programming Applications for Microsoft Windows". It covers application development in Windows. I've taken a class by Jeffrey Richer and I know that he's definately both a good developer and a good teacher, so I'm waiting to read it.

5) Programming the Windows Driver Model (half-read): This is the most tough-to-read and most advanced book in windows driver development. It covers many aspects of windows in depth, however I don't feel that I'm ready to read it yet. I've tried a couple of times in the past and I failed miserably. I think that really understanding this book means that your level is definately above intermediate.

Of course, there are other books, like Subverting the Windows Kernel, however these are my top choices. I'd love to hear your comments, as well as your top choices.

WDF 1.7 RC1 has been released!

Last Friday (12/7/2007) we completed the final build for the Windows Server 2008 RC1 WDK (you can find it here), which includes WDF 1.7 RC1 (both UMDF 1.7 and KMDF 1.7). Beware, though, that this is just the WDF RC1 version and not the RTM one, so you can use it for development and testing, but not to ship it with commercial drivers. I know that everybody in my team is really happy about it, since now KMDF and UMDF are available for many more platforms. KMDF 1.7 supports 2000 Service Pack 4, XP, Server 2003, Vista and Server 2008. UMDF supports XP Service Pack 2, Server 2003, Vista and Server 2008. This is something that we've been asked for many times.

Here's a table, which shows which versions of Windows are supported by each version of WDF:

 

Version	Framework	Windows 2000 SP4	Windows XP SP2	Windows Server 2003 SP1	Windows Vista	Windows Server 2008	Release Date
1.0	KMDF		Yes	Yes			Nov 2005
	UMDF		Yes				Oct 2006 (with WMP 11)
1.1	KMDF	Yes	Yes	Yes			April 2006
1.5	KMDF	Yes	Yes	Yes	Yes		Nov 2006 (with Windows Vista)
	UMDF		Yes		Yes
1.7	KMDF	Yes	Yes	Yes	Yes (SP1)	Yes	With Windows Server 2008
	UMDF		Yes	Yes	Yes (SP1)	Yes

 

What's new with WDF 1.7? Let's start with UMDF:

• Bug fixing in many critical areas (power management. more stable reflector, driver impersonation, hardening against all sorts of errors, etc): Lots of background work to make this version more secure and robust.
• Sample updates: New socket echo driver (uses ATL), Idle and Wake support in Fx2 sample driver, additional Fx2 driver that uses impersonation and shows how to cancel synchronous requests, hybrid sample that uses winusb, etc.
•  New debugger extensions to dump I/O target, USB and File objects.

Now for KMDF:

• New wait-wake policy between parent and children
  
• New samples: PCMCIA smartcard reader,  HID minidriver, NDIS 6.0, bluetooth
• New tools: DDI fault injection tester, driver call tracer
• Lots of bug fixing in key areas

New tool: WdfVerifier

• GUI tool that manages:
• KMDF verifier settings
• UMDF debug/trace and other global settings
• Makes it easier to hook debuggers to UMDF drivers
• Works on all versions of UMDF and KMDF
• Takes care of niceties like KMDF loader spew, flips DbgPrint on and off, if the OS suppresses it, etc

What's changed in the coinstallers?

• Support for all these operating systems! From version 1.7, the coinstaller supports many more operating systems. Also, the way that we update the frameworks in Vista (and later) is very different than what was happening pre-Vista. We managed to overcome lots of issues in this journey.
• File size is kept low: Both coinstallers were modified to use as few update packages as possible (e.g. KMDF uses one package for 2k, XP, 2k3, instead of 3). This decreased the total size of the coinstaller by a lot (the KMDF coinstaller was decreased by 50%!).
• In the previous versions of WDF, many times a user would install a driver without any problem and the driver would be shown in the device manager with a "!". After reboot everything would "magically" be solved. Now we've changed the way that the coinstallers determine, if a reboot is needed (and prompt the user accordingly, instead of being silent) and when an update is necessary.
• Now we have lots more debugging information in %windir%\setupact.log. As I wrote in my previous post, setupact.log is the main logging file for the coinstaller. Now the information is more verbose and in case of an error, there's information that tells a user which additional files to look at. (this also means that I need to write a post about the logging information of the WDF 1.7 coinstallers)
• While fixing all the above, we fixed several bugs in the coinstallers (parsing infs, solving edge cases to help external companies, etc). As far as we know, many of the issues that we've been informed about, have now been fixed. So, please if there are any new issues that you see with the new coinstallers please tell us either by contacting me, or Bob, or the alias wdfinfo@microsoft.com or any of the above blogs for WDF and KMDF that I've mentioned.

Warning: "Ops" we did it (hopefully not again) for ia64

We'd like to warn all the ia64 WDF developers that there are a few "small" problems with the ia64 coinstallers. Unfortunately, we discovered them too late and we didn't have time to fix them for RC1 (as I wrote above, the Windows Server 2008 RC1 WDK deadline had arrived). Currently, these errors have been fixed and won't exist in RTM:

• The ia64 KMDF and UMDF fre coinstallers install chk KMDF/UMDF files and the opposite (i.e. the chk coinstallers install fre files). If this doesn't bother you, then no problem (i.e. you can test your drivers regardless of whether UMDF/KMDF is fre or chk. Just keep in mind that in some cases you might see some asserts in the chk versions). However, if you're using WDF in ia64 and want to use fre files you'll have to use the workaround that's described in Bob's post. This means that you'll have to install the chk files using the fre coinstaller and then extract the fre files from the chk coinstaller. This was caused by an error in the scripts of the build lab.
  
• UMDF doesn't install in ia64 Windows Server 2008 (it works fine in all other ia64 versions of Windows). The UMDF coinstaller checks if the WudfPf service is installed. If it isn't, then it tries to update the framework, however it fails to do so and the installation stops. This is because of a bug in the actual coinstaller (the check for WudfPf should not be there). So, if you want to bypass this issue just use "sc create" to create a fake WudfPf service (it would be nice to make it point to the binary at %windir%\system32\drivers\wudfpd.sys, however apart from that, you can use any arguments that you like). After creating the fake WudfPf service, UMDF should install in ia64 Windows Server 2008 without any problem.

The WDF 1.7 team

The above is just a high-level summary of the coinstaller changes. There were many changes under the covers, lots of testing and hopefully everything will work smoother than before. I know that Bob had a blast testing them all these months :P I also know how much our Test Lead, Shefali, and my manager, Eliyas, had to fight the last couple of months in the War room, so that we can have as many bugs fixed as possible before the Windows Server 2008 deadline. Also, I've lost count of how many times I've gone to Peter and Vishal asking them about the coinstaller internals (for UMDF andf KMDF respectively). And Patrick, who loves to walk in front of my office and stare at me weirdly at random moments for no apparent reason (I don't know why he's doing that... send him an email and ask him :) ). Of course there are more people, who contributed to WDF 1.7. Our devs: Praveen, Abhishek (together with Peter they're the 3 UMDF gurus... or the Three Musketeers of UMDF... ok I just made up the last one, however it sounds so cool!), Egidio (he's a human quality gate... if your code passes his code review, then it has to be good!), Guru (unfortunately for the team, he's decided to switch jobs. Good luck with your new position!) and Kumar (who just moved from the test team to the dev team. Welcome!). Our amazing test team: Shyamal (for UMDF) and Ravi (for KMDF). Our PMs: Jeff and Rob (now it's a good time to bust a long-living myth... NO! PM doesn't stand for "Provider of Meals"! They're doing an excellent job in WDF! :) ). Yeah, we've had lots of fun during this period. I love my team :)

Analyzing the Installation of UMDF and KMDF 1.5 drivers

The goal of this post is to provide an insight about the installation of UMDF and KMDF 1.5 drivers and to show how to overcome any potential problems. For demo purposes I'll be using the UMDF and KMDF echo driver, which can be found at the WDK (%WinDDK%\6000\src\umdf\echo for UMDF and C:\WinDDK\6000\src\kmdf\echo for KMDF).

Let's start with UMDF. In order to install the driver you need to put the following files in the same directory:

• The UMDF coinstaller (%WinDDK%\6000\redist\wdf\x86\WUDFUpdate_01005.dll)
• The echo driver (%WinDDK%\6000\src\umdf\echo\objfre_wlh_x86\i386\WUDFEchoDriver.dll)
• The inf file (%WinDDK%\6000\src\umdf\echo\objfre_wlh_x86\i386\WUDFEchoDriver.inf)
• devcon (%WinDDK%\6000\tools\devcon\i386\devcon.exe)
• Even though it's not mandatory, it might be useful for debugging to have the pdb file of the driver (%WinDDK%\6000\src\umdf\echo\objfre_wlh_x86\i386\WUDFEchoDriver.pdb)
  

All the above paths are based on the fact that I'm using and x86fre build. If you are using a different architecture or want to have a chk build, then please modify the paths accordingly.

Now in order to install the driver, you can go to a command prompt and use the command
devcon install WUDFEchoDriver.inf WUDF\Echo

If everything goes well, you should see the following output:

>devcon install WudfEchoDriver.inf WUDF\Echo
Device node created. Install is complete when drivers are installed...
Updating drivers for WUDF\Echo from WudfEchoDriver.inf.
Drivers installed successfully.

If there's a problem, then you'll see something like:

>devcon install WudfEchoDriver.inf WUDF\Echo
Device node created. Install is complete when drivers are installed...
Updating drivers for WUDF\Echo from WudfEchoDriver.inf.
devcon failed.

So, how do we start to find out what the problem is?

First let's talk a little bit about how the installation works. Devcon calls setupapi (which is an API provided by the PNP manager) to do the installation. Setupapi parses the inf file and sees that the driver needs a coinstaller, in order to install. Then, the PNP manager calls the coinstaller both before doing the actual installation (i.e. add the driver to the driver storage, etc), so that the coinstaller can perform some pre-device installation functions (at this time the WDF coinstallers update the version of the framework that's installed in the system), as well as after doing the actual installation.

So, the first file that we'll look at is %windir%\setupact.log. This file contains the logging that is made by the coinstaller. In the case of succesful installation, it should contain something like the following:

[11/27/2007 17:30.47.097] WudfUpdate: Locating resource stream WUDF_UPDATE_XP.
[11/27/2007 17:30.47.160] WudfUpdate: unpacking update from resource to Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe.
[11/27/2007 17:30.47.191] WudfUpdate: Temporary path is C:\WINDOWS\Temp\WDF1CE.tmp.
[11/27/2007 17:30.47.253] WudfUpdate: Invoking update with command line "/quiet /ER /log:"%WINDIR%\temp\wudf_update.log"".
[11/27/2007 17:30.47.316] WudfUpdate: Invoking ""C:\WINDOWS\Temp\WDF1CE.tmp\Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe" /quiet /ER /log:"C:\WINDOWS\temp\wudf_update.log"".
[11/27/2007 17:30.47.347] WudfUpdate: Waiting for update to terminate.
[11/27/2007 17:31.25.237] WudfUpdate: Update process returned 0.
[11/27/2007 17:31.25.362] WudfUpdate: WUDF version 1.5.0 () was installed successfully.
[11/27/2007 17:31.25.408] WudfUpdate: Cleaning up update.
[11/27/2007 17:31.25.455] WudfUpdate: Loading configuration coinstaller from C:\WINDOWS\system32\wudfcoinstaller.dll.
[11/27/2007 17:31.25.487] WudfCoInstaller: ReadWdfSection: Checking WdfSection [Echo_Install.NT.Wdf]
[11/27/2007 17:31.25.518] WudfCoInstaller: UMDF Service WUDFEchoDriver is already installed - removing existing settings in preparation for setting new ones.
[11/27/2007 17:31.25.549] WudfCoInstaller: Configuring UMDF Service WUDFEchoDriver.
[11/27/2007 17:31.25.580] WudfCoInstaller: Adjusting binpath for service WudfPf to allow start at boot.
[11/27/2007 17:31.25.612] WudfCoInstaller: Changing binpath for service WudfPf from system32\DRIVERS\WudfPf.sys to \SystemRoot\system32\DRIVERS\WudfPf.sys.
[11/27/2007 17:31.26.205] WudfCoInstaller: Service WudfPf started successfully.
[11/27/2007 17:31.28.252] WudfCoInstaller: Service WudfSvc started successfully.
[11/27/2007 17:31.28.346] WudfCoInstaller: Final status: error(0) The operation completed successfully.
[11/27/2007 17:31.30.689] WudfCoInstaller: Created marker file C:\WINDOWS\system32\drivers\umdf\Msft_User_WUDFEchoDriver_01_05_00.Wdf.

So, what do we see here?
First of all, the coinstaller locates the resource WUDF_UPDATE_XP. You can look at all the resources that are included in a dll by doing a drag-and-drop of the dll to an open window of Visual Studio. If you drop the coinstaller, you'll see the WUDF_UPDATE_XP resource. So, the coinstaller extracts the resource to C:\WINDOWS\Temp\WDF1CE.tmp\Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe and calls C:\WINDOWS\Temp\WDF1CE.tmp\Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe" /quiet /ER /log:"C:\WINDOWS\temp\wudf_update.log"". This process updates the system to UMDF 1.5.

This means that the following files were updated to version 1.5:

• %windir%\system32\wudfx.dll (contains the UMDF APIs)
• %windir%\system32\wudfsvc.dll (Manages user-mode driver host processes)
• %windir%\system32\wudfplatform.dll (UMDF internal library)
• %windir%\system32\wudfhost.exe (UMDF host, which loads the driver dlls)
• %windir%\system32\wudfcoinstaller.dll (UMDF configuration coinstaller)
• %windir%\system32\drivers\wudfpf.sys (Provides communciation services for UMDF components)
• %windir%\system32\drivers\wudfrd.sys (Reflects device requests to user-mode driver drivers)
  

After that, the UMDF update coinstaller (WUDFUpdate_01005.dll), which is responsible for updating the system to the newest UMDF version, calls the UMDF configuration coinstaller (%windir%\system32\wudfcoinstaller.dll), which is responsible for configuring the new device. That's why the logging prefix changes from "WudfUpdate" to "WudfCoInstaller". The config coinstaller reads the inf file, finds that I had already installed the echo driver in the past, updates the corresponding driver settings, and starts the 2 UMDF services (WudfPf and WudfSvc). After that, the marker file is created. If you go to the device manager, the echo driver should be installed succesfully under the name "Sample WUDF Echo Driver". Also, the binary of the driver should be installed at %windir%\system32\drivers\UMDF\WUDFEchoDriver.dll.

Another interesting file to look at is %windir%\setupapi.log (for Windows XP) and %windir%\inf\setupapi.dev.log (for Vista). If you search for WUDF\echo inside the file (which is the hardware id that you passed to devcon, when you were installing the driver), then you'll find more information about the installation. In Vista, you can search for lines that begin with !, which correspond to warning, and with !!!, which correspond to error. The former means that something happened, however the installation continued (e.g. the driver was unsigned), but the latter means that the installation stopped because of an error. So, in case of a failed installation, you can search for !!! and find the problem. For example, if an installation fails in Vista 64-bit, because of unsigned drivers this is where the error will be logged.

So, let's see a scenario, where the installation failed. By looking at %windir%\setupact.log we see:

[11/27/2007 17:46.46.379] WudfUpdate: Locating resource stream WUDF_UPDATE_XP.
[11/27/2007 17:46.46.426] WudfUpdate: unpacking update from resource to Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe.
[11/27/2007 17:46.46.473] WudfUpdate: Temporary path is C:\WINDOWS\Temp\WDF1D8.tmp.
[11/27/2007 17:46.46.535] WudfUpdate: Invoking update with command line "/quiet /ER /log:"%WINDIR%\temp\wudf_update.log"".
[11/27/2007 17:46.46.566] WudfUpdate: Invoking ""C:\WINDOWS\Temp\WDF1D8.tmp\Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe" /quiet /ER /log:"C:\WINDOWS\temp\wudf_update.log"".
[11/27/2007 17:46.46.613] WudfUpdate: Waiting for update to terminate.
[11/27/2007 17:46.48.629] WudfUpdate: Update process returned 63745.
[11/27/2007 17:46.48.676] WudfUpdate: WUDF was already installed.
[11/27/2007 17:46.48.707] WudfUpdate: Cleaning up update.
[11/27/2007 17:46.48.754] WudfUpdate: Loading configuration coinstaller from C:\WINDOWS\system32\wudfcoinstaller.dll.
[11/27/2007 17:46.48.785] WudfCoInstaller: ReadWdfSection: Checking WdfSection [Echo_Install.NT.Wdf]
[11/27/2007 17:46.48.816] WudfCoInstaller: Error reading section [(null)] key UmdfLibraryVersion - status(E0000102) <no error text>.
[11/27/2007 17:46.48.848] WudfCoInstaller: Final status: status(E0000102) <no error text>

In this case, we see that the proccess returns 63745, which means that UMDF 1.5 was already installed. The configuration coinstaller is loaded, however there's an error, when the inf file is read. More specifically, I had modifies the UmdfLibraryVersion directive to be empty, so we're seeing this error. So, this way you can find if there are errors with your inf file.

Let's look at another example. The %windir%\setupact.log file has:

[11/27/2007 18:14.09.139] WudfUpdate: Locating resource stream WUDF_UPDATE_XP.
[11/27/2007 18:14.09.186] WudfUpdate: unpacking update from resource to Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe.
[11/27/2007 18:14.09.201] WudfUpdate: Temporary path is C:\WINDOWS\Temp\WDF1DF.tmp.
[11/27/2007 18:14.09.248] WudfUpdate: Invoking update with command line "/quiet /ER /log:"%WINDIR%\temp\wudf_update.log"".
[11/27/2007 18:14.09.311] WudfUpdate: Invoking ""C:\WINDOWS\Temp\WDF1DF.tmp\Microsoft User-Mode Driver Framework Install-v1.0-WinXP.exe" /quiet /ER /log:"C:\WINDOWS\temp\wudf_update.log"".
[11/27/2007 18:14.09.358] WudfUpdate: Waiting for update to terminate.
[11/27/2007 18:14.11.514] WudfUpdate: Update process returned 61442.
[11/27/2007 18:14.11.561] WudfUpdate: update returned error 0xf002 - error(61442) <no error text>.
[11/27/2007 18:14.11.592] WudfUpdate: Cleaning up update.
[11/27/2007 18:14.11.623] WudfUpdate: Error updating UMDF - error(61442) <no error text>.  Aborting installation.

For some reason, the update process returned an error, however no further explanation is provided. In order to find the problem, we need to go to %WINDIR%\temp\wudf_update.log. This is the log file that is created every time that the update process is called. In our case, I have the message:

0.031: ================================================================================
0.031: 2007/11/27 18:14:11.108 (local)
0.031: e:\899f0514c88d175e24bd\update\update.exe (version 6.3.4.0)
0.047: Hotfix started with following command line: /quiet /ER /log:C:\WINDOWS\temp\wudf_update.log
0.047: CheckSystem: GetCheckedFree failed :STATUS_CHECKED_FREE_MISMATCH
0.047: DoInstallation: CheckSystem Failed: 0xf002
0.047: WUDFCustom: Entering callback EndInstallation.
0.047: WUDFCustom: Entering callback FailedInstallation.
0.047: Wudf01005 Setup cannot update a checked (debug) system with a free (retail) version of Wudf01005, or vice versa.
0.047: Update.exe extended error code = 0xf002

So, the reason of the error was that I tried to use the debug version of the UMDF coinstaller (located at %WinDDK%\6000\redist\wdf\x86\WUDFUpdate_01005_chk.dll), even though I'm using a retail version of Windows XP.

If you want to uninstall the driver, you can either go to the device manager, right click on the driver and select uninstall (make sure that you check the box
that's asking you, if you want to delete the driver files, too) or you can use devcon. For this scenarion, you need to go to %windir%\inf and look at the oem*.inf files. You'll find the one that corresponds to the inf file of the echo driver(just looking at the first few lines will do the trick) and issue the command
devcon dp_delete oemXX.inf
(where XX is the number that corresponds to the inf of the echo driver)

So, when there's an installation problem with UMDF, we need to look at the following files:

• %windir%\setupact.log (coinstaller log file)
• %windir%\temp\wudf_update.log (update process log file)
• %windir%\setupapi.log (for Windows XP) and %windir%\inf\setupapi.dev.log (for Vista):PNP manager log file
  

Now it's time for KMDF. In order to install the KMDF you need to create a directory with the following files:

• The KMDF coinstaller (%WinDDK%\6000\redist\wdf\x86\WdfCoInstaller01005.dll)
• The echo driver (%WinDDK%\6000\src\kmdf\echo\sys\objfre_wxp_x86\i386\echo.sys)
• The inf file (%WinDDK%\6000\src\kmdf\echo\sys\objfre_wxp_x86\i386\echo.inf)
• devcon.exe (%WinDDK%\6000\tools\devcon\i386\devcon.exe)
• The pdb file of the driver (%WinDDK%\6000\src\kmdf\echo\sys\objfre_wxp_x86\i386\echo.pdb). This is not mandatory, but it might be helpful to debug the driver.
  

In order to install the driver you can open a command prompt and type
devcon install ECHO.inf root\ECHO

If everything goes well, you should see the following output:

>devcon install echo.inf root\Echo
Device node created. Install is complete when drivers are installed...
Updating drivers for root\Echo from echo.inf.
Drivers installed successfully.

If there's a problem, then you'll see something like:

>devcon install echo.inf root\Echo
Device node created. Install is complete when drivers are installed...
Updating drivers for root\Echo from echo.inf.
devcon failed.

The first file that you look at, in order to find more information about an installation is %windir%\setupact.log. This file contains the logging that is
made by the coinstaller. In the case of succesful installation, it should contain something like the following:

WdfCoInstaller: [11/27/2007 20:15.56.320] DIF_INSTALLDEVICE: Pre-Processing
WdfCoInstaller: [11/27/2007 20:15.56.335] ReadComponents:  WdfSection for Driver Service ECHO using KMDF lib version Major 0x1, minor 0x5
WdfCoInstaller: [11/27/2007 20:15.56.461] VerifyMSRoot: exit: error(0) The operation completed successfully.
WdfCoInstaller: [11/27/2007 20:16.27.844] DIF_INSTALLDEVICE: Post-Processing

Not much to see here. In this case, the following files were update to KMDF 1.5:

• %windir%\system32\drivers\wdf01000.sys (KMDF framework)
• a%windir%\system32\drivers\wdfldr.sys (KMDF loader)
• Also, the echo driver is installed at %windir%\system32\drivers\echo.sys. If you go to the device manager, you'll see the echo driver under the name "Sample WDF Echo Driver".
  

Now let's see at a problematic scenario. In this case I'll try using the debug version of the coinstaller in a release version of Windows XP. The installation
fails and in %windir%\setupact.log I can see:

WdfCoInstaller: [11/27/2007 20:32.19.349] DIF_INSTALLINTERFACES
WdfCoInstaller: [11/27/2007 20:32.19.724] DIF_INSTALLDEVICE: Pre-Processing
WdfCoInstaller: [11/27/2007 20:32.19.771] Path to INF files: c:\echo\
WdfCoInstaller: [11/27/2007 20:32.19.818] ReadWdfSection: Checking WdfSection [ECHO_Device.NT.Wdf]
WdfCoInstaller: [11/27/2007 20:32.19.865] ReadComponents:  WdfSection for Driver Service ECHO using KMDF lib version Major 0x1, minor 0x5
WdfCoInstaller: [11/27/2007 20:32.19.912] Extracting WDFCAB_RESOURCE from wdfcoinstaller01005.dll to C:\WINDOWS\Temp\WdfTemp\WDF.cab
WdfCoInstaller: [11/27/2007 20:32.20.005] file: C:\WINDOWS\Temp\WdfTemp\WDF.cab, time-stamp: [11/28/2007 04:32.19.943], size: 1451454 bytes
WdfCoInstaller: [11/27/2007 20:32.20.162] Check C:\WINDOWS\Temp\WdfTemp\WDF.cab for trusted signature
WdfCoInstaller: [11/27/2007 20:32.20.209] VerifyMSRoot: exit: error(0) The operation completed successfully.
WdfCoInstaller: [11/27/2007 20:32.20.255] CheckWinTrust(C:\WINDOWS\Temp\WdfTemp\WDF.cab) returns S_OK
WdfCoInstaller: [11/27/2007 20:32.20.287] VerifyFileTrust: exit: C:\WINDOWS\Temp\WdfTemp\WDF.cab, error(0) The operation completed successfully.
WdfCoInstaller: [11/27/2007 20:32.20.396] C:\WINDOWS\Temp\WdfTemp\WDF.cab has been extracted to C:\WINDOWS\Temp\WdfTemp
WdfCoInstaller: [11/27/2007 20:32.20.443]  component [0] : C:\WINDOWS\Temp\WdfTemp\wdf01000.inf
WdfCoInstaller: [11/27/2007 20:32.22.553] Update process returned error code :error(1603) Fatal error during installation.
. Possible causes are running free version of coinstaller on checked version of OS or vice versa. Look at the Kmdf documentation as to what steps need to be followed to install the correct versionof the coinstaller
WdfCoInstaller: [11/27/2007 20:32.22.600] Update process returned error code :error(1603) Fatal error during installation.
. Possible causes are running free version of coinstaller on checked version of OS or vice versa. Look at the Kmdf documentation as to what steps need to be followed to install the correct versionof the coinstaller
WdfCoInstaller: [11/27/2007 20:32.22.663] Final status: error(1603) Fatal error during installation.
WdfCoInstaller: [11/27/2007 20:32.22.960] DIF_DESTROYPRIVATEDATA

Indeed, this is more verbose logging. First of all we see that the coinstaller extracts the WDFCAB_RESOURCE resource to C:\WINDOWS\Temp\WdfTemp\WDF.cab. You can drag-and-drop the KMDF coinstaller to a Visual Studio window, in order to look at the resources and see what they contain. After doing some signature checking and making sure that it's a signed file by Microsoft, it calls again the update process, as in the case of the UMDF coinstaller. The process fails and there is a message saying that possibly this happened, because we are running a checked version of the coinstaller in the release version of the operating system. In order to find, if this is the case, we need to look at the log file of the update process. This is located at %windir%\Wdf01005Inst.log. This file has the following output:

0.094: ================================================================================
0.094: 2007/11/27 20:32:22.178 (local)
0.094: e:\7b719b539b272fadb485240b\update\update.exe (version 6.3.4.0)
0.110: Hotfix started with following command line: /quiet
0.125: CheckSystem: GetCheckedFree failed :STATUS_CHECKED_FREE_MISMATCH
0.125: DoInstallation: CheckSystem Failed: 0xf002
0.125: Wdf01005 Setup cannot update a checked (debug) system with a free (retail) version of Wdf01005, or vice versa.
0.125: Update.exe extended error code = 0xf002
0.125: Update.exe return code was masked to 0x643 for MSI custom action compliance.

So, indeed this is the case. Another file that we can look for more information is %windir%\setupapi.log (for Windows XP) and %windir%\inf\setupapi.dev.log (for Vista and later).

If you want to uninstall the driver, you can follow the same steps as with the UMDF driver (either go to control panel or use devcon).

So, when there's an installation problem with KMDF, we need to look at the following files:

• %windir%\setupact.log (coinstaller log file)
• %windir%\Wdf01005Inst.log (update process log file)
• %windir%\setupapi.log (for Windows XP) and %windir%\inf\setupapi.dev.log (for Vista): PNP manager log file

Driver-Driver and Driver-Application Communication

Everybody knows how an application communicates with a driver: call CreateFile to open a handle to the PDO, and use ReadFile(Ex), WriteFile(Ex) and DeviceIoControl to read data, write data or send a control code.

However, what about the opposite? How does a driver send data to an application (whenever it wants)? And how do drivers exchange data? These types of questions come very often in OSR's NTDEV list, so I found a list of resources that provide answers:

• Driver-Driver Communication
  
• Driver to Driver Communication: Explanation of the way that drivers exchange data between each other
  
• Buddy Drivers: An implementation of 2 drivers exchanging data
• Driver-Application Communication
  
• Peter Wieland has written this post on how the driver uses the Push or the Pull model to send data to an application
• Inverted Call Model: An implementation of Peter's model that shows how the driver sends data to the application
• Sharing memory between drivers and applications: Why, when and how?
• Driver-Application communication using IRQs
• OSR's thread on Driver-Application communication: Many driver developers write down their opinions on this issue
• Microsoft's documentation: How to use asynchronous events, in order to send data from a driver to an application
• UMDF drivers
• How an application communicates with a UMDF driver:Overview of the WDK sample that is located at %WinDDK%\src\umdf\usb\fx2_driver\final

How Driver Installation Works

The last few months I've been working on the WDF 1.7 (UMDF+KMDF) coinstallers (that's one of the reasons that I've been silent for quite some time).

Through this process I managed to learn a lot of things about how driver installation works and what is required by the driver developer. Unfortunately, this area is often a black box for driver developers, since their job ends, after the driver is up and running (and hopefully tested :) ). However, it's possible that you might want the driver installation to do something "more", e.g. change the icon of the device in the device manager or add a page in the device manager that shows the driver capabilities, etc. This is when a coinstaller or a class installer is needed.

Let's start from the beginning though. If you have a driver, then how do you install it? The easy way is OSR's Driver Loader. Just point to your sys file, click "Register Service" (i.e. create the appropriate settings in the registry at HKLM\System\CurrentControlSet\Services\<Driver Name>) and "Start Service" (i.e. call the Service Control Manager APIs that load the driver according to the registry settings). If you want to remove the driver, just click on "Stop Service" (stops the driver from running) and "Unregister Service" (deletes the registry settings). Easy, right? You don't need an inf file or anything more apart from your driver. However, this program supports non-pnp (legacy) drivers only.

For WDF drivers we provide a coinstaller (one for UMDF and one for KMDF). The main tasks of the coinstaller is to upgrade the framework (e.g. from 1.5 to 1.7), parse the inf file and configure the driver correctly. In the past, we've had some issues with the previous versions of the coinstallers, as shown by Bob Kjelgaard (part 1, part 2, part 3), however we're trying our best to tackle them. By the way, if you have any problems with the installation of a WDF driver, please look at the Doron's post for information on how to debug it and at Bob's post for information about how you can ask more help from Microsoft.

Microsoft has lots of useful information on Driver Installation at http://www.microsoft.com/whdc/driver/install/default.mspx, however I'd also like to point specifically to:

• Eugene Lin and Jason Cobb's channel9 video: Introductory overview on how the user-mode Plug-and-Play manager works and how the installation works from the point that the device is connected to the machine, until the corresponding driver is selected and the device is running
  
• Jim Cavalaris' presentation on coinstallers and class installers: Excellent presentation that shows how the class installers and the coinstaller cooperate during driver installation. Also look at the resources, which are provided at the last page.
  
• Debugging Device Installation on Windows Vista: Everything that you wanted to know (and even more than that!) about how to debug problems during device installation (and were afraid to ask)
• Device Installation Design Guide: Microsoft's official documentation on everything that has to do with device installation
  

What's New in Windows Vista?

One of the most popular questions that I've been hearing lately is "what's new in Windows Vista?". Therefore, I could not prevent myself from writing a post with links that provide information about this issue:

• Let's start with the changes in the kernel. Mark Russinovich covers this topic very well:
• Part 1: Processes, threads and I/O
• Part 2: Memory management, system startup, shutdown, and power management
• Part 3: Reliability, recovery, security
• Summarizing video: Vista Kernel Changes
• Additional video: User Account Control (UAC)
• In user-mode, Arstechnica provides a deep overview (this is the first post of a three-part review)
• Security-related topics:
• Michael Howard presents the big picture about Windows Vista's security here and focuses on ASLR (here and here)
• Windows Vista Security Enchancements discusses about the changes in more depth
• Channel9 videos about security in Vista (part 1, part 2) and User Account Control (UAC)
• Channel9 has videos that cover many windows components that are new in windows vista
  
• Wikipedia provides a quick overview of all the new additions in Windows Vista

UPDATE:

If you are interested in a detailed comparison between the different Windows Vista versions (Home, Business, Enterprise, etc), then you can look here (4th post).

Poll: Which IDEs are being used by driver developers?

I would like to use this post as a way to find out, what different IDEs are being used for driver development. It would be nice, if you could write your favorite IDE(s) and a small explanation (and possibly a link) about their strengths and weaknesses. It doesn't matter, which company created the IDE, since this could be a nice opportunity for many people to learn what options are available.

Personally, I've tried Visual Studio (http://msdn2.microsoft.com/en-us/vstudio/default.aspx), however I consider it too heavy. One advantage, though, is that it can be combined with DDKBUILD (http://www.osronline.com/article.cfm?article=43), in order to provide seamless interaction with the WDK compiler. You can look at http://hollistech.com/Resources/ddkbuild/ddkbuild.htm for additional information. Hopefully, in the future, Visual Studio might also support WDK help integration. Currently, I'm using Source Insight (http://www.sourceinsight.com/), which is much lighter. I like the fact that it has symbol windows for each file, syntax formatting and many other useful options, such as smart rename of variables (e.g. if I choose to rename a variable called buffer to Buffer that is used inside a function Foo, it won't rename a variable called Stringbuffer and it won't rename any other variable called buffer that is outside of Foo).

So, which IDE(s) are you using? 

Suggestions for new topics

The goal of this post is to be used as a way for you to provide suggestions for introductory driver-related topics that you would like more information about. I would like to hear any suggestions that an new driver developer might have and I'll try to gather information and provide a useful analysis. So, feel free to write a comment and suggest any interesting relative topic that you might think about. Please keep in mind, though, that my main area of focus is UMDF :)