Pointless Blathering

Code

Perhaps I'm just out of ideas, but I've decided to write about my coding style.

Coding style is usually a very personal thing, like any writing style.  And as it usually causes some to foam at the mouth when brought up, I also wanted to put out this little disclaimer before I start. 

You may agree with how I've chosen to do things, you may disagree, you may not care in the least.  Please just take this as writing by someone who has been writing code a while and who likes to think about the process of writing code and building software.

At the least this can help my blog live up to its name :)

Pedantic Coder : Where do braces go?

I've become rather pedantic about my coding style over the years.  I've worked in a number of people's code, and have always felt most comfortable in the core NT code because of the consistency of formatting, naming, etc...  This is a coding style that we often call "Cutler Normal Form" in deference to Dave Cutler.

Despite this I recently made a change in the way I place my braces around blocks of code.  CNF says that braces go at the end of the line that will execute the code block, as follows:

if (foo) {
    do something;
} else {
    do something else;
}

I used to really like this style.  It made it clear to me when the statement being evaluated was continued into the next block.  Of course I always use braces, even when I only want one statement in the if or else clause, so this was a quick way to look and be sure i'd set things up correctly.

I've now transitioned over to:

if (foo) 
{
    do something;
} 
else
{
    do something else;
}

This was the preferred style of the folks in the UMDF group when I joined the project.  It took me a while to warm up to this.  However i eventually found two things that i like about it.

First it leaves more open white space.  In my "old age" (i.e. mid thirties) i find that i like more whitespace in my code.  It improves the readability for me, and makes me work harder to keep my functions fitting on a single page - which is a good threshold for whether they're understandable or not.

The second is that it makes it much, much easier to put part of the block under an IFDEF.  This to me was the winner.  Now i can do:

#if bar
if (foo)
{
    do something specific to bar;
}
else
#endif
{
    do something else;
}

Rather than:

#if bar
if (foo) {
    do something specific to bar;
} else // note that there would otherwise be a { here
#endif
{
    do something else;
}

Or even worse:

#if bar
if (foo) {
    do something specific to bar;
} else {
#else
{
#endif
    do something else;
}

Perhaps it's a silly thing to change something as fundamental(ist) as brace formatting style to get around a little inconsistency in how you preprocess out part of a conditional.  But i like consistency ... the hairs on the back of my neck go up when i see code that's not in my normal format.  So in the end this made me more comfortable.

This has come up quite a bit for me when debugging something or refactoring something.  When refactoring i'll frequently #if out a chunk of impacted non-critical functionality (usually replaced with a failure case) until i'm ready to deal with that chunk of code.  For debugging it can be useful to do the same thing if you're trying to track down the cause of a crash and are at your witts end.

And I've come to find it prettier.

-p

So many ways to send SCSI requests to a driver

(Please excuse the recycled bits.  The 7-month-old is still absorbing most of my blogging time (along with sleeping time, dating time, playing time, cleaning time, working time, etc...))

A question came up on the ntdev mailing list about why there are "so many" ways to send an I/O request to a SCSI driver.

In matter of fact there are two - IRP_MJ_SCSI and IOCTL_SCSI_PASS_THROUGH (and it's direct mapped variant).

IRP_MJ_SCSI == IRP_MJ_INTERNAL_DEVICE_CONTROL. This is easily determined by looking at the headers. No SCSI shouldn't have its own IRP MJ code, and no it shouldn't reuse a number from an existing one. But it's been that way since I joined MS (and I tried to change it once ... not possible without breaking everyone doing storage) and so we all just have to live with it. Someone decided we needed a MJ code for SCSI requests. I suspect IRP_MJ_INTERNAL_DEVICE_CONTROL was reused to avoid the cost of an additional PVOID per driver object (which at the time would have been an issue).

IOCTL_SCSI_EXECUTE_* is set in the stack location for an IRP_MJ_SCSI to (a) allow someone handling IRP_MJ_INTERNAL_DEVICE_CONTROL to differentiate between the two and (b) to provide some indicator of which direction the SRB is transferring data. Its use isn't particularly consistent ... I think of it as more of a debugging aid but I see that some of our drivers (like the RAMDISK driver) use it to decide if the SRB in question is a read/write type command or one of those strange SCSI commands like REQUEST_SENSE which require more complex emulation. I think using the CDB code would have been a much better idea.

IRP_MJ_SCSI comes with an SRB.

IOCTL_SCSI_PASS_THROUGH and IOCTL_SCSI_PASS_THROUGH_DIRECT are to send SCSI requests from user-mode. They come with SCSI_PASS_THROUGH/SCSI_PASS_THROUGH_DIRECT structures so as not to expose SRB to user-mode and so as to avoid validating SRBs which might come from user-mode (besides, how is a user mode component going to provide a value like OriginalIrp).

IOCTL_SCSI_PASS_THROUGH is a buffered operation, while IOCTL_SCSI_PASS_THROUGH_DIRECT direct maps the data transfer. They're separate control codes because they are - I suppose a flag in the PASS_THROUGH structure could have contained the transfer mode ... would that make you happier?

So there are two ways to send a SCSI request. One for kernel-mode components (IRP_MJ_SCSI) and one for user-mode components (IOCTL_SCSI_PASS_THROUGH[_DIRECT]). If you want to convert a pass-through command to a IRP_MJ_SCSI command it's pretty straight-forward to make an SRB and send it down.

-p

WDK available in other languages

This is pretty cool, though my high-school french isn't sufficient to appreciate it completely.  The WDK content on MSDN has been translated into multiple languages. 

It's a machine translation and to make up for the inconsistencies they do side-by-side english & other language.  It's not as good as native documentation, but hopefully it will help bridge the gap for those who don't speak english as their first language.

The language options are:

· Brazilian Portuguese (pt-br)

· French (fr-fr)

· Japanese (ja-jp)

· S. Chinese (zh-cn)

· Spanish (es-es)

· T. Chinese (zh-tw)

Enjoy

-p

Where is Peter

It's been ages since I posted to my blog and I'm sorry about that.  I recently became a dad and it turns up that absorbs a lot of the free time I used to spend on my blog.

I'm hoping to get back into things ... so don't remove me from your OPML files yet.

-p

Catherine van Ingen is up on Channel 9

I used to work with Catherine back when i was on the storage team.  I learned a lot just watching Catherine do what she does.  I went to the taping of the interview and it was quite amusing.  I think this is definately worth the hour it takes to watch.

http://channel9.msdn.com/ShowPost.aspx?PostID=316739

WinHEC Day 2 - So Very Tired

I was up way to late last night and then decided to get up this morning for the keynotes.  And to make it somewhere by 8 i have to get up early since i'm not really functional in the mornings.  So it's been a long day, but a good one. 

Mark Russinovich gave a very good keynote speech about kernel changes for Server 2008.  The Windows Sideshow folks are doing some really amazing stuff with their platform and had several very exciting things to demonstrate. 

The evening was a reception at Univeral Studios (in part of the backlot area).  Also fun, but i'm not sure i had enough energy to really go.  I did finally dig out my camera and take a couple of pictures.  I  like my Fuji F30 for its low-light point and shoot capabilities.

I managed to be one of the last folks out of the park - the line to get on the busses was quite long:

On the ride back i had a chat with an MS employee from Amsterdam who was in for the convention.  That really made my night. 

Sleep now.

WinHEC Day 1

One day and i'm done.  More or less.  All of the KMDF and UMDF talks were front-loaded into the conference so we were finished "talking" by 4:15. 

Eliyas and I gave our talk this morning on what's new in KMDF and UMDF.  It seemed to go pretty well, though it's definately one of the least technical talks i've given on the topic.  It's also my least rehearsed - i didn't start running through the slides until I got into my hotel room last night.  Hopefully it didn't show too badly :).  I did a quick poll of the room and most of the attendees were aware of UMDF and KMDF & a good number are using KMDF.  UMDF adoption is sparser (which isn't surprising) but actually the number of hands was better than I was expecting.

I got to poke my head into Landy's Q&A session on the memory manager.  Landy always manages to pack a room and it's easy to understand why. 

The KMDF chalk talk was pretty lively.  Folks have been playing with KMDF for quite a while so there were a number of good questions.  The UMDF chalk-talk had a large number of "newbies" which we honestly weren't expecting.  I think Praveen did a pretty good job of adapting to that - we walked thorugh the skeleton sample briefly then showed debugging some things with the debugger extensions to walk through your driver once it's up.  Shefalli had a little time at the end to run through her slides on testing your UMDF drivers and the tools that are available for that.

Now that Nar's done talking about the I/O manager changes it's time to go back to the hotel.  I'm ready for a nap.

-p

How do i figure out which host contains my driver?

Let's say you're luck enough to have more than one user-mode driver running on your system.  How would you figure out which is running your driver?

Let's take my laptop.  By luck i happen to have installed both the UMDF skeleton and echo drivers.  I'm interested in debugging the echo driver (the skeleton does almost nothing so it's not really worth debugging :) ).  Device manager shows me this:

And there are two host processes:

    tasklist | findstr -i wudfhost.exe
    WUDFHost.exe 4668 Services 0 3,572 K
    WUDFHost.exe 4296 Services 0 3,540 K

How do i know which is which?

Praveen found a really good trick for this.  Tasklist has a /M flag which lets you look for processes with a particular module loaded.  So i if i want to know which host has wudfechodriver.dll loaded i can simply run:

    tasklist /m wudfechodriver.dll
    Image Name    PID   Modules
    WUDFHost.exe 4668   WUDFEchoDriver.dll

Note that you need to do this from an elevated command window on Vista - WUDFHost runs as LocalService in session 0 and isn't normally visible to an unelevated process.

Also if you're a PowerShell sort of person (as i'm rapidly becoming) you could run:

foreach($p in get-process wudfhost) {
    $p | format-table
    $p.Modules | 
        ? {$_.FileName -like "$env:windir\system32\drivers\umdf\*"} | 
        format-table -autosize
}

to get:

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s)   Id ProcessName
------- ------ ----- ----- ----- ------   -- -----------
    124      3  2092  3508    31   0.05 4296 WUDFHost 

Size(K) ModuleName       FileName
------- ----------       --------
24      UMDFSkeleton.dll C:\Windows\System32\drivers\UMDF\UMDFSkeleton.dll 

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s)   Id ProcessName
------- ------ ----- ----- ----- ------   -- -----------
    124      3  2100  3568    31   0.03 4668 WUDFHost 

Size(K) ModuleName         FileName
------- ----------         --------
28      WUDFEchoDriver.dll C:\Windows\System32\drivers\UMDF\WUDFEchoDriver.dll

I like this technique a little more because it shows all the hosts and lists every UMDF driver that's loaded in that host, not just the one you were asking about.  But it requires you to use PowerShell.

How to enable USB selective suspend and system wake in the UMDF driver for a USB device

My UMDF driver has some data - how do I get it to my app?

This came up as a question on NTDEV today. After I thought about it for a bit I realized it was actually a pretty good question. You've started writing your driver, or at least thinking about your driver. You know you'll get some data back from the device and you need to get it to the app. How do you do this?

For simplicity's sake, let's say there are two models for a driver to operate in. The "push" model and the "pull" model (of course there are more, but this captures the big ones.) When I say "push" and "pull" here I'm talking about both how the driver decides it should talk to the device, and how it returns the data to the driver.

But before I dive in, realize that I'm talking here about how the driver operates in "steady state" - the time between PNP and Power Management operations, between recovering from device errors/resets, the time when it's just servicing I/O requests.

Pull Model

We'll start with the Pull model because it's easier to understand. In steady state the typical pull-model driver does nothing on its own. When the app wants to talk to the device it sends a request to the driver, which triggers a driver callback. In the simplest case the driver formats a command to send to the device and uses the buffers from the application to hold any data the command returns. When the command returns the app's buffers contain the necessary data. The driver sets the information property on the request to indicate how many bytes of data the app is getting back then completes the request. When the app sees the request completion the data your driver provided will be in its buffers.

It's rarely this simple. Often you'll need to do apply some transformation to the data before you can return it. In that case your driver would allocate a buffer to use in the command you send to the device. When the command completes you use its result to write the correct data into the buffers on the app's requests. Then set information, complete the app's request, and free the intermediate buffer. Better yet, you can make the WDF memory object for the intermediate buffer a child of the request so that it's freed automatically when you complete the request.

Push Model

Say your device is going to bubble data up to your driver when something happens. It might be GPS coordinates, printer status, a finger print scan, etc… How do you get this data back to an application? Clearly you don't want the application to poll for data. All you want to do is push the data you're getting from the device up to the application. How do you do this?

The first thing to understand is that you can only give the application data that it asks for*. Unless the app makes a request and gives you a buffer you don't have a way to put data in its address space. So clearly the Push model is just a variant of the pull model.

In the pull model you let the app send you one or more requests asking for data. You queue those requests for later processing. In the background you have some code which is reading data from the device. When those read commands complete you pickup an app request off the queue, copy your command data into the app's request buffers then complete the app request.

You may choose to copy the data into multiple requests, you may choose not to complete the request if you don't have enough data to fill its buffer with just one command. And if you don't have any pending requests you need to decide whether to just throw the data away or whether to hold it in an internal buffer until the app comes down to ask for more.

The key difference between the push and pull models is whether the driver is sending commands to the device on its own, or whether every device command comes because the app sent your driver a request.

As I said before - it's never this simple. Looking at the UMDF Fx2 sample you see that it uses the pull model for read/write operations on the device and for most of the i/o controls that it supports. However it uses the push model to handle requests to read the state of the DIP switches.

-p

* You can use PNP device events if you need to push the fact that something has happened to any app which might be listening. But it's an unreliable mechanism and isn't suitable as a way to stream data.

Do UMDF drivers require signing?

To quote from Rev. Lovejoy:

Shoo, short answer: "Yes" with an "If", long answer "No" - with a "But.

There are two types of driver signing, which complicates the answer. As usual any answers I provide are probably vague and can be overridden by standard documentation. In this case the documentation seems to be available on MSDN.

Driver Package Signing:

First there's the standard signing of the 'driver package' (all the files needed to install the driver) which has been around since Win2k. Signing your driver package allows you to avoid the "unsigned driver" popup, and allows end-users to determine whether your package has been tampered with. Personally I think it's goodness, but having never had to go through the signing process I realize it's easy for me to feel that way.

To sign your driver package your INF should have a CatalogFile directive in the [Version] section. The catalog file contains hashes for all the files in your driver package and is then signed to ensure it can't be tampered with.

Code Integrity Signing:

In Vista there's a new code-signing requirement. Kernel-mode binaries that load in 64-bit Windows Vista (or drivers on 32-bit & 64-bit windows that stream protected content) need to be code signed so that the OS can determine whether they have been tampered with.

From a quick scan of the documentation it sounds like this mostly affects the type of certificate you can use when you do your signing. The Code Integrity system (which checks your driver signatures when the driver is loaded) requires something called a "Software Publisher Certificate (SPC)", while regular driver signing can use either an SPC or a "commercial release certificate". However I believe that if you are installing a kernel-mode boot driver you have to make sure your binaries are self signed (signature embedded in the binary) while if you are installing non-boot kernel-mode drivers that you can use a catalog signature (signature embedded in the .cat file installed with the package) instead.

Regardless your driver package will still require a signature in order to get through installation.

So what does all this mean for UMDF?

UMDF drivers do not require Code Integrity Signing since they aren't kernel-mode code. However your driver package still needs to be signed so that it can't be tampered with & can be traced back to the author (or you continue to have unsigned driver popups). And if your package includes any non-Microsoft kernel-mode drivers in addition to the user-mode driver then the Code Integrity Signing rules apply to that too.

To finish with the quote from Rev. Lovejoy:

Uh, if you need additional solace, by the way, I've got a copy of something or other by Art Linkletter in my office.

-p

Fixing the UMDF USB samples to install both on XP & Vista

UMDF's content in the WDK contains a handful of examples for controlling a USB device. Unfortunately they only show how to do it on Vista. The problem was that WinUSB's co-installer wasn't added to the WDK until very late - too late for us to be able to change our samples to show how to use it.

WINUSB introduces a number of new things into the INF because of the complexity of installing it on XP and making it work with the in-box Vista support. In Vista WinUSB is present on the disk but the driver & service aren't "installed" until some INF copies them into place. The WinUSB folks provided an in-box INF (winusb.inf) that you can reference in your INF with the Includes/Needs directives so that you don't have to setup the service entries yourself.

On XP WinUSB doesn't exist until someone installs it. The preferred installation route is to use the WinUsbCoinstaller. However because WINUSB.INF isn't already on the machine in XP, you can't depend on it to setup the WinUSB service entries. And we couldn't rely on the WinUsbCoinstaller to do it because the device installer requires that your INF have an AddService directive for the driver which will act as the FDO.

The result is that installing WinUSB requires you to add a few things to your INF. Some of these are already demonstrated in the existing sample INFs. Others are new, and there's one "bug fix". I've added the full INF as an attachment to this blog entry.

So starting with the Fx2 driver's INF, here are the necessary changes to build a driver package that will install both on XP and Vista.

1. Include sections from WINUSB.INF in your INF

   Specifically in your [DDInstall.NT] section you add:

   Include=WINUSB.INF ; Import sections from WINUSB.INF
   Needs=WINUSB.NT ; Run the CopyFiles & AddReg directives
   

   The first tells Device Installation to locate WINUSB.INF and read in all of its sections. The second tells DI that it should import any directives in the "WINUSB.NT" section into this section as well. WINUSB.NT contains CopyFiles & AddReg directives that tell Vista to copy the WinUSB.Sys binary as part of your driver package & sets up some registry entries.

   Now you may recall I said that there was a problem on XP because this INF doesn't exist. However when you ask DI to include an INF file that doesn't exist (or say you Need a section that doesn't exist) the installer just ignores you. Normally that would drive me nuts (I hate hidden errors) but it works in my favor in this case.

   On XP these two directives are benign and we depend on the WinUsbCoinstaller to copy the files into place and setup the registry entries. On Vista WinUsbCoinstaller doesn't do anything and we depend on these directives to copy the binaries.

   This leads us to step 2 ...

2. Copy & Invoke the WinUsbCoinstaller in your INF

   To invoke the WinUSB coinstaller requires three things in your INF:

   1. Add the file to [SourceDisksFiles] so that DI knows where to get the file from

      [SourceDisksFiles]
      ...
      WinUsbCoinstaller.dll=1
      

   2. Add the file to [CoInstallers_CopyFiles] so that DI knows that it needs to copy it

      CoInstallers_CopyFiles]
      ...
      WinUsbCoinstaller.dll
      

   3. Add the name of the coinstaller to the CoInstallers32 registry entry so that DI knows to invoke it.

      [CoInstallers_AddReg]
      HKR,,CoInstallers32,0x00010000,"WudfUpdate_$UMDFCOINSTALLERVERSION$.dll",\
                                     "WinUsbCoinstaller.dll", \
                                     "WdfCoInstaller01005.dll,WdfCoInstaller"
      

   Note that the names of the last two sections need to match what you provide to CopyFiles and AddReg in your INF's [DDInstall.Coinstallers] section.

3. Fix the bug in the [WinUsb_Install] section

   WinUsb requires version 1.5 of KMDF, not version 1.0. We didn't catch this before the samples shipped. Change the section to look like:

   [WinUsb_Install]
   KmdfLibraryVersion=1.5
   

4. Include WinUsbCoinstaller.dll as part of your signed driver package

   Along with WudfUpdate_01005.dll & WdfCoinstaller01005.dll you also now need to copy WinUsbCoinstaller.dll & add it to the catalog for your INF. You can find this file in c:\winddk\6000\redist\winusb\<arch>. 

   Like the KMDF and UMDF coinstallers, it comes in a _chk version also. If you're installing your driver (whether you built it checked or free) on a checked system you should use the _chk version of the DLL.

Four little changes and you're done. The same changes apply to any of the UMDF USB samples.

-p

Don't trip over the skin change

I finally got back to my blog after coming back from vacation (two week drive down to disneyland with the wife … lots of fun) and noticed that the CSS overrides I'd done looked even worse than before.

However they also loaded a lot of new skins into MSDN blogs, so I picked a different one.

Expect the look to be in flux while I find one that I like.

-p

Glossary Definition of "WDM"

I got looped into a discussion about the WDK glossary entry for "Windows Driver Model". It got me thinking about how difficult it is to explain things in the driver space to non-driver folks.

When I was working in storage I used to have this problem all the time. Someone, usually a family member, would ask me what I did and I'd have to talk about what a "storage device" was, about what miniports were, etc… For a while I gave up and described my job as "something technical and utterly uninteresting". Finally I settled on "I make the light on your hard-disk blink".

Then I moved to the WDF team and everything became complicated again. Explaining what WDF is to a non-driver person is a bit like explaining what colors are to someone who is color blind.

So here's my attempt to explain WDM to someone sort of technical who has no idea what a device driver does:

Windows Driver Model

An architecture for Windows extensions (device drivers) which allow the operating system core to communicate with a wide variety of devices.  WDM defines a Device Driver Interface (DDI) for drivers to interact with each other and the OS, and a set of required behaviors that device drivers must implement in order to work well with the rest of the system.  WDM and WDM drivers work together to provide device I/O, Plug and Play, Power Management and WMI support.

WDM is a packet-based I/O architecture.  Applications (as well as the OS and other drivers) can make requests of particular devices.  WDM represents these requests as I/O Request Packets (IRPs) and sends them to a particular set of device drivers for processing.  The drivers typically work in a layered fashion – each performing some portion of the request then handing the request off to the next driver – until the bottom most driver programs the device to perform the requested operation(s).

Not all devices are supported by WDM drivers.  Some device classes (such as storage devices or network adapters) provide their own class-specific DDIs and behaviors which represent a subset of WDM.  Other drivers may be implemented using the Windows Driver Frameworks, which provides a simpler general-purpose infrastructure for writing drivers.