Queue Callbacks are NOT Dispatch Routines
One of our SDETs recently made an error in a Win7 WDK sample that prompted a discussion. After I explained the error, Wei gave me some feedback that my explanation made more sense than the WDK documentation did. So I'll try to elaborate a bit on what I see as the underlying misconception, and perhaps it will help others avoid this conceptual error. The title perhaps gives my key point away...
The Wild and Wooly World of the Original Driver Model
I'm with some of the other purists here- some will call this the WDM driver model, but this model is older than that- within Windows, it was in the original release of Windows NT [and it's based on VMS, IIRC- that's even older]. The driver object has a table of dispatch routines, which are what the I/O manager calls to tell the driver it has to process some form of I/O request [in the form of an I/O request packet or IRP]. There is one entry for each "major code" [which is an enumeration of types of I/O request which is somewhat arbitrary, but for compatibility reasons is unlikely to change much- the advent of WDM added several new major codes, for instance, but the previous ones remain]. This isn't meant to be a tutorial, so that's enough explanation for now.
The main things to consider about dispatch routines for this discussion:
- There are instances [for instance, top level driver] where you can safely assume you are called at passive level and in context of the calling process and thread.
- You are almost always called at passive level anyway.
- There is some synchronization of I/O done by the IO subsystem (Serialization on a handle and of non-overlapped I/O, for example), but you cannot generally assume any synchronization between routines or even between a routine and itself.
That last bullet is where a lot of problems come in- unless you design the right kind of test cases, you can miss synchronization issues in a driver. But the first two bullets also matter, particularly if you are trying to be a good citizen and make code and data paged when they are only relevant at passive level.
WDF Offers Abstractions With More Precise Semantics (If You Accept Them)
WDF handles dispatch routines for you through its request queuing models [and I use the plural deliberately, there are multiple models to let you tailor your driver better]. One thing in particular these provide is the capacity to synchronize request processing in multiple ways:
- The sequential queue dispatching model guarantees that only one request is processed at a time- a new request is not dispatched until the previous one is completed. This is the simplest model, but it isn't desirable for a lot of applications. But if it works for you, it's the simplest mechanism of all to understand.
- New in KMDF 1.9 is a counted parallel queue, which can have a bounded number of concurrent requests- once this limit is reached, new requests are not dispatched until an existing one completes. [The sequential queue can be called a counted queue of one request]. If some or all of these requests need the same resource, you will need additional synchronization, of course.
- A device can use device level synchronization on its queues- this means that only one callback at a time will execute, and a new request will not be dispatched until the current callback completes. This is not the same as sequential dispatching in that it affects all queues, and the point at which the next request is dispatched is different. If your callback doesn't complete a request [which is often the case, it is often enough for it to simply set up an operation], then the dispatch will occur sooner than it would in a sequential queue and you have overlapping of requests [potentially quite a bit of it, depending upon how you manage this sort of asynchronous I/O]. This model can also be extended to work items, Dpcs, and timers parented to this device. If your callback DOES complete a request, the dispatch occurs a bit later [because the callback exit and not the completion triggers the dispatch of the next request]. This model is good if you have a device level resource you want to serialize access to, but you don't have to wait for completion to use it for a further operation (something along the lines of a command FIFO, for instance)
- For further granularity, you can synchronize at queue level instead- in this case, callbacks within a queue are serialized with respect to each other, but not with respect to other queues- again this can be extended to work items, etc, but in this case, make sure they are parented to the queue and not the device. This model is good if you have independently programmable subdevices, each of which still requires some additional synchronization.
The error I alluded to was related to paging and the last two types of synchronization. You can specify that synchronization has to happen at passive level. If you do, then a passive level locking mechanism is used, and you can make your callbacks paged code. If someone sends you a request at dispatch level, a work item will be used by the framework to dispatch that request to you [the caller will get STATUS_PENDING back].
But if you DON'T state you want passive level execution, then the potential is always there that you have to synchronize with dispatch level code. That means a spin lock is necessary. If you are using either of the last two synchronization mechanisms, and have not explicitly used passive level synchronization [this has to be set at the device level or driver level] you MUST NOT make the code paged, because it will ALWAYS be called with the spin lock held [and hence at dispatch level].
Yes, you may know the I/O came to the dispatch routine at passive level, but the callback is not a dispatch routine- if you've asked for a synchronization model that isn't workable at passive level, your callback won't be there, because the framework had to change IRQL to meet your synchronization requirements. Be careful what you ask for- you might well have received it!
One afterthought about queues- you can use multiple queues and even take requests from one and forward them to another. So if, for instance, you have some particular IOCTL that would be perfect for sequential dispatch, you can use an IOCTL handler in a top-level queue, create a sequential queue for just that particular IOCTL and forward the requests from one to the other. You still get the benefits of cancellation handling and a more focused model when you do this.
I'm still not sure this is clearer, but at least it isn't another Life At Microsoft tag, this time around!
Now Playing: Seatbelts(Yoko Kanno) Cowboy Bebop- "Bad Dog No Biscuit"
