A Hole In My Head : KMDF

WDFREQUESTs are for sharing in KMDF v1.9

doronh — Fri, 06 Mar 2009 00:48:14 GMT

In my last post I described why a WDFREQUEST is unique to a particular WDFDEVICE. There is one particular programming pattern where this is not the behavior you want. This pattern is when you have each PDO accepting IO requests which it then forwards on to the parent WDFDEVICE for processing. One great in box example of this is usbhub.sys. Each usbhub PDO receives URBs which are then forwarded to the parent FDO and the FDO is where all IO processing occurs.

If you want to apply this pattern to a KMDF driver written to a v1.7 or earlier and take advantage of WDFQUEUEs you had to send the requests from the PDO to the FDO with WdfRequestSend so that they were represented to the FDO. The easiest way to do this is to create a WDFIOTARGET for the FDO itself and then have each PDO send IO to that WDFIOTARGET as shown in the following 3 code snippets.

EvtDriverDeviceAdd for the FDO

NTSTATUS EvtDriverDeviceAdd(WDFDRIVER Driver, PWDFDEVICE_INIT DeviceInit) 
{
   WDFDEVICE device;

   WDF_OBJECT_ATTRIBUTES woa;
   WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&woa, FDO_EXTENSION);

   // ...initialize the DeviceInit...
   status = WdfDeviceCreate(&DeviceInit, &woa, &device);

   if (!NT_SUCCESS(status)) {
      return status;
   }

   PFDO_EXTENSION pFdoExt = GetFdoExt(device);
   status = WdfIoTargetCreate(device, WDF_NO_OBJECT_ATTRIBUTES, &pFdoExt->SelfTarget);
   if (!NT_SUCCESS(status)) {
      return status;
   }

   // open the WDFIOTARGET to point our own PDEVICE_OBJECT
   WDF_IO_TARGET_OPEN_PARAMS openParams;
   WDF_IO_TARGET_OPEN_PARAMS_INIT_EXISTING_DEVICE(openParams, WdfDeviceWdmGetDeviceObject(device));

   status = WdfIoTargetOpen(pFdoExt->SelfTarget, &openParams);
   if (!NT_SUCCESS(status)) {
      return status;
   }
   ...
   return status;
}

EvtChildListCreateDevice for the PDO

NTSTATUS EvtChildListCreateDevice(
    WDFCHILDLIST ChildList, 
    PWDF_CHILD_IDENTIFICATION_DESCRIPTION_HEADER IdentificationDescription,
    PWDFDEVICE_INIT ChildInit
    )
{
   WDFDEVICE pdo;
   NTSTATUS status;

   WDF_OBJECT_ATTRIBUTES woa;
   WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&woa, PDO_EXTENSION);

   status = WdfDeviceCreate(&ChildInit, &woa, &device);
   if (!NT_SUCCESS(status)) {
      return status;
   }

   PPDO_EXTENSION pPdoExt = GetPdoExt(device);
   PFDO_EXTENSION pFdoExt = GetFdoExt(WdfPdoGetParent(device));

   pPdoExt->ParentTarget = pFdoExt->SelfTarget;
   ...
   return status;
}

EvtIoDefault for the PDO

typedef
VOID EvtIoDefault(WDFQUEUE Queue, WDFREQUEST Request)

{
    // ...  extract Request type ...
    if (RequestShouldBeSentToParent()) {
       WdfRequestFormatRequestUsingCurrentType(Request);
       WdfRequestSend(Request, GetPdoExt(WdfIoQueueGetDevice(Queue))->ParentTarget);
    }
}

As you can see, this is a bit cumbersome! While it works, it is not ideal. The KMDF team addressed this issue in v1.9 by adding 2 new DDIs that must be used together

WdfPdoInitAllowForwardingRequestToParent which tells KMDF that you will be forwarding IO from a PDO to the parent FDO. Internally this sets up some bookkeeping and, more importantly, sets the PDO’s PDEVICE_OBJECT’s StackSize to the FDO’s PDEVICE_OBJECT StackSize+1 so that there will be enough stack locations in the underlying PIRP for both the parent and child stacks
WdfRequestForwardToParentDeviceIoQueue which removes the need for the custom WDFIOTARGET and WdfRequestSend and directly presents the PDO’s WDFREQUEST to the FDO’s WDFQUEUE

Let’s now rewrite the 3 code snippets to make use of these new DDIs, new code in red

NEW EvtDriverDeviceAdd for the FDO

NTSTATUS EvtDriverDeviceAdd(WDFDRIVER Driver, PWDFDEVICE_INIT DeviceInit) 
{
   WDFDEVICE device;

   WDF_OBJECT_ATTRIBUTES woa;
   WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&woa, FDO_EXTENSION);

   // ...initialize the DeviceInit...
   status = WdfDeviceCreate(&DeviceInit, &woa, &device);

   if (!NT_SUCCESS(status)) {
      return status;
   }

   PFDO_EXTENSION pFdoExt = GetFdoExt(device);

   status = WdfIoTargetCreate(device, WDF_NO_OBJECT_ATTRIBUTES, &pFdoExt->SelfTarget);
   if (!NT_SUCCESS(status)) {
      return status;
   }

   // open the WDFIOTARGET to point our own PDEVICE_OBJECT
   WDF_IO_TARGET_OPEN_PARAMS openParams;
   WDF_IO_TARGET_OPEN_PARAMS_INIT_EXISTING_DEVICE(openParams, WdfDeviceWdmGetDeviceObject(device));

   status = WdfIoTargetOpen(pFdoExt->SelfTarget, &openParams);
   if (!NT_SUCCESS(status)) {
      return status;
   }



   // initialize a WDF_IO_QUEUE_CONFIG to your needs
   status = WdfIoQueueCreate(device, ..., &pFdoExt->>ChildProcessingQueue);
   if (!NT_SUCCESS(status)) {
      return status;
   }

   ...
   return status;
}

NEW EvtChildListCreateDevice for the PDO

NTSTATUS EvtChildListCreateDevice(
    WDFCHILDLIST ChildList, 
    PWDF_CHILD_IDENTIFICATION_DESCRIPTION_HEADER IdentificationDescription,
    PWDFDEVICE_INIT ChildInit
    )
{
   WDFDEVICE pdo;
   NTSTATUS status;

   WDF_OBJECT_ATTRIBUTES woa;
   WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&woa, PDO_EXTENSION);


   WdfPdoInitAllowForwardingRequestToParent(ChildInit);
   

   status = WdfDeviceCreate(&ChildInit, &woa, &device);
   if (!NT_SUCCESS(status)) {
      return status;
   }

   PPDO_EXTENSION pPdoExt = GetPdoExt(device);
   PFDO_EXTENSION pFdoExt = GetFdoExt(WdfPdoGetParent(device));

   pPdoExt->ParentTarget = pFdoExt->SelfTarget;
   ...
   return status;
}

NEW EvtIoDefault for the PDO

VOID EvtIoDefault(WDFQUEUE Queue, WDFREQUEST Request)

{
    // ...  extract Request type ...
    if (RequestShouldBeSentToParent()) {

       WdfRequestFormatRequestUsingCurrentType(Request);
       WdfRequestSend(Request, GetPdoExt(WdfIoQueueGetDevice(Queue))->ParentTarget);


       WDF_REQUEST_FORWARD_OPTIONS options;
       WDF_REQUEST_FORWARD_OPTIONS_INIT(&options);
       options.Flags = WDF_REQUEST_FORWARD_OPTIONS_FLAGS;


       status = WdfRequestForwardToParentDeviceIoQueue(
           Request, 
           GetFdoExt(WdfPdoGetParent(WdfIoQueueGetDevice(Queue)))->ChildProcessingQueue,
           &options);

    }
}

WDFREQUESTs are not for sharing

doronh — Thu, 05 Mar 2009 03:58:00 GMT

FYI: this is a bit of a long post, but I wanted to be thorough and illustrative and give some insight into how the framework works and potential design that could have been made, but were not for the sake of simplicity and performance

A common misconception a WDFREQUEST handle is the assumption that the WDFREQUEST handle value “follows” (e.g. stays the same) the PIRP around everywhere that the PIRP goes to. Basically, the idea is that everywhere that the PIRP is sent or presented, the same WDFREQUST handle will be used. The reality is that the same WDFREQUEST handle value will only be used while the PIRP is owned by the WDFDEVICE for which it was created. This means that if you send the WDFREQUEST to another driver with a call to WdfRequestSend (e.g. transfer ownership to the driver you are sending it to), the driver which receives the PIRP will have a different handle for the WFDREQUEST.

This means that the WDFREQUEST cannot be used to send extra data or buffers to the driver which will receive the sent request. For instance, this pattern does not work. First, in the sending driver (let's say request's handle value is 0xA) you format a request context, format the request and send it to the WDFIOTARGET

typedef struct _EXTRA_BUFFER {
   PVOID Data;
   ULONG DataLength;
   ...
} EXTRA_BUFFER, *PEXTRA_BUFFER;

PEXTRA_BUFFER pExtra = NULL;
WDF_OBEJCT_ATTRIBUTES woa;
WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&woa, EXTRA_BUFFER);
NTSTATUS status;

status = WdfObjectAllocateContext(request, &woa, (PVOID*) &pExtra);

// .. initialize pExtra ...
// ... format request as an internal IOCTL ...

if (WdfRequestSend(request, ...) == FALSE) {
   WdfRequestComplete(request, WdfRequestGetStatus(request));
}

And then in the receiving driver's WDFQUEUE dispatch routine would have a new WDFREQUEST handle value (let’s say 0xAA) and not the sender’s WDFREQUEST (0xA).

VOID EvtInternalDeviceIoControl(
    __in WDFQUEUE Queue,
    __in WDFREQUEST Request,
    __in size_t OutputBufferLength,
    __in size_t InputBufferLength,
    __in ULONG IoControlCode
    ) { ... }

Think about how KMDf would have been implemented if the same WDFREQUEST was presented to the lower driver. One of two things would have had to occur:

The first option would have been for the WDFREQUEST handle value to be smuggled somewhere in the PIRP (with the most likely candidate being DriverContext[]), but that would have been unreliable and prone to compatibility issues with non KMDF drivers in the stack.
The second option would have been to maintain a global mapping in KMDF that mapped from PIRP pointer value to WDFREQUEST handle. This would have been very expensive to maintain because we would have had to look up the mapping every time a WDFQUEUE handled PIRP arrived in the framework. It would have grown more even more expensive because this proposed map would have been guarded by a global lock which meant that all KMDF drivers would have been acquiring and releasing this one lock, making it very contentious and a huge performance problem.

Let’s take a step back and look at why there is a WDFREQUEST handle value when you send a request to another device. Here is what happens when a PIRP arrives in a KMDF driver

Call the potentially registered WDM preprocess routine
If it is a PIRP that will be processed by a WDFQUEUE (a read/write/IOCTL/internal IOCTL), allocate a WDFREQUEST for the PIRP. If it is not one of these types, pass it to the appropriate pipeline in the framework
Call the potentially registered in process callback
Pass the WDFREQUEST to the WDFQUEUE handler so that it will be presented on the appropriate WDFQUEUE
The WDFQUEUE calls the right IO event callback that you registered when the WDFQUEUE was created

Step #2 is the key here. The WDFREQUEST is always allocated (from a lookaside if that matters). It means that there is no 1:1 correspondence between a PIRP and WDFREQUEST value. Let’s say we wanted to have a 1:1 correspondence for a singular WDFDEVICE though(and not across multiple WDFDEVICE as in the first example). We would need a WDFDEVICE based mapping of active PIRPs to WDFREQUEST handles to see if the PIRP is an active PIRP in the WDFDEVICE. As in the case with the previously proposed global mapping, such a WDFDEVICE wide mapping would also be prohibitively expensive. While the lock contention would move from a global scope to a WDFDEVICE scope and thus reducing some contention, such a lock would still be a performance hot spot since all PIRPs arriving into the driver would be contending on this lock.

When you look at how KMDF allocates a WDFREQUEST it becomes clear that the WDFRQUEST loosely maps back to the current stack location in the PIRP, not the PIRP itself! Think about the sending the WDFREQUEST to another driver case. The sending driver has its own stack location and its own WDFREQUEST. The receiving driver has its own stack location and WDFREQUEST as well. Just to reinforce this idea, let’s consider a final example. Let’s say

Your driver received a WDFREQUEST formatted for IOCTL ‘A’ in your dispatch routine
In processing the request your driver formatted (the next stack location) it for IOCTL ‘B’
The request is sent to the top of your stack (which means your driver will see the PIRP in its dispatch routine eventually as IOCTL ‘B’).

When PIRP enters your driver as IOCTL 'B', it will have a new WDFREQUEST handle! This is completely by design. The first stack location (IOCTL ‘A’) has one WDFREQUEST, the subsequent stack location (IOCTL ‘B’) has another WDFREQUEST.

In conclusion, WDFREQUEST handles are local to a specific WDFDEVICE. In fact, they are local to a specific stack location. The context off of a WDFREQUEST can only be used by your driver for that particular WDFDEVICE and is not meant to be shared across WDFDEVICEs.

How do I cancel an IRP that another thread may be completing at the same time?

doronh — Mon, 30 Jun 2008 09:16:00 GMT

Let's say that you allocated a PIRP and sent it down your device stack. You free the PIRP in the completion routine and then return STATUS_MORE_PROCESSING_REQUIRED. To make life more fun, you decide that you want to be able to cancel the sent IRP after you have sent it so you try to do it simple like this

typedef struct _DEVICE_EXTENSION {
    KSPIN_LOCK SentIrpLock;
    PIRP SentIrp;
} DEVICE_EXTENSION;

Sending thread:

KeAcquireSpinLock(&devext->SentIrpLock, ...);
devext->SentIrp = Irp;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

Canceling thread:

KeAcquireSpinLock(&devext->SentIrpLock, ...);
if (devext->AllocatedIrp != NULL) {
   IoCancelIrp(devext->SentIrp);
}
KeReleaseSpinLock(&devext->SentIrpLock, ...);

Completion routine:

PIRP irp;

KeAcquireSpinLock(&devext->SentIrpLock, ...);
irp = devext->SentIrp;

devext->SentIrp = NULL;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

IoFreeIrp(irp);

return STATUS_MORE_PROCESSING_REQUIRED;

And it then deadlocks ;). If the call to IoCancelIrp causes the IRP to be completed in the calling context (e.g. the one which has acquired the lock), the completion routine will run and try to acquire the lock (SentIrpLock) on the same thread which holds it.

So, life is not that simple and you have to do something more. The basic solution is that you need extra state to track who is touching the PIRP and who can free it. Walter Oney's book has a solution (IIRC, it is in the self initiated I/O section, but I do not have the book handy), but IMHO it is a bit complicated. KMDF has a solution to this problem which I like much more (imagine that ;)).

You need an extra LONG, calling it CompletionCount per PIRP that you want to be able to cancel.

You initialize CompletionCount to 1 before sending it down the stack and storing it in devext.
Whenever there is a thread that wants to cancel the PIRP, it tries to interlocked increment CompletionCount only if and only if the current CompletionCount value is > 0. For this you need to roll your own InterlockedIncrementWithFloor which is fortunately not that hard and I have already shown you how to do that, http://blogs.msdn.com/doronh/archive/2006/12/06/creating-your-own-interlockedxxx-operation.aspx.
After the canceling thread has called InterlockedIncrementWithFloor and IoCancelIrp, it calls InterlockedDecrement.
Whomever wants to complete the PIRP, like the completion routine, interlock decrements CompletionCount.

If the returned value from InterlockedDecrement is zero, the caller can complete the PIRP. If not, somebody else is trying to touch the PIRP and you must leave the PIRP alone. So here is the revised code:

typedef struct _DEVICE_EXTENSION {
    KSPIN_LOCK SentIrpLock;
    PIRP SentIrp;
    ULONG CompletionCount;
} DEVICE_EXTENSION;

Sending thread:

KeAcquireSpinLock(&devext->SentIrpLock, ...);
devext->CompletionCount = 1;
devext->SentIrp = Irp;

KeReleaseSpinLock(&devext->SentIrpLock, ...);

Canceling thread:

PIRP irp = NULL;
KeAcquireSpinLock(&devext->SentIrpLock, ...);
if (devext->AllocatedIrp != NULL && MyInterlockedIcrementeWithFloor(&devext->CompletionCount, 0) > 0) {
   irp = devext->SentIrp;
}
KeReleaseSpinLock(&devext->SentIrpLock, ...);

if (irp != NULL) {
    IoCancelIrp(irp);
    if (InterlockedDecrement(&devext->CompletionCount) == 0) {
        IoFreeIrp(irp);
    }
}

Completion routine:

PIRP irp = NULL;

KeAcquireSpinLock(&devext->SentIrpLock, ...);
irp = devext->SentIrp;devext->SentIrp = NULL;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

if (InterlockedDecrement(&devext->CompletionCount) == 0) {
    IoFreeIrp(irp);
}

return STATUS_MORE_PROCESSING_REQUIRED;

The beauty of this solution is that if you add more actors (let's say a timer for an async timeout, all you have to do is bump the CompletionCount to account for them to asynchronously rundown if you cannot cancel them.

EvtDevicePreprocessWdmIrp is not entirely free

doronh — Thu, 03 Apr 2008 00:14:09 GMT

One of the WDM escapes in KMDF is EvtDeviceWdmIrpPreprocess (or EvtDevicePreprocessWdmIrp in the API in which you register it) which you can register for by calling WdfDeviceInitAssignWdmIrpPreprocessCallback. This function allows you to process a WDM PIRP before KMDF sees it and potentially processes it. From a KMDF adoption point of view, this functionaltiy was a very strong requirement. Without, there would be not be a defined way for a KMDF to support IRPs that KMDF did not natively support. For instance, the KMDF serial example registers a preprocess routine for IRP_MJ_FLUSH_BUFFERS, from [ddkroot]\src\kmdf\serial\pnp.c:

//
// Since framework queues doesn't handle IRP_MJ_FLUSH_BUFFERS,
// IRP_MJ_QUERY_INFORMATION and IRP_MJ_SET_INFORMATION requests,
// we will register a preprocess callback to handle them.
//
status = WdfDeviceInitAssignWdmIrpPreprocessCallback(
    DeviceInit,
    SerialFlush
    IRP_MJ_FLUSH_BUFFERS,
    NULL, // pointer minor function table
    0); // number of entries in the table

You should use this functionality in your driver only if this is the last resort you have; it does not come for free. When you register for any preprocess routine, KMDF will increase the StackSize in the underlying WDM PDEVICE_OBJECT. This is called out in the documention for "Preprocessing and Postprocessing IRPs" in KMDF. By increasing the stack size, all PIRPs that are sent to your device will have an extra IO stack location. This means that the PIRP is a bit larger than it would otherwise be; larger not only for the IRP_MJ code that you registered for, but for each and every PIRP that is sent to your device regardless of IRP_MJ code. Additionally, if your StackSize goes over an internal threshhold in the IO manager, the way the PIRP is allocated could be different (for instance, devices with a StackSize of <= N might have a PIRP allocated out of a lookaside list while a StackSize > N might always be allocated by calling ExAllocatePool each time). Mind you, the internal threshhold is a bit high so if you have a root enumerated device or your initial StackSize is low, this is not something that you will see.

So while registering a preprocess routine is a very useful thing to be able to do, you should consider your options carefully and only register a preprocess routine if you have no other option.

How to share HW resources with another driver not in the same PnP hierarchy

doronh — Thu, 25 Oct 2007 01:56:23 GMT

First, I have to say that I don't agree with this design pattern at all. I think it leads to too many problems and complications that are not worth the pain. The only reason I am writing this entry is that I have seen so many people get this wrong or not account for some details and I want to make sure that if such a driver is installed, the stability of the system is not compromised. The scenario I am covering is not the case where you are enumerating child devices and want to share resources assigned to the parent amongst the children. Rather, this is sharing HW resources assigned to your device (device A) with another device (device B). Device B can be a legacy NT4 style device or another PnP device stack, it doesn't matter.

There are 3 aspects to the issue that you must solve:

Handing off the resources to device B. This means that device B will open a handle against device A and send an internal IOCTL (e.g. IOCTL_INTERNAL_GET_RESOURCES) which device A completes with the appropriate values. If the number of resources is fixed, this is easy. If the number of resources varies, this can get complicated.
Coordinating access to the resources between device A and device B. If both drivers want to touch the resources at the same time, you now have to synchronize between 2 components and that can get very complicated, especially if there are callbacks involved.
Only touching the resources when the device has power. This is easy for device A to do since it is the power policy owner for the device and sees pnp state changing and device power irps. This is not easy for device B to do though.

Aspects number 1 and 2 I will leave for you to solve. Number 3 is where things get interesting. There are scenarios where the device can lose power:

PnP graceful remove (query remove, remove)
PnP surprise removal
PnP stop
System power state change (e.g. a standby or hibernate)

The first 2 scenarios in green are easily solved. Device B registers for PnP notifications on the file handle that it opens and it will be told when a PnP remove or surprise removal occurs. If you are using KMDF, the WDFIOTARGET automatically does this for you. But there is a catch to the surprise removal notification...it is sent to device B after device A has already processed the PnP IRP. This means that there is a window where device B thinks it has valid resources, but in reality it does not. File handle notifications do not solve the last 2 scenarios though and the surprise removal case does not work well, so it would be nice if there was one solution for all 4 scenarios.

The solution is that device B must register 2 callbacks with device A, a power up and power down callback (they can be condensed to one callback with an additional passed in parameter indicating power state if you want). These callbacks should be set at the same time that the resources are acquired so that there is no window where the resources have been acquired, the device loses power and the callback has not yet been set notifying device B of the power state change. These callbacks can be passed from device B to device in any number of ways. Two traditional ways are

To pass the callbacks in a buffer in an internal IOCTL (you can overload IOCTL_INTERNAL_GET_RESOURCES if you want)
To send a IRP_MN_QUERY_INTERFACE with these callbacks as input values in the INTERFACE based structure as I describe in this post

Once the callbacks have been set, device A must call them at the right time. In a KMDF driver, it is a very simple implementation. You call device B's power up callback in EvtDeviceSelfManagedIoRestart and device B's power down routine in EvtDeviceSelfManagedIoSuspend. This covers all cases because KMDF treats PnP state changes as implicit power changes, powering down the device and using the same callbacks for all cases. This also covers the surprise removal window because device B is now notified at the beginning of device A's power down instead of after it. One final thing to consider is what device B thinks the initial power state of device A is in. Ideally, device B should assume device A is in low power and should not touch the retrieved resources until its power up callback has been invoked. This would mean that the initial call to device B's power up call might occur outside of EvtDeviceSelfManagedIoRestart, perhaps immediately after completing the IOCTL_INTERNAL_GET_RESOURCES request.

If you are writing a WDM driver, well, I will leave that implementation up to you ;).

WdfDeviceRetrieveDeviceInterfaceString and PDOs

doronh — Tue, 23 Oct 2007 02:56:46 GMT

When you initially create a PDO, it takes a few steps for PnP to recognize it. I wrote this problem of determining when a PDO becomes a PDO last year. At the end of the post I mentioned that KMDF handles all of this state management for you underneath the covers. This works for 99% of the client drivers out there who just register a device interface and then let KMDF manage the state and do not care about the interface string.

Unfortunately, this platform issue does bleed out to the driver writer in one DDI, WdfDeviceRetrieveDeviceInterfaceString (which, fortunately, is not a very commonly called DDI). This DDI returns the string generated by the call to IoRegisterDeviceInterface, but since this string cannot be generated (and subsequently returned to the client driver) until the created WDFDEVICE PDO is a PnP recognized PDO, this function returns error. (Currently the returned error is STATUS_INVALID_DEVICE_STATE, but that can change so do not rely on a particular error value, use !NT_SUCCESS instead.) To solve this timing problem, the client driver must query for the interface string at a later time.

I recommend querying for the string in EvtDeviceSelfManagedIoInit. It has two benefits over registering a WDM preprocess routine and querying for the string in IRP_MJ_PNP/IRP_MN_QUERY_RESOURCE_REQUIREMENTS.

You don't have to register a WDM preprocess routine and deal with the underlying WDM structures and not leveraging the state tracking that KMDF provides.
EvtDeviceSelfManagedIoInit is called only on the first time a PDO is started, any subsequent PnP starts will not invoke this callback. This is not true for the WDM preprocess routine, it can be invoked multiple times. Since the preprocess routine can be invoked multipel times, you must now not only track the state of the PDO as reported to PnP, but also the registered state of the interface and the returned interface string.

Setting a security descriptor on a legacy device object

doronh — Wed, 17 Oct 2007 02:45:00 GMT

Setting the security descriptor allows you to control who can open a handle to the device object. Typically you can call IoCreateDeviceSecure to create the device object and have the correct DACL from the start. One issue with IoCreateDeviceSecure is that the SDDL string is limited to what it can describe, primarily you can only specify predefined SIDs. If you need a SID that is not predefined you must build the security descriptor by hand and the code to do it is not very easty to implement (let alone figure out which DDIs you need to call in the first place).

Well, Paul Sliwowicz kindly donated this code snippet which will set a security descriptor on a legacy device object. It does require that you are able to open a handle to the device object first, so it might be a good idea to initially create the device with a very restrictive SDDL string that limits opens to the system only and then relax it to your needs with the following code which works in either a WDM or KMDF driver. This can also work on device objects that are created by other drivers, but you have a built in race between when the device is created and when you set the security descriptor that you must also account for.

NTSTATUS
SetDeviceDacl(
    PDEVICE_OBJECT DeviceObject
    )
{
    SECURITY_DESCRIPTOR sd = { 0 };
    ULONG aclSize = 0;
    PACL pAcl = NULL;

    NTSTATUS status;

    status = ObOpenObjectByPointer(DeviceObject,
                                   OBJ_KERNEL_HANDLE,
                                   NULL,
                                   WRITE_DAC,
                                   0,
                                   KernelMode,
                                   &fileHandle);

    if (!NT_SUCCESS(status)) {
        goto Exit;
    } 

    //
    // Calculate how big our ACL needs to be to support one ACE (in
    // this case we'll use a predefined Se export for local system
    // as the SID)
    //
    aclSize = sizeof(ACL);
    aclSize += RtlLengthSid(SeExports->SeLocalSystemSid);
    aclSize += RtlLengthSid(SeExports->SeAliasAdminsSid);
    aclSize += RtlLengthSid(SeExports->SeAliasUsersSid);
   
    //
    // Room for 3 ACEs (one for each SID above); note we don't
    // include the end of the structure as we've accounted for that
    // length in the SID lengths already.
    //
    aclSize += 3 * FIELD_OFFSET(ACCESS_ALLOWED_ACE, SidStart);

    pAcl = (PACL) ExAllocatePoolWithTag(PagedPool, aclSize, TAG);

    if (pAcl == NULL) {
        status = STATUS_INSUFFICIENT_RESOURCES;
        goto Exit;
    }

    status = RtlCreateAcl(pAcl, aclSize, ACL_REVISION);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    status = RtlAddAccessAllowedAce(pAcl,
                                    ACL_REVISION,
                                    GENERIC_READ | GENERIC_WRITE | DELETE,
                                    SeExports->SeLocalSystemSid);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    status = RtlAddAccessAllowedAce(pAcl,
                                    ACL_REVISION,
                                    GENERIC_READ | GENERIC_WRITE | DELETE,
                                    SeExports->SeAliasAdminsSid );
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    status = RtlAddAccessAllowedAce(pAcl,
                                    ACL_REVISION,
                                    GENERIC_READ,
                                    SeExports->SeAliasUsersSid );

    if (!NT_SUCCESS(status)) {
        goto Exit;
    }
 
    //
    // Create a security descriptor
    //
    status = RtlCreateSecurityDescriptor(&sd, SECURITY_DESCRIPTOR_REVISION);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    //
    // Associate the above pAcl with the security descriptor
    //
    status = RtlSetDaclSecurityDescriptor(&sd, TRUE, pAcl, FALSE);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    //
    // Set security on the object
    //
    status = ZwSetSecurityObject(fileHandle, DACL_SECURITY_INFORMATION, &sd);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

Exit:
    ZwClose(fileHandle);
    fileHandle = NULL;

    if (pAcl != NULL) {
        ExFreePool(pAcl);
        pAcl = NULL;
    }

    return status;
}

Having two names is not necessarily better than one, part 2

doronh — Thu, 12 Jul 2007 21:10:29 GMT

Previously I wrote about what happens when there are 2 devices in a stack with a name and all of the associated issues that arise from it. In that vein I want to write about what happens when you give your device object a name in KMDF and compare it to WDM.

In a WDM driver, if you specify a name for your device object when you call IoCreateDevice nothing related to security descriptors occurs. The device is accessible to everything on the machine. On a side not, if you specify certain types of DeviceTypes a different security descriptor will be applied to your device and it will limit who can open your device.

On the hand, when you assign a name to the device in a KMDF driver (by calling WdfDeviceAssignName), KMDF will either use an SDDL string you provide, or if one is not provided, a different security descriptor other than the WDM default. As opposed to the WDM default which allowed anyone to open the device, the KMDF default specifies that only administrators can open the device using the SDDL string "D:P(A;;GA;;;SY)(A;;GA;;;BA)" (which is defined as the constant SDDL_DEVOBJ_SYS_ALL_ADM_ALL in wdmsec.h). If you don't like this default you can specify your own SDDL when creating the device by calling WdfDeviceInitAssignSDDLString. If you want the WDM default, I think that SDDL_DEVOBJ_SYS_ALL_ADM_RWX_WORLD_RWX_RES_RWX (see wdmsec.h) will give you what you want.

This behavior is documented in the online help, you can read it here. There is also a good page on SDDL for device objects.

So why did we do this? This is obviously a breaking change from WDM behavior and it does make writing or porting to a KMDF a little harder because you will probably encounter this issue only after you execute your application as a non administrator and see that it cannot open your device. Well, we had good reason: we wanted a KMDF driver to be secure by default and to have access open up explicitly unlike the WDM model where you could open yourself up to anyone and had to explicitly restrict access. By making access explicit you have to think about who should access the device as a part of the creation of the driver. Being explicit about access is pervasive through other types of device object creation as well, both raw PDOs (WdfPdoInitAssignRawDevice) and control devices (WdfControlDeviceInitAllocate) require an SDDL string to be created.

Problems with not having a current IRP stack location, part 2

doronh — Mon, 09 Apr 2007 19:04:00 GMT

This problem falls into the category being hidden by a macro that does not indicate in its name what it touches. If you call IoMarkIrpPending on an IRP that you allocated in your driver, chances are that you are corrupting memory. First, let's look at the implementation of this function (from wdm.h):

#define IoMarkIrpPending( Irp ) ( \
    IoGetCurrentIrpStackLocation( (Irp) )->Control |= SL_PENDING_RETURNED )

This function retrieves the current stack location and sets a flag in one of the fields (IO_STACK_LOCATION::Control). Since the current IRP stack location is not valid, and by not valid I mean that the pointer value points to undefined memory, not that it is NULL, so the call corrupts memory (a nefarious single bit flip) somewhere in the system. It so happens that based on how IRPs are currently allocated today, you will corrupt memory immediately after the IRP allocation making this a little bit easier to diagnose later.

How would a mistaken call to IoMarkIrpPending even get into your driver? Well, most code is copied from somewhere else. A lot of completion routines have the following code in them. These completion routines were written to manipulate an I/O manager presented IRP (which works because there is a current stack location), but when copied over they have a time bomb waiting to go off in them. Here is an example completion routine which is copied:

NTSTATUS CompletionRoutine(PDEVICE_OBJECT DeviceObject, PIRP Irp, PVOID Context)
{
    ...
    if (Irp->PendingReturned) {
         IoMarkIrpPending(Irp);
    }
    ...
    IoFreeIrp(Irp);

    //
    // Must return STATUS_MORE_PROCESSING_REQUIRED because we cannot let the IRP complete
    // back to the I/O manager
    //
    return STATUS_MORE_PROCESSING_REQUIRED;
}

What happens if you have an I/O completion routine that handles both types (driver created, I/O manager presented) IRPs? In the I/O manager presented IRP case you need to propagate the pending state by calling IoMarkIrpPending, while in the driver create case you obviously should not propagate the state. KMDF must do this since it sets the same completion routine for all requests sent to a WDFIOTARGET. Here is how KMDF manages this functionality. If Irp->PendingReturned is TRUE and the current stack location count is less than or equal to the total stack count (in the driver allocated case, CurrentLocation will be > StackCount), propagate the pending into the current stack.

VOID
PropagatePendingReturned(
    PIRP Irp
    )
{
   if (Irp->PendingReturned && Irp->CurrentLocation <= Irp->StackCount) {
       IoMarkIrpPending(Irp);
   }
}

BTW, unaccredited part 1 is why you don't have a DeviceObject in your I/O completion routine

Filtering HID collections and setting I/O flags

doronh — Tue, 27 Mar 2007 03:19:00 GMT

Over the past 3 years or so, I have been casually referring to KMDF as the ultimate driver compat layer. Just like Windows has an application compatibility layer which shields applications from OS changes, KMDF provides the same type of compatibility across device stacks. For the most part each stack implements WDM as per the documentation and there is nothing special that KMDF must compensate for when inserted into the stack. HID is not one of these stacks ;). First, let me describe the HID stack.

+---------+
| Raw PDO |
+---------+
     |
     |    +----------------+
     +----| HID miniport + |
          | HIDclass       | 
          +----------------+
                   |
          +----------------+
          |       PDO      |
          +----------------+

HIDClass (the port driver) will enumerate a raw PDO for each top level collection (TLC) in a device's HID descriptor (in this picture there is only one raw PDO, but there could be many). A raw PDO is special in that it does not need an additional INF to be started, all of the functionality that is usually found in a stack's FDO is in the raw PDO. This includes the DeviceObject Flags which specify the buffering type for reads and writes on the device (more on this later). A raw PDO can have a FDO and filters layered on top of it, this is how the HID keyboard and mouse stacks work. A TLC which is neither a keyboard nor a mouse can have a filter installed on it as well.

Setting the flags is where HID does not play by the rules. Typically a bus driver will specify the flags (DO_DIRECT_IO or DO_BUFFERED_IO) which dictate the buffering in read and write IRPs when the device object is created. This would allow an FDO or filter to determine the I/O type during the FDO or filter's AddDevice routine and copy them into its own device object (since the I/O manager looks at the top of the stack to determine buffering for reads and writes). HIDClass on the other hand sets one of the flag (DO_DIRECT_IO) during IRP_MN_START_DEVICE! This means that in a WDM driver filtering on top of a raw HID PDO, you must special case this behavior and set these flags in your device object during IRP_MN_START_DEVICE processing. KMDF takes care of this for you, so your HID filter code is no different than a filter for any other device stack (this is where the driver compat layer is).

What makes the situation worse is that there is a HID filter sample, firefly, which does not change the I/O type in its device object's Flags, meaning it is now a generic HID filter sample, rather only a HID mouse sample. But how does firefly even work in the mouse stack? Well, the upper edge of the mouse stack (mouclass.sys) always sets DO_BUFFERED_IO and read IRPs are never sent down the stack, so HIDClass would never see a read IRP with the wrong buffer type. This means that the propagation of the buffering flags is not needed.

So how is it that this bug existed HIDClass for so long without being discovered? Remember that HIDClass enumerates its PDOs as raw and 99% of the time, nobody loads on top of them, so there was no other WDM component to interact with this behavior of setting the Flags field at the wrong time.

Supporting query interface in KMDF

doronh — Mon, 12 Mar 2007 21:25:00 GMT

In my last post I talked about bidirectional interfaces which can have both input and output parameters. KMDF supports both unidirectional and bidirectional interfaces. As is the case with many KMDF APIs and behaviors, we allow you to implement the easy pattern with little to no work and let you opt in to more complicated functionality as needed. Query interface support follows this philosophy.

If you want to support a unidirectional interface which only contains output parameters, you have very little code to write. First, declare and initialize the interface as you would want it to be returned to the requestor. Second, declare and initialize a WDF_QUERY_INTERFACE_CONFIG structure with your interface. Finally, you call WdfDeviceAddQueryInterface(). Here is a snippet from the WDK KMDF toaster bus sample which demonstrates how easy this is

WDF_QUERY_INTERFACE_CONFIG  qiConfig;
TOASTER_INTERFACE_STANDARD  ToasterInterface;

//
// Create a copy of the interface that the framework will use when
// completing the query interface request.
//
RtlZeroMemory(&ToasterInterface, sizeof(ToasterInterface));

ToasterInterface.InterfaceHeader.Size = sizeof(ToasterInterface);
ToasterInterface.InterfaceHeader.Version = 1;
ToasterInterface.InterfaceHeader.Context = (PVOID) hChild;

//
// Let the framework handle reference counting.
//
ToasterInterface.InterfaceHeader.InterfaceReference =
    WdfDeviceInterfaceReferenceNoOp;
ToasterInterface.InterfaceHeader.InterfaceDereference =
    WdfDeviceInterfaceDereferenceNoOp;

ToasterInterface.GetCrispinessLevel  = Bus_GetCrispinessLevel;
ToasterInterface.SetCrispinessLevel  = Bus_SetCrispinessLevel;
ToasterInterface.IsSafetyLockEnabled = Bus_IsSafetyLockEnabled;

WDF_QUERY_INTERFACE_CONFIG_INIT(&qiConfig,
                                (PINTERFACE) &ToasterInterface,
                                &GUID_TOASTER_INTERFACE_STANDARD,
                                NULL);

status = WdfDeviceAddQueryInterface(hChild, &qiConfig);
if (!NT_SUCCESS(status)) {
    return status;
}

But what about other scenarios? There are 3 other patterns that apply to handling a query interface request:

You want to selectively succeed, fail, or pass down the query interface request based on state or previous requests
You want to capture input parameters in the request and optionally set output parameters
You want forward the request down to the parent stack (PDO only)

Selectively succeed, fail, or pass down the request

Typically a query interface request is processed regardless of the previous queries or device state, but not always. For instance, let's say you want to only handle out the interface to one requestor (it would be available once it is dereferenced). Another example might be that you want to determine the status of the query interface request based on the capabilities exposed lower in the device stack. Finally, to process the query interface, you might be required to allocate memory (which could lead to failure if you cannot allocate the required buffer). To allow your driver to determine the status, check state, perform your custom action, etc. you specify a EvtDeviceProcessQueryInterfaceRequest function pointer in your WDF_QUERY_INTERFACE_CONFIG structure after calling WDF_QUERY_INTERFACE_CONFIG_INIT(). To succeed the query interface request, return a value which is NT_SUCCESS(), likewise to fail a request, return a value which is !NT_SUCCESS(). To instruct the framework to pass down the request to the lower stack, you return a special value, STATUS_NOT_SUPPORTED.

Capturing input parameters in the request

There is a field in WDF_QUERY_INTERFACE_CONFIG which controls how the framework treats the interface buffer itself. If ImportInterface is FALSE (the default value), the interface buffer is treated as a pure output buffer and the framework will copy over the interface to return directly into the requestor's buffer (and subsequently overwriting any values the requestor initialized the buffer with). If ImportInterface is TRUE, the framework does not touch the requestor's buffer. This allows you to inspect the requestor's buffer first and copy data from it. By setting ImportInterface to TRUE, there are two direct side effects that you must deal with. First, you must specify a EvtDeviceProcessQueryInterfaceRequest function when adding the interface. Second, and most importantly, your driver must now copy all output values into the requestor's buffer because the framework no longer does this for you!

Forwarding the request to the parent's stack

There are some query interface requests that must be forwarded from the PDO down to the parent's stack (and most likely to be completed by the parent's own PDO). For example, look at usbccgp (the USB generic parent driver). This driver does not support every single USBHUB interface, yet it must maintain compatibility with the hub's supported interfaces. To do this, it forwards all unhandled requests down from the PDO it created down the parent's stack. The framework can automatically provide this functionality to your driver if you set SendQueryToParentStack to TRUE.

Power relationships in a bus driver

doronh — Sat, 03 Mar 2007 04:28:00 GMT

A small but important rule in WDM is that while a PDO is in D0, the parent must be in D0 as well. A very simple statement that can cause a lot of trouble ;). What I have seen is that very few WDM drivers enforce this rule (while KMDF does implement this rule, so all KMDF drivers inherit this for free) since it can be quite difficult to get right. It is hard to get right because there are a few scenarios that affect how this rule is interpreted. Here are a few scenarios that might affect its interpretation:

The machine is entering an Sx state and the parent is powering down
The machine is running (e.g. S0) and all of the children have idled out and powered down
The children are software devices which are not connected to any real hardware
A child is started while the parent is in Dx because it idled out
The parent who enumerated the PDO is not the power policy owner for the stack

The machine is entering an Sx state the and parent is powering down

In this scenario, if all of the power policy owners (i.e. a PPO) for your child stacks are implemented properly, there is nothing for your bus driver to do. A proper PPO implementation will pend the Sx IRP, request a Dx IRP, and once the Dx IRP is completed by the entire stack, complete the Sx irp. The OS's power manager will guarantee that the parent FDO will not see an Sx request until all of its children have completed their Sx IRPs. This means that in this case, the OS enforces the power relationship. But what happens if one of the child stacks has a PPO implementation which is wrong? For instance, it requests the Dx IRP but does not wait for it to complete and lets the Sx IRP complete asynchronously before Dx has completed. It introduces a race condition. The Dx could complete before the parent is powered off and everything is OK, or it could lead to the parent powering off before the child. What happens if the PDO does not have a PPO at all? In this case, the PDO is always in D0. In both cases, you can have a parent that is in Dx and a child that is in D0. What should you do as a bus driver in this case? You can't block the Sx IRP until all of the children have powered down because that could prevent the machine from actually reaching the intended Sx state. So you do nothing because this is a bug in the PDO's PPO and there is nothing you can do about.

The machine is running and all of the children have idled out and powered down

This is the trickiest scenario to implement. You do not have Sx irps to guard the states, so the driver must guard its own state. When you are in this state, the bus may want to power down as well. To implement a power down of the parent, two things must occur. First you must decide to power down (typically by starting a timer so that you don't immediately power down the parent). Second, you must put in a barrier so that if a child PDO stack decides to power up (as indicated to the bus driver by a D0 IRP arriving at the PDO) after you have made the decision to power down or are in the middle of powering down, the PDO's D0 IRP is pended until the parent is completely powered down and then powered back up again. Once the parent returns to D0, the PDO's D0 IRP can be processed. Getting all of this right is not trivial, KMDF does it, but even in KMDF it took us until v1.5 to handle these small edge cases that are hard to hit.

The children are software devices

Since the children are software devices (a software device is one that does not control hardware nor is it enumerated by a bus that controls hardware (like USB)), device power state has no real effect on the functionality of the child stack. This means that the PDO's PPO will never put the device into Dx. The bus driver doesn't care about the parent/child power relationship in this case either because there is nothing gained by it. You can only take this shortcut in a WDM driver. In KMDF we always enforce the rule.

A child is started while the parent is in Dx because it is idled out

Let's say that the parent is in a Dx state and then a child stack is started. (You can get into this state if the PDO was previously enumerated and disabled and then the user enables the PDO stack.) This is a variation on the previous idle scenario. In this scenario, the relationship must be enforced, so the PDO's handling of the PnP start IRP must be asynchronously pended until the parent moves into D0. After the parent returns to D0, the child's PnP start IRP processing can resume. In KMDF, the PDO calls the equivalent of WdfDeviceStopIdle() on the parent WDFDEVICE when processing the start IRP (and calls WdfDeviceResumeIdle() on the parent when it is put into the powered down state so that the parent can optionally idle out). The power state machine in KMDF makes handling this scenario very simple.

The parent who enumerated the PDO is not the power policy owner for the stack

Let's take the previous scenario, but remove the fact that the enumerator is the PPO for the parent stack. Now, there is no way for the PDO's enumerator to guarantee that the parent is in D0 when the child PDO is in D0. This completely breaks the rule, but there is nothing you can do as a bus driver (KMDF is aware of this scenario and doesn't try to change the parent enumerator's power state). What this boils down to is that the child stack is a software device (see previous scenario for a description of a software device) where the device power state does not matter.

Problems associated with arming yourself for wake

doronh — Wed, 28 Feb 2007 01:59:54 GMT

Hindsite in this case is 20/20, but arming your device for wake can open yourself up to multitude of race conditions and hard problems. Some of them you can solve in your driver, some of them you must accept that they are there and leave them be. Arming yourself for wake can be broken down into 2 scenarios, arming yourself for wake when your device has gone idle while the machine is running (S0) and arming yourself for wake while when the machine is going into low power (Sx). Here are a few things to consider when writing a WDM driver, a KMDF driver has many of these scenarios handled for you by the framework:

Common races

After you send the wait wake irp down the stack to the PDO, you don't know when it actually arrives at the PDO and is processed. This is important to know because you don't want to send the Dx power irp until the bus driver has had a chance to process the wake request. The best you can do is request the Dx irp after you send the wait wake irp down the stack and then hope that no driver in between yours and the PDO pends the wait wake irp or changes any of the timing.
Once you have armed yourself for wake, your device triggers wake while still in D0 (before you can send the Dx irp). Another race that you just have to accept, but one that you can fix in firmware. The device's firmware can be changed to only trigger wake if the device is in a low power state.

Idle while in S0

After you have requested the wait wake irp and before it arrives, you decide you need to longer arm yourself for wake because of some external event. You must now track this, let the wait wake irp flow through, then cancel it. Another variation is that the wait wake irp has been processed and you make the decision to power up after the Dx irp has been requested, but before the Dx irp has arrived. Same solution as before as well, you must wait for the Dx irp to flow through the stack and then request a D0 irp.
USB selective suspend introduces another state. You must be idle and then the usb core must let you know when to actually power down and arm your device. This can lead to another state in which you must account for trying to back out of if you are no longer idle.

Wake from Sx

You have armed yourself for wake and put the device into a lower power state, but the machine is not yet in lower power. At this point your device triggers wake. For instance, let's say you are arming a mouse to wake the machine and the user moves the mouse after it is armed and in low power but the machine is still powered on because it is turning off other devices). There are 2 usability problems here. First, your device triggered too early. Second, your device will not actually wake up the machine once it goes into low power and the user may thing something is wrong!
Your device has triggered wake and awoken the machine. When do you cancel the wait wake request so that that the underlying bus can turn off wake signaling in the hardware? When you get the S0 irp or after you sent the D0 irp to the PDO? There are some devices which reflect the wake status into an interrupt when they are fully powered on. If you send the D0 irp without canceling the wait wake irp, the device may trigger an interrupt storm when the PDO processes the D0 irp (and you cannot turn off your device's functionality yet because driver's process D0 from the bottom up to the top of the device stack). If your hardware does this, you will have to cancel the wait wake irp and wait for it to complete before requesting a D0 irp when you receive the S0 irp.

I can't imagine how much work it would be to actually /write/ a book

doronh — Wed, 21 Feb 2007 04:27:00 GMT

All I can say is that it is a ton of work to review one. We just finished all of the reviews for the WDF book and man was it alot of work! We have an awesome set of writers, I really don't know how they managed it all. From creating the content, taking a ton of feedback and edits from a large number of people, and keeping the same voice throughout, the writers worked tirelessly to make it happen. I could have never imagined how much work was involved on all sides of the equation.

Go buy the book when it comes out! ... and no I don't get any royalties, I just think that if you are writing a WDF driver (either UMDF or KMDF), this is a must read.

This also explains why I have not been posting for the past few weeks or so. I promise that things will pick up again!

Which PnP and power IRPs are synchronized against each other?

doronh — Wed, 07 Feb 2007 09:26:00 GMT

In my previous post, I talked about how state changing PnP IRPs (refered to from now own as just PnP IRPs) are serialized against each other and briefly mentioned which power IRPs they were synchronized against. This merits its own entry. In short, PnP IRPs are only synchronized against system power IRPs (e.g. IRP_MN_SET_POWER/SystemPowerState irps). PnP IRPs are not synchronized against any other power IRP, which are

IRP_MN_SET_POWER/DevicePowerState
IRP_MN_WAIT_WAKE
IRP_MN_QUERY_POWER/SystemPowerState
IRP_MN_QUERY_POWER/DevicePowerState
IRP_MN_POWER_SEQUENCE (unimplemented)

If you think about it, it makes sense that some of these power IRPs are not synchronized against PnP IRPs. Let's take wait wake as an example, your device has idled out and armed itself for wake, which means a wait wake IRP is pending in the stack at the PDO. While idled out, the device is surprise removed from the system. If the wait wake power irp was synchronized against PnP IRPs, the PnP IRP_MN_SURPRISE_REMOVAL IRP would never arrive in the until the wait wake IRP was completed by the PDO or canceled by the device stack's power policy owner. While you could argue that the bus driver should complete the wait wake IRP when it reports the device as missing and have the wait wake IRP be synchronized against PnP IRPs, there are two problems with this approach. First, it places an additional burden on the bus driver and writing a bus driver is complicated enough ;). Second, any driver in the stack can cause a IRP_MN_SURPRISE_REMOVAL IRP to be sent to the stack (by invalidating the device state with IoInvalidateDeviceState and then returning a failed state in IRP_MN_QUERY_PNP_DEVICE_STATE (not to mention the problem that IRP_MN_QUERY_PNP_DEVICE_STATE is also a state changing PnP IRP so there is no way you could query the state while a wait wake irp was pended) without the knowledge of the bus driver, so the bus driver or power policy owner would not be able to proactively cancel the wait irp to allow the surprise removal IRP to arrive.

On the other hand, you might wonder why the OS did not serialize IRP_MN_QUERY_POWER/DevicePowerState power IRPS against PnP IRPs. Unlike wait wake IRPs, device power IRPs are not pended in the stack, they flow through the stack and return back to the PnP manager rather quickly. I wondered the same thing. The only reason I could think of why they are not synchronized is that on Windows 2000 the power policy owner was supposed to pend the IRP_MN_SET_POWER/SystemPowerIrp until the requested IRP_MN_SET_POWER/DevicePowerIrp has completed. This means that you be able to have both types of power IRPs pended in the stack at the same time, which is not possible if both of these types of IRPs are synchronized against each other.

So, what does this mean for you as a driver writer? If your driver is WDM, you have manually synchronize the sending and waiting for power irps during PnP and IRP_MN_SET_POWER/SystemPowerState state transitions (for instance, waiting for a D0 irp to complete when returning from idle and receiving a IRP_MN_SET_POWER/SystemPowerIrp where you will have to send a Dx IRP). This will involve a lot of state management and is not a simple task. On the other hand, if your driver uses KMDF, there is nothing you need to do; KMDF will take care of all the state management for you.