A Hole In My Head : WDF

The WDF 1.7 cointstallers are now available

doronh — Fri, 18 Apr 2008 02:20:00 GMT

After a long wait (thank you for your patience!), the WDF 1.7 coinstallers are now up on the connect site. To get the bits

go to http://connect.microsoft.com
Log in using your passport account
Navigate to the WDF page (I don't know where it lives in the connection directory, sigh)
Choose Downloads on the left
The package is dated 4/17/2008, you may be able to get it directly from here once you have logged on

Enjoy and let the signing and shipping of v1.7 WDF drivers begin!

Making sure the IO manager evaluates the security of your device

doronh — Thu, 04 Oct 2007 19:00:11 GMT

Last time I wrote about how the IO manager handles the creation of file handles and pointed out a potential security hole. If there is a namespace (or path) after your device's name in the path passed to CreateFile, the IO managed does not evaluate the security settings set on your device and relies on your driver to do the evaluation. For instance, if you create a device object and specify that only administrators have access to it and you do not validate access during IRP_MJ_CREATE processing, any user regardless of privilege level may open a handle to your device if they specify any namespace (i.e. "\Device\FooDevice\Blah") when opening a handle.

A gut shot reaction to this behavior is that is a very large tax for every device driver to pay and the NT kernel developers agreed. To rectify the situation a new characteristics flag, FILE_DEVICE_SECURE_OPEN, was added. This flag tells the IO manager that when a device namespace is present during handle creation that the IO manager should no longer skip the security check and should still evaluate if the caller has sufficient rights to open the device. By setting this flag, the caller's access to the object is always evaluated at the IO manager level and the driver does nothing more then set this flag.

You might say to yourself, "well that is backwards, shouldn't the driver writer get this behavior by default and choose to opt out of it instead of knowing that the device must opt in to get the most secure behavior?" and I would agree with you. The reason the most secure behavior was not made default was for backwards compatibility. This hole was not discovered until after NT 3.1 shipped and if the IO manager behavior was flipped, already shipping drivers would stop functioning.

So when should you set this flag? In my opinion you should always set this flag in a WDM driver! The one exception is if you are creating a device upon which a file system will be mounted (if you set the flag in this case, the file system will not be able to evaluate the security of the namespace string). In the opinion of the WDF team, this flag is so important that we tried to guarantee that this flag is set for all WDF device objects, even if you tried to clear it by calling WdfDeviceInitSetCharacteristics or WdfDeviceSetCharacteristics. If you are writing a KMDF driver which will have a file system mounted on top of it, this is how you would clear the flag after creating the WDFDEVICE

WdfDeviceWdmGetDeviceObject(device)->Characteristics &= ~FILE_DEVICE_SECURE_OPEN;

This KB article also discusses the side effects of the FILE_DEVICE_SECURE_OPEN flag on inbox drivers in previous OS releases.

Of Pipes and Endpoints

doronh — Wed, 11 Jul 2007 03:13:23 GMT

If you look at the KMDF headers, you will see two names, pipes and endpoints, that refer to the same concept. They can be confusing, I had to spend a few minutes explaining them to one our technical writers awhile back explaining why we had two names in the first place (and, yes, there is a method to the madness).

If you have read the USB specification or looked at descriptors before, the term endpoint should not be anything new to you. An endpoint is an address on the device. The endpoint is unidirectional (unless it is a control endpoint), has a type (control, interrupt, bulk, isoch) and other properties. The endpoint is self descriptive, you can retrieve the endpoint descriptor by retrieving the config descriptor and then parsing for each endpoint. Endpoint is a term that relates to the hardware itself, independent of the host operating system.

Pipes on the other hand is purely an OS term (IIRC, the USB core spec does mention pipes as well, but uses them as an abstract term). A pipe is an endpoint that has been configured through a select configuration or selecting an interface's alternate setting. A pipe has all the properties of an endpoint, but it is active and be used to communicate with the host. To put it bluntly, an unconfigured endpoint is called an endpoint while a configured endpoint is called a pipe.

Why even make this distinction?

We made the distinction because what you do with an endpoint and a pipe are completely different. If a driver is going to look at endpoint information, it will be before the device has been configured or during selection of an alternate setting. The driver will iterate over all of the interfaces (WdfUsbTargetDeviceGetNumInterfaces, WdfUsbTargetDeviceGetInterface) and then iterate over each interfaces list of settings (WdfUsbInterfaceGetNumSettings, WdfUsbInterfaceGetNumEndpoints) and look at the properties of each endpoint (WdfUsbInterfaceGetEndpointInformation) or the entire set of endpoints in the setting. Looking at the endpoint information is cheap and does not affect the configured state of the device. Note that each of these DDIs use the term "Endpoint" in their name; any DDI which has "Endpoint" in its name operates on an unconfigured endpoint.

While (IMHO) very few drivers will search for specific unconfigured endpoint information, every USB driver will be looking at configured pipe information after selecting a configuration. A driver will iterate over an interface's pipes (WdfUsbInterfaceGetNumConfiguredPipes, WdfUsbInterfaceGetConfiguredPipe) and then retrieve the pipe's information (WdfUsbTargetPipeGetInformation). If a driver has multiple interfaces then this is repeated for each interface. Note that each of these DDIs use the term "Pipe" to in their name; any DDI which has "Pipe" in its name operates on a configured pipe.

One question that I have been asked is why the WdfUsbTargetPipeXxx DDIs aren't called WdfUsbTargetConfiguredPipeXxx and the reason is that a WDFUSBPIPE can only be a configured pipe and the only way to get a WDFUSBPIPE is to make a call to WdfUsbGetConfiguredPipe. Also while I have referred to the KMDF DDIs, the UMDF DDIs have very similar names and follow the same pattern.

Why does WDF have its own (Wdf)False?

doronh — Fri, 22 Jun 2007 21:00:00 GMT

I mean, come on! Another false value? You have the C++ false, the Windows FALSE and now another one. Furthermore, another boolean type (BOOL, BOOLEAN, bool)? … at least we didn't add one a traditional Boolean type. What gives? If I were not a member of the WDF team I would question their design skills, but since I am a member of the team that designed the usage of this value, I feel I have some explaining to do. WdfFalse is a part of an enumeration, WDF_TRI_STATE, which also defines WdfTrue and WdfUseDefault.

typedef enum _WDF_TRI_STATE {
    WdfFalse = FALSE,
    WdfTrue = TRUE,
    WdfUseDefault = 2,
} WDF_TRI_STATE, *PWDF_TRI_STATE;

As you can see, WdfFalse is equal to FALSE (as WdfTrue is equal to TRUE). Why would we redefine the traditional TRUE and FALSE values instead of just using the traditional values as is? Well, WDF has data structures (WDF_DEVICE_PNP_CAPABILITIES, WDF_DEVICE_POWER_CAPABILITIES, etc.) with lots of BOOLEAN like fields in them and we wanted to be able to only set a subset of them at a time and then call a DDI. What we didn't want is that the developer had to set each field before making the call. The framewoek had to be able to tell if a field was intentionally set or was just the default value as set from the INIT() routine (so that it could be ignored). There are 2 common ways implement this pattern. The first way is to have a separate field in the structure which is a bit field of flags and you set a flag in this field to indicate that another field was set (many Win32 common controls use this pattern, especiall the listview and treeview controls). The second way is to choose a value which indicates that the field should be ignored. For a BOOLEAN value, only TRUE and FALSE are valid values, but we didn't want to define a value like NOT_TRUE_OR_FALSE, so we created a new enum to represent true, false, and not set. WDF_TRI_STATE is that enumeration. Originally WDF_TRI_STATE looked like this:

typedef enum _WDF_TRI_STATE {
    WdfUseDefault = 0,
    WdfTrue = 1,
    WdfFalse = 2,
} WDF_TRI_STATE, *PWDF_TRI_STATE;

This made the the INIT() routine very simple since WdfUseDefault was zero. The INIT() routine could zero the structure (which would set all of the WDF_TRI_STATE fields to WdfUseDefault), set the Size field and then return (assuming all of the fields were WDF_TRI_STATE, if not, other initialization might occur). Conceptually this enum definition was clean and simple. The problem was that developers treated the WDF_TRI_STATE fields as BOOLEANs and used TRUE and FALSE instead of WdfTrue and WdfFalse. This meant that if the developer assigned FALSE to a field, it was not really false, it was WdfUseDefault (which meant the field was ignored)! To make matters worse, the compiler never flagged the error because TRUE and FALSE are #define constants, not real enumerants and C/C++ is very loose about translations between constant values and enumerations. We created a type that was almost always used incorrectly and the error was not easily discoverable. Very bad for usability and writing correct the first time.

By the time we realized this problem, getting rid of WDF_TRI_STATE was not viable (and besides, getting rid of it didn't do us any good, we still needed a way to set a few fields and know how to ignore other fields), so we changed the definition of WDF_TRI_STATE to its current state. The way it is currently defined makes it easier on the driver writer, WdfFalse and WdfTrue map to the traditional FALSE and TRUE which means they can be used interchangably. More importantly, there is no undiscoverable mistake for the driver writer to make. On the other hand, this definition does place the burden on WDF to manually initialize each WDF_TRI_STATE field the structure correctly in the INIT() routine. For any structure which has a WDF_TRI_STATE field, the structure must be zeroed, the Size field is set, and then each WDF_TRI_STATE field must be set to WdfUseDefault (see WDF_DEVICE_PNP_CAPABILITIES_INIT in wdfdevice.h for an example of all the gory details).

So, in the end, we were not crazy or out of our minds :). We were well intentioned, got bit by a usability bug and a lack of type safety in the compiler and altered our design accordingly. In retrospect, we could have used values like WdfYes and WdfNo instead of WdfTrue and WdfFalse, but these alternate names would still be equal to their BOOLEAN counterparts and I don't think it woudl have been any better in the end.

Your completion routine context can be anything you want

doronh — Sat, 07 Apr 2007 00:25:00 GMT

It sounds obvious, but sometimes it needs to be stated. For instance, let's say that you are allocating your own IRP, your context contains I/O related data (like a URB) and you encounter the issue where the DeviceObject passed to your I/O completion routine is NULL. Adding another stack location is one solution I wrote about, but it is a little complicated and has a certain black magic feeling to it, besides it does not work with any IoBuildXxx API calls. Since you are allocating memory for your context, just allocate a little bit more. Instead of allocating just the URB

PURB urb = ExAllocatePoolWithTag(NonPagedPool, sizeof(URB), ...);
PIRP irp = IoAllocateIrp(...);
...
// format the URB
IoSetCompletionRoutine(Irp, CompletionRoutine, urb, TRUE, TRUE, TRUE);
...
IoCallDriver(..., Irp);
...

NTSTATUS CompletionRoutine(PDEVIC_OBJECT Device, PIRP Irp, PURB Urb)
{
     // Do not touch Device, it is NULL!
     // Process Urb and Irp results
     ExFreePool(Urb);
     IoFreeIrp(Irp);

     return STATUS_MORE_PROCESSING_REQUIRED;
}

allocate a structure to contain the URB and your DeviceObject pointer (or WDFDEVICE or anything else)

typedef struct _COMPLETION_DATA {
    PDEVICE_OBJECT DeviceObject;
    URB Urb;
} COMPLETION_DATA, *PCOMPLETION_DATA;

PCOMPLETION_DATA data = ExAllocatePoolWithTag(NonPagedPool, sizeof(COMPLETION_DATA), ...);
PIRP irp = IoAllocateIrp(...);
...
// format data->Urb
data->DeviceObject = DeviceObject;
IoSetCompletionRoutine(Irp, CompletionRoutine, data, TRUE, TRUE, TRUE);
...
IoCallDriver(..., Irp);
...

NTSTATUS CompletionRoutine(PDEVIC_OBJECT Device, PIRP Irp, PCOMPLETION_DATA Data)
{
     // Do not touch Device, it is NULL, but Data->Device is valid!
     
     // Process Data->Urb and Irp results
     ExFreePool(Data);
     IoFreeIrp(Irp);

     return STATUS_MORE_PROCESSING_REQUIRED;
}

Like I said, it is an obvious thing to do :), but when you are in the depths of debugging a bugcheck, sometimes the obvious solution is the last thing you think of.

Which callback is called on surprise or graceful removal?

doronh — Mon, 26 Feb 2007 21:14:00 GMT

WDF performs a lot of state tracking for you, from PnP to power state to the number of outstanding I/Os. In particular to this question, WDF abstracts surprise removal and graceful removal from you. WDF also handles all of the standard things that you must perform on surprise or graceful removal: disabling device interfaces, destroying symbolic links, unregistering from WMI, etc. But what if you need release your own device specific state (such a device index number that is constant for this device instance) in the context of the correct PnP IRP? Remember that you can't just clean up on the PnP remove IRP since your device can be stuck in the surprise removed state as long as there is an open handle against it. Since the difference between the removals is abstracted away, this would appear to be a difficult problem to solve by staying within the framework. I'll first the correct solution and then present 2 potential solutions which should not be used, but appear to be very attractive solutions.

The correct solution is to register for EvtDeviceSelfManagedIoFlush or IPnpCallbackSelfManagedIo::OnSelfManagedIoFlush. It is called in the correct PnP IRP context for both situations. It is called before these PnP IRPs are completed, so there is no chance of the same device instance enumerating again while this callback is running (e.g. the original device instance is stuck in the surprise removed state and the device is plugged in again). It is synchronized against the other power up/down callbacks as well.

KMDF only: one potential solution is to set a WDM IRP preprocess routine for IRP_MN_SURPRISE_REMOVAL and IRP_MN_REMOVE_DEVICE. This would require you to track your own state and you would have to remove your device specific state before KMDF processed either of these IRPs. A lot of work without a lot of gain, plus you lost out on all the WDF contracts that are provided to you.

Another potential solution is to register for EvtDeviceSurpriseRemoval or implement IPnpCallback::OnSurpriseRemoval. This provides you with half the solution, but ignores the other half of the problem, graceful removal. You could track state in the surprise removal callback and then check this state in your device's cleanup routine/destructor. This does have the benefit of working within the framework and its contracts, but it still requires you to track state. Furthermore, the surprise removal callback is not synchronized with the other power up/down callbacks which is another detractor.