A Hole In My Head : WDM

How do I cancel an IRP that another thread may be completing at the same time?

doronh — Mon, 30 Jun 2008 09:16:00 GMT

Let's say that you allocated a PIRP and sent it down your device stack. You free the PIRP in the completion routine and then return STATUS_MORE_PROCESSING_REQUIRED. To make life more fun, you decide that you want to be able to cancel the sent IRP after you have sent it so you try to do it simple like this

typedef struct _DEVICE_EXTENSION {
    KSPIN_LOCK SentIrpLock;
    PIRP SentIrp;
} DEVICE_EXTENSION;

Sending thread:

KeAcquireSpinLock(&devext->SentIrpLock, ...);
devext->SentIrp = Irp;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

Canceling thread:

KeAcquireSpinLock(&devext->SentIrpLock, ...);
if (devext->AllocatedIrp != NULL) {
   IoCancelIrp(devext->SentIrp);
}
KeReleaseSpinLock(&devext->SentIrpLock, ...);

Completion routine:

PIRP irp;

KeAcquireSpinLock(&devext->SentIrpLock, ...);
irp = devext->SentIrp;

devext->SentIrp = NULL;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

IoFreeIrp(irp);

return STATUS_MORE_PROCESSING_REQUIRED;

And it then deadlocks ;). If the call to IoCancelIrp causes the IRP to be completed in the calling context (e.g. the one which has acquired the lock), the completion routine will run and try to acquire the lock (SentIrpLock) on the same thread which holds it.

So, life is not that simple and you have to do something more. The basic solution is that you need extra state to track who is touching the PIRP and who can free it. Walter Oney's book has a solution (IIRC, it is in the self initiated I/O section, but I do not have the book handy), but IMHO it is a bit complicated. KMDF has a solution to this problem which I like much more (imagine that ;)).

You need an extra LONG, calling it CompletionCount per PIRP that you want to be able to cancel.

You initialize CompletionCount to 1 before sending it down the stack and storing it in devext.
Whenever there is a thread that wants to cancel the PIRP, it tries to interlocked increment CompletionCount only if and only if the current CompletionCount value is > 0. For this you need to roll your own InterlockedIncrementWithFloor which is fortunately not that hard and I have already shown you how to do that, http://blogs.msdn.com/doronh/archive/2006/12/06/creating-your-own-interlockedxxx-operation.aspx.
After the canceling thread has called InterlockedIncrementWithFloor and IoCancelIrp, it calls InterlockedDecrement.
Whomever wants to complete the PIRP, like the completion routine, interlock decrements CompletionCount.

If the returned value from InterlockedDecrement is zero, the caller can complete the PIRP. If not, somebody else is trying to touch the PIRP and you must leave the PIRP alone. So here is the revised code:

typedef struct _DEVICE_EXTENSION {
    KSPIN_LOCK SentIrpLock;
    PIRP SentIrp;
    ULONG CompletionCount;
} DEVICE_EXTENSION;

Sending thread:

KeAcquireSpinLock(&devext->SentIrpLock, ...);
devext->CompletionCount = 1;
devext->SentIrp = Irp;

KeReleaseSpinLock(&devext->SentIrpLock, ...);

Canceling thread:

PIRP irp = NULL;
KeAcquireSpinLock(&devext->SentIrpLock, ...);
if (devext->AllocatedIrp != NULL && MyInterlockedIcrementeWithFloor(&devext->CompletionCount, 0) > 0) {
   irp = devext->SentIrp;
}
KeReleaseSpinLock(&devext->SentIrpLock, ...);

if (irp != NULL) {
    IoCancelIrp(irp);
    if (InterlockedDecrement(&devext->CompletionCount) == 0) {
        IoFreeIrp(irp);
    }
}

Completion routine:

PIRP irp = NULL;

KeAcquireSpinLock(&devext->SentIrpLock, ...);
irp = devext->SentIrp;devext->SentIrp = NULL;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

if (InterlockedDecrement(&devext->CompletionCount) == 0) {
    IoFreeIrp(irp);
}

return STATUS_MORE_PROCESSING_REQUIRED;

The beauty of this solution is that if you add more actors (let's say a timer for an async timeout, all you have to do is bump the CompletionCount to account for them to asynchronously rundown if you cannot cancel them.

Inconceivableable

doronh — Wed, 19 Mar 2008 02:55:00 GMT

I have no idea who created the name for PNP_DEVICE_NOT_DISABLEABLE, but I probably have the same reaction as you ... "seriously? that is what they named?" I mean come on, I think it could have at least been named PNP_DEVICE_CANNOT_BE_DISABLED. I am sure you can think of some better names too. If so, please leave a comment with your suggestions! While we had a chance to rectify this in KMDF in the WDF_DEVICE_STATE structure, we chose to keep the field name (NotDisableable) similar to the WDM name to avoid confusion.

For any readers who have not encountered this bit, it is a part of PNP_DEVICE_STATE. You set this bit in the IRP_MJ_PNP/IRP_MN_QUERY_PNP_DEVICE_STATE IRP after calling IoInvalidateDeviceState.

Once not disableable, forever not disableable

doronh — Tue, 18 Mar 2008 18:21:12 GMT

One interesting quirk about the PNP_DEVICE_NOT_DISABLEABLE state is that once it has been set and the PnP manager has processed it, the state is sticky. By sticky I mean that even if you attempt to clear this bit on a subsequent IRP_MN_QUERY_PNP_DEVICE_STATE IRP, the PnP manager ignores your changes to this state. This state remains stuck until any of the following occur

The machine is rebooted and the device is reenumerated
The device (or any device in its ancestry) is surprise removed
The device (or any device in its ancestry) is ejected

Why I should not be writing applications ;)

doronh — Fri, 09 Nov 2007 22:12:00 GMT

The first driver I owned when I started at Microsoft in 1997 was i8042prt.sys, the driver that controls your PS2 mouse and keyboard. I had the job of upgrading it from an NT4 legacy style driver to a PnP enabled Windows 2000 driver. Of course my dad asked "didn't keyboards and mice work before you got there? what are you doing over there anyways?" My manager at the time said all I had to do was get the mouse and keyboard working after resume from Sx and that would be it. I ended up owning the driver for 5+ years and not the advertised couple of months ;), but I learned a lot.

One of things I did while owning this driver was unify crash dump support for all builds. Previous to my owning the driver, Microsoft Far East Asia PSS added the ability to crash (aka crash dump) the machine for any given key and modifier combination (see I8xServiceCrashDump() for initialization of state and I8xProcessCrashDump() on processing the keys at runtime in 6000\src\input\pnpi8042 in the WDK for the gory details). This code was under #ifdefs and was only live for far east builds of the driver. This was a universally useful feature and I was asked to make it available for all builds. I decided to make it easy for folks to enable it as well. Configuration was not easy. First, you had to know the scan code for the trigger key (instead of the character itself). Second you had to describe the modifiers using a set of flags that were only documented in the driver's header. Third, you had to know where to put these registry values (which typically meant you also had to create a reg key as well). Yuck, not friendly.

To fix this I created one registry value (CrashOnCtrlScroll) that setup the default crash dump key sequence for you, right ctrl + scroll lock. This KB article describes how to set the key. That worked well ... except that many laptops did not have a right control key! I didn't want to create another key for crashing on left ctrl + scroll lock (or something as arbitrary but completely different) so I created a GUI application that let you type the key and select the modifiers. Not rocket science, but functional. Here is a snapshot

Issue #1: While functional, it's ugly

Issue #2: I named it cdsetup.exe (c[rash]d[ump]setup.exe). Logical name except that many cdrom setup applications are also called cdsetup.exe so there was a bit of confusion. To add to the confusion, I think my version of cdsetup.exe was a part of the Windows 200 resource kit. The one saving grace to this name is that it automatically requests elevation on Vista (probably because it has "setup" its name).

Issue #3: After recompiling the application yesterday to test it on Vista the red X, Cancel and OK buttons all stopped working which meant I could not dismiss the dialog box. Weird. I found an old build and it worked just fine so something I did during the recompile altered the behavior. To get the application to build and run I had to add the following 2 lines to the sources file

USE_MSVCRT=1
_NT_TARGET_VERSION=$(_NT_TARGET_VERSION_WIN2K)

Specifying the taret version has no effect on the runtime so it must have been the conversion to the MS VC runtime instead of the old C runtime that shipped with Windows. I decided to debug the application's DlgProc. It is rather simple:

LRESULT CALLBACK CrashDumpSetup::s_DlgProc(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
    CrashDumpSetup *cdi;

    if (message == WM_INITDIALOG) {
        cdi = new CrashDumpSetup(hDlg);

        if (!cdi)
            return FALSE;

        SetWindowLongPtr(hDlg, DWLP_USER, (ULONG_PTR) cdi);
        return (LRESULT) cdi->Initialize();
    }

    cdi = (CrashDumpSetup *) GetWindowLongPtr(hDlg, DWLP_USER);
    if (cdi) {
        return cdi->DialogProc(message, wParam, lParam);
    }

    return FALSE;
}

LRESULT CrashDumpSetup::DialogProc(UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message) {
    case WM_COMMAND:
        switch (LOWORD(wParam)) {
        case IDC_DUMP_DISABLE:
        case IDC_DUMP_DEFAULT:
        case IDC_DUMP_ALT:
            ToggleAltUI(LOWORD(wParam) == IDC_DUMP_ALT);
            break;

        case IDOK:
            if (Apply() == FALSE) {
                return FALSE;
            }
            __fallthrough;

        case IDCANCEL:
            delete this;
            EndDialog(m_hDlg, LOWORD(wParam));
            break;

        }
    }

    return FALSE;
}

When pressing the red X, Cancel or OK keys my WM_COMMAND processing code ran and called EndDialog() as I expected, but like I said before, the dialog didn't dismiss itself. I looked at the docs for EndDialog and saw that it returns a BOOL, so I rewrote EndDialog to capture the return value and it was returning FALSE! Ugh. Since I was single stepping through the function, !gle came to the rescue

0:000> !gle
LastErrorValue: (Win32) 0x578 (1400) - Invalid window handle.
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found.

An invalid window handle? m_hDlg worked before, maybe some component was overwriting it. So dumped the variable and then the hDlg from s_DlgProc,

0:000> dt this m_hDlg
Local var @ 0x1ff798 Type CrashDumpSetup*
0x00a01f88 
   +0x000 m_hDlg : 0x00a064d0 HWND__
0:000> .f+
01 001ff7cc 775af512 cdsetup!CrashDumpSetup::s_DlgProc+0x8f [d:\work\cdsetup\cdsetup.cpp @ 576]
0:000> dt hDlg
Local var @ 0x1ff7d4 Type HWND__*
0x004e05f0 
   +0x000 unused           : 49595140

Looking at the 2 values, it is obvious that m_hDlg was altered. But how? After staring at the code I wrote 7 years ago, I thought about what I changed (using the MS C runtime) and I immediate saw it. I was deleting the object and then touching freed memory. The old C runtime left the freed memory as it was before itw as freed so the old code had the bug the entire time, it was just that the new runtime caught made the mistake apparent immediately. The fix was simple, capture the value before deleting the object and then calling EndDialog (I guess I could have called EndDialog in the destructor but that made cleanup in the failure to initialize the object potentially problematic). After applying the fix, all of the buttons now worked as expected ;).

        case IDCANCEL:
            HWND h = m_hDlg;
            delete this;
            EndDialog(h m_hDlg, LOWORD(wParam));
            break;

How to share HW resources with another driver not in the same PnP hierarchy

doronh — Thu, 25 Oct 2007 01:56:23 GMT

First, I have to say that I don't agree with this design pattern at all. I think it leads to too many problems and complications that are not worth the pain. The only reason I am writing this entry is that I have seen so many people get this wrong or not account for some details and I want to make sure that if such a driver is installed, the stability of the system is not compromised. The scenario I am covering is not the case where you are enumerating child devices and want to share resources assigned to the parent amongst the children. Rather, this is sharing HW resources assigned to your device (device A) with another device (device B). Device B can be a legacy NT4 style device or another PnP device stack, it doesn't matter.

There are 3 aspects to the issue that you must solve:

Handing off the resources to device B. This means that device B will open a handle against device A and send an internal IOCTL (e.g. IOCTL_INTERNAL_GET_RESOURCES) which device A completes with the appropriate values. If the number of resources is fixed, this is easy. If the number of resources varies, this can get complicated.
Coordinating access to the resources between device A and device B. If both drivers want to touch the resources at the same time, you now have to synchronize between 2 components and that can get very complicated, especially if there are callbacks involved.
Only touching the resources when the device has power. This is easy for device A to do since it is the power policy owner for the device and sees pnp state changing and device power irps. This is not easy for device B to do though.

Aspects number 1 and 2 I will leave for you to solve. Number 3 is where things get interesting. There are scenarios where the device can lose power:

PnP graceful remove (query remove, remove)
PnP surprise removal
PnP stop
System power state change (e.g. a standby or hibernate)

The first 2 scenarios in green are easily solved. Device B registers for PnP notifications on the file handle that it opens and it will be told when a PnP remove or surprise removal occurs. If you are using KMDF, the WDFIOTARGET automatically does this for you. But there is a catch to the surprise removal notification...it is sent to device B after device A has already processed the PnP IRP. This means that there is a window where device B thinks it has valid resources, but in reality it does not. File handle notifications do not solve the last 2 scenarios though and the surprise removal case does not work well, so it would be nice if there was one solution for all 4 scenarios.

The solution is that device B must register 2 callbacks with device A, a power up and power down callback (they can be condensed to one callback with an additional passed in parameter indicating power state if you want). These callbacks should be set at the same time that the resources are acquired so that there is no window where the resources have been acquired, the device loses power and the callback has not yet been set notifying device B of the power state change. These callbacks can be passed from device B to device in any number of ways. Two traditional ways are

To pass the callbacks in a buffer in an internal IOCTL (you can overload IOCTL_INTERNAL_GET_RESOURCES if you want)
To send a IRP_MN_QUERY_INTERFACE with these callbacks as input values in the INTERFACE based structure as I describe in this post

Once the callbacks have been set, device A must call them at the right time. In a KMDF driver, it is a very simple implementation. You call device B's power up callback in EvtDeviceSelfManagedIoRestart and device B's power down routine in EvtDeviceSelfManagedIoSuspend. This covers all cases because KMDF treats PnP state changes as implicit power changes, powering down the device and using the same callbacks for all cases. This also covers the surprise removal window because device B is now notified at the beginning of device A's power down instead of after it. One final thing to consider is what device B thinks the initial power state of device A is in. Ideally, device B should assume device A is in low power and should not touch the retrieved resources until its power up callback has been invoked. This would mean that the initial call to device B's power up call might occur outside of EvtDeviceSelfManagedIoRestart, perhaps immediately after completing the IOCTL_INTERNAL_GET_RESOURCES request.

If you are writing a WDM driver, well, I will leave that implementation up to you ;).

Setting a security descriptor on a legacy device object

doronh — Wed, 17 Oct 2007 02:45:00 GMT

Setting the security descriptor allows you to control who can open a handle to the device object. Typically you can call IoCreateDeviceSecure to create the device object and have the correct DACL from the start. One issue with IoCreateDeviceSecure is that the SDDL string is limited to what it can describe, primarily you can only specify predefined SIDs. If you need a SID that is not predefined you must build the security descriptor by hand and the code to do it is not very easty to implement (let alone figure out which DDIs you need to call in the first place).

Well, Paul Sliwowicz kindly donated this code snippet which will set a security descriptor on a legacy device object. It does require that you are able to open a handle to the device object first, so it might be a good idea to initially create the device with a very restrictive SDDL string that limits opens to the system only and then relax it to your needs with the following code which works in either a WDM or KMDF driver. This can also work on device objects that are created by other drivers, but you have a built in race between when the device is created and when you set the security descriptor that you must also account for.

NTSTATUS
SetDeviceDacl(
    PDEVICE_OBJECT DeviceObject
    )
{
    SECURITY_DESCRIPTOR sd = { 0 };
    ULONG aclSize = 0;
    PACL pAcl = NULL;

    NTSTATUS status;

    status = ObOpenObjectByPointer(DeviceObject,
                                   OBJ_KERNEL_HANDLE,
                                   NULL,
                                   WRITE_DAC,
                                   0,
                                   KernelMode,
                                   &fileHandle);

    if (!NT_SUCCESS(status)) {
        goto Exit;
    } 

    //
    // Calculate how big our ACL needs to be to support one ACE (in
    // this case we'll use a predefined Se export for local system
    // as the SID)
    //
    aclSize = sizeof(ACL);
    aclSize += RtlLengthSid(SeExports->SeLocalSystemSid);
    aclSize += RtlLengthSid(SeExports->SeAliasAdminsSid);
    aclSize += RtlLengthSid(SeExports->SeAliasUsersSid);
   
    //
    // Room for 3 ACEs (one for each SID above); note we don't
    // include the end of the structure as we've accounted for that
    // length in the SID lengths already.
    //
    aclSize += 3 * FIELD_OFFSET(ACCESS_ALLOWED_ACE, SidStart);

    pAcl = (PACL) ExAllocatePoolWithTag(PagedPool, aclSize, TAG);

    if (pAcl == NULL) {
        status = STATUS_INSUFFICIENT_RESOURCES;
        goto Exit;
    }

    status = RtlCreateAcl(pAcl, aclSize, ACL_REVISION);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    status = RtlAddAccessAllowedAce(pAcl,
                                    ACL_REVISION,
                                    GENERIC_READ | GENERIC_WRITE | DELETE,
                                    SeExports->SeLocalSystemSid);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    status = RtlAddAccessAllowedAce(pAcl,
                                    ACL_REVISION,
                                    GENERIC_READ | GENERIC_WRITE | DELETE,
                                    SeExports->SeAliasAdminsSid );
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    status = RtlAddAccessAllowedAce(pAcl,
                                    ACL_REVISION,
                                    GENERIC_READ,
                                    SeExports->SeAliasUsersSid );

    if (!NT_SUCCESS(status)) {
        goto Exit;
    }
 
    //
    // Create a security descriptor
    //
    status = RtlCreateSecurityDescriptor(&sd, SECURITY_DESCRIPTOR_REVISION);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    //
    // Associate the above pAcl with the security descriptor
    //
    status = RtlSetDaclSecurityDescriptor(&sd, TRUE, pAcl, FALSE);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

    //
    // Set security on the object
    //
    status = ZwSetSecurityObject(fileHandle, DACL_SECURITY_INFORMATION, &sd);
    if (!NT_SUCCESS(status)) {
        goto Exit;
    }

Exit:
    ZwClose(fileHandle);
    fileHandle = NULL;

    if (pAcl != NULL) {
        ExFreePool(pAcl);
        pAcl = NULL;
    }

    return status;
}

Fast Resume and how if affects your driver

doronh — Mon, 15 Oct 2007 20:33:02 GMT

Fast resume, which was introduced in Windows XP, is often mentioned when implementing power support in your WDM driver. But what does "fast resume" mean and when implementing fast resume, what side effects occur in your driver? I'll to answer both of these questions as well as the reasoning behind this feature. When I started to write this entry, I tried to look up fast resume in the WDK and found absolutely nothing! I was shocked since this is a very common term (at least internally at Microsoft); after speaking with the power team, there is a white paper that describes fast resume but you would never know it from the title "Measuring System Resume Performance on Windows Vista."

First, let's go over how to handle IRP_MN_SET_POWER/SystemPowerState for PowerSystemWorking (aka S0) IRP. You are supposed to pend the S0 irp and subsequently complete the S0 irp after you have requested a IRP_MN_SET_POWER/DevicePowerState PowerDeviceD0 IRP and it has been processed by the entire stack (which means you must register a Po completion routine which is where you complete the S0 IRP). In short, this means that the S0 IRP is pended in your driver as long as it takes for your device to power up. The definition of fast resume is simply that instead of pending the S0 IRP until the D0 IRP completes, you do not pend the S0 IRP and let it complete before the D0 IRP has been processed in your stack.

Sounds simple, right? Well, the devil is in the details. Here are a couple of issues with fast resume that I have encountered:

Device power IRPs are not synchronized with PnP state changing IRPs as I wrote about earlier this year. This has a direct side effect on a fast resume implementation. Previous to fast resume, your D0 IRP could never be concurrently processed while a PnP state changing IRP (like a surprise remove) was being processed. This is due to the fact that system power IRPs are synchronized against PnP state changing IRPs and by pending the S0 IRP, you guard your device power up path from running concurrently with a PnP state change. This made life simple for both PnP and power IRP handling in your driver, but with fast resume this is no longer true. Now the state changing PnP can be running concurrently with your power up path and you must make sure that the PnP dispatch routine does not assume the device's power state nor does it free resources that the power dispatch path may be using.
If this is a bus driver, the "free" power state synchronization between your parent device and your enumerated children goes away. WDM dictates that for a child device to be powered on, the parent must be powered on. By default you can get the right behavior, only after the parent device's S0 IRP has completed back to the power manager will the power manager send S0 IRPs to your child devices. This means that if you complete the S0 IRP before processing the D0 irp in the parent device, a new race condition occurs. The children could receive their S0 IRP and request their D0 IRP before the parent has processed (or even requested!) its own D0 IRP. This can be handled in the bus driver by pending each child's D0 IRP until the parent has completed its D0 IRP, but it can get complicated quickly (especially in the face of failure to power up).

So given these issues, why would you even want to implement fast resume? The answer lies in the history of fast resume, or rather, Windows 2000 which did not have this feature. Resume times on Windows 2000 laptops was not great. There were a causes for the poor resume time, but a large contributor was device resume time. Since the OS serialized S0 IRPs to all devices in the PnP tree, the time for each device to resume added sequentially to the system's resume time. The OS serialization of the S0 IRPs could not change for Windows XP, so the problem was attacked at the other end...each driver would complete the S0 IRP as fast as it could so the OS could resume quickly and then asynchronously power up the device. This way each device could power up in parallel and the total time to power up was not sequential (nor was it only the longest of any device to power up since there is still ordering between parent and child devices) and resume time could be dramatically slower.

Given the perceivable value you are giving to the user by implementing fast resume, I recommend that you implement it in your (non bus) function driver and handle the new races between PnP state changing IRPs and power IRPs (which if you implemented idle during S0, must also be accounted for already. Considering the complicated aspects of asynchronously managing a parent's power state and a child's power state, I would not recommend that you implement fast resume in a bus driver. This follows the recommendations in the aforementioned white paper.

Obligatory KMDF plug: KMDF automatically implements fast resume for you and handles both issues behind the scenes. You will not process a pnp state changing irp while power up (or down) and if you are a bus driver, you will get an implementation which maintains the parent/child relationship with respect to power state.

Vista IO manager changes in handling FILE_DEVICE_SECURE_OPEN

doronh — Fri, 05 Oct 2007 18:00:00 GMT

After having the IO manager developer review my last 2 posts, he pointed out to me that the IO manager handling of FILE_DEVICE_SECURE_OPEN (FDSO) has changed slightly in Vista. News to me and probably news to all of you as well. The change involves the case where there is a file system mounted on a device object, or in other words, a device object which implements a namespace with no file system attached is not affected by the change. The old behavior (as I described earlier) skipped the security descriptor check on the device object if there was a path remaining after parsing for the device object name. In new Vista behavior, if the device object has the FDSO flag set, has a mounted file system, and its descriptor is not the OS default descriptor, the device’s descriptor will now be evaluated, whereas previous to Vista, it would have been skipped.

Why make this change? It was driven by a usage scenario for removable devices. The driver which enumerates the removable media volume can now alter the security descriptor and apply volume wide security settings regardless of the actual file system mounted on the drive. For instance, the volume device could be assigned a security descriptor which only allowed administrators to write to the volume or make the entire volume read only.

Making sure the IO manager evaluates the security of your device

doronh — Thu, 04 Oct 2007 19:00:11 GMT

Last time I wrote about how the IO manager handles the creation of file handles and pointed out a potential security hole. If there is a namespace (or path) after your device's name in the path passed to CreateFile, the IO managed does not evaluate the security settings set on your device and relies on your driver to do the evaluation. For instance, if you create a device object and specify that only administrators have access to it and you do not validate access during IRP_MJ_CREATE processing, any user regardless of privilege level may open a handle to your device if they specify any namespace (i.e. "\Device\FooDevice\Blah") when opening a handle.

A gut shot reaction to this behavior is that is a very large tax for every device driver to pay and the NT kernel developers agreed. To rectify the situation a new characteristics flag, FILE_DEVICE_SECURE_OPEN, was added. This flag tells the IO manager that when a device namespace is present during handle creation that the IO manager should no longer skip the security check and should still evaluate if the caller has sufficient rights to open the device. By setting this flag, the caller's access to the object is always evaluated at the IO manager level and the driver does nothing more then set this flag.

You might say to yourself, "well that is backwards, shouldn't the driver writer get this behavior by default and choose to opt out of it instead of knowing that the device must opt in to get the most secure behavior?" and I would agree with you. The reason the most secure behavior was not made default was for backwards compatibility. This hole was not discovered until after NT 3.1 shipped and if the IO manager behavior was flipped, already shipping drivers would stop functioning.

So when should you set this flag? In my opinion you should always set this flag in a WDM driver! The one exception is if you are creating a device upon which a file system will be mounted (if you set the flag in this case, the file system will not be able to evaluate the security of the namespace string). In the opinion of the WDF team, this flag is so important that we tried to guarantee that this flag is set for all WDF device objects, even if you tried to clear it by calling WdfDeviceInitSetCharacteristics or WdfDeviceSetCharacteristics. If you are writing a KMDF driver which will have a file system mounted on top of it, this is how you would clear the flag after creating the WDFDEVICE

WdfDeviceWdmGetDeviceObject(device)->Characteristics &= ~FILE_DEVICE_SECURE_OPEN;

This KB article also discusses the side effects of the FILE_DEVICE_SECURE_OPEN flag on inbox drivers in previous OS releases.

Devices and namespaces (or how the IO manager handles file creation)

doronh — Wed, 03 Oct 2007 21:55:00 GMT

Ever wonder how the creation of a handle works? It doesn't matter type of resource the handle you are opening is backed by (a COM port, a file, a network share, a custom piece of hardware, etc), it all goes through CreateFile (which should be a little obvious since the only way to open an type of handle is by calling it (with the exceptions like this function or this function)). The IO manager creates a file handle for any resource type the same way. In theory, the way the IO manager handles file creation should be transparent to most drivers, but in reality a driver writer should understand how it works so that you can configure your device properly and implement IRP_MJ_CREATE properly.

The key concept to understand here is that the IO manager does not have hard coded knowledge of each of these resources, it is completely generic. This means that the IO manager does some parsing/path evaluation and then relies on the driver to perform some additional optional path evaluation. Very broadly, here is what the IO manager does

It tries to find the device name in the lpFileName parameter. More than likely the path will contain a symbolic link to the device name instead of the device name itself. This means that the IO manager will evaluate the symbolic link to find the target string and then restart the parse.
Once the device name is found, it used to find the device object (i.e. a PDEVICE_OBJECT).
If there is no remaining path name after the device name/symbolic link, use the device object's security descriptor to evaluate if the caller has sufficient rights to open the device object. If there is a remaining path, no security check occurs (*)!
Create a file object, set the FileName to the remaining path and send a IRP_MJ_CREATE IRP to the device

That's it. The FileName is the important part here. This is how the driver provides additional path evaluation. It is up to the driver to evaluate the FileName properly and apply its own security checks based on the string. In WDM terms, the FileName value is a part of the device's namespace. Each device defines its own namespace, the WDK topic "Controlling Device Namespace Access" explains device namespaces and does a decent job, but without the context of how the IO manager works, it is a bit indecipherable.

Let's look at a few sample file names and see how they would parse and the consequences of each example

lpFileName	Device name	FileName	Security access check in IO manager
"COM1"	"\Device\Serial0"	NULL	Yes
"COM1\Foo"	"\Device\Serial0"	"\Foo"	No
"C:\Windows\win.ini"	"\Device\HarddiskVolume1"	"\Windows\win.ini"	No
"C:"	"\Device\HarddiskVolume1"	NULL	Yes

The "COM1" case is straightforward. The COM port is being opened, the IO manager evaluates whether the caller has access and it is done, but why "COM1\Foo" ? I want to illustrate that you can append a path name to any symbolic link that you want and that it might result in failure, different behavior or be completely ignored (it will nearly always be ignored). When opening "COM1\Foo", the IO manager will pass "\Foo" to the serial driver and it will be subsequently ignored. The path name can have meaning though as I wrote about earlier if you want to distinguish which device interface is being opened.

"C:\Windows\win.ini" is what many consider to the normal usage for CreateFile, a file is being opened. "C:" is evaluated to particular volume and then the create is sent to the file system mounted on this volume. The file system then parses "\Windows\win.ini" to determine if the caller has sufficient rights to open the file. But what about the last example, "C:"? This path name (without the root directory "\") is asking for raw access to the volume itself (allowing you to read and write sectors directly).

I used these last 2 examples to show why the IO manager would skip the security check (step #3) if there is any remaining path. Access to "C:" itself requires administrative privileges while access to a file on C: may not have any security at all. Both the volume and file system must participate in evaluating whether the current caller has sufficient rights to open a particular file handle depending on its path. But this also brings up a very nasty security hole. I can append any path to a non file system device name, cause the IO manager to skip the security check, and if the driver does not do its own security check (which it probably doesn't since it is relying on the IO manager to do it), gain access to the device regardless of the device's security descriptor. This is a very valid concern and I will cover how to handle this in my next post.

(*) Actually, it is not always skipped, but that is the next topic ;).

Creating symbolic links for an anonymous FDO or filter

doronh — Thu, 03 May 2007 20:21:00 GMT

As I wrote about previously, naming your FDO has some side effects that you may not want to incur. But what if you want to give your device a fixed symbolic link name (by calling IoCreateSymbolicLink), such as \DosDevices\Foo1, in addition to your device interface GUID? Does an unnamed FDO (or filter exclude the possibility of creating the fixed symbolic link? The short answer is that it is still possible to create a fixed symbolic link for any unnamed device in a PnP stack, you just have a little more work ahead of you if you are writing a WDM driver (KMDF does implements the recommendation in this post for you). First, let's take a look at how you would create a fixed symbolic link name if your had a named device

DECLARE_CONST_UNICODE_STRING(deviceName, L"\\Device\\Foo1");
DECLARE_CONST_UNICODE_STRING(symbolicLinkName, L"\\DosDevices\Foo1");

IoCreateDevice(..., &deviceName, ...);
IoCreateSymbolicLinkName(&symbolicLinkName, &deviceName);

The key takeaway here is that the symbolic link name points to some device's name. If you had an unnamed device object, you have no device name to pass to IoCreateSymbolicLink. The saving grace here is that at least one device in the stack is guaranteed to have a name, the PDO. All you need to do is to get the name of the PDO and create a symbolic link against the PDO's name. IoGetDeviceProperty(DevicePropertyPhysicalDeviceObjectName) will give us the PDO name (as would ObQueryNameString, but only until recently this was only documented in the IFS kit). To retrieve the name you must call IoGetDeviceProperty twice, first to get the length of the name buffer and second to retrieve the name. Here is how you would retrieve the PDO name and create the symbolic link against it (feel free to reuse the code in your own driver).

PDEVICE_OBJECT pPdo;
ULONG length;
NTSTATUS status;
PWSTR pName;
UNICODE_STRING pdoName;

pPdo = DeviceExtension->PhysicalDeviceObject;
length = 0;

status = IoGetDeviceProperty(pPdo,
                             DevicePropertyPhysicalDeviceObjectName,
                             0,
                             NULL,
                             &length);

if (status != STATUS_BUFFER_TOO_SMALL && !NT_SUCCESS(status)) {
    return status;
}
else if (length > USHORT_MAX) {
    return STATUS_INTEGER_OVERFLOW;
}
else if (length == 0) {
    //
    // We can get zero back if the PDO is being deleted.  This can only happen if we are 
    // creating the symbolic link outside of AddDevice() or a PnP start.
    //
    return STATUS_INVALID_DEVICE_STATE;
}

pName = (PWSTR) ExAllocatePoolWithTag(PagedPool, length, [tag]);
if (pName == NULL) {
    return STATUS_INSUFFICIENT_RESOURCES;
}

status = IoGetDeviceProperty(pPdo,
                             DevicePropertyPhysicalDeviceObjectName,
                             length,
                             pName,
                             &length);

if (!NT_SUCCESS(status)) {
    ExFreePool(pName);
    pName = NULL;
    return status;
}

pdoName.Buffer = pName;
pdoName.Length = (USHORT) length - sizeof(UNICODE_NULL);
pdoName.MaximumLength = (USHORT) length;

status = IoCreateSymbolicLink([SymbolicLinkName], &pdoName)

if (NT_SUCCESS(status)) {
    // the type of DeviceExtension->PdoName is UNICODE_STRING
    RtlCopyMemory(&DeviceExtension->PdoName, &pdoName, sizeof(pdoName));

    // now that we stored the buffer in our extension, remove all references to the buffer
    // so that we don't accidentally free it while still pointing to it in the extension
    pName = NULL;
    RtlZeroMemory(&pdoName, sizeof(pdoName));
}
else {
    ExFreePool(pName);
    pName = NULL;
}

return status;

Remember that you must also destroy the symbolic on a surprise remove or graceful remove which is why I stored the PDO name in the device extension. Otherwise, you would have to query, allocate and requery during remove processing, which just makes your driver more complicated (and if the alloc fails, there is no way for you to destroy the symbolic link name, so by storing it upon creation, we can guarantee we can destroy it later). Also, remember to free DeviceExtension->PdoName.Buffer during remove processing when you are freeing resources! Again, KMDF does all of this for you (including the automatic destroying of the link upon deletion), all you have to do is create an unnamed WDFDEVICE and then call WdfDeviceCreateSymbolicLink and you are done.

If you think about this a little more deeply, this is exactly how device interfaces work. When you create a device interface, the OS creates a symbolic link name on your behalf and has it point to the PDO's name. The difference between your symbolic link name and the OS generated name based on the device interface GUID is that the device interface has built in mechanisms to notify an application of arrival or departure and has a name which can be queried for at runtime. The fixed symbolic link has no such notifications and there is no way to tell how many of these links exist.

Special thanks to Mark Roddy who intially suggested this technique during KMDF beta testing.

Having two names is not necessarily better than one

doronh — Wed, 18 Apr 2007 23:20:00 GMT

Every physical device object (PDO) must have a name. Furthermore, if you read the entire MSDN page, you see that any device attached to the PDO must not have a name. Why does such a rule exist? To answer this question, let's explore what happens if more than one device in the stack has a name. Furthermore, each named device has a symbolic link created against it so that a user mode application can easily create a file handle, as shown by this diagram

    +----------------+
    |      FDO       |  
    | \Device\Name2  | <-- Symbolic link \DosDevices\Name2
    +----------------+
             |
    +----------------+
    |      PDO       |
    | \Device\Name1  | <-- Symbolic link created by IoRegisterDeviceInterface 
    +----------------+

To process the CreateFile request, the I/O manager resolves the symbolic name to the underlying device name and sends the IRP_MJ_CREATE to the top of the stack. If the create succeeds, all I/O flows through the top of the stack and it is up to the stack to forward or process I/O requests as appropriate. This means that no matter which name (\DosDevices\Name2 or the device interface name returned by SetupAPI) you used to open the device stack, the I/O will always be sent to the top. It is not as if you can open the stack by the PDO's name and send I/O to the PDO directly without the FDO seeing the I/O requests. So, why are having additional names a problem in this scenario? The answer is that this is not the reason that this rule exists.

So, again, why does this rule exist? The rule exists because of how the I/O manager uses the device object derived from the device name/symbolic when creating a file handle. Each device has its own set of properties (exclusive, secure open, security descriptor) that are specified when the device is created by calling IoCreateDevice or IoCreateDeviceSecure and uses the properties associated with only this device object and not any other device object in the stack. This means that if the PDO has a security descriptor that allows access only to user A and excludes B, while the FDO has a security descriptor that allows access only to user B, user B can open the stack by using the FDO's name and circumvent the PDO's security settings. In other words, you have a potential security hole! Since each name is evaluated on its own, every named device in the stack can be opened with their own set of rights and properties.

This leads to 2 more questions.

How do I specify the properties (exclusivity, security, etc) I want for the stack if I can't have a name and some other driver is creating the PDO?

This is a legitimate problem. If your driver has no control over how the PDO is created, how can you affect its settings? Or to turn it around, how is the bus driver going to know what settings are appropriate for the PDO it is creating? Most bus drivers do not know what specific driver will be loaded on the PDO it is enumerating. To fix this, the PnP manager will adjust the settings of each device object in the stack to a specific value. These values are written to the device node during installation by the INF. You use the hardware AddReg directive to tell the PnP manager what to set for the stack. Using the INF, you can set stack wide settings for device characteristics, device type, security, and exclusivity.

Why do some stacks have named device objects in their stacks that are not the PDO?

Some stacks have named FDOs or filters because of legacy reasons that predate PnP. For instance, the storage stacks name their device objects because other system components expect to find these names. In these stacks which have multiple device objects with a name, you can still use the INF directives to create stack wide settings.

Problems with not having a current IRP stack location, part 2

doronh — Mon, 09 Apr 2007 19:04:00 GMT

This problem falls into the category being hidden by a macro that does not indicate in its name what it touches. If you call IoMarkIrpPending on an IRP that you allocated in your driver, chances are that you are corrupting memory. First, let's look at the implementation of this function (from wdm.h):

#define IoMarkIrpPending( Irp ) ( \
    IoGetCurrentIrpStackLocation( (Irp) )->Control |= SL_PENDING_RETURNED )

This function retrieves the current stack location and sets a flag in one of the fields (IO_STACK_LOCATION::Control). Since the current IRP stack location is not valid, and by not valid I mean that the pointer value points to undefined memory, not that it is NULL, so the call corrupts memory (a nefarious single bit flip) somewhere in the system. It so happens that based on how IRPs are currently allocated today, you will corrupt memory immediately after the IRP allocation making this a little bit easier to diagnose later.

How would a mistaken call to IoMarkIrpPending even get into your driver? Well, most code is copied from somewhere else. A lot of completion routines have the following code in them. These completion routines were written to manipulate an I/O manager presented IRP (which works because there is a current stack location), but when copied over they have a time bomb waiting to go off in them. Here is an example completion routine which is copied:

NTSTATUS CompletionRoutine(PDEVICE_OBJECT DeviceObject, PIRP Irp, PVOID Context)
{
    ...
    if (Irp->PendingReturned) {
         IoMarkIrpPending(Irp);
    }
    ...
    IoFreeIrp(Irp);

    //
    // Must return STATUS_MORE_PROCESSING_REQUIRED because we cannot let the IRP complete
    // back to the I/O manager
    //
    return STATUS_MORE_PROCESSING_REQUIRED;
}

What happens if you have an I/O completion routine that handles both types (driver created, I/O manager presented) IRPs? In the I/O manager presented IRP case you need to propagate the pending state by calling IoMarkIrpPending, while in the driver create case you obviously should not propagate the state. KMDF must do this since it sets the same completion routine for all requests sent to a WDFIOTARGET. Here is how KMDF manages this functionality. If Irp->PendingReturned is TRUE and the current stack location count is less than or equal to the total stack count (in the driver allocated case, CurrentLocation will be > StackCount), propagate the pending into the current stack.

VOID
PropagatePendingReturned(
    PIRP Irp
    )
{
   if (Irp->PendingReturned && Irp->CurrentLocation <= Irp->StackCount) {
       IoMarkIrpPending(Irp);
   }
}

BTW, unaccredited part 1 is why you don't have a DeviceObject in your I/O completion routine

Your completion routine context can be anything you want

doronh — Sat, 07 Apr 2007 00:25:00 GMT

It sounds obvious, but sometimes it needs to be stated. For instance, let's say that you are allocating your own IRP, your context contains I/O related data (like a URB) and you encounter the issue where the DeviceObject passed to your I/O completion routine is NULL. Adding another stack location is one solution I wrote about, but it is a little complicated and has a certain black magic feeling to it, besides it does not work with any IoBuildXxx API calls. Since you are allocating memory for your context, just allocate a little bit more. Instead of allocating just the URB

PURB urb = ExAllocatePoolWithTag(NonPagedPool, sizeof(URB), ...);
PIRP irp = IoAllocateIrp(...);
...
// format the URB
IoSetCompletionRoutine(Irp, CompletionRoutine, urb, TRUE, TRUE, TRUE);
...
IoCallDriver(..., Irp);
...

NTSTATUS CompletionRoutine(PDEVIC_OBJECT Device, PIRP Irp, PURB Urb)
{
     // Do not touch Device, it is NULL!
     // Process Urb and Irp results
     ExFreePool(Urb);
     IoFreeIrp(Irp);

     return STATUS_MORE_PROCESSING_REQUIRED;
}

allocate a structure to contain the URB and your DeviceObject pointer (or WDFDEVICE or anything else)

typedef struct _COMPLETION_DATA {
    PDEVICE_OBJECT DeviceObject;
    URB Urb;
} COMPLETION_DATA, *PCOMPLETION_DATA;

PCOMPLETION_DATA data = ExAllocatePoolWithTag(NonPagedPool, sizeof(COMPLETION_DATA), ...);
PIRP irp = IoAllocateIrp(...);
...
// format data->Urb
data->DeviceObject = DeviceObject;
IoSetCompletionRoutine(Irp, CompletionRoutine, data, TRUE, TRUE, TRUE);
...
IoCallDriver(..., Irp);
...

NTSTATUS CompletionRoutine(PDEVIC_OBJECT Device, PIRP Irp, PCOMPLETION_DATA Data)
{
     // Do not touch Device, it is NULL, but Data->Device is valid!
     
     // Process Data->Urb and Irp results
     ExFreePool(Data);
     IoFreeIrp(Irp);

     return STATUS_MORE_PROCESSING_REQUIRED;
}

Like I said, it is an obvious thing to do :), but when you are in the depths of debugging a bugcheck, sometimes the obvious solution is the last thing you think of.

Filtering HID collections and setting I/O flags

doronh — Tue, 27 Mar 2007 03:19:00 GMT

Over the past 3 years or so, I have been casually referring to KMDF as the ultimate driver compat layer. Just like Windows has an application compatibility layer which shields applications from OS changes, KMDF provides the same type of compatibility across device stacks. For the most part each stack implements WDM as per the documentation and there is nothing special that KMDF must compensate for when inserted into the stack. HID is not one of these stacks ;). First, let me describe the HID stack.

+---------+
| Raw PDO |
+---------+
     |
     |    +----------------+
     +----| HID miniport + |
          | HIDclass       | 
          +----------------+
                   |
          +----------------+
          |       PDO      |
          +----------------+

HIDClass (the port driver) will enumerate a raw PDO for each top level collection (TLC) in a device's HID descriptor (in this picture there is only one raw PDO, but there could be many). A raw PDO is special in that it does not need an additional INF to be started, all of the functionality that is usually found in a stack's FDO is in the raw PDO. This includes the DeviceObject Flags which specify the buffering type for reads and writes on the device (more on this later). A raw PDO can have a FDO and filters layered on top of it, this is how the HID keyboard and mouse stacks work. A TLC which is neither a keyboard nor a mouse can have a filter installed on it as well.

Setting the flags is where HID does not play by the rules. Typically a bus driver will specify the flags (DO_DIRECT_IO or DO_BUFFERED_IO) which dictate the buffering in read and write IRPs when the device object is created. This would allow an FDO or filter to determine the I/O type during the FDO or filter's AddDevice routine and copy them into its own device object (since the I/O manager looks at the top of the stack to determine buffering for reads and writes). HIDClass on the other hand sets one of the flag (DO_DIRECT_IO) during IRP_MN_START_DEVICE! This means that in a WDM driver filtering on top of a raw HID PDO, you must special case this behavior and set these flags in your device object during IRP_MN_START_DEVICE processing. KMDF takes care of this for you, so your HID filter code is no different than a filter for any other device stack (this is where the driver compat layer is).

What makes the situation worse is that there is a HID filter sample, firefly, which does not change the I/O type in its device object's Flags, meaning it is now a generic HID filter sample, rather only a HID mouse sample. But how does firefly even work in the mouse stack? Well, the upper edge of the mouse stack (mouclass.sys) always sets DO_BUFFERED_IO and read IRPs are never sent down the stack, so HIDClass would never see a read IRP with the wrong buffer type. This means that the propagation of the buffering flags is not needed.

So how is it that this bug existed HIDClass for so long without being discovered? Remember that HIDClass enumerates its PDOs as raw and 99% of the time, nobody loads on top of them, so there was no other WDM component to interact with this behavior of setting the Flags field at the wrong time.