Trouble Ahead- Trouble Behind : KMDF

New WDF debugging videos on WHDC

BobKjelgaard — Tue, 22 Sep 2009 16:41:32 GMT

Several for KMDF here, and several more for UMDF here.

Thus endeth today’s PR blitzen. Have fun!

The Case of the Unexpected Overload

BobKjelgaard — Mon, 03 Aug 2009 17:31:44 GMT

So are you using C++ in a KMDF driver and scratching your head over some unexpectedly unresolved references?

I recently engaged in a somewhat prolonged conversation with Doron and a dev I won’t name [since I didn’t ask for permission] about C++ in KMDF driver. This dev (a very good dev, by the way) had hit a stumbling point converting a driver from C to C++ and was wondering if we had ever tried this in QA.

Yes, I know that means he/she doesn’t read this blog, since I’ve mentioned doing just that on more than one occasion. But that’s OK, not many people do- and since I’ve got a tendency to write about whatever strikes my fancy at any moment, and I’m not the most dazzlingly brilliant human on the face of the planet, that’s not much of a surprise to me. You are of course free to believe what you like about that statement.

But in an attempt to be useful today- the problem was one of those that I guess I’ve hit so many times I don’t give it much thought. But it gave our dev a bit of trouble, and not much gives this person that kind of trouble. So perhaps a word to the wise will give the three or four of you still reading this a leg up the next time you use C++ in a KMDF driver.

The Lesson for Today Is…

It was just a simple conversion of a driver from C to C++ (to gauge my familiarity with the process, I did that with a half dozen or more drivers last week, and it was far from the only thing I did that week- in fact the big job was actually that I also made them W4 and PFD clean along with quite a few more drivers I’d already converted to C++- almost two dozen drivers total), but it ran afoul of this little bit of goodness:

typedef
__drv_functionClass(EVT_WDF_OBJECT_CONTEXT_CLEANUP)
__drv_sameIRQL
__drv_maxIRQL(DISPATCH_LEVEL)
VOID
EVT_WDF_OBJECT_CONTEXT_CLEANUP(
    __in
    WDFOBJECT Object
    );

typedef EVT_WDF_OBJECT_CONTEXT_CLEANUP *PFN_WDF_OBJECT_CONTEXT_CLEANUP;

That’s the “role typedef” for any WDF object’s context cleanup routine. The SDV tool can use role typedefs to do a better job of statically verifying your driver code, so some special cleanup typedefs are defined to make it a better tool, and you can see them in this (by the way, everything I’m posting is from the WDK- no deep dark secrets here).

typedef EVT_WDF_OBJECT_CONTEXT_CLEANUP EVT_WDF_DEVICE_CONTEXT_CLEANUP;
typedef EVT_WDF_OBJECT_CONTEXT_DESTROY EVT_WDF_DEVICE_CONTEXT_DESTROY;
typedef EVT_WDF_OBJECT_CONTEXT_CLEANUP EVT_WDF_IO_QUEUE_CONTEXT_CLEANUP_CALLBACK;
typedef EVT_WDF_OBJECT_CONTEXT_DESTROY EVT_WDF_IO_QUEUE_CONTEXT_DESTROY_CALLBACK;
typedef EVT_WDF_OBJECT_CONTEXT_CLEANUP EVT_WDF_FILE_CONTEXT_CLEANUP_CALLBACK;
typedef EVT_WDF_OBJECT_CONTEXT_DESTROY EVT_WDF_FILE_CONTEXT_DESTROY_CALLBACK;

So, what the original driver, being a good sample [because SDV suitability also matters], had was something like this (not exactly like this- nobody with my twisted sense of humor writes sample code, because humor never works globally- someone always takes offense- I’m just not inclined to care about that anymore, because I’m a callous old bastard with less than a dozen years to retirement age and not much “career” left to lose):

// In the header file:

EVT_WDF_DEVICE_CONTEXT_CLEANUP HideTheBodies;

// In the implementation
VOID
HideTheBodies(
    __in WDFDEVICE MortTheMortuaryTechnicianIsOnTheJob
    )
{
...

Of course, in C, that sort of works- but in C++, that says you have two overloaded versions of HideTheBodies- one takes a WDFOBJECT as input, and isn’t defined, because nobody wrote it that way. The one that is defined takes a WDFDEVICE as input. More annoying, of course, is that from the typedef name, you’d expect it would take a WDFDEVICE as input, so it isn’t an error that’s glaringly obvious.

As is- the code will compile fine, and PFD will be happy, too [they’re not the same function, remember]. But when you link it, you get an unresolved reference for HideTheBodies- decorated to indicate it has a WDFOBJECT parameter. Now, unless you really look closely at such decorations (something my fine history of mistyping the simplest of code has taught me to do), you’re looking at a mystery. It took my colleague digging through preprocessor output and object files and such to finally hit upon the error- a reasonably annoying circumstance!

Another thing you can do (and this is what I usually do) is tie the callbacks to the context itself (with the context a class) as static member functions, as I described earlier (effectively you do the cleanup by calling the context’s “delete” operator). PFD currently doesn’t match the roletype if you do this, so I suppress those warnings (and there is no SDV for C++)- I think we’re trying to get that fixed… But the compiler will catch the parameter type mismatch- a bit earlier than the linker.

But I’ve hosed this both ways- so, until something on the subject gets posted by a recognized authority, this post can at least be found by the diligent searcher…

2008 DDC slides now available to all!

BobKjelgaard — Wed, 22 Jul 2009 14:24:33 GMT

Got an email late last week stating that the material I and others presented at the 2008 Driver Developers Conference (which Dieter and James want renamed to Driver Development and Test Conference- more power to that idea!) is now available here.

The sessions I participated in are a description of several WDF test technologies and some hand waving about troubleshooting WDF installation problems.

If you are a tester, you might also want to take a look at James and Dieter’s presentation on concurrency testing, and who wouldn’t want to know about the new Driver Verifier features for Win7 (ably presented by Daniel Mihai)? I also recommend the presentation on DSF, because in many ways I live and die by DSF.

That would be “Device Simulation Framework” if the acronym is unknown to you. Sooner or later, having an easily reproduced software simulator for a device trumps going to EBay to buy some long out-of-production hardware your test driver must have… That’s just one reason I like it, but that’s all I have time for now…

There’s more- but that’s why it’s called a “browser”, eh? By all means, browse!

As for why it took so long to make public- remember this conference occurred under non-disclosure agreements. Takes some time to unravel things and get all the requisite hoops jumped through.

He’s Gone

BobKjelgaard — Fri, 26 Jun 2009 05:15:16 GMT

It has been a tough week, and a busy one. I’m getting ready to leave on a vacation, and trying to get all those last-minute things done.

While my last post discussed the genesis of what I call the Configuration Agent and the WDF QA Universal Setup Job, this week’s focus has been entirely on another of my oddball notions- it started as a desire to build a more precise fault injection engine for testing the KMDF loader. With that California background, the project name was easy- Fault? Why, San Andreas, of course!

At a high level it was a programming model built around IAT filtering of calls from a driver into the kernel. I wanted precision, so I could specify behavior by thread and IRQL, program each DDI independently, allow a driver to do its own filtering [i.e. get a callback with context for a DDI call under given conditions]- and with enough flexibility and locking behavior that the behavior could be reprogrammed within such a callback. For instance, in one working application of mine a filter on calls to IoCallDriver from the KMDF runtime were routed on specific threads to a callback in the test driver which then turned off all call logging, as it meant the call was leaving it’s stack location- I was using the logs from calls to various resource allocation DDI to make sure the framework did not allocate resources under conditions in which we contractually would not fail- so I wanted to stop logging before I called IoCallDriver and to restart it after it returned [I knew the driver handling it was synchronous- otherwise I would have to hook the completion side of things, of course].

So the concept went well beyond fault injection, but it’s always been a side-line. My initial tests wound up being very white box, and a refactoring of the loader code meant a similar rewrite of the test was necessary, but there was no time to do that… But with Windows 7 winding up [since we’ve announced dates, that can’t qualify as a secret anymore], I have time to go back and try to at least do a few new things beyond mere maintenance. Such time is rare and fleeting, so I’ve been busily making the most of it.

So today was busy- I worked a bit of overtime on a COM automation server (in process) exposing a scriptable interface to my engine for some of the other team members to use, and got it at least working well enough to demo. Still have my packing to do, and dinner to make and so forth, so home I headed.

Then I turned on the television, and immediately heard the news about Michael Jackson. Whoa…

On those old tapes I’ve been scraping for my archives I’ve got many covers of his material from the time I was playing in bar bands: Billie Jean, Thriller, Beat It- and more. I’ve not posted them because frankly I never thought I did any of them justice- a lot of my material from those days has rough edges [anyone who listens to what I have posted can hear that], but these were just not good enough. But add me to the billions of fans, then and since, sure…

The other thought that came early on was that in my previous post I mentioned having learned to keep some of my more controversial opinions to myself- one of those was intense skepticism about the molestation accusations. I knew I wasn’t likely to sway anyone there- so why bother?

He’s gone, and he was younger than I am- scary stuff. Life is fleeting, and it was an unexpected reminder of that.

But, it’s time to pack my CDs and games and controllers, write those last few checks for bills coming due during the vacation. eat that dinner, and get ready for a night’s rest prior to that final day of work, so another slapdash post, and back to work…

There was some offsetting good news today- besides finding out I really do still remember enough about COM to roll out an in-process automation server on the QT- without using ATL of course, where would the challenge be in that [answer: in learning something new for a change- you old dog!]? But of course it falls into that semi-confidential area where I’ll just have to sit on it for a while [not that it’s likely to shatter the world if revealed anyway, but circumspection is worthy of practice even for an old blowhard like myself].

I’ll try to do something more substantial someday- these days I’ve just not got the time to really write anything too terribly technical. My apologies to those diehards who still try to read this awful stuff I’m churning out of late…

Not all of them

BobKjelgaard — Thu, 07 May 2009 20:44:33 GMT

Alas, all that stuff I mentioned in my last post won’t make it into the Windows 7 WDK.

As we get close to the final days of the product cycle, it gets harder to justify code changes. In this case, my own choices in handling odd cases were my undoing.

In a nutshell, if I can’t identify the KMDF client driver version, I treat it as if it is the same as the runtime. Since the only version I can’t identify is KMDF 1.9, and a 1.9 driver won’t even load unless the runtime is 1.9 or higher and there is nothing higher, I can’t really claim something is broken besides some text that says what the real version is. All the rest of the relevant UI function will be correct, as de facto, it will be treated as a 1.9 driver, and that is what it is.

Well, they say there will be service packs for the WDK, perhaps this will make one. If not, at least I had my fun, and the tool does work pretty much as it should.

I just designed it too well- yeah, that’s the excuse du jour…

Exceptions rule!

BobKjelgaard — Sat, 18 Apr 2009 19:24:55 GMT

I’ve decided to revert to storytelling mode today. So sit right back and you’ll hear a tale, a tale of my fateful trip (or click something useful- you’re the one deciding, and I’m the one typing to satisfy whatever inner demon has driven me to do this today)…

It began when someone found a bug in the static version of KMDF- the one almost nobody gets to use and which we have been trying to discontinue any support. But people were still using it, and we had no clear migration path, and there was a bug. So we decided to begin testing it. Poor Neslihan got that task (well at least it wasn’t me, this time). Soon we had added a static KMDF driver to our list of test drivers used in our daily automated testing.

Then, of course, we got the bugcheck. As often happens, I played “Johnny-on-the-spot” and (virtually speaking) leapt onto the remote debug session accompanying the report. Buffer overflow? In FxDriverEntry? But only on Itanium (I checked the other architectures that ran the same test, of course- no problems)? What in blazes is this about?

Well, the method is well documented, and not all that hard to understand- a “cookie" gets initialized, it gets stored to the local stack frame in such a way that code overflowing buffers in the stack frame will overwrite the stored copy. On exit, code gets called that checks that value to see if that happened. It’s a little more complicated than that, and I’m not going to be precise, because part of that complication is related to securing the method against the sort of people who like to exploit buffer overflows [so I’m not going to make the method clear for them- they can do their own research without my assistance]. The bugcheck analysis will even tell you the two values- expected and actual, making it easy for people who really think overflows are a bad idea to make a quick guess as to where the overflow came from.

But the circumstances seemed suspicious, so I went and read the source code- H’mm- there’s a default value the cookie is initialized to… Interesting- that’s the value we have- but it doesn’t match the expected value.

At which time, something I’ve known for years hit me like the proverbial sledgehammer- our entry code calls GsDriverEntry (which supports the stack probes inserted by the GS compiler switch, hence its name) when it finishes. GsDriverEntry initializes the cookie, which means that that was why the values didn’t match. Like the sword of Damocles, this had been hanging over our heads for years- the first time the compiler decided to do stack probes in our entry code, everything would break. Ouch…

I’ll leave a bunch of story out here- things got “interesting” at that point, but the problem eventually got solved, both for us and for other people facing similar issues, as well. Short answer is that everyone now initializes the cookie right at the start, just as should happen…

But a while back I had occasion to tweak the build process for the WdfVerifier WDK applet, and afterwards I ran it briefly to make sure I didn’t break anything in the process. Oh, joy of joys- NONE of the 1.9 client drivers are being properly identified. They are all identified as “Inbox”, which is what I do when the client identification method I described last year fails on me.

Already beginning to panic, I think- did someone change FxDriverEntry, and I didn’t even notice it? So I go to our source control system, and look at the change history. The most recent change is the fix for the problem with the stack probes (yes it occurred on static, but what scared me then was it could have happened to any version, because were weren't doing things properly). But that just calls library routines to initialize said cookie… Oh, blazes, those routines must be calling an import I wasn’t accounting for!!! Why??? Because I NULL the IAT to force access violations, and I handle the exception by giving up on getting the true version, and fall back to calling it “Inbox”- which is, after all, exactly what I am seeing. Oh, well- so much for my having thoroughly considered all the test and product consequences of that change when it was made…

Well, easy enough to find out what that import might be. One nice trick Ilias showed me is that you can open any binary in WinDbg as a crash dump, and happily resolve symbols and disassemble code. So I pick a random driver on my dev box, and do so.

What, no imports? But, but--- hmm- it uses a fixed address (load of a 64-bit immediate value into RAX, since my dev box is an X64, thank you- you x86 dinosaurs can keep your wimpy processors). What’s with this?

The answer lies in wdm.h, of course- KeQueryTickCount turns out to always be a macro- on x86 and IA64, it accesses KeTickCount [which is an import, and if you followed my earlier tale, it’s clear adding an import to my existing hack-o-matic mechanism is a trivial task]- but on X64, it accesses an essentially hardcoded address in a “SharedUserData” area. This snippet is from wdm.h, so you can see for yourself…

#define KI_USER_SHARED_DATA 0xFFFFF78000000000UI64

#define SharedUserData ((KUSER_SHARED_DATA * const)KI_USER_SHARED_DATA)

#define SharedInterruptTime (KI_USER_SHARED_DATA + 0x8)
#define SharedSystemTime (KI_USER_SHARED_DATA + 0x14)
#define SharedTickCount (KI_USER_SHARED_DATA + 0x320)

#define KeQueryInterruptTime() *((volatile ULONG64 *)(SharedInterruptTime))

#define KeQuerySystemTime(CurrentCount)                                     \
    *((PULONG64)(CurrentCount)) = *((volatile ULONG64 *)(SharedSystemTime))

#define KeQueryTickCount(CurrentCount)                                      \
    *((PULONG64)(CurrentCount)) = *((volatile ULONG64 *)(SharedTickCount))

My. my. my- this is a problem- no import I can just tweak to point into my code. For a plus, at least the value can’t change (if it did, existing drivers would, after all, cease to work)- well, at least not easily, so I’ll save any remaining paranoia about that for another time. But I’m going to access violate trying to access that address in user mode (not to mention the reams of experimental evidence I had just accumulated by noticing the problem in the first place)…

So in reading about exception handling, a phrase happens to catch my eye- putting it in my words, it says that when continuing an exception, you can alter the context record supplied to you with the exception (and I know this process well, but that’s perhaps a story for a different time). Whoa- that must mean I can change the contents of the registers programmatically, and then tell it to repeat the failed instruction! Now, I can’t find that illustrated in any of the samples the material I’m reading points me to [nor could I find it an an internet search, although the latter was by no means exhaustive]. But that HAS to be what it means, right?

So I start coding- first the half dozen or so lines needed to handle the KeTickCount import. Then an exception filter [only for AMD64, of course]. The logic is my usual bit of precise work: If it is an access violation, and if it is a read, and it is a read of that exact address, and one of the integer registers in the context record has that same address, then change that register’s entry in the context record to the address of the proxy I’ve decided to keep in WdfVerifier, and tell the OS to repeat the failed instruction. I began with RAX, because after all, that’s what I had seen in my investigation, and it seemed the most likely place for a while, but I added the whole set since under the circumstances, it seemed unlikely to do any harm. Anything that didn’t match that exact pattern, and I gave up- just execute the handler, which does nothing. The attempt to run the driver entry code to extract the client version will fail, but WdfVerifier keeps running, and the machine itself is still quite safe from my hackery.

Worked the first time, it did (not counting my usual compiles to get rid of typos). Problem solved. Total work time from start to finish- something like 5 or 6 hours- good enough- after all, I did have to do some research… Of course, there’s always somebody who can do it better, faster, and quicker- and they usually jump out of the woodwork if I start talking about things in that fashion, but it’s my story, so I’ll brag now and regret it later…

Exceptions do indeed rule! I love it when a plan comes together…

A glimpse now at handling the new import…

    //  The descriptors are an array of entries per module, each terminated with an all-0 entry.
    //  Each entry has an RVA for the module image name, and a second for a 0-terminated list of RVAs to the structures used for
    //  resolving names of exported entries from the module (in loading these get resolved to real addresses and are plugged into
    //  the loaded image's import address table in the same order).  So we now know how to write an image loader if we need to...

    //  First, find the descriptor for the KMDF loader, and quit if it isn't there.
    DWORD   StringCopy = (DWORD) -1, WdfBind = (DWORD) -1, InitEvent = (DWORD) -1, TickCount = (DWORD) -1; 

    //  Then look for the Bind and Unbind entry points.  Case counts!

    enum {OutdatedTechnology, HasBind, HasUnbind, OneCoolDriver};
    
    BYTE        TargetIdentification = OutdatedTechnology;

....

            else if  (IsKernel)
            {
                if  (0 == strcmp("RtlCopyUnicodeString", (PSTR) NameDescriptor.Name))
                    StringCopy = ImportsIndex;
                else if  (0 == strcmp("KeInitializeEvent", (PSTR) NameDescriptor.Name))
                    InitEvent = ImportsIndex;
                else if  (0 == strcmp("KeTickCount", (PSTR) NameDescriptor.Name))
                    TickCount = ImportsIndex;
                
                ImportsIndex++;
            }
.....


        DWORD       IATSize, RelocationsSize;
        LONGLONG    OurFakeTickCount = 0x8badf00ddeadbeefI64;   //  ersatz Tick count         
        PVOID*  ImportAddresses = (PVOID*) 
            ImageDirectoryEntryToDataEx(DriverImage, TRUE, IMAGE_DIRECTORY_ENTRY_IAT, &IATSize, &Unused);
        
        PIMAGE_BASE_RELOCATION  Relocations = (PIMAGE_BASE_RELOCATION)
            ImageDirectoryEntryToDataEx(DriverImage, TRUE, IMAGE_DIRECTORY_ENTRY_BASERELOC, &RelocationsSize, &Unused);

        if  (NULL == ImportAddresses)
        {
            FreeLibrary((HMODULE) DriverImage);
            return  true;
        }

        //  Plug them in here, and we are good to go!

        DWORD   OldProtect;

        if  (!VirtualProtect(ImportAddresses, IATSize, PAGE_READWRITE, &OldProtect))
        {
            FreeLibrary((HMODULE) DriverImage);
            return  true;
        }

        memset(ImportAddresses, 0, IATSize);

        ImportAddresses[StringCopy] = FakeOutStringCopy;
        ImportAddresses[WdfBind] = CollectVersion;
        if  (InitEvent != (DWORD) -1)
            ImportAddresses[InitEvent] = FakeEventInit;
        if  (TickCount != (DWORD) -1)
            ImportAddresses[TickCount] = &OurFakeTickCount;

        VirtualProtect(ImportAddresses, IATSize, OldProtect, &OldProtect);

So for some other glimpses, this is one part of the other hackery (complete with my elementary-level annotations):

class AllKMDFDrivers
{
    MachineKey&             Owner;
    KMDFDriverList&         InstalledKMDFDrivers;
    LoadedDrivers&          CurrentlyLoadedDrivers;
    LoaderDiagnosticsFlag   LoaderFlag;
    String                  RuntimeVersion;
    DWORD                   Major, Minor, Build;

    bool            ServiceUsesKMDFLoader(__in PCWSTR ServiceName, __in RegistryKey& Service, __out MyBindInfoAlias& Binding);
    static void     FakeOutStringCopy(__in PVOID, __in PVOID);  //  Fake RtlCopyUnicodeString entry
    static void     FakeEventInit(__in PVOID, __in ULONG, __in ULONG);  //  Fake KeInitializeEvent entry
    static ULONG    CollectVersion(__out MyBindInfoAlias& BindingOut, __in PVOID, __in MyBindInfoAlias& BindingIn, __in PVOID);
#if defined(_AMD64_)
    static LONG     Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount);
#endif

While the rest of it looks like this (and again, this is partial)- yes, feel free to hate my stylistic indifference to the herd- perhaps I’ll get fired for this, and all can breathe a vast sigh of relief…

First, the code that calls the filter

        MangledDriverEntry  CrashAndBurn = (MangledDriverEntry) ((PBYTE) DriverImage + EntryRva);

        __try
        {
            CrashAndBurn(Binding, &Binding);
        }
#if !defined(_AMD64_)
        __except(EXCEPTION_EXECUTE_HANDLER)
#else     
        __except(Filterx64Exception(GetExceptionInformation(), OurFakeTickCount))
#endif
        {
        }

and now our filter:

#if defined(_AMD64_)
/**********************************************************************************************************************************

LONG    AllKMDFDrivers::Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount)

Ahh, the joys of low-level intervention by the truly incorrigible!  On AMD64, we will get an exception when the code tries to get
the tick count.  This value resides at a known address (which cannot change because if it did, all existing drivers would then fail
to work- although I suppose someone could resort to what I am about to do- bring it on, I'll see if I can keep up with the absurd
arms race).

So, first, verify from the exception record that we are getting an access violation of some sort reading that known address.  Then
see if one of the integer registers has that address (start with RAX, since that's where it currently is, and I bet it doesn't
change much).  If it does, change it to point to the value given to this function (which resides in WdfVerifier, since I made it
a reference, and will let the compiler play enforcer), and then continue execution.

In all other cases, execute the handler, which will do nothing (meaning the version will continue to be unknown).

**********************************************************************************************************************************/

LONG    AllKMDFDrivers::Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount)
{

    //  cf definition of SharedTickCount in wdm.h
    static const DWORD64    KnownTickCountAddress = 0xFFFFF78000000320UI64;

    if  (NULL == ExceptionInfo || NULL == ExceptionInfo->ExceptionRecord || NULL == ExceptionInfo->ContextRecord)
        return  EXCEPTION_EXECUTE_HANDLER;

    switch  (ExceptionInfo->ExceptionRecord->ExceptionCode)
    {
    case    EXCEPTION_ACCESS_VIOLATION:
    case    EXCEPTION_IN_PAGE_ERROR:    //  Unlikely, but what the heck...
        if  (2 > ExceptionInfo->ExceptionRecord->NumberParameters)
            return  EXCEPTION_EXECUTE_HANDLER;
        break;

    default:
        return  EXCEPTION_EXECUTE_HANDLER;
    }

    if  (EXCEPTION_NONCONTINUABLE == ExceptionInfo->ExceptionRecord->ExceptionFlags)
        return  EXCEPTION_EXECUTE_HANDLER;

    if  (EXCEPTION_READ_FAULT != ExceptionInfo->ExceptionRecord->ExceptionInformation[0] ||
        KnownTickCountAddress != ExceptionInfo->ExceptionRecord->ExceptionInformation[1])
        return  EXCEPTION_EXECUTE_HANDLER;

    if  (KnownTickCountAddress == ExceptionInfo->ContextRecord->Rax)
    {
        ExceptionInfo->ContextRecord->Rax = (DWORD64) &FakeTickCount;
            return  EXCEPTION_CONTINUE_EXECUTION;
    }
    else if  (KnownTickCountAddress == ExceptionInfo->ContextRecord->Rbx)

The rest I leave as an exercise to the reader, having already put all 0.378 of you through the treadmill today…

Using C++ in a KMDF driver part 1- a pattern for using contexts as objects

BobKjelgaard — Thu, 12 Mar 2009 16:36:59 GMT

This is an article I’ve started probably close to a dozen times since I started this blog, but never published. In part because of all the heat the topic of using C++ in the kernel generates, and the rest perhaps because of my reaction to that heat.

So I’ll get one thing off my chest at the start and perhaps that will be enough to let me proceed.

I’ve been writing drivers with C++ in the kernel (or the Win 3.1/9x/ME VMM) pretty much since I began using the language (in the very early 90’s). I routinely use paged code (paged data not so much- I’ve never had designs where there seemed to be any benefit to it). That spans more than a decade- in fact, its close to two decades now.

I’ve NEVER had a problem with the issues raised in the paper. I’m not even going to link to it. If you want to find it- it isn’t hard to find. On top of that, in a determined effort to find people who had, I didn’t find many- and the one clear case I did find was cured with a #pragma that would have made sense if you were a C programmer and had a basic understanding of where VTABLES and such get emitted to. So I personally have a sense of mismatch between my own experience and the strongly worded severity there.

I’ve been told that if I persist, I’ll deserve all the paging problems I get [and that nobody will help me with them]. Well, I do get them (always have), and I’ve been developing long enough in this environment that I don’t really need anybody’s help to find the root cause of a bug like that. The causes are always the ones I remember- acquiring a spin lock in pagable code- making a routine pagable that can be called at elevated IRQL- all the usual ways to screw up in that other language that is the hallmark of a true first-class kernel developer. But none of them had anything to do with my choice of programming language. Not once. Not saying that it never happened across all those years, because in the early ones, the ability to catch a bug like that was severely limited- there was no Driver Verifier, no static analysis tools. So it may not have been noticed early on. But now that I have them, it’s still not happening…

Now there can be plenty of reasons for that- but I look at what people take away from the paper, and I do a lot of the things they think aren’t safe. I use polymorphism and inheritance freely. I don’t use multiple inheritance a lot, but I have used it and have not observed problems arising from its use. On the other hand, there are common features that I don’t use as a matter of personal preference or style that may bear on this. I don’t expose the implementation of functions in header files (meaning the compiler is not going to start out by inlining them and then give up later, dumping them in some unintended segment) with the exception of trivial accessor functions. I don’t use templates (I’m not sure they were a language feature when I started, but at any rate, during my learning curve I never needed them, so while I can handle code with templates, I don’t use them myself). I almost always code my own constructors, destructors, copy constructors and assignment operators (I usually have to- I prefer references to pointers and if you have reference members, the compiler can’t generate default code for most of those routines).

So now that its clear I’m not going to walk the company line on that topic [the opinions I expressed ought obviously to be clearly my own], I’ll proceed to something more useful…

Leveraging the KMDF Object Model

KMDF provides a nice object model, with managed lifetimes and one of the most delightful observations I had in my first KMDF driver was that it was easy for me to blend this with my usual coding patterns.

As an aside, I’ll note that I never use sample code to learn anything. I take the reference materials and code to them. If there are samples, I treat them as the last resort- and I will deliberately change as much as I can of them [in part to see what knowledge of things that can go wrong wasn’t explicitly represented in the sample]. I judge the quality of the reference material by how rarely I have to look at a sample to figure something out [yes, I don’t find much reference material i would call “good”].

So my first KMDF driver was a software bus driver, and I didn’t go near toaster in writing it. In fact it was one of those “ActiveX” (OLE automation) test drivers I mentioned in our DDC presentation. Now we had some samples for them, too- and not surprisingly, virtually no reference material. The samples were fastidious about one thing- all the COM parts were in C++ (not a lot of choice there)- but all the parts using KMDF were in C- precisely because of said paper. I might add that this was even though no attempt was made to mark any of the code or data pagable in those samples. Well, I’m a stubborn <expletive of your choice here>, so I decided then and there I was going to write the whole driver in C++ in spite of the objections I received. I was still in my “trial period” so they could always fire me if they wanted to, but the job market was good enough at that time that I was pretty sure I could find something…

Back to the proper topic- one fine thing is that most of the KMDF macros are agnostic enough they can handle at least straightforward C++. That was one of the happiest discoveries of that time.

So the basic pattern as I use it:

Declare a static function in your class that takes a WDFOBJECT as input and returns a pointer to an object of your class.
Declare a class-specific placement form new operator that takes as its additional input the type of handle you expect your object to live in the context of (that is, it can be more precise than the preceding function can and you can benefit from stronger typing in C++)
Declare a class-specific delete operator that basically does nothing if you need to have your destructor invoked.
If you have such a delete operator, also declare a static member with a void return that takes a WDFOBJECT as input.
Use the WDF_DECLARE_CONTEXT_TYPE_WITH_NAME macro to get the compiler to write that first function for you.
Have your new operator implementation use the first function to return the address of the underlying context (you basically ignore or validate the size parameter) from the passed-in handle.
When creating the context, and you need your destructor called, use a WDF_OBJECT_ATTRIBUTES structure with the EvtCleanupCallback set to the routine in item 4.
Code that routine to use the routine in item 1 to get the context address out of the object handle- and delete that pointer. This causes your destructor to be called at cleanup time (which is much more sensible than destroy time) and your do-nothing delete operator will also be invoked (or inlined out of existence if your compiler is any good).

There- you “create” your object when the KMDF object is created (or via a WdfObjectAllocateContext call if you add your object later), and “delete” it when the object dies. But KMDF manages the memory and most of the object lifetime for you. Sure works for me (a lot).

The following snippet is from that first driver (I’ve since dropped the usage of “C” on class definitions in my general drive toward anarchic style). This is slightly convoluted because I have logic allowing only one instance of a device with this driver- so I’ve deliberately intermingled driver-level and device-level usages (always pushing those boundaries- but I think that’s a good way for an SDET to think). I’ll admit this is slightly doctored (I removed some things related to the COM technology as that I can’t disclose, plus I tried to include support for the standard bus interface and that just complicates things without illustrating this method), but it should show I practiced what I am preaching…

class CTargetTestBus
{
    static WDFDEVICE                Owner;              //  WDF device that "owns" this bus object

    static void*    operator new(size_t size, WDFDEVICE OwningDevice);
    CTargetTestBus(WDFDEVICE OwningDevice);
    ~CTargetTestBus() {}

    //  Private callbacks (ie, accessed from within this class' code)
    static EVT_WDF_IO_QUEUE_IO_DEFAULT      OnIoDispatchDefault;

    static void operator delete(void*) {}
public:

    NTSTATUS    NewChild(LPCWSTR Name, int InstanceID);
    NTSTATUS    RemoveChild(LPCWSTR Name, int InstanceID);
    void        DoneWithBus();

    static CTargetTestBus*  GetThisTargetTestBus(__in WDFOBJECT Object);    //  WDF macro writes this code
    static CTargetTestBus&  GetTheBus(bool& BusPresent);
    
    //  Driver callbacks (public, because used in DriverEntry)
    static EVT_WDF_DRIVER_DEVICE_ADD        OnDriverDeviceAdd;
    static void             OnDriverUnload(IN WDFDRIVER Driver);

    //  This one is accessed from a member in CChildInfo
    static EVT_WDF_CHILD_LIST_CREATE_DEVICE OnAddNewChild;
};


// Macros to get the context
WDF_DECLARE_CONTEXT_TYPE_WITH_NAME(CTargetTestBus, CTargetTestBus::GetThisTargetTestBus);

Now playing: Me (that old recording of Johnny B Goode again)!

Bad Advice No Matter Where it Came From

BobKjelgaard — Sat, 07 Feb 2009 19:34:34 GMT

Just a short post- in the “don’t do this at home” category.

I was recently asked to assist someone with another KMDF installation issue, and in this case, the Internet proved itself to be everything it shouldn’t be- a treasure trove of unchallenged bad advice.

I don’t care where you read it- deleting wdf01000.sys and wdfldr.sys from %windir%\system32\drivers of a machine is:

So rarely necessary as a problem-solving step you shouldn’t even be thinking about doing it.
Very likely to make things worse than they were before you started. If you are using any OS beginning with Windows Vista, it can turn your machine into an unusable paperweight, and you may find it very difficult to recover. Plus there are enough drivers in general use now even the earlier OS are not immune.

What I was looking at was a machine where someone attempted to “cure” a KMDF 1.5 installation issue this way [I was fortunate enough to have the entire setupapi.dev.log, so I could actually tell within a couple of hours when this was done]. On Windows Vista, no less- now doing that requires overriding system file protection, so it isn’t like we haven’t tried to save you from yourself. This particular user was very lucky- no boot devices using the framework [and this is the first Vista machine where I’ve ever seen this].

But once you’ve done this, you’ve given yourself one huge problem- none of the already installed drivers work, and no amount of reinstalling them is going to save you.

Why?

Because our coinstallers won’t even attempt to update an OS where KMDF is a part of the OS unless they were released AFTER the OS was. They don’t need to- it is already there, built-in.

Nobody’s driver is going to work after those files are deleted, until an update is applied. If your Vista machine has had the SP1 upgrade installed [and by now, the vast majority of them have], then it is using KMDF 1.7- if you delete our binaries then NOBODY can fix it for you, because that is the current released version [1.9 is still in Beta]. There is no driver on the market you can install that will fix this, because we haven’t given anyone a package that is capable of that.

The only cure is to find the right binaries and put them back where they belong. Then consider not taking advice from some internet blowhard [if you feel like considering me as one- hey, so be it- but I actually helped engineer the product, and that claim is verifiable- what’s your source’s claim of expertise based upon?].

My apologies for the tone here, but I track access to this blog, and I know “deleting wdf01000.sys” is a search term that pops me up. So maybe some advice here may save you a world of grief. There are better ways to solve your problem, whatever it is- even the problems where this was actually suggested as a “working” solution at one time.

Now Playing: Bob Kjelgaard- “Let’s Escape” (not available anywhere)- just me and my acoustic with a song a couple of my college buddies wrote [and I renamed, revised, and arranged]…

When Progress Is Required…

BobKjelgaard — Thu, 15 Jan 2009 01:33:44 GMT

Today’s post is going to be a quick overview of one of the great new features in KMDF 1.9- built-in support for “guaranteed forward progress”.

What is it, and why would I want to have it?

The essential case occurs in storage device drivers. If one of the system’s paging files resides on that device, then the driver is going to be asked to read and write memory pages to and from the paging file. Failure of this I/O isn’t an option- the system itself could be dead if it were to fail.

Since it is common for drivers to subdivide requests into smaller pieces (for instance), this can mean that the driver may need IRPs or MDLs or other items to complete this work. But if it asks for them from the system, those calls may fail. So, to guarantee forward progress under low-memory conditions (and of course paging is critical in such cases), it is a common practice to pre-allocate IRPs, MDLs and other items that may be needed when memory is tight, and to use them only when this situation occurs. Basically a rainy day account used to keep things moving under stress, but not touched in the normal course of events.

So what’s this new stuff about?

Well, KMDF has to create WDFREQUEST objects for incoming I/O, and this is a memory allocation. So, beginning with KMDF 1.9, you can now set a “Forward Progress Policy” on a WDFQUEUE object while setting up your device.

This policy consists of these things:

A number of WDFREQUEST objects that the framework is to pre-allocate, use only as needed, and not let go of.
An optional event callback that allows you to create additional items for each of these reserved requests ass they are created (so you can do all of your pre-allocation when the policy is established).
A similar callback for normal requests (so all requests can be assumed to have similar contexts in your IO callbacks).
A choice of policies regarding when this feature is utilized (more on that in the next list).

The choices I alluded to (things the framework will do when it fails to get a functional WDFREQUEST for an incoming IRP destined for your queue) are these (they are mutually exclusive):

Always use a reserved request in the event of a failure to wrap an incoming request in a WDFREQUEST (if you have the callback which adds items to a normal request, failure of that callback also triggers this condition- a factoid that came in handy when testing this feature, of course).
Provide an optional “examination” callback which will look at the incoming IRP and decide whether you want to just fail it or use a reserved request.
Examine the IRP to see if it is paging I/O (the OS marks these)- if it is, then use a reserved request, otherwise, fail it.

A new DDI was added to set this policy on a queue, and another was added to allow you to see if a given WDFREQUEST is a reserved request or not.

If requests continue to fail, and all the reserved requests are in use, then the incoming requests get pended and remain in a list at the queue- as each reserved request is completed, it will get recycled as needed until there are no more failing requests to guarantee progress on. So a queue with this configured becomes a counted queue of sorts under low-memory conditions.

For the most part you can treat these requests in the normal fashion- you can forward them to other queues, for instance. When completed, they return to their owner- the bookkeeping behind all of that makes this a challenging feature from a test perspective.

Details on this are in the Win7 Beta WDK- see WdfIoQueueAssignForwardProgressPolicy and WdfRequestIsreserved as a starting point for further reading- this post is just meant for the mile-high overview [so I feel like I did something besides practice my guitar during today’s long builds].

Now playing: Yuki Kajiura- Madlax OST Volume 2- Lost Command

Final Thoughts on KMDF Installation for 2008

BobKjelgaard — Tue, 30 Dec 2008 19:34:28 GMT

I should get at least one post in this month, and of the topics I want to discuss [but haven't had time because I've either been on other tasks or forced away from my blog by inclement weather], this one is the easiest. I'll cover a few errors that have arisen within the last month or so related to installation of KMDF that are worthy of note.

Things Not To Be Done (But They Were Done, Anyway)

Don't alter the settings of the KMDF runtime service in your installer. They belong to Microsoft, and you shouldn't be touching them! We have an unfortunate situation that arose recently where one vendor's installer changed the load group for our runtime, potentially [and not just theoretically, as it turned out] breaking a subset of the previously installed boot devices that were also using KMDF. This is an automatic blue screen at boot [and very difficult to fix] for the affected user, but it is the previous vendor [and Microsoft, of course] who gets blamed in the message on that screen. There are cases where we will tell you what to set them to- specifically a device that ships in an OEM system and has to be used during the earliest (or "text-mode") portion of setup. But that mechanism doesn't involve an installer, and has so far been disclosed only on a need-to-know basis. Hands off our stuff, please!
Make sure that you use the DDInstall section name [and this includes any decorations such as NT or NTAMD64] for your device when naming the Coinstallers and WDF sections in your INF. If you don't do the first, then our coinstaller doesn't get copied or invoked, and if you don't do the latter, then we won't install your device if the coinstaller is invoked. One case I saw formed both of these from a service name instead, meaning the device would only work if the required version of KMDF was already there. We are improving the ChkInf tool to catch this error in the future, but until it arrives, you'll have to verify this yourself.
The first name in the KmdfService directive in your DDInstall.WDF section has to match the first name in the corresponding AddService directive in your DDInstall.Services section. This is particularly critical if you are using the KMDF 1.5 coinstaller, as it can result in a permanently broken installation [similar to this problem] for everyone [until a KMDF 1.7 device comes to that machine and fixes things, or someone takes the steps to fix it manually]. If you check your logs, you'll see tons of errors in %windir%\setupact.log if you do this. Since the ChkInf work is going on now, I'll make sure I recommend we check this as well...

Please note I named no names- to err is human- to blame is often pointless...

Other Trivia

Well, my capturing the music performances of my 20's and 30's has had some benefit- it's given me two hobbies- a new one is capturing this stuff and processing some of the noise and warts out of it. The second is I'm playing my guitar again- keep my Guild acoustic in my office, and my left hand already has developed a fine set of calluses. I try to do my practicing during long builds at 3-5 AM or so to avoid disturbing the neighboring offices- wouldn't want to interfere with the Windows 7 WDK production effort, after all...

Hope all have a fine 2009!

Now Playing: One of my final performances of Chuck Berry's Johnny B Goode (ca 1987) Narcissism abounds for me these days (but I'll get over it soon enough- I always do)

WinHEC Presentations you might want to view

BobKjelgaard — Thu, 20 Nov 2008 00:01:56 GMT

Since I was asked in response to my previous post "so what else is new in WDF?", how about this link to the presentation Eliyas and Peter made on exactly that topic at this year's WinHEC?

I suppose I could also flog this link to the presentation Ilias and I made [but it's not quite so broadly of interest- WDF coinstallers and introduction to WDF logo requirements and tests].

Now Playing: Turning Point (a bar band I played in during the 80's)- The Other Woman (ahh, I could play a decent bass once in a while, back in the day)

Which reminds me, I've put that one up here (about 2 MB WMA) if you really want to hear it yourself. I've put up a smattering of others available from this page(including a few after I switched to lead guitar and vocals)...

Now Playing: Grateful Dead- Europe '72- Sugar Magnolia

Queue Callbacks are NOT Dispatch Routines

BobKjelgaard — Wed, 19 Nov 2008 04:22:21 GMT

One of our SDETs recently made an error in a Win7 WDK sample that prompted a discussion. After I explained the error, Wei gave me some feedback that my explanation made more sense than the WDK documentation did. So I'll try to elaborate a bit on what I see as the underlying misconception, and perhaps it will help others avoid this conceptual error. The title perhaps gives my key point away...

The Wild and Wooly World of the Original Driver Model

I'm with some of the other purists here- some will call this the WDM driver model, but this model is older than that- within Windows, it was in the original release of Windows NT [and it's based on VMS, IIRC- that's even older]. The driver object has a table of dispatch routines, which are what the I/O manager calls to tell the driver it has to process some form of I/O request [in the form of an I/O request packet or IRP]. There is one entry for each "major code" [which is an enumeration of types of I/O request which is somewhat arbitrary, but for compatibility reasons is unlikely to change much- the advent of WDM added several new major codes, for instance, but the previous ones remain]. This isn't meant to be a tutorial, so that's enough explanation for now.

The main things to consider about dispatch routines for this discussion:

There are instances [for instance, top level driver] where you can safely assume you are called at passive level and in context of the calling process and thread.
You are almost always called at passive level anyway.
There is some synchronization of I/O done by the IO subsystem (Serialization on a handle and of non-overlapped I/O, for example), but you cannot generally assume any synchronization between routines or even between a routine and itself.

That last bullet is where a lot of problems come in- unless you design the right kind of test cases, you can miss synchronization issues in a driver. But the first two bullets also matter, particularly if you are trying to be a good citizen and make code and data paged when they are only relevant at passive level.

WDF Offers Abstractions With More Precise Semantics (If You Accept Them)

WDF handles dispatch routines for you through its request queuing models [and I use the plural deliberately, there are multiple models to let you tailor your driver better]. One thing in particular these provide is the capacity to synchronize request processing in multiple ways:

The sequential queue dispatching model guarantees that only one request is processed at a time- a new request is not dispatched until the previous one is completed. This is the simplest model, but it isn't desirable for a lot of applications. But if it works for you, it's the simplest mechanism of all to understand.
New in KMDF 1.9 is a counted parallel queue, which can have a bounded number of concurrent requests- once this limit is reached, new requests are not dispatched until an existing one completes. [The sequential queue can be called a counted queue of one request]. If some or all of these requests need the same resource, you will need additional synchronization, of course.
A device can use device level synchronization on its queues- this means that only one callback at a time will execute, and a new request will not be dispatched until the current callback completes. This is not the same as sequential dispatching in that it affects all queues, and the point at which the next request is dispatched is different. If your callback doesn't complete a request [which is often the case, it is often enough for it to simply set up an operation], then the dispatch will occur sooner than it would in a sequential queue and you have overlapping of requests [potentially quite a bit of it, depending upon how you manage this sort of asynchronous I/O]. This model can also be extended to work items, Dpcs, and timers parented to this device. If your callback DOES complete a request, the dispatch occurs a bit later [because the callback exit and not the completion triggers the dispatch of the next request]. This model is good if you have a device level resource you want to serialize access to, but you don't have to wait for completion to use it for a further operation (something along the lines of a command FIFO, for instance)
For further granularity, you can synchronize at queue level instead- in this case, callbacks within a queue are serialized with respect to each other, but not with respect to other queues- again this can be extended to work items, etc, but in this case, make sure they are parented to the queue and not the device. This model is good if you have independently programmable subdevices, each of which still requires some additional synchronization.

The error I alluded to was related to paging and the last two types of synchronization. You can specify that synchronization has to happen at passive level. If you do, then a passive level locking mechanism is used, and you can make your callbacks paged code. If someone sends you a request at dispatch level, a work item will be used by the framework to dispatch that request to you [the caller will get STATUS_PENDING back].

But if you DON'T state you want passive level execution, then the potential is always there that you have to synchronize with dispatch level code. That means a spin lock is necessary. If you are using either of the last two synchronization mechanisms, and have not explicitly used passive level synchronization [this has to be set at the device level or driver level] you MUST NOT make the code paged, because it will ALWAYS be called with the spin lock held [and hence at dispatch level].

Yes, you may know the I/O came to the dispatch routine at passive level, but the callback is not a dispatch routine- if you've asked for a synchronization model that isn't workable at passive level, your callback won't be there, because the framework had to change IRQL to meet your synchronization requirements. Be careful what you ask for- you might well have received it!

One afterthought about queues- you can use multiple queues and even take requests from one and forward them to another. So if, for instance, you have some particular IOCTL that would be perfect for sequential dispatch, you can use an IOCTL handler in a top-level queue, create a sequential queue for just that particular IOCTL and forward the requests from one to the other. You still get the benefits of cancellation handling and a more focused model when you do this.

I'm still not sure this is clearer, but at least it isn't another Life At Microsoft tag, this time around!

Now Playing: Seatbelts(Yoko Kanno) Cowboy Bebop- "Bad Dog No Biscuit"

Adaptive Fault Injection

BobKjelgaard — Wed, 12 Nov 2008 22:18:43 GMT

One of the topics I covered in Ilias' and my joint presentation at WinHEC this year was new requirements coming for getting a WHQL signature for drivers using KMDF (UMDF as well, but this discussion doesn't currently apply to them). One part of those new requirements is that your driver survives a rigorous fault injection from the WdfTester tool. The method used for that is what I intend to discuss briefly in this post.

By "survives", I mean no bugchecks, hangs, or leaks- graceful failure, not fault tolerance, is the goal here.

The method used is one I call adaptive fault injection. The term "adaptive fault injection" is admittedly my invention (and I'm probably the only person that uses it)- but I thought it fit. So this is an attempt to define what I mean by the term.

The Problem

For readily available fault injection, I had two tools available to me in 2006, when I began looking into ways to improve code coverage in the KMDF loader in an automated way.

Driver Verifier Low Resource Simulation- this did boost our numbers, but it took a long time,, and since it was random, it took extensive analysis to find out what was covered in terms of code paths, and the only way to cover a particular fault was to run until you finally hit it. Not very predictable. To me, it was a shotgun handed to me when what I really wanted was a needle.
The always on, but programmable threshold mechanism in the KMDF Verifier. For one thing, it didn't help me with the loader (but I was also looking for generic solutions), but for another, I wanted to be able to say- "I want just this one fault, period". So it was still a bludgeon, and not the precise instrument I desired.

Now, we had experimented with IAT hooks and had even used them experimentally for fault injection [the tool just failed pool allocations]. I could easily see that I could use them for precision, but the result could be a very high-maintenance mechanism- if I had to individually code each fault, the price for the technique would be way too high.

I wanted something that could examine the system under test, find all the failure points, and then cover each and every one of them. Something that would adapt to the system's behavior as it evolved.

Hence "adaptive fault injection". So perhaps the basic idea is less of a mystery?

Feedback is almost always your friend

If you have an IAT hook, you can log the activity through the intercepted call- both the data going into the called routine, and the values returned. You can't see state changes occurring on either side of the hook, but at least you've got a point you can begin at [and this situation reminds me a lot of the days when we were testing circuit boards via their external connectors].

So the basic idea for the first iteration was quite simple:

Monitor all the points you can, and record the activity on them.
Analyze the recorded activity, and determine where inputs going back into the system can be modified to simulate failures.
Count the number of such failures you have identified.
Repeat the original activity as many times as needed, and on each cycle, inject each failure in turn. Log this activity much as was done in the first step [and the logging mechanism should report the injection activity, of course].

WdfTester was under development at the same time as my tool [which I named "SanAndreas" after the infamous fault line running up the West Coast of the USA], and utilized the same concept, although in a different form. My trigger was a simple integral counter, making for a simple loop. WdfTester instead injects a fault on a specific count of a given DDI call. Neither method is perfect [I'll get back to that in a bit], but in highly repeatable traces, there's no real difference between them. By the way, I have no idea if we both had the same idea, or if we discussed this [although we probably did, because when I first began that task I said I wanted this sort of injection mechanism, long before I did the implementing].

So, this provides more precise injection, at the cost of potentially longer run times, but the times are deterministic and computable once the analysis step has completed. But it also requires little programming or reprogramming, as it does adapt itself to the observed behavior of the system under test.

Where this still falls short

At least a couple of places come quickly to mind:

In multithreaded cases, observed behavior may not be that deterministic. In that case, we at least try to inject something, but there is potential for things to be missed, or even for the same fault to be injected multiple times.
The state that cannot be observed may matter- I believe this is primarily a concern when designing the repeatable case you want to inject, or at least this risk can be mitigated by considering the state when deciding what sequence to apply this technique to.
The system may adapt to the faults you inject [for instance a failed I/O may be retried]. This is bad only in the sense that it may prevent you from reaching paths you would still like to reach. One could address this by recursively repeating the analysis and subsequent phases (and then using a more complicated injector- first inject this {series of} fault{s}, then in turn inject these]. Such a mechanism may need to have runtime bounds placed on it to prevent endless recursion...

Still, even with these flaws, I've found the technique even in this most primitive of forms to be a step forward in having a more precise fault injection method available to me in my bag of tools.

As long as I'm typing...

Gee, Patrick- I've played all three [but I preorder at GameStop, even though the price is higher]- which reminds me that I need to pick up Call of Duty: World At War [and a day later- World of Warcraft: Wrath of the Lich King]. Didn't get far in Fallout 3, but one play-through each on Fable II and Gears of War 2.

I've been pulling tunes off some really old cassette tapes [bands I played in during the 70's and 80's- recorded in mono on hand-helds, for the most part, along with some solo practice recordings]- think I'll use that nice USB stick I got at WinHEC to bring a few of the livable ones to the office- see if anyone can figure out which tunes I'm the guitarist or bass player and / or vocalist on... Of course, the poor quality of the audio ought to be a clue. If I find one I can live with, maybe I'll link it somewhere [I have a few I wrote myself, are public domain tunes, or are just instrumental noodling, so I can avoid the copyright bogeyman].

Also, since things are now disclosed- this is what the new 1.9 controls in WdfVerifier will look like.

Now Playing: Grateful Dead Album: Built To Last- Victim or the Crime- seem to end on Dead tunes of late, not that I don't listen to plenty else.

Determining a KMDF driver's client version

BobKjelgaard — Tue, 11 Nov 2008 23:37:00 GMT

This was probably the most fun I've had in a while. The initial versions of WdfVerifier simply collected the version information recorded by the KMDF coinstaller when it is used to install the framework for a driver. Unfortunately:

not every driver needs to use the coinstaller, so the registry tracks may not even be there (for instance, any KMDF driver shipping in an OS with KMDF native to it won't have them)
the information comes from the INF, and by now many of us know how trustworthy that information is.
the information could be modified or even removed any number of ways, since it is just unsubstantiated registry tracks.

This was annoying (I wound up labeling all of these cases as "Inbox" in the UI), but with KMDF 1.9 it became important, as we are adding new keys that will only be used if you are actually a 1.9 driver (they won't affect a 1.7 or earlier driver when run with KMDF 1.9, in part so we can minimize regressions in behavior- a lot of attention has been paid to this sort of thing).

Since we have now revealed these at WinHEC, I'll give a short summary of those relevant to WdfVerifier:

A new registry value for advanced verification options:
Callback checking- where we will verify that you do not change IRQL or critical region status during your callback [yes, that has happened].
The ability to inject faults on queues establishing guaranteed forward progress policies- i.e., we will mimic failure to allocate WDFREQUEST on incoming requests- either randomly at the Driver Verifier standard 6% rate, or else do it for ALL I/O destined for your queue
If your driver does not have a setting for VerifierOn, we will automatically turn KMDF verifier on and off based upon whether you are using Driver Verifier on the driver.

The controls for these settings were not visible in the screen shots I posted here, because we hadn't yet said these features would be there. But in all cases, they apply only to 1.9 drivers, so as to not disrupt existing behaviors for earlier versions.

There sure is a catch here

Well, there are no easily discernible tracks pointing to your driver's version. The only certain way to tell is to load your driver- its entry code will tell the KMDF loader what its version is [which is why the !wdfldr debugger extension can tell you], but that is it. I later found out an attempt had been made to put a version string in the library code the driver must be linked with so some sort of scan could locate the version. But the linker removes unreferenced data, and in part because we didn't know it was supposed to be there, nobody had tests to verify that it was [not to imply the tests would exist if we had known- we have to prioritize test efforts, and testing something you aren't even using isn't really high on that list].

So, the real knowledge is buried in the kernel, and I want to extract it in user mode, no ways have been engineered for me to get it, and even if I get them in the new product, I personally prefer solutions that work with what is already there.

Mission Impossible- now THAT'S something I find interesting.

The Inescapable Lunacy of Bob

So fine, all I have to do is load a kernel mode binary in user mode, call its entry point, intercept the call to the loader that gives the correct version, harvest the version information, and return failing status so the rest of the initialization code won't run. Piece of cake!

Well, I actually understand the PE format pretty well- I have an internal test tool, which we mentioned briefly at DDC, where I do IAT hooking, dynamic linking [as in I have a basic GetProcAddress implementation for anything in the kernel- not just the kernel and HAL entry points MmGetSystemRoutineAddress will return, but any PE file loaded in the kernel process], and also share events and map memory between processes to implement the kinds of interfaces people get flamed for suggesting. Around the IAT hooks I built a fairly sophisticated logging, and fault injection or targeted callback mechanism- so I could do things like inject faults along one thread, while leaving others calling the same OS interface alone, etc.

So in this case, I was working in user mode, and that's a lot more forgiving than the kernel. As I said before- fun!

So, already WdfVerifier would scan binaries with no Parameters\Wdf- I would map them into memory, walk through the PE structures, and see if they imported the linking support functions from the KMDF loader. If yo understand the PE format, this means picking up the RVA (relative virtual address- how far from the start of the binary image something is) for its entry point was simple. It also meant I knew where the Import Address Table would live, and where any linkage point in that table would be.

Fine- just a little surgery on code I already have.

But I want more- the code will probably need relocation, after all, and some pieces load a bit differently in memory than they do as just a mapped data file. So let's take a quick look at the Win32 API. Ah, fine- LoadLibraryEx has several promising options. I especially like the one about loading without resolving external references...

Why? Because I am going to hook calls normally intended for other KM binaries and [since I didn't mention that before] you can't load them successfully in user mode! Why make the loader try to do something I am going to undo, and which won't succeed, anyway?

So I tried it, and it worked. So cool!

Well, now I need to think about how this has to work- The driver is going to expect a pointer to a device object, and a second to a Unicode string. So, what happens to these before the call I already know I will intercept? Well, here I cheated a bit, and actually looked at our code [but I could have done it from assembler quite easily, the path was rather short]. Hmm- nothing touched the driver object, but it did get passed by pointer into the routine I was hijacking. The Unicode string was copied with a system DDI, but not otherwise touched. So, let's intercept that string DDI and make it a no-op, and then make our "Driver object" the structure I want to return the version information in- it gets passed to the loader code, but after all, I'm spoofing like crazy.

So I tried this with some stack parameters, and it actually worked- sort of. First surprise (not a big one) was a code obfuscation technology injected some additional code into at least one driver, meaning I had an additional routine to supply and no-op.

Then I found out that not only did that option not resolve references, it didn't do any relocation! So, I had to do that myself, too...

To make a horribly long story short, I added them, it worked on all three processor platforms, secured it a bit by doing things like allocating the inputs on pages not on my stack (and well zeroed out, so they'd most likely access violate if someone ws really pulling a fast one).

Do all that the first time it is run, then set a flag in the registry that all of them have been inventoried. Allow it to be run again upon request, but most of the time, you just trust what is registered...

Now playing: Grateful Dead- untitled [usually called Skull and Roses] Not Fade Away / Going Down The Road (Feelin' Bad)- another classic jam...

Determining WDF runtime and client versions

BobKjelgaard — Wed, 05 Nov 2008 17:29:00 GMT

What a long, strange trip, it was, getting to WinHEC! I waited for a shuttle to the airport at my apartment, and the rain [which was fairly heavy] turned into hail! But by the time we got near the airport, everything was dry.

Well, LA is probably still LA (I lived in El Segundo for about a year 6 years ago, and of course went to school in Pasadena, so I'm not a total loss], so that's enough travelogue.

UMDF

Easy- Peter and his team obviously planned ahead. There is an exported data item linked into each driver and in one of the framework binaries. So LoadLibrary and GetProcAddress, and you're there. The data is three ULONGs [IIRC, as I'm not about to carry Windows source on my laptop to WinHEC without a better reason than a desire to blog]- major version, minor version and "SP level".

KMDF runtime

I won't say Doron and his peers didn't plan ahead- I believe it was more a case of having more important problems to solve at the time... But I did mention it, so I might as well face it.

Again, I'm going to have revisit a few details again, later. This is my recollection [and it will be pretty close]:

KMDF 1.0 had nothing to identify it.
KMDF 1.1 and 1.5 have a resource in the disk file that identifies major version, minor version, and build number, similar to the UMDF version info described above. LoadLibraryEx (as datafile) and the usual FindResource, etc. route.
As part of our installation improvements in KMDF 1.7, the runtime now also records its version in the registry under its software key when it is loaded, and removes it when it unloads. That's why the WdfVerifier image I showed here said "In Memory". I like this because it also works remotely (and I'm not going to inspect binaries remotely).

So, I first look for an in-memory version, as that is the most reliable. If I am remote, then the only other thing I can do is look at the registered OS version information. If it is Windows Vista, I assume it must be KMDF 1.5 [because 1.7 would tell me it is in memory, as Vista has running inbox KMDF drivers]- the message indicates this, although how I say it is by inference escapes me. Otherwise, it is 1.5 or below, but I can't tell- I don't remember the message, but I do hope I changed it from my original "Can't tell- you tell me!".

If it is local, then when I first scan all driver binaries, I pick out the runtime, and examine the version information just as I describe above.

KMDF client

This is the fun one for me, so I'm going to save it for a later post- it's time to get dressed and find that shuttle to the convention center, so I can spend a day in WinHEC [oh, the pun potential for that one...].

No tunes today- wonder if the WDK folks I now sit amongst are enjoying time away from the endless noise emanating from my environs!