Trouble Ahead- Trouble Behind : test development

2008 DDC slides now available to all!

BobKjelgaard — Wed, 22 Jul 2009 14:24:33 GMT

Got an email late last week stating that the material I and others presented at the 2008 Driver Developers Conference (which Dieter and James want renamed to Driver Development and Test Conference- more power to that idea!) is now available here.

The sessions I participated in are a description of several WDF test technologies and some hand waving about troubleshooting WDF installation problems.

If you are a tester, you might also want to take a look at James and Dieter’s presentation on concurrency testing, and who wouldn’t want to know about the new Driver Verifier features for Win7 (ably presented by Daniel Mihai)? I also recommend the presentation on DSF, because in many ways I live and die by DSF.

That would be “Device Simulation Framework” if the acronym is unknown to you. Sooner or later, having an easily reproduced software simulator for a device trumps going to EBay to buy some long out-of-production hardware your test driver must have… That’s just one reason I like it, but that’s all I have time for now…

There’s more- but that’s why it’s called a “browser”, eh? By all means, browse!

As for why it took so long to make public- remember this conference occurred under non-disclosure agreements. Takes some time to unravel things and get all the requisite hoops jumped through.

He’s Gone

BobKjelgaard — Fri, 26 Jun 2009 05:15:16 GMT

It has been a tough week, and a busy one. I’m getting ready to leave on a vacation, and trying to get all those last-minute things done.

While my last post discussed the genesis of what I call the Configuration Agent and the WDF QA Universal Setup Job, this week’s focus has been entirely on another of my oddball notions- it started as a desire to build a more precise fault injection engine for testing the KMDF loader. With that California background, the project name was easy- Fault? Why, San Andreas, of course!

At a high level it was a programming model built around IAT filtering of calls from a driver into the kernel. I wanted precision, so I could specify behavior by thread and IRQL, program each DDI independently, allow a driver to do its own filtering [i.e. get a callback with context for a DDI call under given conditions]- and with enough flexibility and locking behavior that the behavior could be reprogrammed within such a callback. For instance, in one working application of mine a filter on calls to IoCallDriver from the KMDF runtime were routed on specific threads to a callback in the test driver which then turned off all call logging, as it meant the call was leaving it’s stack location- I was using the logs from calls to various resource allocation DDI to make sure the framework did not allocate resources under conditions in which we contractually would not fail- so I wanted to stop logging before I called IoCallDriver and to restart it after it returned [I knew the driver handling it was synchronous- otherwise I would have to hook the completion side of things, of course].

So the concept went well beyond fault injection, but it’s always been a side-line. My initial tests wound up being very white box, and a refactoring of the loader code meant a similar rewrite of the test was necessary, but there was no time to do that… But with Windows 7 winding up [since we’ve announced dates, that can’t qualify as a secret anymore], I have time to go back and try to at least do a few new things beyond mere maintenance. Such time is rare and fleeting, so I’ve been busily making the most of it.

So today was busy- I worked a bit of overtime on a COM automation server (in process) exposing a scriptable interface to my engine for some of the other team members to use, and got it at least working well enough to demo. Still have my packing to do, and dinner to make and so forth, so home I headed.

Then I turned on the television, and immediately heard the news about Michael Jackson. Whoa…

On those old tapes I’ve been scraping for my archives I’ve got many covers of his material from the time I was playing in bar bands: Billie Jean, Thriller, Beat It- and more. I’ve not posted them because frankly I never thought I did any of them justice- a lot of my material from those days has rough edges [anyone who listens to what I have posted can hear that], but these were just not good enough. But add me to the billions of fans, then and since, sure…

The other thought that came early on was that in my previous post I mentioned having learned to keep some of my more controversial opinions to myself- one of those was intense skepticism about the molestation accusations. I knew I wasn’t likely to sway anyone there- so why bother?

He’s gone, and he was younger than I am- scary stuff. Life is fleeting, and it was an unexpected reminder of that.

But, it’s time to pack my CDs and games and controllers, write those last few checks for bills coming due during the vacation. eat that dinner, and get ready for a night’s rest prior to that final day of work, so another slapdash post, and back to work…

There was some offsetting good news today- besides finding out I really do still remember enough about COM to roll out an in-process automation server on the QT- without using ATL of course, where would the challenge be in that [answer: in learning something new for a change- you old dog!]? But of course it falls into that semi-confidential area where I’ll just have to sit on it for a while [not that it’s likely to shatter the world if revealed anyway, but circumspection is worthy of practice even for an old blowhard like myself].

I’ll try to do something more substantial someday- these days I’ve just not got the time to really write anything too terribly technical. My apologies to those diehards who still try to read this awful stuff I’m churning out of late…

Crazy is as crazy does

BobKjelgaard — Thu, 18 Jun 2009 23:57:00 GMT

When I was an undergraduate at CalTech, my student residence- Ricketts House- had a tradition where we relieved some of the tedium and stress of the annual election for house officers [President, Vice-President, Social Vice-President, Secretary and Treasurer was the lineup if memory serves me well] by nominating and electing people to “Minor Offices”. [Looking at the Wikipedia link, I see I probably fit the Scurve profile more than I had realized- I’ve had many problems since with holding controversial and unpopular opinions- now that I’m old I just keep them to myself more than I used to].

This process was largely temporized on the spot, and usually fueled with socializing, various recreational substances and a great deal of levity. Some offices had “duties”- for instance, the RLPL (Resident Lecturer on Pornographic Literature) was in charge of the acquired collection of said “literature”- and in self-defense I’ll add that this had been an all-male school until a year or two before I arrived. Others (such as the “House Christian”) were largely symbolic- often made up and discarded as seemed appropriate to the assembled wits at the time. [To put the RLPL in context, I remember utilizing the service to read a Playboy interview from the mid-60’s, of Jerry Garcia, so it wasn’t always the porn that was the draw].

My purpose for the aside and the title of this post combine. During my tenure, while I never held an elected office, I was elected several times to one particular office- “House Crazy <a word I cannot use on MSDN, or email, or say on public broadcast- I suppose “F-bomb” comes close enough>”.

I’ve always had a quirky sense of humor, and I displayed it a lot during those years, so I suppose I deserved the title- things like taking a philistine swipe at some of the campus art being acquired by building a “sculpture garden” that included discarded urinals I dragged up from the basements [I swear you could find anything down there, if you looked long enough], stacks of chairs, umbrellas, discarded lumber- all tastefully arranged for display of course- and I really liked building things off-the-cuff with precarious and eccentric balance to them, so there was some “art” involved, in at least one sense of the word. The darbs sort of liked it- I doubt the Master of Student Housing liked it, since we titled the display with his name. Never can tell, though- some people actually can take a joke [I usually try to]…

Another factor was being a pyromaniac- I can remember holding a drinking glass with some burning gasoline in it [not a lot- maybe a few millimeters, before it ignited] at one point [only because I had to find a place to toss it, to avoid injury to myself- with a bunch of people crowded around it took a while to find one- even at that somebody insisted I was throwing it at them- not the case- at that distance I wouldn’t have missed…], and at another being asked to detonate an explosive mixture someone had painted on an exterior window and then decided they didn’t feel safe either leaving or igniting- in normal form, I used a long rolled up piece of newspaper, ignited one end, set the other against the frame, and then ran, barefoot as always, across the roof tiles, which were often littered with shards of gritty nasty stuff, but my feet were long since tougher than that [of course this wasn’t on the ground floor] to get out of range. But it was a bit of a letdown, that time- not sure it even shattered the window- where’s the fun in that?

I suppose I did earn the title [and I believe I paid for most of the damages, as well, but that would be even more off-topic stories]…

So today I find other outlets, and wanting to write something before I go on vacation without having posted anything in almost two months, I’ll pick one at random and see if it’s worth talking about.

Endless Dogfood

“Dog Food” is what we refer to new and unreleased software as, here at Microsoft. Using it means you are helping those who are developing the software as an adjunct tester, and being a helpful sort [as are most of the other ‘softies I know], I participate when I can. But it brings its issues, particularly if you are using it for real work- and if you’re not using it for real work, then just how is it you are helping?

To test KMDF and UMDF, we use a lot of tools- many of which you know. But to maximize the potential return of our test effort, I want to “dog food” as much of it as possible. That’s not as simple as it sounds- these are just a few of the issues experience has shown me:

most products under development are built frequently, with multiple versions shared out from their “release shares”.
The form in which those shares are presented [in terms of paths to specific processor architectures, for instance] has some variations.
Sometimes when a product goes bad, you have to get it from somewhere else, such as a private build.
Or you may have to stick with a previous fixed version for a while.
But most of the time you want to get as much as possible from the same place, because some people won’t even talk to you if they see you doing much mixing and matching.
If that mixing and matching includes things that go into driver packages [and it does for us], then cataloging and signing becomes an issue.
Worse yet, since we do our own servicing, we sometimes have to run signed content used to test our older products whose signatures have expired and are no longer valid.

We used to make horrid ad-hoc changes to handle these things- breaking paths down into “parameters” banged together in explicit paths to form a full path, for instance. Done differently for each automated test, of course. To get a test run by someone else, you had to communicate a complicated list of these “parameters” to whomever was running it, and they had to get all of them right.

One day I decided I’d had enough, and out of that was born one of those custom designed solutions that is probably as quirky as the problem it solves. Not the global sort of solution Microsoft sells to people, but the kind of stuff a lot of people in IT or support roles have to build just to survive.

In the amalgam that is my work life, this became part and parcel of a task I was doing at the same time, which was to bring some commonality to all of our test setups, so it was easier to look at a machine on which a problem had been reported and know what you were actually looking at.

Simple Mind, Simple solution

So I started with some basic rules:

One setup job goes out, finds all the tools we use, copies them to the test machine, and sets them up.
Individual tests never go to the network for anything- if someone needs something, we add procurement of that to the setup job.
The setup job will own making sure that everything is signed.
The setup job is going to be very smart- no more complicated instructions. Our most common test requirements should be its defaults, and if we need something run outside our group, all that should be needed is one reasonably mnemonic string value, used as a parameter, to identify the machine configuration we are testing.
The setup job is going to log where everything that is pulled off the network came from in easily accessible form, including a history of such locations from previous runs of that same job if it gets run more than once after a new OS is installed.
It will be easy to tell it to get the most recent version of anything we use- but it will also be possible to tell it to get a specific version. Our team will own making sure that knowledge is there in a fashion that we can meet bullet #4 above. Bet you thought I’d forgotten I’d mentioned dog food earlier, didn’t you?
Everything we use is to come from a supported source- not some file someone stuck on a share somewhere that had no clear ownership [or symbol files, usually…].
Experience says there will always be some need for temporary exceptions to that last one- so put all those eggs in one basket and watch that basket closely.

Well, WTT does provide some facilities aiding in what we wanted, but it was not at all close to everything we needed. So it was time to invent.

First, I broke the job into phases:

An Analysis phase where it takes that single parameter, along with some database of recorded setup knowledge, and determines for itself [by checking the appropriate shares] if all the pieces needed are available, and if they are, which specific places to get them from. The results of this phase are “cast in concrete”, and the subsequent phases will use this information and only this information- not go looking again [which causes problems when things change between the analysis and when they get around to looking, for instance].
A “staging” phase where anything that isn’t already on the machine is copied to the machine. The places copied from are determined by the preceding analysis. If it starts out on the network, and you need it on two places on the machine- this phase has to copy it to the machine first, and then from one place on the machine to the other. It is also not a problem if later jobs also do such copying- as long as they never go to the network again.
Finally, a “configuration” phase where tools are installed [from MSI or such already copied in the staging process], registries tweaked, and the machine is made ready for testing. At this point, any of our jobs should be able to be run in any order [and one benefit here was we can now test either KMDF or UMDF where we had previously been one or the other].

That done, it was time to make it a reality…

Maybe not so simple, after all

First of all I needed that way to record values so they were usable both by WTT and by scripts and command line apps and whatever else without too much intervention. Some research and experience aforethought turned up the answer, and it was almost as old as I am- the machine’s global environment is accessible to all of them, it can be updated with simple registry operations, and you can even make the changes immediately recognizable to all the pieces that really count by broadcasting a Windows message when you finish the updating [Wei indirectly clued me in on the last part- he observed that if he used the control panel the changes were immediate, even on open command prompts, but not after using one of the early versions of this code].

Then there perhaps would have been the signing issue, except that we had been working on that one- we had found that we could catalog and self-sign everything and that seemed to cover everything [eventually this has had some holes, on pre-Vista OS, but I’m working on them when I can]. But we were using a copy from some WDK stuck on a share- see rule #7 above!

So, I now needed some way to encode directions about where to find things how those directions changed under various circumstances, and how they differed under certain test requirements, and something that would take that information and create those environment variables. It also had to meet my logging and reporting requirements.

I wound up with another ancient solution- Windows private profiles [aka INI files]. As I’ve already taken up too much time composing this, I’ll have to continue the story at a later date…

Exceptions rule!

BobKjelgaard — Sat, 18 Apr 2009 19:24:55 GMT

I’ve decided to revert to storytelling mode today. So sit right back and you’ll hear a tale, a tale of my fateful trip (or click something useful- you’re the one deciding, and I’m the one typing to satisfy whatever inner demon has driven me to do this today)…

It began when someone found a bug in the static version of KMDF- the one almost nobody gets to use and which we have been trying to discontinue any support. But people were still using it, and we had no clear migration path, and there was a bug. So we decided to begin testing it. Poor Neslihan got that task (well at least it wasn’t me, this time). Soon we had added a static KMDF driver to our list of test drivers used in our daily automated testing.

Then, of course, we got the bugcheck. As often happens, I played “Johnny-on-the-spot” and (virtually speaking) leapt onto the remote debug session accompanying the report. Buffer overflow? In FxDriverEntry? But only on Itanium (I checked the other architectures that ran the same test, of course- no problems)? What in blazes is this about?

Well, the method is well documented, and not all that hard to understand- a “cookie" gets initialized, it gets stored to the local stack frame in such a way that code overflowing buffers in the stack frame will overwrite the stored copy. On exit, code gets called that checks that value to see if that happened. It’s a little more complicated than that, and I’m not going to be precise, because part of that complication is related to securing the method against the sort of people who like to exploit buffer overflows [so I’m not going to make the method clear for them- they can do their own research without my assistance]. The bugcheck analysis will even tell you the two values- expected and actual, making it easy for people who really think overflows are a bad idea to make a quick guess as to where the overflow came from.

But the circumstances seemed suspicious, so I went and read the source code- H’mm- there’s a default value the cookie is initialized to… Interesting- that’s the value we have- but it doesn’t match the expected value.

At which time, something I’ve known for years hit me like the proverbial sledgehammer- our entry code calls GsDriverEntry (which supports the stack probes inserted by the GS compiler switch, hence its name) when it finishes. GsDriverEntry initializes the cookie, which means that that was why the values didn’t match. Like the sword of Damocles, this had been hanging over our heads for years- the first time the compiler decided to do stack probes in our entry code, everything would break. Ouch…

I’ll leave a bunch of story out here- things got “interesting” at that point, but the problem eventually got solved, both for us and for other people facing similar issues, as well. Short answer is that everyone now initializes the cookie right at the start, just as should happen…

But a while back I had occasion to tweak the build process for the WdfVerifier WDK applet, and afterwards I ran it briefly to make sure I didn’t break anything in the process. Oh, joy of joys- NONE of the 1.9 client drivers are being properly identified. They are all identified as “Inbox”, which is what I do when the client identification method I described last year fails on me.

Already beginning to panic, I think- did someone change FxDriverEntry, and I didn’t even notice it? So I go to our source control system, and look at the change history. The most recent change is the fix for the problem with the stack probes (yes it occurred on static, but what scared me then was it could have happened to any version, because were weren't doing things properly). But that just calls library routines to initialize said cookie… Oh, blazes, those routines must be calling an import I wasn’t accounting for!!! Why??? Because I NULL the IAT to force access violations, and I handle the exception by giving up on getting the true version, and fall back to calling it “Inbox”- which is, after all, exactly what I am seeing. Oh, well- so much for my having thoroughly considered all the test and product consequences of that change when it was made…

Well, easy enough to find out what that import might be. One nice trick Ilias showed me is that you can open any binary in WinDbg as a crash dump, and happily resolve symbols and disassemble code. So I pick a random driver on my dev box, and do so.

What, no imports? But, but--- hmm- it uses a fixed address (load of a 64-bit immediate value into RAX, since my dev box is an X64, thank you- you x86 dinosaurs can keep your wimpy processors). What’s with this?

The answer lies in wdm.h, of course- KeQueryTickCount turns out to always be a macro- on x86 and IA64, it accesses KeTickCount [which is an import, and if you followed my earlier tale, it’s clear adding an import to my existing hack-o-matic mechanism is a trivial task]- but on X64, it accesses an essentially hardcoded address in a “SharedUserData” area. This snippet is from wdm.h, so you can see for yourself…

#define KI_USER_SHARED_DATA 0xFFFFF78000000000UI64

#define SharedUserData ((KUSER_SHARED_DATA * const)KI_USER_SHARED_DATA)

#define SharedInterruptTime (KI_USER_SHARED_DATA + 0x8)
#define SharedSystemTime (KI_USER_SHARED_DATA + 0x14)
#define SharedTickCount (KI_USER_SHARED_DATA + 0x320)

#define KeQueryInterruptTime() *((volatile ULONG64 *)(SharedInterruptTime))

#define KeQuerySystemTime(CurrentCount)                                     \
    *((PULONG64)(CurrentCount)) = *((volatile ULONG64 *)(SharedSystemTime))

#define KeQueryTickCount(CurrentCount)                                      \
    *((PULONG64)(CurrentCount)) = *((volatile ULONG64 *)(SharedTickCount))

My. my. my- this is a problem- no import I can just tweak to point into my code. For a plus, at least the value can’t change (if it did, existing drivers would, after all, cease to work)- well, at least not easily, so I’ll save any remaining paranoia about that for another time. But I’m going to access violate trying to access that address in user mode (not to mention the reams of experimental evidence I had just accumulated by noticing the problem in the first place)…

So in reading about exception handling, a phrase happens to catch my eye- putting it in my words, it says that when continuing an exception, you can alter the context record supplied to you with the exception (and I know this process well, but that’s perhaps a story for a different time). Whoa- that must mean I can change the contents of the registers programmatically, and then tell it to repeat the failed instruction! Now, I can’t find that illustrated in any of the samples the material I’m reading points me to [nor could I find it an an internet search, although the latter was by no means exhaustive]. But that HAS to be what it means, right?

So I start coding- first the half dozen or so lines needed to handle the KeTickCount import. Then an exception filter [only for AMD64, of course]. The logic is my usual bit of precise work: If it is an access violation, and if it is a read, and it is a read of that exact address, and one of the integer registers in the context record has that same address, then change that register’s entry in the context record to the address of the proxy I’ve decided to keep in WdfVerifier, and tell the OS to repeat the failed instruction. I began with RAX, because after all, that’s what I had seen in my investigation, and it seemed the most likely place for a while, but I added the whole set since under the circumstances, it seemed unlikely to do any harm. Anything that didn’t match that exact pattern, and I gave up- just execute the handler, which does nothing. The attempt to run the driver entry code to extract the client version will fail, but WdfVerifier keeps running, and the machine itself is still quite safe from my hackery.

Worked the first time, it did (not counting my usual compiles to get rid of typos). Problem solved. Total work time from start to finish- something like 5 or 6 hours- good enough- after all, I did have to do some research… Of course, there’s always somebody who can do it better, faster, and quicker- and they usually jump out of the woodwork if I start talking about things in that fashion, but it’s my story, so I’ll brag now and regret it later…

Exceptions do indeed rule! I love it when a plan comes together…

A glimpse now at handling the new import…

    //  The descriptors are an array of entries per module, each terminated with an all-0 entry.
    //  Each entry has an RVA for the module image name, and a second for a 0-terminated list of RVAs to the structures used for
    //  resolving names of exported entries from the module (in loading these get resolved to real addresses and are plugged into
    //  the loaded image's import address table in the same order).  So we now know how to write an image loader if we need to...

    //  First, find the descriptor for the KMDF loader, and quit if it isn't there.
    DWORD   StringCopy = (DWORD) -1, WdfBind = (DWORD) -1, InitEvent = (DWORD) -1, TickCount = (DWORD) -1; 

    //  Then look for the Bind and Unbind entry points.  Case counts!

    enum {OutdatedTechnology, HasBind, HasUnbind, OneCoolDriver};
    
    BYTE        TargetIdentification = OutdatedTechnology;

....

            else if  (IsKernel)
            {
                if  (0 == strcmp("RtlCopyUnicodeString", (PSTR) NameDescriptor.Name))
                    StringCopy = ImportsIndex;
                else if  (0 == strcmp("KeInitializeEvent", (PSTR) NameDescriptor.Name))
                    InitEvent = ImportsIndex;
                else if  (0 == strcmp("KeTickCount", (PSTR) NameDescriptor.Name))
                    TickCount = ImportsIndex;
                
                ImportsIndex++;
            }
.....


        DWORD       IATSize, RelocationsSize;
        LONGLONG    OurFakeTickCount = 0x8badf00ddeadbeefI64;   //  ersatz Tick count         
        PVOID*  ImportAddresses = (PVOID*) 
            ImageDirectoryEntryToDataEx(DriverImage, TRUE, IMAGE_DIRECTORY_ENTRY_IAT, &IATSize, &Unused);
        
        PIMAGE_BASE_RELOCATION  Relocations = (PIMAGE_BASE_RELOCATION)
            ImageDirectoryEntryToDataEx(DriverImage, TRUE, IMAGE_DIRECTORY_ENTRY_BASERELOC, &RelocationsSize, &Unused);

        if  (NULL == ImportAddresses)
        {
            FreeLibrary((HMODULE) DriverImage);
            return  true;
        }

        //  Plug them in here, and we are good to go!

        DWORD   OldProtect;

        if  (!VirtualProtect(ImportAddresses, IATSize, PAGE_READWRITE, &OldProtect))
        {
            FreeLibrary((HMODULE) DriverImage);
            return  true;
        }

        memset(ImportAddresses, 0, IATSize);

        ImportAddresses[StringCopy] = FakeOutStringCopy;
        ImportAddresses[WdfBind] = CollectVersion;
        if  (InitEvent != (DWORD) -1)
            ImportAddresses[InitEvent] = FakeEventInit;
        if  (TickCount != (DWORD) -1)
            ImportAddresses[TickCount] = &OurFakeTickCount;

        VirtualProtect(ImportAddresses, IATSize, OldProtect, &OldProtect);

So for some other glimpses, this is one part of the other hackery (complete with my elementary-level annotations):

class AllKMDFDrivers
{
    MachineKey&             Owner;
    KMDFDriverList&         InstalledKMDFDrivers;
    LoadedDrivers&          CurrentlyLoadedDrivers;
    LoaderDiagnosticsFlag   LoaderFlag;
    String                  RuntimeVersion;
    DWORD                   Major, Minor, Build;

    bool            ServiceUsesKMDFLoader(__in PCWSTR ServiceName, __in RegistryKey& Service, __out MyBindInfoAlias& Binding);
    static void     FakeOutStringCopy(__in PVOID, __in PVOID);  //  Fake RtlCopyUnicodeString entry
    static void     FakeEventInit(__in PVOID, __in ULONG, __in ULONG);  //  Fake KeInitializeEvent entry
    static ULONG    CollectVersion(__out MyBindInfoAlias& BindingOut, __in PVOID, __in MyBindInfoAlias& BindingIn, __in PVOID);
#if defined(_AMD64_)
    static LONG     Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount);
#endif

While the rest of it looks like this (and again, this is partial)- yes, feel free to hate my stylistic indifference to the herd- perhaps I’ll get fired for this, and all can breathe a vast sigh of relief…

First, the code that calls the filter

        MangledDriverEntry  CrashAndBurn = (MangledDriverEntry) ((PBYTE) DriverImage + EntryRva);

        __try
        {
            CrashAndBurn(Binding, &Binding);
        }
#if !defined(_AMD64_)
        __except(EXCEPTION_EXECUTE_HANDLER)
#else     
        __except(Filterx64Exception(GetExceptionInformation(), OurFakeTickCount))
#endif
        {
        }

and now our filter:

#if defined(_AMD64_)
/**********************************************************************************************************************************

LONG    AllKMDFDrivers::Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount)

Ahh, the joys of low-level intervention by the truly incorrigible!  On AMD64, we will get an exception when the code tries to get
the tick count.  This value resides at a known address (which cannot change because if it did, all existing drivers would then fail
to work- although I suppose someone could resort to what I am about to do- bring it on, I'll see if I can keep up with the absurd
arms race).

So, first, verify from the exception record that we are getting an access violation of some sort reading that known address.  Then
see if one of the integer registers has that address (start with RAX, since that's where it currently is, and I bet it doesn't
change much).  If it does, change it to point to the value given to this function (which resides in WdfVerifier, since I made it
a reference, and will let the compiler play enforcer), and then continue execution.

In all other cases, execute the handler, which will do nothing (meaning the version will continue to be unknown).

**********************************************************************************************************************************/

LONG    AllKMDFDrivers::Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount)
{

    //  cf definition of SharedTickCount in wdm.h
    static const DWORD64    KnownTickCountAddress = 0xFFFFF78000000320UI64;

    if  (NULL == ExceptionInfo || NULL == ExceptionInfo->ExceptionRecord || NULL == ExceptionInfo->ContextRecord)
        return  EXCEPTION_EXECUTE_HANDLER;

    switch  (ExceptionInfo->ExceptionRecord->ExceptionCode)
    {
    case    EXCEPTION_ACCESS_VIOLATION:
    case    EXCEPTION_IN_PAGE_ERROR:    //  Unlikely, but what the heck...
        if  (2 > ExceptionInfo->ExceptionRecord->NumberParameters)
            return  EXCEPTION_EXECUTE_HANDLER;
        break;

    default:
        return  EXCEPTION_EXECUTE_HANDLER;
    }

    if  (EXCEPTION_NONCONTINUABLE == ExceptionInfo->ExceptionRecord->ExceptionFlags)
        return  EXCEPTION_EXECUTE_HANDLER;

    if  (EXCEPTION_READ_FAULT != ExceptionInfo->ExceptionRecord->ExceptionInformation[0] ||
        KnownTickCountAddress != ExceptionInfo->ExceptionRecord->ExceptionInformation[1])
        return  EXCEPTION_EXECUTE_HANDLER;

    if  (KnownTickCountAddress == ExceptionInfo->ContextRecord->Rax)
    {
        ExceptionInfo->ContextRecord->Rax = (DWORD64) &FakeTickCount;
            return  EXCEPTION_CONTINUE_EXECUTION;
    }
    else if  (KnownTickCountAddress == ExceptionInfo->ContextRecord->Rbx)

The rest I leave as an exercise to the reader, having already put all 0.378 of you through the treadmill today…

Using C++ in a KMDF driver part 1- a pattern for using contexts as objects

BobKjelgaard — Thu, 12 Mar 2009 16:36:59 GMT

This is an article I’ve started probably close to a dozen times since I started this blog, but never published. In part because of all the heat the topic of using C++ in the kernel generates, and the rest perhaps because of my reaction to that heat.

So I’ll get one thing off my chest at the start and perhaps that will be enough to let me proceed.

I’ve been writing drivers with C++ in the kernel (or the Win 3.1/9x/ME VMM) pretty much since I began using the language (in the very early 90’s). I routinely use paged code (paged data not so much- I’ve never had designs where there seemed to be any benefit to it). That spans more than a decade- in fact, its close to two decades now.

I’ve NEVER had a problem with the issues raised in the paper. I’m not even going to link to it. If you want to find it- it isn’t hard to find. On top of that, in a determined effort to find people who had, I didn’t find many- and the one clear case I did find was cured with a #pragma that would have made sense if you were a C programmer and had a basic understanding of where VTABLES and such get emitted to. So I personally have a sense of mismatch between my own experience and the strongly worded severity there.

I’ve been told that if I persist, I’ll deserve all the paging problems I get [and that nobody will help me with them]. Well, I do get them (always have), and I’ve been developing long enough in this environment that I don’t really need anybody’s help to find the root cause of a bug like that. The causes are always the ones I remember- acquiring a spin lock in pagable code- making a routine pagable that can be called at elevated IRQL- all the usual ways to screw up in that other language that is the hallmark of a true first-class kernel developer. But none of them had anything to do with my choice of programming language. Not once. Not saying that it never happened across all those years, because in the early ones, the ability to catch a bug like that was severely limited- there was no Driver Verifier, no static analysis tools. So it may not have been noticed early on. But now that I have them, it’s still not happening…

Now there can be plenty of reasons for that- but I look at what people take away from the paper, and I do a lot of the things they think aren’t safe. I use polymorphism and inheritance freely. I don’t use multiple inheritance a lot, but I have used it and have not observed problems arising from its use. On the other hand, there are common features that I don’t use as a matter of personal preference or style that may bear on this. I don’t expose the implementation of functions in header files (meaning the compiler is not going to start out by inlining them and then give up later, dumping them in some unintended segment) with the exception of trivial accessor functions. I don’t use templates (I’m not sure they were a language feature when I started, but at any rate, during my learning curve I never needed them, so while I can handle code with templates, I don’t use them myself). I almost always code my own constructors, destructors, copy constructors and assignment operators (I usually have to- I prefer references to pointers and if you have reference members, the compiler can’t generate default code for most of those routines).

So now that its clear I’m not going to walk the company line on that topic [the opinions I expressed ought obviously to be clearly my own], I’ll proceed to something more useful…

Leveraging the KMDF Object Model

KMDF provides a nice object model, with managed lifetimes and one of the most delightful observations I had in my first KMDF driver was that it was easy for me to blend this with my usual coding patterns.

As an aside, I’ll note that I never use sample code to learn anything. I take the reference materials and code to them. If there are samples, I treat them as the last resort- and I will deliberately change as much as I can of them [in part to see what knowledge of things that can go wrong wasn’t explicitly represented in the sample]. I judge the quality of the reference material by how rarely I have to look at a sample to figure something out [yes, I don’t find much reference material i would call “good”].

So my first KMDF driver was a software bus driver, and I didn’t go near toaster in writing it. In fact it was one of those “ActiveX” (OLE automation) test drivers I mentioned in our DDC presentation. Now we had some samples for them, too- and not surprisingly, virtually no reference material. The samples were fastidious about one thing- all the COM parts were in C++ (not a lot of choice there)- but all the parts using KMDF were in C- precisely because of said paper. I might add that this was even though no attempt was made to mark any of the code or data pagable in those samples. Well, I’m a stubborn <expletive of your choice here>, so I decided then and there I was going to write the whole driver in C++ in spite of the objections I received. I was still in my “trial period” so they could always fire me if they wanted to, but the job market was good enough at that time that I was pretty sure I could find something…

Back to the proper topic- one fine thing is that most of the KMDF macros are agnostic enough they can handle at least straightforward C++. That was one of the happiest discoveries of that time.

So the basic pattern as I use it:

Declare a static function in your class that takes a WDFOBJECT as input and returns a pointer to an object of your class.
Declare a class-specific placement form new operator that takes as its additional input the type of handle you expect your object to live in the context of (that is, it can be more precise than the preceding function can and you can benefit from stronger typing in C++)
Declare a class-specific delete operator that basically does nothing if you need to have your destructor invoked.
If you have such a delete operator, also declare a static member with a void return that takes a WDFOBJECT as input.
Use the WDF_DECLARE_CONTEXT_TYPE_WITH_NAME macro to get the compiler to write that first function for you.
Have your new operator implementation use the first function to return the address of the underlying context (you basically ignore or validate the size parameter) from the passed-in handle.
When creating the context, and you need your destructor called, use a WDF_OBJECT_ATTRIBUTES structure with the EvtCleanupCallback set to the routine in item 4.
Code that routine to use the routine in item 1 to get the context address out of the object handle- and delete that pointer. This causes your destructor to be called at cleanup time (which is much more sensible than destroy time) and your do-nothing delete operator will also be invoked (or inlined out of existence if your compiler is any good).

There- you “create” your object when the KMDF object is created (or via a WdfObjectAllocateContext call if you add your object later), and “delete” it when the object dies. But KMDF manages the memory and most of the object lifetime for you. Sure works for me (a lot).

The following snippet is from that first driver (I’ve since dropped the usage of “C” on class definitions in my general drive toward anarchic style). This is slightly convoluted because I have logic allowing only one instance of a device with this driver- so I’ve deliberately intermingled driver-level and device-level usages (always pushing those boundaries- but I think that’s a good way for an SDET to think). I’ll admit this is slightly doctored (I removed some things related to the COM technology as that I can’t disclose, plus I tried to include support for the standard bus interface and that just complicates things without illustrating this method), but it should show I practiced what I am preaching…

class CTargetTestBus
{
    static WDFDEVICE                Owner;              //  WDF device that "owns" this bus object

    static void*    operator new(size_t size, WDFDEVICE OwningDevice);
    CTargetTestBus(WDFDEVICE OwningDevice);
    ~CTargetTestBus() {}

    //  Private callbacks (ie, accessed from within this class' code)
    static EVT_WDF_IO_QUEUE_IO_DEFAULT      OnIoDispatchDefault;

    static void operator delete(void*) {}
public:

    NTSTATUS    NewChild(LPCWSTR Name, int InstanceID);
    NTSTATUS    RemoveChild(LPCWSTR Name, int InstanceID);
    void        DoneWithBus();

    static CTargetTestBus*  GetThisTargetTestBus(__in WDFOBJECT Object);    //  WDF macro writes this code
    static CTargetTestBus&  GetTheBus(bool& BusPresent);
    
    //  Driver callbacks (public, because used in DriverEntry)
    static EVT_WDF_DRIVER_DEVICE_ADD        OnDriverDeviceAdd;
    static void             OnDriverUnload(IN WDFDRIVER Driver);

    //  This one is accessed from a member in CChildInfo
    static EVT_WDF_CHILD_LIST_CREATE_DEVICE OnAddNewChild;
};


// Macros to get the context
WDF_DECLARE_CONTEXT_TYPE_WITH_NAME(CTargetTestBus, CTargetTestBus::GetThisTargetTestBus);

Now playing: Me (that old recording of Johnny B Goode again)!

Automated Jobs and Library Jobs

BobKjelgaard — Thu, 12 Mar 2009 14:30:00 GMT

I received an IM late yesterday evening (my time) from “James” (via the Web) asking what the difference is between an automated job and a library job. Unfortunately, I wasn’t around to answer it (I started work about 2 AM PDT today, and the IM was from about 9:30 PM PDT yesterday), but it’s rare I get asked something I can flip an answer to off the top of my head, so…

I’m assuming this is in the context of the Device Test Manager (DTM) in the Windows Logo Kit (WLK). That is a derivative of a larger internal-use product we call WTT (Windows Test Technologies), so I’ll assume the same answer applies to both, because I’m sure it does (and if I’m wrong, feel free to correct me).

If you’re a programmer, the fastest rough analogy is that an Automated Job is like a C / C++ “main” while the Library job is like a “function” in the runtime library (or some DLL, etc). The important difference here being that a job can “call” a library job and pass it parameters, while you cannot “call” an automated job- you can schedule it (and give it parameters) just like you can invoke a program (and give it parameters) from a command shell- but we don’t have analogs like piping and redirection of I/O etc in this execution environment.

They are a lot alike- both can be scheduled and given run-time parameters, and possess fairly similar capabilities.

So why not make all Automated jobs Library Jobs? One strong reason is that Automated Jobs can be constrained [have a set of circumstances spelled out to detail the requirements for running the job- one common one in device testing is the presence of a particular kind of hardware, for instance] while library jobs cannot. Another one I use occasionally perhaps ventures into the realm of the political- making a job an automated job so that it doesn’t wind up being called as a library job if you expect such a usage to have untoward consequences.

The best practice, especially initially, is to make most jobs library jobs- this gives you the most flexibility as you combine them later. You can work on them in pieces and then combine the pieces fairly easily.

Well, James- if you’re still searching for that answer, perhaps this will help. If not, maybe it will help someone else later.

Well, yes and no...

BobKjelgaard — Fri, 23 Jan 2009 10:46:00 GMT

I got involved (probably quite foolishly) in a discussion on the NTDEV mailing list that really wasn't much of my business yesterday, but I thought I'd expand a bit on my thoughts there, anyway.

It probably sounds in one of my first posts there that I'm completely disagreeing with Peter Viscarola, although in fact, I don't. I just think there's a lot more to the story than that- both the reasons why things are as they are, and the broad general effect it has.

The Purina Effect

I think I'll structure the rest as a Q & A with some imaginary and perhaps occasionally hostile interviewer- we'll see where that takes me. Not that there's a need for this, but it's my blog, so I get to make these choices...

So why do so many people at Microsoft run the latest of everything?

Because we hire a lot of technophiles- we are people who like new technology and believe innately in its transformative power. We know such people do well in our environment, and make great contributions. Its a good trait to have if you're seeking a job here. You'll get picked on sometimes for using Office 2003 for the same reason some people will give you grief about out of date fashions, or a stodgy car, or... So the desire for the latest and greatest is inbuilt.
Because so many of us are involved in developing new products, we understand quickly the economics of their test, and that includes the idea that finding problems earlier is fixing them cheaper. In part due to that first bullet, there's usually a ready pool of people available to help test your new product, and that's a good thing. Most of the time, the problem is keeping them away from things before you're prepared for their input. This is also good over the long run- we call it "dog food" [short for "eating the dog food"], and it is well-baked into the corporate culture. It occurs on voluntary and ad-hoc bases and in planned deployments. We always use things before you do. We always will. [And unfortunately we will still miss things- there is no perfect test process, at least, not that I know of]. I began running Windows 7 within weeks of the release of Windows Vista, and used it for most of my work (and most of my blog posts, except at times like this when I post from home). Unlike most driver developers, I am used to an OS whose code changes each and every day, and exists in multiple forms at nearly the same time.

Just looking at the last couple of years, I've run beta or earlier versions of a couple of OS, Office, Windows Live, Halo 3, and Halo Wars. I didn't have the big XBox Live update because I knew I wouldn't have time to work with it [but I know people who did]. I'm not particularly on the bleeding edge, usually, even with all that.

Why do things change and old ways no longer work?

Sometimes, it is much as Peter said [with my demurrals above flavoring it a bit]- a lot of focus on the new. But many times these are done deliberately, and "cool" [in the sense of some form of hipness] may occasionally be a factor, but not one that gets much weight [sticking to the OS- different products, different markets, differenf factors].

Most of the cases I can think of were the result of analysis of real data- watching and observing how things are really used. Looking for the problem areas- trying to find ways to minimize the number of steps to accomplish the most important and the most common tasks. Making things work better, and more easily. I don't think that should be news- I expect several of the Windows team blogs say so directly or indirectly. Now we think of that as "cool", but that's not the sense the word was being used in in that thread.

But we don't always know the full impact of changes sometimes, and it never hurts to check things in the real world. That's one reason we have beta tests. The TARGETPATH issue is (in my opinion) something we will take as a lesson for the future, but I also think this is proof the beta test process works. Would it have been better for it to be this way in the final version? Was the entire beta so good that the only conclusion was that we would not change anything from the feedback resulting from the beta, making it some sort of trial balloon we'd pull back, buff up, and then reissue as the final thing?

Is your "love of the new" [I won't repeat calling it the Purina Effect, although perhaps they'd be happy I find the name mnemonic for pet food] the sole reason you miss down-level breaks?

No, we also miss them, in my opinion, because:

We develop blind spots in our test procedures and our thinking about how our products are used. Most test scenarios have large variability and we have to narrow down the problem spaces to develop workable solutions. Sometimes this process leaves holes. More on that in a minute.
Peculiar to this case is the mechanisms through which our build tools reach the WDK. As I'm sure has been discussed on the Engineering Windows 7 blog, we put a lot of time and effort into improving our internal processes, and that definitely included how we build Windows. The new mechanisms for binplace versus the old targetpath mechanisms has been a big win for us in terms of working with the source code. But to minimize the impact to product teams, a lot of the initial legwork for it was done by the tool developers, and pushed out to us along with the tool changes. I did some rework in our test code [things I privately had but weren't part of the main Windows source], so I was aware of some of this work, but I suspect that for most developers, this just happened [and that's good- it saves a lot of R&D money when things work that way]. This "magic elf" phenomenon may have helped develop something of a blind spot.

Did you, personally, test the WDK at all?

Actually, yes [and it has its own test organization, as well]. I and other members of our test team have installed it, built samples, run them, read the documentation and the source code, used the tools, and filed bugs as a result of what we've found. That's happened more than once. Periodically we have held what we call "bug bashes" where everyone in the organization that includes our team goes to an out-of-office location, splits up into teams, and competes to see who can find the most bugs, or the most interesting bugs, and so on. But no, we didn't take SP1 DDK code and run it against the latest WDK [and there are multiple reasons that wouldn't work for WDF, anyway]. But I wouldn't be surprised to hear someone's considering that after all of this.

Is it your fault this happened?

I certainly ask myself that- or at least, if I shouldn't have done more, or paid more attention, and if I had, maybe I could have realized TARGETPATH would be an issue and flagged it before it went out the door. But probably not- it was one of those mostly deprecated things that stayed in the WDK long after its need in Windows proper had passed on. I developed a blind spot.

I bet I'm not the only SDET in Device Platform Technologies (which encompasses the WDK and WDF groups) who's doing that kind of thinking these days, although I certainly haven't conducted a survey on the subject. I said above I'd get back to test holes, and this is a good place for it.

They exist. We prefer when they're complicated enough it makes some sense to have missed them, but we're human enough it doesn't always work that way. In a way, it should be hard to get too big a sense of oneself as a computer programmer. The blasted thing always does exactly what you tell it to do. You often don't tell it the right thing, even when you're sure that you have. But being human, we get that sense sometimes anyway. Maybe I had more of it than I should have when I made my first post on that thread.

Time will tell.

Layoffs?

Not funny. I've been laid off twice in my career, both times as an SDE, and both times in a difficult economy, and this economy is worse than it was either of those times. Nobody I know well is affected, but I am on some mailing lists where affected people made their brief farewells.

I sympathize with their situation and hope for the best for them. I'll add that as far as I'm concerned, whomever hires them gets a good deal, because I can't remember the last time I saw a Microsoft FTE I thought wasn't a good employment catch. I'll also add that I have a lot of faith in the people who made those decisions- I've seen enough to believe it had to be done, and it wasn't done lightly.

So why did you hint at unemployment?

Someone taking that thread and blowing it up into a news story about Microsoft not trusting its own products? Not that it would (or will- I can't predict the future on something like this) get me terminated, but I wouldn't blame anyone but myself if it did. For that matter, I may have said more about internal affairs than I should have. Business procedures (and that includes security procedures) are proprietary information and covered by employment agreements. By not checking first, I may have doomed myself, and that's life [corporate at any size- the principles don't differ that much from place to place].

I don't think I was discussing anything other than recommended best practices, though, so I'm not really as worried now as I was when I first thought about it.

Next time I intend to discuss something [anything] else- that's a certainty.

Adaptive Fault Injection

BobKjelgaard — Wed, 12 Nov 2008 22:18:43 GMT

One of the topics I covered in Ilias' and my joint presentation at WinHEC this year was new requirements coming for getting a WHQL signature for drivers using KMDF (UMDF as well, but this discussion doesn't currently apply to them). One part of those new requirements is that your driver survives a rigorous fault injection from the WdfTester tool. The method used for that is what I intend to discuss briefly in this post.

By "survives", I mean no bugchecks, hangs, or leaks- graceful failure, not fault tolerance, is the goal here.

The method used is one I call adaptive fault injection. The term "adaptive fault injection" is admittedly my invention (and I'm probably the only person that uses it)- but I thought it fit. So this is an attempt to define what I mean by the term.

The Problem

For readily available fault injection, I had two tools available to me in 2006, when I began looking into ways to improve code coverage in the KMDF loader in an automated way.

Driver Verifier Low Resource Simulation- this did boost our numbers, but it took a long time,, and since it was random, it took extensive analysis to find out what was covered in terms of code paths, and the only way to cover a particular fault was to run until you finally hit it. Not very predictable. To me, it was a shotgun handed to me when what I really wanted was a needle.
The always on, but programmable threshold mechanism in the KMDF Verifier. For one thing, it didn't help me with the loader (but I was also looking for generic solutions), but for another, I wanted to be able to say- "I want just this one fault, period". So it was still a bludgeon, and not the precise instrument I desired.

Now, we had experimented with IAT hooks and had even used them experimentally for fault injection [the tool just failed pool allocations]. I could easily see that I could use them for precision, but the result could be a very high-maintenance mechanism- if I had to individually code each fault, the price for the technique would be way too high.

I wanted something that could examine the system under test, find all the failure points, and then cover each and every one of them. Something that would adapt to the system's behavior as it evolved.

Hence "adaptive fault injection". So perhaps the basic idea is less of a mystery?

Feedback is almost always your friend

If you have an IAT hook, you can log the activity through the intercepted call- both the data going into the called routine, and the values returned. You can't see state changes occurring on either side of the hook, but at least you've got a point you can begin at [and this situation reminds me a lot of the days when we were testing circuit boards via their external connectors].

So the basic idea for the first iteration was quite simple:

Monitor all the points you can, and record the activity on them.
Analyze the recorded activity, and determine where inputs going back into the system can be modified to simulate failures.
Count the number of such failures you have identified.
Repeat the original activity as many times as needed, and on each cycle, inject each failure in turn. Log this activity much as was done in the first step [and the logging mechanism should report the injection activity, of course].

WdfTester was under development at the same time as my tool [which I named "SanAndreas" after the infamous fault line running up the West Coast of the USA], and utilized the same concept, although in a different form. My trigger was a simple integral counter, making for a simple loop. WdfTester instead injects a fault on a specific count of a given DDI call. Neither method is perfect [I'll get back to that in a bit], but in highly repeatable traces, there's no real difference between them. By the way, I have no idea if we both had the same idea, or if we discussed this [although we probably did, because when I first began that task I said I wanted this sort of injection mechanism, long before I did the implementing].

So, this provides more precise injection, at the cost of potentially longer run times, but the times are deterministic and computable once the analysis step has completed. But it also requires little programming or reprogramming, as it does adapt itself to the observed behavior of the system under test.

Where this still falls short

At least a couple of places come quickly to mind:

In multithreaded cases, observed behavior may not be that deterministic. In that case, we at least try to inject something, but there is potential for things to be missed, or even for the same fault to be injected multiple times.
The state that cannot be observed may matter- I believe this is primarily a concern when designing the repeatable case you want to inject, or at least this risk can be mitigated by considering the state when deciding what sequence to apply this technique to.
The system may adapt to the faults you inject [for instance a failed I/O may be retried]. This is bad only in the sense that it may prevent you from reaching paths you would still like to reach. One could address this by recursively repeating the analysis and subsequent phases (and then using a more complicated injector- first inject this {series of} fault{s}, then in turn inject these]. Such a mechanism may need to have runtime bounds placed on it to prevent endless recursion...

Still, even with these flaws, I've found the technique even in this most primitive of forms to be a step forward in having a more precise fault injection method available to me in my bag of tools.

As long as I'm typing...

Gee, Patrick- I've played all three [but I preorder at GameStop, even though the price is higher]- which reminds me that I need to pick up Call of Duty: World At War [and a day later- World of Warcraft: Wrath of the Lich King]. Didn't get far in Fallout 3, but one play-through each on Fable II and Gears of War 2.

I've been pulling tunes off some really old cassette tapes [bands I played in during the 70's and 80's- recorded in mono on hand-helds, for the most part, along with some solo practice recordings]- think I'll use that nice USB stick I got at WinHEC to bring a few of the livable ones to the office- see if anyone can figure out which tunes I'm the guitarist or bass player and / or vocalist on... Of course, the poor quality of the audio ought to be a clue. If I find one I can live with, maybe I'll link it somewhere [I have a few I wrote myself, are public domain tunes, or are just instrumental noodling, so I can avoid the copyright bogeyman].

Also, since things are now disclosed- this is what the new 1.9 controls in WdfVerifier will look like.

Now Playing: Grateful Dead Album: Built To Last- Victim or the Crime- seem to end on Dead tunes of late, not that I don't listen to plenty else.

DDC starts in less than an hour

BobKjelgaard — Mon, 29 Sep 2008 16:21:36 GMT

Plenty of good stuff for today- begin with Eliyas and Peter going over what's new in WDF for Windows 7 (or as we prefer to think of it WDF 1.9, since much of it also works all the back to Windows 2000 [KMDF] or Windows XP [UMDF]). It continues later with our first presentation of talks on the "shared secrets" of the Windows Driver Frameworks. The first talk discusses the implementation of the frameworks themselves, while the second presentation (my first of this conference, and my first in about 16 years) discusses how the frameworks are tested. The latter talk will also be one of the final talks of the day.

Tomorrow will include Neslihan's talk about getting a logo for your WDF driver (yes we are adding logo requirements specifically for WDF drivers- primarily for KMDF at the moment). It will include some excellent talks by Praveen Rao about work we have done on WDF drivers in the storage space, and again the final talk (at least in that room) is Ilias and I discussing the WDF coinstallers.

Wednesday we'll repeat all the talks I am in- the shared secrets talks will be back to back in the morning, but I won't catch part 1 that day because I'll be in another room with Ilias repeating the presentation on the coinstallers.

There's so much more, too- SDV, DSF, the innards of driver verifier, annotations for PFD and SDV, new test techniques from my colleagues James Moe and Dieter Achstettler, excellent talks on device installation, many driver technologies (several on printing, I can't help but notice). I could go on and on, except I've got some work to get to this morning [and I'll be leaving soon so I can register and begin my bit of supporting the conference].

Ask the experts session tomorrow night, bowling tonight, space set aside for us to talk with developers before and after talks, labs, panel discussions, Q & A with our most senior engineering staff, and a few surprises I'm not at liberty to mention even now.

My plan for now is to try to attend the whole thing- how well that works depends upon how things go in the lab [we've got some critical work going on, but let's see how the junior staff handles things for a bit], and whether I'm providing any benefit by being there [if I can be more effective working on test tasks in my office, I'll return to it]. Of course, assuming my laptop works there, I may just blog a bit more than usual. Not having a cell phone, it might be a bit hard for people to contact me if they need me back at the office [although they will know where I am, anyway].

So, I hope to see a few of my readers there (both those within Microsoft, and especially those from the outside). Hope all enjoy there time.

Now playing: The acid rock classic jam "Dark Star" from Live / Dead (of course by the Grateful Dead)

Looking Ahead- Conferences!

BobKjelgaard — Thu, 11 Sep 2008 07:23:00 GMT

Since I noticed Patrick has spilled the beans [just before the second question], I might as well cover this topic while I wait for my dinner to bake [I eschew microwave ovens these days- just seems better to let it cook the old-fashioned- if early-to-mid-twentieth century is old-fashioned- way].

I've been through some of my belongings of late and put a few items up on the corkboard in my office- pictures of me from the last several decades, ads from bands I played with in the 70's and 80's, and so on [along with my Master Chief calendar and pictures of my daughter and InuYasha]. Two of the recent additions are from my first time around at Microsoft. One of them, which I showed off recently at a WDF team meeting related to this article's topic, is what I believe was my original Microsoft card key- a much different technology from those in use today, which are basically proximity read- I believe this was a magnetic stripe like a credit card [but can't be sure, because that part of the back of it is gone]. I think the big surprise for most was that 19 years ago, I had no facial hair and very short hair [almost no gray then, either]. Weighed a lot less, too, and still had two working eyes [even if they were near-sighted].

The other [finally getting to my point] is a speaker's badge for "Microsoft Windows NT Device Driver Developer's Conference". That was from 1992 [I wouldn't know for sure except that it turns out I still had the conference material with the dates on it], and it was the real "first DDC". I was the last speaker on the final day- the track was about Win32 printing [and this was the very first release of Windows NT and the first 32-bit OS Microsoft released], and I was speaking about testing and logoing printer drivers- that was my first time hearing about WHQL, by the way- when I was preparing the talk and they asked me to add some slides related to it.

Of course, my prized possession is the "Ship-It" award for OS/2 3.1 [which of course didn't exist as a Microsoft product- it was a typo, but I snuck mine out before the manual update processes in use then could correct it], but that just made this paragraph irrelevant and disruptive of the flow. I'm like that...

DDC- what a long, strange trip it's been

The Driver Developer's Conference is an intensive high-level conference with some NDA-type restrictions [I'm no lawyer- it appears these aren't as strict as the MVP summit, but who am I to judge or care?]. It will be held on the Microsoft campus later this month. The meeting I am alluding to was our team reviewing one of the presentations I will be participating in [and I've been slide-smithing ever since]. In one session, Ilias and I will be discussing the WDF coinstallers. Ilias will discuss the whys and hows, while I will finish up by telling you what you can do when things go wrong. In the other technical session, I, Shyamal and Wei will tell you about how we test the frameworks, including things we do that can translate into your own test problems.

There will be some Ask The Experts sessions, and I know WDF will be there, but I don't think the final lineup for that's been determined. I can live with it myself, either way...

As for the rest of the conference, well--- one of the more attractive nuisances of the past several days is that all the presentations go on a single sharepoint site, so I've been previewing a few that interested me. Lots of good stuff- device installation, DSF [my favorite non-WDF tool], new technologies all across the space, etc. I also see the world of printer testing tools is in great hands [probably better ones than mine ever were!].

WinHEC- a first time for everything

The Windows Hardware Engineering Conference- well, you can use the link to find out what you need to know. I'll be there too, with somewhat stripped-down versions of both talks [at least, I think that's what's happening]. It's still several weeks away, and there's DDC to get through first. But it will be in LA. Not certain who all else is going, yet, but I know Ilias is, and I can't believe it can be done without the likes of Doron and Peter.

I've been on the mailing list for years, but never gone (I was supposed to go a couple of years back, when it was in Seattle, but had car trouble).

So perhaps I'll meet a few of you there. I'll be the one with all the hair and the wacky left eye [unless I can't pass the airport security checks, or the other passengers find me scary and demand I leave the plane]. Well, I've lived in LA a few times and driven from here to there and back before...

Passion and Persistence

BobKjelgaard — Mon, 26 May 2008 01:06:29 GMT

Well, it's the Memorial Day weekend here in Redmond, and I am in my office working for the second of the three days of the weekend. Just can't stay away.

Why?

Because for months now I have been trying to improve the efficiency of our team's automated tests, and also get all the content in them under control. Over the years (I'm afraid it was the KMDF side of the team, from which I originally came, that was the biggest culprit), bits and pieces of content- test tool binaries, script files, and the like- were placed on a server accessible to both us and the lab- none of it under source control, nor with any idea in most cases where it could be built from or symbols to it procured. Shefali and I have been whittling away at this problem for months now, and I'm finally close enough to having it work I'm determined to make that one last push to get the job done once and for all (except for the inevitable maintenance, and I've tried to be smart enough about designing my solution that this should also be easy).

We've got many hours of tests overall between KMDF and UMDF, and I've got to make sure all of them work before I tell the lab that this new stuff is the way it's going to be from here on out [not to mention there is going to be some peer review first, as well]. It takes a lot of time, and of course, none of this is committed or budgeted work. I'm filling in the idle time [among other things, I'm doing abusive searches of build release servers so I can figure out where some of the tools we've just stuck up there before actually appear in Windows builds- so my being here when almost no one else is proves to be somewhat considerate of me, as well] by writing blog entries, updating code, and working on even further out ideas I want to try out.

Not that it's all hard work. I've got the music turned way up (a live version of the Grateful Dead's "Casey Jones" of all things, right at the moment- so I hear my blog's title being raucously sung), and I'm having my usual whimsical fun with comment fields and the like. For instance, the job itself (which is working rather well, so far) is named "Sauron", and the Details tab (which is supposed to describe it) begins with this absurd off-the-cuff plagiarized quatrain [my apologies to the Tolkiens]:

One job to to run for all-

One job to find them;

One job to bring them all

And at the test box bind them.

So why do something this exhaustive,

and do it when I'm supposed to be relaxing, and when its unpaid time, and so on?

Because I care about it (passion) and because I'm not going to just give up because it isn't easy (persistence).

Besides, this reminds me of my early days at Microsoft (the days of OS/2, Windows 3.1, and what became Windows NT 3.1). The days where the 70 and 80 hour week were common and I usually didn't mind. The days when I slept on the floor of my office because I got to be too tired to drive home, even though it was only a block away [yes, I really should walk, but I find the idea daunting at the 1-5 AM hours I am usually coming to work- Redmond is usually safe, but why take the chance?]. The days when I wrote code because I thought I'd need it or because I wanted to try something that struck me as cool at the time. The days when I may have worked hard, but I was enjoying it so much it didn't feel like work.

They came to an end- the reasons why don't matter. At one point after that I was interviewed by a journalist / author (he was writing a book about the development of Windows NT, and I guess he thought I'd make for a mildly interesting highlight or side story), and I remember telling him "I'd sit at the keyboard and I couldn't type- I didn't want to type. I wasn't interested in programming anything- I'd never had that happen before". While I didn't put it that way, it was like being dead, but still being capable of breath and movement. In a very real way, that was behind my resigning at the time I did- partly protective (no way could I do a decent job in that state, and sooner or later that would result in some black marks on the record), and partly reactive (some of the roots were environmental, or so I felt- doesn't matter really, it was all so long ago).

Ever since then, I've been looking for that feeling- that sense of wonder- that confident feeling that if I could think of it, I'd find a way to use the technology available to me and my own capabilities to make it happen. I'd get occasional flashes of it, but never the full-blown, out and out desire to do nothing but work on something for endless hours, and actually achieving that goal. The closest I came was gaming, but to me that was always "play". "Work" was a better use of time, but it just wasn't engaging me.

When I decided to rejoin Microsoft, I was hoping I might finally find that desire and enthusiasm capable of being rekindled. Working as a contractor, I'd had urges in that direction [but you must charge for those hours- so the kind of thing I am doing now would be an abuse of the business relationship to either my employer or to their customer]. But it just seemed to sputter- there were brief flashes, but nothing consistent.

But now it looks like it is happening at last. Perhaps I do get another bite at the apple- I already feel younger for some reason. Perhaps I can take on big challenges and bring them to fruition, even though I am highly prone to "the lone wolf" style of addressing things. If it's happened- I know who to thank- Shefali, Abdullah, Scott, Darren, and everyone from them right on up to Messrs. Ballmer and Gates. For providing an environment (and yes, there is much more to that environment than just the chain I've listed) where I can be myself, and still accomplish things- maybe not great things, but at least things worthy of accomplishing. For listening to sometimes petty concerns and putting up with my foibles, inconsistencies and mistakes (especially Shefali in this case).

I hadn't thought about that before- perhaps the problem previously is I felt I had to be something other than what I am to succeed. Now I finally feel otherwise, and it is very liberating.

Well, perhaps I am deluded. While I see this as a good thing to do, and an improvement over the existing conditions, perhaps it will be judged otherwise. I am not using all the latest and greatest technologies to solve the problem, so perhaps it will lack appeal because of that. Time will tell, in either event. I'll deal with that as it comes.

In the meantime, I am going to do my level best to make this thing work, and I don't mind the time it is taking.

Because believe it or not, I am actually having quite a lot of fun...

[Segue into Grateful Dead [Wake of the Flood] Mississippi Half Step Uptown Toodeloo.... "along the Rio Grande, oh-..."]

The Forever Wait

BobKjelgaard — Sat, 24 May 2008 03:31:00 GMT

Many thanks to Patrick for this reminder about another practice an SDET should avoid (actually, most SDE's should, as well- but they have their own masters and I'll leave that job to them). Not that it's easy to pick the "right" amount of time. In this much older post of mine, I mention a test case that took an incredible amount of time when we turned Driver Verifier on. Well, that test case has been a rascal in many ways (it even found some unusual and rare bugs in Driver Verifier itself). I was initially using the same wait time on all of my tests, but I had to stop- I set the wait on this one out to many minutes, but special cased it so the others wait a few seconds- if I didn't, a hardcore hang would potentially set runtime out to hours, not good for automation.

Well, I ran it today [I'm spending time I shouldn't be trying to streamline our automation- in this case making the installation aspects of it smart enough to find their own content, instead of having people type an ever changing set of locations into WTT as parameters every time we have to make a run, which is, after all, every day or more- but at the same time I removed the copying of things we no longer use, or are getting from insecure places when there are secure places, etc- to make sure I didn't mess it up, I'm running all of our tests to make sure I didn't prune out the collection of something we need], and the test case AFTER it failed because it timed out. I already know this is because the cleanup from all those allocations is hogging resources (I've even hit the "too long at dispatch level" break on machines with way too much memory), preventing me from getting time to get that next instance spawned off my test bus (serves me right for trying to squeeze some test performance by spinning up the next test while the first was spinning down).

The forever wait would make life easier in this case- but it wouldn't be near as much fun.

Now if any of you are thinking- "wait- why aren't you even talking about breaking and forcing people to look at the hang when it times out like this?"- well, we still have an opening or two for more SDETs on our team (and our close buddies in Storage have plenty of good opportunities for you as well), and you're obviously beginning to think like one! The best answer I ahve at the moment is that I serve multiple masters, one of which is regular automated checking, for which recording a failure and moving on is OK (as long as it isn't new, anyway). I've been thinking about ways over the last few days to accommodate this [break conditionally, but not all of the time in way that's smart enough to suit our needs], but haven't hit upon that most desirable solution yet. Not that I can't think of ways- but I need ways I can achieve with what's at hand- all the aggressive work we've done on hardening the test code so it doesn't break more often than the product it's testing have been working, so I've at least got time to move from "Stop all the bugchecks" to "what about attempt rates and failure rates" phase. Actually, it's not just that- it's that Patrick and Wei have been stepping up and helping with some of that fixing and the never-ending job of triaging failures in our test labs.

But, thanks to Patrick, that "things to do next time you crack that DDI test code" list now has another item:

Things to do the next time you crack that moldy old test code

Set USER_C_FLAGS to /TP
Set the warning level to 4 (convert warnings to error is already on)
PFD green before you're done, period.
Look for suspicious coding patterns where failures are being masked or ignored- at the very least in the places the above changes have forced you to look.
Examine KeWait... calls for NULL timeout and WaitFor... calls for INFINITE. Take them out with the rest of the trash!

[Edit 5/24] Sorry , I just realized I revealed someone's email address wtihout permission [it made a fine pun, but still not the right thing to do]- stats say not likely anyone realized it yet, but mea culpa, anyway...

Just had to add this one

The title of this post keeps reminding me of The Forever War- a science fiction novel that earned both Nebula and Hugo awards. The awards probably speak for themselves- I'm quite pleased to have a copy of it in my personal library (I guess I'm pleased to have a personal library, but I leave that sort of thing to my personal blog.

As for the music- happens to be another bit of mellow fineness from Field of View (sorry, I didn't come up with a more useful link, but I'm short on time, as always) at the moment- I may have paid what seemed an outrageous fee to import this CD set (not the one I linked to, that would take me a while to find, I'm afraid) from Japan, but it was soooooo worth it!

Why SDETS should be the most fastidious and paranoid coders in existence

BobKjelgaard — Sat, 10 May 2008 03:41:00 GMT

Well, to follow up some more on my adventures in test code maintenance, I bring you a case study.

As I whittled away at the 100+ PFD warnings I mentioned in my previous article, I did my paranoid best to look for dubious code above and beyond what PFD was telling me about when I noticed this in one of our oldest test drivers:

NTSTATUS  _stdcall 
CTestDriver::SetupObjectHandleParameter(
    IN ULONG eHandle, 
    IN OBJECT_TYPE ObjectType,
    IN OUT PVOID * Handle
    )
{
    NTSTATUS status = STATUS_SUCCESS;

    //
    //setup Queue Handle parameter
    //
    switch(eHandle){
        
        case HANDLE_INVALID_NULL:
            *Handle = NULL;
            break;

        case HANDLE_INVALID_OBJECT_DELETED:
            //
            //we will treat this as HANDLE_INVALID_NULL since we can not delete the default queue
            //
            *Handle = NULL;
            break;

        case HANDLE_INVALID_WRONG_OBJECT_TYPE:
            
            if(ObjectType != OBJECT_TYPE_DRIVER){ 
                
                *Handle = (PVOID)WdfGetDriver();
                
            }else{
            
                *Handle = (PVOID)this->m_pFdo;
            }
            break;

        case  HANDLE_VALID:
            
            switch(ObjectType)
            {
                case OBJECT_TYPE_DEVICE:
                    *Handle = (PVOID)this->m_pFdo;
                    break;
                    
                case OBJECT_TYPE_QUEUE:
                    *Handle = WdfDeviceGetDefaultQueue(this->m_pFdo);
                    break;
                    
                case OBJECT_TYPE_REQUEST:
                    status = WdfRequestCreate(NULL, NULL, (WDFREQUEST *)Handle);
                    break;
                    
                default:
                    *Handle = NULL;
                    
            }
            break;
            
        default:
            DbgBreakPoint();
#pragma prefast(suppress:__WARNING_UNREACHABLE_CODE, "Always runs under debugger, so not unreachable, merely deliberately annoying")
            *Handle = NULL;
            status = STATUS_INVALID_PARAMETER;
    }

The breakpoint and suppression pragma are my later addition [and I removed some invalid cases prior to this- in fact, it was the act of removing them that led me to even look at this code]. Why did I add them? Because this silently changes an invalid and unexpectedly out-of-range value without any notice or any indication that something unexpected and unwanted has happened. I wanted to make sure that if this EVER happened, someone was forced to take a look (and yes, this is why people use ASSERT of course).

Well, I finally got all my warnings cleaned up, and I built the dozens of drivers and assembled everything I needed so I could make a "dry run" of all the affected code under lab conditions. I hit that breakpoint about a half hour into the run. So I took a look, of course- that IS why I did this, after all [to force someone to look- I just also turned out to BE that someone].

As one would expect from even middlin'-to-fair practice, those constants are in a header file, the appropriate snippet being this:

typedef enum {
    HANDLE_INVALID_NULL,
    HANDLE_INVALID_OBJECT_DELETED,
    TOTALLY_INVALID_AND_DO_NOT_USE_UNTIL_ALL_ARE_REMOVED_AND_THIS_CAN_BE_RECYCLED,
    HANDLE_VALID,
    HANDLE_INVALID_NOT_SET,
    HANDLE_VALID_NULL,
    HANDLE_INVALID_NOT_OPEN,
    HANDLE_INVALID_WRONG_OBJECT_TYPE,
    HANDLE_CALL_DDI_MULTIPLE_TIMES = 23
    } HANDLE_STATES;

The long name was my addition- it was for the invalid case I was trying to prune from all those drivers [it passed -1 as an "invalid handle value"- fine practice in user mode, but in KMDF they [wisely, IMO] decided NULL was enough to guard against]. The oddball value at the end is there because of one of the items that factors into my tale [but I won't mention it again].

Soon after these early test drivers were developed, we off-shored the work of developing more of them to test the burgeoning KMDF DDI. They basically made a copy of what we had, and then began developing more, extending and refining our techniques. Now I'm about to describe an error of theirs, but if the original developers had been more paranoid, these folks could have known about this long before I found put about it! People make mistakes, especially during those early stages of development, period. They continue to make them in maintenance and addition of new features. This isn't a "look at this fool" story- it's a "this is good and intelligent people just being human" story- please don't misunderstand me!

To go back now to the present, a quick check in the debugger showed that "9" was the value for eHandle. You'll notice it doesn't match any of the enumerated values. So where did it come from? Well these tests are driven from scripts, and the original developers being fastidious used the same names and definitions in their script-based engine. [I told you these people were good!] So I found the test case used because it was readily available from the driver at the point this failed (drum roll, please):

    var    eQueue    = HANDLE_INVALID_WRONG_OBJECT_TYPE;
    var    bStopComplete = POINTER_VALID;
    var    eContext   = POINTER_VALID_NULL;

Now HANDLE_INVALID_WRONG_OBJECT_TYPE is defined, but its value is 7, not 9- so why the mismatch? Well, like I said, all the original files were forked off, and the offshore team maintained their copies. At first I diff'd the original header to see if I'd done something totally bad, like deleting values from the enumeration without maintaining the value of those that were left- after all a stream of 14+ hour days leaves me quite capable of such a move! But it wasn't me, this time- so I went to their version of that same header, and found:

typedef enum 
{
    HANDLE_INVALID_NULL                             = 0,
    HANDLE_INVALID_OBJECT_DELETED                   = 1,
    INVALID_VALUE_NOT_TO_BE_USED_1                  = 2,
    HANDLE_VALID                                    = 3,
    HANDLE_INVALID_NOT_SET                          = 4,
    HANDLE_VALID_NULL                               = 5,
    HANDLE_INVALID_NOT_OPEN                         = 6,
    HANDLE_INVALID_REACQUIRE                        = 7,
    HANDLE_INVALID_NOTACQUIRED                      = 8,
    HANDLE_INVALID_WRONG_OBJECT_TYPE                = 9,

Oops (and yes my names for the value I ripped out were inconsistent, but then maybe some consistent inconsistency is useful in doing this sort of maintenance activity, and besides, none of this is finished yet]!

I then went to the trouble of trolling through our configuration management system (mostly out of curiosity, as this bug isn't a killer in and of itself] to see when this breaking code change was made, and found it happened to have been made about 10 days after I walked out of New Employee Orientation and into my second SDET career at Microsoft [I probably couldn't even begin to use WTT {DTM to you} at that time]. This defect silently caused us to stop testing the effects of passing the wrong kind of object to the DDI tested in this driver [passing NULL instead, which as you can probably divine was already being done]. The test failed, but it was supposed to, so no visible change in outward results- just a silent loss of coverage of some paths in the product code for the last 3 years.

Umm, so what?

Granted- this is hardly the most important problem found on the planet this week (probably even in the last 10 seconds). But I think there's a point here for any serious SDET. Be paranoid! Check everything! Trust nothing! Before you punt that oddball case, at least consider using an ASSERT or some other mechanism to focus attention on questionable results or behaviors. If the code looks ratty as a result, and you'd never ship it, so what? Your purpose isn't to make pretty code- leave that to the SDEs- your task is to find bugs [and preferably in the product- but if you find your own with these practices, the odds of your doing the same for your product are that much improved]. I don't personally care much where the braces go in your code- but I sure care a lot about what you do between them!

Yes, someone inclined to point out errors and less-than-optimal practices can point out many more [like, oh, say- keeping two versions of the same header file about?] But it takes time to sort out improving what you've got, and there's the continual need to add new stuff [which is what I was doing when I began this little detour this week, and I'd better get back to it, and soon]. I'll add removing that superfluous header to the ever growing list of "things that ought to get done when there's finally time to do it"...

Enough of the soapbox rant for now. I'll go put on my hat and find some of the other curmudgeons [someday I'll have to take a photo of it perkily perched on top of the replica Spartan helmet from the Halo 3 Legendary edition [someone once told me it gave them visions of John-117 as Indiana Jones, bullwhip in hand] in my office]...

Odds and Ends

Well, PFD eventually found well over 30 IRQL misusages by the times I was done- many of the pattern I showed in that last article, others more subtle- and some of those were more of a "risk of future maintenance trouble" issue- exactly the sort of maintenance error behind this bug. PFD- it isn't just for developers- your test code ought to be as good as theirs and preferably better [so when something breaks you've got all the information you need to say what's going on]. It also focused attention on dozens more problem areas in this code base. Being PFD clean doesn't guarantee your driver is rock-solid bulletproof, but it sure makes a nice foundation!
Wei reminded me that you can combine the values when suppressing by separating them with white space. Figures- I'd tried commas and got into too much of a hurry to look up the basics of those pragmas. Also that the suppression is suppose to apply to a single source line (but I think stacking them as I did also works- can't say for sure).
I've begun using some Analytical / tracking software on my blog to understand my readership better [nothing personally identifiable]. I see that I get a reasonable number of non-English speakers, so I'll see if I can rein in my vocabulary and use of idioms in the future.
The web site I keep some of my blog content on (like my photograph) is undergoing maintenance for a couple of days. The text files for fixing KMDF installation failres are there too. They'll be back by Monday, May 12, 2008.
No song lists- too busy to try to cpture thm this time [lucky you!].
A good weekend to all!

Reaping the Benefits of Static Analysis Tools

BobKjelgaard — Wed, 07 May 2008 19:00:00 GMT

Our team has a huge amount of legacy test driver code, much of it rather old, and originally created by somewhat less experienced developers. I continually find myself going back and performing maintenance on that code, and as I've noted previously- Static Analysis Tools are a great help in identifying problems in this code. I'll put this "down time" (among other things, I've got PFD analyzing 40+ files that are rather large as I write this) to good use by describing both some further benefits and showing how easy it is to annotate your own code so you can recognize this sort of benefit.

Irql Tracking Finds Yet Another Flaw

One of the benefits PFD adds above regular PreFast is tracking IRQL changes in your code. Of course, there are additional annotations used to support this, and I found them to be most useful in a very real-world way.

In our tests of the KMDF DDI, we try to make a lot of invalid calls to assess how well the framework verifier works. One of the things that we do is make calls at improper IRQL levels- since efficiency suggests you usually have one place that tests a particular call, this change has to be programmable. Not the sort of thing you expect a static analysis tool to be able to do for you, eh? Well one of our SDETs made an initial pass at this issue [after I told the team we were going to make all of our test drivers PFD clean over the next few months, instead of having me piecemeal it as I fixed bugs as I had done previously], and encountered some difficulties

The original declarations looked like this:

/*******************************************************************************
 KmdfTestRaiseAndPrintIRQL

 Synopsis - Raises IRQL to DISPATCH_LEVEL and prints IRQL

 Parameters -
   eIRQL    -   Enumerator for IRQL
   OldIRQL  -   IRQL prior to raising to DISPATCH_LEVEL

******************************************************************************/
VOID 
KmdfTestRaiseAndPrintIRQL(
    __in    ULONG   eIRQL,
    __out   KIRQL   *OldIRQL
    );

Initial code- nice enough, and PreFast annotated, but it changes IRQL and doesn't say a thing about that other than in comments

There was no declaration for a function to lower IRQL- typically eIRQL was compared to a constant and KeLowerIrql was called when needed. Well, PFD quickly noticed this function elevates IRQL on some paths, so the attempt was made to annotate it, and a paired restore function was also defined- thus:

/*******************************************************************************
 KmdfTestRaiseAndPrintIRQL

 Synopsis - Raises IRQL to DISPATCH_LEVEL and prints IRQL

 Parameters -
   eIRQL    -   Enumerator for IRQL
   OldIRQL  -   IRQL prior to raising to DISPATCH_LEVEL

******************************************************************************/
__drv_raisesIRQL(DISPATCH_LEVEL)
VOID 
KmdfTestRaiseAndPrintIRQL(
    __in    ULONG   eIRQL,
    __out   __drv_out_deref(__drv_savesIRQL) PKIRQL   OldIRQL
    );

/*******************************************************************************
 KmdfTestRestoreIRQL

 Synopsis - Restore IRQL

 Parameters -
   eIRQL    -   Enumerator for IRQL
   OldIRQL  -   IRQL returned from KmdfTestRaiseAndPrintIRQL

******************************************************************************/
VOID
KmdfTestRestoreIRQL(
    __in    ULONG   eIRQL,
    __in __drv_restoresIRQL __drv_nonConstant KIRQL OldIRQL
    );

First attempt- this says IRQL is changed and a restore function is added, but is it enough?

However, this didn't work out so well, when the implementation of these routines was analyzed by PFD, and since I had mandated cleanliness, warnings from PFD were suppressed, thus:

__drv_raisesIRQL(DISPATCH_LEVEL)
VOID __stdcall
#pragma prefast(suppress:__WARNING_IRQL_NOT_SET, "Intentionally save irql at specified condition");
KmdfTestRaiseAndPrintIRQL(
    __in    ULONG   eIRQL,
    __out   __drv_out_deref(__drv_savesIRQL) PKIRQL   OldIRQL
    )

// and then this

VOID __stdcall
#pragma prefast(suppress:__WARNING_IRQL_NOT_USED, "Intentionally restore irql at specified condition");
KmdfTestRestoreIRQL(
    __in    ULONG   eIRQL,
    __in __drv_restoresIRQL __drv_nonConstant KIRQL OldIRQL
    )

These annotations must not be enough, because PFD is smart enough to find paths that put the lie to the annotated effects of these functions

Why the need to suppress? Because PFD correctly analyzed that there were paths through the code that did not do what the annotations say these functions do (raise IRQL to dispatch and restore it, respectively). These actions were conditional, and the annotations didn't reflect this.

Conditional Annotations: One Powerful Tool Makes One Effective Solution

I encountered the problem in different form. I found myself working with the earliest of our drivers, written before these library routines were used. They defined similar functions inline, At first, I made these same annotations there [but didn't suppress], as that let me borrow from the work already done. But since I didn't suppress, I got the identical errors.

When I realized why I was getting errors, I recalled reading about conditional annotations, so I read up on them and really, for a case like this they just weren't that hard to figure out. I fixed the code I was working with, and then when I was sure this was the right thing to do, I made the same fix to the library code and header file I snipped above. The end result looked like this (and I removed the suppression pragma from the library code, of course).

/*******************************************************************************
 KmdfTestRaiseAndPrintIRQL

 Synopsis - Raises IRQL to DISPATCH_LEVEL and prints IRQL

 Parameters -
   eIRQL    -   Enumerator for IRQL
   OldIRQL  -   IRQL prior to raising to DISPATCH_LEVEL

******************************************************************************/
__drv_when( eIRQL == EXECUTION_LEVEL_DISPATCH, __drv_raisesIRQL(DISPATCH_LEVEL))
VOID __stdcall
KmdfTestRaiseAndPrintIRQL(
    __in    ULONG   eIRQL,
    __out __drv_when( eIRQL == EXECUTION_LEVEL_DISPATCH, __drv_out_deref(__drv_savesIRQL))
            PKIRQL  OldIRQL
    );

/*******************************************************************************
 KmdfTestRestoreIRQL

 Synopsis - Restore IRQL

 Parameters -
   eIRQL    -   Enumerator for IRQL
   OldIRQL  -   IRQL returned from KmdfTestRaiseAndPrintIRQL

******************************************************************************/
VOID __stdcall
KmdfTestRestoreIRQL(
    __in    ULONG   eIRQL,
    __in __drv_when( eIRQL == EXECUTION_LEVEL_DISPATCH, __drv_restoresIRQL)
            KIRQL   OldIRQL
    );

Adding conditional annotations lets PFD correctly analyze the functions and code that uses them, removing the need to suppress perfectly valid warnings.

It also more clearly states how the function works in a fairly readable way.

The annotations are now more precise. EXECUTION_LEVEL_DISPATCH is defined in a header always known when this is included, so PFD can properly analyze all usages. It now knows that we change and restore when the eIRQL parameter is set to this value, and leave it alone otherwise.

Reaping The Benefits, Part 1- Finding a Real Bug at Build Time

Well, it didn't take long for it to turn up a real flaw [in one of those 40 files I alluded to earlier]. This is pseudo-code illustrating what it found.

KmdfTestRaiseAndPrintIRQL(
    eIRQL, 
    &OriginalIRQL
    );

if (condition) {
    MakeTheCall();
    KmdfTestRestoreIRQL(
        eIRQL,
        OriginalIRQL
        );
}
Pseudocode for a bug the above annotations exposed- found by PFD in that ancient test code.

Out of the 10's of thousands of source lines I'm working with, I don't think I could have noticed that- but PFD zeroed right in on it. If the condition isn't met, we can raise IRQL without lowering it. It's rare for the condition to not occur, which is one reason why nobody had found this before [another is that intermittent failures in these tests used to be routinely overlooked- but I don't work that way- I only give up when I absolutely have to]. Easily fixed, of course, just move the raise inside the conditional block as well.

Reaping the Benefits, Part 2- Clearer Documentation of Test Practices

By requiring the code to be clean (no errors or warnings) we have to use suppression in those places where we deliberately break the rules. Now I could call this an annoyance, but actually I think it makes it a lot easier to understand better how we are testing the DDI in these cases. So I'll leave this final sample:

        TraceEvents(TRACE_LEVEL_INFORMATION, DBG_INFO, 
            "\nCalling DDI WdfDmaEnablerCreate\n\n");

#pragma prefast(suppress:__WARNING_INVALID_PARAM_VALUE_1, "Intentionally passing 0 to test the DDI")
#pragma prefast(suppress:__WARNING_IRQ_TOO_HIGH, "Deliberately done to test framework verifier")
        status = WdfDmaEnablerCreate(
                    device,          // Handle to device object
                    pDmaConfig,      // Pointer to WDF_DMA_ENABLER_CONFIG structure
                    pAttributes,     // Pointer to WDF_OBJECT_ATTRIBUTES
                    pDmaEnabler      // Pointer to WDFDMAENABLER to create
                    );

For a further benefit, places where rules are deliberately broken for test purposes are now clearly marked by the suppression pragmas

Now, we actually pass NULL values in some other parameters- one reason this isn't reflected is there are common routines that establish some of these pointer values, and they are not annotated to reflect that the values can be NULL on output. Someday maybe I'll get to that task [or just try to make sure we do it upfront on new test code].

Well, that run finished and I've got 128 bits of further errata to investigate [I sort by warning number, and I'd just finished the warnings for Irql too high when I saved it all off, recompiled to make sure I hadn't broken anything, and went on- I started with over 180 failures, so I'm making headway]. In case you were unaware of it, you can get the mnemonic constants for the errors [typically reported by number] from the file suppress.h in the sdk\inc directory.

So, I'm off to do more cleanup [I started with one test driver, and this just sort of ballooned on me when I decided to rip out some less desirable things I've been meaning to rip out], and I'll let you get back to your regularly scheduled workday!

Tunes

Grateful Dead

[Aoxomoxoa] China Cat Sunflower

[Workingman's Dead] High Time, Black Peter

[Europe 72] Sugar Magnolia

[Wake Of The Flood] Eyes of the World

[Shakedown Street] If I had the World to Give [Bonus Track]

[Go To Heaven] Alabama Getaway

[Dead Set] Brokedown Palace, Rhythm Devils

[Reckoning] It Must Have Been The Roses

Various Artists

[Record of Lodoss War] Instrumental [name untranslated]

[InuYasha TV OST] Kagome and InuYasha 1

Tsuneo Imahori

[Trigun the First Donuts] Knives

Yoko Kanno /Seatbelts

[Cowboy Bebop] Car 24, The Egg and I

[Cowboy Bebop Blue] Farewell Blues

[Cowboy Bebop Knockin On Heaven's Door] Diggin'

Yuki Kajiura / See-Saw

[.hack//liminality] [name untranslated- also on "Dream Field"], [name untranslated- also on "Dream Field"], [name untranslated]

[.hack//sign OST1] silent life, kiss

[Fiction] Fiction

The light at the end of the WDF 1.7 tunnel

BobKjelgaard — Thu, 17 Apr 2008 18:04:55 GMT

I know people want to know about this, but firm dates still prove elusive. My best guess is today or tomorrow at this point.

Basically we're trying to put an interim package up that satisfies everyone and isn't going to cause us foreseeable problems in the future. Eliyas keeps tweaking this MSI to satisfy one concern or another raised by members of the Dev, PM and QA teams, and once it's done, it will go to Microsoft Connect and you'll have your coinstallers. Sometime after that we'll have an updated WDK with everything tied together nicely again. As I understand it, the links for it on MSDN will work eventually, but I have no idea when.

As far as the real testing- all done, all good. All the legal and licensing stuff, good. Just have to make an MSI with everything in it that doesn't look like it was slapped together at the last minute- even though it sort of is.

Now playing: Yoko Kanno [Ghost in the Shell Stand Alone Complex, V2]: Psychedelic Soul

Some follow up

Kumar fixed the UMDF coinstaller build problem (which you'll never see) in fairly short order, and then went on vacation himself. So of course, more build problems turned up- but until we get that MSI out, not likely I'll get any traction. This time the x64 update package for WinXP and Win2K3 is missing some pieces- blocks me, but again, I doubt this is a problem that affects anybody but my team [call this an attempt to give you an idea what it can be like sometimes "on the inside", because it's easy to get the idea everything is endlessly smooth and we spend all our time pondering the whichness of the why].

Now Playing: Grateful Dead [Live / Dead] St Stephen [always a welcome album]

So that's what they look like!

Yesterday I gave a brief presentation on WdfVerifier and WdfTester in a session on Driver Testing at the Global MVP summit. Met Mikhail Vodicka, Gianlucca Varenni, Martin O Brien and Maxim Shatskih- or rather at least exchanged greetings face-to-face, either in the session or out in the hall afterward. I'd dropped off the OSR lists for a bit to help manage my email and workload, but after talking to Martin a bit, I decided to re-subscribe again. Hopefully I can better manage my impulses to jump in on some of the esoteric issues that occasionally surface there...

Now Playing: Kraftwerk [The Mix] Musique Non Stop