Trouble Ahead- Trouble Behind : debugging

New WDF debugging videos on WHDC

BobKjelgaard — Tue, 22 Sep 2009 16:41:32 GMT

Several for KMDF here, and several more for UMDF here.

Thus endeth today’s PR blitzen. Have fun!

Exceptions rule!

BobKjelgaard — Sat, 18 Apr 2009 19:24:55 GMT

I’ve decided to revert to storytelling mode today. So sit right back and you’ll hear a tale, a tale of my fateful trip (or click something useful- you’re the one deciding, and I’m the one typing to satisfy whatever inner demon has driven me to do this today)…

It began when someone found a bug in the static version of KMDF- the one almost nobody gets to use and which we have been trying to discontinue any support. But people were still using it, and we had no clear migration path, and there was a bug. So we decided to begin testing it. Poor Neslihan got that task (well at least it wasn’t me, this time). Soon we had added a static KMDF driver to our list of test drivers used in our daily automated testing.

Then, of course, we got the bugcheck. As often happens, I played “Johnny-on-the-spot” and (virtually speaking) leapt onto the remote debug session accompanying the report. Buffer overflow? In FxDriverEntry? But only on Itanium (I checked the other architectures that ran the same test, of course- no problems)? What in blazes is this about?

Well, the method is well documented, and not all that hard to understand- a “cookie" gets initialized, it gets stored to the local stack frame in such a way that code overflowing buffers in the stack frame will overwrite the stored copy. On exit, code gets called that checks that value to see if that happened. It’s a little more complicated than that, and I’m not going to be precise, because part of that complication is related to securing the method against the sort of people who like to exploit buffer overflows [so I’m not going to make the method clear for them- they can do their own research without my assistance]. The bugcheck analysis will even tell you the two values- expected and actual, making it easy for people who really think overflows are a bad idea to make a quick guess as to where the overflow came from.

But the circumstances seemed suspicious, so I went and read the source code- H’mm- there’s a default value the cookie is initialized to… Interesting- that’s the value we have- but it doesn’t match the expected value.

At which time, something I’ve known for years hit me like the proverbial sledgehammer- our entry code calls GsDriverEntry (which supports the stack probes inserted by the GS compiler switch, hence its name) when it finishes. GsDriverEntry initializes the cookie, which means that that was why the values didn’t match. Like the sword of Damocles, this had been hanging over our heads for years- the first time the compiler decided to do stack probes in our entry code, everything would break. Ouch…

I’ll leave a bunch of story out here- things got “interesting” at that point, but the problem eventually got solved, both for us and for other people facing similar issues, as well. Short answer is that everyone now initializes the cookie right at the start, just as should happen…

But a while back I had occasion to tweak the build process for the WdfVerifier WDK applet, and afterwards I ran it briefly to make sure I didn’t break anything in the process. Oh, joy of joys- NONE of the 1.9 client drivers are being properly identified. They are all identified as “Inbox”, which is what I do when the client identification method I described last year fails on me.

Already beginning to panic, I think- did someone change FxDriverEntry, and I didn’t even notice it? So I go to our source control system, and look at the change history. The most recent change is the fix for the problem with the stack probes (yes it occurred on static, but what scared me then was it could have happened to any version, because were weren't doing things properly). But that just calls library routines to initialize said cookie… Oh, blazes, those routines must be calling an import I wasn’t accounting for!!! Why??? Because I NULL the IAT to force access violations, and I handle the exception by giving up on getting the true version, and fall back to calling it “Inbox”- which is, after all, exactly what I am seeing. Oh, well- so much for my having thoroughly considered all the test and product consequences of that change when it was made…

Well, easy enough to find out what that import might be. One nice trick Ilias showed me is that you can open any binary in WinDbg as a crash dump, and happily resolve symbols and disassemble code. So I pick a random driver on my dev box, and do so.

What, no imports? But, but--- hmm- it uses a fixed address (load of a 64-bit immediate value into RAX, since my dev box is an X64, thank you- you x86 dinosaurs can keep your wimpy processors). What’s with this?

The answer lies in wdm.h, of course- KeQueryTickCount turns out to always be a macro- on x86 and IA64, it accesses KeTickCount [which is an import, and if you followed my earlier tale, it’s clear adding an import to my existing hack-o-matic mechanism is a trivial task]- but on X64, it accesses an essentially hardcoded address in a “SharedUserData” area. This snippet is from wdm.h, so you can see for yourself…

#define KI_USER_SHARED_DATA 0xFFFFF78000000000UI64

#define SharedUserData ((KUSER_SHARED_DATA * const)KI_USER_SHARED_DATA)

#define SharedInterruptTime (KI_USER_SHARED_DATA + 0x8)
#define SharedSystemTime (KI_USER_SHARED_DATA + 0x14)
#define SharedTickCount (KI_USER_SHARED_DATA + 0x320)

#define KeQueryInterruptTime() *((volatile ULONG64 *)(SharedInterruptTime))

#define KeQuerySystemTime(CurrentCount)                                     \
    *((PULONG64)(CurrentCount)) = *((volatile ULONG64 *)(SharedSystemTime))

#define KeQueryTickCount(CurrentCount)                                      \
    *((PULONG64)(CurrentCount)) = *((volatile ULONG64 *)(SharedTickCount))

My. my. my- this is a problem- no import I can just tweak to point into my code. For a plus, at least the value can’t change (if it did, existing drivers would, after all, cease to work)- well, at least not easily, so I’ll save any remaining paranoia about that for another time. But I’m going to access violate trying to access that address in user mode (not to mention the reams of experimental evidence I had just accumulated by noticing the problem in the first place)…

So in reading about exception handling, a phrase happens to catch my eye- putting it in my words, it says that when continuing an exception, you can alter the context record supplied to you with the exception (and I know this process well, but that’s perhaps a story for a different time). Whoa- that must mean I can change the contents of the registers programmatically, and then tell it to repeat the failed instruction! Now, I can’t find that illustrated in any of the samples the material I’m reading points me to [nor could I find it an an internet search, although the latter was by no means exhaustive]. But that HAS to be what it means, right?

So I start coding- first the half dozen or so lines needed to handle the KeTickCount import. Then an exception filter [only for AMD64, of course]. The logic is my usual bit of precise work: If it is an access violation, and if it is a read, and it is a read of that exact address, and one of the integer registers in the context record has that same address, then change that register’s entry in the context record to the address of the proxy I’ve decided to keep in WdfVerifier, and tell the OS to repeat the failed instruction. I began with RAX, because after all, that’s what I had seen in my investigation, and it seemed the most likely place for a while, but I added the whole set since under the circumstances, it seemed unlikely to do any harm. Anything that didn’t match that exact pattern, and I gave up- just execute the handler, which does nothing. The attempt to run the driver entry code to extract the client version will fail, but WdfVerifier keeps running, and the machine itself is still quite safe from my hackery.

Worked the first time, it did (not counting my usual compiles to get rid of typos). Problem solved. Total work time from start to finish- something like 5 or 6 hours- good enough- after all, I did have to do some research… Of course, there’s always somebody who can do it better, faster, and quicker- and they usually jump out of the woodwork if I start talking about things in that fashion, but it’s my story, so I’ll brag now and regret it later…

Exceptions do indeed rule! I love it when a plan comes together…

A glimpse now at handling the new import…

    //  The descriptors are an array of entries per module, each terminated with an all-0 entry.
    //  Each entry has an RVA for the module image name, and a second for a 0-terminated list of RVAs to the structures used for
    //  resolving names of exported entries from the module (in loading these get resolved to real addresses and are plugged into
    //  the loaded image's import address table in the same order).  So we now know how to write an image loader if we need to...

    //  First, find the descriptor for the KMDF loader, and quit if it isn't there.
    DWORD   StringCopy = (DWORD) -1, WdfBind = (DWORD) -1, InitEvent = (DWORD) -1, TickCount = (DWORD) -1; 

    //  Then look for the Bind and Unbind entry points.  Case counts!

    enum {OutdatedTechnology, HasBind, HasUnbind, OneCoolDriver};
    
    BYTE        TargetIdentification = OutdatedTechnology;

....

            else if  (IsKernel)
            {
                if  (0 == strcmp("RtlCopyUnicodeString", (PSTR) NameDescriptor.Name))
                    StringCopy = ImportsIndex;
                else if  (0 == strcmp("KeInitializeEvent", (PSTR) NameDescriptor.Name))
                    InitEvent = ImportsIndex;
                else if  (0 == strcmp("KeTickCount", (PSTR) NameDescriptor.Name))
                    TickCount = ImportsIndex;
                
                ImportsIndex++;
            }
.....


        DWORD       IATSize, RelocationsSize;
        LONGLONG    OurFakeTickCount = 0x8badf00ddeadbeefI64;   //  ersatz Tick count         
        PVOID*  ImportAddresses = (PVOID*) 
            ImageDirectoryEntryToDataEx(DriverImage, TRUE, IMAGE_DIRECTORY_ENTRY_IAT, &IATSize, &Unused);
        
        PIMAGE_BASE_RELOCATION  Relocations = (PIMAGE_BASE_RELOCATION)
            ImageDirectoryEntryToDataEx(DriverImage, TRUE, IMAGE_DIRECTORY_ENTRY_BASERELOC, &RelocationsSize, &Unused);

        if  (NULL == ImportAddresses)
        {
            FreeLibrary((HMODULE) DriverImage);
            return  true;
        }

        //  Plug them in here, and we are good to go!

        DWORD   OldProtect;

        if  (!VirtualProtect(ImportAddresses, IATSize, PAGE_READWRITE, &OldProtect))
        {
            FreeLibrary((HMODULE) DriverImage);
            return  true;
        }

        memset(ImportAddresses, 0, IATSize);

        ImportAddresses[StringCopy] = FakeOutStringCopy;
        ImportAddresses[WdfBind] = CollectVersion;
        if  (InitEvent != (DWORD) -1)
            ImportAddresses[InitEvent] = FakeEventInit;
        if  (TickCount != (DWORD) -1)
            ImportAddresses[TickCount] = &OurFakeTickCount;

        VirtualProtect(ImportAddresses, IATSize, OldProtect, &OldProtect);

So for some other glimpses, this is one part of the other hackery (complete with my elementary-level annotations):

class AllKMDFDrivers
{
    MachineKey&             Owner;
    KMDFDriverList&         InstalledKMDFDrivers;
    LoadedDrivers&          CurrentlyLoadedDrivers;
    LoaderDiagnosticsFlag   LoaderFlag;
    String                  RuntimeVersion;
    DWORD                   Major, Minor, Build;

    bool            ServiceUsesKMDFLoader(__in PCWSTR ServiceName, __in RegistryKey& Service, __out MyBindInfoAlias& Binding);
    static void     FakeOutStringCopy(__in PVOID, __in PVOID);  //  Fake RtlCopyUnicodeString entry
    static void     FakeEventInit(__in PVOID, __in ULONG, __in ULONG);  //  Fake KeInitializeEvent entry
    static ULONG    CollectVersion(__out MyBindInfoAlias& BindingOut, __in PVOID, __in MyBindInfoAlias& BindingIn, __in PVOID);
#if defined(_AMD64_)
    static LONG     Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount);
#endif

While the rest of it looks like this (and again, this is partial)- yes, feel free to hate my stylistic indifference to the herd- perhaps I’ll get fired for this, and all can breathe a vast sigh of relief…

First, the code that calls the filter

        MangledDriverEntry  CrashAndBurn = (MangledDriverEntry) ((PBYTE) DriverImage + EntryRva);

        __try
        {
            CrashAndBurn(Binding, &Binding);
        }
#if !defined(_AMD64_)
        __except(EXCEPTION_EXECUTE_HANDLER)
#else     
        __except(Filterx64Exception(GetExceptionInformation(), OurFakeTickCount))
#endif
        {
        }

and now our filter:

#if defined(_AMD64_)
/**********************************************************************************************************************************

LONG    AllKMDFDrivers::Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount)

Ahh, the joys of low-level intervention by the truly incorrigible!  On AMD64, we will get an exception when the code tries to get
the tick count.  This value resides at a known address (which cannot change because if it did, all existing drivers would then fail
to work- although I suppose someone could resort to what I am about to do- bring it on, I'll see if I can keep up with the absurd
arms race).

So, first, verify from the exception record that we are getting an access violation of some sort reading that known address.  Then
see if one of the integer registers has that address (start with RAX, since that's where it currently is, and I bet it doesn't
change much).  If it does, change it to point to the value given to this function (which resides in WdfVerifier, since I made it
a reference, and will let the compiler play enforcer), and then continue execution.

In all other cases, execute the handler, which will do nothing (meaning the version will continue to be unknown).

**********************************************************************************************************************************/

LONG    AllKMDFDrivers::Filterx64Exception(__in EXCEPTION_POINTERS* ExceptionInfo, __in LONGLONG& FakeTickCount)
{

    //  cf definition of SharedTickCount in wdm.h
    static const DWORD64    KnownTickCountAddress = 0xFFFFF78000000320UI64;

    if  (NULL == ExceptionInfo || NULL == ExceptionInfo->ExceptionRecord || NULL == ExceptionInfo->ContextRecord)
        return  EXCEPTION_EXECUTE_HANDLER;

    switch  (ExceptionInfo->ExceptionRecord->ExceptionCode)
    {
    case    EXCEPTION_ACCESS_VIOLATION:
    case    EXCEPTION_IN_PAGE_ERROR:    //  Unlikely, but what the heck...
        if  (2 > ExceptionInfo->ExceptionRecord->NumberParameters)
            return  EXCEPTION_EXECUTE_HANDLER;
        break;

    default:
        return  EXCEPTION_EXECUTE_HANDLER;
    }

    if  (EXCEPTION_NONCONTINUABLE == ExceptionInfo->ExceptionRecord->ExceptionFlags)
        return  EXCEPTION_EXECUTE_HANDLER;

    if  (EXCEPTION_READ_FAULT != ExceptionInfo->ExceptionRecord->ExceptionInformation[0] ||
        KnownTickCountAddress != ExceptionInfo->ExceptionRecord->ExceptionInformation[1])
        return  EXCEPTION_EXECUTE_HANDLER;

    if  (KnownTickCountAddress == ExceptionInfo->ContextRecord->Rax)
    {
        ExceptionInfo->ContextRecord->Rax = (DWORD64) &FakeTickCount;
            return  EXCEPTION_CONTINUE_EXECUTION;
    }
    else if  (KnownTickCountAddress == ExceptionInfo->ContextRecord->Rbx)

The rest I leave as an exercise to the reader, having already put all 0.378 of you through the treadmill today…

DDC starts in less than an hour

BobKjelgaard — Mon, 29 Sep 2008 16:21:36 GMT

Plenty of good stuff for today- begin with Eliyas and Peter going over what's new in WDF for Windows 7 (or as we prefer to think of it WDF 1.9, since much of it also works all the back to Windows 2000 [KMDF] or Windows XP [UMDF]). It continues later with our first presentation of talks on the "shared secrets" of the Windows Driver Frameworks. The first talk discusses the implementation of the frameworks themselves, while the second presentation (my first of this conference, and my first in about 16 years) discusses how the frameworks are tested. The latter talk will also be one of the final talks of the day.

Tomorrow will include Neslihan's talk about getting a logo for your WDF driver (yes we are adding logo requirements specifically for WDF drivers- primarily for KMDF at the moment). It will include some excellent talks by Praveen Rao about work we have done on WDF drivers in the storage space, and again the final talk (at least in that room) is Ilias and I discussing the WDF coinstallers.

Wednesday we'll repeat all the talks I am in- the shared secrets talks will be back to back in the morning, but I won't catch part 1 that day because I'll be in another room with Ilias repeating the presentation on the coinstallers.

There's so much more, too- SDV, DSF, the innards of driver verifier, annotations for PFD and SDV, new test techniques from my colleagues James Moe and Dieter Achstettler, excellent talks on device installation, many driver technologies (several on printing, I can't help but notice). I could go on and on, except I've got some work to get to this morning [and I'll be leaving soon so I can register and begin my bit of supporting the conference].

Ask the experts session tomorrow night, bowling tonight, space set aside for us to talk with developers before and after talks, labs, panel discussions, Q & A with our most senior engineering staff, and a few surprises I'm not at liberty to mention even now.

My plan for now is to try to attend the whole thing- how well that works depends upon how things go in the lab [we've got some critical work going on, but let's see how the junior staff handles things for a bit], and whether I'm providing any benefit by being there [if I can be more effective working on test tasks in my office, I'll return to it]. Of course, assuming my laptop works there, I may just blog a bit more than usual. Not having a cell phone, it might be a bit hard for people to contact me if they need me back at the office [although they will know where I am, anyway].

So, I hope to see a few of my readers there (both those within Microsoft, and especially those from the outside). Hope all enjoy there time.

Now playing: The acid rock classic jam "Dark Star" from Live / Dead (of course by the Grateful Dead)

Now for a different UMDF Coinstaller story...

BobKjelgaard — Sat, 12 Apr 2008 01:40:08 GMT

This one can never affect you, so your blood pressure can start easing now. It's just been a while since I tried to put on my "war stories" hat and tried tellin' one of them tales...

In the Beginning was...

the need to make sure drivers we build for test are signed so we can automate their usage and avoid all the nasty workarounds unsigned driver installation entails- particularly since we test on so many operating system platforms. In the early versions of KMDF, we did this as part of our normal build process, getting them signed just like all the rest of the OS is. Even when we went to produce releases (this happens in build lines not under our direct control) they built all the samples and test code as well, and life was good. We always had a coinstaller with the right update packages in it, and everything was signed.

You still see vestiges of those halcyon days in the KMDF samples, where entries for the catalog kmdfsamples,cat can be found even now [and this will be the case for as long as we are responsible for them]. All of our test drivers are also catalogued in there, so we can mix and match with impunity and it all works as long as it's from the same build.

The price of success

But we also became a part of the OS beginning with Windows Vista. Now for a host of reasons, the build lines that build the OS produce coinstallers for KMDF and UMDF that contain no update packages. We did continue to build coinstallers with update packages on our private build machines, though. We utilized various nefarious techniques to undo the system's file protection and place KMDF on a Vista machine when testing so we could test our latest versions, so things still weren't too bad. But sometimes we wanted to use coinstallers and test binaries from differing builds- signing was becoming a problem. More importantly, when we approached WDK release times, our external builders now only produced the coinstallers. So we no longer had a single nice signed package automagically produced for us.

We worked around this as best we could- typically one of the SDETs assembled a "build" out of disparate pieces and then re-ran the signing steps with a bcz in the proper directory. Like all manual processes, errors happened, but we muddled along.

Finally, late last year things got to be too much for Shefali and I- we had to run really old test content and found the signatures were no longer valid. I should explain that a bit, if I can. Developers working on Windows have certificates created for them identifying components they build that chain to a special test root certificate (a term you can look up, for instance- this MSDN article touches upon them) that is recognized by most interim builds produced of Windows. This means all of our content is "signed" (and it also means that if any of those signed binaries show up where they should not, they identify who produced it, giving one easy place to start searching for a leak). When we approach releases a switch is made to more official forms of signing- our test drivers are also part of those builds, although they never ship to anyone, so we're still good to install on those- but we can't use them anywhere else because the coinstallers are still what we call "thin" [no update packages]. Of course, since nobody should need them for very long, those certificates also have a very short shelf life, which is what I was referring to at the start of this rambling paragraph of mine.

We also had a hard time making clear to the software test engineers who were trying to run our rapidly changing test mixes what parameters to use (or even to figure out for ourselves which combinations of parameters really did what). This led to delays, confusion, dissatisfaction [one of those poor STEs must have been sure he was on the verge of being dismissed- and that bothered me because I knew it wasn't all his fault], and other generally bad and stressful things.

If you want it done right, DIY

So, if self-signing driver packages and using test-signing approaches is good enough for our customers, it ought to be good enough for us. I redesigned our entire automation process around this approach (with much encouragement and prodding from Shefali). I solved both problems at the same time, but since I like to wander when telling tales, and I'm the one with the keyboard, I shall tangent...

Too many cooks

Normally a good test automation design in WTT (known to you as DTM) is fairly self-contained. It gets its stuff, does its work, and cleans up after itself. You see this in the three phases- setup, regular, and cleanup. Virtually all of our test jobs worked this way, meaning any one could run independently of the others.

We have literally hundreds of jobs that work this way- the bulk of them testing the KMDF DDI. They took parameters with little bits and pieces of path names [because most of the time everything came from a specific machine, or if the machine changed, parts of the path were known, etc] and assembled them together to locate things, install them, run them, and clean up.

But the names weren't consistent, the portions mapped weren't consistent and while it was sometimes possible to get a correct path by using .. in path entries and even blank entries for some parameters, determining those values was a logic puzzle in and of itself. Worse, you couldn't be sure after a run that all of those jobs had really run the same thing. Since I now found myself the only cook left in the KMDF QA kitchen, I took advantage of the situation to impose order on the chaos.

Slicing the knot

The story of Alexander the Great and the Gordian Knot has been with me all these last few weeks for some reason, and this may have been another of my "Brute Force" solutions. I broke our test pass into three stages:

Staging- in this phase, all of the tools and content used is copied from disparate and myriad sources to the test machine in a known location. Common tools like DSF are installed. The key to the underlying narrative here is that this job also creates a test certificate on the machine, creates a catalog containing the entire contents that had been copied earlier, and signs that catalog with the new certificate. It then sets the machine up so it works with test-signed binaries effectively. So there's still a kmdfsamples.cat- but now it gets built fresh and piping hot right at your table [that thought makes me want to visit Benihana].
Setting the framework on the machine. In this phase, if we need to overwrite the normal version of KMDF already there we do- either brute force (by overriding the system protection on it) or elegantly (by using a "fat" coinstaller containing the appropriate update package). As mentioned somewhere in here, you have to reboot the machine if the coinstaller is used (in fact, you have to do it either way). Sometimes we don't even need this phase [XP, for instance]. Whenever possible we try to utilize real coinstallers in random configurations to more closely duplicate the end user experience, after all.
The tests themselves.

I had a single ground rule- only the staging job would have any parameters. All the subsequent jobs would use what was staged. There was a corollary based on previous experience: those parameters would be substantially complete path names. They might take longer to type, but it was easier to switch and accommodate quirks in how paths were assembled as you tried to get things from elsewhere if you just always took entire paths.

I finished most of that work in one weekend (in November, if I remember correctly). The most mind-numbing part of it was modifying the existing tests- I'd go through task by task with the new "known" staged path in the clipboard, and selected each directory name I found and pasted it in. There were some deviations that I wound up adjusting in the initial setup job because they were done too many places. There were places where parameters were passed down into library jobs that I left untouched (I actually set any such parameters to totally invalid values to make sure nothing escaped my wrath).

After all that surgery I now had the ability to mix and match with much greater flexibility, and simple instructions with four basic parameters that covered all the known variations we had seen. I then created a Wiki on the internal Microsoft network where I listed the instructions for all of the normal passes we did so we could clearly communicate what settings were to be used each day- at first, the STE could literally cut and paste from my instructions. Once they were familiar with the new setup, they could do more of the work themselves. You wouldn't recognize that same STE today.

It worked pretty well, even if underneath there are a lot of rough edges (if you like clean setups, this isn't one- the sheer scale of the task is too big to justify yet). It also made testing test changes easier- I build everything on my machine, and can schedule a job to pick up the content from there. If I'm doing even more aggressive mixing and matching than usual, I actually assemble the binaries on the test machine and let the setup job copy them from there into the new official staged location [one part of the hard drive to another, but it's all with a tool we use continually and rarly needs to be done]. To be fully fair, I should add that I didn't get all the tests at this time, just the ones that we absolutely had to keep running. For instance, our stress mix fell out. But Shefali later chipped in on her own and got them working.

Life was good- and there was now time to work on problems in the test code instead of trying to figure out how to continually tweak creaky automation into doing something slightly different every few days. Shefali seemed pleased, and what the heck, making the boss happy is generally a good idea in the business world...

Trouble in Paradise

Until I deployed a new test. Or rather a new variation on an old test. I have a rather elaborate setup I use to verify operation of the IoTarget and IRP processing function in KMDF [although I don't go totally into queues- just the very basic configurations], and to support some new features in WDF 1.7 you'll be hearing about soon, I added UMDF drivers into that test. I had to add another parameter to make sure I had all the flexibility I needed in finding a UMDF update coinstaller, but that's not a problem. I put it all together, tested it quite a bit, rejoicing somewhat in how easy this new process made it for me to do a test that was now creating 88 different devices on top of a virtual test bus, installing the proper drivers with no popups in sight, and then putting those devices through their paces- and it looked good. So I called it complete. got it reviewed, and checked it all in. [For the 2 or 3 regular readers [overestimating my impact again?] this is the test with the targets and "hunters" where I showed some code here].

But early this week, it failed. Makecat wouldn't process the UMDF coinstaller from our own build machines (to debug it, I forced WTT to halt when this failed- we were losing logs due to some problems not worth going into here- the following is an email snippet):

I set the task to freeze if it fails. This is weird- this is the tail of the self-sign log:

processing: <hash>C:\kmdftest\WUDFSvc.dll.mui

processing: <hash>C:\kmdftest\WUDFUpdate_01007.dll

NOT processed: calculating the indirect data (C:\kmdftest\WUDFUpdate_01009.dll)

Failed: CryptCATCDFEnumMembersByCDFTagEx. Last Error: 0x80004005

Errors found in parsing the CDF file

A comedy of errors ensued for a while after that, as I tried to find out why the 1.7 RC1 coinstaller was there when the job parameters I was told used pointed to locations that couldn't have contained it [if I'd dug into the job reports, I'd have seen that when the set they gave didn't work, they pointed to a server containing that and tried it again]. Once that was settled, I began focusing on why makecat was giving me the ever-so-helpful E_FAIL parsing a file that seemed perfectly good.

Well, it was the file itself for some reason- take it out of the CDF, makecat worked. Have it as the only one in the CDF, same error. Since we recently had some changes made, I was wondering how they could affect hashing the file- so I went back and tried earlier versions. This led to another comedy of errors when I inadvertently mistyped a path and the process worked [because it couldn't find the coinstaller, and without going too deep into why, I couldn't treat it as an error at this point in our setup job].

Well, the world was looking strange- I know I ran this dozens of times while I was developing this, didn't I? We'd run it the previous week, and there'd been no problems. It got stranger when I ran the same thing on a Windows 7 machine, and it worked flawlessly. Why should the OS have made a difference? It was the same set of binaries, tools and all... Now if I weren't stressed and well-befuddled by then, I might not have continued to be so stressed and befuddled at that point, but of course I was and I did...

But this is serious after all- in this design, all the test pass eggs are in one basket called the staging or "Unified Setup Job" and with it broken, NOTHING works!

Fools Rush In

And this old fool is no exception.

Ilias sends me an email about the problem late in the day with this link and the comment "real programmers use butterflies". I had decided what I was going to do, so at about 5 AM the next morning (I started sometime between 1 and 2 AM) in reply I said, "A sledgehammer is more my style"- ahh, well- afterthought says "Real SDETs use sledgehammers" might have been a better retort...

Onward- "Take the bull by the horns!", Papa sez to himself, and loads the debugger package on the machine. Point it to makecat, give it the command line to process the CDF, and go. Make sure we've got all the symbols [miracle of all, they were there the first time!], and set a breakpoint on the routine name which was most helpfully displayed in that error message above, and go. Then step into the code [now while I did have symbols, I don't normally need to work with that part of the Windows source, and this is Windows 2003, anyway- so I'm doing it the old-fashioned way, reading the assembler and using the old noggin to cipher out what's up... I wasn't totally cheating- but because I had symbols I could see internal names and also the names and types of local variables, so I wasn't flying entirely blind].

Now before I did this, I went through a phase where I thought there was a defect I could note externally that would tell me what had happened- and in the process, I dumped the headers of the coinstaller with the linker (link /dump /headers <file name>). It seems to me that my long-term memory is beginning to suffer the ravages of age, but short-term is still pretty good, so I still remember things like this:

    10 number of directories
11440 [      8D] RVA [size] of Export Directory
10974 [      50] RVA [size] of Import Directory
15000 [ 12A53C] RVA [size] of Resource Directory
     0 [       0] RVA [size] of Exception Directory
13400 [     FB8] RVA [size] of Certificates Directory
140000 [     AE0] RVA [size] of Base Relocation Directory
1210 [      1C] RVA [size] of Debug Directory
     0 [       0] RVA [size] of Architecture Directory
     0 [       0] RVA [size] of Global Pointer Directory
     0 [       0] RVA [size] of Thread Storage Directory
5530 [      40] RVA [size] of Load Configuration Directory
     0 [       0] RVA [size] of Bound Import Directory
1000 [     1D0] RVA [size] of Import Address Table Directory
     0 [       0] RVA [size] of Delay Import Directory
     0 [       0] RVA [size] of COM Descriptor Directory
     0 [       0] RVA [size] of Reserved Directory

what was odd, was that even though this says a certificate was there, I couldn't see one in Explorer. Odd, but it didn't raise any red flags to this old bull, so on he went.

Well, after much digging and a bit of backtracking, I found the place makecat decided to make that error. So I followed the preceding call deeper and deeper and got into code that was preparing to hash the binary and was looking for parts of the PE image to exclude. Now it happens I've done lots of hacking to binaries- stripping resources out, putting them back in, altering tables and all sort of general mayhem, so following this code is a snap, even in assembler [with those handy locals about, anyway]. I find a path where it is clearly failing, and looking back up through the registers shown as I single-stepped the code, the values FB8 and 13400 caught my eye. Hurrah for what memory remains! A quick check confirmed they were the header values. Bashing them against the values in dv, I had my cause...

It had refused to hash the binary because the certificate was not at the end of the file's memory image- specifically, the resources followed it. It turns out this also caused the certificate to be invisible to explorer and made signtool verify most unhappy [but signtool also happily replaced the certificate in situ every time I tried, alas].

I then sent a rather rambling and somewhat incoherent email to Ilias telling him we'd been building an unsignable UMDF coinstaller [too much stress and too little sleep- forgot that it had worked in Windows 7] since time immemorial. Probably boosted his blood pressure since I wasn't all that clear I meant only on our private build line [obviously WHQL signs the official versions from time to time, after all]. I also wasn't making any clear distinction between signing the binary by embedding a certificate and signing it by having it properly hashed in a catalog signed with an embedded certificate [that duplication of terms has always been a source of confusion] . After a face to face and some more coherent and detailed explanations from me we had it down- that only the UMDF coinstaller from our private line was unsignable, and then only on Windows 2003 and earlier [I'm afraid you'll have to repeat some of what I did to see why]...

As I describe here, I can disassemble the coinstallers quite readily and in converse I know how they're put together- the problem is clearly that we signed it before we added the update package as a resource [now the package that does this could just be made smarter...]. We found out how that's happening, but fixing it is proving a challenge.

Well, the main build lines have well-funded and trained staff to handle all those scripts that handle all those things we do after build- on a private line like ours, you have something a bit more seat of the pants, and Ilias is not the originator of most of those scripts. If you've been around software long enough, you probably get the picture- logging not quite up to snuff, commentary a bit lacking, and so on. He's still working on it [actually, he's going on vacation, so my old buddy Kumar probably gets to hold this hot potato].

So you unhappy souls we've held up with the Server WDK [and I mean this with all sympathy and respect- you've got good reason to feel that way] aren't the only ones with coinstaller issues- but at least this one is never going to affect you.

I also got to tell him somewhere in the middle of all that about my fascination with the Gordian Knot, him being Greek and all [alas, I kept wondering if Greeks regarded Alexander as Greek, since Alexander was Macedonian- but I kept saying Mycenaean and totally fudging the issue- that aging memory again]. This time I figured my brute force approach to the knot was debugging it myself instead of doing the usual thing and trying to find someone who could just tell me or wanted to find out why something didn't work on such an old operating system [lets face it, the main focus is on Windows 7 around here]. He admitted to being one of those closet WdfVerifier users I occasionally speculate exist, and so it went [our conversations are usually quite a bit of fun, even when the situation isn't all that much fun- similar sense of humor, perhaps].

Those who hate my endless music lists can rejoice- this was much too long to bother trying to accumulate one. But I at least got to hear all that good stuff (ahh, Garcia's "Bird Song"- "tell me all that you know- I'll show you snow and rain")

L8r!

So Nice I Tried it Twice

BobKjelgaard — Fri, 01 Feb 2008 03:58:12 GMT

Yeah, the disco daze wuz good uns... As long as I'm on the "Papa B Dumb" kick, I'll add a most embarrassing errata for WdfVerifier- this will be there in the official 6001 WDK, much to my eternal shame...

With all the things to test on WdfVerifier, you'd think Papa would make sure that UMDF debugger launching would work with the default debugger locations from installing Debugging Tools For Windows. Papa sure would have said so, right up until he did his late-in-the game manual tests for said WDK.

Papa forgot that if a path has spaces in it, you need to use double quotes around it to form a working command line. Well, actually not even Papa would forget that- as soon as the debugger launched and then complained it couldn't find "Files\Debugging Tools For Windows 64-bit -<debugger switches papa done fergot>" he knew what he'd done, and he knew it wasn't good, and he also knew it was too late to do anything about it but realize he was a dunderhead and that work item he filed for getting some regular automated testing was a lot more necessary than hoped.

Not what anyone should expect from QA. Live I shall, learn- well, we'll find out about that one if I'm lucky enough to get another shot at that sort of thing.

There is a workaround, but it's ugly- install your package (or copy it after installation) to a path with no spaces in it.

It was easy enough to fix, of course- but it won't get out there for a while, now.

On the bright side, at least, I'm not sure anybody actually uses this feature [nobody mentioned this flaw, after all], nor this tool. So p'rhaps 'tis much ado about nothing...

With that thought, I'll go see if there's anything useful I could be doing.

Papa's "Wall of Dumb"

BobKjelgaard — Fri, 01 Feb 2008 02:06:42 GMT

If memory serves correctly, always an iffy thing with us old fogies, Chloe Sullivan (on "Smallville") maintained a "Wall of Weird" covering all the strange events in her benighted hometown.

So Papa's decided to inaugurate his "Wall of Dumb" today- but don't worry about showing up on it- this wall is for Papa's dumb ideas. If I get ambitious enough to think about and and describe all of those I've had, 'twould be a grand wall indeed!

Today's entry (the fuel for this disgruntled posting) is a DDI test for a new KMDF 1.9 feature. Can't tell you about the feature, of course, but as a result of one of the relevant DDI, the driver writer can allocate large numbers of objects if he or she so desires. The original plan had a cap on the amount you could request, so of course I had a test that hit just above that mark. But then the cap was removed, the idea being to allow as much as there was memory for.

So Papa said- "hey fine, the number I want is a ULONG, so I'll ask for 4 billion or so [(ULONG) -1]- that ought to fail eventually [sure will on any machine I own], and I can verify it all cleans up nicely afterwards, and so forth". Possibly a dumb idea as it turned out, but we'll get back to that.

Later Papa was looking at the possibility that if there was a bug in the driver or framework, he might hang forever, so he put an explicit 20 second timeout on each test. The timeout was reported, and the test had very specific history tracking and such that it would be easy to look at the logs and see just about where it hung. The idea here being that an automated run would detect and report the failure, the logs would give a good idea where to look, and if needed, good clues for either code review of suspect areas or good breakpoints for repro could be easily determined. Thinking ahead, Papa thought he was.

Perhaps you see where this is headed...

Papa ran his test, debugged it, found bugs in the feature, and eventually got it into the regular automation runs for KMDF. All was well, the test had no false positives or negatives, and no feature regressions turned up [in part because by then it was the winter holidays and all the folks that could have been breaking stuff were on vacation].

Meanwhile, in other parts of Testland, trouble was a brewin'- older test drivers developed by more junior folks and at a time when we were still becoming familiar with the technologies that would just as soon bugcheck as look at ya. So after picking out the worst of that lint, Papa enabled driver verifier on the framework runtime, loader and the most likely of the corrupting drivers.

In some ways this was a very good thing. New bugs were found, even some framework bugs we should have been catching sooner. Many more horrible test bugs also surfaced- nice icky big ones that steal your lunch when you're not looking and slap you in the back of your head with their tails as the scuttle out of sight... Papa had more fun than an SDET should trying to squash all them 'orrible critters.

But Papa's prize test, the one where he pushed some gnarly hacks for quicker tracking and coding, and where he did things like tear down one software device stack while spawning another [to increase throughput at runtime, of course]- it began crashing, timing out on one test, and sometimes even bugchecking. Papa was mortified.

But Papa was very busy with all his other problems [including some installation issues people have presented through this blog- keep 'em coming], so he just had to wait for some time to investigate, wondering in the meantime what could be wrong with his bus driver [had to be the bus driver- couldn't possibly be an issue with the newest code- that blind spot has always troubled that old dude].

So today he finally got his chance. The test quickly hit the overlong case, so Papa broke into the debugger, set up his symbols, loaded the extension, and thinking "well, let's see how far this got- maybe I'll just see what handles and such are in my driver":

!wddfdriverinfo <DriverNameSuppressedToPreventPapaBeingDismissedForLeakingConfidenshulInfermashuns> ff

That command is still running, and it's been over an hour, and it will probably be a lot longer- the same object type, over and over.

So let's see, standard settings include special pool, so instead of each object taking a few bytes, it consumes a page of pool. Pages are fewer in number than bytes, and there is extra overhead for all that neat tracking driver verifier does. So that 20 seconds that was enough with it off to run out of memory wasn't near enough with special pool on.

Papa is an idiot. Really should have seen that one coming...

WdfVerifier tricks- Controlling DbgPrint on Vista and later OS

BobKjelgaard — Wed, 17 Oct 2007 01:35:00 GMT

This is one you can use even if you never use WDF at all (of course, you may find WdfVerifier to be a bit annoying for such a use, but if you've got a memory like mine...). I wound up doing this a couple of days back, and thought it would make good fodder for the "gotta add more blog content" jones.

As I mentioned earlier, the KMDF loader (since it has to work all the way back to Win2K) uses DbgPrint for its diagnostic messages. Having no memory, I thought it would be nice if WdfVerifier took care of turning it on and off for me so I wouldn't have to look up the post in Doron's blog that I used as a crutch whenever I had to do this.

As a result, the latest version (6001 RC0 WDK, available on an MS Connect near you) of WdfVerifier will turn DbgPrint on and off- but it is actually more precise than the method listed almost anywhere else- it enables only the specific mask bit the DbgPrint implementation actually uses! Of course, almost nobody bothering to use DbgPrintEx uses the default setting, but hey, if they ever do- I've already got you covered! With my usual attempt to be thoughtful, you can configure the way this support works to suit you, which is crucial to this discussion.

I'm going to assume for this post that you could care less about the KMDF loader, or KMDF in general- but you use DbgPrint, and find having to look up that key name or remembering the debugger method more than a bit annoying. WdfVerifier just might make even one WDF-less development task a tad less annoying.

To begin, you point WdfVerifier at your Vista or Server 2008 box (or copy it there and run it directly- whichever method matches your working style). You then:

Go to the Tool Preferences tab, and go down to the group box entitled On Committing changes to:, and drop down the list next to KMDF verifier settings:
Select the item named Simply Exit With No Prompts, which of course I now think should have been labeled something else, since "Apply" won't actually exit anything...
Now go to the group box labeled Vista DbgPrint Enabling.
If you want to turn DbgPrint ON, select Turn DbgPrint output on if needed for loader diagnostics, but do not turn it off.
If you want to turn it OFF, select Turn DbgPrint output on and off as I turn KMDF Loader diagnostics on and off.
Now go to the Kernel Mode Driver Settings tab.
Click the Checkbox Enable KMDF Loader Diagnostic Messages to Kernel Debugger, then press Apply.
Repeat the last step.
If the checkbox is ON (meaning it was on when you started, which would be rather weird if you don't care about KMDF), do it a third time.

So that checkbox is always OFF when you exit. What that means is DbgPrint ON or OFF, you won't have any KMDF loader messages interfering with your trace.

How all that worked ought to be self-explanatory, of course. But you know, it might be more convenient than firing up RegEdit, remembering key names and keywords, etc. Just a few clicks and very little typing. It reads harder than it does, if you catches my drift...

That's how I like my tools- but after all, it's both optional and free- if you don't like it, no need to use it...

The adept among you no doubt noticed I didn't use a spell checker, and my first group box typo'd "committing". I used to have no problem with that rule, but after stubbing my mind against "editing" so many times, I now can mess up all my doubled consonants...

Ah well, that and the second bullet are what next versions are for- assuming anyone actually uses it and we don't decide to use that valuable WDK space for something useful...

WdfVerifier Basics- Assistance in debugging UMDF drivers

BobKjelgaard — Mon, 08 Oct 2007 23:01:00 GMT

One of the most interesting set of requests for the WdfVerifier tool in its latest incarnation involved simplifying the attachment of debuggers to UMDF drivers. I believe that WdfVerifier can now be used to great advantage here. Without it, you must find the PID (Process ID) of the host process in which your driver is running, and then launch a debugger and direct it either through the command line or (in the case of WinDbg or the visual studio debugger) its UI to attach to that PID.

The earliest version of WdfVerifier which appeared in the WDK at least listed the PIDs, but you were still on your own to attach the debugger. But we wanted to do better by our developer users, and I believe we now have.

This is a list of things WdfVerifier will assist the UMDF driver developer with:

If you use the Debugging Tools For Windows package, it can usually find all of the debuggers for you and can pick whichever is your favorite for launching a debugging session either when needed, or at the click of a button.
If it can't find the debuggers, it is easily pointed to them.
You can use it to force your driver to be restarted if you want to attach a debugger at process start or driver load (although driver load is the logical choice 90+% of the time).
You can use a different debugger as long as it can handle getting a PID in the command line.
You can attach a debugger to any running UMDF driver at the click of a button.
The debugger command line which is built in is sufficient for most uses, but you can easily change it if you wish to.

There are also extended controls for UMDF in this version, including control of framework tracing, failure retries, and kernel debugger interactions (for those that simply must live the hard way).

Debugger Auto-launch- or automatic "Set and forget" debugger attachment.

This is how we've referred to one of these features. If you have asked the host process to wait for debugger attachment (all you need to do is enter a non-zero number in the edit control between "wait" and "seconds" about halfway down the tab), and your debugger preference is confirmed (meaning WdfVerifier was able to find the executable file for the debugger), then the "Automatically launch user-mode debugger when requested" checkbox is available.

If that checkbox is on, then WdfVerifier watches for new UMDF host processes to start running. When one does, it automatically starts the preferred debugger, with the proper command line to attach it to the process. Since this is an asynchronous process, I recommend you set the wait to at least 3-5 seconds- but that is still much faster than what you can normally do if you're trying to look it all up yourself. Most of the time when I try it, it seems virtually instantaneous (a setting of "1" works for me, but why push your luck?).

I've found this useful in conjunction with a second feature: if you change trace settings for UMDF, they do not apply until all host processes exit and are restarted. So WdfVerifier will ask if you want this done when those settings change. So, for safe and sure attachment of a debugger to your driver, just set up the auto-launch feature, then change a trace setting (my favorite is toggling the "Send log output to the kernel debugger" setting, since I don't often need one in debugging UMDF drivers) and press "Apply now". In a few short seconds, I have a debugger session up with my driver ready to debug.

Why not just attach to a running driver at any time?

You can do this by selecting the PID from the list on the UMDF tab, and pressing a button. So that was also made easy- however, if you did NOT have UMDF set up for user mode debugging at the time that particular process started, then the kernel-side reflector will think the process is hung if you have it broken in the debugger. If this happens when it tries to handle a PnP or power event, the process you are trying to debug will be shut down. Since that's probably not what you want, I usually recommend the auto-launch method.

Feedback?

Let us know what you think of these features, or if there is anything else that might make life better as a UMDF developer. If we can make UMDF development easier to do as well as having it produce a product that is safer for end users, I think everybody wins- which isn't such a bad outcome!

Additionally, I'm not certain any further description of this tool is needed. If there's something else you want to know about, send some feedback on that, as well.

Root Causing a Not Reproducible KMDF Installation Issue- part 3: Pay Dirt!

BobKjelgaard — Thu, 16 Aug 2007 19:00:00 GMT

I'll begin by recapping where we've been. In part 1, an interesting installation issue was reported- interesting in that it was 100% reproducible on customer machines scattered about Europe, but not all such machines, and neither we nor the IHV who reported it to us could reproduce it. After we got enough information to understand the symptoms, a workaround was provided to the IHV so their customers could use their devices, while we worked to identify the underlying cause. In part 2, an attempt that should have been successful to capture the access violation behind it all failed, and I am left wondering Where Did I Go Wrong?

Rechecking assumptions

For a while, I reviewed previously known issues with the developers and/or PMs for the various installer technologies. Many were registry based- I spend hours one day running the entire installation under a debugger checking every registry access, planning if necessary to script the debugger to log them all for me the next time I tried a pass at the customer's machine.

We also discussed the issue quite a bit. At one point we were considering buying / renting /borrowing one of the failing systems if we still couldn't find a way to solve the problem remotely.

I also reviewed the coinstaller source code, and it was this review, coupled with all that time in the debugger on a working system, that finally revealed the worst of my problems.

Doh!

Can't say it better than Homer! Update.exe is where we've had problems, and the source code mentions it by name. BUT, it is not the program we extracted from the coinstaller, spawned and were checking the results of! Instead, that program is a self-extracting CAB which has a very long name identifying the target OS and KMDF version. In this particular case, it is Microsoft Kernel-Mode Driver Framework Install-v1.1-WinXP.exe. The fatal access violation probably happened there after update.exe completed successfully!

Try, Try Again

After making such a blunder, I decide to be more careful the next time around. This time, I script the debugger (after all an end user's machine is the only place this happens) to:

Debug the program and all of its child processes
Take and overwrite the AV dump on each occurrence as before, but also
Create a specific dump after each process exits (this is a point at which the debugger normally breaks, but I had turned it off on the previous try).
Keep a log of everything for further reference in case I am making even more mistakes.

So I will have three dumps when all is done- the final two will include the stack with the ExitProcess call on it, and I can tell from such a trace what the return code was, so I will know (assusming as I had that the application is returning the code) which application set that code..

The new registry file (note that the switches and key identity have changed):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Microsoft Kernel-Mode Driver Framework Install-v1.1-WinXP.exe]
"Debugger"="C:\\Program Files\\Debugging Tools For Windows\\Cdb.Exe -o -logo C:\\Dumps\\Debugger.Log -c \"$<C:\\\\Dumps\\\\Script.Txt\""

Second try- debug the self-extracting CAB and its children, also keep a log for later analysis.

The script is also a bit more complicated:

sxd -c ".dump /ma /o C:\\Dumps\\FinalAV.Dmp" av;g;g;.dump /ma /o C:\Dumps\EndUpdate.Dmp;g;.dump /ma /o C:\Dumps\EndExtractor.Dmp;q

The new script is a bit more sophisticated, but still primitive, compared to what I've seen elsewhere.

For this, I'll give a brief breakdown:

sxd -c ".dump /ma /o C:\\Dumps\\FinalAV.Dmp" av

Issue the .dump command (this creates an "all process-related memory" user mode minidump, which is large but usually has everything you're likely to want) each time an access violation occurs- report the AV in the debugger log, but don't interfere with the normal handling of it by the program which is being debugged.

Proceed with the self-extracting CAB. The debugger will break again when update.exe, which is a child process of the one we began debugging, has been loaded and is ready to begin execution.

Let update.exe run until it ends (I know this will happen because it did the last time we tried this). I expect the debugger to break again when the process is exited.

.dump /ma /o C:\Dumps\EndUpdate.Dmp

Create a similar user mode minidump for update.exe at exit. The kb debugger command will show me the exit code from the process, and if I need to look at anything, there should at least be traces in this dump.

Resume execution now of the self-extracting CAB. It should normally break when the CAB process exits.

.dump /ma /o C:\Dumps\EndExtractor.Dmp

Assuming this is the exit point for the extractor, create the final dump.

Leave the debugger and stop the cab process. If it wasn't done, we return anyway. I will be able to tell from the dump if I've missed something.

You must have accidentally fixed it, or sent me the wrong stuff- it works now!

That was one user's reaction. Actually, in exiting by quitting the debugger, I caused the coinstaller to see a successful completion- so it looked like a successful install, and everything else then worked. Good result, but not what I'd propose for a workaround!

Success!

This time, FinalAV.Dmp has exactly what I am looking for. It also reveals my fallacy for good- this was an unhandled access violation after all (so the Watson tool could have been deployed as a much more elegant solution). Still, I've got more data than Watson would normally provide, and you, gentle reader, now have a way to get a dump yourself if you ever find yourself in a similar situation...

For the rest, I'll return to the record- this snippet is from the email I sent to the IHV's engineer, edited to remove names to protect privacy and IP. The DWORD of 0's I highlighted in red is the return code passed to the Win32 ExitProcess API.

0:000> r

eax=00060800 ebx=00000001 ecx=00263e90 edx=00260608 esi=007a0031 edi=00000000

eip=10106022 esp=0006fd34 ebp=0006fd78 iopl=0         nv up ei ng nz ac pe cy

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00010297

xxxxxxxx!clearxxHook+0xxxxx:

10106022 8b4010          mov     eax,dword ptr [eax+10h] ds:0023:00060810=???????? <- Note this is the address <user name> reported (both the IP and Excepting address]

0:000> kb

ChildEBP RetAddr Args to Child

WARNING: Stack unwind information not available. Following frames may be wrong.

0006fd78 7c9111a7 10100000 00000000 00000001 xxxxxxxx!clearxxHook+0xxxx <- This is where the AV occurs

0006fd98 7c933f31 101081b4 10100000 00000000 ntdll!LdrpCallInitRoutine+0x14

0006fe1c 7c81cd76 0006ffc4 00082430 00000000 ntdll!LdrShutdownProcess+0x14f

0006ff10 7c81cdee 00000000 77e8f3b0 ffffffff kernel32!_ExitProcess+0x42

0006ff24 01005971 00000000 00011970 7c9218f1 kernel32!ExitProcess+0x14 <- IE we were returning 0 (ERROR_SUCCESS)

0006ff30 7c9218f1 0006fff0 7ffde000 00000000 Microsoft_Kernel_Mode_Driver_Framework_Install_v1_1_WinXP+0x5971

My first surprise- we are exiting the process when we take an access violation in cleanup code in a 3rd party's Windows global input hook DLL.

I poked around with the debugger until I had separated the failing routine to see if I could understand why it had the AV, and this snippet (edited for the privacy reasons noted above) finishes the story:

This is the offending code (with some guesses, perhaps incorrect, as to what it is doing):

10106010 51              push    ecx

10106011 56              push    esi

10106012 8b35f81a1110    mov     esi,dword ptr [xxxxxxxx!yyyyyyyy+0xxxxx (10111af8)] ß fixed address, so must be in data segment

10106018 8b06            mov     eax,dword ptr [esi]

1010601a 3bc6            cmp     eax,esi ß Looks like check for empty list (points to head)

1010601c 89442404        mov     dword ptr [esp+4],eax ß Current list head saved to local variable???

10106020 7427            je      xxxxxxxx!clearxxHook+0xxxxx (10106049) <- just leave if the list is empty

10106022 8b4010          mov     eax,dword ptr [eax+10h] ß Attempt to access some value in the list item (this is where the AV occurs)

10106025 85c0            test    eax,eax ß check for some embedded structure?

10106027 740f            je      xxxxxxxx!clearxxHook+0xxxxxx (10106038) ß if not in use, jump ahead to what looks like a memory free / deallocate?

10106029 50              push    eax <- looks like call to cleanup or deallocate said structure?

1010602a e887180000      call    xxxxxxxx!yyyyyyyy+0xxxxx (101078b6)

1010602f 8b35f81a1110    mov     esi,dword ptr [xxxxxxxx!yyyyyyyy+0xxxxx (10111af8)] <- get list head again (esi must not be preserved)

10106035 83c404          add     esp,4 ß must be __cdecl…

10106038 8d4c2404        lea     ecx,[esp+4] ß this looks like __thiscall which does something to the list item (cleanup / unlink?)

1010603c e8cff9ffff      call    xxxxxxxx!clearxxHook+0xxxxx (10105a10)

10106041 8b442404        mov     eax,dword ptr [esp+4] ß get next item and repeat if list still not empty

10106045 3bc6            cmp     eax,esi

10106047 75d9            jne     xxxxxxxx!clearxxHook+0xxxxx (10106022)

10106049 5e              pop     esi

1010604a 59              pop     ecx

1010604b c3              ret

My guesses (note, though, it’s possible the “list” was at least partially good, and we’re seeing something that actually happened after several passes through it)- they’re trying to walk a list, apparently, anchored in their data segment, but the list is pretty obviously trashed.

Why? Well, let’s look at the segment section that’s trashed…

0:000> dc 10111a80

10111a80 00000000 00000000 00000000 00000000 ................

10111a90 00000000 00000000 00000000 00000000 ................

10111aa0 00000000 00000000 00000000 00000000 ................

10111ab0 00000000 00000000 00000000 00000000 ................

10111ac0 00000000 00000000 7263696d 666f736f ........microsof

10111ad0 656b2074 6c656e72 646f6d2d 72642065 t kernel-mode dr

10111ae0 00000000 61726620 6f77656d 69206b72 .... framework i

10111af0 6174736e 762d6c6c 007a0031 00000000 nstall-v1.z.....

Kind of looks like they didn’t expect our program name to be as long as it is, and never checked (also looks like they stopped at the first period they found in the name, rather than the final one, since “.1-WinXP” would be needed to complete the full file name).

Copying it into their data segment trashed at least one critical structure (for instance, note that something wrote a NULL right in the middle of the name after it was presumably copied).

I’m doing an internet search to see if this is a known problem and if so, if there is a newer version of <Product> that fixes this, and we can make the dump available to <Vendor> if it isn’t and they need to see it.

Snippet of my email reporting back to the IHV- I was close- the cause reported back to us was a "byte count vs character count" problem in a Unicode->MBCS conversion.

While I didn't mention it there, I actually did a search through the dump for the address at which the buffer started, to see if I could decipher that routine, but with no symbols, and calls going on to snake through, I cut it short after several minutes. I clearly had enough to make an accurate report.

The Aftermath...

I did indeed find at least one 3rd party developer hit this bug before, but from the forum in which the report appeared, it did not sound like they actually reported it back to the producer of the faulty software. We have contacted the provider of the suspect software, and they have confirmed that there is a known defect in that software that has since been fixed. The product in this case is fairly old.

I am not going to identify them, though. If you are a developer using KMDF (this problem can affect all versions, and we have informed them of that), and you believe you are currently affected by this, contact us at kmdffdbk@microsoft.com, send me an email from this blog, or any other route you choose to get to us, and we will investigate the problem and make sure it gets solved if at all possible. All indications (and this matches the vendor's report back to us) is that there is a small though finite probability this problem will affect you. This isn't what I'd prefer to say, but it's the best I can do, for now.

Lessons?

I took away several, some of which apply to developers in a general sense:

You can often debug effectively even without source code. Strange as it may seem, I have no access to the source code for the (non-KMDF) Microsoft components involved, nor, of course, did I have it to the code which faulted.
An openly cooperative stance toward your customers can make a world of difference. In this case, our direct customer had such a stance with their end-user customers, and this transitive property of cooperativeness is what enabled us to finally identify the root cause.
No matter how carefully you design and test, things are going to happen when you integrate various pieces of technology that you could not have predicted.
If you're determined, persistent, and can establish a cooperative link to the person with a problem, you can solve that problem even if you can't reproduce it yourself. Maybe not every time, but that doesn't mean you shouldn't at least try.
Debugging Tools For Windows are AWESOME! I barely scratched the surface of their capability in this task.
It can take a lot of heads to solve a problem like this. Before being done, at least a dozen people contributed useful work in solving this problem.

Bye for a while...

Vacation coming- when I get back, I will do some articles on a WDK tool named WdfVerifier that should make it easier for you to test and debug your WDF drivers!

Root causing a not reproducible KMDF installation issue- part 2: Not stupid, merely human

BobKjelgaard — Fri, 10 Aug 2007 20:39:00 GMT

In the last installment, we had a workaround, so people could get on with their lives. BUT, there's still that problem of an access violation of unknown origin that happens every time we try to install KMDF 1.1 on a machine a continent or two away...

I'll get right to the stupid / human part. In my befuddled mind, I am convinced that this AV is being handled within the failing program itself. Actually, it is being handled by the OS code that manages the process. But being lucky, my "direct approach" is still a right one- if I try to use the normal post-mortem techniques, then the faulting stack will already have unwound to the exception handler that invoked a post-mortem debugger (however, the stack is typically still available using the debugger's .cxr command). Being a simpleton, I want to see what's actually failing, rather than try to make guesses [because weeks have elapsed, there's been a whole lotta guessin' goin' on, and nothing really useful has come of it]. So I may be about to make things harder than they needed to be.

So, the plan (simple minds like simple plans, and this is pure brute force]:

Have the user install the Debugging Tools For Windows package, using the default settings.
Create a REG file that uses Image File Execution Options to debug just the program I think is failing.
Create a debugger script that, as each access violation occurs (aka "first chance" exception), captures a rather exhaustive minidump, and just keeps doing this [overwriting it each time] until the process ends.
Provide batch files for setting it all up and tearing it down, so that the user isn't left with a big mess on their machine when we're done [we are guests, after all- if we can't do the dishes, we can at least pick up after ourselves].

The user received a ZIP file with several files and some instructions. The first file here is this batch file, run to start the process:

@echo off
@Echo Preparing to capture dumps from broken installation...
md c:\dumps
md c:\dumps\backups
@Echo Preparing to capture dumps from broken installation... > C:\Dumps\Preparation.log
reg import DebugUpdateExe.reg >> C:\Dumps\Preparation.log
net stop hypkern >> C:\Dumps\Preparation.log
net stop hypaudio >> C:\Dumps\Preparation.log
net stop wdf01000 >> C:\Dumps\Preparation.log
sc delete hypkern >> C:\Dumps\Preparation.log
sc delete hypaudio >> C:\Dumps\Preparation.log
sc delete wdf01000 >> C:\Dumps\Preparation.log
del %windir%\system32\drivers\hypkern.sys >> C:\Dumps\Preparation.log
del %windir%\system32\drivers\hypaudio.sys >> C:\Dumps\Preparation.log
del %windir%\system32\drivers\wdf01000.sys >> C:\Dumps\Preparation.log
del %windir%\system32\drivers\wdfldr.sys >> C:\Dumps\Preparation.log
copy script.txt c:\dumps >> C:\Dumps\Preparation.log
copy %windir%\setup*.log C:\Dumps\Backups >> C:\Dumps\Preparation.log
copy %windir%\wdf*.log c:\Dumps\Backups >> C:\Dumps\Preparation.log
del %windir%\setup*.log >> C:\Dumps\Preparation.log
del %windir%\wdf*.log >> C:\Dumps\Preparation.log
Echo You are now ready to attempt the installation again >> C:\Dumps\Preparation.log
Echo You are now ready to attempt the installation again

A Batch file for a roll your own "try an install, get a minidump with the final AV, and make sure you get all the backup info you need in case anything in this massive hack breaks"

The batch file does these things in order (I leave matching numbers to batch file lines up to the reader, but you can always ask if you wants to).

Directories are made to hold the dump and to backup setup-related files.
A log file for the batch process is maintained, so we can review it in case anything breaks.
We set things up so the debugger will debug the update binary when it spawns.
Just to be safe, all of their driver services are stopped and then deleted (as is KMDF). This is insurance, as they were told to uninstall first.
To be even safer, all of the driver files are deleted.
The debugger script is copied to a known directory, since the command line we registered references it.
The setup-related log files are cached and then deleted.
Just to be nice, we spit out a message saying they are now ready to retry the installation (the message being in a foreign language for them being hopefully forgivable under the circumstances).

Our second file is this registry file that starts the secret debugging sauce:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\update.exe]
"Debugger"="C:\\Program Files\\Debugging Tools For Windows\\Cdb.Exe -G -c \"$<C:\\\\Dumps\\\\Script.Txt\""

Secret sauce? No- just a way to make sure we debug only what we want to debug.

Key things to note here:

I used CDB instead of WinDbg because by default, WinDbg is going to throw a popup when the session starts, and I don't want to confuse the user any further than I already will.
The path is hard coded, but it is the default when the MSI on WHDC is used. I verified this, and provided the user with instructions to allow them to double-check, just in case.
The command line itself skips unneeded breaks [I imagine having a console window pop up in the middle of the installation is going to be shock enough, why make them punch keys?], and executes the script, which is what I will show next.

The third file in the package is the debugger script itself:

sxd -c ".dump /ma /o C:\\Dumps\\UpdateAV.Dmp" av;g

A simple script for catching the final access violation in the program when it occurs, and then taking a "full minidump"

Now this is pretty brute force- this script creates a dump every single time an access violation occurs! They occur a lot in user mode, but most are silently handled. But it overwrites it each time. Furthermore, if the final dump isn't the right one [as in maybe some AV occurs after the handler has decided to reflect the failure out], I can change a switch in the .dump command and get one each time (and have them send the last half dozen or so, which I could mine to see if they had what I was looking for).

Finally, being the good houseguest, there is the final file- the "cleanup script":

@Echo off
echo Cleaning up the registry and collecting logs...
echo Cleaning up the registry and collecting logs... > C:\Dumps\Cleanup.Log
reg delete "hklm\software\microsoft\windows NT\currentversion\Image File Execution Options\Update.Exe" /v Debugger /f >> C:\Dumps\Cleanup.Log
copy %windir%\setup*.log c:\Dumps >> C:\Dumps\Cleanup.Log
del %windir%\setup*.log >> C:\Dumps\Cleanup.Log
copy %windir%\wdf*.log c:\Dumps >> C:\Dumps\Cleanup.Log
del %windir%\wdf*.log >> C:\Dumps\Cleanup.Log
copy c:\Dumps\Backups %windir% >> C:\Dumps\Cleanup.Log
@Echo Cleanup complete >> C:\Dumps\Cleanup.Log
@Echo Cleanup complete

Cleaning up after the fact- turn off the debugger intervention, grab the installation logs, and put the original installation records back where you found them.

On my end, I not only try these pieces out to make sure they work, I save the final dump for comparison against the one they send back. If it is the same AV, then we missed something. I also make the change to catch "all" AVs (actually it doesn't- the switch uses the timestamp to name files, but the granularity isn't quite enough to literally catch all of them)- I'm looking at 100+MB of dumps, but I decide we can cross that bridge when we get to it.

So who gets to play host?

I already know from previous submissions that our valiant European consumers can ZIP files, but even so, these dumps are going to be too big for an email. Somebody needs an FTP site. Yes, Microsoft has them, but its not like I know anything about that! So I ask the IHV engineer who's acting as the middleman if they have the capacity. Turns out they do- so Bob can continue his slacker ways, and let them host the files...

When the middleman tries my technique, his installation breaks. Might be the debugger return code is creeping out the code that calls it? I decide that if we got the dumps, then we'll get what we really need so I urge him (perhaps against his better judgement) to pass it on to our end users [long may their devices function!].

No fairy tale ending here...

So, when all is said and done, what happens? I get the same dump from the end user that I got on my own machine [cue Obi-Wan, who gestures and says "this is not the AV you are looking for"]. I could make the changes to ask for more, but for the moment, doubt about the whole chain of thought that has led us to this pass is the order of the day.

Well, if George Lucas can end the second part of his trilogy on a down note, why can't I? Since suspense won't change my compensation, I'll say now that I do eventually figure out where it all went wrong [and the above technique is basically sound, or I wouldn't have listed it here]- but I'm saving that for the next and final installment.

Root causing a not reproducible KMDF installation issue- part 1: Looking for trouble, finding it, and working around it

BobKjelgaard — Thu, 02 Aug 2007 17:48:00 GMT

Working in the Windows Driver Frameworks QA department has its challenges. One of them for me has been that occasionally there is a paucity of the interesting problems that make for an intense workday. Since those are the days you write code or blog, they aren't wasted- they just lack that adrenaline...

For one thing, in my opinion the development team is usually very good about testing before checking in. Their unit tests are excellent and keep getting better [a topic I hope to return to someday]. For another, the other members of my team have put together additional tests that are also quite effective. For instance, we have exhaustive DDI-level tests for the frameworks. Microsoft has an excellent set of code coverage measurement tools, and we utilize these to further improve test coverage [the more you cover at even the path level, the less there is of "the easy stuff" to find]. In my case, a final factor is I'm Johnny-come-lately, so KMDF was much more stable by the time I began working on it than is often the case.

But this is good- there are other aspects of quality than the simple bugcheck or hang, and that leaves time to go after them. For instance, it leaves time to pursue "the ones that got away".

This time, the problem could not be reproduced except on the end user's machine, and there were oceans, time zones, and a few national borders separating us. Good reason to give up?

Well, what would you do for a customer with a problem? The answer says a lot about how your business will fare [sorry- I used to run my own, so it's hard for me to not think in those terms].

Making House Calls? Or Just "Beating the Bushes" to Search for Trouble?

As an aside, this is how I got to where I could hear of this problem.

The frameworks are developer tools, and developers of course have Internet communities. Being shy to begin with and something of the lone wolf [although I prefer the term self-sufficient], I have rarely participated in these. But my manager insists I commit to doing something to "engage customers", so I decide a mailing list is a good place to start.

OSR's NTDEV mailing list is recommended to me by several colleagues as an active community of likely early adopters of KMDF, and also having a high level of expertise. So, with a general idea of monitoring traffic to see if we missed anything in our test processes, I subscribe. I also begin participating [often quite foolishly, because my judgement's never been terribly good in any community- I also tend to shoot from the hip, which doesn't always go well with such an audience], but that's not of much further relevance.

An Opportunity to Help

So: many days later, a post arrives (http://www.osronline.com/showthread.cfm?link=110819). A developer has used KMDF in their product, and the installation is failing mysteriously on a very small percentage of machines, all of them in Europe. This sounds like a potential test escape to me! After exchanging a few posts on the mailing list, I ask the developer if we can work via email to address the problem. He agrees.

So, just what seems to be the problem?

Those who read Doron Holan's blog know that there are several files that can tell us quite a bit of history about what may have broken for you when a KMDF driver installation fails. Our customer is obviously conversant with these facts, because his very first post has the problem that I focus on from the beginning.

WdfCoInstaller: [03/27/2007 23:59.53.609] DIF_INSTALLDEVICE: Pre-Processing
WdfCoInstaller: [03/27/2007 23:59.53.718] ReadComponents: WdfSection for Driver Service hypkern using KMDF lib version Major 0x1, minor 0x1
WdfCoInstaller: [03/27/2007 23:59.53.875] VerifyMSRoot: exit: error(0) Op?ration r?ussie.

WdfCoInstaller: [03/28/2007 00:01.11.203] Update process returned error code :status(C0000005) <no error text>. Possible causes are running fre version of coinstaller on checked version of OS and vice versa

WdfCoInstaller: [03/28/2007 00:01.17.875] DIF_INSTALLDEVICE: Pre-Processing
WdfCoInstaller: [03/28/2007 00:01.17.890] ReadComponents: WdfSection for Driver Service hypaudio using KMDF lib version Major 0x1, minor 0x1
WdfCoInstaller: [03/28/2007 00:01.17.953] DIF_INSTALLDEVICE: GetLatestInstalledVersion install version major 0x1, minor 0x1 is less then or equal to latest major 0x1, minor 0x1, asking for post processing

WdfCoInstaller: [03/28/2007 00:01.18.859] DIF_INSTALLDEVICE: Post-Processing
WdfCoInstaller: [03/28/2007 00:03.19.437] DIF_INSTALLDEVICE: Pre-Processing

Take a look at the bolded line, here- looks like someone has a serious problem on their hands!

C0000005 is the exception code for an access violation [touching non-existent memory]. What seems strange (and this issue resurfaces later) is that this is usually accompanied by a popup, or an error report, and not one mention has been made of this.

The rest of the error message indicates that perhaps we should be more explicit in checking error codes, because the customer was wasting his time pursuing a red herring, so I make that suggestion to the development team.

Favoring the Direct Approach

This gentleman's company apparently has an enviable dynamic with its customers, because they have several who are helping them troubleshoot the problem. Not everyone has such a good relationship, and it made a world of difference here.

Still, first you try to reproduce the problem right where you can get your hands on it if you can. We know it's KMDF 1.1, and XP SP2 French and German- not much else seems relevant, although we do try more things than I'm going to bore you further with. After a few fruitless attempts on our part to reproduce the problem (not a big surprise, since their engineers have also been unable to reproduce it), I begin to focus on two things:

What can we do for a workaround? If the end user (and our IHV customer, who remains involved throughout this process) can see progress, then they will have more confidence that we can deliver a solution, and will be more motivated to continue to work with us to solve this problem. Since it looks like we have to debug transcontinentally, it may take a while- lose the engagement, and you may never find out what really happened.
How do we find the source of that access violation, when the only cases of it are thousands of miles away, OCA/Watson/WER doesn't seem to be happening, and we don't even share the same language?

The first is the more important- it is clear that we at least have clear reproduction of the problem somewhere, and a workaround will make the IHV's (and by extension our) customers able to enjoy their product as intended, while we work on the second problem. As I mentioned earlier, it will also boost confidence, helping to guarantee we keep the essential players involved.

So after some of my typically incoherent requests for information, I write a batch file that generates a simple text file that will let us assess the current state of the installation on those machines. It tells me if the files are where they should be, what their vintage is, and also if the registry shows the appropriate services installed, and also that the framework registry support has been properly created.

reg export hklm\System\CurrentControlSet\Services\wdf01000 Service.Prt
reg export hklm\System\CurrentControlSet\Control\wdf Loader.Prt
Dir %windir%\system32\wdf* > CoinstallerFiles.Prt
Dir %windir%\system32\drivers\wdf* > RuntimeFiles.Prt
copy *.prt InstallationStatus.Log
del *.prt

Bob's handy-dandy batch file for assessing KMDF installation health in one swell foop. Copy it, run it, and send us InstallationStatus.Log.

What we find is that on each machine, the framework's files have been copied to where they belong, but none of the registry entries have been made. This and some reading of the coinstaller's source code provides the insight needed for solving the first priority.

It is pretty clear now that this is what has happened:

We provide developers with a coinstaller for KMDF. They include this in their device package and make specific INF changes to invoke it when their device is installed.

The KMDF coinstaller first checks to see if there is any need to install the version of the KMDF runtime it contains. If it finds that KMDF is not present, or that the version that is present is not as new as the one it contains, then it will try to install its payload.

To do this, we have an executable hidden in the coinstaller's resources, which is extracted as a temporary file, and then executed.

Normally, this just works, or else it returns a specific error code related to checked/free mismatches between the coinstaller and the OS. The latter should never happen for a typical end user, because checked OS versions are only provided to software developers, and are not redistributable by them.

But this time it failed, and then [as often happens with unexpected failures] things got worse. The installation has actually done its work, and all the files needed for the framework are on the machine. But since it failed, we will not complete the process and take the steps needed to make the framework usable.

Unfortunately, the way the check for prior version installation works is this: we check if the files are where they belong, and if they are, we see what version they are. On these machines, this check tells us that the correct version is present, so all subsequent installations "succeed". But none of them actually work (because the steps to make KMDF usable were never taken).

So I make another "suggestion"- that initial check needs to be strengthened- some malicious soul could just drop a file in the right place on an end user's machine that wasn't using our technology and we could never successfully install on it (for me clearing up an earlier mystery from NTDEV, where a developer trying to troubleshoot an installation issue did this to himself more than once).

Now it's time for that workaround- the coinstaller's developer has some REG files handy that we can use to complete the installation properly [normally used when working on WinPE]. We send these and instructions on how to use them to complete the aborted installation. If they use these, and retry the installation, everything should work.

Get out the way...

After the usual transcontinental delays, we get back some results. We now know that we have a workaround!

That should ease some tensions and build confidence for both our direct and indirect customers. After several weeks of frustrating delays, some progress!

The job isn't over, yet- time now for the second problem- what is causing that access violation?

This being plenty long for an initial post, I'll save THAT story for the next time around.

Later...