Trouble Ahead- Trouble Behind : UMDF

New WDF debugging videos on WHDC

BobKjelgaard — Tue, 22 Sep 2009 16:41:32 GMT

Several for KMDF here, and several more for UMDF here.

Thus endeth today’s PR blitzen. Have fun!

2008 DDC slides now available to all!

BobKjelgaard — Wed, 22 Jul 2009 14:24:33 GMT

Got an email late last week stating that the material I and others presented at the 2008 Driver Developers Conference (which Dieter and James want renamed to Driver Development and Test Conference- more power to that idea!) is now available here.

The sessions I participated in are a description of several WDF test technologies and some hand waving about troubleshooting WDF installation problems.

If you are a tester, you might also want to take a look at James and Dieter’s presentation on concurrency testing, and who wouldn’t want to know about the new Driver Verifier features for Win7 (ably presented by Daniel Mihai)? I also recommend the presentation on DSF, because in many ways I live and die by DSF.

That would be “Device Simulation Framework” if the acronym is unknown to you. Sooner or later, having an easily reproduced software simulator for a device trumps going to EBay to buy some long out-of-production hardware your test driver must have… That’s just one reason I like it, but that’s all I have time for now…

There’s more- but that’s why it’s called a “browser”, eh? By all means, browse!

As for why it took so long to make public- remember this conference occurred under non-disclosure agreements. Takes some time to unravel things and get all the requisite hoops jumped through.

He’s Gone

BobKjelgaard — Fri, 26 Jun 2009 05:15:16 GMT

It has been a tough week, and a busy one. I’m getting ready to leave on a vacation, and trying to get all those last-minute things done.

While my last post discussed the genesis of what I call the Configuration Agent and the WDF QA Universal Setup Job, this week’s focus has been entirely on another of my oddball notions- it started as a desire to build a more precise fault injection engine for testing the KMDF loader. With that California background, the project name was easy- Fault? Why, San Andreas, of course!

At a high level it was a programming model built around IAT filtering of calls from a driver into the kernel. I wanted precision, so I could specify behavior by thread and IRQL, program each DDI independently, allow a driver to do its own filtering [i.e. get a callback with context for a DDI call under given conditions]- and with enough flexibility and locking behavior that the behavior could be reprogrammed within such a callback. For instance, in one working application of mine a filter on calls to IoCallDriver from the KMDF runtime were routed on specific threads to a callback in the test driver which then turned off all call logging, as it meant the call was leaving it’s stack location- I was using the logs from calls to various resource allocation DDI to make sure the framework did not allocate resources under conditions in which we contractually would not fail- so I wanted to stop logging before I called IoCallDriver and to restart it after it returned [I knew the driver handling it was synchronous- otherwise I would have to hook the completion side of things, of course].

So the concept went well beyond fault injection, but it’s always been a side-line. My initial tests wound up being very white box, and a refactoring of the loader code meant a similar rewrite of the test was necessary, but there was no time to do that… But with Windows 7 winding up [since we’ve announced dates, that can’t qualify as a secret anymore], I have time to go back and try to at least do a few new things beyond mere maintenance. Such time is rare and fleeting, so I’ve been busily making the most of it.

So today was busy- I worked a bit of overtime on a COM automation server (in process) exposing a scriptable interface to my engine for some of the other team members to use, and got it at least working well enough to demo. Still have my packing to do, and dinner to make and so forth, so home I headed.

Then I turned on the television, and immediately heard the news about Michael Jackson. Whoa…

On those old tapes I’ve been scraping for my archives I’ve got many covers of his material from the time I was playing in bar bands: Billie Jean, Thriller, Beat It- and more. I’ve not posted them because frankly I never thought I did any of them justice- a lot of my material from those days has rough edges [anyone who listens to what I have posted can hear that], but these were just not good enough. But add me to the billions of fans, then and since, sure…

The other thought that came early on was that in my previous post I mentioned having learned to keep some of my more controversial opinions to myself- one of those was intense skepticism about the molestation accusations. I knew I wasn’t likely to sway anyone there- so why bother?

He’s gone, and he was younger than I am- scary stuff. Life is fleeting, and it was an unexpected reminder of that.

But, it’s time to pack my CDs and games and controllers, write those last few checks for bills coming due during the vacation. eat that dinner, and get ready for a night’s rest prior to that final day of work, so another slapdash post, and back to work…

There was some offsetting good news today- besides finding out I really do still remember enough about COM to roll out an in-process automation server on the QT- without using ATL of course, where would the challenge be in that [answer: in learning something new for a change- you old dog!]? But of course it falls into that semi-confidential area where I’ll just have to sit on it for a while [not that it’s likely to shatter the world if revealed anyway, but circumspection is worthy of practice even for an old blowhard like myself].

I’ll try to do something more substantial someday- these days I’ve just not got the time to really write anything too terribly technical. My apologies to those diehards who still try to read this awful stuff I’m churning out of late…

WinHEC Presentations you might want to view

BobKjelgaard — Thu, 20 Nov 2008 00:01:56 GMT

Since I was asked in response to my previous post "so what else is new in WDF?", how about this link to the presentation Eliyas and Peter made on exactly that topic at this year's WinHEC?

I suppose I could also flog this link to the presentation Ilias and I made [but it's not quite so broadly of interest- WDF coinstallers and introduction to WDF logo requirements and tests].

Now Playing: Turning Point (a bar band I played in during the 80's)- The Other Woman (ahh, I could play a decent bass once in a while, back in the day)

Which reminds me, I've put that one up here (about 2 MB WMA) if you really want to hear it yourself. I've put up a smattering of others available from this page(including a few after I switched to lead guitar and vocals)...

Now Playing: Grateful Dead- Europe '72- Sugar Magnolia

Determining WDF runtime and client versions

BobKjelgaard — Wed, 05 Nov 2008 17:29:00 GMT

What a long, strange trip, it was, getting to WinHEC! I waited for a shuttle to the airport at my apartment, and the rain [which was fairly heavy] turned into hail! But by the time we got near the airport, everything was dry.

Well, LA is probably still LA (I lived in El Segundo for about a year 6 years ago, and of course went to school in Pasadena, so I'm not a total loss], so that's enough travelogue.

UMDF

Easy- Peter and his team obviously planned ahead. There is an exported data item linked into each driver and in one of the framework binaries. So LoadLibrary and GetProcAddress, and you're there. The data is three ULONGs [IIRC, as I'm not about to carry Windows source on my laptop to WinHEC without a better reason than a desire to blog]- major version, minor version and "SP level".

KMDF runtime

I won't say Doron and his peers didn't plan ahead- I believe it was more a case of having more important problems to solve at the time... But I did mention it, so I might as well face it.

Again, I'm going to have revisit a few details again, later. This is my recollection [and it will be pretty close]:

KMDF 1.0 had nothing to identify it.
KMDF 1.1 and 1.5 have a resource in the disk file that identifies major version, minor version, and build number, similar to the UMDF version info described above. LoadLibraryEx (as datafile) and the usual FindResource, etc. route.
As part of our installation improvements in KMDF 1.7, the runtime now also records its version in the registry under its software key when it is loaded, and removes it when it unloads. That's why the WdfVerifier image I showed here said "In Memory". I like this because it also works remotely (and I'm not going to inspect binaries remotely).

So, I first look for an in-memory version, as that is the most reliable. If I am remote, then the only other thing I can do is look at the registered OS version information. If it is Windows Vista, I assume it must be KMDF 1.5 [because 1.7 would tell me it is in memory, as Vista has running inbox KMDF drivers]- the message indicates this, although how I say it is by inference escapes me. Otherwise, it is 1.5 or below, but I can't tell- I don't remember the message, but I do hope I changed it from my original "Can't tell- you tell me!".

If it is local, then when I first scan all driver binaries, I pick out the runtime, and examine the version information just as I describe above.

KMDF client

This is the fun one for me, so I'm going to save it for a later post- it's time to get dressed and find that shuttle to the convention center, so I can spend a day in WinHEC [oh, the pun potential for that one...].

No tunes today- wonder if the WDK folks I now sit amongst are enjoying time away from the endless noise emanating from my environs!

They're here!

BobKjelgaard — Fri, 31 Oct 2008 20:24:28 GMT

For some of you, this may be old news, but it appears the 1.7 WDF coinstallers are finally back in the downloadable WDK (I just got my notification from Microsoft Connect much as others registered for the release program presumably did).

This would be the link to the download site [if you've signed up for Connect].

Lifted directly from said notice:

Here are few highlights in this release:

New Setup Environment
Updates/fixes to Headers
Added Bluetooth 2.1 headers and samples
Added support for Server 2008 to DifX
Updated WDF 1.7 co-installers
Updated Device Simulation Framework (DSF)
Updated Debugger

New in WdfVerifier- the tweaks and fixes

BobKjelgaard — Thu, 30 Oct 2008 21:24:38 GMT

Now that the PDC is underway, and WinHEC is nigh, I'm going to provide a quick peek at changes coming to the WdfVerifier tool (these will be in the Windows 7 Beta WDK, which isn't far away, but obviously I still can't say more than that).

Doh!

First off, fixes for a few of the usual boneheaded bugs (please note these were my bugs, not bugs in the underlying KMDF or UMDF code):

Handle Tracking settings for KMDF were being managed in an incorrectly named value.
Paths to UMDF debugger with spaces in them weren't handled properly
Any attempt to change UMDF host process restart limit set it to the maximum value (one of those last minute changes that slipped past an ad-hoc test procedure).
The thread that handles UMDF auto-launch never idled. Didn't think it mattered, as it was normal priority, but it sucks laptop batteries down fast when it doesn't.
There was a reference counting error in my object management code- if you had UMDF drivers installed and ran for a very long time, the resulting leak from the auto-launch thread would eventually do the usual low-memory nasty things to you.

There were a few typos and a lot of rather poorly expressed concepts (probably some grammatical errors as well) in the tooltips [although I'm often tempted to rewrite the lot of them every time I look at them].

Added touches

I now show you the start type of each KMDF driver (when you select it).
When running locally (and where I can, I also do this remotely for KMDF runtime) the true WDF version information for the runtime and each client driver are presented. I overwrite any version info the coinstaller wrote (which we know is untrustworthy) with the real stuff when I find it.
Tool settings for the app itself are now managed by machine name (they have always been per-user, as well)- so you can have different preferences for local and remote, and even different settings for each remote machine you manage with it [not that this was in high demand- it was just easy to do and harmless enough].
I tried to unclutter the UI a bit (but new features cluttered it back up fairly quickly). I may try to do something more drastic after Windows 7.
There are some new features in KMDF 1.9 that only apply to KMDF 1.9 client drivers- so the UI now changes a bit when you have a 1.9 or pre 1.9 driver selected.
I realized there was a smarter way to show how the handling of WdfVerifierBreakPoint and the WDFVERIFY macro work with respect to the overall Framework verifier settings (drop list instead of checkboxes that mysteriously changed settings on you in unexpected ways).
I added a splash screen- it's a bit klutzy, but at least you don't sit wondering what's happening after you invoke it- it provides some progress messages
Eliyas said he wanted an icon, so being cheap, I made one (the letters WDF in blue with a green check mark sketched in as a background), rather than try to find a real artist.

The method for determining the client version of a KMDF driver and for that matter of the runtime was fun and might interest some readers, so I'll blog about that shortly. But don't expect much code- I don't believe in samples, I believe in concepts, but I hashed out my thoughts on the subject of sample code in part some time ago.

So here's the next 1000 words (I'd show you a 1.9 inbox driver, but that would reveal things to be disclosed at WinHEC) - KMDF tab (on all snapshots I've added some highlighting or markup for new stuff):

Here's the UMDF tab- the list of drivers would also show, but I didn't have any handy and am short on time (I'll snap one for a later post, perhaps):

Preferences page has a new control related to the harvesting of KMDF version info (you only see this when running against the local machine)- it's there because it can take some cycles, so you only want to repeat it when you really think you need the info (first time through registers it results, so once might be enough):

Did I mention a (less than awesome, admittedly) splash screen so you don't get bored while it loads? When it is scanning for drivers, it lists each service- I could keep trying to get a snapshot right when a real driver is being looked at, but I'm not. So the one here has nothing to do with drivers...

Enough pictures for now- probably more than you needed. Next time I'll talk about the easy parts of getting version information, then when I find time, I'll talk about the fine hacks I made to get the real KMDF client driver versions, and you can decide if you even want to let this code near your machine...

Now Listening: Grateful Dead- Built To Last [album]- Blow Away

DDC Day 3 (and final) soon to come

BobKjelgaard — Wed, 01 Oct 2008 16:32:58 GMT

In my post about yesterday, I left out an observation I feel the need to make. My intent for listing the folks I met in the morning wasn't so much to drop names as to observe that (a) they were appreciative of what I have done in terms of blogging an newsgroup assistance (which is always easy feedback to accept, particularly from a stalwart group such as that) and (b) it's still something of a kick for me to be able to talk to them, and get some of the conversation you just can't carry out via email. As a developer, it's good to see what experts do, and as a provider of developer tools, its good to get the chance to collect all the unvarnished feedback you can about the utility of what you are providing.

I missed a lot of sessions yesterday I would have liked (such as Daniel Mihai's presentations on Driver Verifier). But I did catch Neslihan's presentation on WDF logo requirements [and I need to remember a couple of things we might want to add to that occurred to me overnight], and then Ilias and I did our first pass at the coinstaller presentation. Numbers were a bit light- maybe 2 dozen people was all, but I think we did fairly well overall in terms of just presenting our material, at least.

The Ask The Experts session was a bit lopsided- lots of experts, and something of a paucity of people wanting to ask. We did get a few questions (unfortunately both Doron and Eliyas were unavailable, and the rest of us still have some gaps- at least Peter was there for some of the tougher technical questions- some of the questions we didn't know sure answers to weren't actually all that technical- more along the lines of adoption numbers, which I sort of pay attention to, but only in the fuzzy, order of magnitude sense).

Got a couple more good test-related questions in hallway conversations, and finally met Penny Orwick, whom we all owe a debt to for her work on the WDF book, and to whom we owed a further debt as presenters because she edited our slides.

And now for something of greater utility...

As part of my preparation for the coinstaller debugging presentations at DDC and WinHec, I have put together this collection of information about WDF components for all shipped versions, along with related information, so there's a single place usable for triaging those occasional installation issues. Parts of it are still under construction (and some things I want to include require me to link to some files I host elsewhere), but I'm open to comments and suggestions, as always.

Finally, a warning about the shipping versions of WdfVerifier: I messed up the registry keyword for handle tracking in KMDF, so the tool won't actually manage that setting for you. If we provide a WDK with the "pre-beta" release of Windows 7 about to hit the streets [at WinHec and PDC], it won't have this fixed- but it will be there in the WDK tied to the Win7 Beta.

DDC starts in less than an hour

BobKjelgaard — Mon, 29 Sep 2008 16:21:36 GMT

Plenty of good stuff for today- begin with Eliyas and Peter going over what's new in WDF for Windows 7 (or as we prefer to think of it WDF 1.9, since much of it also works all the back to Windows 2000 [KMDF] or Windows XP [UMDF]). It continues later with our first presentation of talks on the "shared secrets" of the Windows Driver Frameworks. The first talk discusses the implementation of the frameworks themselves, while the second presentation (my first of this conference, and my first in about 16 years) discusses how the frameworks are tested. The latter talk will also be one of the final talks of the day.

Tomorrow will include Neslihan's talk about getting a logo for your WDF driver (yes we are adding logo requirements specifically for WDF drivers- primarily for KMDF at the moment). It will include some excellent talks by Praveen Rao about work we have done on WDF drivers in the storage space, and again the final talk (at least in that room) is Ilias and I discussing the WDF coinstallers.

Wednesday we'll repeat all the talks I am in- the shared secrets talks will be back to back in the morning, but I won't catch part 1 that day because I'll be in another room with Ilias repeating the presentation on the coinstallers.

There's so much more, too- SDV, DSF, the innards of driver verifier, annotations for PFD and SDV, new test techniques from my colleagues James Moe and Dieter Achstettler, excellent talks on device installation, many driver technologies (several on printing, I can't help but notice). I could go on and on, except I've got some work to get to this morning [and I'll be leaving soon so I can register and begin my bit of supporting the conference].

Ask the experts session tomorrow night, bowling tonight, space set aside for us to talk with developers before and after talks, labs, panel discussions, Q & A with our most senior engineering staff, and a few surprises I'm not at liberty to mention even now.

My plan for now is to try to attend the whole thing- how well that works depends upon how things go in the lab [we've got some critical work going on, but let's see how the junior staff handles things for a bit], and whether I'm providing any benefit by being there [if I can be more effective working on test tasks in my office, I'll return to it]. Of course, assuming my laptop works there, I may just blog a bit more than usual. Not having a cell phone, it might be a bit hard for people to contact me if they need me back at the office [although they will know where I am, anyway].

So, I hope to see a few of my readers there (both those within Microsoft, and especially those from the outside). Hope all enjoy there time.

Now playing: The acid rock classic jam "Dark Star" from Live / Dead (of course by the Grateful Dead)

Looking Ahead- Conferences!

BobKjelgaard — Thu, 11 Sep 2008 07:23:00 GMT

Since I noticed Patrick has spilled the beans [just before the second question], I might as well cover this topic while I wait for my dinner to bake [I eschew microwave ovens these days- just seems better to let it cook the old-fashioned- if early-to-mid-twentieth century is old-fashioned- way].

I've been through some of my belongings of late and put a few items up on the corkboard in my office- pictures of me from the last several decades, ads from bands I played with in the 70's and 80's, and so on [along with my Master Chief calendar and pictures of my daughter and InuYasha]. Two of the recent additions are from my first time around at Microsoft. One of them, which I showed off recently at a WDF team meeting related to this article's topic, is what I believe was my original Microsoft card key- a much different technology from those in use today, which are basically proximity read- I believe this was a magnetic stripe like a credit card [but can't be sure, because that part of the back of it is gone]. I think the big surprise for most was that 19 years ago, I had no facial hair and very short hair [almost no gray then, either]. Weighed a lot less, too, and still had two working eyes [even if they were near-sighted].

The other [finally getting to my point] is a speaker's badge for "Microsoft Windows NT Device Driver Developer's Conference". That was from 1992 [I wouldn't know for sure except that it turns out I still had the conference material with the dates on it], and it was the real "first DDC". I was the last speaker on the final day- the track was about Win32 printing [and this was the very first release of Windows NT and the first 32-bit OS Microsoft released], and I was speaking about testing and logoing printer drivers- that was my first time hearing about WHQL, by the way- when I was preparing the talk and they asked me to add some slides related to it.

Of course, my prized possession is the "Ship-It" award for OS/2 3.1 [which of course didn't exist as a Microsoft product- it was a typo, but I snuck mine out before the manual update processes in use then could correct it], but that just made this paragraph irrelevant and disruptive of the flow. I'm like that...

DDC- what a long, strange trip it's been

The Driver Developer's Conference is an intensive high-level conference with some NDA-type restrictions [I'm no lawyer- it appears these aren't as strict as the MVP summit, but who am I to judge or care?]. It will be held on the Microsoft campus later this month. The meeting I am alluding to was our team reviewing one of the presentations I will be participating in [and I've been slide-smithing ever since]. In one session, Ilias and I will be discussing the WDF coinstallers. Ilias will discuss the whys and hows, while I will finish up by telling you what you can do when things go wrong. In the other technical session, I, Shyamal and Wei will tell you about how we test the frameworks, including things we do that can translate into your own test problems.

There will be some Ask The Experts sessions, and I know WDF will be there, but I don't think the final lineup for that's been determined. I can live with it myself, either way...

As for the rest of the conference, well--- one of the more attractive nuisances of the past several days is that all the presentations go on a single sharepoint site, so I've been previewing a few that interested me. Lots of good stuff- device installation, DSF [my favorite non-WDF tool], new technologies all across the space, etc. I also see the world of printer testing tools is in great hands [probably better ones than mine ever were!].

WinHEC- a first time for everything

The Windows Hardware Engineering Conference- well, you can use the link to find out what you need to know. I'll be there too, with somewhat stripped-down versions of both talks [at least, I think that's what's happening]. It's still several weeks away, and there's DDC to get through first. But it will be in LA. Not certain who all else is going, yet, but I know Ilias is, and I can't believe it can be done without the likes of Doron and Peter.

I've been on the mailing list for years, but never gone (I was supposed to go a couple of years back, when it was in Seattle, but had car trouble).

So perhaps I'll meet a few of you there. I'll be the one with all the hair and the wacky left eye [unless I can't pass the airport security checks, or the other passengers find me scary and demand I leave the plane]. Well, I've lived in LA a few times and driven from here to there and back before...

It's soup!

BobKjelgaard — Fri, 18 Apr 2008 04:41:00 GMT

You can now get the good WDF 1.7 coinstallers from Microsoft connect. What you will get is an MSI that will copy them and some accompanying notices to your machine. If you have the WS08 RTM WDK (18000) replace those coinstallers with these. If you have the 18001 WDK (with no WDF coinstallers) just use these.

In a while (hope it's a few weeks, anyway, I'm getting real tired of coinstallers) a WDK 18002 will be produced and take the place of the WDK sans coinstallers plus coinstaller MSI. We've started the process to get this stuff into MSDN proper, but that process has been a bit sluggish lately, so I'm not about to offer any estimates as to when it will make it there.

Short and to the point...

Now Playing: Yuki Kajiura [Noir Vol 2]: Secret Game- interesting anime- gotta spring for "Madlax" sometime...

The light at the end of the WDF 1.7 tunnel

BobKjelgaard — Thu, 17 Apr 2008 18:04:55 GMT

I know people want to know about this, but firm dates still prove elusive. My best guess is today or tomorrow at this point.

Basically we're trying to put an interim package up that satisfies everyone and isn't going to cause us foreseeable problems in the future. Eliyas keeps tweaking this MSI to satisfy one concern or another raised by members of the Dev, PM and QA teams, and once it's done, it will go to Microsoft Connect and you'll have your coinstallers. Sometime after that we'll have an updated WDK with everything tied together nicely again. As I understand it, the links for it on MSDN will work eventually, but I have no idea when.

As far as the real testing- all done, all good. All the legal and licensing stuff, good. Just have to make an MSI with everything in it that doesn't look like it was slapped together at the last minute- even though it sort of is.

Now playing: Yoko Kanno [Ghost in the Shell Stand Alone Complex, V2]: Psychedelic Soul

Some follow up

Kumar fixed the UMDF coinstaller build problem (which you'll never see) in fairly short order, and then went on vacation himself. So of course, more build problems turned up- but until we get that MSI out, not likely I'll get any traction. This time the x64 update package for WinXP and Win2K3 is missing some pieces- blocks me, but again, I doubt this is a problem that affects anybody but my team [call this an attempt to give you an idea what it can be like sometimes "on the inside", because it's easy to get the idea everything is endlessly smooth and we spend all our time pondering the whichness of the why].

Now Playing: Grateful Dead [Live / Dead] St Stephen [always a welcome album]

So that's what they look like!

Yesterday I gave a brief presentation on WdfVerifier and WdfTester in a session on Driver Testing at the Global MVP summit. Met Mikhail Vodicka, Gianlucca Varenni, Martin O Brien and Maxim Shatskih- or rather at least exchanged greetings face-to-face, either in the session or out in the hall afterward. I'd dropped off the OSR lists for a bit to help manage my email and workload, but after talking to Martin a bit, I decided to re-subscribe again. Hopefully I can better manage my impulses to jump in on some of the esoteric issues that occasionally surface there...

Now Playing: Kraftwerk [The Mix] Musique Non Stop

Now for a different UMDF Coinstaller story...

BobKjelgaard — Sat, 12 Apr 2008 01:40:08 GMT

This one can never affect you, so your blood pressure can start easing now. It's just been a while since I tried to put on my "war stories" hat and tried tellin' one of them tales...

In the Beginning was...

the need to make sure drivers we build for test are signed so we can automate their usage and avoid all the nasty workarounds unsigned driver installation entails- particularly since we test on so many operating system platforms. In the early versions of KMDF, we did this as part of our normal build process, getting them signed just like all the rest of the OS is. Even when we went to produce releases (this happens in build lines not under our direct control) they built all the samples and test code as well, and life was good. We always had a coinstaller with the right update packages in it, and everything was signed.

You still see vestiges of those halcyon days in the KMDF samples, where entries for the catalog kmdfsamples,cat can be found even now [and this will be the case for as long as we are responsible for them]. All of our test drivers are also catalogued in there, so we can mix and match with impunity and it all works as long as it's from the same build.

The price of success

But we also became a part of the OS beginning with Windows Vista. Now for a host of reasons, the build lines that build the OS produce coinstallers for KMDF and UMDF that contain no update packages. We did continue to build coinstallers with update packages on our private build machines, though. We utilized various nefarious techniques to undo the system's file protection and place KMDF on a Vista machine when testing so we could test our latest versions, so things still weren't too bad. But sometimes we wanted to use coinstallers and test binaries from differing builds- signing was becoming a problem. More importantly, when we approached WDK release times, our external builders now only produced the coinstallers. So we no longer had a single nice signed package automagically produced for us.

We worked around this as best we could- typically one of the SDETs assembled a "build" out of disparate pieces and then re-ran the signing steps with a bcz in the proper directory. Like all manual processes, errors happened, but we muddled along.

Finally, late last year things got to be too much for Shefali and I- we had to run really old test content and found the signatures were no longer valid. I should explain that a bit, if I can. Developers working on Windows have certificates created for them identifying components they build that chain to a special test root certificate (a term you can look up, for instance- this MSDN article touches upon them) that is recognized by most interim builds produced of Windows. This means all of our content is "signed" (and it also means that if any of those signed binaries show up where they should not, they identify who produced it, giving one easy place to start searching for a leak). When we approach releases a switch is made to more official forms of signing- our test drivers are also part of those builds, although they never ship to anyone, so we're still good to install on those- but we can't use them anywhere else because the coinstallers are still what we call "thin" [no update packages]. Of course, since nobody should need them for very long, those certificates also have a very short shelf life, which is what I was referring to at the start of this rambling paragraph of mine.

We also had a hard time making clear to the software test engineers who were trying to run our rapidly changing test mixes what parameters to use (or even to figure out for ourselves which combinations of parameters really did what). This led to delays, confusion, dissatisfaction [one of those poor STEs must have been sure he was on the verge of being dismissed- and that bothered me because I knew it wasn't all his fault], and other generally bad and stressful things.

If you want it done right, DIY

So, if self-signing driver packages and using test-signing approaches is good enough for our customers, it ought to be good enough for us. I redesigned our entire automation process around this approach (with much encouragement and prodding from Shefali). I solved both problems at the same time, but since I like to wander when telling tales, and I'm the one with the keyboard, I shall tangent...

Too many cooks

Normally a good test automation design in WTT (known to you as DTM) is fairly self-contained. It gets its stuff, does its work, and cleans up after itself. You see this in the three phases- setup, regular, and cleanup. Virtually all of our test jobs worked this way, meaning any one could run independently of the others.

We have literally hundreds of jobs that work this way- the bulk of them testing the KMDF DDI. They took parameters with little bits and pieces of path names [because most of the time everything came from a specific machine, or if the machine changed, parts of the path were known, etc] and assembled them together to locate things, install them, run them, and clean up.

But the names weren't consistent, the portions mapped weren't consistent and while it was sometimes possible to get a correct path by using .. in path entries and even blank entries for some parameters, determining those values was a logic puzzle in and of itself. Worse, you couldn't be sure after a run that all of those jobs had really run the same thing. Since I now found myself the only cook left in the KMDF QA kitchen, I took advantage of the situation to impose order on the chaos.

Slicing the knot

The story of Alexander the Great and the Gordian Knot has been with me all these last few weeks for some reason, and this may have been another of my "Brute Force" solutions. I broke our test pass into three stages:

Staging- in this phase, all of the tools and content used is copied from disparate and myriad sources to the test machine in a known location. Common tools like DSF are installed. The key to the underlying narrative here is that this job also creates a test certificate on the machine, creates a catalog containing the entire contents that had been copied earlier, and signs that catalog with the new certificate. It then sets the machine up so it works with test-signed binaries effectively. So there's still a kmdfsamples.cat- but now it gets built fresh and piping hot right at your table [that thought makes me want to visit Benihana].
Setting the framework on the machine. In this phase, if we need to overwrite the normal version of KMDF already there we do- either brute force (by overriding the system protection on it) or elegantly (by using a "fat" coinstaller containing the appropriate update package). As mentioned somewhere in here, you have to reboot the machine if the coinstaller is used (in fact, you have to do it either way). Sometimes we don't even need this phase [XP, for instance]. Whenever possible we try to utilize real coinstallers in random configurations to more closely duplicate the end user experience, after all.
The tests themselves.

I had a single ground rule- only the staging job would have any parameters. All the subsequent jobs would use what was staged. There was a corollary based on previous experience: those parameters would be substantially complete path names. They might take longer to type, but it was easier to switch and accommodate quirks in how paths were assembled as you tried to get things from elsewhere if you just always took entire paths.

I finished most of that work in one weekend (in November, if I remember correctly). The most mind-numbing part of it was modifying the existing tests- I'd go through task by task with the new "known" staged path in the clipboard, and selected each directory name I found and pasted it in. There were some deviations that I wound up adjusting in the initial setup job because they were done too many places. There were places where parameters were passed down into library jobs that I left untouched (I actually set any such parameters to totally invalid values to make sure nothing escaped my wrath).

After all that surgery I now had the ability to mix and match with much greater flexibility, and simple instructions with four basic parameters that covered all the known variations we had seen. I then created a Wiki on the internal Microsoft network where I listed the instructions for all of the normal passes we did so we could clearly communicate what settings were to be used each day- at first, the STE could literally cut and paste from my instructions. Once they were familiar with the new setup, they could do more of the work themselves. You wouldn't recognize that same STE today.

It worked pretty well, even if underneath there are a lot of rough edges (if you like clean setups, this isn't one- the sheer scale of the task is too big to justify yet). It also made testing test changes easier- I build everything on my machine, and can schedule a job to pick up the content from there. If I'm doing even more aggressive mixing and matching than usual, I actually assemble the binaries on the test machine and let the setup job copy them from there into the new official staged location [one part of the hard drive to another, but it's all with a tool we use continually and rarly needs to be done]. To be fully fair, I should add that I didn't get all the tests at this time, just the ones that we absolutely had to keep running. For instance, our stress mix fell out. But Shefali later chipped in on her own and got them working.

Life was good- and there was now time to work on problems in the test code instead of trying to figure out how to continually tweak creaky automation into doing something slightly different every few days. Shefali seemed pleased, and what the heck, making the boss happy is generally a good idea in the business world...

Trouble in Paradise

Until I deployed a new test. Or rather a new variation on an old test. I have a rather elaborate setup I use to verify operation of the IoTarget and IRP processing function in KMDF [although I don't go totally into queues- just the very basic configurations], and to support some new features in WDF 1.7 you'll be hearing about soon, I added UMDF drivers into that test. I had to add another parameter to make sure I had all the flexibility I needed in finding a UMDF update coinstaller, but that's not a problem. I put it all together, tested it quite a bit, rejoicing somewhat in how easy this new process made it for me to do a test that was now creating 88 different devices on top of a virtual test bus, installing the proper drivers with no popups in sight, and then putting those devices through their paces- and it looked good. So I called it complete. got it reviewed, and checked it all in. [For the 2 or 3 regular readers [overestimating my impact again?] this is the test with the targets and "hunters" where I showed some code here].

But early this week, it failed. Makecat wouldn't process the UMDF coinstaller from our own build machines (to debug it, I forced WTT to halt when this failed- we were losing logs due to some problems not worth going into here- the following is an email snippet):

I set the task to freeze if it fails. This is weird- this is the tail of the self-sign log:

processing: <hash>C:\kmdftest\WUDFSvc.dll.mui

processing: <hash>C:\kmdftest\WUDFUpdate_01007.dll

NOT processed: calculating the indirect data (C:\kmdftest\WUDFUpdate_01009.dll)

Failed: CryptCATCDFEnumMembersByCDFTagEx. Last Error: 0x80004005

Errors found in parsing the CDF file

A comedy of errors ensued for a while after that, as I tried to find out why the 1.7 RC1 coinstaller was there when the job parameters I was told used pointed to locations that couldn't have contained it [if I'd dug into the job reports, I'd have seen that when the set they gave didn't work, they pointed to a server containing that and tried it again]. Once that was settled, I began focusing on why makecat was giving me the ever-so-helpful E_FAIL parsing a file that seemed perfectly good.

Well, it was the file itself for some reason- take it out of the CDF, makecat worked. Have it as the only one in the CDF, same error. Since we recently had some changes made, I was wondering how they could affect hashing the file- so I went back and tried earlier versions. This led to another comedy of errors when I inadvertently mistyped a path and the process worked [because it couldn't find the coinstaller, and without going too deep into why, I couldn't treat it as an error at this point in our setup job].

Well, the world was looking strange- I know I ran this dozens of times while I was developing this, didn't I? We'd run it the previous week, and there'd been no problems. It got stranger when I ran the same thing on a Windows 7 machine, and it worked flawlessly. Why should the OS have made a difference? It was the same set of binaries, tools and all... Now if I weren't stressed and well-befuddled by then, I might not have continued to be so stressed and befuddled at that point, but of course I was and I did...

But this is serious after all- in this design, all the test pass eggs are in one basket called the staging or "Unified Setup Job" and with it broken, NOTHING works!

Fools Rush In

And this old fool is no exception.

Ilias sends me an email about the problem late in the day with this link and the comment "real programmers use butterflies". I had decided what I was going to do, so at about 5 AM the next morning (I started sometime between 1 and 2 AM) in reply I said, "A sledgehammer is more my style"- ahh, well- afterthought says "Real SDETs use sledgehammers" might have been a better retort...

Onward- "Take the bull by the horns!", Papa sez to himself, and loads the debugger package on the machine. Point it to makecat, give it the command line to process the CDF, and go. Make sure we've got all the symbols [miracle of all, they were there the first time!], and set a breakpoint on the routine name which was most helpfully displayed in that error message above, and go. Then step into the code [now while I did have symbols, I don't normally need to work with that part of the Windows source, and this is Windows 2003, anyway- so I'm doing it the old-fashioned way, reading the assembler and using the old noggin to cipher out what's up... I wasn't totally cheating- but because I had symbols I could see internal names and also the names and types of local variables, so I wasn't flying entirely blind].

Now before I did this, I went through a phase where I thought there was a defect I could note externally that would tell me what had happened- and in the process, I dumped the headers of the coinstaller with the linker (link /dump /headers <file name>). It seems to me that my long-term memory is beginning to suffer the ravages of age, but short-term is still pretty good, so I still remember things like this:

    10 number of directories
11440 [      8D] RVA [size] of Export Directory
10974 [      50] RVA [size] of Import Directory
15000 [ 12A53C] RVA [size] of Resource Directory
     0 [       0] RVA [size] of Exception Directory
13400 [     FB8] RVA [size] of Certificates Directory
140000 [     AE0] RVA [size] of Base Relocation Directory
1210 [      1C] RVA [size] of Debug Directory
     0 [       0] RVA [size] of Architecture Directory
     0 [       0] RVA [size] of Global Pointer Directory
     0 [       0] RVA [size] of Thread Storage Directory
5530 [      40] RVA [size] of Load Configuration Directory
     0 [       0] RVA [size] of Bound Import Directory
1000 [     1D0] RVA [size] of Import Address Table Directory
     0 [       0] RVA [size] of Delay Import Directory
     0 [       0] RVA [size] of COM Descriptor Directory
     0 [       0] RVA [size] of Reserved Directory

what was odd, was that even though this says a certificate was there, I couldn't see one in Explorer. Odd, but it didn't raise any red flags to this old bull, so on he went.

Well, after much digging and a bit of backtracking, I found the place makecat decided to make that error. So I followed the preceding call deeper and deeper and got into code that was preparing to hash the binary and was looking for parts of the PE image to exclude. Now it happens I've done lots of hacking to binaries- stripping resources out, putting them back in, altering tables and all sort of general mayhem, so following this code is a snap, even in assembler [with those handy locals about, anyway]. I find a path where it is clearly failing, and looking back up through the registers shown as I single-stepped the code, the values FB8 and 13400 caught my eye. Hurrah for what memory remains! A quick check confirmed they were the header values. Bashing them against the values in dv, I had my cause...

It had refused to hash the binary because the certificate was not at the end of the file's memory image- specifically, the resources followed it. It turns out this also caused the certificate to be invisible to explorer and made signtool verify most unhappy [but signtool also happily replaced the certificate in situ every time I tried, alas].

I then sent a rather rambling and somewhat incoherent email to Ilias telling him we'd been building an unsignable UMDF coinstaller [too much stress and too little sleep- forgot that it had worked in Windows 7] since time immemorial. Probably boosted his blood pressure since I wasn't all that clear I meant only on our private build line [obviously WHQL signs the official versions from time to time, after all]. I also wasn't making any clear distinction between signing the binary by embedding a certificate and signing it by having it properly hashed in a catalog signed with an embedded certificate [that duplication of terms has always been a source of confusion] . After a face to face and some more coherent and detailed explanations from me we had it down- that only the UMDF coinstaller from our private line was unsignable, and then only on Windows 2003 and earlier [I'm afraid you'll have to repeat some of what I did to see why]...

As I describe here, I can disassemble the coinstallers quite readily and in converse I know how they're put together- the problem is clearly that we signed it before we added the update package as a resource [now the package that does this could just be made smarter...]. We found out how that's happening, but fixing it is proving a challenge.

Well, the main build lines have well-funded and trained staff to handle all those scripts that handle all those things we do after build- on a private line like ours, you have something a bit more seat of the pants, and Ilias is not the originator of most of those scripts. If you've been around software long enough, you probably get the picture- logging not quite up to snuff, commentary a bit lacking, and so on. He's still working on it [actually, he's going on vacation, so my old buddy Kumar probably gets to hold this hot potato].

So you unhappy souls we've held up with the Server WDK [and I mean this with all sympathy and respect- you've got good reason to feel that way] aren't the only ones with coinstaller issues- but at least this one is never going to affect you.

I also got to tell him somewhere in the middle of all that about my fascination with the Gordian Knot, him being Greek and all [alas, I kept wondering if Greeks regarded Alexander as Greek, since Alexander was Macedonian- but I kept saying Mycenaean and totally fudging the issue- that aging memory again]. This time I figured my brute force approach to the knot was debugging it myself instead of doing the usual thing and trying to find someone who could just tell me or wanted to find out why something didn't work on such an old operating system [lets face it, the main focus is on Windows 7 around here]. He admitted to being one of those closet WdfVerifier users I occasionally speculate exist, and so it went [our conversations are usually quite a bit of fun, even when the situation isn't all that much fun- similar sense of humor, perhaps].

Those who hate my endless music lists can rejoice- this was much too long to bother trying to accumulate one. But I at least got to hear all that good stuff (ahh, Garcia's "Bird Song"- "tell me all that you know- I'll show you snow and rain")

L8r!

Make that April 16th for WDF 1.7...

BobKjelgaard — Fri, 11 Apr 2008 19:59:00 GMT

That seems to be the new consensus- only off by a day in that last post (which also has some background on why).

Some "Thank You"s

First off to Emil Protalinksi, for this article about our updated release date on Ars Technica. Not many people read my blog (which is fine with me), so having some coverage in an outlet of that caliber helps alleviate the flood of emails we get asking when we're going to be ready. Heck, I wind up there at least a couple times a month without even trying...

A second to Mr Evgeny Balykov, who just this week has joined us (and Microsoft) as our latest SDET (due to visa issues he is working across the border in Canada, but he's worth the hassle). Jenya's credentials are impressive, and he's eager to help us not have problems like this one [and to make WDF a better product overall]. One position left to fill!

Tons of Tunes, again...

Grateful Dead [Dead set]: Loser, Space

[Hundred Year Hall]: Me and My Uncle

Yoko Kanno [untranslated] : untranslated name- nice Japanese ballad

[Ghost In The Shell Stand Alone Complex]- Pet Food

Yuki Kajiura [Madlax]: Inside Your Heart

Unknown [InuYasha SoundTrack]- instrumental, sounds like the little fox demon's theme...

A New Target Date For WDF 1.7!

BobKjelgaard — Thu, 03 Apr 2008 18:15:49 GMT

As noted in the previous post (among others), I've not been saying anything about when WDF 1.7 will reappear for all those eager to begin using it, biding my time and waiting for a date that sounds achievable and has been given to a customer from our Program Management team.

Well, that's happened, and it's a day familiar to most U.S. taxpayers: April 15th, 2008. For those not in the US, that's the day income tax payments and their associated paperwork for the previous year are due [although you can get an extension on the paperwork by just filing more paperwork].

I've been involved in the testing, and this date certainly looks achievable. So hopefully this sad sorry saga is soon to be behind us. Still, I feel the need to stress- this is not guaranteed to happen. I could never guarantee such a thing, especially given what's already occurred.

Could I be a little more vague?

I've decided to shed a bit more light on what happened here. As I mentioned here and here, our problems trace to the fact that we are a part of the OS itself beginning with Windows Vista, and this introduced us to new update technologies [both new to us and new to the world of OS updating in general] that we just didn't know as well as one could have hoped.

The initial problem (and the one we were well on our way to fixing by that original March 15th date) was that the mechanism we used to update KMDF 1.5 to KMDF 1.7 was undone by a must-have fix to KMDF 1.5 that was finalized and released later than Server 2008. We knew about it, but we did them in the reverse order, and thus were quite surprised by this.

But in the world of OS fixes, there are two classes of fixes- those we know apply to all users (general distribution) and those that apply only to specific cases (limited distribution). When broad sets of fixes like service packs are made, a decision has to be made which fixes are included and which can be updated or superseded. Turns out we didn't understand this, properly, either.

So our second issue- the one mentioned in the previous post (link above)- came when we found out the UMDF 1.7 coinstaller failed to work when one of those "limited distribution" fixes was present. It seems our coinstallers could not update ANY fix of that sort. Worse, fixing it so they could meant changing the way they were built and packaged- I show here how to disassemble it, so that should give you an insight into the underlying complexity. But it's even harder than that would look, because the packages I extract there contain more internals you don't need to look at to solve the problem I was describing in that article.

So, learning what needed to be changed, getting the build processes changed, verifying the new processes [and there's a bunch of manual work all the way through all of that], and finally getting the ultimate packaged product and going through all of these new scenarios we now know we need to watch out for- all of that took a lot of time.

The silver lining

Well, the ongoing staffing problems I've been mentioning are getting handled. This is the first round of manual coinstaller testing in what seems like a very long time where I was not doing much of the QA single-handed (speaking only of KMDF, and I've always had some assistance, even then). Instead Wei Mao (whom I first mentioned here) did the super-critical stuff while I tried to get some new test features [behind schedule due to a host of reasons] ready.

Patrick, Shyamal, myself and Wei then spread out and did all that legacy stuff- will WDF 1.7 update WDF 1.5 or less? On all the OS we support? Do OS upgrades disturb this? Will it install when no WDF version was previously present? Do we still get public symbols (in case the build changes disrupted anything)? Do the debugger extensions still work? I even picked at things a bit with WdfVerifier- and so on. While Ilias was involved in the earliest stages, he was able to fix a bunch of bugs [many of those coinstaller "suggestions" of mine] while we did this.

So the team is beginning to function like it should again, and that is always a good thing.

Now playing- Yuki Kajiura [Fiction]: Fake Wings

Tsuneo Imahori [trigun the first donuts]: Philosophy in a Tea Cup

There were a slew of others, but for a change, I wasn't at all diligent about noticing- these were the final two...