Software Sleuthing

A co-worker of mine was working on a tool to watch our release share for the appearance of a test result file, for which he would then publish up to a TFS server. The "trx" results file in question is actually being created on the Xbox console by our custom test harness, and historically was around a 10 megabyte (Unicode) file for a typical daily regression run. Yet on a Friday afternoon (when it actually happened to be nice outside), I get the following email from the tool writer:

Any idea why daily\testResults.trx is 1.10GB?  It’s too large to load and publish.

I'm thinking to myself: "One gig? Ay corumba! Did something go into a bad loop or something?"

I go out to the release server and start digging through the various log files, and sure enough, that thing is huge. Luckily we also produce a little rollup by component area which is written out to a text file and only contains the totals. The previous day's totals came out to 6422 tests for this suite. Today's number: an unorthodox 454,216. Yea, I guess adding 447,794 test cases will definitely blow out the size of a TRX file.

Now to dig a little deeper. The four hundred and forty-seven thousand test cases were just in one feature area. Looking at the test code it was definitely following a brute-force pattern over multiple axis, the first instance I noticed was O(n³).

Cranking through half-a-million test cases for a single feature is probably not the best use of your time (or the computer's). And I would hate to be a developer asked to run this suite in order to isolate a failure down deep in a nest of loops ("sorry Joe Developer, but it starts to fail around the 300,000th test case, good luck with that breakpoint"). Instead you should be trying to pick the fewest (reasonable) cases that adequately test the component. Here are some ideas to get you started:

·         Use Equivalence Classes/Partitioning. You can even use Code Coverage to help determine groups.

·         Use All-Pairs or Pairwise pruning techniques (Microsoft's PICT tool can be downloaded for free).

·         Use data generation (e.g. Fuzzing) or Data-driven techniques to increase your coverage over time. You don't need to run every single variation all at once, instead you increase your surface area a little bit each day.

So we have these fancy touch screen displays on the outside of our conference rooms which will tell us when the room is booked (and by whom). They are definitely handy as they help resolve the inevitable conflict when multiple groups arrive at the same time and then fight over who owns the room. They also have the added bonus of allowing an individual to walk-up and immediately reserve an available conference room using the touch screen interface.

This is all assuming, of course, that the device hasn't crashed. In the past several weeks that our team has been in the new buildings I've now seen two conference rooms who's displays have crashed (on this floor alone).

And yes, not only do they crash with a splendid Blue-Screen-Of-Death, but they crash upside down.

The only reason I could think of for inverting the normal, run-time image, would be if you could access the unit from the back: if you flipped the screen down in a diagnostics mode, then everything would be right-side-up. Unfortunately these units aren't accessible from the back. They are truly embedded in the wall.

As I see it, there are two failures here. One is obviously in the uptime testing/certification of the devices. These things are on and running 24x7x365 and apparently something got missed. The second failure is a design issue. There is no user means of resetting the device. Every time these things crash, a technician has to come out, crack open the front panel and reboot the hardware.

Here is my semiannual post on testing conferences that will be occurring later this year.

For a simple tool I was working on I needed to talk to a Microsoft Access database from my program. Now, most of the database work I've done in the past has been against SQL Server, so I needed to dig up the magic voodoo which represents the "connection string" for Access 2007 files. The connection string is used to tell the engine underlying the APIs how to talk to a specific type of database or file (i.e. what driver to use). Unfortunately most of the samples I found in MSDN were for an older version of Access (dealing with the old .mdb files) and not for 2007's .accdb files.

After consulting my good friend Google, I stumbled across the "Connection String" site which gave me exactly what I needed and I figured was useful enough to share.

Clear communication is essential to our jobs and yet it always amazes me how poorly we do it. I received this chart in an email (about stress results).

Now, the interesting part: I've just given you as much context as to the meaning of the chart (and the email) as the author did. From the legend, I now know what color green, yellow, and red are, but aside from that, I've learned absolutely nothing. I may be able to go away with a little bit of comfort since the "red" line has gone down (and hopefully that is a good thing), but otherwise this email has done nothing but waste my time (especially since I've emerged from my blogging hiatus to write about it).

When you communicate with someone, you are hopefully trying to deliver some kind of message. If that message isn't brutally clear, then you have failed. In the case of the stress email, what was missing was a substantial amount of context. But even if you had been privy to the dark mysteries which made up that missing information, you would still have had to do your own analysis and basically connect the dots between what a "green" line is and what it actually represents, because that link itself was not brutally obvious to begin with.

If you are going to take the time to deliver a message, then take the time to actually think about what you are saying and how you are saying it. If the author had actually thought about the chart for a second he probably would have realized that units would be a good thing and the default/existing legend was completely useless.

SANS, along with a coalition of individuals and software organizations, just released their list of top "25 most dangerous programming errors". There has been some talk of making these a "requirement" or some sort of logo program to help protect customers when deciding which software to use or purchase.

From a developer's perspective you should review these and ensure your design and code are doing the right thing. From a tester's perspective, be sure you have cases in place to validate and document how you are verifying these. If the items do become a hard requirement your group will probably have to provide proof of what was tested and how.

Ideally your team is doing all this anyway, but if not, here is a handy checklist to get started.

Before the holidays, there was a big re-organization of personnel here in Xbox land. As a result, I figured that this would be a good time to update my Black Book and have some more copies printed out for all the folks which are now a part of our new-and-improved software testing family.

The version which Bj just recently blogged about here was last updated a year ago, so the new one obviously contains more content. The changes also include formatting for better printing out in a 5.5" x 8.5" booklet (if you are so inclined). And as an aside, I'm not sure I deserve space in the same blog as the likes of Binder etc, but I will take what I can get :)

Link to PDF: Black Book

For those of you which haven't seen it yet, it just contains a plethora of random reference material. The likes of which I have no desire to memorize, but for some reason I find myself having look up more than once. At least having it all in one place means I no longer have to spend time searching.

A fellow reader sent a comment outlining the following problem:

I'm using the DMPSTK example from the Debugging Tools SDK. If I generate a crash dump from within Visual Studio 2005, I can see the call stack perfectly. However, if I create my own crash dump (the same as  you do in your blog), I get several LoadLibrary(ext), LoadLibrary(exts), ... failures. What's more important to notice is that the callstacks are different.

The LoadLibrary messages are benign and just an artifact of the debugger attempting to load additional plug-ins. But with regard to the stack traces, things get more interesting.

It turns out that the crash dump files created with Visual Studio are slightly different from the ones created via the API MiniDumpWriteDump. On top of that, WinDBG and the debugging engine exposed by WinDBG handle things a little bit differently with regard these dumps. The main difference stems from the use of embedded "contexts". A context contains the basic state of the system (CPU registers for example), and crash dumps can contain an embedded context which gets added in if you pass down the appropriate exception details to MiniDumpWriteDump.

If you were to load up one of these API generated crash dump files in WinDBG itself, the program will kindly tell you that there is an additional context and in order to use it, you need to issue the ".ecxr" command. Until you do this, the stack trace you will see possibly is not the stack you are interested in. This is where things are going wrong for our reader, who was displaying the default stack via the code in DMPSTK sample.

Luckily for us, we can do the equivalent to .ecxr in our code allowing us to get the "correct" stack.

To start with we replace IDebugControl for IDebugControl4. This then gives us access to the GetStoredEventInformation and GetContextStackTrace methods.

If we were to modify the initial code from this blog entry, we would end up with something like this which lets us try to get the embedded context, otherwise just fall back on the old standby (GetStackTrace).

Just don't forget to delete the newly allocated "contextData" buffer at the end of the function.

Here are the links and dates for some of the testing related conferences which are approaching.

If you want to get a jump start on conferences in the latter half of 2009, you can browse through the ones that were around in 2008. And thanks to Alan's comment on that page, I'm now pushing these through the Tester Center site as well, so they should be showing up there in the near future.

This blog has been pretty quiet for a while since we've been busy doing the last push getting the "New Xbox Experience" polished and ready for release later this month. The new UI is pretty cool and is a nice break from the "Blade" interface which I've always thought was too web'ish (and very flat coming from the green 3D space on the original Xbox).

As for functionality, I'm the most excited about the Netflix integration and the ability to install games to the hard drive. Both of these features have been written about elsewhere, so there isn't much need to go into detail here. But they will definitely improve the entertainment and gaming enjoyment people experience from their Xbox console.

And since the whole "Avatars" thing is a big topic as well, here is an Xbox rendition of myself.

I'm off at StarWest, so the blog will be a bit quieter this week. If you happen to be in Anaheim, look for the pale bald guy.

The relatively new "Metadata Working Group" just released their first publication on guidelines for making the metadata stored in image files more portable across applications and file formats. Ever since I started photographing with a Digital SLR camera, I've been paranoid about preserving the rich metadata stored in every file. Paranoid to the point of writing my own tool for modifying/adding to the information as pretty much every single application I've ever tried has corrupted the information to some extent (deleting tags, moving them around, and generally doing bad things).

The goals of the working group are admirable, now we just need more applications to start following them.

As seen earlier, it can be problematic to run scripts which rely on 32bit COM objects on a 64bit platform. To make things easier, I wrote basic some script code to detect that the wrong scripting host was run and fork out to the correct one. This just uses a variation of the "detect cscript vs. wscript" pattern, but instead we look at the current processor and do a check to see if we are running the wrong flavor of the script host.

In this sample we are just checking for AMD64, so you may want to add logic for Itanium or other platforms as well.

Be sure to notice the "ATTEMPT" comment when transposing the command line options from the current process into the new one. The way that the command shell handles double quotes makes it hard to reproduce the exact command line from the support we have in script. For example, if you pass /abc="hello world" as an argument, the quotes will get stripped out and you will actually see /abc=hello world from your script code. If we pass this raw argument (as shown above) into the new process then the one argument will be interpreted as two because of the missing quotes. It is left as a reader exercise to detect spaces and reconstruct the exact arguments if needed.

I've got a handful of stand-alone script files (mostly JavaScript) which I run from cscript.exe or wscript.exe to perform various tasks. Unfortunately, when I was setting up one of these scripts on a Server 2008 box it started to fail when attempting to instantiate one of my COM objects.

If you go lookup the failure code (0x80040154) you will see that it is the value for the "class not registered" error. Ok, no problem, just re-register the DLL again (via regsvr32.exe), and re-run the script. But, alas, this gave me the same error. So now I go digging through the registry to manually ensure the chain of keys and all their nodes look valid. I then try oleview.exe, and it is able to see the COM object and instantiate a version of it, so now things are really starting to get interesting (and a simple task which should have taken me a couple of minutes is now dragging out).

After a bit of further fruitless digging around, it occurs to me that the COM object is in a 32 bit DLL, and I'm running on a 64 bit version of Server 2008. Hmm, I wonder if that is the issue. I then go searching through my windows directory for all the copies of cscript.exe, and come across two: one in "system32" and one in "syswow64". So I then go and run the syswow64 version on my script file, lo and behold, it executes flawlessly.

So if your computer tells you that your COM object isn't installed, but you are pretty darn sure it is, check to ensure you are also running a compatible script host...

Throughout the course of my testing career, I've often heard this dreaded statement. Despite the causes behind it, I think that there will always be times when you just won't get any formal documentation or specification on how a piece of software is supposed to work. Not to mention the equally evil (or perhaps more so) times, when the documentation is completely obsolete and doesn't represent reality. 

When I run into this, my work commonly falls into the following pattern consisting of four different phases.

Investigation

Here I spend some time upfront to get an understanding of what the software currently does and how it acts. A good chunk of time is used reading through and reviewing the code itself. I will also typically write a very basic (straight line) test making it easier to trace through the code in a debugger, which gives me a better feel for the program flow and organization.

Write the tests

Once I have some understanding of what the code is supposed to do, I then start cranking out test cases. Throughout the investigation and while I'm writing and running the tests, I'm also communicating with the developer (and the PM if there is one) in order to answer specific questions about the behavior.

Conflict resolution

After I've written a chunk of tests, there are undoubtedly a number of areas where a bug may or may not exist depending on someone's point of view. Since I've written my tests based on my perceived conception of how the software should behave (and the developer has done likewise with the code) there will be areas where my test expects one behavior, but the program does something else. And without a specification to arbitrate, this often becomes a heated battle. Especially since the developer has already written the code (and is undoubtedly working on something else now), he/she will be reluctant to go back and change it based on the whim of a tester. Thisis where your persuasion and diplomacy skills come into use as you need to make a case and argue on behalf of the customer.

Documentation

After finishing up the tests and the stability of the software has solidified, I typically go back and provide some form of documentation. At the very least I ensure to comment specific weirdness along with my tests. But depending on the area, I may also provide some high level information to co-workers. For example, after working with the (undocumented) custom file system used on the Xbox, I created a basic table outlining the characteristics compared to other common file systems. This included information useful to my colleagues such as maximum filename length, maximum filesize, etc. At the other end of the documentation scale, you could always go and write the specification yourself, which would definitely help to improve the maintainability of the system.