Software Sleuthing

After many moons of sitting in a queue somewhere, the MSDN Tester Center just posted my intro article on special numbers you probably want to think about when running tests. The paper addresses several interesting domains of integers (all the way up to 64-bits) and gives you samples of important values to use:

In testing software, we often miss an entire class of bugs just because we don't happen to think about one category of inputs and how they might adversely affect the system. To help combat this in with my own tests, I've pulled together a listing of numbers which are good to use in testing for various independent reasons.

http://msdn.microsoft.com/en-us/library/ee621251.aspx

While looking at some of the traffic statistics for the MSDN Tester Center, I noticed that a bunch of people arrived at the site as a result of doing a Google search for "test plan template". The primary starting point for a template is really the IEEE standard 829-2008 which breaks apart the various types of documents and goes into detail on what should be present in each. In the Xbox software organization, we've taken the Level Test Plan and tweaked it to better fit our needs, but a (generalized) version of it is posted for your viewing pleasure: Word Template (dotx) or PDF.

A co-worker of mine was working on a tool to watch our release share for the appearance of a test result file, for which he would then publish up to a TFS server. The "trx" results file in question is actually being created on the Xbox console by our custom test harness, and historically was around a 10 megabyte (Unicode) file for a typical daily regression run. Yet on a Friday afternoon (when it actually happened to be nice outside), I get the following email from the tool writer:

Any idea why daily\testResults.trx is 1.10GB?  It’s too large to load and publish.

I'm thinking to myself: "One gig? Ay corumba! Did something go into a bad loop or something?"

I go out to the release server and start digging through the various log files, and sure enough, that thing is huge. Luckily we also produce a little rollup by component area which is written out to a text file and only contains the totals. The previous day's totals came out to 6422 tests for this suite. Today's number: an unorthodox 454,216. Yea, I guess adding 447,794 test cases will definitely blow out the size of a TRX file.

Now to dig a little deeper. The four hundred and forty-seven thousand test cases were just in one feature area. Looking at the test code it was definitely following a brute-force pattern over multiple axis, the first instance I noticed was O(n³).

Cranking through half-a-million test cases for a single feature is probably not the best use of your time (or the computer's). And I would hate to be a developer asked to run this suite in order to isolate a failure down deep in a nest of loops ("sorry Joe Developer, but it starts to fail around the 300,000th test case, good luck with that breakpoint"). Instead you should be trying to pick the fewest (reasonable) cases that adequately test the component. Here are some ideas to get you started:

·         Use Equivalence Classes/Partitioning. You can even use Code Coverage to help determine groups.

·         Use All-Pairs or Pairwise pruning techniques (Microsoft's PICT tool can be downloaded for free).

·         Use data generation (e.g. Fuzzing) or Data-driven techniques to increase your coverage over time. You don't need to run every single variation all at once, instead you increase your surface area a little bit each day.

So we have these fancy touch screen displays on the outside of our conference rooms which will tell us when the room is booked (and by whom). They are definitely handy as they help resolve the inevitable conflict when multiple groups arrive at the same time and then fight over who owns the room. They also have the added bonus of allowing an individual to walk-up and immediately reserve an available conference room using the touch screen interface.

This is all assuming, of course, that the device hasn't crashed. In the past several weeks that our team has been in the new buildings I've now seen two conference rooms who's displays have crashed (on this floor alone).

And yes, not only do they crash with a splendid Blue-Screen-Of-Death, but they crash upside down.

The only reason I could think of for inverting the normal, run-time image, would be if you could access the unit from the back: if you flipped the screen down in a diagnostics mode, then everything would be right-side-up. Unfortunately these units aren't accessible from the back. They are truly embedded in the wall.

As I see it, there are two failures here. One is obviously in the uptime testing/certification of the devices. These things are on and running 24x7x365 and apparently something got missed. The second failure is a design issue. There is no user means of resetting the device. Every time these things crash, a technician has to come out, crack open the front panel and reboot the hardware.

Here is my semiannual post on testing conferences that will be occurring later this year.

For a simple tool I was working on I needed to talk to a Microsoft Access database from my program. Now, most of the database work I've done in the past has been against SQL Server, so I needed to dig up the magic voodoo which represents the "connection string" for Access 2007 files. The connection string is used to tell the engine underlying the APIs how to talk to a specific type of database or file (i.e. what driver to use). Unfortunately most of the samples I found in MSDN were for an older version of Access (dealing with the old .mdb files) and not for 2007's .accdb files.

After consulting my good friend Google, I stumbled across the "Connection String" site which gave me exactly what I needed and I figured was useful enough to share.

Clear communication is essential to our jobs and yet it always amazes me how poorly we do it. I received this chart in an email (about stress results).

Now, the interesting part: I've just given you as much context as to the meaning of the chart (and the email) as the author did. From the legend, I now know what color green, yellow, and red are, but aside from that, I've learned absolutely nothing. I may be able to go away with a little bit of comfort since the "red" line has gone down (and hopefully that is a good thing), but otherwise this email has done nothing but waste my time (especially since I've emerged from my blogging hiatus to write about it).

When you communicate with someone, you are hopefully trying to deliver some kind of message. If that message isn't brutally clear, then you have failed. In the case of the stress email, what was missing was a substantial amount of context. But even if you had been privy to the dark mysteries which made up that missing information, you would still have had to do your own analysis and basically connect the dots between what a "green" line is and what it actually represents, because that link itself was not brutally obvious to begin with.

If you are going to take the time to deliver a message, then take the time to actually think about what you are saying and how you are saying it. If the author had actually thought about the chart for a second he probably would have realized that units would be a good thing and the default/existing legend was completely useless.

SANS, along with a coalition of individuals and software organizations, just released their list of top "25 most dangerous programming errors". There has been some talk of making these a "requirement" or some sort of logo program to help protect customers when deciding which software to use or purchase.

From a developer's perspective you should review these and ensure your design and code are doing the right thing. From a tester's perspective, be sure you have cases in place to validate and document how you are verifying these. If the items do become a hard requirement your group will probably have to provide proof of what was tested and how.

Ideally your team is doing all this anyway, but if not, here is a handy checklist to get started.

Before the holidays, there was a big re-organization of personnel here in Xbox land. As a result, I figured that this would be a good time to update my Black Book and have some more copies printed out for all the folks which are now a part of our new-and-improved software testing family.

The version which Bj just recently blogged about here was last updated a year ago, so the new one obviously contains more content. The changes also include formatting for better printing out in a 5.5" x 8.5" booklet (if you are so inclined). And as an aside, I'm not sure I deserve space in the same blog as the likes of Binder etc, but I will take what I can get :)

Link to PDF: Black Book

For those of you which haven't seen it yet, it just contains a plethora of random reference material. The likes of which I have no desire to memorize, but for some reason I find myself having look up more than once. At least having it all in one place means I no longer have to spend time searching.

A fellow reader sent a comment outlining the following problem:

I'm using the DMPSTK example from the Debugging Tools SDK. If I generate a crash dump from within Visual Studio 2005, I can see the call stack perfectly. However, if I create my own crash dump (the same as  you do in your blog), I get several LoadLibrary(ext), LoadLibrary(exts), ... failures. What's more important to notice is that the callstacks are different.

The LoadLibrary messages are benign and just an artifact of the debugger attempting to load additional plug-ins. But with regard to the stack traces, things get more interesting.

It turns out that the crash dump files created with Visual Studio are slightly different from the ones created via the API MiniDumpWriteDump. On top of that, WinDBG and the debugging engine exposed by WinDBG handle things a little bit differently with regard these dumps. The main difference stems from the use of embedded "contexts". A context contains the basic state of the system (CPU registers for example), and crash dumps can contain an embedded context which gets added in if you pass down the appropriate exception details to MiniDumpWriteDump.

If you were to load up one of these API generated crash dump files in WinDBG itself, the program will kindly tell you that there is an additional context and in order to use it, you need to issue the ".ecxr" command. Until you do this, the stack trace you will see possibly is not the stack you are interested in. This is where things are going wrong for our reader, who was displaying the default stack via the code in DMPSTK sample.

Luckily for us, we can do the equivalent to .ecxr in our code allowing us to get the "correct" stack.

To start with we replace IDebugControl for IDebugControl4. This then gives us access to the GetStoredEventInformation and GetContextStackTrace methods.

If we were to modify the initial code from this blog entry, we would end up with something like this which lets us try to get the embedded context, otherwise just fall back on the old standby (GetStackTrace).

Just don't forget to delete the newly allocated "contextData" buffer at the end of the function.

Here are the links and dates for some of the testing related conferences which are approaching.

If you want to get a jump start on conferences in the latter half of 2009, you can browse through the ones that were around in 2008. And thanks to Alan's comment on that page, I'm now pushing these through the Tester Center site as well, so they should be showing up there in the near future.

This blog has been pretty quiet for a while since we've been busy doing the last push getting the "New Xbox Experience" polished and ready for release later this month. The new UI is pretty cool and is a nice break from the "Blade" interface which I've always thought was too web'ish (and very flat coming from the green 3D space on the original Xbox).

As for functionality, I'm the most excited about the Netflix integration and the ability to install games to the hard drive. Both of these features have been written about elsewhere, so there isn't much need to go into detail here. But they will definitely improve the entertainment and gaming enjoyment people experience from their Xbox console.

And since the whole "Avatars" thing is a big topic as well, here is an Xbox rendition of myself.

I'm off at StarWest, so the blog will be a bit quieter this week. If you happen to be in Anaheim, look for the pale bald guy.

The relatively new "Metadata Working Group" just released their first publication on guidelines for making the metadata stored in image files more portable across applications and file formats. Ever since I started photographing with a Digital SLR camera, I've been paranoid about preserving the rich metadata stored in every file. Paranoid to the point of writing my own tool for modifying/adding to the information as pretty much every single application I've ever tried has corrupted the information to some extent (deleting tags, moving them around, and generally doing bad things).

The goals of the working group are admirable, now we just need more applications to start following them.

As seen earlier, it can be problematic to run scripts which rely on 32bit COM objects on a 64bit platform. To make things easier, I wrote basic some script code to detect that the wrong scripting host was run and fork out to the correct one. This just uses a variation of the "detect cscript vs. wscript" pattern, but instead we look at the current processor and do a check to see if we are running the wrong flavor of the script host.

In this sample we are just checking for AMD64, so you may want to add logic for Itanium or other platforms as well.

Be sure to notice the "ATTEMPT" comment when transposing the command line options from the current process into the new one. The way that the command shell handles double quotes makes it hard to reproduce the exact command line from the support we have in script. For example, if you pass /abc="hello world" as an argument, the quotes will get stripped out and you will actually see /abc=hello world from your script code. If we pass this raw argument (as shown above) into the new process then the one argument will be interpreted as two because of the missing quotes. It is left as a reader exercise to detect spaces and reconstruct the exact arguments if needed.