The Galactic Patrol : Software Testing

Software Testing on Mars

bwill — Tue, 10 Feb 2004 12:34:00 GMT

Here is an interesting summary of the problem encountered by the Spirit probe on Mars. Important test lessons:

“test real-world conditions”

“run long-haul stress tests”

“SYSTEM BEHAVIOR: Spirit began acting up last week, when it stopped sending data and began rebooting its computer, resetting it roughly 130 times. At one point, the rover thought it was 2053.

BUG DESCRIPTION: Engineers found that the rover's 256 megabyte flash memory had retained hundreds of files containing flight data and was still juggling them along with the daily flood of new data from its activities in Mars' Gusev Crater.

WORKAROUND: By commanding Spirit each morning into a mode that avoids using the flash memory, engineers plan to begin deleting hundreds of unneeded files to make the memory more manageable for the rover's RAM.

WHY WASN'T THIS CAUGHT IN TEST?: The bug had not been detected in operational tests of the rover on Earth because the longest tests lasted only eight or nine days.”

From http://www.testgeek.com

how to contact me

bwill — Fri, 12 Dec 2003 08:12:00 GMT

I published this info when I started my GotDotNet blog; with the move, now is a good time to re-present the info:

Bruce Williams

I am a test developer for Indigo – current and future Web Services technology at Microsoft.

e-mail: bwill@microsoft.com

IM: domanite@hotmail.com

If you develop on Windows - use AppVerifier!

bwill — Thu, 11 Dec 2003 03:47:00 GMT

I read a post recently from a person who was disappointed that they hadn’t heard of the AppVerifier tool that Microsoft ships. So let’s fix that – if you do commercial software development on Windows, you owe it to your customers to download AppVerifier and test your product with it. It is an extremely easy way to find bugs that might otherwise be very difficult to track down.

Be verwy verwy quiet - we're hunting bugs!

bwill — Fri, 26 Sep 2003 09:16:00 GMT

Eric Lippert has a great post describing some of his adventures with security issues in VBScript. I particularly liked this segment:

“Of course, it could be worse. There was a bug in early versions of the CLR (which I believe was fixed before the first beta shipped, fortunately) where you could get an error message something like

Path discovery security exception: You are not allowed to determine the name of directory 'c:\foo\bar'

Super! Thanks for letting me know!”

Fault injection tools from Microsoft

bwill — Wed, 24 Sep 2003 18:52:00 GMT

In a response to my recent post on fault injection, Mark asks if I know of any publicly-available fault-injection frameworks. Yes! In fact, much of the fault-injection work done at Microsoft is built on top of work done by Microsoft Research, and you can also take advantage of their work. In particular, look at the Detours tool for injecting code into an existing binary.

Unfortunately, Detours only does native code, not managed code. We do have managed code injection tools internally, but I don’t see them on the research site, so I guess they are ‘top secret’. You are in luck, however; I’ve also heard of a well-known fault-injection tool called Holodeck, that does do managed code. This also has the advantage of providing a nice user interface and support tools – a point-n-click operation. Detours is a bit more low-level, and requires some coding and work on your part. Unfortunately Holodeck does appear to be pretty pricy - $5000 for a single license. Let’s keep looking…

A google search on fault injection finds a bazillion hits – take a look there and see if you can find a tool that meets your needs.

For a good time, call 1-800-FAULT-INJECTION-TESTING

bwill — Tue, 23 Sep 2003 08:49:00 GMT

We have occasional debates here at Microsoft on the value of ‘white-box’ testing. (For the sake of this discussion, ‘white-box’ testing means testing that takes advantage of internal product knowledge, while ‘black-box’ testing relies solely on the published API and behavior.)

On the one hand, if the person who developed code writes the test for it, or if the tester was intimately involved in the design of the feature, then the test design is often blinded by the dev design. For example, if the development of the feature goes to great lengths to ensure that the threading and locking behavior is correct, then the tester may spend a majority of their test effort there. This is a problem if, for example, there are major flaws in the way the feature interacts with its config files. You don’t want a certain focus in product development dictating a similar focus in test development.

On the other hand, there are some kinds of bugs that can *only* be found with white-box testing. If code has an off-by-one error that only kicks in at the exact buffer size that switches between the fast-path algorithm and the slow-path algorithm, there is every possibility that even well-designed equivalence classes against the published API won’t find the problem. A quick peek at the code, however, can tell you exactly what buffer size to use, if you want to give the product a conniption fit.

However, there is one kind of white-box testing that is pretty much universally seen as useful – fault injection testing. This is where you inject code at compile- or run-time to force faults that would not normally occur. These are great! Once you have a fault-injection test framework set up, a fault scenario that might ordinarily require an IBM mainframe database and a female goat can be duplicated in a few couple minutes, with a few simple lines of code. If you are hard-core about testing error handling in your program, you owe it to yourself to investigate fault-injection schemes. It is similarly good for increasing code-coverage numbers, if you use that metric.

Now, there are a lot of different ways you can do fault injection. Recently I’ve seen some work done injecting faults at the protocol level during SOAP message transmission. It is interesting to send SOAP messages from one place to another, and have a little bitty guy who sits in between and drops, alters, or otherwise interferes with the messages flowing by. (Would that be an example of MaXMLwell’s Demon?) Going further back in the past, I’ve seen a system that systematically generated every possible memory allocation fault while running our test suite.

Now, that last example brings up the ‘dark side’ of fault injection testing. You see, there were a couple problems with that memory allocation testing. Here is how it worked:

1. Pick a single test variation from your test suite, and run it. Keep a count of the number of memory allocations executed.

2. Run the variation again. Fail the first memory allocation, and all subsequent ones.

3. Run the variation again. Fail the second memory allocation, and all subsequent ones.

4. Keep doing this until you’ve gone through the entire set of memory allocations that a successful variation run does.

5. Go to the next test variation.

Now, the most obvious problem is that this test takes *forever* to run. It literally took weeks to run through a simple set of COM tests. (Oops, did I accidentally reveal the product in question?) You really have to look at the bugs you find with this method, and ask yourself “is finding these bugs worth the cost of the testing?” There might be other testing you could be doing that would find better bugs, or find bugs more efficiently. So if you are doing ‘high-volume’ fault injection, you really need to balance the cost of the testing against its value to you.

Another problem, really more of a nit, is that while this method seems comprehensive, there are many cases it doesn’t hit. What if the memory allocations are only failing intermittently? If something fails only on the pattern “allocation failed, allocation succeeded, allocation succeeded again, allocation now fails again”, then the test above won’t find it. I don’t have an answer to this one, except to give the sad truth that you can always think of more things to test, than you could possibly actually write (or run) tests for.

Another, much more significant problem, is this: how do you know if it failed? We took the easy way out, and said “if it AV’s, blue-screens, or otherwise crashes – it failed.” So if it appears to succeed; or if it returns a wrong error code; or if the behavior of the program is just wrong, in a non-crashing way – we would not detect it. Why did we do this? Well, remember our discussion of costs and tradeoffs. I’ll give you a million lines of test code, developed over ten years by 40 different people. Are you gonna track through that and fix it so it reports errors correctly *for every possible memory allocation failure path*? Note that I’m not claiming that level of detailed testing isn’t useful – it just wasn’t feasible in that case, on the scale of the entire test suite. Picking a set of core functionality and writing some exceptionally robust test code specifically for this type of fault injection scenario might be a useful endeavor. We’ll see. <grin>

For those concerned about Microsoft code quality, note that these days we do also have some static analysis tools that will churn through a reasonable subset of possible call graphs in our programs, and report possible problems in error paths. It even files bugs automatically – the Windows folks love that, I’m sure!

Some of my readers may be familiar with Microsoft’s stress-testing efforts, where we often hammer a machine with tests to the point of program failure. While stress testing is useful, don’t be fooled into thinking that it is an adequate replacement for fault-injection testing. The biggest problem with stress testing is the “early exit” problem. If you are crushing a machine to the point that memory allocations are failing, then most programs are just going to die immediately when you run them. You are only going to end up testing the first 10% of the program, which presumably is not the intent. Another issue is that stress test failures tend to be non-deterministic; you don’t necessarily know when (or if) a particular failure will occur, and it can be very difficult to determine the actions that led up to the failure. This makes debugging a stress failure much more, well, stressful; than debugging an equivalent failure in a fault injection test.

Lastly, note that (as with so many of the topics I write about), there is much, much, more to it than I have written. For example, there are many different kinds of faults that you can inject. We talked about memory faults and protocol faults; but there are also file access faults, security faults (a biggie!), system object access faults (events, mutexes, etc), registry faults – you can even feel free to inject faults into your own internal product code – what happens when your lower-level stuff throws an exception to the higher-level stuff?

It is really not that hard to get started with, however; and I highly recommend it.

Test tracing - it is not optional!

bwill — Sun, 21 Sep 2003 02:15:00 GMT

I’d like to talk about one of the first lessons I learned as a tester at Microsoft:

Anticipate failure investigations, and log everything.

Let me tell you a fairly rambling tale of my early days at Microsoft:

We’d just shipped Windows NT 4.0 (and IIS 1.0). I’d recently moved from a contract position in the Windows build lab; to a full-time position (aka FTE, aka ‘blue-badge’) in the COM test team.

In those days, the COM team was just beginning its transition from a consumer, drag-n-drop, linking-and-embedding technology; to a distributed computing, enterprise-application sort of focus. This was all before COM+; DCOM was the new thing. Given that history, it is perhaps not surprising that we had a bunch of old tests covering a variety of those aforementioned consumer-oriented applications of COM. Linking and embedding, COM common dialogs, and more; these were tests that had been lovingly handed down from tester to tester over the years.

You’ll recall I mentioned a transition. Now, those who have been in the industry for a while know what it means when the product changes focus like that. Resources shift. Things happen, like taking these “upper layer” tests from the five experienced testers who owned them, and giving them to a new, green tester fresh from the build lab. Sink or swim, as they say. Let me tell you, taking over maintenance of a large test suite (written by a number of different people, in a number of different styles *ping*) is a great way to learn some valuable lessons about test code design.

For example, there was a test for the OLE common dialogs, used for inserting OLE objects and other operations. This test basically ran a large script (*ping*) that iterated through a bunch of scenarios exercising these dialogs. In total the scripts performed thousands of operations – click that button, move that window, select this other menu option, etc. Now, any time you have one program executing a script like that, you basically have one program running another program. This means it is particularly difficult to figure out, from a debugger, what the dang thing is doing. When you break in, the stack just shows you what the script interpreter is doing, and it requires some investigation of the program variables to figure out where you are in the script being interpreted.

Now, the test had a log file output, but unfortunately when I acquired the test this output was fairly minimal. Let me tell you, debugging a failure in this test was an adventure! It was time-consuming, and technically challenging. (Technically educational, too, I will say.) So I went into the interpreter and did something very simple – I just logged each unique operation as it executed:

LOG TRACE: Menu Select ‘File Open’

The logs got a lot bigger, and the test ran marginally more slowly (those UI tests tended to be dog slow, anyway), but now when something failed, I could look at the log and see the precise script actions that led up to the failure.

So what I learned is that it is really, really important in a test to log *everything*. Log your inputs; log your outputs; log all the program variables that seem even halfway interesting. It will make your life so much easier. It will make the life of the next test maintainer so much better. If it causes a severe performance hit, *then* you can perhaps think about placing some of the trace output under a switch, that you can enable or disable as needed for debugging purposes. Even there, make sure you can turn on the trace output without having to recompile the program – you’ll thank yourself later.

On a side note, in the last couple years I’ve noticed a trend in Microsoft product design to add this kind of detailed tracing to our products. Debug level tracing, in the production code – something you can enable or disable with a program or registry entry. I know that DTC has done this for a while, and I believe it has been standard practice in the mainframe world for decades. Of course, for performance reasons the tracing is typically turned off in normal execution – but it is there, and it doesn’t require a re-compile to enable.

PSS loves this – it really helps them track down and solve customer problems.

more Scoble stuff - there are many parts, but only one Microsoft

bwill — Sat, 20 Sep 2003 23:14:00 GMT

Again, while reading this same Scoble post, it got me thinking about my role at Microsoft. When I tell people what I do at Microsoft, I say "I do software testing. If you find a bug in a Microsoft product, it is my fault."

Of course, that is a vast oversimplification. I work on one part of one product, that isn't even released yet. I could easily disclaim responsibility for the particular problems people are having. But that’s not the kind of person I want to be. That’s not the kind of company I want to be part of. And if I don’t take the responsibility, who will?

“The first priority, young man, is to find the bugs…”

bwill — Wed, 10 Sep 2003 18:25:00 GMT

Prologue

One of my teammates recently set up a Wiki site for my team, and I must admit that I have become quite addicted to it. One of the pages I’ve created has been a list of common test areas, a checklist of things most of our tests will want to concern themselves with. Things like “bad parameters”, and “different transport protocols”.

Let me interject at this point that this concept of a common test checklist is by no means a new one. Anyone among my three readers who has done software testing before knows what I mean. Despite all the prior art, however, I believe it is useful to have a team-specific list; something that targets those test areas particularly relevant to that team. To pick a team completely at random, for example, .NET Remoting likely has specific test areas and checklist items that are particularly relevant to it.

My checklist currently has about 20-30 items on it; today I would like to focus on a single one of them; two simple words: Multithreaded Testing

In reality, it’s a topic that can fill a library. I’m going to chat about my empirical experiences in this area, but if you are interested, there is also considerable computer science research in this area.

As I write this, I find myself mentioning test topics that deserve more discussion, in a future post. I’ll mark those with a *Ping* so I remember to go back to them some other day.

“The first priority, young man, is to find the bugs…”

Many moons ago, in the before-time, I wrote a test suite for the IErrorInfo interface and the associated COM infrastructure. (Yes, it is my fault if it doesn’t work right…) One set of tests that I wrote was designed to discover race condition problems when multiple objects on different threads got error objects back from the same target object.

I was actually quite proud of these tests. In one case, I would have threads A and B call object O, where the calls arrived in the order A, then B, but by judiciously blocking the calls in the target object, they would return in the order B, then A. In another case, I would force the order as A-calls, B-calls, A-returns, B-returns.

I had about four of these test variations. I had beautiful charts in my test specification describing the control flow. I had bountiful program output, describing the scenario in loving detail for anyone who might happen to need to debug a failure. I loved those, because if anything failed, I could point to the exact repro scenario and documentation needed to demonstrate and debug the bug. *Ping*

What I forgot (or hadn’t learned yet), was that documentation and easy repros, while important, are all “priority two”. The first priority is to *find the bugs*.

Hindsight is 20/20

You see, by artificially controlling the ordering of the thread actions, I’m also artificially constraining the product code paths that my test explores. For example, my test would never try the case where a call arrives at the target object at the exact same time as another call returned from that object; the care I took in synchronizing the scenario prohibited it.

What I should have done was kick off about a hundred threads, set up some loops so they continuously hit the target object, and let it run for a minute or two. Sure, random testing is not deterministic; there is no guarantee that a given failure will repro, and figuring out what happened when something does fail is a major pain in the rear. But remember, that is all “priority two”. Think about all those calls, twisting and twining, overlapping and conflicting, throughout the internal IErrorInfo infrastructure. Its gonna be *tons* better at finding bugs than the four simple variations I wrote years ago.

Now, even that random test case isn’t enough. There may be races that will just never show up on your machine, in your configuration. To catch stuff like that, you’ll want to induce errors or delays; the joys of fault injection. *Ping*

You also might want to run tests for longer than a minute or two, which brings us to our next topic…

How I learned to stop worrying and love the stress

Sometimes, a race only shows up once every couple months. Sometimes it will only show up on a single machine; the one that has the magic combination of system components that demonstrate the failure. To catch these, the Windows team has this thing called “Office Stress”. This runs a bunch of different tests, exercising many of the features of Windows. It crushes the machine – office stress will routinely peg the CPU, and things run so slowly that failures are the norm.

Now, honestly, that is mostly useful for testing software that cannot fail. Things like winlogon, or rpcss; if they fail, the machine fails. These core system components have to keep functioning even if 90% of their memory allocations start failing – and office stress will force that condition.

Office stress is not so useful for testing other kinds of programs. Your typical application doesn’t expect to keep working with memory failures – it just dies, hopefully with some appropriate error message. Typically when you run these programs under office stress, they’ll die in the first 10% of the program, so you end up never testing the other 90% of the program. For these programs, a lower-intensity variant of stress is appropriate. We’ll tune the test configuration so that it runs at about 70% resource utilization, and just let it run continuously. The ASP.NET team does a lot of their stress testing like this; in addition to finding race conditions and other multithreading issues, it is also good for finding slow resource leaks.

On a side note, we also have a concept of “long-haul” stress. Teams at Microsoft often have ship criteria where we won’t ship a product unless it has run on so-many-hundreds of machines, for a certain number of days, under stress, without failures. For Windows, for example, I believe it is something like forty days. (Once we start the last one of these forty-day test passes, it is a major pain in the rear if someone finds a showstopper bug that resets testing…) Depending on the team, long-haul stress may consist mainly of high-intensity or low-intensity stress testing.

In the managed world, we do an interesting variation on fault-injection combined with stress. We have this tool called GCStress, which basically forces the garbage collector to do a collection on every program step. Yeah, its really slow. By running this along with our tests, we can surface memory failures pretty much at the point where they occur, which makes them much easier to debug. (Similar to using the appverifier and turning on full pageheap, in the unmanaged world.)

Anyway, I’m definitely rambling away from the original topic of this entry, so I’ll sign off now. I’ve enjoyed writing this up; it helps me clarify the concepts in my mind. Please do let me know if you find it interesting as well!

Escape Analysis; or, 'how did that bug slip through testing?'

bwill — Wed, 10 Sep 2003 03:57:00 GMT

After the Indigo team's last milestone (M4), most of our feature test teams did an analysis of the bugs in their feature area, and extracted some useful 'lessons learned' to help us improve our testing in future milestones. Here are some of the lessons we learned:

1. Do breadth-first testing, sanity testing, and test prototyping as early as possible. A higher-than-expected number of our bugs were found by developers and casual users of the release; bugs that we expected test to find, and ones that they later did find. We could have saved those folks some wasted time if we had a basic level of testing available in all feature areas as early as possible in the milestone.

2. Be very thorough in design and spec reviews; it pays off. Explore the alleyways of the design, and ask obscure questions about threading and error handling. You'll save yourself a lot of grief by working these issues out in design, rather than discovering them through exhaustive investigation during later testing.

3. Be very clear about how testing of cross-team dependencies will be done. Documentation, communication, and review across feature teams is important. If you ever find yourself saying "we don't need to test that, team XYZ is doing it", stop yourself, and go double-check that team XYZ agrees with your conclusion.

4. Participate in dev code reviews and set aside some time for white-box testing. My team tends to go back-and-forth on the value of white-box testing. In our M4 milestone, however, did identify a number of bugs that could have been caught if we (test) had a better concept of the internal product design. There were also several bugs that would have been extremely difficult to find without using some kind of white-box testing technique like fault injection.

I work on "Indigo"

bwill — Mon, 08 Sep 2003 19:26:00 GMT

I noticed at this link that folks are starting to hear (and speculate) about Indigo. That’s cool to see, because that’s my team! I know, I shouldn’t tease, because I’m not allowed to talk about it yet – but still, it is just cool to start getting some public recognition. Even if it is just guesses about what we are about. Insert obligatory PDC reference here.

ASML - a modeling tool for testing

bwill — Wed, 03 Sep 2003 06:57:00 GMT

John Tobler talks a bit about Abstract State Machine Language (ASML) on his blog. This is actually a really cool technology – one my test team has been looking at for a little while now, and one that I personally have been struggling to wrap my mind around. At first I thought it was a test-case generator; we have other state-based tools here at Microsoft that do that kind of work, so that is where my mind goes when I think about “state machine” tools. However, I’ve come to realize that its true role (ok, one of its true roles) is as an oracle.

Suppose you write a test, and want to check if the result is correct. How do you know? If you hand-wrote the test, then you probably “know” the expected result. But if you automatically generate thousands of test cases, it gets harder. What you really want is a second, completely independent version of the program, whose results you can compare to the real program. This second source of expected test results is what testers call an ‘oracle’. ASML is a computer language (a .NET language, even) that lets you write that oracle really fast. The oracle will encapsulate the broad strokes of the real thing, while likely glossing over a few of the details – but its close enough to serve as a good point of comparison against the production code. Thus your ASML program is a ‘model’ of the real thing.

It’s a really cool concept, especially since the more I do this testing thing, the more I realize that automation is the only way to fly. Being clever about the automation and about choosing what to test is the future of software testing.

Another role of ASML is as a spec validator; the process of implementing your spec in ASML can show spec flaws a lot faster than doing so in a production development effort.

Another super-cool thing about ASML is that it is a full-fledged .NET language, so it has access to the whole .NET Framework. It’s a real programming language, and you can write real programs with it. I have a colleague who has an exercise he does to learn a new programming language; he writes ‘Tetris’ in that language. I’ve seen a functional version of Tetris written in ASML.

Closing the cycle between test and customers

bwill — Wed, 13 Aug 2003 19:19:00 GMT

I was thinking about the role of test this morning. We measure the quality of the product - but in some sense we have to take an educated guess what "quality" means. In the end, the customer determines if it is a quality product, and they can't make that decision until we ship the product. (Well, we do have betas and previews to help out.) Waiting for summary quality rollups from PSS after shipping seems like a pretty slow and cumbersome way to have a "quality connection" between test and the customer.

So what can we do to tighten the customer-test cycle of information? Clearly looking at beta feedback is critical, but is there anything more "automatic" that we could do?

How about if we had a sampling profiler that ran automatically when people run the beta? Sampling profilers are low-impact, and spread over all beta testers, we'd get solid coverage numbers. Upload and integrate the data, and compare it to our test code coverage data. Bang! We have an automated method to compare how we test the product against how customers actually use it. Heck, if we have internal customers using our stuff, I'd love to get their data too!

I wonder what the next step would be? Maybe learn more about sampling profilers... I'll see if I can find some time for that...

An example of Test-Driven Development

bwill — Wed, 13 Aug 2003 19:16:00 GMT

A fairly long description of a project using TDD can be found at:

http://redsquirrel.com/dave/work/babySteps/#200306

I've skimmed it, and look forward to perusing it in more depth when I have time.

If a metric falls in the forest, did it make a sound?

bwill — Wed, 13 Aug 2003 19:07:00 GMT

I heard a comment today that really struck home with me:

"If you aren't willing to act on a metric, you might as well not measure it."

This makes sense to me, and also strikes a nerve. I know I've seen any number of metrics over the years - bug charts, task progress charts, anything and everything that can be measured. I admit I've been guilty at times of looking at some numbers and saying "that looks bad!" and waiting for a response, and happy that we had this great information available.

But having the information is only half the story.

If a metric is off, if it indicates something is late or not getting done, the right comment is "this measurement isn't what we expect or need - and here is what we can do about it." Cut a feature, extend the milestone, shuffle some people around - if you aren't willing to define and accept the remediation actions, you might as well not even look at the measurement.