JW on Test

tour of the month: the exit-stage-right tour

2009-05-21T23:11:00Z

All tours much eventually come to an end and thus it is with my tour with Microsoft. I have resigned my position and am leaving the company. It was a great ride.

But the tours will continue. My book Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Manual Testers is in press and will appear through Addison-Wesley sometime this summer. I am truly thankful for the many wonderful testers at Microsoft who contributed wisdom, thoughts and even case studies to the effort. Special thanks go to Nicole Haugen, Geoff Staneff, David Gorena Elizondo, Shawn Brown and Bola Agbonile. Microsoft is full of great testers and even here, these guys manage to stand out.

I imagine that I will not be long in setting up a new blog as I have very much enjoyed this experience and being the only tester in Developer Division's top ten bloggers was quite an honor. For that I thank you.

In case you are interested in my landing place, I can imagine that one or two of the more popular testing blogs around town will be talking about it.

Wish me luck ...

tour of the month: the landmark tour

2009-04-06T21:18:00Z

Every location that covets tourists must have some good reasons for them to come. For Las Vegas it’s the casinos and the strip, for Amsterdam it’s the coffee shops and red light district, for Egypt it’s the pyramids. Take these landmarks away and the place is no longer an attraction.

Software is much the same – there has to be some reason for users to buy it. Some landmarks that make it what it is. If you identify the landmark features that draw users, that’s where you need to concentrate a lot of testing. You can tell where the tourist landmarks are by noticing crowds and watching where they spend their money. For exploratory testers finding the landmarks also means following the money, literally. And money usually leads directly to the sales force.

Sales folk spend a great deal of time giving demos of applications. One might imagine that since they get paid based on fulfilling their sales quota, they would be very good at it and would include any interesting nuances of usage that make the product look its very best. They also excel at shortcuts to smooth out demos and often come up with scenarios that sell the product but weren’t part of any specific requirements or user story. The long and short of it is that sales people are a fantastic source of information for the landmark tour.

Testers performing the landmark tour should sit in on sales demos, watch sales videos, and accompany salespeople on trips to talk to customers. Each demo or demo script identifies the landmark features and can even give you insight into their relative importance by how often those features appear.

To execute the tour, you could simply run through the demos yourself and look for problems. As the product code is modified for bug fixes and new features, it may be that the demo breaks and you’ve not only found a great bug, but you’ve saved your sales force from some pretty serious embarrassment (perhaps even salvaging a sale). I have found enough bugs this way to privately wonder if there is a case to be made for testers sharing in sales commissions!

But we’ve found a more powerful way is to inject variation into the demo scenarios by “landmark hopping.” Ever use a compass in the woods? That’s the effect we are going for here. My army-trained older brother taught me to locate landmarks (rocks, trees, hillsides) in a compass scope and then to make a path from landmark to landmark in the general direction you want to go. It kept me from getting lost in the woods of Kentucky where I grew up and it creates excellent variation in demo scripts and end-to-end sceanrios. Pick the landmarks in advance, scramble them up and tour from one to the next. Rinse and repeat trying to use the landmark feature in a different way every time.

A powerful variation of this tour is the skeptical customer tour in which you execute the landmark tour but pretend there is a customer constantly stopping the demo and asking ‘what if’. ‘What if I wanted to do this?’ they may ask, or, ‘how would I do that?’ requiring you to go off script and include a new feature into the demo. This happens a lot in customer demos, especially the serious ones where a purchase is imminent and the customer is kicking the tires one last time. It’s a powerful way to create test cases that will matter to end users.

Landmarks can be "back of the box" features, features used as part of sales demos, features popular to users (in the event you collect such statistics), features prominently displayed in the UI and so forth. Clearly any bugs you find on this tour or any of its variations are very important ones as they are likely to be seen by real customers.

Enjoy the tour and don't get lost.

testing sucks

2009-04-03T00:11:00Z

Bet that got your attention. It's true, but let me qualify it: Running test cases over and over in the hope that bugs will manifest sucks. It’s boring, uncreative work and since half the world thinks that is all testing is about, it is no great wonder few people covet testing positions. Testing is either too tedious and repetitive or it’s downright too hard. Either way, who would want to stay in such a position?

What is interesting about testing is strategy, deciding what to test and how to combine multiple features and environmental consideration in a single test. The tactical part of testing, actually running test cases and logging bugs is the least interesting part.

Smart test managers and test directors need to recognize this and ensure that every tester splits their time between strategy and tactics. Take the tedious and repetitive parts of the testing process and automate them. Tool development is a major creative task at Microsoft and is well rewarded by the corporate culture.

For the hard parts of the testing process like deciding what to test and determining test completeness, user scenarios and so forth we have another creative and interesting task. Testers who spend time categorizing tests and developing strategy (the interesting part) are more focused on testing and thus spend less time running tests (the boring part).

Testing is an immature science. There are a lot of insights that a thinking person can make without inordinate effort. By ensuring that testers have the time to take a step back from their testing effort and find insights that will improve their testing, teams will benefit. Not only are such insights liable to improve the overall quality of the test, but the creative time will improve the morale of the testers involved.

So all the managers out there need to ask themselves what they've done lately to make their testers more creative. If you don't have an answer, then testing isn't the only thing that sucks.

live webinar next week

2009-03-25T23:23:00Z

I am doing another webinar on March 31 which will be hosted by uTest. This is the second webinar I have done for them and I quite enjoyed the first. I have a lot of new material that I have created (and am still creating) for this one which I've called "5 Insights to Revolutionize Your QA."

The link to register is http://blog.utest.com/?p=122. Note the use of the words 'sucks' in the blurb. I love it when I get marketing people to say sucks...

tour of the month: the intellectual's tour

2009-03-11T21:09:00Z

As promised, here is the first tour on the tour-of-the-month parade. It's probably not the best place to start, but it's finding so many good bugs for so many testers around the company that I wanted to get it in the hands of others sooner rather than later where it might make more logistical sense.

Enjoy!

I was once on a walking tour of London in which the guide was a gentleman in his fifties who claimed at the outset to have lived in London all his life. A fellow tourist happened to be a scholar who was knowledgeable in English history and was constantly asking hard questions of the guide. He didn’t mean to be a jerk, but he was curious and that ended up being a dangerous combination … at least to the guide.

Whenever the guide would talk about some specific location on the tour whether it was Oscar Wilde’s former apartment in Chelsea, details of the great fire, or what life was like when horses were the primary mode of transportation, the scholar would second guess him or ask him some hard question that the guide struggled to answer. The poor guide had never worked so hard on any past tour. Every time he opened his mouth he knew he was going to be challenged and he knew he had to be on his toes. He wasn’t up to the task and finally admitted that he had only actually lived in London for five years and he had memorized the script of the tour. Until he met the scholar, his ruse had worked.

What a fantastic bug! I was so impressed I bought both the guide and the scholar a pint when the tour ended at a pub (a place, incidentally, where the hapless guide was infinitely more knowledgeable than the scholar).

When applied to exploratory testing, this tour takes on the approach of asking the software hard questions. How do we make the software work as hard as possible? Which features will stretch it to its limits? What inputs and data will cause it to perform the most processing? Which inputs might fool its error checking routines? Which inputs and internal data will stress its ability to produce any specific output?

For folks who test word processors this tour would direct them to create the most complicated documents possible, ones full of graphics, tables, multiple columns, footnotes and so forth. For folks testing online purchasing systems, what is the hardest order we can place? Can we order two hundred items? Can we place multiple items on backorder? Can we keep changing our mind about the credit card we want to use? Can we make mistakes on every field in the data entry forms? This tour is going to be different for every application, but the idea is the same: ask your software hard questions. Just as the scholar did with the London guide, you are likely to find gaps in its logic and capabilities.

A variation of this tour is the arrogant American tour that celebrates a stereotype of my countrymen when we travel abroad. Instead of asking hard questions, we ask silly questions otherwise intended simply to annoy or impede the tour and draw attention to ourselves. We intentionally present obstacles to see how the software reacts. Instead of the most complicated word processing document, we make the most colorful, invert every other page, print only prime number pages, or place something in a location that makes little sense. On a shopping site we’ll seek out the most expensive items only to return them immediately. It doesn’t have to make sense … we do it because we can.

links you've been asking for

2009-02-25T20:26:00Z

I’ve gotten enough requests for links to my interviews, lectures and pointless videos that I am finally tired of responding individually. Here are a few recent ones I have been pointing people to:

A video interview on Channel 9 about software testing

A video interview on Channel 9 about Visual Studio Team System

A radio interview on Dot Net Rocks with two funny guys (I had a blast with this one and it will be one reason I never run for political office in the future)

The Future of Testing webinar for uTest

No, “Piece of Crap” is not included. Sorry but it’s time we put that one to sleep. Lee Copeland is widely known to play it at his lectures so try and catch him on his speaking circuit if you just can’t resist. That video is another reason I can’t run for political office. Neil Young has my sincerest apologies.

Some internal links for my MS readers:

The Future of Testing

An Update on the Future

Online Security Testing Course (this is officially a classic since it is the last time I taught this material)

Sorry to post internal links that many of you can’t get to. But if you do manage to get to them, report the bug you exploited so I can include it in my future talks!

the touring test

2009-02-12T21:55:00Z

(I couldn’t resist the play on Alan Turing’s famous test when naming this testing metaphor.)

When I think of software, I naturally think of it in testing terms. In my mind software is made up of components which are defined by structural boundaries (code files/assemblies) where multiple components communicate through some defined interface, a set of features that define the functionality for each component, and capabilities that produce behavior that the user desires. It’s extremely useful to me to think about it this way because I can now more purposefully explore my application.

For example, I can choose a set of capabilities to test as an ensemble or focus on a single one. I can purposefully explore the capabilities of many features or a single feature. I can exercise capabilities that cross component boundaries or choose to stay within a component. I can use my component/feature/capability map to guide my testing and ensure that I cover the interesting combinations to the extent possible.

I call these tests tours and think in terms of a tourism metaphor when I test software. Ideas from tourism fit well into testing. If you think about a big, complicated tourist destination like New York or London you realize that wandering aimlessly around its streets is a pretty lousy way to tour. If you understand a little bit about its districts and destinations (components and features) you have a better feel for how to get around. If you go further and use published guidebooks, consult locals and read tourist maps you’ll discover the best places to go and things to do (capabilities) a lot faster. Travelers want to get the most out of their brief vacation time and as testers we can learn from them.

At Microsoft a group of us test folk from around the division and around the company are experimenting with tour-guided testing. We have the Supermodel Tour in which all we do is window shop … look at the interface and find superficial (no intended disrespect to any actual supermodel who might be reading this blog) bugs. We have the Money Tour where we watch salespeople demo the product and create variations of those demos as test cases. We have the Intellectual Tour where we ask the software hard questions. We have the Garbage Collector’s Tour, the Competitor’s Tour, the Museum Tour, the After Hours Tour and a host more that we are experimenting with. All of these tours represent specific, focused testing advice that guides the tester through paths of the software. The tours help a tester think through inputs, data and environment settings that satisfy the goals of the tour. It's a more methodical and purposeful way to test.

We created the tours as exploratory test guidance and even more as a way of partitioning test concerns. Each tour is guidance about a very specific type of capability. Add all the tours together and they collectively should encompass most all interesting scenarios, but there is more to them that that. They can also be associated with the component/feature/capability map to help testers associate a tour with a specific capability. Want to test the interface between a database component and the application? Run the FedEx Tour. Want to test legacy code? Run the Museum Tour and the Prior Version Tour. Want to test how two features interact? Run the Guidebook Tour between them. The touring tests take the discussion of how to test up to a higher level. Instead of talking about specific test cases, they guide us toward talking about test concepts because each tour embodies testing advice for a very specific capability. Such higher level concepts have been missing from our testing vocabulary for far too long.

Tours also take the discussion of bugs to a higher level. Want to find UI bugs? Run the Supermodel Tour. Want to find stress bugs? Run the Saboteur (get it?) and the TOGOF Tour. Want to find timing bugs? Run the Rained Out Tour.

Want to learn more about the details of these tours? Stay tuned to this site, I am going to start running a tour-of-the-month post to explain these concepts in more detail. Perhaps your own team will join us and pass the touring test!

of moles and tainted peanuts

2009-02-06T22:47:00Z

There was a full page ad for Jif peanut butter in my morning paper that caught my attention. (For those non-US readers, our nation is experiencing a salmonella bacteria outbreak which has been traced back to contaminated peanuts.) The ad touted Jif’s rigorous testing processes and reassured readers that testing for salmonella was a long time habit for the Jif company and we should feel confident in consuming their products.

Now clearly peanut butter is not software. I very much doubt that the processes for making peanut butter have changed much over the past few decades. I also imagine that one batch of peanut butter is about the same as the last batch. I concede that we have a harder problem.

But, the term ‘long time habit’ really caught me. Because I haven’t seen too many long term habits established in the testing industry. We plan tests, write test cases, find bugs, report bugs, use tools, run diagnostics and then we get a new build and start the process all over again. But how much do we learn in the process? How much do we retain from one build to the next? Are we getting better each time we test? Are we getting better purposefully or are we just getting more experienced? In many ways, the only real repository of historical wisdom (those long term habits of Jif) is embodied in our tools.

My friend Alan Page likens testing to playing whack-a-mole. You know the one: chuck a quarter in and plastic moles pop up through a random sequence of holes and you whack them on the head with a mallet. Whack one, another appears and even previously-whacked moles can pop up again requiring additional mallet treatment. It’s a never ending process, just add quarters.

Sound familiar? Testing is whack-a-mole with developers applying the quarters liberally. Now, defect prevention notwithstanding, we can take a lesson from Jif. They understand that certain risks are endemic to their business and they’ve designed standard procedures for mitigating those risks. They’ve learned how to detect salmonella and they have wired those tests into their process.

Have we paid enough attention to our risks that we can codify such routine test procedures and require their consistent application?

Clearly software is not peanut butter. Every piece of software is different; Office’s salmonella is likely irrelevant to Windows and vice versa. But that is no excuse to play whack-a-mole with bugs. We have to get better. Perhaps we can’t codify a salmonella testing procedure into a cookbook recipe but we can start being more proactive with the whole business of learning from our mistakes.

I propose that testers take a break from finding bugs and just take some time to generalize. When the bug pops its head from some coded hole, resist the temptation to whack it. Instead, study it. How did you find it? What was it that made you investigate that particular part of the app? How did you notice the hole and what was happening in the app that caused the bug to pop out? Is the test case that found the bug generalizable to find more bugs similar to the one you are ready to whack? Is there some advice you could pass along to other testers that would help them identify such holes?

In other words, spend part of your time testing the current product you are trying to ship. Spend the rest of the time making sure you learn to test the next product better. There is a way to do this, a metaphor we use here at Microsoft to help.

I’ll discuss the metaphor and how we apply it to form long time habits, peanut butter style, in my next post.

getting away from it all

2009-02-01T22:39:00Z

When you’re on vacation do you think about work? Not thoughts of dread, worry or angst but reflection, planning and problem solving. I just did. Last Sunday I awoke in Seattle to freezing temps and a dusting of snow. By midday I was building a sandcastle on Ka’anapali Beach, Maui in 79 degree sunshine. If that’s not getting away from it all, I don’t know what is.

Yet my mind wasn’t really away. In fact, I thought about work all the time. Given that software was everywhere I looked, it’s not hard to see why. My entire trip was booked online, even the taxi to the airport. Not a single person besides myself took part in the process. Just me … and a load of software.

The taxi cab itself contained software, as did the airplane. The baggage carousel, the espresso machine, the car rental counter (no person there, just a self serve terminal) and even the surveillance camera that watched my son juggle his soccer ball while I packed our bags in the trunk. All alone, except for the software. Even the frozen concoction machine had software that helped it maintain the right temperature. (It broke, incidentally, making me thankful that I am a beer drinker.)

Is it possible for anyone in this field to really get away from it all? (Don’t get me started on the motion sensors that control the air conditioning in the hotel room. I’m all for turning them off when they are not in use, but apparently sitting still and being cool was not one of their end-to-end scenarios.)

The truth of the matter is that getting away from it all just isn’t necessary for me. I like seeing software in action and I enjoy brooding over problems of testing it. Vacations free my mind from the daily grind and leave my mind to question things that back home I might overlook. Does this make me work obsessed or just indicate that I really like what I do?

Vacations have always been like this for me. When I was a professor, two students who led my research lab, Ibrahim El-Far and Scott Chase, actually avoided me when I returned from a trip, afraid of the work my new insights would bring. They never quite managed to successfully do so.

Which brings me back to the motion sensor in my room. The problem isn’t so much a poor tester, rather poor testing guidance. The sensor does exactly what it is designed to do and testing it based on those requirements got me in the sit-and-sweat loop. The problem is that no one thought to give it a field try … what I call ‘day in the life’ testing. Had the tester thought to take the sensor through a 24 hour cycle of usage they would have identified that problematic ten hour period (yes, ten, it’s a vacation after all) when motion is low and the desire to be cool is high. But what tool gives such guidance? Modern tools help testers in many ways, but helping them think of good test scenarios isn’t one of them. They help us organize, automate, regress and so forth, but do they really help us to test?

That’s the tool I want. Tomorrow, when I return, I am going to direct someone to build it for me. Ibrahim and Scott, you are off the hook this time.

more about test case reuse

2009-01-22T22:33:00Z

We mostly write test cases that are specifically tied to a single application. This shouldn’t come as any big surprise given that we’ve never expected test cases to have any value outside our immediate team. But if we want to complete the picture of reusable test cases that I painted in my last post we need to write test cases that can be applied to any number of different apps.

Instead of writing a test case for an application, we could move down a level and write them for features instead. There are any number of web applications, for example, that implement a shopping cart so test cases written for such a feature should be applicable to all such apps. The same can be said of many common features like connecting to a network, making SQL queries to a database, username and password authentication, and so forth. Feature-level test cases are far more reusable and transferable than application-specific test cases.

The more focused we make the scope of the test cases we write, the more general they become. Features are more focused than applications, functions and objects are more focused than features, controls and data types are more focused than functions and so forth. At a low enough level, we have what I like to call “atomic” test cases. A test atom is a test case that exists at the lowest possible level of abstraction. Perhaps you’d write a set of test cases that simply submits alphanumeric input into a text box control. It does one thing only and doesn’t try to be anything more. You may then replicate this test atom and modify it for different purposes. For example, if the alphanumeric string in question is intended to be a username, then a new test atom that encoded the structure of valid usernames would be refined from an existing atom. Over time thousands (and hopefully orders of magnitude more) of such test atoms would be collected.

Test atoms can be combined into test molecules. Two alphanumeric string atoms might be combined into a test molecule that tests a username and password dialog box. I can see cases where many independent test authors would build such molecules and then over time the best such molecule would win out and yet the alternatives would still be available. With the proper incentives, test case authors would build any number of molecules that could then be leased or purchased for reuse by application vendors that implement similar functionality.

At some point, enough test atoms and molecules would exist that the need to write new, custom tests would be minimal. I think that Wikipedia, a site with user supplied, policed and maintained content, would be what the industry would need to store all these tests. Perhaps such a community Testipedia can be constructed or companies can build their own internal Testipedias for sensitive applications. But a library of environment-carrying (see my last post) test atoms and molecules would have incredible value.

A valuable extension of this idea is to write atoms and molecules in such a way that they will understand whether or not they apply to an application. Imagine highlighting and then dragging a series of ten thousands tests onto an application and having the tests themselves figure out whether they apply to the application and then running themselves over and over within different environments and configurations.

Ah, but now I am just dreaming.

test case reuse (in the future)

2009-01-16T22:47:00Z

I’ve given my ‘future of testing’ talk four times (!) this week and by far the part that generates the most questions is when I prophesize about test case reuse. Given that I answered it differently all four times (sigh), I want to use this space to clarify my own thinking and to add some specifics.

Here’s the scenario: one tester writes a set of test cases and automates them so that she can run them over and over again. They are good test cases so you decide to run them as well. However, when you do run them, you find they won’t work on your machine. Your tester friend used automation APIs that you don’t have installed on your computer and scripting libraries that you don’t have either. The problem with porting test cases across machine boundaries is that they are too specific to their environment.

In the future we will solve this problem with a concept I call environment-carrying tests (nod to Brent Jensen). Test cases of the future will be written in such a way that they will encapsulate their environment needs within the test case using virtualization. Test cases will be written within virtual capsules that embed all the necessary environmental dependencies so that the test case can run on whatever machine you need it to run on.

The scope of technological advances we need for this to happen are fairly modest. However, the Achilles heel of reuse has never been technological so much as economic. The real work required to reuse software artifacts has always been on the consumer of the reused artifact and not on its producer. What we need is an incentive for testers to write reusable test cases. So, what if we create a “Testipedia” that stored test cases and paid the contributing tester, or their organization, for contributions? What is a test case worth? A dollar? Ten dollars? More? Clearly they have value and a database full of them would have enough value that a business could be created to host the database and resell or lease test cases on an as-needed basis. The more worthy a test case, the higher its value and testers would be incentivized to contribute.

Reusable test cases will have enough intrinsic value that a market for test case converters would likely emerge so that entire libraries of tests could be provided as a service or licensed as a product.

But this is only part of the solution. Having test cases that can be run in any environment is helpful, but we still need test cases that apply to the application we want to test. As it turns out, I have an opinion on this and I’ll blog about it next.

explaining exploratory testing

2009-01-09T01:59:00Z

I just got finished talking (actually the conversation was more like a debate) to a colleague, exploratory testing critic and a charter member of the plan-first-or-don’t-bother-testing-at-all society.

I am happy to say, he conceded the usefulness (he would not grant superiority, if he had I would be on my way to the pub right now with his credit card) of exploratory testing. Perhaps I have finally found a way to explain the utility of exploration. Here’s what I said:

“Software testing is complicated by an overload of variation possibilities from inputs and code paths to state, stored data and the operational environment. Indeed, whether one chooses to address this variation in advance of any testing by writing test plans or by an exploratory approach that allows planning and testing to be interleaved, it is an impossible task. No matter how you ultimately do testing, it’s simply too complex to do it completely.

However, exploratory techniques have the key advantage that they encourage a tester to plan as they test and to use information gathered during testing to affect the actual way testing is performed. This is a key advantage over plan-first methods. Imagine trying to predict the winner of the Super Bowl or Premier League before the season begins … this is difficult to do before you see how the teams are playing, how they are handling the competition and whether key players can avoid injury. The information that comes in as the season unfolds holds the key to predicting the outcome with any amount of accuracy. The same is true of software testing and exploratory testing embraces this by attempting to plan, test and re-plan in small ongoing increments guided by full knowledge of all past and current information about how the software is performing and the clues it yields in the testing results.

Testing is complex, but effective use of exploratory techniques can help tame that complexity and contribute to the production of high quality software.”

the Zune issue

2009-01-07T01:01:00Z

As you can imagine there is a pretty lively debate going on over the Zune date math issue here in the hallways and on our internal mailing lists. There are plenty of places one can find analyses of the bug itself, like here, but I am more interested in the testing implications.

One take: this is a small bug, a simple comparator that was ‘greater than’ but should have been ‘greater than or equal to.’ It is a classic off-by-one bug, easily found by code review and easily fixed then forgotten. Moreover, it wasn’t a very important bug because its lifespan was only one day every leap year and it only affected the oldest of our product line. In fact, it wasn’t even our bug; it was in reused code. Testing for such proverbial needles is an endless proposition, blame it on the devs and ask them not to do it again. (Don’t get your knickers in a twist, surely you can detect the sarcasm.)

Another take: this is a big bug, in the startup script for the device and thereby affected every user. Moreover, its effect is nothing short of bricking the device, even if only for a day (as it turns out, music is actually a big deal on that specific day). This is a pri-1, sev-1, run-down-the-halls-screaming-about-it kind of bug.

As a tester can I take any view but the latter? But the bug happened. Now we need to ask what can we learn from this bug?

Clearly, the code review that occurred on this particular snippet is suspect. Every code review I have ever been part of, a check on every single loop termination condition is a top priority, particularly on code that runs at startup. This is important because loop termination bugs are not easily found in testing. They require a “coming together” of inputs, state and environment conditions that are not likely to be pulled out of a hat by a tester or cobbled together using unthinking automation.

This brings me to my first point. We testers don’t do a good job of checking on the quality of code reviews and unit testing where this bug could have been more easily found. If I was still a professor I would give someone a PhD for figuring out how to normalize code review results, unit test cases and system test cases (manual and automated). If we could aggregate these results we could actually focus system testing away from the parts of the system already covered by upstream ‘testing.’ Testers would, for once, be taking credit for work done by devs, as long as we can trust it.

The reason that system testing has so much trouble dealing with this bug is that the tester would have to recognize that the clock was an input (seems obvious to many, but I don’t think it is a given), devise a way to modify the clock (manually or as part of their automation) and then create the conditions of the last day of a year that contained 366 days. I don’t think that’s a natural scenario to gravitate toward even if you are specifically testing date math. I can imagine a tester thinking about February 29, March 1 and the old and new daylight savings days in both Fall and Spring. But what would make you think to distinguish Dec 31, 2008 as any different from Dec 31, 2007? Y2K seems an obvious year to choose and so would 2017, 2035, 2999 and a bunch of others, but 2008?

This brings me to my second point. During the discussions about this bug on various internal forums no less than a dozen people had ideas about testing for date related problems that no one else involved in the discussions had thought of. I was struck by a hallway debate between two colleagues who were discussing how they would have found the bug and what other test cases needed to be run for date math issues. Two wicked smart testers that clearly understood the problem date math posed but had almost orthogonal approaches to testing it!

The problem with arcane testing knowledge (security, y2k, localization all come to mind) is that we share our knowledge by discussing it and explaining to a tester how to do something. “You need to test leap year boundaries” is not an ineffective way of communicating. But it is exactly how we are communicating. What we should be doing is sharing our knowledge by passing test libraries back and forth. I wish the conversation had been: “you need to test leap year boundaries and here’s my library of test cases that do it.” Or, “counting days is a dangerous way to implement date math, when you find your devs using that technique, run these specific test cases to ensure they did it right.”

The testing knowledge it took to completely cover the domain of this specific date math issue was larger than the set of folks discussing it. The discussion, while educational and stimulating, isn’t particularly transportable to the test lab. Test cases (or models/abstractions thereof) are transportable and they are a better way to encapsulate testing knowledge. If we communicated in terms of test cases, we could actually accumulate knowledge and spread it to all corners of the company (we have a lot of apps and devices that do date math) much faster than sitting around explaining the vagaries of counting time. Someone who doesn’t understand the algorithms to count time could still test those algortihms using the test assets of someone else who did understand it.

Test cases, reusable and reloadable, are the basis for accumulated knowledge in software testing. Testing knowledge is simply far too distributed across various experts’ heads for any other sharing mechanism to work.

new year's resolutions

2009-01-05T22:53:00Z

Welcome to the new year!

2009 will be the year I publish a new book on testing and the year I ship my first testing tool since Holodeck oh so many years ago.

I’m thinking about calling the book exploratory testing. Believe it or not the title isn’t taken. Any thoughts on that title? Am I too cheeky by claiming it? By happy coincidence the material I am preparing is on manual testing of a strikingly exploratory nature.

I don’t get to name the tools though as they will be shipping as part of Visual Studio 2010 and are currently code named after various islands in the Puget Sound. My boss (yes, there is some poor unlucky soul here at the empire who is forced to claim me as a direct) is in the process of blogging about those tools our team is building. Allow me to introduce Amit Chatterjee at his corner of MSDN. Please send along your condolences for him having to be the one to write my review.

Also, based on reader feedback, it will also be the year that I expose more insider details about testing culture and practice at Microsoft. Those are the most read of my blog posts and I’m going to take that as a hint. Although, much of the thunder has already been aired by my colleagues in their new book How We Test Software at Microsoft. That book's a good read and a high bar for me to match when my own book comes out in a few months.

google v. microsoft, and the dev:test ratio debate

2008-12-09T22:51:00Z

Every since I gave a talk at Google’s GTAC event here in Seattle this past October, I’ve had the chance to interact with a number of Google testers comparing and contrasting our two companies’ approach to testing. It’s been a good exchange.

Now it seems that, their toilets notwithstanding, Google focuses on testing with an intensity that is in the same general ballpark as ours. We both take the discipline and the people who do it seriously. But I think that there are some insights into the differences that are worth pondering.

Specifically, the disparity between our respective developer-to-tester ratios is worth a deeper look. At Microsoft the dev:test ratio varies somewhat from near 1:1 in some groups to double or triple that in others. At Google just the opposite seems to be the case with a single tester responsible for a larger number of bug-writing devs (clearly we have that in common).

So which is better? You tell me, but here are my thoughts (without admission of any guilt on Microsoft’s part or accusations against Google):

1. 1:1 is good. It shows the importance we place on the test profession and frees developers to think about development tasks and getting the in-the-small programming right. It maximizes the number of people on a project actively thinking about quality. It speeds feature development because much of the last minute perfecting of a program can be done by testers. And it emphasizes tester independence, minimizing the bias that keeps developers from effectively testing their own code.

2. 1:1 is bad. It’s an excuse for developers to drop all thoughts of quality because that is someone else’s job. Devs can just build the mainline functionality and leave the error checking and boring parts to the testers.

It’s interesting to note that Microsoft testers tend to be very savvy developers and are often just as capable of fixing bugs as they are of finding bugs. But when they do so, do devs really learn from their mistakes when they have someone else cleaning up after them? Are testers, when talented and plentiful, an excuse for devs to be lazy? That’s the other side of this debate:

3. Many:1 is good. When testers are scarce, it forces developers to take a more active role in quality and increases the testability and initial quality of the code they write. We can have fewer testers because we our need is less.

4. Many:1 is bad. It stretches testers too thin. Developers are creators by nature and you need a certain number of people to take the negative viewpoint or you’re going to miss things. Testing is simply too complicated for such a small number of testers. Developers approach testing with the wrong, creationist attitude and are doomed to be ineffective.

So where’s the sweet spot? Clearly there are application specific influences in that big server apps require more specialized and numerous testers. But is there some general way to get the mix of testers, developers, unit testing, automated testing and manual testing right? I think it is important that we start paying attention to how much work there really is in quality assurance and what roles are most impactful and where. Test managers should be trying to find that sweet spot.