This afternoon was a half-day tutorial on exploratory testing. Elisabeth Hendrickson gave an engaging talk - complete with in-class exploratory testing - that presented four techniques for doing successful exploratory testing:

  • variables (anything whose value can be changed, e.g., window size), values (a value given to a variable), and heuristics (rules of thumb that lead you to good tests for particular types of values, e.g., "It you can count it, try zero, one, and many" or "If it has an order, try first, middle, last"): List everything you can change and then use hueristics to develop sets of values to try for each variable.
  • state models: As you identify possible states think about how the actions you can and can't take change and how the results of those actions change. As you identify events that transfer you between states, think about various triggers for each of those events. Now use that state model to apply every controllable event to every state (regardless of whether it seems interesting) and force exits from every state.
  • nouns and verbs: Describe feature interactions as nouns (what kinds of things can you create or manage) and verbs (what can different types of users do); use adjectives and adverbs to further describe the nouns and verbs (i.e., the values you could give the variables you identified). Often the grammar you develop will be rich enough to enable you to write stories about your software - stories that you can easily translate to test cases. Writing these stories is simple: randomly pick a series of nouns and verbs and then turn these words into a short action sequence. (Yes, this sounds a whole lot like model-based testing - one more way to develop those models.)
  • personae and soap operas: Identify several types of users with distinct needs and personalities. Inhabit each persona and use your app the way that person would - i.e., find the same bugs they will. <g/> A very helpful technique for doing this is to write soap opera scripts about your user and their travails and triumphs with your software.

Throughout Elisabeth made sure we understood very well that exploratory testing is *not* just pounding on the keyboard but rather is a means of discovering new information (aka surprises - which are not necessary bugs, just something you didn't realize about your product) about the software under test by methodically exploring its capabilities and limitations.

Lectures are all well and good, and some people can actually learn that way, but I need to apply what I hear to actually internalize it. Elisabeth handed out a bunch of Pixter toys - a roughly PDA-sized electronic (black and white) coloring book. Once we figured out how to turn the speaker off these turned out to be surprisingly (ha!) capable little buggers that served as excellent foils for applying the techniques we were learning.

Perhaps the most important piece of information was that exploratory testing is best done in packs. Each group found that the other groups had identified variables they hadn't. The same holds true for your app: one person doing exploratory testing is good, two people doing exploratory testing is better, the whole Dev+Test+Everybody Else team doing exploratory testing is better yet, and the whole team doing exploratory testing in pairs is best. Case in point: one team found a diagnostic mode -- something no one else in the other twenty-some classes Elisabeth has used Pixter in had ever done!

*** Comments, questions, feedback? Want a fun job on a great team? I need a tester! Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. Great coding skills required.