A portion of the material from How We Test Software at Microsoft comes from some of the classes we teach to testers at Microsoft. Bj’s chapters on functional and structural testing, for example, cover much of what we teach to new testers at Microsoft. I recently taught a class to a group of experienced testers at Microsoft and had a chance to reflect on some of the material that exists in both the book and this particular course.

Testability and patterns of software test are two topics a recent group of senior testers and I discussed (both topics are discussed briefly in chapter 4 of HWTSAM). When I first started talking with testers about testability a few years ago, there wasn’t a whole lot happening with the concept around the company. Today, just about every team understands the importance of the concept and we have a huge amount of material available internally for guidance and education. There’s still more to do, but it’s nice to see the growth and maturation.

The use of patterns for software testing is also gaining some traction. Design patterns, of course, have been popular among software developers for years, and most developers recognize the value of communicating common solutions to software design problems. One of the things that has always bugged me about software testing is that we don’t have a common language to communicate testing ideas. Testers throughout the industry often refer to the same concept by different names. Worse, we don’t have a common way to teach new testers about testing. Most of the time we sit testers in front of an application and say, “here test this”. Describing testing as patterns is one way to communicate common testing approaches to new testers. In the book – and at Microsoft, we use a variation on Robert Binder’s patterns described in Testing Object Oriented Systems. The simplified pattern includes eight elements:

• Name Give a memorable name—something you can refer to in conversation
• Problem Give a one-sentence description of the problem that this pattern solves
• Analysis Describe the problem area (or provide a one-paragraph description of the problem area) and answer the question of how this technique is better than simply poking around
• Design Explain how this pattern is executed (how does the pattern turn from design into test cases?)
• Oracle Explain the expected results (can be included in the Design section)
• Examples List examples of how this pattern finds bugs
• Pitfalls or Limitations Explain under what circumstances and in which situations this pattern should be avoided
• Related Patterns List any related patterns (if known)

In the book, we use the example of boundary-value analysis – or more simply stated, the technique of testing around the edges of valid input values (e.g. testing for “off by one” errors). Another example of a pattern could be the “All Pairs” test pattern (another technique discussed in-depth in hwtsam) . Using the template above, that pattern may be described like this:

• Name All Pairs or Pairwise testing
• Problem Some testing requires a large amount of combinations generated by multiple variables with multiple values. It is often infeasible to exercise every single combination and a more efficient methodology is necessary.
• Analysis Combinations are created in such a way that all pairs of values are combined with each other. This yields highly efficient, but not exhaustive, test cases
• Design
  • List all variables in the problem set
  • For each variable, list all possible values (or choices)
  • Use a tool such as PICT (or orthogonal array) to get a list of tests to perform
  • Supplement and modify your list of tests to:
    • Change weights (certain variables should show up more often)
    • Add additional tests that are high probability of failure or execution (or both)
• Oracle Expected results are then added by hand to each individual test above. Most often, all tests are supposed to simply pass (as in a browser/OS matrix).
• Examples See the PICT tool (normally, we’d include more here)
• Pitfalls or Limitations All Pairs is not exhaustive testing, and therefore care must be taken to add additional coverage in more important areas.
• Related Patterns Decision Table Test Pattern

I find test patterns to be a great way to communicate and educate testers about design techniques, heuristics and approaches to solving software testing problems. Students in our classes and I have created dozens of patterns using the template above. Some are merely “good”, but others are outright fantastic. Once we get them all polished up, I’ll make sure we share them somewhere (perhaps even in another MSPress book) so that other testers can learn test design through patterns.