Yesterday I wrote about the need to reduce the number of things a project attempted to do in order to deliver a great product. Too many seemingly good ideas can make a product late or fragmented or both. The same is true of testing a product. Great testing is more about deciding what not to test than deciding what to test.
There is never enough time to test everything about a product. This isn’t just the fault of marketing which has a go-to-market date in mind. It is a physical reality. To thoroughly test a product requires traversing the entire state tree in each possible combination. This is analogous to the traveling salesman problem and is thus NP-Complete. In layman’s terms, this means that there is not enough time to test everything for any non-trivial program.
When someone first starts testing, thinking up test cases is hard. We often ask potential hires to test something like a telephone or a pop machine. We are looking for creativity. Can they think up a lot of interesting test cases? After some time in the field, however, most people are able to think up a lot more tests than they have time to carry out. The question then becomes no longer one of inclusion, but one of exclusion.
In Netflix’s case the exclusion was for focus. This is not the right exclusion criteria for testing. It is improper to not test the UI so that you can test the backing database. Instead, the criteria by which tests should be excluded is more complex. There is no single criteria or set of criteria that work for every project. Here are some to consider which have wide applicability:
There are many more criteria that could be used. The important point is to have criteria and to make intentional decisions. A test planning approach that merely says, “What are the ways we can test this product?” is insufficient. It will generate too many test cases, some of which will never be carried out due to time or cost. It is important to prune the decision tree up front so that the most important cases are done and the least important ones are left behind. Do this up front, in the test spec, not on the fly as resources dwindle.