My previous post stirred up a bit of controversy and spurred a bit of discussion. Fun stuff! The test plan for my new team states:
All functional, integration, performance, and stress tests are automated and none run manually in test passes. This will allow us to evaluate our product state quickly and spend more time focused on dogfooding, directed exploratory testing, and working with customers. This will set us up to provide better post-release support.
I think a key phrase is missing: which are worth automating. As in "All tests…which are worth automating are automated." I hope this is implicit to my new teammates' thinking. I will be making it explicit. <g/>
Determining which tests to automate is one of the Hard Problems in the testing world. I don't have any absolute answers or rules to follow. I have however developed some heuristics - fallible methods for solving a problem - which have proved useful. Possibly you will find them useful too. The relevance they have for you will to some degree track how closely your context matches with mine: shrink-wrapped software which is localized into many different languages and supported by sustained engineering for five to ten years.
Oftentimes several of these guidelines come into play. Say you have a test which is somewhat difficult to automate, would be tedious to execute manually, where you can afford some manual munging of the results in order to find missed failures and bypass false positives. Should you automate it? It depends. <g/>
A few examples from my personal experience:
Should you automate your tests? Probably. Should you test manually? Almost certainly. How do you choose when to do what? It depends!
*** My new team is hiring! Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great coding skills required.