Treating test automation as a product changes the way we design and implement test code and helps to ensure that the tests provide cost-effective feedback for their entire lifespan.  The time a test spends being used for the verification of new functionality is relatively short, but the same tests must continue to run for many years as a guard against regressions during the creation of service packs for a released version of the software or during development of newer versions.


While it is essential to not lose sight of the importance of the software that is being tested, considering “test automation as a product” provides some interesting additional perspective on what constitutes good test automation.  Some of the implications are:

  • The product should be easy to use.  For tests, that implies they should have ‘push button’ execution in a variety of ways.  Results should be reported consistently, should be easy to access and understand.
  • The product should be reliable.  For tests, the results should not include intermittent failures.  False failures should be minimized and false positives should never happen.
  • The product should perform well.  Tests provide feedback on the state of the software that is being developed.  The speed with which this feedback is provided is important.
  • The product should be easy to maintain.  In the case of test hand-off and re-use, source code is provided and needs to be maintained and extended.  This is a huge requirement for the test automation product.  To be easily maintained, the code must be well designed, use consistent underlying test frameworks, and be implemented with quality engineering standards.
  • The product should be developed using state of the art software engineering practices.  Tests should go through design and code reviews and have supporting documentation.  Test frameworks should go through design and code reviews, include unit tests, be subject to rigorous pre-check-in verification, and must include a set of user documentation. 

Attention to crafting the test automation product involves more than just sound engineering.  There should be a conscious effort to limit the portfolio of frameworks and libraries used in test development.  In many cases, ownership of tests is transferred to other teams for long-term support of the released software.  The down-stream teams are often smaller than the development teams and cannot afford the extra effort that is required to understand and maintain tests running on a diverse set of test tools.  When confronted with the choice of creating a new test framework or test library, think hard about the users of the automation product you are creating.  Do they benefit from the new tools or is it just creating an additional barrier to the uptake and long-term maintenance of these tests?


Recommended Reading

Page, A., Johnston, K. and Bj Rollison.  2009.  How we test software at Microsoft.  Microsoft Press, Redmond, WA.