testing ledger

Automation Accuracy

Tim Lamey — Tue, 24 Aug 2010 18:05:00 GMT

To help drive test quality improvements, Dynamics CRM, Dynamics AX, and Dynamics NAV are using a metric that indicates the percentage of all tests that are passing or which fail due to a product issue. This metric is called Automation Accuracy (although Dynamics NAV has used the term Test Reliability, at least to this point). Automation Accuracy has several desirable characteristics, but it is important to understand the limitations of this metric to avoid misuse or misinterpretation.

In software testing we want tests that are likely to detect product defects when present (high Sensitivity) and which do not result in a large number of false failures (high Specificity; Lamey 2009). It would be nice to be able to measure Sensitivity and Specificity and drive towards high values for each, but we are unable to assign values to these metrics because they suffer from the difficulty of separating True Pass (no issues) from False Pass (undetected product issues).

It is easier to assign a value to Automation Accuracy because it does not differentiate between tests that are passing when there is no product issue and those that do not detect a product issue and pass anyway. As a result, Automation Accuracy is not directly related to either Sensitivity or Specificity. For example, the following diagram depicts the relationship between Automation Accuracy, Sensitivity, Specificity, and Positive Predictive Value when only the percentage of True Pass and False Pass are allowed to vary. Tests with the highest quality should appear in the upper right corner of the diagram (where Sensitivity and Specificity are highest).

Notice that while Automation Accuracy and Positive Predictive Value do not vary, Sensitivity and Specificity vary based on the percentage of True Pass results. If high Sensitivity and Specificity are desirable test quality attributes, Automation Accuracy does not help much in the pursuit of these attributes.

Automation Accuracy can be used as a rough measure of test quality if we assume that the rate of True Passes (pass result correctly indicates there are no product issues) is relatively high. Rigorous test design techniques might help to keep the rate of True Passes high, but we can only verify our assumption by looking at the number of product issues that are discovered in the field and determining if tests should have detected the issue.

Automation Accuracy is a bit like Code Coverage: high values do not indicate high-quality tests, but low values are a sign of a serious problem. Despite its limitations, Automation Accuracy is a useful measure as it exposes poor test quality and because it is a driver of several desirable behaviors, notably

Prompt and complete analysis of failure causes (unanalyzed failures lead to low Automation Accuracy)
Focused attention on test failures that occur due to non-product causes

Making Code Coverage Useful

Tim Lamey — Tue, 02 Mar 2010 11:56:00 GMT

[This is an updated version of a note that was first published to a Microsoft internal blog in November 2008]

There are many opportunities to misuse code coverage (see Marick 1999 or Lamey 2010). Focusing solely on code coverage numbers can lead to some very bad behaviors, but it does not follow that code coverage does not have the potential to add value.

Here is a short list of ways to make code coverage a useful part of the software development process. Additional suggestions for how to get the most from code coverage can be found in Rollison (2009 and 2010).

Identification of Testing Gaps

Code coverage measures whether a test (or suite of tests) exercises the application code. It does not tell us, at least not directly, how well the application code is tested, only that the code is touched.

Code coverage can be used as a tool to help build a better suite of tests by identifying the gaps in the existing testing. Because 0% code coverage is an indication that the test(s) fail to touch a unit of code, 0% code coverage can be used as a quick, coarse-grained way to “triage” areas that need additional testing effort. The metric can be applied sequentially at multiple levels to identify assemblies, files, classes, or even methods that require additional testing focus.

A refinement of this methodology is to treat any code coverage value less than 100% as an indication of some weakness in the design of the test suite (Marick 1999). The idea goes something like this. For non-zero code coverage values, spend time analyzing the areas that are not covered. Consider what categories of tests are missing or what modifications to existing tests could be included to improve the original test design. Modify the test design, create the tests, and measure code coverage again. Iterate between measuring code coverage and modifying the test design until the desired level of code coverage is achieved. The important point is that the focus should be on improving the test design and not on improving code coverage (Marick 1999). Good test design is correlated with high code coverage values, but high code coverage values do not necessarily imply you have a good set of tests (Lamey 2010).

Risk Mitigation

Assuming the tests do a good job of testing the code they cover, code coverage can help point out areas in the code base where there may still be substantial risk of a fault. Code coverage can be a valuable tool when attempting to decide if a product is ready for release, provided there has been some analysis of the areas that were not covered during testing. As long as the code coverage information is used to understand exactly what code is not covered by testing, then the actual code coverage value is of little significance. Regardless of whether the coverage value is 58% or 92% it is important to identify the portions of the application that are not exercised and to come to some agreement that the lack of coverage in these areas is an acceptable risk.

Dead Code Removal

Analysis of code coverage values that are less that 100% can unearth cases where the application code is unreachable (i.e. dead code). Once the dead code is identified, it can be removed to improve the hygiene of the code base. This is another case where the benefit comes from striving to fully understand the reasons for incomplete code coverage and not from the actual code coverage number.

Testability

Product code that is difficult to reach with tests may represent a testability problem. Systems that are testable are likely to the associated with high code coverage values, so code coverage can provide an indirect indicator of the level of testability in the product. Again, it is the analysis of the code that is not covered that provides the insight into the testability or lack of testability of the application, not the actual code coverage number.

Test Case Selection

Code coverage information can be cross referenced with code changes to the application to select the test cases that, because they exercise the affected code, are the best choice to run to detect regressions. Use of code coverage to select a set of tests maximizes the chances of detecting an error as the result of a particular product code change while reducing the need to run all possible tests.

Test Case Prioritization

In some cases, execution of a subset of tests may not be an option (e.g. perhaps because of legal or contractual requirements), but it may still be of value to know which tests are most likely to detect problems so they could be executed first. Code coverage information can used as a heuristic for prioritizing the order in which tests are run to improve the rate of fault detection early in a regression run (Elbaum et al. 2002). These prioritization techniques improve the odds that regression testing will provide valuable early feedback even in cases where for other reason, it is still necessary to run all tests.

In summary, code coverage numbers in and of themselves are of little value. With the exception of using code coverage in test case selection or test case prioritization, code coverage information only adds value to the software development process when the organization is committed to analyzing the implications of code coverage values that are less than 100%. Failure to reach 100% coverage may be indicative of one or more of the following: poor test design, limited testability support, the presence of dead code, and exposure to undiscovered faults. Understanding which issue underlies the observed code coverage number is only possible after spending time analyzing the portions of the product code that are not covered.

Recommended Reading

Elbaum, S., Malishevsky, A., and G. Rothermel. 2002. Test Case Prioritization: A Family of Empirical Studies. IEEE Transactions on Software Engineering, 28-2:159-182. http://www.cse.unl.edu/~elbaum/papers/journals/tse01.pdf

Kaner, C. 1996. Software negligence and testing coverage. http://www.kaner.com/coverage.htm

Lamey, T.C. 2010. Don’t be seduced by numbers. Testing Ledger blog post: http://blogs.msdn.com/tim_lamey/archive/2010/02/08/don-t-be-seduced-by-numbers.aspx

Marick, B. 1999. How to misuse code coverage. International Conference and Exposition on Testing Computer Software. http://www.exampler.com/testing-com/writings/coverage.pdf

Rollison, Bj. 2009. Reconsidering code coverage. I.M. Testy blog post: http://www.testingmentor.com/imtesty/2009/11/25/reconsidering-code-coverage/

Rollison, Bj. 2010. Code coverage: more than just a number. I.M. Testy blog post: http://www.testingmentor.com/imtesty/2010/01/21/code-coverage-more-than-just-a-number/

Don’t Be Seduced By Numbers

Tim Lamey — Mon, 08 Feb 2010 08:24:00 GMT

[This note was first published to a Microsoft internal blog in October 2008]

It is common to include code coverage goals as key criteria for deciding if is time to exit various milestones in the software development cycle. This suggests that code coverage numbers must be providing value; why else would so many teams use them? There appear to be three reasons that code coverage metrics are used as exit criteria:

Code coverage is relatively easy to measure and provides a quantitative value that can be compared against some numeric criteria to determine success or failure.
Code coverage is an indication of the quality of the test suite and/or testing efforts.
There is an underlying assumption that code coverage is correlated in some tangible way with product quality. (That is, teams use code coverage as a surrogate for the admittedly harder to measure product quality.)

Management likes an unambiguous way to make decisions, and code coverage numbers seem to provide exactly the type of hard data that is useful for making a critical ship/no ship decision. However, don’t be seduced by code coverage numbers. The numbers must be used with extreme caution as low code coverage values are associated with low quality test suites, but higher code coverage values do not automatically imply a test suite is of high quality. And the relationship between code coverage and product quality is rarely, if ever validated. Failing to validate the relationship between code coverage and product quality renders code coverage numbers effectively useless as a measure of product quality.

Only very low code coverage values (at or near 0%) have any meaning without additional investigation: these values are a clear indication that the test suite is woefully inadequate. Larger values for code coverage, including 100% coverage, require additional investigation in order to make a useful interpretation of the value. Code coverage measures quality of the test site along a single dimension: how much of the product code is exercised. It does not measure product quality.

Product quality is how well the software meets the user’s needs and expectations. For the sake of illustration, imagine that product quality can be evaluated and assigned a numeric value from 0 to 100 (worst to best, respectively). Consider two extreme cases. In one case, the tests fully exercise product code and verify that the code works as specified, but the product was improperly specified and does not fulfill any of the end user expectations (coverage is 100% and quality is 0). In the other case, the product does exactly what the customer wants it to do, but the tests are poorly designed and exercise none of the product code (coverage is 0% and quality is 100).

We do not have a clear understanding of the relationship between code coverage and product quality. If the two are correlated at all, there is no reason to assume the relationship is linear or that it is consistent from one portion of the product to another (without incorrectly assuming that all teams and developers produce code of similar quality and complexity and that all test teams are equally adept at creating tests that provide the same coverage per code complexity). There are just too many variables to provide confidence in the numbers.

If code coverage cannot be effectively used as a surrogate for product quality and only correlates with test suite quality at low coverage values, then when does it add value? Next time I will discuss some of the scenarios where code coverage is a useful tool in the software development process.

Test Automation as a Product

Tim Lamey — Mon, 25 Jan 2010 07:25:00 GMT

Treating test automation as a product changes the way we design and implement test code and helps to ensure that the tests provide cost-effective feedback for their entire lifespan. The time a test spends being used for the verification of new functionality is relatively short, but the same tests must continue to run for many years as a guard against regressions during the creation of service packs for a released version of the software or during development of newer versions.

While it is essential to not lose sight of the importance of the software that is being tested, considering “test automation as a product” provides some interesting additional perspective on what constitutes good test automation. Some of the implications are:

The product should be easy to use. For tests, that implies they should have ‘push button’ execution in a variety of ways. Results should be reported consistently, should be easy to access and understand.
The product should be reliable. For tests, the results should not include intermittent failures. False failures should be minimized and false positives should never happen.
The product should perform well. Tests provide feedback on the state of the software that is being developed. The speed with which this feedback is provided is important.
The product should be easy to maintain. In the case of test hand-off and re-use, source code is provided and needs to be maintained and extended. This is a huge requirement for the test automation product. To be easily maintained, the code must be well designed, use consistent underlying test frameworks, and be implemented with quality engineering standards.
The product should be developed using state of the art software engineering practices. Tests should go through design and code reviews and have supporting documentation. Test frameworks should go through design and code reviews, include unit tests, be subject to rigorous pre-check-in verification, and must include a set of user documentation.

Attention to crafting the test automation product involves more than just sound engineering. There should be a conscious effort to limit the portfolio of frameworks and libraries used in test development. In many cases, ownership of tests is transferred to other teams for long-term support of the released software. The down-stream teams are often smaller than the development teams and cannot afford the extra effort that is required to understand and maintain tests running on a diverse set of test tools. When confronted with the choice of creating a new test framework or test library, think hard about the users of the automation product you are creating. Do they benefit from the new tools or is it just creating an additional barrier to the uptake and long-term maintenance of these tests?

Recommended Reading

Page, A., Johnston, K. and Bj Rollison. 2009. How we test software at Microsoft. Microsoft Press, Redmond, WA.

Test Quality Metrics: Diagnosing the Problem

Tim Lamey — Sat, 21 Nov 2009 02:00:00 GMT

A key attribute of test quality is that the tests must be effective, that is, there must be a high probability that they will detect defects in the product. Conceptually this is simple, but it is more difficult to come up with simple metrics that can be used to determine how effective existing tests are or if we are making progress in attempts to drive process improvements based on test effectiveness.

The fields of epidemiology and medical diagnostics use some simple quantitative measures (sensitivity, specificity, and positive predictive value) in discussions of how good diagnostic procedures are at detecting the presence of disease (see University of Iowa, Department of Pathology 2007 for additional details).

In medical diagnostics, sensitivity is the percentage of patients with the disease that are correctly identified as having the disease. Specificity is the percentage of patients that do not have the disease that are correctly identified as not having the disease. Positive Predictive Value (PPV) is the percentage of patients correctly identified as having the disease among all individuals identified as having the disease. In medical terms, good diagnostic tests have high sensitivity, specificity, and positive predictive value.

Parallels can be drawn between medical tests for the diagnosis of disease and software testing for the diagnosis of product defects. Conceptually, sensitivity and specificity should translate well to software testing as we want tests that are likely to detect product defects when present (high sensitivity) and which do not result in a large number of false failures (high specificity).

The matrix of the possible software test outcomes is:

	Test Result
	Fail	Pass
Product Issue	True Fail (TF)	False Pass (FP)
No Product Issue	False Fail (FF)	True Pass (TP)

The formulae for the relevant metrics are:

Sensitivity = (TF / (TF + FP)) * 100

Specificity = (TP / (FF + TP)) * 100

Positive Predictive Value = (TF / (FF + TF)) * 100

However, calculation of accurate values for sensitivity and specificity are near impossible in software testing because there is no “golden standard test” that can be used as the reference to distinguish between tests that pass because nothing is wrong with the product (True Pass) and tests that pass but do not detect a defect when present (False Pass). (In this case, sensitivity can be described as the percentage of tests that detect a product issue among all tests that could have detected a product issue. The practical problem is determining how many tests could have detected a problem.)

Positive predictive value (PPV) is relatively easy to calculate for software testing (as long as we have good data on the root cause of test failures). Test quality improvement efforts should track PPV and should include activities designed to improve the metric over time. Any activities which reduce the number of false failures will help to drive up PPV, and while it cannot be measured directly, an increase in PPV will tend to be correlated with an increase in testing specificity. Reducing the number of false failures makes it easier to quickly and accurately diagnose when the product contains a defect.

Recommended Reading

University of Iowa, Department of Pathology. 2007. Online Laboratory Services Handbook (Appendix). http://www.medicine.uiowa.edu/path_handbook/Appendix/Chem/pred_value_theory.html

Testing Efficiency and Effectiveness

Tim Lamey — Mon, 16 Nov 2009 13:45:00 GMT

Efficiency was a recurring sub-theme at the Ørdev software conference in Malmö, appearing explicitly in at least five of the presentation titles. However, most of the speakers were quick to point out that efficiency has little meaning without a focus on effectiveness. Scott Hanselman (http://www.hanselman.com/blog/) made the point most clearly in his keynote “Information Overload and Managing the Flow:”

Efficiency is doing something right
Effectiveness is doing the right thing

This point is very relevant to software testing efforts. We want our software testing efforts to be as efficient as possible, but because we always face more potential tests than can be created and executed in the time that is allotted, it is important that we focus on testing activities that are the most effective. It doesn’t do the users much good if we are very efficient at doing the wrong types of testing.

Efficiency is often an important consideration because management would like to maximize the return on invest (ROI) for the testing efforts. The premise is that being able to complete the planned testing faster or for less effort will have a positive impact on ROI. This makes sense, but it is still a good idea to ask the question,

“Is this the best way to test?”

as improving the effectiveness of the testing is likely to have a larger impact on ROI than performing the existing testing as efficiently as possible.

Qualitatively, effective testing activities are those that maximize the opportunity to discover problems with the software under test. However, the effectiveness of testing is difficult to evaluate quantitatively. In future posts, I will consider quantitative metrics for test effectiveness and and touch a little more on test efficiency.

Cucumber = plain text + Ruby

Tim Lamey — Wed, 11 Nov 2009 08:09:00 GMT

The Agile software development track during the Ørdev conference in Malmö last week included several sessions on Behavior Driven Development (BDD). This development methodology is one of the latest of the "driven development" paradigms and is positioned by its proponents as the evolution of Test Driven Development (TDD).

Like TDD, BDD permits developers to focus on what the code should do, rather than on the technical details of the implementation. And like TDD, BDD retains the process of first writing code that encapsulates "stories" used to validate the system under development before writing the code for the system itself. Code for the system is then written such that the executable validation stories pass. The development rhythm consists of alternating between adding validation stories and writing the code to support those stories.

BDD asserts that the activity of writing the validation first has very little to do with testing and thus it is unfortunate that TDD semantically places such emphasis on "test" (North 2006). The focus in BDD is on describing how the system should behave in a language (either native language or a ubiquitous domain-specific language) that is easily shared with, and understood by, the stakeholders, rather than being semantically bound by the language of "testing."

One of the more interesting presentations, which ironically was part of the Test track, was by Aslak Hellesøy, the creator of Cucumber (http://cukes.info/), a tool for combining plain text descriptions of how a system should behave with step definitions written in Ruby (or a number of other languages, including .NET languages). The presentation was a combination of an introduction to BDD, some coding using Cucumber and Ruby, and a live cooking show. How many software development presentations have you been to that included a cucumber, ginger, a little basil, and a blender? I left hungry for test code that is an easily readable description of the intent of the test rather than a reflection of the implementation details of the application or semantics of the test frameworks that were used.

Recomended Reading

Astels, D. 2009. Introduction to BDD with Cucumber. Engine Yard (23 June 2009). http://www.engineyard.com/blog/2009/cucumber-introduction/

North, D. 2006. Introducinng BDD. http://dannorth.net/introducing-bdd

Welcome

Tim Lamey — Tue, 10 Nov 2009 23:21:00 GMT

I am finally taking the plunge and starting an external blog on software testing.

My views on software testing have been acquired on the job during 13 years of helping to test a number of ERP (Enterprise Resource Planning) systems in the Microsoft Dynamics family of products. I have worked on Dynamics GP, Dynamics AX, and am now working on Dynamics NAV. I was also involved with several projects exploring the possibilities of creating a new framework for building ERP applications.

While the emphasis of this blog will be on issues, techniques, and practices that are common to testing ERP systems, many of the lessons learned will be generally applicable to testing any large, complex software system.