Introduction
Testability Is About Cost
There are lots of definitions of “testability”, but this is my favorite:
“[The] effort required to test a program to insure it performs its intended function”
I like this definition because with its first word it puts the emphasis squarely where it belongs: on the cost of testing software. If you are testing to a defined quality bar, testability means that you get to that quality bar faster. If you are working to fixed time and resources, testability means that you will ship higher quality software.
Testability Is About Automation
To a large extent, imparting testability to a system means making the system more amenable to automated testing, so you can think of testability as “the ability to build test automation that can do more, do it faster, do it more reliably, and do it at lower cost”.
Testability Involves All Disciplines
It is easy to come to the mistaken belief that testability is just something that developers implement for the benefit of testers: more hooks, more interfaces, different architectures, etc. Testability also involves other disciplines. For example, for software to be testable we have to know what it is supposed to do when it is working correctly. This is called the “oracle” requirement, and it is usually PM specs which fulfill it.
Testability Is Also About Process
Testability is more than just an attribute of the system being tested. It is strongly affected by the processes used to build the system. For example, testability is reduced if software changes can abruptly break all the UI test automation.
The Core Concepts of Testability
From a tester’s perspective, there are two core concepts that together make a system more testable: controllability and observability.
Controllability refers to the ability to place the system into any desired state, and to control the inputs to the system. Controllability is closely related to the ability to isolate a system component by building software shims “below” the component and drivers “above” it. To quote from Binder (see below for ref): “A component that can act independently of others is more easily controllable.”
Observability refers to the ability to know the exact state of the system at any time, and to observe the outputs of the system. Observability covers the entire domain of the system, not just the limited information available through public interfaces (client UI, public APIs, etc). For example, the full state of an object in the CRM system includes the data held in the database, instance-specific local data maintained by the Platform, cached data, XML data transferred to the client, and data as rendered in the client UI. Many tools can be used to increase observability: asserts, traces, dumps, logs, etc, as well as the shims and drivers mentioned above.
The other core testability concept that deserves to be mentioned at this time is sensitivity.
Sensitivity refers to the ease with which a fault-causing change in the system (“mutations”) can be detected. For example, when a Platform API change halts the entire CRM system, we would say that the system is showing high sensitivity to the mutation. Despite the dramatic effect and the bad things this scenario might say about change control processes, we all understand that this high degree of sensitivity to mutation is actually a good thing. One of the biggest benefits of test automation is that it increases the sensitivity of the testing system by decreasing the cost of regression testing.
Testability Factors
In his article “Design for Testability in Object-Oriented Systems” (Communications of the ACM, September 1994, Vol 17, No 9, p 87), Robert Binder discusses in detail six factors that influence testability: representation, implementation, built-in testing, test suite, test support environments, and process capability.
Representation
Representation refers to the availability of an accurate and precise model of correct system functioning. If testers do not know how the system is expected to function, they can play with it, but they can not actually test it.
Representation refers in part to the quality of the system architecture documentation; the functional requirements specifications; and the software design specifications. For maximum testability, the representations of the system from which testers work must be complete, correct, and current (meaning that the representation reflects what is actually implemented).
Another useful attribute of the system representation is its traceability, which is the ability to determine which software component implements a given feature, and which features implement a given requirement. Traceability is a book-keeping function, and while it is conceptually straightforward and often requested, it is not easy to introduce.
Implementation
Many aspects of the way a component or system is implemented will affect its testability. Most of these factors are well-known and fall under the general heading of “good software design”, but three general dimensions are note-worthy:
- Complexity. The more complex the implementation, the harder it will be to test.
- Scope. The more the implemented component does, the more tests will need to be run to cover the component’s scope.
- Determinism. If the implemented component requires asynchronous cooperation with other tasks, it will be harder to test. When a system is inherently designed around cooperating asynchronous components (or services), it pays to think hard about how to make the overall system more testable by increasing the determinism of each cooperating component.
Built-In Testing
Highly testable systems have lots of test functions built into the system itself. For example, the ANSI/IEEE 1149.x standards define interfaces that can set the state of any of the millions of gates on a VLSI chip, report on the state of any gate, send specific inputs to any gate and report the outputs. The testability of VLSI chips is improved enormously by the existence of these interfaces.
In a similar vein, an object-oriented system should have interfaces on every object that can set/reset the object into any desired state (possibly including illegal states, and states involving implementation-specific instance variables), and report back all the state data (internal and external) for the object.
Note that set/reset and reporter methods, to be useful, break object encapsulation and therefore have security requirements. A common approach to this problem is to limit these functions to debug builds. While simple, this is not always the best approach, especially for debugging live systems that cannot be taken down to introduce an instrumented version. In this case it is a better approach to build a run-time security scheme into the set/reset and reporter methods.
A well-known way to introduce built-in testing capability is through code assertions (“assert statements”). A lot has been written about the best ways to use assertions (to ensure that the internal logic of an implementation is maintained, to check that invariants are truly invariant, to implement observable tracing or reporting functionality, etc.), so I will not go into details here. I will, however, note that one of the key uses of assertions to increase testability is to police the extent of unit testing or class testing once a new component is integrated into the full system. In other words, assertions can be placed into the component code during unit testing to say “this component is not known to work outside these ranges”.
Test Suite, Test Support Environments, and Test Process Capability
These three factors address the way tests are authored, recorded, etc; the environment and tools used for testing; and the surrounding team processes (change control, build verification tests, spec inspections, etc). Detailed discussion of these factors is beyond the scope of this article.
