Test plans, or test specs, are designed to share with a product team exactly what a tester plans on covering to ensure that their feature is ready to ship to customers.
Thinking about who the test plan is for can greatly influence the results of what comes out of the actual document. There are usually a bunch of vested parties such as developers, program managers, peer testers, test managers, test vendors, sustained engineering teams and security experts just to name a few. Each one is looking for something different out of the test plan document.
Here’s a breakdown of the lifetime of a test spec and who cares about at each milestone:
Milestone
Who Cares
What They Care About
Design Phase
Developers
Program Managers
Peer Testers
Test Managers
Testability of a feature
Design of a feature
Manual and automated test coverage
Impacts to other features
How long testing will take
Test Strategy
Implementation Phase
Test Vendors
Security Experts
Test Coverage
Checklist of signoff items
Security Scenarios
Stabilization Phase
Sustained Engineering
Post Release
Peer testers
Security Scenarios Checklist of signoff items
The two topics that are cared about in every milestone are test coverage and test strategy. Put simply, it is what is covered and why. Most of the time, test specs and test plans focus more on items in the design phase because this is when test plans are usually reviewed and iterated on. The people in the test plan reviews chose to focus on those items.
It isn’t always clear what a test plan is used for once it has been written. Often times, there are notes of work items and costs, lists of tests that need to be run or automated and perhaps specific performance and behavior goals to reference but it usually isn’t looked at again until sustained engineering handoff or something’s gone wrong.
Making test plans something that are consistently referenced, reviewed and used should be a goal for any test team. Otherwise, the exercise of writing the test plan is simply to think about how someone would test the feature and highlight areas of concern in a developer’s design.
Let’s go back to a subset test plan users:
Users
Needs
Feature Tester
Test coverage
Test work items
Test cases
Security scenarios
Test Cases
Test Managers Feature Team
Test strategy
Sustained Engineering Security Experts Future Feature Testers
If developers are following more traditional design then implement models, the test plan will usually be signed off prior to a developer starting implementation work. What happens to the test plan now?
There are three scenarios to consider:
Scenario
Test Plan Needs Changes
Developer Design and Spec do not change, feature is implemented exactly as described
No
User requirements change and minor updates or design changes are needed. Feature goals modestly updated and changed
Yes
Major changes to requirements and design cause implementation to drastically change. Feature is implemented completely differently from original design.
But what usually happens in these situations? If deadlines or milestones are tight, the actual test plan might not be updated. Or, more often than not, the test cases and scenarios are updated on a case by case basis in automation but not necessarily captured in the original document.
So, after all of that what does the Test Plan of the Future™ need? Here are some of the requirements that I thought of that will address the needs of the primary users:
This list of requirements satisfies a majority of the needs laid out above. If test plans can reflect the current state of the product, highlight feature and scenario coverage all while being easy to update, it can become a really valuable tester resource.
As things progress, I’m going to try investigating and implementing different types of Test Plans of the Future™. Stay tuned for future posts on this topic…