Importance of testing in computer applications is increasing day by day. Role of QA is to ensure quality of the product starting from product planning and design phase to final deployment. Efficient testing techniques and best practices at every stage of product life cycle are very important to deliver quality products to customers. The following is a compilation of some best practices in testing at various stages of product life cycle. It is not an exhaustive list. We can add lot more to it. This list is compiled from the trainings and presentations on various testing aspects and my own experience in testing @ Microsoft.
Define the scope of testing and Quality matrix early
One of the key challenges of a tester is to define the quality criteria with a clear agreement to the team and defining the scope of testing. As the product evolves, revisiting these criteria as per the changes is very important.
• Testing is an infinite space! You need to define the boundaries for this!
• How do you say that you are done with testing?
• Identify your quality criteria early
• Define the scope of testing and testing matrix
• Definition comes from whole team, but testing can drive
• Once identified, aim for them, and (most importantly) stick to them
• When you reach the criteria, you’re done, ship it!
• Publish them, make sure the entire team knows and understands them
• Track your progress against the goals
• Establish a formal process for change. They will change over time; just make sure that change is managed.
Test the Specification documents
Prevention is better than cure! At MS, test is involved right from the beginning of the product cycle instead of starting after a code complete stage. Getting involved at the specification stage is very valuable. QA should look from the customer’s viewpoint to ensure the specification is meeting customer needs!
• Many testing teams uses some standard check list for specification doc validations.
• There are standard techniques like model based testing for spec validations.
• ~30% of all bugs could have been avoided in the spec stage
• Testing must be involved in specification reviews
• Testing is the advocate for the customer
• Does it meet the requirements?
• Is it testable? At the same time no security holes later with your testing hooks!
• Bug’s fixed earlier cost less, so fixing an issue at specification stage is very much valuable.
• Want to reduce the number of bugs that get to testing? Reduce the number of bugs that get to development!
Do Design, Code Reviews
Many times test tends to ignore design and code review – at that stage they may be busy with test plan!
• Right design is very important
• Fixing a wrong design may prevent a pattern of bugs
• Consider stress and performance early
• White box testing – this should be an ongoing process
• Hard coding / Localization / Globalization issues
• Special handling / special casing – my experience taught me this area is a bug mine!
Manage your Bugs!
The quality of the bug filed by a tester talks a lot about the tester!
• Describe bugs accurately with clear repro steps
• Clearly give the customer scenario and impact.
• Quality of bugs matters, not just the number of bugs
• Give the right priority and severity
• Promote practices that keep open bug counts down
• Track bug metrics and get blocking bugs fixed with high priority
• File bugs on each and every issue to track them.
• Be proactive; analyze the patterns of bugs to identify weak design areas.
• You should not wait until the end of the milestone to verify the resolved bugs
• Do not put more than one bug in the same bug entry
• Do not resolve a bug not Repro unless it truly is not reproducible
Test Automation
Automation of the tests is very important for efficient regression runs and to save manual efforts in repeated testing. Time to automate, maintenance cost and amount of time for manual testing are crucial deciding factors on deciding on the % of automation at various stages of product life cycle.
• Identify the right tools and frameworks for your automation. Think about long term benefits, extensibility, maintenance and reporting.
• When to automate? Especially UI tests
– Specification documents should be closed – tester should drive for it. If we start automating early without the product specification fixed, it will lead to lots of automation destabilization and maintenance.
• Goal: Minimal tests to get max coverage.
• Start with the key functionality test automation and later expand it to cover the desired feature lists.
• Consider stabilizations and stabilization costs
• Manual tests – Is there a better way to do it? If we are doing it repeatedly, look for a way to automate it. Trade off is the time to automate and maintain vs the amount of time you spend in running manually.
• Frameworks should be globalization/localization ready if you are doing these testing.
• Fix test issues upfront! A stitch in time saves nine J
• Tools – are you reinventing the wheel? Look around and see if you have the same tools available!
• Use measures like code coverage to ensure you have sufficient coverage for your tests.
Test Your Tests!
What if your tests are not catching the bugs? Test automation and test codes are to be reviewed with good attention to the details.
• How will you ensure your test code catches bugs? Is there any false positives?
• Ongoing risk analysis ensures you’re testing the most important things
• Coverage analysis ensures that your tests are testing everything you think they are
• Code reviews for testing are more important than code reviews for dev
Analyze Your Tests
A good tester will keep analyzing his tests to ensure the quality of the product is good at all stages of product cycle.
• Risk Analysis -likelihood of a problem happening and impact if it happens
• Bug Analysis – is the product getting stable? Do I see any patterns?
• Product Analysis – functional, data, platform, code coverage – what changed as the product cycle changes and progress?
• Data Domain Analysis – have you covered all the possible ranges?
• Change Analysis – have you added new tests for the final stage bug fixes/design changes/feature additions?
• Code coverage Analysis – how much of the code is touched by your tests?
• Functional coverage – how much of the functionality is covered by your tests?
Private Testing for late design changes
Late design changes in the product cycle could be risky. Test should ensure proper quality gates are used to do large design changes to avoid destabilization of the product.
• Ensure there is proper code review done before check in.
• Define a check in criteria for risky design changes.
• Do private testing and signoff when there is a large design change.
• Add new tests to your test suites
Communication
Efficient communication is a key to the success of a tester. It could be in the form of calling out the quality concern to the team or collaborating with the peers to leverage their knowledge to improve the component quality or pushing back on enabling key customer scenarios.
• Work in perfect sync with developer and project management team.
• Leverage from all experienced hands – third eye! Ask your friend to dogfood your component or conduct bug bashes.
• No assumptions in testing! Make your assumptions public!
• No biasing/Assuming it works ( if x works 1000x will work)
• Let the team know what you think about product quality – Good/Fair/Bad
• Call out what is tested and what is remaining!
• If there is a decision taken by the development/project management team on not fixing the bug – If you disagree on the decision, always raise it to the team with the justifications. At the same time pick the right bugs and understand the prioritizations.