My QA manager colleague and I have been working on our QA metrics process for the Team Foundation Server test organization. This has been a fascinating process, and I wanted to give out a little insight into how we're thinking about the problem.
We have 2 different metrics processes focused on 2 different entities: 1) the product quality, and 2) the QA process/team.
The focus of this blog post is on the second aspect - measuring the QA process/team. We have lots of metrics for product quality, and I'll get into those in another post series (and in particular, how you can use TFS to help track and report them).
To give the metrics process a framework on which to build, we've come up with the following "values" for our team:
- Self Improving - we're constantly aware of how we're doing and constantly improving
- Highly Productive - we get a lot of work done in a short amount of time
- Increasing Capacity - we're gaining the ability to get more done over time
- Predictable - we can be trusted to deliver what we say when we say it, transparency is a key part of this
- Customer Focused - because we all want happy customers!
We've started referring to these values as SHIPC to help everyone remember them. Mnemonics are great, eh?
We use these values as a way to think about everything we do. For example, if we're developing test plans, are we writing more effective test plans this time than in the past (self improving)? Are we writing more effective test plans in less time (highly productive)? Are we able to cover more new features than in previous product cycles (increasing capacity)? Do the test plans effectively reflect customers priorities, key customer scenarios, customer-focused testing, etc. (customer-focused)? You get the idea...
Personally, I've found that having these 5 values has really helped provide a foundation for the rest of our metrics development/refinement process.
Obviously, there are thousands of possible values to choose from. In coming up with these 5, we spent a good amount of time brainstorming about adjectives we'd like to be able to use to describe our QA team. Then we went through a bucketing process where we grouped similar adjectives. Finally, we came up with this set of 5. Yes, there are overlaps between them, but we really felt like they captured the essence of what we thought of when we pictured a world class QA organization.
Do you use values to guide your metrics process? Are your values very different from these? Do you have any others you think a QA team shouldn't live without?