Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Chris — Tue, 20 Apr 2004 20:34:00 GMT

And even worse is when testers get docked pay for filing duplicate bugs. I've never had more shit from testers than when I've tried to return bugs as duplicates.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Drew — Wed, 21 Apr 2004 01:59:00 GMT

I'll bite on the controversy.

That's a bit one-sided (says Drew the tester).
Blindly-applied metrics like that can motivate developers to be unreasonable, too. When a dev manager says "if you have more than X open bugs you can't do new work on project Y" (yes- this can happen), we have devs who refuse to fix real bugs, resolving "won't fix" or "by design" or even "no repro" so that their bug counts can remain low. That's incredibly frustrating as a tester. Especially when the dev is in a different org or even a separate division of the company - we testers can't even get away with schmoozing to get the fix then.
For the record, I tend to agree that lumping several related problems into one bug is easier to track. The flip side is that I've also had bugs resolved as fixed when not all of the problems in the bug report were fixed yet. The right style seems to depend on the developer the bug is assigned to as much as the tester that filed the bug.

I'm not so sure the validity of the metrics is the problem either. A true measurement could still be meaningless. IMHO, the root of the problem is determining how to make information from data.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Valorie Osterman — Wed, 21 Apr 2004 04:06:00 GMT

Ok, the wife just has to chime in here....

Yup, I think that using the number of bugs filed metric as *the* most important part of a tester's review is silly. I'm a crappy tester from that standpoint. I'm even a crappy tester from Larry's standpoint of spending the extra time to figure out what's going on. Sorry dear, I don't write code, I just break it.

What's missing from the review process is exactly the input that's most important: the customer of the tester or developer who is being reviewed. Why can't a tester be graded using the metric of all bugs filed (by design, won't fix or spec included) and require the developer/closer of the bug to assign a "relevance of bug" scale and an "effort involved" scale? If you find a nit bug (like a spelling mistake), the relevance should be medium but the effort should be minor. If someone finds a nasty race condition, the relevance should be high and the effort (should hopefully) be high. This allows the customers of the testers (aka the developers) to have input in the tester's review. Testers get almost immediate feedback from the developers about what kind of testing they are doing and how much effort the tester percieves they are putting into their job. Wouldn't this help testers to do their jobs better and developers to forge better relationships with the developers? Is it more time and energy? Yes, but the goal is to make better testers (and better developers) on the way to making better products. The adversarial relationship between many testers and developers is nothing more than a time and energy waster.

The same thing can be said of the review of developers. Part of being in a group at Microsoft is the sense of family that develops. I would hope that everyone at Microsoft in any sort of a development position should have the ability to fill out a short questionaire for some number of people in their immediate group. Something simple like
1) I find the individual's code to be understandable.
2) I find the individual easy to work with.
3) I find the individual takes responsibility for errors or ommisions in their code.
4) I find the individual to write code appropriate to their level.
5) I find the individual to be an asset to their group.
(I know, there's a good reason I'm not an HR kinda person...)

Larry should be filling out a short questionaire for everyone in his group as well as anyone he's had a serious technical discussion with in the last month. He should be filling out a questionaire for his primary tester(s)and anyone who has written code that he's had to integrate into his stuff. I would think that everyone who's been at Microsoft for over a year could come up with 5 or 10 individuals about whom they could write some sort of quickie review. These reviews should help a manager get a better view of their employee. I know a good manager doesn't really need this as much as one who is overworked, but I don't percieve it harming the good managers.

Ok, so I gave you $2.00 rather than $.02. It's the decimals that do me in.

Increasing bug quality in your organization - yes, it can be done

Gunnar Kudrjavets — Thu, 22 Apr 2004 22:08:00 GMT

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Robert Wagner — Mon, 26 Apr 2004 19:45:00 GMT

Being an STE myself I must agree with Mrs. Osterman. I have to say that basing evaluations of a Tester solely on bug stats seems completely short sighted. Even if you are taking into account the priority and or severity of the bugs found.

There are other aspects to being a software tester than writing bugs. Quantifiable stats such as test cases written, test scenarios written, test cases completed, test scenarios completed, just to name a few.

Evaluation becomes more difficult when we attempt to quantify the more intangible quality criteria such as leadership, communication, thoroughness, etc.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Alan Bram — Mon, 26 Apr 2004 22:39:00 GMT

I guess you meant "omnipotent", not "idempotent".

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Larry Osterman — Mon, 26 Apr 2004 22:50:00 GMT

No, I meant idempotent :) From Dictionary.Com:
http://dictionary.reference.com/search?q=idempotent

Having said that, a better word would have been "interchangable". My usage follows from the jargon usage (a header file is considered idempotent if it can be safely included twice - thus w.r.t. order of inclusion it is interchangable) but that's a stretch.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Quentin — Tue, 04 May 2004 00:32:00 GMT

Hmmm. Maybe I'm missing something, but there seems to be an even more obvious flaw with such a metric: *if* each tester is working on different pieces of code or testing different apps or modules of an app, then there will exist a correlation between the number of bugs found by a tester and the developer responsible for the code being debugged. That is, you can't separate the coder from the tester. If a group of testers (T1...TN) are testing code written by a group of coders (C1...CN) then it could be possible (given the nature of the org or some such) that some subset of testers reviews the code from a subset of coders more often than others. That is, T1-T3 may get code from C2,C4, C5 more often than from C1,C3,C6-CN. The number of bugs reported by T1-T3 is then proportional to the number of bugs produced by the C2,C4 and C5 coders.

The point being, you would need to ensure that there is no statistically significant correllation between the tester and the person coding the app being tested. Otherwise, you could be measuring the coders skill at coding as well.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Larry Osterman — Tue, 04 May 2004 00:36:00 GMT

An interesting point Quentin.
I think that in general, most metrics of this type ignore quality differences between different members of a given development team, which is, of course silly.

Which gets back to the question of "Is the metric valid?" In the case you're describing above, it's not clear that it is. In other circumstances, it may be (the testers might be testing a completely new product developed by developers whose defect rate is constant). It depends on the circumstances.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

KurtW — Thu, 06 May 2004 01:19:00 GMT

Chiming in way late here, following a link from Joels page of last week, not sure if you've discussed this futher elsewhere, but:
Valorie, isn't the customer of the tester the customer of the business? So rather than bug counts or dev reviews, the testers rewards should be based on bug reports filed by clients?

This'd have the positive impact that the quality of the team is measured, rather than the quality of the dev or of that individual tester - both of which are subjective measures anyway.

It would also make your assessment of severity easier : a spelling mistake mentioned by your 10,000 seat client *may* have more business impact than when the 20 seat client screams at you about the false positive on file save.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Nicholas Urbi — Sat, 08 May 2004 13:32:00 GMT

Isn't it a little bit moronic to discuss how testers spend their time? It is their time that they waste anyway! And you are being paid for fixing those bloody bugs, you should concentrate on fixing them, or better yet, not produce them.

I know I'm asking too much...

If developers like you were not so incompetent, then testers could concentrate on the important stuff. I'm not saying that testers are brilliant, but your jerk attitude isn't helping either.

And of course you can invent the most important metrics for testers, but it will just get fatter and fatter until it explodes on your face. Developers who produce buffer overruns should not be allowed to modify metrics, because you must be a lot more careful when designing metrics. You should get a book on statistics if you insist. But anyway, it would be like saying that someone who can't balance his own checkbook should run for president. Resolve your own misunderstandings before you try to resolve someone else's.

Peace.

Turn brain on

Ben Bucksch — Sun, 20 Jun 2004 10:31:00 GMT

Larry, you are so obviously right that even a discussion about it is sick.

I'm a volunteer Mozilla contributor and now consultant since 1999, mainly as developer, but I also hunted bugs, mainly during dogfood. As it turned out, I was top 12 bug filer ever, top 3 of non-Netscape people, measured by reported bugs (valid and invalid). Top 1-3 are well-known (at least by me) for filing a mass of duplicate and stupid bugs, some even wrong.

Without exaggeration, Top 1 actually filed a number of bugs "http://mozilla.org should be http://www.mozilla.org/", one for each instance of the URL in the product. (For those who don't know: the slash is optional, the URLs are semantically identical. I prefer the version without slash.)

Top 2 was a Netscape developer, he filed bugs without checking for existing bugs at all, didn't think things through, and sometimes even fixed the bug right away, and marked it fixed. Quite often, it was later discovered that it was a dup, and that he missed important facts that were already considered in the inital description or discussion of the older bug.

If you count bugs open or fixed, dups and wontfix count negative etc., the numbers shuffle around a bit (Top 1 gets Top 3, I get Top 18 or so) but not all that much. Top 1-3 are still the same people and no better than before.

As you rightly said, the correctness, clarity/reproducability and severity of a bug are very important for the usefulness of a bug report. But if you try to measure this merely by bug DB fields, you'll get quarrels about Severity: major vs. normal instead of people fixing and finding bugs.

Judging programmers by lines of code produced is just as stupid. The best solutions are often the shortest.
A manager who proposes or uses something like this should be fired.
Any attempt to measure quality (of people, products etc.) based on statistics is bound to fail, in my observation, and does serious harm. Nothing can replace a rational, well-intended and -directed mind in judging.

Typo

Ben Bucksch — Sun, 20 Jun 2004 10:37:00 GMT

Correction, sorry, it should read: "http://www.mozilla.org should be http://www.mozilla.org/", i.e. he only wanted the slash added.

re: Larry's rules of software engineering #2: Measuring testers by test metrics doesn't.

Shrini k — Mon, 26 Jul 2004 12:18:00 GMT

Great post Larry. Thanks for picking up this "burning issue" that test leads/managers world wide face today.

My thoughts are:
Though it has been fiercely fought, extensively debated and beaten to death - "We should not measure tester’s performance only one the basis of # bugs they log", I think if we were to put a metric around that to make it SMART, # of bugs may be closest that one can think of. Among several work products the testers produce, "bugs" are single most important and have direct impact on quality of the final product that goes out customer.

So are there any efforts to classify the bugs logged standardize and normalized so that we can compare 2 bugs. once we have decent framework to classify the bugs to take care of all possible variable that make one bug different from other, it would be fair to measure the performance of two testers.

Having said all this, I strongly believe that there should way that quantifies output of a tester and measure.

Shrini

What does

Larry Osterman's WebLog — Fri, 28 Jan 2005 22:36:00 GMT

Hey, I'm an author too!

Larry Osterman's WebLog — Mon, 20 Jun 2005 20:32:54 GMT

Joel just sent me an email letting me know that the first edition of "Best Software Writing, I" has now...

Links to essays in Best Software Writing I

Nosce te ipsum — Sun, 26 Jun 2005 05:55:44 GMT

John Gruber makes an appearance in the soon to be released book The Best Software Writing I which was put together by Joel Spolsky .

The Best Software Writing I

Archipelagos — Mon, 25 Jul 2005 16:27:13 GMT

I have just finished reading a book compiled, edited and introduced by Joel Spolsky and released by Apress. &quot;The Best Software Writing I&quot; is a collection of some of the best articles on software development, and management written on various w

Another year, another post

Larry Osterman's WebLog — Wed, 15 Mar 2006 20:21:12 GMT

Well, this year I didn't miss the anniversary of my first blog post.
I still can't quite believe it's...

Métricas mal entendidas

ASP.NET Espanol Blogs — Thu, 02 Nov 2006 20:43:32 GMT

Vuelvo a leer aAntonio Quiros explicando su visión sobre la gestión de proyectos, en el

jtheo 2.0 » A proposito di software - Joel Spolsky

jtheo 2.0 » A proposito di software - Joel Spolsky — Wed, 11 Apr 2007 12:47:57 GMT

PingBack from http://www.jtheo.it/2007/04/11/a-proposito-di-software-joel-spolsky/

jace: Introduction to Best Software Writing I

jace: Introduction to Best Software Writing I — Sat, 21 Apr 2007 22:57:15 GMT

PingBack from http://www.livejournal.com/users/jace/385188.html

Introduction to Best Software Writing I by jace () | LjSEEK.COM

Introduction to Best Software Writing I by jace () | LjSEEK.COM — Sat, 21 Apr 2007 22:57:18 GMT

PingBack from http://www.ljseek.com/introduction-to-best-software-writing-i_61114144.html

Joel Spolsky: Best Software Writing I at ???????????? ?????????? ?? ??????????????????

Wed, 06 Jun 2007 14:54:00 GMT

PingBack from http://blog.rushchak.com/index.php/2007/06/06/joel-spolsky-best-soft-book/

Software Information » Larry Osterman’s WebLog : Larry’s rules of software engineering #2 …

Tue, 22 Jan 2008 15:35:49 GMT

PingBack from http://softwareinformation.247blogging.info/larry-ostermans-weblog-larrys-rules-of-software-engineering-2/

Rahul Verma’s Blog on Software Testing Perspective » Blog Archive » Using Bug Count for Performance Evaluation of Testers

Fri, 18 Apr 2008 12:35:42 GMT

PingBack from http://www.testingperspective.com/blog/?p=10

Good reading: The Best Software Writing I, Selected and Introduced by Joel Spolsky at ?????????? ?? ???????????? ????????

Mon, 09 Jun 2008 18:19:13 GMT

PingBack from http://cotoha.info/thoughts/good_reading_the_best_software_writing_i/

Larry Osterman s WebLog Larry s rules of software engineering 2 | Indoor Grills

Larry Osterman s WebLog Larry s rules of software engineering 2 | Indoor Grills — Mon, 01 Jun 2009 22:33:55 GMT

PingBack from http://indoorgrillsrecipes.info/story.php?id=3139

Larry Osterman s WebLog Larry s rules of software engineering 2 | fire pit

Larry Osterman s WebLog Larry s rules of software engineering 2 | fire pit — Sun, 14 Jun 2009 06:21:19 GMT

PingBack from http://firepitidea.info/story.php?id=1489

Larry Osterman s WebLog Larry s rules of software engineering 2 | debt solutions

Larry Osterman s WebLog Larry s rules of software engineering 2 | debt solutions — Fri, 19 Jun 2009 20:26:45 GMT

PingBack from http://debtsolutionsnow.info/story.php?id=2165