Progressive Development : test

Motley says: "More test automation is always better"

James Waletzky — Tue, 19 Aug 2008 18:01:00 GMT

Summary

Motley: A tester's job is to find bugs, so measure them on the amount of bugs they find. More test automation is always better.

Maven: Do not measure testers by the amount of bugs they report. Think of the test team as more a quality assurance team than a quality control team. Too much test automation can be a problem due to brittle test infrastructures, too much time troubleshooting tests, suboptimal investment in tests, and lack of thinking like a user.

______________________________

[Context: Motley has been having issues with another counterpart team in the company]

Motley: I am a little disappointed in our test team. They really haven't generated many bugs in the product as of late. I feel like I have to kick them in the pants to get anything done.

Maven: I always thought we had a great test team. We have a bunch of really smart people, and they have made previous releases great. What is the problem?

Motley: Lately all the bugs are generated either by the development team or by others using the product. We pay the test team to test the product and find issues. If I was the test manager who is measuring them by the number of bugs they find, they would all fail this year's annual review.

Maven: Whoa. Before we explore the real issues of why the test team is not reporting bugs, let's talk about your statement around measuring testers.

Motley: Why? The best way to measure a tester is by the number of bugs they find. You can mine the bug database easily to get the numbers and it makes evaluation of the test team really easy.

Maven: Sorry, bud. Measuring testers by the number of bugs they find is a really dumb thing to do.

Motley: Don't EVER call me "dumb" - <pow>.

Maven: Ouch! Fine. I deserved that, and I apologize. Anyway, just because a tester is not reporting bugs does not mean they are not being productive. Think of the test role less as "quality control" and more as "quality assurance". The best tester can prevent a bug before it happens.

Motley: But they do control quality, and how would they prevent a bug before it happens? Developers hand over the code when they are ready expecting the tester to find all the issues.

Maven: Maybe in the past. Now, particularly with agile teams, testers are involved in the process right from the beginning. They participate in functional specification reviews, contribute to user scenario development, design reviews, code reviews, do private buddy testing, help compute metrics like code coverage, and many other quality-related activities. They do not have a sole test responsibility - they are another member of the team. A good tester helps assure quality by finding issues in all aspects of the development process and helps to improve the team's processes to prevent bugs. The best testers find no bugs in the code because they worked to prevent them right from the beginning. As a result, it makes no sense whatsoever to measure a tester by the amount of bugs they report in the bug database.

Motley: I guess that has some merit. The best tester may actually have a lower bug count than the poor testers. That is kind of an odd way of looking at the performance of a tester, though.

Maven: Odd, but reasonable. The test team should still be testing end-to-end scenarios in the product and reporting test results, however. You are saying that isn't happening? Why?

Motley: I talked to Morgan, the test lead of the multimedia feature. They are focusing all of their time on improving the test automation to ensure we have full coverage. However, the test infrastructure they are using is very unreliable and often reports incorrect results. Another team in the company is responsible for that infrastructure, but because our tests are a little more complicated, Morgan's team ends up troubleshooting the issues. It sucks up so much time that they don't have time to do the tasks that matter, like attend reviews and actually test the product.

Maven: A common problem when teams focus too much on automation.

Motley: What do you mean? Test automation is a good thing. It allows us to re-run tests without human intervention on a regular basis. Without test automation we risk breaking many areas of the product every time we make a change.

Maven: Do not get me wrong - automation can be a great thing. Unit testing is a form of automated testing and something that we have been discussing that absolutely every software developer must be doing. You make a change, you run your tests, and perhaps tests from other teams, to ensure you have not broken any functionality. The more unit tests, the better. The higher the code coverage on those unit tests, the better.

Motley: You are contradicting yourself all over the place Mr. Maven. First you said you can focus too much on automation, and now you are saying more is better. Inquiring minds want to know - which is better?

Maven: More unit tests - better. More scenario-based automated tests - not necessarily better. There is a great value in having a basic test suite that tests the core user scenarios of the product. However, relying too much on automation at the scenario level has disadvantages:

If the architecture of the test infrastructure is very brittle (i.e. not robust), every time the product changes, major changes in the automated tests may be required. Testers spend more time trying to get the automation to work than adding real value to the product.
A large focus on automation perpetually puts the test team behind the development team, if that is how your team is organized. If the developers design for testability, as they should, this situation is less of a problem as the testers can start developing automation fairly quickly. If automation is an afterthought, the developers may be on to the next feature before the test team is done, which means the feature isn't "done" according to our definition. It is also a context switch to go back and diagnose and debug issues later as the code is not as fresh.
If the infrastructure on which the current features are built is expected to change in the next release, often a rewrite of the test automation is required. Should the team spend a large amount of time building automation infrastructure if it will change in the near future? Tough call.
An automated test cannot think like a user, even if you inject some randomness into the tests. Over-reliance on automation creates some nasty test holes in the product that may only get discovered after shipping.

Motley: And here I thought more automation all the way around is always better. You actually make some good points. I am going to have a chat with the test people and ensure we are focusing on the right things. We should give up some work on automation if it means that we ship the product sooner and we do not cause ourselves any long term harm.

Maven: Sounds great. Let me know how you make out. I'm now going to stick some tissue up my nose to control the bleeding thanks to you.

Motley: Ah, you know you deserved it.

______________________________

Maven's Pointer: At Microsoft, I (James) have worked with a test team that seems to spend more time diagnosing and troubleshooting organization-wide automation infrastructure than it does improving processes, working closely with developers on test plans, and manually testing the product as a user would. Automation is a tool to detect regressions earlier in development, but it should not be used as a crutch. Given that philosophy, limit the amount of time you spend on automation to a reasonable number and pay attention to diminishing returns.

Maven's Double Pointer Indirection: We talked about code coverage in the past. One mistake test teams make is that they demand that they own the code coverage numbers, and that coverage should be measured against the test team's automated tests. Scenario-based test automation is not about testing individual components, modules, methods, and lines of code - it is more about ensuring that a scenario works as it should. Leave code coverage metrics to the developers and measure it with unit tests.

Maven's Resources:

I.M. Testy's blog, from a guy in the Microsoft Engineering Excellence who has been teaching about testing for a long time.
Alan Page's Notes and Rants, from another guy in Engineering Excellence who really knows his test material
How We Test Software at Microsoft, by Page, Johnston and Rollison, Microsoft Press, ISBN: 9780735624252, August 2008.

Motley says: "I can't write tests before application code - there's nothing to test."

James Waletzky — Mon, 09 Apr 2007 18:32:00 GMT

Summary

Motley: You can't possibly write tests before code - there's nothing to test.

Maven: Writing tests for a method before coding it has all kinds of design advantages.

______________________________

[Context: Motley has been writing unit tests for a week. Maven wants to take the next step]

Maven: Hey. Looks like you're getting pretty good at this unit testing thing.

Motley: Yeah, I can definitely see the benefits of having a thorough suite of tests. I can make sure that with every code change I don't break anything.

Maven: Want to take it even farther? Do you want to make an even bigger difference in quality?

Motley: You really are a poop disturber, Maven. Fine. Let me hear about your hair-brained scheme.

Maven: How about this? Try writing your tests first before you write your code. It's called test-driven development, or TDD for short.

Motley: I can't write tests before I write any code! There would be nothing to test! You've been sniffing to much glue for your own good.

Maven: No, I am serious. Thinking about how you are going to test your code before you write it has all kinds of advantages, including driving your design. In fact, TDD is more of a design technique than a test technique when you really think about it.

Motley: So how could this possibly drive my design? What difference would it make if I write my tests before my code - which still seems crazy to me - or after? It's like saying that I should edit my book before I write it.

Maven: Maybe if you thought more carefully about how your book was to be edited, you'd do a better job writing the book the first time. With TDD, since you write your tests first, your design automatically is testable, which the test team will love. You can also view your APIs from the perspective of a client even before the API is implemented, so you know right up front whether the API is usable. In addition, coupling and cohesion problems in your code come to light immediately. I am sure you hate programming to difficult APIs. Well, this way you make sure that your APIs are clean before you start because you are using them. TDD essentially forces a good design to emerge based on your tests.

Motley: I'm still not sure about this. So all I have to do is write my tests before the code?

Maven: Well, to truly do TDD, here are the steps to follow:

Start with a requirement. A requirement could be one method you want to implement.
Write a test for that requirement.
Write just enough stub code to make the test and code compile.
Write just enough code to fail the test.
Execute your tests and watch them fail.
Write just enough code to pass the test.
Execute your tests and watch them pass.
Refactor.
Execute your tests to ensure you have not broken anything.
Go to step 1.

Motley: That's a lot of steps. Seems too complicated to really be useful. And what's the point of writing a failing test first?

Maven: Actually, it's not complicated. Once you get in the groove, it actually feels quite natural. Oh, and the point of writing a failing test is essentially to test your test. If you write a test and expect it to fail but it passes, you know you have a problem with either your test or your code. By starting with a failing case and then watching it pass, you are more confident that your test is good.

Motley: Let me finish what I am doing with my current set of modules and then maybe I'll give this a shot. No promises. You'll also have to enlighten me a bit on the refactoring step, as I'm not sure what to do there.

Maven: No problem, Motley. I applaud you for having an open mind to try this stuff out.

Motley: Whatever. The boss says you were hired for your sound engineering ideas, so I guess I should humor him and listen. Hehehe. Kidding, of course. Maybe.

Maven: Stay tuned. I have a lot more to say about TDD. It's a great developer technique.

Motley: Can't wait. Ugh.

______________________________

Maven's Pointer: A precursor to effectively doing test-driven development is understanding how to write good unit tests. Use a proven test framework to make it easy to write tests, and ensure your tests execute quickly. After all, with TDD, you are perpetually running your tests. It's such a great way to develop code that I don't think I'll ever go back to writing tests after code.

Maven's Resources: Test Driven Development: By Example, by Kent Beck, Addison-Wesley, ISBN: 0321146530, 2002.

Motley says: "I am a developer - I don't test. Testing is for the test team."

James Waletzky — Thu, 29 Mar 2007 18:56:00 GMT

Summary

Motley: Unit testing is the test team's job! I am not paid to write tests.

Maven: White box (unit) tests help with code quality and give you confidence to change your code.

______________________________

[Context: During a previous conversation about debugging, Motley stated that he doesn't do unit testing]

Motley: Yeah, that's what I said: unit testing is the test team's job! I am not paid to write tests.

Maven: Did you know that unit testing is a developer technique with one of the biggest returns on investment?

Motley: Great. But it's still not my job. We hire a test team. They test. That's what they do.

Maven: But does the test team know the code the way you do? Unit tests need to be white box tests - they exercise as many code paths as possible - ideally, all - and require knowledge of the code.

Motley: Testers are technical people - they can figure it out.

Maven: But, they have a lot of other stuff to do. The test team has to do performance testing, stress testing, integration testing, platform testing, and of course, the all-important testing of real customer scenarios. Do you think they have time to find all those trivial little bugs that you should have caught before check-in? The answer is NO! The have expectations that developers write high quality code. They then do the validation of the code with test scenarios that would be difficult for developers without a full infrastructure to test.

Motley: They're also the reason why our ship dates are often missed!

Maven: Only because the developers are not doing their job with unit testing. You need to do a root-cause analysis on why we have these long "stabilization" cycles. We should not be stabilizing anything! As of our code complete milestone the code should be stable. The Test team validates on the black box level - without having knowledge of the code - to ensure integrated scenarios work properly.

Motley: So you're saying that developers need to write test code??? Bah!

Maven: YES! Try it on the unit level. In the object-oriented world, a unit is often a class. Write a set of tests using a unit test framework that exercises the contracts of your methods. Run them as often as possible - ideally, after every code change. By having tests in place, you have the confidence to change your code without breaking anything. Plus, you can generate code coverage numbers to help validate the thoroughness of your tests. The test team will thank you because now they can focus on their real jobs and not cleaning up after the developers. The test team also gets some nice sample code that helps them learn more about your APIs.

Motley: Sounds like lots of benefits to have unit tests, but I am still not totally convinced. If I take care in writing my code and ensure it's bug-free, test can still concentrate on doing their real jobs.

Maven: But you can't be sure your code has minimal bugs until you test it! Honestly, if I could get devs to add one technique to their quality arsenal, it would be unit testing.

Motley: Tell you what: I'll give this thing a try. However, if I don't get positive feedback from the test team that it is helping, I'm going to return to writing real code and let them figure out if my stuff works.

Maven: I'm good with that. I think you'll come out a winner! In the future we'll talk about this idea of contracts and code coverage, but start with some basic tests.

Motley: I really look forward to that conversation - kind of like a cat looks forward to being neutered.

______________________________

Maven's Pointer: These days, Microsoft hires testers that are just as technical as developers - they just have a different role. However, since developers know the code in detail, they need to be the ones to test the details. The test team can validate the code on a higher-level to ensure integrated scenarios function correctly. By combining the test responsibilities, we ship sooner and with higher quality.

Maven's Resources: Pragmatic Unit Testing in C# with NUnit, by Andy Hunt and Dave Thomas, Pragmatic Bookshelf, ISBN: 0977616673, 2007.