Progressive Development : unit testing

Motley says: “Slow down to go fast? Do I look like a tortoise?”

James Waletzky — Wed, 22 Apr 2009 08:23:00 GMT

Summary

	Motley: There is no harm shortcutting a few steps of development to meet a deadline. Some up-front check-in steps just take too much time.
	Maven: Slow down to go fast. Obey check-in checklists or you will pay for it later in the development cycle. Pre-check-in tasks such as code reviews, unit testing, static analysis, and buddy builds of various flavors must not be avoided.

______________________________

[Context: The code check-in deadline is approaching. Motley has code that is in progress and he is hurrying to get his change in prior to the build snap]

Maven: Hey, Mot. How's it-

Motley: Quiet! I have to get this code checked-in by the end of the day to meet the deadline. I don't have time to chat and need to rush to get this code in.

Maven: Whoa - maybe we should chat. Please just give me a few minutes here - it won't affect anything in the bigger scheme of things. What do you mean you have to "rush"?

Motley: The daily build starts at 6pm and it's 5:35pm. I only have 25 minutes to get the code checked-in so I have to take a few shortcuts.

Maven: I cannot stand the word "shortcut". Care to clarify?

Motley: It doesn't really matter what you can and cannot stand. But since you asked, I am going to skip writing unit tests for this last class, I am going to forgo code review because my code is always perfect anyway, and I am going to skip running FxCop just this once.

Maven: Have you ever heard the phrase, "Pay me now, or pay me more later"? What you are doing is akin to racking up credit card debt. On the outside it looks like a $5000 purchase, but in the long run it is going to cost you a lot more in interest if you do not pay off the debt as soon as the bill comes due.

Motley: Oh, no. More Maven analogies. I can do without those. And by the way, Boyz II Men called - they want their shirt back.

Maven: Always insulting my clothes! This blue and pink sweater combo isn't that bad, is it? Anyway, taking shortcuts now when the code is cheap to fix pays off in the long run and leads to quicker ship cycles and shorter bug-fix cycles later in the development cycle.

Motley: How so? Bugs are bugs, I need to fix them anyway.

Maven: It's much cheaper to fix bugs now than later. Fixing them later usually involves someone else creating and maintaining bugs in a bug database, fixing them when the code is no longer in context and fresh in your memory, and the tendency to bolt-on fixes and shortcut fixes. Additionally, if the test team does not find an issue, the cost of fixing it post-release is large due to the overhead in working with the customer, the bug getting back to the product group, deploying the fix, the negative press, and other factors. If you find an issue that modifies the design later, you may have to change a lot of code possibly destabilizing the product.

Motley: Yeah, yeah - the cost of change curve. I know it.

Maven: Then you know that you shouldn't rush your development. Would you rather fix bugs later when they are more expensive or now when they are cheap without the overhead of a post-check-in bug? Leaving bugs until later forces long "stabilization" periods on the team, which, in my opinion, are evil and can be avoided by a diligent team.

It is well worth pushing back on the deadline to ensure that you can adhere to the following code check-in checklist:

Unit tests. We have talked about unit testing at length in other discussions. This is the number one quality-driven developer technique. You must have a set of unit tests in place that exercise all of your new code and you can check-in when all the tests pass. This items should be bolded on a checklist.
Code coverage. Hand-in-hand with unit testing is code coverage. Aim for 70%+ code coverage for all of your code. More important than the coverage number is the feedback you receive from the coverage tool regarding source lines that were missed in testing so that you can add new tests.
Debug and retail builds. A debug build is generally different than a retail build, particularly when you make use of debug-time primitives such as assertions. You need to build both flavors to ensure the build succeeds either way. Broken builds are not tolerable as they block the entire team. In fact, do a clean build (vs. incremental build) to be safe.
Static analysis. A static analysis tool like FxCop finds bugs for you. Per the cost of change discussion, why not fix them now instead of later when it is more expensive? Why risk security defects escaping your defenses and being released into the wild where hackers and customers will find them?
Reliability tools. Running a tool like AppVerifier is more relevant to C++ applications. Running this tool is a must prior to check-in as it will help identify heap corruptions, improper use of handles, and misuse of critical sections. Additionally, for native code it will point out memory leaks.
Warning level. Crank up the compiler to generate warnings at level 4 (or whatever is highest for your compiler) to ensure that you have addressed all possible compiler-identified issues. Get into the habit of perpetually building at the highest warning level. Get clean and stay clean.
Other tools. Run any other organization-wide tools such as a copyright header checker, XML documentation checker, or coding style checker (e.g. Source Analysis) prior to check-in. You don't want to be the grunt fixing a ton of these issues later. Trust me on this one.
Code reviews. All code should be at least peer-reviewed, with larger changes (e.g. new features) being put through a code inspection. Having at least one other pair of eyes look at your code is likely to find many issues before even the test team gets their hands on the code.
Change list description. Include complete descriptions for check-in change lists, including how you tested the code, why the change was made, how you built, possible regressions, and other risks.
Buddy build. To ensure you did not forget to add a file to your change list and to protect against other bad assumptions, package up your change and pass it to a buddy for her to build. Most source control systems allow you to package up a change list for distribution. If you have a continuous integration system that builds changes before check-in, this is less necessary.

Motley: You cannot possibly be serious! If I do all that now there is no way I would ever make the deadline, let alone make a check-in.

Maven: All of the above is necessary, particularly in a large, quality-focused organization. We want to exterminate bugs before they are even checked-in. The checklist above helps with that.

Motley: Can you say "Sloooooooooooooowwwwww"? I knew you could.

Maven: Get in touch with your inner tortoise. I heard that in a TED talk on Slowing Down in a World Built for Speed by Carl Honore. Going a little slower up front will save time in the overall development cycle, and I would claim that you ship sooner as a result. Of course, that was not the specific theme of the talk, but the lesson can be applied here. Also, be pragmatic. Perhaps you avoid a step or two for small bug fixes.

Motley: One thing I do not quite buy is the lack of long stabilization periods at the end of the development cycle. We still need to put all the pieces together!

Maven: Ultimately we want to avoid long stabilization cycles and instead be stable when you check-in. A tail of the development cycle needs to be dedicated to integration and overall acceptance test validation, but is not the same thing as "stabilization". We talked about this before in our discussion on developer testing.

Motley: So you are telling me that I need to pay my boss a visit as soon as possible to tell him that I am going to miss the deadline? He is going to kill me.

Maven: Well, seeing as you only have 5 minutes to the build snap-

Motley: <POW>. You deserved that shot in the nose for wasting all my time and missing the build snap!

Maven: Ouch! Okay, I deserved that. But seriously, your boss is a reasonable person and will understand. Firstly, you are communicating bad news as soon as possible (well, not really, but let's pretend), which all managers appreciate. Secondly, present it such that you are focused on quality and want to avoid taking shortcuts, and that you feel this course of action is best for the product. I will bet you lunch that he understands.

Motley: There are some days where I wish I stayed in bed...

______________________________

James' Pointer: The checklist items I mention above are from a real team in at Microsoft in Windows Mobile. Quality is our #1 value as a team. We want solid check-ins that adhere to the checklist to ensure we don't have a long tail of bugs later. Yes, it is a lot of work but ultimately saves time in the long run. Slow down. Don't rush. Nail the checklist on every check-in. You will thank yourself later. Okay, maybe not, but you should.

James' Double Pointer Indirection: A good developer knows how much work goes into each and every check-in and factors these things into their estimates. These things take time.

James' Triple Pointer Indirection: "Slow down to go fast" is real advice in car racing circles. Intuitively it does not make a lot of sense. However, if I go into the corner too fast I have to brake to avoid the wall and then take a lot of time and gas to come back up to speed exiting the corner. If I slow down on entry, roll through the corner and smoothly apply the accelerator on exit, I actually go faster even though I slowed down entering the corner.

Maven's Resources:

TED (http://www.ted.com): A great web site of some of the world's best speakers. There are talks on a multitude of topics. You are guaranteed to find something of interest.
AppVerifier (http://msdn.microsoft.com/en-us/library/ms220948.aspx): A mandatory tool for C++ developers on the Microsoft Windows platform.
Source Analysis (http://code.msdn.microsoft.com/sourceanalysis): A customizable tool that validates coding standards and is a good complement to FxCop.

Motley says: "More test automation is always better"

James Waletzky — Tue, 19 Aug 2008 18:01:00 GMT

Summary

Motley: A tester's job is to find bugs, so measure them on the amount of bugs they find. More test automation is always better.

Maven: Do not measure testers by the amount of bugs they report. Think of the test team as more a quality assurance team than a quality control team. Too much test automation can be a problem due to brittle test infrastructures, too much time troubleshooting tests, suboptimal investment in tests, and lack of thinking like a user.

______________________________

[Context: Motley has been having issues with another counterpart team in the company]

Motley: I am a little disappointed in our test team. They really haven't generated many bugs in the product as of late. I feel like I have to kick them in the pants to get anything done.

Maven: I always thought we had a great test team. We have a bunch of really smart people, and they have made previous releases great. What is the problem?

Motley: Lately all the bugs are generated either by the development team or by others using the product. We pay the test team to test the product and find issues. If I was the test manager who is measuring them by the number of bugs they find, they would all fail this year's annual review.

Maven: Whoa. Before we explore the real issues of why the test team is not reporting bugs, let's talk about your statement around measuring testers.

Motley: Why? The best way to measure a tester is by the number of bugs they find. You can mine the bug database easily to get the numbers and it makes evaluation of the test team really easy.

Maven: Sorry, bud. Measuring testers by the number of bugs they find is a really dumb thing to do.

Motley: Don't EVER call me "dumb" - <pow>.

Maven: Ouch! Fine. I deserved that, and I apologize. Anyway, just because a tester is not reporting bugs does not mean they are not being productive. Think of the test role less as "quality control" and more as "quality assurance". The best tester can prevent a bug before it happens.

Motley: But they do control quality, and how would they prevent a bug before it happens? Developers hand over the code when they are ready expecting the tester to find all the issues.

Maven: Maybe in the past. Now, particularly with agile teams, testers are involved in the process right from the beginning. They participate in functional specification reviews, contribute to user scenario development, design reviews, code reviews, do private buddy testing, help compute metrics like code coverage, and many other quality-related activities. They do not have a sole test responsibility - they are another member of the team. A good tester helps assure quality by finding issues in all aspects of the development process and helps to improve the team's processes to prevent bugs. The best testers find no bugs in the code because they worked to prevent them right from the beginning. As a result, it makes no sense whatsoever to measure a tester by the amount of bugs they report in the bug database.

Motley: I guess that has some merit. The best tester may actually have a lower bug count than the poor testers. That is kind of an odd way of looking at the performance of a tester, though.

Maven: Odd, but reasonable. The test team should still be testing end-to-end scenarios in the product and reporting test results, however. You are saying that isn't happening? Why?

Motley: I talked to Morgan, the test lead of the multimedia feature. They are focusing all of their time on improving the test automation to ensure we have full coverage. However, the test infrastructure they are using is very unreliable and often reports incorrect results. Another team in the company is responsible for that infrastructure, but because our tests are a little more complicated, Morgan's team ends up troubleshooting the issues. It sucks up so much time that they don't have time to do the tasks that matter, like attend reviews and actually test the product.

Maven: A common problem when teams focus too much on automation.

Motley: What do you mean? Test automation is a good thing. It allows us to re-run tests without human intervention on a regular basis. Without test automation we risk breaking many areas of the product every time we make a change.

Maven: Do not get me wrong - automation can be a great thing. Unit testing is a form of automated testing and something that we have been discussing that absolutely every software developer must be doing. You make a change, you run your tests, and perhaps tests from other teams, to ensure you have not broken any functionality. The more unit tests, the better. The higher the code coverage on those unit tests, the better.

Motley: You are contradicting yourself all over the place Mr. Maven. First you said you can focus too much on automation, and now you are saying more is better. Inquiring minds want to know - which is better?

Maven: More unit tests - better. More scenario-based automated tests - not necessarily better. There is a great value in having a basic test suite that tests the core user scenarios of the product. However, relying too much on automation at the scenario level has disadvantages:

If the architecture of the test infrastructure is very brittle (i.e. not robust), every time the product changes, major changes in the automated tests may be required. Testers spend more time trying to get the automation to work than adding real value to the product.
A large focus on automation perpetually puts the test team behind the development team, if that is how your team is organized. If the developers design for testability, as they should, this situation is less of a problem as the testers can start developing automation fairly quickly. If automation is an afterthought, the developers may be on to the next feature before the test team is done, which means the feature isn't "done" according to our definition. It is also a context switch to go back and diagnose and debug issues later as the code is not as fresh.
If the infrastructure on which the current features are built is expected to change in the next release, often a rewrite of the test automation is required. Should the team spend a large amount of time building automation infrastructure if it will change in the near future? Tough call.
An automated test cannot think like a user, even if you inject some randomness into the tests. Over-reliance on automation creates some nasty test holes in the product that may only get discovered after shipping.

Motley: And here I thought more automation all the way around is always better. You actually make some good points. I am going to have a chat with the test people and ensure we are focusing on the right things. We should give up some work on automation if it means that we ship the product sooner and we do not cause ourselves any long term harm.

Maven: Sounds great. Let me know how you make out. I'm now going to stick some tissue up my nose to control the bleeding thanks to you.

Motley: Ah, you know you deserved it.

______________________________

Maven's Pointer: At Microsoft, I (James) have worked with a test team that seems to spend more time diagnosing and troubleshooting organization-wide automation infrastructure than it does improving processes, working closely with developers on test plans, and manually testing the product as a user would. Automation is a tool to detect regressions earlier in development, but it should not be used as a crutch. Given that philosophy, limit the amount of time you spend on automation to a reasonable number and pay attention to diminishing returns.

Maven's Double Pointer Indirection: We talked about code coverage in the past. One mistake test teams make is that they demand that they own the code coverage numbers, and that coverage should be measured against the test team's automated tests. Scenario-based test automation is not about testing individual components, modules, methods, and lines of code - it is more about ensuring that a scenario works as it should. Leave code coverage metrics to the developers and measure it with unit tests.

Maven's Resources:

I.M. Testy's blog, from a guy in the Microsoft Engineering Excellence who has been teaching about testing for a long time.
Alan Page's Notes and Rants, from another guy in Engineering Excellence who really knows his test material
How We Test Software at Microsoft, by Page, Johnston and Rollison, Microsoft Press, ISBN: 9780735624252, August 2008.

Motley says: "Just use a batch file to run unit tests"

James Waletzky — Wed, 19 Sep 2007 04:48:00 GMT

Summary

Motley: Just use a batch file to execute unit tests - it's easy and quick.

Maven: Use a unit test framework to write and execute unit tests. A framework makes tests easy to write, executes tests quickly and consistently, and provides features that make your life easier, such as logging.

______________________________

[Context: Motley was talking to a co-worker about unit testing]

Motley: I was talking with Marty the other day about unit testing. I was telling him that I was using the built-in Visual Studio unit test framework for my testing. He indicated I was nuts. After I popped him in the nose for calling me names, I let him finish.

Maven: Why would he think you're nuts, besides the obvious reasons?

Motley: <pow>

Maven: Ouch, dude! What was that for?

Motley: Calling me names! Anyway, he was telling me about his method of compiling tests as executables adding tests to a batch file, and executing the batch file. He says it's much faster and a big time saver.

Maven: I beg to differ! Using a test framework like Visual Studio has all sorts of advantages, including quick, integrated test execution.

Motley: Are you using the word "quick" and phrase "Visual Studio unit test execution" in the same sentence? You need to kick your drug habit.

Maven: Granted, Visual Studio 2005 was not super efficient in terms of test execution, but boy, you should see Visual Studio 2008! Big improvement. In any event, it's still quicker overall than using a batch file.

Motley: The fact that test execution is integrated into the IDE is definitely a bonus, and tests run reasonably quick after the framework startup has cached its state. However, batch files can still be quicker don't you think? And what if I don't want to use the VS framework?

Maven: I doubt it that batch files are quicker. Think about the context switch in moving from an integrated environment to a command-line environment. It's not very efficient over the long haul. Plus, how do you know the tests were successful? You either have to rely on a kludgy exit code or watch the results. We need something more robust than that.

Motley: Fine. I see a few benefits to using a test framework. What else?

Maven: Writing the tests. A good test framework like Visual Studio or NUnit makes it real easy to not only execute tests but write them. For C# there is a well-defined set of attributes that you can use to tell the test framework how to execute tests. Then the framework takes over, reflects on the compiled assembly, runs the tests you tell it to run, and reports results. It becomes obvious whether your tests pass or fail after execution, and most frameworks will give you a nice log of your results.

Motley: But if I just create executable code it's pretty easy to write the tests.

Maven: Yes, but not as robust. A test framework works consistently across all tests and reports results uniformly. You don't have to remember to set return codes and check them in a batch file, for example. All the common infrastructure is encapsulated behind the framework. This allows you to concentrate on writing your tests, which is where your attention should be. Plus, you can write tests using a common pattern.

Motley: What pattern would that be?

Maven: Most test frameworks support the following pattern for writing tests:

Setup a suite of tests: Write some code that sets up the execution of a set of tests. Tests are contained within a class, and an attribute like [ClassInitialize] is used to run initialization code that is common across all tests.
Setup each individual test: An attribute like [TestInitialize] is used to act as a sort of constructor for each test. Perhaps each test needs some test class member variables to be initialized prior to test execution. Here is a good place to put that code.
Write the test: A test method is typically a method with no parameters that returns void. An attribute like [TestMethod] tells the framework that the method is a test. You use a built-in test framework object like Assert to determine whether a condition passes or fails, and this tells the test framework what the result is.
Clean up each individual test: An attribute like [TestCleanup] is used to act as sort of a destructor for each test. Use it to clean up anything that you set up in the [TestInitialize] method.
Clean up the suite of tests: The [ClassCleanup] attribute is called after the execution of all tests to clean up whatever you set up in the [ClassInitialize] method.

Motley: But you could do that yourself with an executable and a batch file.

Maven: Yeah, you could. But why do a bunch of extra work, have a less robust solution forcing more test debugging, and avoid using a well-implemented framework that is regularly enhanced and maintained and does some work for you?

Motley: Okay, okay. I'm heading over to Marty's desk right now...

______________________________

Maven's Pointer: It's not important what framework you choose - even if its homegrown - but it is important to use a well-written framework. Visual Studio and NUnit are great choices for the .NET platform. If neither of those meet your needs, feed requirements back to the developers. If you decide to implement your own, sticking to the pattern mentioned above will ease the learning curve for test writers.

Maven's Resources:

A good summary of the new features of Visual Studio 2008 unit testing:

A nice table comparing the functionality and attributes between Visual Studio and NUnit for unit testing: Comparing the MSTest and NUnit Frameworks

Motley says: "If-then-else hooks are the best way to inject mock objects"

James Waletzky — Tue, 14 Aug 2007 18:26:00 GMT

Summary

Motley: A simple if-then-else hook is the simplest and best way to inject a mock object.

Maven: First, you have to design to interfaces. Then, techniques like dependency injection, factories, and endo-testing are more effective ways to inject mock objects.

______________________________

[Context: Maven and Motley continue a discussion on mock objects. Maven is just about to show Motley how to design his classes to be mockable]

Motley: As I said, a simple if-then-else hook in my code is probably the best way to inject a mock object for testing. It's simple, which is one of your key design principles, right?

Maven: I'll give you that - it's simple on the outside, but what about maintaining that code into the future? If-then-else hooks all over the place end up looking like spaghetti when you add more over time. Plus, there are other disadvantages.

Motley: First you tell me simple rules, and then you tell me simple leads to spaghetti. You know, Mave, sometimes I want to give you a shot in the-

Maven: Whoa! Stow your temper for a bit, bud. Simplicity, by default, is good. However, there are other slightly m ore elegant solutions that are still simple to follow and maintain. The "if" hooks, although simple to start, lead to complexity. They also require us couple test code to our production code if we inline the "if" statements. Both of those are far from optimal.

Motley: Ok, let's hear about your so-called "elegant" solutions.

Maven: Let's do it! Each require a little bit of object-oriented knowledge, but you're an expert, so no problem.

Motley: Ah, you flatter me. You're also right.

Maven: Above everything else, as we discussed in the past, you need to design to interfaces. Once you have well defined interfaces between your component boundaries, you can plug and play different implementations - mock objects are a great example. But, that's not all. You need to employ a simple technique to invoke the mock objects when required. The first technique is known as "dependency injection."

Motley: Sounds like something a drug addict would do!

Maven: Funny guy. Anyway, let's say we are developing an account class for a bank, and that we are doing some kind of transfer service from one account at our bank to some other remote bank over a secure web service. Perhaps we have code like this:

public class Account

{

public void Transfer(Money amount)

{

remoteBank.Transfer(amount);

}

Pretty straight-forward. Assume that remoteBank talks to a web service somewhere over the wire. We discussed that good unit tests cannot access the network - they have to remain isolated and fast. So, we need to replace the remote bank with something else. We could take your way of putting in an "if" hook, but let's use dependency injection. The remote bank needs a well-defined interface. Assuming that exists, here's what we do:

public class Account

{

public void Transfer(IRemoteBank bank, Money amount)

{

bank.Transfer(amount);

}

Motley: Great. The code basically looks like the same. Are you sniffing something?

Maven: Ah, it may look similar, but it's not the same. A client of the Account class can now pass in an implementation of the remote bank that implements the IRemoteBank interface. That implementation can be a real remote bank (probably by default) or a mock.

Motley: I see some advantages, but jeez, now clients have to know about remote banks, which is arguably an implementation detail of the Account class.

Maven: Very astute, my friend! Definitely a disadvantage mixed in with the advantages. Another technique is to use the factory pattern to return us the right implementation for our needs. Check this out:

public class Account

{

public void Transfer(Money amount)

{

IRemoteBank bank = BankFactory.CreateRemoteBank();

bank.Transfer(amount);

}

public class BankFactory

{

static public IRemoteBank CreateRemoteBank()

{

if (someCondition)

{

return new MockRemoteBank();

}

else

{

return new RemoteBank();

}

The "someCondition" in this case could be a flag or a setting from a configuration file that is true when tests are executing.

Motley: Nice, to some extent. We've now made it such that clients don't need to know anything about implementation detail, but you still have your test code coupled to your core business logic. Ideally we'd like to get rid of that.

Maven: I agree! Factories are nice to separate creation from usage, but they don't solve all our problems. This next technique, called Endo-testing, takes care of some of those disadvantages. Here's the code:

public class Account

{

public void Transfer(Money amount)

{

IRemoteBank bank = this.createRemoteBank();

}

protected virtual IRemoteBank createRemoteBank()

{

return new RemoteBank();

}

Motley: Big deal. So you have an extra helper method to create a new instance of the remote bank object. What does that solve?

Maven: Ah, but notice the virtual on the method declaration. That means that I can subclass the account object, override the createRemoteBank() method on the subclass, and return a mock object for test purposes.

Motley: Oh, um, I was just testing you. You can definitely override that method for test purposes. I guess that has the advantage that our test code is completely decoupled from our product code and clients still don't need to know about the implementation details.

Maven: You bet! There is a disadvantage though-

Motley: Yeah! You can't seal your classes!

Maven: Yes. We have to allow subclassing from our class even though we may not want to. And, we have a "protected" method, which for documentation purposes is essentially public. We really should document it.

Motley: Those seem like small disadvantages though relative to the alternatives. It's a bit more complicated, though, because you have to understand polymorphism, but a good OO developer can deal with that.

______________________________

Maven's Pointer: Know any more good techniques for injecting mock objects? Let us know with a comment below. Of course, all this requires that you design to interfaces, which was one of the main design principles discussed previously.

Maven's Resources:

Working Effectively with Legacy Code, by Michael Feathers, Prentice Hall PTR, ISBN: 0131177052, Sept. 2004.

Motley says: "Test both private and public methods"

James Waletzky — Tue, 12 Jun 2007 10:41:00 GMT

Summary

Motley: To get high code coverage, we should be testing both public and private methods

Maven: Only test public methods. Testing private methods gets in the way of refactoring

______________________________

[Context: Maven checks in on Motley's unit testing practices and notices something odd]

Maven: Hey Mot - how is the unit testing thing going?

Motley: Pretty good, I must say. I'm leveraging Test-Driven Development (TDD) as much as possible and it has really improved my code coverage. Check out all these tests!

Maven: Wow - that is pretty good! In fact, that's more tests than I was expecting. Are you literally testing everything?

Motley: You betcha. I have test cases around every method I write.

Maven: Even the private methods?

Motley: Duh. Of course! I gotta get that 100% code coverage, man! That was your idea don't forget.

Maven: Okay, I am willing to bet you are doing TDD without the refactoring part of the process. Am I right?

Motley: Well, um, I guess. I'm trying to do a bit but it's such a pain in the butt. But wait, how can you tell just by looking at my tests???

Maven: Well, if you are testing private methods that typically leads to a slightly larger number of tests, because a good practice is one test (or more) per method that you are testing. If you just test public methods, you can typically get the same code coverage with a slightly lower number of tests.

Motley: So what? So I have a few more tests. It doesn't harm anything.

Maven: Ah, but it does. Testing private methods makes it really hard to refactor. You see, refactoring does not usually affect a public interface but instead the implementation behind that interface, i.e. the private methods. With TDD, you need to be free to refactor, or you defeat one of its primary purposes - to drive a better design. If you need to update tests every time you make a refactoring change to your implementation because you are testing private methods directly, it really gets in the way and you are not going to do it. I am willing to bet that's why you are skipping an extremely important part of the TDD process.

Motley: The private methods do get in the way of refactoring, but code coverage is everything!

Maven: With TDD you don't write a line of code without having a test in place first. When you do refactoring, you rerun your tests to ensure you haven't broken anything. You should theoretically end up with the same coverage even just testing via public methods. If you can't hit your private methods through your public methods, why are they even there? Besides, the code coverage number in itself is not super interesting. What is more interesting is the analysis of why coverage numbers are low. This can lead you to more tests that you may otherwise miss. Don't go overboard trying to get to 100% - it's not worth it. Some C# language constructs, like foreach, have extra compiler-generated code in the intermediate language, which makes it real tough to get 100%. It's not worth the effort. Oh, and by the way, when I say "public", I typically also mean protected methods (public for subclasses) and even internal methods (public within the assembly).

Motley: So you're saying I should just test public methods and code coverage will just come if I use TDD. You're also saying that getting to that magical 100% code coverage number is often not worth the extra effort, and although code coverage is a good measure of your tests, the analysis is more important.

Maven: You should write a book, Mot. I could not have put it better myself.

Motley: I'm a quick learner, as you can see from my stellar deliverables. I can even teach you a few things, like how to be likeable in a social atmosphere.

Maven: What are you saying?!?

______________________________

Maven's Pointer: You may ask, "How do I test private methods anyway? They are private!". In .NET, there are several ways to test private methods should you have a rare case where you definitely need to test a private:

Leverage a test framework that uses reflection to call methods. The System.Reflection APIs can access private members of a class
Use Visual Studio 2005 (and later) for unit testing. It will generate a test context object when you use the wizard to create a test that lets you access privates.
Create a friend assembly using the InternalsVisibleTo attribute that lets you access internal methods at least.
Create/Run a tool that flips some bits in the .NET assembly metadata to change the scope of classes and methods all to public (you are not testing the original compiled assembly in this case though).
Create/Run a tool that generates a proxy for your class and accesses members via reflection. Your tests then use the proxy instead of the original class.
Use preprocessor definitions to change the scope of the private methods to public in a special build.

Maven's Resources: How to Test Private and Protected Methods in .NET provides similar arguments for/against testing private methods, and talks about a subset of the above list for testing private methods.

Motley says: "Contracts are for baseball players and cell phones"

James Waletzky — Tue, 08 May 2007 19:27:00 GMT

Summary

Motley: Contracts don't apply to software - leave them for baseball players and cell phone providers

Maven: Contracts (preconditions, postconditions, invariants) help drive your unit tests

______________________________

[Context: Motley is all excited as he has started to measure code coverage on his tests]

Maven: Hey Mot. Remember that conversation a while back that you said you were looking forward to as much as a cat looks forward to being neutered? I was looking over some of your unit tests and I think it's time we had it.

Motley: Yippee-ki-yay. You may know what the next part of that phrase is if you are a movie buff.

Maven: Ouch! I assume you're joking <laugh>. Anyway, about those tests. Your code coverage numbers are pretty good.

Motley: Excellent! Glad to hear something positive come out of your mouth!

Maven: Absolutely. I think you are making some great progress and the other devs are really looking up to you as a leader. I do, however, have an idea to make your tests better. Bear with me for a second. Although the coverage numbers are good, you're still missing some cases as per our last conversation.

Motley: Ok, you buttered me up a bit, so I assume you want something. I'll hear you out.

Maven: A good way to ensure you thoroughly test a method is to think about the contract of a method.

Motley: Contracts are for baseball players and cell phone customers and providers. I don't see how this is relevant.

Maven: If you think about your cell phone contract, your cell provider has certain expectations of you, and you have certain expectations of them. In the meantime, they may do some things that violate those expectations but they keep it invisible so that your expectations remain consistent. The same thing works for software. When you write a method, think about the preconditions of the method - what must be true before the method is called, the postconditions of the method - what must be true when the method is complete, and class invariants - what must be true before and after a method, but not necessarily during the execution of the method.

Motley: I think better with examples, dude.

Maven: Sure. An example of a precondition might be that an integer input parameter must fall within a certain range. An example of a postcondition is that the returned object cannot be null. An example of a class invariant is that the state of the class must be some value before and after the method, but could change during the method, like before a call to a private method, and restored after that call.

Motley: Wow. That would be a lot of extra work to think about that for every method!

Maven: It is extra work, but think about the benefits you gain when writing tests against a public method. Contracts help you frame your tests and make sure you have all boundary conditions/corner cases covered. Contracts also enhance the documentation around a method making it easier for someone to understand its usage.

Motley: I can see some value in thinking this way, but going through all this extra work in specifying a contract just to think through my tests seems like lots of work for minimal gain.

Maven: Well, you'd have to thoroughly test the method anyway. This is just a different way to think about it. Let's cover how tools can actually help us out with validating contracts …

______________________________

Maven's Pointer: Using contracts to design software is often referred to as Design by Contract. Thinking about your methods in terms of contracts helps drive quality of those methods. These contracts and associated unit tests actually become the formal specification of your software. Anyone wanting to know how the software is to be used (like the test team) can examine a contract and even without full documentation, can get an accurate picture of how the method is to be used.

Maven's Resources: Design by Contract on Wikipedia, An interview with Bertrand Meyer, who is credited with inventing the technique

Motley says: "100% code coverage tells me there is nothing left to test"

James Waletzky — Wed, 02 May 2007 02:47:00 GMT

Summary

Motley: If I hit 100% code coverage for my tests, then there is nothing left to test.

Maven: 100% line coverage does not necessarily mean you have full test coverage. You have to consider condition coverage as well.

______________________________

[Context: Motley is all excited as he has started to measure code coverage on his tests]

Motley: Hey Mave, check this out. I'm one up on you this time. I've been doing Test-Driven Development for a little while now and discovered that Visual Studio 2005 has code coverage built in! I can see exactly what lines of code my tests are executing and see an overall percentage of lines covered. Pretty cool, eh?

Maven: Very cool! Code coverage can be a very useful tool to help measure quality.

Motley: Of course you knew about it already Mr. Smartypants. I should have known.

Maven: Ah, don't feel bad. Visual Studio and I are best buddies and know each other intimately. What kind of overall code coverage numbers are you getting?

Motley: I'm doing pretty well, if I do say so myself. Check this out: 80% coverage on this module.

Maven: Pretty good, but why aren't you at 100%?

Motley: 100% code coverage?!?! Surely you can't be serious!

Maven: I am. And stop calling me "Shirley".

Motley: Ever been in a Turkish prison? Hehehe. I couldn't resist. "Airplane" is one of my favorite movies - EVER! And these quotes are classic. But seriously, I can't possibly hit 100%.

Maven: Sure you can! Doing TDD you should never be writing any code without a test. As a result, you should be very close to 100% if not at 100%. You can use code coverage to help gauge how thorough your tests are.

Motley: So I guess if I hit 100% on a module there is nothing left to test. Excellent!

Maven: Not so fast. You know there's always a catch. Are you measuring line coverage or condition coverage?

Motley: What's the difference? I think VS said line coverage but I'm not sure.

Maven: Line coverage measures your overall coverage in hitting all lines of code. Condition coverage validates that all possible blocks (accounting for conditions) have been hit.

Motley: Don't these basically boil down to the same thing? If I hit 100% either way, I'm pretty happy.

Maven: Check this example:

If (a == 1)

{

// do some stuff

}

else

{

// other stuff

}

If (b == 1)

{

// yet more stuff

}

else

{

// keep on going with more stuff

}

How many test cases do I need to get 100% line coverage?

Motley: Two. I can test the if/if and then the else/else.

Maven: Correct! 100% line coverage. Do I have enough tests to fully cover all my conditions?

Motley: Fine. No you don't. You need 4: if/if, if/else, else/if, and else/else. I see your point. Even though I have 100% line coverage I may be missing some tests. And by the way, I get diminishing returns by trying to hit 100% - some code constructs compile down to a lower level that make it difficult to get 100%. I'm not going to bother covering those.

Maven: As I have said before, you are a very smart man, Motley.

Motley: Tell me something I don't know. Oh wait, you try and do that every day! Hang on a second. I have to rush off to the kitchen.

Maven: What is it?

Motley: It's the room in the back with a fridge and counter, but that's not important right now.

______________________________

Maven's Pointer: Turning on code coverage in Visual Studio is not super intuitive. First set up your main application or library project. Then add a Test project to your VS solution. Once you have a test project in place, there is a file called localtestrun.testrunconfig that VS adds to your solution. Double-click that file. In the dialog that appears, click "Code Coverage" on the left side and check the box next to your non-test projects.

Maven's Resources: Don't be fooled by the coverage report, How to: Obtain Code Coverage Data (with Visual Studio 2005)

Motley says: "Refactoring is too hard - it's not worth the effort"

James Waletzky — Tue, 24 Apr 2007 21:42:00 GMT

Summary

Motley: Every time I make a change in refactoring I either cause compile problems or break functionality. Refactoring is slow and not worth the effort.

Maven: Use tools to refactor and leverage unit tests to improve your confidence that your refactoring does not break functionality.

______________________________

[Context: Motley is refactoring some code but having a tough time with it.]

Motley: #$%#%! This refactoring thing is a load of crap. It's too hard and not worth the effort. You were definitely high on something when you gave me this little tidbit.

Maven: Maybe I'll take a teeny step back for a second and let you cool off.

Motley: Not if I chase you into the kitchen with my iron fists of fury!

Maven: Well, errr, what's the problem?

Motley: Every time I make a change to "improve" the design I end up breaking something. Sometimes functionality breaks with my change and I discover that when I run the application, and sometimes I just have compiler error after compiler error. It's REALLY frustrating!

Maven: You know what I'm going to say next?

Motley: Grrrr. Yeah. "I can help with that". Well give it to me before I wring your neck!

Maven: I can definitely help with that. Let's start with the compiler errors. Even doing simple things like renaming variables - which is a simple refactoring - can cause compiler problems. Using Find/Replace across all files in a project can help, but you can still miss a file - particularly if you change an interface. But that was the old days. Let's try a basic refactoring example in Visual Studio 2005 that will make your life easier. Double-click the name of the variable you want to rename.

Motley: Grrrr… Fine. Now what?

Maven: Right-click on the variable that is highlighted. See that little "Refactor" menu that appears? Mouse over it, look at the pop-up menu, and click "Rename". Type in the new name of the variable and click "Ok". VS gives you a preview of what it is going to do. When you accept the changes it goes through and takes care of all the renames in a safe way leveraging all of the symbol information it creates when it compiles your project. I've come to trust it quite a lot.

Motley: Now that is pretty cool, I must say. But I don't use VS all the time.

Maven: Most other modern IDEs these days have refactoring support built in. In fact, the idea has been around for quite some time. Smalltalk developers are quite familiar with the technique. It is a huge time saver.

Motley: Ok, and I see there are quite a few different refactorings available. I'll have to play with that. But what about breaking functionality? That's still getting in the way of refactoring!

Maven: You're doing Test-Driven Development, right? Which means you have unit tests?

Motley: Yeah, so?

Maven: Are you running your unit tests after each and every refactoring?

Motley: No. That seems like overkill.

Maven: Well, that's part of the problem. You've spent a bunch of effort creating unit tests. They definitely help drive the design, but they also give you confidence to refactor the code. Every time you make even a small change, passing unit tests give you confidence that you have not broken any functionality.

Motley: Fine. I can see how that would help, but unit tests only cover small increments of code. I could still be breaking other dependencies/clients.

Maven: I can't argue with you there. That's reality and a most excellent point.

Motley: I guess validating the unit has not broken still goes a long way in preventing breaking functionality. I'll stick with this a little while longer and try and prove you aren't as dumb as you look.

Maven: I appreciate that. I think.

______________________________

Maven's Pointer: Visual Studio 2005 offers several different refactorings - mostly from Martin Fowler's refactoring catalog. Examples include rename, extract method, encapsulate field, extract interface, promote local variable to parameter, remove parameters, and reorder parameters. Experiment on some sample code by highlighting chunks of code and applying the refactorings. Once you get used to it, you'll save huge amounts of time. Other IDEs like Eclipse for Java also offer refactoring support.

Maven's Resources: Refactoring C# Code Using Visual Studio 2005

Motley says: "I don't have time to Refactor - I need to get it right the first time"

James Waletzky — Thu, 19 Apr 2007 16:34:00 GMT

Summary

Motley: Refactoring takes too much time - I have to ship you know!

Maven: Refactoring leads to much more maintainable code, and if you have tests in place, the return on investment far outweighs the cost.

______________________________

[Context: Motley has just embarked on his first adventures with TDD, but Maven wants to check-in to ensure Motley really understands the process]

Maven: So how are things going with TDD, Motley?

Motley: I've only been at this for a couple of hours - why are you bugging me already?

Maven: It's good to start off on the right path. Just want to make sure you understand the TDD process.

Motley: Yeah, yeah, yeah. Write a failing test. Write code to pass it. Move on to the next requirement.

Maven: Whoa there, Nelly! You forgot one of the most important steps - refactoring.

Motley: Not really sure what you mean by that. If I have syntax errors I fix them. Duh.

Maven: Refactoring is all about improving the design of your code without changing any existing functionality. You mentioned previously that one of your values is writing great software. Well, refactoring is not just about writing it great the first time, it's about being realistic with yourself that you won't get it right the first time. You make the necessary changes to prevent it from starting just OK and turning into a big turd later as you maintain it. Obviously it takes more time than just writing the code once, but the pay-off down the road is huge.

Motley: You've been sniffing too much glue, Maven. I barely have enough time to write my code once let alone keep redoing it over and over. I have to ship. If I don't ship, the business suffers, and most important, I get fired, and I suffer.

Maven: What about when someone has to maintain that code moving forward? Do you want to make it difficult for them? How much do you think that costs the business?

Motley: Like everything in life, there is a trade-off. I have to do my best given that one shot is all I get. I vote ship and deal with maintenance issues later.

Maven: That's what has gotten us into all this trouble with our legacy code! How do you like fixing bugs in all that old stuff?

Motley: Well… it sucks. Our legacy code is a "turd" - as you so eloquently put it.

Maven: Exactly. We need to make it easier for the next person to maintain. Refactoring is a way to get us there. Write your code, improve it, and then move forward with the next requirement. It's actually quite natural once you get in the rhythm.

Motley: But I'm afraid that once I have it working that if I make changes I'll break something.

Maven: Ah, but you're doing TDD, right? That means you have tests in place so that if you do happen to break something you know about it right away. It's quite beautiful, really. Give it a shot and I'll check in with you later.

Motley: Why do I keep letting you do this to me? Fine, I'll try it. But if I'm late on my deliverables, it's your paycheck. Deal?

Maven: I'm so confident that you're going to rock that yeah, I'll take that bet.

______________________________

Maven's Pointer: Many times the primary reason that developers are afraid to refactor isn't because they don't have the time, but because they don't have the confidence to make a change without breaking existing behavior. Having unit tests in place increases that confidence and makes refactoring easier and fun. Avoid refactoring without first having tests in place!

Maven's Resources: Refactoring: Improving the Design of Existing Code, by Martin Fowler et al, Addison-Wesley Professional, ISBN: 0201485672, 1999.

Motley says: "If I write code to pass tests I'm cheating!"

James Waletzky — Sun, 15 Apr 2007 09:44:00 GMT

Summary

Motley: Tests need to validate code that has already been written - not the other way around.

Maven: Writing tests before you code allows the tests to drive your code while providing more sound requirements validation.

______________________________

[Context: Motley is thinking through test-driven development before he goes off and tries it]

Motley: I've been thinking about this TDD thing and something is weird - if I write code to pass tests that's cheating! Tests need to validate code that has already been written.

Maven: Cheating? Hardly. Your tests are written around the requirements. What's wrong with writing code to execute the requirements? You're actually cheating if you write tests after the code. Now the tests are exercising the code but you are an extra step away from the requirements. Validation that you met the requirements is loose at best.

Motley: But I write my application code to satisfy the requirements. What's the difference?

Maven: The difference is that tests validate that the requirements are met, which is a logical layer between the requirements and the code. It forces you to think about the requirements in greater detail before implementing them. Plus, you get all the other benefits we talked about with TDD.

Motley: Well, maybe, but…

Maven: Just do me a favor and try out TDD for a couple of days, okay?

Motley: Fine, fine, fine. If it gets you off my back. Give me a couple of days to play. Now go make yourself useful and clean up the kitchen or something.

______________________________

Maven's Pointer: "Requirements" as described in TDD are usually described at a lower level than you may be used to. A "requirement" is one small nugget of functionality that you want to implement. It may be an entire (simple) method, or one positive or negative path through the method.

Maven's Resources: Test-Driven Development in Microsoft .NET, by James Newkirk and Alexei Vorontsov, Microsoft Press, ISBN: 0735619484, 2004.

Motley says: "I can't write tests before application code - there's nothing to test."

James Waletzky — Mon, 09 Apr 2007 18:32:00 GMT

Summary

Motley: You can't possibly write tests before code - there's nothing to test.

Maven: Writing tests for a method before coding it has all kinds of design advantages.

______________________________

[Context: Motley has been writing unit tests for a week. Maven wants to take the next step]

Maven: Hey. Looks like you're getting pretty good at this unit testing thing.

Motley: Yeah, I can definitely see the benefits of having a thorough suite of tests. I can make sure that with every code change I don't break anything.

Maven: Want to take it even farther? Do you want to make an even bigger difference in quality?

Motley: You really are a poop disturber, Maven. Fine. Let me hear about your hair-brained scheme.

Maven: How about this? Try writing your tests first before you write your code. It's called test-driven development, or TDD for short.

Motley: I can't write tests before I write any code! There would be nothing to test! You've been sniffing to much glue for your own good.

Maven: No, I am serious. Thinking about how you are going to test your code before you write it has all kinds of advantages, including driving your design. In fact, TDD is more of a design technique than a test technique when you really think about it.

Motley: So how could this possibly drive my design? What difference would it make if I write my tests before my code - which still seems crazy to me - or after? It's like saying that I should edit my book before I write it.

Maven: Maybe if you thought more carefully about how your book was to be edited, you'd do a better job writing the book the first time. With TDD, since you write your tests first, your design automatically is testable, which the test team will love. You can also view your APIs from the perspective of a client even before the API is implemented, so you know right up front whether the API is usable. In addition, coupling and cohesion problems in your code come to light immediately. I am sure you hate programming to difficult APIs. Well, this way you make sure that your APIs are clean before you start because you are using them. TDD essentially forces a good design to emerge based on your tests.

Motley: I'm still not sure about this. So all I have to do is write my tests before the code?

Maven: Well, to truly do TDD, here are the steps to follow:

Start with a requirement. A requirement could be one method you want to implement.
Write a test for that requirement.
Write just enough stub code to make the test and code compile.
Write just enough code to fail the test.
Execute your tests and watch them fail.
Write just enough code to pass the test.
Execute your tests and watch them pass.
Refactor.
Execute your tests to ensure you have not broken anything.
Go to step 1.

Motley: That's a lot of steps. Seems too complicated to really be useful. And what's the point of writing a failing test first?

Maven: Actually, it's not complicated. Once you get in the groove, it actually feels quite natural. Oh, and the point of writing a failing test is essentially to test your test. If you write a test and expect it to fail but it passes, you know you have a problem with either your test or your code. By starting with a failing case and then watching it pass, you are more confident that your test is good.

Motley: Let me finish what I am doing with my current set of modules and then maybe I'll give this a shot. No promises. You'll also have to enlighten me a bit on the refactoring step, as I'm not sure what to do there.

Maven: No problem, Motley. I applaud you for having an open mind to try this stuff out.

Motley: Whatever. The boss says you were hired for your sound engineering ideas, so I guess I should humor him and listen. Hehehe. Kidding, of course. Maybe.

Maven: Stay tuned. I have a lot more to say about TDD. It's a great developer technique.

Motley: Can't wait. Ugh.

______________________________

Maven's Pointer: A precursor to effectively doing test-driven development is understanding how to write good unit tests. Use a proven test framework to make it easy to write tests, and ensure your tests execute quickly. After all, with TDD, you are perpetually running your tests. It's such a great way to develop code that I don't think I'll ever go back to writing tests after code.

Maven's Resources: Test Driven Development: By Example, by Kent Beck, Addison-Wesley, ISBN: 0321146530, 2002.

Motley says: "Unit tests take too long to execute, so don't bother"

James Waletzky — Wed, 04 Apr 2007 18:38:00 GMT

Summary

Motley: Unit tests take too long to execute. I'm not going to bother.

Maven: Unit tests need to run quickly - you need to execute them frequently. Mock objects can help.

______________________________

[Context: Motley has been writing unit tests for a couple of days, but is not sold on developers writing test code]

Motley: I've been trying this unit testing thing for a couple of days now but I can't really say I like it. I developed this little executable to run my tests and that took time. You stated that I should run them regularly, but they just take too long to run.

Maven: Whoa - hold on a second. Let's address these things in turn. You're coding using C#, right?

Motley: Yeah. I wouldn't have it any other way.

Maven: Well, it just so happens there are some existing unit test frameworks out there that you should leverage.

Motley: Framework? For what? I just wrote a little executable to run my tests.

Maven: A good C# unit test framework like Visual Studio 2005 or NUnit can do some work for you. They quickly run tests, provide some good visual feedback, tell you the details about any failures, and make it dead simple to write new tests. You definitely need to be using a unit test framework!

Motley: I didn't even know you could do that with Visual Studio - might be worth a try. BUT, these tests just take too long to run so I'm not even sure it's worth the effort.

Maven: There are certain rules of unit testing that you need to follow. First, tests need to run quickly! If they don't run quickly, like in the order of seconds, developers won't run them and they get stale and unused.

Motley: But my tests have to run over a network to talk to a web service that has a computational intensive algorithm. It takes time. Nothing I can do.

Maven: You're testing your own code, right? Not the web service on the other end?

Motley: Yeah, so?

Maven: Replace the web service! You can create a stub or a mock object to take the place of the web service. The stub simply simulates the web service and returns a result for your test immediately. No network or long computation required.

Motley: That's cheating, dude! I'm not really testing the system then. It's fake!

Maven: Ahhhh, but we're unit testing here. The keyword is unit. You are testing your code that interacts with the web service, not doing a full integration test from end-to-end. Unit tests cover small, isolated chunks of code and simply validate the implementation of those small chunks.

Motley: But I still have to test the web service results!

Maven: Yes, but the web service will have its own set of unit tests. And just for the record, you also want your unit tests to:

○ Leave the system in the same state you found it (e.g. don't leave files sitting around that your test created)

○ Test very small chunks and not combine test cases (helps you quickly pinpoint the problem code)

○ Cover positive cases (the main logic) as well as negative logic (error cases)

○ Not be dependent on each other (don't expect unit tests to run in a particular order)

Motley: Okay, I'll try your suggestions. I guess if I want to replace components with stubs, I need to make sure they have well-defined interfaces so I can plug and play different implementations.

Maven: You are a smart man, Mot, a smart man.

______________________________

Maven's Pointer: Check out the infrastructure components that already exist to help with unit testing. Free tools for managed code include NUnit (http://www.nunit.org/) as a test framework and NMock (http://nmock.org/) for creating mock objects.

Maven's Resources: Pragmatic Unit Testing in C# with Nunit, by Andy Hunt and Dave Thomas, Pragmatic Bookshelf, ISBN: 0977616673, 2007.

Temporary Link: Technorati Profile

Motley says: "I am a developer - I don't test. Testing is for the test team."

James Waletzky — Thu, 29 Mar 2007 18:56:00 GMT

Summary

Motley: Unit testing is the test team's job! I am not paid to write tests.

Maven: White box (unit) tests help with code quality and give you confidence to change your code.

______________________________

[Context: During a previous conversation about debugging, Motley stated that he doesn't do unit testing]

Motley: Yeah, that's what I said: unit testing is the test team's job! I am not paid to write tests.

Maven: Did you know that unit testing is a developer technique with one of the biggest returns on investment?

Motley: Great. But it's still not my job. We hire a test team. They test. That's what they do.

Maven: But does the test team know the code the way you do? Unit tests need to be white box tests - they exercise as many code paths as possible - ideally, all - and require knowledge of the code.

Motley: Testers are technical people - they can figure it out.

Maven: But, they have a lot of other stuff to do. The test team has to do performance testing, stress testing, integration testing, platform testing, and of course, the all-important testing of real customer scenarios. Do you think they have time to find all those trivial little bugs that you should have caught before check-in? The answer is NO! The have expectations that developers write high quality code. They then do the validation of the code with test scenarios that would be difficult for developers without a full infrastructure to test.

Motley: They're also the reason why our ship dates are often missed!

Maven: Only because the developers are not doing their job with unit testing. You need to do a root-cause analysis on why we have these long "stabilization" cycles. We should not be stabilizing anything! As of our code complete milestone the code should be stable. The Test team validates on the black box level - without having knowledge of the code - to ensure integrated scenarios work properly.

Motley: So you're saying that developers need to write test code??? Bah!

Maven: YES! Try it on the unit level. In the object-oriented world, a unit is often a class. Write a set of tests using a unit test framework that exercises the contracts of your methods. Run them as often as possible - ideally, after every code change. By having tests in place, you have the confidence to change your code without breaking anything. Plus, you can generate code coverage numbers to help validate the thoroughness of your tests. The test team will thank you because now they can focus on their real jobs and not cleaning up after the developers. The test team also gets some nice sample code that helps them learn more about your APIs.

Motley: Sounds like lots of benefits to have unit tests, but I am still not totally convinced. If I take care in writing my code and ensure it's bug-free, test can still concentrate on doing their real jobs.

Maven: But you can't be sure your code has minimal bugs until you test it! Honestly, if I could get devs to add one technique to their quality arsenal, it would be unit testing.

Motley: Tell you what: I'll give this thing a try. However, if I don't get positive feedback from the test team that it is helping, I'm going to return to writing real code and let them figure out if my stuff works.

Maven: I'm good with that. I think you'll come out a winner! In the future we'll talk about this idea of contracts and code coverage, but start with some basic tests.

Motley: I really look forward to that conversation - kind of like a cat looks forward to being neutered.

______________________________

Maven's Pointer: These days, Microsoft hires testers that are just as technical as developers - they just have a different role. However, since developers know the code in detail, they need to be the ones to test the details. The test team can validate the code on a higher-level to ensure integrated scenarios function correctly. By combining the test responsibilities, we ship sooner and with higher quality.

Maven's Resources: Pragmatic Unit Testing in C# with NUnit, by Andy Hunt and Dave Thomas, Pragmatic Bookshelf, ISBN: 0977616673, 2007.

Motley says: "Rely on the debugger to write solid code"

James Waletzky — Sun, 25 Mar 2007 21:05:00 GMT

Summary

Motley: I rely on the debugger when I am writing code. Without it, I'd have a really hard time tracking down all the bugs in my code.

Maven: Write code as if the debugger does not even exist. Include unit tests, assertions, tracing, logging, static analysis, and internal consistency checks in your code to pinpoint problems as they occur. Time in the debugger is generally not efficient.

______________________________

[Context: Maven interrupts Motley as he is debugging some code]

Maven: Hey Motley. What are you up to?

Motley: Just debugging some code I just wrote. The debugger is every developer's best friend don't you know.

Maven: Actually, when I am writing new code I try and avoid the debugger at all costs. It is my most inefficient time spent as a developer.

Motley: Avoid the debugger?!?! You're insane! How can I expect to get my job done without the debugger. It lets me see the current state of my application and figure out why things went awry. Do you really have any experience in this industry?

Maven: How long does it take you to debug a typical problem using the debugger?

Motley: It depends on the problem of course. On average I would say it takes me about 15 minutes to come up with a decent repro and then another 15 minutes to get to the bottom of the issue. From there the fix is usually just a couple of lines and I'm off and running to the next issue. I suppose now you are going to tell me that your code is bug-free?

Maven: Of course not. Writing bug-free new code is next to impossible. Even "Hello World" typically has bugs. BUT, when I am writing new code I like to think the debugger doesn't even exist.

Motley: Boy, I bet you are really efficient. Can you sense my sarcasm?

Maven: You bet - I AM efficient. By incorporating the right level of unit tests, debug assertions, tracing, logging, and embedded self-tests, I can get to the bottom of an issue in no time. In fact, I can usually nail down the exact lines of code that caused the issue in a few seconds flat.

Motley: Did you say "unit testing"? That's the test team's job! I am not paid to write tests.

Maven: Oh jeez. One thing at a time - we'll have the unit testing conversation next. In the meantime, try inserting some defensive programming techniques into your code, like debug assertions and tracing. Not only will they help you get to root of an issue more quickly, but they will also help the support team once your software ships in the unlikely event a problem occurs. An assertion that fires provides an immediate indication that something is wrong and gives some great information about the error - as long as you take care writing your assertions.

Motley: By assertions you mean something like System.Diagnostics.Debug.Assert in my .NET code, right? I supposed I could sprinkle a few more in there. Including more debug traces would be easy. Unit tests? Not a snowball's chance in a high temperature furnace!

Maven: Okay, at least that's a start. The support team will thank you, and your time spent in the debugger will be minimized. I guarantee that the little bit of extra effort expended to incorporate those things into your code will still give you back lots of the time you would have otherwise spent debugging.

______________________________

Maven's Pointer: Use every tool in your arsenal to tell you about problems in your code before they appear. That involves running tools like Visual Studio's built-in code analysis (FxCop and PREfast) and cranking up compiler warnings to the highest level.

Maven's Resources: Code Complete 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, 2004.