Being Cellfish : Test

Test readability

cellfish — Fri, 12 Jun 2009 07:23:00 GMT

I read this excellent post on test readability a few days ago. I like the way he shows how he refactors a test for readability but personally I would have written the example test a little bit differently. First of all I would have added another extension method:


   1:      public static void PushOkButtonMultipleTimes(this ILoginView view, int times)   
   2:      {   
   3:          for (int i=0; i<times; i++)
   4:              view.PushOkButton();   
   5:      }

With that extension method my version of the test would have looked like this:


   1:      [Test]
   2:      public void only_three_login_attempts_are_allowed()
   3:      {
   4:          presenter = given_a_login_presenter_with_bad_credentials();
   5:   
   6:          view.PushOkButtonMultipleTimes(3);
   7:   
   8:          view.ShouldBeClosed();
   9:      }
  10:      
  11:      public ILoginView given_a_login_view_with_bad_credentials()
  12:      {
  13:          ILoginView view = CreateLoginView("BadUser", "BadPassword");
  14:          ILoginService loginService = CreateLoginService(false);
  15:   
  16:          LoginPresenter presenter = new LoginPresenter(loginService);
  17:          presenter.Initialize(view);
  18:          return view;
  19:      }

I personally prefer to move as much as possible of test setup to separate methods that are called explicitly by the test. The use of the extension method to remove the three times duplicated PushOkButton call is not something I'd always do. I was considering having the for-loop in the test method but I think an in place for-loop is less readable than making the calls three times. The extension method wrapping the for-loop however removes the duplication without decreasing the readability of the test.

When to write passing tests before fixing a bug

cellfish — Tue, 21 Apr 2009 23:45:00 GMT

If you're a BDD/TDD practitioner you're probably used to create a failing test whenever you have a new bug reported and you want to fix it. But sometimes a situation arises where you have to do the opposite; creating a passing test before you fix a bug.

Let me explain with an example. Let's say that you have code that uses date and time to calculate something. It all works fine. Then you go into daylight savings time and the test fails. Your first reaction might be to change the code to work but since you only have one test you might end up changing the code so that in half a year when you stop using daylight savings time the test fails again. So a better approach to handle this problem is to first create a test that passes before you change the code. Once that is done you have your (old) failing test and you fix the problem. Now you know you have code that works both with and without daylight savings time.

CAT - Continuous Acceptance Testing

cellfish — Sat, 17 Jan 2009 12:17:00 GMT

A few days ago I wrote about Continuous Unit Testing (CUT) and why the need for CUT was a sign of other problems. The last days I have been thinking about CUT a little bit more and I still find CUT questionable but there is another thing - CAT (Continuous Acceptance Testing) that I find very useful.

Acceptance tests are typically slower than unit tests since they involve several components that are not faked in any way and that may need extensive setup work. So acceptance tests are often much slower than unit tests. The unit tests created as part of the BDD (or TDD) cycle are probably enough to catch most problems but in some cases things will be caught in automated acceptance tests. And since that is the case you have much to gain from running those acceptance tests as often as possible. So why not run them AL the time (continuously) in the background?

Why slow unit tests are a bad smell:

cellfish — Wed, 07 Jan 2009 12:07:00 GMT

Earlier I promised to elaborate on why slow unit tests are a sign of problems (or a smell if you like). So here it goes.

The first thing I would like to look at is when the complete test suite takes too long to run to be part of a tight TDD-loop. If it is just the number of tests that makes the difference and each single test is super fast you will have a very large number of tests. Probably several thousands of tests. This can only mean two things. The least bad thing is that you have a very large application that really need all these tests. But you could probably split things up into different modules and test each module by it self thus reducing the number of tests that need to be run when you're working on a single module. Any dependencies between the modules can be faked in your day to day work. A much worse problem is that you're over-testing your code. That means testing the same thing in several similar (but not identical) ways. While over-testing it self should not be a problem it impacts TDD bad in two ways. First of all writing more tests takes more time so you get the feel that TDD is much slower than "developing the old way". It also makes the feedback slower since tests takes longer to run and that means to loose focus.

The second reason for slow tests is when a single test actually takes to long to execute just by itself. This is a sign of trouble since it probably means you're having badly written tests or even worse; badly designed production code. The reason for the test to be slow is typically because it takes some time to set up the circumstances for the test. The reason for this is typically a bad design since it is difficult or impossible to fake/stub/mock dependencies away in your test code. If you're having trouble faking dependencies you probably have a too tightly coupled design which most people I know tend to think is a bad design.

So whenever you feel the complete test suite takes too long to run, don't start looking at what is wrong with the tests - start looking at what is wrong with your production code...

CUT - Continuous Unit Testing

cellfish — Mon, 05 Jan 2009 12:05:00 GMT

During the Christmas holidays I read about something I would like to call CUT (Continuous Unit Testing). CUT is a technique where you run your unit tests in the background all the time rather than after each time you compile (I'm assuming TDD/BDD is being used all the time). There are a number of reasons I'm skeptical toward this approach.

My first concern is why the tests are run in the background and not part of the regular build process. It could be because the build environment does not have the ability to run custom actions after each build. Hm... Let me rephrase that: The developer does not know how to add custom build steps to the build process. And that is not the only problem. Every time I write a new test I have to wait for the background process to finish, start a new test run and wait for that result. This means the average turn around time each time I write a test is not only the time it takes to compile and make a single test run, it is the compile time plus 1.5 times the test time.

Another argument might be that the use of a background process to run tests makes it possible to continue writing code while the tests run. But then you're not really following TDD practices, are you? You're just mixing in your test & code writing continuously running the tests.

But there is one relevant reason for running unit tests in the background. If it takes longer than you can wait to run all tests you will probably start running only a small subset of tests for the code you're actually working on and then run all tests just prior to check-in (or have the build server run all the tests for you). This will however give you feedback that you've changed something you didn't perceive quite late. Under these circumstances I think it makes perfectly sense to run all unit tests in the background and only have a small subset running each time you compile for that quick feedback in your TDD-loop.

IMHO, having the ability to run unit tests fast is important if you are to be successful with TDD/BDD. And sometimes you end up with a number of tests that are not fast enough (or you have so many tests that they all together takes too long). Under these circumstances the background test runner makes sense but not otherwise. Also, having unit tests that are slow is typically a sign of problems but I think I have to elaborate on that later.

Yet another reason for loving xUnit.net

cellfish — Wed, 26 Nov 2008 13:26:00 GMT

I recently noticed a thing about xUnit.net that makes me love that framework even more than before. As you probably already know, an important aspect of (unit) tests is that they can be executed in any order. xUnit.net helps you remember this since it always runs your tests in random order. So simple, so brilliant...

Dangers of using Visual Studio 2008 Team System Code Coverage Tool for Native C++

cellfish — Tue, 18 Nov 2008 16:18:00 GMT

So now you know how to get coverage reports for native C++ using Visual Studio 2008 Team System (if not - read this). There are a few things you need to know before you get excited. First of all the only metrics you get are line and block coverage. A block is basically a statement and each line typically consists of one or more blocks. Unless you have 100% coverage I think these metrics both are pretty useless when measuring quality. For example consider a function consisting of ten lines of code. There is an IF-statement checking for an error and it throws an exception if the error occurs. If the error never occurs during the test-run you still get 90% line coverage since the other nine lines are executed. I think this is pretty common in production code. Most of the code is for the common state and fewer lines are used to handle errors. So you get pretty high line coverage even if you do not test any of error cases.

Block coverage is even worse. For example consider the following line:

SimpleClass* o = new SimpleClass();

That line produces two blocks of which only one is covered. And there is no reasonable way test the uncovered block since it probably has to do with when the process runs out of memory.

Identifying functions that are not called at all is often considered an important part of the code coverage report. Here we have another problem with the visual studio tool. Functions never referenced by the code will be excluded from the report completely (I suspect this is the case since the linker will remove all unreferenced symbols as part of the optimization at link time). This means the following class will report 100% coverage it it is instantiated and only GetA is called.

class SimpleClass
{
public:
    SimpleClass(int a, int b)
        : m_a(0) , m_b(b)
    { }

    int GetA() { return m_a; }
    int GetB() { return m_b; }

private:
    int m_a;
    int m_b;
};

So with all these potential problems there is another tool I'd recommend you consider. It's called BullsEye. It is a more "puristic" tool so there is no way to get block or line coverage, basically since those metrics are bad. In stead you can get Decision and Condition/Decision coverage. Basically Decision coverage checks that each conditional evaluates to both true and false and Condition/Decision coverage is when each part of a boolean expression is evaluated to both true and false. Consider the following line:

if(a || b)

There are two different decisions (either "a || b" is true or false) but four different conditions (both "a" and "b" must evaluate to true and false). BullsEye also adds instrumentation at compile time so the GetB method in the example above will not be lost but be part of the report as an uncovered function even if not referenced anywhere in the code. In the initial example (ten lines with 90% line coverage) we would get 50% decision coverage which is a much better indicator of quality.

And on using code coverage as a quality metric...I must insist you read one of my previous posts if you haven't done that already...

Native C++ Code Coverage reports using Visual Studio 2008 Team System

cellfish — Sun, 16 Nov 2008 13:16:00 GMT

The code coverage tool in Visual Studio 2008 Team System is quite easy to use from within the IDE unless you want code coverage for your native C++ code. In order to generate a code coverage report for native C++ you have to use the command line tools. This is how you do it:

First of all your project must be compiled using the /PROFILE link option. If you bring up your project properties it can be found here:
Configuration Properties -> Linker -> Advanced -> Profile
The profiler tools can then be found in the following directory:
C:\Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools
You need to add some instrumentation code to your EXE or DLL file and that is done with this command:
vsinstr.exe <YOUR_EXE_OR_DLL> /COVERAGE
This will copy the original file to an ".orig"-file and create a new file with the original name that contains instrumentation code needed to gather coverage data.
Now start the listener with this command:
VSPerfMon.exe /COVERAGE /OUTPUT:<REPORT_FILE_NAME>
Now run your EXE or some test suite that uses the file you want to calculate coverage for.
The listener started in step four (4) will not stop by it self once your test suite is finished so you have to stop in manually using this command (from a second command prompt):
VSPerfCmd.exe /SHUTDOWN
When the listener has stopped you just drag-n-drop the created ".coverage"-file into Visual Studio and you can view the results.

Exit Mocks. Enter Stubs!

cellfish — Tue, 14 Oct 2008 12:13:00 GMT

Since I'm not a fan of mocks I guess my prayers have been heard. Microsoft Research will soon release a stub framework. That's right. Stubs and not mocks! It is part of Pex that I have mentioned before. And you don't need to use Pex if you don't want. You can just use the stub framework by it self.

The new stub framework can currently be used to stub interfaces. It creates the stubs by generating code. At first this scared me since I feared it would not regenerate the stubs when the interface changed (which might happen quite common when creating the interface using BDD). But it works just fine and detects changes in the assembly and regenerates the stubs. Very convenient.

Another convenient thing is that it supplies default implementations for methods if I want. The default is to throw an StubNotImplementedException but I can change that and return default values (null, 0, empty string etc) if that is what I want. And the change can be made globally for all stubs created after the change or for a single stub object.

So how does this work? Consider the following interface:

public interface IMyInterface
{
    int DoStuff();
}

I just tell the stub framework to generate stubs for the assembly containing the interface and it will generate a new namespace with the stubs. If IMyInterface is part of MyNamespace the stub framework will create a new namespace MyNamespace.Stubs with a class SMyInterface. Now we want to make a test. The test below doesn't really make since since it tests the stub instead of something using the stub, but I want to show the code that makes the stub work without the surrounding stuff so bare with me...

void TestStub()
{
    MyNamespace.Stubs.SMyInterface o = new MyNamespace.Stubs.SMyInterface();
    MyNamespace.IMyInterface i = o;
    Assert.Equal(42, i.DoStuff());
}

Now the test will fail since the stub will throw an exception since the stub is not implemented. Let's use the default implementation:

void TestStub()
{
    MyNamespace.Stubs.SMyInterface o = new MyNamespace.Stubs.SMyInterface();
    o.DefaultStub = Microsoft.Stubs.DefaultStub<MyNamespace.Stubs.SMyInterface>.DefaultValue;
    MyNamespace.IMyInterface i = o;
    Assert.Equal(42, i.DoStuff());
}

Still a failure since the stub returns zero for DoStuff(). Let's stub that method.

void TestStub()
{
    MyNamespace.Stubs.SMyInterface o = new MyNamespace.Stubs.SMyInterface();
    o.DefaultStub = Microsoft.Stubs.DefaultStub<MyNamespace.Stubs.SMyInterface>.DefaultValue;
    o.DoStuff = (stub) => { return 42; };
    MyNamespace.IMyInterface i = o;
    Assert.Equal(42, i.DoStuff());
}

Now the test passes! I think this new framework is an excellent addition to the developer's toolbox. And I think it is a better starting framework than all the mock frameworks out there. In the few cases that a mock framework is really needed (over a stub framework) you can easily extend your stub to verify behavior in the same way as a mock does. Currently the stub framework have a few limitations such as only being able stub interfaces. But I'll guess we'll have to wait and see what will be part of the release once it is available to the public (when writing this I've been using an internal release available to Microsoft staff only).

My new favorite .Net unit test framework

cellfish — Wed, 24 Sep 2008 11:53:00 GMT

I recently had a chance to really sink my teeth into xUnit.net. Since I've been stuck with CPPUnit for quite some time and working with xUnit.net was a relief. Compared to NUnit I find xUnit much easier and convenient to work with (and compared with CPPUnit it's in a class of it own).

I feel I must give you four good reasons (no particular order) for why I prefer xUnit.net:

xcopy install makes it incredible easy to use it anywhere it is needed and in any form. Even different versions.
No need to mark up fixtures and test suits with attributes. A test method can basically be put anywhere.
No weird assertion names that feels like just a tick in the feature list. A small set of useful asserts.
Testing for exceptions is an assertion and not a test method attribute.

So earlier I suggested you to take a look at xUnit. Now I'm telling you. You won't be disappointed!

Nothing is "Too simple to test"

cellfish — Mon, 23 Jun 2008 09:18:00 GMT

A common practice taught in many TDD classes is the principle of too easy to test. The principle basically says that you should not test really simple things since then you will spend time writing tests that not really add value.

This is a fair recommendation and actually makes a lot of sense. Especially if you're implementing TDD in a team where people are a little bit skeptical. But there are a few dangers with allowing the use of this rule.

Exactly what is considered too easy to test may be different if you ask different developers. Those who are reluctant to embrace TDD tend to stretch this line quite far.
If something that once was really simple and then didn't need a test later on is changed to something more complex there is a risk that the developer will not add a new test since there was no tests before.

Personally I think that if something is considered to be so simple it does not need a test, it must be really simple to test. And if it so simple to test why shouldn't you use it. The relative cost might be high but the absolute cost for adding a real simple test for some real simple functionality is worth it in my book since it removes the decision from the developer. "No new functionality unless you have failing tests" is much simpler to follow and remember than "no new functionality unless you have failing tests or you consider the functionality to be really simple". Also consider the code coverage issues.

Code Coverage

cellfish — Mon, 16 Jun 2008 09:00:00 GMT

Measuring code coverage is often perceived as a good measure of test quality. It is not. Good tests finds problems when changes are made to the code. But if you just want to have large code coverage you can easily make a number of tests calling all your methods but not checking the results. The only thing high code coverage values really tell you is that the code is at least not crashing with the given input.

If you however are using BDD/TDD, code coverage values might be of interest. For example if you do not have 100% function coverage (i.e. not 100% of the functions are called) then you aren't really using BDD/TDD are you? Because how could you write a method that is not called by anyone? Well actually you might have created methods that are never called by your tests. Many TDD practitioners use a rule "to simple to test" which applies to real simple, property like methods. I don't really like that philosophy, but more on that in a later post.

So now you think with 100% function coverage you will also have 100% line coverage with BDD/TDD, right? Well, yes and no. Typically you don't since you will add error handling with logging methods that will never occur since the errors never occur in your tests. It might be that you open a file and if that fails you log an error and exit your application. This never happens in the tests since you always manage to open the file in your tests. So how did those lines get in there if never called? Well one other rule often used by TDD practitioners is that you "should not do stupid things". If a system call may fail, you check the result regardless of tests or not. With dependency injection you'll probably get close to 100% but there is no point in bending back-wards in order to achieve high code coverage.

Does this mean there is no point in measuring code coverage? I think it is great to measure code coverage if you use the result correctly. You should not add more tests just to increase coverage since test added just to increase coverage tend to just exercise code and not really testing something interesting. But low code coverage when using BDD/TDD is definitely a warning signal that something is wrong. The team is not using the methodology correctly. So what is considered OK coverage levels? From personal experience I think anything below the levels listed in the table below should be considered bad since type without injectiontype withoutinjection with with you should have no problem at all achieving the given values.

Coverage type	without dependency injection	with dependency injection
Function	90%	99%
Line	75%	95%

But sometimes there is someone (usually a manager) that thinks coverage should be above some level. So even though you know those tests will not really be useful you have to add more tests for coverage. What do you do? Either you can try to ignore the coverage fact and just try to add more tests, testing interesting things. Or you could try using Pex. Pex is a tool from Microsoft Research that is an automated exploratory testing tool. It can be used to make a small test suite with high code coverage and with only a few simple examples I get the impression it is quite good at finding border cases in your code. This will not however replace your traditional tests/specifications written as part of your TDD/BDD process. But it can help you test some cases you did not think of and that way increase code coverage even more without any extra effort from you. At least it is better than adding coverage tests by hand.

I don't like Mock objects

cellfish — Sat, 24 May 2008 23:44:00 GMT

When looking at BDD and TDD examples it is very common to see the use of mock objects in the code. This however strikes me as a little bit strange since I think you should use stubs and not mocks when working with BDD/TDD. A mock is basically a stub with the added knowledge that a mock object knows how it is expected to be used. In my opinion this fact disturbs the focus when applying BDD/TDD. Additionally I think maintainability is decreased since the tests will now have a dependency on implementation internals and you risk having to change your test not because the API has changed but how things are done internally.

Let me explain what I mean with an example. Consider a data object used to handle transactions and the ability to read and modify the amount money on a given bank account. Now assume this data object is used as a parameter to a transfer method that is used to transfer funds between two accounts. One way to implement the transfer method could be:

Start a transaction.
Get amount from the to-account.
Get amount from the from-account.
Set new account amounts for both involved accounts.
Check that the from-account does not have a negative balance.
End the transaction.

From a maintainability point of view I think mocking the data object will be risky since the mock will verify that the correct number of calls are made in the correct order. It wouldn't surprise me if we some day saw a new transfer implementation looking something like this:

Start a transaction.
Get amount from the from-account.
Check that the amount is enough.
Get amount from the to-account.
Set new account amounts for both involved accounts.
End the transaction.

The example may seem a little far-fetched but still I think it points toward a maintainability risk involved when using a mock. But OK, let's assume that maintainability will not be a problem. We're still shifting focus from what is important I think. As soon as a mock is involved when I write my test/scenario I do not only have to think about the behavior I expect. I must also think about how my mock will be used since that may or may not affect my test code. Using stubs instead of mocks will reduce that focus shift since stubs are just doing what they're told without regard of context.

Sure one might argue that using stubs will also involve having to think about how the stub is used. That is true but without the risk of having order or number of calls to the stub mess up your test/scenario setup. And the less time you spend on thinking about setup and the more you spend on defining behavior, the better I think. And thinking about behavior of internals and/or dependent objects should just be kept to an absolute minimum if you ask me.

Yet another unit test framework for .Net released

cellfish — Thu, 24 Apr 2008 09:09:00 GMT

Version 1.0 of xUnit.net has been released. So do we need another unit test framework? And I think the answer is yes. There are a number of things that I've found a little bit weird in NUnit and with xUnit.net they have been removed. I definitely think that you should take a closer look at why xUnit.net was written and the comparison with other unit test frameworks. xUnit.net definitely feels like a new fresh wind in the TDD/BDD toolbox.

Mocks are not Stubs

cellfish — Tue, 22 Apr 2008 09:24:00 GMT

As many before me I was recently looking at different mocking frameworks in order to find one that suited my needs, and was written in C++. And there are not many alternatives out there if you're using C++. I've found one open source and two internal (Microsoft staff can access what can be described as an internal codeplex) ones. Pretty soon it turned out that none of these frameworks did what I wanted. Actually they suggested using no framework at all for the situations I was looking to "solve" with a framework.

At this point I rediscovered an old article by Martin Fowler and it became crystal clear that the reason I didn't find what I wanted in the mocking frameworks was because I really wanted a stub framework (or rather a stub helper). Because what I wanted was a simple way to reuse the algorithm "return 17 the first 7 times called and then throw an exception after that, unless called with argument 4711 in which case you should always return 42". A pretty simple method to write in your stub if you just subclass and override the object you want to mock/stub but a little bit cumbersome in the long run if you do this a lot.

During a discussion on a company internal mailing list for TDD it became clear that people generally use the term mock object for almost every kind of non-production object used during testing. And when another large group do make the distinction between mocks, stubs, fakes and dummies you quickly run into trouble since you are no longer discussing the same thing. So here is yet another recap on what the different kinds of objects are so I can contribute to the cause, making world a better place to be. At least when talking about mocks & friends.

Object	Description	Comment
Dummy	Passed around and should never be used.	Should fail the tests if ever used.
Fake	Working implementation but takes short-cuts making them unsuitable for production.	For example an in memory database.
Stub	Quickly responds with a small number of known values.	Used during testing to mimic a certain behavior.
Mock	I like to think of mocks as stubs with the knowledge of usage added.	Used to verify that the user of the (mock)object makes the correct number of calls in the correct order. Many mock frameworks are designed to make verification of number of calls and/or order optional. But if you do not verify order or number of calls you're basically using a stub (but it is generated from a mock framework).