Dennis "D.C." Dietrich : Exceptions

The ultimate ExceptionAssert.Throws() method

ddietric — Wed, 07 Jan 2009 10:15:00 GMT

A while ago I published a post titled Pimp your VSTT exception tests in which I pointed out some disadvantages of VSTT's declarative approach to testing for exceptions and provided an alternative. I recently gave this some more thought while writing some new tests and noticed that my original version could be significantly improved by a simple tweak leading to the ultimate version of the Throws() methods (still, don't be too surprised if I publish a post titled The ultimate ExceptionAssert.Throws() method v2.0 at some point).

First of all, when writing my original post it seemed like having overloads would be a good idea because of how anonymous methods are treated by the compiler and the potential side-effects. Looking back at it I'd say that was an overly cautious approach. In the tests I've written since I published the post there wasn't any instance where accessing fields and locals would cause problems. The second point is the really interesting one though. Consider the following code:

public static void Foo(int i)

{

if (i < 0)

throw new ArgumentOutOfRangeException(

"i",

"The value of parameter 'i' must not be less than zero.");

}

When testing whether or not the method throws as expected, the test code will often just check if an exception of the correct type was thrown. If the code is part of an application and only used internally this may actually be enough and just checking the type is somewhat encouraged by how it is achieved in VSTT with the ExpectedExceptionAttribute. The same is true for my original Throws() method. But if you're building components for others to use in their applications you certainly want to make sure that the exception object looks exactly the way it was specified.

In the sample above you would want to verify the properties ActualValue, Message and ParamName. But generally speaking the approach to exception testing used when creating libraries must allow the tester to perform arbitrary validation on the exception object. While Throws() could for example simply return the exception it caught - probably the simplest solution - this would be not very consistent with how a typical assert method works. It is void-returning and only returns if the assert passed. A slightly more difficult approach which is equally flexible in this context leads to more consistent test methods. Since we're already dealing with delegates for telling Throws() what code exactly should throw, why not pass in a second delegate which verifies the exception object using standard assert methods so that a test for the method above would look like this:

[TestMethod]

public void TestMethod()

{

ExceptionAssert.Throws(

() => MyClass.Foo(Int32.MinValue),

(ArgumentOutOfRangeException e) =>

{

Assert.AreEqual("i", e.ParamName);

Assert.AreEqual(Int32.MinValue, e.ActualValue);

Assert.IsTrue(e.Message.StartsWith("The value of parameter 'i' must not be less than zero."));

});

}

The biggest disadvantage is that test methods will probably rely on anonymous methods in many cases to keep the code compact and make it easy to follow the control flow but it can take a while get used to reading/writing this kind of code. Still, if you want/need to go deeper than just looking at exception types I'd say that this is a good way of doing it and - unless I've overlooked something (again) - the only Throws() method you'll ever need (especially since the second parameter is optional).

public static class ExceptionAssert

{

// The type parameter is used for passing in the expected exception type.

// The where clause is used to ensure that T is Exception or a subclass.

// Thus an explicit check is not necessary.

public static void Throws<T>(Action action, Action<T> validator) where T : Exception

{

if (action == null)

throw new ArgumentNullException("action");

// Executing the action in a try block since we expect it to throw.

try

{

action();

}

// Catching the exception regardless of its type.

catch (Exception e)

{

// Comparing the type of the exception to the type of the type

// parameter, failing the assert in case they are different,

// even if the type of the exception is derived from the expected type.

if (e.GetType() != typeof(T))

throw new AssertFailedException(String.Format(

CultureInfo.CurrentCulture,

"ExceptionAssert.Throws failed. Expected exception type: {0}. Actual exception type: {1}. Exception message: {2}",

typeof(T).FullName,

e.GetType().FullName,

e.Message));

// Calling the validator for the exception object if one was

// provided by the caller. The validator is expected to use the

// Assert class for performing the verification.

if (validator != null)

validator((T)e);

// Type check passed and validator did not throw.

// Everything is fine.

return;

}

// Failing the assert since there was no exception.

throw new AssertFailedException(String.Format(

CultureInfo.CurrentCulture,

"ExceptionAssert.Throws failed. No exception was thrown (expected exception type: {0}).",

typeof(T).FullName));

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Pimp your VSTT exception tests

ddietric — Thu, 31 Jul 2008 13:40:00 GMT

Writing unit tests in VSTT is great since it comes with a nice unit testing framework and the test tools are neatly integrated into Visual Studio. There is one exception though: Exception testing (yes, that pun was intended). Consider the following example:

public class Foo

{

private object[] _array;

public Foo(object[] array)

{

if (array == null)

throw new ArgumentNullException();

if (array.Length == 0)

throw new ArgumentException();

_array = array;

}

public object GetItem(int index)

{

if (index < 0 || index >= _array.Length)

throw new ArgumentOutOfRangeException();

return _array[index];

}

This contrived example contains three throw statements requiring four distinct checks in order to hit all code paths which will cause an exception to be thrown (the if statement in GetItem(int) contains two blocks due to short-circuit evaluation). Since the way to test for exceptions in VSTT is the ExpectedException attribute we need one test method for each code path:

[TestMethod]

[ExpectedException(typeof(ArgumentNullException))]

public void ctor_array_null()

{

Foo foo = new Foo(null);

}

[TestMethod]

[ExpectedException(typeof(ArgumentException))]

public void ctor_array_empty()

{

Foo foo = new Foo(new object[0]);

}

[TestMethod]

[ExpectedException(typeof(ArgumentOutOfRangeException))]

public void GetItem_index_minus_one()

{

Foo foo = new Foo(new object[1]);

foo.GetItem(-1);

}

[TestMethod]

[ExpectedException(typeof(ArgumentOutOfRangeException))]

public void GetItem_index_array_length_plus_one()

{

Foo foo = new Foo(new object[1]);

foo.GetItem(1);

}

While there is nothing wrong with this in general it does bloat both the test code base as well as the test list a little bit. Also, if I need to go through a couple of complex initialization steps before actually triggering the exception I'll have to execute those for every single test method. Another more serious potential issue is that with this implementation I can only verify that an exception caused by the test method is of a certain type. But I can never know for sure that the statement I expected to throw actually did throw the exception without stepping through the code or paying close attention to my code coverage while writing the tests. The worst case scenario here is a bug in my test method that causes the expected exception to be thrown before the statement that was supposed to throw even gets executed. This could go unnoticed since the test would still pass.

Now, if you've worked with VSTT for a while you may have noticed that there are a couple of classes for assertions. The Assert class is probably the one that is most well-known. But there are also the classes CollectionAssert and StringAssert which complement the general purpose Assert class. What's missing is an ExceptionAssert class! So let's create one.

using Microsoft.VisualStudio.TestTools.UnitTesting;

using System;

public static class ExceptionAssert

{

public static void Throws(Type exceptionType, Action action)

{

if (exceptionType == null || action == null)

throw new ArgumentNullException();

if (exceptionType != typeof(Exception) && !exceptionType.IsSubclassOf(typeof(Exception)))

throw new ArgumentException();

try

{

action();

}

catch (Exception e)

{

if (e.GetType() != exceptionType)

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. Expected exception type: {0}. Actual exception type: {1}. Exception message: {2}",

exceptionType.FullName,

e.GetType().FullName,

e.Message));

return;

}

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. No exception was thrown (expected exception type: {0}).",

exceptionType.FullName));

}

Parameter validation aside, Throws(Type, Action) takes the type of the expected exception and a delegate of the type Action pointing to the code that is supposed to throw. This delegate is called in a try block since we expect it to throw and the corresponding catch block catches all managed exceptions (this a rare instance in which it is actually okay to do so). If an exception is caught but it is not of the expected type an AssertFailedException is thrown to signal the test failure to the test runtime which is also what for example the Assert class that ships with VSTT does if the condition is not met. Finally, if calling the delegate did not cause an exception we also throw an AssertFailedException this time with a different message. Using this new assertion method simplifies our test code for the constructor of the Foo class to this:

[TestMethod]

public void Constructor()

{

ExceptionAssert.Throws(typeof(ArgumentNullException),

delegate { new Foo(null); });

ExceptionAssert.Throws(typeof(ArgumentException),

delegate { new Foo(new object[0]); });

}

You may argue that this alone is not that much of a simplification but it's a start. The Throws() method above only takes delegates that don't have parameters and return void. What we need to do is add overloads for different kinds of delegates. I just want to show one more example for delegates taking one parameter and returning void. While the remaining test code I haven't replaced yet could also be written using anonymous methods this overload will allow me to demonstrate the usage of lambda expressions in the test method (for all those programmers out there who love them - I think I'm slowly turning into one myself).

public static void Throws<T>(Type exceptionType, T target, Action<T> action)

{

if (exceptionType == null || target == null || action == null)

throw new ArgumentNullException();

if (exceptionType != typeof(Exception) && !exceptionType.IsSubclassOf(typeof(Exception)))

throw new ArgumentException();

try

{

action(target);

}

catch (Exception e)

{

if (e.GetType() != exceptionType)

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. Expected exception type: {0}. Actual exception type: {1}. Exception message: {2}",

exceptionType.FullName,

e.GetType().FullName,

e.Message));

return;

}

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. No exception was thrown (expected exception type: {0}).",

exceptionType.FullName));

}

As you can see most of the new overload is exactly the same as before (differences in bold) and it allows us to replace the remaining tests with the following single test method:

[TestMethod]

public void GetItem()

{

object content = new object();

Foo foo = new Foo(new object[] { content });

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(-2));

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(-1));

Assert.AreEqual(content, foo.GetItem(0));

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(1));

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(2));

}

This new test method does not only add a regular assert in-between exception checks but it also calls GetItem(int) with two more invalid values without the need for adding additional test methods. As cool as this may (or may not) be, I'd like to close this post with a little warning because there is something I failed to mention before which I consider to be an advantage of the attribute-based approach:

When in doubt you should assume that an exception is thrown because of literally exceptional circumstances which may leave your objects under test in an undefined state. Using an attribute and catching the expected exception in the test runtime makes it more difficult for you to shoot yourself in the foot by reusing objects in your test method which may be in a bad state (or looking at it from the other side, my ExceptionAssert class makes it easier since allowing you to keep going after an exception is exactly the point). That said, I only recommend this approach of reusing instances for multiple exception checks when you know for sure that the objects under test are not in an undefined state after throwing an exception because the exception is part of parameter validation before any state changes occur or the object can reliably recover from the error condition for example by rolling back changes.

This posting is provided "AS IS" with no warranties, and confers no rights.