Dennis "D.C." Dietrich : Testing

DTACM (D.C.'s Test Automation Coding Manifesto)

ddietric — Sat, 21 Nov 2009 02:06:00 GMT

I find myself getting more and more frustrated whenever I see test code that is best described as quick and dirty. Now, sometimes I do acknowledge that the world is not perfect (has never been and will never be) so I realize that the following is really my personal ideal that might not fully work in the real world. However, I don't see why that should be an excuse for not trying to get as close as possible. Furthermore, I do realize that some of you will consider parts (or all) of this posting to be highly controversial. This is indeed on purpose and meant as an invitation to comment or, better yet, start a conversation.

So here is my personal set of rules for successfully building test infrastructure and tooling (e.g. runtime environments and automation frameworks) and automated tests:

Leadership

Every test team has a test architect (this can be an additional role for one of the testers on a team, especially if the number of team members is less than or equal to one). The responsibilities of the test architect include:

The overall architecture of the test infrastructure and automation
The quality of the code base
Defining and enforcing the development process used by the test team

Consequently, the test architect has the final say in all matters.

Specifications

All development starts by specifying the behavior of the software to be created in writing with the kind and scope of the specification depending on the chosen development process. An exception can be made only for projects completely owned by one individual if no other project depends on it. In that case it is acceptable to create the specification in parallel.

Code Quality

All code meets the following criteria:

It compiles without any warnings with the warning level on the highest setting
It passes static code analysis (e.g. FxCop) without any warnings
Suppressing individual occurrences of compiler or static code analysis warnings is acceptable if there is a technical reason for it and the reason is properly documented (preferably in code)
It is checked into a source control system
It is reviewed before check in
Test infrastructure and tooling code must be covered by an automated test suite with code paths that cannot (easily) be covered through automated tests covered through manual test cases or code inspection
It is commented on the type and member level and additional comments within member implementations should be added for all non-trivial implementations

Additional requirements can be imposed at the architect's discretion.

Reliability

The reliability of test infrastructure and tooling is proven by the associated test suite. Regression tests are added for all defects found.

Automated tests are reliable when there are no false positives and no false negatives. This implies that rerunning unreliable tests until they pass is unacceptable. Test failures are always tracked down to a product or test defect and fixed accordingly.

User Experience

Usability is a primary goal for all infrastructure and tooling development. User interfaces are as self-explanatory as possible; the output of all software clearly indicates success or failure. The output ends with a summary for software that outputs large amounts of information like test harnesses or deployment utilities. The summary contains enough information for enabling the user to fix all issues in case of failures. Note that this can be as simple as a URL to a website with step-by-step instructions. Any software that requires more than copying files from one location to another in order to install it or requires more than deleting files in one location in order to uninstall it ships as a proper setup package with uninstall capability.

Documentation

The documentation of any project contains the following:

User manual
Sustained engineering documentation

In addition, the documentation for any software that is extensible or a library includes an API reference and sample projects.

Legacy Code

Any legacy code that was written with lower or no quality standards is replaced or refactored.

This posting is provided "AS IS" with no warranties, and confers no rights.

All the wonderful colors of code coverage

ddietric — Thu, 22 Oct 2009 00:51:00 GMT

I'd like to share my colors with you:

	R	G	B
Coverage Not Touched Area	230	128	165
Coverage Partially Touched Area	191	210	249
Coverage Touched Area	180	228	180

These are the colors I prefer for code coverage in Visual Studio and if you've used the first Beta of Visual Studio 2005 they'll probably seem very familiar. Now, let's talk about what each of these (do not) mean and I have to admit that my choice for Coverage Touched Area is a bit "dangerous" since people often associate green with "everything okay".

Coverage Not Touched Area

This looks straightforward at first. There is code that hasn't been executed, that simple. So all we need to do is add test code that does execute the uncovered code paths. Not necessarily. Some people will tell you that you need 100% code coverage and this simply doesn't make sense. If you just happen to reach 100% without any major obstacles that's great. But in many projects you will hit scenarios that are not easily automatable up to the point where automation would be much more expensive than manual testing (so you hopefully won't do it). And it's even more likely that you'll hit an issue like this:

public void Foo()

{

switch (_someInternalState)

{

case SomeInternalState.State1:

// Do the right thing for state 1

break;

case SomeInternalState.State2:

// Do the right thing for state 2

break;

case SomeInternalState.State3:

// Do the right thing for state 3

break;

default:

throw new Exception("The internal state is invalid.");

}

Using the default case for raising an exception is a common pattern when there is no meaningful default behavior. If the component containing this code works as intended there is no way for the caller to get the component into an invalid state in the first place. In case of managed code you may be able to force the component into an invalid state but it's not always that easy (no private reflection because of trust level, unmanaged code, the code in question is actually invoked by an external process, etc.). Even when it is you need to ask yourself if it's worth it. The alternative is a code review in order to verify that the default case does exactly what it's supposed to do in the rare event that it gets executed and call it a day.

Long story short, uncovered paths require your attention. They must be covered by additional automated or manual tests or through code review/inspection. If do not create automated tests be sure to document the results of manual test passes or code reviews. Finally, if it turns out that the uncovered code is dead code remove it (thanks to source control there is no reason for keeping code around that is not needed).

Coverage Partially Touched Area

This tends to cause some confusion. How can a single statement be only partially executed? Let's consider the following example:

[TestMethod]

public void FooTest()

{

Assert.AreEqual("At least one parameter is not true.",

Class1.Foo(true, false));

}

public static string Foo(bool b1, bool b2)

{

if (b1 && b2)

return "Both parameters are true.";

return "At least one parameter is not true.";

}

While this might be somewhat intuitive since the && operator uses short-circuit evaluation the real and only reason for that line of code being marked as partially touched is that not all of its corresponding IL instructions were executed. This may be the case when the compiled code contains branch instructions like for example the IL code equivalent to the if statement above. What all of this implies is an important fact if you want to understand code coverage with Visual Studio: Code coverage is performed on the IL in the compiled assemblies which is also the reason while it is possible to end up with uncovered blocks although every line of code is covered according to the source editor.

The code coverage feature, however, doesn't know anything about the high-level language you are using. It could be C# but it could as well be [insert favorite language here]. This causes an interesting side-effect. The code coverage shown above and the IL shown below are from a debug build without optimization giving us a coverage of 5 out of 7 blocks.

.method public hidebysig static string Foo(bool b1, bool b2) cil managed

{

.maxstack 2

.locals init (

[0] string CS$1$0000,

[1] bool CS$4$0001)

L_0000: nop

L_0001: ldarg.0

L_0002: brfalse.s L_000a

L_0004: ldarg.1

L_0005: ldc.i4.0

L_0006: ceq

L_0008: br.s L_000b

L_000a: ldc.i4.1

L_000b: stloc.1

L_000c: ldloc.1

L_000d: brtrue.s L_0017

L_000f: ldstr "Both parameters are true."

L_0014: stloc.0

L_0015: br.s L_001f

L_0017: ldstr "At least one parameter is not true."

L_001c: stloc.0

L_001d: br.s L_001f

L_001f: ldloc.0

L_0020: ret

}

Now people probably think of performance when they hear optimization. But in this context you have to remember that optimization usually means generating code that is faster to execute rather than trying to preserve the exact structure of the original source code. That said, let's see what happens when we take the previous example and rerun on a build with optimization turned on.

public static string Foo(bool b1, bool b2)

{

if (b1 && b2)

return "Both parameters are true.";

return "At least one parameter is not true.";

}

All of a sudden the code coverage result is 3 out of 4 blocks covered although we haven't changed either the product or the test code. Turns out the compiler has rewritten the code in order to optimize it. Though the generated code is completely equivalent to the original, it messes with our results and defies our expectation:

.method public hidebysig static string Foo(bool b1, bool b2) cil managed

{

.maxstack 8

L_0000: ldarg.0

L_0001: brfalse.s L_000c

L_0003: ldarg.1

L_0004: brfalse.s L_000c

L_0006: ldstr "Both parameters are true."

L_000b: ret

L_000c: ldstr "At least one parameter is not true."

L_0011: ret

}

So, unoptimized builds may be more useful for getting code coverage data since the IL is closer to the original source than that of optimized builds. In other words, the same reasons that make it easier to debug builds that are not optimized also make it easier to understand code coverage numbers. In any case, if you have any partially touched areas identify which parts exactly are not covered and then treat them like any other untouched area.

Coverage Touched Area

[TestMethod]

public void FooTest()

{

Class1.Foo();

}

public static string Foo()

{

return "This is a hardcoded string.";

}

Looking at the code above it probably seems painfully obvious what's going on. Method Foo() is fully covered but since the test method is not doing any verification the only thing this tells us is that Foo() doesn't cause an unhandled exception. While this is important it is probably not what was intended. The problem with real tests is that usually there a couple of calls into the product and a couple of verification steps to check state and return values. Whether or not we've hit a certain code block is something code coverage tells us. But making sure that we have verified everything that needs to be verified is something the machine can't do for us which should be encouragement to double-check test code (ideally through code review) before trusting it.

This posting is provided "AS IS" with no warranties, and confers no rights.

EMTF 2.0 Beta released

ddietric — Tue, 14 Jul 2009 13:04:00 GMT

It's been a couple of months since I first blogged about EMTF. Since then I've been busy refining and extending the framework and I'm happy to announce that the beta release of version 2.0 is finally done! Special thanks to Josh Poley since a lot of the new functionality is based on his feedback. So, allow me to give you a tour of what's new in ~~Reality~~ EMTF 2.0:

Pre- and post-test actions

The PreTestActionAttribute and PostTestActionAttribute attributes have been added to allow defining methods that are executed before/after each test method in a test class. This is most commonly used for shared setup and teardown of the environment required by a set of tests.

[TestClass]

public class Tests

{

// Will be executed before every test in the Tests class

[PreTestAction]

public void PreTestAction() { }

// Will be executed after every passed test in the Tests class

[PostTestAction]

public void PostTestAction() { }

}

Aborting tests, custom logging and the TestContext

Test methods as well as pre- and post-test actions can take one parameter of the type TestContext. If they do an instance will be automatically provided by the runtime. The test context allows aborting a test from within the test implementation e.g. in case test initialization fails. It also allows adding arbitrary information to the test log.

[Test]

public void Test(TestContext context)

{

context.LogLine("Initialization failed");

context.AbortTest();

}

Test groups

The TestGroupsAttribute attribute allows tagging test methods as belonging to one or multiple test groups. At the same time, there are new TestExecutor.Execute() overloads which limit test execution to tests belonging to the specified group(s).

[TestClass]

public class Tests

{

[Test]

[TestGroups("BVT")]

public void Bvt() { }

[Test]

[TestGroups("BVT", "STRESS")]

public void StressBvt() { }

[Test]

[TestGroups("STRESS")]

public void Stress() { }

}

class Program

{

static void Main(string[] args)

{

// Will only run Bvt() and StressBvt()

new TestExecutor().Execute(new String[] { "BVT" });

}

Concurrent test runs

EMTF now supports concurrent test runs in order to take advantage of multi-core CPUs. Concurrent test runs will execute tests on multiple threads instead of just one. The number of threads used is equal to the number of logical processors on the local machine.

TestExecutor executor = new TestExecutor();

executor.ConcurrentTestRuns = true;

executor.Execute();

Plain text logging to a stream

The logging library has also been extended and now includes the StreamLogger which writes a plain text log to a stream.

TestExecutor executor = new TestExecutor();

using (Stream stream = new FileStream("testrun.log", FileMode.Create))

{

StreamLogger logger = new StreamLogger(executor, stream);

executor.Execute();

logger.Close();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Creating an API testing framework 101

ddietric — Wed, 25 Feb 2009 07:40:00 GMT

Writing a fully featured API testing solution is a lot of work. You have to deal with things like test case management, integration with code coverage tools or test execution on server farms. However, the core of a (reflection-based) API testing framework is quite simple. This post explains the basics with EMTF as an example.

Implementing asserts

Assertions are the building blocks for actual API test cases (my previous post Introducing EMTF contains a simple example). A single failing assertion fails the whole test and causes the execution of the current test method to be cancelled. How does a method that doesn't even return a value achieve this? Fairly simple: it throws an exception.

[DebuggerHidden]

public static void IsTrue(Boolean condition, String message)

{

if (!condition)

throw new AssertException("Assert.IsTrue failed.", message);

}

Since it throws a specific exception type, a failed assertion can be distinguished from a crashing test method based on the type of the exception. But isn't it bad to use exceptions to return from an operation? It usually is and this is one of the few exceptions to this rule. First of all, a failing test is (hopefully) still an exceptional event. Usually we want and expect our tests to pass. Second, there is no better way to allow the test runtime to cancel the execution of a test method when an assertion fails.

Another little detail is the DebuggerHiddenAttribute on all methods of the Assert class (Assert.cs) as well as the public constructors of the AssertException class (AssertException.cs). This tells the debugger that the methods/constructors marked with this attribute should be omitted from stack traces so they won't show up when debugging assertion failures.

Identifying test classes and test methods

A common way of marking classes as test classes and methods as test methods is the use of attributes. This allows the test runtime to automatically find and execute all tests and has the advantage of keeping the object model of the test suite flexible. An alternative is to define a base class or interface that test classes must derive from or implement respectively and treating all public, non-abstract, non-generic, void returning, parameter less methods as test method. Though this simplifies both the test code and the code that finds all test methods through reflection a little bit it comes at the expense of pretty much dictating what a test class looks like.

EMTF contains the type TestClassAttribute (TestClassAttribute.cs) for test classes and TestAttribute (TestAttribute.cs) for test methods. These attributes can contain additional (optional) information or be supplemented by additional attributes. EMTF's TestAttribute for example has a property for the test description.

[AttributeUsage(AttributeTargets.Class)]

public sealed class TestClassAttribute : Attribute

{

}

[AttributeUsage(AttributeTargets.Method)]

public sealed class TestAttribute : Attribute

{

private String _description;

public String Description

{

get

{

return _description;

}

public TestAttribute()

{

}

public TestAttribute(String description)

{

_description = description;

}

Finding all tests

private static Collection<MethodInfo> FindTestMethods()

{

Collection<MethodInfo> testMethods = new Collection<MethodInfo>();

// Iterate through all assemblies in the current application domain

foreach (Assembly assembly in AppDomain.CurrentDomain.GetAssemblies())

{

// Iterate through all exported types in each assembly

foreach (Type type in assembly.GetExportedTypes())

{

// Verify that the type is:

// 1. Marked with the TestClassAttribute

// 2. Not a generic type definition or open constructed type

// 3. Not abstract

if (type.IsDefined(typeof(TestClassAttribute), true) &&

!type.ContainsGenericParameters &&

!type.IsAbstract)

{

// Iterate through all public instance methods

foreach (MethodInfo method in type.GetMethods(BindingFlags.Instance | BindingFlags.Public))

{

// Verify that the method is:

// 1. Marked with the TestAttribute

// 2. Not a generic method definition or open constructed method

// 3. Its return type is void

// 4. Does not have any parameters

if (method.IsDefined(typeof(TestAttribute), true) &&

!method.ContainsGenericParameters &&

method.ReturnType == typeof(void) &&

method.GetParameters().Length == 0)

{

// Add method to the list of

// test methods to execute

testMethods.Add(method);

}

return testMethods;

}

Executing the tests

The last piece missing is a method that executes all test methods and logs the results or at least raises events which can be handled by the caller or a dedicated logger. The basic logic for test execution used by EMTF's TestExecutor (TestExecutor.cs) and other API testing runtimes looks like this:

)" alt="TestExecutor.ExecuteImpl(IEnumerable)" align=baseline src="http://blogs.msdn.com/photos/ddietric/images/9443552/original.aspx" mce_src="http://blogs.msdn.com/photos/ddietric/images/9443552/original.aspx">

private void ExecuteImpl(IEnumerable<MethodInfo> testMethods)

{

// Raise event signaling the start of a test run

OnTestRunStarted();

// Instance of the current test class

Object currentInstance = null;

// Iterate through all test methods

foreach (MethodInfo method in testMethods)

{

// Get the TestAttribute on the method

object[] testAttribute = method.GetCustomAttributes(typeof(TestAttribute), true);

string testDescription = null;

// Get the test description if TestAttribute is defined on the method

if (testAttribute.Length > 0)

testDescription = ((TestAttribute)testAttribute[0]).Description;

// Try to instantiate the test class if necessary

// Skip the test if an instance cannot be created

if (!TryUpdateTestClassInstance(method, testDescription, ref currentInstance))

continue;

// Raise an event signaling the start of a test

OnTestStarted(new TestEventArgs(method, testDescription));

// Invoke the test method in a try block so we can

// catch assert failures and unexpected exceptions

try

{

method.Invoke(currentInstance, null, true);

}

// Assert failed

catch (AssertException e)

{

// Raise event signaling the test failure

// then immediately run the next test

OnTestCompleted(

new TestCompletedEventArgs(

method,

testDescription,

e.Message,

e.UserMessage,

TestResult.Failed,

null));

continue;

}

// Test threw unexpected exception

catch (Exception e)

{

// Raise event signaling the test failure

// then immediately run the next test

OnTestCompleted(

new TestCompletedEventArgs(

method,

testDescription,

String.Format(

CultureInfo.CurrentCulture,

"An exception of the type '{0}' occurred during the execution of the test.",

e.GetType().FullName),

null,

TestResult.Exception,

e));

continue;

}

// No exceptions were thrown

// Raise event signaling that the test passed

OnTestCompleted(

new TestCompletedEventArgs(

method, testDescription, "Test passed.", null, TestResult.Passed, null));

}

// Raise event signaling the completion of the test run

OnTestRunCompleted();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Introducing EMTF

ddietric — Tue, 17 Feb 2009 10:45:00 GMT

A while ago a needed to do what I like to call quick and dirty testing, i.e. getting as much test coverage as possible in a given period of time. To make things worse, I was working on a client-server feature requiring a bit of setup in order to work in the first place and wanted to cover both sides with my tests and be able to easily run all tests. So I thought, what if instead of trying to get this done with one of the existing test frameworks that typically host the test target as well as its tests, I just created a new solution which already had all the pieces in place to use the feature and put a tiny test runtime in there. I did exactly that and got my testing done the way I wanted to. Again, there is a reason why I call this kind of approach quick and dirty testing and I definitely wouldn't pitch it as a best practice but it has its place.

Because of my objective this "embedded" test runtime I created was extremely small but also highly specific to the test scenarios for which it was written and thus virtually impossible to reuse. That wasn't a surprise but rather one of the tradeoffs I had opted in for. Still, I was wondering what a generalized version would look like that, next time I wanted to do something like this, I could simply drop into my application and get going. The answer to that question is EMTF.

Yet another test framework called EMTF

The Embeddable Micro Test Framework is similar to its big brothers MSTest or NUnit though obviously smaller and simpler and with the twist of it being supposed to be hosted by the application under test rather than coming with its own test host. It is available in source and binary form on CodePlex and by now I'd say that its greatest value is actually in using it for educational purposes since the reduced complexity makes it easy to understand for people without prior API testing experience (that goes for both how to use it and - at the core - how it works). That said the rest of this post is an EMTF quickstart (though I do consider a small series on how to implement an API test framework using EMTF as an example).

The easiest way to add EMTF to your project (especially if you're not using C#) is to add references to the two assemblies Emtf.dll and Emtf.Logging.dll. The first assembly contains the actual test framework that doesn't even include any logging capabilities. The second one contains a logging base class, a logger writing to the debug output and a logger writing to the console (the latter one only for the .NET Framework). There are separate versions of the two assemblies for Silverlight and the .NET Framework with the Silverlight version being a subset.

Writing tests

Let's say your project is a console application, you added EMTF to it and now want to test the following class which is part of your project (once again, please ignore the fact that the class is completely useless):

public class LameExample

{

private String _string;

public String String

{

get

{

return _string;

}

set

{

if (value == String.Empty)

throw new ArgumentException(

"New value must not be an empty string.",

"value");

_string = value;

}

We need to verify that:

The default value of the String property is null
The setter of the String property throws an ArgumentException if we pass in String.Empty
We can set the property to arbitrary values (except for String.Empty)

Ideally we want the test framework to automatically find all the tests and execute them. So in order to tell EMTF which classes contain tests, they need to be marked with the TestClassAttribute. Furthermore, test classes must be public, non-abstract and non-generic since the runtime wouldn't know which type parameters to use. Similarly, test methods are marked with the TestAttribute and must be public, non-abstract and non-generic. In addition, their return type must be void and they must not take any parameters. Tests for state and behavior of an object are performed using assertions.

[TestClass]

public class LameExampleTests

{

[Test]

public void StringProperty()

{

LameExample le = new LameExample();

// Verifies the default value of LameExample.String

Assert.IsNull(le.String);

// Verifies that the setter throws

Assert.Throws(

// Action that is expected to cause the exception

() => le.String = String.Empty,

// Action(T) verifying the exception object where

// T is the type of the expected exception.

(ArgumentException e) =>

{

Assert.IsTrue(e.Message.StartsWith("New value must not be an empty string."));

Assert.AreEqual("value", e.ParamName);

});

// Verifies setting the property to a non-null,

// non-empty string.

le.String = "fhqwhgads";

Assert.AreEqual("fhqwhgads", le.String);

// Verifies resetting the property to null.

le.String = null;

Assert.IsNull(le.String);

}

Running the tests

Since EMTF does not include any sort of test host we need to instantiate the engine, the TestExecutor, ourselves. In addition, we need to hookup the engine's events so we actually get to know the outcome of the test run. The loggers in Emtf.Logging.dll do this out of the box. Finally, calling TestExecutor.Execute() will execute all the tests we have defined that conform to the rules explained above.

private static void RunTests()

{

TestExecutor executor = new TestExecutor();

ConsoleLogger logger = new ConsoleLogger(executor);

executor.Execute();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

The ultimate ExceptionAssert.Throws() method

ddietric — Wed, 07 Jan 2009 10:15:00 GMT

A while ago I published a post titled Pimp your VSTT exception tests in which I pointed out some disadvantages of VSTT's declarative approach to testing for exceptions and provided an alternative. I recently gave this some more thought while writing some new tests and noticed that my original version could be significantly improved by a simple tweak leading to the ultimate version of the Throws() methods (still, don't be too surprised if I publish a post titled The ultimate ExceptionAssert.Throws() method v2.0 at some point).

First of all, when writing my original post it seemed like having overloads would be a good idea because of how anonymous methods are treated by the compiler and the potential side-effects. Looking back at it I'd say that was an overly cautious approach. In the tests I've written since I published the post there wasn't any instance where accessing fields and locals would cause problems. The second point is the really interesting one though. Consider the following code:

public static void Foo(int i)

{

if (i < 0)

throw new ArgumentOutOfRangeException(

"i",

"The value of parameter 'i' must not be less than zero.");

}

When testing whether or not the method throws as expected, the test code will often just check if an exception of the correct type was thrown. If the code is part of an application and only used internally this may actually be enough and just checking the type is somewhat encouraged by how it is achieved in VSTT with the ExpectedExceptionAttribute. The same is true for my original Throws() method. But if you're building components for others to use in their applications you certainly want to make sure that the exception object looks exactly the way it was specified.

In the sample above you would want to verify the properties ActualValue, Message and ParamName. But generally speaking the approach to exception testing used when creating libraries must allow the tester to perform arbitrary validation on the exception object. While Throws() could for example simply return the exception it caught - probably the simplest solution - this would be not very consistent with how a typical assert method works. It is void-returning and only returns if the assert passed. A slightly more difficult approach which is equally flexible in this context leads to more consistent test methods. Since we're already dealing with delegates for telling Throws() what code exactly should throw, why not pass in a second delegate which verifies the exception object using standard assert methods so that a test for the method above would look like this:

[TestMethod]

public void TestMethod()

{

ExceptionAssert.Throws(

() => MyClass.Foo(Int32.MinValue),

(ArgumentOutOfRangeException e) =>

{

Assert.AreEqual("i", e.ParamName);

Assert.AreEqual(Int32.MinValue, e.ActualValue);

Assert.IsTrue(e.Message.StartsWith("The value of parameter 'i' must not be less than zero."));

});

}

The biggest disadvantage is that test methods will probably rely on anonymous methods in many cases to keep the code compact and make it easy to follow the control flow but it can take a while get used to reading/writing this kind of code. Still, if you want/need to go deeper than just looking at exception types I'd say that this is a good way of doing it and - unless I've overlooked something (again) - the only Throws() method you'll ever need (especially since the second parameter is optional).

public static class ExceptionAssert

{

// The type parameter is used for passing in the expected exception type.

// The where clause is used to ensure that T is Exception or a subclass.

// Thus an explicit check is not necessary.

public static void Throws<T>(Action action, Action<T> validator) where T : Exception

{

if (action == null)

throw new ArgumentNullException("action");

// Executing the action in a try block since we expect it to throw.

try

{

action();

}

// Catching the exception regardless of its type.

catch (Exception e)

{

// Comparing the type of the exception to the type of the type

// parameter, failing the assert in case they are different,

// even if the type of the exception is derived from the expected type.

if (e.GetType() != typeof(T))

throw new AssertFailedException(String.Format(

CultureInfo.CurrentCulture,

"ExceptionAssert.Throws failed. Expected exception type: {0}. Actual exception type: {1}. Exception message: {2}",

typeof(T).FullName,

e.GetType().FullName,

e.Message));

// Calling the validator for the exception object if one was

// provided by the caller. The validator is expected to use the

// Assert class for performing the verification.

if (validator != null)

validator((T)e);

// Type check passed and validator did not throw.

// Everything is fine.

return;

}

// Failing the assert since there was no exception.

throw new AssertFailedException(String.Format(

CultureInfo.CurrentCulture,

"ExceptionAssert.Throws failed. No exception was thrown (expected exception type: {0}).",

typeof(T).FullName));

}

This posting is provided "AS IS" with no warranties, and confers no rights.

It's all about systems - even if you're testing only a component

ddietric — Thu, 23 Oct 2008 05:00:00 GMT

Earlier this year I bought a brand new computer and wanted to play what turned out to be one of my favorite games of all times. However, it also turned out that this new computer of mine came with a pretty bad graphics card. It did work but the performance was so bad that even full screen DVD playback was impacted. So I got a new one, installed it and was ready to play. Except for the fact that the game which ran fine before crashed now during initialization. I did a quick search on the web and found reports from other users hitting a similar if not the same issue. I thought about sending an email to the developer but noticed that since others had already posted about this on the support forum it would be nothing more than a "me too" mail. Unless I provided some additional information and fortunately I had already installed Visual Studio on that machine. So I started the debugger and tried to gather some information about the crash. The first thing I saw was this:

Unhandled exception at 0x6e75f0ec (d3d8.dll) in SamMax101.exe:

0xC0000005: Access violation reading location 0x00000000.

Looked like someone was dereferencing a null pointer and then tried to read from that memory location. I did not pay enough attention to the module in which this occurred. If I had I could already have been done at this point but I kept going and checked the stack traces. There were seven threads of which only the main thread did something useful and it was also the thread on which the access violation occurred.

d3d8.dll!CD3DHal::CreateVertexShaderI() + 0x24c bytes

d3d8.dll!CD3DBase::CreateVertexShader() + 0x33e bytes

SamMax101.exe!005cc3a6()

[Frames below may be incorrect and/or missing, no symbols loaded for SamMax101.exe]

SamMax101.exe!0070a4b8()

SamMax101.exe!005cc79d()

SamMax101.exe!007a4aed()

SamMax101.exe!00556619()

SamMax101.exe!00730061()

kernel32.dll!_FindFirstFileA@8()

7ffdfbf8()

ntdll.dll!@RtlpAllocateHeap@20() + 0x3ce bytes

SamMax101.exe!00592208()

user32.dll!_NtUserPeekMessage@20() + 0xc bytes

user32.dll!__PeekMessage@24() + 0x2d bytes

The last two frames on the stack refer to functions in Direct3D of DirectX 8 suggesting that the game might be innocent after all (and it was). The function in which the access violation occurred was CreateVertexShaderI() according to the trace. Even without knowing all the details it's obvious that creating shaders involves the graphics card driver so there actually was another potential source for the issue that wasn't even in the stack trace. An inner voice finally said "update the graphics card driver, Luke" and after doing so the game worked flawlessly again (and with the new card also with a very high frame rate).

A couple of months later I received an email about Visual Studio crashing with an access violation when attempting to create a new Silverlight or WPF project. That sounded a bit strange so I asked for a stack trace and the trace I got back contained the following frame:

d3d9!CD3DDDIDX10TL::CreateVertexShaderFunc+0x88

With this piece of the puzzle the issue started looking similar to the game crashing because of a bad driver. So I suggested updating the graphics card driver and that once again did fix the issue. But why am I telling you all this. Well...

It's all about systems
First, let's go back to the title of this post. As testers we usually own a specific feature (area) that we focus on and there is nothing wrong with that - a complex product can't be tested by one person alone. But it is important to have an understanding of the components that your feature relies on. And of the components those components rely on and so on going down all the way to the operating system. Of course you cannot have a deep understanding of everything in a software stack down to the OS. The "further away" you are from the components you own the more basic/general your understanding of that part of the system will be. Still, any kind of knowledge you have will be an advantage. Going back to the sample of Visual Studio crashing, I have to admit I was irritated at first. But if you happen to know that WPF uses DirectX for rendering it's not surprising anymore that trying to use the WPF designer (which starts automatically when you create a new WPF application) can bring down VS in case of a bad driver. Ultimately this is about being able to determine the root cause of an issue which is essential for logging good bugs because otherwise a tester will log the issue against the feature through which the issue surfaced. That in turn puts an unnecessary burden on the developer to determine the actual cause. In my example above the developer might not even be able to repro the issue on his/her machine.
Good testers need broad debugging skills
I'm a "managed developer/tester". I love the .NET Framework and Silverlight and I try not to touch unmanaged code unless I really have to. But when you look at the software we use every day - starting with the OS itself - there is still a lot of unmanaged code in the mix which hosts our managed solutions and with which we interface/interop. Again my point is not to say that you should or even can be an expert in all possible fields. But because of the circumstances you need to be able to debug into the managed and the unmanaged parts of a system to determine the root cause of an issue or at least narrow it down. Just being able to figure out the area in which an issue originally occurred is pretty useful since it will allow you to take all the information you already gathered and hand it over to the person who actually owns the corresponding feature. More generally speaking - and also including debugging of managed code that is simply owned by someone else - testers need to be comfortable debugging into "foreign" (parts of the) codebase(s) in order to write detailed and accurate bug reports even if the issue extends beyond the features assigned to said tester.
Noticing patterns in defects
Granted, this post may not contain the best example. Either way, recognizing patterns is important as it can save you a lot of time. In my examples it was just about an access violation happening somewhere during vertex shader creation and your reoccurring issues may be more complex. The point is that if you have similar looking issues and you've already found a solution for one of them it's definitely worth checking if the same solution applies of the other ones.

This posting is provided "AS IS" with no warranties, and confers no rights.

Which Microsoft virtualization solution to use for software testing

ddietric — Mon, 29 Sep 2008 13:35:00 GMT

I have to admit the title is a bit misleading this time since - if you ask me - there is only one answer: Hyper-V! It is the first Microsoft virtualization product to support 64-bit guest operating systems, SMP for some guests (see Supported Guest Operating Systems for a full list) and with Enlightened I/O it is incredibly fast. The only problem - and also the real reason for writing this post - is that you can only get Hyper-V as part of Windows Server 2008 or as the new Hyper-V Server 2008. So what if that is not an option?

The simplest alternative is good old Virtual PC. It certainly gets the job done but I really appreciate the option of being able to use Remote Desktop with all my machines no matter if they are physical or virtual. While nothing is preventing me from connecting to a VM running in Virtual PC through Remote Desktop, Virtual PC itself is "just" a regular application. Or rather, it is not a service, meaning that you cannot logout as long as you still need that VM running.

Fortunately, there is still Virtual Server which - like any well-behaved server - runs as a service. That addresses what I dislike about Virtual PC but now I got a different problem. The standard management solution for Virtual Server is web-based which of course means that you need at least one IIS instance in order to configure Virtual Server and manage your VMs. This is - for the purpose of testing applications in VMs - an unnecessary resource penalty. Even worse, if your host OS is Windows Vista setting IIS up so the Virtual Server management website will actually work is quite painful. Good thing no one - not even the installer - is forcing us to install the Virtual Server Web Application as the feature is officially called. So let's not.

So if you install with these options, how do you actually use your Virtual Server? This is where a little unsupported freeware tool comes in. Since it's unsupported it's not suitable for production (then again, so is running Virtual Server on Vista anyway) but it's perfect for testing. It's called VMRCplus (the full name is Virtual Machine Remote Control Client Plus... but let's be honest, that's simply too long), available for download from the Microsoft Download Center and while it does not cover the full functionality of the web application it most certainly includes everything you typically need (just remember to use Run as administrator on Vista).

This posting is provided "AS IS" with no warranties, and confers no rights.

A poor man's approach to testing with databases

ddietric — Fri, 26 Sep 2008 09:12:00 GMT

Say you are working on software that accesses a database server and you need to test it. To some extent you'll probably try to not actually hit the database with your tests by using a mock. However, this is not always possible, especially when we're dealing with integration/acceptance tests which need to verify the system as a whole. In case your software only reads from a database there isn't really an issue. You setup a database server, create the required test database, the actual tests and start doing your test passes. Unfortunately, it's not that simple anymore as soon as your software changes the data in the database. While there are approaches that perform actions regardless of the state of the database and include verification steps that are able to analyze the resulting state changes this comes at a cost and might be overkill for your project.

So what to do if you want to have tests that are comparable to unit tests in terms of the level of complexity (just another way of saying, you want to keep your test code simple)? Referring to unit tests already implies that each test requires the software we are testing to be in a certain state. This requirement of course includes the state of the database. As long as we have a backup of the test database in its initial state we can ensure this by simply restoring the database form the backup during the initialization phase of every test. At this point I'd like to point out that this does not make sense for large databases (based on the time required for a full database restore) since the overall execution time will be unacceptable.

Restoring a database can be done in T-SQL allowing us to create a stored procedure for this purpose so the test code does not contain any file paths which may be specific to the test server used. The following example creates such a stored procedure in the master database (though it should be obvious: this is a bad idea in a production environment):

USE [master]

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

CREATE PROCEDURE [dbo].[sp_Restore_Northwind]

BEGIN

RESTORE DATABASE Northwind

FROM DISK = 'C:\Database Backups\Northwind.bak'

WITH FILE = 1,

MOVE 'Northwind' TO 'C:\Databases\Northwind.mdf',

MOVE 'Northwind_log' TO 'C:\Databases\Northwind_log.ldf',

RECOVERY;

END

Then we can add a test initialization routine to our test suite that executes the stored procedure so the database is always in the same state before each test no matter whether or not the previous test has altered its content. The following example uses the .NET Framework Data Provider for SQL Server.

[TestInitialize]

public void ResetDatabase()

{

using (SqlConnection connection = new SqlConnection(Settings.Default.MasterConnectionString))

{

connection.Open();

using (SqlCommand command = new SqlCommand("EXEC sp_Restore_Northwind", connection))

command.ExecuteNonQuery();

connection.Close();

}

And we're done! Except for a small problem. The .NET Framework Data Provider for SQL Server uses a connection pool to improve performance. This unfortunately gets in our way. The first test - including its initialization - will run just fine. The second one however will fail since, because the data provider uses a connection pool, there is still an active connection to the test database by the time we are trying to restore it again. The end result is that the second initialization will timeout. Since the problem is caused by connection pooling it can be solved by disabling it which can be done without any code changes by adding Pooling=False to the connection string (thanks to Mathew Charles for pointing this out):

Data Source=DbServerName;Initial Catalog=Northwind;Integrated Security=True;Pooling=False

This posting is provided "AS IS" with no warranties, and confers no rights.

Pimp your VSTT exception tests

ddietric — Thu, 31 Jul 2008 13:40:00 GMT

Writing unit tests in VSTT is great since it comes with a nice unit testing framework and the test tools are neatly integrated into Visual Studio. There is one exception though: Exception testing (yes, that pun was intended). Consider the following example:

public class Foo

{

private object[] _array;

public Foo(object[] array)

{

if (array == null)

throw new ArgumentNullException();

if (array.Length == 0)

throw new ArgumentException();

_array = array;

}

public object GetItem(int index)

{

if (index < 0 || index >= _array.Length)

throw new ArgumentOutOfRangeException();

return _array[index];

}

This contrived example contains three throw statements requiring four distinct checks in order to hit all code paths which will cause an exception to be thrown (the if statement in GetItem(int) contains two blocks due to short-circuit evaluation). Since the way to test for exceptions in VSTT is the ExpectedException attribute we need one test method for each code path:

[TestMethod]

[ExpectedException(typeof(ArgumentNullException))]

public void ctor_array_null()

{

Foo foo = new Foo(null);

}

[TestMethod]

[ExpectedException(typeof(ArgumentException))]

public void ctor_array_empty()

{

Foo foo = new Foo(new object[0]);

}

[TestMethod]

[ExpectedException(typeof(ArgumentOutOfRangeException))]

public void GetItem_index_minus_one()

{

Foo foo = new Foo(new object[1]);

foo.GetItem(-1);

}

[TestMethod]

[ExpectedException(typeof(ArgumentOutOfRangeException))]

public void GetItem_index_array_length_plus_one()

{

Foo foo = new Foo(new object[1]);

foo.GetItem(1);

}

While there is nothing wrong with this in general it does bloat both the test code base as well as the test list a little bit. Also, if I need to go through a couple of complex initialization steps before actually triggering the exception I'll have to execute those for every single test method. Another more serious potential issue is that with this implementation I can only verify that an exception caused by the test method is of a certain type. But I can never know for sure that the statement I expected to throw actually did throw the exception without stepping through the code or paying close attention to my code coverage while writing the tests. The worst case scenario here is a bug in my test method that causes the expected exception to be thrown before the statement that was supposed to throw even gets executed. This could go unnoticed since the test would still pass.

Now, if you've worked with VSTT for a while you may have noticed that there are a couple of classes for assertions. The Assert class is probably the one that is most well-known. But there are also the classes CollectionAssert and StringAssert which complement the general purpose Assert class. What's missing is an ExceptionAssert class! So let's create one.

using Microsoft.VisualStudio.TestTools.UnitTesting;

using System;

public static class ExceptionAssert

{

public static void Throws(Type exceptionType, Action action)

{

if (exceptionType == null || action == null)

throw new ArgumentNullException();

if (exceptionType != typeof(Exception) && !exceptionType.IsSubclassOf(typeof(Exception)))

throw new ArgumentException();

try

{

action();

}

catch (Exception e)

{

if (e.GetType() != exceptionType)

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. Expected exception type: {0}. Actual exception type: {1}. Exception message: {2}",

exceptionType.FullName,

e.GetType().FullName,

e.Message));

return;

}

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. No exception was thrown (expected exception type: {0}).",

exceptionType.FullName));

}

Parameter validation aside, Throws(Type, Action) takes the type of the expected exception and a delegate of the type Action pointing to the code that is supposed to throw. This delegate is called in a try block since we expect it to throw and the corresponding catch block catches all managed exceptions (this a rare instance in which it is actually okay to do so). If an exception is caught but it is not of the expected type an AssertFailedException is thrown to signal the test failure to the test runtime which is also what for example the Assert class that ships with VSTT does if the condition is not met. Finally, if calling the delegate did not cause an exception we also throw an AssertFailedException this time with a different message. Using this new assertion method simplifies our test code for the constructor of the Foo class to this:

[TestMethod]

public void Constructor()

{

ExceptionAssert.Throws(typeof(ArgumentNullException),

delegate { new Foo(null); });

ExceptionAssert.Throws(typeof(ArgumentException),

delegate { new Foo(new object[0]); });

}

You may argue that this alone is not that much of a simplification but it's a start. The Throws() method above only takes delegates that don't have parameters and return void. What we need to do is add overloads for different kinds of delegates. I just want to show one more example for delegates taking one parameter and returning void. While the remaining test code I haven't replaced yet could also be written using anonymous methods this overload will allow me to demonstrate the usage of lambda expressions in the test method (for all those programmers out there who love them - I think I'm slowly turning into one myself).

public static void Throws<T>(Type exceptionType, T target, Action<T> action)

{

if (exceptionType == null || target == null || action == null)

throw new ArgumentNullException();

if (exceptionType != typeof(Exception) && !exceptionType.IsSubclassOf(typeof(Exception)))

throw new ArgumentException();

try

{

action(target);

}

catch (Exception e)

{

if (e.GetType() != exceptionType)

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. Expected exception type: {0}. Actual exception type: {1}. Exception message: {2}",

exceptionType.FullName,

e.GetType().FullName,

e.Message));

return;

}

throw new AssertFailedException(String.Format(

"ExceptionAssert.Throws failed. No exception was thrown (expected exception type: {0}).",

exceptionType.FullName));

}

As you can see most of the new overload is exactly the same as before (differences in bold) and it allows us to replace the remaining tests with the following single test method:

[TestMethod]

public void GetItem()

{

object content = new object();

Foo foo = new Foo(new object[] { content });

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(-2));

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(-1));

Assert.AreEqual(content, foo.GetItem(0));

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(1));

ExceptionAssert.Throws(typeof(ArgumentOutOfRangeException), foo, f => f.GetItem(2));

}

This new test method does not only add a regular assert in-between exception checks but it also calls GetItem(int) with two more invalid values without the need for adding additional test methods. As cool as this may (or may not) be, I'd like to close this post with a little warning because there is something I failed to mention before which I consider to be an advantage of the attribute-based approach:

When in doubt you should assume that an exception is thrown because of literally exceptional circumstances which may leave your objects under test in an undefined state. Using an attribute and catching the expected exception in the test runtime makes it more difficult for you to shoot yourself in the foot by reusing objects in your test method which may be in a bad state (or looking at it from the other side, my ExceptionAssert class makes it easier since allowing you to keep going after an exception is exactly the point). That said, I only recommend this approach of reusing instances for multiple exception checks when you know for sure that the objects under test are not in an undefined state after throwing an exception because the exception is part of parameter validation before any state changes occur or the object can reliably recover from the error condition for example by rolling back changes.

This posting is provided "AS IS" with no warranties, and confers no rights.

Unit testing non-public types using reflection

ddietric — Fri, 20 Jun 2008 05:35:00 GMT

Let's say you want to unit test the following class:

using System;

internal class Foo

{

internal Foo(object parameter)

{

if (parameter == null)

throw new ArgumentException();

}

internal int ReturnZero()

{

return 1;

}

First of all, besides being pointless, it is buggy. The constructor should throw an ArgumentNullException and ReturnZero() should return 0. More importantly for this post though, it is not public. Now, you could sign your test assembly and use the InternalsVisibleTo attribute; however, that won't help you with types and members that are declared as private or members declared as protected. But I love reflection and with reflection I can call into anything I want though the code looks more complicated than it should. That said, I may decide to write the following test code to be executed by MSTest.exe/VSTT.

using Microsoft.VisualStudio.TestTools.UnitTesting;

using System;

using System.Reflection;

[TestClass]

public class FooTests

{

private Type fooType;

private ConstructorInfo fooConstructor;

private MethodInfo returnZeroMethod;

public FooTests()

{

fooType = Type.GetType("Foo, Foo", true, false);

fooConstructor = fooType.GetConstructor(BindingFlags.Instance | BindingFlags.NonPublic,

null,

new Type[] { typeof(object) },

null);

returnZeroMethod = fooType.GetMethod("ReturnZero",

BindingFlags.Instance | BindingFlags.NonPublic);

}

[TestMethod]

[ExpectedException(typeof(ArgumentNullException))]

public void Foo_ctor_parameter_Null()

{

fooConstructor.Invoke(new object[] { null });

}

[TestMethod]

public void Foo_ReturnZero()

{

object fooInstance = fooConstructor.Invoke(new object[] { new object() });

Assert.AreEqual<int>(0, (int)returnZeroMethod.Invoke(fooInstance, null));

}

This works fine and both tests fail as expected. However, because we are using reflection and Invoke() throws a TargetInvocationException when the target methods throws with the InnerException property set to the original exception our test result for Foo_ctor_parameter_Null() contains the slightly cryptic message "Test method FooTests.Foo_ctor_parameter_Null threw exception System.Reflection.TargetInvocationException, but exception System.ArgumentNullException was expected. Exception message: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Value does not fall within the expected range.". Not a big deal but not very nice to read when examining test failures. If only there was an overload of Invoke() that allows us to tell it to let the original exception through. But there isn't and we can't create one. But we can create an extension method to mimic this behavior:

using System.Reflection;

public static class ConstructorInfoExtensions

{

public static object Invoke(this ConstructorInfo constructor,

object[] parameters,

bool throwOriginalException)

{

if (throwOriginalException)

{

try

{

return constructor.Invoke(parameters);

}

catch (TargetInvocationException e)

{

if (e.InnerException != null)

throw e.InnerException;

else

throw e;

}

else

return constructor.Invoke(parameters);

}

This extension method looks like an overload which I prefer over a method with a name like InvokeAndSuppressTargetInvocationException(). If throwOriginalException is true and a TargetInvocationException occurs it will catch it and throw TargetInvocationException.InnerException instead unless InnerException is null. Now we can change Foo_ctor_parameter_Null() to:

[TestMethod]

[ExpectedException(typeof(ArgumentNullException))]

public void Foo_ctor_parameter_Null()

{

fooConstructor.Invoke(new object[] { null }, true);

}

The test of course still fails but now we get the message "Test method FooTests.Foo_ctor_parameter_Null threw exception System.ArgumentException, but exception System.ArgumentNullException was expected. Exception message: System.ArgumentException: Value does not fall within the expected range." which is exactly what we would get if there was no reflection involved.

By the way, talking of ExpectedException and reflection: There is the general issue that Invoke() can actually throw a couple of different exceptions including the very common ones ArgumentException and InvalidOperationException. In case you are expecting one of those exception types (see the API reference for a full list) you need to be extra careful to make sure that you don't end up with tests that pass simply because Invoke() is throwing instead of the target of invocation.

Last but not least I'd like to mention that we can create another extension method like the one above to be used in case we are invoking test methods through reflection. Granted this is a special case but as shown in How would you test a... C# code generator with Visual Studio Team Test there are use cases for this.

using System.Reflection;

namespace SnapshotGeneratorTests

{

public static class MethodBaseExtensions

{

public static object Invoke(this MethodBase method,

object obj,

object[] parameters,

bool throwOriginalException)

{

if (throwOriginalException)

{

try

{

return method.Invoke(obj, parameters);

}

catch (TargetInvocationException e)

{

if (e.InnerException != null)

throw e.InnerException;

else

throw e;

}

else

return method.Invoke(obj, parameters);

}

This posting is provided "AS IS" with no warranties, and confers no rights.

How would you test a... C# code generator with Visual Studio Team Test

ddietric — Sat, 14 Jun 2008 15:50:00 GMT

Introduction

I spent some time this week thinking about how to test a code generator that generates C# code and is itself written in C# (so it's once again all about managed code). At first I thought about how to verify the generator output directly. Since the output is C# code - which ultimately is just a bunch of text - one approach is to create a collection of base line outputs and tests that run the generator with different inputs and compare the outputs to the base lines which have been verified when then the tests were written. There are two major flaws in this idea: Verifying the base lines in this scenario will be a manual task and even small changes to the generator output will cause tests to fail even if the new code behaves exactly the same way then before. Another approach would be to actually analyze the generator output but at the very least this would require parsing C# and I'll leave that to people who enjoy (and know how to) build compilers.

And? How would you test it?

So, coming to the conclusion that verifying the output was not an option I started thinking about the next best thing: Writing tests against the generated code and thus implicitly testing the generator. Makes sense; however, the question remains how to do that. One could run the generator manually for all interesting inputs, add the outputs to a test project and then write tests for them. This is actually worse than the approach with base lines since now there are a bunch of source files to update every time the generator output changes but the tests won’t automatically break when that happens. What I really wanted was running the code generator with every test pass. That said - and without further ado - let's look at some code. Consider a C# class library project containing the following "generator":

public static class CodeGenerator

{

public static string Generate(string name)

{

return @"

public static class PersonalizedHelloWorld

{

public static string Name { get { return " + (name != null ? "@\"" + name.Replace("\"", "\"\"") + "\"" : "null") + @"; } }

public static string GetHelloWorldMessage()

{

if (Name == null)

return ""Hello World! Hello anonymous user!"";

else

return ""Hello World! Hello "" + Name + ""!"";

}

}";

}

I say "generator" because obviously it doesn't do anything useful and I just wrote it for demonstrating my approach to testing generators which is also why I'm not using the CodeDOM. Now, let's think about the output we get for the string John "\/\/" Doe as the input. What I would normally do to test that code is create a new test project and write a test class like this one:

[TestClass]

public class NonemptyNameTests

{

[TestMethod]

public void NameProperty()

{

Assert.AreEqual<string>(@"John ""\/\/"" Doe",

PersonalizedHelloWorld.Name);

}

[TestMethod]

public void GetHelloWorldMessageMethod()

{

Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",

PersonalizedHelloWorld.GetHelloWorldMessage());

}

Apparently I can't do that because the class this code calls into does not exist when the test assembly is built. So instead I add a new source file to the test project with the following code and set its build action to Embedded Resource.

public static class NonemptyNameTests

{

public static void NameProperty()

{

Assert.AreEqual<string>(@"John ""\/\/"" Doe",

PersonalizedHelloWorld.Name);

}

public static void GetHelloWorldMessageMethod()

{

Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",

PersonalizedHelloWorld.GetHelloWorldMessage());

}

The idea is to compile this code together with the compiler output for the input John "\/\/" Doe during test execution. In order to do this I introduce a little helper method - we need to run and test the generator multiple times with different inputs after all.

public static class TestCompiler

{

private static string unitTestFrameworkAssemblyPath = typeof(Assert).Assembly.Location;

public static Assembly Compile(string generatedCode,

string generatedCodeFileName,

string dynamicTestCodeResourceName,

string dynamicTestCodeFileName,

string assemblyFileName)

{

string dynamicTestCode;

using (StreamReader reader = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(dynamicTestCodeResourceName)))

dynamicTestCode = reader.ReadToEnd();

File.WriteAllText(generatedCodeFileName, generatedCode);

File.WriteAllText(dynamicTestCodeFileName, dynamicTestCode);

CompilerParameters parameters = new CompilerParameters();

parameters.GenerateExecutable = false;

parameters.GenerateInMemory = false;

parameters.OutputAssembly = assemblyFileName;

parameters.ReferencedAssemblies.Add(unitTestFrameworkAssemblyPath);

#if DEBUG

parameters.IncludeDebugInformation = true;

#else

parameters.IncludeDebugInformation = false;

#endif

CSharpCodeProvider codeProvider = new CSharpCodeProvider();

CompilerResults results = codeProvider.CompileAssemblyFromFile(parameters,

generatedCodeFileName,

dynamicTestCodeFileName);

#if DEBUG

foreach (string message in results.Output)

Debug.WriteLine(message);

#endif

return results.CompiledAssembly;

}

All Compile() does is take a couple of names and generated code and return a new assembly. A key characteristic of this implementation is that the source code as well as the assembly are written to the file system which allows further analysis in case the code does not compile or test execution fails (the code provider does write code passed in as strings to the file system automatically - but it writes it to a temp directory and it is very quick to delete those temporary files as soon as the compiler finishes execution). Now we can add what VSTT/MSTest.exe will consider to be the test class:

[TestClass]

public class NonemptyNameTests

{

public const string AssemblyFileName = "NonemptyNameTests.dll";

public const string GeneratedCodeFileName = "NonemptyNameTests.target.cs";

public const string DynamicTestCodeFileName = "NonemptyNameTests.dynamic.cs";

public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NonemptyNameTests.dynamic.cs";

private static Type dynamicTestType;

[ClassInitialize]

public static void ClassInitialize(TestContext context)

{

string generatedCode = CodeGenerator.Generate(@"John ""\/\/"" Doe");

Assembly assembly = TestCompiler.Compile(generatedCode,

GeneratedCodeFileName,

DynamicTestCodeResourceName,

DynamicTestCodeFileName,

AssemblyFileName);

dynamicTestType = assembly.GetType("NonemptyNameTests");

}

[TestMethod]

public void NameProperty()

{

dynamicTestType.GetMethod("NameProperty").Invoke(null, null);

}

[TestMethod]

public void GetHelloWorldMessageMethod()

{

dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);

}

Note that the only real work this class does is run the code generator, build another test assembly with the generator output and the test code we embedded as a resource and store a Type instance for the new test class. Even more important, it does so in the class initialization method meaning that if an exception is thrown during the generation or the compilation step it will cause all tests in this class to fail. Furthermore, for every test to be added you need to add two methods: One in the class with the actual test logic which gets compiled at runtime and one in the class above that just calls the first one using reflection. And that's actually all we need to make this work. What's left for this post is a non-exhaustive list with the advantages and disadvantages of this approach.

Advantages

No manual steps
Running the code generator is an integral part of the test pass (which I considered to be a requirement) and so is compiling the code.
Unit test framework friendliness
Just to be clear - it doesn't matter if you consider any of this to be unit testing or not. The point is that you can use any unit testing framework you're already using anyway (like VSTT/MSTest.exe, NUnit, [insert favorite unit test framework here]) as long as it allows defining an initialization step for a test class.
Agnostic to changes not affecting behavior
Tests do not have to be updated in case of changes to the generated code unless its behavior changes (meaning changes in comments, white space, formatting and refactoring without API changes are free).
Test assembly is self-contained
In terms of test execution and from VSTT's/MSTest.exe's point of view there is absolutely no difference between the test project I described in this post and a "normal" one. All you need to run the tests is the assembly with the code generator (and of course its dependencies) and the test assembly (and its dependencies). This is also the reason why I chose to embed the test code to be compiled during text execution as a resource instead of keeping it as an external file.

Disadvantages

Can be difficult to write actual test code
Since the test classes actually doing the work are embedded as resources and refer to types not available when building the test project it can be more difficult to write them since IntelliSense and some other IDE features are not available unless you keep switching the build action from Embedded Resource to Compile and back and - if that is not enough help - temporarily copy the generator output into one of the test project's source files.
Can be difficult to debug
This applies primarily to the compilation failing since you have to rely on the raw compiler output to figure out why it failed. Failing tests are less of a problem because the debugger will open the source files compiled at runtime when it is configured to break when a first chance exception occurs. However, this can also be confusion at times because one must refrain from editing the file the debugger opened and fix issues in the original source file or generator code respectively. Also, since the test methods marked with the TestMethod attribute invoke the actual test methods through reflection, failing asserts will cause the test to fail because of a TargetInvocationException instead of the failed assert. Luckily, the original assert information is available through the InnerException property.
Cannot verify comments, formatting, etc.
One of the strengths of this approach is also a weakness. As the tests never look at the generated source code there is also no way to easily hook in steps to check things like comments or formatting.

Full annotated sample source code

/**********************************

* Project: CodeGenerator *

* File Name: CodeGenerator.cs *

* Build Action: Compile *

**********************************/

public static class CodeGenerator

{

// Useless code generator method for

// demonstration purposes only.

// DO NOT USE IN PRODUCTION ENVIRONMENTS!

public static string Generate(string name)

{

return @"

public static class PersonalizedHelloWorld

{

public static string Name { get { return " + (name != null ? "@\"" + name.Replace("\"", "\"\"") + "\"" : "null") + @"; } }

public static string GetHelloWorldMessage()

{

if (Name == null)

return ""Hello World! Hello anonymous user!"";

else

return ""Hello World! Hello "" + Name + ""!"";

}

}";

}

/************************************

* Project: CodeGeneratorTests *

* File Name: TestCompiler.cs *

* Build Action: Compile *

************************************/

using Microsoft.CSharp;

using Microsoft.VisualStudio.TestTools.UnitTesting;

using System.CodeDom.Compiler;

using System.Collections.Generic;

using System.Diagnostics;

using System.IO;

using System.Reflection;

public static class TestCompiler

{

private static string unitTestFrameworkAssemblyPath = typeof(Assert).Assembly.Location;

// Use by test class initializers to compile the generated

// code together with actual test code into an assembly

// during the test pass.

public static Assembly Compile(string generatedCode,

string generatedCodeFileName,

string dynamicTestCodeResourceName,

string dynamicTestCodeFileName,

string assemblyFileName)

{

string dynamicTestCode;

// Reads the actual test code from a resource

// stream in the static test assembly.

using (StreamReader reader = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(dynamicTestCodeResourceName)))

dynamicTestCode = reader.ReadToEnd();

// Writes the generated code as well as the actual

// test code from the resource stream to the file

// system in order to allow debugging the actual test code.

File.WriteAllText(generatedCodeFileName, generatedCode);

File.WriteAllText(dynamicTestCodeFileName, dynamicTestCode);

// Parameters for the code provider telling it to generate

// a DLL and write it to the file system.

CompilerParameters parameters = new CompilerParameters();

parameters.GenerateExecutable = false;

parameters.GenerateInMemory = false;

parameters.OutputAssembly = assemblyFileName;

// The paths to/names of all assemblies containing types

// referenced in the code to compile need to be added to

// the CompilerParameters object ReferencedAssemblies collection.

parameters.ReferencedAssemblies.Add(unitTestFrameworkAssemblyPath);

#if DEBUG

// Need to make debugging easy in case a class initializer or

// test fails by having the compiler include debug information

// and create a PDB file.

parameters.IncludeDebugInformation = true;

#else

parameters.IncludeDebugInformation = false;

#endif

// Creating code provider instance and invoking C# compiler.

CSharpCodeProvider codeProvider = new CSharpCodeProvider();

CompilerResults results = codeProvider.CompileAssemblyFromFile(parameters,

generatedCodeFileName,

dynamicTestCodeFileName);

// Writing full compiler output to trace listeners in debug

// builds to preserve all errors in case the generated/dynamic

// test code does not compile.

#if DEBUG

foreach (string message in results.Output)

Debug.WriteLine(message);

#endif

// Returning new assembly to caller.

return results.CompiledAssembly;

}

/**************************************

* Project: CodeGeneratorTests *

* File Name: NonemptyNameTests.cs *

* Build Action: Compile *

**************************************/

using Microsoft.VisualStudio.TestTools.UnitTesting;

using System;

using System.Reflection;

// Test class verifying the code generator

// by testing its compiled output.

[TestClass]

public class NonemptyNameTests

{

// File and resource name constants for this test class.

public const string AssemblyFileName = "NonemptyNameTests.dll";

public const string GeneratedCodeFileName = "NonemptyNameTests.target.cs";

public const string DynamicTestCodeFileName = "NonemptyNameTests.dynamic.cs";

public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NonemptyNameTests.dynamic.cs";

private static Type dynamicTestType;

[ClassInitialize]

public static void ClassInitialize(TestContext context)

{

// Running the code generator we want to test

// with one set of the inputs we need to cover.

string generatedCode = CodeGenerator.Generate(@"John ""\/\/"" Doe");

// Compiling the generated code together with the test code

// for the chosen generator input into a new assembly. Note

// that if the generator produced code with syntax errors

// all tests in this test class will fail because of an

// exception in the class initializer.

Assembly assembly = TestCompiler.Compile(generatedCode,

GeneratedCodeFileName,

DynamicTestCodeResourceName,

DynamicTestCodeFileName,

AssemblyFileName);

// Storing a Type instance for the test class type in the new

// assembly to be used by the test methods in this class.

dynamicTestType = assembly.GetType("NonemptyNameTests");

}

// The following two test methods do not contain actual

// verification code. Instead they call methods in the new

// assembly using reflection.

[TestMethod]

public void NameProperty()

{

dynamicTestType.GetMethod("NameProperty").Invoke(null, null);

}

[TestMethod]

public void GetHelloWorldMessageMethod()

{

dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);

}

/**********************************************

* Project: CodeGeneratorTests *

* File Name: NonemptyNameTests.dynamic.cs *

* Build Action: Embedded Resource *

**********************************************/

using Microsoft.VisualStudio.TestTools.UnitTesting;

// Class containing the actual verification logic for a specific set

// of code generator input. This code is not compiled into

// CodeGeneratorTests.dll but embedded as a resource. It is compiled

// together with the generator output and called by methods marked

// with the TestMethod attribute using reflection during the test pass.

public static class NonemptyNameTests

{

public static void NameProperty()

{

Assert.AreEqual<string>(@"John ""\/\/"" Doe",

PersonalizedHelloWorld.Name);

}

public static void GetHelloWorldMessageMethod()

{

Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",

PersonalizedHelloWorld.GetHelloWorldMessage());

}

/************************************

* Project: CodeGeneratorTests *

* File Name: NameNullTests.cs *

* Build Action: Compile *

************************************/

using Microsoft.VisualStudio.TestTools.UnitTesting;

using System;

using System.Reflection;

[TestClass]

public class NameNullTests

{

public const string AssemblyFileName = "NameNullTests.dll";

public const string GeneratedCodeFileName = "NameNullTests.target.cs";

public const string DynamicTestCodeFileName = "NameNullTests.dynamic.cs";

public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NameNullTests.dynamic.cs";

private static Type dynamicTestType;

[ClassInitialize]

public static void ClassInitialize(TestContext context)

{

string generatedCode = CodeGenerator.Generate(null);

Assembly assembly = TestCompiler.Compile(generatedCode,

GeneratedCodeFileName,

DynamicTestCodeResourceName,

DynamicTestCodeFileName,

AssemblyFileName);

dynamicTestType = assembly.GetType("NameNullTests");

}

[TestMethod]

public void NameProperty()

{

dynamicTestType.GetMethod("NameProperty").Invoke(null, null);

}

[TestMethod]

public void GetHelloWorldMessageMethod()

{

dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);

}

/******************************************

* Project: CodeGeneratorTests *

* File Name: NameNullTests.dynamic.cs *

* Build Action: Embedded Resource *

******************************************/

using Microsoft.VisualStudio.TestTools.UnitTesting;

public static class NameNullTests

{

public static void NameProperty()

{

Assert.IsNull(PersonalizedHelloWorld.Name);

}

public static void GetHelloWorldMessageMethod()

{

Assert.AreEqual<string>("Hello World! Hello anonymous user!",

PersonalizedHelloWorld.GetHelloWorldMessage());

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Fun with events and delegates

ddietric — Mon, 01 Oct 2007 12:57:00 GMT

So how can you verify that a given class raises an event? Well, you can simply write some code that registers an event handler and then cause the target to raise that event. Now, how do you verify that a given class does actually handle an event raised by another class? Basically it’s the same thing – just the other way round. You create test code which raises the event in question, makes the target register an event handler, raises the event and verifies that the target handles the event correctly. But how do you verify that a given class does not handle an event raised by another class? And why do you even need to worry about this?

The last two questions only make sense in a scenario in which a class under test registers an event handler in the first place and is then expected to unregister that handler later on. The obvious reason why you need to care is that the behavior of your software is probably incorrect if it keeps on handling an event although it shouldn't. The less obvious reason is that event handlers affect the lifetime of the object that handles the event because the delegate passed to the object that raises the event contains a reference back to the event handling object. This means that failure to unregister event handlers before getting rid of all other references to an object can lead to issues that are hard to debug. The best case scenario is that because of the remaining event handlers your software is behaving strangely so you at least notice that there is something going wrong (which might still be difficult to track down). The worst case scenario is that the remaining event handlers won't really do anything because of their internal state so unless you inspect your software and specifically look for method invocation counts, class instance counts or memory consumption you may not even notice the defect.

With all that said, how can we actually verify that a given object unregisters its event handlers? As mentioned before, simply raising the event may not give us any clue on whether or not there are handlers left. However, we can examine an event at runtime. But before I show some simple code that will help us to achieve this let's have a look at what an event really is. I most cases when you declare an event it will probably look something like this:

public event EventHandler SomeEvent;

The fun begins when we look at the IL generated by the C# compiler because of that one innocent line of code:

.field private class [mscorlib]System.EventHandler SomeEvent

.event [mscorlib]System.EventHandler SomeEvent

{

.addon instance void TestProject1.TestClass::add_SomeEvent(class [mscorlib]System.EventHandler)

.removeon instance void TestProject1.TestClass::remove_SomeEvent(class [mscorlib]System.EventHandler)

}

.method public hidebysig specialname newslot virtual final instance void add_SomeEvent(class [mscorlib]System.EventHandler 'value') cil managed synchronized
{
    .maxstack 8
    L_0000: ldarg.0
    L_0001: ldarg.0
    L_0002: ldfld class [mscorlib]System.EventHandler TestProject1.TestClass::SomeEvent
    L_0007: ldarg.1
    L_0008: call class [mscorlib]System.Delegate [mscorlib]System.Delegate::Combine(class [mscorlib]System.Delegate, class [mscorlib]System.Delegate)
    L_000d: castclass [mscorlib]System.EventHandler
    L_0012: stfld class [mscorlib]System.EventHandler TestProject1.TestClass::SomeEvent
    L_0017: ret
}

.method public hidebysig specialname newslot virtual final instance void remove_SomeEvent(class [mscorlib]System.EventHandler 'value') cil managed synchronized
{
    .maxstack 8
    L_0000: ldarg.0
    L_0001: ldarg.0
    L_0002: ldfld class [mscorlib]System.EventHandler TestProject1.TestClass::SomeEvent
    L_0007: ldarg.1
    L_0008: call class [mscorlib]System.Delegate [mscorlib]System.Delegate::Remove(class [mscorlib]System.Delegate, class [mscorlib]System.Delegate)
    L_000d: castclass [mscorlib]System.EventHandler
    L_0012: stfld class [mscorlib]System.EventHandler TestProject1.TestClass::SomeEvent
    L_0017: ret
}

First, we have a private field of the type we used to declare our event. This type is a delegate type derived from MulticastDelegate which means that it holds an array of delegates. The field being declared as private is also the reason why you cannot raise an event from a derived class. Second, there is the actual event which points to the methods add_SomeEvent() and remove_SomeEvent(). This is similar to how properties are compiled into getter and setter methods. The add and remove methods of an event are automatically called when you use the += and -= operators. Now that we have a clear understanding of what an event is we can go through some sample code. Say I have an interface that contains an event and a class that hooks up event handlers for that event in my production code:

public interface IRaisesEvent

{

event EventHandler SomeEvent;

}

public static class MyClass

{

private static Collection<IRaisesEvent> registeredObjects = new Collection<IRaisesEvent>();

public static void RegisterObject(IRaisesEvent obj)

{

if (obj == null)

throw new ArgumentNullException();

registeredObjects.Add(obj);

obj.SomeEvent += new EventHandler(EventHandler);

}

public static void ReleaseObjects()

{

foreach (IRaisesEvent item in registeredObjects)

item.SomeEvent -= new EventHandler(EventHandler);

registeredObjects.Clear();

}

private static void EventHandler(object sender, EventArgs e)

{

Debug.Print("Object of type {0} raised IRaisesEvent.SomeEvent", sender.GetType());

}
}

The next thing we need is a test class that implements the interface so we have something to pass to the static class. In addition to implementing the interface I also defined a read-only property that allows a caller to access the private MulticastDelegate instance. Please note that I do not advice you to do so in production code since this exposes detailed information about the event (which, however, is exactly what we want for testing purposes).

public class TestClass : IRaisesEvent

{

public event EventHandler SomeEvent;

public MulticastDelegate SomeEventMulticastDelegate

{

get

{

return SomeEvent;

}

This actually does the trick. We can now examine the state of the event (i.e. its currently registered event handlers) by calling MulticastDelegate.GetInvocationList() and the properties Delegate.Method and Delegate.Target allow us to get detailed information on each of those delegates. Of course creating some extension methods once for the most interesting scenarios will simplify our actual test code.

public static class MulticastDelegateExtensions

{

public static int GetDelegateCount(this MulticastDelegate d)

{

if (d == null)

return 0;

else

return d.GetInvocationList().Length;

}

public static Type[] GetTargetTypes(this MulticastDelegate d)

{

Delegate[] invocationList;

Type[] types;

if (d == null)

return new Type[0];

else

{

invocationList = d.GetInvocationList();

types = new Type[invocationList.Length];

for (int i = 0; i < invocationList.Length; i++)

types[i] = invocationList[i].Method.ReflectedType;

return types;

}

The methods above for example allow us to get the number of event handlers and the types of the objects that will handle the event. This is a reasonable setup for the test given that you can reuse the extension methods throughout your test projects. The test method itself is short and easy to understand:

[TestMethod]

public void DelegateTest()

{

TestClass c = new TestClass();

MyClass.RegisterObject(c);

Assert.AreEqual<int>(1, c.SomeEventMulticastDelegate.GetDelegateCount());

Assert.AreEqual<Type>(typeof(MyClass), c.SomeEventMulticastDelegate.GetTargetTypes()[0]);

MyClass.ReleaseObjects();

Assert.AreEqual<int>(0, c.SomeEventMulticastDelegate.GetDelegateCount());

}

This posting is provided "AS IS" with no warranties, and confers no rights.

A good reason for calling GC.Collect()

ddietric — Sun, 23 Sep 2007 12:11:00 GMT

You've probably heard that you should not call GC.Collect() and generally speaking this is absolutely correct. Yes, there are probably exceptions to that rule but unless you know for sure that calling GC.Collect() will improve the runtime behavior of your application you should refrain from making that call. Also, let's not forget that implementation details of the garbage collector can change between releases of the CLR. However, the possibility of forcing a garbage collection can come in very handy for testing or debugging purposes. Let's consider the following static class:

public static class MyClass

{

private static Collection<object> objects = new Collection<object>();

public static void RegisterObject(object o)

{

objects.Add(o);

}

public static void ReleaseAllObjects()

{

objects.Clear();

}

This class is of course pretty pointless. But it is suitable for illustrating a scenario in which a class holds references to an object created by a caller thus influencing the lifetime of that object. Instead of a ReleaseAllObjects() method you could for example think of a class that releases references when certain events occur. Either way, you might want to write a test which actually verifies that all object references have been deleted at a certain point. This is a problem GC.Collect() helps us to solve as an object without any references to it is automatically destructed by the garbage collector and because of the nondeterministic nature of the collector we have to manually trigger a collection. So, a step-by-step description of the test looks like this:

Create an object
Pass a reference to the MyClass class by calling RegisterObject(object)
Delete the local reference on the caller side
Call MyClass.ReleaseAllObjects()
Call GC.Collect()
Verify that the object has been destructed

Obviously we are still missing an important piece that allows us to do the actual verification that the object is no more. And that piece is a weak reference. Automated tests checking the lifetime of objects aside, weak references allow an object to keep a reference to another object without keeping the garbage collector from collection it (in other words, the garbage collector will ignore all weak references when determining if an object is still referenced or not). By creating a WeakReference referencing the object before deleting the regular reference to it we can verify that it has been collected by checking the IsAlive property which must be false at the end of the test:

[TestClass]

public class MyClassTests

{

[TestMethod]

public void ObjectLifetimeTest()

{

object someObject = new object();

WeakReference weakObjectReference = new WeakReference(someObject);

MyClass.RegisterObject(someObject);

someObject = null;

MyClass.ReleaseAllObjects();

GC.Collect();

Assert.IsFalse(weakObjectReference.IsAlive);

}

This posting is provided "AS IS" with no warranties, and confers no rights.