Dennis "D.C." Dietrich : EMTF

EMTF 2.0 Beta released

ddietric — Tue, 14 Jul 2009 13:04:00 GMT

It's been a couple of months since I first blogged about EMTF. Since then I've been busy refining and extending the framework and I'm happy to announce that the beta release of version 2.0 is finally done! Special thanks to Josh Poley since a lot of the new functionality is based on his feedback. So, allow me to give you a tour of what's new in ~~Reality~~ EMTF 2.0:

Pre- and post-test actions

The PreTestActionAttribute and PostTestActionAttribute attributes have been added to allow defining methods that are executed before/after each test method in a test class. This is most commonly used for shared setup and teardown of the environment required by a set of tests.

[TestClass]

public class Tests

{

// Will be executed before every test in the Tests class

[PreTestAction]

public void PreTestAction() { }

// Will be executed after every passed test in the Tests class

[PostTestAction]

public void PostTestAction() { }

}

Aborting tests, custom logging and the TestContext

Test methods as well as pre- and post-test actions can take one parameter of the type TestContext. If they do an instance will be automatically provided by the runtime. The test context allows aborting a test from within the test implementation e.g. in case test initialization fails. It also allows adding arbitrary information to the test log.

[Test]

public void Test(TestContext context)

{

context.LogLine("Initialization failed");

context.AbortTest();

}

Test groups

The TestGroupsAttribute attribute allows tagging test methods as belonging to one or multiple test groups. At the same time, there are new TestExecutor.Execute() overloads which limit test execution to tests belonging to the specified group(s).

[TestClass]

public class Tests

{

[Test]

[TestGroups("BVT")]

public void Bvt() { }

[Test]

[TestGroups("BVT", "STRESS")]

public void StressBvt() { }

[Test]

[TestGroups("STRESS")]

public void Stress() { }

}

class Program

{

static void Main(string[] args)

{

// Will only run Bvt() and StressBvt()

new TestExecutor().Execute(new String[] { "BVT" });

}

Concurrent test runs

EMTF now supports concurrent test runs in order to take advantage of multi-core CPUs. Concurrent test runs will execute tests on multiple threads instead of just one. The number of threads used is equal to the number of logical processors on the local machine.

TestExecutor executor = new TestExecutor();

executor.ConcurrentTestRuns = true;

executor.Execute();

Plain text logging to a stream

The logging library has also been extended and now includes the StreamLogger which writes a plain text log to a stream.

TestExecutor executor = new TestExecutor();

using (Stream stream = new FileStream("testrun.log", FileMode.Create))

{

StreamLogger logger = new StreamLogger(executor, stream);

executor.Execute();

logger.Close();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Creating an API testing framework 101

ddietric — Wed, 25 Feb 2009 07:40:00 GMT

Writing a fully featured API testing solution is a lot of work. You have to deal with things like test case management, integration with code coverage tools or test execution on server farms. However, the core of a (reflection-based) API testing framework is quite simple. This post explains the basics with EMTF as an example.

Implementing asserts

Assertions are the building blocks for actual API test cases (my previous post Introducing EMTF contains a simple example). A single failing assertion fails the whole test and causes the execution of the current test method to be cancelled. How does a method that doesn't even return a value achieve this? Fairly simple: it throws an exception.

[DebuggerHidden]

public static void IsTrue(Boolean condition, String message)

{

if (!condition)

throw new AssertException("Assert.IsTrue failed.", message);

}

Since it throws a specific exception type, a failed assertion can be distinguished from a crashing test method based on the type of the exception. But isn't it bad to use exceptions to return from an operation? It usually is and this is one of the few exceptions to this rule. First of all, a failing test is (hopefully) still an exceptional event. Usually we want and expect our tests to pass. Second, there is no better way to allow the test runtime to cancel the execution of a test method when an assertion fails.

Another little detail is the DebuggerHiddenAttribute on all methods of the Assert class (Assert.cs) as well as the public constructors of the AssertException class (AssertException.cs). This tells the debugger that the methods/constructors marked with this attribute should be omitted from stack traces so they won't show up when debugging assertion failures.

Identifying test classes and test methods

A common way of marking classes as test classes and methods as test methods is the use of attributes. This allows the test runtime to automatically find and execute all tests and has the advantage of keeping the object model of the test suite flexible. An alternative is to define a base class or interface that test classes must derive from or implement respectively and treating all public, non-abstract, non-generic, void returning, parameter less methods as test method. Though this simplifies both the test code and the code that finds all test methods through reflection a little bit it comes at the expense of pretty much dictating what a test class looks like.

EMTF contains the type TestClassAttribute (TestClassAttribute.cs) for test classes and TestAttribute (TestAttribute.cs) for test methods. These attributes can contain additional (optional) information or be supplemented by additional attributes. EMTF's TestAttribute for example has a property for the test description.

[AttributeUsage(AttributeTargets.Class)]

public sealed class TestClassAttribute : Attribute

{

}

[AttributeUsage(AttributeTargets.Method)]

public sealed class TestAttribute : Attribute

{

private String _description;

public String Description

{

get

{

return _description;

}

public TestAttribute()

{

}

public TestAttribute(String description)

{

_description = description;

}

Finding all tests

private static Collection<MethodInfo> FindTestMethods()

{

Collection<MethodInfo> testMethods = new Collection<MethodInfo>();

// Iterate through all assemblies in the current application domain

foreach (Assembly assembly in AppDomain.CurrentDomain.GetAssemblies())

{

// Iterate through all exported types in each assembly

foreach (Type type in assembly.GetExportedTypes())

{

// Verify that the type is:

// 1. Marked with the TestClassAttribute

// 2. Not a generic type definition or open constructed type

// 3. Not abstract

if (type.IsDefined(typeof(TestClassAttribute), true) &&

!type.ContainsGenericParameters &&

!type.IsAbstract)

{

// Iterate through all public instance methods

foreach (MethodInfo method in type.GetMethods(BindingFlags.Instance | BindingFlags.Public))

{

// Verify that the method is:

// 1. Marked with the TestAttribute

// 2. Not a generic method definition or open constructed method

// 3. Its return type is void

// 4. Does not have any parameters

if (method.IsDefined(typeof(TestAttribute), true) &&

!method.ContainsGenericParameters &&

method.ReturnType == typeof(void) &&

method.GetParameters().Length == 0)

{

// Add method to the list of

// test methods to execute

testMethods.Add(method);

}

return testMethods;

}

Executing the tests

The last piece missing is a method that executes all test methods and logs the results or at least raises events which can be handled by the caller or a dedicated logger. The basic logic for test execution used by EMTF's TestExecutor (TestExecutor.cs) and other API testing runtimes looks like this:

)" alt="TestExecutor.ExecuteImpl(IEnumerable)" align=baseline src="http://blogs.msdn.com/photos/ddietric/images/9443552/original.aspx" mce_src="http://blogs.msdn.com/photos/ddietric/images/9443552/original.aspx">

private void ExecuteImpl(IEnumerable<MethodInfo> testMethods)

{

// Raise event signaling the start of a test run

OnTestRunStarted();

// Instance of the current test class

Object currentInstance = null;

// Iterate through all test methods

foreach (MethodInfo method in testMethods)

{

// Get the TestAttribute on the method

object[] testAttribute = method.GetCustomAttributes(typeof(TestAttribute), true);

string testDescription = null;

// Get the test description if TestAttribute is defined on the method

if (testAttribute.Length > 0)

testDescription = ((TestAttribute)testAttribute[0]).Description;

// Try to instantiate the test class if necessary

// Skip the test if an instance cannot be created

if (!TryUpdateTestClassInstance(method, testDescription, ref currentInstance))

continue;

// Raise an event signaling the start of a test

OnTestStarted(new TestEventArgs(method, testDescription));

// Invoke the test method in a try block so we can

// catch assert failures and unexpected exceptions

try

{

method.Invoke(currentInstance, null, true);

}

// Assert failed

catch (AssertException e)

{

// Raise event signaling the test failure

// then immediately run the next test

OnTestCompleted(

new TestCompletedEventArgs(

method,

testDescription,

e.Message,

e.UserMessage,

TestResult.Failed,

null));

continue;

}

// Test threw unexpected exception

catch (Exception e)

{

// Raise event signaling the test failure

// then immediately run the next test

OnTestCompleted(

new TestCompletedEventArgs(

method,

testDescription,

String.Format(

CultureInfo.CurrentCulture,

"An exception of the type '{0}' occurred during the execution of the test.",

e.GetType().FullName),

null,

TestResult.Exception,

e));

continue;

}

// No exceptions were thrown

// Raise event signaling that the test passed

OnTestCompleted(

new TestCompletedEventArgs(

method, testDescription, "Test passed.", null, TestResult.Passed, null));

}

// Raise event signaling the completion of the test run

OnTestRunCompleted();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Introducing EMTF

ddietric — Tue, 17 Feb 2009 10:45:00 GMT

A while ago a needed to do what I like to call quick and dirty testing, i.e. getting as much test coverage as possible in a given period of time. To make things worse, I was working on a client-server feature requiring a bit of setup in order to work in the first place and wanted to cover both sides with my tests and be able to easily run all tests. So I thought, what if instead of trying to get this done with one of the existing test frameworks that typically host the test target as well as its tests, I just created a new solution which already had all the pieces in place to use the feature and put a tiny test runtime in there. I did exactly that and got my testing done the way I wanted to. Again, there is a reason why I call this kind of approach quick and dirty testing and I definitely wouldn't pitch it as a best practice but it has its place.

Because of my objective this "embedded" test runtime I created was extremely small but also highly specific to the test scenarios for which it was written and thus virtually impossible to reuse. That wasn't a surprise but rather one of the tradeoffs I had opted in for. Still, I was wondering what a generalized version would look like that, next time I wanted to do something like this, I could simply drop into my application and get going. The answer to that question is EMTF.

Yet another test framework called EMTF

The Embeddable Micro Test Framework is similar to its big brothers MSTest or NUnit though obviously smaller and simpler and with the twist of it being supposed to be hosted by the application under test rather than coming with its own test host. It is available in source and binary form on CodePlex and by now I'd say that its greatest value is actually in using it for educational purposes since the reduced complexity makes it easy to understand for people without prior API testing experience (that goes for both how to use it and - at the core - how it works). That said the rest of this post is an EMTF quickstart (though I do consider a small series on how to implement an API test framework using EMTF as an example).

The easiest way to add EMTF to your project (especially if you're not using C#) is to add references to the two assemblies Emtf.dll and Emtf.Logging.dll. The first assembly contains the actual test framework that doesn't even include any logging capabilities. The second one contains a logging base class, a logger writing to the debug output and a logger writing to the console (the latter one only for the .NET Framework). There are separate versions of the two assemblies for Silverlight and the .NET Framework with the Silverlight version being a subset.

Writing tests

Let's say your project is a console application, you added EMTF to it and now want to test the following class which is part of your project (once again, please ignore the fact that the class is completely useless):

public class LameExample

{

private String _string;

public String String

{

get

{

return _string;

}

set

{

if (value == String.Empty)

throw new ArgumentException(

"New value must not be an empty string.",

"value");

_string = value;

}

We need to verify that:

The default value of the String property is null
The setter of the String property throws an ArgumentException if we pass in String.Empty
We can set the property to arbitrary values (except for String.Empty)

Ideally we want the test framework to automatically find all the tests and execute them. So in order to tell EMTF which classes contain tests, they need to be marked with the TestClassAttribute. Furthermore, test classes must be public, non-abstract and non-generic since the runtime wouldn't know which type parameters to use. Similarly, test methods are marked with the TestAttribute and must be public, non-abstract and non-generic. In addition, their return type must be void and they must not take any parameters. Tests for state and behavior of an object are performed using assertions.

[TestClass]

public class LameExampleTests

{

[Test]

public void StringProperty()

{

LameExample le = new LameExample();

// Verifies the default value of LameExample.String

Assert.IsNull(le.String);

// Verifies that the setter throws

Assert.Throws(

// Action that is expected to cause the exception

() => le.String = String.Empty,

// Action(T) verifying the exception object where

// T is the type of the expected exception.

(ArgumentException e) =>

{

Assert.IsTrue(e.Message.StartsWith("New value must not be an empty string."));

Assert.AreEqual("value", e.ParamName);

});

// Verifies setting the property to a non-null,

// non-empty string.

le.String = "fhqwhgads";

Assert.AreEqual("fhqwhgads", le.String);

// Verifies resetting the property to null.

le.String = null;

Assert.IsNull(le.String);

}

Running the tests

Since EMTF does not include any sort of test host we need to instantiate the engine, the TestExecutor, ourselves. In addition, we need to hookup the engine's events so we actually get to know the outcome of the test run. The loggers in Emtf.Logging.dll do this out of the box. Finally, calling TestExecutor.Execute() will execute all the tests we have defined that conform to the rules explained above.

private static void RunTests()

{

TestExecutor executor = new TestExecutor();

ConsoleLogger logger = new ConsoleLogger(executor);

executor.Execute();

}

This posting is provided "AS IS" with no warranties, and confers no rights.