Dennis "D.C." Dietrich : Open Source

EMTF 2.0 Beta released

ddietric — Tue, 14 Jul 2009 13:04:00 GMT

It's been a couple of months since I first blogged about EMTF. Since then I've been busy refining and extending the framework and I'm happy to announce that the beta release of version 2.0 is finally done! Special thanks to Josh Poley since a lot of the new functionality is based on his feedback. So, allow me to give you a tour of what's new in ~~Reality~~ EMTF 2.0:

Pre- and post-test actions

The PreTestActionAttribute and PostTestActionAttribute attributes have been added to allow defining methods that are executed before/after each test method in a test class. This is most commonly used for shared setup and teardown of the environment required by a set of tests.

[TestClass]

public class Tests

{

// Will be executed before every test in the Tests class

[PreTestAction]

public void PreTestAction() { }

// Will be executed after every passed test in the Tests class

[PostTestAction]

public void PostTestAction() { }

}

Aborting tests, custom logging and the TestContext

Test methods as well as pre- and post-test actions can take one parameter of the type TestContext. If they do an instance will be automatically provided by the runtime. The test context allows aborting a test from within the test implementation e.g. in case test initialization fails. It also allows adding arbitrary information to the test log.

[Test]

public void Test(TestContext context)

{

context.LogLine("Initialization failed");

context.AbortTest();

}

Test groups

The TestGroupsAttribute attribute allows tagging test methods as belonging to one or multiple test groups. At the same time, there are new TestExecutor.Execute() overloads which limit test execution to tests belonging to the specified group(s).

[TestClass]

public class Tests

{

[Test]

[TestGroups("BVT")]

public void Bvt() { }

[Test]

[TestGroups("BVT", "STRESS")]

public void StressBvt() { }

[Test]

[TestGroups("STRESS")]

public void Stress() { }

}

class Program

{

static void Main(string[] args)

{

// Will only run Bvt() and StressBvt()

new TestExecutor().Execute(new String[] { "BVT" });

}

Concurrent test runs

EMTF now supports concurrent test runs in order to take advantage of multi-core CPUs. Concurrent test runs will execute tests on multiple threads instead of just one. The number of threads used is equal to the number of logical processors on the local machine.

TestExecutor executor = new TestExecutor();

executor.ConcurrentTestRuns = true;

executor.Execute();

Plain text logging to a stream

The logging library has also been extended and now includes the StreamLogger which writes a plain text log to a stream.

TestExecutor executor = new TestExecutor();

using (Stream stream = new FileStream("testrun.log", FileMode.Create))

{

StreamLogger logger = new StreamLogger(executor, stream);

executor.Execute();

logger.Close();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

HTTP debugging for Silverlight developers (Mac OS X edition)

ddietric — Mon, 08 Jun 2009 13:00:00 GMT

Many posts have been written about how to debug problems with issuing HTTP requests from Silverlight applications, particularly in the context of web services and WCF (see for example Debugging Web Service Usage in Silverlight 2 or Faults and Exceptions when using Web Services in Silverlight 2). As they point out there are severe limitations to what information is passed to applications in case of non-200 HTTP status codes. And they also point out that sometimes you need to know what requests and responses are exactly sent over the wire in order to figure out what's going on, usually mentioning Fiddler for doing so. It is a very powerful tool allowing you to not only monitor HTTP traffic from certain applications but also letting you manipulate requests and responses (which is probably more than you need for "simple" debugging anyway).

This is all great until you hit an issue that only seems to repro on that other platform. Using a Windows application to debug on Mac OS X is impractical at best (impossible would be the better word in this case though). So, what to use instead? For my personal application development I use Wireshark which some of you may have encountered in the past under its old name, Ethereal. There are, however, a couple of differences. Wireshark is open source released under the GPL 2 but more importantly it is a sniffer and not an "HTTP debugging proxy". This means that in comparison to Fiddler it does not necessarily "know" HTTP as well since to Wireshark it is just another high-level protocol. It also means that a typical Wireshark capture picks up much more traffic than we care for (not using promiscuous mode in case of shared medium topologies at least reduces the amount of noise in the capture).

Wireshark is an X11 application so be sure that you have X11 installed and that it is up-to-date. Also, the OS X version of Wireshark uses BPFs to capture network traffic; however, access to those is denied for all users except for root (see also Enabling and using the "root" user in Mac OS X) so their permissions need to be changed first. This has to be redone after every reboot. While automatically making these changes during every startup or login has some appeal to it I prefer making them manually when they are needed. Given the nature of BPFs, keeping them locked down seems like a good idea to me.

Now we're finally ready to start Wireshark. The first start of the application can take a while (if I remember correctly this is due to X11 filling its font cache) so be patient. Also, all builds I have tried so far greet you with a nice pop-up error message. Luckily just ignoring it doesn't cause any trouble at least not in case of what I'm talking about in this post. At this point you can start capturing by simply selecting an interface from the list.

As mentioned before, this will give you much more information than you need since you'll see all traffic sent to/from your machine or all traffic hitting your interface if you are using promiscuous mode which is on by default. The screenshot below indicates 12752 captured packets after about 100 seconds which shouldn't surprise you since unlike Fiddler which is only concerned about HTTP Wireshark is capturing Ethernet frames. The simplest way to get to the information you really care about is to filter for http which is also shown in the screenshot. There's a lot more you can do like filtering for source or destination address so pay a visit to the Wireshark Wiki.

Lastly I'd like to mention that you can view a packet in its own window by selecting the command Show Packet in New Window from the View menu or by simply double-clicking on a packet in case you're running the latest development release.

This posting is provided "AS IS" with no warranties, and confers no rights.

Introducing EMTF

ddietric — Tue, 17 Feb 2009 10:45:00 GMT

A while ago a needed to do what I like to call quick and dirty testing, i.e. getting as much test coverage as possible in a given period of time. To make things worse, I was working on a client-server feature requiring a bit of setup in order to work in the first place and wanted to cover both sides with my tests and be able to easily run all tests. So I thought, what if instead of trying to get this done with one of the existing test frameworks that typically host the test target as well as its tests, I just created a new solution which already had all the pieces in place to use the feature and put a tiny test runtime in there. I did exactly that and got my testing done the way I wanted to. Again, there is a reason why I call this kind of approach quick and dirty testing and I definitely wouldn't pitch it as a best practice but it has its place.

Because of my objective this "embedded" test runtime I created was extremely small but also highly specific to the test scenarios for which it was written and thus virtually impossible to reuse. That wasn't a surprise but rather one of the tradeoffs I had opted in for. Still, I was wondering what a generalized version would look like that, next time I wanted to do something like this, I could simply drop into my application and get going. The answer to that question is EMTF.

Yet another test framework called EMTF

The Embeddable Micro Test Framework is similar to its big brothers MSTest or NUnit though obviously smaller and simpler and with the twist of it being supposed to be hosted by the application under test rather than coming with its own test host. It is available in source and binary form on CodePlex and by now I'd say that its greatest value is actually in using it for educational purposes since the reduced complexity makes it easy to understand for people without prior API testing experience (that goes for both how to use it and - at the core - how it works). That said the rest of this post is an EMTF quickstart (though I do consider a small series on how to implement an API test framework using EMTF as an example).

The easiest way to add EMTF to your project (especially if you're not using C#) is to add references to the two assemblies Emtf.dll and Emtf.Logging.dll. The first assembly contains the actual test framework that doesn't even include any logging capabilities. The second one contains a logging base class, a logger writing to the debug output and a logger writing to the console (the latter one only for the .NET Framework). There are separate versions of the two assemblies for Silverlight and the .NET Framework with the Silverlight version being a subset.

Writing tests

Let's say your project is a console application, you added EMTF to it and now want to test the following class which is part of your project (once again, please ignore the fact that the class is completely useless):

public class LameExample

{

private String _string;

public String String

{

get

{

return _string;

}

set

{

if (value == String.Empty)

throw new ArgumentException(

"New value must not be an empty string.",

"value");

_string = value;

}

We need to verify that:

The default value of the String property is null
The setter of the String property throws an ArgumentException if we pass in String.Empty
We can set the property to arbitrary values (except for String.Empty)

Ideally we want the test framework to automatically find all the tests and execute them. So in order to tell EMTF which classes contain tests, they need to be marked with the TestClassAttribute. Furthermore, test classes must be public, non-abstract and non-generic since the runtime wouldn't know which type parameters to use. Similarly, test methods are marked with the TestAttribute and must be public, non-abstract and non-generic. In addition, their return type must be void and they must not take any parameters. Tests for state and behavior of an object are performed using assertions.

[TestClass]

public class LameExampleTests

{

[Test]

public void StringProperty()

{

LameExample le = new LameExample();

// Verifies the default value of LameExample.String

Assert.IsNull(le.String);

// Verifies that the setter throws

Assert.Throws(

// Action that is expected to cause the exception

() => le.String = String.Empty,

// Action(T) verifying the exception object where

// T is the type of the expected exception.

(ArgumentException e) =>

{

Assert.IsTrue(e.Message.StartsWith("New value must not be an empty string."));

Assert.AreEqual("value", e.ParamName);

});

// Verifies setting the property to a non-null,

// non-empty string.

le.String = "fhqwhgads";

Assert.AreEqual("fhqwhgads", le.String);

// Verifies resetting the property to null.

le.String = null;

Assert.IsNull(le.String);

}

Running the tests

Since EMTF does not include any sort of test host we need to instantiate the engine, the TestExecutor, ourselves. In addition, we need to hookup the engine's events so we actually get to know the outcome of the test run. The loggers in Emtf.Logging.dll do this out of the box. Finally, calling TestExecutor.Execute() will execute all the tests we have defined that conform to the rules explained above.

private static void RunTests()

{

TestExecutor executor = new TestExecutor();

ConsoleLogger logger = new ConsoleLogger(executor);

executor.Execute();

}

This posting is provided "AS IS" with no warranties, and confers no rights.

Transitionals - About a little piece of Acropolis that wanted to become open source

ddietric — Thu, 01 May 2008 02:44:00 GMT

While Microsoft is certainly not an open source company there are some instances of code drops from, and open source projects started by teams at Microsoft that are released under licenses which comply with the definition of open source as defined by the Open Source Initiative (OSI). One well-known example is the Windows Installer XML (WiX) toolset, a more recent example is the source code for the additional Silverlight controls shipped during MIX which was released using the OSI-approved Microsoft Public License (Ms-PL).

The very latest addition to this list is a little project called "Transitionals" which - quoting from the project description - "[...] is a framework for building and using WPF transitions which provide an easy way to switch between UI views in a rich and animated way". The reason I wanted to highlight this project is that it was originally part of the incubation project "Acropolis" and while we are of course using the experience gained during the project for new technologies we are working on it is good to see that a small part of it has been released to the community in form of an open source project under the Ms-PL. Kudos to Jared Bienz for coming up with the idea and patiently pushing to make it happen (he documented the journey on his blog).

This posting is provided "AS IS" with no warranties, and confers no rights.