That Pesky MSTest Execution Ordering..

nnaderi — Sat, 17 Feb 2007 04:41:00 GMT

I have come upon several groups who are puzzled by the intricacies of execution of the MSTest Framework. I admit - it is quite confusing, so I hope that these postings will help to clean some of the confusion.

I think that most confusion comes from some user’s expectation of MSTest to execute like the Nunit framework. They execute differently since Nunit instantiates a test class only once when executing all the tests contained in it, whereas MSTest instantiates each test method’s class separately during the execution process, with each instantiation occurring on a separate thread. This design affects 3 specific things which often confuse users of MSTest:

1. ClassInitialize and ClassCleanup: Since ClassInitialize and ClassCleanUp are static, they are only executed once even though several instances of a test class can be created by MSTest. ClassInitialize executes in the instance of the test class corresponding to the first test method in the test class. Similarly, MSTest executes ClassCleanUp in the instance of the test class corresponding to the last test method in the test class.

2. Execution Interleaving: Since each instance of the test class is instantiated separately on a different thread, there are no guarantees regarding the order of execution of unit tests in a single class, or across classes. The execution of tests may be interleaved across classes, and potentially even assemblies, depending on how you chose to execute your tests. The key thing here is – all tests could be executed in any order, it is totally undefined.

3. TextContext Instances: TestContexts are different for each test method, with no sharing between test methods.

For example, if we have a Test Class:

[TestClass]

public class VSTSClass1

{

private TestContext testContextInstance;

public TestContext TestContext

{

get

{

return testContextInstance;

}

set

{

testContextInstance = value;

}

[ClassInitialize]

public static void ClassSetup(TestContext a)

{

Console.WriteLine("Class Setup");

}

[TestInitialize]

public void TestInit()

{

Console.WriteLine("Test Init");

}

[TestMethod]

public void Test1()

{

Console.WriteLine("Test1");

}

[TestMethod]

public void Test2()

{

Console.WriteLine("Test2");

}

[TestMethod]

public void Test3()

{

Console.WriteLine("Test3");

}

[TestCleanup]

public void TestCleanUp()

{

Console.WriteLine("TestCleanUp");

}

[ClassCleanup]

public static void ClassCleanUp()

{

Console.WriteLine("ClassCleanUp");

}

(This consists of 3 Test Methods, ClassInitialize, ClassCleanup, TestInitialize, TestCleanUp and anexplicit declaration of TestContext)

The execution order would be as follows:

Test1 [Thread 1]: new TestContext -> ClassInitialize -> TestInitialize -> TestMethod1 -> TestCleanUp

Test2 [Thread 2]: new TestContext -> TestInitialize -> TestMethod2 -> TestCleanUp

Test3 [Thread 3]: new TestContext -> TestInitialize -> TestMethod2 -> TestCleanUp -> ClassCleanUp

The output after running all the tests in the class would be:

Class Setup

Test Init

Test1

TestCleanUp

Test Init

Test2

TestCleanUp

Test Init

Test3

TestCleanUp

ClassCleanUp

(A special thanks to Dominic Hopton for his edits of my hackery on this post :))

Testmundo : Execution Order

That Pesky MSTest Execution Ordering..