Clark Sell : Testing

Bye Bye VSMDI

csell — Mon, 06 Nov 2006 17:39:04 GMT

That blasted VSMDI file. Well Buck, Tom and crew did it. They have finally given us the ability to run tests ( albeit all ) found in an assembly regardless of test lists. http://blogs.msdn.com/buckh/archive/2006/11/04/how-to-run-tests-without-test-metadata-files-and-test-lists-vsmdi-files.aspx I am currently integrating this into TfsAlert and will post my findings.

Simple NMock Working Sample

csell — Tue, 24 Jan 2006 05:17:47 GMT

Based on some conversations I had today, here is a simple NMock example.

NMockExample.zip (17.19 KB)

Converting a csproj into a test csproj

csell — Sat, 21 Jan 2006 17:36:27 GMT

Short answer.

It's as simple as adding one line to your csproj which in turn tells the IDE to treat that project like a test project. Add the following line to your main PropertyGroup:

<ProjectGuid>{04082EBA-C85C-4336-B3FD-9891096BAA0F}</ProjectGuid>

Then just reference Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll and you are on your way. Be sure to verify that Guid between releases.

Long Answer.

When I start a project I create the base set of known assemblies that project will need. I also add an additional test project which will contain any Scenario, Load and Stored Procedure tests. Then I convert all assemblies to test projects. I am a firm believer in Test Driven Development. IMHO it is the only way to properly write code. Having said that over the years I have developed my own set of TDD practices; one of which is the physical placement of your unit test classes in the assembly their testing. Why? Well I will explain that in a future post call Unit Testing with Partial Classes.

With any decision there are of course trade-offs. In this case the trade-off is around the IDE interaction with hosting type projects, ie exe or web. After you convert an exe to a test project, when you decide to run ( F5 ) it will actually fire off Test Manager rather than the launch app. If you are a TDD bigot like me this is actually the correct action; if your not, I know your already complaining. BUT all is not lost! There is a little tool us TDD bigots keep in our back pocket at all times, TestDriven.Net. This is an Visual Studio add-in and it allows you to execute your unit tests with just a right click. If you choose to use this tool it isn't necessary to convert your project to a test project but rather just reference the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll. This will allow you to fire your app as intended and run your tests with the same project configuration.

Disclaimer: this is not a Microsoft recommend practice. You can find more information about the Visual Studio Team System Testing Framework here.

Unit Testing with Anonymous Methods

csell — Sat, 21 Jan 2006 04:19:45 GMT

Mark Seemann recently published a great post demonstrating the use of anonymous methods in unit testing events. Testing Events Using Anonymous Methods

That just rocks, thanks Mark!

Multiple Test Runs in one BuildType

csell — Wed, 18 Jan 2006 08:51:14 GMT

If you are using TeamBuild then this post is for you.

In TeamExplorer when you create a new build type it will create a *.proj file. During the creation of this build type, you will be asked which TestLists you would like executed in that build type. Once completed if you open that proj file, you will see an item group that looks something like the following:

<ItemGroup>
<MetaDataFile Include="$(SolutionRoot)\TestProject1\TestProject1.vsmdi">
<TestList>TestList1;TestList2</TestList>
</MetaDataFile>
</ItemGroup>

For this build type there are actually 2 different TestLists, TestList1 and TestList2. These TestLists will get executed and reported on in the build. With the out of box ItemGroup configuration both lists will actually roll up into one set of test results; but what if you wanted to to actually have multiple test runs for a given build.

We can make a very slight modification to that generated code block and actually get a separate test run per test list.

<PropertyGroup>
<VsmdiPath>$(SolutionRoot)\TestProject1\TestProject1.vsmdi</VsmdiPath>
</PropertyGroup>

<ItemGroup>
<MetaDataFile Include="$(VsmdiPath)">
<TestList>TestList1</TestList>
</MetaDataFile>

<MetaDataFile Include="$(VsmdiPath)">
<TestList>TestList2</TestList>
</MetaDataFile>
</ItemGroup>

More to come....

ExpectedException Exception Message Validation

csell — Mon, 16 Jan 2006 18:28:06 GMT

While you cannot validate you exception's message in the ExpectedException attribute all is not lost. Lets walk through three scenarios.

In all three tests you will see an Assert.Fail in the try block. You need this statement in case the target doesn't throw any exceptions, therefor failing the test. The first example is a passing test, throw and exception with a message, catch it and compare the message string.

   [TestMethod]
   public void PassingException()
   {
      const string expectedExceptionMessage = "Foobar Message";

      try
      {
         throw new ApplicationException( expectedExceptionMessage );
         Assert.Fail( "Expected an exception to get thrown by target" );
      }

      catch ( ApplicationException ex )
      {
         Assert.AreEqual<string>( expectedExceptionMessage, ex.Message );
      }
   }

Here we see the same test failing only because the message contained in the exception is different.

   [TestMethod]
   public void FailingOnExceptionMessage()
   {
      const string expectedExceptionMessage = "Foobar Message";

      try
      {
         throw new ApplicationException( "Invalid" );
         Assert.Fail( "Expected an exception to get thrown by target" );
      }

      catch ( ApplicationException ex )
      {
         Assert.AreEqual<string>( expectedExceptionMessage, ex.Message );
      }
   }

While this last test follows the same pattern seen above the test is really only testing for the type of exception being throw. This should be refactored to use the ExpectedException attribute.

   [TestMethod]
   public void FailingOnExceptionType()
   {
      const string expectedExceptionMessage = "Foobar Message";

      try
      {
         throw new Exception( expectedExceptionMessage );
         Assert.Fail( "Expected an exception to get thrown by target" );
      }

      catch ( ApplicationException ex )
      {
      }

      catch
      {
         Assert.Fail( "Incorrect Exception type was thrown by target" );
      }
   }

Same test refactored, this test still fails but only to illustrate that example.

   [TestMethod, ExpectedException ( typeof ( ApplicationException) )]
   public void FailingOnExceptionType()
   {
      throw new Exception( "..." );
   }

ExpectedException might not be what you've expected

csell — Sat, 14 Jan 2006 20:34:50 GMT

If you've switched from NUnit to the VSTS Unit Testing then I am pretty sure you've used the ExpectedException attribute. If you haven't it looks something like this:

[ExpectedException( typeof( ApplicationException ), "Message" )]

Using it with a test would look like the following:

[TestMethod, ExpectedException( typeof ( ApplicationException ) ) ]
public void ut1()
{
throw new ApplicationException();
}

[TestMethod, ExpectedException( typeof( ApplicationException ), "foo" )]
public void ut2()
{
throw new ApplicationException("bar");
}

You would naturally expect test ut1 to pass and ut2 to fail. ut2 should fail on the bases that the message contained within the exception was in fact different. Not true. Both actually pass and not because of a bug but rather a new feature introduced to the attribute. The Message property is actually a message that prints out with the test when it fails; just like the optional message property you find on all of the Assert methods.

Assert.IsTrue( false == true, "Developer put explanation if this test was to fail" );

That message will print with the output of the test like so:

Error 1 TestCase 'ExpectedExceptionTest.ExpectedException.ut3' failed: Assert.IsTrue failed. Developer put explanation if this test was to fail
at ExpectedExceptionTest.ExpectedException.ut3() in C:\MyPrograms\ExpectedException\ExpectedExceptionTest\ExpectedException.cs:line 26 C:\MyPrograms\ExpectedException\ExpectedExceptionTest\ExpectedException.cs 26

Lets correct ut2:

[TestMethod, ExpectedException( typeof( ApplicationException ), "I failed because I am looking for the wrong excpeption type" )]
public void newUt2()
{
throw new Exception( "foobar" );
}

This will in fact fail because the Exception type thrown was different. When it fails we will see that message in the output.

Having said all that, there is in fact a bug. When you run a test with that ExpectedException attribute in Visual Studio you will not see the message on the failure output; if you run it in MSTest.exe you will.

ExpectedException.cs (.75 KB)

MSTest.exe

csell — Sat, 14 Jan 2006 18:12:01 GMT

Stumbled across a command line tool ( MSTest ) which executes your VSTS test list. You can call if from a Visual Studio command prompt or @ C:\Program Files\Microsoft Visual Studio 8\Common7\IDE

Upon execution this will create the same output as if you executed your tests inside of Visual Studio.