Dev Oriented Testing and Other Ramblings

ooh Camano… ooh planning testing in the November CTP

In several of the organizations that we've visited we've found some that some of the more experienced ones tend to put much thought into planning their testing efforts.  One of the organizations that we visited, followed an agile testing process with sprints of four weeks in duration as follows:

Before the sprint, the test manager created a test plan for the release with her team.  The test plan served to identify the area to be tested in the sprint.  For the next two weeks, the testers wrote test cases in excel to specify the steps to verify the new functionality.  Each row in the excel file consisted of a test case with columns used to specify its steps and record the results of its execution.  In the closing weeks of the sprint, the testers executed all the test cases that they have authored in the prior weeks twice - once in week 3, once in week 4. 

While the very organized effort above is bound to locate most of the holes in the product, the tools that the organization used to plan and manage the testing effort were not helping them with the process.  The programs provided them with no easy way to organize their test cases, track the requirements associated to them, quickly view test case results, plan a testing effort or view the overall progress of their testing.

In the November CTP release of Rosario we introduce Codename 'Camano' - a tool designed to help testers and test managers to plan, organize and analyze a testing effort.   In Camano, each test case is a database object that can be executed by Microsoft Test Pilot.  Testers can organize their test cases into "Suites" that can by either "Dynamic" (query defined) or "Static" (list defined). A lead can schedule a suite of tests to be executed by placing the suite in a new test plan.  When creating a test plan, the lead can also divide up the testing effort by assigning different members of their org to execute specific test cases.  Once a plan is created, the testers execute test cases out of it. 

Once a test plan is created, managers will be able to track its progress, and add and remove test cases to the plan as new test cases are authored and the plans change.  Management will also be able to generate reports against the execution of test cases contained in the plan.  We are hoping that users will find it very useful to be able to quickly generate reports on the number of test cases currently failing, the number of blocked test cases, the testers that are finding the most bugs, the speed at which bugs are being fixed and the number of testers petitioning to put Arrested Development back on the air :).  Test Managers at Microsoft love getting such statistics, we have been planning that those outside of Microsoft will find them quite handy as well (we have 3.5 testers on our team currently currently petetioning). 

Have any thoughts on our thinking around planning a testing effort?  Are we completely off wack?  A little off wack?  I would be very interested in your feedback - as we really want to get the planning experience right.

Cheers,

- Naysawn

Who are Generalist Testers?

New Unit Testing Features in Orcas (Part 2)

This is a part of a series of postings on the details of some of the new features appearing in Orcas.  All these features will be available in Orcas beta 2.

Gaining Access to Private Code under test:

In Whidbey, we introduced the concept of private accessors to provide users with an easy way to access private members of code under test. Private Accossors served as a wrapper for reflection, consisting of shadow implementations of the methods of a class which provided access to a private field or method through a reflection call - hiding all the nitty gritties of reflection from the user.  Although, it seems that users love the concept, we decided to rewrite the feature to make it more robust in Orcas.

In Orcas, we decided to change private accessors from being code based to assembly based.  The assembly based assessors are more robust and are picked up by intelliscene and auto-regeneration when the code under test has changed.  As in Whidbey, Orcas private accessors are automatically generated whenever generating unit tests for private methods, they can be manually generated by right clicking inside the code editor and selecting Create Private Accessor.  Generating private accessors can also be achieved via the command line via running publicize.exe (new in Orcas).  I have the impression that the majority of users are still unaware of this cool feature, so I hope more will play with it in Orcas Beta 2 and RTM.

Generating Unit Tests:

For those new to unit testing, VSTS contains a nice feature to generate an outline of a unit test for a given method of code under test.  In Orcas, we have re-written the generation engine to be more robust, to work more intelligently with the code under test and to functional with the new assembly based private accessors.  We have also beefed it up to better support .Net languages, specifically .Net reflection.

Support for Device Projects:

In Orcas, we introduce support for unit testing on embedded devices.  All the functionality described, in these posts, and that which was present in Whidbey will be accessible to device programmers.

Automatic Cleanup of Test Results and Their Associated Deployments

One of the pain points we often hear from customers concerns deployments inadvertently filling their hard drive as they execute more and more tests.  To assist these users, we have designed Visual Studio Orcas to automatically remove old deployments when more than 25 test results have filled the Test Project’s Test Results directory.   One can change the number of old deployments maintained on disk in Tools->Options->Test Tools->Test Execution.

New Unit Testing Features in Orcas (Part 1)

This is a part of a series of postings on the new unit testing features appearing in Orcas.  All these features will be available in Orcas beta 2.

I know that MSTest isn’t yet the darling of the TDD world, but I’d like to think that our TDD story has improved a decent amount in Orcas.  These are some of the improvements:

1 – Better Execution Times: We have rewritten our test execution engine to significantly improve the time that it takes to execute unit tests from the command prompt and via the IDE. 

2 – Run Tests: We have added a menu item to execute the tests in the current context of the test class’ code.  Executing “Run Tests” from a right click context menu will execute the best guess of the tests we think you would like to execute.  Therefore if the focus is on a test method, Run tests will execute the test method, if the focus is in a test class, Run Tests will execute all the tests in the test class, if the focus is in the test results window, Run Tests will execute the tests which are checked, etc.  Ctl R, T also accesses this command. I think you guys will love it. 

3 – Short Cut Keys to Run Tests: We have added in a lot of short cut keys to kick off a test quickly.  They are as follows:

·         Ctl R, T: Run Tests in Current context (see #2)

·         Ctl R, C: Run Tests in Current Test Class

·         Ctl R, N: Run Tests in Current Namespace

·         Ctl R, S: Run All Tests in Solution

·         Ctl R, D: Run the Tests in the Last Test Run

·         Ctl R, F: Run the Failed Tests of the Last Test Run

Holding Ctl while pressing the second key in the key combo executes the tests with the debugger.  We hope that these additions will significantly improve the TDD flow.

4 – Disable Deployment:  As many of you are aware, MSTest currently “deploys” tests for every test run it executes.  This means that in addition to generating a Test Results (trx) file, it also copies all the test binaries, code coverage files and debug information to a deployment folder, where the tests are executed.  This design provides for convenient features for collaboration, specifically publishing old test results and their deployments, allowing another team member to  re-run the tests and step through a test run.  It is however, inconvenient for TDD users, since most TDD’ers want their tests to execute as fast as possible.  We have therefore added an option in the Test Run Configuration to disable deployment.  To do so, go to the Test Run Configuration file in a Test Project (double click on *.testrunconfig in solution explorer), Deployment, and uncheck “Enable deployment.”

5 – Inheritance of Tests:  In Whidbey, we did not support the inheritance of tests methods or initializations.  We have added support for inheritance in Orcas.  I’m planning to use a future post to dive in to the nitty gritties of Test Inheritance.

6 – Directly go to the Point of Failure: We have enabled an option in Orcas to directly jump to the point of failure.  We have made the stack trace in the Test Results window clickable as well.  To specify that double clicking on a failed test result takes you to the first element in the stack trace, go to Tools->Options->Test Tools -> Test Execution and check the last box in the window.

Props to Alan Ridlehoover, Peter Provost and Peli de Halleux for their help in trying to get the TDD experience right. J

Test Lists Going Pro

Unit Testing Trickling into Pro!

That Pesky MSTest Execution Ordering..

I have come upon several groups who are puzzled by the intricacies of execution of the MSTest Framework.  I admit - it is quite confusing, so I hope that these postings will help to clean some of the confusion.

I think that most confusion comes from some user’s expectation of MSTest to execute like the Nunit framework.  They execute differently since Nunit instantiates a test class only once when executing all the tests contained in it, whereas MSTest instantiates each test method’s class separately during the execution process, with each instantiation occurring on a separate thread. This design affects 3 specific things which often confuse users of MSTest:

1.      ClassInitialize and ClassCleanup: Since ClassInitialize and ClassCleanUp are static, they are only executed once even though several instances of a test class can be created by MSTest.  ClassInitialize executes in the instance of the test class corresponding to the first test method in the test class. Similarly, MSTest executes ClassCleanUp in the instance of the test class corresponding to the last test method in the test class.

2.      Execution Interleaving: Since each instance of the test class is instantiated separately on a different thread, there are no guarantees regarding the order of execution of unit tests in a single class, or across classes. The execution of tests may be interleaved across classes, and potentially even assemblies, depending on how you chose to execute your tests. The key thing here is – all tests could be executed in any order, it is totally undefined.

3.      TextContext Instances: TestContexts are different for each test method, with no sharing between test methods.

For example, if we have a Test Class:

(This consists of 3 Test Methods, ClassInitialize, ClassCleanup, TestInitialize, TestCleanUp and anexplicit declaration of TestContext)

The execution order would be as follows:

Test1 [Thread 1]: new TestContext -> ClassInitialize -> TestInitialize -> TestMethod1 ->  TestCleanUp

Test2 [Thread 2]: new TestContext -> TestInitialize -> TestMethod2 ->  TestCleanUp

Test3 [Thread 3]: new TestContext -> TestInitialize -> TestMethod2 ->  TestCleanUp -> ClassCleanUp

The output after running all the tests in the class would be:

Class Setup

Test Init

Test1

TestCleanUp

Test Init

Test2

TestCleanUp

Test Init

Test3

TestCleanUp

ClassCleanUp

 (A special thanks to Dominic Hopton for his edits of my hackery on this post :))

Comparing the MSTest and Nunit Frameworks

I haven't seen much information online comparing the similarities and differences between the Nunit and MSTest Frameworks.  Here I will define the similarities and some of the differences.  If there is anything else which you come upon, please do add it to the comments.

 The order of execution is similar in both frameworks, but there are some differences between the two:

• In Nunit, tests are not executed in parallel.  Rather, it appears that all tests execute on a single thread.  In MSTest, each test is instantiated on a separate thread, this results the runs being interleaved.  Therefore, if test A depends on test B for its success, it likely will fail as test B will likely start running as test A is running.
• The time at which ClassCleanUp/TestFixtureTearDown executes is different between the two frameworks.  In Nunit, TestFixtureTearDown is executed immediately following the completion of the last test in a TestFixture or after TearDown if the attribute exists for the test in question.  In the Whidbey implementation of MsTest, ClassCleanUp executes at the end of a test run, before AssemblyCleanUp but not necessarily immediately after the last test in a TestClass has completed executing.
• In Whidbey, support for test class inheritance was missing.  In Nunit, it is fully supported.  This will be rectified in Orcas.

I should also mentioned that in MsTest, TestContext exists for passing information about the test run.  There is no equivalent in Nunit tests.  This can serve as a handy tool for pulling information from datasources on the disk to the unit tests, as well as other uses.  More can be read about it here.