As most of you are already aware, we released RC version of Visual Studio 2010 recently. For those who are not aware, you can download it from this location.

I would like to discuss a few changes that we have done in Test Runner post Beta 2. The changes are mainly made after receiving feedback from our Beta 2 users and also from the amazing dogfooding effort that happened inside our team.

You might be wondering what dogfooding is. In Microsoft, we believe in using the product ourselves long before it ships to our customers. By using, I mean using it in our current project as a part of our development cycle.

For instance, we have Microsoft Test Manager(MTM) which is a Test Case Management tool and the test runner which is used in executing manual tests. For a long time, our test teams have been using MTM to test MTM. The manual tests for MTM have been executed using the Test Runner. This not only helped us uncover quite a few bugs but also made us question the usability of the product. The point is, if we do not find it useable how can we expect our customers to find it useable.

We made a few changes in Test Runner as a result of feedback from dogfooding.

Changes in Save behavior

Those who have used Test Runner in Beta 2 will remember that the test would be disabled after you saved it in the Test Runner. This was a one way operation and the user could not do anything on the test after saving it. This behavior was found to be very limiting by our users. Post Beta 2, we changed this behavior. Now the user can save test anytime after starting the test. The test is not disabled after save operation. The user can perform all operations on the test even after saving it. This is more like saving a document in an editor. Of course there are a few subtleties here. How and when are diagnostic data saved? What happens to automation? e.t.c. We have tried to handle these scenarios in an intuitive way. After this change, the manual test execution model has definitely looked better.

Notice the (*) after test case id in the image below indicating a dirty test. At this point, if you save it, the test is saved and the (*) goes away and you can continue with your testing.

Cautioning user to mark validate step

Another thing that users found confusing was related to auto-computation of test result when they ended the test. Sometimes, not marking any step result would auto-compute to pass and sometimes to fail. The main difference is the presence of validate step in the test case. A validate step is one which has an expected result. If the user does not mark a validate step to pass, then the test result is always auto-computed to fail. If there are no validate steps, then not marking any step auto-computes to pass. We have included some gestures in the Test Runner to caution users about this.

This has helped us in handling fast-forwarding scenario too. How? Currently, we do not support validate actions in our action recorder. For instance, we can create action recording for actions corresponding to steps “Open calculator”, “Click 3 * 3”, and “Click =”. But what about the action recording for verifying that the result is 9. We do not have a way to automate it from our Test Runner. This has to be validated manually. The expected workflow here is to create action recording region for steps that can be automated which ends in a validate step. See figure below for more on this.

What I have here is an action recording region that ends in the validate step. Next time when I run this test, I just playback all actions until “Check the result” (I do not perform them manually). Since an action recording region ends there, I can play until that step. Then I validate the result manually and go on with rest of the actions.

While recording actions for the test case, we give a different warning message if the validate step is not marked.

Launching application after starting the test

In Beta 2, to collect diagnostic data for Intellitrace and Test Impact, we expected the users to invoke their application after starting the test. This was a bit confusing because sometimes the user would just forget this and do it out of order only to realize at the end of testing that this data has not been collected. We changed this behavior post Beta 2 to let user open the application soon after tests are loaded in the Test Runner. Now the user does not have to remember to start the test before opening the Application under test.

Performance Improvements

Another revelation that we had as a part of dogfooding effort is the performance issues we had while loading tests in Test Runner, closing test runner e.t.c. The issues were very annoying in a remote server scenario with the server located in Redmond and client machines located in India. The server team did a lot of improvements in this regard and we were able to bring down test load time significantly. Just to share one data point, we used to take 75s to load 24 tests in Beta 2(in remote server scenario) and this would increase linearly. We brought it down to 12s post Beta 2. The good news is that the load time is no longer a linear function of number of tests loaded. I was able to load 150 tests in less than 16s.

I will share more changes we have done in my subsequent blogs. For now, I request you to move to RC bits and start using the product. As always feedback is welcome.