Anutthara's WebLog : VSTS Dev10

Our first raving fan for Team Test in VSTS 2010!

anutthara — Thu, 16 Jul 2009 08:09:00 GMT

Yay - I finally got to see our Dev10 Team Test bits being used in production! E&D team in in MSIT, India is going live on VSTS 2010 for their test case management. These guys are pretty avid dogfooders and have taken to the product pretty fast - creating their test cases, plans, suites...on the server and using Test and Lab Manager in their daily lives. We have a whole slew of early adopters using VSTS 2010 for production and MSIT, our internal IT division is an important dogfooder for us.

As Amit tells us all the time, one of our goals for VSTT 2010 is to create “raving fans” of our product - and it's great to see our first local raving fan for Team Test! :-)

Video recording crashing your Test Manager in VSTS 2010 Beta1?

anutthara — Fri, 26 Jun 2009 08:00:00 GMT

Getting a video recording of your test to attach to the bug you file on your developer can be a really cool thing. And the Microsoft Test and Lab Manager tool makes it a breeze to do that.

However, if your MTLM is crashing or erroring out when you try to pick up a recording,try the following:

1. Check if WME encoder 9 is installed on the box. If not, install it from here

2. If you are running Vista, Windows 7 or Windows 2008, install the patch for the encoder from here

As Thomas Schissler, one of our most active Team System MVPs points out, the docs indicate this patch is required for Vista, but you need it for Win7 and W2K8 also

UI automation breaking due to window title changing each time?

anutthara — Thu, 11 Jun 2009 07:29:00 GMT

When testing UI applications, specially web apps, we frequently encounter the problem of window title changing with the state of the app or operations done on the app. For instance, when checking web mail, the title of the page could read “Inbox(35) - Microsoft Internet Explorer” where 35 really represents the number of unread email in your inbox. Now, when I open it at any other time, it might read “Inbox(36) - Microsoft Internet Explorer” based on unread mail. For GUI testers, this poses a problem since object identification works usually on the basis of window title and the title changing each time I run the test causes object identification to fail. :(

To solve this problem, the API of VSTS 2010 has a “Smart Match” feature built into it. Thej Kumar is our smart young developer that works on the record and play engine of Team Test and today, he will explain how smart match works. So, here you go:

Baseline assumptions:

The assumption is that the title of the window is of the form <Title> - <Application Name> or <Application Name> - <Title>.
The assumption is that the last hyphen separates the title from the application name.

How does the heuristic work:

If the application name matches completely, we ignore it in the calculation of score for a window. In the above example, “Microsoft Internet Explorer” is a complete match. Hence the scores are calculated only for Inbox(35) and Inbox(36).

Whichever window has the longest common string will have a higher score. Example: “Inbox, 45 unread” and “Inbox, 46 unread” will be a 50% match because the longest common string is 8 (Inbox, 4) out of the total length of 16. On the other hand, “Inbox, unread(45)” and “Inbox, unread(46)” will be excellent matches because the longest common string is 15 out of 17.

Special case for MDI windows:

Title changes based on whether or not the MDI window is maximized or restored. Eg: if the title of the main window is A and the title of MDI window is B, if B is maximized, the title of A changes to A – [B]. To handle these cases, we give a high score to windows whose one half matches and the other half of only one of the windows is empty.

If two or more windows get a good score, the window on the top is picked up.

Good things to do regarding titles to ensure smart match picks works correctly:

Ensure that the last hyphen separates the title from the application name and in general, avoid too many hyphens in the title.

Ensure that part of the title that changes is present either in the beginning or in the end.

Ensure that the window which you want to be picked up is present on top of the other windows.

Sometimes on a rare occasion, smart match does end up being oversmart ;-) and gives false positives, but those scenarios can be worked around. Tell us what you think of smart match!

Your developer can never say "no repro" to your bug ever again!

anutthara — Wed, 05 Dec 2007 17:03:00 GMT

You have already used the bug tracking system (WIT) in TFS before. So, we integrated this bug tracking system with MTR so that you can file bugs with a single click from the runner itself. What's the big deal about this you might ask?

Well, the big deal is that we not just create a bug for you from the runner itself, we also populate it with the test case that you have just been executing when you opened the bug. And that too in this neat little repro steps control with a tab of its own in the bug form. (Yes, we modified the MSF template to include some extra tabs in the default bug) So, all comments that the tester has written, notes she has taken, attachments that she has made are all uploaded automatically to the bug from the first step to the last failed step.

Now, how often have you hit a bug during testing that would just repro on your test machine but simply would not repro on the dev machine. And how often has it turned out that it was a configuration related issue - different service packs, different CPU usage, different memories, the list is endless. To ensure that the developer gets a complete picture of the environment where the bug was actually produced, we fill in system information about the test machine automatically into the newly created bug. Check out the "system info" tab within the bug form:

Now, even with all of these, what if the developer still cannot repro the bug? Perhaps it was a domain expert that filed the bug and not a technical tester. He/she may not have noticed subtleties that might have affected the system and caused the bug. Or the bug repro steps or description maybe woefully inadequate for the developer to produce a reasonable repro. In such cases, the video recording that the runner makes in the background during all test execution, will be considerably useful. Playing back the video will give the developer a very detailed and accurate picture of the exact steps leading up to the bug. This video is filed as an attachment to the bug

What if the test itself took 10 long minutes. Now, the failure is in the last step perhaps. The developer may want to look at only the failed step instead of the whole long video. Video markers are the way to go for that. Each time the tester marks a test step in the test case, a video marker is inserted with that timestamp in the video file. These markers are then populated against the test steps in the repro steps of a bug filed via the runner. When I click on these markers, I can play back from the point where the previous step was marked. This gives the developer finer control over what portions of the video file may be useful.

So, how do you like our actionable bug? Are there any other ways we could help developers get the maximum effective amount of information from the bugs?

Rosario November CTP - did you play it back yet?

anutthara — Sat, 01 Dec 2007 09:11:01 GMT

Right after we RTM-ed Orcas last Monday, we are out with the November CTP of Rosario VSTS. Go on over here and give it a spin. We have a ton of stuff that's new with VSTS in there - more on BHarry's blog.

But, of course, I am excited about the manual test and AFN portions that we built out of my team here. Check out the awesome background recording that we do while you are out executing your manual tests in MTR. Next time you want to run the same test, just "play back" the recording of your manual tests. We support web apps for now, winforms is on the way...

We've also re-done the test case manager from the Aug CTP - there is now this sleek and stylish stand-alone test case manager where you manage all test activity from. We figure Visual Studio may not be the best environment for testers to go do all of this work from - so we have it completely stand alone now. Our team blog has a nice walkthrough of the TCM features - do skim through the post.

So, what are you waiting for? Go try it already and tell us what you think...

PS: Oh, given the huge size of the VPC, you might just want to install it in convenient parts as provided on the download page.

What's new in Rosario for VSTT: Manual Test Runner

anutthara — Fri, 10 Aug 2007 05:10:08 GMT

Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester and make her life much easier than it currently is. The primary intent of the MTR is to help the manual tester create an actionable bug with which the developer is much more empowered to debug issues without the bug ping-pong happening between the dev and tester. Like I have mentioned before in my blog, it is pretty tedious for a tester today to even capture screenshots while filing bugs. MTR gives the tester one-click screenshot capture which operates in 3 different convenient modes and attaches itself to the test result as well. There is also a one-click bug filing option in which the bug is pre-populated with test step details from the test case that has already been loaded up in the MTR.

In this blog entry, I will describe the basic end to end scenario using the MTR to execute test cases and publish test results. The last entry, we saw how to create test cases and test runs in the TCM client. To launch MTR, open the VS command prompt, and type msrun. This launches the runner window on your desktop. Here, point to File->Open. You will be prompted with a dialog to connect to a TCM server similar to the "Connect to server" dialog that pops up when you open the team explorer window to connect to a TFS server. Choose the TCM server (in case of the VPC, the server is on the same machine) and choose the team project on the server.

Now, you will see the "Open Test Run" dialog, where you can pick any test run that you want. Let's choose a test run and say OK.

The runner window is populated with the test cases that are the result of the query associated with the run and selected in the dialog. Now, you can open your application under test (aut) and go about executing the tests that you loaded in the run. You can mark each test step with results of pass/fail or inconclusive through multiple ways including a handy single click.

You can enter comments in the comments text box or add attachments using the "Attach" option in the menu. Once the test steps are marked, click on "End test case" to compute the result of the test case from the test step results. You could also mark test cases globally without getting into the test steps at all. And if dissatisfied, you can just right click on the test case and say "Reset test case" to erase all results marked till now and start afresh. Once the test case is complete, you can publish the test case result to the server through the "Publish" button on the toolbar. These results are then published to the server and the test case grayed out to prevent further marking of steps.

In future entries, we will look at screenshots and bug filing in more detail. And of course, our glitzy video recording functionality will feature in its own entry as well. :)

What's new in Rosario for VSTT: Test case management

anutthara — Tue, 07 Aug 2007 05:33:00 GMT

Rosario has test case management in a full fledged fashion as part of the VSTST feature set. So, now test cases are first class citizens in the VSTS world with them having their own exclusive test case manager tool window inside the IDE. Test cases are a separate work item type with pre-defined fields like Owner, Priority, Test steps etc.

Click on the Test->Windows->Test Case Manager option in the IDE to launch the test case manager window shown at the right most side of the IDE below. The test case manager shows the selected team project TestTeamProject with its set of test cases, test run definitions, test runs. You can create a new test case from the TCM window. A new test case pops up with the test case form.

The test case form has a "Steps" tab as shown below:

You can enter descriptive steps for the test case detailing all the actions that need to be performed as part of the test case. You can mark a test step as "Validate step" explicitly using the icon shown in the pic (highlighted using a red dot next to the icon). The test steps shown are also parameterized using data driving. More on that in later posts. Now, you can define a test case query akin to the work item query. All those defined queries are shown in the test cases node in the TCM tool window as shown above.

With the test cases created, you can also create a run definition with the test case query specified as shown in screen shot below. This run definition is used to create a logical cluster of test cases - say tests for a certain component/feature etc.

Now, you can instantiate a new test run using the test run definition created above. The test run can be associated with a build so that the test run results can be published against the build.

Once the test run is created, you can open the test run from the TCM by querying all test runs. The current state of another sample test run in a graph is as shown below:

Now that we have a test case, run definition and test run created, we will delve into using MTR to execute our test cases in further blog posts.

Rosario is here!

anutthara — Mon, 06 Aug 2007 10:05:00 GMT

If you still(!!!) haven't heard, Rosario CTP is out in the form of a VPC here :) Doug's whitepaper should give you enough info to take it out for a spin.

Manual Test Runner(MTR) is a key part of the CTP. It is a tool designed to help manual testers to load up their test plans and execute those with the support of additional tools like one-click screenshot, one-click bug with pre-filled repro and more such exciting stuff. Test case management(TCM) is another key piece where we have tried to fill the missing piece in the existing VSTT SKU. There is a tool window right inside VS IDE where you can create/modify/manage test cases, test run definitions and test runs and save them in the TCM server which has your team project's source code and work items. Then, load up the test runs in MTR and go. Publish the test results back into the TCM server and look at the cool reports that show how your test team is doing.

Do check it out and tell me how it goes!

We signed off on the first CTP release for Rosario

anutthara — Thu, 26 Jul 2007 12:43:14 GMT

Rosario is the next in line version for Team System after Orcas. And we are doing this CTP when Orcas has not even RTM-ed yet! Isn't that pretty cool? It's been a bit crazy managing multiple releases, but then, we want Rosario to hit the shelves in a short time after Orcas releases - faster new release. And no - I am not allowed to define "short time" :P

I've been working on the VSTT portions of this CTP for some time now and this is certainly one of the most exciting set of features I have seen for testers. Though I cannot wait to talk about all the awesome new stuff in the release, I'll not give out any more info until the CTP is out. Stay tuned for more...

We'll put it out as a VPC download on our website so that there are no installs that you need to do separately. Just set up the VPC and go...

Testers only, please!

anutthara — Thu, 17 May 2007 06:53:00 GMT

Here is the address for a new testers' club: http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=1600&SiteID=1

Ok, so, it's yet another forum where testers can get together and share info, solve problems collectively. Why would I be excited? Well, that's because this is the first time we have had something like this on our very own MSDN forums - it's like a party for your buddies at your own house :)

I have been at Microsoft for four years now and I know for sure that we take testing really seriously here. We have some kick ass people thinking about process, tools and engineering practices specifically for testing. IMO, we have not been talking about our testing processes and tools externally as much as we could. Conversely, I am sure there are other testers with some pretty cool processes, tools and ideas that they would love to share with folks outside. This forum looks like the right place to share such stuff.

As a tester in Microsoft, I am glad that we have a forum now on MSDN, where we can talk to other testers in our ilk, not just about Microsoft products, but testing in general. I am pretty thrilled about the new forum and I hope this is a useful platform for testers to build a great community.

And yes - you can expect to see me shouting over the roof tops about my current favorite topic of manual testing and why I think it's the black sheep of all testing topics ;-) See you there!