Anutthara's WebLog : Team Test

Video recording crashing your Test Manager in VSTS 2010 Beta1?

anutthara — Fri, 26 Jun 2009 08:00:00 GMT

Getting a video recording of your test to attach to the bug you file on your developer can be a really cool thing. And the Microsoft Test and Lab Manager tool makes it a breeze to do that.

However, if your MTLM is crashing or erroring out when you try to pick up a recording,try the following:

1. Check if WME encoder 9 is installed on the box. If not, install it from here

2. If you are running Vista, Windows 7 or Windows 2008, install the patch for the encoder from here

As Thomas Schissler, one of our most active Team System MVPs points out, the docs indicate this patch is required for Vista, but you need it for Win7 and W2K8 also

UI automation breaking due to window title changing each time?

anutthara — Thu, 11 Jun 2009 07:29:00 GMT

When testing UI applications, specially web apps, we frequently encounter the problem of window title changing with the state of the app or operations done on the app. For instance, when checking web mail, the title of the page could read “Inbox(35) - Microsoft Internet Explorer” where 35 really represents the number of unread email in your inbox. Now, when I open it at any other time, it might read “Inbox(36) - Microsoft Internet Explorer” based on unread mail. For GUI testers, this poses a problem since object identification works usually on the basis of window title and the title changing each time I run the test causes object identification to fail. :(

To solve this problem, the API of VSTS 2010 has a “Smart Match” feature built into it. Thej Kumar is our smart young developer that works on the record and play engine of Team Test and today, he will explain how smart match works. So, here you go:

Baseline assumptions:

The assumption is that the title of the window is of the form <Title> - <Application Name> or <Application Name> - <Title>.
The assumption is that the last hyphen separates the title from the application name.

How does the heuristic work:

If the application name matches completely, we ignore it in the calculation of score for a window. In the above example, “Microsoft Internet Explorer” is a complete match. Hence the scores are calculated only for Inbox(35) and Inbox(36).

Whichever window has the longest common string will have a higher score. Example: “Inbox, 45 unread” and “Inbox, 46 unread” will be a 50% match because the longest common string is 8 (Inbox, 4) out of the total length of 16. On the other hand, “Inbox, unread(45)” and “Inbox, unread(46)” will be excellent matches because the longest common string is 15 out of 17.

Special case for MDI windows:

Title changes based on whether or not the MDI window is maximized or restored. Eg: if the title of the main window is A and the title of MDI window is B, if B is maximized, the title of A changes to A – [B]. To handle these cases, we give a high score to windows whose one half matches and the other half of only one of the windows is empty.

If two or more windows get a good score, the window on the top is picked up.

Good things to do regarding titles to ensure smart match picks works correctly:

Ensure that the last hyphen separates the title from the application name and in general, avoid too many hyphens in the title.

Ensure that part of the title that changes is present either in the beginning or in the end.

Ensure that the window which you want to be picked up is present on top of the other windows.

Sometimes on a rare occasion, smart match does end up being oversmart ;-) and gives false positives, but those scenarios can be worked around. Tell us what you think of smart match!

Latest trends in software testing

anutthara — Tue, 21 Apr 2009 08:20:00 GMT

Check out this article in ITMagz.com that covers the latest trends in software testing. This is written by our very own industry expert, Tanuj Vohra that runs the Program Management team for VSTT.

Virtualization testing is something that has taken off pretty well lately - and we have been using it a fair bit internally at the empire too. With the Lab Management SKU being the latest addition to the VSTS family, we have been attempting to dogfood it in our test lab. Before the dogfood though, we have an internal team that's built a tool called Virtual Lab which can create disposable virtual machines super-fast based on similar technology. This tool has clearly upped the productivity of the team - no more fighting for repro machines, elaborate installs etc. Everyone has a host machine that can put up upto 4 VMs.

Tell me if your team uses virtualization to speed up your testing!

Your developer can never say "no repro" to your bug ever again!

anutthara — Wed, 05 Dec 2007 17:03:00 GMT

You have already used the bug tracking system (WIT) in TFS before. So, we integrated this bug tracking system with MTR so that you can file bugs with a single click from the runner itself. What's the big deal about this you might ask?

Well, the big deal is that we not just create a bug for you from the runner itself, we also populate it with the test case that you have just been executing when you opened the bug. And that too in this neat little repro steps control with a tab of its own in the bug form. (Yes, we modified the MSF template to include some extra tabs in the default bug) So, all comments that the tester has written, notes she has taken, attachments that she has made are all uploaded automatically to the bug from the first step to the last failed step.

Now, how often have you hit a bug during testing that would just repro on your test machine but simply would not repro on the dev machine. And how often has it turned out that it was a configuration related issue - different service packs, different CPU usage, different memories, the list is endless. To ensure that the developer gets a complete picture of the environment where the bug was actually produced, we fill in system information about the test machine automatically into the newly created bug. Check out the "system info" tab within the bug form:

Now, even with all of these, what if the developer still cannot repro the bug? Perhaps it was a domain expert that filed the bug and not a technical tester. He/she may not have noticed subtleties that might have affected the system and caused the bug. Or the bug repro steps or description maybe woefully inadequate for the developer to produce a reasonable repro. In such cases, the video recording that the runner makes in the background during all test execution, will be considerably useful. Playing back the video will give the developer a very detailed and accurate picture of the exact steps leading up to the bug. This video is filed as an attachment to the bug

What if the test itself took 10 long minutes. Now, the failure is in the last step perhaps. The developer may want to look at only the failed step instead of the whole long video. Video markers are the way to go for that. Each time the tester marks a test step in the test case, a video marker is inserted with that timestamp in the video file. These markers are then populated against the test steps in the repro steps of a bug filed via the runner. When I click on these markers, I can play back from the point where the previous step was marked. This gives the developer finer control over what portions of the video file may be useful.

So, how do you like our actionable bug? Are there any other ways we could help developers get the maximum effective amount of information from the bugs?

Rosario November CTP - did you play it back yet?

anutthara — Sat, 01 Dec 2007 09:11:01 GMT

Right after we RTM-ed Orcas last Monday, we are out with the November CTP of Rosario VSTS. Go on over here and give it a spin. We have a ton of stuff that's new with VSTS in there - more on BHarry's blog.

But, of course, I am excited about the manual test and AFN portions that we built out of my team here. Check out the awesome background recording that we do while you are out executing your manual tests in MTR. Next time you want to run the same test, just "play back" the recording of your manual tests. We support web apps for now, winforms is on the way...

We've also re-done the test case manager from the Aug CTP - there is now this sleek and stylish stand-alone test case manager where you manage all test activity from. We figure Visual Studio may not be the best environment for testers to go do all of this work from - so we have it completely stand alone now. Our team blog has a nice walkthrough of the TCM features - do skim through the post.

So, what are you waiting for? Go try it already and tell us what you think...

PS: Oh, given the huge size of the VPC, you might just want to install it in convenient parts as provided on the download page.

Odds and ends

anutthara — Mon, 26 Nov 2007 11:16:00 GMT

So, we've been heads down building the latest Rosario CTP of VSTT for you guys. Some good new stuff heading your way... more details when we release it ;-)

Meanwhile, like you must be knowing already, Orcas is on it's way to RTM. VSTT has some great stuff lined up on the load testing side - do check it out. And as with all releases, we are waiting nervously for this one to get out of the door safe and sound. Onto Rosario then!

Among other news, we had an EE day here at IDC, where we get to hear about cool new tech/processes emerging in other parts of the company. MSFT, being as big as it is currently, we certainly need a forum where we can share and learn new stuff across teams. The keynote was very Sinofsky - enjoyable and interesting as always. Peter Spiro did a fiery session as well on day 2 - where he spoke about his journey inside and outside of Microsoft. If I were given a 100 rupees for each time he swore, I'd be rich! LoL! Hard hitting and candid as usual - this guy inspires awe from even the most paranoid of people.

Alan Page gave a talk on testability, that was pretty neat. He also did this course for senior testers; that I'd highly recommend... lots of useful nuggets stringed well and sprinkled with some amusing and interesting anecdotes from him! And it was good seeing him in person - now I have a face to match the blog/mail/IM :)

Oh - and there will be a couple of guest posts from folks at my team on Rosario bits. They wanna test drive blogging and check if it's really their cup of tea...so my blog will be their guinea pig for now :)

What's new in Rosario for VSTT: Manual Test Runner

anutthara — Fri, 10 Aug 2007 05:10:08 GMT

Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester and make her life much easier than it currently is. The primary intent of the MTR is to help the manual tester create an actionable bug with which the developer is much more empowered to debug issues without the bug ping-pong happening between the dev and tester. Like I have mentioned before in my blog, it is pretty tedious for a tester today to even capture screenshots while filing bugs. MTR gives the tester one-click screenshot capture which operates in 3 different convenient modes and attaches itself to the test result as well. There is also a one-click bug filing option in which the bug is pre-populated with test step details from the test case that has already been loaded up in the MTR.

In this blog entry, I will describe the basic end to end scenario using the MTR to execute test cases and publish test results. The last entry, we saw how to create test cases and test runs in the TCM client. To launch MTR, open the VS command prompt, and type msrun. This launches the runner window on your desktop. Here, point to File->Open. You will be prompted with a dialog to connect to a TCM server similar to the "Connect to server" dialog that pops up when you open the team explorer window to connect to a TFS server. Choose the TCM server (in case of the VPC, the server is on the same machine) and choose the team project on the server.

Now, you will see the "Open Test Run" dialog, where you can pick any test run that you want. Let's choose a test run and say OK.

The runner window is populated with the test cases that are the result of the query associated with the run and selected in the dialog. Now, you can open your application under test (aut) and go about executing the tests that you loaded in the run. You can mark each test step with results of pass/fail or inconclusive through multiple ways including a handy single click.

You can enter comments in the comments text box or add attachments using the "Attach" option in the menu. Once the test steps are marked, click on "End test case" to compute the result of the test case from the test step results. You could also mark test cases globally without getting into the test steps at all. And if dissatisfied, you can just right click on the test case and say "Reset test case" to erase all results marked till now and start afresh. Once the test case is complete, you can publish the test case result to the server through the "Publish" button on the toolbar. These results are then published to the server and the test case grayed out to prevent further marking of steps.

In future entries, we will look at screenshots and bug filing in more detail. And of course, our glitzy video recording functionality will feature in its own entry as well. :)

What's new in Rosario for VSTT: Test case management

anutthara — Tue, 07 Aug 2007 05:33:00 GMT

Rosario has test case management in a full fledged fashion as part of the VSTST feature set. So, now test cases are first class citizens in the VSTS world with them having their own exclusive test case manager tool window inside the IDE. Test cases are a separate work item type with pre-defined fields like Owner, Priority, Test steps etc.

Click on the Test->Windows->Test Case Manager option in the IDE to launch the test case manager window shown at the right most side of the IDE below. The test case manager shows the selected team project TestTeamProject with its set of test cases, test run definitions, test runs. You can create a new test case from the TCM window. A new test case pops up with the test case form.

The test case form has a "Steps" tab as shown below:

You can enter descriptive steps for the test case detailing all the actions that need to be performed as part of the test case. You can mark a test step as "Validate step" explicitly using the icon shown in the pic (highlighted using a red dot next to the icon). The test steps shown are also parameterized using data driving. More on that in later posts. Now, you can define a test case query akin to the work item query. All those defined queries are shown in the test cases node in the TCM tool window as shown above.

With the test cases created, you can also create a run definition with the test case query specified as shown in screen shot below. This run definition is used to create a logical cluster of test cases - say tests for a certain component/feature etc.

Now, you can instantiate a new test run using the test run definition created above. The test run can be associated with a build so that the test run results can be published against the build.

Once the test run is created, you can open the test run from the TCM by querying all test runs. The current state of another sample test run in a graph is as shown below:

Now that we have a test case, run definition and test run created, we will delve into using MTR to execute our test cases in further blog posts.

Rosario is here!

anutthara — Mon, 06 Aug 2007 10:05:00 GMT

If you still(!!!) haven't heard, Rosario CTP is out in the form of a VPC here :) Doug's whitepaper should give you enough info to take it out for a spin.

Manual Test Runner(MTR) is a key part of the CTP. It is a tool designed to help manual testers to load up their test plans and execute those with the support of additional tools like one-click screenshot, one-click bug with pre-filled repro and more such exciting stuff. Test case management(TCM) is another key piece where we have tried to fill the missing piece in the existing VSTT SKU. There is a tool window right inside VS IDE where you can create/modify/manage test cases, test run definitions and test runs and save them in the TCM server which has your team project's source code and work items. Then, load up the test runs in MTR and go. Publish the test results back into the TCM server and look at the cool reports that show how your test team is doing.

Do check it out and tell me how it goes!

We signed off on the first CTP release for Rosario

anutthara — Thu, 26 Jul 2007 12:43:14 GMT

Rosario is the next in line version for Team System after Orcas. And we are doing this CTP when Orcas has not even RTM-ed yet! Isn't that pretty cool? It's been a bit crazy managing multiple releases, but then, we want Rosario to hit the shelves in a short time after Orcas releases - faster new release. And no - I am not allowed to define "short time" :P

I've been working on the VSTT portions of this CTP for some time now and this is certainly one of the most exciting set of features I have seen for testers. Though I cannot wait to talk about all the awesome new stuff in the release, I'll not give out any more info until the CTP is out. Stay tuned for more...

We'll put it out as a VPC download on our website so that there are no installs that you need to do separately. Just set up the VPC and go...

But will the dog eat the dog food?

anutthara — Wed, 11 Jul 2007 08:40:59 GMT

Since I am a tester on the Test edition of VSTS - I get this q a lot of times: "Do you use VSTST for testing VSTST?" And the answer is "You bet, I do!" :)

As you might know already, this is called "Dogfooding". Many Microsoft products are dogfooded extensively before release - Vista, MSN Search, Visual Studio, Team System...

Our devs use unit test types to write their developer regression tests. Our QA team uses unit tests to write our functional tests. Yes, I know this is not a perfect fit for UI heavy testing, but it does suffice for the time being. We use generic tests to wrap our legacy tests that may be scripts or batch files. Load tests are added to the test bed to do stress testing on the app. All of this goes into the TFS server that we have deployed for the entire Developer Division and testing performed out of it.

Like any other user, I have my own set of pet features and pet peeves in TST. I absolutely love the code coverage coloring, the "Run test" from the editor (Orcas feature), the ease with which load tests can be configured and the controller-agent functionality. It makes my life so much easier as a tester. Not to mention that it's incredibly easy to point out to devs where the problems are and associate results with bugs since we use TFS for work item tracking. Now for the bad parts - I think the test list feature is absolutely useless for large test projects. The UI confuses me - more than once, I have selected a test in the window, right clicked and said "Run checked tests", but some other tests start running!! That's because selecting a run is not the same as checking it. :( I have a wishlist as well - stuff like test case management being easier to handle, the test run results being easier to analyze etc.

The best part of dogfooding is that this is a good chance to know which features are likely to be real relief and which ones are mostly going to tick the users off. Testers being the first customers of the tool is literally true in this case. Plus the QA team gets additional testing in the process so that bugs/suggestions are filed before outside users get a chance to go at it.

So, do you eat your own dogfood? ;-)

Team System how-to videos up on asp.net

anutthara — Tue, 03 Jul 2007 07:53:39 GMT

There are some really sweet videos over on the asp.net site to help kick start users of VSTS. The videos are typically 8-10 minutes long and to-the-point. Very useful for those that are looking for quick starters or help with a specific scenario. Personally, I love these kind of videos as compared to the lengthy hour long tutorials and talks. Short and fast - gets me started right away.

Eric Lee gives a short intro to unit testing with VSTS as well as using code coverage inside of VS. There are some good videos on load testing and web application perf profiling by Chris Menegay as well. Check out the videos and let us know if you want any specific videos up!

TechMela, ahoy!

anutthara — Wed, 13 Jun 2007 08:56:00 GMT

TechMela is the "biggest technical event of the year" in India and will replace the Tech Ed events that Microsoft traditionally had in many cities in India. I'll be part of a VSTT talk on 16th June, specifically talking more about how we use our own Team System features internally. Brian Harry does a fantastic job of bringing transparency into this with his mind boggling set of stats that he uploads monthly on his blog. I'll extend into how we dogfood TST as well - so testers outside of MSFT can get a peek at how we use our own products for testing. Abhishek Mathur, our PM will present the new Orcas features of VSTST as well.

So, do drop by to say hi or rave/rant about VSTS if you are planning on attending. (Yeah - I know, even though my blog maybe read by a grand 2 and a half readers, I still wrote this to humour myself :)

How do you add same test to multiple test lists?

anutthara — Mon, 06 Nov 2006 18:26:00 GMT

Ok - I have to log this one...I have heard this q from at least half a dozen folks. In the VSTT Test Manager, how do you add tests to multiple test lists? For those of you that have used pre RTM builds, you must have seen the drag and drop feature where I could drop tests into multiple lists by just dragging the test into those lists as many times. But, now you can't do that - the drag and drop functions like a Move versus the earlier Copy functionality. Once I move a test into a test list, the next time I drag and drop the test, it shifts to the new test list instead of replicating itself.

The trick is to hit Ctrl while dragging the test. This overloads the drag and drop action into an action where the test is now part of multiple test lists instead of getting moved about. Personally, I don't like this model too much - it's sorta cryptic to first time users! But then, I can't think of a great alternative for a move action either. So, I guess, it is the Ctrl drag that we will use to cop into multiple lists, for now at least.