Musings of a tester testing test tools! : Team System

WPF controls not getting identified in Beta2 CUIT?

anutthara — Wed, 02 Dec 2009 13:43:48 GMT

Are you having trouble identifying a few of your WPF controls in Coded UI Test after upgrading to Beta2 or getting a warning during recording? You need to install the automation api 3.0 on the client to get CUIT to identify the virtualized controls in WPF.

Note that this update gets automatically downloaded if you are running Vista as part of the Vista platform update. In case of XP or W2K3, you can install the standalone version from here

Designing the Coded UI Test Builder

anutthara — Tue, 27 Oct 2009 08:48:00 GMT

The Coded UI test interface has undergone a redesign in Beta2 to make it easier and more intuitive. While designing this UI, we went through a common dilemma – do we need this to be a guided flow or a choice of operations, all available in parallel? So, is this a wizard or a toolbar?

The basic flows are:

Record method > Generate code > Repeat > Add control > Add assertion > Generate code
(Generate code from MTR strip before) > Add control > Add assertion > Repeat > Generate code
Add control > Repeat > Generate code : for folks that will write code on the controls added instead of recording

Additional flows focusing on incremental ops were many. Primary among those were:

Edit the UI elements to rename or delete them
Add additional assertions to pre-existing controls

A wizard would give ample guidance on when to hit the red button, when to drop the cross hair etc. creating a nice little workflow. But it would also constrain the user to one or two of the new flows above. Incremental flows were becoming tedious with the wizard. Plus the wizard was taking up significant space on the desktop contesting with the app under test.

A stack bar would allow the user to choose from the options available to create the shortest desirable workflow. But it was also confusing to first time users on how to use the builder. We have tried to address that through tooltips and detailed walkthroughs, that should hopefully help.

So, the stack bar now appears at the bottom of the screen with a large tooltip that draws the user’s attention to itself:

Users can choose to record actions through the red button or choose to drag and drop the cross hair icon to capture controls. The last button is for the eventual action of generating code.

On dropping the cross-hair on a control of your choice, the UI Control Locator comes up listing the properties of the control along with the navigate bar and a button that lets you add assertion on the chosen property.

Once you choose a property, you can hit the add assertion button to see the dialog where you can choose the comparator and the value to compare with. the generate code dialog comes up:

The UI map is available for viewing in the expand bar to the left of the props window. It’s a bit like the collapsible calendar in Outlook. Click on the expand button and you can see the list of controls available in the UI Map – this consists of all the UI elements you have acted upon in either recording or adding assertions.

Once you are done with recording actions and adding assertions, hit generate code and that will complete the operation by emitting code into VS.

Testing Silverlight applications with VSTS 2010 – coded UI, MTLM and web tests

anutthara — Thu, 27 Aug 2009 13:05:00 GMT

Lately, we’ve been asked by some of our users if they can use VSTS 2010 test features to test their Silverlight apps. Well, web tests will still work on Silverlight web apps since they don’t operate at a control level, but functional tests that require to identify the control type and act on each control does not yet have support for SL. Therefore, we still don’t have support for testing Silverlight controls via the Coded UI Test and MTLM in VSTS 2010. But the good news is that we are planning to plug this gap after the VSTS 2010 release to enable our users to test Silverlight apps with coded UI test.

Another alternate is users can employ the extensibility support in the record and play engine in VSTS to write a plugin to enable support for Silverlight and use it in MTLM or Coded UI Test earlier. You can write up your plugin, compile and drop it onto your test machine – and you are ready to go. More on RnP extensibility in the coming posts.

Latest trends in software testing

anutthara — Tue, 21 Apr 2009 11:20:00 GMT

Check out this article in ITMagz.com that covers the latest trends in software testing. This is written by our very own industry expert, Tanuj Vohra that runs the Program Management team for VSTT.

Virtualization testing is something that has taken off pretty well lately - and we have been using it a fair bit internally at the empire too. With the Lab Management SKU being the latest addition to the VSTS family, we have been attempting to dogfood it in our test lab. Before the dogfood though, we have an internal team that's built a tool called Virtual Lab which can create disposable virtual machines super-fast based on similar technology. This tool has clearly upped the productivity of the team - no more fighting for repro machines, elaborate installs etc. Everyone has a host machine that can put up upto 4 VMs.

Tell me if your team uses virtualization to speed up your testing!

Your developer can never say "no repro" to your bug ever again!

anutthara — Wed, 05 Dec 2007 20:03:00 GMT

You have already used the bug tracking system (WIT) in TFS before. So, we integrated this bug tracking system with MTR so that you can file bugs with a single click from the runner itself. What's the big deal about this you might ask?

Well, the big deal is that we not just create a bug for you from the runner itself, we also populate it with the test case that you have just been executing when you opened the bug. And that too in this neat little repro steps control with a tab of its own in the bug form. (Yes, we modified the MSF template to include some extra tabs in the default bug) So, all comments that the tester has written, notes she has taken, attachments that she has made are all uploaded automatically to the bug from the first step to the last failed step.

Now, how often have you hit a bug during testing that would just repro on your test machine but simply would not repro on the dev machine. And how often has it turned out that it was a configuration related issue - different service packs, different CPU usage, different memories, the list is endless. To ensure that the developer gets a complete picture of the environment where the bug was actually produced, we fill in system information about the test machine automatically into the newly created bug. Check out the "system info" tab within the bug form:

Now, even with all of these, what if the developer still cannot repro the bug? Perhaps it was a domain expert that filed the bug and not a technical tester. He/she may not have noticed subtleties that might have affected the system and caused the bug. Or the bug repro steps or description maybe woefully inadequate for the developer to produce a reasonable repro. In such cases, the video recording that the runner makes in the background during all test execution, will be considerably useful. Playing back the video will give the developer a very detailed and accurate picture of the exact steps leading up to the bug. This video is filed as an attachment to the bug

What if the test itself took 10 long minutes. Now, the failure is in the last step perhaps. The developer may want to look at only the failed step instead of the whole long video. Video markers are the way to go for that. Each time the tester marks a test step in the test case, a video marker is inserted with that timestamp in the video file. These markers are then populated against the test steps in the repro steps of a bug filed via the runner. When I click on these markers, I can play back from the point where the previous step was marked. This gives the developer finer control over what portions of the video file may be useful.

So, how do you like our actionable bug? Are there any other ways we could help developers get the maximum effective amount of information from the bugs?

Odds and ends

anutthara — Mon, 26 Nov 2007 14:16:00 GMT

So, we've been heads down building the latest Rosario CTP of VSTT for you guys. Some good new stuff heading your way... more details when we release it ;-)

Meanwhile, like you must be knowing already, Orcas is on it's way to RTM. VSTT has some great stuff lined up on the load testing side - do check it out. And as with all releases, we are waiting nervously for this one to get out of the door safe and sound. Onto Rosario then!

Among other news, we had an EE day here at IDC, where we get to hear about cool new tech/processes emerging in other parts of the company. MSFT, being as big as it is currently, we certainly need a forum where we can share and learn new stuff across teams. The keynote was very Sinofsky - enjoyable and interesting as always. Peter Spiro did a fiery session as well on day 2 - where he spoke about his journey inside and outside of Microsoft. If I were given a 100 rupees for each time he swore, I'd be rich! LoL! Hard hitting and candid as usual - this guy inspires awe from even the most paranoid of people.

Alan Page gave a talk on testability, that was pretty neat. He also did this course for senior testers; that I'd highly recommend... lots of useful nuggets stringed well and sprinkled with some amusing and interesting anecdotes from him! And it was good seeing him in person - now I have a face to match the blog/mail/IM :)

Oh - and there will be a couple of guest posts from folks at my team on Rosario bits. They wanna test drive blogging and check if it's really their cup of tea...so my blog will be their guinea pig for now :)

What's new in Rosario for VSTT: Manual Test Runner

anutthara — Fri, 10 Aug 2007 08:10:08 GMT

Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester and make her life much easier than it currently is. The primary intent of the MTR is to help the manual tester create an actionable bug with which the developer is much more empowered to debug issues without the bug ping-pong happening between the dev and tester. Like I have mentioned before in my blog, it is pretty tedious for a tester today to even capture screenshots while filing bugs. MTR gives the tester one-click screenshot capture which operates in 3 different convenient modes and attaches itself to the test result as well. There is also a one-click bug filing option in which the bug is pre-populated with test step details from the test case that has already been loaded up in the MTR.

In this blog entry, I will describe the basic end to end scenario using the MTR to execute test cases and publish test results. The last entry, we saw how to create test cases and test runs in the TCM client. To launch MTR, open the VS command prompt, and type msrun. This launches the runner window on your desktop. Here, point to File->Open. You will be prompted with a dialog to connect to a TCM server similar to the "Connect to server" dialog that pops up when you open the team explorer window to connect to a TFS server. Choose the TCM server (in case of the VPC, the server is on the same machine) and choose the team project on the server.

Now, you will see the "Open Test Run" dialog, where you can pick any test run that you want. Let's choose a test run and say OK.

The runner window is populated with the test cases that are the result of the query associated with the run and selected in the dialog. Now, you can open your application under test (aut) and go about executing the tests that you loaded in the run. You can mark each test step with results of pass/fail or inconclusive through multiple ways including a handy single click.

You can enter comments in the comments text box or add attachments using the "Attach" option in the menu. Once the test steps are marked, click on "End test case" to compute the result of the test case from the test step results. You could also mark test cases globally without getting into the test steps at all. And if dissatisfied, you can just right click on the test case and say "Reset test case" to erase all results marked till now and start afresh. Once the test case is complete, you can publish the test case result to the server through the "Publish" button on the toolbar. These results are then published to the server and the test case grayed out to prevent further marking of steps.

In future entries, we will look at screenshots and bug filing in more detail. And of course, our glitzy video recording functionality will feature in its own entry as well. :)

What's new in Rosario for VSTT: Test case management

anutthara — Tue, 07 Aug 2007 08:33:00 GMT

Rosario has test case management in a full fledged fashion as part of the VSTST feature set. So, now test cases are first class citizens in the VSTS world with them having their own exclusive test case manager tool window inside the IDE. Test cases are a separate work item type with pre-defined fields like Owner, Priority, Test steps etc.

Click on the Test->Windows->Test Case Manager option in the IDE to launch the test case manager window shown at the right most side of the IDE below. The test case manager shows the selected team project TestTeamProject with its set of test cases, test run definitions, test runs. You can create a new test case from the TCM window. A new test case pops up with the test case form.

The test case form has a "Steps" tab as shown below:

You can enter descriptive steps for the test case detailing all the actions that need to be performed as part of the test case. You can mark a test step as "Validate step" explicitly using the icon shown in the pic (highlighted using a red dot next to the icon). The test steps shown are also parameterized using data driving. More on that in later posts. Now, you can define a test case query akin to the work item query. All those defined queries are shown in the test cases node in the TCM tool window as shown above.

With the test cases created, you can also create a run definition with the test case query specified as shown in screen shot below. This run definition is used to create a logical cluster of test cases - say tests for a certain component/feature etc.

Now, you can instantiate a new test run using the test run definition created above. The test run can be associated with a build so that the test run results can be published against the build.

Once the test run is created, you can open the test run from the TCM by querying all test runs. The current state of another sample test run in a graph is as shown below:

Now that we have a test case, run definition and test run created, we will delve into using MTR to execute our test cases in further blog posts.

Rosario is here!

anutthara — Mon, 06 Aug 2007 13:05:00 GMT

If you still(!!!) haven't heard, Rosario CTP is out in the form of a VPC here :) Doug's whitepaper should give you enough info to take it out for a spin.

Manual Test Runner(MTR) is a key part of the CTP. It is a tool designed to help manual testers to load up their test plans and execute those with the support of additional tools like one-click screenshot, one-click bug with pre-filled repro and more such exciting stuff. Test case management(TCM) is another key piece where we have tried to fill the missing piece in the existing VSTT SKU. There is a tool window right inside VS IDE where you can create/modify/manage test cases, test run definitions and test runs and save them in the TCM server which has your team project's source code and work items. Then, load up the test runs in MTR and go. Publish the test results back into the TCM server and look at the cool reports that show how your test team is doing.

Do check it out and tell me how it goes!

We signed off on the first CTP release for Rosario

anutthara — Thu, 26 Jul 2007 15:43:14 GMT

Rosario is the next in line version for Team System after Orcas. And we are doing this CTP when Orcas has not even RTM-ed yet! Isn't that pretty cool? It's been a bit crazy managing multiple releases, but then, we want Rosario to hit the shelves in a short time after Orcas releases - faster new release. And no - I am not allowed to define "short time" :P

I've been working on the VSTT portions of this CTP for some time now and this is certainly one of the most exciting set of features I have seen for testers. Though I cannot wait to talk about all the awesome new stuff in the release, I'll not give out any more info until the CTP is out. Stay tuned for more...

We'll put it out as a VPC download on our website so that there are no installs that you need to do separately. Just set up the VPC and go...

But will the dog eat the dog food?

anutthara — Wed, 11 Jul 2007 11:40:59 GMT

Since I am a tester on the Test edition of VSTS - I get this q a lot of times: "Do you use VSTST for testing VSTST?" And the answer is "You bet, I do!" :)

As you might know already, this is called "Dogfooding". Many Microsoft products are dogfooded extensively before release - Vista, MSN Search, Visual Studio, Team System...

Our devs use unit test types to write their developer regression tests. Our QA team uses unit tests to write our functional tests. Yes, I know this is not a perfect fit for UI heavy testing, but it does suffice for the time being. We use generic tests to wrap our legacy tests that may be scripts or batch files. Load tests are added to the test bed to do stress testing on the app. All of this goes into the TFS server that we have deployed for the entire Developer Division and testing performed out of it.

Like any other user, I have my own set of pet features and pet peeves in TST. I absolutely love the code coverage coloring, the "Run test" from the editor (Orcas feature), the ease with which load tests can be configured and the controller-agent functionality. It makes my life so much easier as a tester. Not to mention that it's incredibly easy to point out to devs where the problems are and associate results with bugs since we use TFS for work item tracking. Now for the bad parts - I think the test list feature is absolutely useless for large test projects. The UI confuses me - more than once, I have selected a test in the window, right clicked and said "Run checked tests", but some other tests start running!! That's because selecting a run is not the same as checking it. :( I have a wishlist as well - stuff like test case management being easier to handle, the test run results being easier to analyze etc.

The best part of dogfooding is that this is a good chance to know which features are likely to be real relief and which ones are mostly going to tick the users off. Testers being the first customers of the tool is literally true in this case. Plus the QA team gets additional testing in the process so that bugs/suggestions are filed before outside users get a chance to go at it.

So, do you eat your own dogfood? ;-)

Team System how-to videos up on asp.net

anutthara — Tue, 03 Jul 2007 10:53:39 GMT

There are some really sweet videos over on the asp.net site to help kick start users of VSTS. The videos are typically 8-10 minutes long and to-the-point. Very useful for those that are looking for quick starters or help with a specific scenario. Personally, I love these kind of videos as compared to the lengthy hour long tutorials and talks. Short and fast - gets me started right away.

Eric Lee gives a short intro to unit testing with VSTS as well as using code coverage inside of VS. There are some good videos on load testing and web application perf profiling by Chris Menegay as well. Check out the videos and let us know if you want any specific videos up!

TechMela, ahoy!

anutthara — Wed, 13 Jun 2007 11:56:00 GMT

TechMela is the "biggest technical event of the year" in India and will replace the Tech Ed events that Microsoft traditionally had in many cities in India. I'll be part of a VSTT talk on 16th June, specifically talking more about how we use our own Team System features internally. Brian Harry does a fantastic job of bringing transparency into this with his mind boggling set of stats that he uploads monthly on his blog. I'll extend into how we dogfood TST as well - so testers outside of MSFT can get a peek at how we use our own products for testing. Abhishek Mathur, our PM will present the new Orcas features of VSTST as well.

So, do drop by to say hi or rave/rant about VSTS if you are planning on attending. (Yeah - I know, even though my blog maybe read by a grand 2 and a half readers, I still wrote this to humour myself :)

Using Team Build with VSS or your own source control

anutthara — Tue, 11 Apr 2006 18:06:00 GMT

So - you saw Team Build and liked it's cool features that show which check ins went into the build, what work items were resolved in the checkins that went in, running unit tests as part of the build process and displaying results in that nifty little table on the build report. But, what if you have your own source control system? You may not want to use the VSTS source control system right away since you have a huge repository on VSS or maybe some other source control system. What then?

Team Build builds a sequence of MSBuild targets bunched together in a file called TFSBuild.proj on the build machine. One of these targets has the "Get" task that syncs source code from your source control system. Now, the default task that we ship has this task getting code from the VSTS Source Control server. So, now if you write a custom task that gets code from your own source control, you should be done. But, wait - what about the integration aspects? Which checkins went in and which work items were resolved/associated are going to be churned out in further tasks like GetChangesetsWorkItemsList that integrate with VSTS Source Control system. Alright - say you deleted those tasks from the TFSBuild.proj file. Then what? One more problem - apart from these tasks, Team Build also syncs certain config files (the very TFSBuild.proj that you are building on the build machine plus some other files) from VSTS Source Control itself. This is inbuilt in the product though. Which means you have to have these on the VSTS Source Control in a hierarchy that Team Build expects. So, theoretically, even if you replaced all your TFSBuild.proj tasks that interact with VSTS Source Control with custom tasks that interact with your source control instead, you would still have to have the VSTS Source Control in place to store your configuration files created each time you create a build type.

Well, so the short answer to the q then would be that we don't really recommend users to use Team Build to integrate with your own custom source control system. However, we do want to know how important this scenario is to you. On one hand, we do understand that migrating to a new system could be painful and you might want to do this one step at a time and source control would probably be the last item due to it's size and indispensability. So, for folks looking to do incremental adoption of VSTS, you can't move to Team Build first and VSTS Source Control, next really. But, on the other hand, Team Build has this great set of integration stufflike reporting changesets, work items associated etc. that we believe are really some of the most compelling set of features in the product. Using your own custom source control would not really let you benefit out of those scenarios. And, if your intention is to do incremental adoption, one option would be to use VSSConverter, a tool that migrates source code from VSS to VSTS Source Control. When you run the converter on your existent code base and migrate your code to VSTS Source Control, you could launch Team Build on that appropriately. Scheduling incremental migration every night on your original source code base will port daily changes to the new Source Control and Team Build will build these new bits too.

What is your take on this? Are we being naive in assuming whatever we have? Or is this the best possible option to work with? Do share your thoughts...

I can't figure out it's a configuration based property!! Can you?

anutthara — Fri, 24 Feb 2006 17:38:00 GMT

Many times, I have been fooled as Andrew has been. In fact, the same error has been reported more than twice from different channels where users had no idea where they were wrong.

The properties in the Code Analysis tab of the project properties is as shown. As a new user, I won't pay much attention to the two dropdowns on top which are anyway set to Active. But when I happily choose my rules and save the project, I build it to see that the rules are being applied fine. Then I proceed to make a build type to build the solution. But, the code analysis rules are no longer honored!! Why would that be? That is because the default configurations are Release, Any CPU for build type creation while default configurations are Debug, Any CPU for setting code analysis rules. Hmm... so whose fault is it? Project properties or Team Build or worse, the user ;-) ?

Team Build chooses a default value of Release based on a very logical assumption that when daily builds are created, people usually want usable, crisp and testable builds without loads of debug info, which means the release builds are the logical defaults. Project properties will have an equally logical choice of default value, Debug where projects are usually built in this mode in VS. But the hapless user also is not in a happy position when he needs to set code analysis rules after paying careful attention to the 2 drop downs and remembering the effect of those long after. Having a check box per configuration with all configs checked cannot be feasible since custom configurations are going to explode the combinations possible and make no space available for that many chcek boxes. Choosing All Platforms and All Flavors as a default option may not make much sense in a Code Analysis setting itself and obviously the Build settings would be worse. Well...

I can't think of any creative solution to this. Perhaps you can?