Musings of a tester testing test tools!

Tool to import test cases from Excel to MTLM (VS 2010 Beta2) now available on codeplex

2009-11-06T12:48:22Z

Importing test cases from Excel to VS2010 is a no longer a pain. Check out Test Case Migrator (Excel) on codeplex - an automated tool that gives end to end support to move test cases from Excel to the new TCM capabilities with MTLM in VS 2010!

It has a nifty wizard that lets you map columns in Excel with fields in your work item type and also lets you split out your test steps with different delimiters or columns. Plus this works across both Agile and CMMI methodologies out of the box. Note this is a one time import only and won’t let you continue to keep your test cases in Excel in sync with TCM though.

So, go on – give it a shot and tell us what you think!

Designing the Coded UI Test Builder

2009-10-27T08:48:00Z

The Coded UI test interface has undergone a redesign in Beta2 to make it easier and more intuitive. While designing this UI, we went through a common dilemma – do we need this to be a guided flow or a choice of operations, all available in parallel? So, is this a wizard or a toolbar?

The basic flows are:

Record method > Generate code > Repeat > Add control > Add assertion > Generate code
(Generate code from MTR strip before) > Add control > Add assertion > Repeat > Generate code
Add control > Repeat > Generate code : for folks that will write code on the controls added instead of recording

Additional flows focusing on incremental ops were many. Primary among those were:

Edit the UI elements to rename or delete them
Add additional assertions to pre-existing controls

A wizard would give ample guidance on when to hit the red button, when to drop the cross hair etc. creating a nice little workflow. But it would also constrain the user to one or two of the new flows above. Incremental flows were becoming tedious with the wizard. Plus the wizard was taking up significant space on the desktop contesting with the app under test.

A stack bar would allow the user to choose from the options available to create the shortest desirable workflow. But it was also confusing to first time users on how to use the builder. We have tried to address that through tooltips and detailed walkthroughs, that should hopefully help.

So, the stack bar now appears at the bottom of the screen with a large tooltip that draws the user’s attention to itself:

Users can choose to record actions through the red button or choose to drag and drop the cross hair icon to capture controls. The last button is for the eventual action of generating code.

On dropping the cross-hair on a control of your choice, the UI Control Locator comes up listing the properties of the control along with the navigate bar and a button that lets you add assertion on the chosen property.

Once you choose a property, you can hit the add assertion button to see the dialog where you can choose the comparator and the value to compare with. the generate code dialog comes up:

The UI map is available for viewing in the expand bar to the left of the props window. It’s a bit like the collapsible calendar in Outlook. Click on the expand button and you can see the list of controls available in the UI Map – this consists of all the UI elements you have acted upon in either recording or adding assertions.

Once you are done with recording actions and adding assertions, hit generate code and that will complete the operation by emitting code into VS.

What’s new for testers doing UI automation in VS 2010 Beta2?

2009-10-26T05:31:06Z

We are back in Beta2 with a bunch of new stuff in Coded UI test for UI automation testers. First up, the code generation in Coded UI test has been changed to structure the code in a much more usable and intuitive way:

1. All the UI elements you interact with, whether part of recording or added from the UI control locator, all go into a single UI Map – a single collection of all the UI objects in your test. So, you can now edit objects from a single store

2. Every assertion you add is now a method in itself which takes parameters where you can define the expected value to assert on. So, you can now separate the test into logical actions and assertions

3. The recorded methods also have input parameters defined on them that help make data driving easier to do. So, you can now data drive the method by changing the values of the input parameters that go into the recorded method.

For example, if you were testing the new user sign up in hotmail.com and checking if a certain id’s availability is displayed correctly, you can record steps to open hotmail.com, go to sign up page, enter a particular id and press “check availability” button.

The coded UI test project generated has 4 files – the coded UI test .cs file (TestHotmail.cs) containing the test method, UIMap.uitest file which is an XML representation of all the UI elements and actions/assertions performed on each UI object, the UImap.designer.cs file which is the code-behind for the .uitest file containing code for the objects and actions, the UIMap.cs is a file containing a partial definition of the UImap class to help preserve user customizations to the test code.

TestHotmail.cs – contains the actual test method and invocations to recorded methods and assertions

[CodedUITest]

public class TestHotmail

{ [TestMethod]public void TestNewUserSignUp(){

this.UIMap.GotoSignUpPage();

this.UIMap.CheckAvailabilityOfId();

this.UIMap.VerifyIdAvailable();

}}

UIMap.designer.cs – contains the definitions of the recorded methods, assertions and UI objects. Plus also contains the parameter class definitions passed into each class

public partial class UIMap {

public void GotoSignUpPage()

{ HtmlInputButton signupButton = this.SignInWindowsInterneWindow.HttploginlivecomlogiClient.SignInDocument.SignupButton;

// Click 'Sign up' button

Mouse.Click(signupButton, new Point(47, 11)); }

public void CheckAvailabilityOfId()

{ HtmlEdit imembernameliveEdit = this.SignupWindowsLiveWinWindow.HttpssignuplivecomsiClient.SignupWindowsLiveDocument.ImembernameliveEdit;

// Type 'anutthar' in 'imembernamelive' text box

imembernameliveEdit.Text = this.CheckAvailabilityOfIdParams.ImembernameliveEditText;

// Click 'Check availability' button

Mouse.Click(checkavailabilityButton, new Point(88, 15)); }}

// Input parameter into the CheckAvailabilityOfId recorded method

public class CheckAvailabilityOfIdParams

{ public string ImembernameliveEditText = "anutthar"; }

Other than that, the CUIT builder has gone through a major redesign and morphed into a stack bar at the bottom of your screen with pop up dialogs for the control locator, actions viewer, generate code etc. More on that in the next post

We’ve also added support for virtualized controls in WPF. You will need to wait until the UI Automation 3.0 APIs are available to get this working though. But this should certainly help WPF programmers trying to write tests on their UI apps.

Go ahead, give CUIT a spin and tell us what you think…

Where are my Team System features?

2009-10-23T11:09:54Z

With the new SKU structure in Visual Studio, there is no longer any SKU called “Team System”. So, how do you figure out which features are available in what SKU? Here is a table listing where each of the Team System features are in the new SKUs:

Does the new SKU structure make things simpler for you?

Testing Silverlight applications with VSTS 2010 – coded UI, MTLM and web tests

2009-08-27T13:05:00Z

Lately, we’ve been asked by some of our users if they can use VSTS 2010 test features to test their Silverlight apps. Well, web tests will still work on Silverlight web apps since they don’t operate at a control level, but functional tests that require to identify the control type and act on each control does not yet have support for SL. Therefore, we still don’t have support for testing Silverlight controls via the Coded UI Test and MTLM in VSTS 2010. But the good news is that we are planning to plug this gap after the VSTS 2010 release to enable our users to test Silverlight apps with coded UI test.

Another alternate is users can employ the extensibility support in the record and play engine in VSTS to write a plugin to enable support for Silverlight and use it in MTLM or Coded UI Test earlier. You can write up your plugin, compile and drop it onto your test machine – and you are ready to go. More on RnP extensibility in the coming posts.

Using regular expressions to locate controls in Coded UI Test

2009-08-19T13:14:00Z

When testing UI applications, you often want to locate a control on the app under test that has dynamic properties. For instance, when you login into a website, there may be a link/label on the first page that says “Welcome user, UserName1”. Now this username is going to change as per the user that logged in.

While writing a coded UI test to locate this label however, the search properties are set to the value that was found during recording. While playing back if you logged in as a new user, the search fails since it was looking for “UserName1” instead of your logged in “UserName2”. To address this, you can tweak the search properties of the control you are using:

UITestControl lblObj = new UITestControl();
lblObj.SearchProperties.Add("InnerText", "Welcome", PropertyExpressionOperator.Contains);

The above code uses “Contains” to ensure that it is searching for a string containing the substring “Welcome”. If you need a more accurate search, you might want to use regular expressions for matching the title. Unfortunately, we don’t have built in support to match the a search property against a given regular expression. However, you can write code to find the set of labels that are under a given hierarchy and match each of their inner text properties against a regular expression of your choice using the standard .NET APIs

Migrating test cases from Excel to VSTS 2010

2009-08-19T09:04:00Z

Lately, there has been a hurried slew of queries on migrating test cases to VSTS 2010 from other sources. The most popular q of course is to move test cases from Excel to VSTS 2010.

We have this facility to import work items from Excel to VSTS, right? Well, test cases are yet another kind of work items. So, you can leverage this feature to import your stuff from Excel to VSTS. You'll need to map your columns in Excel to the right test case fields though (and this can be a pain to do manually) However, test cases in 2010 have this field called "Test Steps" that are actually XML fields. The content in this is represented as an XML blob. So, when you export a test case from the 2010 server to any Excel worksheet using the regular Excel integration, you see this field containing some unformatted steps. So, if you have your test steps typically written in multiple rows in Excel, it becomes conversely hard to import those into VSTS 2010.
To solve this problem plus give users greater flexibility over what and how to import, we wrote an internal tool that lets you do mapping from Excel columns to test case fields via a small UI. It can also automatically pick up test steps from multiple rows and import them properly into VSTS without all the formatting hassle. This is still work-in-progress however and we don't have it available externally yet. I know a couple of our customers that just used the TCM Object model and whipped up a migration tool of their own. I believe we should shortly have *something* out to help folks that want to migrate - either a solution from our partners in the community or a tool on codeplex.

Once there is a tool to do this, migration from other sources should not be a problem. All you need to do is export from old TCM to Excel and then to VSTS 2010.

Our first raving fan for Team Test in VSTS 2010!

2009-07-16T11:09:00Z

Yay - I finally got to see our Dev10 Team Test bits being used in production! E&D team in in MSIT, India is going live on VSTS 2010 for their test case management. These guys are pretty avid dogfooders and have taken to the product pretty fast - creating their test cases, plans, suites...on the server and using Test and Lab Manager in their daily lives. We have a whole slew of early adopters using VSTS 2010 for production and MSIT, our internal IT division is an important dogfooder for us.

As Amit tells us all the time, one of our goals for VSTT 2010 is to create “raving fans” of our product - and it's great to see our first local raving fan for Team Test! :-)

My top 3 favorite features of Outlook in Office 2010

2009-07-15T12:00:00Z

Well – the tech preview for Office 2010 is now public – and I am bursting to tell you guys my favorite features of Outlook in Office 14. Being a Microsoftie, I practically live in Outlook…and Outlook 14 has some pretty nifty improvements that I absolutely loved! Here are my personal favorites:

Conversation view – saves a whole lot of inbox viewing space, lets me “cleanup” redundant mail in the thread. The really cool thing is that it shows a logical view of the conversation with mail not only from the current folder, but also strings together mail I sent or mail I moved to other folders.

Scheding meetings – you can now reply to a conversation with a meeting req – just right click and “reply with meeting” – neat. Plus the incoming meeting req now shows your calendar right inside the meeting req – so no extra clicking to navigate and see your calendar whether you are free, if not what meetings are going on. For instance, I can see right away that I can’t accept Nivedita’s meeting due to a conflict, nor move it out by an hour due to other meetings.

“Seeing” people – Having the photo of the sender integrated right into the mail header – very nice touch. I always underestimated how powerful that can be in a cross-site environment. Seeing photos of people right next to their email/contact list really makes me feel more as if I am really conversing with real people versus just sending messages to a faceless entity in Redmond or NC several thousand miles away. My personal favorites are Dominic’s photo with a VS tattoo on his head(!) and Ed grinning away into the camera :-)

Of course, some of these were available as power toys and other stand alone tools for Outlook 2007, but it’s really nice to see the scenario come together in 2010. There are other small but very thoughtful features like showing space remaining in your mailbox on the left corner when you start getting close to the limit, the quick contacts tab available in the bottom right to expedite IM. Being a tech preview, it has its share of hangs and crashes(aargh!) but I was impressed enough by the new features to keep it installed on my laptop :-)

[Edited: Duh - removed screenshots showing my mailbox!]

Video recording crashing your Test Manager in VSTS 2010 Beta1?

2009-06-26T11:00:00Z

Getting a video recording of your test to attach to the bug you file on your developer can be a really cool thing. And the Microsoft Test and Lab Manager tool makes it a breeze to do that.

However, if your MTLM is crashing or erroring out when you try to pick up a recording,try the following:

1. Check if WME encoder 9 is installed on the box. If not, install it from here

2. If you are running Vista, Windows 7 or Windows 2008, install the patch for the encoder from here

As Thomas Schissler, one of our most active Team System MVPs points out, the docs indicate this patch is required for Vista, but you need it for Win7 and W2K8 also

UI automation breaking due to window title changing each time?

2009-06-11T10:29:00Z

When testing UI applications, specially web apps, we frequently encounter the problem of window title changing with the state of the app or operations done on the app. For instance, when checking web mail, the title of the page could read “Inbox(35) - Microsoft Internet Explorer” where 35 really represents the number of unread email in your inbox. Now, when I open it at any other time, it might read “Inbox(36) - Microsoft Internet Explorer” based on unread mail. For GUI testers, this poses a problem since object identification works usually on the basis of window title and the title changing each time I run the test causes object identification to fail. :(

To solve this problem, the API of VSTS 2010 has a “Smart Match” feature built into it. Thej Kumar is our smart young developer that works on the record and play engine of Team Test and today, he will explain how smart match works. So, here you go:

Baseline assumptions:

The assumption is that the title of the window is of the form <Title> - <Application Name> or <Application Name> - <Title>.
The assumption is that the last hyphen separates the title from the application name.

How does the heuristic work:

If the application name matches completely, we ignore it in the calculation of score for a window. In the above example, “Microsoft Internet Explorer” is a complete match. Hence the scores are calculated only for Inbox(35) and Inbox(36).

Whichever window has the longest common string will have a higher score. Example: “Inbox, 45 unread” and “Inbox, 46 unread” will be a 50% match because the longest common string is 8 (Inbox, 4) out of the total length of 16. On the other hand, “Inbox, unread(45)” and “Inbox, unread(46)” will be excellent matches because the longest common string is 15 out of 17.

Special case for MDI windows:

Title changes based on whether or not the MDI window is maximized or restored. Eg: if the title of the main window is A and the title of MDI window is B, if B is maximized, the title of A changes to A – [B]. To handle these cases, we give a high score to windows whose one half matches and the other half of only one of the windows is empty.

If two or more windows get a good score, the window on the top is picked up.

Good things to do regarding titles to ensure smart match picks works correctly:

Ensure that the last hyphen separates the title from the application name and in general, avoid too many hyphens in the title.

Ensure that part of the title that changes is present either in the beginning or in the end.

Ensure that the window which you want to be picked up is present on top of the other windows.

Sometimes on a rare occasion, smart match does end up being oversmart ;-) and gives false positives, but those scenarios can be worked around. Tell us what you think of smart match!

Where are the 1.4 million developers of India?

2009-05-13T20:16:20Z

As I sit here @ Tech Ed 2009 in Hyderabad, that’s what I am wondering looking at the sparse audience in the first session of the day. (Yeah right, I was expecting half of the million developers to turn up here ;-) ) Sigh - given the current economic condition and the fact that Tech Ed is still a fairly new event in Hyderabad, the participation might just be expected to be not-so-high after all.

Anyway, AmitC gave a great demo of all the wonderful features we’ve built as part of VSTS 2010. It covered all our value propositions in breadth and there were several “wow” moments in the crowd when he demo-ed test impact, no more no repro and the virtualization capability added newly in VSTS 2010. Bunch of interesting qs from the audience around specific features in test.

SteveB’s keynote opened the talk – he talked a little bit about how IT is impacted by the current economy, gave a peek of the product lineup we have coming out this year from MSFT and answered a host of qs around Azure, Win7 and VSTS 2010. (Oh – our branding is now official for Test and Lab)

Neelesh gave a terrific talk on the load and web test features in Orcas. It is actually funny to see so many folks say they have VSTS 2008 Team Suite, but a large chunk of them not know any of the Team Test features. :-) Of course, with 2010, we have a complete first class test offering – so Team Test should really make a splash with our users. Looking forward to feedback on our Beta1…

Creating UI automation without writing a single line of code or script!

2009-05-06T23:33:25Z

Check out this totally kickass feature in VSTS 2010 aka Dev10 – the Coded UI Test (CUIT). This is a brand new test type in VSTT that lets users do functional UI automation. As GUI testers will already know, UI automation is one of the most tedious to write and get all the timing right. What CUIT will do for you is to automatically generate C# or VB.NET code from a series of recorded actions that you did while testing your app. Adding validations is a breeze – just point and click. Point to the object you want, pick the property you want to validate and type in the expected value – this validation gets automatically generated.

For more on CUIT, check out this cool video on Channel 9 – this shows an end to end demo of how to create and run a CUIT. Let me know what you think!

Latest trends in software testing

2009-04-21T11:20:00Z

Check out this article in ITMagz.com that covers the latest trends in software testing. This is written by our very own industry expert, Tanuj Vohra that runs the Program Management team for VSTT.

Virtualization testing is something that has taken off pretty well lately - and we have been using it a fair bit internally at the empire too. With the Lab Management SKU being the latest addition to the VSTS family, we have been attempting to dogfood it in our test lab. Before the dogfood though, we have an internal team that's built a tool called Virtual Lab which can create disposable virtual machines super-fast based on similar technology. This tool has clearly upped the productivity of the team - no more fighting for repro machines, elaborate installs etc. Everyone has a host machine that can put up upto 4 VMs.

Tell me if your team uses virtualization to speed up your testing!

My favorite engineering practice at Microsoft

2009-03-25T22:41:00Z

Here at MSFT, specifically at DevDiv, we have specific testing categories called "Exit Criteria" - basically gates that you need to pass through in order to ship a release at any milestone. These criteria represent different flavors of testing that your product has to go through in order to be pronounced "fit for release". For eg: localization, accessibility, performance. At DevDiv, we have 26 different exit criteria that we need to pass through before we ship!
Now, as is the case with any "process", this one has also been the subject of passionate debate among the MSFT denizens as to the value this really brings to our product quality. I have heard arguments around how this is a lot of overhead at a very busy time in the ship cycle. Well - I have to concede that these set of tests really do not bring in a lot of bugs. Especially, criteria like "branding and copyright" - where we just test to see if the product is carrying the right trademarks everywhere, the tool names are mentioned correctly etc. Also, when criteria like "configuration matrix" is in question - this is basically the gate where we ensure our products run ok on different operating systems, different versions of CLR, different versions of IE, different bitness...some of us testers tend to go berserk and many times, products get overtested. And of course, there is this eternal danger of manual testing (which is still the best source of the most important bugs) getting undermined and sacrificed at the altar of checking off items in a checklist.

In spite of all of this, I still opine that the exit criteria proc is one of the best engineering practices I have seen here at home. If you are a tester testing a product with a customer base numbering tens of thousands or more, I am sure you appreciate the sheer breadth of things you need to cover in your test planning. The number of dimensions is really mind boggling - security, performance, localization, globalization, backward compatibility, accessibility, stress, endurance, limit...it sure gets crazy. Exit criteria really bring a semblance of sanity in this whole universe of testing. I would term this the distilled essence of the different testing dimensions that go into testing a world class product. It serves as a map to help testers navigate the maze of testing requirements and builds confidence into the quality of the product step by step.
The Globalization EC for instance, is so very well thought out - we have different configurations like Italian, Turkish, Arabic; with each configuration defined by what is different in that config and what are the kind of bugs you can expect to hit. For instance, ITA is one of the "most localized" OSes, which help catch issues where we hardcode stuff like security group names. I remember Team Build had this bug in 2005, where we had hardcoded to check if user belongs to "Administrators"! The test passed on JPN, because the admin group was still "Administrators" on the JPN OS, but it failed on German OS, since the group was "Administrateren" on DEU! Turkish has the infamous sort problems with the quirky dotted/undotted I's. Testing on Arabic configs throw up specific problems rendering bidirectional text or right-to-left UI mirroring etc. To me, this is a treasure trove of info put together by several years of testing experience. Who wouldn't want to use this cheat sheet? :-)