Musings of a tester testing test tools!

Resources for Visual Studio 2010 Test Tools

Our team has been busy putting together content for the Test Tools in Visual Studio 2010, and here is a compilation of those resources including videos, walkthroughs, whitepapers and posts. Also listed are relevant blogs and twitter ids.

Content index for Microsoft Test Manager is:

Resources for Coded UI test are updated on my resources page

We plan to add the content index for Load Test and other test types and roll up this data onto the team blog. Since this data will be getting updated often, I have put this up as a page on my blog and refresh it periodically. I suggest readers link to that page for an up-to-date content index. 

Scaling up your CUIT UI automation for real world projects

One of the complaints that we heard often from Coded UI test users in Beta2 was that the coded UI tests could not scale up to large projects. The single UI map that contained all the UI elements and recorded methods would quickly become unwieldy and confusing to use. To address that problem, we introduced the concept of multiple UI maps in coded UI tests in the VS 2010 RC.

You can now add more than one UI map to a test project containing Coded UI tests by just using the Project level menu of Add new itemà Coded UI Map to add new UI map to the test project. If you need to edit the UI map, you can simply right click on the .uitest file node in Solution Explorer and load the UI map file in the Coded UI test builder. From here, you can add new methods/controls, edit controls, rename controls etc.

So, why do we recommend using multiple UI maps in your test project? When your application under test is be a large app with over 1000 forms or web pages, using CUIT in the default single UI map format can result in overloading the UI map and make it cumbersome. Splitting the single map into multiple ones provides the following benefits:

• Each UI map is logically separated and associated with a smaller construct of UI – easier to understand and maintain
• Each tester can individually work on a separate UI map and checkin code when they are ready without interfering with other testers working on other portions of the UI
• Easier to scale UI incrementally by adding new UI maps as compared to conflict resolution using a single UI map

Now, how do you determine when to create a new UI map? If you have a complex composite UI control set doing a logical operation or an independent aggregate of controls that you can navigate to separately and perform specific operations on, it is recommended you create a UI map for that. Eg: a registration page for a new user in a website, a dialog in a winforms app or a WPF app etc. You can have multiple UI maps per single UI aggregate also – for example, for a wizard, you can either create one UI map per wizard page or create a single UI map for the entire wizard, depending on how complex the interactions with the wizard and it’s child controls are.

Note that in many cases, the top level window(TLW) of your application is constant(if your app is a website, the TLW will correspond to the browser window or if it is a winforms/WPF app, the TLW will be the main application window). Since CUIT searches for controls beginning with the ancestor control, which in most cases is the TLW, the instance of the TLW will repeat across multiple UI maps leading to duplication of controls. This can give rise to problems like having to make duplicate fixes across UI maps when the TLW search props change or perf problems when switching between UI maps since the TLW will have to searched again, once per UI map. In order to resolve this, you can equate top level windows across UI maps in the test code using the CopyFrom() API. We’ll see an illustration of how to do this in the walkthrough in my next post.

Pricing for VS 2010 SKUs announced

The pricing for the VS2010 SKUs are finally available. Check out the table below for retail prices

Suggested Retail Pricing (USD) for Visual Studio 2010

* Subscription contents vary by purchased product.

Note that if you are buying VS for your company, volume licensing is a preferred mode of buying SKUs and that will usually be lesser than the retail prices.

A production license for TFS 2010 and the CAL for TFS 2010 is included in every MSDN subscription. And the Visual Studio Agents SKU will be free additional software with the VS Premium, VS Ultimate and VS Test Pro SKUs 

For a detailed look at what each SKU contains you can check out the table in my previous entry here or from this nice little poster from the Rangers’ VS ALM documentation

How to simulate more than 250 virtual users on Load Test in VS2010 Beta2?

VS2010 Beta2 comes with a default license of only 250 virtual users per client. So, if you want to run a load test that needs to simulate more than 250 virtual users, you need to install an additional VU pack.

As written in Ed’s blog, the easiest way to install an extra VU pack is to install a controller and agent on the box. You can also install many agents on the same box or different m/cs.

To  add virtual users, start a Visual Studio Command prompt as Administrator (on Vista or Win7, just type Visual Studio Command Prompt in the search window, on XP you’ll find this under the Visual Studio group in the start menu). Then run this command line:

TestControllerConfig licenses /addkey:V10000

This will license the controller for 10,000 virtual users. Note that this license is temporary, and will only work in beta2. For RTM, we will only offer the 1,000 user license pack. If you have a Volume License, you will be able to get your license key from the volume licensing site, and then enter a count of licenses you want to install.

Once you have a virtual user license installed, that license can also be used for local runs (that is, the runs do not have to go through the controller).

If you just want to do local runs, and not use the controller and agent, you can run this from the command line to install the virtual user license:

VSTestConfig licenses /addkey:V10000

This enables you to exceed 250 users for local runs, and unlocks all the cores on your CPU. Pretty cool, eh?

What apps and browsers does Coded UI test work on?

Here is a snapshot of the platform support provided by Coded UI (and also ability to Fast Forward in Manual Test Runner)

This will evolve over time based on the support that we, the product team or the community, our partners will add to CUIT. The snapshot on this blog should be updated regularly.

Wanna know about VS2010 ALM in 5 mins?

The VSTS Rangers have come up with this great set of content on VS2010 ALM that is basically intended to answer the q “Tell me about VS2010 ALM in 5 mins”.

The Visual Studio 2010 Quick Reference Guidance consists of compact cheat sheets for Team Foundation Server (TFS) 2010 and Visual Studio (VS) 2010, addressing folks that are unaware of Visual Studio and Team Foundation Server capabilities or have little time to invest in detailed education.

The artifacts include an overview document and poster that allows you to quickly focus on individual areas like testing, by providing crisp and compact guidance sheets and quick reference posters. You can take these to your 5-min coffee break discussions or use them as a stepping stone to the more detailed and in-depth guidance you will find on MSDN.

Check out the testing posters – I loved them! :)

VS2010 demos for testing tools – MTLM, CUIT and related features

Our PM team has put together a great series of demo videos to illustrate the new set of features available for testers as part of VS2010. We will eventually post these on C9 and team sites, but I wanted to publish this now to let you guys get a quick dekko at the videos. Here you go:

Running tests in mstest without installing the VS IDE

Yes, you CAN run your unit tests or any other VS test type on a machine without having Visual Studio IDE installed on that machine. You need the Visual Studio Team Agent to be installed on your test machine to run your tests. This is a 15 min lightweight install and is pretty easy to configure.

Those of you familiar with remote testing from our previous versions, will already know that you can trigger a test run on the client to run tests on a remote machine with the test agent installed. Just use the test settings to point to a test agent where your tests can run remotely. With the VS Team Agent installed on machine A and the client (VS Ultimate or Pro or Test Elements) installed on machine B, you can run tests remotely on A by triggering the run from client B.

Another alternate scenario where you would want to run tests without having the Visual Studio IDE installed is during builds. If you use Team Build to do builds and run tests as part of your build process, you can setup the build agent on a machine with no VS IDE installed. You can now go ahead and run all your tests without any additional installs on the build machine in VS 2010.

What if you want to run mstest from the command line though? You can choose to run mstest.exe from either the test agent or the build agent to run tests, but note that you can run only the unit test and ordered test types.

How to remove unwanted actions from the recording in Manual Test Runner/MTLM?

If you were recording your application while doing manual testing inside MTLM, you must have come across a scenario where you wanted to clean up your recording by deleting a few unnecessary actions that were recorded inadvertently. In the example below, I illustrate a case where I opened msn.com and searched for a string – an extra mouse hover has been recorded over the search text box.

Now, there are 2 options:

1. Don’t edit the recording –  since the tool is smart enough to ignore failed hovers during playback and won’t interfere with your test playing back.

2. Edit the recording and delete the extra action – this is useful in cases where the extra action(s) are to dismiss one-time dialogs like the “Do you want to remember passwords” dialog for IE that can be suppressed for future times. Now, if you encounter that dialog during record, but are sure you won’t hit it during playback, you can delete this action from your recording.

Just expand the section at the bottom of MTR to reveal the list of actions being recorded. Select the action to delete and right click –> delete action. There you go – your recording will now be devoid of the extra action.

Wanna win a Hyundai i20 car? Answer some qs on VS 2010

Check out the VS ALM challenge at http://www.vsalmchallenge.com/ – exciting stuff! Take the challenge, answer simple questions on VS 2010 and you shall be on your way to win a Laptop, Yamaha Fazer bike or even a Hyundai i20 car!

I thought this is a really cool offer for VS enthusiasts. Should figure out if employees are eligible to take the test – specially those of us that built VS 2010 ;-)

WPF controls not getting identified in Beta2 CUIT?

Are you having trouble identifying a few of your WPF controls in Coded UI Test after upgrading to Beta2 or getting a warning during recording? You need to install the automation api 3.0 on the client to get CUIT to identify the virtualized controls in WPF.

Note that this update gets automatically downloaded if you are running Vista as part of the Vista platform update. In case of XP or W2K3, you can install the standalone version from here

What's new for manual testers in VS2010 Beta2?

I just gave a talk with the same title at Virtual Tech Days 2010. Deck and videos shared on my skydrive

It's always fun to talk about your product. In this session, voice was enabled only for the presenter - so it was kinda weird talking into the phone for an hour without hearing anything in return. The QnA session at the end helped reinforce the fact that there was an audience that was listening on the other side :)   

Tool to import test cases from Excel to MTLM (VS 2010 Beta2) now available on codeplex

Importing test cases from Excel to VS2010 is a no longer a pain. Check out Test Case Migrator (Excel) on codeplex - an automated tool that gives end to end support to move test cases from Excel to the new TCM capabilities with MTLM in VS 2010! 

It has a nifty wizard that lets you map columns in Excel with fields in your work item type and also lets you split out your test steps with different delimiters or columns. Plus this works across both Agile and CMMI methodologies out of the box. Note this is a one time import only and won’t let you continue to keep your test cases in Excel in sync with TCM though.

So, go on – give it a shot and tell us what you think!

Designing the Coded UI Test Builder

The Coded UI test interface has undergone a redesign in Beta2 to make it easier and more intuitive. While designing this UI, we went through a common dilemma – do we need this to be a guided flow or a choice of operations, all available in parallel? So, is this a wizard or a toolbar?

The basic flows are:

• Record method > Generate code > Repeat > Add control > Add assertion > Generate code
• (Generate code from MTR strip before) > Add control > Add assertion > Repeat > Generate code
• Add control > Repeat > Generate code : for folks that will write code on the controls added instead of recording

Additional flows focusing on incremental ops were many. Primary among those were:

• Edit the UI elements to rename or delete them
• Add additional assertions to pre-existing controls

A wizard would give ample guidance on when to hit the red button, when to drop the cross hair etc. creating a nice little workflow. But it would also constrain the user to one or two of the new flows above. Incremental flows were becoming tedious with the wizard. Plus the wizard was taking up significant space on the desktop contesting with the app under test.

A stack bar would allow the user to choose from the options available to create the shortest desirable workflow. But it was also confusing to first time users on how to use the builder. We have tried to address that through tooltips and detailed walkthroughs, that should hopefully help.

So, the stack bar now appears at the bottom of the screen with a large tooltip that draws the user’s attention to itself:

Users can choose to record actions through the red button or choose to drag and drop the cross hair icon to capture controls. The last button is for the eventual action of generating code.

On dropping the cross-hair on a control of your choice, the UI Control Locator comes up listing the properties of the control along with the navigate bar and a button that lets you add assertion on the chosen property.

Once you choose a property, you can hit the add assertion button to see the dialog where you can choose the comparator and the value to compare with. the generate code dialog comes up:

The UI map is available for viewing in the expand bar to the left of the props window. It’s a bit like the collapsible calendar in Outlook. Click on the expand button and you can see the list of controls available in the UI Map – this consists of all the UI elements you have acted upon in either recording or adding assertions.

Once you are done with recording actions and adding assertions, hit generate code and that will complete the operation by emitting code into VS.

What’s new for testers doing UI automation in VS 2010 Beta2?

We are back in Beta2 with a bunch of new stuff in Coded UI test for UI automation testers. First up, the code generation in Coded UI test has been changed to structure the code in a much more usable and intuitive way:

1. All the UI elements you interact with, whether part of recording or added from the UI control locator, all go into a single UI Map – a single collection of all the UI objects in your test. So, you can now edit objects from a single store

2. Every assertion you add is now a method in itself which takes parameters where you can define the expected value to assert on. So, you can now separate the test into logical actions and assertions

3. The recorded methods also have input parameters defined on them that help make data driving easier to do. So, you can now data drive the method by changing the values of the input parameters that go into the recorded method.

For example, if you were testing the new user sign up in hotmail.com and checking if a certain id’s availability is displayed correctly, you can record steps to open hotmail.com, go to sign up page, enter a particular id and press “check availability” button.

The coded UI test project generated has 4 files – the coded UI test .cs file (TestHotmail.cs) containing the test method, UIMap.uitest file which is an XML representation of all the UI elements and actions/assertions performed on each UI object, the UImap.designer.cs file which is the code-behind for the .uitest file containing code for the objects and actions, the UIMap.cs is a file containing a partial definition of the UImap class to help preserve user customizations to the test code.

TestHotmail.cs – contains the actual test method and invocations to recorded methods and assertions

[CodedUITest]

    public class TestHotmail

    {  [TestMethod]public void TestNewUserSignUp(){

            this.UIMap.GotoSignUpPage();

            this.UIMap.CheckAvailabilityOfId();

            this.UIMap.VerifyIdAvailable();

        }}

UIMap.designer.cs – contains the definitions of the recorded methods, assertions and UI objects. Plus also contains the parameter class definitions passed into each class

public partial class UIMap    {

        public void GotoSignUpPage()

        {    HtmlInputButton signupButton = this.SignInWindowsInterneWindow.HttploginlivecomlogiClient.SignInDocument.SignupButton;

            // Click 'Sign up' button

            Mouse.Click(signupButton, new Point(47, 11)); }

       public void CheckAvailabilityOfId()

        { HtmlEdit imembernameliveEdit = this.SignupWindowsLiveWinWindow.HttpssignuplivecomsiClient.SignupWindowsLiveDocument.ImembernameliveEdit;

            // Type 'anutthar' in 'imembernamelive' text box

            imembernameliveEdit.Text = this.CheckAvailabilityOfIdParams.ImembernameliveEditText;

           // Click 'Check availability' button

            Mouse.Click(checkavailabilityButton, new Point(88, 15));      }}

// Input parameter into the CheckAvailabilityOfId recorded method

    public class CheckAvailabilityOfIdParams

    {      public string ImembernameliveEditText = "anutthar";    }

Other than that, the CUIT builder has gone through a major redesign and morphed into a stack bar at the bottom of your screen with pop up dialogs for the control locator, actions viewer, generate code etc. More on that in the next post

We’ve also added support for virtualized controls in WPF. You will need to wait until the UI Automation 3.0 APIs are available to get this working though. But this should certainly help WPF programmers trying to write tests on their UI apps.

Go ahead, give CUIT a spin and tell us what you think…