Musings of a tester testing test tools! : Automation

Running tests in mstest without installing the VS IDE

anutthara — Wed, 16 Dec 2009 10:52:05 GMT

Yes, you CAN run your unit tests or any other VS test type on a machine without having Visual Studio IDE installed on that machine. You need the Visual Studio Team Agent to be installed on your test machine to run your tests. This is a 15 min lightweight install and is pretty easy to configure.

Those of you familiar with remote testing from our previous versions, will already know that you can trigger a test run on the client to run tests on a remote machine with the test agent installed. Just use the test settings to point to a test agent where your tests can run remotely. With the VS Team Agent installed on machine A and the client (VS Ultimate or Pro or Test Elements) installed on machine B, you can run tests remotely on A by triggering the run from client B.

Another alternate scenario where you would want to run tests without having the Visual Studio IDE installed is during builds. If you use Team Build to do builds and run tests as part of your build process, you can setup the build agent on a machine with no VS IDE installed. You can now go ahead and run all your tests without any additional installs on the build machine in VS 2010.

What if you want to run mstest from the command line though? You can choose to run mstest.exe from either the test agent or the build agent to run tests, but note that you can run only the unit test and ordered test types.

What’s new for testers doing UI automation in VS 2010 Beta2?

anutthara — Mon, 26 Oct 2009 05:31:06 GMT

We are back in Beta2 with a bunch of new stuff in Coded UI test for UI automation testers. First up, the code generation in Coded UI test has been changed to structure the code in a much more usable and intuitive way:

1. All the UI elements you interact with, whether part of recording or added from the UI control locator, all go into a single UI Map – a single collection of all the UI objects in your test. So, you can now edit objects from a single store

2. Every assertion you add is now a method in itself which takes parameters where you can define the expected value to assert on. So, you can now separate the test into logical actions and assertions

3. The recorded methods also have input parameters defined on them that help make data driving easier to do. So, you can now data drive the method by changing the values of the input parameters that go into the recorded method.

For example, if you were testing the new user sign up in hotmail.com and checking if a certain id’s availability is displayed correctly, you can record steps to open hotmail.com, go to sign up page, enter a particular id and press “check availability” button.

The coded UI test project generated has 4 files – the coded UI test .cs file (TestHotmail.cs) containing the test method, UIMap.uitest file which is an XML representation of all the UI elements and actions/assertions performed on each UI object, the UImap.designer.cs file which is the code-behind for the .uitest file containing code for the objects and actions, the UIMap.cs is a file containing a partial definition of the UImap class to help preserve user customizations to the test code.

TestHotmail.cs – contains the actual test method and invocations to recorded methods and assertions

[CodedUITest]

public class TestHotmail

{ [TestMethod]public void TestNewUserSignUp(){

this.UIMap.GotoSignUpPage();

this.UIMap.CheckAvailabilityOfId();

this.UIMap.VerifyIdAvailable();

}}

UIMap.designer.cs – contains the definitions of the recorded methods, assertions and UI objects. Plus also contains the parameter class definitions passed into each class

public partial class UIMap {

public void GotoSignUpPage()

{ HtmlInputButton signupButton = this.SignInWindowsInterneWindow.HttploginlivecomlogiClient.SignInDocument.SignupButton;

// Click 'Sign up' button

Mouse.Click(signupButton, new Point(47, 11)); }

public void CheckAvailabilityOfId()

{ HtmlEdit imembernameliveEdit = this.SignupWindowsLiveWinWindow.HttpssignuplivecomsiClient.SignupWindowsLiveDocument.ImembernameliveEdit;

// Type 'anutthar' in 'imembernamelive' text box

imembernameliveEdit.Text = this.CheckAvailabilityOfIdParams.ImembernameliveEditText;

// Click 'Check availability' button

Mouse.Click(checkavailabilityButton, new Point(88, 15)); }}

// Input parameter into the CheckAvailabilityOfId recorded method

public class CheckAvailabilityOfIdParams

{ public string ImembernameliveEditText = "anutthar"; }

Other than that, the CUIT builder has gone through a major redesign and morphed into a stack bar at the bottom of your screen with pop up dialogs for the control locator, actions viewer, generate code etc. More on that in the next post

We’ve also added support for virtualized controls in WPF. You will need to wait until the UI Automation 3.0 APIs are available to get this working though. But this should certainly help WPF programmers trying to write tests on their UI apps.

Go ahead, give CUIT a spin and tell us what you think…

Using regular expressions to locate controls in Coded UI Test

anutthara — Wed, 19 Aug 2009 13:14:00 GMT

When testing UI applications, you often want to locate a control on the app under test that has dynamic properties. For instance, when you login into a website, there may be a link/label on the first page that says “Welcome user, UserName1”. Now this username is going to change as per the user that logged in.

While writing a coded UI test to locate this label however, the search properties are set to the value that was found during recording. While playing back if you logged in as a new user, the search fails since it was looking for “UserName1” instead of your logged in “UserName2”. To address this, you can tweak the search properties of the control you are using:

UITestControl lblObj = new UITestControl();
lblObj.SearchProperties.Add("InnerText", "Welcome", PropertyExpressionOperator.Contains);

The above code uses “Contains” to ensure that it is searching for a string containing the substring “Welcome”. If you need a more accurate search, you might want to use regular expressions for matching the title. Unfortunately, we don’t have built in support to match the a search property against a given regular expression. However, you can write code to find the set of labels that are under a given hierarchy and match each of their inner text properties against a regular expression of your choice using the standard .NET APIs

But will the dog eat the dog food?

anutthara — Wed, 11 Jul 2007 11:40:59 GMT

Since I am a tester on the Test edition of VSTS - I get this q a lot of times: "Do you use VSTST for testing VSTST?" And the answer is "You bet, I do!" :)

As you might know already, this is called "Dogfooding". Many Microsoft products are dogfooded extensively before release - Vista, MSN Search, Visual Studio, Team System...

Our devs use unit test types to write their developer regression tests. Our QA team uses unit tests to write our functional tests. Yes, I know this is not a perfect fit for UI heavy testing, but it does suffice for the time being. We use generic tests to wrap our legacy tests that may be scripts or batch files. Load tests are added to the test bed to do stress testing on the app. All of this goes into the TFS server that we have deployed for the entire Developer Division and testing performed out of it.

Like any other user, I have my own set of pet features and pet peeves in TST. I absolutely love the code coverage coloring, the "Run test" from the editor (Orcas feature), the ease with which load tests can be configured and the controller-agent functionality. It makes my life so much easier as a tester. Not to mention that it's incredibly easy to point out to devs where the problems are and associate results with bugs since we use TFS for work item tracking. Now for the bad parts - I think the test list feature is absolutely useless for large test projects. The UI confuses me - more than once, I have selected a test in the window, right clicked and said "Run checked tests", but some other tests start running!! That's because selecting a run is not the same as checking it. :( I have a wishlist as well - stuff like test case management being easier to handle, the test run results being easier to analyze etc.

The best part of dogfooding is that this is a good chance to know which features are likely to be real relief and which ones are mostly going to tick the users off. Testers being the first customers of the tool is literally true in this case. Plus the QA team gets additional testing in the process so that bugs/suggestions are filed before outside users get a chance to go at it.

So, do you eat your own dogfood? ;-)