Visual Studio Team Test

How to remove unwanted actions from the recording in Manual Test Runner/MTLM?

VSTS Quality Tools — Wed, 23 Dec 2009 17:21:00 GMT

If you were recording your application while doing manual testing inside MTLM, you must have come across a scenario where you wanted to clean up your recording by deleting a few unnecessary actions that were recorded inadvertently. In the example below, I illustrate a case where I opened msn.com and searched for a string – an extra mouse hover has been recorded over the search text box.

Now, there are 2 options:

1. Don’t edit the recording – since the tool is smart enough to ignore failed hovers during playback and won’t interfere with your test playing back.

2. Edit the recording and delete the extra action – this is useful in cases where the extra action(s) are to dismiss one-time dialogs like the “Do you want to remember passwords” dialog for IE that can be suppressed for future times. Now, if you encounter that dialog during record, but are sure you won’t hit it during playback, you can delete this action from your recording.

Just expand the section at the bottom of MTR to reveal the list of actions being recorded. Select the action to delete and right click –> delete action. There you go – your recording will now be devoid of the extra action.

- Anu

Cross posted from Musings of a tester testing test tools

Running tests in mstest without installing the VS IDE

VSTS Quality Tools — Tue, 22 Dec 2009 11:53:40 GMT

Yes, you CAN run your unit tests or any other VS test type on a machine without having Visual Studio IDE installed on that machine. You need the Visual Studio Team Agent to be installed on your test machine to run your tests. This is a 15 min lightweight install and is pretty easy to configure.

Those of you familiar with remote testing from our previous versions, will already know that you can trigger a test run on the client to run tests on a remote machine with the test agent installed. Just use the test settings to point to a test agent where your tests can run remotely. With the VS Team Agent installed on machine A and the client (VS Ultimate or Pro or Test Elements) installed on machine B, you can run tests remotely on A by triggering the run from client B.

Another alternate scenario where you would want to run tests without having the Visual Studio IDE installed is during builds. If you use Team Build to do builds and run tests as part of your build process, you can setup the build agent on a machine with no VS IDE installed. You can now go ahead and run all your tests without any additional installs on the build machine in VS 2010.

What if you want to run mstest from the command line though? You can choose to run mstest.exe from either the test agent or the build agent to run tests, but note that you can run only the unit test and ordered test types.

- Anu

Cross posted from Musings of a tester testing test tools

How Test Case result is computed in Test Runner?

VSTS Quality Tools — Mon, 21 Dec 2009 16:04:31 GMT

Note: Cross posted from Rubel's Blog.
Permalink

“I’ve NOT marked any test step BUT test case result is auto-computed as passed sometimes and failed sometimes.“

“I’ve marked a few test steps to pass / fail but auto-computed test case result is not what I had expected.”

Have you also faced similar kind of issues around auto-computation of test case result? Do you find test case auto-computation logic confusing sometimes? if yes then it won’t be any longer confusing for you after reading this post :)

Before understanding test result auto-computation logic, you need to understand difference between the simple test steps and validation test steps. While creating a test case in Microsoft Test and Lab Manager you can list the action and the expected result for each test step in your manual test case:

Under Action, there is action that the tester takes to perform this test step.
Under Expected Result (Optional), there is result that user should expect after the action has been performed.

Simple test steps just have an action assigned to them and expected results is NOT set for them. If you set expected results for a test step, it automatically becomes a validation test step (shown by the red check mark on the step icon). E.g. in test case shown below, 2^nd and 4^th are validation test steps whereas 1^st and 3^rd are normal test steps:

In Test Runner also validation test steps are differentiated from normal test steps by an extra red check mark on the test step result icon as shown below:

So now as you understand the difference between validation test steps and normal test steps, this is simple algo used for auto-computation of test case results in Test Runner:

If your test case has ZERO validation test steps then your test case result will be auto-computed as “Passed” unless you’ve explicitly marked one / more test steps as failed.
If your test case has one / more validation test steps then your test case result will be auto-computed as “Passed” if and only if:
- ALL validate test steps as marked as passed
- NO other test step is marked as failed

In short, it is mandatory to mark Validation Test Steps. If you don’t mark any validate test step to pass / fail then it is considered as FAILED so please do mark ALL validation test steps in your test case to ensure that test case result is auto-computed properly.

Note: Cross posted from Rubel's Blog.
Permalink

Another Prerelease Of VS10

VSTS Quality Tools — Fri, 18 Dec 2009 18:45:39 GMT

We’re very excited about the recently announced plans to make a publicly available RC (release candidate) of VS10. We’ve done a lot of work since Beta 2 went out, and we’ve been cranking on bug fixes as well as integrating customer feedback.

For more information, check out these blog posts by Soma, Scott, and Brian.

http://blogs.msdn.com/somasegar/archive/2009/12/17/visual-studio-2010-and-net-framework-4-beta-period-extended.aspx

http://weblogs.asp.net/scottgu/archive/2009/12/17/visual-studio-2010-and-net-4-0-update.aspx

http://blogs.msdn.com/bharry/archive/2009/12/17/vs-2010-net-4-public-release-candidate-planned.aspx

Blogger Dandino Discovers Shared Steps

VSTS Quality Tools — Wed, 16 Dec 2009 06:18:51 GMT

http://blog.nfocus.co.uk/2009/12/share-and-share-alike.html

How can I configure MTLM to use my custom bug / test case type?

VSTS Quality Tools — Tue, 15 Dec 2009 12:37:00 GMT

Note: Cross posted from Rubel's Blog.
Permalink

How can I configure MTLM to use my custom bug / test case type?

I got this Q on my previous blog and thought of posting a separate post on it as it's a bit tricky to do it and hence other folks may need this info.

In Microsoft Test and Lab Manager (MTLM), we don’t provide any UI way to set a particular work item type as default work item type for a category. Basically, if I’ve added a new bug type say “CustomBug” to TFS and want MTLM to refer to that type always while opening a new bug etc then there is no UI way to achieve it.

But yes, you can do it through witadmin using importcategories / exportcategories command.

First you need to exports the XML definition of categories defined on the Team Foundation Server by using exportcategories command e.g.

The following command exports the work item type categories defined for the AdventureWorks project to the Categories XML file:

C:\Program Files\Microsoft Visual Studio 10.0\VC>witadmin exportcategories /collection:"http://AdventureWorksServer:8080/AWTeam/Collection1”

/p:AdventureWorks /f:"Categories.xml"

Categories.xml will look this:

<?xml version="1.0" encoding="utf-8"?>

<cat:CATEGORIES xmlns:cat="http://schemas.microsoft.com/VisualStudio/2008/workitemtracking/categories">

</CATEGORY>

</CATEGORY>

</CATEGORY>

</CATEGORY>

</cat:CATEGORIES>

As you can see that there is default work item type for each category like Bug, Test Case, Shared Step.

You just need to modify the xml file so as to point DEFAULTWORKITEMTYPE of required category to yours newly created work item type. E.g. If I want to use work item type CustomBug as default work item type for Bug category then my modified XML will look like this:

<?xml version="1.0" encoding="utf-8"?>

<cat:CATEGORIES xmlns:cat="http://schemas.microsoft.com/VisualStudio/2008/workitemtracking/categories">

</CATEGORY>

</CATEGORY>

</CATEGORY>

</CATEGORY>

</cat:CATEGORIES>

After that you just need to import this XML back into team project of TFS using importcategories. e.g. The following example imports categories from the myCategories XML file to the AdventureWorks project:

C:\Program Files\Microsoft Visual Studio 10.0\VC>witadmin importcategories /collection:"http://AdventureWorksServer:8080/AWTeam/Collection1”

/p:AdventureWorks /f:"Categories.xml"

Now if you open any bug in Test Runner / MTLM then it will open your CustomBug work item type form instead of default bug form. Similarly you can do required changes to use customized Test Case / any other work item.

Required Permissions

For the team project where the work item types are defined, you must have the following permissions set:

· To export categories of work item types, you must be a member of the Readers group or have your View work items in this node permission set to Allow

· To import categories of work item types, you must be a member of the Team Foundation Administrators security group or the Project Administrators security group.

For more details on TFS permissions check here

Note: Cross posted from Rubel's Blog.
Permalink

Is Coded UI Test under Load Test supported?

VSTS Quality Tools — Tue, 15 Dec 2009 12:03:34 GMT

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

I have seen multiple queries on “Is running Coded UI Test under Load Test supported?”.

Well, the short answer is – yes, it is supported with restriction of one user-load per test agent and in most cases it ...

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

Reinstalling your Test Controller keeping environments intact

VSTS Quality Tools — Mon, 14 Dec 2009 17:36:00 GMT

So you have probably quickly found out that if you ever have to rebuild your test controller isn’t possible from the installer provided.

Here are some steps to help make the process a little easier:

Locate the QTControllerConfig.xml file and back it up to a safe location. This file is usually located in your Visual Studio install location in the Common7\IDE folder.
Rebuild the machine with the same name as needed
Reinstall the test controller on the machine you just built out.
Replace the default QTControllerConfig.xml file on the machine with the file you backed up in step 1
Restart the machine.

Thanks

Bruce Taimana
Visual Studio Test Team
Program Manager

How to delete a Test Case / any other Work Item?

VSTS Quality Tools — Fri, 11 Dec 2009 10:29:44 GMT

Note: Cross posted from Rubel's Blog.
Permalink

I saw some Qs on forums asking about how to delete a test case from MTLM (Microsoft Test and Lab Manager). Please note that you can’t delete a test case from MTLM. But as test case is eventually a work item so you can achieve the same using witadmin provided you’ve admin permissions. witadmin is a command-line utility and is located in %Program Files%\Microsoft Visual Studio 10.0\Common7\IDE. Please note that this is an admin operation and we don’t encourage using it a lot in general. The usual guideline is to close the obsolete / not required test cases and move it out of your test suite.

Anyhow, here is the way to delete the test case or any other work item using witadmin:

witadmin destroywi /collection:CollectionURL /id:id [/noprompt]

The following example deletes the work items with IDs, 12, 15, and 23 from the database for Collection1 on the AdventureWorksServer server:

witadmin destroywi /collection:http://AdventureWorksServer:8080/AWTeam/Collection1 /id:12,15,23

You can read in detail about permanently removing work items using witadmin here.

Note: Cross posted from Rubel's Blog.
Permalink

Logging a message in test result as part of an automated test

VSTS Quality Tools — Wed, 09 Dec 2009 06:44:52 GMT

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

Multiple folks have asked in forums on how to log a message from a Coded UI Test for tracing\debugging purpose. I know of as many as 5 ways of doing this from any test type (Coded UI Test, Unit Test etc). Try this code out -...

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

Test Runner- Using Global Shortcut Keys

VSTS Quality Tools — Wed, 09 Dec 2009 05:31:50 GMT

Note: Cross posted from Rubel's Blog.
Permalink

While executing a test case in Test Runner you’ll have to mark test steps and browse through test steps quite frequently. It won’t be very good experience for tester if he has to move focus back and forth from AuT to Test Runner and Test Runner to AuT. To solve this issue, we’ve provided Global HotKeys for some frequently used operations in Test Runner:

1. Pass Active Test Step: Win+Ctrl+Q

2. Fail Active Test Step: Win+Ctrl+W

3. Move to Next Test Step: Win+Ctrl+X

4. Move to Previous Test Step: Win+Ctrl+Z

5. Capture Image: Win+Ctrl+C

Please note that Global HotKeys are system-wide hotkeys and enables us to perform action on an application even if focus is NOT on that application. So if you working over your AuT and want to mark a test step to pass in Test Runner then you can simply use “Win+Ctrl+Q” for it without moving focus to Test Runner window. We’ve kept these global hotkeys limited because we shouldn’t register too many global hotkeys for any application. This is because more global shortcut keys an app registers more chances are there for collision of its global shortcut keys with other applications which may have already registered that key.

You can face Global shortcut key registration failure while starting Test Runner if you already have some app running on your box which has registered one / more keys out of the 5 keys mentioned above. In that scenario you’ll see this message box while starting Test Runner from MTLM (Microsoft Test and Lab Manager):

As it is clear from the message box content you need to modify the global hotkeys for Test Runner from the mtlm.exe.config file which is present at:

%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE

You’ll see there entries for global keys as shown below:

<add key="GlobalPassTestStep" value="Windows, Control, Q"/>

<add key="GlobalFailTestStep" value="Windows, Control, W"/>

<add key="GlobalNextTestStep" value="Windows, Control, X"/>

<add key="GlobalPreviousTestStep" value="Windows, Control, Z"/>

<add key="GlobalCaptureImage" value="Windows, Control, C"/>

You need to modify the clashing entry so as to use some other key and restart MTLM.

Note: Cross posted from Rubel's Blog.
Permalink

How To: Enable Tracing for “UI Test” Components

VSTS Quality Tools — Wed, 09 Dec 2009 05:26:34 GMT

Note: Cross posted from Gautam's Blog.
Permalink

Like any other good product, the Testing Tools in VS also have tracing support to generate the extra trace logs that can help the product team debug the customer issues better. Though this trace logs could be used by the customers in certain scenarios, these are meant for the product team only. When reporting a bug or asking for support from the product team via forums or other channels, it is always good to generate the repro with the tracing ON and submit these trace logs too.

Below are various different ways to enable tracing -

The quick and easiest way to enable tracing for all Testing Tools in VS is to use the registry files in the zip attached. The zip has two files – one to enable and another to disable tracing. SECURITY WARNING – Please validate the registry file before importing. The registry file is -

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools\Diagnostics]
"EnableTracing"=dword:00000001
"TraceLevel"=dword:00000004

In the above, the EnableTracing variable controls whether tracing is ON or OFF.

1 means tracing is ON
And 0 means OFF.

The TraceLevel variable is for level (granularity) of tracing.

1 means Errors only
2 is Errors and Warnings
3 is Errors, Warnings and Information
4 is All including Verbose Information

Typically for reporting issues, it is good to set it to 4. Once you have reported the issue, you should turn the tracing off by changing EnableTracing to 0. Alternatively, use the DisableTrace.reg file in the attached zip.

Set EqtTraceLevel or UITestTraceLevel switch as appropriate in the corresponding *.exe.config file. (Typically these values are there towards the end in the config files.) The EqtTraceLevel will turn tracing ON for all Testing Tools modules of the executable whereas UITestTraceLevel will turn ON the tracing for only the UI Test modules of the executables. Here the EqtTraceLevel and UITestTraceLevel has the same 1-4 values like the TraceLevel variable above.

The following are the *.exe.config that you need to modify depending on your scenario -

For Action Log and Fast Forward feature of MTLM – mtlm.exe.config

For recording\assertion in Coded UI Test Builder – CodedUITestBuilder.exe.config.

For playback of Coded UI Test - QTAgent.exe.config. (Note – In case of remote playback, you need to modify QTAgent.exe.config of the remote machine.)

All the above config files are located at “%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE” where the part in bold is the default installation directory. If you have changed the installation directory of the product, you need to replace it appropriately here too. To disable you will have to turn off the setting in the same file.

Note that for trace settings changes to take affect, the executable (VS or MTLM or others) need to closed and restarted.

For UI Test, the trace file is generated in %temp%\UITestLogs\*\LastRun\UITestLog.html where * could be empty for VS or could be exe name like CodedUITestBuilder for other cases. The LastRun here indicates that this is THE file for last run – the earlier ones would be stored as PreviousRunXX. The tool automatically cleans up old trace file if it exceeds ‘n’ (today it is 25) previous runs.

Note: Cross posted from Gautam's Blog.
Permalink

Troubleshooting Data Diagnostic Adapters- Log is not getting generated?

VSTS Quality Tools — Mon, 07 Dec 2009 05:42:19 GMT

Note: Cross posted from Rubel's Blog.
Permalink

As I had mentioned in my earlier posts that you can enable a number of Diagnostic Data Adapters in test setting so as to collect various types of data while running tests in Test Runner. Sometimes it is confusing if you don’t get any log from a diagnostic data adapter though you’ve enabled it in test setting. There can be many reasons for this behavior like data collector is not expected to generate log in that particular scenario, data collector pre-reqs are not met successfully, that particular data collector is not expected to generate log at all :) e.g Network Emulation. This post intends to explain various reasons for log generation failure from each data diagnostic adapter. This is NOT an advanced guide for troubleshooting all data diagnostic adapters’ warnings / errors but you can quickly refer to it if you are not seeing logs from a data diagnostic adapter.

IntelliTrace and Test Impact

Both IntelliTrace and Trace Impact data collectors have a number of pre-reqs:

1) They work only for applications written with managed code

2) AuT process need to be started after launching the Test Runner as they needs to instrument modules, binaries or executables that are loaded into a process.

3) ASP .NET Client Proxy for IntelliTrace and Test Impact needs to be enabled in test setting for client role while collecting test data from ASP. NET applications hosted on remote boxes.

4) Computer’s IP address / hostname need to be used instead of “127.0.0.1” “local host” while collecting data from locally hosted ASP.NET apps e.g. please use http://microsof-7f734b/webapplication1/default.aspx instead of http://localhost/webapplication1/default.aspx

5) AuT Process / required modules are included for data collection in configuration settings of collectors. Please check articles mentioned below for details on configuring IntelliTrace and Test Impact:

a. Configuring IntelliTrace

b. Configuring Test Impact

So if one / more prereqs mentioned above are not satisfied then you won’t get IntelliTrace / Test Impact data collector log. Another common mistake that you may do: Launching AuT after launching MTR but from an already opened command prompt. Please avoid it.

One more important thing to note:

You’ll never get both IntelliTrace and Test Impact log for same Test Case result. This is because IntelliTrace log is generated only for Test Cases marked as failed in Test Runner whereas Test Impact log is generated only for Test Cases marked as passed in Test Runner. Additionally, Test Impact log is never attached to bug irrespective Test Case is passed / failed whereas IntelliTrace log will always be attached to bug irrespective Test Case is passed / failed.

Network Emulation

It NEVER generates any log and just used to emulate the slow network connection speed. It affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up.

ASP .NET Client Proxy for IntelliTrace and Test Impact

It NEVER generates any log. This proxy enables you to collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters

Video recorder

I had talked about Video Recorder data collector pre-reqs in my earlier post. These can be the reasons if you are not getting any log from Video Recorder data collector:

1) Window Media Encoder is not installed on box

2) Window Media Encoder hotfix is not installed which is required for Windows Vista, Windows Server 2008 or Window 7 boxes.

3) Desktop experience is not enabled. This is required only for Windows Server 2008 boxes.

4) Test Agent was not running as interactive process. This is required in case you are collecting video data for a remote role.

5) Desktop session was not active during test execution. If session is minimized / locked then you won’t get proper video log.

6) Collector is configured not to generate log for passed TCs as shown below:

In that scenario you won’t get video log for passed TCs.

System Information

System Information is the simplest data diagnostic adapter and is expected to generate log ALWAYS.

Event Log

There is no specific pre-reqs for Event Log data collector and it is expected to generate log ALWAYS.

Action Log

These can be the reasons if you are not getting any action log:

1) No actions are done on AuT (Application Under Test)

2) You don’t have appropriate permissions to record actions on AuT. It may happen if AuT is running under high privileges.

Note: Cross posted from Rubel's Blog.
Permalink

Installing Windows Automation API 3.0 for enabling CodedUITest on WPF Applications

VSTS Quality Tools — Sun, 06 Dec 2009 09:50:08 GMT

Note: Cross posted from Rituparna's Blog.
Permalink

If you are using CodedUITest to test WPF applications then installing Windows Automation API 3.0 is a pre-requisite. Without this the virtualized controls in WPF wont be played back correctly and since lot of the WPF controls like ListBox/Datagrid are virtualized by default , this will lead to failures.

Windows automation API is pre-installed on WIN7 and can be installed on Vista/XP as well. This blog has all the details on howto get Windows Automation API 3.0 installed.

In case the machine does not have WAA3.0 code generated on doing action on WPF application will have the following comment

// An update for Windows Automation API 3.0 must be installed to enable automation on Windows Presentation Foundation controls. Please refer http://go.microsoft.com/fwlink/?LinkId=162159 for more information.

Similar comment is displayed in the MTLM UI for viewing recorded action as well.

This let’s the user know that the recording is not proper and user has to install WAA3.0 and re-record. Also a useful way to identify if the machine actually has Windows Automation API 3.0 installed :)

ciao

Diagnostic Data Adapters getting timed out?

VSTS Quality Tools — Thu, 03 Dec 2009 05:19:30 GMT

Note: Cross posted from Rubel's Blog.
Permalink

In Test Runner you can face Data Collectors time out issue while creating bug, ending test case, starting Test Runner etc basically at all points wherever data collectors are generating logs /getting initialized / disposed. Time out can happen for a number of reasons like network is slow, Client / Test Agent/ Test Controller machine is slow, large number of Data Collectors are enabled in test setting etc.

You’ll see these kinds of dialogs in Test Runner while facing time out issue:

Time Out while starting Test Runner

Time Out while creating a Bug from Test Runner

Time out while ending Test Case in Test Runner

Good thing is that these time outs are configurable in the mtlm.exe.config file:

<add key="DataCollectorInitializationTimeoutInSeconds" value="50"/>

<add key="DataCollectorEventTimeoutInSeconds" value="30"/>

<add key="DataCollectorCleanupTimeoutInSeconds" value="60"/>

There is obviously no “right” default value for this. We want to keep it as low as possible for manual testers but high enough to not timeout in common / 80% scenarios. But if you facing time out issues too frequently then it will be good to modify it the mtlm.exe.config file.

If you are running a test agent also then you need to modify these files also on your machine running test agent:

· QTAgent.exe.config

· QTAgent32.exe.config

· QTDCAgent.exe.config

· QTDCAgent32.exe.config

Note: Cross posted from Rubel's Blog.
Permalink