VS Team System Test

Open Items and Manage Queries

In my last post, I discussed whether Manage Queries is a manager or artifact, and promised to cover other aspects of behavior.

One such distinction of behavior between managers and artifacts is that artifacts show up in Open Items. Managers are just relaunched from the chrome (navigation area), or you can get back to them with the Back and Forward buttons. Artifacts were a little harder to find in the first place since you had to find an item in a list, run a query, or use Control+G to specifically open the work item by ID, so we show them in the Open Items list.

Notice when you first start Manage Queries, the Open Items count grows by 1. Without doing anything in that view, browse away and relaunch from the chrome, and the same view will open. Your Open Items count did not grow.

Now select a query and run it, then browse away. Navigate back to Manage Queries from the chrome. A new item is opened and your Open Items count grows by 1. If you drop down Open Items you’ll see two Manage Queries views open.

This is designed so you can have multiple different queries open at the same time. However we don’t want to open a bunch of blank Manage Queries, so if a blank one exists it is simply brought to the foreground. However if an instance of Manage Queries has info, then you can decide whether you want a new one (via the Chrome) or the old one (via Open Items).

Cheers,

David Williamson

Engineering Lead, MTLM

Manage Queries – Artifact or Manager?

If the most recognizable part of MTLM for anybody who is familiar with VS’s Team Explorer from 2005 and 2008 is the work item, the next most recognizable piece of UI would be Manage Queries. This view is akin to the Work Items node in Team Explorer. It has a Team Queries node, and a My Queries node. Inside you’ll find all the team and personal queries that are also seen in Team Explorer.

So from that aspect this view acts more like a MTLM manager, much like Test Plan Manager, Test Case Manager, Shared Steps Manager, Test Configuration Manager, and Test Settings Manager. All of these views list a bunch of artifacts, which can be opened to view or change. These views can be refreshed, but they cannot be saved (since there is no way to get into a ‘dirty’ state).

However, unlike those manager views, Manage Queries opens the selected artifact inside the same view, rather than in a new window. As long as the item has not changed, the view still has nothing to be saved. As soon as a change happens, we do several things. First, the title gets an asterisk to indicate a save must happen to retain the changes. Second, we disable the tree view which lists all the items since selecting a new item would also lose your changes. If you press the refresh button, you will be prompted to discard your changes. If you try to close the view, same thing.

In this sense, it is definitely an artifact. I like to call it an artifact with a manager view built in. It is definitely a different kind of beast than the other manager and artifact views in MTLM, but it is also a unique kind of activity as well. It seems more likely one will be running several different queries in a row, without changing them. From this aspect, it is convenient to be able to do all that inline – no opening and closing of windows.

Stay tuned tomorrow when I’ll talk about another interesting aspect of this view’s behavior.

Cheers,

David Williamson

Engineering Lead, MTLM

Screenshots!

I’ve been silly lately. I’ve failed to include screenshots with my blog posts. I didn’t realize how easy it would be with Windows Live Writer. I’ve been using the app, but didn’t realize it would handle posting the images somewhere for me.

So, expect all future blogs to have screenshots… and I’ll work on going back through the old blogs and adding screenshots to each.

Humbly,

David Williamson

Engineering Lead, MTLM

 

P.S. Nice job Live Writer team. This app rocks!

Using Shared Steps

In MTLM, test cases can reference a reusable, sharable set of test steps. This cuts down on the amount of work it takes to author new test cases, because many test cases contain some of the same steps.

The ideal use for this is in common tasks. The quintessential shared step is for logging into the application. Assuming your application cares about identity as many do, this is something nearly every test case will need to do as part of the process to get where you want to test.

If you didn’t have shared steps, you would either have to rewrite those steps for every test case, or you’d simply have a test case that handled login and you’d position that test ahead of the others in the test suite.

Rewriting the steps in every test case is painstaking and annoying. I want to be productive as a tester, not waste my time doing repetitive tasks! Worse, if the steps change for logging in, I have to go and change every test case. That is a terrible option, and I would prefer to avoid it as much as possible.

I could just ignore obvious steps like this, but that might be difficult for people who take over my work someday to understand what is expected. It is best if I’m explicit

The other solution I mentioned was ordering test cases in a suite. With this option, I add a test case to the top of my suite called Login, and then have other test cases that require the tester to be logged in appear after. This has worked for many testers for years, and I think it is better than the previous option. However, it is a bit tedious. For one, I have to be very sure the test suite is always ordered correctly. Let’s say I have my Logout test case at the end of the suite. When a new test case is added, I have to be sure to update the order (again) to make sure it is inside of the Login/Logout scope. Also, if I want to include a test case that requires Login in another suite, then I have to make sure that suite also includes the Login (and probably Logout) test. So again, this is just more maintenance.

Also, now I’m testing Login and Logout explicitly in every test suite which may not be interesting to me.

If I go the Shared Steps route, then every test case that needs Login explicitly references it. There’s no need to keep track of the association every suite my test case is in.

The downside is that this implies I need to login and logout with every test case. This really could slow down execution time. Can’t I just run through all my test cases without logging in and out? Sure, I think you could. The steps don’t have to be mandatory. You might make the Login steps conditional (if not already logged in, then do the next few steps), and Logout could also be conditional (Logout if you will close the application).

In the test step control, it is easy to reference existing shared steps. Just click the ‘Insert shared steps’ button, and find the shared step you want in the list. If you need to create a shared step for the first time, you can even promote steps you’ve written to be a new shared step by selecting those rows and clicking the ‘Create shared steps and insert at the current position’ button.

Cheers,

David Williamson

Engineering Lead, MTLM

New features in Beta2 (compared to Beta1)

We recently announced Beta2 of VS2010. If you have installed the Beta1 release and played with it or even if you didn’t and are experiencing the product for the first time, you are perhaps wondering what is new in Beta2. The table below is an enumeration of the new features in Beta2. As you can see from the list below the team has been very busy reacting to your feedback and adding a lot of features to round off the product to address your needs. In a future post I will expand on each of the features and explain how to get to them so you can experience these for yourself. Do let me know which of these features you would like me to blog about first so I can prioritize them higher.

MTLM/Test Runner

·         Number of performance and scale improvements on both client and server side ·         Support for resetting results in a run ·         Improved handling of attachments when filing bugs ·         Better start up experience ·         History (Back & Forward) ·         Full support for any field on a work item being shown in grids ·         Single panel Test Runner with an easier and more intuitive interface ·         Advanced Test Settings support with a much better user experience ·         Full query editing in Test Case/Shared step manager and Add Requirement/Add Tests activities

Coded UI

·         Redesigned the Coded UI Test Builder UI to be more intuitive ·         Code generation enhanced to provide automatic parameterization and  ease of maintenance ·         Support added for WPF Datagrid and virtualized controls. ·         Support for specialized classes

Test Execution

·         Simplified test settings ·         Better surfacing of warnings and errors from data collectors

Remote execution

·         Controller manager in MTLM – improved functionality and usability ·         Interactive agent UI with config UI and systray app

Web Access

·         Repro steps and system information controls on the bug form now usable and editable from web access ·         Read-only Test Steps control on test case form now available in Web Access

Load Test

·         Support for comparing two load test performance runs in Excel

Others

·         Ability to publish test results from VS2008 to a VS2010 server (using the newly released VSTS2008 GDR)

Cheers,

Ram Cherala

Principal Program Manager, Visual Studio Test Tools

Test Case Authoring from the Test Plan Contents

A test case is just a work item. You can create a new one from Team Explorer in Visual Studio. You can open Excel and connect to the server from the Team add-in, and create one from there. In MTLM, you can also go to Test Case Manager and create test cases.

Those are all fine, but if you are authoring test cases in the context of a test plan or requirement, there is a better way. When you are in Test Plan Contents (the suite manager of MTLM) you can add existing test cases to a suite, and you can also create test cases from there as well. You get the same experience of the work item form, but when you are done, the item is also added to the suite.

Note, this works only for static and requirement-based suites.

In the static suite case, the test case is added to the selected suite and all default configs specified by that suite are added to the new test case.

In the requirement-based suite, the test case is linked to the requirement, which adds it to the suite, and all the default configs specified by that suite are added to the new test case.

If you didn’t author the test case from Test Plan Contents, then you’d have to manually go add the test to the suite of your choice after the fact. I said this was better, because it helps save you time by accomplishing both tasks at once (create, add to suite).

Cheers,

David Williamson

Engineering Lead, MTLM

‘Active’ Tests

In the Run Tests activity, you’ll see test cases grouped up by various states including Active, Failed, Blocked, and Completed.

This idea comes from the workflow many customers report to using where they run many tests only once per iteration. So the intent is to take a test that is Active (meaning it is ready and available to be run) and test it in the Test Runner.

Based on the outcome of that test result, the test state will change to Failed or Completed.

Completed tests are done, unless you decide you want to run that test again. You can simply select it and run, you can select it and hit Reset so it moves back to Active (ideally for another person to pick it up), or discover it and reset it as a recommended test based on code churn in the product code that the test covered.

Failed tests will generally have a bug associated with them, so they’ll be rerun from the Verify Bugs activity when the bug is resolved back to the tester.

Through these cycles, the goal is to get every test into the Completed state. Then it is time to do exploratory testing, or identify existing test cases to run again.

When you run tests from the right-hand pane of the Run Tests activity, the selected items are always taken into the run. However, if you select a suite and try to run it, only the tests in the Active state will be taken into the run. This is to avoid taking a Completed tests (one that reached the end of the cycle) into the run, or a Failed test which is still waiting on a bug fix.

Cheers,

David Williamson

Engineering Lead, MTLM

Lab Management?

MTLM stands for Microsoft Test and Lab Manager. This blog is dedicated to the testing aspects of this application (as well as the other testing features in VS).

Even though testers need to use the lab management activities they tend to be doing one type of activity or another, so we have a separate blog just for lab management-specific activities.

Test Configurations in a Test Plan

For many, many test teams, we hear that they don’t typically test on more than one configuration. This is a big departure from what we at Microsoft are used to. Obviously we have many operating system versions including languages, versions of IE, versions of Office, etc. We’re used to dealing with testing on many platforms because of the configuration matrix.

So for testing teams that don’t use different configurations, we are here to make your life as simple as possible. There’s no reason to make your life harder than it has to be. For you, we have an out of the box, default configuration. You can edit it if you want, or just ignore it entirely. Go on about your testing, and don’t worry about configs.

For all the poor souls who do need to test on multiple configurations, we haven’t forgotten you either. We use MTLM ourselves, so it must meet our own needs too.

Creating configurations is fairly straight forward. First, you visit the Test Configuration Manager (in Organize), and open the Edit Test Variables dialog. In here you can define the different factors you will have variations in, such as OS, Browser, language, etc. Then for each factor, define the variations in the right pane. Then you go back to the manager and begin defining test configurations. Give it a descriptive title, and then add the relevant factors to the grid at the bottom, giving each the correct value. This will be helpful later in reporting where you can look for test result trends across the factors rather than simply reporting on each independent configuration.

The other important value on a test config is whether it is the default for new test plans. If it is set to default, whenever a new test plan is created it will automatically include those configs. Yes, you can remove project defaults from your plan, and add non-defaults. The goal for the person defining the test config project defaults is to make sure new plans have the configs you expect.

Test config defaults on the plan are automatically applied to test cases added to the plan. In Test Plan Properties, you’ll find a test configuration selector where this is specified. Again this is meant to make your life easier, so plan ahead and make sure you have the configs set up on your plan before test cases are added. If you get it wrong, don’t worry, you can change configurations applied to test cases after the fact.

There’s yet another level of ways to scope defaults. We’ve already talked about project-level defaults, test plan defaults, and now we’ll introduce test suite level defaults. Any new suite in the plan will inherit from the parent. That can be changed so that all test cases add to it get a different set of configs applied. In the suite header in Test Plan Contents, look for a drop down next to the label that says ‘Test configurations (x):’.

Finally, test cases can be changed independently of the plan and suite. Just select the test cases in question, and click the Configurations button in Test Plan Contents.

One last thing, when you are looking at test cases in the suites in the Test Plan Contents activity, you’ll see one instance of the test case. However, since many configurations can be applied to that once instance of the test case in the suite, these entries will be multiplied when they are viewed in the Run Tests activity. This is because the test case needs to be run multiple times, and selected independently from the other instances so they can be run with other similar configs. To assist in that work flow, you can use the Filter button on the Run Tests suite tree view toolbar to only show a specific config. Now you can focus on testing just on that one config, then change the filter later when you are ready to move onto another config.

Cheers,

David Williamson

Engineering Lead, MTLM

What is a Test Plan?

A common question we hear is how and when to use a test plan in MTLM.

To most testers, a test plan is a document that describes the intended approach to testing for a feature, or more likely a group of features, for a specific release. It typically reviews the features to be tested, discusses the approach to testing, and it may even list some or all of the specific test cases to be run. It definitely covers timelines and deadlines.

In MTLM, we wanted to expand the concept of a test plan beyond just a document. The information in the test plan should be structured, able to be queried, and directly reference the test artifacts that eventually help define when the testing is complete.

So we start with the Test Plan Properties activity that exposes the high level values. We have a title, description, owner, state, start date, and end date. These values show up in the Test Plan picker and Test Plan Manager to help you find the right test plan you want to work in. It also holds information for running tests, like Test Settings and Test Environments for both manual and automated tests.

 

The test lead will probably set up this test plan artifact for the team. There is a lot of flexibility in defining scope and duration for a test plan. It really depends on how you want to use it. However, we can offer some guidance on when scope is too big or small, or when the duration is too short or too long.

Since we define a test plan as a test effort for an area for an iteration, it is up to you to define the area and iteration. For a product release, you could have 3 feature teams and 5 iterations… so should that be 15 test plans? 3? 1? 80?

Well, the answer to that, for me, is whatever is most convenient for you. However there are some hard requirements that we should review first. I think the biggest factor is who is driving and monitoring the test effort. If a project leader has to maintain and switch between several test plans, that ends up being tedious. So perhaps there are several feature teams working in a single test plan because there is a single owner, they’d each work out of different test suites. That way they could avoid stepping on each other, but… they could also have common areas which might be a bonus. It also means reporting can easily be scoped to the whole effort based on what is in the plan. The second most important factor has to do with test run settings. Do these efforts use the same build definition? Do they move forward to new builds at the same time? If not, it can be inconvenient for testers when they kick off test runs to have to manually select the real build they want to test with. So to sum up, the main factors for me are ownership, reporting, team affinity, and build selection.

The other activity that helps define a test plan, is the Test Plan Contents. This is where test cases are added to the plan, and optionally organized into test suites.

Both of these activities live under the Plan activity group.

Cheers,

David Williamson

Engineering Lead, MTLM

Test Steps in MTLM

In a test case, there is a control to enter test steps in the first tab. It has several columns, but typically a person will be adding a test title, and sometimes some expected text. There’s also the ability to add attachments to steps.

What you’ll see off the bat is an insertion row. Click inside it with your mouse, or Tab down to it with the keyboard, and type in your first step title. Assuming you don’t have any expected text for the first step, just hit Enter.

Continue to type in more steps hitting Enter after each one. If you want expected text, just Tab over, type it, then hit Enter. Note that steps with expected text automatically become ‘Validation Steps’. This means the step will be represented and treated differently in the Test Runner since the expectation is that the tester will ensure the application under test has met the requirements. Use these judiciously since you may not want to mark every step of a test. If you are like me, once you know your test, you’ll want to blow through several steps at a time without indicating success or failure.

Each time you hit Enter it commits the current row and gives you a new row. Feel free to save as you go with Control+S.

You can add attachments via the toolbar button, or via the context menu.

There are lots of helpful toolbar buttons including some to help you move steps up or down (supports multi-select too!), insert a step, and insert a reference to shared steps.

The final thing to add here is data parameters. Sometimes you may want your test case to use different data for a step. You can define a parameter here to refer to data you’ll define later. A good example of this is the user name and password to be used to login. While you are typing your step text, prefix the word that you want to represent your parameter with an at sign (@) like: Login to the application as @user with @password. In the grid below, you’ll see an entry for each data parameter appear. Later you can drop into that grid and begin adding values. Note the @ symbol needs to have a space before it, so an email address which has an @ symbol in the middle won’t signal what follows as a data parameter.

Cheers,

David Williamson

Engineering Lead, MTLM

Migrating Test Cases

Anutthara blogged about how to migrate test cases using Excel. This wouldn't be hard to do, except getting test steps into a test case via the standard Excel integration is not possible. Our team has written a migration tool to help with that.

Activity Centers

When you load up Microsoft Test and Lab Manager (MTLM) and after connecting to a server, project, and plan, you’ll notice the top part of the application is devoted to navigation. Let’s break it down.

The first part is the back and forward buttons. This isn’t a browser, but more and more apps these days include a back/forward navigation paradigm because we’re all used to having that luxury. Note you can use your mouse’s or keyboard’s back/forward buttons, along with Alt-Left and Alt-Right.

To the right is a home button that takes you back out to the place where you select TFS servers and projects. In this out-space, you can also see your app registration information, an about box, etc.

Next up is the activity center switcher, which appears as a large drop down box control. It doesn’t quite look like one because we’ve styled it in WPF to look smoother. After install, you’ll see 2 different activity centers: Test and Lab. Depending on which one you have selected, the rest of the center groups and links will change.

In the Testing Center, you have the following center groups: Plan, Test, Track, and Organize. Plan is where you’d go to change the settings of your current test plan, and organize the test suites with test case membership. Test is where you’ll go to start running tests. You can also view previous test runs here; this is primarily for analysis of automated runs. Also, this is where you’ll go to see bugs you’ve logged before, and bugs that are assigned back to you. This is all part of the typical testing workflow: run tests, analyze failures and log bugs, follow up on bugs that have been resolved. In Track, you are looking for a work item using Manage Queries. You can also assign a new build to your plan here, and look at the changes you’ve taken into test since a previous build. Finally, we also have a link to the TFS project portal. The last center group, Organize, is a project-wide view of assets that are related to testing. For instance, a full list of test plans is available here. You’ll also find all the test configurations, test cases, and shared steps in the project. Although this center group will probably not be used very often, when you need it then it’ll come in handy.

In the Lab Center, you’ll manage your test environments, VMs, VM templates, and test settings.

Also up in the navigation area is a link to the current test plan you are connected to. Clicking on that will take you back out to select a different plan, or go back further to select a different TFS project or even server.

We also have the Open Items drop down. Any time you have an artifact such as a test case or test result opened (i.e. anything that is savable) it will show up here.

Cheers,

David Williamson

Engineering Lead, MTLM

Keyboard Shortcuts in MTLM

I don’t know about you, but I’m a huge fan of keyboard shortcuts. The right shortcuts can really speed up your computer use. For instance, some of my favorite ones in Windows include Win+Break to open computer properties, Win+E to open file explorer, and in Win7 all the Win+<shift>+ArrowKey combinations to move and position windows.

We have a few keyboard shortcuts in Beta2 for Microsoft Test and Lab Manager, with more planned for the future. I wanted to share the shortcuts we have in now, and ask for your feedback on what else you’d like to see.

The back and forward navigation buttons can be operated with Alt-Left and Alt-Right, or with your keyboard and mouse back/forward buttons.

Control+W will close the active View (or what I call an activity).

Control+S saves the active View.

F1 brings up contextual help from online documentation, which loads in your default browser.

F5 will refresh or revert the current View.

Of course, you can Tab (or Shift+Tab) around the view and navigation area, if you are not using a mouse. Spacebar, Enter, and Escape work where you’d expect them to. F2 should work anywhere you are renaming something (like in Manage Queries and Test Plan Contents tree views). Context menus for the currently focused item will open with Shift+F10 or the context menu key to the right of your spacebar, assuming your keyboard has it. We often hook up the Delete key for actions that make sense.

We also include accelerators to many toolbar buttons like typical Windows apps. If you don’t see the underlined letter which indicates the Alt+key combination, just press the Alt key and they should show up.

When you get into Test Runner, there are even more keyboard shortcuts. Some will work only when the application has focus (like Control+> and Control+< to move forward and back between test cases in the list), and some are meant to be used no matter which application has focus (like Win+Control+Q, by default, to pass the current step). Those make it easy to keep the test result up to speed with you without having to change application focus back and forth.

So what other keyboard shortcuts do we have planned? Mostly everything around navigation. The idea is to prevent you from having to Tab around the application for common things. For instance, getting to the Open Items drop down, and switching activities. What else would you like to see?

Cheers,

David Williamson

Engineering Lead, MTLM

Blogger Chris Eargle discovers MTLM

In his recent post titled ‘Microsoft Test and Lab Manager’, Chris reports on reading about MTLM on Jason Zander’s blog. Looks like he is excited about the overview of features he saw.

He’s right, this will make life easier for generalist testers out there. :)