Testmundo

Making the most of your bugs

2009-07-31T08:00:00Z

(This is a re-post of my post on the Team Test blog)

I like to think of bugs as glue which links software teams together. Developers write the code, testers test it, and bugs serve as the gooey battle ground where folks try to make sure that the right thing is done for the customer.

While the ideal flow of bugs probably should look something like this:

Tester: finds an issue –> creates a bug –> assigns it to a developer
Developer: fixes the bug –> assigns it back to the tester to be verified
Tester: verifies the fix –> closes the bug.
Everyone sings Kumbaya

If your development process look anything like it does at Microsoft, where there is lots of ambiguity around what the “right” thing to do with a bug is and there exists differing opinions on what a “fixed” bug looks like, your process may look a little more like this:

Tester: files a bug with 1 line of information (“x is broken!”) and assigns it to a Developer
Developer: looks at issue tries to repro it, finds that x isn’t broken on their box, marks the bug as “no repro” and assigns back to Tester to be closed
Tester: finds the bug still exists on their box, complains to the developer who agrees that this indeed is a bug, so they expand on the 1 line in the bug and assign the bug back to the developer
Developer: fixes 1 component of “x being broken” but x is still broken, Assigns back to the Tester stating that the bug is “fixed”
Tester: tries the bug finds that it is slightly better, re-activates the bug noting out the additional failures.
Developer: Fixes the additional components, marks the bug as fixed, assigns to tester for confirmation
Tester: Confirms the fix by going over the issues that originally caused the issue and closes the bug

While not all bugs go through this much back and forth, with misunderstandings around what constitutes a bug and what should be included in a “fix,” I think that it is understandable for there to exist some back and forth between the team. In fact, it is probably healthy to for some friction to exist.

Tools for a Fair Fight

While I think it is healthy for bug disagreements to occur, folks need the tools to fight a fair fight:

Developers need bugs which are super “rich” and provide as much information as possible about the defect under discussion (with screenshots, video, actions about the application under test, etc)
Testers need tools to track the issues that they open and visibility into what is going on in the dev team to make sure that bugs are acted upon

As there is quite a lot of info already written on making rich bugs in VS2010, I will discuss a little below how you can track bugs and verify when fixes have been made.

Making the most of your bugs

In Test and Lab Manager, we have introduced an activity called “My Bugs” a dedicated manager for tracking the things that you care about and seeing to it that they work their way to completion. You can try it out in the Beta 1 bits. A screenshot of it is shown below:

My Bugs, quite simply lists out the bugs that you should care about. The activity is split up into three sections:

Assigned to me: These are bugs on your plate, waiting for you to act upon them
Created by me: These are bugs created by you which you should track over time to see to it that they are fixed.
Custom: This an area for you to define a custom query for the bugs which you care about

While their are 5 buttons available to you (New, Open, Create Test Case from Bug, Verify, and Create Copy), I think Create Test Case from Bug and Verify are both pretty cool so I will describe them below.

The Verify Button

Like many organizations, teams at Microsoft have a sign off process to determine when bugs are fixed. This ensures that both the person who fixed the bug and the person who created the bug agree when something is “fixed.”

When a bug is fixed, it transitions from state to state as multiple people “sign off” on the bug being fixed. In our team just one level of sign off is required: so bugs transition from Active (not fixed) –> Resolved (fixed) –> Closed (fix confirmed). We created the verify button to make it easy for you to transition bugs from the Fixed –> Confirmed state.

The diagram shows how the verify functionality works in action. (1) Clicking on the Verify button in the my bugs activity will load (2) Test Runner with the Test Case associated to the bug. (3) If you pass the test case and return to Test and Lab Manager, (4) you will be presented with a dialog to immediately take action on the test case, the transition to the next state will automatically occur for you. This dialog provides you with a quick way to act on the bug in question.

The “Create Test Case from Bug” Button

I am a big fan of combining exploratory testing and manual testing. I like the idea of leveraging exploratory tests to explore the application in an planned manner and using manual tests to verify scenarios which should just work.

I know that several test orgs ask their testers to exploratory test until they find an issue. Once they find an issue, they file a bug and continue exploratory testing. After completing their exploratory session, they ask their testers to create several scripted test cases covering the issue to make sure that they will catch the issue if it regresses later on. These new test cases are then scheduled to be run on a regular basis. The "Create Test Case from Bug” button was designed to support this workflow.

When a user take this action on a bug in the My bugs activity, we will bring up a test case pre-loaded with the text from the “Action log” (the log of actions captured about the application under test) contained in the bug. You can then add and remove steps from the test case and tweak it as you like to properly cover the steps which cause the bug. This effectively means that you can author test cases by having the tool create the steps of test cases for you!

Upon saving the test case, Test and Lab Manager automatically places the new test case in the suite where the exploratory test case was located. This ensures that it will be picked up and run by your testers.

I hope that these cool features will help you to make the most of your bugs! If you have any comments on how we can make them better, please do let me know!

The Evolution of the UI Design of Test and Lab Manager

2009-05-29T08:08:00Z

In the last post, I discussed how our team approaches designing user interfaces in an Agile manner. In this post, I will discuss how the design of Test and Lab Manager, aka codename Camano, has evolved as assumptions were tested and feedback was received.

Starting out

After doing extensive research on the testing market, we were fairly certain that in the next release, we wanted to create a product targeting Generalist Testers (70% of the testing market) who are uncomfortable with the amount of options available to them in Visual Studio. We were however fairly uncertain on the exactly which features to include in the product all the scenarios which we would have time to handle.

Being an Agile team who was open to change along the way, we decided to start designing and developing the product with the understanding that change would come along the way.

Iteration 1: Test Case Management inside Visual Studio

To get going on the project, while our other release (Visual Studio 2008) was winding down, we decided to create the basics of testing tools inside Visual Studio. Our thinking was that we wanted to get some tools around Test Case Management in the box so we could leverage the infrastructure which we had created in future sprints.

Below are a couple of shots of how we envisioned doing Test Case Management inside Visual Studio. The first shows managing configurations, the second shows authoring a test case (which is a work item). The mockups below were created by Moneta Ho.

Iteration 2: Creating codename Camano

As more research studies became available and we learned more about our target market's likes and dislikes, the team decided to create a standalone environment for generalist testers and to scrap the abilities to do TCM in Visual Studio. Our studies had told us that our target audience was not extremely technical and was a little overwhelmed by Visual Studio. We hoped that in doing so, testers would feel much more comfortable using our tools for their testing. The sprint of work that we did inside Visual Studio was not throw away work since the majority of it was merged into Camano later on.

UI Breakdown: Although, we were unsure which actual features would end up in the product, we envisioned that the majority of the tasks would fall in to 5 broad categories: Dashboard Information, Planning, Testing, Triaging Bugs and Reporting. To accommodate these groups, we chunked the UI into 5 groups: Home, Planning, Testing, Defects and Reports, with the perl on the left reserved for items which didn't seem to fit into any group.

Workflows: We wanted to make the product as simple to use as possible so we thought to guide the user through the workflow of various tasks. To this end, we modeled much of the UI after traditional tasked based UIs such as Microsoft Money, with the left portion the UI dedicated to guiding the user through various workflows via hyperlinks. Tasks which were specific to the activity displayed in the activity window would be displayed as "common activities," while tasks specific to whatever is selected in the page would appear under "selected activities."

Multi-tasking: As we felt that it would be quite common to work on multiple things in parallel, we introduced a work in progress area to the left hand pane. The area worked by like a stack in that when a user navigated away from an activity, the activity from which the user went was placed at the top of the stack.

Below are a few mock ups, created by Moneta Ho, of the initial design of Camano. The first shows the broad breakdown of the UI, the second shows the UI with a few artifacts.

Iteration 3: Simple Tweaks

Problems

We conducted a round of usability studies with local testers. We took away the following details from the study:

Many people did not understand where they were in the UI at any given point
Many didn't understand why the contents of the left hand pane was changing all the time or find it useful
Nobody used the work in progress area to switch between items, rather they preferred to just navigate to it directly

After observing users crying out in frustration in the usability study, we decided to make some tweaks to the Camano shell.

Solution

Since most users did not correlate between selecting an item in the grid and then selecting an action to be performed on it far away, we moved all actions that could be performed on items in a grid to a toolbar directly above the item. Further, we created a toolbar to contain items for the page displayed in the Activities pane. Together, this allowed us to get rid of the selected item activities and paved the way to getting rid of the left hand pane entirely later on.

As we iterated, we found that we really did not need the Perl in the top left hand corner in the UI. We therefore removed it from the product as well.

Below is a screenshot of the tweaks that were made, one with a blue theme, one with a dark theme. Both themes were in implemented in the the product at various times.

Iteration 4: Major Overhaul

Problems

As more of Camano began to take shape and more users internally began to use it, we increasingly received feedback from almost all of our channels including TAP, MVPs, the SIG and internal users that the product was not easy to use. In our effort to make something that would allow people to plan all the steps of their testing, we made something which exposed too many artifacts in the UI at one time, did not guide a user through any particular workflow and to some extent got in the way of a user doing their testing. Although we initially designed band aid solutions such as activities dedicated to streamlining the out of the box experience via a wizard, filtering a tester’s view by various factors, and workflow diagrams built into the page, we felt we needed to do something more dramatic.

Further, it was felt that our current UI:

Had too much space (around 40%) dedicated to navigation. Much of the navigation was redundant with 2 or 3 areas dedicated to doing the same thing.
Our history metaphor was not working. Our work in progress area quickly filled with items which the user was not interested in navigating back to.
Did not make it easy to perform simple tasks. 2 or 3 steps were needed to just perform simple things without the tool guiding the user through the process.

In a typical tester’s day, we began to understand that:

Most testers would work within the context of a Single Test Plan. Examining items that span multiple plans would be a 20% scenario.
The majority of their time would be spent in 3 types of activities: Authoring Tests, Running Tests, and Examining Bugs

Solution

Object Model: After much discussion, between both server and client teams, it was realized that the object model on which Camano was based could be dramatically simplified if we always forced the user to plan their test cases for execution at the plan level rather than as stand-alone suites which could be used from plan to plan.

Some of the initial sketches of this idea by Ed Glas and the server team were as follows:

User Interface: To assist with the issues mentioned above, we decided to make several changes to the UI. Specifically, we decided to optimize the UI on common workflows, make it easier to access the most common activities, eliminate the left hand pane, change the history metaphor, add back and forward buttons to the UI and to force the user to work within a single plan.

By doing so, I think that we were able to naturally filter the perspective of the tester to the items they are interested in (their currently connected plan), and streamline workflows. With these changes made, workflows become much more natural since objects are always created in the context of a plan.

Below is a mock up of design, by Nigel Wolters, that shows the changes, followed by a screenshot of what was actually implemented. Note that the amount of space dedicated to navigation was minimized and that when the user is in the “Tester’s Center,” the user’s perspective is filtered to the Test Plan to which they are currently connected. Further, access to the most common areas (to author or run tests or look at bugs), are just one or two clicks away at any given point.

Iteration 5: Simple Tweaks

After making the major overhaul to the object model and accompanying changes in the UI, we started to get raving reviews. Folks were particularly happy to have the full horizontal screen for their activity content and that they were able to get testing quickly out of the box. All of our feedback channels started telling us that the product felt much more polished and easy to use.

As folks in Visual Studio started to look at it, it was pointed out that the colors were not in sync with the new blue Visual Studio theme in VS2010. We decided to do a simple update to make Camano blue in addition to making it a little sexier. I’m quite happy with how it ended up.

Below is a mockup created by David Culberton in conjunction with Nigel Wolters. It is very close to how the Beta 1 product looks and feels.

Conclusion

As you can see, in the run up to the Beta 1 release of codename Camano, the product has been completely redesigned several times. My hat goes off to the Agile process and management which has allowed us to respond to various levels of feedback to make design changes when appropriate.

I should note how much of a team effort designing Camano has been. Each design decision has been thoroughly scrutinized to make sure that the right trade-offs are considered and the right optimizations are made. Nigel Wolters, David Culbertson, Moneta Ho, Mark Mydland, David R. Williamson, Ed Glas, Michael Rigler, Dominic Hopton, Euan Garden and myself (Naysawn Naderi) have have all been thoroughly involved in making Camano as easy on the eye as we think it is easy to use.

Along these lines, we are also incredibly indebted to our MVPs, SIG representatives and many other unsung feedback heros who have been invaluable in praising us when appropriate and holding us out on the ringer when we deserve to as well. My thanks goes out to you for helping to shape the design of codename Camano.

How We Approach Agile Design

2009-05-08T21:28:47Z

For approximately the past year and a half, I have been working on building the next generation of testing tools. Our standalone UI, Microsoft Test and Lab Manager, was built to allow testers to plan their testing and overview their results. We have built the UI in an Agile manner and followed Agile processes. In this post, I'll go over some of the practices that we have been following to plan and design in an agile manner. In the next post, I'll go over an example of this by looking at how the UI for Camano has evolved in the past year and a half.

Breaking down the Planning

The Camano team follows an Agile Development process with Sprints of 5 weeks. I'm sure most of you are familiar with Agile. One of its premises is that requirements and priorities are constantly changing and that software is best developed by breaking work into bite size chunks which can be coded, tested and completed in a single Sprint. The idea is to incrementally add the most pressing features to the product in a manner where the overall quality of the product is maintained.

To accommodate planning in this manner, our team maintains a priority sorted product backlog in a Team Foundation Server (TFS) database. Each backlog item has a corresponding work item that contains a field for its relative priority (RankInt) to other items on the backlog. We leverage the integration that TFS has with Excel, to view the list of all the backlog work items in Excel so that we can move the work items around in the list and change their relative rankings. We are constantly adding more items to the backlog and shifting around their relative priorities.

This is a screenshot of how the Camano backlog looked at one point in time. Each row in excel corresponds to a TFS work item

About 4 weeks before a Sprint begins, we flag the items at the top of the backlog to be planned and developed in the next Sprint. Us Program Managers, take it as our goal to fully spec out the features for the Sprints so that developers have all unanswered questions hashed out by the time they need to plan out its implementation. We don't always hit our mark of identifying which features to develop in time or manage to close open issues by design week, but we our best to achieve the goal.

Rapid Design to Accommodate Agile Development

PMs and designers typically have about 5 weeks to grind out a design for a feature to be implemented in Camano. Although this may sound like a while, striving to go from nothing to something which all the important stake holders agree upon can be a challenging task.

Our team has attempted to create rapid designs by having Program Manger spearhead a design who presents to higher ups the design at regular intervals. When there are 4 weeks to develop a design, it typically looks like this:

Prior to Week 1: Management and PMs decide on the components to add to the product in the next Sprint.

Week 1: PM chats with a few passionate people about a new and cool feature, develops a low fidelity wireframe of the experience & discusses the experience with higher ups at the end of the week

Week 2: PM iterates on feedback from the discussion & works with a designer to create a high fidelity mock ups of their features and presents it again to the higher ups for feedback

Week 3: PM iterates on feedback and does another review with higher ups and engineering leads to see to it that they are content with the final solution. The PM also typically will run a design by other effected teams to be sure that dependencies are flagged properly.

Week 4: PM presents the design to the engineering team. Engineering team has feedback which is jotted down to make design trade-offs.

Week 5: PM updates the design from feedback from the engineering team.

End of Week 5: The Sprint begins and the designs begin to be implemented by the engineering team. The PM team starts to design the features for the next Sprint.

Feedback along the way

Since we are developing in an Agile manner, we are able to demonstrate functionality to customers along the way and get feedback. Although our product is being released as a part of Visual Studio 2010, we have shown the completed portions to customers in pre-betas, called Customer Technology Previews (CTPs), at regular milestones.

In addition to getting input via customer use of CTPs, we use a variety of other mechanisms to get feedback from customer as the product is developed.

Hands on Labs: With each CTP release, we have adopted the practice of getting both internal and external testers to run through a scripted scenario. Although it is painful to set up machines for all parties to run through, it is a good means to hear feedback from first time users of the product.

Focus Groups: We have 2 groups of customers, one made up of local software testers, one of remote testers. We routinely discuss priorities and features before they are coded and ask them to find gaps in our story.

Usability Studies: We routinely run usability studies where we bring in local testers and observe their efforts to perform various tasks on the developed product. I've found it quite eye opening to see users struggling to perform tasks with a product that you've designed. It is also quite motivating to fix an issue when the whole engineering team observes a user yelling at the screen!

Customer Chats: Its nice to work on a team where people care about the product that you are developing. The Camano team has been fortunate to be able to have many customers eager to provide feedback via the Microsoft TAP program. Via this program, most members on the team are assigned one member at an interested company to speak with about the product in development on a regular basis. Through these informal chats, we strive to keep everyone on the development team focused on solving real world problems.

Responding to Feedback

As we get feedback on implemented code, we do our best to respond to this feedback by making design changes and tweaks in subsequent sprints. This can be as simple as adding another button or changing a color to overhauling an entire design and dropping functionality to go in another direction.

While it is challenging to plan in an iterative manner, I find it all worthwhile as through its use we can show customers a design, find out it needs more work, and make appropriate changes before we ship. In the next post, I'll go over how designs have changed as we have progressed by looking at how the UI of Camano has evolved as we have gotten feedback.

My moment on Channel 9

2009-01-22T04:13:28Z

A little while ago, I did an interview with Brian Keller and the Channel 9 team to explain some of the Manual Testing experiences that were brought to light in the September CTP of Visual Studio 2010. If you haven't had a chance to see video, I would recommend looking it over. In it you'll see that we have created a new UI, Camano, optimized for the generalist tester.

James Whittaker, around the same time, did a channel 9 interview as well. In it he explains some of the steps that we are taking to make software testers first class citizens in VS2010.

What Would Alan Cooper Do?

2009-01-16T04:41:00Z

I have been recently thinking about designing simple yet powerful UI. I recently sat down one weekend to take notes from Alan Cooper, one of the thought leaders in interface design, from his book About Face. Below are some of the points from the book that I found particularly interesting that I have been trying to keep in mind as we have been designing the Camano and Manual Testing UIs. In future postings, I'll discuss how we have applied these design principles to recent versions of the Test product.

Design for intermediates users

Cooper argues that the vast majority of users of any software product can be categorized as intermediate users - users who basically know how to use the product and generally use it for doing the same thing all the time. He estimates that such users are around 80% of the users who use the product, with the remaining 20% equally distributed between advance and beginner users.

However, he interestingly points out that although they are the majority of the users of the product, often their needs are often neglected in the design. He states that management tend to advocate most for the needs of beginners since beginners are the users that they tend to most interact with, while developers are by definition expert users of the product and tend to think of end users as experts too. Cooper feels that intermediates tend to get lost in the mix, even though they are the majority of the users.

He explains that the needs of the 3 types of users differ due to their familiarity with product. While experts want quick ways to do things and the ability to customize the app, beginners are most concerned with overviewing the basics of the tool. Intermediates tend to want to do the same things with the tool repeatedly and do not want to have their work processes obstructed.

Use tools that help beginners to become intermediates

While only 10% of users at any point in the project are beginners, all users of the product must pass through the state of being a beginner user. Special attention should therefore be paid to designing the interface to help new users to learn the basics of the application.

Although migrating users to intermediates is a goal of the UI design, Cooper is quick to note that once a user has moved from being a beginner user to an intermediate user, the UI that helped them to achieve this task will be of little use to the user. In fact, it will likely obstruct the user's flow.

There are numerous products that try to educate beginner users but at the same time really annoy intermediate users of the product. In VS2008 for example, we inserted comments into generated unit tests help beginner users to understand the various components of a unit test. While they serve to help users understand the components of the unit test, once they understand the components, they get in the user's way.

To help users become intermediates, he suggests adding something to the UI that provides overview information to the user on the use of the product. He suggests that it should be possible to remove this information from the UI when the user wants to turn it off. This classically takes the form of a "getting started" video which overviews the features of the product.

Cooper thinks that online help and tooltips actually best serve intermediate users rather than beginners since their needs are slightly different. He suggests that intermediates search for reference educational materials, while beginners tend to be most interested in material that overviews the materials.

Less is more

I love this quote: "No matter how cool your interface is, less of it would be better."

Sometimes in UI design, I think that we forget that users use a product as a tool to accomplish a specific goal. Sometimes we get carried away with cool navigation tools such that we put too much of it in the UI. A purple hammer which glows in the dark is useless if it can't effectively put nails into wood.

Cooper is an advocate of minimalist design where every options should be purposeful and direct. He advocates for fewer more direct options in an interface. He states that in a well orchestrated UI design, the UI becomes almost transparent to the user since it naturally follows their mental model.

Design for the probable, provide for the possible

Cooper is an advocate for following the user's mental model when designing the product and optimizing for the mainline flow through the application. While this often means that one should eliminate more options from a dialog than a programmer may feel comfortable doing, it does not imply that there should not be ways to do more complex and sophisticated things, just that they should be more difficult to access.

If one were to outline the types of activities that a user can perform at any one stage in an application workflow, Cooper feels that there will always be one or two items that are the most likely to be performed. Thus, the interface should be optimized to help the user find these activities and guide them through this workflow rather than just listing all options at all times. There should be a way for the user to access non-mainstream activities, but the UI should not be optimized for its use.

I think the office ribbon did a fantastic job of optimizing for the probable but allowing for the possible. In the ribbon, the items that are likeliest for the user to want to use are 1 click away, the items that they will use less are 2 clicks away, the items that very unlikely to be used are 3+ clicks away. They clearly optimized their UI for the probable.

Eliminate Error and Confirmations Dialogs

Keeping with the theme of following a user's mental model and removing obstacles that can obstruct intermediate user's flow, Cooper advocates for removing all error and confirmation dialogs from a product. He feels that applications should do what statistically has a good chance of being correct but provide users with an option to undo their actions.

Cooper states that users like to see their options available to them, perform an action, and then get confirmation that the action had successfully occurred. Without confirmation than an action occurred, users wonder if their actions actually took place. Rather than provide pop-up notifications, in order to communicate the result of actions, Cooper favors inline status alerts and positive auditory feedback.

While I agree that it would be ideal to get rid of confirmation dialogs in all circumstances, getting rid of them entirely feels too drastic for my taste. To implement what Cooper suggests, developers need to make each action that the user performs reversible, which is both costly and in my opinion, not needed. I would prefer to make the best effort to eliminate confirmation dialogs and make actions reversible, but in the case of drastic actions, like disk format and file delete, popping a confirmation dialog seems to be fine, in fact users may expect it.

On the other hand, I tend to agree with him that error dialogs should be entirely removed from the product. Although they make sense to a programmer, I think they tend to make little sense to the end user. I've seen many users getting very frustrated because their workflow is hampered when they appear. Cooper notes that while they are used to signal that something went wrong with the code, the user tends to interpret them as “I’ve done something wrong.” When users are told that they are wrong repeatedly, they start to hate your product. I've been playing with some of the Mac UI recently and have been surprised to see they seem to be constantly eating their exceptions. Sure, problems occur - they just don't surface to the user.

I got a lot of value out of reading the About Face and would highly recommend it. In the next post, I will discuss how we have iterated on Camano and the Test Runner UI in an Agile development process to produce the design that we have today.

ooh Camano… ooh planning testing in the November CTP

2007-12-15T04:28:00Z

In several of the organizations that we've visited we've found some that some of the more experienced ones tend to put much thought into planning their testing efforts. One of the organizations that we visited, followed an agile testing process with sprints of four weeks in duration as follows:

Before the sprint, the test manager created a test plan for the release with her team. The test plan served to identify the area to be tested in the sprint. For the next two weeks, the testers wrote test cases in excel to specify the steps to verify the new functionality. Each row in the excel file consisted of a test case with columns used to specify its steps and record the results of its execution. In the closing weeks of the sprint, the testers executed all the test cases that they have authored in the prior weeks twice - once in week 3, once in week 4.

While the very organized effort above is bound to locate most of the holes in the product, the tools that the organization used to plan and manage the testing effort were not helping them with the process. The programs provided them with no easy way to organize their test cases, track the requirements associated to them, quickly view test case results, plan a testing effort or view the overall progress of their testing.

In the November CTP release of Rosario we introduce Codename 'Camano' - a tool designed to help testers and test managers to plan, organize and analyze a testing effort. In Camano, each test case is a database object that can be executed by Microsoft Test Pilot. Testers can organize their test cases into "Suites" that can by either "Dynamic" (query defined) or "Static" (list defined). A lead can schedule a suite of tests to be executed by placing the suite in a new test plan. When creating a test plan, the lead can also divide up the testing effort by assigning different members of their org to execute specific test cases. Once a plan is created, the testers execute test cases out of it.

Once a test plan is created, managers will be able to track its progress, and add and remove test cases to the plan as new test cases are authored and the plans change. Management will also be able to generate reports against the execution of test cases contained in the plan. We are hoping that users will find it very useful to be able to quickly generate reports on the number of test cases currently failing, the number of blocked test cases, the testers that are finding the most bugs, the speed at which bugs are being fixed and the number of testers petitioning to put Arrested Development back on the air :). Test Managers at Microsoft love getting such statistics, we have been planning that those outside of Microsoft will find them quite handy as well (we have 3.5 testers on our team currently currently petetioning).

Have any thoughts on our thinking around planning a testing effort? Are we completely off wack? A little off wack? I would be very interested in your feedback - as we really want to get the planning experience right.

Cheers,

- Naysawn

Who are Generalist Testers?

2007-11-07T04:22:00Z

I have been recently focusing on a project to improve the lives of generalist testers for our next release ‘Rosario.’ In order to improve their lives, I’ve been trying to understand them better, their methods for testing software and what like to eat for lunch. Here’s what I’ve learned so far:

Background: Testers have a different background than developers. While many developers tend to love coding and usually have a technical degree, most manual testers do not have a technical background and do not code or have any desire to do so in the future.

Testing Philosophy: Most testers tend to approach testing differently from developers. While developers like to test their code by unit testing it in small fragments, testers tend to prefer to test end to end scenarios. While developers are comfortable writing code to perform their automation, testers seem to shy away from automation tools, preferring to manually step through written test cases. While testers tend to search for bugs by striving to take the unhappy path through the software, while developers often take the “yellow brick road” when ensuring that their code meets the requirements.

Focus on Quality: Testers seem to be motivated by a desire of creating a quality product. They love to break things, are delighted to find bugs in the product that they are testing, and to hound developers to see to it that they fixed. I find that this differs from most developers who hunger to implement new functionality and who view bug fixing as a chore.

Good Representatives of the Customer: On many software development teams, testers understand the ins and outs of the product in development better than anyone else in the company. They are therefore a very good source of information on the product and of future customer requests.

Treated as Second Class Citizens: We’ve also observed that testers are often treated as second class citizens of the software development process. In some organizations, they are given the worst workspaces in the building (one tester was placed right next to the company foosball table), while in others they are merely not trusted to test the right components in each sprint.

Similarly, they are often not given the best of tools to manage the testing effort. While it seems that every week a new tool is created to allow developers to create a flashier UI and collaborate more easily, manual testers are forced to work with a much more limitted toolset. Many testers author test cases in excel, step through test cases while constantly juggling which window is displayed and struggle to create bugs that provide developers with a clear picture of what needs to be fixed. We are therefore hoping to make a tool that will significantly improve the life of testers and help them to feel like first class citizens. More details on our plans shortly...

New Unit Testing Features in Orcas (Part 2)

2007-05-14T03:07:00Z

This is a part of a series of postings on the details of some of the new features appearing in Orcas. All these features will be available in Orcas beta 2.

Gaining Access to Private Code under test:

In Whidbey, we introduced the concept of private accessors to provide users with an easy way to access private members of code under test. Private Accossors served as a wrapper for reflection, consisting of shadow implementations of the methods of a class which provided access to a private field or method through a reflection call - hiding all the nitty gritties of reflection from the user. Although, it seems that users love the concept, we decided to rewrite the feature to make it more robust in Orcas.

In Orcas, we decided to change private accessors from being code based to assembly based. The assembly based assessors are more robust and are picked up by intelliscene and auto-regeneration when the code under test has changed. As in Whidbey, Orcas private accessors are automatically generated whenever generating unit tests for private methods, they can be manually generated by right clicking inside the code editor and selecting Create Private Accessor. Generating private accessors can also be achieved via the command line via running publicize.exe (new in Orcas). I have the impression that the majority of users are still unaware of this cool feature, so I hope more will play with it in Orcas Beta 2 and RTM.

Generating Unit Tests:

For those new to unit testing, VSTS contains a nice feature to generate an outline of a unit test for a given method of code under test. In Orcas, we have re-written the generation engine to be more robust, to work more intelligently with the code under test and to functional with the new assembly based private accessors. We have also beefed it up to better support .Net languages, specifically .Net reflection.

Support for Device Projects:

In Orcas, we introduce support for unit testing on embedded devices. All the functionality described, in these posts, and that which was present in Whidbey will be accessible to device programmers.

Automatic Cleanup of Test Results and Their Associated Deployments

One of the pain points we often hear from customers concerns deployments inadvertently filling their hard drive as they execute more and more tests. To assist these users, we have designed Visual Studio Orcas to automatically remove old deployments when more than 25 test results have filled the Test Project’s Test Results directory. One can change the number of old deployments maintained on disk in Tools->Options->Test Tools->Test Execution.

New Unit Testing Features in Orcas (Part 1)

2007-05-12T01:03:00Z

This is a part of a series of postings on the new unit testing features appearing in Orcas. All these features will be available in Orcas beta 2.

I know that MSTest isn’t yet the darling of the TDD world, but I’d like to think that our TDD story has improved a decent amount in Orcas. These are some of the improvements:

1 – Better Execution Times: We have rewritten our test execution engine to significantly improve the time that it takes to execute unit tests from the command prompt and via the IDE.

2 – Run Tests: We have added a menu item to execute the tests in the current context of the test class’ code. Executing “Run Tests” from a right click context menu will execute the best guess of the tests we think you would like to execute. Therefore if the focus is on a test method, Run tests will execute the test method, if the focus is in a test class, Run Tests will execute all the tests in the test class, if the focus is in the test results window, Run Tests will execute the tests which are checked, etc. Ctl R, T also accesses this command. I think you guys will love it.

3 – Short Cut Keys to Run Tests: We have added in a lot of short cut keys to kick off a test quickly. They are as follows:

· Ctl R, T: Run Tests in Current context (see #2)

· Ctl R, C: Run Tests in Current Test Class

· Ctl R, N: Run Tests in Current Namespace

· Ctl R, S: Run All Tests in Solution

· Ctl R, D: Run the Tests in the Last Test Run

· Ctl R, F: Run the Failed Tests of the Last Test Run

Holding Ctl while pressing the second key in the key combo executes the tests with the debugger. We hope that these additions will significantly improve the TDD flow.

4 – Disable Deployment: As many of you are aware, MSTest currently “deploys” tests for every test run it executes. This means that in addition to generating a Test Results (trx) file, it also copies all the test binaries, code coverage files and debug information to a deployment folder, where the tests are executed. This design provides for convenient features for collaboration, specifically publishing old test results and their deployments, allowing another team member to re-run the tests and step through a test run. It is however, inconvenient for TDD users, since most TDD’ers want their tests to execute as fast as possible. We have therefore added an option in the Test Run Configuration to disable deployment. To do so, go to the Test Run Configuration file in a Test Project (double click on *.testrunconfig in solution explorer), Deployment, and uncheck “Enable deployment.”

5 – Inheritance of Tests: In Whidbey, we did not support the inheritance of tests methods or initializations. We have added support for inheritance in Orcas. I’m planning to use a future post to dive in to the nitty gritties of Test Inheritance.

6 – Directly go to the Point of Failure: We have enabled an option in Orcas to directly jump to the point of failure. We have made the stack trace in the Test Results window clickable as well. To specify that double clicking on a failed test result takes you to the first element in the stack trace, go to Tools->Options->Test Tools -> Test Execution and check the last box in the window.

Props to Alan Ridlehoover, Peter Provost and Peli de Halleux for their help in trying to get the TDD experience right. J

Test Lists Going Pro

2007-04-24T03:38:00Z

Judging from your feedback, it seems that the majority of you feel that the authoring and execution of Test Lists is an integral component of the unit testing experience. In short, we agree with you and have therefore decided to trickle Test Lists down to the Pro Sku in Orcas as well. The changes will be present in the Beta 2 release. Cheers!

Unit Testing Trickling into Pro!

2007-03-27T20:40:00Z

Due to popular demand we have decided to add the majority of the unit testing features of Team System to the Pro Sku of Visual Studio. With the release of Orcas, the support for authoring, generating and running unit tests, will become available to the users of the Pro Sku of Visual Studio. Pro customers will also take advantage of the some of the unit testing improvements we have added into Orcas, specifically generating for generics, performance improvements, the ability to unit test devices and better IDE integration (I’ll try my best to blog on the details soon). We are in general very open to the concept of trickling down other functionality introduced in Team System into other Skus over time, so please let us know if you feel that other items should trickle down as well. Keeping this pattern keeps us on our toes to ensure that we are always adding high value features higher up the stack. We love hearing your feedback and take your suggestions very seriously (I’m not just saying this - I have been continually surprised at how much time is spent on user’s feedback).

Again, we are very excited about the trickle down as we hope that it will introduce the concept of unit testing to the average .Net developer. Our team hopes that every developer will see the major benefits of unit testing and will regularly author and execute the tests throughout the product life cycle.

To the beta users: you may notice that a few pieces of the unit testing puzzle is missing from the Pro Versions of Beta 1 – specifically, authoring test lists, remotely executing tests and generating code coverage results. We have been debating if some of these features should also trickle down and would be very interested in your feedback. For example, the ability to author test lists has been excluded from the Pro Sku since many felt that its chief benefit comes to those which author test lists and run them as a part of a Build Verification Test prior to checking code into Team Foundation Server. Some, however feel that it is still convenient to organize tests in a list regardless of check-ins. How do you feel? Does the test list editor (formerly called Test Manager) belong in Pro?

Happy Unit Testing,

Naysawn

That Pesky MSTest Execution Ordering..

2007-02-17T04:41:00Z

I have come upon several groups who are puzzled by the intricacies of execution of the MSTest Framework. I admit - it is quite confusing, so I hope that these postings will help to clean some of the confusion.

I think that most confusion comes from some user’s expectation of MSTest to execute like the Nunit framework. They execute differently since Nunit instantiates a test class only once when executing all the tests contained in it, whereas MSTest instantiates each test method’s class separately during the execution process, with each instantiation occurring on a separate thread. This design affects 3 specific things which often confuse users of MSTest:

1. ClassInitialize and ClassCleanup: Since ClassInitialize and ClassCleanUp are static, they are only executed once even though several instances of a test class can be created by MSTest. ClassInitialize executes in the instance of the test class corresponding to the first test method in the test class. Similarly, MSTest executes ClassCleanUp in the instance of the test class corresponding to the last test method in the test class.

2. Execution Interleaving: Since each instance of the test class is instantiated separately on a different thread, there are no guarantees regarding the order of execution of unit tests in a single class, or across classes. The execution of tests may be interleaved across classes, and potentially even assemblies, depending on how you chose to execute your tests. The key thing here is – all tests could be executed in any order, it is totally undefined.

3. TextContext Instances: TestContexts are different for each test method, with no sharing between test methods.

For example, if we have a Test Class:

[TestClass]

public class VSTSClass1

{

private TestContext testContextInstance;

public TestContext TestContext

{

get

{

return testContextInstance;

}

set

{

testContextInstance = value;

}

[ClassInitialize]

public static void ClassSetup(TestContext a)

{

Console.WriteLine("Class Setup");

}

[TestInitialize]

public void TestInit()

{

Console.WriteLine("Test Init");

}

[TestMethod]

public void Test1()

{

Console.WriteLine("Test1");

}

[TestMethod]

public void Test2()

{

Console.WriteLine("Test2");

}

[TestMethod]

public void Test3()

{

Console.WriteLine("Test3");

}

[TestCleanup]

public void TestCleanUp()

{

Console.WriteLine("TestCleanUp");

}

[ClassCleanup]

public static void ClassCleanUp()

{

Console.WriteLine("ClassCleanUp");

}

(This consists of 3 Test Methods, ClassInitialize, ClassCleanup, TestInitialize, TestCleanUp and anexplicit declaration of TestContext)

The execution order would be as follows:

Test1 [Thread 1]: new TestContext -> ClassInitialize -> TestInitialize -> TestMethod1 -> TestCleanUp

Test2 [Thread 2]: new TestContext -> TestInitialize -> TestMethod2 -> TestCleanUp

Test3 [Thread 3]: new TestContext -> TestInitialize -> TestMethod2 -> TestCleanUp -> ClassCleanUp

The output after running all the tests in the class would be:

Class Setup

Test Init

Test1

TestCleanUp

Test Init

Test2

TestCleanUp

Test Init

Test3

TestCleanUp

ClassCleanUp

(A special thanks to Dominic Hopton for his edits of my hackery on this post :))

Comparing the MSTest and Nunit Frameworks

2007-02-02T00:40:00Z

I haven't seen much information online comparing the similarities and differences between the Nunit and MSTest Frameworks. Here I will define the similarities and some of the differences. If there is anything else which you come upon, please do add it to the comments.

MSTest Attribute	NUnit Attribute	Purpose
[TestMethod]	[Test]	Indentifies of an individual unit test
[TestClass]	[TestFixture]	Identifies of a group of unit tests, all Tests, and Initializations/Clean Ups must appear after this declaration
[ClassInitialize]	[TestFixtureSetUp]	Identifies a method which should be called a single time prior to executing any test in the Test Class/Test Fixture
[ClassCleanup]	[TestFixtureTearDown]	Identifies a method in to be called a single time following the execution of the last test in a TestClass/TestFixture
[TestInitialize]	[SetUp]	Identifies a method to be executed each time before a TestMethod/Test is executed
[TestCleanUp]	[TearDown]	Identifies a method to be executed each time after a TestMethod/Test has executed
[AssemblyInitialize]	N/A	Identifies a method to be called a single time upon before running any tests in a Test Assembly
[AssemblyCleanUp]	N/A	Identifies a method to be called a single time upon after running all tests in a Test Assembly

The order of execution is similar in both frameworks, but there are some differences between the two:

In Nunit, tests are not executed in parallel. Rather, it appears that all tests execute on a single thread. In MSTest, each test is instantiated on a separate thread, this results the runs being interleaved. Therefore, if test A depends on test B for its success, it likely will fail as test B will likely start running as test A is running.
The time at which ClassCleanUp/TestFixtureTearDown executes is different between the two frameworks. In Nunit, TestFixtureTearDown is executed immediately following the completion of the last test in a TestFixture or after TearDown if the attribute exists for the test in question. In the Whidbey implementation of MsTest, ClassCleanUp executes at the end of a test run, before AssemblyCleanUp but not necessarily immediately after the last test in a TestClass has completed executing.
In Whidbey, support for test class inheritance was missing. In Nunit, it is fully supported. This will be rectified in Orcas.

I should also mentioned that in MsTest, TestContext exists for passing information about the test run. There is no equivalent in Nunit tests. This can serve as a handy tool for pulling information from datasources on the disk to the unit tests, as well as other uses. More can be read about it here.