Making the most of your bugs

I like to think of bugs as glue which links software teams together. Developers write the code, testers test it, and bugs serve as the gooey battle ground where folks try to make sure that the right thing is done for the customer.

While the ideal flow of bugs probably should look something like this:

Tester: finds an issue –> creates a bug –> assigns it to a developer
Developer: fixes the bug –> assigns it back to the tester to be verified
Tester: verifies the fix –> closes the bug.
Everyone sings Kumbaya

If your development process look anything like it does at Microsoft, where there is lots of ambiguity around what the “right” thing to do with a bug is and there exists differing opinions on what a “fixed” bug looks like, your process may look a little more like this:

Tester: files a bug with 1 line of information (“x is broken!”) and assigns it to a Developer
Developer: looks at issue tries to repro it, finds that x isn’t broken on their box, marks the bug as “no repro” and assigns back to Tester to be closed
Tester: finds the bug still exists on their box, complains to the developer who agrees that this indeed is a bug, so they expand on the 1 line in the bug and assign the bug back to the developer
Developer: fixes 1 component of “x being broken” but x is still broken, Assigns back to the Tester stating that the bug is “fixed”
Tester: tries the bug finds that it is slightly better, re-activates the bug noting out the additional failures.
Developer: Fixes the additional components, marks the bug as fixed, assigns to tester for confirmation
Tester: Confirms the fix by going over the issues that originally caused the issue and closes the bug

While not all bugs go through this much back and forth, with mis-understandings around what constitutes a bug and what should be included in a “fix,” I think that it is understandable for there to exist some back and forth between the team. In fact, it is probably healthy to for some friction to exist.

Tools for a Fair Fight

While I think it is healthy for bug disagreements to occur, folks need the tools to fight a fair fight:

Developers need bugs which are super “rich” and provide as much information as possible about the defect under discussion (with screenshots, video, actions about the application under test, etc)
Testers need tools to track the issues that they open and visibility into what is going on in the dev team to make sure that bugs are acted upon

As there is quite a lot of info already written on making rich bugs in VS2010, I will discuss a little below how you can track bugs and verify when fixes have been made.

In Test and Lab Manager, we have introduced an activity called “My Bugs” a dedicated manager for tracking the things that you care about and seeing to it that they work their way to completion. You can try it out in the Beta 1 bits. A screenshot of it is shown below:

My Bugs, quite simply lists out the bugs that you should care about. The activity is split up into three sections:

Assigned to me: These are bugs on your plate, waiting for you to act upon them
Created by me: These are bugs created by you which you should track over time to see to it that they are fixed.
Custom: This an area for you to define a custom query for the bugs which you care about

While their are 5 buttons available to you (New, Open, Create Test Case from Bug, Verify, and Create Copy), I think Create Test Case from Bug and Verify are both pretty cool so I will describe them below.

The Verify Button

Like many organizations, teams at Microsoft have a sign off process to determine when bugs are fixed. This ensures that both the person who fixed the bug and the person who created the bug agree when something is “fixed.”

When a bug is fixed, it transitions from state to state as multiple people “sign off” on the bug being fixed. In our team just one level of sign off is required: so bugs transition from Active (not fixed) –> Resolved (fixed) –> Closed (fix confirmed). We created the verify button to make it easy for you to transition bugs from the Fixed –> Confirmed state.

The diagram shows how the verify functionality works in action. (1) Clicking on the Verify button in the my bugs activity will load (2) Test Runner with the Test Case associated to the bug. (3) If you pass the test case and return to Test and Lab Manager, (4) you will be presented with a dialog to immediately take action on the test case, the transition to the next state will automatically occur for you. This dialog provides you with a quick way to act on the bug in question.

The “Create Test Case from Bug” Button

I am a big fan of combining exploratory testing and manual testing. I like the idea of leveraging exploratory tests to explore the application in an planned manner and using manual tests to verify scenarios which should just work.

I know that several test orgs ask their testers to exploratory test until they find an issue. Once they find an issue, they file a bug and continue exploratory testing. After completing their exploratory session, they ask their testers to create several scripted test cases covering the issue to make sure that they will catch the issue if it regresses later on. These new test cases are then scheduled to be run on a regular basis. The "Create Test Case from Bug” button was designed to support this workflow.

When a user take this action on a bug in the My bugs activity, we will bring up a test case pre-loaded with the text from the “Action log” (the log of actions captured about the application under test) contained in the bug. You can then add and remove steps from the test case and tweak it as you like to properly cover the steps which cause the bug. This effectively means that you can author test cases by having the tool create the steps of test cases for you!

Upon saving the test case, Test and Lab Manager automatically places the new test case in the suite where the exploratory test case was located. This ensures that it will be picked up and run by your testers.

I hope that these cool features will help you to make the most of your bugs! If you have any comments on how we can make them better, please do let me know!

Thanks - Naysawn Naderi