Welcome to MSDN Blogs Sign in | Join | Help

Mining Work Items for Opportunities to Improve Your Engineering Process

One of the things I spend a lot of time doing as a test manager is mining for interesting data in our work item tracking system (TFS of course). 

For example, this morning, I went through a query of bugs that had been resolved as “won’t fix” but were also marked as regressions from previous releases.  Most of these were correctly resolved because we’ve rewritten some features and the new features have different/improved functionality.  But a few of them had truly worked in past releases and were now broken and for whatever reason our feature teams had decided not to fix them.  Because we had added the “regression” field to our bug form and because people had correctly filled out these fields, we can easily identify such issues and dig into whether or not these decisions were correct.

Out of the box, the bug forms in the Agile and CMMI processes are fairly sparse (at least compared to our internal bug form).  If they don’t have fields to track the information you need, you’ll have to customize the forms.  Learn how to customize WIT forms here (TFS 2005, TFS 2008, TFS 2010 Beta)

7 Tips to Help Your Data Mining

Of course, there’s a fine balance to aim for between overloading your bug form with tons of fields and having a bug form that’s simple enough as to not discourage people from actually filling out all the fields properly.  Here are some tips I’ve learned to help improve your chances of success:

  1. Work with your project stakeholders to decide which engineering processes you want to improve
  2. Prioritize the list
  3. Determine which bug form fields will give you the necessary insight into how those processes are working today
  4. Determine what length of time you’ll need to look for patterns.  Don’t assume a few data points means a trend.  Consider observing data during different phases of your development cycle (e.g. feature development vs. stabilization) and across phases.
  5. Do regular spot-checks to make sure the data your team is supplying is valid.  i.e. – are they taking the time to fill in the fields properly?  are they skipping the fields altogether?  check in with your team to get their feedback on the bug form.
  6. Don’t make the bug form part of the process problem by overloading it with too many fields.
  7. For critical process questions, mark the associated fields as required on the bug form.
  8. Understand the normal ranges for each of your metrics and drill down to understand anything that varies from the norm.  You could find areas where some teams are working much more efficiently than others and find opportunities to spread those practices to other teams.  Alternatively, you could also find areas in need of improvement.

Data Mining Examples

Here are some other interesting queries I’ve run recently, the fields I use to gather data, and some follow-up questions I ask for each.

Example 1 – What is the ratio of bugs in automated tests due to product changes vs. bugs due to poorly coded tests?

Fields:

  • Issue Type (code defect, test defect)
  • Issue Level 01 (product change, test bug, infrastructure/lab/network issue)

Follow-up Questions:

  • Why do we have product changes getting checked in that break tests? 
  • Could those changes have been caught before check-in?  How? 
  • Did anyone run the tests before the code was checked in?

Example 2 - Do we have high priority bugs that are regressions from previous releases (points to test holes or cowboy check-ins)?

Fields (values):

  • Priority
  • Regression (previous milestone, previous release, not a regression)

Follow-up Questions:

  • Do we have unit tests for those scenarios we should have run before check-in?
  • Did anyone run the tests?
  • Why aren’t we fixing these high priority regressions? Is this the right decision?

Example 3 - Who is finding our bugs and how are they finding them?

Fields (values):

  • Source (feature tester, feature developer, feature test team, feature developer team, customer, etc.)
  • How Found (exploratory testing, unit test, integration test, dogfooding)

Follow-up Questions:

  • Who is finding the high priority bugs? 
  • Are testers/developers finding more bugs in functional units/end-to-end scenarios?  Why?  Is this the “right” mix?

Example 4 – Which bugs were not fixed in feature crews and added to our technical debt?

This one might take a little explaining.  We use a parallel development model called “feature crews”.  This means small teams work on new features in branches until the features are “complete”, then they merge those changes up with a reverse-integration into a higher level branch.  We guide teams to fix all their feature bugs before integrating their work back into the main lines to avoid accumulating technical debt.

Fields (values):

  • Bug type (product bug, feature bug)
  • Feature ID (which feature work item was this bug related to?)

Follow-up Questions:

  • What types of feature bugs are getting turned into “product bugs” / i.e. what technical debt are we accumulating and why?
  • Is one particular feature team accumulating more technical debt on average than others?  Why?
  • When were those bugs moved to product bugs?  Did the team push technical debt out of their feature crew too aggressively?
  • Which teams can I reward for consistently not contributing to technical debt?

Wrapping Up…

I hope this post helps ignite some conversations around how to identify opportunities for process improvement in your organizations. 

Remember, you get what you measure, so measure what you get!

In testing, you have to cross jurisdictions

The other day on the way to work, I was almost side-swiped by a big rig weaving all over I-40.  The truck’s erratic driving continued, and after I watched a few more cars quickly veer out of its way, I decided to call 911 to report the truck.  Although I described to the triage operator that I was on the Interstate, I was patched through to the police department for the city I was passing through.  They politely explained to me that by the time they could get an officer to general area, the truck would be out of their jurisdiction.  Worse, when I asked them how to contact the highway patrol, they couldn’t even tell me.  Clearly, they had triaged the problem to someone else’s plate and were not interested in the final outcome.

Unfortunately, I see this same type of mentality repeated all the time in software testing.  At the end of the day, customers don’t care (and shouldn’t have to care) whether the feature was owned by tester A or B or whether the bug was created by this developer or that developer.  At the end of the day, they care whether their software meets their needs or not.  Anything else is just noise.

In testing, we have to cross jurisdictions all the time.  We can’t conceptual lines of responsibility keep us from doing what’s right for our customers.  For example, when we run into bugs outside our team’s feature areas, we must log the defect, follow up with the other team to make sure it’s properly filed, and then keep a watchful eye on it to make sure the right thing happens.  I tell my teams to “aim high” and keep a long term view of their work and think about how the customer will use features from an end-to-end perspective rather than just how they’ll use the single feature they’re currently working on.

Sometimes as SDETs, the boundaries we cross include discipline roles.  Many testers at Microsoft have worn the PM or developer hat from time to time to varying degrees.  Every tester knows that if they can narrow down a bug to a line of code and provide a verified fix along with the defect report, they stand a pretty good chance of getting the bug fixed.  Often too, testers find themselves playing customer advocate to PMs or devs because they’re the ones actually using the software rather than just imagining it from a design perspective.

Of course, there are valid reasons to stay within the lines sometimes, but I think the lines aren’t as always as clear as people perceive them.  Yes, we have to manage risk, and we have to use our limited resources appropriately.  That said, there are also other ways to make sure the right thing happens in the end without going too far outside the lines.  For example, in the case of the runaway big rig, the officer I spoke to could have offered to follow up with highway patrol himself, recognizing that he was at a desk and I was in my car on a mobile phone trying to stay out of this driver’s way.  Instead, he took the concept of jurisdiction a little too far (in my opinion) and handed the problem back to me to solve.  If you do that with software, your customers will simply hand their dollars to someone else.

So next time you think about handing off a bug to another tester or team, ask yourself whether you should maybe run with it a little bit longer to ensure the right thing happens in the end for your customers.

Posted by JasonBa | 1 Comments
Filed under:

Help! Files in my bin directory are automatically added to source control!

You’re pulling your hair out.  Any file saved to the bin directory (even log files and temporary files you generate) end up getting added to source code control.  You’re getting more frustrated by the minute, especially when you try to build and some other developer already has binary files checked out exclusively (because you can’t merge them).  What’s going on here?!  I am reminded of the pain around website projects nearly weekly on Twitter, so I finally decided to blog about it in hopes someone will read this and save themselves a bit of trouble. 

My magic 8-ball says, you’re using a website project. 

The solution to this problem is to convert your website project into a Web Application Project.  I’m sure this “feature” was initially added as a convenience to website builders, but it’s also the cause of frequent confusion and angst from my observations.  Web Application Projects were created to solve this and a slew of other issues with website projects. 

Here’s an overview of Web Application Projects with links to more documentation for creating them, building them, and deploying them.  Hope this helps!

Yet another reason I love Twitter

If you read this blog, you’re probably already following me on Twitter (@JasonBarile), and you probably already know that I really love Twitter.  I stumbled upon a new reason for loving it today – real time crowd sourcing of data for a bug triage session.  We were discussing some bugs in meeting today when the conversation turned to speculation about whether or a specific feature was “commonly used”.  Of course, that’s a subjective measure, so I decided to take a very subjective temperature check via Twitter.  Within 5 minutes, I got replies from 5 users who told me they never used the feature.

Granted, this isn’t a scientific approach at all.  I’m positive some users have used this particular feature in the past.  Some may even depend on it for critical business needs.  That said, the fact that everyone who responded said they haven’t used it tells me it’s not critical for a majority of our users, which was exactly the data point we needed at the time.

This was so much more efficient than saying “I’ll get back to you with some data in a couple of weeks”.  It helped us come to a quick decision and move on.  We’ll probably still follow up on this by talking with some of our customers to validate the decision, but the urgency of doing that is lower now because we already have a good idea of what the responses will be like.

Twitter brings you closer to more customers more faster.

Posted by JasonBa | 4 Comments
Filed under: ,

How to Add a Placeholder Account to Your TFS Bug Form For Pre-Triage Bugs

This question came across Twitter recently, and I had to do a little research to answer it.  I thought I’d pass along what I found in case others are wondering how to do the same thing.

The scenario – You want to add a “placeholder” account to the available list of options in the AssignedTo field so you can assign bugs that have not yet been triaged to this value.  Ideally, you want this to be the default value.  When your team members file bugs, they get assigned to this fake user.  Then, bugs that have not yet been triaged by the team can easily be searched.

Note, this is not the only way to solve this problem.  Other solutions I’ve seen include adding a new state value for bugs called something like “submitted”.  Then, when bugs are accepted by the triage team/process, they’re moved into the “active” or “accepted” state.  If there is sufficient interest, I’ll blog about how to do that.

 

Step 1: Export the work item metadata for the work item type you want to modify.  See this page for details on how to use witexport.exe.  The command I used to export the “Bug” work item type from my team project called “MovieApp” looked like this:

C:\Program Files\...\IDE>witexport.exe /f bug.xml /t http://jasonba-dev10:8080 /p "MovieApp" /n “Bug”

This command connects to the server jasonba-dev10 and exports the metadata for the “Bug” work item type from the Team Project called “MovieApp” to a file called “bug.xml” in the current directory.

 

Step 2:  Next, open the bug.xml file and find the section of the metadata that defines the “Assigned To” field.  Assuming you created your team project using the MSF Agile template, it would look something like this:

      <FIELD name="Assigned To" refname="System.AssignedTo" type="String" reportable="dimension">
        <VALIDUSER />
      </FIELD>

The <VALIDUSER /> tag means that any member of the Valid Users group is a valid option for the “Assigned To” field.  Unfortunately, this limits you to just those members.  So, since we want to add a placeholder user, we’ll need to remove that tag and change a few other things.

 

Step 3:  Remove the <VALIDUSER /> line (just that line only!) and replace it with the following text:

        <DEFAULT from="value" value="Not Yet Assigned" />
        <ALLOWEDVALUES expanditems="true">
          <LISTITEM value="[global]\Team Foundation Valid Users" />
          <LISTITEM value="Not Yet Assigned" />
        </ALLOWEDVALUES>

Let’s walk through what these new lines mean.  First, the <DEFAULT…> line specifies that the field should be get to “Not Yet Assigned” by default. 

Next, the <ALLOWEDVALUES…> section defines the set of values that appear in the “Assigned To” list.  The expanditems=”true” attribute means that if you specify a group in the list of allowed values, any sub-groups that happen to be members of that group should be expanded and their members should also show up as options in the list.  The default value for this attribute is true, so it’s really not necessary, but I include it for completeness.

The first <LISTITEM…> line pulls in all members of the global server group called “Team Foundation Valid Users”.  Essentially, anyone with access to the server will now show up in the “Assigned To” dropdown list.

Finally, the second <LISTITEM…> line is what defines our fake user named “Not Yet Assigned” (hopefully you don’t have a real team member with that name!).  Note that this text must match the text you specified in the <DEFAULT…> tag in order for the default setting feature to work properly.

Note, you can add other similar values here, such as “Closed” for bugs that have been moved into the “Closed” state.

 

Step 4: – Upload your template changes back to the server with the witimport.exe command.  For example:

C:\Program Files\...\IDE>witimport.exe /f bug.xml /t http://jasonba-dev10:8080 /p "MovieApp"

 

Step 5 – Finally, if your import was successful, refresh your Team Explorer window to pick up the changes.  When you create a new bug, you should see the value “Not Yet Assigned” as the default value for the “Assigned To” field.

Posted by JasonBa | 4 Comments

Make Sure You Reinstall VS 2008 SP1 After Installing Team Explorer

If you installed Team Explorer into Visual Studio after you applied Visual Studio 2008 Service Pack 1, you’ll need to re-apply the service pack.  If you don’t, then you’ll be running an unsupported “mixed mode” installation, meaning the core Visual Studio components will be at the SP1 servicing level and TE will be at the RTM servicing level. 

Here’s a quick way to determine if your installation is in this state:

1. Click on Help->About Microsoft Visual Studio

2. On the Help screen, click “Copy Info”

3. Paste the text into Notepad (or your favorite editor)

4. Check the version of Visual Studio:

   Microsoft Visual Studio 2008
   Version 9.0.30729.1 SP  <—indicates you have SP1 installed on Visual Studio
   Microsoft .NET Framework
   Version 3.5 SP1

5. Now check the version of Team Explorer.  If you see this, then you' have SP1 installed on Team Explorer:

   Microsoft Visual Studio 2008 Team Explorer
   Version 9.0.30729.1

However, if you see this, then you need to re-apply Service Pack 1 to bring Team Explorer up to the right servicing level:

   Microsoft Visual Studio 2008 Team Explorer
   Version 9.0.21022.8

You can download the service pack installer here.

Check out the new VSTS Community Website on MSDN

We've just launched a new page on MSDN to help give some visibility to the great contributions from the VSTS community.  Check it out at http://msdn.microsoft.com/en-us/vsts2008/bb964396.aspx.

TFS Build Tweeter renamed

TFS Build Tweeter has been renamed to TFS Build Monitor to better refelct its purpose.  This cool community project can send out alerts for TFS Build events via Twitter or light up a USB powered "build light".  Visit the project's page on Codeplex for more information.  Thanks to Steve for the tip!

NotAtPDC - A Conference for the Rest of Us

If you're like me, you're sitting at your desk working today instead of attending PDC '08.  Luckily, sessions are available from Channel9 24 hours after each event (tags: PDC 2008, PDC08), so you can still get all the scoop.  First up for TFS today will be Brian Harry's talk from yesterday, "Team Foundation Server 2010: Cool New Features".  Also be sure to watch Stephanie Saad's talk on how we use TFS at Microsoft later this week.

Additionally, several MVPs and other community folks have put together an ad hoc virtual conference, NotAtPDC, for people who couldn't attend the PDC.  The presentations will all be given virtually via LiveMeeting, so you can watch them in real time and participate by chatting or asking questions.  Find more information at NotAtPDC.com.  The event schedule is evolving constantly as word spreads, so check back often for updates.  You can also follow developments on Twitter by following @NotAtPDC.

Enjoy!

Twitter Poll Results – How Testers Define Their Jobs

It’s a bit late, but I’m finally posting the results of my recent Twitter poll.  This was definitely an experiment.  I only received 4 responses for this poll, but it’s a start.  For future polls, I’ll allow blog comments as well.

The 4 responses I received were:

  • “I'm the person who investigates and reports on the state of our applications compared to software developed by other companies”
  • “At the moment work consists mainly of trying to find business!”
  • “Develop Software for Testing Software. Nurse Out-Source staff. Design Low Maintenance Test Automation Solutions.”
  • “System questioner and information provider”

If I missed any replies, I apologize.  Please feel free to leave comments to this post with your job description if you’d like.  In particular, I’m interested to know if you think your job stops at reporting defects or if you think your role includes actually driving changes you feel are important in the product.  When I ask this question in crowds of testers, I tend to get strong responses on both sides of the fence.  Which side are you on?

Thanks to PhilK for pointing me to a similar question posted at the Software Testing Club.

Posted by JasonBa | 0 Comments
Filed under:

Preview of next TFS Power Tools Release

Chances are you’ve already heard about this, but if you don’t read Brian Harry’s blog regularly, this is one post you don’t want to miss.  Brian has posted some screenshots and details about our upcoming TFS Power Tools release.  This isn’t your typical can of “nice to have” widgets… This release will introduce some new features we’ve been getting lots of requests for, in particular, more “Team-ness” and Windows Shell integration.  I’m not going to go into all the details here because Brian’s done a great job of explaining all the new goodies.  I have to say, I’m really looking forward to helping test this out!!

Twitter poll for testers: How do you define your job?

I'm curious to know what testers think their job descriptions are. Please send me your replies via Twitter to @jasonbarile. In the spirit of brevity, this keeps your answers to <= 140 characters :)


I'll take descriptions until this noon EDT (GMT -05:00) on Sept. 29, 2008 and I'll post all the replies back here.

Posted by JasonBa | 3 Comments
Filed under:

BuildTweeter - TFS Build Publisher for Twitter

If you’re looking for even more ways to get pinged by your build server, check out release v0.1 of BuildTweeter on CodePlex.  It’s a handy little Windows service that monitors build events and build quality change events and will post them to the Twitter account of your choosing.  Of course, this opens your build quality up to the general public, but it’s all about transparency anyway these days, right?  ;)  You can use the service with either TFS Build 2005 or TFS Build 2008.

Photosynth of our building in Raleigh/Durham, NC

msftsynth

I thought it'd be fun to share a Photosynth of our building in Durham, NC.  Note we don't have a Microsoft sign on the front, so if you're coming for a visit, just look for this building and stop by front desk in the lobby :).

If you haven't tried Photosynth.net yet, I highly recommend it!

Posted by JasonBa | 0 Comments

What I Look for in Test Plans

I've been reviewing a lot of test plans recently.  As I review them, I've compiled this list of things I look for in a well written test plan document.  Here's a brain dump of things I check for, in no particular order, of course, and it is by no means a complete list.  That said, if you see a major omission, please chime in to let me know what you expect from or include in your test plans.  These requirements may or may not work for you based on your development methodology, but it's what I look for :).  I'm sure I'll think of several more points as soon as I publish this post!

Test Strategy

I always like to see the testing strategy called out early in the test plan.  In general, it's always best to vet your testing strategy with your team as early as possible in the product/feature cycle.  This helps set the stage for future test planning discussions and most importantly, it gives your teammates an opportunity to call out any flaws in the strategy before you've done a bunch of tactical work such as creating individual test cases and developing test automation.  Some key points I look for in this section are:

    • An Understanding of the Feature Architecture - For the bits you are testing through "glass box" approaches, reflecting a simple understanding of the feature architecture in the test strategy section is very helpful. I don't expect to see a full copy of the feature architecture spec - just the "interesting" points called out and how the tester is using that knowledge to create an efficient testing strategy.  For example, if the UI is build using MVC, a lot of testing can be done "under the hood" against the controller rather than driving all the tests via automation through the UI.
    • Customer-Focused Testing Plan - Are you going to do any "real world" testing on your feature?  Some examples include "dogfooding" (either using the product yourself or deploying frequent drops to real customers to get feedback), testing with "real" data, install/uninstall testing on "dirty" machines (as opposed to cleanly installed VMs, for example).
    • Test Case Generation Methodology - List the approach you're taking to generating test cases.  Are you using the typical "sit around a table and brainstorm" approach?  Model based testing?  Pairwising?  Automatic unit test stubbing tools?
    • Identification of Who Will Test What - For example, developers will write and run unit tests before check-ins, testers will focus on system level, integration, and end-user scenarios, etc. This serves multiple purposes: reducing redundancy, driving accountability for quality across the feature area

Testing Risks and Mitigation Plans

What known risks are you aware of?  Some examples might be building on top of a new platform/framework that hasn't been fully tested yet, tight schedules, or relying on components delivered from external sources or other teams.  Once you've called out the major risks, list what you are doing to mitigate the chances each one could impact your project?  For example, do you have escalation paths identified in case you find bugs in those external components? 

Testing Schedule

Ideally, this would be merged into a common team schedule so you can see dates from each discipline alongside each other.  Dates/testing events listed should include dependencies (e.g. Test Plan Initial Draft Complete" probably depends on "Feature Architecture Spec Complete" and "End User Scenario Spec Complete" dates being hit).  Sharepoint is a great place to keep a team calendar for feature development, and the test plan can simply contain a link to the current schedule.

Test Case Details

Each test case listed should be written in such a way that someone who is only loosely familiar with the feature (and armed with the feature specifications) can execute the test cases.  This is helpful for several reasons: churn among testers on the project, it facilitates hand-off to test vendors or servicing teams (if you're lucky enough to have one!), and most importantly, clarifying what is and isn't being tested by reducing ambiguity in the test plan.

  • a clear description of the test case & steps
  • a description of expected output/behavior
  • things to verify to prove the test case worked (or didn't, in the case of negative testing)
  • "clean up" steps, in the case that the test changed something that might invalidate the entry point for other test cases
  • mapping back to an end-user scenario (this can be accomplished by grouping your tests by scenario, for example)
  • priority of the test case (from the end-user's perspective usually, though there are an infinite number of ways to prioritize)

And last but not least...

Make sure you use the right cover sheet!  (Did you get that memo?)  :)  Seriously though, if your team has a test plan template, by all means start with that!  Chances are it's much more than a "TPS report".  Templates give you a framework to help you think through your test planning process.  You probably don't need to include each and every section in your final plan - just include the sections that are relevant to the feature/product you are testing.

There you go.  I hope this helps you think through your own test planning process.  Again, if you have suggestions for other important points to cover in a test plan, please leave comments to let me know!

Posted by JasonBa | 1 Comments
Filed under:
More Posts Next page »
 
Page view tracker