John Socha-Leialoha's Blog

How to Name Test Methods

John Socha-Leialoha - MSFT — Sun, 11 Mar 2012 17:30:46 GMT

I really like BDD (Behavior Driven Development) and it’s structure for helping to focus on behavior instead of implementation when you write unit tests. Some time ago I ran across a blog post by my former colleague in the patterns & practices group Daniel Cazzulino: Writing meaningful, self-documenting, behavior-oriented tests. Another good post, which has more links to BDD is this one: Approaching Behavior Driven Development (BDD) from a Test Driven Development (TDD) perspective.

As a result, I started naming my tests with names like this, using underscores to separate the different BDD keywords:

[TestMethod]
[Description("Column order is different, but tables match")]
public void ShouldNotThrow_WhenAreEqual_GivenSameData_ButColumnOrderDifferent()

This worked reasonably well in Visual Studio 2010 tests results window because I could show the Description column. That way the test results included an easy to read description (the Description attribute), as well as the Given/When/Then behind the test (although I put the Then part first and called it Should):

Then along came Dev11 beta. They’ve completely changed the test results experience. The new version, at least in the beta, doesn’t support the description attribute, so you only see the test method names, resulting in my test results looking like this:

Argh! That makes it really hard to read test methods. Plus, I wasn’t completely happy with the long test methods anyway. What to do?

Rethinking Test Method Naming

I had a nice conversation with another former p&p colleague, Peter Provost, who is a PM working on the test experience. He suggested that I put the BDD description into the Description attribute. I decided to try this out, and so far I’m liking the change:

// Column order is different, but tables match
[TestMethod]
[Description("Given two tables And same data But different column order | When TableAssert | Then true")]
public void ColumnOrderDoesntMatter()

Here is what this looks like in the new test runner:

One nice thing about this approach is that the BDD description is easier to read and it can be richer without overwhelming the method name.

Wishful Thinking

I think it would be really cool (just my opinion) if this had a rich tooltip that showed the class information (the current tooltop does) and also the Description attribute. It would also be really cool if I could write an VS extension that would format my descriptions into multiple lines with keywords like Given, When, and Then in bold. Hint, hint VS guys…

Retrieving TFS Results from a Tree Query

John Socha-Leialoha - MSFT — Wed, 22 Feb 2012 18:22:22 GMT

I recently needed to retrieve the results from a TFS tree query using .NET code, via calls to the TFS API. It was easy to find examples for retrieving a set of work items from a flat query, but I couldn’t find any information about how to retrieve a tree. I did find a few people talking about some ways to do it, but they said it was too slow.

In this blog post I’ll show you a very fast way to retrieve a tree of results. Let’s start with the query I’m using:

SELECT [System.Id], 
       [System.Title], 
       [Microsoft.VSTS.Common.BacklogPriority], 
       [System.AssignedTo], 
       [System.State, 
       [Microsoft.VSTS.Scheduling.RemainingWork], 
       [Microsoft.VSTS.CMMI.Blocked], 
       [System.WorkItemType] 
  FROM WorkItemLinks 
 WHERE (
        Source.[System.TeamProject] = @project 
    AND Source.[System.WorkItemType] IN ('Product Backlog Item', 'Bug')
    AND (
        Source.[System.State] <> 'Removed' 
        AND Source.[System.State] <> @closedState
      )
    )
    AND [System.Links.LinkType] = 'System.LinkTypes.Hierarchy-Forward'
    AND Target.[System.WorkItemType] <> ''
ORDER BY [Microsoft.VSTS.Common.StackRank], [Microsoft.VSTS.Common.Priority], [System.Id] mode(Recursive)

I’ve highlighted the FROM location, which in this case is WorkItemLinks, which you need to use in order to retrieve a hierarchy. Flat querys pull from WorkItems.

Retrieving a tree of work items actually takes two queries. The first query returns a list of work item IDs that contains SourceId and TargetId values (parent and child IDs when the link type is Hierarchy-Forward as here). However, the query does not return any of the details of the work item. For that you’ll need a second query (described later).

The tree query above also contains two parameters (@project and @closedState) that will need values. Here is a code snippet that shows supplying values and retrieving the list of IDs:

Dictionary<string, object> parameters = new Dictionary<string, object>();
parameters.Add("project", _tfsProjectName);
parameters.Add("closedState", "Done");

var treeQuery = new Query(_workItemStore, wiql, parameters);
WorkItemLinkInfo[] links = treeQuery.RunLinkQuery();

The first part of the code creates a dictionary with the parameter values to use when running the query. You must use the RunLinkQuery method to run the query, instead of the RunQuery method, when working with WorkItemLinks. Calling RunQuery when you reference WorkItemLinks will throw an exception.

The links array returned from this query might look something like this:

SourceId	TargetId	LinkTypeId	IsLocked
0	23268	0	FALSE
23268	23339	2	FALSE
23268	23464	2	FALSE
0	23633	0	FALSE
23633	24664	2	FALSE

These results represent a tree. In our case the tree has just one level of children. As an example, work item 23268 is the parent of two other work items: 23339 and 23464. However, more importantly, the results of running this query do not return any of the fields from the query. How do you get those fields? You have to run another query that returns the results. The second query needs to use the set of columns from the original query, but pull from the WorkItems “table” instead of WorkItemLinks.

//
// Build the list of work items for which we want to retrieve more information
//
int[] ids = (from WorkItemLinkInfo info in links
             select info.TargetId).ToArray();

//
// Next we want to create a new query that will retrieve all the column values from the original query, for
// each of the work item IDs returned by the original query.
//
var detailsWiql = new StringBuilder();
detailsWiql.AppendLine("SELECT");
bool first = true;

foreach (FieldDefinition field in treeQuery.DisplayFieldList)
{
    detailsWiql.Append("    ");
    if (!first)
        detailsWiql.Append(",");
    detailsWiql.AppendLine("[" + field.ReferenceName + "]");
    first = false;
}
detailsWiql.AppendLine("FROM WorkItems");

//
// Get the work item details
//
var flatQuery = new Query(_workItemStore, detailsWiql.ToString(), ids);
WorkItemCollection details = flatQuery.RunQuery();

Here is what this code does:

Create an array of work item ids in the same order as returned by the tree query
Build a new query that has this form: “SELECT {field list} FROM WorkItems”. The Query instance used to retrieve the tree of work item IDs has a property called DisplayFieldList that contains the the list of fields contained in the query. The code uses this list to build a new flat query that returns all the fields from the original tree query.
Call the RunQuery method on the new flat query

The set of work items returned by this new query will be in exactly the same order as the link query results at the top of this post. You can then “combine” these two results sets to build a tree and use this tree as you need.

Using this approach is very fast because there are just two queries. I don’t have any inside knowledge about how the TFS client using the SDK to retrieve results, but I wouldn’t be surprised if it’s very much like what I’ve outlined here.

Allowing Localizing after-the-fact using MUI

John Socha-Leialoha - MSFT — Wed, 14 Dec 2011 22:46:39 GMT

Here is the problem. We built an English-only C++ application. However, now some of our customers are asking how they can translate the application into other languages. We don’t have the budget to provide translated versions ourselves, and they’d be happy to do the work themselves. Is there a way they can translate our application after the fact? And without our involvement?

I thought the answer was “no,” until someone suggested MUI. My first reaction was “what’s that?” MUI, as it turns out, stands for Multilingual User Interface and has been supported in one way or another since Windows XP. I did not know this. The idea is that you remove the resources from your main binaries and instead put them into MUI files, which are just DLLs with a different extension. For example:

MyApplication.exe <— No resources
en-US
    MyApplication.exe.mui <— Language-neutral (in this case English)
de
    MyApplication.exe.mui <— German

In the picture above, the first MUI file (under the en-US directory) contains the English resources (strings, dialog boxes, etc.). This MUI file is marked as both the en-US file and the language-neutral file that will be used when Windows can’t find a better match.

The second MUI (under the de directory) contains the German resources that will be used whenever the Windows’ current language is German (weather that be in Germany, or any other country).

With very little work (less than a day), we were able to build a version of our application that our customers can translate to their heart’s content. Below I describe the steps for a developer to make their application translatable after the fact.

Before you Start

In order to create an MUI-aware application, you’ll need to install the Windows 7 SDK, which is where you’ll find MUIRCT.

Creating the MUI Files

You may not need to make many changes in your application to make all of this work if you only support Vista or later. If you require Windows XP support, you’ll need to read the section below.

The first step is to run a program called MUIRCT that will move the resources from your main executables into satellite executables. This program requires a config file that describes which resources to move over. Here is the file I used:

<?xml version="1.0" encoding="utf-8"?> 
<localization>
  <resources>
    <win32Resources fileType="Application">
      <neutralResources>
        <resourceType typeNameId="#16" />
      </neutralResources>
      <localizedResources> 
         <resourceType typeNameId="#2" />
         <resourceType typeNameId="#3" />
         <resourceType typeNameId="#4" />
         <resourceType typeNameId="#5" />
         <resourceType typeNameId="#6" />
         <resourceType typeNameId="#9" />
         <resourceType typeNameId="#11" />
         <resourceType typeNameId="#16" />
      </localizedResources> 
    </win32Resources>
  </resources>
</localization>

For more details, read Preparing a Resource Configuration File on MSDN.

Next you’ll need to run MUIRCT to split your files:

muirct.exe -q muiconfig.xml -v 1 -x 0x0409  ..\MyApplication.exe MyApplication.exe en-US\MyApplication.exe.mui

The “..\MyApplication” refers to the original EXE that contains all the language-neutral resources (the 0x409 says they’re en-US resources). The second-to-last parameter, “MyApplication.exe” is the name that will be used for the new version that contains none of the resources being moved. Finally, the last parameter contains the name of the MUI file that will be created to contain all the resources that are called out in the config file as resource you want to move.

If you have no need to support Windows XP, you’re done.

Supporting Windows XP

Windows Vista and later automatically handle locating and using the correct MUI file for resources. However, if your application needs to be able to run on Windows XP as well, you’ll need to make some changes to your code. In our case that was really easy because the calls to load strings were all encapsulated in a helper class. The idea is that you’ll need to call a new API that will located and load the best-match MUI file:

std::vector<WCHAR> moduleName(MAX_PATH);

GetModuleFileName(hModule, &moduleName[0], moduleName.size());
hInstance = ::LoadMUILibrary(moduleName.data(), MUI_LANGUAGE_NAME, 0);

The first two lines get the name of the executable that we’re running. The last line is where the magic happens. LoadMUILibrary is a function that in Windows XP will locate the appropriate satellite MUI file, load that file, and return an instance handle to that file. In Windows Vista and later, this call returns the instance handle for your main executable because the system automatically handles resource location in calls like LoadString.

Additionally, you’ll need to ensure you link to muiload.lib. I like to use a pragma in my source files, rather than setup additional libraries in the link options, using a line like this:

#pragma comment(lib, "muiload.lib")

Localizing MUI Files

I’m not an expert on localization, so I’m not sure what most companies use to localize native Windows applications. Here at Microsoft we use a tool called LocStudio, and I understand there are similar programs available outside of Microsoft.

Whichever tool you use, here’s what you have to do:

Create a new directory for the new locale using the two-letter language identifier (like “de” for German)
Copy the language-neutral MUI file
Translate the strings (and possibly change control sizes and positions in dialogs)
Copy the localized MUI file into the new directory (“de” in this example)

Just a quick note about Visual Studio. You can open native Windows executable files and edit the resources in them with Visual Studio. However, you cannot save changes you make to MUI files, so you’ll have to use another tool. Again, I don’t have knowledge of those other tools.

Working with Signed Binaries

What happens if the original binaries and MUI files are code signed? It just works. I didn’t know the answer to this so I tested it. I used MUIRCT to split the files, and then code-signed both the main executable and the MUI file. Then I made a localized copy of the MUI file for German that was not signed. I was able to run this and see the English strings when I ran with English as the default language for Windows. When I switched to German and ran the program again, it showed the German strings from the un-signed German MUI files.

Capturing Code Coverage for Coded UI Tests

John Socha-Leialoha - MSFT — Fri, 02 Dec 2011 18:45:27 GMT

You would think that simply running coded UI tests with everything setup for capturing code coverage would just work, right? And if you’re running your coded UI tests on a 32-bit operating system, it will.

However, if you’re running your tests on a 64-bit operating system, chances are you’ll see the test results (pass/fail), but you won’t see any code coverage results. Why not? And how do you fix it?

By default, the test runner in Visual Studio (and on the TFS build server) run as a 32-bit process, which also means that the code coverage tools are the 32-bit versions. When you run unit tests, the test runner loads all the assemblies into the test runner’s process, which means that .NET assemblies will be run as 32-bit assemblies.

The problem occurs when you used coded UI tests. In this case, the executable assembly is not loaded into the test runners process. Instead, it’s launched in a new process. If you assembly is marked as Any CPU, and you running on a 64-bit OS, the application you’re testing will now be a 64-bit application, which means the 32-bit code coverage code won’t be able to capture any code coverage for your code UI tests.

How do you fix this? Basically you want to make sure the builds used for test runs have the EXE targeted for 32-bit rather than Any CPU. One way is to modify the solution so it builds targets the main EXE to 32-bit, as shown here (this is the Configuration Manager dialog for the solution):

The other option is to change the target platform in the build definition

Using either approach will force the assembly that you want to test to be built for the 32-bit platform.

Shrinking the Arrow on the WPF Expander

John Socha-Leialoha - MSFT — Thu, 06 Oct 2011 01:57:57 GMT

My goal was simple. I needed a smaller arrow in the WPF Expander control to help conserve screen real-estate. Here is the before and after:

Before:
After:

The arrow on the right is 80% of the original size.

So how did I accomplish this? It turned out to be far simpler than any of the blog posts I found on the subject. They all suggested using Expression Blend to copy the full template and them modify it. But you can use a RenderTransform to reduce the size:

<Expander Header="{Binding Name}" FontSize="16">
    <Expander.RenderTransform>
        <ScaleTransform ScaleX=".8" ScaleY=".8" />
    </Expander.RenderTransform>
</Expander>

Notice that I bumped the font size up to 16. When multiplied by .8, this gives a final font size of 12.8, which is a nice size.

Delay Signing a C++/CLI Assembly

John Socha-Leialoha - MSFT — Sat, 10 Sep 2011 00:40:51 GMT

This should be easy, right? Just set the link options to delay sign the assembly with an SNK file. There were two issues I ran into, which I’ll describe here.

Fixing Delay Signing in SP1

I used the /DELAYSIGN and /KEYFILE settings in the properties page for my C++/CLI project and everything compiled just fine. However, when I tried to use an object from this DLL, I got this message::

CS1577: Assembly generation failed … does not have a strong name

This certainly wasn’t what I expected. My first step was to double-check the signing on the assembly. I used the following command-line command to check:

sn -T AssemblyName.dll

When I ran this command, it reported “AssemblyName.dll does not represent a strongly named assembly". Obviously the properties I set in the project settings dialog weren’t working. After quite a bit of searching, I found a blog post by Amit Mohindra that explained the problem, which is a change in Visual Studio 2010 SP1.

The blog post had suggestions on how to fix the problem, but it seemed like quite a bit of work. I tried another approach, which worked just fine.

Signing via AssemblyInfo.cpp

I added an new file called AssemblyInfo.cpp to my project, which has the following contents:

#include "stdafx.h"
 

using namespace System;
using namespace System::Reflection;
 

[assembly:AssemblyKeyFile("myKey.snk")];
[assembly:AssemblyDelaySign(true)];

This took care of signing the assembly, which I verified using the sn command.

Dealing with C++/CLI vs C# Assemblies

One this was all working, we added a call from our C# EXE into the C++/CLI assembly, and got this run-time error:

Could not load file or assembly 'AssemblyName, Version=0.0.0.0, Culture=neutral, PublicKeyToken=…' or one of its dependencies. An attempt was made to load a program with an incorrect format.

Incorrect format? Huh?

With a little more searching, I found the problem, and now that I understand the problem, it makes perfect sense. First, the solution. You need to change the Platform from Any CPU to x86 for you main EXE (or x64 if you’re building a 64-bit version) to match the bitness of your C++/CLI assembly:

Here’s why we had to make this change. Our C++/CLI assembly is calling not a pure .NET assembly since it’s making calls to windows APIs. As such, the C++/CLI assembly needs to be either 32- or 64-bit. It can’t be both because of the native C++ code it contains. A .NET assembly that is built for Any CPU can run as either a 32- or 64-bit process, but you can’t load a 32-bit DLL into a 64-bit process, hence the error. Building a 32-bit .NET EXE that loads the 32-bit C++/CLI DLL fixes the problem.

Interpreting C++ Code Coverage Results

John Socha-Leialoha - MSFT — Wed, 17 Aug 2011 19:48:58 GMT

You’ve finally gotten code coverage results for your C++ code. But now your code coverage numbers are lower than you expected. Why is that? And how do you get around it?

There are several issues that make C++ code coverage data “noisy” and/or inaccurate. First, if you use the standard C++ libraries, you’ll find the code coverage results littered with entries from the std namespace, which represents classes and methods provided by Microsoft. Clearly, you don’t want library classes provided by Microsoft to impact your code coverage results. Especially if it lowers your numbers.

Another issue has to do with using template classes. Look at this example, which is a real example from some of our code:

You would probably expect this to show 100% code coverage in Visual Studio (or TFS). It doesn’t. In fact, it reports 72% code coverage. The problem is the return statement at the end. Visual Studio reports code coverage in terms of blocks covered/not covered (also this article). There are blocks in the wstring class that aren’t being covered in this method.

How do you fix this problem?

Building a Code Coverage Report

There are probably many ways to provide better code coverage numbers for C++ code. I chose to write a report for TFS that reads results from the data warehouse and process them to produce more “useful” results. Here is an overview of what the report does with the data (I’ll describe each in more detail):

Use lines instead of blocks for coverage information
Treat partially covered lines as covered
Collapse concrete template instances into a “common” instance
Use the method results with the highest coverage
Filter out namespaces and functions

The following sections describe each of these bullets in more detail.

Use lines instead of blocks, and treat partially covered as covered

I debated a long time before making these changes. Visual Studio reports code coverage results based on block coverage, so I wasn’t sure how people would react to using lines instead. However, switching to lines and also treating partially covered lines as covered mitigates the issue I described above. It does mean we’re over-reporting coverage for lines that have conditional code, but our style guidelines call for using multiple lines in such cases, so this is probably a small issue.

Alternatively, there are Visual Studio APIs, found in the Microsoft.VisualStudio.Coverage.Analysis namespace that could be used to process the code coverage results. I haven’t tried this, and it would have taken more time than writing a report.

Collapse template instances

We have some template classes that we use as base classes. The methods are well covered with unit tests. However not all concrete instances call the base methods, and we have a number of concrete instances of these classes. The result is that we get partial code coverage multiplied by the number of concrete instances. Our desire is to know how much coverage we achieve for the code we wrote, not for compiler-generated code. As such, I chose to collapse concrete instances. For example, we have a template called BaseControl, with a concrete instance called BaseControl<ItreeView>. In the report, I collapse this into BaseControl__ so that we can “combine” the results for each base method from sub classes. This brings us to the next topic.

Use the highest coverage results

We have several different runs on our build server, and we want to combine these in the report results. For example, we have unit tests run during gated check-ins, and we runs all our automated acceptance tests nightly. The acceptance tests cover some code (such as UI code) that isn’t covered by the unit tests, and visa versa.

Because of the testing approach we’re using, the unit tests are written in C++/CLI and include the product code directly into the test DLL (see Writing Unit Tests in Visual Studio for Native C++) whereas our acceptance tests are written in C# and use the production DLL. This results in the DLL names being different between these two runs. The report I created replaces these different names with a common name so the data can be combine.

One the report collapses template instances and combined results from DLLs with different names, but representing the same code, there may be more than one record (result) for each method in the product code. How do you combine these records? TFS stores the name of the class and method, as well as lines covered, not covered, and partially covered (and also block information). I chose to use the record for each method that has the highest coverage. So if you have different test runs that cover different parts of a method, choosing the result with the highest coverage will under-report code coverage since there’s no way to tell how much overlap there is between different result records.

Filter out namespaces and functions

In my MSDN article, Agile C++ Development and Testing with Visual Studio and TFS, I talked about filtering out common patterns, such as “std::”, from the results because this represents library code that we don’t want to include. Since then, I’ve found even more patterns, so the new report I created excludes a much larger list.

Using the Results

The results of all this work were very significant. Switching from block to line coverage moved our code coverage results up from 62% for one DLL to about 70%. Doing all the other work took this number up to 87%, which is more of what I expected to see since we used TDD to write all the code in that DLL.

Using the Report

If you’ve never added a new report to your reporting site, you can find some instructions here: http://blogs.msdn.com/b/aaronbjork/archive/2010/07/30/microsoft-visual-studio-scrum-1-0-updated-sprint-burndown-report.aspx.

Code Coverage.rdl

Once you install the report in your team project, you’ll see something that looks like this:

Customizing the Report

There are several places where you can customize the report. Open the report in Report Manager (right click on the Reports node in Team Explorer inside Visual Studio and select Show Report Site), click the Properties tab, then click Parameters. You’ll see something like this:

There are several hidden parameters you can use to provide more control over what you see in the report:

MinStartOffset	The default is -5, which means the report will only show builds within the last 5 days.
IgnoreBuilds	A set of build definitions that you don’t want to show up in the Build Definitions parameter. I’ve used the name “empty” when there are no builds I want to ignore.
IgnoreAssemblies	A list of assemblies that you want to exclude from the report. This is useful to us because we have several projects that share source, so they’re all part of the same gated build definition.
DefaultBuildDefinitions	The list of build definitions that you want to be selected when you first run a report.

Remember to click Apply to save changes you make to these parameters.

Capturing C++ Code Coverage with Visual C++

John Socha-Leialoha - MSFT — Tue, 16 Aug 2011 23:32:38 GMT

I wrote an article about six months ago on code coverage in C++, which is part of my article Agile C++ Development and Testing with Visual Studio and TFS. In that article, I showed how to use a command line to create code coverage because I was under the impression that Visual Studio 2010 didn’t support C++ code coverage. I had tried it, but couldn’t get it to work at the time.

It turns out there is a way to use capture code coverage directly in Visual Studio (or in on a TFS build server) without any customization. The trick, as it turns out, is that you have to turn on the /Profile switch in the linker.

Including ALL of your Code

If you’re using static libraries, as described in Writing Unit Tests in Visual Studio for Native C++, you will need to make a change to ensure all of your code is included. By default, the linker only includes code from static libraries that is actually used. So if you don’t have a unit test that hits some code, it won’t show up in the code coverage results. That means you won’t be able to find dead code. To fix this problem, you need to turn off linker optimizations:

The /OPT:NOREF switch will include all your code in the test DLL.

Turning on C++ Code Coverage

Right click on your DLL (the test DLL in my case since I use a static library to link the test library into the C++/CLI test project, see Writing Unit Tests in Visual Studio for Native C++) and turn on the /PROFILE linker switch:

Once you do this, you’ll be able to use the testsettings file to specify which DLLs to instrument. Double-click the testsettings file, click Data and Diagnostics, then click Code Coverage to check the box in the Enabled column:

Finally, click Configure, and then check the DLLs that you want to be instrumented, which in this case is the MyCodeTests.dll file that contains both the test code and the product code.

Excluding Test Classes from Code Coverage

If you’re using static libraries and C++/CLI for testing as I described in Writing Unit Tests in Visual Studio for Native C++, you may have noticed that your test methods also show up in the code coverage results. You can exclude all the methods in a test class from code coverage using the ExcludeFromCodeCoverageAttribute in front of the class. For example:

[TestClass]
[ExcludeFromCodeCoverage]
public ref class MyClassFixture
{

Now all the methods in your test class will be excluded from the code coverage results.

Note: You’ll need SP1 of Visual Studio 2010 in order to see lines colored to show coverage information.

Building for Testability in C++

John Socha-Leialoha - MSFT — Tue, 12 Jul 2011 23:37:59 GMT

Ten months ago, when I joined my current team, we had a C++ application that was written as many applications are, with a number of classes, inheritance, and classes creating other classes they needed in order to do their work. The trouble was that writing tests for this code was very hard because it was very difficult to isolate a single class for testing. We had some tests, but they were all written as black-box tests that exercised the UI. Unfortunately, as I mentioned in my blog post on No Regressions, UI tests are hard to write, hard to maintain, run slowly, and are fragile (in other words, small, intentional changes to the product can easily break the UI test).

In this blog, I’ll describe the architecture that I proposed and then implemented in order to allow us to test at the different levels of the pyramid. We’re dealing with a fairly small codebase (less than 10,000 lines), so we were able to rewrite the program over a period of about six months, while also adding new functionality. As I write this, we have a total of about 1,000 automated tests, divided into roughly 700 unit tests and 300 acceptances/UI tests (written by three people).

When I set out to build this architecture, I had the goal of being able to write tests in C# that would exercise our C++ code. Most of our unit tests are written in C++/CLI (see Writing Unit Tests in Visual Studio for Native C++ and C++/CLI to C++ Tips and Tricks) because we didn’t have all the pieces in place to write C# tests until a month or so ago. If I were to do this project again, I think I would have tried to get the C# part working earlier because our testers are more comfortable writing in C# (when we searched for testers, far more have C# than C++ experience). The 300 acceptance/UI tests are written in C#. More importantly, we can write mock objects in C#.

Using C# Mocks with C++ Product Code

Think about that last statement, because this is pretty interesting. Our testers can write mocks in C# that replace production C++ components during testing. In other words, it looks something like this:

How can you write a mock in C# (or VB.NET) that will replace a C++ component? Using COM (Component Object Model). COM has been around for a long time and is used inside Windows by a number of components. Some people dislike COM, and if you’re one of those people, please read on. I chose to adopt the very core of COM, and nothing else. Since .NET interoperates very well with COM, and it’s fairly easy to write COM components in C++, the COM object model acts very nicely as a bridge between the .NET test methods and the C++ product code that we’re testing.

Why Choose COM?

Why did I chose COM? Basically, because both C++ and .NET support COM. Here is the .NET support for COM that I needed:

.NET supports calling methods on COM interfaces
.NET supports writing classes that implement COM interfaces

When people think of COM, they don’t always think of the same thing, as there are many different levels (OLE, DCOM, COM++, etc.). But the very core of COM is actually quite simple. Every component supports at least one interface called IUnknown plus one or more other interfaces. And that’s really all there is to the core of COM.

The IUnknown interface supports object lifetime through reference-counting, and it also has a method, called QueryInterface, that allows access to other interfaces supported by the component. Here is the notation that is used for a COM component that supports two interfaces:

This component is a button that implements IUnknown and IControl. All interfaces inherit from IUnknown, so the three methods from IUnknown are always available. I’m not going to go into details on how to implement a COM component in C++ here—there are a lot of resources where you can find that, plus I want to focus on the high-level approach.

Using Private Classes and Class Factories

Traditionally C++ applications are written with the class declaration in a header file and the implementation in a “.cpp” file. This allows you to subclass an existing class. However, subclassing can make it more difficult to test classes, particularly when the base class contains implementation that could change and break a number of subclasses.

I decided to go the pure component route. Using a component model, you can’t subclass a component, nor can you create create a class directly. This approach makes it easier to test and reuse components since dependencies are explicit (I’ll explain dependencies between components in the next section), and it also helps focus design with separation of concerns in mind. In this section I’ll focus on using the factory pattern, where you define a public class with a public method for creating a new instance of a component. The class for the component, on the other hand, is private:

Using this approach has several advantages:

It ensures that no class is dependent on any other class. You can still have dependencies on other components, which is done by using interfaces
When building a component, you don’t have to flip back and forth between the header and implementation—it’s all in one file. You can have the implementation directly inline instead of having a separate class definition and implementation
Components are easier to replace with other components, which helps with testability (replacing a component with a mock, for example)
If a class depends on another component, you’ll need to supply that class to the component. This also make mocking easier

Here is a concrete example of this, showing the full header file, as well as an abbreviated implementation file.

RadioButtonWrapper.h

#pragma once

#include "IRadioButton.h"

class RadioButtonWrapperFactory
{
public:
    static void Create(HWND hwndControl, HWND hwndDailog, IRadioButton **ppInstance);
};

This is the class factory. Pretty simple.

RadioButtonWrapper.cpp

#include "stdafx.h"
#include "BaseControl.h"
#include "RadioButtonWrapper.h"
#include "UnknownImpl.h"
class RadioButtonWrapper : public BaseControl<IRadioButton>
{
public:
    RadioButtonWrapper(HWND hwndControl, HWND hwndDialog) : BaseControl(hwndControl), m_hwndDialog(hwndDialog)
    {
    }
 

private:
    ...
    virtual void EnableRadio(int id, BOOL enable)
    {
        HWND hWnd = ::GetDlgItem(m_hwndDialog, id);
        ::EnableWindow(hWnd, enable);
    }

    
    const HWND    m_hwndDialog;
};
 
void RadioButtonWrapperFactory::Create(HWND hwndControl, HWND hwndDialog, IRadioButton **ppInstance)
{
    *ppInstance = static_cast<IRadioButton *>(new RadioButtonWrapper(hwndControl, hwndDialog));
}

I’ve removed all except for one method. As you can see, the component is entirely contained within the CPP file, and therefore completely private. Only the factory is public, and it’s implementation is very simple.

Creating a Mock in C#

I’ll be filling the details on how to make all this work in some future posts. I also plan to provide a downloadable Visual Studio solution that has all the details working. For now, I’m going to provide a high-level overview of how you would write a mock object in C#.

There are a few things you’ll need. The interfaces that we use are all defined in header files, but .NET requires an interop assembly. In a later blog post I’ll describe how to create an interop assembly based on the C++ code. For now, let’s assume that assembly exists. Here is an actual mock object from our test suite:

class MockCPUCheck : ICpuInfo
{
    private bool m_Is64bit;
 
    public MockCPUCheck(bool is64bit)
    {
        m_Is64bit = is64bit;
    }
 
    public int Is64Bit()
    {
        return m_Is64bit ? 1 : 0;
    }
}

In this case, ICpuInfo is an interface that was defined in the C++ product code that is present in the interop assembly. To create a mock, all you have to do is subclass from this interface and implement the methods for that interface, which in this case is the method Is64Bit. In the product there is a “real” class that checks to see if the CPU is a 64-bit CPU. But when we’re testing the code with component/acceptance tests, we want some tests pretend to run on a 32-bit machine, and other tests pretend to run on a 64-bit machine.

When you’re testing a component that expects an ICpuInfo interface, simply create an instance of this class and pass it to the C++ method, which ties into the next section on dependency injection.

Isolating Components with Dependency Injection

It can be very hard to test an object that creates other objects because of the tight coupling between the object you want to test, and the objects it creates. Dependency injection is a technique that helps decouple dependencies. The idea is that the caller of a component, rather than the component, is responsible for creating dependent components. The caller creates the other components and then passes them to the component it wants to use. This may seem a little backwards, but it allows test code to provide mock instead of the real implementations of these required components, thus completely isolating a component during testing.

To make this more concrete, let’s say you have a component that can use regular expressions to validate the contents of a text box control. But this component, called RegExValidator, uses a shared component called RegEx to actually evaluate the regular expressions. RegExValidator handles reading text from a text box, calling the RegEx methods, and returning status (perhaps including a message). The signature might look something like this:

HRESULT Init(IRegEx *pRegEx, LPCWSTR pattern);
HRESULT Validate(ITextControl *pText, LPBSTR pMessage);

Any consumer of RegExValidator is required to call Init before calling Validate. The caller of the Init method is injecting the dependency on IRegEx into the RegExValidator component.

In your test code, you would create a mock object that implements IRegEx, called MockRegEx. You test code would create an instance of MockRegEx and then pass this to RegExValidator’s Init method before calling Validate. You can now test RegExValidator isolated from any “real” component, which means your test methods have complete control over the environment in which your tests run. Isolating the components makes it much easier to write tests that pinpoint regressions to a single class, and hopefully a single method.

Using a Factory Registry to Inject Dependencies

In some of our code, we use a factory registry as a way components can obtain dependent components rather than using dependency injection. For example, the RegExValidator could ask the factory registry for the factory that, in turn, creates a new instance of RegEx. Test code can overwrite the registered factory for a specific type so that a component will receive a mock instance when it asks for a new instance.

This approach is particularly useful for acceptance tests, as apposed to unit tests. Acceptance tests are typically higher-lever tests that exercise parts of the entire application rather than individual components. As a result, the product code, such as RegExValidator described in the previous section, will be created as a result of running the application. Because the test won’t be creating such components directly, you need a way to inject the mock components at a deeper level.

As an example, our application has code that shows different results on 32- and 64-bit CPUs. But since this code that uses the process type isn’t accessible directly to the acceptance tests, our acceptance tests register a mock class factory with the registry, and this factory returns a mock CPU test:

MockCPUCheckFactory cpuCheckFactory = new MockCPUCheckFactory();
m_page.Wizard.RegisterFactory(WizardComponents.CpuInfo, cpuCheckFactory);

Now when the code asks for information about the CPU type, it will get this mock component instead of the real component. This means the acceptance tests can validate both 32- and 64-bit results regardless of the actual CPU type.

Here is the definition of the class factory, which implements the IClassFactory interface:

class MockCPUCheckFactory : WizardTestLibrary.IClassFactory
{
    public bool Is64Bit { get; set; }

    public int CreateInstance(IntPtr pUnkOuter, ref Guid riid, out IntPtr ppvObject)
    {
        ppvObject = Marshal.GetIUnknownForObject(new MockCPUCheck(Is64Bit));
        return 0;
    }

    public int LockServer(bool fLock)
    {
        return 0;
    }
}

I’ll go into more detail in a future post. The main thing to notice here is the call to Marshal.GetIUnknownForObject. This method will return a native IUnknown interface from the C# mock object, which can then be passed back to the native code as an IntPtr, which represents a pointer.

How do you Make this Work?

Obviously, I’ve left out al lot of details in this post. There is some more groundwork I’ll need to lay, which I plan to do in future blog posts. My next blog post, for example, will cover how to define an interface in C++ that will appear in an interop assembly that can be consumed in .NET.

Balancing Work By Discipline Instead of Individuals

John Socha-Leialoha - MSFT — Thu, 02 Jun 2011 20:19:02 GMT

As part of our effort to become a highly-effective agile team, we’ve been working on switching from an individual to a team focus. During sprint planning, we want to know that we haven’t taken on too much work for each discipline. And on a daily basis, we’d like to see if we’re running behind on a discipline. For example, if we’re running behind on test, are there things developers can do to help?

The Iteration Backlog workbook that is included in the MSF for Agile Software Development v5.0 process template is a really nice workbook. As implemented, it allows you to see how much work is assigned to each individual so you can tell if they have the right amount of work.

However, not all agile teams like to assign work up-front to individuals—it’s fairly common to have group ownership of work within a discipline. As a result, I decided to modify the Iteration Backlog workbook to report on discipline rather than individuals. The changes are quite involved, so I’m only going to provide a high-level overview of the changes I made.

Overview of Changes

If you want to make this type of changes, you’ll want to read my blog post: Customizing the Iteration Backlog Workbook.

Here is a brief overview of the changes I made in order to have the capacity worksheet use discipline rather than team member. First, I needed to add a separate table that maps team members into disciplines:

Add a new Team Members table to the Settings worksheet:

You can actually add the same person more than once if they’re sharing their time between disciplines. The team member names here can be anything you want—they’re not tied to a field in the query results. The Discipline column, however, is tied to values from the Activity field. You do this by turning on data validation (in the Design tab of the ribbon) and setting the Source field to =Disciplines, which is a named range on the HiddenCapacityData worksheet.

The Remaining Capacity cells have the following formula:

=IF(AND(ISNUMBER(IterationStart),ISNUMBER(IterationEnd),ISBLANK([@[Team Member]])=FALSE),
    [@[Hours/Day]]*NETWORKDAYS(TODAY(),IterationEnd, Holidays[Date])
        - SUMIFS(PlannedInterruptions[Remaining Hours],PlannedInterruptions[Team Member],[@[Team Member]]),
    ""
)

Modify the Planned Interruptions table so the drop-down list of team members comes from the new Team Members table instead of the Assigned To field of the query results. This change required modifications to the VBA code

The finished Capacity worksheet looks something like this (I’m using dummy data):

VBA Changes

In the sections below I’m going to provide some of the details of changes I made to the VBA code. You’ll need some understanding of working with VBA in Excel in order to be successful following these examples. Because of the large number of changes I had to make to the workbook, I’m not going to provide you with all the details, but rather enough so with some effort on your part, you can get this to work, or perhaps make other types of changes.

VBA Changes for the Interruptions Worksheet

The interruptions worksheet needs to pull a list of users from the new Team Members table. Here are the changes you’ll need to make to the VBA.

First, add a new variable definition, m_disciplines, near the top of the file:

Private m_users() As String
Private m_disciplines() As String

Then at the end of the code, modify the UpdateLists sub, and add a new sub:

Private Sub UpdateLists()
    UpdateList FIELD_ACTIVITY, NAME_DISCIPLINE_LIST, m_disciplines, False
    UpdateUserList
End Sub

Private Sub UpdateUserList()
    Dim r As Range
    Dim c As Range
    Dim n As name
    Dim list As ListObject
    Dim columnIndex As Integer
    Dim vals As New Collection
    
    '
    ' Build a collection with only the unique values
    '
    Set list = GetListByName(Settings, LIST_TEAM_MEMBERS)
    
    columnIndex = list.ListColumns(Settings.Range(NAME_TEAM_MEMBER_CAPTION).value).index
    If columnIndex = 0 Then Exit Sub
    
    Set r = list.ListColumns(columnIndex).DataBodyRange
    
    For Each c In r.Cells
        On Error Resume Next
        vals.Add c.value, c.value
        On Error GoTo 0
    Next

    '
    ' Sort the values we got back
    '
    Dim sortedVals() As String
    SortCollection vals, sortedVals
    
    UpdateListRange NAME_USER_LIST, sortedVals, m_users
End Sub

VBA Changes for Capacity Worksheet

The changes in the code behind the Capacity worksheet involve changing formulas, and also using a list of disciplines instead of people.

At the top of the file, change m_users to m_disciplines:

Private m_disciplines() As String

In the code around line 92 (in the Sub called UpdateFormulas), change the code so it looks like this:

formula = "=IF(ISBLANK(" + _
          LIST_INDIVIDUAL_CAPACITY + "[[#This Row],[" + FIELD_TEAM_MEMBER + "]]),""""," + _
          "SUMIFS(" + _
          tableName + "[" + FIELD_REMAINING_WORK + "], " + _
          tableName + "[" + FIELD_ITERATION_PATH + "], " + NAME_CURRENT_ITERATION + " & ""*"", " + _
          tableName + "[" + FIELD_AREA_PATH + "], " + NAME_AREA_PATH + " & ""*"", " + _
          tableName + "[" + FIELD_ACTIVITY + "], " + "[" + FIELD_TEAM_MEMBER + "]," + _
          tableName + "[" + FIELD_WORK_ITEM_TYPE + "], """ + WORK_ITEM_TYPE + """" + _
          "))"

This changes the formula to group by the Activity field instead of the AssignedTo field.

Finally, change the UpdateLists sub at the end of the file:

Public Sub UpdateLists()
    UpdateList FIELD_ACTIVITY, NAME_DISCIPLINE_LIST, m_disciplines, False
End Sub

This will retrieve the unique set of activity values from the query results and update a named range in the hidden CapacityData worksheet. You’ll also need to update the named range in that worksheet, as well as update the set of named ranges in the GlobalConstants module.

Using the Discipline Iteration Backlog

Using this workbook is very much like using the version shipped with the process template, with some important differences.

Enter Team Members

First, you’ll need to add your team members to the Team Members table on the Settings worksheet. It doesn’t matter what you type in for each team member. You should, however, ensure that you use unique names for team members (if you don’t, Planned Interruptions won’t behave correctly).

If a team member is available to work in more than one activity, such as both development and testing, add one line for each combination of team member and activity. That way their time can be shared between activities.

Select Disciplines

In the Discipline Capacity table, make sure you have one row for each activity that you’re tracking. The rows in this table control what you see in the graphs on this page.

That’s really all there is to using the Discipline version of the Iteration Backlog workbook.

Driving Automation of Test Cases

John Socha-Leialoha - MSFT — Thu, 02 Jun 2011 01:37:31 GMT

As I’ve mentioned in previous posts, our team has been working our way up the testability pyramid. We started with unit tests, and now have about 670 C++/CLI unit tests that are testing our C++ code. However, we didn’t have any automated acceptance or UI tests. Our tester was going nuts because he was always behind the curve, and was doing entirely manual testing.

Starting about two sprints ago (our sprints are two weeks), we finally had both the resources and infrastructure to be able to start automating acceptance tests. I’ll describe our infrastructure in a separate blog post. But once we start write tests, we wanted to know how were were doing. To answer that question, I wrote a Reporting Services report:

You can download this report from here: Test Automation Status.rdl.

The report above is real data from one of our projects. Green is good—it shows how many test cases we’ve automated.

This report pulls data from the Test Case work item’s Automation Status field. This field can be Not Automated (the default for new test cases), Planned, or Automated. You can switch to automated by clicking on the Associated Automation tab of the Test Case and selecting a test method that is in the currently open solution. Alternatively, if you have the Test View tool window open in Visual Studio, you can right click a test, click Associate Test to Test Case and then choose a test case.

Since we plan to automate almost all of these test cases, we’re using the Planned automation state to represent test cases where we’re writing an automated test. In other words, in our case, Planned represents work in flight.

Hope you find this report as useful as we do in both driving test case automation, and knowing where you’re at.

Fixing Cube Processing ODBC Errors

John Socha-Leialoha - MSFT — Fri, 27 May 2011 18:20:02 GMT

Our Team Foundation Server (TFS) cube has been running just fine. But a couple of days ago, right after we updated the passwords on the service account, it stopped updating the cube. Here I’ll briefly describe how I found the problem, and also what appears to be the fix.

In order to find the problem, I downloaded the Admin Report Pack and installed the Cube Status report. You can find much more information on this report in Grant Holliday’s post: Administrative Report Pack for Team Foundation Server 2010.

When I ran this report, I saw error messages like this:

[Incremental Analysis Database Sync]:
AnalysisDatabaseProcessingType=Full, needCubeSchemaUpdate=True.
Microsoft.TeamFoundation.Server.WarehouseException: TF221122: An error occurred running job Incremental Analysis Database Sync for team project collection or Team Foundation server TEAM FOUNDATION.
Microsoft.TeamFoundation.Server.WarehouseException: Failed to Process Analysis Database 'Tfs_Analysis'.
Microsoft.TeamFoundation.Server.WarehouseException: Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of 'Tfs_AnalysisDataSource', Name of 'Tfs_AnalysisDataSource'.

It seems to indicate that there is a network-related issue. However, all we did was change the password on the service accounts running TFS and SSAS.

The Solution

Here is what I did:

Change the service account for Analysis Services from a domain account to Local System.
Restart SSAS

After that, everything started to work just fine. I then set the service account back to the domain account, using the updated password, and everything is continuing to run correctly. What I didn’t try, and what may work just fine, is to simply restart the Analysis Services service. If I run into this error again, I’ll try that fix (I’ve had reports from some other people that restarting the service is enough to fix this problem).

Building Mixed C# and C++ Projects in TFS 2010

John Socha-Leialoha - MSFT — Thu, 12 May 2011 17:04:59 GMT

We recently added C# unit test projects to our existing C++ product and unit test projects, but they weren’t being built by our gated check-ins. I did some research and found lots of blog posts. The closes one, and the blog post that got me pointed in the right directory, was Aaron Hallberg’s post on Solution Configurations. He talks about the Mixed Platforms custom platform, as well as creating new platform definitions. However, it turns out the solution to our problem was even simpler. All we had to do is check the C# projects so they would be built as part of the Win32 platform:

There are two things to notice about this dialog. First, notice that the active platform is set to Win32, and second notice that the Build checkbox is checked for the second project, which is a C# unit test project built using the Any CPU platform.

Here is the dialog box showing the configurations that are built by our gated check-in build definition on TFS:

In other words, the configurations defined in the build definition need to match up with the configurations you’ve defined in the solution files, not in the project files that are included in that solution.

We now have both C++ and C# unit tests running as part of gated check-ins. Life is good!

C++/CLI to C++ Tips and Tricks

John Socha-Leialoha - MSFT — Wed, 02 Mar 2011 18:04:22 GMT

In my previous post, Writing Unit Tests in Visual Studio for Native C++, I described the approach I’m using to write unit tests for native C++ code by using C++/CLI, which is C++ code that runs inside the .NET runtime. Because of this “mixed” programming model, there are some techniques you may need to employ between C++ and C++/CLI code.

Class Instance Variables

The instance variables in C++/CLI classes are different from instance variables in native C++, which means you can’t mix the two blindly. For example, you can’t use native C++ classes as instance variables in your test class. The following, for example, will not compile:

public ref class MyFixture
{
    MyClass m_instance;
};

You’ll get the following error message:

error C4368: cannot define 'm_instance' as a member of managed 'Tests::MyFixture': mixed types are not supported

So how do you deal with this? Instead of using a class as the type, you have to use pointers and then create/destroy the instance in the TestInitialize and TestCleanup code:

public ref class MyFixture
{
    [TestInitialize]
    void Initialize()
    {
        m_pInstance = new MyClass();
    }


    [TestCleanup]
    void Cleanup()
    {
        delete m_pInstance;
    }


    MyClass *m_pInstance;
};

The methods marked by TestInitialize and TestCleanup run before and after each test method.

Passing a Pointer to an Instance Variable

C++/CLI instance variables have a type of interior_ptr<T> instead of the type you wrote in your code. This makes a difference if you attempt to pass either the address or a reference to this instance variable to a native C++ method or function. For example, given the class above, you might think you could write a call to one of your native methods like this:

p->SomeMethod(&m_pInstance);

Compile this and you’ll see this message:

error C2664: 'Tests::MyFixture::MyTest' : cannot convert parameter 2 from 'cli::interior_ptr<Type>' to 'MyClass **'

This error appears because .NET uses a heap—items on the heap can be moved as a result of garbage collection. In order to send a pointer to a native method/function, you need to “pin” the pointer for the duration of the call, which you can do like this:

cli::pin_ptr<MyClass *> p = &m_pInstance;
SomeMethod(static_cast<MyClass **>(p));

Once the variable p either goes out of scope, or is assigned a new value, the pointer will be free to change again.

When you’re dealing with helper methods in your test code, you can also write this type of code:

Helper(&m_pInstance);
...
void Helper(cli::interior_ptr<MyClass *> ppInstance)
{
    **ppInstance = new MyClass();
}

Using the Product Planning Workbook with the Scrum Template

John Socha-Leialoha - MSFT — Wed, 23 Feb 2011 18:14:00 GMT

The Product Planning Workbook does not work with the Microsoft Visual Studio Scrum 1.0 process template out of the box. Fortunately, it doesn’t take much work to modify the workbook so it will work with the Scrum template.

Note: For information on the Iteration Backlog workbook, see my other post: Using The Iteration Backlog Workbook with the Scrum Template.

Create the Product Planning query

The existing Product Backlog query is very close to what you’ll need. However, there are two differences, one minor and one not so minor. The not so minor difference is that it needs to show product backlog items that have been closed so it can show you velocity of previous sprints. As a result, I chose to create a new query specifically for this workbook, which you can do by following these steps:

In the Team Explorer window, right click Product Backlog, which is under the Team Queries folder of your team project
Click Copy
Right click Team Queries and click Paste
Type Product Planning – Workbook in the New Name text box
Click OK
Right click this new query and click Edit Query
Above the query results, click on the Column Options button
Click Area Path in the Available columns list
Click the > button to add this to the Selected columns list
Repeat to add the Activity column
Click OK
Click on the left side of the clause that excludes the Done state (where you see the right arrow to the left of the selected line):
Click the Delete Clause button () in the toolbar (at the top, right in the image above)
Repeat for the Iteration Path clause
Click Save Query at the top of the query editor window

Your query results should now contain the Area Path column and show Done product backlog items.

Modify the Product Planning Workbook

You’ll need to obtain a copy of the Product Planning workbook, which is located in the MSF for Agile Software Development v5.0 process template. You can follow the instruction in my previous blog post on how to retrieve an “unbound” copy of this workbook: Customizing the Iteration Backlog Workbook.

Once you have the Product Planning workbook, here are the instructions on how to modify the workbook:

Press Alt+F11 to display the VBA editor
In the Project window, click on LocalizableStrings:
In the Properties window, click Visible and then select –1 – xlSheetVisible from the drop-down:
Return to the Workbook and click the LocalizableStrings to display this now visible worksheet
Change Story Points to Effort, which is probably in cell B5
Change Active to Committed, which is probably in cell C2
In the Project window, click on LocalizableStrings
In the Properties window, click Visible and then select 2 – xlSheetVeryHidden from the drop-down
Save the workbook

Binding to Your Project

Now you’re ready to go. You can create copies of this workbook and then bind them to a specific team project. Once you’ve bound the workbook (instructions below), you can’t use the workbook with any other team project.

Click cell A1 of the Product Backlog worksheet
Click on the Team ribbon
Click New List in the Team ribbon
Select your project and click Connect
Click Query List
Click the drop-down under Query List and select Product Backlog from your team project
Click OK

Your worksheet should now be ready to use.

Using The Iteration Backlog Workbook with the Scrum Template

John Socha-Leialoha - MSFT — Mon, 21 Feb 2011 20:11:09 GMT

Our group has switched over from the MSF for Agile to the Microsoft Visual Studio Scrum 1.0 process template. We really like the Scrum process template. Particularly now that we’ve fixed the work items so we can resolve bugs and task: Marking Tasks and Bugs as “Done” During a Check-in. However, there is one piece we really miss from our days using the MSF for Agile process template: the Iteration Backlog workbook. In particular, we really miss the capacity page.

So in this blog post, I’ll show you how you can modify Iteration Backlog workbook to work with the Scrum process template.

If you plan to use your modified workbook on more than one team project, you should start by read my previous blog post: Customizing the Iteration Backlog Workbook. This will ensure that any changes you make are “portable.” Otherwise, your modified workbook will be tightly bound to a single team project.

Note: For information on the Product Planning workbook, see my other post: Using the Product Planning Workbook with the Scrum Template.

Modifying the Sprint Backlog Query

The Iteration Backlog workbook needs a few more columns than are provided in the Sprint Backlog query. You’ll see the following message when you click on the Settings worksheet (or any of the other worksheets):

One or more required fields filters are not present in the BurndownData pivot table: Area
Path, Iteration Path, Completed Work.

Modify the Sprint Backlog query to include the Area Path and Iteration Path columns

Fixing the Lack of Completed Work

The Scrum work items don’t have completed work, which is something I don’t miss as a user of this process template. However, the burndown worksheet requires this field. Furthermore, there is validation code that requires this field be present in order for the workbook to be functional.

The easiest “solution” to this problem is to hide the burndown worksheet from this workbook (removing it requires a lot more changes). After all, the Sprint Burndown report in the Scrump process template is really nice. I’m going to take that approach:

Press Alt+F11 to open the Visual Basic for Applications (VBA) editor
In the Project window, expand the Modules folder
Double-click UtilityModule to open this file
Remove the following line from the AreRequiredFieldsPresent subroutine:
AddIfFieldNotPresent list, FIELD_COMPLETED_WORK
Click Burndown in the Project window
Click Visible in the Properties window and select 2 – xlSheetVeryHidden, as shown here:

That’s basically it. Now the Iteration Backlog, Settings, and Capacity worksheets should behave correctly. If there’s enough interest in getting the Burndown worksheet to work with Scrum, I'll address that in a future post.

Customizing the Iteration Backlog Workbook

John Socha-Leialoha - MSFT — Mon, 14 Feb 2011 23:18:21 GMT

The MSF for Agile process template that ships with TFS 2010 includes a nice Iteration Backlog workbook. Because this is an Excel workbook, it’s completely customizable. However, as far as I know, there isn’t any documentation on how to go about customizing this workbook. The Iteration Backlog workbook contains a fair amount of VBA code, so it may be a bit daunting when you try to make changes. I did much of the development on this workbook when I was in TFS, so I can provide some insight into how best to make changes to this workbook.

Getting an Unbound Workbook

When you double-click the Iteration Backlog workbook inside one of the team projects, you’re getting a copy of the workbook that has been bound to a specific team project. Once a workbook is bound, it’s very hard to unbind it. How is this important? If you’re only going to be changing the workbook for one team project, it isn’t important. However, if you want to make changes that will work with more than one team project, it’s far easier to make changes to a single unbound workbook, and then bind copies of that workbook to the different team projects.

Here is how you can get unbound copy of the workbook:

In Team Explorer, right click the server node.
Click Team Project Collection Settings
Click Process Template Manager
Click MSF for Agile Software Development v5.0 to select this process template in the list
Click Download, select the folder where you’d like to save and click Select Folder

These steps will download the entire process template to a new folder created inside the folder you selected. You’ll find the Iteration Backlog workbook in the following path under the process template’s folder:

Windows SharePoint Services\Shared Documents\Project Management

Preparing the Workbook for Development

Now that you have an unbound workbook, there are some steps you’ll need to take in order to temporarily connect the workbook to TFS without actually binding the workbook to TFS. I’m using Office 2010, so the instructions below are for that version of Excel. If you have people who will be using Excel 2007, you should probably use Excel 2007 to modify the workbook just to be safe.

The first step is to retrieve a set of work items you can use while you’re making changes. You’ll save these as a plain text file to ensure that the workbook you’re changing is not bound to TFS, or that it has links to another workbook.

From Team Explorer, double-click the Iteration Backlog workbook that is in one of your team projects
Select the entire table on the Iteration Backlog tab and copy this to the clipboard
Paste the results into Notepad and save it as a text file
Quit Excel

Now open the unbound copy of the Iteration Backlog workbook and follow these steps:

If you see a security warning, click the Enable Content button:
Open the text file you saved in the previous step in Notepad, select all the text, and copy it. Note: make sure Word Wrap… is not checked under the Format menu in Notepad before you copy the text
Select cell A1 in on the Iteration Backlog tab and past the text from the previous step. Note: It’s important that you copy the text from Notepad so you’re not pasting in a reference to another workbook.
With the cells containing the copied text still selected in Excel, click the Insert tab on the ribbon, and then click Table
Ensure the My table has headers checkbox is checked and click OK.
Press Alt+F11 to display the Visual Basic for Applications window
In the Project pane, expand the Modules node
Double-click the DevelopmentModule to display the VBA code for this module:

The subroutine inside this module is called Prepare, and it should look like this:

.Add tableName & "_ASServerName", False, msoPropertyTypeString, ""
.Add tableName & "_ASDatabaseName", False, msoPropertyTypeString, ""
.Add tableName & "_TeamProjectName", False, msoPropertyTypeString, ""
.Add tableName & "_TeamProjectId", False, msoPropertyTypeString, ""

You will to insert some text inside the “” at the end of each line. Here is a summary of what will need to go into each of those lines:

_ASServerName	The name of the SQL Server machine that contains your TFS cube. You may have to contact your TFS administrator to determine this name.
_ASDatabaseName	Unless you administrator made any changes, this will be “Tfs_Warehoue”
_TeamProjectName	The name of your team project
_TeamProjectId	This is a GUID that you can obtain by right clicking your team project in Team Explorer and selecting Properties. The Url property will contains something that looks like this: vstfs:///Classification/TeamProject/guid, which guid will be the value you want to use here.

Once you’ve made these changes, click anywhere inside the Prepare subroutine and press F5 to run this method. Your workbook should now function as if it is bound to TFS without actually being bound (the content on the Iteration Backlog tab is “static” because it’s not actually bound to a TFS query).

Sharing the Customized Workbook

Once you’ve finished making changes to the workbook, you’ll need to undo the temporary “binding” you setup in the previous section. This is actually very simple. Place the cursor inside the Clean subroutine in DevelopmentModule and press F5. This will set the workbook back to a clean state that you can then put into a process template or share with anyone on your team.

If you want to bind to an actual team project, make sure you make a copy of the workbook (so you don’t bind your “development” copy).

Make sure the Iteration Backlog tab is the active tab
Click cell A1 to make this the active cell
Click the Team ribbon
Click New List
Select your team project and click Connect
Select the Iteration Backlog query and click OK

Then click the Team ribbon and click New List

When and Why VBA

You may be wondering why there is so much VBA in this workbook. And there is quite a bit. In this section I’ll provide an overview of the VBA used by each tab of the workbook.

Iteration Backlog Tab

The VBA code on this tab runs whenever you switch to this tab. It sets up custom formatting of the State and Remaining Work columns, as shown here:

The state has a green background whenever it’s Active. The Remaining Work column shows bars that give you an idea of the size of the different tasks, in terms of the number of hours remaining. You can easily modify these macros to change the formatting to work the way you want.

If you’ve changed the name of states, you don’t need to modify the macros. Instead, there is a hidden worksheet that contains the list of names. I’ll cover this a little later.

Settings Tab

The main job of the macros for this tab are to update the validation list used for the Area and Iteration cells. These lists display all the areas/iterations that are present in the query result. The actual code to update these lists is in UtilityModule, and is rather long. Chance are you won’t need to modify these macros.

Interruptions Tab

The macros on this tab handle updating the team member list (from the Assigned To field on the Iteration Backlog tab) and doing some validation.

Capacity Tab

This tab is probably where you’ll want to make customizations. There are a number of moderately complex formulas on this page, and many of them are actually set by the VBA code. Why, you ask? The formulas in this tab need to reference the query results table in the Iteration Backlog tab. However, this table doesn’t exist until TFS creates it. Since you can’t have formulas reference something that doesn’t exist, we had to use macro code to setup the formulas. That means any changes you make to the formulas, without updating the VBA code, will be overwritten by the VBA code whenever you switch back to this worksheet.

The easiest way to modify the formulas is to try making changes directly in the worksheet until you have it working the way you want. Then copy the formula into Notepad, for example. And finally update the VBA code that sets the formulas to use your modified formula.

Burndown Tab

I would venture to say that the majority of the VBA code in the workbook is devoted to this tab. The burndown chart is driven from a PivotTable on the hidden BurndownData worksheet that is connected to the TFS cube (which is a special type of database designed for historical reporting).

Customizing State Names and Other Strings

The names of states and fields that are used by macros are stored in a “very hidden” worksheet called LocalizeableStrings. In order to display this worksheet, you’ll need to switch to the VBA editor (press Alt+F11), select LocalizeableStrings, and then change the Visible property to –1 – xlSheetVisible, as shown here:

You’ll now be able to see that worksheet in the workbook.

Summary

This blog post just scratches the surface, but I hope it helps people customize the Iteration Backlog (and Product Backlog) workbooks for their own needs.

I Have a Dream…No Regressions

John Socha-Leialoha - MSFT — Fri, 11 Feb 2011 16:55:58 GMT

OK, so “I have a dream” isn’t original, and perhaps neither is my dream. However, the path to my dream is the road less traveled, and I believe we should have a super-highway instead of a dirt road.

Imagine you could build software without fear of breaking existing behavior. That’s my dream. As a developer, I want to be able to constantly improve our architecture to make it easy to add new functionality. But I don’t want to break anything. I don’t want to break the build. I don’t want to break someone else’s code. I don’t want to break the product. And I don’t want to break a customer’s scenario after shipment.

You may be wondering if this is even possible. I believe it is through a suite of automated regression tests. In our case, it’s mapping Mike Cohn’s test pyramid, which I first encountered while reading the book Agile Testing, into the Visual Studio/Team Foundation Server stack.

Traditionally I’ve heard people say it’s the job of testers to ensure quality. Rubbish. It’s the team’s responsibility. Without developer involvement, it would be very expensive to get to the goal of no regressions. As developers, it’s our job ensure our code is testable. I’ll go into more detail on this later.

Path to No Regressions

In my experience working on teams, there are several things that lead to regressions and loss of productivity for the team. On one team, we were constantly fighting with build breaks. Before checking in I would get the latest code from source, only to find it wouldn’t build. How frustrating is that. Then once I got it to build, behavior we depended on would be broken, so we’d have to figure out who’s code it was and work with them to get a fix. Double grump. I really don’t want to go through that again.

Here is a short list of things you can do to prevent this type of problem, which I’ll cover in more detail below:

Gated check-ins
Suite of automated regression tests
Failing tests for new bugs

At the core of all of this is discipline. Our team has to believe in “No Regressions” and work together to achieve that dream. My job is to help inspire the team to achieve this goal.

Automating Regression Testing

Automated regression tests should run automatically. No, that’s not redundant. In one team I worked on, it was a huge effort to get the regression tests to run. Testers needed to manually start a test run. And even then, the run often required manual intervention in order to get the tests to run. Ideally a regression suite should run automatically, without any intervention. And the results should be easy to access by anyone on the team.

It’s also important to have the right mix of tests. On the left is a version of Mike Cohn’s test automation pyramid found on p. 277 of the book Agile Testing. I’ve modified this slightly. The idea is that each level of the pyramid provides a general idea of how many tests should be at that level. Each unit test should exercise just one public method of a class in isolation, should run in less than 20ms, and should run on any computer. Developers should be writing the tests in the bottom section of the pyramid.

The upper two levels of the pyramid are generally where testers will create automated regression tests, but we as developers still have a responsibility in these two levels—we need to design and write our code so testers can write tests below the UI, where they can be less fragile and run faster.

Stage 1: Unit Tests & Gated Check-Ins

Here is where we enter the dream. All of the tests in the pyramid should be run automatically and frequently on servers. Today our team’s unit and component tests run automatically whenever we check code into Team Foundation Server. We’ve set this up with gated check-in turned on. If the build fails, or if any of the unit tests fail, the check-in is rejected. In other words, we don’t have build breaks anymore. Woo hooo!

Of course, we can still break the product, and we have. This is where the next two layers come into play.

Stage 2: Acceptance Tests

The next level up is where things get trickier, especially when we’re writing software that works with server software, such as System Center Configuration Manager in our case. Here we want/need to run the tests in a real environment. Of course, this means the tests can have side effects that destroy the integrity of the environment. I once wrote a simple test that ended up corrupting a SharePoint database so you couldn’t even run the administration tools. I had to wipe out the databases and create new ones.

This is where Visual Studio Lab Management comes into play. You can configure a set of virtual machines in an isolated lab. Once you have this all setup, you can spin off a new isolated lab for each test run. This means you can have several developers/testers running tests or debugging software with “real” environments without stepping on each other (this is big!). You also have a well-known environment each time you spool up a test run. Finally, and this is one that sounds pretty exciting to me as a developer, if a test fails, you can go back to the snapshot of the lab at the point where the test failed, which means you can actually look at the repro environment. Very nice! I want it now!

Stage 3: UI Tests

The last layer is the smallest because as many of the acceptance/integration tests as possible should be below the UI. UI tests tend to be fragile, slow to run, and hard to maintain. As a result, you want just enough UI tests to ensure that the UI still works. Without support from us developers, testers are usually forced into writing far more UI tests than headless tests below the UI.

If we take it as a given that we have good testing in the layers below, testers can use Test Manager to create recorded tests that are used as smoke tests. Now imagine that these tests are also run in Lab Management automatically. Wouldn’t it be great if every night the build server would build the latest source and then kick off all of the tests in the pyramid?

Stage 4: One Button Delivery

I decided to throw in one more goal because I’ve just started to read the book Continuous Delivery. The dream here is that you can deliver/deploy quickly and easily with a single button click. Then if anything goes wrong, which is detected automatically, the system automatically rolls back to the last known-good state.

Where Next?

I don’t know if our small team will realize this dream, but I certainly hope so. I will blog more about our progress towards this dream as we make that progress.

Reading and Writing Task Sequence Variables in .NET

John Socha-Leialoha - MSFT — Tue, 01 Feb 2011 01:44:33 GMT

I’m working on a program that needs to read Configuration Manager task-sequence variables while running in the full operating system (as apposed to Windows PE). Reading task sequence variables is really quite simple. However, all the examples are either in VB Script, or C++. I’m writing this program in C#, so I need a managed way to read task sequence variables.

First, some background. You can find more information about task-sequence variables is this article on MSDN: About Task Sequence Variables. The object you need to use is a COM object. C#, as well as other .NET languages, has good support for working with COM objects. However, it’s not always obvious how to reference a COM object in .NET. The normal way is to open the Add Reference dialog box, click the COM section, locate the DLL and then click Add. However, there are cases where this isn’t sufficient, and working with the COM object for task-sequence variables is one of these cases.

Some COM DLLs, such as the TsCore.dll that we need to reference, have what’s known as an embedded type library. This is a description of the COM objects that are exposed by the DLL. In order to use such a DLL, you need to create an interop assembly that provides a bridge between .NET and COM. This is done using a program called TlbImp.

The following command line will run TlbImp and create a new DLL called TsEnvironmentLib.dll:

"%WindowsSdkDir%\Bin\TlbImp"”%SystemRoot%\CCM\TSCore.dll”

Once you run this command, you can open the Add Reference dialog box, click the Browse button and select TsEnvironmentLib.dll, which is a .NET assembly. At this point, reading a task sequence variable looks something like this:

TSEnvClassClass ts = new TSEnvClassClass();
return ts["MyVarName"];

There is one final issue. You need to make sure you use a version of TlbImp that matches the .NET version you’re targeting. In the case above, I needed to use .NET 3.5 since the target machine probably won’t have .NET 4 installed on it. I tried the different versions of TlbImp that were on my computer, and the one in the Windows SDK folder on my computer targets .NET 3.5. You can see which version TlbImp targets by running it without any parameters.

Writing Unit Tests in Visual Studio for Native C++

John Socha-Leialoha - MSFT — Fri, 19 Nov 2010 21:14:53 GMT

My team frequently creates wizards that run in Windows PE as part of a Configuration Manager task sequence, and we write them in C++. I’m a big fan of TDD, so when I started to work on a new architecture for our wizards, I wanted to write all the code using TDD.

When I asked around about writing C++ tests, I was told there isn’t any support for C++ unit tests in Visual C++. Not true. In this blog post I’ll describe what’s required in order to write unit test for native C++ code using Visual Studio.

For new C++ projects, you can set them up as I’ll describe below. However, for your existing code, you may have to reorganize your project structure before you can start writing unit tests.

Here is a quick overview of the setup for testing C++ code:

The unit tests need to be written in C++/CLI in order to use the .NET unit test framework
The code you’re testing should be in a static library that is linked into both your production EXE/DLL, as well as the test DLL

Setting up the projects

As I mentioned above, the first step is to setup the projects correctly:

Create a new Win32 Project and select Static library in the Application Settings screen. This is where you’ll add your production code
Create a second Win32 Project for your production EXE or DLL
Create a Test Project (in the C++ section of the Add New Project dialog box)
In the Properties dialog box for the test project, change the Target Name to the name you want to use for you unit tests. The default is always DefaultTest, rather than the name of the project you just created

I’m using Visual Studio 2010, but these instruction will most likely work with prior versions, especially Visual Studio 2008.

Next we need to link the static library into the test and production projects, and provide both of those projects access to the header files in the static library:

Right click the test project and select Properties
In the Configuration combo box, select All Configurations to ensure the changes below are made to both Debug and Release configurations
Click the Common Properties node and then Framework and References
Click Add New Reference…, select the static library and click OK
Click the Configuration Properties node, then click the C/C++ node
Click in the Additional Include Directories property and type something like this (substitute the name of your library project for ProductLibrary):
$(SolutionDir)\ProductLibrary;%(AdditionalIncludeDirectories)
Change the Common Language RunTime Support property to Common Language RunTime Support (/clr)

You’ll want to repeat steps 1-6 for your EXE/DLL project.

Writing C++/CLI Unit Tests

At this point you should be able to build your solution, so the next step if you’re using TDD is to write a test method. If you’ve written test methods in C#, most of this should be familiar to you, although with a different syntax. But if you’re a C++ programmer and you’ve never written C# unit tests, there is a bit of a learning curve. Here I’ll explain enough to get you started.

Creating a new C++ Unit Test project initially creates a file called UnitTest1.cpp that contains one unit test. I usually delete the extra code and just leave the single test method. If you have a simple class called SomeClass with a method called SomeValue that returns an int, you can write a test that looks like this:

[TestMethod]
void ShouldReturn1()
{
    std::unique_ptr<SomeClass> pClass(new SomeClass());
    Assert::AreEqual<int>(1, pClass->SomeValue());
};

There are several elements to this test that you may not be familiar with, but you can find full details on MSDN. First is the TestMethod attribute. This identifies the method as being a test method. When you run the tests in Visual Studio, it will run any public methods marked with this attribute.

Next, notice the use of the Assert class static method called AreEqual. There are a number of static methods that allow you to validate the results of running a command. Again, you’ll find all of these documented on MSDN.

Some Tips and Tricks

The most significant limitation is the lack of IntelliSense in C++/CLI in Visual Studio 2010 (the C++ team plans to correct that in a future release). Fortunately, there is a third-party product called Visual Assist X from Whole Tomato Software that brings back IntelliSense to C++/CLI in Visual Studio 2010. I’ve been using this now for a few weeks and I really love it.

There are some tricks you’ll want to be aware of when you’re using Assert. Because Assert is designed for .NET code, there are some methods that won’t apply, such as IsNotNull, which expects to receive a managed pointer rather than a native pointer. Here are some ways you can deal with this limitation:

Assert::AreNotEqual<DWORD_PTR>(0, (DWORD_PTR) pClass, "Should not be null");
Assert::AreEqual<DWORD_PTR>(0, (DWORD_PTR) pClass, "Should be null");
Assert::IsTrue(pClass == nullptr, "Should be null");

Finally, if you want to test a string value returned by a function, you can do something like this:

Assert::AreEqual<String^>("Expected", gcnew String(pClass->StringValue());

Note in particular the use of gcnew, which creates a new instance of a managed class. Likewise, you’ll notice the ^ at the end of String in the AreEqual. This tells the compiler that we’re working with a pointer to a managed instance of type String.

Update

I’ve create a new blog post with more tips and tricks: C++/CLI to C++ Tips and Tricks.

For code coverage, see Capturing C++ Code Coverage with Visual C++.

Final Thoughts

I’ve been using C++/CLI unit tests with native C++ code now for about 1-1/2 months and I find it a really nice environment. I can debug my unit tests just as easily as C# unit tests. At this point I have about 240 tests and they run in about a 1-2 seconds, which means I can easily run all these tests after making changes to ensure I haven’t broken anything.

After so many years writing in C#, I never thought I would enjoy C++ programming again. I was wrong. Using TDD to write C++ code is almost as nice as writing C# code, and I’m really enjoying the experience.

I want to thank my colleague Mike Schmidt for getting me pointed in the right direction. He had some C++/CLI unit tests, but they were testing public functions of a DLL. I did some research and added the part of about using a static library, which provides full access to the internals of the code—just the thing you need for writing code with TDD.

Feature creep, gold plating, and the Product Backlog

John Socha-Leialoha - MSFT — Thu, 21 Oct 2010 20:41:54 GMT

How many times have you been working on a feature only to discover a new scenario that you feel is important, but isn’t included in the estimates for the current sprint? And what did you do? If you’re like most programmers, you probably just wrote the code to handle the scenario.

If so, you’re experiencing feature creep and/or gold plating. What this really means is that you’re adding code that perhaps should never be in the product. How’s that you ask? Shouldn’t I ensure that I handle all scenarios?

In a word, no. You see, not all scenarios have the same value to a customer. If you just go ahead and write code for a scenario, you might be supporting a scenario that really isn’t that important. But you’ve actually taken time away from working on what are most likely more important scenarios.

So what do you do when you find a scenario that you’re not handling? Add it to the product backlog.

The product backlog represents value that you’d like to add to the product, stacked from the most important to the least important. When you add a new item to the product backlog, it’s a good idea, as a team, to agree on a stack rank value (like 1000) that you’ll use for new items that haven’t been stacked yet. Also assign the user story (or product backlog item) to yourself so the product owner will know who to talk to when they have questions about the importance of your item.

The idea of agile is to build as much code as you need to deliver the value, but no more. Building more depth into a feature (gold plating) or adding extra features (feature creep) only takes time away from adding more important value. So stay focused on building the minimum code required to satisfy the acceptance criteria for the current sprint, and put all other ideas on the product backlog.

A Young Agile Team

John Socha-Leialoha - MSFT — Sun, 17 Oct 2010 04:25:55 GMT

My team is fairly new to agile. During my first week I talked to the PMs and the developers to get a sense as to where we stood and what the issues were. I first asked about user stories.

Me: “Do you have user stores?”

B: “Yes. We have these three stories.”

Hmmm. Three stories. It didn’t take too long to discover that these were epics rather than sprint-sized user stories. I also looked into the tasks and how they were handling/prioritizing the work. Here is what I found:

They were using two-week sprints. Good.
They were working on one user story for a number of sprints. Not good.
Tasks often covered multiple days of work. Not good.
Developers were checking in at the very end of the sprint. Not good.
Developers often wrote either more than needed, or not enough. Not good.

Improving User Stories

First, we needed more and smaller user stories. I worked with the PMs to break the existing stories into ones small enough to fit into a single sprint. Ideally we want to have 4-6 user stories in a sprint. The format I like to use for a user story is based on the Behavior Driven Development (BDD) community:

Title
In order to business value
As a role
I want to feature

The common format for user stories is pretty close to this: “As a role I want to feature so that business value.” So why change the order? There are several reasons. First, and most importantly, leaving the business value until last means that it’s all to easy to leave it out entirely. I’ve seen many user stories that are really just restatements of features. For agile teams, the business value should be the most important thing because it’s what we exist to deliver. So placing the business value first means you have to write it. Writing the business value is hard, and it takes some practice to learn how to be good at it. But when you do focus on the value first, you’ll often discover that you end up with a very different feature than you had in mind when you first started. That’s a good thing, because it means you’re reacting to the value, not to the feature.

We use Team Foundation Server to store all of our tasks and user stories. We place the main part of the story in the description field. But you don’t want to have to read the entire story when you’re trying to stack rank user stories—it’s just too hard to scan a long story. The title, therefore, is a short-hand for the entire story, something like a short feature description.

Acceptance Criteria

Next I worked with the teams to add acceptance criteria to the user stories. User stories tend to be somewhat abstract, so they leave a lot of room for interpretation. This, of course, leads to problems when a developer builds something and says they’re done, but someone else says “but you didn’t handle this.” Acceptance criteria makes a user story concrete. If your story is about performance, and performing an operation needs to be fast, the acceptance criteria defines what “fast” means. For example, it might say that response time needs to be less than 2 seconds. That’s measureable and testable.

Acceptance criteria also helps constrain the work so you don’t build too much either. This last part is a little hard for programmers to get used to. For some reason, as programmers we think we have to cover all case we can think of. However, all cases may not have equal business value. Good acceptance criteria focuses just on what’s important for this particular user story. If other cases are important, add more user stories on the backlog to cover those cases.

When do you add acceptance criteria to a user story? There are a couple of pieces to this answer. First, acceptance criteria should be in user stories that are going to be in the next sprint. The product owner needs to spend time preparing for the next sprint by working on the acceptance criteria. Often they’ll involve the testers, who in the agile world are customer advocates. But that doesn’t mean the acceptance criteria need to be complete before the sprint planning meeting.

During the sprint planning meeting, the acceptance criteria helps the team have a very focused discussion about what done means for a user story, which helps the team do a better job of estimating the size of the user story. The acceptance criteria will most likely be modified or extended as a result of these discussions.

Testers usually really like the acceptance criteria because they can start to write test cases right away. In fact, each line in the acceptance criteria often maps directly into one or more test cases. Having the test cases early also means programmers can use them to guide the work they do (we’re not there yet as a team).

Working in Stack Rank Order

In the next sprint, we had much better user stories, so we were able to have five user stories in a single sprint for one project (we have three projects). While watching the team during the standups, I noticed that they were not focusing on completing user stories in stack rank order. They looked at the list of tasks and chose one to work on. I’ve been reading the book Coaching Agile Teams by Lyssa Adkins and she recommended that a coach (which is the role I was playing at the time) sit back and let the team succeed or fail on their own. So I sat back and watched.

When we got to the sprint review the team had only finished one of the five user stories. They were almost finished with the other four, but each had tasks not yet completed. I had the feeling that they were about to pat themselves on the back for finishing most of the work. When I pointed out that they had only finished one user story and had to return the other four stories to the backlog, you could hear a pin drop. They were not happy.

This “failure” turned into a win for the team. During the next sprint they really focused on doing the work in stack-rank order, and they paid a lot more attention to finishing one story before moving on to the next.

A Regular Cadence

To say that the testers were being squeezed is an understatement. When developers waited until the end of the sprint to check in their work, there is no way the testers could possibly test anything until the next sprint. As a result, testing was constantly one sprint behind. So my next goal was to get developers to check in more frequently. Ideally I felt they should check in at least after they finish each task, but at worst when they finish a user story.

When we tried this in a sprint, the tester was very happy because he was able to start testing early in the first week, and by the end of the sprint he had tested much of the work done during the sprint. Success!

One Month Later

In just one month, we’ve gotten to the point where:

We have sprint-sized user stories.
We have more user stories, and the backlog is now growing on a regular basis as we think of things that should be in the product
The acceptance criteria is helping the team build just enough, and no more
The testers a much better understanding of what we’re going to build
Testers are able to test much earlier in the sprint, and can test during the sprint
User stories are being implemented in stack rank order

We’re by no means done learning how to do all of this well, but the team has made a lot of progress in a short time. I’m glad we’re using two-week sprints because this learning curve would have been slower if we had longer sprints.

A New Job, a New Journey

John Socha-Leialoha - MSFT — Sun, 17 Oct 2010 00:43:00 GMT

On September 13, 2010, I started a new journey. This was my first day as an official Microsoft employee, and my first day in a new group, MPSD (Managed Platform and Service Delivery), which is a part of System Center. Before this I spent almost 5 years as a vendor here at Microsoft—first as a subject matter expert on mobile development in the patterns & practices group. And then the last 3-1/2 years as a developer in the process team of Team Foundation Server. It was quite a ride, and I’m very proud to have been part of the team that built Team Foundation Server 2010. You can find my blog posts about the work I did there at this location: http://blogs.socha.com/.

So what caused me to switch to a completely different product and a different team? Agile. And my new team found me—they were looking for a senior developer to help them be more agile, and with development practices. I’d been wanting for a while to be involved in making a highly effective agile team, and this appeared to be a chance for me to do just that.

I plan to blog about agile, and also about development. I’m a long-time developer, and I love building applications that customers use, so in addition to agile practices, I’ll also be blogging about programming—both in C# and C++. Before coming to this team, it had been nearly 12 years since I last did any serious C++ development. And since I learned C#, I thought I’d never again enjoy C++. Well, I was wrong, and I’ll have more to say about that in a future blog.