vsts process blog

Team System and Sarbanes Oxley

2008-04-04T22:15:00Z

I'm pleased to say MSDN has just published our paper on Team System and Sarbanes-Oxley (SOX).

There seems to be some confusion around SOX and software development. SOX isn't a standard for software, and no software tool can make a business "SOX compliant". SOX relates to the management of transactions that affect assets, and is undertaken with the help of a qualified appraiser who defines a risk management framework for your business. Some of the risks identified may involve your software development activities. Because Team System closely shadows the software development process, it can be a good platform for gathering data in support of SOX objectives.

The paper looks at several example risk scenarios for a business and suggests ways to use Team System. (The paper doesn't talk about more general functions like auditing with Team System, etc.)

I hope the paper helps, you can find it here:

Sarbanes-Oxley 404 and Team System 2008

Andrew Delin

...more on team processes and innovation at ProcessofInnovation.com

Bugs past, present and future - predicting the bug trend for your next project

2008-03-24T22:07:00Z

Recently we’ve been reviewing what Excel reports we could offer to support TFS customers. We think Excel is a powerful platform that allows manipulation and exploration of your TFS data beyond RDL reports (SQL Server Reporting Services).

The particular report I want to describe is used by some larger teams at Microsoft and is useful for predicting and tracking the expected future bug count on projects that have long standard cycle times (e.g. a year or more) and fixed end dates. In basic form, the Excel chart looks like this:

The blue area represents the predicted active bug count for the whole eight-month project, while the dashed red line tracks the actual active bug count from the start of the project until today. Looking at this chart you might start to ask some probing questions about what was happening on the project: why is there such a divergence between the predicted and the actual bug count in April? We were expecting a bug spike during mid-May when we released a CTP (for example), but why are we getting this big increase in bugs earlier in the project? Researching the answers to these questions could help to unblock testing (i.e. improve the closure rate of active bugs) and could also help inform us about not accepting additional scope or new features until the bug count is more closely tracking our prediction.

Reading the Excel chart this way assumes certain things, most notably that our prediction makes some kind of sense! How can we produce a prediction of future active bug counts? As mentioned above, the technique relies on certain fundamentals including planning to a fixed end-date and having comparable cycle times and consistent milestones in each major release of the software. The ability to predict bugs for a future cycle of a project depends on being able baseline the rates at which the team can both produce and fix bugs. This means we need to know the number of testers (bug producers) and developers (bug fixers) and the estimated ability of the team to raise and resolve bugs each week. These latter numbers will vary as the team moves through different activities during the project (setup, development, stabilization, CTP, more development, etc).

The Excel spreadsheet is structured in this form, as a series of weekly data predictions about opening and closing bugs given the size of the testing and development teams:

For each week we record the number of testers, based on what we know about our resourcing. Below this we calculate the maximum number of bugs we expect a testing team of this size to be able to open, which is based on the greatest number of new bugs opened in any week during the previous cycle of the project. Next we estimate the percentage ability of the testing team to apply effort to bug opening during each week – this may vary depending on the other testing activities (such as test setup, server configuration, etc., which are not bug-finding activities as such). We also record the actual percentage effort the test team was able to apply to opening bugs. Following this you can see the projected number of bugs we expect the testing team to be able to open this week, which is a simple calculation based on the numbers above. Finally we record the actual number of bugs opened this week, and the cumulative count of open bugs as at this week.

The Excel cells for the development team are similar to the testing team.

The following is a chart from a project, showing the original ‘locked’ bug prediction based on our estimated bug generation and resolution rates over time, and our current tracking against the prediction (the thick red line). The original prediction is fixed so that you can continue to track actual against the original estimate. In addition, you can see a dotted red line which plots our most recent estimation of the future bug count.

You will note a couple of milestones have been added to the chart, Beta 2 and Beta 3 (these are simply drawing objects added to the chart for annotation). While the chart described so far allows for bugs produced and closed by our team, the issuing of a Beta causes external bugs (from outside the team) to be delivered in greater numbers. For this reason we extend the Excel data collection sheet to add an additional column each week, being the number of non-test-team bugs opened. Whenever a build is issued for wider examination (CTP, Beta, etc) we expect to see an increase in bugs coming from outside the test team, so we record this in the workbook prediction as follows:

Charts of this nature are useful because of the questions they cause us to ask. Bug counts can be considered one form of proxy for progress of the solution under development. For example, we may decide, based on a large divergence from our predicted bug count, that we won’t accept additional scope to be added to the project until we have understood in more detail why we are diverging from our original bug trend prediction.

With projects that have a reasonably standard cycle associated with development and release, we can plot the bug trends of several releases on the same chart to examine the profile of bug counts over many versions. This encourages us to try to understand the behavior of projects. In combination with other reports/charts from TFS we can shed light on the complex flow of software development.

Here is a chart showing two past versions of a project (v4 in blue, and v5 in orange). The current development (v6) is shown by the dotted line in red:

You can see the effect of the various milestone dates on the past bug counts – the counts rise shortly after the release of a Beta version. We have also added predicted dates for the v6 Beta milestones (red vertical bars) and added our estimates of increased external bug counts at those milestones.

Some teams at Microsoft use these Excel-based reports as one tool to help understand the progress of complex developments and to signal problems for further investigation. Would such an Excel report be useful to you? Should we ship something like this? How should we deliver it? We could develop the report above into an in-box offering – let us know if you think this is a good idea. Our view is that this is a non-trivial workbook to setup and maintain, being more of a tool than a report, and is applicable to a particular style of management of larger, fixed cycle projects.

We’d be interested to hear feedback on this, and also how you predict bug counts on your larger projects – or any other techniques you have developed. I look forward to your comments.

What sample docs do you want? What's useful?

2007-11-14T04:45:00Z

On the VSTS process team we're currently looking at the next round of Sample documents to include in a future Agile template.

We'd be very interested to know what you'd like to see included.

Our goal is to be "broadly applicable", which means we extend beyond what might be considered 'pure agile'.

This is what we have in mind today:

Vision document and solution concept
Project start-up checklist
Personas defined
Design artefacts capturing design decisions and directions chosen
Data model & Use cases
Risk plans connected with risk work items
Threat models
Logistics / roll-out plans (Operations)
Test plans
Triage meeting notes - decisions and justifications
Iteration retrospective – meeting notes and documented learnings/changes to team working
Stakeholder matrix

Let us have your feedback. What's useful? What do you use on your teams? What should we include (or not)?

Andrew

So tell us what you think... about the MSF guidance

2007-09-24T20:17:00Z

Recently I've been talking with the MSF team about the future of the MSF guidance in Team System. Two themes have come up: firstly, how to make it better, more focussed, more relevant. And secondly, how to make the MSF guidance "enactable", in other words to have it appear in the right places in the shell, when you need it, to help the team's efforts.

To get this right, we really need to hear from you. We'd love to have your feedback on what you think about the MSF guidance today, and what needs to be improved or fixed.

If you have 5 minutes to spare, please take our "MSF guidance experience" survey and let us know what you really think.

Start the survey here.

(When I say 'MSF guidance', I mean these HTML pages that you can access from the Team Explorer tree and also in your browser)

Fragile Agile?

2007-09-12T07:40:00Z

Recently I reviewed a troubled project that over-ran (and over-spent) in delivering a custom Sharepoint solution to a customer. The project claimed to be "agile", by which they meant there was very little structure around the development process.

Looking deeper into the behaviour of this project, it was clear that certain fundamental Agile practices weren't in place. Notably, there was a failure to "get specific early" by writing decent user scenarios. These scenarios are intended to drive sufficient design activity so that development tasks can be articulated in enough detail. I call this "earthing the design", just like you ground an electric circuit. If the scenario isn't sufficiently detailed and the corresponding design is improperly resolved, a lot of churn will follow. This quickly leads to customer dissatisfaction.

("How detailed should my scenarios be?" There's a thread I like on this topic in the MSF forum, here.)

I also found this project had minimal testing practices and an immature approach to work backlog. So the measurement and transparency of progress were missing. The results of the final solution speak volumes: there were lots of areas where functionality didn't work properly, and a lot of post-implementation rework was required. The solution was fragile, brittle.

It's an unfortunate thing when a project is labelled "agile" under such circumstances; instead, it was really "unstructured".

The discipline of Agile development is in the execution of team practices that are intended to capture and clarify specific scenarios of value, to illuminate design direction, enable incremental coding, and promote testing as a transparent indicator of progress to all stakeholders.

Our aim with MSF is to make these things easier by having the tools 'understand' these practices and to support the team in applying them as they work: we call this 'process enactment'.

This led me to thinking whether using the MSF Agile template in Team System would have helped this customer? They were new to the Agile space and could have used some guidance to get started. Would our Agile offering have been helpful, or would it have been hard to understand and apply?

I'd be interested to hear any feedback on how easy or hard it is to use the MSF Agile template for the first time: the more specific the feedback, the better.

Hello, (process) world!

2007-09-07T18:34:00Z

By way of introduction, my name is Andrew Delin and I am the new product planner for the Microsoft Solutions Framework (MSF) on the Patterns and Practices (P&P) team at Redmond, responsible for the direction of MSF in Visual Studio Team System. I joined from the Microsoft Services organisation and have a long history of advising customer software projects about getting (and staying) on track. Like many of you, I've felt both the success and failure of team software development efforts...

I look forward to listening and sharing ideas with the process community, that is anyone who cares about how teams perform the task of software development and how the tools could support your chosen methodology better.

I am very excited to have joined P&P: this is a group that has the ability to offer both in-sync and out-of-band content releases - we don't have to wait for the next major product cycle.

More to follow. Please feel free to contact me via this blog or through the MSF forum.

Andrew Delin