Adventures in Software Engineering

Drinking from the fire hose – a retrospective

This year has been such a continuous wave that I am still catching my breath. How do I start?

I had a brief stint with the MSF Team (now called VSTS Process Team) that finished in October 2007. I wanted but couldn’t stay – my family is not yet ready to move from sunny Austin to cloudy Seattle. Maybe in the future. I joined the ALM team in MCS in early December 2007, as it was a natural follow up to the work I was already doing.

With ALM team my first work was on I presenting the talk “Using Key Performance Indicators (KPIs) to streamline development” in December and January of 2008. I also helped in the internal review of the SOX whitepaper.

In January and February I went a lot to the West Coast, both North and South, and was able to meet with a several development teams newly adopting TFS. I couple of them were still getting out of the woods in using waterfall SDLCs. In one of the cases, it was not because they didn’t know about Agile or iterative/incremental software development, but instead the platform and tools they were using imposed a waterfall structure, and not just at a governance level, but at the day to day development activities. What if took you a month and a half to create new build because your product consisted of a 4GL that generated 15K+ (that’s 15 thousand!) DLLs? Unsuitable technology trumps Agility. (For more on that, take a look at Kent Beck’s paper on “Tools for Agility”.)

I also had the opportunity to review and help recover a couple of projects with issues. One of those issues was a lack of understanding that the adoption of better practices in ALM has a cycle of its own. Tie it to a major development project with a different objective, and it will be run over. Choosing a proper pilot for ALM adoption is a major decision that if left to whimsical decisions taken in a tactical mode will derail both projects, the major application development and the ALM process improvement one. This has been by far the biggest cause for failure in adopting newer ALM practices.

In February, right after our internal TechReady event I went to the Netherlands to teach a workshop in MSF Agile and MSF CMMI to Europol. I was impressed by the quality of their development expertise. It doesn’t seem an isolated case but rather similar to other areas of Europe. MSF CMMI adoption seems higher in Europe than in the US. The message that it is possible to apply CMMI to an Agile process has been understood at heart over there.

Meanwhile back in the US I was at a customer helping them develop a customized version of FDD for TFS. FDD was well suited for their process as they were creating a major application framework to be reused by hundreds of other applications. An architectural design cycle fit it like a glove. It also reminded me that given the diversity of the original Agile methodologies, that there is no clear cut answer to process adoption. You make the process work for you. On that I subscribe to Alistair Cockburn’s philosophy of “a process per project team”. It’s the same as in life -  living things are not exact copies. They exhibit variances here and there. There is enough to see that they share a common inheritance, but the uniqueness of time, location, team and project makes them individuals with their own identity.

Then in mid-March Ajoy Krishnamurthy called me back to help represent Microsoft at SEPG 2008. For the first time I saw a reversal in a trend I had been observing since 2005: there were many more young faces at the conference. For 3 years while I was covering both the Agile and SEPG conferences I had the following experiences: I would feel old in the Agile one, and young at SEPG (the cynics will say I just got 3 years older, so now everyone seemed younger).

It seems that the renewal approach from SEI has been paying off, and its support to out of the box thinking in connecting Agile and CMMI is also bringing new faces to the table. Among others we had Microsoft’s input with David Anderson’s revolutionary approach in MSF CMMI. At SEPG 2008 I met [again] David Anderson and Hillel Glazer, who would by November last release with other SEPG members a widely circulated paper on Agile and CMMI.

I delved deep for the next 10 months in the deliveries of the ALM team:

• Application Lifecycle Management Assessment
• Application Lifecycle Management Team Foundation Server Production Pilot
• Application Lifecycle Management Team Foundation Server Lab Pilot
• Application Lifecycle Management Build Management Jumpstart
• Application Lifecycle Management Release Management
• Application Lifecycle Management Workshops for Project Management

It’s been a nice mixture of consulting, training and coaching. A common trend has surfaced from talking to so many different customers in so many different situations:

• Public or semi-public companies currently have their sweet spot for ALM process improvement in adopting TFS as a source control tool, and look forward to implement some sort of Release Management discipline.
• Private companies have implemented TFS as a SCM tool a while ago, and are now moving into adopting TFS to manage their Application Project Portfolio. Aside from the occasional new project,  they have a standard set of developed applications that are mostly in maintenance, and need a place to track change requests and bugs. They are almost Agile, but are held back by siloed requirements/change request “phases”. On top of that, there is rarely the concept of a multidisciplinary Agile team, with the same team member fulfilling all conflicting quality goals which leads to a tactical prioritization of customer needs.
• ISVs have a firm grasp of their process, and are primarily concerned about using TFS to achieve more productivity by establishing a Metrics program as part of a wider ALM optimization effort.

This is of course based on just my experience with a few dozen customers, and I can name several exceptions already. This induced perception however has led me to realize that I have had the privilege of getting first hand experience in improving over ALM baselines, at several adoption phases and at different levels of maturity. Having been part in implementing those steps has definitely helped me in assisting other customers as it is now easier to weave them into a continuous path for improvement.

So the “fire hose” is in reality this incessant learning experience with customers all over the place to enact improved ALM processes using Team Foundation Server. More than that: I feel like I have been watching, from the first row, the dawn of a new way of working with software development using the Microsoft platform. It reminds me the old days in 1996 when VSS was a novelty. “So you don’t use source control? Here’s VSS. It will double your productivity”. And it made a difference. After a couple of years, you couldn’t call yourself a professional developer if you didn’t use some kind of source control.

And today the same is happening with ALM 2.0: a few years from now, you won’t be called a professional developer unless you have a set of integrated tools that will finally make the overhead of capturing project management metrics a thing of the past. Better and more transparent tools such as TFS will fade in the background leading to the enactment of that Agile pillar principle: Individuals and interactions over processes and tools.

Implementing SOX with TFS

I touched on this topic on a previous post in December, when I mentioned that Andrew Delin from the VSTS Process team was working on a comprehensive whitepaper to be delivered sometime in 2008.

And here it is: Sarbanes-Oxley 404 and Visual Studio Team System 2008. There isn't a one-size-fits-all solution for SOX implementation with TFS, but this paper provides the guidelines that will allow you to sort out the many implementation options. Enjoy.

Error 29112 when installing TFS 2008 and Reporting Services as part of a scale-out installation

Normally I would post the resolution for this issue in the appropriate VSTS/TFS forum, but since posting there does not include pictures, here it goes.

Note: Error 29112 is a catch-all code. This blog post only handles one possible case. A blog that also mentions other causes for this error is Will Buffington's WebLog, and one TFS forum thread about it is Team Foundation Server - Installation Error 29112.

Error message:

---------------------------
Microsoft Visual Studio 2008 Team Foundation Server Setup
---------------------------
Error 29112.Team Foundation Report Server Configuration: Either SQL Reporting Services is not properly configured, or the Reporting Services Web site could not be reached.  Use the Reporting Services Configuration tool to confirm that SQL Reporting Services is configured properly and that the Reporting Service Web site can be reached, and then run the installation again. For more information, see the Team Foundation Installation Guide.
---------------------------
Retry   Cancel  
---------------------------

Right before this in the installation log was the following message:

"TFSUI: [2] wsschecker.exe : *** ERROR: Unauthorized access of sharepoint url http://xwyz/: The remote server returned an error: (401) Unauthorized."

Context: I was trying to install SSRS as part of a scale-out SSRS installation.

Troubleshooting steps: By looking at the Reporting Services Configuration Manager, I found out that there was probably some version mismatch.

The highlighted message in the picture says:

"You specified a connection to a report server database that contains encryption keys for another report server. If you are configuring a scale-out deployment, that feature is not supported by this edition of Reporting Services. If you want to use this report server database with the current report server instance, remove the existing encryption keys first."

I checked out and found out that the customer was using SQL Server Standard Edition on the TFS application tier, while the database tier already had SQL Server Enterprise edition.

From the documentation on Configuring a Report Server Scale-Out Deployment:

"The Reporting Services edition must be Enterprise, Developer, or Evaluation. Standard edition does not support a scale-out deployment. You can create a scale-out deployment using a combination of editions as long as the edition supports the scale-out feature."

So here was the issue. However, the TFS 2008 installation program does not details this and aborts with error 29112.

Resolution:  I reinstalled the app tier SSRS with a SQL 2005 Enterprise version, and the installation was able to finish.

An old metaphor for project management

Our culture owes a lot to the influence of Newtonian mechanics in shaping the thinking in other areas, such as in Psychology with the notion of "energy". Even though this usage is obsolete within professional circles, the expression is still an active part of the popular culture.

The fabric of our thinking also has another Newtonian concept as a background: the mechanical "static equilibrium". We usually see stability in a way that is contradictory to what happens in life, where equilibrium is a delicate balancing of multiple forces woven in a dynamic exchange.

This cultural influence extends to ideas on project management. We tend to see projects as a collection of static endeavors where everything is predictable, punctuated in between with checkpoints at which some change might be added, in a predictable way, into the system.

I would rather refer to Heraclitus "everything changes" idea when thinking about project management. And that implies continuous adaptation, not just at staggered checkpoints or milestones.

Juggling has always been a great metaphor for project management, for although we don't think about it while watching their mesmerizing performance, jugglers are constantly adapting to where the balls go. Even though they kind of have an idea of a ball's trajectory through their experience, their greatest skill is going after it in the nick of time.

The following video by Fatboy Slim is a homage to adaptive project managers of all times, from Heraclitus to Agilistas today, with eyes on the ball and ears on the rhythm. Enjoy...

Keep your SOX clean

I have been to a few customers who have implemented or are implementing Sarbanes-Oxley (SarbOx or SOX) compliance in their development processes using VSTS. Andrew Delin from the VSTS Process team is creating a whitepaper on how to do that with VSTS. In the meantime, here are some reflections based on my personal work with this topic so far.

[The next is a PPT-like intro to the topic. For those who know what SOX, you can skip it].

What is SOX?

• Federal legislation signed into law in July 2002
• It requires higher accounting standards, improved trustworthiness in corporate reporting, and greater financial transparency
• Two key sections of the law that have drawn the most attention
  • Section 302: Requires executives to personally certify the validity of financial statements
  • Section 404: Requires complete documentation of financial controls and auditor attestation to management's evaluation

Section 404

Requires “an internal control report, which shall

1) State the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting;

and

2) Contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.”

[end of introduction]

Ok, given this very brief summary, I can now tell you that the best general guide I have found so far to understand how to implement SOX in an IT environment is "IT Control Objectives for Sarbanes-Oxley, 2nd Edition".

This book explains the rationale for establishing the controls needed from the IT perspective, starting with SEC's own recommendation:

"Historically, assertions on control by an organization have been mostly voluntary and based on a wide variety of internal control frameworks. To improve consistency and quality, the SEC mandated the use of a recognized internal control framework established by a body or group that has followed due-process procedures, including the broad distribution of the framework for public comment. Specifically, the SEC referred to COSO".

and

"For Sarbanes-Oxley compliance efforts, it is important to demonstrate how IT controls support the COSO framework. An organization should have IT control competency in all five of the components COSO identifies as essential for effective internal control. They are:
• Control environment
• Risk assessment
• Control activities
• Information and communication
• Monitoring"

How does that relate to the normal IT framework controls that we are used to, such as ITIL/MOF, and SDLCs such as MSF for CMMI Process Improvement?

Here is a short summary plot:

• SOX recommends COSO per SEC
• COSO maps to COBIT (Control Objectives for Information and related Technology) standard
• portions of COBIT map to relevant parts of CMMI
• other parts of COBIT map to ITIL and other IT management standards

Said in this way it would seem that by implementing ITIL/MOF, and by using MSF CMMI as the standard SDLC, we would be covered in SOX compliance. This seems like a lot of overhead. However, you don't need all that, as we will see next.

SOX is about financial reporting

This was very eloquently mentioned by Dave Erickson:

“Sarbox is about assessing risk. While risk assessment is an element of ITIL, it isn’t the framework’s primary focus. Furthermore, CIOs who put ITIL or any other IT framework in place solely to comply with Sarbox will have gone overboard, says Erickson. The Sarbanes-Oxley Act requires only that companies establish controls over the systems relating directly to financial reporting. ITIL, Cobit and other frameworks for IT help companies put in place general controls for IT—a good thing to have, but much broader than the narrow scope required by law.”

So one of the first things that needs to be established from an IT perspective is a control that identifies the application being developed as impacting financial reporting. These type of applications will need to follow SOX constraints. Other types of application do not need all the overhead, especially if you are doing Agile development.

Usually SOX compliance teams will keep their own database of such applications. In VSTS it is possible to create a work item to identify those for reporting purposes. That would be the first of several work items that might be needed for SOX compliance.

So given that part of what is needed in already in the MSF CMMI template, it is clear that a few items need improvement. Remember that this just a sample of what might be needed, not a comprehensive list:

• Strategic planning alignment
  • One way to implement this might be to have new work items explicitly owned by upper management and/or CIO, and mapped to strategic objectives (such as in Microsoft's Development Division with Value Propositions/Experiences/Features)
• Risk management process
  • We need to add risk reports per project and across portfolio (slice risk management by financial management applications)
• Traceability
  • We need to implement reports that show traceability of work items that impact financial reporting. This will be easier to do with
    • Adding new fields to work items (such as a task work item with a tag “SOX regulation” )
    • Adding work items that have have more workflow steps to deal with regulations
• SCM (as part of change management)
  • Add work items that correspond to checkpoints for branching (see article by John Jacob et alii on branching guidance)
• Audit trails
  • Have additional reportable fields, pivoted with the SOX attribute, and provide more reports for auditors
• Security
  • Existing process guidance already handles part of this, but it is not enacted in tooling
  • We need to implement Secure Development Lifecycle with work items as checkpoints, and corresponding work products and reports

As mentioned above, another big part of SOX compliance is covered by ITIL/MOF. I won't go into the infrastructure topics per se (see the book above for that), but there is one clear common implementation point with VSTS/TFS/MSF CMMI in security groups. Segregation of duties is mandated by SOX. However the currently default process template security groups are loosely defined, allowing persons without the proper authority to review/modify documents.

• The full implementation of security model described in MSDN documentation is a solution.
• Reporting needs enhancement to provide evidence of compliance showing that groups are separated in their duties.

Finally, part of SOX compliance is covered by IT Portfolio Management. Therefore, reporting needs enhancement to provide evidence of compliance using, for instance, a portfolio view of a dashboard showing compliance status. This view could used departments as pivots.

So as I mentioned above, these are just initial thoughts in a very complex topic. Andrew Delin and the VSTS Process team are working on getting more comprehensive guidance on how to tackle this subject.

Presentation on ALM foundational concepts

I did a presentation for the VSTS Inner Circle in September 11th, and I am still getting requests for the video link and slides. Here they go:

Fundamentals of ALM

Abstract: What you should know to elevate an enterprise to an intermediate or higher level of maturity regarding SDLC and ALM. Includes discussion of the features of VSTS that enable integrated ALM, and an overview of what is coming in the next couple versions of VSTS (Orcas and Rosario).

View Recording
Recording Details
    Subject: Fundamentals of ALM
    Recording URL: https://www112.livemeeting.com/cc/microsoft/view
    Recording ID: K7K7ZZ
    Attendee Key: PFSN5?2$m

This presentation has a five minute delay to start (recording started too soon). I have asked the organizers to edit those minutes out, and I will post the link to the edited version when it is available.

I want to thank Sam Guckenheimer who co-authored an earlier version of this deck which was co-presented at TechReady 4 (an internal Microsoft conference).

Guidelines to choose your ALM pilot project and pitfalls to avoid

Some Agile and/or ALM adoption efforts are canceled midstream because of lack of understanding of the basics of finding a suitable candidate development project. I have seen in more than a single situation that the chosen project is cutting edge in all three aspects of technology, process and people:

• The technology is brand new, or new to the team, sometimes in even more than a single tier (for instance, new database software coupled with new UI development tools and a new programming language)
• The development process is being changed (say from waterfall to Agile)
• New people are being added to the team just after receiving their first training in the new technology

But the biggest mistake with Pilot efforts is to to use a strategic, high profile brand new project as the proofing ground for all these aspects, all at the same time. This is more common than expected. It starts as something like this:

1. Business has some urgent need for strategic functionality that allows them to challenge the existing technical architecture
2. However, the effort still has to abide by the usual existing waterfall processes that dictate that all must be done in a single pass
3. So the project is approved, but no cycle is allowed to try out the new tools and processes in a smaller context , and multiple changes to the environment are bundled together in an insurmountable ticking bomb that will later explode as a "death march" project.

To add insult to injury, sometimes on top of all this no proof-of-concept is ever tried with the new technology and processes (Proof-of-concept differs from pilot in that it is done in a lab environment, with no impact on business). Pilot projects do have business justification, but usually one chooses a minor project instead of betting the "jewels of the crown" on risk upon risk.

The mistake on all these lies usually in the governance management tier(PMO, office of the CIO, etc). It is usually associated with just enforcing the status quo, and it takes some brand new business need to act as a catalyst to challenge it. This governance tier should be the one to understand how to evolve their environment through carefully taken steps, and to know how to spread the risk underlying the business need into preparatory small projects (using proof of concepts and pilots) that will pave the ground for more ambitious ones.

If a governance tier is not active in doing this, the new project decays into a rogue that just reinforces the "didn't tell you so" attitude of those who see governance only as keeping IT madness in straightjackets.

Allowing this to happen is like building on moving sand: the construction might be impeccable but will collapse upon itself if it doesn't have firm ground to support the pressure of adding new layers.

The best practices for selection of a Pilot project are widely known, and for quite a long time. Here is an excerpt from a Microsoft Official Curriculum course from 1993. It is part of Course 124 - Managing the Migration to Client-Server Architectures. I modified the text to fit ALM adoption (the text in brackets [] replaces "client-server" and updates the context of other points):

"Start small - with a Pilot Project

We suggest you start your exploration of [new ALM processes and tools] with a pilot project.

• Maintain excitement:
  • Sponsors will lose enthusiasm
  • Team members will lose enthusiasm
  • Reduce risk of turnover
• Need strategic answers quickly to be of value.
• Avoid management problems of large projects:
  • Large projects require more layers of management
  • Coordination of client developers and server developers is critical
  • Coordination will be much easier in a small group that talks to each other

Selection Criteria of Pilot Projects

• Well defined data requirements
  • Don't want to get bogged down in data analysis
  • Could be existing application
  • Could be part of a new application, where data analysis has been completed
• Benchmark available
  • If don't have, need to build in-house benchmarking capability
  • Performance criteria
  • Quantify savings and benefits
  • Define ball-park expectation
  • Use to validate tool selection
  • Use for quality control in future projects
• Decision support application [Business Intelligence in today's jargon] as opposed to data entry application
  • More showy, if that's what's needed
  • Safe place to start - it won't disrupt business operations
  • Usually a simpler system
  • Deliverable flexibility - keep concentration on testing the [ALM processes and tools]
• Committed and supportive users
  • Might be #1 critical success factor [that includes not only end users of the application in the role of product managers, but also developers, project managers and upper management]
  • Willing to work with the team
  • Willing to allocate the time required for the project
  • Could use internal IT system so "end users" are IT
• Low Cost
  • Use equipment you already have [for instance, VPCs]
  • Look for idle equipment [for instance, a PC with Windows XP can be a build server for a small project]
• Low Risk
  • It's better if this might be considered a throw-away project
    • Objective is to evaluate [new ALM processes and tools], not build an application. Concentrate on tools and platform rather than application development"
  • If you need to choose a project that is mission critical, at least let it not be time-critical

As you can see, those best practices are nothing more than codified common sense, and I highly recommend you have those in mind when scoping your next ALM project.

Busy with VSTS and TFS

It has been a busy year, and as any Agilista will tell you, it is time for a retrospective. Here is a picture of what my mind has been over the last year - all nice and fun, but very busy:

Well, that's not actually the picture - my mind is more organized than that J. So here is what I have been up to (BTW thanks Steven Borg for the photo).

Last November I presented at TechEd Brazil on MSF CMMI, and wrote a chapter on MSF Agile (in Portuguese) for a VSTS book led by Brazilian MVP Fábio Câmara, plus several other authors.

Besides consulting engagements, I worked with the creator of MSF Agile, Randy Miller (now with MCS) on a MSF Agile training course, and helped my team create an "ALM Assessment", a version of which is now online at www.microsoft.com/almassessment.

Then in January I presented at TechReady (an internal Microsoft conference) with Sam Guckenheimer on "Fundamentals of ALM", and learned about the latest and greatest in upcoming Microsoft technologies in sessions raging from VSTS Orcas and Rosario, to WCF and others.

Soon after that I embarked on a 8-month stint with the MSF Team (now called "VSTS Process" team) as one of their product planners, while we were waiting to get Andrew Delin on board.

In February I had fun at SEPG 2007 in Austin (where I live - it's nice to have a major conference in your city once in a while). I was at the Microsoft booth talking about MSF CMMI and TFS, and also had interesting conversations with other booth presenters: Osellus (on the future of process enactment and authoring), Fujitsu (on how Macroscope for VSTS uses VSTS/TFS/Project), and Personify Design (with the new products to manage work items from Outlook, and requirements from Word).

At the conference I had the opportunity to talk a lot with David Anderson, the Architect of the MSF CMMI process template (now at Corbis), as well as Mike Konrad and Paul Nielsen from SEI on the past, present and future of MSF CMMI. I also met Hillel Glazer, one of the few fully certified assessors who also works with Agile development in depth.

In March I was at the SxSW Music and Media conference. I helped at the Microsoft booth for the Happy Hour sponsored by Microsoft Expression Studio where I had the opportunity of again talking to the Usability guru Chris Bernard about creating a UX whitepaper for MSF Agile (which is the only Agile methodology AFAIK to explicitly adopt Personas and Scenarios for User Experience).

While with the MSF product team my focus was especially on getting customer feedback. I helped to coordinate a workshop on Reporting in March, monthly calls to the MSF Council members, and hundreds of discussions on how to enact process in VSTS. I have also been in the MSF Forum a lot. After all this and two SDRs (Software Design Reviews) we got great feedback on how we can create better process templates for VSTS.

Then in June I presented at TechEd 2007 on MSF CMMI, helped at the Patterns and Practices booth, and rubbed shoulders with Ivar Jacobson from IJC, Sam Guckenheimer and Ajoy Krishnamoorthy while talking on the cool ESSup implementation for VSTS, Mike Azocar (creator of a lightweight Scrum process template), Colin Bird from Conchango (one of the authors of the first and most widely used Scrum process template), and a host of other VSTS MVPs, among them Chris Menegay, Steven Borg, Will Stott, Martin Donnell, Joel Semeniuk, Richard Hundhausen, Jeff Levinson, Jean-Luc David, Juan Perez, and customers from the MSF Council such as Brian Hinton and Wayne Miller.

Then at the end of the June/early July I worked from London for a week and a half on the process templates redesign with Ian Spence from IJC, and Alan Wills. We also worked on the CMMI 1.2 update and SOX for the next version of TFS (Rosario time frame). A second session on this work was done in September, this time with Andrew Delin as well. I couldn't travel for this latter one, so I worked with the London team from 2:00 AM to 11:00 AM CST every day - an interesting experience of remote work using Live Meeting.

In August I followed the "Scaling Agile" topic at Agile 2007 in Washington DC - an interesting topic which I am still working on with a few other attendees and customers. The best presentation was from Sanjiv Augustine on "Transitioning to Agile Project Management" as he showed how to avoid the friction between PMs using traditional techniques, and the new agile thinking as a company scales Agile from small teams to reach the whole enterprise.

Then I switched gears back to the ALM business - this time with a nation-wide ALM Business team. Last September I did a talk to the VSTS Inner Circle, again on "ALM Fundamentals" (recording started too early - you will have to wait 5 minutes for it to begin. I might post a trimmed version later).

October was the "bug month", not of the software kind, but of the "being sick" kind, which put me out of work for two weeks (one of them with a continuous migraine that now makes me see with even more sympathy those that have this condition).

Finally, in early November I participated of the Alt.Net conference in Austin - a great forum for users who want to use OSS tools for .NET development.

So what is the latest?  I will be doing the keynote in São Paulo on the 4th of December on "Delivering Value Through Application Lifecycle Management" at the Simpros (International Symposium of Process Improvement for Software)  event, meet a few customers and then if I still have time, pass by TechEd Brazil. 

I am now working on a couple of workshops on MSF Agile and MSF CMMI Project Management, and a webcast on "Using KPIs to Streamline Development" with VSTS on the 13th of December, all that while engaged on a few VSTS ALM Projects. I can't complain of not having work to do J....

All in all, in retrospective it has been a very productive year, and hopefully I will repeat the dosage as I delve even more on implementing Software Engineering best practices with Visual Studio Team System.

Why have a separate risk work item

The XP literature includes sorting work items by risk, although it doesn’t seem to be the current standard: William Wake’s is a well known Agilista that has included it in his early but very nice and still useful XP description in here. This February 2000 article is the basis for chapter 7 of his book, Extreme Programming Explored.

Notice that when the book was published in 2001, he cut out the topic on sorting work items by risk, and commented in his book that “[o]lder descriptions of release planning described the programmers sorting the stories by risk. Many teams don’t bother with this; most of the risk seems to show up in longer estimates, so the sorting is redundant. I recommend that a team think about ways to reduce risk in longer stories, but allow the customer to choose the order of the stories.”

That said, I agree that we don’t need to attach a Risk field in Stories/Scenarios. The ROM (Rough Order of Magnitude) field implicitly covers the overall impact anyway.

The Risk work item exists in MSF Agile and MSF CMMI for a different reason: to allow the management of “environmental” risks (including iffy architectural elements, and complexity underlying too many scenarios) other than straightforward “coding impact risks” such as in Bill Wake’s examples.

Here is an example of a typical environmental risk: say a USA-based (living in Dallas) employee in your team has a pending greencard receipt but has some work to do on site in France. He won’t be able to travel in case the receipt doesn’t arrive in time. So the project manager adds this risk to the backlog, and a contingency plan for this situation is crafted (“Plan B: work using LiveMeeting from Dallas, from 2:00 am to 11:00 am”), and a trigger is put in place (“if he doesn’t get the receipt by week X, cancel the trip and follow Plan B”).

So why add those types of risk work items to the backlog? Because Agile is about transparency and trust, and adding it there makes it clear to everyone in the team, including the customer, that there are risks that need to be handled along with requirements. It is the team’s responsibility to establish the Impact of a risk, but when it comes to establishing its Priority, MSF Agile is mute on that. I personally like to allow the customer to prioritize risks along with scenarios and QoS requirements, so she shares the team vision on why it is important to mitigate them.

Earlier SDLCs such as Evo from Tom Gilb, and Spiral from Barry Boehm were heavily risk-driven on all aspects, including technical risk. Initial versions of MSF (circa 1993) were allegedly the same, but in practice it followed the same approach that Bill Wake describes. MSF Agile is an offspring from XP with a MSF slant, so risk management is less emphasized but still present. On the other hand, MSF CMMI implements the standard MSFv4 process more fully to comply with CMMI.

All said, Risk work items have a place in software development, and if possible we should implement the minimum “Probability X  Impact = Exposure” relationship, as it is the least common denominator when properly talking about risk management.

BTW, my favorite piece of canonical XP literature on Risk is the excellent article by Erdogmus and Favaro in chapter 43 of Extreme Programming Perspectives, where they describe XP as an “options-driven process”. It maps neatly to the “risk-driven management” thinking that drove the creation of the Team, Risk Management and Process Models from MSF 1.0  in 1993.

Agile Austin inauguration was a huge success

The first meeting happened last 4th of September, and you can get more details at the Agile Austin website.

In the first talk, Jim Van Riper, VP of Product Planning and Development from Troux Technologies, shared the inside story of adopting Agile in a fast changing environment.

Jim's thinking about Product Management is also interesting: they should always be "Pigs" (in Scrum parlance), not "Chickens". In my experience with MSF in general, and MSF Agile in particular, is that it solves this lack of Product Management involvement by having the Product Manager as someone from the customer side, and as an integral part of the team at all times, not just when interfacing with project managers.

One of his quotes ("turn over is good for you") called my attention when he explained that most of those who could not adapt to Agile chose to leave - and that the company was better after that. Jeffrey Palermo's blog has more on the meeting.

I don't think that even the organizers were expecting to see such a full house with people standing inside and outside of the conference room. This tells a lot about the need for such a community in Austin.

Congratulations to all who put this together, among them Scott Killen and Jeffrey Palermo.

TechEd 2007: Presentation on Agile Software Development for small teams was a hit

There was a session I forgot to mention from my post yesterday, but that was one of the best. Here is the complete summary so you have an idea of what was covered:

DEV02-TLC - Microsoft Visual Studio Team System for Small Agile Teams

   Monday, June 4 4:45 PM - 6:00 PM, Blue Theater 13 

Speaker(s): Will Stott

Everyone says that Agile approaches like Extreme Programming only work when you’ve got a small team of highly talented developers. I agree that Agile works best for small teams, but it’s not talent that really matters, it’s a willingness to adopt certain practices and values. Over the next forty minutes I want to explain how Visual Studio Team System (VSTS) supports the Agile practices and values that allow everyone in this room to create a top performing small team. Topics to be covered include: • The case for creating small Agile teams – how Agile provides a middle-way between having too much process and too little • Adapting the Team Foundation Process Framework – how to create a light weight process for a small Agile team doing Extreme Programming. • Setting-up Team Foundation Build – why frequent integration is important and how to implement practices like ‘Continuous Integration’ and the ‘Ten Minute Build’ with TFB. • Test-driven Development – demonstration of the new VSTS Tools in a Pair-programming session (James Newkirk / Will Stott) with particular emphasis on Refactoring and writing good programmer tests. • Customer Testing using Generic Tests and FIT – how functional tests written by Customers in a Word document can be run as part of an automated build process to verify that your code satisfies its business requirements. James Newkirk and Will Stott have written a book entitled ‘Visual Studio Team System – Better Software Development for Agile Teams’ (700 pages) which Addison-Wesley will launch at Tech·Ed 2007. This book is targeted at people who want to know how VSTS might be used by a small team transitioning to Extreme Programming and includes a series of exercises which take readers through the entire software development lifecycle from inception to deployment.

I arrived there on time, and there were already as many people outside as inside the presentation space.

You could actually see people standing on top of chairs to see and hear what he was presenting. Some people watched the whole presentation while standing up,  which is quite a compliment.

Doug Seven replaced Jim Newkirk as a co-presenter, as his flight was late.

Will and Jim had just launched their new book (as mentioned in the summary "‘Visual Studio Team System – Better Software Development for Agile Teams". The companion site (where you can get a link to buy it from Amazon.com) is here.

As part of this work, Will and Jim released a new software process template for Visual Studio Team System, MSF for XP.

I will blog again at length about this book later. Right now I can tell you though that this is one of the best books that you could buy at this moment if you are planning to adopt an Agile method with Visual Studio Team System.

TechEd 2007 starts...

So today TechEd 2007 starts. I will be at the event helping on the Patterns and Practices booth, and also presenting on MSF for CMMI Process Improvement at the end of the week. Here is what I am planning to attend (my presentation included), all related in one way or another to MSF/software process: 

ARC10-TLC - Agile Talk on Agility

   Thursday, June 7 2:45 PM - 4:00 PM, Blue Theater 9 

Speaker(s): Peter Provost

One of the best ways to learn about agile software development is by doing it. This talk is facilitated using an agile project management approach to prioritize and answer questions from the audience by a panel of agile experts. Do you have questions about TDD? About MSF-Agile? About who to manage risk and predictability on an agile project? Come experience agile first-hand and get your questions answered at the same time.

Track(s): Architecture

Level: 300

BOF08 - Microsoft Solutions Framework 4.0: A Year Later

    Tuesday, June 5 10:15 AM - 11:30 AM, S331 A 

Microsoft Solutions Framework 4.0 shipped in 2006 with Team Foundation Server. It's a lightweight process used to guide development for both Agile and "less Agile" teams. In this BOF we explore how to use it, how to customize it and how to adopt it, based from a one-year experience from different projects in different environments. The discussion will be based on practical experiences and we particularly invite people who have already used MSF 4.0 and those are starting to adopt it.

DEV01-TLC - Tips and Tricks for Editing Your Process Template

    Friday, June 8 10:45 AM - 12:00 PM, Blue Theater 13 

Speaker(s): Mickey Gousset

Many initial users of Microsoft Visual Studio Team System start with using the MSF Agile or MSF CMMI process templates. When companies begin to want to implement their own process templates, they can be a little daunted by the syntax and rules. This session hopes to change that perspective. This session covers several tips and tricks for editing your process template, showing some of the most common requests seen for changes to someone's process template. While this session explains the basics of editing your process template, it is not Process Template 101. Instead, it covers how to implement real-world requests for process template customization. It looks at process template customization through the lens of personal experience.

Track(s): Developer Tools and Technologies

Level: 300

DEV04-TLC - Demystifying CMMI Adoption with MSF for CMMI Process Improvement

    Friday, June 8 1:00 PM - 2:15 PM, Blue Theater 13 

Speaker(s): Clementino MENDONCA

Learn how to help customers adopt CMMI in organizations with a "Stretch to Fit" approach and "MSF for CMMI Process Improvement" methodology, and how its integration with Microsoft Visual Studio Team System/Team Foundation Server helps in satisfying SCAMPI assessment requirements.

Track(s): Developer Tools and Technologies

DEV210 - Microsoft Solutions Framework 4.0 Core and its Families

    Friday, June 8 10:45 AM - 12:00 PM, N210 B 

Speaker(s): Rafal Lukawiecki

For over 11 years Microsoft Solutions Framework (MSF) has been helping larger groups of developers deliver great software on time, on budget, and fully satisfying their customers—always! Being, perhaps, one of Microsoft’s best-kept secrets, MSF has grown a strong community also outside of the fields of software development, most notably in the area of infrastructure deployment. As the times moved on, it turned out that the larger community used MSF in a number of different ways, each benefiting different scenarios and applications. Agile and more formal practices grew in importance all around our industry. While Microsoft’s Visual Studio Team System has already introduced two forms of MSF, known as MSF for Agile Software Development and MSF for CMM Integration Process Improvement, almost no one outside of a small group of people realize that Microsoft had been working on a wider group of MSF families. To keep all of the families cohesive, MSF 4.0 Core has been defined recently, and it forms the basis of all of the individual MSF families and instantiations. Indeed, while the Core has remained a framework, most of the actual instantiations have developed in-depth to become methodologies. This session gives you an overview of what is at the heart of all of those MSFs, the Core, and it shows you how some of the new concepts help structure efficient project teams. For the benefit of MSF v3 practitioners we point out the main version differences, but we concentrate on the new aspects, stressing the features of the Agile approach. As an important side benefit of attending this session you will see a few ways how you could improve your team’s efficiency, reduce the number of difficulties and make everyone feel happier working as a developer.

Track(s): Developer Tools and Technologies

Level: 200

DEV301 - Integrating the Database into the Application Lifecycle Using Microsoft Visual Studio Team System for Database Professionals

    Friday, June 8 2:45 PM - 4:00 PM, S210 E 

Speaker(s): Duncan Davenport

Visual Studio Team Edition for Database Professionals provides a new set of tools for working with databases throughout the development process and beyond. But just having tools is not enough. In order to reap the benefits of this new way of working with databases, you also need to embrace a new process for the way you do the work. In this session, we discuss the new MSF Agile and CMMI processes and show how with a combination of the new Visual Studio Team Edition for Database Professionals tools and the adoption of slightly new processes, you can see significant benefits both in productivity and quality.

Track(s): Developer Tools and Technologies

Level: 300

The other sessions I will also be attending as possible are related to VSTS/process in one way or another. The ones in red are the most promising from the work I have been doing with the Patterns and Practices MSF Team: 

BOF02 - Agile Development with .NET

    Monday, June 4 12:00 PM - 1:00 PM, S331 A 

Agile development methodologies are gaining more and more traction in the software industry. Join Jeffrey Palermo and many others for this popular BOF that has attracted 70+ attendees in past Tech·Eds. We discuss basic principles of Agile development including people over process and delivering working software at the beginning of a project, not just at the end. We share common successes and challenges from all the attendees, so bring your story.

BOF52 - The Need for an Agile Project Planning Tool

    Tuesday, June 5 6:30 PM - 7:30 PM, N310 E 

In the world of Agile Project management, no single tool has emerged as being the de facto standard for Agile Project Planning and Management. In this session, we discuss the need for such a tool and how such a tool can extend the value expressed in tools such as Team System.

Track(s): Developer Tools and Technologies

Level: 200

BOF63 - Creating Flexible Software: TDD, Mocking, and Dependency Injection

   Tuesday, June 5 7:45 PM - 8:45 PM, N320 A 

There has been much discussion of agile development techniques, but what do they really mean? How can they help you develop better software that is more flexible in the face of change? What does it mean for software to be flexible? This Birds of a Feather session promotes a lively discussion of how to be successful in creating software that is resilient in the face of ever-changing business requirements.

Track(s): Developer Tools and Technologies

Level: 400

DEV11-TLC - Microsoft Visual Studio Team System and Software Factories

    Monday, June 4 10:30 AM - 11:45 AM, Blue Theater 13 

Speaker(s): Jack Greenfield

In this chalk talk, Jack Greenfield, co-author of the Software Factories book, talks about the current generation of factories from p&p, future directions for software factories in Visual Studio Team Architect, and integrating software factories with other parts of Visual Studio Team System. The second half of the session will be Q&A, addressing questions from the audience on these topics.

Track(s): Developer Tools and Technologies

ARC13-TLC - Adopting Process One Bite at a Time

    Wednesday, June 6 2:00 PM - 3:15 PM, Blue Theater 9 

Speaker(s): Ivar Jacobson, David West

In this chalk talk Dr. Ivar Jacobson and David West describe a series of innovative new ideas that enable practitioners to effectively deploy, configure, and use process. They introduce the concept of a practice, and show how practices can be used to ‘cut up’ heavy, often irrelevant processes creating ‘developer friendly’ process guidance into smaller, digestible chunks. The session provides a framework for thinking about practices, a set of ideas for structuring a practice, an approach to delivering practices and a mechanism for people to use them all from within inside the Visual Studio Team System environment. This is a very interactive whiteboard type session providing the audience with the chance to ask questions of the father of the Unified Process and Use Cases on the next big thing in process.

Track(s): Architecture

Level: 300

ARC305 - Practice-Centric Software Development with Microsoft Visual Studio Team Foundation Server

    Wednesday, June 6 5:30 PM - 6:45 PM, S230 A 

Speaker(s): Ivar Jacobson, David West

Today there are many methods, processes, techniques, etc. that attempt to help project teams conduct their work. While there are indeed some important differences between them, the commonalities are far greater—the end goal is for all of us to get working software. Thus adopting complete processes does not make practical sense. Instead the focus should be on the ability to mix and match ideas from many different sources inside or outside our own world and compose these ideas to get a better way to work. Over the years we have more and more come to realize that these separate ideas are well represented by practices. Practices are first class citizens—process is just a composition of practices. Practices are to software development teams what use cases are to software systems: they have a beginning and an end and they provide identifiable values to the stakeholders. Practices come from individuals from different camps around the world, such as from the agile camp or the process improvement camp. In this session, gain an understanding of a practice-centric view of software development, as and how that relates to developing software using .NET applications. We examine the core essential practices that are critical to success and how these can be created using templates within Visual Studio Team Foundation Server.

Track(s): Architecture

Level: 300

DEV309 - Best Practices for Team-Based Software Development

    Tuesday, June 5 10:15 AM - 11:30 AM, N310 A 

Speaker(s): Joel Semeniuk

A highly cohesive software development team is critical to a successful project. In this session, we examine all of the different ways teams members, from project managers to developers and QA, can work together to help enhance team collaboration, and ultimately process success. We then take a look at how these opportunities map into Microsoft Visual Studio Team System and other Microsoft products to show you how you can take advantage of these best practices on your teams today.

Track(s): Developer Tools and Technologies

Level: 300

See you there.

The 90% rule

Once again a few friends (in this case, Fabio Camara, and Andrew Delin, both in the same week :-) ) ask me why I haven't been blogging. It seems that every time we have a conversation, they point out it could easily have been made into a blog entry.

After some reflection on the issue, I realized that it is because I have not been applying the "90% rule" to blog publishing: 90% of the ideal quality is good enough for most readers. In fact, I have seen so many spelling errors in some blogs that even so continue to be taken as a good reference that the bar seems to be to be at 60% or maybe 50%.

The same analogy applies to requirements capturing, coding, or for that matter, any knowledge transformation activity. If it takes too long to produce some information to be consumed, when it is finally done in perfection it might no longer be relevant (although it is always possible to recycle an episode if we can extract its intrinsic timeless value - another way of saying that all experiences teach us).

Being stuck in such an endless loop situation in software development prompted the infamous "analysis paralysis". Agile teams have a way to deal with it in the formula YAGNI ("Ya Ain't Gonna Need It"), which helps you cut the cycle by focusing on why you are doing something. Which reminds me, I would better post this now, or I might never publish it once the never ending editing cycle starts.

I guess I have to apply the 90% rule more.  However, my bar for spelling will continue to be 100%, or else I will put the blame on Word's spellchecker if something wrong passes through.

 If only we could blame compilers for our business logic defects :-) ...

McCarthy's video on "23 and a half Rules of Thumb for Software Development"

McCarthy also recently released it as podcasts at http://www.mccarthyshow.com/McCarthyShow/TwentyThreeandaHalfRulesofThumb/tabid/808/Default.aspx

Agile 2006

The weather in Minneapolis was warm this year, and so was figuratively speaking the Agile 2006 Conference, which had all sorts of interesting discussions going on anywhere - lobbies, dining rooms, sessions. I felt at home for its informality, and definitely will come back next year.

While being a newbie to the gathering, I have been working with agile software development since 1999 when I was first introduced to the topic by Bill Addington, whose last job at Compaq was as a software development quality and methodologies guru.

Aside from MSF (which to me was inexplicably absent from the manifesto meeting in 2000), my first book on agile was Adaptive Software Development by Jim Highsmith, then Xtreme Programming Explained (first edition) by Kent Beck,  followed by Java Modelling in Color, a beautiful book by Coad, Lefebvre and DeLuca, which introduced FDD.

So while at the conference it was a pleasure to meet in person several luminaries who have been the backbone of the Agile movement.

The first one was Alistair Cockburn [pronounced 'co-burn']. We had a chance of chatting for a few minutes after his introductory presention on Agile Software Development in general, and Crystal Clear in detail. My favorite book of his though is Writing Effective Use Cases.

We talked about the origin of the name "Agile"  for the manifesto, with a few interesting points I will bring back later.

Alistair Cockburn in a display of positive agile energy while talking to a conference attendee