Grigori Melnik: Thoughts on Agile Software Engineering and Beyond : agile

Enterprise Library 5.0 … and we’re off!

gmelnik — Mon, 20 Apr 2009 20:56:37 GMT

We have officially started our development work on EntLib5.0. Earlier I’ve shared our product backlog, which was largely driven by you, the community. Our initial plan is to ship in early 2010.

If you want to stay tuned with what we’re doing, the best way is to keep an eye on the home page of our community site at http://codeplex.com/entlib. Here, you can view recent posts aggregated from individual team members’ blogs, participate in online discussions, get the most recent bits of code, see file bugs/issues, and access other useful resources.

As you probably know, we run our projects in agile fashion (XP@Scrum). For those wondering, yes, we do perform TDD, pairing (I kid you not! – even distributed pairing), automated acceptance testing, refactoring, daily standups, regular retrospectives, customer proxy demos, and iteration planning; plus frequent releases, continuous integration, collective code/content ownership, low-fi information radiators, and progress tracking. We are working in two-week iterations and will be doing regular source drops on Codeplex.

Now it’s time to introduce the mighty team. You will recognize some familiar faces (many of us have worked on one or more of the previous EntLib releases) and also see some new ones. I am privileged to work with this group of accomplished professionals. I give you… <drum roll please>:

from left to right: David Hill (architect), Alex Homer (documentation and user experience lead/writer), Chris Tavares (dev lead/architect), Bob Brumfield (dev/architect), Fernando Simonazzi (dev/architect), François Tanguay (system test/dev), Nelly Delgado (release lead), Grigori Melnik (product owner/program manager). Not shown are: Rohit Sharma (test lead), RoAnn Corbisier (technical writer/content coordinator), Dennis DeWitt (technical writer), Carlos Farre (perf test consultant), Mani Krishnaswami (test), Nicolas Botto (system test/dev), Olaf Conijn (dev), Mr X (security test).

For the kickoff iteration, we were able to get the core of the team together in Redmond. The level of energy was remarkable! This team has all necessary expertise, diversity of perspectives, and desire to deliver even more goodness and value with EntLib v5.0.

You should realize that, in reality, the team is way bigger. It includes 20 members of the advisory board who we meet with on a regular basis (every 2-3 weeks). These experts help steer our direction. I must say that they are quite direct and honest, and are not shy in pushing back on any of our ideas that don’t make sense to them or do not provide sufficient value for the buck. The advisory board is representative of our target audience with experts from large and small companies, from a wide variety of domains (such as retail, transportation, government, insurance, finance, military, health care, education, media, IT services, and more).

We are also leveraging much of the learning from the previous four major releases of EntLib. Therefore, we appreciate everyone who has ever been a part of the EntLib team. We are smarter today because of you!

In addition, seven members of the patterns & practices directorate (John deVadoss, Ade Miller, Ajoy Krishnamoorthy, Don Smith, Javed Sikander, Mohammad Al-Sabt, and Steve Elston) blessed our project. Their trust and support in our abilities to self-organize and self-manage are real tokens of confidence.

All other teams at patterns & practices are involved and provide their feedback informally on the daily basis. We also interact with many members of other Microsoft product groups via regular workshops, brownbags, and internal discussion forums (with 300+ members).

I would also like to acknowledge our sustained engineering team from Avanade, who have been doing a terrific job in providing online technical support.

And, of course, you – the community! Anyone who ever participated in our surveys, submitted requests/bug reports through Codeplex, shared their EntLib extensions via EntLibContrib, participated in the customer workshops, came to our booth at the conventions (TechEd, PDC, p&p summit, etc.), or in any other way provided feedback or suggestions has contributed to this development effort. For that, we salute and thank you! We build EntLib for you and with you!

We all look forward to yet another exciting ride developing EntLib v5.0!

(Un)training puppies or adventures into the world of agile

gmelnik — Sat, 21 Mar 2009 03:18:16 GMT

As I was going through postings from our internal agile discussion forum (yes, we do have one at Microsoft and it’s pretty vibrant too, with, on average, about a dozen of messages daily), I came across the draft of Sara Ford’s story about how she learnt to program manage an agile team after 6 years of waterfall. Sara is a colleague from Codeplex team. She was asking for feedback on it prior to posting it publicly, which she’s a got tones of. I don’t want to re-tell her story but encourage you to read the final version as I do think it’s an awesome first person account of how a convinced waterfallian ventured into the sea of agile and what kind of mindset conversions took place.

The paragraph that I thought was quite notable and also funny is when Sara talks about the spec:

“…during my first week on the team, I had asked a specification question, like how many characters do we allow for a password? And Jim says, “we’ll have to look it up in the test cases.” And I said very proudly from my Visual Studio experience, “Gotta love it when the test cases are the specs.” (yes, I said this to Jim Newkirk, my manager.) To which Jim very calmly replied, “That’s the way it is supposed to be.” And I laughed, and then after a pause, I said in slight horror, “you’re serious, aren’t you?” It was like finding out there was no Santa Claus or you really don’t vote for the President of the United States. The mind just simply can’t comprehend.”

With time Sara’s mind learnt to appreciate and make the most of test cases as requirements. This resonates with the The Equivalence Hypothesis Bob Martin and I formulated several years ago stating that “as formality increases, acceptance tests and requirements become indistinguishable”.

When I saw Sara by the coffee machine today and commended her on posting the story, she mentioned that the actual look Jim had on his face at that moment of telling her to look up the spec in the test cases was identical to the one of a pet-owner who was ready to punish a new puppy for ruining the carpet and stopped short from smacking it while thinking to himself “Wait, it’s a puppy!”

After all, it looks like old tricks can be unlearnt.

Where to find useful stuff from the Agile conference?

gmelnik — Sat, 14 Mar 2009 07:31:55 GMT

Agile conference traditionally publishes proceedings (that’s where you find all research papers and experience reports).

Attendees also receive a CD or a memory stick with tutorial materials (decks, handouts, demos etc.).

Several main stage talks are available as webcasts.

What about workshops? A workshop is really an opportunity for attendees to learn more from each other as opposed to from a lecturer/facilitator/presenter. Thus, typically, there is not much material produced ahead of time, when the proceedings and conference cds are being assembled. However, a good wealth of knowledge has been captured in workshop summaries that we encouraged workshop facilitators to publish on the conference public wiki after the conference: http://agile2008toronto.pbwiki.com/Agile2008+Sessions

Take a look!

Agile Perspectives PDC Workshop

gmelnik — Sat, 01 Nov 2008 05:14:00 GMT

As promised, here’s the deck Mary Poppendieck and I used during the workshop. We hope it was worth the attendees’ time. We encourage you to try new techniques we talked about and let us know how your agile adoption is going. For more information, visit Mary’s site.

Agile PreCon at PDC 2008

gmelnik — Tue, 30 Sep 2008 05:38:57 GMT

Mary Poppendieck and I are set to run a pre-conference workshop on Agile Perspectives: Industry and Microsoft.

In it, we will provide our observations on what works, what doesn’t in the world of agile, and why (we call this segment of the workshop “Reality of agile”). The objective is to provide pragmatic advice, recommendations and actions that you could use tomorrow to improve the way your teams produce software.

Mary, the author of popular Lean Software Development and Implementing Lean books, will provide her lean perspective from many years in the industry (if you want to find out more about Mary, check out this really good interview that Scott Hanselman took with both Mary and Tom Poppendieck, in which they challenge the notion of success in software industry). In turn, I am going to share my experience from several projects at Microsoft patterns & practices. Several other colleagues from Microsoft will join our panel at the end of the day to debate hot issues of agility. The floor will be open to audience participation. Mary and I identified several hot issues to bring up. However, it would be really good to know what kind of stuff you’d prefer us to touch upon. Please post your At-Issue suggestions in response to this post or simply email me. Thanks and see you in LA!

Agile2008 selected sessions online

gmelnik — Mon, 18 Aug 2008 05:26:00 GMT

You wish you could have made it to Agile2008 in Toronto 2 weeks ago? A small number of sessions is now available online.

18 sessions have been recorded, 3 of which are already made available:

− Natural Laws of Software Development - Deriving Agile Practices by Ron Jeffries, Chet Hendrickson

− "We suck less!" Isn't mediocrity great? by David Douglas, Robin Dymond

− New Car Development in Toyota by Kenji Hiranbe

Another 15 will be gradually published at http://www.infoq.com/agile2008/:

− Future Directions for Agile by David Anderson

− 10 ways to screw up with Scrum and XP by Henrik Kniberg

− Agile and Beyond - The power of aspirational teams! by Tim Mackinnon

− Agile in Practice: What is ACTUALLY Going On Out There by Scott Ambler

− Prioritizing Your Product Backlog by Mike Cohn

− Manager's Introduction to Test-Driven Development by Dave Nicolette, Karl Scotland

− When Working Software Is Not Enough: A Story of Project Failure by Mitch Lacey

− Extremely Short Iterations as a Catalyst for Effective Prioritization of Work by Mishkin Berteig

− Fostering Software Craftsmanship in a Corporate Setting by Scott Dillman

− Measuring Agile in the Enterprise: 5 Success Factors for Large-Scale Agile Adoption by Michael Mah

− Colossal, Scattered, and Chaotic (Planning with a Large Distributed Team by Wes Williams, Mike Stout

− Now that the team's head is in the game, how do you get their heart in? by Jim McCarthy

− Technical lessons learned turning the agile dials to eleven! by Craig Smith, Paul King

− Introduction to Lean Software Development by Alan Shalloway

− Fully Distributed Scrum: The Secret Sauce for Hyperproductive Outsourced Development Teams by Jeff Sutherland

In addition, Alan Cooper’s "The Wisdom of Experience" keynote is available as a slide deck with speaker’s notes at http://www.cooper.com/journal/agile2008/

Note, InfoQ also hosts videos of several excellent sessions from the previous Agile2007 conference, including:

− The Role of Leadership in Software Development by Mary Poppendieck

− Succeeding With Agile: A Guide To Transitioning by Mike Cohn

− Agile in the Waterfall Enterprise by Michele Sliger

− The Agile Enterprise: Real World Experience in Creating Agile Companies by Jeff Sutherland

All experience reports and research papers will be available online via IEEEExplore. Search for 'Agile' under 'Conference Proceedings'.

Acceptance Test Engineering guide - How to Decide if Software is Ready for You or Your Customer - Community Preview 2 released

gmelnik — Wed, 06 Aug 2008 17:32:00 GMT

On Monday, we released our CTP2 on Codeplex: http://codeplex.com/TestingGuidance.

You’ll see it now includes the consumption model and also some recommendations on how to read the guide. There are three parts. Part I. Narratives and Models - this is a short section that is meant to be read end-to-end. This is where we define readiness assessment and acceptance testing and introduce several models that help conceptualize software acceptance. Among other things we discuss decision making process, risk evaluation, the doneness model and its variants for agile and Tayloristic processes. Part II consists of what we refer to as “Thumbnails”. Those are the short descriptions of various useful practices that also provide information on their applicability, risks addressed, implementation options and additional resources. The Thumbnails are complemented by the actual samples (Part III) that we’ve produced on a sample system (Global Bank). Both thumbnails and samples are meant to be used as references and in any order.

You’ll notice there are still a number of empty sections. If you see an empty section, assume we will fill in the section before we complete the book.

Please provide it by posting a reply to the corresponding thread under Discussions on codeplex or as a reply to this blog post.

Happy reading! We look forward to your feedback!

Upcoming conferences - where I'll be (May - Oct 2008)

gmelnik — Sat, 31 May 2008 17:45:00 GMT

Tech-Ed 2008 Developers, June 2-6, Orlando, FL. I’ll be giving a talk on the Enterprise Library (DVP02-TLC). I’ll be also hanging out in the exhibition areas (mainly, Process & Project Management (DVP- VSTS 2008)).
CAST 2008 , July 14-17, Toronto, ON. I look forward to good interaction with testing experts and thought leaders from around the world.
Microsoft Product Fair, July 22-23, Redmond, WA.
TechReady7, July 28-Aug 1, Seattle, WA. I’ll do a couple of talks on the Enterprise Library and agility.
Agile 2008, Aug 4-8, Toronto, ON – the major world conference on agile software engineering. Mary Poppendieck and I are program-charing the conference.
STARWest 2008, Sep 29- Oct 3, Anaheim, CA. Jon Bach and I will give a talk on Acceptable Acceptance Testing, telling the tale of our team producing the guide on acceptance testing and dogfooding our own recommendations.
PNSQC2008, Oct 13-15, Portland, OR. We have a paper "Acceptance Testing: a love story in two acts". No Shakesperian-level drama, but some exciting stuff for sure.
PDC 2008, Oct 26-30, Los Angeles, CA. Together with Mary Poppendieck and other colleagues from p&p, we’ll be presenting the Agile Perspectives pre-conference event.

If you are planning on attending any of these conferences, please come by and say hello. I would love to hear your stories of using or reasons for not using the Enterprise Library, Unity, other p&p assets. I would also be glad to chat with you about our upcoming guidance on Acceptance Test Engineering and hear your experiences, lessons learnt, and suggestions.

Also, I talk to user groups regularly. If you are from the city I am visiting (or nearby) and would like me to give a talk to your user group (agile, .NET, software testing), feel free to ping me.

Empirical Evidence of Agile Methods: ce que nous savons, ce que nous pensons savoir et ce que nous voulons savoir

gmelnik — Tue, 13 May 2008 16:37:00 GMT

I gave two versions of my talk on the Empirical Evidence of Agile Methods : What We Know, What We Think We Know, and What We Would Like to Know at the patterns & practices summit in Québec City, Canada and the Agile Québec user group last week. Here’re the materials of the talk I promised to post.

As a side note: I’ve noticed that the Quebec audience mainly interacts with their eyes. They are so polite that they don’t interrupt the speaker, they even turn off their cell phones.

Unity project heartbeat

gmelnik — Wed, 05 Mar 2008 06:43:00 GMT

Those who follow the work of our team on Unity have probably noticed a certain rhythm we follow – of weekly Unity CTP drops on Tuesdays. Today, we did another one. It has some changes to the API (described at http://www.codeplex.com/unity/Release/ProjectReleases.aspx?ReleaseId=11319). Please check it out and provide us with your feedback. The community involvement (through codeplex) has been quite meaningful. Keep sending your comments and voting on existing work items on codeplex.

Tests and Requirements, Requirements and Tests: A Möbius Strip

gmelnik — Wed, 23 Jan 2008 00:43:00 GMT

Over a year ago, Robert Martin and I wrote a paper on exploring and disambiguating requirements with acceptance tests. In it we formulate the equivalence hypothesis - as formality increases, acceptance tests and requirements become indistinguishable. At the limit, acceptance tests and requirements are equivalent. IEEE Software magazine published it in the January/February’08 issue (http://www.computer.org/portal/cms_docs_software/software/homepage/2008/s1mel.pdf).

As usual, comments are welcomed.

Agile 2008 Conference - new model, many changes - part 1

gmelnik — Tue, 15 Jan 2008 11:21:00 GMT

Catching up on blogging about Agile 2008 Conference, for which I am a Program Chair. Luckily, I am part of an amazing squad of 40+ enthusiasts and their teams, led by the fearless trio: Rachel Davies (who is the General Chair), Mary Poppendieck (who shares Program Chair’s responsibilities) and yours truly.

This year we are experimenting with a new model based on a music festival metaphor. I’ll tell you how we are adopting it in a minute. But firstly let me explain why we needed a change. After all, the conference has been very successful in the past (selling off several months in advance for the past three years). So, why would we change a model that works? Well, over the years due to increasing popularity of agile methods, we rapidly grew from several hundred attendees to over a thousand and realized that the current model cannot scale any further. Last year in Washington, DC, with several dozens of simultaneous sessions in various places, it was hard not only to find the room but to simply navigate the program and decide where to go. We believe the new model will address these issues, allow the conference to grow (we expect 1,600 delegates) and even make it more community-driven.

So, I’ve mentioned the music festival metaphor earlier. What festivals do I have in mind? Blue Skies Festival in Clarendon, Ontario, Canada is one. It grew up from a small beginning and now attracts thousands of people. Glastonbury Festival of Contemporary Performing Arts in the UK is another example with tens of thousands of performers and organizers entertaining hundreds of thousands of attendees (I believe it is the biggest greenfield music festival in the world!) Both started back in 1970s and still running and selling off annually. Only about one third of those who would like to come actually get tickets. How did these festivals manage to grow so big and yet remain so successful for such a long time?

It occurred to us it’s the multiple stage metaphor with individual producers and production crews focusing on their own stage (e.g. Acoustic Tent, Dance Village, JazzWorld etc.) that helps these events grow. Importantly, these producers have a high degree of autonomy (in recruiting and selecting performers, scheduling appearances etc.)

Just like those festivals, the stages within our conference program are designed and organized by experts (acting as stage producers) who are truly passionate about their particular areas. Each stage will have a feel of a smaller, focused mini-conference whilst providing the conference attendee with a wide choice of stages to choose from:

§ Agile and Organizational Culture

§ Breaking Acts

§ Chansons Françaises

§ Committing to Quality

§ Customers and Business Value

§ Designing, Testing, and Thinking with Examples

§ Developer Jam

§ Distributed Agile

§ Leadership and Teams

§ Learning and Education

§ Live Aid

§ Main Stage

§ Open Jam

§ Questioning Agile

§ Research

§ Tools for Agility

§ User Experience

You can find more info on stages here.

This makes a lot of sense. In the past, the conference was organized around the session types (tutorials, workshops, experience reports etc.). Individual committees were formed to review and select sessions for the program. Instead of organizing the conference around the session types we are organizing it around the topics (the content types), which is more aligned with the agile principles. Focus on the value delivered to the customer. Focus on substance in the first place, not form. After all, our customers (attendees) care less about the session type and more about the content presented and discussed. We want to have a good and broad coverage of topics and also avoid redundancy. The producers are in a better position to balance the program of their stage.

(To be continued...)

Scrum at Macro and Micro Levels

gmelnik — Mon, 17 Sep 2007 20:47:00 GMT

Scrum has come of age. Not only it is entering the mainstream of modern software development, but it is, IMHO, the most popular/successful agile method for project management today. It was not the case a few years ago, when XP dominated the agile world. Ken Schwaber reconfirmed this with some stats on Scrum penetration, the number of Certified Scrum Masters, Certified Practicing Scrum Masters, etc.

Historically, when agile methods were in the early adolescence period, one of the main criticisms was that agile methods did not scale. At the Canadian Agile Workshop dedicated to the very issue of scaling agile methods (2003, http://www.agilenetwork.ca/ws2003/ ), Martin Fowler argued strongly against scaling up while Ken Schwaber described his experiences working with Scrum on large projects (100+ team members) and several techniques he had successfully used. One of the techniques was the Scrum of Scrums (in a large team of teams, one representative of each comprising team regularly (Schwaber recommends daily; in practice, it tends to be 2-3 times/week) participates in a scrum of scrums meeting, that is conducted similarly to the regular scrum). Mike Cohn gives a nice description and various techniques on how to run Scrum of Scrums meetings effectively (http://www.mountaingoatsoftware.com/article/35).

In his Agile 2007 talk, Schwaber goes beyond the issue of scaling Scrum. He addresses the matter of how to roll out Scrum adoption enterprise-wide. Schwaber emphasizes the fact that there is no “enterprise” version of Scrum. It is a simple empirical framework for managing the development and deployment of (complex) products/systems. There is, however, Scrum used throughout the enterprise. Schwaber describes the “etc.” project - the Enterprise Transition project, not for development and maintaining of products/systems, but rather for optimization of the enterprise’s overall productivity, quality, value and competitiveness. The idea is to apply Scrum to transition the enterprise to Scrum. The lean principle of ruthless elimination of waste is fundamental. Similarly to a development project, a team (in charge of transition) composes a project backlog (Transition product backlog) with items requiring improvement. These could be fed from the impediments identified by current individual development projects. 30-day sprints are conducted to communicate Scrum values and the impact, establish and meet preconditions that must be met before a project can use Scrum, identify which projects to use Scrum next, identify Scrum masters, provide training, define Scrum metrics and mechanisms for their collection, assess compensation policies that encourage teamwork, and so on. Clearly, such transition project is a major undertaking. Schwaber estimates that it would take 6 months to roll-out, 3-5 years to make it stick and then continuous improvement.

Fundamentally, for Scrum to be successful, a change in thought process is necessary. Those who get Scrum seem to take and apply it beyond their immediate teams – some to a macro level (Scrum of Scrums) and some to a micro level (personal work management and self-improvement). The examples of such adoptions are not hard to find at patterns & practices. The new approach to portfolio management and execution is one of them (will post a separate entry on this). The other two are Peter Provost’s efficient computer repaves (http://www.peterprovost.org/archive/2007/07/16/23053.aspx) and J.D.Meier’s improvement/learning sprints (http://blogs.msdn.com/jmeier/archive/2007/03/09/30-day-improvement-sprints.aspx). Similarly, I execute the MoveFromCanada project with my family in a Scrum-like fashion (actually, it’s also an example of a distributed Scrum). Essentially it all comes down to 1) maintaining and learning from the prioritized backlog; 2) having clear acceptance criteria and knowing your status; 3) using discipline and good practices to be able to respond positively to change.

With the new portfolio management and execution methodology, it appears that we’ve focused on addressing the first of these for now (building a prioritized backlog). Acceptance criteria and how we are going to respond positively to change is not as apparent now. Still, I am enthused.

What all these examples demonstrate is that Scrum ideas can be applied at macro and micro levels, inevitably penetrating other aspects of our work and life (beyond software engineering). The reality contradicts Craig Larman’s prediction that we all would need to die first before the agile ideas become the mainstream. Agile is already there, and as far as I can tell I am still alive. So is my team!