Progressive Development : agile

Motley says: "What's wrong with mixing Scrum and Waterfall? You get a nice 'Scrummerfall'!"

James Waletzky — Tue, 25 Aug 2009 17:45:00 GMT

Summary

	Motley: An effective modification to Scrum is to do a requirements sprint, followed by a design sprint, implementation sprint, test sprint, and stabilization sprint. It has the benefits of Scrum and has commonality with our older processes, which makes the developers happier.
	Maven: Scrummerfall, or mixing traditional Waterfall with Scrum, is less effective than Scrum itself. Deliver real business value with each sprint for early and frequent feedback, and improve collaboration amongst team members as no one is left blocking and waiting for others to finish their tasks.

______________________________

[Context: Motley and his team of able developers have been brainstorming ways to improve their Scrum process, and he is about to upset Maven with the details of his "improvements"]

Motley: Hey, Mave. Where have you been hiding lately? I haven't seen you around the office much.

Maven: Summer time is busy time. I had softball to play, a week of karate class with my sensei's sensei, and a couple of backpacking trips. I've been taking my vacation in small chunks.

Motley: I thought you were just slacking off, as usual.

Maven: Funny guy. This year I've had lots of small vacations instead of one big one. It really doesn't allow you to recharge enough, but I do like the variety.

Motley: Ok, enough about your relatively boring personal life. I'm thinking of making some changes to our Scrum process. With our next milestone fast approaching, the team really wants to transform Scrum into something more effective.

Maven: That's great! A key tenet of agile development is frequent introspection and continuous improvement. Glad to hear it.

Motley: Yeah. The team is thinking of having a requirements sprint, followed by a design sprint, followed by an implementation sprint, followed by a test sprint, followed by a stabilization sprint. Because the company wants to see these document deliverables as early on as possible, it might make more sense to get these done in sequence like this. The team also likes it because it is more similar to our old process.

Maven: I take it back - that's not so great.

Motley: What's the problem? The changes sound reasonable to me.

Maven: What you are proposing can be referred to (depending on the circle) as "Scrummerfall" or "WaterScrum". I'll call it "Scrummerfall", as I like that term better. In essence, you are proposing to combine two methodologies that typically do not work well together. With the proposal, you are basically doing Waterfall development, with more frequent check-ins (more akin to the way Waterfall was supposed to be, but typically is not). You lose the benefits of being an agile team.

Motley: How is it really different? We obviously still preserve the core of Scrum, including the daily stand-up meetings, planning, burndown chart, retrospectives, continuous improvements, and all the other pieces.

Maven: The proposed process is fraught with issues. In fact, you are no longer an agile team - you are back to waterfall, like it or not. Remember our previous chats on agile development? This process would break many of those principles. For example:

Scrummerfall abandons the idea of vertical slices, or delivering end-to-end functionality in short iterations of 2-4 weeks. Your deliverable at the end of a "sprint" in this model is a document, such as a requirements specification. Does that add real customer value that you can solicit meaningful feedback on? In my experience, it is hard for a user or fellow engineer to provide real valuable feedback on something that is not real (i.e. there is no working software, only a description of what that software will do).
Scrummerfall loses the benefits of the improved collaboration that Scrum provides. Sure, you can still have your daily stand-up meeting, but the feeling of marching toward a common goal is somewhat lost. What are the developers doing while the "Requirements Sprint" is happening? What is the test team doing while the "Design Sprint" is happening? Scrummerfall just doesn't work in a truly collaborative way vs. delivering real business value at the end of a sprint. Additionally, it forces you into doing too much up-front work with requirements and design, leading to overthinking the problem.
Scrummerfall violates the "working software over comprehensive documentation" piece of the agile manifesto. Don't get me wrong - documentation is necessary even with agile teams. Here, however, you tend to generate much more of it, which often falls into the waste (i.e. wasted effort) category, and we want to eliminate waste.
Scrummerfall makes change harder. If you get to implementation and test and realize something is amiss, you have a bunch of rework to do, including potentially large adjustments to design and documentation. Don't forget, with agile, we want to embrace change.

The team needs to deliver working software every sprint for early and often feedback. Yes, you can apply some of the practices of the Scrum methodology that contribute to team success, but Scrummerfall violates the fundamental principles of agile, which put you back in a Waterfall world.

Motley: But we still only need to plan for short iterations instead of long term crystal ball-like planning, which is a benefit and better than pure Waterfall.

Maven: Scrummerfall forces the team into a longer-term detailed planning phase because the end date of specific functionality in working software is important to the management team. You'll be predicting the future farther in advance for each feature instead of delivering features in just a few weeks. Of course, if you have a management team that wants crystal clear concrete delivery dates for all functionality up front, then you have less leeway here and have to do up front planning anyway to increase their confidence and minimize risk (even though that may be a fallacy anyway).

Motley: What about bugs? I don't want the team to be distracted with a bunch of bug fixes while we are doing feature development. We need that stabilization sprint at the end once everything is complete to fix all our bugs.

Maven: Don't forget - agile thinks about bugs differently. Instead of leaving bugs until the very end of the cycle, address them in the next sprint after a small end-to-end scenario is implemented. Bugs are a sign that the feature developed in the iteration is not done. Address the bugs as soon as possible, as they lead to increased technical debt (plus potential bug postponement, forgetting context of the bug, fixes growing more complicated, etc.) if they hang around. Avoid leaving them until a stabilization sprint at the very end. As we discussed previously, it is okay to have a quality sprint where you just focus on bugs as your iteration goal - it is more about when you fix the bugs.

Motley: Although I think the proposed process can work, I see some of your points. Small iterations of complete functionality, focused shorter-term planning, better collaboration and efficiency of team members, and less emphasis on documentation.

Maven: I actually agree with you - the proposed process can work. But, you are either an agile team or you are not. Mixing the two models can confuse people and actually make team effectiveness worse. With small chunks of deliverable software, you can get the test team private builds quickly to hammer on, and the test team does not fall behind as quickly. Because the team is marching to a common goal, the constant collaboration leads to priority adjustments and work breakdown to help others succeed (e.g. dev can build stubs and the test team can start to write test automation).

Motley: Ok, ok. I'll have a meeting with the team to discuss putting the brakes on this proposed change. They might be a little upset.

Maven: There is no one right way to build software. The idea here is to set the team up for greater success than it would have had otherwise. Other factors can get in the way of putting in your own completely customized process, but that is a topic for another day.

______________________________

Maven's Pointer: Brad Wilson, a blogger on the "Agile Programmer" site defines Scrummerfall as:

Scrummerfall. n. The practice of combining Scrum and Waterfall so as to ensure failure at a much faster rate than you had with Waterfall alone.

Interesting definition. His description of Scrummerfall is slightly different than above. It involves embedded Waterfall within each sprint in a one-two-one pattern - one week of design, two weeks of implementation, one week of test. Although, in my opinion, this situation is not nearly as bad as full requirements, design, implementation and test sprints, it is still somewhat outside the spirit of Scrum unless you parallelize the tasks and help each other succeed.

Maven's Resources:

Scrummerfall - Mixing Scrum with Traditional Software Methods, blog by Mitch Lacey, http://blogs.msdn.com/mitchl/archive/2006/08/18/706897.aspx

Motley says: “"Vertical Slices"? Sounds like something you do to salami”

James Waletzky — Fri, 27 Mar 2009 16:41:00 GMT

Summary

	Motley: Build an application according to architectural layers, from the bottom-up.
	Maven: Build an application using vertical slices. Build just enough UI coupled with underlying layers such as an object and data model, to satisfy a user scenario. Deliver customer and business value at the end of every sprint to ensure feedback early and often.

[Context: Motley thought he was doing iterative development the right way, but his boss has been all over him for not delivering tangible results]

Motley: I'm telling you, I can't win! We have been doing iterative development with the Scrum agile development model and my boss is still screaming at me that he is not seeing anything tangible coming out of it. I keep telling him that it will take a few sprints for us to show real value.

Maven: Have you forgotten one of the key aspects of agile development? Working software sooner. Agile teams should concentrate on delivering customer and business value at the end of every sprint. Get something real into the customers' hands quickly so that they can provide feedback early and often.

Motley: It takes time to get something real! We have to do a bit of infrastructure work first, and then build on that.

Maven: Ah, have you heard of "vertical slices", my boy??

Motley: You call me "your boy" one more time, I'll turn your shirt inside out with you still in it. "Vertical Slices"? Sounds like something you do to salami. I'm not going to bother asking what you mean by this because I know you'll tell me anyway.

Maven: Correct, my- friend. Vertical slices help deliver customer and business value at the end of every development iteration. Instead of developing one large feature requiring multiple iterations to see results, we break the work into usable pieces. Take a typical, but simplified, architecture:

One approach to development would be to first build the data layer (the lowest level of infrastructure), then build the object model on top of it, and then finally end with the UI. This is pretty common practice.

Motley: What's wrong with that? I need the lowest levels to build the middle and upper tier. You HAVE to agree with that! And besides, I am delivering business and customer value if I build the data layer and deliver that piece of software in one iteration.

Maven: Does the customer really care that you have a "data layer"? I think not. There is nothing for the customer to tangibly see until you have built the UI in later iterations. Additionally, once the customer does see the UI a couple months later and they can provide feedback, you risk having to do a lot of rework in the infrastructure if the customer wants changes. Even a small UI change can render various APIs in lower layers useless.

Motley: But I still need a data layer (in this architecture) to build the entire application.

Maven: Yes, but chunk up the work. Avoid building from the ground up starting from data to object model to UI. Instead take a vertical slice of the overall architecture and deliver a user scenario (user story). You build "just enough" of the data layer, object model, and UI to satisfy the user scenario. You get real working software sooner upon which the user is capable of delivering feedback. The focus switches from delivering technology to delivering real user value.

Motley: Flaw. Should I say it louder? FLAW. Putting software together in a piecewise fashion is going to lead to a bunch of spaghetti without a real architecture. This will never work.

Maven: Don't be such a pessimist! Or is it a cynic? I can never keep those two words straight. Anyway, you still need to have a vision for the product architecture. This agile type of approach does not preclude planning the entire system. It says that you plan the architecture at a high-level (components, interactions) and then build it in vertical pieces that span all layers. I still want a holistic architecture plan in place to guide development across several iterations. I just don't go into tremendous detail around exactly what the specific component APIs will look like (for example).

Additionally, I avoid building a full horizontal layer that may contain a bunch of code that will not ever get used, or get thrown away, once I get feedback.

Motley: Well, I may have to take some shortcuts to implement scenario #1 and then redo some work in the first slice to implement scenario #2. I just wasted effort! As we all know, eliminating waste is a key agile principle.

Maven: Well, it's actually a lean-

Motley: Yeah, yeah - a lean principle. Don't nitpick and respond to my comment.

Maven: It is true that sometimes you may have to refactor some code in the first slice to start adding to it in the second slice. That's fine - you have good unit tests in place that make refactoring easier. You may also have rework based on user feedback. This is all fine in that the rework for one small iteration is typically far less than major changes that could be forced by late feedback. Additionally, developing in vertical slices does not preclude you from using proven design principles. Design to accommodate change so that making changes later is easier. Although you don't want to anticipate future changes in successive slices and build functionality that may never get used later, you want to make your design easy to change and extend.

Motley: So I build a small snippet of the UI and just enough infrastructure to make that UI go. I am going to be left with some incomplete functionality by the end of the sprint.

Maven: Yes, but that's okay. It is usually enough to get going with feedback and validate your overall direction and approach. The customer can much more easily grasp a demo of working software than pictures in documents. Give them something they can touch.

Motley: And from the engineering side, we have some fairly incomplete layers at the end of every sprint.

Maven: Yes, sort of. You have layers of minimal complexity that support only the core scenarios. This helps keep the design and code simple. Think of this as adhering to YAGNI, or "You Ain't Gonna Need It". The gist of that principle is that you avoid implementing stuff until you actually need it, and you never try to anticipate future change. You avoid wasting time creating that functionality, testing it, debugging it, supporting it, and you keep things simple. You build enough to support the scenario. Over time your layers emerge.

Motley: I guess that makes sense. We end up incrementally building the product over time with tangible functionality and integrating features piecewise. I would have to try it before being convinced.

Maven: Absolutely. It takes some practice to chunk up the work into vertical slices, but if you focus on user scenarios/user stories instead of building technology, you'll end up on the right side of the fence.

______________________________

Maven's Pointer: A good way to help yourself think in vertical slices is to formulate your product backlog in terms of user stories. The basic way to think of a user story is to phrase your product backlog item as "As a <user type>, I want to do <task> to accomplish <goal>". We'll talk about this more later.

Maven's Double Pointer Indirection: On my team at Microsoft, we try to break up work into vertical slices as much as possible. Take an address bar in a web browser. Instead of building the data providers that provide auto-suggest functionality and then tacking the UI on top of it, we try to build a basic UI with one suggestion (maybe even hard-coded) and then incrementally tack on from there. Okay - it doesn't always work out that way in terms of work chunks, but that's our goal.

Maven's Resources:

YAGNI, "You Ain't Gonna Need It" - http://en.wikipedia.org/wiki/You_Ain't_Gonna_Need_It
Iterative and Incremental Development (IID), by Robert Martin, http://www.objectmentor.com/resources/articles/IIDII.pdf

Point for Maven - Wideband Delphi Estimation works!

James Waletzky — Tue, 06 Jan 2009 18:54:00 GMT

Summary

Motley: I admit it - Wideband Delphi works! It helped us generate fairly accurate estimates for longer-term planning and the documentation of our assumptions kept everyone on the same page and provided rationale for our numbers.

Maven: Um, what Motley said.

______________________________

[Context: For the past few months, Motley and his team have been using the Wideband Delphi process for their estimates. Over the winter holidays, Motley got curious about the effectiveness of the technique]

Maven: Hey, Mot. Happy New Year, and welcome back to the office! Did you have a good holiday season?

Motley: Yeah, it was a good break. Although most of the time was spent with family and friends, I did do a bit of an analysis of some previous estimation data.

Maven: You have me curious - what's up?

Motley: A few months ago you gave the team some pretty good advice, I must admit. We used Wideband Delphi Estimation as you talked about before to estimate some key features for the product. The team used Scrum and gathered a bunch of data on how long it actually took us to implement the features, and I thought I would show it to you.

Maven: Sweet! Glad you were able to put it to use and see some positive results. Don't keep me in suspense!

Motley: Hold your horses, Mr. Impatient. First, a bit of background. This feature team was responsible for developing a feature with a bunch of subcomponents. We used Wideband Delphi to estimate all of the subcomponents early on in the project when we really did not know a whole lot about what we were estimating. Wideband Delphi was perfect for this exercise due to the large amount of project variability at the beginning.

Maven: How many people were involved with the feature team?

Motley: There was one program manager (PM), two developers, and two testers for the majority of the project to this point. That's not relevant for this analysis, however, as all work was estimated for one developer and computed the hours across all developers.

Maven: Makes sense. Any more background?

Motley: Yeah. One thing I would recommend to everyone is to have whoever is responsible for the initial high-level requirements put together a short 2-3 page document that describes the feature at a basic level, including some mock screenshots if necessary. It doesn't have to be highly detailed but should provide a high-level idea of what we will be building. Think of user stories, but to a slightly lower level of detail (but not much). Then distribute the doc to everyone involved in the estimation session to prepare. Have the author of the short document give an overview just prior to following the Wideband Delphi process to set context.

Maven: Great suggestion. Who did you invite to the session?

Motley: We invited the PM, the two developers, one tester and one senior developer from another team. One thing that worked really well was having that expert external to the feature team to help bring another perspective.

Maven: Okay - let's see the data already!

Motley: Hmmm… maybe I should make you wait a little longer. I love seeing you squirm.

Maven: You really are a mean person.

Motley: Just the way I like it! Fine. Fine. Here are a couple of graphs of the data I gathered:

Maven: Not bad data, Mot! Looks like many of your estimates were fairly close to the actual work on inspection. How do you explain the estimate for feature 6, which was quite far off?

Motley: That's easy - what we thought we were going to build at the beginning of the iteration actually turned out to be quite different. We did the estimate fairly early in the project and made some key assumptions, but later we discovered a couple of those assumptions were not valid. As a result, when we planned the iteration, we revisited our assumptions and realized we had more work than we thought we did. Don't forget - I am comparing our original estimates at the start of the project to our actual numbers. Secondarily, a dependency that we relied upon for that feature was late on their deliverable, so we did a bit of extra work to mitigate that.

Maven: Sounds reasonable. What about intuition - do you feel that Wideband Delphi gave you increased accuracy in your estimates.

Motley: Overall, gut feel is that Wideband Delphi gave us greater accuracy than we would have gotten had we left the estimate up to one developer. Everyone was happy with the process and has committed to keep using it.

Maven: As we talked about in the past, Wideband Delphi helps the team get on the same page at the start of the project. Did you observe that?

Motley: Wideband Delphi really helped us get on the same page with assumptions and come up with an initial direction people were happy with. Even though some of our assumptions changed later in the project, for long-term planning purposes at the start of the project, the data was accurate enough for the release management team to make reasonable predictions.

Maven: So you would use Wideband Delphi again?

Motley: Absolutely. You hit a home run with that one, Mave.

______________________________

Maven's Pointer: Our team at Microsoft uses Wideband Delphi consistently for early-project planning. When the management team forces us to come up with estimates that help define the milestone plan for the next 3 months, we use Wideband Delphi to generate not only fairly accurate numbers but also document the rationale behind the numbers. Even though detailed planning and scheduling for the next 3-6 months is fraught with peril - a realization agile teams come to - to be an agile team in a waterfall organization you must periodically give in to the will of the release management team.

Maven's Resources:

Agile Estimating and Planning, by Mike Cohn, Prentice Hall PTR, ISBN: 0131479415, November 2005.

Motley says: "Milestones are useless for agile development"

James Waletzky — Tue, 24 Jun 2008 17:55:00 GMT

Summary

Motley: Milestones are useless for agile development. Our feature team can ship at the end of every iteration, so milestones have no value for us.

Maven: Milestones provide a synchronization point across a set of features, helping to ensure the overall product is of high quality. Every milestone has a set of exit criteria that the integrated product must satisfy before moving on.

______________________________

[Context: Motley is analyzing his team's upcoming deliverables and is questioning some of the Cynthesis Software development process]

Motley: Overhead, overhead, overhead. Why is the project manager making our team go through all the hassle of hitting a milestone when we are doing iterative development anyway? We have mini-milestones every time we finish an iteration. We produce a shippable piece of functionality that is ready to go. Milestones are useless for us.

Maven: Actually, milestones have use even in an agile world. You are not the only team contributing functionality to the release, right? There are other teams building features?

Motley: Yeah, so? Some of the other teams are doing agile as well so they likely have the same questions as we do. For the others not doing agile, well, they need to wise up and get with the program.

Maven: It's not so easy to change a company, as we discussed in previous conversations on change management. At some point you need to bring all the feature teams together and make sure everyone is synchronized. That's one of the primary purposes of a milestone. It is a checkpoint that all teams have in common with a set of exit criteria. A milestone brings everyone together to ensure all teams can integrate their features and come together to create a product. Milestones may not be technically necessary if there is one small team developing a release. You can ship at the end of every iteration, at least in theory. However, when many teams are involved in creating a large product, you need to periodically bring everyone together.

Motley: Milestones just seem too process-heavy. Why not just sync-up on an as-needed basis?

Maven: Don't forget - milestones also have use to people outside the company.

Motley: Why should anyone outside our development group care about our internal milestones? Perhaps your girlfriend needs to know so she can expect you to work some overtime around the end of a milestone, but other than that, I can't think of any reason.

Maven: With any significantly-sized product, Cynthesis typically ships technical previews, does integrated product demos for customers, and ships beta releases. Typically these interim releases prior to ship align with milestones to ensure product quality is high. Because each small feature team may have their own iteration schedule with their sprints, the milestones provide that date where sprints can align.

Motley: Fine. I can accept all that. It still seems process heavy though. You mentioned the phrase "exit criteria". That phrase has "formal process" stamped all over it. Pepto Bismol is my exit criteria after eating bad seafood.

Maven: Ah, dude, that really wasn't necessary.

Motley: I know, but potty humor is always fun anyway <laugh>.

Maven: Back to business. Yes, "exit criteria" is one of the more formal-sounding software engineering phrases, but it does have use. At the end of any given milestone there is typically a checklist of quality criteria that each team and the overall product has to satisfy before the milestone is considered done. Think of it as a team-wide definition of that ever-so-important definition of "done".

Motley: Ah, so it's a fancy phrase meaning that once we put all the pieces together that they all work well together. That would include overall performance, end-to-end integrated scenario testing, stress testing over time, lack of cross-product memory leaks, general usability and consistency across features, consistency of documentation, how it works with the rest of the system, and other general product health issues.

Maven: Exactly! We put all the pieces of the puzzle together and hopefully it produces a beautiful picture. Without that checkpoint you could ship a product with puzzle pieces that don't match creating a jumbled mess.

Motley: I see. Makes sense. One thing that I would encourage the project managers to do, however, is not make those exit criteria too strict and that we focus on the right things.

Maven: That's a very astute observation, Mot. I worked in one company where the exit criteria list was pages and pages long. After about the first third in priority order, the rest just created needless overhead and diminishing returns. It felt like the project managers were micromanaging the feature teams. Instead of speeding up the development process, it slowed everybody down, in my opinion.

Motley: Ouch. I guess we should consider ourselves lucky here at Cynthesis. I can put up with a reasonable list of exit criteria, but you don't want to know what I do when someone tries to micromanage me.

Maven: Imagine nag nails for every little thing like high priority bugs you haven't updated in the bug database for a few hours, or nag mails because the project manager thinks you haven't triaged your bugs but your process is different than other teams, or people screaming at you because bugs are not assigned to a specific individual person and are instead on a bug backlog. I'll stop ranting now, but just know that these things can be a reality even in great software companies.

Motley: Enough! Enough already! You are going to make me start pulling my hair out just thinking about it, and I really don't have that much hair to begin with.

I understand the need for milestones. From my perspective of being on an agile team, we keep going as usual, make sure we integrate on a regular basis, be strict about our meaning of "done" for tasks, fold the exit criteria for milestones into our everyday development and guess what? Our team won't struggle through trying to meet exit criteria in the high pressure closing of a milestone. Smooth sailing.

Maven: That's the spirit. Your team is going to rock.

______________________________

Maven's Pointer: Learn the intricate details of the milestones in your company. At Microsoft, practically every team has a "code complete" milestone. Most developers take that milestone for granted. However, the definition of "code complete" is different on every team. Do all features need to be done? Are to-do's allowed in the code? Do you need to run static analysis prior to code complete? Is some measure of API documentation required by the milestone exit? Learn what "code complete" and the other milestones mean on your team and address the exit criteria early.

Maven's Double Pointer Indirection: Avoid cheating on the exit criteria when the end of a milestone draws near and the project manager does not want to change the date. You are only cheating yourself. Do your best to meet the exit criteria and maintain high product quality even in early milestones. Revisit the exit criteria for future milestones if the exit definition is too strict.

Maven's Resources:

None this time

Motley says: "Bug fix sprints are a Scrum anti-pattern"

James Waletzky — Tue, 27 May 2008 17:57:00 GMT

Summary

Motley: Bug fix sprints are a Scrum anti-pattern. Quality should be kept high so as not to have to focus on bug fixing.

Maven: A clear meaning of done for sprint tasks is important, but even if you follow this best practice, there will always be integration issues, bugs, and changes required late in a cycle. A sprint focused on these tasks is reasonable.

______________________________

[Context: Motley and his team of merry developers have been fixing bugs for the past week, and Maven has noticed something odd]

Maven: Morning, Mot. How are things going?

Motley: Not too bad, although I'm a bit late getting in this morning. It's tough to get out of bed when all you have to look forward to is fixing bugs for the next couple of weeks. The bug database is the team's to-do list for the next little while.

Maven: Yeah, this is a part of the development cycle that is rarely a developer's favorite. Is morale still high in the daily Scrum meeting?

Motley: Daily Scrum? Why bother? We are done with Scrum now that we are integrating and bug fixing. Besides, bug fix sprints are a Scrum anti-pattern. Didn't you tell me the other day that Scrum only works for new feature development?

Maven: Um, you must have heard that statement from someone else, or misunderstood something I said. I would partially agree with your statement that bug fix sprints are a Scrum anti-pattern. In fact, most of the Scrum/agile purists would likely say that you are absolutely right. However, past experience and the reality of software development have made me realize that bug fix sprints can be a necessity in large organizations practicing agile development.

Motley: Why?

Maven: One of the most important concepts of Scrum, and where many teams fail, is having an explicit meaning of done for each and every task on the sprint backlog. For example, a typical coding task may involve the following in its "done" definition:

Design document reviewed (if necessary)
Feature fully implemented (including error handling, performance, reliability, security, maintainability and other quality concerns)
Buddy build complete
Buddy test complete
Unit tests written
Code coverage from unit tests > 80%
Static analysis run
Code reviewed by at least one peer
Code checked-in to the source tree

Motley: That's getting pretty detailed isn't it? What's the point?

Maven: The point is that we all have a common understanding for what it means to complete a specific coding task, and understand what "shippable quality" means. If you do not explicitly define the task, some tasks may get missed costing you valuable time and effort later in the cycle when change is more difficult. With agile, we want to produce high quality deliverables very early and not rely on a long "stabilization" phase at the end. If we do everything we can to prevent bugs up front, we'll actually save time in the long run and theoretically ship sooner. Having defined meanings of "done" for all our tasks is a big key.

Motley: Which reinforces my point that a bug fix sprint is an anti-pattern. Jeez, Mave - did you not have your morning jolt of caffeine?

Maven: If everything goes according to plan and you integrate components as you go, then yes, longer bug fix periods are somewhat of an anti-pattern. But let's be realistic. There is some measure of integration and late discovery of unknowns in any large software project. For a small project of a few thousand lines of code, you may be able to avoid a bug fix stage like this. However, take James' experience in working for a previous company. One thing his team was responsible for was porting a large code base from one platform to another. The team added unit tests where possible but much of the code was inherited legacy code that did not come with tests. The test team worked closely with the development team to find as many problems as possible up front, but when you have tens of thousands of lines of code running on a platform that is changing underneath you, you must expect to find bugs later.

Motley: Not to mention when people in the organization start hammering on your code as real users would. I think James called that "dogfooding" when I talked to him, as in eating your own dogfood. That does not sound very appetizing.

Maven: Correct. You cannot possibly predict all the issues that will arise and fix them proactively. As a result, his team undertook what they called a "quality sprint", which focused on fixing bugs. They took an agile approach and refused to develop new features until the product was at shippable quality, and they hit ZBB.

Motley: What the heck is a "ZBB"?

Maven: "Zero Bug Bounce". It's a moment in time where you have no bugs that are older than, say, three days and there are no ship stoppers. At this point, the team can make a call that quality is high and new features can be developed.

Motley: You called it a "quality sprint". You could also call it an "integration sprint" depending on what your goals are. In fact, that might be a better name for what we are doing now.

Maven: Agreed. The term I don't like ,however, is "stabilization sprint". To me, this implies that the code you wrote was crap to begin with. In reality, there may be things beyond your control that crop up upon integration even though you focused on quality, so I prefer to give it a more positive spin. "Stabilization" also typically implies a huge focus on testing. Testing is part of what happens here, but testing is also happens in every sprint.

Motley: Game it if you want. If it looks like a duck, and quacks like a duck, well, you know. You still have not answered the question as to why I would want to practice Scrum when I am fixing bugs!

Maven: You don't necessarily have to practice Scrum while bug fixing, but you may find it valuable. When your team needs to fit within a larger ship cycle on a big team, you may not have full control over how to structure your bug database. In addition, your team may live within the confines of a waterfall-oriented organization, but you can be agile as a feature team by practicing Scrum and transferring prioritization of fixes to the sprint backlog. The team does the usual sprint planning figuring out which bugs are most important to fix. The sprint backlog is then a prioritized queue of bugs, and devs grab bugs from the queue as soon as they are done fixing their current bug and validating with test.

Motley: Is it really worth the overhead though? I already have my list of things to do in the bug database. Why duplicate it?

Maven: I generally don't consider Scrum as "overhead" - it's a very lightweight process, although I would agree that with any process you arguably get some level of overhead. You could fairly easily write a tool that automatically populated the sprint backlog with the top bugs from the bug database. Note, however, that you also gain the following benefits by practicing Scrum while fixing bugs:

The daily meetings ensure improved communication and collaboration
Data gathered is useful for future planning sessions
Frequent retrospectives allow the team to constantly improve
The team can adjust priorities during the sprint
The team need not rely on a rigid bug database schema, and can use the sprint backlog to customize information about the bugs
The team remains focused, and has a Scrum Master to help

Motley: Fine, fine, fine. We'll try it your way for a sprint and see if it pays off. I did kind of lose track of what Marvin was doing with his bugs and the daily meeting should help.

Note: I'd love to hear some better suggestions on how to deal with the problems discussed in this story. Arguments based on reality (vs. theory) are much appreciated in comments!

______________________________

Maven's Pointer: Here's another argument for tracking time spent on tasks in a sprint: over several short quality-oriented sprints you can gather data on the amount of time it takes to fix each bug. By mining that data, you can compute an average length of time per bug, which helps you with effort estimates for capacity planning on future bug fixes. Our team, for example, knows that it takes just under 5.4 hours of effort to fix an average bug. We can then estimate how many we can fix during any given sprint, and set expectations accordingly.

Maven's Resources:

Scaling Testing in Scrum, by Bob Galen, http://www.agile2007.org/downloads/handouts/Galen_532.pdf

Motley says: "What are we burning down? Your house?" (Scrum Part V)

James Waletzky — Tue, 19 Feb 2008 20:09:00 GMT

Summary

Motley: A burndown chart? What are we burning down? Your house?

Maven: Track time remaining for tasks to generate a burndown. Combine time remaining with time spent to generate a cumulative flow diagram with burndown. Use the charts to get an accurate picture of project progress.

______________________________

[Context: Maven and Motley just arrived at work in the morning and are continuing their discussion on Scrum, this time discussing data analysis]

Maven: Morning, Mot! Want to pick up where we left off in our Scrum conversation?

Motley: Does Superman want kryptonite for breakfast?

Maven: C'mon - this is important stuff. Let's talk a bit about data gathering in Scrum.

Motley: Sounds about as exciting as cleaning the bathroom. I've got stuff to do.

Maven: This won't take long. Previously we talked about the importance of retrospection for a sprint. You had some questions around why we gathered data during the sprint. Well, the basic data that we gather every day is the amount of time remaining on our tasks. We want to be realistic with our estimates - as we execute, we feed the learning back into the plan and make adjustments. For example, if we underestimated a task and it will take longer than expected, we may want to cut some tasks off the bottom of the priority list to set expectations with our stakeholders that we won't deliver everything we set out to deliver. Tracking our new estimates on a day-to-day basis gives us a burndown chart.

Motley: What are we burning down? I'll bring the matches.

Maven: Time in the sprint. We use the burndown chart to get an idea of how the project is trending and whether we are going to meet expectations. The goal is for the burndown line to reach zero at the end of the sprint.

Motley: I guess given the burndown and the fact that we were able to complete a certain number of hours in a sprint, we can use that data to help us plan next time.

Maven: See? I knew you were enthusiastic about this! You are absolutely correct.

Motley: Am I never not correct? I thought you would have learned by now, Mave.

Maven: You need a bit more humility, Mot. Like a friend of mine always says about himself, "I'm a 9, because nobody's perfect." Errrr… wait. Maybe that's not a good example. But I digress. The burndown also works on the product level, so that over several sprints you can see how your overall release is trending.

Also note that you need not estimate in hours as the chart above indicates. Many agile teams use "story points."

Motley: "Story points"? What the heck is that? I am building software, not telling a story!

Maven: Story points are essentially a unit-less measure that describe the size of a requirement in relation to other requirements. Think of a story as a basic requirement written as a scenario from the user's perspective. The reason we use a generic measure like points is to get us out of thinking about hours. The important concept here is that we understand how much work we can do in a sprint, i.e. our velocity, and use that measure to plan future sprints. Story points remove the focus on time and remove the desire to compare across contexts or other teams. The story points become unique to the particular team.

Motley: But how could I possibly have any idea how many "points" a feature will take to build?

Maven: It's tough at the beginning when you don't have anything to compare to. Once you know your team's velocity and how many points you can deliver in an iteration, it becomes more natural.

Motley: Sounds interesting, but I think I'll stick with hours, thank you very much.

Maven: That's fine - whatever works. Let's get back to the data. While the burndown chart is very useful, you can generate much more valuable data if you not only track time remaining on tasks during a Daily Scrum, but also track time spent.

Motley: You have got to be kidding. Tell me you're kidding. You want everyone on the team to report time spent as part of their status?? I am telling you right now that no one on the team is going to like that. They will feel as if their manager is closely tracking every hour of their day. It will lead to micromanaging, morale will decrease, and people will be afraid to take a break. Not going to work.

Maven: I admit, it's controversial, and to be honest, outside the core Scrum process. However, you get much richer data. Plus, the data is for the team only - no one outside the core Scrum team gets access to the individual data. All that matters is the roll-up of time spent. Let me give you an example of a valuable pictorial representation of the data. The chart below is called a "Cumulative Flow Diagram (CFD) with Burndown".

Motley: I'm not convinced on gathering effort expended, but the chart looks interesting. Tell me more.

Maven: What do you think happened in this project?

Motley: Well, the red line seems to be the same as the burndown you showed earlier. There was a hump around day 16, which could indicate that the team underestimated or the plan grew. The grey indicates the total combined hours of tasks that the team has not started, the yellow the number of hours of tasks in progress, and the green the number of hours corresponding to tasks that have been completed. This team had lots of work in progress, had a slow start, grew and then later shrunk the plan, didn't finish all their tasks, and had some areas of poor productivity. Lots of things to look at in a retrospective.

Maven: Nice analysis! See how straight forward the chart is to follow? You can get some great observations out of one diagram. You mentioned that the hump may be due to underestimating or plan change. We can tell for sure by coupling this CFD based on hours with another one having a y-axis of number of work items. A corresponding hump in the work items CFD indicates that the plan grew. Notice anything about the yellow area?

Motley: It's big, but I don't see a problem with that.

Maven: A wide yellow area is generally not a good thing. With a chart like this you only get credit for work that is completed. The minute you log an hour on a large task, you end up with a large yellow area. Let's talk about the danger of that large yellow area later.

Here is another example of a CFD with burndown:

Motley: Wow - this team significantly over-planned. They didn't finish many of the tasks they set out to do. There is also a flat spot that we want to analyze. Why was the team unproductive during that time? Was there a major distraction? They did adjust the plan somewhat, but perhaps not enough. Actually, now that I look closer, this isn't a good sprint. Although they had a good velocity throughout the sprint, it lasted way too long and they didn't adjust the plan enough based on new knowledge. Stakeholders may have been expecting more to be delivered by the end if they checked-in midway. Bad Scrum Master.

Maven: Nicely done, Mot. You would almost think you invented that graph.

Motley: I am brilliant, of course. However, I cannot take credit for inventing that graph. They are easy to understand though, provided we can convince the team to track time spent. I'll see what I can do.

Maven: Just reassure them that the data will not be used against them. In addition, it doesn't take much extra time to track time spent in addition to time remaining. It does allow us to get a complete picture of the project over a short iteration and adjust the plan on-the-fly.

Motley: One more thing - I am still not quite seeing why a wide yellow area is a bad thing.

Maven: Let's save that discussion for another time.

Motley: Ah, that's typically code for "I don't have a clue".

______________________________

Maven's Pointer: There are generally two reasons why wide yellow areas are a sign of trouble. First, it may be a sign that the team has too many tasks in progress at one time.

Human beings are terrible at multi-tasking, but more on that later. Second, it may indicate that the tasks on the sprint are not broken down enough. For example, if you put 1 hour against a 40 hour task, you get 40 hours worth of yellow in the CFD. Estimate tasks from 1 hour to 16 hours in length. It is easier to define "done" and shows that you have more completely thought out the task. Otherwise, the team does not really know what is involved in completing that task, and the estimate is far less accurate.

Maven's Resources:

Tool: Microsoft Scrum Kit (http://go.microsoft.com/fwlink/?LinkId=98602&clcid=0x409): An Excel-based tool that James helped write for internal Microsoft use that is now available via the web through the most excellent "Hard Code" book by Eric Brechner, Director of Development Excellence at Microsoft. See the "Chapter 2" directory in the download.
Using Cumulative Flow Diagrams, by David Anderson, May 2004, http://dn.codegear.com/article/32410
Managing Lean Software Development with Cumulative Flow Diagrams, David Anderson, May 2004, http://bdn1.borland.com/borcon2004/article/paper/0,1963,32096,00.html

Motley says: "I fear public speaking more than I fear dying!"

James Waletzky — Tue, 11 Dec 2007 17:40:00 GMT

Summary

Motley: Speak at a conference?!? Are you insane?!? No public speaking for me!

Maven: Speaking at a conference is a great way to build confidence and influence not just your company, but the industry.

______________________________

[Context: We'll continue with HPT Part 3 next time. For now, Motley and Maven's buddy James just returned from speaking at a conference]

Maven: Hey - did you hear James just returned from a conference? He actually did a speaking gig at the Agile Development Practices 2007 conference in Orlando from Dec 3-6. He said it went quite well.

Motley: Crazy nut job. Don't you know that the average person fears public speaking MORE than dying? First time I heard that statistic I was quite shocked, yet not surprised all at the same time. There's no way you'd catch me doing that.

Maven: Ah, but it's great experience! What better way to practice your communication skills, meet influential people in the industry, make new contacts, help others, and have a great time while you are doing it! Plus, when you speak at a conference they usually compensate you a little, reimburse some of your fees, and give you free conference registration. Lots of fabulous benefits.

Motley: if you say so. Not sure I could get up in front of 100 people or more and do it. I think I'll expand my scope of influence and work on my communication skills in some other way. What did he speak about anyway?

Maven: A great topic: "Balancing Emergent Design with Big Design Up Front (BDUF)". Here is the abstract:

"Big Design Up Front (BDUF) is a design technique that has been part of the development cycle for decades. Unfortunately, fully specifying a software design in the presence of change without a crystal ball is rarely effective. Agile principles and practices leverage feedback-oriented techniques such as emergent design to embrace change and design “just-in-time.” By balancing BDUF and agile emergent design practices such as test-driven development to avoid “cowboy coding,” we can develop just enough design documentation to guide our development toward the project’s big-picture goals. This balanced approach has been employed successfully at Microsoft to develop large software systems. James Waletzky discusses the pitfalls of BDUF and how agile methods help you reduce design risk. Learn what emergent design is and is not, how refactoring keeps designs clean, and ways to document your design with “just enough” detail. James introduces tools to help your design efforts, including XML comments, Sandcastle, and a C# Design By Contract language extension called Spec#. Take away practical design techniques that will improve your software designs in a world where predicting the future is impossible."

Pasted from <http://www.sqe.com/agiledevpractices/Concurrent/Default.aspx?Day=Wednesday>

Motley: Well, that does sound like a cool topic given other conversations we've had. Can I see the slides?

Maven: Sure. Here they are:

http://waletzky.no-ip.com/download/BalancingEmergentDesignAndBDUF.pdf

NOTE: This is a low-bandwidth connection. Let me know if you want me to e-mail the slides to you in PPTX format.

Motley: There's some good stuff in there. I like the little script. Think it's based on us?

Maven: I wouldn't doubt it. James seems to listen in quite a bit. He was wondering if any of our eavesdroppers have any comments or questions about the topic. Should we pass the word that he would like to hear from them.

Motley: I think you just did. Pay attention, Mave!

______________________________

Maven's Pointer: At Microsoft, to get to the more senior ranks of software development, you have to do a few things well. One is to be technically very good at what you do. Two, you have to expand your scope of influence. This may require some explanation: instead of concentrating on just feature development at small granularity, you have to influence multiple features, a product, a product line or the industry. Three, you need to have good communication skills. Speaking at an industry conference with reputable speakers is a great way to get that experience. I strongly recommend it. Once you get over your initial fear of public speaking (ugh), it's actually quite fun!

James' Pointer: I really enjoyed the Agile Development Practices 2007 conference. If you ever get a chance to go to an agile conference, go. You'll meet lots of great people and you will learn a lot about software development. The Agile Development Practices conference is is happening again in November 2008 in Orlando, and another big one entitled Agile 2008 is from August 4-8, 2008 in Toronto.

Maven's Resources:

Agile Development Practices 2007 Conference Web Site: http://www.sqe.com/agiledevpractices
Agile 2008: http://www.agile2008.org

Motley says: "The best way to control build breaks is to slap around the responsible developer"

James Waletzky — Tue, 11 Sep 2007 19:03:00 GMT

Summary

Motley: The best way to control build breaks is to slap around the responsible developer.

Maven: Continuous integration is an agile technique that minimizes daily build failures and minimizes integration problems in real-time.

______________________________

[Context: Motley is expressing some frustration that his team keeps breaking the daily build]

Motley: Damn! Yet another broken build last night. What is going on with this team? Lately we've had some really bad check-ins to source control causing issues with our daily build. The daily build is our lifeline here - if a build is broken that means a good portion of the team is blocked. ARRRRGGGHHH!

Maven: Calm down, bud. Why are the builds breaking?

Motley: Combination of things. Sometimes people are sloppy and forget to add a file to their source control change list. The build works locally on their machine, but once another machine syncs and builds, all hell breaks loose. Other times, it is obvious the developer has not run the unit test suite prior to check-in and there is a functionality break. A further problem is conflicting check-ins between developers causing havoc when the central build runs. In addition, there are times when the developer obviously has not run static analysis tools, which could have found a few problems even before check-in.

Maven: All common problems in many software companies. And all should be preventable.

Motley: I agree! It's just sloppiness and I am getting tired of it. You trust devs to do the right thing and it doesn't happen. Time for the hammer to come down.

Maven: Don't jump to conclusions just yet. Devs get put under lots of pressure to meet deadlines and periodically things get overlooked. What we need to do is put a system in place that helps them limit their mistakes without much in the proactive action relying on memory.

Motley: Me slapping them upside the head ought to be enough for them to remember!

Maven: The only time violence solves any problems is, well, I can't think of a scenario. Anyway, have you heard of "continuous integration"?

Motley: I'm not quite ready to live with my girlfriend, thank you very much.

Maven: Always the comedian! One recommended practice in agile development circles, most notably Extreme Programming, is continuous integration. The idea is that developers should be integrating their code together at regular intervals to head off build breaks before the daily build takes place. Leaving integration until the daily build happens or worse, until the end of a coding cycle, is a proven recipe for disaster.

Motley: I agree, in principle. But, how do you do it? Do I have to walk around and make sure syncs and builds happen on a regular basis?

Maven: Definitely not - tools to the rescue! One popular application in the .NET community is a free tool called CruiseControl.NET. You set it up on a server somewhere and have it monitor your source control system for check-ins. It works with many different types of source control systems, including Perforce, CVS, SourceSafe and others. Once a check-in happens, the server takes over and performs whatever actions you want it to - most notably, a sync and build of the complete system. You can even set it up to run other actions like a unit test suite, static analysis, or others.

Motley: Sweet! You mean once the server is set up it will do all of this on its own without manual intervention??

Maven: You bet! Individual developers can monitor the results by using either (a) a web site that they hit for integration status; (b) a Windows tray application that always indicates current build status (red icon = failure; green icon = successful run); or (c) an e-mail notification. It will track historical runs and tell you what change list triggered the run.

Motley: Oh, I must say, that is way cool! Now we don't have to babysit developers, and the code is being integrated constantly. That should definitely cut down on build breaks. How long does it take to set up?

Maven: All you need is a machine capable of performing builds, the CC.NET installation package, and an installation of IIS and ASP.NET if you want to use the web portal. Give it an hour or so to set up and you'll be off and running.

Motley: I am definitely going to try this out. It would be even cooler if Visual Studio had this kind of functionality built in.

Maven: Your wish is Microsoft's command! Visual Studio 2008 ("Orcas") Team Foundation Server supports continuous integration. You can specify a trigger for a build when you create a new build definition or modify an existing one. You can use on-demand builds, rolling builds, and continuous integration where each check-in starts a build. This all integrates with Visual Studio source control support.

Motley: Not bad, Mave. Now give me some more information on how I can get you out of my hair for a couple of hours so I can get this stuff implemented and communicated to the team.

______________________________

Maven's Pointer: Continuous integration (CI) is quickly becoming a mandatory practice for all development shops. Risk of breaking the software is substantially decreased, and as a result, the team becomes more efficient by minimizing blockages due to build breaks. In addition, you integrate as you go and avoid the "big bang" integration at the end. Also, since build breaks flagged by a CI tool are immediate and generally tied to one or two changes to the source, it is easy to narrow down the problem and fix it.

Maven's Resources:

CruiseControl.NET - free server-based continuous integration for .NET projects
Blog summary describing changes in Visual Studio 2008 Team System for builds
Extreme Programming practice description for continuous integration
Martin Fowler's description of continuous integration (great write-up)

Motley says: "Lean is for meat, not software"

James Waletzky — Tue, 05 Jun 2007 20:20:00 GMT

Summary

Motley: I like my steak lean. Software? Not so much.

Maven: Lean goes hand-in-hand with agile, and focuses on eliminating waste, learning, building in quality, deferring commitment, delivering fast, respecting people, and optimizing the whole.

______________________________

[Context: Maven has been doing some reading in his spare time and wants to share some ideas with Motley]

Maven: Hey Mot, I've been doing some reading on Toyota and their product development and manufacturing techniques. It's pretty cool stuff.

Motley: So what does car manufacturing have to do with me? I am a software developer. You pretend to be a software developer. Nothing to do with cars.

Maven: Au contraire mon ami. If your French sucks, you probably didn't get that. Anyway, Toyota invented something called "Lean Development" that has all sorts of applicability to software. In fact, agile development - of which test-driven development is an agile practice - has many principles in common with lean development.

Motley: I like my steak lean. Software? Not so much.

Maven: Ah, but wait. Hear me out. Lean Development embodies certain guiding ideas and insights (i.e. principles) about engineering and manufacturing that lead to extreme efficiencies in the development of a product - regardless if the product is a car or software.

Motley: Jeez, more principles. Don't we have enough principles to live by with agile development? Why more?

Maven: Well, in fact, the principles from lean and the principles from agile play well with one another. Lean is all about developing products quickly and efficiently while creating as much value for the customer as possible. You let quality drive your speed by building in quality up front and with increased speed and quality comes lower cost and easier maintenance of the product moving forward. Care to hear the core principles of lean development?

Motley: I have come to know that if I say "no" we both know that you never accept "no".

Maven: I love your enthusiasm. There are 7 key principles. The first one is eliminate waste.

Motley: Yeah, I do that every morning before my shower.

Maven: You're pretty funny. But seriously, much of what we do generates waste. Examples of waste are extra features that we build that never or rarely get used, documentation that we generate for no useful purpose other than our process requires it, and any barriers in communication imposed by our environment. The second principle is focus on learning.

Motley: I learn a lot by watching "Are You Smarter Than a 5th Grader?" every night. Great show.

Maven: Let's go deeper. We always want to learn throughout a development cycle and apply that feedback to improve our processes. One of the agile principles is continuous improvement, so this fits right in. A scientist would never do a huge amount of analysis and risk everything on one guess at a theory. Instead, they form a hypothesis, conduct many quick experiments, learn from the results, and iterate until they find the best alternative. Same for software - short quick iterations that we can learn from. Yet another principle from agile. The third principle is build quality in.

Motley: Quality from the start is definitely something to take from agile development. I am a big believer.

Maven: You bet. We want to focus on avoiding finding defects in our quality assurance process. We've been talking a lot about quality-driven practices like Test-Driven Development and test automation, so no real need to go into more detail here. You are the example everyone now looks up to! The fourth principle is defer commitment.

Motley: Thanks for the kind words! Hmmmm…. "defer commitment." Sounds like something I do on a regular basis with my girlfriend. I definitely respect her viewpoint there, but deferral is definitely the way to go for me.

Maven: Although not necessarily the best advice for dealing with your girlfriend, it is great advice for software. Dependencies, for example, can get us in all kinds of trouble if they don't deliver. It's a big risk. We want to avoid taking them on as long as possible and have a contingency plan if things fail. We also want to embrace change - another agile principle - and that means we cannot rely on customer requirements being right early on. We don't want to solidify customer requests until we're just about ready to ship the functionality. Change is good from our perspective, and big up front requirements analysis phases lead to waste and expensive change later. Another principle is deliver fast.

Motley: Pregnant mothers want doctors to live by that rule.

Maven: Exactly! Software too. Delay is an evil waste in lean development. Delay leads to developers not being efficient. We want to minimize delay and deliver software in small chunks as quickly as possible. Speed baby! Minimize the overhead in the form of queues between organizations and disciplines. Test should not be standing around idle waiting for the code from the developers. We also want to focus not on doing many projects in parallel, which leads to context switching and thrashing, but instead do one at a time to a clear meaning of "done." The next principle is respect people.

Motley: Sing it Aretha!

Maven: You stated earlier that even though you defer commitment you respect your girlfriend. You're right on it! The idea is to empower people to make the right decision and have leaders in place that are willing to do that. Trust is a big factor here. We'll talk more about empowering the team later on when we talk about Scrum. The last principle is optimize the whole.

Motley: What do you mean by "the whole"? If we optimize the development team, for example, we'll make the rest of the organization more efficient. After all, I am a machine! Hehehe.

Maven: To fully minimize cycle time, you need to address the system as a whole. You find the constraints in the system, elevate them, and address them. If optimizing one small piece of the system leads to delays elsewhere, you really haven't fixed anything.

Motley: Well, there are some interesting concepts in there, and I see how the ideas can help development, but where does the rubber meet the road?

Maven: We've already been putting some practices in place that exercise lean principles without even thinking about being lean. That's the cool part. Let's save the discussion of lean practices (vs. lean principles) for another time. Some of your practices, like TDD, already build on these principles.

Motley: So to summarize, the lean principles you mentioned are:

Eliminate waste
Focus on learning
Build quality in
Deliver fast
Respect people
Optimize the whole

Motley: In the meantime, I feel like a steak. Care to join me? No? Bummer.

______________________________

Motley's Pointer: Another key idea in Toyota's Product Development System is set-based design. If a new brake system is needed for a car, for example, three teams may design solutions to the same problem. Each team learns about the problem space and designs a potential solution. As a solution is deemed unreasonable, it is cut. At the end of a period, the surviving designs are compared and one is chosen, perhaps with some modifications based on learning from the others - a great example of deferring commitment until the last possible moment. Software decisions could also benefit from this practice to minimize the risk brought on by big up-front design.

Motley's Resources: Lean Software Development: An Agile Toolkit, by Mary and Tom Poppendieck, Addison-Wesley, ISBN:0321150783, 2003, http://www.poppendieck.com/ld.htm

Motley says: "Agile just means going fast"

James Waletzky — Tue, 29 May 2007 21:16:00 GMT

Summary

Motley: Agile software development is all about going fast.

Maven: Agile development leads to high quality efficient development, and is based on these principles: simplicity, embracing change, incremental iteration, continuous feedback and improvement, collaboration, and eliminating waste.

______________________________

[Context: Reflecting on his recent development, Motley is wondering what 'agile' really means]

Motley: Hey Mave, check out this photo of my cat, Mr. Pnurf.

Maven: Whoa! That's extreme close-up dude! He doesn't look too happy!

Motley: Oh, he's happy. You just can't see the smile on his face. He was just outside scampering along on the top of the fence chasing birds. Quick… like a cat, I suppose. As I was watching him, I kept thinking to myself, "man, that's one agile creature!". His reflexes were lightning quick. That got me thinking… that's just like software. You've been sharing some of these agile techniques with me like Test-Driven development. They're all about going fast!

Maven: Well, your cat definitely sounds agile. Just keep in mind - both for cats and for software - agile is about much more than being quick. The "quickness" is kind of a side effect.

Motley: The bottom line is that we - both the cat and me - are quick at what we do.

Maven: Yeah, but what gets them that way? For the cat, there are fundamental concepts that make them agile - their muscle make-up, stealthy movement, and predatory instinct. They need them all to catch all those little birdies.

Motley: Ok, so you're looking deeper. I'll give that to you - cats are not just quick. There are some fundamental concepts that make them quick. I guess with software there has to be some fundamental concepts that make development quick too, right?

Maven: You are soooo right, and I bet you already know what they are. Let's try and come up with a few principles behind agile development. There are various takes on what the principles are, but I bet we can come up with our own list that makes sense. Let's start. What is the best kind of design?

Motley: A good one? Hehehe. That's probably not what you were after. Well, good designs are often simple right? That makes them easier to implement, understand, and maintain.

Maven: You bet! Simplicity is a core principle of agile development. Always doing the simplest thing possible and only adding complexity when necessary keeps us quick. Let's try another. What's generally the biggest problem with waterfall processes?

Motley: Waterfalls make pretty photos. But for software, hmmmm… to be honest, I'm not sure. Waterfall processes have lots of good characteristics.

Maven: There are some solid ideas in waterfall, but what happens when I spend a few months specifying requirements, a few months designing, and then the user changes their mind?

Motley: Well, you're f-

Maven: Motley! This is a family-friendly conversation! But you're right - you have a lot of rework to do if you mess up and don't catch it until later. That's why embracing change is another agile principle. With everything we do, we cannot be afraid of change and have to adjust on-the-fly. The next one is-

Motley: I bet a I know. To cure the problem you just described with waterfall, what about doing a bunch of mini-waterfalls, so to speak. If we implement in smaller chunks, mistakes are not so costly.

Maven: You the man! That's great - incremental iteration is another key principle. Software should be shippable (as is realistic) and iterations limited to 30 days or less. Now, if you decide during an iteration that developers are not checking in code with high enough code coverage, what should you do?

Motley: You mean before or after you beat them? Just kidding of course. Bottom line: you fix it!

Maven: Right again! Continuous feedback and improvement is another fundamental principle. Always take a look back at what you just did, keep doing the good stuff, and fix the bad stuff. Ok, the next one involves a concept around a frequent activity that we do. What is it?

Motley: Well, you talk my ear off and I can't get you to leave me alone.

Maven: Yes! Well, sort of. We spend a lot of time communicating and collaborating. The more frequently your communication is with the team, the more in tune everyone is with the internal workings of the team. It's even better if you have frequent collaboration with the customer - then you help make sure that any change they want is incorporated sooner rather than later when the cost is less. Last one, Mot. What's the problem with a huge design document that no one reads?

Motley: There really is no problem if you use it for toilet paper. But alas, I guess what you are getting at is that you just wasted a bunch of time for no reason.

Maven: You are an agilista! We want eliminate waste whenever possible. Doing stuff that doesn't add value to our development only slows us down. We spend time doing activities that don't matter. Implementing extra rarely used features and doing long documents that are of no use are examples.

Motley: I have to admit, this all makes sense. I've been living by these principles for the past little while. In your typical annoying way, you just pointed out what I have already been doing and put some simple concepts on it.

Maven: You bet. It's not rocket science. If you live by the principles you can be as agile as your cat, and leverage them to make yourself a more efficient developer. Of course, there are values that go with the principles - like quality - but that's another discussion.

Motley: Great. Oh, and don't call me an "agilista" again or I'll pop you one. I don't know what it means, but it sounds weird. I'm heading home to learn more about software development from Mr. Pnurf. I wonder how I can apply throwing up a hairball to software development?

______________________________

Maven's Pointer: For guidance on what it takes to develop software with agility, check out the Agile Manifesto at http://www.agilemanifesto.org, which was written by a group of software industry veterans while out skiing for a weekend.

Maven's Resources: Exercising Agility, James Waletzky, MSDN Magazine, May 2007.