Update: this blog is no longer active. For new posts and RSS subscriptions, please go to http://saintgimp.org.

A recent internal email thread asked about the difference in philosophy between Agile development and Waterfall-style development (or anything that promotes BDUF).  There’s the Agile Manifesto which clearly articulates the basic assumptions of that movement, but what would a Waterfall Manifesto look like?

Someone observed that there are some actual ideas behind the Waterfall model that make it attractive to organizations.  Software costs a lot to develop so we should do everything possible to push down the cost of development.  If we can avoid making any major mistakes during the development process, that should logically drive down cost.  So let’s lay out a comprehensive, detailed design before we write any code, generate tons of documentation, and lock our feature set so it doesn’t change.  It all sounds good in theory but it usually breaks down in practice, and here’s why.

Waterfall-style engineering starts with the fundamental assumption that software development can be adequately modeled or approximated using some abstraction that’s cheaper, faster, or less error-prone than actual software development.  If you can satisfactorily solve the problem using this abstraction, then you just translate the abstraction into real code and you’re finished with less expense than if you’d simply written the code.

However, it’s starting to look like software development can’t be adequately approximated with any model that’s simpler than the development process itself.  The most efficient way to understand what a software system needs to look like is to actually build the software system in question.  Software development is fundamentally an experimental, empirical discipline.

An analogy might be weather forecasting.  We used to hope that with the right tools we’d be able to built perfect weather models that would tell us if it’s going to rain on Tuesday four weeks from today.  Turns out that we can’t do that, and barring a fundamental change in our understanding, we’ll probably never be able to do it.  The only way to find out if it’s going to rain on Tuesday four weeks from now is to wait four weeks (well, maybe four weeks minus a couple of days) and see what happens.  We don’t know how to build adequate approximations of weather systems that don’t catastrophically break down over time.

Of course, short-term weather forecasting is still useful as long as you understand that nothing’s guaranteed.  In the same way, planning documents and design exercises and the like are still useful in software development as long as you understand that they’re a pretty poor representation of reality and they’ll break down if you use them to predict the long-term future.  Use them to identify the next few steps in the process, take those steps, then build a new plan.