Requirements aren't evil, we are. Really. We, the software developers of the world are solely responsible for the problems of our industry. It's not the process, methodology, or whatever school of thought is winning today’s beauty contest with its sleek bikini elegance and wholesome customer love. It's not the ever more sophisticated tools that promise to simplify one of the most dizzyingly complex activities man has ever conceived. It’s us. Each and every man woman and child (do we allow children to code? Aren't there laws against that kind of thing?) that looks a customer, user, or stakeholder in the eye and says, in a most believable tone, "I understand, and together we will build <fill in the blank with whatever pie in the sky wishful thing your customer, user, or stakeholder is asking for>".
That's the moment of truth. The moment when customer expectations are set. When we tell them, with as much sincerity as we can muster, that they are not only a part of the team but the sole owner of the definition of success.
Read that last sentence again ... go ahead, I'll wait right here.
Do you see it? The hypocrisy the abdication of authority to the customer. The slimily, irresponsible, verbal sleight of hand that forces the burden of success onto the shoulders of the very person who has come to you for help. If that isn't evil incarnate I don't know what is. Don't get me wrong, I'm right there with you. I've done it. Worse yet, I've done it knowingly. After all that’s the business we’re in. Customers want something and we want them to pay us to build it, or at least try to build it.
We are unique, but not in a good way. Unlike the power classes that have come before we have had authority handed to us without the burden of contributing back to the system we have been so boldly authorized to defile. Contrast that with the titans of the age of enlightenment who did prosper on the backs of others but they did so while also providing socially beneficial innovations. Innovations in the form of new types of government, education for all, and my personal favorite, plumbing. Sure they were shunned as oppressive but in broad strokes they improved things. Even the power barons of the industrial age brought innovation to the masses. They built real, tangible things. OK, so they endangered workers with factory conditions that made smoking look downright good for you. More important, they used their own money (well, their families’ money) to control markets, exploit workers, and manipulate society. The consumers of their goods paid for things that were, by the manufacturers’ standards, complete and useable. This actually gave the barons of industry the moral high ground over the lowest form of capitalist in their day, the stock brokers. The brokers were peddlers of the wares produced by others. They taxed the transaction with little value added, essentially making money from money without material contribution.
That was then. Let’s get back to now, back to us. We, the titans of the information age don't produce anything tangible. We don't even come up with the good ideas: our customers do that. Yet we dare to demand payment for our willingness to try to build what they envision. We make promises of delivery that are statistically* not worth the air consumed to express them. Even our contracts fail the basic premise of a contract in every other industry. Elsewhere, in modern and older times, contracts provide a durable, binding way to ensure all parties knowingly and, on occasion, equally share risks and legal protections. We don't have building standards. No association ensures we are self-policing. No law requires our failures injure us as they do our customers. Did I mention evil?
I know what you're thinking. I can hear your voice in my head as I write this. "It's not our fault! How can we possibly be held responsible when the customer is unclear in their wants, needs, and desires?" Right you are. What are we, mind readers?
This is where I have to reintroduce our hero and villain, that favorite of all software development fall guys, the requirements. Requirements have, for most of software developments history, been our best defense against the endless ambiguity we suffer at the hands of our customers. After all, requirements make those contracts work. Oh how I love them.
In the hands of a master software developer requirements are both carrot and stick. They provide a means to entice the customer to participate. They can deeply affect the timing and value of the solution delivered. When seen as opportunities, not obstacles, requirements cradle a project in the warm embrace of constraints. They define boundaries within which the solution lays and beyond which no developer should trespass. They set the bar (achievable or not) that the team can focus on and on good days exceed.
Of course master software developers are few and far between. And to be fair so are really irresponsible, malicious, incompetent software developers. The world as we know it is, by its very definition, filled with average software developers. If you want to get really picky it’s the 75% middle portion of the bell curve, not the mathematical average but let's not drift too far off topic. The more important point is that we are focused on the bulk of the population of all software developers who do the bulk of the work.
Here's where it starts to get tricky.
I'm going to define an average developer as having three to five years of real experience. You know, paid work with customers they aren't related to, or somehow close to anyone they meet in college. During those three to five years it is likely that our average developer has only worked on one or two projects requiring more than four or five people including themselves. They have been on only one or two projects from the very beginning and even then are statistically* not likely to have stayed engaged from inception to delivery.
Consider the now infamous chaos report. Over the last 10 years the industry has never done better than 35% of projects started actually ending in success. If you are willing to accept that as true, and you should ... I read it on the internet... then it is not too much of a leap to believe 65% (or more) of the industry has NEVER delivered successfully. They have never tasted the blood in the water that is success. What form of insanity is it that we believe this average software developer could ever deliver on time, in budget, and within scope?
And what about all those project leads, technical leads, project managers, program managers, high end consultants and coaches (some of which spent as much as two full days achieving the vaulted Master credential), that bring their vast experience to bear guiding and molding our average software developers into better, more capable members of our little IT society? What do they bring to the table to better ensure success? Let’s count the ways shall we?
First among all credentials is proof of experience, and yes, by proof of experience I mean resume. It is the rarest of seasoned IT professional that doesn't have 15 to 20 years of experience proudly displayed in chronological order. They have been working with computers since that first fateful college course where they spilled "that" box of punch cards and spent most of the night sleeping on the couch in the computer lab eating pizza and waiting to hear that their punch card program did or did not run. They have worked with a dozen big name companies leaving one very important project (before it delivered) to take on a more complex, more challenging project with an even bigger name company.
Impressive, yes. Comprehensive, you bet. And look at that font!
The trouble isn't what they have done but what they can do. Are you working with punch cards? I'd be willing to bet a Euro you aren't. Do the leading edge technologies they helped create years ago still lead anything? Probably not. Last, what happens when you ask what I believe to be the most important question of all; what did they materially contribute to their most recent success? Not what their team did, but what exactly did they do, with their own hands, under their own power. What problem did they correctly predict, and ultimately prevent or solve?
In a recent round of candidate interviews I couldn't help but replay a scene from that seminal movie of the 20th century, Office Space. When asked by the Bobs to describe his job our proto-typical middle manager proclaimed that he "... was a people person damn it!" [He] takes the requirements from the customer and gives them to the engineers." It is an all too common, and in my opinion, sad end to many otherwise brilliant technical careers when the decision is made to promote them into management. Expecting their hands-on delivery skills to magically transform into leadership and organizational skills is little more than a folly worthy of a Shakespearian tragedy.
If we don't have a vast army of experienced leaders who can guide our average developers to success, what else could we possibly provide them to improve their chances of actually delivering value to users? Every culture since the dawn of time has been defined by how lessons learned are passed from generation to generation. Being good at passing learning forward to enables subsequent generations to not only avoid the mistakes of their predecessors but to exceed their greatest achievements with even greater, uh, achievements is an indication of maturity our industry has yet to achieve.
Unfortunately software developers have more in common with the Neanderthal than modern man. While modern man has a lifespan creeping up on 100 years, allowing multiple generations to overlap, our current software developers struggle with generations as short as a few years and technology shifts likely counted in months.
The ever increasing speed of technical change has only amplified our problems. Lessons learned during one generation simply don't apply to the next. Memory management in the 80's isn't relevant to any but the rarest few who build the runtime compilers the rest of use to handle those low level tasks. Even those lessons that do apply are largely left to oral historians to pass through mentor / mentee relationships. I know that sounds ridiculous but as a group we have failed to create highly available, searchable repositories where less experienced software developers can find relevant information that would allowed them to avoid the mistakes of the past. The web while arguably comprehensive and searchable is a bit too volatile and fails to provide vetted, definitive information. When faced with a new challenge our average developer will likely take one of the top few returns from a search and try to blindly apply it to the problem at hand.
Our skills may be atrophying with each rapidly passing generation but that's nothing compared to the speed of change in user expectations. Long gone are the days when users were impressed by two documents side-by-side on the screen. Not only do they expect every solution to use the very latest (frequently untested if not unstable) technology but they want to be involved in the less technical portions of a solutions’ creation. They demand to dictate the colors, the layout, the features, the workflow, and more. After all who better to decide how to elegantly layout a user interface than the business subject matter expert who struggles with document page numbers and printing labels?
That brings us full circle to requirements. Once we agreed that a customer could have their way with a solutions’ design we needed to capture their ideas, both good and oh-my-god randomly insane. Way back in the 70s and 80s this wasn't as much of a problem. Technology was simpler and the wants of our customers were easy to describe in plain text. Those simpler days are long, long gone but we continue to both define requirements in plain text and allow users to drive the design of solutions.
You see, it’s not a failure of requirements management; it’s a failure of the requirements that haunt us. Contractual language is not expressive enough to capture what a system should / shall / must do. Current efforts to describe solution parts like user experience, navigation, workflow, business rules, component design, system constraints, system goals, as well as the limitless negative things a solution should not do, more closely resemble iambic pentameter than software design goals. Similarly the technical skills of past generations simply did not include the necessary soft skills that separate success from failure on today’s complex solutions. Customer management, team dynamics, interface design, and multi-disciplinary tasking are but a few skills that didn't even exist in the software development world until recently except for the truly rarified air that built multi-million dollar, sophisticated, large, complex and mission critical systems.
The answer lies, as I mentioned at the beginning of this rant, with each of us. Not only must we let go of the comfortable ways that have served us for so many years but we need to do what for many is simply unthinkable. We need to work together. We need teams of skills that are applied as part of a thoughtful and well-choreographed improvisational theater. We need to accept user input as rough cut story boards from which clear decisions will be refined. User input is not a legal or moral mandate handed down from on high. User requirements are just one of the many variables that will go into the stew that is a modern software solution. After all, this is far more art than science.
Our job is to deliver a high quality solution to the users. This is far more likely to be successful if the entire effort is an ongoing collaboration with the very same users. Just doing our job hasn't been, and continues not to be, enough. I believe it’s our mission to build upon the successes of the past, to leverage the lessons learned, and, with each and every solution delivered, improve our industries ability to take user requirements and turn them into more than they had hoped for.
The sooner we; * convince ourselves and our customers that the definition of success includes, but is not fully defined by, early stage requirements, * accept our role in managing expectations and driving non-user requirements and constraints into the definition of a complete, high quality solution * see getting paid on acceptance of a high quality delivery as the norm and shun those who accept payment for merely trying, the sooner we will achieve reasonable, consistent, and equal relationships with our customers.
As soon as we look to ourselves, our team, and our customers as the source of success instead of failure, then we will be free from the evil that requirements have become. In short, although I suspect I past short many paragraphs back, we need to participate. Much like any other part of the software development process, requirements need to be managed. I'm not talking about the formal requirements management process. I'm talking about acting with, for, and against them intentionally.
This should be the kind of problem software developers enjoy. It's a multi-dimensional puzzle with both slowly changing dimensions and temporal components. Requirements exist on multiple planes immune to simple-minded linear thinking. Requirements don't have a lifecycle. They are simultaneously known and unknown, both immutable facts and infinitely malleable concepts.
We routinely refer to requirements as high level or detailed, as core, original, technical, system, or functional but we fail to evaluate these and many other labels over time. Somehow requirements have become some kind of guide rail protecting our nearly out of control car from careening off the well-known road to success and exploding into the horrific but certain-to-have-killed-all-occupants ball of fire used in every cliché car chase Hollywood has ever produced. Just like the movies, it’s an illusion if not more correctly branded as a delusion.
Guard rails won't prevent a truly out of control car from leaving the road and cars don't burst into flame on impact. Similarly requirements aren't going to keep your project in check and you won't fail if you move, change, or even disregard them. Safety comes from skillfully negotiating the turns in real time. It comes from looking ahead, sometimes far ahead, identifying threats both potential and real, and changing course.
That's the secret. Pay attention, make choices, and act on those choices.
If you want to succeed in software development, if you want to deliver value to real users, if you ever hope to mature from the current generation repeating the mistakes or prior generations, if you are to set aside the hubris that you are somehow in control and managing the chaos that is the creation of a software solution, you need to stop treating the needs of your customers as static. Whatever form requirements may take they are merely snapshots of the shadows cast by fleeting ideas. Good clues but hardly stable, contractual definitions of success.
This is particularly onerous on large projects spanning more than 6 or 12 months. Everyone learns, things are discovered, staff comes and goes, and priorities change every day from the very first. What kind of crazy is it to believe that this one aspect of the project is somehow immune? Software, really great software, while a reflection of the developers who create it, is an extension of the people that use it. People, not a person. People with a broad spectrum of wants and needs. People who are affected by time and politics. Great software gets to version 2. The users will never stop wanting change.
While every solution must be technically correct, performant, and stable, it is equally important that it be inventive, capable, engaging, and even fun. I have never seen a requirement for fun but I know solutions that are enjoyable to use seize market share and live for a long, long time.
We need to step into the light, banish the evil, and act like the mature professionals we claim to be. Through the continuous re-evaluation of the definition of 'done' we will be able to increase the likelihood of successfully helping our customers. As software developers it is our ability to listen to our customers throughout the process not just during some artificially time boxed requirements gathering phase and tell them the truth about their wants, needs, hopes, and desires, with all the complexities, and painful realities that transform our customers from being part of the problem to being part of the solution. Unlike those who hire us to turn dream into virtual reality we need to be proactive and deeply, personally responsible.
*any similarity to real statistical analysis living or dead is entirely unintentional. These are completely unfounded statements.