So, I am picking you are here because you are looking for an answer to the previous posting on why software development projects are continuing to fail to deliver. 'The Vicious Software Development Cycle'.

I should apologise that the previous posting leading to this one was a little more in depth and long-winded than I had planned. So I had to separate them for readability. If you haven't read the other one, then this post is probably a little out of context.

I'm gonna dive straight back into it and look at the 'root causes' of this vicious cycle. In other words why are we constantly seeing the same failing behaviour over and over again in this industry, and why does the cycle typically not diverge and result in successful execution, rather than spiralling in on itself into the vicious cycle.

The basic reason of all of this happening is because - projects today are under too much time pressure to deliver -, that's the fact of it. (it's also the symptom not the cause).

That is not to say that we are not giving them a reasonable amount of time to deliver, that is to say that, realistically they take too much time to deliver given the people, scope, expectations, requirements, budget, tools, understanding of technology and processes to deliver. Or another way to put it is: we are trying to build too much. This is really a failing of the people, tools, process and attitude we use today.

Let's not forget the factors that drive this failing:

  • Customer requirements are increasing in complexity (due to advancements in technology), and these requirements are getting harder to define fully in concrete terms up front in the project scoping phase.

  • Experience & specialised skills are under-valued, and thought to be too expensive (upfront) to invest in, and therefore the actual 'effective capability of the solution' to deliver on the requirements is significantly reduced.

  • Development tools are still too generic to realise significant productivity gains in specific domains to yield overall gains in solution development. In other words we don't use tools specific enough to solve the problem.

  • Developers are still spending too much time hand crafting solutions from the ground up (for many different reasons, mostly related to lack of knowledge/experience). And we rely on the knowledge and experience of too few to deliver the whole solution. (We also don't invest in them deeply enough to gain more).

  • Component re-use is difficult to achieve in a climate where:
    • There is no easy way to standardise, describe, catalogue or shop for reusable component parts.

    • Technical project teams still address software development with the approach of an artist and a blank canvas, where they are free will to create anything they desire, mainly because they 'think' their solution is unique to their problem space.

So what is it driving this strange behaviour? If you think about it, the only activity in the software development process that is absolutely, without exception, the only activity that HAS to take place - is the construction of the solution. All other activities are optional! Without a deliverable, there is no project; it must deliver. Therefore all other activities may fall away under the scoping/budget hammer.

Think about building a new home for your family, without the physical structure to keep out the cold, rain and water, all the planning, design, construction materials, skills and knowledge in the world is not going make any difference if the house is not standing as a result of your budget/efforts. Even a half-constructed house is as good as no house at all.

In a competitive market where margins are necessarily small (to win competitive business), projects are driven by even tighter and tighter budgets, and it is 'he whom can promise the most, for the least' (upfront cost), in the quickest time who wins the deal.

Nobody (customers especially) expects to pay more for something today than they did yesterday, but everyone today wants it to deliver more than it did yesterday. The whole equation is out of balance for the simple reason that it's no quicker/cheaper to execute today than it was yesterday. Today, the best we can do is build the solutions of yesterday - cheaper. For some reason we don't see the same thing in many other modern industries.

Something has to give, and the vicious cycle starts turning and one pitfall feeds the next. Let's dive into this for the moment and look at the effects.

The net result of all these pressures is that solution planning is usually the first to drop away, products, tools processes are all blindly discarded along with any sort of structured architecture in favour of a minimal design. 'Think-time' about the solution holistically from a business perspective falls away, so focus is (rightly) shifted to the technology to deliver the solution and (wrongly) shifted to individuals to deliver it. The tools to describe the solution holistically, in its terms, are poorly built, used, or non-existent.

Nobody - well far too few - at this point would dare suggest spending a bunch of money on getting the expertise in to help them in the right direction (although plenty of people would admit they need it). This is interesting in itself - mostly driven by individual's e-penis and ego, and the shear denial that they, IT-professionals (who are being paid to do this), don't possess the knowledge and skills to perform this complex task.

Instead, the focus becomes, "let's just get on with getting started right away with creating the solution", and is often reasoned with: "we have enough skills, resources and new technology to do it ourselves, we just hired a bunch of new developers/architect/consultant/etc - surely they will figure this stuff out".

The answer to this naive approach is - yes they will, given enough time and relevant resources, yes they sure could, but what is the price of that experience they need to gain to be successful? You will pay for it one way or the other; upfront is cheaper, during a project or during stability/maintenance/support is more expensive, and losing customers because of it is even more expensive.

Of course along with good wholesome planning usually goes any structured development process into the "too expensive/hard" basket. Testing, Q&A, deployment and operations are often pushed to the end of the project "if we have the time and the money"!

So, as a result, the responsibility of the delivery of the solution, (which, by the way, is now the only activity of any value to the project now), falls squarely on the shoulders of the construction workers - the developers.

Everyone in the project team (and the customers too) now look towards the 'development team' as the 'Project Saviours' - whilst being solely at the mercy of them.

(Its not hard to imagine why developers and architects in this industry have such large pressures - and such large uncompromising egos, is it?)

So, who becomes accountable for this masterpiece effort? No artist is accountable for the impression they create with their work, that's in the eye of the beholder isn't it? Software engineers/developers have a similar attitude, since they think they do the same thing. I know, if you are a developer reading this you are saying to yourself "that's just not true, I have discipline and pride for my work, I never take shortcuts, and I am always conscious of what people think of my code, I always do the right thing".

That maybe true in an ideal world (which we/you are far from). If you put a developer under pressure to deliver something, you see some interesting and eye opening behaviours typically occur. Let's face it, we all do this/have done this/continue to do this as long as we need to, to get the job done in the climate and pressures we have to work in. Developers simply can't be accountable for this! can they?

Here are some behaviours/attitudes we see all the time from real-world developers:

  • They don't care about the way they do this stuff, as long as they have a way to do it.

    • "I found it from the internet", "so-and-so blogged about it", "it came from MSDN!"

    • "No one will see that code anyway, or they won't understand it!"

    • "We used a code generator for that part, so I don't need to care about the code, it is managed for me" (that code is somehow immune for the other code).

  • They are too busy (and the budget too small) to take sufficient time to learn all this 'new technology stuff':

    • "We are too busy to invest in time to learn that new stuff"

    • "Just show me the API for it"

    • "What does that guy really know about it!"

    • "I saw a demo of that the other day"

    • "I found an example of that on the web, we need to change it a little to suit our purpose, but it should work."

    • "Let's just approach it the same way we always have" (The same way they always knew, using a previous technology, tool, process etc.)

  • Most developers don't maintain or support their 'creations' once they finish the first version. "That's left for the maintenance developers/support staff/newbies" whoever those legends are!

    • "It compiles fine"

    • "It works on my machine, there is something wrong with your machine or you!"

    • "This is a hack, it will work for now"

    • "Nobody will ever want to re-use this component/artefact/application/solution".

    • "Sorry, I have moved onto another project now, go talk to so-and-so, they should be able to figure out my masterpiece".

    • "What do you need tracing/auditing/logging for?"

    • "Just get the user to send through the error message"

    • "I need to go down there and debug their production system" and by the way, I need to install Visual Studio on that machine.

    • "We need to do whatever we need to get the job done."

    • "Just tell the customer they need to run as an 'administrator' to use this component/application/solution and it will be fine"

    • "You just have to make an exception in your (customers') operations policy for 'our' brilliant component/application/solution"

    • "Don't you read the technical manuals we wrote?, it's all in there" (manual still in version 0.1, created haphazardly, by a developer, for a developer and way out-of-date at the beginning of the project).

    • "Sorry, I have moved onto another project now, go talk to so-and-so, they should be able to figure my masterpiece" (repeated)

Developers don't pay for projects nor the overruns they create.

The bottom line is this; creating software solutions is very hard to get right with what we have to work with today. We are not making it any easier to do with either the tools, people or processes we have, or knowledge we are 'prepared' to gain to get it done properly. Often, the knowledge required to do it right is not available when it is needed and it becomes a financial exercise in getting it - because need it you do!.

So, our basic problem is lack of expertise knowledge (information & experience) in all areas of solution development. We lack the optimised tools. We lack standardised components. We lack a way to locate these tools and components. Its costing us in all the money we should be using to solve the requirements posed by the business problem.

The knowledge we need is contained within individuals. If you don't have the right individuals (that is, the ones containing the right knowledge), and you are not prepared to pay for the right ones (or give them that knowledge through training), then you are likely to pay more for that knowledge in the long run through experience; presumably because you refuse to pay for it up front.

Realistically, you cannot expect any single individual to contain all the knowledge/experience you need to construct a whole solution. Not even a few individuals will suffice.

How would you feel if you wanted to build your own house and you invested all your life's savings into it. Of course architects are expensive so you went with your mate the builder. Your builder says to you,

'Look no worries mate, I'll build the house, then I will run all the plumbing around the place, I can do the electrical work too and a few licks of paint on this place it will be perfect! - it will save you a bunch of money to use me, so you won't need to worry about all paying all those other expensive contractors, you just won't be able to afford them.'

'Oh, and by the way, we are about half done at the moment, and I've run out of cash, I spent all the budget on baking my own bricks and cutting down and sawing the trees for wood - that's bloody hard work my friend!'

The tools we use today should encapsulate the specialist knowledge, so that we don't need to rely on the individuals to master every skill. The tools must evolve with the new technology (actually, delivered as part of the technology), so that the capability of the new technology is realised at the time of its availability.

The components we use today should not have to be built from scratch every time because of lack of knowledge and skills. We should be assembling a careful selection of components specific to a particular requirement. In the mainstream, that requirement needs to be constrained to a 'standardised requirement, with a well known standard solution. Custom solutions are still expected and viable, but also expect to incur an increase in risk and cost. The components should be widely available, well catalogued well understood and very specific to a given requirement. In the same way that you only buy one type of plug for all the appliances in your house.

Our roles as solution developers should be assembling these carefully chosen components using specific optimised tools for the technology, so that we can deliver a solution in the time given.

We should be spending our time focused on what makes solutions different (business requirements) rather that that which is all the same (structure, architecture, plumbing, components etc.).