Welcome to MSDN Blogs Sign in | Join | Help

Inside Architecture

Notes on Enterprise Architecture, Systems Integration, and anything else that interests me this week...
Applications ARE the leaky abstractions

Recently, a message called my attention to a classic article from Joel on Software on the Law of Leaky Abstractions.  It's a fun article and I recommend it heartily.  In this article, Joel argues that most techologies are designed to abstract something more complex under them, but nearly all abstractions are "leaky" in that they fail to completely hide the complexity underneath. 

Therefore, you can teach a software developer student about the "higher level" stuff, but sooner or later, they will need to know more about the underlying technology to cope with some unexpected or misunderstood effect caused by a leaky abstraction.

As a result, we cannot forget the underlying technology and "move up" to the new abstraction every time a new library or language appears.  We have to continuously cram more knowledge into our brains.  This makes all software development increasingly complex. 

This little reminder of Joel's classic article came from a developer who complained that software can never be truly disposable, and can never be free, because software is the first, last, and only place where you can document the business process.  Every place else, including process diagrams and documents, are leaky abstractions.

I believe that notion is backwards.  I believe that software is a leaky abstraction of the business process.

Software, to our end users, is there to solve a problem.  The problem is caused by the business needed to implement a business process.  Software automates a portion of that business process.  In doing so, software must abstract the process details away from the users, hiding complexity.  But if the process is complex, or if the software is not able to abstract the process well, it is very likely that the complexity of the business process will "leak through" the abstraction.

In other words, using code as documentation of a business process is a leaky abstraction. 

This is just one of the reasons why I do not believe that code is a good place to document business processes.

  1. As explained above, software is a leaky abstraction of the business process, which nearly always starts with, involves, and ends with human beings accomplishing a business task. 
     
  2. Code is not accessable to most of the people who actually need the process to change to meet the needs of the business.  Especially Object Oriented code, where large portions of the functional bits are hidden in base classes or where a lot of code is spent dealing with Object Relational Impedence.  Therefore, the cost of changing a business process increases dramatically due to the need for specialized skills to understand the existing automation.
     
  3. Most software modeling technologies focus their strengths on the static and structural elements of software (class diagrams and deployment diagrams) while the business folks want to deal with dynamic diagrams (state diagrams and transition diagrams).  As a result, we've refined a language that the business doesn't speak.  When you reverse engineer a diagram from code, you usually only get the unhelpful (to the business) static diagrams. 
     
  4. Code is an indirect representation of the business process steps.  A software developer, or team of developers, designed a technical solution to a business problem.  There were probably many ways to perform the particular task in technology, yet they chose one.  This narrowing is fine when going from general (business process) to specific (implementation) but when going back to general, as we would need to do when using code to document a business process, we would inherently add value to the choices already made.  This is flawed thinking that constrains business flexibility needlessly.

You see, my notion of inexpensive software is not just to make it easy on developers.  My notion is to create an abstraction that is so good that it provides over 90% of all business problems a "minimally leaky" abstraction.  Of course, developers (not just product teams within MS, but IT developers in the field) will need to create this abstraction for our business users.

I believe those abstractions exist in the services spanning layers predicted by Middle Out SOA Architecture.

Of course, there will always be the need for "very deep" software development to expand the boundaries of technology or to create altogether new abstractions for the business to use.  We don't put developers out of work.  On the contrary, we allow them to lift themselves out of the details and into the language that the business speaks: business process.

Software is the leaky abstraction.  It makes poor documentation for a business process. 

Posted: Wednesday, July 11, 2007 8:45 AM by NickMalik

Comments

Brian said:

Show me a business process not documented in the code and I'll show you an out of date, innacurate process documentation.

Regarding #2, I hear this a lot from analysts and other "non-coders" who don't want to exert the effort to look at code to see documentation. Code isn't something magical that is impossible to access. It's usually just text files. And normally there's some tool like javadoc that goes through and makes the comments real pretty and linky. If you're an analyst or project manager and you can't look into a codebase to view comments then I don't want you on my team.

What would you do if someone said "I can't look at that documentation, it's spread across 10 word files. Too complex, I'm not a MS Office Certified Word Operator." That's what I would do if someone said they couldn't look to a .java file to see what roles are involved in a service operation.

Your other points are valid, and documenting the process in the code (which usually means building the documentation from the code itself) isn't perfect but it's better than the other means that get outdated or never even created properly.

I think you're better off being realistic, adding some specific comment tags to your code and auto-generating documentation. This is the best way to get accurate documentation.

# July 12, 2007 2:45 PM

NickMalik said:

@Brian

"Show me a business process not documented in the code and I'll show you an out of date, innacurate process documentation."

And I will show you an organization at Level 1 of process maturity.

"If you're an analyst or project manager and you can't look into a codebase to view comments then I don't want you on my team."

The feeling is mutual, I'm sure.  Anyone unwilling to document the business process, and maintain it, and drive application requirements FROM the process (and not vice versa) has no place on my team either.

Your attitude is common, but indicative of poor practices.  You have learned the wrong way to develop business processes and you need to actually work in a progressive environment, where processes are considered valuable, before you will get the opportunity to learn a better way.  

In the days when Object Oriented code was a new practice, many folks expressed real reticence about learning OO coding.  "There's nothing I can do in C++ that I can't do in C!"  There are a few of those folks left, and they don't like their jobs.  

Get some training. Your career will benefit.

# July 12, 2007 7:12 PM

Udi Dahan said:

While I agree that automated forward and backward engineering of processes, diagrams to code and back, is a lofty goal I think that we need an evolutionary path to get there.

The problems I run into when getting handed some kind of process description is that it often leaves out critical details, specifically those around failure and partial failure conditions. Furthermore, I also experience variable resolution within each process - some parts are highly specific, others merely hand-waving.

The methodology of implementing such processes involves requirements excavation, analysis, design, prototypes, etc. We iterate that a bunch of times. This is a two-way learning process; both business and IT come out of it with a deeper understanding of the what the process SHOULD be.

Even if we could get a full and coherent mapping from the code to artifacts the business could understand, it would not mitigate the real problem. That of the request to change the process coming with similar fidelity as the original process description.

The solution, if any exists, is a maturation of the whole organization - people first. Then we enter into the oh-so-problematic domain of organizational learning and memory. There we see our best efforts crumble under promotions, reorgs, and quality people leaving.

Woe be us enterprise architects doomed to our promethean fate (http://en.wikipedia.org/wiki/Prometheus).

But that's why we get the "big bucks" :)

# July 13, 2007 3:02 PM

Peter said:

Nick,

You speak from my heart. I was wishing to write the exact thing out of myself so long ago. Now you have done it:)

However using SOA to provide that 90% real-life BP mapping sounds a little weird to me but time will tell you are right or not. I agree: at this time there is no other concise and commonly accepted technology (or maybe methodology???) out there than using SOA like (de-)composition.

Thanks!

Peter

# July 14, 2007 5:24 AM

NickMalik said:

Hi Udi,

I'm not sure that I want to automate the generation of code from process diagrams.  I suppose we will get there, but it will be a professional member of the IT team that builds the diagram.  It is just visual code at that point.  Still: no design required.

I also agree with the low consistency and quality of the business process diagrams.  That said, if we only have developers and no testers, we get crappy code.  If we only have business process analysts, and no business process testing tools, processes, and people, then we get crappy processes.  We need to test the process.  

A business process compiler would be nice.  Developers rely on their language compilers (and IDE) to find the simple syntax errors that are scattered throughout the process diagrams.  Perhaps with simulation software, we won't have to have a coder working side by side with an analyst to work out the details.  You are completely correct: we are immature in this space at the moment.  People, process and tools.

That said, most organizations have the maturity to create an iron-clad process that they will defend.  Think of Sarbanes-Oxley.  It takes executive will, but it can be done.

I would be honored to be considered as effective as Prometheus.  At least he provided fire to man.  Most days, I feel like Sisyphus, rolling a rock uphill and never making it to the top before it escapes and rolls back down.

Thanks for the feedback.

# July 14, 2007 9:08 PM

Craig Brown said:

Nice article & I love your first response to Brian.  

Technologists need to remember they aren't patronised as artists.  They are hired as artisans to service their client's business needs.

And on the topic of BP documetation; don't you see a determined effort by more mature organisations to develop business process competencies these days?  Things are getting better, gradually.

Cheers

# July 15, 2007 3:32 AM

NickMalik said:

@Craig,

"Things are getting better, gradually."

Yes.  They are.  I didn't drive this bandwagon, but I'm jumping on.  It's the right thing to do, and now is the right time to do it.  

I do see major movements in standards, tools, and mindshare.  It will take some time to get the business and technology schools to teach the right material, but movement is happening.  I am hopeful.

# July 15, 2007 10:02 AM

Captain Kirk said:

NickMalik,

 I have been on the darkside of development for too long.  We could not trust external documentation, it was always out of date (And this is at a big insurance company).  We trusted the code.  PDP-11 Days.

 But I have seen the light...  So I would like your opinion on the following...

 Where do you see Using the Testing System to encapsulate the business requirements?

 We are in the process of developing new software, and have decided to embrace a Test Driven Development attitude on top of an rather Agile Process.  BUT Only after very thorough analysis.  (I disagree that design should be done ad hoc, 22 years of development has taught me the best decisions are made with the best SOLID analysis).

 So, our answer to the documentation delimma was that the Diagrams would be "STORED" in test modules that excericised the logic.  And as bugs are discovered they are put here and fixed, and the documentation updated.

 Again, I am interested in your opinion on this...

We believe it solves a couple key problems and helps

to keep the developers and the business people working from the same pages....

Thanks,

Kirk Out!

# July 18, 2007 7:21 PM

NickMalik said:

@Captain Kirk,

I'm a trekkie too.  First series.  Anyway...

Testing and Business Requirements...

Once you have the overall business process documented, I would consider using Fitnesse (a tool on top of FIT) for documenting the acceptance cases.  In a truly agile development process, it matters more that you have defined the boundaries of "working code" than having defined all of the business rules and process flows.  (It's an interesting approach, to say the least).

That said, as you may guess from my description, I do not believe that it makes sense to define the acceptance cases before you define the business process.  That is simply backwards.  You have to assume a good understanding of the business process in order to create acceptance cases.

One thing that always surprises me is that developers will often assume that they are responsible for developing software according to the business process defined by the business.  That is nonsense.  It is more collaborative than that.  Agile really shines in that you can work directly with the business to show them the actual effects of creating the pie-in-the-sky process that they said they wanted.

I have often heard a business person say "I need process X" and thought to myself (WTF) "you must be joking."  With an agile dev process, instead of being the bearer of bad news and the outside expert who will make the business person feel stupid, you get to play the role of "helpful agent" who shows the business user that they want something simpler, because what they asked for is probably not a good idea.

I do not believe that you store diagrams in test modules.  That tells me that either (a) you don't trust your developers to open up the diagrams if you put them outside the code, or (b) you don't put diagrams in your source code control system.  

My expectations would be reverse of both of those:

1) Developers will use the diagrams during the sprint perspective and presentation meeting with the business to demonstrate how the app performs according to their diagram, and 2) the diagrams must be stored in source control with the source code in order to insure that you have good version control.

Two kinds of bugs: either the code matches the process, and the process is wrong, or the code is wrong.  If the process is wrong, it needs an entirely different kind of workflow and review process than if the code is wrong.   Therefore, you WANT to have these documents in seperate files.  They have different lifespans, stakeholders, and concerns.  It doesn't make sense to couple them.

# July 18, 2007 9:43 PM
New Comments to this post are disabled
Page view tracker