Applications ARE the leaky abstractions

re: Applications ARE the leaky abstractions

Brian — Thu, 12 Jul 2007 21:45:51 GMT

Show me a business process not documented in the code and I'll show you an out of date, innacurate process documentation.

Regarding #2, I hear this a lot from analysts and other "non-coders" who don't want to exert the effort to look at code to see documentation. Code isn't something magical that is impossible to access. It's usually just text files. And normally there's some tool like javadoc that goes through and makes the comments real pretty and linky. If you're an analyst or project manager and you can't look into a codebase to view comments then I don't want you on my team.

What would you do if someone said "I can't look at that documentation, it's spread across 10 word files. Too complex, I'm not a MS Office Certified Word Operator." That's what I would do if someone said they couldn't look to a .java file to see what roles are involved in a service operation.

Your other points are valid, and documenting the process in the code (which usually means building the documentation from the code itself) isn't perfect but it's better than the other means that get outdated or never even created properly.

I think you're better off being realistic, adding some specific comment tags to your code and auto-generating documentation. This is the best way to get accurate documentation.

re: Applications ARE the leaky abstractions

NickMalik — Fri, 13 Jul 2007 02:12:04 GMT

@Brian

"Show me a business process not documented in the code and I'll show you an out of date, innacurate process documentation."

And I will show you an organization at Level 1 of process maturity.

"If you're an analyst or project manager and you can't look into a codebase to view comments then I don't want you on my team."

The feeling is mutual, I'm sure. Anyone unwilling to document the business process, and maintain it, and drive application requirements FROM the process (and not vice versa) has no place on my team either.

Your attitude is common, but indicative of poor practices. You have learned the wrong way to develop business processes and you need to actually work in a progressive environment, where processes are considered valuable, before you will get the opportunity to learn a better way.

In the days when Object Oriented code was a new practice, many folks expressed real reticence about learning OO coding. "There's nothing I can do in C++ that I can't do in C!" There are a few of those folks left, and they don't like their jobs.

Get some training. Your career will benefit.

re: Applications ARE the leaky abstractions

Udi Dahan — Fri, 13 Jul 2007 22:02:30 GMT

While I agree that automated forward and backward engineering of processes, diagrams to code and back, is a lofty goal I think that we need an evolutionary path to get there.

The problems I run into when getting handed some kind of process description is that it often leaves out critical details, specifically those around failure and partial failure conditions. Furthermore, I also experience variable resolution within each process - some parts are highly specific, others merely hand-waving.

The methodology of implementing such processes involves requirements excavation, analysis, design, prototypes, etc. We iterate that a bunch of times. This is a two-way learning process; both business and IT come out of it with a deeper understanding of the what the process SHOULD be.

Even if we could get a full and coherent mapping from the code to artifacts the business could understand, it would not mitigate the real problem. That of the request to change the process coming with similar fidelity as the original process description.

The solution, if any exists, is a maturation of the whole organization - people first. Then we enter into the oh-so-problematic domain of organizational learning and memory. There we see our best efforts crumble under promotions, reorgs, and quality people leaving.

Woe be us enterprise architects doomed to our promethean fate (http://en.wikipedia.org/wiki/Prometheus).

But that's why we get the "big bucks" :)

re: Applications ARE the leaky abstractions

Peter — Sat, 14 Jul 2007 12:24:52 GMT

Nick,

You speak from my heart. I was wishing to write the exact thing out of myself so long ago. Now you have done it:)

However using SOA to provide that 90% real-life BP mapping sounds a little weird to me but time will tell you are right or not. I agree: at this time there is no other concise and commonly accepted technology (or maybe methodology???) out there than using SOA like (de-)composition.

Thanks!

Peter

re: Applications ARE the leaky abstractions

NickMalik — Sun, 15 Jul 2007 04:08:01 GMT

Hi Udi,

I'm not sure that I want to automate the generation of code from process diagrams. I suppose we will get there, but it will be a professional member of the IT team that builds the diagram. It is just visual code at that point. Still: no design required.

I also agree with the low consistency and quality of the business process diagrams. That said, if we only have developers and no testers, we get crappy code. If we only have business process analysts, and no business process testing tools, processes, and people, then we get crappy processes. We need to test the process.

A business process compiler would be nice. Developers rely on their language compilers (and IDE) to find the simple syntax errors that are scattered throughout the process diagrams. Perhaps with simulation software, we won't have to have a coder working side by side with an analyst to work out the details. You are completely correct: we are immature in this space at the moment. People, process and tools.

That said, most organizations have the maturity to create an iron-clad process that they will defend. Think of Sarbanes-Oxley. It takes executive will, but it can be done.

I would be honored to be considered as effective as Prometheus. At least he provided fire to man. Most days, I feel like Sisyphus, rolling a rock uphill and never making it to the top before it escapes and rolls back down.

Thanks for the feedback.

re: Applications ARE the leaky abstractions

Craig Brown — Sun, 15 Jul 2007 10:32:29 GMT

Nice article & I love your first response to Brian.

Technologists need to remember they aren't patronised as artists. They are hired as artisans to service their client's business needs.

And on the topic of BP documetation; don't you see a determined effort by more mature organisations to develop business process competencies these days? Things are getting better, gradually.

Cheers

re: Applications ARE the leaky abstractions

NickMalik — Sun, 15 Jul 2007 17:02:29 GMT

@Craig,

"Things are getting better, gradually."

Yes. They are. I didn't drive this bandwagon, but I'm jumping on. It's the right thing to do, and now is the right time to do it.

I do see major movements in standards, tools, and mindshare. It will take some time to get the business and technology schools to teach the right material, but movement is happening. I am hopeful.

re: Applications ARE the leaky abstractions

Captain Kirk — Thu, 19 Jul 2007 02:21:15 GMT

NickMalik,

I have been on the darkside of development for too long. We could not trust external documentation, it was always out of date (And this is at a big insurance company). We trusted the code. PDP-11 Days.

But I have seen the light... So I would like your opinion on the following...

Where do you see Using the Testing System to encapsulate the business requirements?

We are in the process of developing new software, and have decided to embrace a Test Driven Development attitude on top of an rather Agile Process. BUT Only after very thorough analysis. (I disagree that design should be done ad hoc, 22 years of development has taught me the best decisions are made with the best SOLID analysis).

So, our answer to the documentation delimma was that the Diagrams would be "STORED" in test modules that excericised the logic. And as bugs are discovered they are put here and fixed, and the documentation updated.

Again, I am interested in your opinion on this...

We believe it solves a couple key problems and helps

to keep the developers and the business people working from the same pages....

Thanks,

Kirk Out!

re: Applications ARE the leaky abstractions

NickMalik — Thu, 19 Jul 2007 04:43:58 GMT

@Captain Kirk,

I'm a trekkie too. First series. Anyway...

Testing and Business Requirements...

Once you have the overall business process documented, I would consider using Fitnesse (a tool on top of FIT) for documenting the acceptance cases. In a truly agile development process, it matters more that you have defined the boundaries of "working code" than having defined all of the business rules and process flows. (It's an interesting approach, to say the least).

That said, as you may guess from my description, I do not believe that it makes sense to define the acceptance cases before you define the business process. That is simply backwards. You have to assume a good understanding of the business process in order to create acceptance cases.

One thing that always surprises me is that developers will often assume that they are responsible for developing software according to the business process defined by the business. That is nonsense. It is more collaborative than that. Agile really shines in that you can work directly with the business to show them the actual effects of creating the pie-in-the-sky process that they said they wanted.

I have often heard a business person say "I need process X" and thought to myself (WTF) "you must be joking." With an agile dev process, instead of being the bearer of bad news and the outside expert who will make the business person feel stupid, you get to play the role of "helpful agent" who shows the business user that they want something simpler, because what they asked for is probably not a good idea.

I do not believe that you store diagrams in test modules. That tells me that either (a) you don't trust your developers to open up the diagrams if you put them outside the code, or (b) you don't put diagrams in your source code control system.

My expectations would be reverse of both of those:

1) Developers will use the diagrams during the sprint perspective and presentation meeting with the business to demonstrate how the app performs according to their diagram, and 2) the diagrams must be stored in source control with the source code in order to insure that you have good version control.

Two kinds of bugs: either the code matches the process, and the process is wrong, or the code is wrong. If the process is wrong, it needs an entirely different kind of workflow and review process than if the code is wrong. Therefore, you WANT to have these documents in seperate files. They have different lifespans, stakeholders, and concerns. It doesn't make sense to couple them.