Marco Dorantes' WebLog

Egoless architecture

2013-03-01T13:26:00Z

What is architecture? Which could be the shared properties among diverse schools of thought and practice of architecture? Complexity management is possibly a common property among these kinds of design disciplines. Such complexity is present in many respects; some are of essential complexity, some of accidental complexity; some are about people interaction, some others about process, and others about technology —to mention just a few. Yet another respect, which is all-encompassing, is the usage of the thing being architected.

My main point in this brief comment is that complexity management is pervasive and architecture needs to be practiced by all participants in a project for building any non-trivial system, not just by those with the word ‘architect’ in their job title. Mainly because, in software, the role of the bricklayer is perfectly fulfilled by the compiler and the computer itself —as these core technologies are very good at repetitive and non-creative tasks.

The same key observations and good practices proposed by Gerald M. Weinberg can be applied here: egoless architecting. For an evidence of why this is relevant, reflect on possible responses from a so-called architect when his work is challenged or its defects are pointed out.

Composing software is a recent activity, perhaps just 60 years old. It is like an infant among other adult professions, like architecture —the well-established one. The tendency of a child to mimic the adults around him is a completely normal human attitude, just as normal as the overcoming, later in life, of such attitude. The same child, and if he grows up, starts to be aware of his own identity and the quest for independence continues.

The software industry has tried to mimic old and outdated perspectives from manufacturing; for instance, the notion of division of labor. This notion has been, in part, behind the idea of roles like ‘analyst’, ‘designer’, ‘programmer’, ‘tester’. These roles can be mapped exactly with related stages of the waterfall lifecycle model: analysis, design, programming, testing. Hence, ‘architect’ and ‘architecture’, as an independent role and as a discrete stage in the lifecycle, respectively, is another evidence of the imitation many people in this industry continue doing; also, it is an evidence of the childish age of this industry. If we think in a linear sequence and pick one of those activities as the first stage, followed by another as a second stage, and so on, then accidental complexity is added unnecessarily.

On the other hand, in more mature sectors of this industry, composing software that works robustly and flexibly across many successive releases requires all capacities at once. In this way, specification, testing, architecting, deploying, analyzing, etc., are not stages in the software lifecycle but concurrent activities. As we grow up as more complete and more mature software professionals and software consumers, we can focus more in the usage of the software, and less in our egos, and we can also realize that it is about architecting, not about “architecture” or about “architect”.

Good programming styles

2013-02-10T16:22:50Z

A presentation by Bjarne Stroustrup, creator of C++, about good programing styles. This presentation was dedicated to Dennis Ritchie, creator of C.

Note of relevance: The C programming language, in one way or another, is at the bottom of all existing computing.

Day 1 Keynote - Bjarne Stroustrup: C++11 Style

Empirically controlled?

2013-02-08T17:56:00Z

The creation of software-based business solutions, also known as application development, could benefit a lot from specific kinds of development processes; in particular those controlled empirically, like those with an iterative and incremental kernel inside, like Adaptive, Crystal Clear, DSDM, Scrum, FDD, Lean, XP, and others.

What does it possibly mean to be empirically controlled? Well, to start with, it should mean that a process is not controlled by «a priori» rationalizations alone —which is still a major trend in today industry, with big up-front architectures, Gantt charts with timelines expressed in months or years, and fear-based strategies and tactics for risk management. Instead, on the other hand, an empirically controlled process springs from «a posteriori» strategies and tactics executed continuously and in batches of small size.

One example of such strategies and tactics are found in musical performances. If you are interested in an increased level of performance within application development projects, then «rehearsal, rehearsal, rehearsal!» is a basic tenet of excellent performances both, in musical performances as well as in creative performances of software-based business solutions. For example, as deployment is an important part of a value stream in a software delivery performance then let’s execute, early and often, a rehearsal for such a procedure in a realistic way before touching the end-user.

More, much more, about this is coming...interested?

Further steps in theoretical groundwork for software development

2013-01-31T21:17:00Z

Time ago I posted about the obsolescence of some theories for project management (The underlying Theory of Project Management is Obsolete) and, hence, about the need for better theories not just for management. In my journey as a software professional, to the extent that I was becoming aware of how important is the theoretical effort to improve practice —and vice versa—, to the same extent I have searched for theories that scientifically uphold our activity. Then, it was very gratifying to know that more professionals were doing theoretical efforts (Working on software theory) in our field.

Now, it is very pleasing to learn more about the current progress on the theoretical work of people like Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, and Svante Lidman:

The Essence of Software Engineering: Applying the SEMAT Kernel

«The kernel in this book represents a software development effort as a continuously operating abstract mechanism composed of components and relationships. The project does not move from position to position within this mechanism as in the assembly line metaphor. Rather, there is a continuous flow through the mechanism as opportunities are transformed into requirements, and then into code and tests, and then into deployments.» —from the foreword by Robert C. Martin

«Software projects everywhere look for methodology and are not finding it. They do, fortunately, find individual practices that suit them; but when it comes to identifying a coherent set of practices that can guide a project from start to finish, they are too often confronted with dogmatic compendiums that are too rigid for their needs. A method should be adaptable to every project’s special circumstances: it should be backed by strong, objective arguments; and it should make it possible to track the benefits.» —from the foreword by Bertrand Meyer

Architecture and empowerment

2012-12-10T03:22:00Z

How bad could organizational hierarchies be for the advance of professionalism in the business of software creation? In particular, command and control organizational hierarchies derived from misinterpretations of the concept of ‘governance’.

I suggest that the advance of professionalism should be part of the strategic interests of a business for both, the short and the long term. That is why the inquiry of the opening question is relevant to a software business.

A proper association between authority and responsibility, that is, a proper empowerment, is crucial to a mature professionalism; otherwise professional accountability has little ground on which to build upon. Hence, unbalanced levels of authority and responsibility work against the advance of professionalism; therefore, against the business. That is, if a level of authority has been given to a person to take strategic design decisions then the same person should have the same level of responsibility for the consequences of those very same design decisions. How authoritarian hierarchies help or hindrance a proper empowerment for a given software project? Of course, the original design of those authoritarian hierarchies could not be intended to care for the professionalism in software.

This inquiry is not about the formal discourse on the matter but about the actual happenings on a given software endeavor when unmanaged complexity shows its ugly face. Also, this inquiry is intended, first, to understand what those actual happenings are; and, second, to transform those happenings into other possibilities based on a greater level of professionalism.

One cause for improper empowerment is contextual mismatch. For example, if architectural decisions, or most design decisions, are based on previous experiences in the same business domain —at best— and those decisions are taken by people at a different level of empowerment than those in the actual development team, then there are good chances that those previous experiences relate to an older context, even in the same domain, because of a non-trivial number of changes in the factors of the current context, which by now has become a significantly different context. Most of the people are different, most of the technology is different and, most importantly, most of the problem is different.

“..but, the difference is not that much...”, if there is not enough and solid evidence to support such opinion then it should be regarded as a mere manifestation of wishful thinking. Traits of mature professionalism relate less with wishful thinking than with critical thinking.

How important critical thinking is for emotional and intellectual maturity. Examples of the traits of critical thinking: before taking a position and having a discussion with someone on any given subject, learn about the counter arguments. The purpose of critical thinking is to gain better understanding and to get to the truth of the matter, not about "winning" a debate. If you want to truly understand counter arguments to your position and to be persuasive with people, learn to put yourself in the shoes of the person holding different views. Think of yourself as an actor taking on a role which is different than yours: learn about the person, their values and their background. Do not only focus on attempting to refute their position.

Some references for further thinking:

What Killed Waterfall could Kill Agile.

Scrum will never be the same

What Critical Thinkers Can Learn From Actors

Notes:

1. My post could be discussed in an application architecture community, but it could also be relevant for a discussion within an enterprise architecture community because it could relate to The generalized "Peter Principle".

2. Do not miss an historical perspective from the first referenced hyperlink in my post: What Killed Waterfall could Kill Agile.

3. I am aware of the schools of thought, in the realm of philosophy of art and literature, that regard architecture as something not related to the concrete nuances of engineering; whereas I see great transformational value in those schools of thought, especially from their ability to bring about new narratives with different interpretations of reality, I also see that, in software, those “engineering nuances” are not so different than a well-crafted ontological reasoning. So, in software, there is not a radical dichotomy between ‘architecture’ and ‘engineering’.

Measure of trust

2012-11-22T23:54:00Z

Building trust is precisely what I look for when I think about adaptive development in general. We, as creators and as consumers of software, need better levels of trust at many levels. The directors of the European Space Agency surely wanted a better level of trust in their software after the Ariane 5 explosion (caused by a software design error). I want a better level of trust in the WCF service I am designing and programming right now, which will be enterprise-level deployed and to be called by dozens of enterprise applications.

How do I know I am not fooling myself, and my project team, into believing that my effort estimate is right?

How do I know I am not fooling myself, and my project team, into believing that my WCF service is ready for deployment?

For better levels of trust, of course, the first principle is, as said by Richard P. Feynman in regard to a trait of scientific thought, do not fool yourself, neither your team, neither your customer. The methods of science have built-in ways to not fool ourselves —from all those useful combinations of different schools of thought known as rationalism, empiricism, and skepticism.

As of right now, for example, an increasing number of automated tests passing at each build of my set of WCF services amounts for an increasing level of trust. But those tests are not all. There is a number of feedback loops from an increasing set of client applications and, ultimately, end-users which also helps to not fool ourselves into believing that “we have the belief that we are on the right track” or “we want to believe...”. If we have that belief or not is not important, what is relevant is what we can prove based on a number of kinds of evidence, diverse, wide, and non-trivial kinds of evidence.

Should I trust my teenage daughter that she is going to do her duties? Is there any evidence or pattern of behavior for that? Then I have my answer. The words in themselves are not evidence. It is not a matter of trust, it is ultimately a matter of evidence.

The historic record to deliver good software on time and on budget, by a particular project team, could be good evidence on which to base my trust. The plans and good intentions are not evidence in themselves.

So, a measure of trust in X is going to be based on measures of actual evidence for X —for any relevant value of X.

If there are only words, plans and good intentions without an historic record backing up those words, plans and good intentions then there is little trust to measure. In that case a plausible action is to start building that historic record for particular project teams.

Dark Manifesto for Agile Software Development. Take 2

2012-11-11T01:09:00Z

In addition to this:

Do you think that instead of "We are uncovering better ways of developing software by doing it and helping others do it." people tend to adopt "We are uncovering the only ways of developing software teaching others."?

Yes, I have often seen a kind of indoctrination into the new «one and only true agile cult». But that is because people do not actually read and reflect on what writing software and agile are about. So the problem is radical dogmatism: people imposing unquestionable ideas over other people; that kind of dogmatism has the consequence you are contrasting. There is another kind of dogmatism that is chosen by an individual upon herself provisionally, as a stage in her apprentice path; about this look for ‘Shu-Ha-ri’ in the agile literature.

Do you think that instead of "Individuals and interactions over processes and tools" people tend to adopt "Individuals and interactions and not processes and tools"?

Yes, I have recently seen a development director that forced their teams to use Post-it on the wall whereas they have a fully functional development suite of wonderful tools to have a ‘single system of record’ of project reality; one consequence is that those teams now have two systems to keep in synchrony, wasting effort that could otherwise be in more useful activities.

Do you think that instead of "Working software over comprehensive documentation" people tend to adopt "Working software and not comprehensive documentation"?

I have seen no significant change with the problem of documentation. I still see much of both, too much ineffective documents and too little concise and useful forms of documents that help to communicate, and to preserve, important justifications of why the software is as it is.

Do you think that instead of "Customer collaboration over contract negotiation" people tend to adopt "Customer collaboration and not contract negotiation"?

I would be interested to see what might come out of such a tendency: "Customer collaboration and not contract negotiation". I still see too much protectionist focus on the terms of the contract, out of pure fear from both the customer and the provider. I would like to see more about designing value-streams that end at the end-user level.

Do you think that instead of "Responding to change over following a plan" people tend to adopt "Responding to change and not following a plan"?

I see that teams must respond to change but they do it in different ways depending of their context, and with different business consequences. Some, by contract restrictions, certainly must wait until the current plan ends; others, who provisioned proper contract clauses, can adapt to new business priorities more quickly. Many could respond to change quickly but also can break things more quickly; I still see fewer teams who are able to actually change quickly and not breaking a single feature at the same time.

Do you think that instead of "That is, while there is value in the items on the right, we value the items on the left more." people tend to adopt "That is, while since there is no value in the items on the right, we value only the items on the left."?

I think that we people fool ourselves into thinking that we understand something new without proper application of the tenets of critical thinking. We often fail to challenge properly our own presuppositions and then misread new concepts, relating them with what we already have in our memory; mere opinion is misinterpreted as knowledge. So, little or nothing new is actually learned. For example, an agile development process is seen as a sequence of steps or discrete iterations instead of an integrated set of value streams.

Dark Manifesto for Agile Software Development

2012-11-10T20:38:00Z

For those interested in answer a survey or questionnaire by Giancarlo Succi and Andrea Janes: http://darkagilemanifesto.org

Let me also post here my comments for further discussion:

Part 1

If “doing it” is removed then there are lesser chances to reflect, to inspect, and to adapt on the search for the specific business value —which often is a moving target— in a given context. Key parts of software development could be seen as epistemological endeavors; thus, if the empirical or experimental part is removed then the justification of a claim for an achieved business value could be dramatically diminished.

Do you think that instead of "Individuals and interactions over processes and tools" people tend to adopt "Individuals and interactions and not processes and tools"?

If the processes and tools do not help for better communication between individuals then those processes and tools should not be followed or used. Instead, an open dialog among practitioners should define proper processes and tools that help them to practice their values and professional principles.

Do you think that instead of "Working software over comprehensive documentation" people tend to adopt "Working software and not comprehensive documentation"?

The lack of a proper way to remember or to communicate relevant things to future or distant people is certainly a problem that should be solved in a project or product. That is a different problem than the problem of effort wasted in obscure and ineffective documents that nobody read. Besides, there is a plethora of recording technologies that could help to preserve and transmit relevant and valuable things about a project.

Do you think that instead of "Customer collaboration over contract negotiation" people tend to adopt "Customer collaboration and not contract negotiation"?

And for that matter: who, on the face of the Earth, is going to listen to the end-user?

Do you think that instead of "Responding to change over following a plan" people tend to adopt "Responding to change and not following a plan"?

If «a plan» no longer follows discovered and corroborated reality, then it is wise to make all sort of proactive adjustments or an entire new plan. The key to follow, of course, is not the outcome —a plan— but the frequent practice of planning. Conversely, blind reactions to whimsical moves, without awareness of the related costs, pave the path to diminished business value or business failure.

A part of the problem is a psychological pattern of neglected interpretation. It is not, of course, something exclusive of agile methods but it is a pervasive lack of education that permeates politics, economics, and many more spheres of society. The root cause is a very, very poor foundation in important areas of life, whereas it is called science, philosophy, history, mathematics, ethics, etc. In addition to that, the absolute reign of the hunt for short-term gratifications aggravates the odds to potential exits from the previous condition.

I am fully aware that a commercial interest in profits with software does not have an inherent responsibility to advance the state of the practice of the software development profession at the individual level, but it is not difficult to see that doing so is for the best of precisely that interest. Otherwise, an increasing number of consumers in the general public will start to not pay for software that does not deliver value fast and robustly.

The need for better software practices has been stated constantly, and yet, there is still way too much wishful thinking in our industry, mainly from non-practitioners, which persist in their lack of awareness about the long-term consequences.

What prevents us from learning a new concept is the preconception that usurps its place. See what Derek A. Muller has to say in the case of scientific concepts:

http://youtu.be/eVtCO84MDj8

Sugar-free C# – Part 2: Iterators 1

2011-04-10T00:28:35Z

Task: as a breadth developer, I need to analyze a set of VB6 legacy applications and enlist all stored procedures invoked per subroutine per application.

Is this a task for a breadth developer? Let’s consider this criterion: if the number of concepts needed to successfully apply an effective solution to the problem exceeds what the developer could understand and apply in one day, then it is not a task for the breadth developer.

Let’s try C#’s iterators to scan a large set of VB6 legacy projects to group invoked stored procedures from each subroutine from each VB6 form and module. In our problem, all the VB6 projects’ code at hand has the same basic organization: classes, forms, and modules with subroutines from which stored procedures are invoked, the names for all stored procedures are in the code of each subroutine and start with the prefix “usp_”. The folder structure where all VB6 projects reside is a two-level only structure: a root folder with child folders for each VB6 project and its related files.

A sample of the required output is:

project1.vbp
- module1.bas
  - subroutine1
    - usp_select1
    - usp_insert2
- form1.frm
  - subroutine2
    - usp_select3
  - subroutine3
    - usp_delete4
    - usp_select5

With C# iterators we could return all the .vbp files scanning the root folder structure where the VB6 code is located. For each VB6 project file, we could use iterators to return all its forms and modules. From the text of each code file, we could use iterators to return all its subroutines. For each subroutine, we could use iterators to return all the names of stored procedures in it.

The overall structure for a C#’s iterators-based solution could be like this:

void f(DirectoryInfo VB6Root)

{

foreach (DirectoryInfo folder in VB6Root.GetFolderHierarchy())

foreach (FileInfo vbproject in folder.GetVB6Projects())

foreach (FileInfo vbcodefile in vbproject.GetVB6CodeFiles())

foreach (VB6Subroutine subroutine in vbcodefile.GetSubroutines())

foreach (string storedprocedure in subroutine.GetInvokedStoredProcedures())

{

//...

}

The definition for the methods in the structure above could be like this: to be continued...

Sugar-free C# – Part 1: Introduction

2011-03-01T15:04:00Z

There is the notion of breadth developer, which encompasses the attitudes of the practical mindset: How can I do such and such task in a straightforward way? For example, I need to analyze a set of VB6 legacy applications and enlist all stored procedures invoked per subroutine per application, how can I use C#’s iterators to scan a large set of VB6 projects and VB6 source text files to group invoked stored procedures from each subroutine from each VB6 project?

There is the notion of depth developer, which —seems to me— encompasses some attitudes of Elvis and Einstein developer stereotypes: Why such and such works this way? For example, why yield return statements are allowed within C#’s iterator blocks whereas return statements are not?

I like to think about breadth or depth developer, or Mort, Elvis, or Einstein developer stereotypes, as roles we could play accordingly to the task at hand. One difference between these stereotypes is the number of concepts that you need to understand before you can do any production-grade work. Whereas the breadth developer enjoys any amount of syntactic sugar that helps to get the job done with a reduced set of conceptual prerequisites, the depth developer enjoys sugar-free programming in order to fully grasp what is behind the scenes of what was just done while in the breadth developer role; also, to jump from a mere effective to a more efficient solution.

I have done some contemplative explorations on the MSIL code generated by the C# compiler that have been found useful to satisfy my curiosity about iterators, deferred execution, extension methods, eager/lazy evaluation, lambda expressions (anonymous function expressions, anonymous method expressions), anonymous delegates, query expressions, etc.

I will share my findings with those contemplative explorations in future posts.