The New World of Emergent Architecture and Complex Adaptive Systems

Over the years, we architects have been taught to design systems that maintain equilibrium.   We designed for service levels, deployed security countermeasures, and built complicated solutions to meet 100% of our requirements.   We strove for perfection.   This mindset comes from the established manufacturing era of the last century where systems must be managed and controlled.   This Newtonian machine metaphor was revaluated by scientists over 60 years ago, and was proven to be incorrect.    The real world does not work that way, and the world of IT is now starting to figure this out.   In this new century, there must be a shift in thinking of how we design software based systems from one that resists change to one that embraces change.   

The manufacturing mindset that we architects have embraced taught us that the enterprise and its systems as a linear place where a simple set of rules of cause and effect apply.    So if I would want to comprehend the system:   I would take it apart, understand the parts, therefore I would understand the whole.  In order for me to optimize the system:    I would make each part work better, then the whole would be better.   We are very comfortable believing that everything we build could be predicted and controlled.

This is merely an illusion, in the real world comprehending the "system" is more challenging than was previously thought because there are forces in which we do not completely understand or control.   The best example in recent history is the financial meltdown that occurred in 2008.   The new world economy and its system do not play by cause and effect, or by pure supply and demand.     Economists where blinded by their models, and failed to understand or consider the behaviors of investors, mortgage brokers, scam artists, etc.   The idea that the "market" would correct itself is another illusion.  When oscillations occur in a system where feedback occurs, it can rip the entire structure of the system apart.   

In the last twenty years, we as architects have perhaps also been blinded by the beauty of our structures.   The structure of Active Directory, the structure of SharePoint server farms required to support collaboration, the network topologies, or object oriented UML diagrams, and the structure of  components and interfaces of our SOA based applications to name a few.    We see these structures as solid and enduring and we architect/engineer/design "stuff" to maintain the state of equilibrium that we all desire.   As economists now have discovered a harsh realization that control over complex systems is an merely illusion.   You can either fight it... or accept it and work with it.

In my previous blog entry, I mentioned the basic parts of speech that make up a sentence.   A sentence is defined as a complete unit of thought, and it consists of subjects, predicates, and objects.    An architecture by definition is the fundamental organization of a system embodied in its components [described by structures], their relationships [understood by behaviors] to each other and to the environment and the principles [driven by stakeholder goals] guiding its design and evolution.   The verb is what describes the relationship between subject (active structures) and object (passive structure).    Verb can be stative or dynamic or both.   These verbs or behaviors are a fundamental part of architecture, and it is the one we tend to ignore in designing systems.  When we ignore the behaviors (events, functions, processes, services, interactions)  of a system we design systems that are rigid and maladaptive or ones that just blow up in our face.   

There are two Laws of nature that we conveniently ignore:   The Second Law of Thermodynamics and The Law of Requisite Variety.

The second law of thermodynamics essentially states that within a given system, there is a level of disorder than can be measured.   This is known as entropy.      The Law of Requisite Variety comes from the field of cybernetics.   It essentially states that the survival of a system depends on its ability to cultivate (not just tolerate) variety within its internal structure.    There is that word structure again.

Welcome to the world of Complex Adaptive Systems.     These systems are diverse and made up of multiple interconnected elements and they are adaptive in that they have an aspect of cognition by learning from previous experiences in order to respond to change both positive and negative.   Complex adaptive systems are less about structure and more about how behaviors shape and mold the system through the concept of emergence.   Emergence is how the system evolves through patterns that form from a basic set of rules.    Studying the concept of "emergence" leads us to understanding systems that evolve as a result of the relationships within the system itself at an elemental level.   This is known as weak emergence.  Emergence then becomes a language or model that is needed to describe a system's behavior.   Then there is also the system where some emergent properties relate to the system as a whole , where qualities are not directly traceable to a specific system element or elements.   The system and how it emerges is related to its interaction as a whole, where the whole is greater than the sum of its parts.   This is known as strong emergence.  The human body, a termite mound, and taxi cab system are some examples of complex adaptive systems.   Complex Adaptive Systems are not complicated, the emerging patterns have variety, but the rules governing the function of the system are often quite simple.

Adapted from:   Wikipedia:  Complex Adaptive System

Another interesting aspect of Complex Adaptive Systems is that they tend to thrive on the Edge of Chaos.   This goes with the notion that if a system is designed with too much rigidity, it will eventually cease to exist.  On the other hand, if a system lacks enough structure and is too formless, it will cease to function.    The most productive state to be in is at the "edge of chaos" where there is maximum requisite variety, innovation, and fitness of the system which leads to new possibilities.   (Look whas is happened in the practice of Enterprise Architecture where only governance is considered and cannot change due to new paradigms, then the practice of Enterprise Architecture will die and business may look elsewhere for answers.   On the other hand if there no Enterprise Architecture, there are too many siloed solutions with isolated information where the organization cannot make decisions quickly in order to function.  That is another topic for another blog.)

Understanding emergence and this interesting duality of order and chaos will be critical for us architects in the future, where we will be pushed more into the world of architecture and less in the design and engineering of structures.  The discipline of architecture is about relationships and interactions that are expressed through strategic statements which define paradigm, style, principles, constraints, and rules that guide the evolution of the system.     

So what are the architectural qualities of a complex adaptive system?    Recently I had the privilege at our internal Enterprise Strategy and Architecture Summit on leading an open space discussion with some of my distinguished colleagues to help us the describe the architectural qualities of complex adaptive systems.   First I selected an example of complex adaptive system to focus our thinking:  The Cloud.   

After a good discussion there seem to be consensus on the following architectural quality attributes for complex adaptive systems (CAS):

• Adaptive (Positive Delta). A CAS can adjust in the face of positive change. In the cloud example, we should be easily provision new resources to meet new demand. For the cloud to be adaptive, it has to be engineered for elasticity.
• Resilient (Negative Delta). A CAS can rapidly recover in the event of negative change. In the cloud, we should be able to move workloads in the event of failure or decay to healthy physical systems. For the cloud to be resilient, it has to be engineered for failure.
• Appropriately Partitioned. A CAS consists of many elements, and in many cases there are self-organized in a way that is provides a level of equilibrium. When a CAS is partitioned it provides a level of isolation of each "sub system" to emerge independently. This is related to "weak emergence" where relationship and inter-dependencies of components are isolated to provide the maximum fitness of the solution. The cloud is appropriately partitioned to provide a level of isolation for tenancy, performance, and health diagnosis. The human system has many sub systems that allow for specialization.
• Well Integrated. A CAS elements are also well integrated to promote the concept of strong emergence, where the sum of the parts are greater than the whole. A well-integrated cloud can provide more benefits by providing integration to other cloud systems. A well-integrated cloud system within the system can deliver smooth operations and promotes flexibility.
• Efficient. A CAS system is not wasteful. Every element has a purpose and is not over engineered. Resources are never wasted or underutilized. There is an optimum level of fitness that lives at the edge of chaos (more on this later). For the cloud to be effectively leveraged, the right level of resources need to be provisioned based on demand, and released when not utilized to conserve resources. This allows for cloud computing users to pay only for what is consumed, while the provider minimizes its investment.
• Innovative / Cognitive. A CAS system should promote innovation through cognition. If the system is aware, it can adjust automatically to changing conditions through learning based on previous knowledge. A self-adjusting system at the edge of chaos is where innovation takes place.

The concept of Zen and Complex Adaptive Systems go hand in hand.   One has to accept that it impossible to fully comprehend and anticipate everything.   Zen is not about perfection, it is about harmony and thriving in the environment and living within an uncomplicated (dare I say simple) set of rules.    It forces us to think about what is necessary when we are architecting, engineering, and designing systems that serve others, and not ourselves.