Architecture + Strategy : Enterprise Architecture

SOA – End of Life 2009.01.01

dachou — Fri, 09 Jan 2009 11:59:37 GMT

It has just been a few days since Anne Thomas Manes at Burton Group published her post “SOA is Dead; Long Live Services”, and it has stirred up quite a storm of comments in the blogosphere. Most of what I read though, seem to be in alignment with what Anne Thomas Manes said -

SOA met its demise on January 1, 2009, when it was wiped out by the catastrophic impact of the economic recession. SOA is survived by its offspring: mashups, BPM, SaaS, Cloud Computing, and all other architectural approaches that depend on “services”.

Her article clarified that it is the “SOA as we know” (and the terminology used) has faded into irrelevance, the SOA that called for a comprehensive transformation of an organization’s view and management of its portfolio of data, technology, process, and people. Indeed, many people (such as David Linthicum, Eric Roch, JP Morgenthal, and the ongoing debate on InfoQ), for a number of years now, have been cataloguing why most enterprise SOA efforts fail miserably.

In general, I think the community is coming to the realization that SOA really is an architectural approach, not a set of technologies to implement. From that perspective Microsoft actually has been spot-on in terms of not offering “SOA”-branded products, but instead advocating customers to carefully design and build the right type of SOA for their organizations.

While there are many, many identified technical reasons why most SOA projects don’t succeed, Mike Kavis in his post has articulated one perspective nicely at a high level (just summarizing his list here):

We think process is a bad thing and it slows us down
We are impatient
We don’t understand what an architect really is
We don’t understand what architecture really is
We lose sight of the value and argue semantics
We lack leadership skills and emotional intelligence

This highlights one area why SOA hasn’t been successful: the human factor. But this doesn’t only apply to SOA; it’s just that the SOA requirements for organizational transformation and consistency amplify issues associated with the human factor. So what aspects of the human factor that make SOA difficult to implement?

Lack of patience, persistence, and perseverance

I think this is applicable on many levels. SOA requires a long-term, incremental build approach, but many projects are required to justify immediate or short-term ROI. Or from a different perspective, people just naturally expect to see some form of immediate benefits, and lose interest/motivation when the reality of SOA hits after the first few initial projects, which are often ESB-driven infrastructure optimization efforts, or point-to-point integration efforts. The lack of immediate business agility and cost savings gives people excuses to question the approach, reduce level of support, etc.

There is the aspect of jumping on the bandwagon simply because SOA was the acronym du jour and that it seemed smart to talk about it, without investing sufficient research, discipline, and due diligence to do it right. Of course, those who are impatient to jump on the bandwagon would just as (if not more) quickly to jump off at the first sign of trouble. There was sufficient intention or willingness to invest in SOA endeavors, but the impatience of not acquiring necessary expertise resulted in failures.

And truth is, SOA is not easy. Many organizations lose sight of the most important aspect of SOA - “how” to do SOA, not “what” we do SOA with. To many organizations it just seems simpler to follow marketing hype and implement products that are branded as SOA suites and think that an SOA can be constructed using the new infrastructure.

Resistance to change

People are naturally resistant to change, especially tough changes like SOA. Large organizations stand to gain more from SOA, but at the same time, those large organizations that have operated for many years in traditional functional silos have always resisted enterprise-level efforts that require them to build more dependencies on shared resources.

And SOA meant changes across all aspects of IT disciplines as we know. Operationally, traditional SLA management processes need to be adapted as downstream systems may need to inherit availability and performance requirements from upstream systems. Design-wise, it’s not just about exposing functionality as services, but more in the context of how a function is useful from the enterprise’s perspective; but that requires a higher level of collaboration beyond one department’s development teams.

Also, distributed computing is not simple. When we build process-level dependencies on other systems, efforts required to troubleshoot issues that one does not have full control over become magnitudes more challenging. This requires a major adjustment from the ways IT teams work today, and in those cases it’s often easier to point the finger at others first.

Organizational dynamics

The above aspects often apply to individuals. But when we look at an organization as a whole, the effects are also amplified. SOA requires a higher level of collaboration between teams in an organization. Each team or department used to having a higher level of autonomy in terms of managing their budgets, schedules, clients, technology, etc.; relatively independently from other teams. How to find the right balance between organizational consistency and flexibility with sufficient local autonomy, is unfortunately in itself requiring a uniform understanding and approach within the organization.

And politics. Not everyone likes to work with everyone, and individuals used to be relatively shielded within their own teams/departments/silos. But SOA requires breaking down the walls of silos, and can expose people more to personalities they may not like to work with, causing more contention among people. If not managed carefully, such as not positively reinforcing the correct behaviors, this can more quickly send the wrong signals to workers and hinder progress.

Lastly, strategic thinkers who understand what it takes to do SOA, tend to be the minority in today’s IT organizations which typically focus on tactical goals and are also measured as such. It’s just difficult for a few individuals to influence and steer an organization to adapt new changes.

So what now?

Thus it’s the “how” we do SOA that is the most important. It is the architectural disciplines, organizational cohesion, strategic leadership, etc. that most significantly impact the outcome of SOA efforts. And from that perspective the architectural principles of service-oriented architecture are still sound. In fact, as many people are already jumping into the next new big things such as cloud computing, “service”-oriented or driven concepts and considerations become much more important than before. Furthermore, cloud computing, in my opinion, has to do more about services than simply moving existing on-premises infrastructure into a utility-based cloud.

Perhaps it’s time to take a hard look at each SOA effort and ask the hard questions. Is it really meaningful, or valuable, to do SOA for your organization? Does your enterprise really need real-time process-driven integration, or traditional data integration, or a hybrid model, can suffice? Can people in your organization work collaboratively towards common goals and standards? Does your organization have what it takes to undergo and withstand such transformation? And so forth.

This doesn’t mean we should stop doing SOA (and regardless of what name we use to call it), but we should do it for the right reasons, and do it right. And more importantly, having the right people in the right places to see the plan through. This means having the right skills to plan and lead an organization to transform all aspects of data, technology, processes, and people (in knowing how to deal with the human factors mentioned above). There are still very significant benefits that SOA can bring, evident in the few organizations that have been successful with it.

Another lesson that can be learned from this ongoing discussion is that, SOA was deemed unsuccessful because it presented very significant gaps to the way existing IT organizations work today (such as what Dana Gardner mentioned in his post). However, we have to be careful in thinking that “abandoning SOA” and moving on to the next big thing – cloud computing, will solve all of our issues (doesn’t that sound eerily familiar?). It is evident that these gaps present such large gaps to many organizations, that the organizational aspects become the biggest impediments to progress. As technologists it is easy for us to say that the next major innovative technology trend will bring about sweeping changes and transformational benefits. But the reality is, cloud computing, as an extension of SOA, will require even more maturity and competencies in working with SOA to implement successfully. Indeed cloud computing presents a more prominent influencing factor to transform legacy IT, but it won’t make it any easier than SOA did. Organizations that want to take advantage of these transformative technology trends need to not only understand the technologies involved, but really pay attention to planning the organizational and people side of the endeavors.

Using Events in Highly Distributed Architectures

dachou — Fri, 14 Nov 2008 13:23:35 GMT

I had the privilege of publishing another article, "Using Events in Highly Distributed Architectures", in the Architecture Journal, issue 17, after the previous one on Strong User Authentication on the Web in issue 16.

I discussed Event-Driven Architecture (EDA) in this article, in terms of its concepts, and how it can be applied in enterprises as the next step of evolution in SOA initiatives.

The discussion took the direction of building on top of existing SOA infrastructures, so I didn't go into some key aspects for EDA, such as the importance of a robust and reliable messaging infrastructure that can ensure the reliable delivery of messages. It needs to support capabilities such as durable subscribers, message persistence and delivery tracking, idempotence, etc. Without a reliable messaging infrastructure, the collective architecture will not achieve a high level of data consistency, which is what EDA is ideally suited for.

I also did not go into much details, in the article, regarding modeling business processes as another logical layer on top of the asynchronous event distribution models. Now most practices today when modeling business processes follow very BPEL-like approaches - sequential logical workflows and orchestrations. And there are observations that many of these efforts, attempts to catalog or define enterprise business processes, often don't succeed at achieving the intended results. One possible explanation is that it is kind of "unnatural" to try to describe series of business activities as a sequentially linked list of tasks (with conditional branches, loops, etc.). Consequently, many architects end up not modeling the processes appropriately.

On the other hand, we can more easily map out business tasks as lifecycles or state transitions for each object, because that model is actually closer to how business analysts perceive business activities. The EDA perspective is that, in the context of asynchronous eventing models, these state transitions can then be defined into a state machine. Then, a layer of sequential workflows from a traditional business process definition perspective, can be added simply by drawing relationships across the state transitions of each object. It is a more abstracted view of business processes, but the underlying state machine model may bridge many of the gaps in traditional process modeling approaches, and can be mapped directly to an EDA technical implementation.

Lastly, asynchronous systems are inherently more scalable than synchronous systems. Today when people use asynchronous communication patterns to connect systems, they often integrate processes at a functional level (i.e., having logical dependencies on the outcomes of other distributed processes). As we move towards an environment where applications become much more inter-connected, both internally within an enterprise SOA environment, and with external partners via the open Internet (from a B2B perspective); and applications becoming more dependent on a growing number of externally distributed components, higher levels of functional decoupling will be needed to improve many aspects of the distributed architecture. EDA approaches seem to provide some relatively good answers from that perspective, but in practice, just like over-arching SOA initiatives, still have many challenges to overcome.

Talking about Service Oriented Architecture

dachou — Tue, 15 Apr 2008 13:29:45 GMT

I had the privilege to speak at the April monthly meeting at the Los Angeles Java Users Group. The meeting was held on April 1st at the Sun Microsystems office in LA, and so that in itself was the source of a few jokes lobbed at me. At times I was also referred to as Darth Vader, but all in good fun.

I had a great time chatting with the group, and was fortunate enough to see some familiar faces, especially a few back in the days when I worked at Sun Microsystems at this same office.

The topic of my presentation was "Service Oriented Architecture". This was picked as SOA is something that can be talked about from a technology-agnostic perspective, especially if we're focusing on the "A" in SOA.

| View | Upload your own

The presentation was mostly repeating what is considered "common knowledge" in SOA these days, such that SOA is an "architectural style", and that most of the fundamental principles consist of best practices and learned lessons in software and systems engineering in distributed computing, in the IT industry; and applied towards enterprise IT and systems integration efforts. And SOA is a pretty overloaded term too, such that many different perspectives exist, but all are valid:

Organizationally - developers, architects, managers, business stakeholders, executives, etc.

Architecturally - enterprise, infrastructure, security, data, integration, application, etc.

But in general, there is no one form of SOA that fits all organizations. Large enterprises tend to have a different set of issues and solution approaches at different priorities compared to medium-to-small businesses. The kind of SOA implemented at one organization may not necessarily be effective or needed for another organization. Just the same as all problem-solving scenarios, it is most effective to fully understand the issues, and then figure out approaches to address them.

Similarly, a maturity model-based approach is an excellent way to plan an SOA journey, but no existing maturity model (available from most of the technology vendors and analysts) is necessarily the right one to use. It's more important to pick out the aspects in a few maturity models that work for each organization, than to try to follow/implement a specific one religiously.

Meanwhile, many organizations looking at SOA are also faced with a number of questions at the implementation level:

Process vs. data integration
SOAP vs. REST
Data federation vs. data replication
Synchronous (RPC) vs. asynchronous (EDA)
Transactional (2PC / compensational) or not
Trust vs. impersonation
Centralized vs. federated ESB (or no ESB at all)
Stateful (BPM) vs. stateless (orchestrations)
Real-time vs. latencies

Conceptually, transforming a traditionally silo'ed enterprise environment into one logical real-time entity does seem to be a really beneficial proposition. However, that is often very difficult to obtain. Some of the factors include transforming all systems to be 24/7 instead of having independent maintenance outage windows, bear the transaction volumes of the highest trafficked systems, coordinating integration tests between multiple teams (or the entire enterprise), be subjected to the highest security compliance, etc.; when many didn't have to be when they were back in the silo environments.

Lastly, what may SOA look like in the future? There's talk about event-driven architecture (EDA) as the next step in evolution, which could be a more natural way of integrating business processes than the current RPC-style of tying everything together. There is also talk about extending SOA out to the Web, for enterprises, and additional advances in cloud computing, such as infrastructure services like internet service bus(es), federated security, cloud-based identity management and privacy controls, cloud-based data transformation services, etc. Continued progress in the semantic and interpretive Web may also play a major role in adding context to Web as a platform. And advances in model-driven programming and integrating them into service-oriented compositional architectures (such as Oslo on the Microsoft side, and SCA on the Java side) that change traditional multi-tiered application architectures to fully composite application architectures in all tiers (client-side mash-ups plus server-side mash-ups).

SOA Security - Enterprise Architecture Perspective

dachou — Thu, 18 Oct 2007 04:04:52 GMT

This week I had the opportunity to speak at the IT Architect Regional Conference in San Diego, on the subject of architecting enterprise SOA security. It is an interesting event, with speakers from Microsoft, IBM, Oracle, TIBCO, Fair Issac, and many other organizations. We even gave away a brand new XBox 360 and a Zune!

In a nutshell, my presentation was intended to point out the security aspects of planning an enterprise SOA, and a few topics that don't seem to be covered very often, and with an emphasis towards the future and navigating the organizational and cultural issues.

A brief overview -

Basically, some of the fundamental changes in SOA, such as:

Moving from low-volume batch-oriented data replication architectures to highly interactive real-time architectures between connected systems
Plus the migration towards Event-Driven Architectures (EDA) means an exponential growth in real-time (though asynchronous) communication, as each event can potentially trigger off a number of downstream events which can trigger off more events being sent across the network
All this moves the security concerns from the traditionally isolated infrastructure and application groups, into the integration layer that becomes a cross-cutting concern for everyone involved
SOA can also magnify existing issues such as identity management (or the lack of), and create new issues such as exposing mainframes directly to web traffic (for sake of real-time access into legacy applications and data)
The ideal state of "everything talking to everything in real-time" also means a breakdown of traditional physical network zones/perimeters, where DMZ becomes more like a reception/lobby area instead of a quarantine area, and data centers can no longer be considered locked down
Lastly, the threat environment has also evolved from single PC attacks, to DoS system attacks, and to today's application and data-level attacks, with lowered complexity and lowered barrier of entry (facilitated by vastly improved competencies in using XML)

Then of course, these changes also bring along many questions. Particularly many that represent conflicting approaches and each organization may come up with different solutions based on varying trade-offs.

For example,

Trust vs. impersonation/delegation. There are many security groups that believe enterprise network environments are inherently unsafe (which is agreeable), and thus all systems will need to require end-user authentication (regardless whether they are user-facing or intermediaries or downstream producer systems), and that "trust" cannot be trusted
From a different perspective, this debate is also centered on the concept of implementing end-to-end vs. peer-to-peer security contexts
There is also a lot of recent discussion on moving security intelligence (w/ centralized management) into the endpoints (laptops, mobile devices, etc.), or moving intelligence into the network (like recent advances in NAC)

In my opinion, trust-based architectures are much more flexible and scalable, and implementable by today's technology standards. And we couldn't completely eliminate trust in an impersonation/delegation model anyway. For example, a connected node/system has to "trust" service wrappers, agents, and/or local system components to verify user credentials against a centralized repository (such as Active Directory, LDAP, etc.) anyway.

On the other hand, having end-to-end security contexts is indeed conceptually more secure, as it can help better address the man-in-the-middle attacks, but in an SOA with a number of intermediaries between consumers and producers, there is still not an effective solution in managing public keys to support end-to-end message-level data encryption.

It's always interesting to try to take a peek at what may be possible in the future.

Most SOA discussions still seem to be focused on implementing "SOA in the enterprise". While that is very important, as enterprise architects we should also start to look at the growing trend of becoming more open on the Web, to an environment where enterprises essentially have no physical perimeters and security zones, largely due to the increasing number of direct and real-time connections into an enterprise (for sake of facilitating transactions with business partners).
Plus at that time we would also need to be concerned with the connections going from inside an enterprise out to the Web, as more and more internal systems becoming service consumers themselves
Thus a potential trend is moving away from trying to secure one large environment for the entire enterprise, migrating to a model where numerous (and potentially overlapping) smaller logical partitions (or zones) can be implemented to be provisioned with more targeted and effective security solutions (depending on data sensitivity). Rationale behind this is that it'll be more effective to try to protect smaller attack surfaces, even from a systems architecture perspective
Another interesting trend already underway is the growing centralization of data and content. Instead of consolidating everything into one or a few large enterprise content management deployments, organizations are creating smaller islands of data and content using collaboration platforms such as SharePoint. The point here is moving from mass distribution of data and content, and smaller islands seem to be lower hanging fruits at this point

Finally, some overall talking points. One important and interesting point that was kind of new to many people is that security in SOA has to be planned and designed just like another process layer. If we overlook security and not plan it carefully, we may end up creating tightly coupled elements in the overall architecture, and impacting the agility we intended to create.

The most visible example of this is trying to implement message-level encryption for the sake of data integrity (message digests) and confidentiality. In order to establish an end-to-end security context (so that intermediaries, including the ESB, should not be able to decrypt sensitive data on transit to the destination), both the intended consumer and producer have to know exactly how to encrypt and decrypt data. And that depends on a previous exchange of public keys, which in this case had to occur directly between the consumer and producer endpoints. That in a way is tight coupling, as the consumer and producer endpoints have to know about each other, and are required to establish a one-to-one, peer-to-peer relationship in terms of public keys exchange used for encryption/decryption. To alleviate the situation, a centralized public key infrastructure can be implemented in an enterprise so that the management and decisions on public key usage can be externalized from endpoints and centralized. However, enterprise solutions in this area are still evolving, and we haven't yet seen effective solutions for doing similar things beyond the enterprise and on the Web.

Lastly, the most important point is that, just like SOA governance, security is also a huge factor of the organization and corporate culture. We have to take a process-first approach to the problem (instead of technology-first), then weave in the technology delivery part of it.

For those interested, the entire slide deck I used can be downloaded from my Windows Live SkyDrive. If you don't have Office 2007, you can download the free PowerPoint Viewer 2007.

Share this post :

IT Architect Regional Conference 2007

dachou — Sun, 23 Sep 2007 20:45:31 GMT

Scheduled for October 15-16th, 2007, the ITARC 2007 conference is an event focused on the architecture topics in IT. The organizers at IASA (International Association of Software Architects) have arranged over 30 session in 4 concurrent tracks covering enterprise architecture, infrastructure architecture, software architecture, and architecture fundamentals.

Many notable speakers (just to list a few) are scheduled to present at the conference:

Chris Haddad, VP, Burton Group - Infrastructure Architecture in the Business Domain
Fred Waskiewicz, Director of Standards, OMB - Service Oriented Architecture: Making the Leap, Leveraging the Standards
Scott Ambler, Practice Leader, Agile Development, IBM - Agile Strategies for Enterprise Architects
Denise Cook, Rational Method Architect, IBM - Software Architecture Analysis Methods
David Chappel, VP & Chief Technologist, Oracle - Next Generation Grid Enabled SOA
Vince Casarez, VP, Portal Platform, Oracle - Web 2.0 for the Enterprise
Simon Guest, Director, Microsoft - Putting the User back into Architecture
Harry Pierson, Architect, Microsoft - Moving Beyond Industrial Software
Lynn Langit, Developer Evangelist, Microsoft - SharePoint Architecture, Lessons from the Trenches
David Chou (myself), Architect, Microsoft - Architecting Enterprise Security

The full agenda can be found at http://www.iasahome.org/web/itarc/socal/agenda

Also, it just so happens that Microsoft is sponsoring this conference as well. ;) As a result, we have a booth at the event. Looking forward to see you there!