Architecture + Strategy : SOA

Architect Council | 2010: A Cloud Odyssey

dachou — Tue, 10 Nov 2009 04:39:00 GMT

The forecast for 2010 is looking very cloudy, with every technology company delivering some type of cloud offering, and analysts estimating that almost everyone will try some form of cloud computing. But from a practical stand point, how should organizations leverage cloud computing effectively? Is it to simply host existing and legacy assets somewhere else, or to create next-generation applications that enable differentiated capabilities? In addition, what role does SOA play in the age of cloud computing? And does this impact the way we manage and interact with data?

Facilitated by Microsoft, the premise of this half-day event is to provide an open forum where architects can meet to discuss technologies and insights with peers. This forum will provide first-hand experiences and best practices intended to help the community to collectively discover and generate knowledge. Please join us for this opportunity to learn, share and network with your peers and other thought leaders.

Event Summary

Provide a clear and profound understanding of Microsoft’s cloud strategy and roadmap, and latest updates on the Windows Azure Platform
Discuss the state of SOA, and its relevance in the age of cloud computing
Review case studies and models of cloud usage, and examine approaches to build next-generation capabilities leveraging the cloud
Explore trade-offs, best practices, and guidance on today’s myriad of data storage options
Explain the differences and patterns of using various data interaction frameworks

Who Should Attend

Technology and business stakeholders, including CTOs, CIOs, VPs, Directors, Program and Product Managers, Architects, Lead Engineers, and IT Managers

When & Where

12/01/09: Irvine, CA

12/02/09: Los Angeles, CA

12/03/09: Phoenix, AZ

12/08/09: Mountain View, CA

12/09/09: San Francisco, CA

12/10/09: Seattle, WA

12/15/09: Denver, CO

How to Attend

This is a FREE event; only an online registration is required. Please follow the links on the event page at http://blogs.msdn.com/sac/pages/council-2009q4.aspx

Cloud Computing and the Microsoft Platform

dachou — Tue, 13 Jan 2009 14:09:14 GMT

It has been a couple of months since I wrote about cloud computing and Microsoft’s plans and strategies. Now that Azure Services Platform has been unveiled at PDC2008, and after having the opportunities to discuss it with a community of architects from major enterprises and startups via the Architect Council series of events, I can talk about cloud computing from the perspective of the Microsoft platform, and the architectural considerations that influenced its design and direction.

Okay – cloud computing today is a really overloaded term, much more than SOA (service-oriented architecture) when it was the hottest ticket in IT. There are a lot of different perspectives on cloud computing, adding to the confusion and the hype. And unsurprisingly, there are a lot of confusion around Microsoft’s cloud platform too. So here is one way of looking at it.

Microsoft’s cloud includes SaaS (Software-as-a-Service) offerings as shown in the top row of the above diagram, such as Windows Live and the Business Productivity Online Suite; and the PaaS (Platform-as-a-Service) offering currently branded as the Azure Services Platform. For the rest of this article we will focus on the Azure Services Platform, as it represents a platform on top of which additional capabilities can be developed, deployed, and managed.

Comprehensive Software + Services Platform

At Microsoft, we believe that the advent of cloud computing does not necessitate that existing (or legacy) IT assets be moved into the cloud, as it makes more sense to extend to the cloud as opposed to migrate to the cloud. We think that eventually, a hybrid world of on-premise software and cloud-based services will be the majority norm, although the balancing point between the two extremes may vary greatly among organizations of all types and sizes. As a platform company, Microsoft’s intention is to provide a platform that can support the wide range of scenarios in that hybrid world, spanning the spectrum of choices between on-premises software and cloud-based services.

Thus Microsoft’s cloud platform, from this perspective, is not intended to replace the existing on-premises software products such as our suite of Windows Server products, but rather, completes the spectrum of choices and the capabilities required for a Software + Services model.

Cloud Platform as a Next-Generation Internet-Scaled Application Environment

So what is a cloud platform? It should provide an elastic compute environment that offers auto-scalability (small to massive), and ~100% availability. However, while some think that the compute environment means a server VM (virtual machine) allocation/provisioning facility that provides servers (i.e., Windows Servers, Linux Servers, Unix Servers, etc.) for administrators to deploy applications into, Microsoft’s approach with the Azure Services Platform is remarkably different.

Azure Services Platform is intended to be a platform to support a “new class of applications” – cloud applications.

On the other hand, the Azure Services Platform is not a different location to host our existing database-driven applications such as traditional ASP.NET web apps or third-party packaged applications deployed on Windows Server. Cloud applications are a different breed of applications. Now, the long-term roadmap does include capabilities to support Windows-Server-whichever-way-we-want-it, but I think the most interesting/innovative part is allowing us to architect and build cloud applications.

To clarify, let us take a quick look at the range of options from an infrastructure perspective.

The diagram above provides a simplified/generalized view of choices we have from a hosting perspective:

On-premises: represents the traditional model of purchasing/licensing and acquiring software, install them, and manage them in our own data centers
Hosted: represents the co-location or managed outsourced hosting services. For example, GoGrid, Amazon EC2, etc.
Cloud: represents cloud fabric that provides higher-level application containers and services. For example, Google App Engine, Amazon S3/SimpleDB/SQS, etc.

From this perspective, “Hosted” represents services that provide servers-at-my-will, but we will interact with the server instances directly, and manage them at the server level so we can configure them to meet our requirements, and install/deploy applications and software just as we have done with existing on-premises software assets. These service providers manage the underlying infrastructure so we only have to worry about our servers, but not the engineering and management efforts required to achieve auto-scale and constant availability.

“Cloud” moves the concerns even higher up the stack, where application teams only need to focus on managing the applications and specifying to the environment their security and management policies, and the cloud infrastructure will take care of everything else. These service providers manage the application runtimes, so we can focus on deploying and managing business capabilities, as well as higher-level and differentiating aspects such as user experience, information architecture, social communities, branding, etc.

However, this does not mean that any one of these application deployment/hosting models is inherently better than the other. Yep, while most people look at “hosted” and “cloud” models as described here, both as cloud platforms, they are not necessarily more relevant than the on-premises model for all scenarios. These options all present varying trade-offs that we as architects need to understand, in order to make prudent choices when evaluating how to adopt or adapt to the cloud.

Trade-Offs in the Cloud

Let us take a closer look at the trade-offs between the on-premises model and the cloud (as differences between “hosted” and “cloud” models are comparatively less).

At the highest level, we are looking at trade-offs between data consistency and scalability/availability. This is a fundamental difference between on-premises and cloud-based architectures, as “traditional” on-premises system architectures are optimized to provide near-real-time data consistency (sometimes at the cost of scalability and availability), whereas cloud-based architectures are optimized to provide scalability and availability (by compromising data consistency).

One way to look at this, for example, is how we used to design and build systems using on-premises technologies. We used pessimistic locking, optimistic locking, two-phase commit, etc., methods to ensure proper handling of updates to a database via multiple threads. And this focus on ensuring the accuracy and integrity of the data was deemed one of the most important aspects in modern IT architectures. However, data consistency is achieved by compromising concurrency. For example, in DBMS design, the lowest transaction isolation level “serializable” means all transactions occur in a serial manner (in a way, single-threaded) which promises safe updates from multiple clients. But that adversely impacts performance and scalability in highly concurrent systems. Raising the isolation level helps to improve concurrency, but the database loses some control over data integrity.

Furthermore, as we look at many of the Internet-scale applications, such as Amazon S3/SimpleDB, Google BigTable, and the open source Hadoop; their designs and approaches are very different from traditional on-premises RDBMS software. Their primary goal is to provide scalable and performant databases for extremely large data sets (lots of nodes and petabytes of data), which resulted in trading off some aspects of data integrity and required users to accommodate data that is “eventually consistent”.

Amazon Web Services CTO, Werner Vogels, has recently updated his thoughts on “eventual consistency” in highly distributed and massively scaled architectures. An excellent read for more details behind the fundamental principles that contribute to this trade-off between the two models.

Thus, on-premises and cloud-based architectures are optimized for different things. And that means on-premises platform are still relevant, for specific purposes, just as cloud-based architectures. We just need to understand the trade-offs so each can be used effectively for the right reasons.

For example, an online retailer’s product catalog and storefront applications, which are published/shareable data that need absolute availability, are prime candidates to be built as cloud applications. However, once a shopping cart goes into checkout, then that process can be brought back into the on-premise architecture integrated with systems that handle order processing and fulfillment, billing, inventory control, account management, etc., which demand data accuracy and integrity.

The Microsoft Platform

I hope it’s kind of clear why Microsoft took this direction in building out the Azure Services Platform. For example, the underlying technologies used to implement Azure include Windows Server 2008, but Microsoft decided to call the compute capability Windows Azure, because it represents application containers that operate at a higher level in the stack, instead of Windows Server VM instances for us to use directly. In fact, it actually required more engineering effort this way, but the end result is a platform that provides extreme scalability and availability, the transparency of highly distributed and replicated processes and data, while hiding the complexities of the systems automation and management operations on top of a network of globally distributed data centers. This should help clarify, at a high level, as to how Azure can be used to extend existing/legacy on-premise assets, instead of being just another outsourced managed hosting location.

Of course, this is only what this initial version of the platform looks like. From a long-term perspective, Microsoft does plan to increase parity between the on-premise and cloud-based platform components, especially from a development and programming model perspective, so that the applications can be more portable across the S+S spectrum. But the fundamental differences will still exist, which will help to articulate the distinct values provided by different parts of the platform.

Thus the Azure Services Platform is intended for a “new class of applications”. Different from the traditional on-premise database-driven applications, the new class of “cloud applications” are increasingly more “services-driven”, as applications operate in a service-oriented environment, where data can be managed and provisioned as services by cloud-based database service providers such as Amazon S3/SimpleDB, Google MapReduce/BigTable, Azure SQL Services, Windows Azure Storage Services, etc., and capabilities integrated from other services running in the Web, provisioned by various private and public clouds. This type of applications inherently operate on an Internet scale, and are designed with a different set of fundamentals such as eventual consistency, idempotent processes, federated identity, services-based functional partitioning and composition (loose-coupling), isolation, parallel and replicated data and process architecture, etc.

This post is part of a series of articles on cloud computing and related concepts.

SOA – End of Life 2009.01.01

dachou — Fri, 09 Jan 2009 11:59:37 GMT

It has just been a few days since Anne Thomas Manes at Burton Group published her post “SOA is Dead; Long Live Services”, and it has stirred up quite a storm of comments in the blogosphere. Most of what I read though, seem to be in alignment with what Anne Thomas Manes said -

SOA met its demise on January 1, 2009, when it was wiped out by the catastrophic impact of the economic recession. SOA is survived by its offspring: mashups, BPM, SaaS, Cloud Computing, and all other architectural approaches that depend on “services”.

Her article clarified that it is the “SOA as we know” (and the terminology used) has faded into irrelevance, the SOA that called for a comprehensive transformation of an organization’s view and management of its portfolio of data, technology, process, and people. Indeed, many people (such as David Linthicum, Eric Roch, JP Morgenthal, and the ongoing debate on InfoQ), for a number of years now, have been cataloguing why most enterprise SOA efforts fail miserably.

In general, I think the community is coming to the realization that SOA really is an architectural approach, not a set of technologies to implement. From that perspective Microsoft actually has been spot-on in terms of not offering “SOA”-branded products, but instead advocating customers to carefully design and build the right type of SOA for their organizations.

While there are many, many identified technical reasons why most SOA projects don’t succeed, Mike Kavis in his post has articulated one perspective nicely at a high level (just summarizing his list here):

We think process is a bad thing and it slows us down
We are impatient
We don’t understand what an architect really is
We don’t understand what architecture really is
We lose sight of the value and argue semantics
We lack leadership skills and emotional intelligence

This highlights one area why SOA hasn’t been successful: the human factor. But this doesn’t only apply to SOA; it’s just that the SOA requirements for organizational transformation and consistency amplify issues associated with the human factor. So what aspects of the human factor that make SOA difficult to implement?

Lack of patience, persistence, and perseverance

I think this is applicable on many levels. SOA requires a long-term, incremental build approach, but many projects are required to justify immediate or short-term ROI. Or from a different perspective, people just naturally expect to see some form of immediate benefits, and lose interest/motivation when the reality of SOA hits after the first few initial projects, which are often ESB-driven infrastructure optimization efforts, or point-to-point integration efforts. The lack of immediate business agility and cost savings gives people excuses to question the approach, reduce level of support, etc.

There is the aspect of jumping on the bandwagon simply because SOA was the acronym du jour and that it seemed smart to talk about it, without investing sufficient research, discipline, and due diligence to do it right. Of course, those who are impatient to jump on the bandwagon would just as (if not more) quickly to jump off at the first sign of trouble. There was sufficient intention or willingness to invest in SOA endeavors, but the impatience of not acquiring necessary expertise resulted in failures.

And truth is, SOA is not easy. Many organizations lose sight of the most important aspect of SOA - “how” to do SOA, not “what” we do SOA with. To many organizations it just seems simpler to follow marketing hype and implement products that are branded as SOA suites and think that an SOA can be constructed using the new infrastructure.

Resistance to change

People are naturally resistant to change, especially tough changes like SOA. Large organizations stand to gain more from SOA, but at the same time, those large organizations that have operated for many years in traditional functional silos have always resisted enterprise-level efforts that require them to build more dependencies on shared resources.

And SOA meant changes across all aspects of IT disciplines as we know. Operationally, traditional SLA management processes need to be adapted as downstream systems may need to inherit availability and performance requirements from upstream systems. Design-wise, it’s not just about exposing functionality as services, but more in the context of how a function is useful from the enterprise’s perspective; but that requires a higher level of collaboration beyond one department’s development teams.

Also, distributed computing is not simple. When we build process-level dependencies on other systems, efforts required to troubleshoot issues that one does not have full control over become magnitudes more challenging. This requires a major adjustment from the ways IT teams work today, and in those cases it’s often easier to point the finger at others first.

Organizational dynamics

The above aspects often apply to individuals. But when we look at an organization as a whole, the effects are also amplified. SOA requires a higher level of collaboration between teams in an organization. Each team or department used to having a higher level of autonomy in terms of managing their budgets, schedules, clients, technology, etc.; relatively independently from other teams. How to find the right balance between organizational consistency and flexibility with sufficient local autonomy, is unfortunately in itself requiring a uniform understanding and approach within the organization.

And politics. Not everyone likes to work with everyone, and individuals used to be relatively shielded within their own teams/departments/silos. But SOA requires breaking down the walls of silos, and can expose people more to personalities they may not like to work with, causing more contention among people. If not managed carefully, such as not positively reinforcing the correct behaviors, this can more quickly send the wrong signals to workers and hinder progress.

Lastly, strategic thinkers who understand what it takes to do SOA, tend to be the minority in today’s IT organizations which typically focus on tactical goals and are also measured as such. It’s just difficult for a few individuals to influence and steer an organization to adapt new changes.

So what now?

Thus it’s the “how” we do SOA that is the most important. It is the architectural disciplines, organizational cohesion, strategic leadership, etc. that most significantly impact the outcome of SOA efforts. And from that perspective the architectural principles of service-oriented architecture are still sound. In fact, as many people are already jumping into the next new big things such as cloud computing, “service”-oriented or driven concepts and considerations become much more important than before. Furthermore, cloud computing, in my opinion, has to do more about services than simply moving existing on-premises infrastructure into a utility-based cloud.

Perhaps it’s time to take a hard look at each SOA effort and ask the hard questions. Is it really meaningful, or valuable, to do SOA for your organization? Does your enterprise really need real-time process-driven integration, or traditional data integration, or a hybrid model, can suffice? Can people in your organization work collaboratively towards common goals and standards? Does your organization have what it takes to undergo and withstand such transformation? And so forth.

This doesn’t mean we should stop doing SOA (and regardless of what name we use to call it), but we should do it for the right reasons, and do it right. And more importantly, having the right people in the right places to see the plan through. This means having the right skills to plan and lead an organization to transform all aspects of data, technology, processes, and people (in knowing how to deal with the human factors mentioned above). There are still very significant benefits that SOA can bring, evident in the few organizations that have been successful with it.

Another lesson that can be learned from this ongoing discussion is that, SOA was deemed unsuccessful because it presented very significant gaps to the way existing IT organizations work today (such as what Dana Gardner mentioned in his post). However, we have to be careful in thinking that “abandoning SOA” and moving on to the next big thing – cloud computing, will solve all of our issues (doesn’t that sound eerily familiar?). It is evident that these gaps present such large gaps to many organizations, that the organizational aspects become the biggest impediments to progress. As technologists it is easy for us to say that the next major innovative technology trend will bring about sweeping changes and transformational benefits. But the reality is, cloud computing, as an extension of SOA, will require even more maturity and competencies in working with SOA to implement successfully. Indeed cloud computing presents a more prominent influencing factor to transform legacy IT, but it won’t make it any easier than SOA did. Organizations that want to take advantage of these transformative technology trends need to not only understand the technologies involved, but really pay attention to planning the organizational and people side of the endeavors.

SOA Change Management Strategies

dachou — Wed, 23 Apr 2008 06:55:14 GMT

By today's standards, it is pretty well-understood that governance is a critical success factor for enterprise SOA initiatives. And there is already a considerably saturated/consolidated market providing the SOA governance solutions (see Gartner's Magic Quadrant for Integrated SOA Governance Technology Sets, 2007, Forrestor's SOA Service Life-Cycle Management Q1 2008, and SOA Governance Conference 5 for some content from HP, IBM, Progress Software, and SOA Software).

A quick glance over the product features and discussions finds that the SOA governance tools in the market today focus on a set of key capabilities:

Design-time lifecycle management (contracts & metadata & artifacts management, change & reporting notification plus automation, QA and test automation, dependencies mapping, policy compliance auditing, etc.)
Run-time lifecycle management (versioning, decommissioning, monitoring & reporting, change deployment management, usage and exceptions metrics collection, policy compliance enforcement, etc.)
Security and access control (policy-driven fine-grained service authorization)
Integration with service infrastructure (ESB, identity management, single sign-on, MDM, service registries, metadata repositories, PKI, etc.)

Just one way of categorizing the capabilities. Most vendors have their own ways of categorizing/describing these products, and some provide more built-in features. These capabilities are quite advanced and do address a wide range of governance needs. And then there is another set of products that aims to address SOA testing and automation needs.

How to Validate Incremental Changes Deployed to a Live, Real-Time, Inter-Connected, and Inter-Dependent Distributed Architecture?

The SOA governance tools support this from the perspective of making sure services are developed in compliance to policies and defined contracts, then managed in runtime after deployment and release. The SOA testing tools support this from managing and automating test efforts against component-based service deployments. However, there seems to be a considerable gap in terms of validating and managing changes in an enterprise SOA environment.

A closer look uncovers many tough questions:

How do we validate changes in an SOA where a set of (sometimes hundreds of) physically distributed systems and services have been managed and governed into one logical/virtual entity? Specifically, how do we ensure a particular change being released into the live production environment won't break anything else, which often are other connected mission-critical systems that are running concurrently, based on traditional multi-staging change management strategies?
Do we trust that changes verified in a QA environment will behave exactly the same in production? If so, is the QA environment an exact replica of production, including the operational data that processing logic depends on?
Do we just "unit test" the service components associated with a unit of change, or do we work with other teams to conduct a full integration test in QA? And in an integration test, is the whole virtual enterprise involved, or just the directly connected system components?
How do we ensure that downstream components connected in multi-part, distributed transactions are not impacted if we don't conduct integration tests on everything?
How do we ensure that the QA environment is pristine and how do we coordinate among multiple team's project schedules, which are often more different than similar?
In a highly inter-connected and inter-dependent environment, how do we manage the worst-case scenario where hundreds of service components are impacted by a change?
If change verification/testing in production is allowed, how do we facilitate synthetic transactions so that actual production data is not interfered by test cases?

SOA Requires Different Methods of Managing Changes

Well that's obvious, right? :) But fundamentally it probably requires a different way of thinking too. For example, traditional multi-staging change migration strategies (dev, test, QA, staging/regression, prod, etc.) don't lend themselves very well anymore as they were more effective at managing changes that are more autonomous and local in nature. Now that changes are inter-related and inter-dependent, and often impacting a high number of systems not under any one team's management, full integration tests may mean coordinating schedules, code/data versions, security, etc. all bundled into one massive enterprise-wide test. Which would be too difficult and complex to undertake on a regular basis, and as a result what happens to the agility SOA was intended to deliver?

The SOA governance tools today address this change management need mainly via service lifecycle management, so that newer versions of services can be deployed with minimal initial dependencies. Then over time consumers can be migrated over from the older versions in their own independent schedules, and eventually the older versions can be decommissioned once no one is using them anymore. However, it isn't always that applications can support multiple versions of the same service (and best practices on when a new version is required as opposed to hot fixes is still unclear), or the trade-offs in management costs may not justify doing so.

And is the only effective solution to manage changes in an SOA environment, to implement SOA governance tools? Tools are tools, and they do help, but often they also bring a layer of complexity as well. And governance tools are best suited to support specific processes defined in specific architectures; they don't actually solve problems in this area, as the problems are due to the collective processes and systems bound together in an SOA. Thus, well-defined processes and architectures are still required, then tools can be used for automation and enforcement.

Build Layers of Encapsulation and Abstraction Into an SOA

This concept is markedly different from the initial intention of transforming a disconnected and silo'ed enterprise into one seamless entity. But basically, one massive logical SOA may actually be more difficult to manage, than a set of smaller localized/partitioned SOA's federating as one. Even though more costly from an infrastructure perspective, there are many benefits to this approach (especially for larger enterprise environments):

Layers of abstraction/encapsulation provide boundaries where changes can be localized instead of being required to be verified against the entire end-to-end architecture
Allows for shrinking and localizing the scope of impacted components in integration tests, into smaller and more discrete units which become easier to coordinate and schedule between smaller number of involved teams
Still not effective at addressing changes that impact a high number of systems, but smaller and localized changes no longer have to be tied up and wait for the "big test" to complete, to be released
Over time, the entire architecture is re-validated
From a security perspective, this supports defense in depth

ESB vendors will like this, as these products are the most effective solutions to build in layers of encapsulation/abstraction into an SOA. But there are many different kinds of ESB's in the market. Point is, from an enterprise architecture perspective, we really don't need to migrate to a full centralization model when implementing an SOA. A model where local SOA's federate into one enterprise SOA may work out better, providing sufficient local autonomy (type of ESB, local governance, etc.) while coherently organizes the enterprise into one logical entity, and likely higher scalability, reliability, and agility.

Also, data integration/replication, even though often cited as a major anti-pattern in SOA, when applied appropriately, is often an effective way to add a layer between different systems, when an encapsulation layer is preferred. Basically, inter-dependencies are minimized if there are no distributed transactions binding systems together at the process level.

A Different Process-Oriented Approach

A resulting SOA validation strategy is to have a centralized management of integration testing schedule in the enterprise QA environment, so that at any one point in time, only one set of changes is being validated. As a result, most integration tests should occur in localized groups and at more discrete intervals/schedules, as opposed to trying to get everyone to undergo validation at the same time, or cause people to run over each other with conflicting changes.

Thus there are three testing models: unit test, localized integration test, and full integration test. Full integration tests are usually preferred (perceived to be more accurate and comprehensive), but also too cost-prohibitive to undertake. The best trade-off is localized integration tests performed at more discrete and distributed schedules, as each validation can assume it’s done in a pristine environment, and logically the entire architecture is re-validated over time.

In addition, from a SLA or security management perspective, systems are often categorized into different criticality tiers. In an ideal SOA where everything is connected to everything, it shouldn’t mean that everything is now molded into the same tier. Consequently, different strategies can be devised to re-validate systems in different tiers. For example, only require unit test for systems in lower tiers.

The enterprise perspective can be that, all three types of tests are done; it’s just management effort that is required. For example, a full integration test can be scheduled on a quarterly basis, while localized integration tests are used to release regular changes.

SOA Change Management Requires a Multi-Faceted Approach

There are still some areas where the current set of SOA governance and testing tools don't address very well. It's not that these products lack maturity; it's just some issues are inherent in distributed computing and are created by the collection of design decisions, processes and methodologies, and technologies implemented in an SOA (which obviously, can be different for each organization). The SOA governance solution vendors themselves state that governance is a people-oriented process.

Thus, when architecting SOA governance, additional thought needs to be placed in these areas, in a change management context, and integrated into many different aspects of an SOA, leveraging an integrated approach across people, processes, and technologies.

See Related Content

Talking about Service Oriented Architecture

dachou — Tue, 15 Apr 2008 13:29:45 GMT

I had the privilege to speak at the April monthly meeting at the Los Angeles Java Users Group. The meeting was held on April 1st at the Sun Microsystems office in LA, and so that in itself was the source of a few jokes lobbed at me. At times I was also referred to as Darth Vader, but all in good fun.

I had a great time chatting with the group, and was fortunate enough to see some familiar faces, especially a few back in the days when I worked at Sun Microsystems at this same office.

The topic of my presentation was "Service Oriented Architecture". This was picked as SOA is something that can be talked about from a technology-agnostic perspective, especially if we're focusing on the "A" in SOA.

| View | Upload your own

The presentation was mostly repeating what is considered "common knowledge" in SOA these days, such that SOA is an "architectural style", and that most of the fundamental principles consist of best practices and learned lessons in software and systems engineering in distributed computing, in the IT industry; and applied towards enterprise IT and systems integration efforts. And SOA is a pretty overloaded term too, such that many different perspectives exist, but all are valid:

Organizationally - developers, architects, managers, business stakeholders, executives, etc.

Architecturally - enterprise, infrastructure, security, data, integration, application, etc.

But in general, there is no one form of SOA that fits all organizations. Large enterprises tend to have a different set of issues and solution approaches at different priorities compared to medium-to-small businesses. The kind of SOA implemented at one organization may not necessarily be effective or needed for another organization. Just the same as all problem-solving scenarios, it is most effective to fully understand the issues, and then figure out approaches to address them.

Similarly, a maturity model-based approach is an excellent way to plan an SOA journey, but no existing maturity model (available from most of the technology vendors and analysts) is necessarily the right one to use. It's more important to pick out the aspects in a few maturity models that work for each organization, than to try to follow/implement a specific one religiously.

Meanwhile, many organizations looking at SOA are also faced with a number of questions at the implementation level:

Process vs. data integration
SOAP vs. REST
Data federation vs. data replication
Synchronous (RPC) vs. asynchronous (EDA)
Transactional (2PC / compensational) or not
Trust vs. impersonation
Centralized vs. federated ESB (or no ESB at all)
Stateful (BPM) vs. stateless (orchestrations)
Real-time vs. latencies

Conceptually, transforming a traditionally silo'ed enterprise environment into one logical real-time entity does seem to be a really beneficial proposition. However, that is often very difficult to obtain. Some of the factors include transforming all systems to be 24/7 instead of having independent maintenance outage windows, bear the transaction volumes of the highest trafficked systems, coordinating integration tests between multiple teams (or the entire enterprise), be subjected to the highest security compliance, etc.; when many didn't have to be when they were back in the silo environments.

Lastly, what may SOA look like in the future? There's talk about event-driven architecture (EDA) as the next step in evolution, which could be a more natural way of integrating business processes than the current RPC-style of tying everything together. There is also talk about extending SOA out to the Web, for enterprises, and additional advances in cloud computing, such as infrastructure services like internet service bus(es), federated security, cloud-based identity management and privacy controls, cloud-based data transformation services, etc. Continued progress in the semantic and interpretive Web may also play a major role in adding context to Web as a platform. And advances in model-driven programming and integrating them into service-oriented compositional architectures (such as Oslo on the Microsoft side, and SCA on the Java side) that change traditional multi-tiered application architectures to fully composite application architectures in all tiers (client-side mash-ups plus server-side mash-ups).

SOA Security - Enterprise Architecture Perspective

dachou — Thu, 18 Oct 2007 04:04:52 GMT

This week I had the opportunity to speak at the IT Architect Regional Conference in San Diego, on the subject of architecting enterprise SOA security. It is an interesting event, with speakers from Microsoft, IBM, Oracle, TIBCO, Fair Issac, and many other organizations. We even gave away a brand new XBox 360 and a Zune!

In a nutshell, my presentation was intended to point out the security aspects of planning an enterprise SOA, and a few topics that don't seem to be covered very often, and with an emphasis towards the future and navigating the organizational and cultural issues.

A brief overview -

Basically, some of the fundamental changes in SOA, such as:

Moving from low-volume batch-oriented data replication architectures to highly interactive real-time architectures between connected systems
Plus the migration towards Event-Driven Architectures (EDA) means an exponential growth in real-time (though asynchronous) communication, as each event can potentially trigger off a number of downstream events which can trigger off more events being sent across the network
All this moves the security concerns from the traditionally isolated infrastructure and application groups, into the integration layer that becomes a cross-cutting concern for everyone involved
SOA can also magnify existing issues such as identity management (or the lack of), and create new issues such as exposing mainframes directly to web traffic (for sake of real-time access into legacy applications and data)
The ideal state of "everything talking to everything in real-time" also means a breakdown of traditional physical network zones/perimeters, where DMZ becomes more like a reception/lobby area instead of a quarantine area, and data centers can no longer be considered locked down
Lastly, the threat environment has also evolved from single PC attacks, to DoS system attacks, and to today's application and data-level attacks, with lowered complexity and lowered barrier of entry (facilitated by vastly improved competencies in using XML)

Then of course, these changes also bring along many questions. Particularly many that represent conflicting approaches and each organization may come up with different solutions based on varying trade-offs.

For example,

Trust vs. impersonation/delegation. There are many security groups that believe enterprise network environments are inherently unsafe (which is agreeable), and thus all systems will need to require end-user authentication (regardless whether they are user-facing or intermediaries or downstream producer systems), and that "trust" cannot be trusted
From a different perspective, this debate is also centered on the concept of implementing end-to-end vs. peer-to-peer security contexts
There is also a lot of recent discussion on moving security intelligence (w/ centralized management) into the endpoints (laptops, mobile devices, etc.), or moving intelligence into the network (like recent advances in NAC)

In my opinion, trust-based architectures are much more flexible and scalable, and implementable by today's technology standards. And we couldn't completely eliminate trust in an impersonation/delegation model anyway. For example, a connected node/system has to "trust" service wrappers, agents, and/or local system components to verify user credentials against a centralized repository (such as Active Directory, LDAP, etc.) anyway.

On the other hand, having end-to-end security contexts is indeed conceptually more secure, as it can help better address the man-in-the-middle attacks, but in an SOA with a number of intermediaries between consumers and producers, there is still not an effective solution in managing public keys to support end-to-end message-level data encryption.

It's always interesting to try to take a peek at what may be possible in the future.

Most SOA discussions still seem to be focused on implementing "SOA in the enterprise". While that is very important, as enterprise architects we should also start to look at the growing trend of becoming more open on the Web, to an environment where enterprises essentially have no physical perimeters and security zones, largely due to the increasing number of direct and real-time connections into an enterprise (for sake of facilitating transactions with business partners).
Plus at that time we would also need to be concerned with the connections going from inside an enterprise out to the Web, as more and more internal systems becoming service consumers themselves
Thus a potential trend is moving away from trying to secure one large environment for the entire enterprise, migrating to a model where numerous (and potentially overlapping) smaller logical partitions (or zones) can be implemented to be provisioned with more targeted and effective security solutions (depending on data sensitivity). Rationale behind this is that it'll be more effective to try to protect smaller attack surfaces, even from a systems architecture perspective
Another interesting trend already underway is the growing centralization of data and content. Instead of consolidating everything into one or a few large enterprise content management deployments, organizations are creating smaller islands of data and content using collaboration platforms such as SharePoint. The point here is moving from mass distribution of data and content, and smaller islands seem to be lower hanging fruits at this point

Finally, some overall talking points. One important and interesting point that was kind of new to many people is that security in SOA has to be planned and designed just like another process layer. If we overlook security and not plan it carefully, we may end up creating tightly coupled elements in the overall architecture, and impacting the agility we intended to create.

The most visible example of this is trying to implement message-level encryption for the sake of data integrity (message digests) and confidentiality. In order to establish an end-to-end security context (so that intermediaries, including the ESB, should not be able to decrypt sensitive data on transit to the destination), both the intended consumer and producer have to know exactly how to encrypt and decrypt data. And that depends on a previous exchange of public keys, which in this case had to occur directly between the consumer and producer endpoints. That in a way is tight coupling, as the consumer and producer endpoints have to know about each other, and are required to establish a one-to-one, peer-to-peer relationship in terms of public keys exchange used for encryption/decryption. To alleviate the situation, a centralized public key infrastructure can be implemented in an enterprise so that the management and decisions on public key usage can be externalized from endpoints and centralized. However, enterprise solutions in this area are still evolving, and we haven't yet seen effective solutions for doing similar things beyond the enterprise and on the Web.

Lastly, the most important point is that, just like SOA governance, security is also a huge factor of the organization and corporate culture. We have to take a process-first approach to the problem (instead of technology-first), then weave in the technology delivery part of it.

For those interested, the entire slide deck I used can be downloaded from my Windows Live SkyDrive. If you don't have Office 2007, you can download the free PowerPoint Viewer 2007.

Share this post :

IT Architect Regional Conference 2007

dachou — Sun, 23 Sep 2007 20:45:31 GMT

Scheduled for October 15-16th, 2007, the ITARC 2007 conference is an event focused on the architecture topics in IT. The organizers at IASA (International Association of Software Architects) have arranged over 30 session in 4 concurrent tracks covering enterprise architecture, infrastructure architecture, software architecture, and architecture fundamentals.

Many notable speakers (just to list a few) are scheduled to present at the conference:

Chris Haddad, VP, Burton Group - Infrastructure Architecture in the Business Domain
Fred Waskiewicz, Director of Standards, OMB - Service Oriented Architecture: Making the Leap, Leveraging the Standards
Scott Ambler, Practice Leader, Agile Development, IBM - Agile Strategies for Enterprise Architects
Denise Cook, Rational Method Architect, IBM - Software Architecture Analysis Methods
David Chappel, VP & Chief Technologist, Oracle - Next Generation Grid Enabled SOA
Vince Casarez, VP, Portal Platform, Oracle - Web 2.0 for the Enterprise
Simon Guest, Director, Microsoft - Putting the User back into Architecture
Harry Pierson, Architect, Microsoft - Moving Beyond Industrial Software
Lynn Langit, Developer Evangelist, Microsoft - SharePoint Architecture, Lessons from the Trenches
David Chou (myself), Architect, Microsoft - Architecting Enterprise Security

The full agenda can be found at http://www.iasahome.org/web/itarc/socal/agenda

Also, it just so happens that Microsoft is sponsoring this conference as well. ;) As a result, we have a booth at the event. Looking forward to see you there!