No Spin Architecture : ESB (Enterprise Service Bus)

Microsoft's Enterprise Services Bus (ESB) Strategy - Part 4/4

makif — Tue, 23 Jan 2007 17:22:00 GMT

About: This is the fourth post in a series of four about Microsoft’s strategy for Enterprise Service Bus. This post discusses Microsoft offerings in the area of ESB

Greetings,

Let me start by wishing all of you a very happy 2007. In the previous three posts I had described the core concepts behind an ESB, tried to dispel some myths about it and discussed the tangible manifestations of the concept. This post focuses on Microsoft’s offerings in the area of ESB.

Microsoft does not believe on a one-size-fits-all approach to ESB and has so far resisted the pressure to rename Biztalk or a variation of it as an ESB. Microsoft’s approach to ESB is to understand the customer’s core reasons for seeking an ESB and then propose a solution that suits the need of that enterprise. I strongly believe that this is a better approach than selling ESB-in-a-box; however, it has made some analysts unhappy who believe that Microsoft should have a product marketed under the ESB label. Based on my experience of what is being sold in the marketplace under the ESB label, Biztalk could definitely qualify as one, but I am glad that we have refrained for renaming it so far.

Microsoft has contributed to the idea of ESB as an architectural pattern and has made available some of the technology-agnostic thinking and best practices in the integration patterns book and associated articles.

In terms of the technologies some of the key offerings that address the integration issues that ESB try to resolve are as follows:

Windows Communication Foundation

Windows Communication Foundation is a messaging API that allows you to develop a standards-based integration mechanism; WCF can help you implement an ESB as an architectural pattern. One of the best things about WCF is the richness of the API which reduces the amount of code that you have to write for infrastructure plumbing (reliable messaging, transactions, security etc.) by many orders of magnitudes. It is of course possible to implement an ESB pattern without WCF but not having to write, customize and/or generate tens of thousands of lines of code that you have to maintain, is a good reason for using WCF

Biztalk 2006

Based on the reasons for why you want to have an ESB, Biztalk provides features like Brokered Communication, Web Services Support, integration with development tools, Business Activity Monitoring, Service orchestration and a number of others ESB features. Considering what is selling today in the market place as an ESB, Biztalk plus web services and base connecter classes can provide a superset of features and functionalities. The decision to use Biztalk as an ‘ESB’ would depend on your needs and whether you will end up building Biztalk yourself. One of the key benefits of Biztalk is the availability of adapters that greatly ease the process of ‘plugging’ in systems, for a complete set of adapters and other information about Biztalk please visit http://www.microsoft.com/biztalk/default.mspx

Host Integration server

If your reason for implementing an ‘ESB’ was to expose your legacy mainframe systems as part of SOA than you can consider Host Integration Server. I know that it does not qualify as a pure ESB but in the real world that seems to be one of the core driving reasons behind why customers want to buy an ESB.

Messaging software + Web Services

Finally by combining .NET 3.0 and your existing messaging software you can construct a number of features provided by the ESB

In summary, do not buy an ESB-in-a-box (yes even Biztalk) unless you understand the benefits it would provide to your IT and business, also do not build something that can buy J I know it sounds contradictory, what I am trying to communicate is that rather than starting from what’s selling as an ESB, start with your organizational needs, construct a requirements matrix and then apply that on the available choices to reach a conclusion that suits you.

Stay tuned for a series on Web 2.0 Demystified

Best regards,

Mohammad

Microsoft's Enterprise Service Bus (ESB) Strategy - Part 3/4

makif — Fri, 24 Nov 2006 02:59:00 GMT

About: This post is the third in a series of four about Microsoft’s strategy for Enterprise Service Bus. This post discusses the various tangible manifestations that an ESB can take; the fourth and final part will discuss Microsoft offerings in this area

Greetings,

Let me start by thanking all the people who have expressed their views on the first two postings by sending emails and postings on their blogs. The number of emails I have received about the ESB postings have been overwhelming, one of my favorite ones was a criticism of my position by Udi Dahan who stated that the word ‘Bus’ and ‘Broker’ are not synonyms and that “SOA is not about hooking together poorly design, hastily shipped legacy systems. Each service is responsible for its own communications with other services. If it needs to "transform" messages, whatever that means, it should do it by itself”. I do agree with the majority of what Udi has written (including the differences he points between Bus and Broker), I respectfully disagree with his position on SOA as I think that in addition to a number of other advantages and building the ‘new’ services right, SOA can enable an organization to encapsulate its legacy systems, that is an important part of the reality in the field and typically is a critical component of adoption of SOA in an organization. Microsoft has technologies like WCF that greatly helps an organization in realizing SOA in the manner described by Udi but we also need to take a realistic perspective in terms of market needs and current state.

Getting back to what ESB can mean to you, when thinking about the ESB the first thing you should try to do is determine the capabilities that are relevant to your organization, ask questions like ‘Why do I want an ESB’ and ‘How does it help improve our offering as an IT group to our internal and external customers”?. If you cannot determine the how having an ‘ESB’ would make your offerings faster, better and/or cheaper then I suggest that you take a step back and reassess your needs.

Depending on what you want to get out of an ESB, it can take various shapes and forms, please note that they are not necessarily mutually exclusive but different ways of thinking about an ESB, you may end up with an implementation that is a combination of these ideas

ESB as a pattern

ESB can be thought-of as an architectural integration pattern, irrespective of which technologies you choose to implement it, you can think of ESB as a middle tier concept that connects the various end points in a bus topology.

ESB in a box

Personally, I do not think that it is possible to have an ESB-in-a-box solution that fulfils your needs. Whether it is Microsoft or Sonic or Tibco, it is unlikely that the ESB-in-a-box would resolve all the issues that you are attempting to resolve through this mechanism. However, in most of the cases the ESB-in-a-box, when chosen based on your organizational needs, can provide value and through customization and extensions you can reach the desired outcome. This is especially true if one of your main needs around ESB is to externalize translations, transformations and business logic from individual systems and for having centralized interaction and connection mechanisms between systems implemented in different technologies

Messaging software + Web Services

In some cases the outcome you desire can be achieved through building web services on top of your messaging layer, i.e. you may not need to buy a COTS ESB solution.

Legacy integration

If your primary purpose of having an ESB is to encapsulate and expose your legacy system and make them part of the new architecture in your organization, you can achieve that goal through specialized software designed for exposing/integrating particular types of legacy system (e.g. exposing mainframe assets through Host Integration Server).

In the fourth and final post, I will be discussing Microsoft’s offerings in the area of ESB and how you can use these technologies and best practices to implement an ESB in your organization.

Best regards,

Mohammad

Microsoft's Enterprise Service Bus (ESB) Strategy - Part 2/4

makif — Sun, 17 Sep 2006 21:08:00 GMT

About: This post is the second in a series of four about Microsoft’s strategy for Enterprise Service Bus. This post discusses the various definitions of the term ESB, defines the attributes of an ESB and tries to dispel some myths about it

Greetings,

If you want seamless integration between systems implemented on disparate technologies, if you want to plug legacy systems in an SOA environment, if you want to have composite applications based on multiple services, if you want dynamic discovery and binding capabilities, if you want true interoperability, in short if you want to resolve all of your integration nightmares, then ladies and gentlemen I give you the ENTERPRISE SILVER BULLET (ESB).

The ESB is a magical and nebulous piece of software that plugs itself in the middle of your enterprise with neat bidirectional arrows going to boxes labeled ‘mainframe systems’, ‘Java Systems’, ‘.NET systems’ and ‘other legacy assets’, it is all you will ever need to reduce costs, increase your time to market and optimize your IT operations etc. etc. and etc.

As all of you know, despite sounding familiar, most of what I have written is incorrect or minimizes the amount of effort involved in achieving the desired outcome. Unfortunately most of what I have written is based on the descriptions and presentation of ESBs I have come across.

Let us explore a the brief history of ESB and beyond-the-hype attributes of this type of software

A brief history of ESB and its many definitions

The term ESB was first used by Sonic in 2002 to refer to their SonixXQ product that was an XML-enabled MOM product, the product was later re-branded as Sonic ESB. A few months later, Gartner called ESB a strategic investment and then very soon every thing from integration servers to messaging products were re-branded as ESBs, I even attended a presentation where a vendor showcasing their SOA offerings displayed the Ethernet wire as their official ESB strategy.

In addition to the re-branding there was also a plethora of definitions put out, everything from Gartner’s “A Web-services-capable infrastructure that supports intelligently directed communication and mediated relationships among loosely coupled and decoupled biz components”, to Burton’s group “The ESB label simply implies that a product is some type of integration middleware product that supports both MOM and Web services protocols” to IBM’s “To put it bluntly: If you have WebSphere MQ and other WebSphere brokers and integration servers, you have an ESB (Bob Sutor, IBM).”

In general, the market quickly realized the potential of promising a new pill for what was essentially an integration problem. In many cases the companies who bought the ESB products were disillusioned by the amount of effort and customization it takes to ‘plug’ a new system in to an ESB, the licensing costs involved in having an ESB node license in a distributed model, the learning effort involved in creating customizations/extensions for ‘plugging’ legacy systems and the maintenance nightmares. Ironically, in many cases businesses ended up deploying multiple ‘ESBs’ and once you have multiple ESBs what do you need to integrate them?....... Yes you guessed it right; you need another ESB to integrate the ESBs, one BUS to rule them all!

I am not against ESB as an idea, product or integration pattern, I am against setting unrealistic expectations around a buzzword that has resulted in to considerable grief for many early adopters.

Let us explore the basic attributes of this type of software

Attributes of an ESB

There are some attributes of an ESB that all major industry experts have agreed upon, an ESB at the very minimum should support:

Brokered communication: The core purpose of an ESB is to act as an intermediary between systems that need to send and receive data from each other. The notion of an intermediary is essential to the ability of an ESB to provide address indirection, rules-based routing, message transformation and other capabilities. The core value of having brokered communication is to increase loose coupling between systems and externalize integration logic for potential reuse and ease of maintenance

Routing: In order to integrate systems in a loosely coupled manner, an ESB typically has some type of name space or directory which resolves a logical service name to a specific implementation at run time. An ESB should also be able to maintain and execute rules for topic-based, property-based or content-based message routing.

Basic Web services: An ESB should at the minimum support basic Web Services standards and their foundation, i.e. it should provide support for WSDL, SOAP over HTTP, TCP/IP and XML

Endpoint metadata: An ESB must be capable of storing or accessing information about service interfaces and message schemas.

Non-functional requirements: An ESB should be able to support deployment in a high availability configuration; it should also be scalable and reliable and offer management capabilities

At the very minimum, any decent ESB product will include the above five properties as they form the core value proposition for buying a piece of software that eases integration and interoperability problems in an organization.

In addition, there are other attributes that lend themselves to be placed in an ESB, whether an organization would need some or all of these attributes will differ based on a variety of factors that I plan to discuss in my next post. Some of the other attributes that could be considered with respect to an ESB include

Externalizing business processes and rules. An ESB could contain capabilities that allow for externalizing business process and rules from applications and placing them in the ESB.

Business Activity Monitoring. An ESB is a natural place to conduct business activity monitoring, if it is indeed the middleware through which systems are communicating, interacting and interoperating with each other then having capabilities that allow for analysis of the data for various stakeholders can be quiet useful

Interoperability and integration. Web Services are increasingly the ideal choice to implement interoperability between different systems, however, Web Services are sometimes not the best or the most optimal option. An ESB should ideally contain capabilities to facilitate interoperability without Web Services reducing the amount of custom effort required for each type of system.

Based on the above attributes, you can think of an ESB as

Enterprise: The “Enterprise” in ESB refers to the fact that an ESB will generally be installed and managed within one virtual enterprise - one company and possibly some of its customers and suppliers.

Services: The “Service” in ESB refers to the value added services provided by the ESB, including routing, management and transformation.

Bus: The “Bus” in ESB refers to its support for non-disruptive service substitution. A logical bus topology facilitates these properties as a bus enables any endpoint to talk to any other endpoint in a decoupled manner.

Conclusion

In this article I discussed the history and basic definition of the term ESB. In my opinion rather than buying an ESB and retrofitting your needs around it, you should define what your organization requires from an ESB right now and what you will need from it in future. Based on the needs you can come up with an evaluation criteria that will allow you to choose an ESB (as a product or a pattern, something I will discuss in the next post) that provides the maximum value to your organization.

In the next post I will attempt to describe Microsoft’s strategy for ESB and discuss the future of this type of software based on the new developments in the industry around interoperability and integrations standards.

Best regards,

Mohammad

Microsoft's Enterprise Service Bus (ESB) Strategy - Part 1/4

makif — Fri, 11 Aug 2006 21:05:00 GMT

About: This post is the first in a series of four about Microsoft’s strategy for Enterprise Service Bus, this post introduces the series.

Part 2: http://blogs.msdn.com/mohammadakif/archive/2006/09/17/759414.aspx

Hello,

Over the past one year there has been considerable discussion in the IT circles about Microsoft’s ESB strategy, there seems to be confusion about what an ESB really means and whether it is possible to implement an ESB (which some believe stands for Enterprise Silver Bullet) using Microsoft technologies, over the next three posts I will attempt to address the following catrgories which encompass the bulk of questions I have heard

What is an ESB
Common ESB characteristics
Microsoft’s ESB Strategy
Microsoft’s ESB Solution
Myth Busters
Case studies
Microsoft’s ESB Roadmap
Getting started

Stay tuned and let me know if I have missed a broad category of questions.

Part 2: http://blogs.msdn.com/mohammadakif/archive/2006/09/17/759414.aspx

Best regards,

Mohammad