BizTalk Adapter Development

Announcing the Biztalk Adapter Pack Poster

 http://msdn.microsoft.com/en-us/biztalk/bb545017.aspx, as well as part of the MSDN library at

http://msdn.microsoft.com/en-us/library/cc627292.aspx.   

Here is a summary of its content:

This paper describes the use of the BizTalk WCF adapters for hosting and consuming WCF services. As a key objective, it provides an in-depth understanding of the means of integrating a custom WCF channel stack with the BizTalk WCF adapters. Furthermore, key industrial infrastructure concerns such as transaction management and security are discussed in the context of integrating WCF services with the BizTalk Messaging Engine. The paper also emphasizes a few pragmatic paradigms such as the use of dynamic ports in invoking WCF services. Finally, some of the key integration challenges are discussed to streamline the correct use of the adapters for solving complex business problems.

With the realization of the benefits of developing and orchestrating unified business solutions, today’s business organizations adopt the paradigm of Service Oriented Architecture (SOA)-based solution development for the execution of dynamically composed business processes. However, due to the disparate nature of the diverse business applications that exist in an enterprise, many organizations experience daunting challenges in integrating these business applications and services.

This paper discusses the use of Microsoft® BizTalk® Server 2006 R2 and its WCF adapters as a platform for service orientation that alleviates the aforementioned integration challenges. The paper emphasizes the configuration and integration of custom WCF channels with BizTalk solutions, through the use of the above adapter platform. Furthermore, the paper provides concrete solutions for the identified integration concerns that apply to the orchestration of services through the use of custom WCF channels.

I have recently posted three WCF Adapter samples off the BizTalk Server Developer Center at http://msdn2.microsoft.com/en-us/biztalk/bb608378.aspx, Two samples were released today. The first one is of a WCF message intemediary where an incoming WCF message to BizTalk is dynamically channeled to the correct version of the the WCF service. The second one is shows custom message interception during BizTalk Server's binding processing of an outgoing WCF message. The third sample was published in January and is an example of using pure WCF messaging with the WCF adapters.

Hi

Is there a specific WCF adapter sample or whitepaper you feel you would like to see written? If so, please email me directly and let's discuss to see if I can get what is needed any WCF adapter content.Please put on the subject line "WCF Adapter Documentation Recommendation" ,and send to mckeown@microsoft.com. Thank you.

Believe me, the adapter is telling the truth! It really doesn’t understand the action it got! To understand what’s going on, read on…

When you use the ‘Consume Adapter Service’ wizard to generate LOB schemas in a BizTalk project, you also get a binding file which you can import (using the BizTalk Administration Console) to create the corresponding physical WCF-Custom port (Send or Receive Port, based on whether the contract was Outbound or Inbound respectively) when deploying your BizTalk Application. While this does save you the pain of creating and configuring the port manually, there is a ‘not-so-obvious’ caveat which you need to be aware of. (This applies only to send ports i.e. Outbound Contracts)

The WCF-Custom Adapter allows the SOAP Action to be specified in the following two ways on the static send ports:-

1. As a single line of text – the port is used for a single intended operation (say ‘ABC Action’).
2. As an xml, mapping each of a set of ‘Operations’ to the corresponding SOAP Action. This xml looks something like this:- (namespaces omitted for clarity)
   
   <BtsActionMapping>
               <Operation Name =”ABC” Action = “ABC Action” \>
               <Operation Name = “XYZ” Action = “XYZ Action” \>
   </BtsActionMapping>

The binding file created by the ‘Consume Adapter Service’ wizard chooses the second method to specify the action, even if you generate schemas for a single operation.

Now the question is, when a message comes to a port using such a mapping, how does it know what the intended operation was? Or more specifically, how does it know which action to use for which incoming message?

To make this decision, the WCF-Custom adapter compares the message’s BTS.Operation context property with the list of operations in the map (attribute ‘Name’). If a match is found, the corresponding action is used.

However, if the WCF-Custom adapter fails to resolve the operation name from the map, it assumes that the entire thing specified is a single line action (as in 1 above) and sends the whole string to the target adapter binding.

The most common scenario this could happen is when you create an orchestration and forget to change the operation names on the logical ports to appropriate names. By default, when you create a logial port in an orchestration, it’d be created with a single operation called ‘Operation_1’. The next operation you add (either on the same port or while creating another port) would be called ‘Operation_2’ and so on. You need to modify these default names to the appropriate names as the BTS.Operation context property derives it’s value from the operation name on the logical orchestration port which published the message to the message box. (That also means if you are not using orchestration and still want to use the Action-Mapping feature, you’ll probably need to write a custom pipeline component to populate this context property).

So if you are seeing the error mentioned in the title, what is possibly happening is this:-

Your FOO Adapter supports two operations with the corresponding actions being ‘ABC Action’ and ‘XYZ Action’. You generate the schemas and the binding file, create an orchestration (but forget to change the default operation names), tie the message types, deploy the project and import the binding file.

At runtime when you send a message, because you did not change the operation name on the logical port, the BTS.Operation property is set to ‘Operation_1’. The WCF-Custom adapter tries to look up this name in the map you specified on the send port (by importing the binding file). Since it does not find any operation by the name ‘Operation_1’, the WCF-Custom adapter believes that whatever you specified on the port is a single line action and passes the same to the FOO adapter.

So, instead of getting “ABC Action” or “XYZ Action”, which it understands, the FOO adapter gets “<BtsActionMapping><Operation Name =”ABC” Action = “ABC Action” \><Operation Name = “XYZ” Action = “XYZ Action” \></BtsActionMapping>”.

Clearly, it doesn’t understand that action!

Ok, so what do I do to fix this?

 In the above example, you should change the operation names on the logical ports in the orchestration to ‘ABC’ and ‘XYZ’ instead of the default ‘Operation_1’, ‘Operation_2’ etc. (Of course, you can also do it the other way round – change the mapping on the port to use names in the orchestration or, if there is just one action to use, specify the action directly on the port instead of using the action map.)
Basically, you need to make sure that the WCF-Adapter is able to resolve the operation name on the message to use the correct action, as explained above.

There are two ways the adapters belonging to Adapter Pack can be used by SQL Service Integration Services:

1. Using ADO layer built on top of base adapters. This option is available with SAP and Siebel adapters.

2. Writing custom SSIS task which directly invokes adapter to communicate with LOB system.

Option 1 covers all scenarios where data need to be moved from LOB system into SQL server. The reason being that ADO layer exposes only Query and Execute interfaces. Using these ADO interfaces, we can only pull out data from LOB system.

If usage scenario is such that data has to be loaded into LOB system, then option 1 doesn't help. But that is the not end of story. We can write custom SSIS task. This custom task essentially wraps the adapter as destination component. Once adapter is exposed as SSIS component, it is available for use in any SSIS scenario.

Q. It seems WCF LOB adapters works different from normal WCF service as there is no WCF Service hosted. Do we only use channelfactory to create custom channel for send wcf message to LOB without channellisterner? I just wonder how WCF LOB adapter run time works behind scene.

WCF Service is hosted using a class called ServiceHost.  The ServiceHost encapsulates the functionality regarding bindings, channels, dispatchers and listeners. 

WCF LOB Adapter is surfaced to the consumer as a WCF Binding, where the binding is used to create the channel stack. This binding can be considered a peer to other predefined WCF bindings such as BasicHttpBinding, WsHttpBinding, NetTcpBinding, etc and can be set via app.config or in code by the client application when calling a service.  This binding contains an ordered set of binding elements, with adapter being the key binding element that derives from TransportBindingElement class.  In an outbound scenario, the WCF LOB Adapter SDK runtime uses channel factory to create the adapter (i.e. transport channel).  In an inbound scenario, the WCF LOB Adapter SDK runtime uses channel listeners that listen for incoming channels in a service application. 

When a .NET application set the endpoint information with the WCF LOB Adapter binding, WCF under-the-covers calls

• BindingElement::BuildChannelFactory, if the application uses the adapter in client endpoint or
• BindingEement::BuildChannelListener, if the application uses the adapter in service endpoint.

When an application uses the WCF LOB Adapter in an outbound scenario, the WCF LOB Adapter SDK runtime interprets the BindingElement::BuildChannelFactory to build a Channel Factory for a specific type of channel.  When ChannelFactory::OnCreateChannel call is made by WCF, WCF LOB Adapter SDK runtime performs some of the following steps –

• Checks the type of channel (IOutputChannel, IOutputSesionChannel, IRequestChannel, IRequestSessionChannel) and creates appropriate adapter send channel
• Create connection pool
• Create metadata cache

Then, on channel’s OnOpen() method, WCF LOB Adapter SDK runtime creates the connection (or uses the connection from pool), builds appropriate outbound handler that implements interface IOutboundHandler and/or IAsyncOutboundHandler and call the Execute/BeginExecute/EndExecute method for message exchange.

Similarly, when an application uses the WCF LOB Adapter in an inbound scenario, the WCF LOB Adapter SDK interprets the BindingElement::BuildChannelListner to accept channels of a specific type from the binding (i.e. the adapter).  The adapter checks for the type of channel (IInputChannel, IInputSessionChannel, IReplyChannel, IReplySessionChannel) and creates appropriate adapter receive channel.   Then on channel’s OnOpen() method, WCF LOB Adapter SDK retrieves the connection from the pool, builds appropriate inbound handler that implements IInboundHandler and/or IAsyncInboundHandler, and invoke the TryReceive()/WaitForMessage() methods to map the lob-specific method calls to WCF messages.   The listening application/service then uses this channel to receive incoming messages.

Check the EchoAdapterSample's TestEchoAdapter_Channel (outbound) and TestEchoAdapter_Channel_Inbound (inbound) projects to see how the channel model is used directly to communicate with the adapter.  Then see how the WCF Service Model sample consumer projects TestEchoAdapter_Proxy (outbound) and TestEchoAdapter_Service_Inbound (inbound) hide the channel related details from the end-user.  The EchoAdapterSample is available from the WCF LOB Adapter SDK install directory under \Documents\Samples.

WCF LOB Adapter SDK is now generally available for download . . .

We are pleased to announce the release of WCF LOB Adapter SDK. 

Here are some of the related links:

• WCF LOB Adapter SDK Technology Site
• WCF LOB Adapter SDK Download Site
• WCF LOB Adapter SDK MSDN Forum
• BizTalk Server Roadmap
• Building Custom WCF Channels

While the team is working on providing the official documentation, see some of the following posts to learn more about WCF LOB Adapter SDK.

• About WCF LOB Adapter SDK
• Basic Q & A
• Adapter usage patterns
• WCF LOB Adapter SDK and its relationship with WCF Channel Model
• Using adapter code generation wizard
• Consuming WCF LOB Adapter in BizTalk Server 2006 R2
• Consuming WCF LOB Adapter in custom .NET applications
• How to browse/search metadata in WCF LOB Adapter
• How to generate WSDL from WCF LOB Adapter

The WCF LOB Adapter SDK ships with two samples - EchoAdapterSample and ContosoAdapterSample. The samples are installed as part of the setup in the {install_location}\WCF LOB Adapter SDK\Documents\Samples.

Download WCF LOB Adapter SDK

In this post, I will show how WCF LOB Adapter SDK relates to WCF Service Model and WCF Channel Model. 

WCF Service Model provides an intuitive programming model to create service-oriented interfaces.   There is plethora of information available on WCF (aka Indigo) now since its inception few years ago.   Typically when you are reading about WCF, the most common usage is using classes/interfaces/attributes in System.ServiceModel namespace.

WCF Channel Model can be used for development of custom protocol and transport channels to communicate with existing systems that expose its functions through proprietary network API.   The channels adapt the proprietary network API to WCF. 

WCF LOB Adapter SDK utilizes the extensibility features of WCF Channel Model and surfaces an adapter to target line-of-business system as a WCF Binding.

You can write a custom channel using WCF LOB Adapter SDK and using WCF Channel Model.   Here are some of the resources you can use when developing a custom channel (aka creating an adapter) using WCF Channel Model:

1)      Channel Model Overview

2)      Developing Channels

3)      Nicholas Allen’s Blog (an excellent resource)

4)      Building Custom Channels

So if you can create a WCF binding using WCF Channel Model, why WCF LOB Adapter SDK? Whatever you can do with WCF LOB Adapter SDK, it is definitely doable with WCF Channel Model. 

WCF Channel Model fits in very well for creating transport adapters.  Here is an excerpt from Building Custom Channels:

Custom channels are a powerful and versatile extensibility point. You'd write a custom protocol channel if you want to implement a wire protocol. Examples of wire protocols include the WS-ReliableMessaging protocol (WCF already implements that one for you as a channel) and a chunking protocol (see the chunking channel sample below). You'd write a custom transport if you want to send/receive WCF messages over a transport other than the ones we ship in the box (we ship TCP, HTTP, Named Pipes, MSMQ, and PeerChannel). For example, you could write a UDP transport (there's a sample of that in the WCF SDK) or an SMTP transport. Another reason to write a WCF transport is to integrate with an existing system that's not exposed as a Web service. In this case your transport acts as an adapter adapting WCF messages to the existing system's message format or API allowing a WCF client to talk directly to the existing system. An example of that is the WSE 3.0 TCP transport sample.

If you are writing a pure transport and/or protocol channel, Microsoft recommends utilizing WCF Channel Model. WCF LOB Adapter SDK is an application of WCF Channel Model and provides following additional features more suitable for systems that can be described using rich metadata.   

1)      Code Generation Wizard

It facilitates creation of WCF channels and bindings through Adapter Development Code Generation Wizard.  The adapter developer only focuses on three key aspects of building an adapter – connectivity, metadata and message exchange.  The channel concepts like message exchange patterns, channel factory, channel listener, binding element, binding, WSDL export/import and Policy export/import are not exposed to the adapter developer. 

The adapter developer focuses on mapping of the message from/to their line-of-business messages in inbound and outbound handlers.

2)      Metadata Explorer

It provides an ability to generate WSDL / WCF Contract dynamically at design-time.  Let’s say you are creating a database adapter with each stored procedure exposed as an operation.  The number of stored procedures in the database can vary from implementation to implementation and the parameters/return values can change.  Such functionality shouldn’t be captured in a static contract.  WCF LOB Adapter SDK promotes inspecting the live line-of-business system for its metadata and allows generation of contract from this live metadata.  Even though you can provide metadata support for a WCF binding element in WCF Channel Model using IWsdlExportExtension.ExportContract, WCF LOB Adapter SDK provides the API to enable browsing and searching of metadata.  And … when the message flows over the channel during run-time, the code has knowledge of metadata of the message (via SOAP action) and can use it during construction, transformation and verification.   

The adapter developer can provide the metadata exploring and export functionality through metadata browse handler, metadata search handler and metadata resolver handler.

3)       Connection Pooling

Typically when a line-of-business adapter is created for a database, messaging system and/or packaged application, there is a concept of a connection.  WCF LOB Adapter SDK provides an efficient way of using these connections through a pool of connections.  Using the WCF Channel Model, the adapter developer can provide own implementation of connection pooling.

The adapter developer can tweak the adapter settings for connection pool and connection pool manager.

4)      Adapter Data Reader / Adapter Data Writer

Rather than having to use XmlDictionaryReader / XmlDictionaryWriter to walk through the WCF Message, since the adapter has the knowledge of the metadata of the message, the adapter operation and type readers/writers can get callbacks (e.g. OnAfterOperationParameterWrite) in which the adapter developer can then do line-of-business system specific mapping and transformation.

The adapter developer can use these readers and writers within the message exchange handlers.

* If the adapter developer is using WCF Channel Model to create the line-of-business system adapter, then the N/A items can be supplemented with developer's own implementation.

Enterprise Application Integration is the secure and orchestrated sharing of processes and/or data between applications which weren’t originally designed for such sharing. To enable such applications to participate in integration processes one has typically had to write an integration adapter or write a web service wrapper around the existing line-of-business system APIs to enable them to be “normalized” for consumption by integration brokers such as BizTalk Server.   Adapters traditionally have achieved the normalization by reducing the target systems to appear like “messaging systems” and capture the LOB specific aspects in the actual message that is transferred to the backend systems. While this approach is simple it is practically unusable in regular applications where typically the levels of abstraction are much higher and semantically richer than just exchanging messages.  Web services on the other hand offer a much more “human friendly” programming model and the appeal of wide reach and support for several popular WS-* standard protocols. But web services today require a static monolithic service contract declared a priori describing a strongly typed service interface that is the public face of the service.

For LOB systems which have an enormous amount of operations that are constantly changing and evolving it is impractical for such a contract to capture the capabilities of the system with 100% fidelity.  One of the more common ways of exposing such a dynamic system to EAI systems is to have a generic InvokeWebMethod() operation exposed as the extensibility mechanism to deal with any new or changing metadata.  Here is an example of one such implementation.  So the client of the system will invoke the newer methods by passing in the appropriate method name, parameters etc packaged in generic array data structures to be “interpreted” by the InvokeWebMethod() function and turn around and make the actual call.  Although this provides the “dynamic” functionality in being able to call any method, it expects the client to be fully aware of the details of the new methods on the LOB system. This is a fundamental violation of the SOA tenets where the boundaries between client and service are expected to be strictly limited to what is exposed in the service contract and nothing more. To expect the client to be deeply aware of the internals of the LOB system leads to tight coupling between service and client and this is total anathema in the SOA world.

What we need in this situation is an intermediary sitting in between the LOB system and the consuming client that lets the client “carve out and construct” a strongly typed service contract dynamically from the live metadata in the LOB system. The WCF LOB Adapter SDK enables the development of adapters which act as an “uber service” that can incarnate any service from the available backend metadata.  This allows the clients to keep up with the changing functionality in the LOB system at design-time without being intimately aware of these changes.  If the target system changes, the client can easily regenerate updated proxy and tremendously increase the develop-test-deploy web service cycle.  This approach also enables a more consumer- friendly experience for exposure of LOB systems with large and unwieldy metadata although the metadata may remain more or less static.

Additionally, adapters written to the WCF LOB Adapter SDK eventually manifest themselves as WCF bindings which can be invoked via WCF Service Programming Model and WCF Channel Programming Model (via the client proxy generated from one such composed contract) thereby combining the advantages of both traditional adapters as well as web services.

Summary

WCF Service Model is the next-generation distributed application technology for developing and consuming services in Windows. The client and service communicate with each other using endpoints.

WCF Channel Model provides extensibility by enabling creation of custom protocol and transport channels for a binding in an endpoint.

WCF LOB Adapter SDK is a utility of WCF Channel Model to surface target line-of-business systems as WCF Bindings to the WCF clients.  This means a client can access an existing line-of-business system the same way as they would any other WCF Service.   The client is abstracted from having to learn the proprietary access mechanism of the target line-of-business system.

An adapter built using WCF LOB Adapter SDK can be used in variety of topologies and scenarios.  Follow your organization’s n-tier architecture principles and design patterns to determine where the adapter can fit best for your requirements.

Usage Pattern 1 – Consume adapter in a .NET managed application (Point-to-Point)

Design Time: In Visual Studio 2005, use in project templates such as Windows Application, Class Library and Console Application. 

1.1 Outbound

At design-time, use Add Adapter Service Reference Visual Studio Plug-In to generate the WCF Client Proxy for use in the .NET application.

1.2 Inbound

The Microsoft BizTalk Server 2006 R2 will ship with a BizTalk WCF Adapter.  The BizTalk WCF Adapter is a bridge between BizTalk and WCF.

It supports features such as: