Yasser Shohoud

Blog moved

What Exactly Is BizTalk?

I now work as a Technology Architect at the Microsoft Technology Center in Austin which is part of the Microsoft Sales org. In this role I help customers understand how to architect solutions to their business problems/needs using Microsoft technologies. As a result I'm learning a lot of technologies that, as a member of the WCF product team, I just wasn't focused on in the past few years. When I learn a new technology I like to start by coming up with (or learning) a short and accurate description of that technology's focus, i.e. what is it for? 

One of the technologies I'm (re)learning is BizTalk Server. So I looked around for a short and accurate description of what BizTalk is. I knew what it was, but I wanted a 2 - 3 sentence description. I found some interesting documents like the BizTalk Overview and David Chappell's excellent Understanding BizTalk Server 2006 (a must read). While those docs offered valuable information, I couldn't find the sentences I was looking for. This is probably due in part to my coming from a development background and wanting to relate things back to the developer world. So I came up with this short and, in my opinion, accurate description:

BizTalk is to business process automation what Visual Studio and the .NET Framework are to application development. It is a runtime and tools that enable you to automate and manage business processes more productively.

That's basically it. If I only get 10 seconds to explain what BizTalk is to a customer, the above description is what I'm most comfortable with. If I have more time, I would elaborate on that description as follows. 

The term "application development" is worth further exploration. Looking at the kinds of applications that people build with Visual Studio and the .NET Framework today, the vast majority are focused on specific tasks rather than end to end processes. Not to say that such applications are isolated, they're actually often connected to other apps or data sources via Web services or other means. But the focus of the app is to automate a particular task or set of related tasks and the connectivity serves that purpose. However the business reality is that businesses run on processes which span multiple tasks and of course people. So to automate a business process end-to-end you wind up wanting to connect some of the applications that were written to automate the tasks that make up that business process. As an example, consider the task of updating contact information in the company's address book for a newly hired employee. That employee may launch a .NET desktop app or go to an intranet Web site and enter they home address, cell phone number, emergency contact etc. Now zoom out and look at the big picture and what you see is an end-to-end on-boarding process that includes things like provisioning credentials, certificates, smart card, automatic deposit setup, hardware procurement, office furniture procurement, 401K account setup, setting up a mentor, and the variety of other things a new employee needs to get/set up. The address book update app automates one task in the end to end process but there's no app that automates the entire process.  BizTalk Server allows you to connect existing apps that automate the various tasks of this process and automate the tasks or parts that don't yet have automation to form a complete end-to-end process automation solution. It also gives you an operational or runtime view of that automation allowing you to see all the moving parts from one vantage point and start/stop/change/maintain them.

While it's certainly possible to build VS and .NET apps that automate end-to-end business processes, it's not the most productive use of the customer's/developer's time because they'd have to reinvent (or re-implement) several wheels in the process. It's a bit like building an document management Web site using nothing but ASP.NET: You can do it, but you can do it a lot better and faster if you used Windows SharePoint Services (WSS). The operations folks would also love you for it because of the management features they get out of the box with WSS.

Living In Austin

On Saturday 6/17 I boarded a flight from Seattle to Austin and I didn't have a return ticket. You see, for family reasons I moved to Austin and joined the Microsoft Technology Center. The goal of the MTC is to help customers map their business problems to solutions based on Microsoft products and services. MTC Architects work with customers in three primary ways: Strategy Briefings (1day), Architecture Design Sessions(2days), and Proof-of-Concept Workshop(2-3weeks). You can read more about these activities at the above link, but basically they are about giving the customer the big picture of Microsoft technologies that can help with their business problems, drilling deeper into customer needs and architecting a solution to meet those needs, and building a proof of concept of that solution with the customer's own developers.
Needless to say, this role is quite different from my previous role on the WCF product team and I'm excited about that change and looking forward to spending a lot more time with customers and getting first hand experience with the customer-facing side of Microsoft. It's only been a week but I'm already having a ton of fun. For my first two weeks I'm shadowing Christian Thilmany in a POC based on Windows Sharepoing Services and Microsoft Office Sharepoint Server. Both technologies and in beta and I have no prior hands-on experience with them so I'm learning and having fun doing it. In addition to the exciting work, people on the MTC team have been warmly welcoming me and offering assistance whenever I need it.
Besides the exciting new role and the great team, Austin has some bonus features. There are some really great schools here (we're looking to live in NW Austin around Canyon Vista Middle and Westwood High Schools), UT Austin is a high ranking university, and the Austin live music scene provides some unique entertainment opportunities. Yesterday I discovered another bonus feature: Twisty roads. Moving here from Seattle's East Side (foot hills of the Cascades), I was worried I won't find good places to ride my bike. So yesterday I went out exploring and I found several well paved, twisty roads that were nearly empty at 5PM on a Saturday. In fact I got tired and turned back before the road showed any signs of ending. Turns out because Austin is in the Texas Hill Country, there are plenty of hills, canyons, and lakes and and abundance of twisty roads. Couple that with the drier and warmer weather, and you're pretty close to sport bike rider's heaven! I'm already loving it here.

More on the WCF RSS Sample

Last week we put up a sample named WCF RSS Toolkit on wcf.netfx3.com and I posted a sample service using it here. Since I wrote most of the toolkit’s code I thought I’d talk a bit about its goals, design and the future. There were two primary goals for this sample:

1. To explore/prototype RSS/Atom support in WCF and get customer feedback
2. Provide a sample to show WCF developers how to expose a service as RSS and Atom over HTTP GET using WCF v1

Note the “and” not “or” in #2, meaning the same service implementation can be exposed as RSS, Atom, or any other format simultaneously. If you know WCF, you’ll know this means generating multiple wire formats (RSS, Atom, whatever) from the same contract which can be done with some extensibility. The key is to not rely on the contract projecting the desired info set, rather to transform whatever info set projected by the contract to the desired info set. That transformation would be different depending on the desired wire format.

 To make it easier for developers to emit RSS and Atom, I defined a DataContract with some of the commonly used RSS/Atom data and I defined a transform from that DataContract’s info set to RSS and another transform to Atom. All of this is pluggable/replaceable: You can use your own DataContract if you like, the transforms will transform what they recognize and copy the rest out as extension elements. You can also plug in your own transform to support new formats. Since it’s source, it’s also customizable so you can modify the DataContract and/or the transform if you don’t like how I make DataContract members to RSS/Atom elements.

 There was also the HTTP GET part of the equation. WCF’s HTTP transport supports GET out of the box but there are other places in the stack that need to be extended for the scenarios I had in mind. For example, dispatching to the right operation based on the path segment following the endpoint’s address in the request URL. Also, parsing the request query string and decoding parameters from it. And of course, supporting ETag and Last-Modified HTTP header. All of these are commonly needed in RSS/Atom scenarios and when we do build RSS/Atom support into WCF, we will have these features. For the sample however, the first two are included in the sample but the last one isn’t. The good news is that the WCF HTTP transport gives you full read/write access to HTTP headers via the Message properties HttpRequestMessageProperty and HttpResponseMessageProperty so you could build ETag support on top of the current WCF bits. In fact, one of the benefits of having the RSS toolkit sample source code is it gives you a starting point that you can take and modify with ETag support and other features you want. You can even publish you modified sample for the community to learn from and use.

WCF RSS Sample

Mark is looking for some sample code for the recently released WCF RSS Toolkit. I've put the readme up here. I won't repeat the whole readme here but here's a peek at the programming model. There are several shortcuts for feeds from databases and feeds using existing data types so you should really check out the readme.

An example service:

using System;
using System.ServiceModel;
using System.Collections.Generic;
using Microsoft.ServiceModel.Samples.Syndication;
namespace Microsoft.ServiceModel.Samples.Syndication
{

    [ServiceContract]
    public class ItemListService
    {
       [OperationContract]
       ContentFeed<FeedItem> GetItemList()
       {

            //instantiate the feed
            ContentFeed<FeedItem> feed = new ContentFeed<FeedItem>();

            //populate the list of feed items
            feed.FeedItems = GenerateItemList();

            //set some feed-level properties
            feed.FeedAuthor = wcfuser@contoso.com;
            feed.FeedTitle = "WCF Item List!";

            ...

            //return the feed
            return feed;       

        }

        ItemList<FeedItem> GenerateItemList()

        {

            ItemList<FeedItem> list = new ItemList<FeedItem>(); 

            FeedItem item1 = new FeedItem();
            item1.ItemTitle = "This is item 1";

            ...

            list.Add(item1);

            ...

            return list;

        }

    }

}

            host.Open();

            Console.WriteLine("Service started ...");
            Console.WriteLine("Press [Enter] to exit");
            Console.ReadLine();

            host.Close();

        }

    }

}

Enjoy!

WCF Custom Channel Ecosystem

Back in January, a few of us on the WCF team kicked off a side project dubbed "the channel ecosystem project". The goal is to create a community and an ecosystem around WCF custom channels. The first step was to make it easier for developers to build custom channels. To do this we decided to provide detailed documentation, samples, and a couple of tools. We've been creating this content over the last 3 - 4 months as a part-time effort next to our full-time jobs. Thanks to the hard work of many people, the content is now in a state where it can be shared with the world as a pre-release to solicit feedback. It's been published on windowscommunication.net. The samples are also linked from the samples gallery.
We want to hear your feedback on this content as well as what else you'd like to see to help you build custom channels. Send feedback to ces at microsoft.com.

It's here

Moving lots of data

Recently I've been getting lots of questions about moving large files (or lots of data) between a WCF service and client. The question comes in multiple forms, e.g.

How do I send a file that's many GB in size from the service to the client?
Should I use MTOM to send large files?
I have a huge object graph and I want to send it over the wire, how should I do it?

So let's explore the problems associated with moving lots of data and the solutions offered by WCF.
Problem: Bandwidth Utilization
Sending gigabytes of data means lots of bandwidth usage. While that may not be a problem if it happens infrequently on a 100 Mbit or 1 Gigabit LAN, it is definitely an issue when bandwidth is scarce and/or is being paid for. In interop scenarios, where messages are encoded as text XML, the encoding of binary data using Base64 exacerbates the problem because it inflates the size by 1/3. There are three solutions to this problem depending on your scenario. First, compression could really help especially if the data is text or gets encoded as text (using Base64). Compression/decompression can be implemented using a custom WCF encoder/decoder (and IIS 6 offers built-in response compression). Second, you can avoid using text encoding when interop is not required. WCF provides a binary encoding which is far more bandwidth-efficient than the text encoding especially when sending binary data. Third, if you want interop and you need to send large binary content, you can use MTOM which allows you to send the binary content outside of the SOAP envelope (as a separate part of a multi-part MIME message) without Base64 encoding it.

Problem: Memory Utilization
By default, WCF buffers messages to support protocols like WS-ReliableMessaging and WS-Security that require buffered messages. For extremely large messages this can lead to out-of-memory conditions especially on servers that try to send or receive multiple of those messages simultaneously. Fortunately, WCF supports streaming on HTTP, TCP, and Named Pipes allowing you to send infinitely large messages without hitting out-of-memory exceptions (actually, message size is constrained to Int64.MaxValue or 9,223,372,036,854,775,807 but hopefully that's infinite as far as your app is concerned).

Problem: Recovering from Failures
So what happens if half way through sending your 4GB stream the TCP connection fails? Well, your app must catch the exception and recover. If the other side has been processing the stream as it receives it (e.g. saving it to disk) you may be able to have the sending app coordinate with the receiving app to figure out what was the last received byte and restart from there.
Another alternative is to use Reliable Messaging to recover from connection failures. RM will detect the failure and automatically re-establish a connection and resend the failed message. The problem here is that when you're streaming, the entire stream is one message so resending means basically starting over. Another problem, is that RM requires buffering (so it can resend on failure) so it doesn't work with streaming! The solution is to use chunking. This is where the sending application divides up the 4GB file into say 1 million messages each 4KB in size and sends them in buffered mode. The receiving application reconsitutes the file by appending all 1 million messages to form the original 4GB file. Here, RM buffers only a few of the 4KB messages at a time. In case of a failure, RM will automatically resend the 1 or few failed messages. The chunking/dechunking functionality can be encapsulated in a general purpose chunking channel that lets the applications program against a Stream and handles the chunking under the covers. This is what my sample chunking channel from PDC does. Note that because of the chunking and the protocol overhead added by RM, the resulting throughput is expected to be lower than direct streaming. However, you are getting reliability which comes in handy when the connection fails especially when you're almost done sending the 4GB and don't want to start all over!

To summarize, my recommendations are:
1. Use binary encoding if you have WCF on both sides or MTOM if you need interop.
2. Consider compression if you are forced to use text encoding or if the data itself is highly compressible (e.g. large text data).
3. Use streaming, or if you want reliability, use chunking. One restriction to keep in mind is that to stream, there can be only one parameter (or return value, depending on which direction you're streaming) and it must be of a type that derives from System.IO.Stream.

Meet the Channel Model: ICommunicationObject

A Common a State Machine

public interface ICommunicationObject : IDisposable
{
    CommunicationState State { get; }

    event EventHandler Closed;
    event EventHandler Closing;
    event EventHandler Faulted;
    event EventHandler Opened;
    event EventHandler Opening;

    void Abort();

    void Close();
    void Close(TimeSpan timeout);
    IAsyncResult BeginClose(AsyncCallback callback, object state);
    IAsyncResult BeginClose(TimeSpan timeout, 
                            AsyncCallback callback, object state);
    void EndClose(IAsyncResult result);

    void Open();
    void Open(TimeSpan timeout);
    IAsyncResult BeginOpen(AsyncCallback callback, object state);
    IAsyncResult BeginOpen(TimeSpan timeout, 
                           AsyncCallback callback, object state);
    void EndOpen(IAsyncResult result);
}

public enum CommunicationState
{
    Created,
    Opening,
    Opened,
    Closing,
    Closed,
    Faulted
}

States and Transition
The figure below shows the ICommunicationObject states and state transitions. State transitions can be caused by calling one of the three methods: Abort, Open, or Close. They could also be caused by calling other implementation-specific methods. Transitioning to the Faulted state could happen as a result of errors while opening or after having opened the communication object. Every ICommunicationObject starts out in the Created state. In this state, an application can configure the object by setting its properties (e.g. setting the address of a channel listener by calling its SetUri method). Once an object is in a state other than Created, it is considered immutable. For example, calling SetUri on a channel listener in the Opened state should throw an InvalidOperationException.

WCF Architecture Overview

Looking for people who want to change the world

We have a few open positions on the Indigo (WCF) team for people who want to change the world. In this role, you would get the opportunity to really change how millions of developers write distributed applications (if you're a distributed app developer yourself, you know that's a big part of your world!). I'm not exaggerating here, we're not just talking tweaks or incremental improvements, I really mean significant change.
We're looking for program managers to help innovate, design, build and ship Indigo v1 and future versions/technologies in the distributed applications space. We have positions for both experienced folks and recent college grads. If you or someone you know is interested, just email me at yassers@microsoft.com.

Meet the WCF Channel Model – Part 2

Most WCF developers won’t interact directly with channels or have to think about them much. The Service Model, WCF’s programming model, provides abstractions that layer on top of channels to give a “method and typed parameters” programming model. However, many developers will want to write custom channels for the reasons I mentioned in my previous post. For those developers, it’s interesting to understand how the channels are used (indirectly) by a developer building a WCF service and/or client.

WCF services and clients are based on the concept of endpoints. An endpoint consists of an address, a binding and a contract. The address is the endpoint’s network location (the where). The binding specifies how the endpoint communicates with the world (the how) and the contract is the collection of operations the endpoint exposes (the what). For the purpose of this discussion, the binding is the most interesting of the three.

A service developer defines an endpoint by specifying its three components. For example:

public static void Main()
{
 ServiceHost host = new ServiceHost(typeof(StreamServer));

 host.AddEndpoint(
             typeof(IStreamService),
             new ChunkingTcpBinding(),
             new Uri("net.tcp://localhost/StreamService"));

 host.Open();

 Console.WriteLine("stream server started....");
 Console.ReadLine();
}

On the send side (e.g. on the client side), the Service Model calls BuildChannelFactory to get back a ChannelFactory. A ChannelFactory is used to build a channel that can be used for sending messages. Here is this part visually:

You may be wondering why there are two factory abstractions: binding element and channelfactory/channellistener. Turns out there are different two categories of things one may want to do in a factory class that creates channels and those two categories are best separated in two factories. The categories are: programming model-specific work and channel-specific work. Programming model-specific work that belongs in binding elements includes things like varying behavior based on settings on the endpoint’s contract or implementation and reading from config. Channel-specific work that belongs in channel factories/listeners includes things like pooling resources across channels (e.g. network connections).

The next interesting thing to talk about is how channel factories create channels and the concept of channel shapes. That will be the subject of my next posting.

Meet the WCF Channel Model - Part 1

For the past 5 months or so, I’ve been working on the WCF Core Communications team (with Kenny, Matt, and Steve) where we focus on WCF transports and hosting features. In this period, I’ve learned a lot about the WCF channel model and I’m still learning a lot. I want to share some of what I’ve learned with you on this blog. So here’s a first of what I hope to be many posts on this topic. Kenny also has several posts about this subject on his blog.

The WCF runtime consists of two layers: The channel stack and the dispatcher/proxy. The channel stack is responsible for sending/receiving messages between the sender and receiver while the dispatcher/proxy layer is responsible for translating those messages to/from service method calls.

The channel stack is where WCF gets a lot of its power and versatility. It’s similar to the traditional OSI model in that there are cleanly separated layers with each layer using the services of, and only calling on, the next layer down. It’s also very different from the OSI model in that there aren’t n well defined layers each with their specific role. Instead, there are two types of channels with implementations of these types allowed to compose in different ways depending on the application’s needs.

At the bottom of the stack is a transport adapter channel (aka a transport) whose role is to adapt an underlying transport to WCF’s channel model. For example, WCF ships with HTTP, TCP, Named Pipes, Peer to Peer, and MSMQ transport adapters allowing you to use any of these technologies as a means of transporting messages between sender and receiver. Each transport adapter can use an encoder that transforms WCF messages to the appropriate format for the transport. WCF ships with three encoders: text XML, binary XML, and MTOM. You can mix and match transport adapters and encoders unless the transport adapter has some restriction on the type of encoder it needs.

Above the transport adapter is zero or more protocol channels. These are channels that perform some kind of message transformation as a means of communicating with the equivalent protocol channel on the other side. For example, WCF ships with security, reliable messaging, and transaction flow channels. Each of these channels add/read headers to/from the message and some of them modify message content and even send their own messages (reliable messaging sends ACK messages).

One of the key differences between WCF and existing distributed app technologies like ASMX or ES, is this ability to use any transport (through transport adapters) and compose any number of protocol implementations within the communications stack. .NET Remoting has similar features (extensibility allowing you to plug in custom transport channels and chains of message sinks that could act as protocols) but WCF takes this further by providing a richer channel model that captures the fundamental communication patterns (input, output, duplex, request/reply).

So why is all this flexibility and extensibility good? Those of you who know me will know I’m a pragmatic guy so I would not say something is good just because it’s cool technology. WCF is the next generation distributed applications framework. It will be used for a long time and for many diverse scenarios many of which we don’t even know yet. Product extensibility is essential in this case to satisfy such diverse scenarios. For example WCF/WSE 3.0 interop over HTTP is supported by WCF out of the box but there are some scenarios where customers may want to interoperate over TCP.  In our PDC talk, Kenny showed a custom TCP transport adapter that allows Indigo to talk to WSE 3.0 over TCP. The fact that it was possible, and relatively easy (OK, Kenny owns our TCP transport adapter so he knows this stuff inside out) to enable such a scenario speaks volumes for WCF’s channel extensibility. Another powerful scenario is when a customer wants to use WCF to communicate with a existing system that exposes its functions only through a proprietary network API. The customer can write a custom transport adapter that adapts the proprietary network API to WCF’s channel model allowing WCF developers to access the existing system without having to learn a new API.

These rich scenarios are what excite me the most about WCF’s channel extensibility. As more customers start playing with WCF, I think we’ll see a proliferation of custom channels both from ISV’s as well as within the enterprise.

Emacs for 2007

Don Box, Chris Anderson, Anders Hejlsberg and Scott Guthrie did a 40 minute demo in Jim Allchin's keynote this morning. The demo started showing off the new Language Integrated Query or LINQ technology. Anders and Don wrote an app to query the list of running process with more than 4MB of working set. They then joined that result with a list of process descriptions from a SQL database. The description for devenv.exe was "Emacs for 2007".
Next, Don and Chris wrote an Indigo service to expose this list of processes to other apps. Then Scott got up and talked about Atlas and showed how easy it is to write an interactive Web page that gets data from an Indigo service. I think there were two key points in this part of the demo that may have been lost:

1. The Web page had a text box and a "Get data" button. Using traditional ASP.NET, clicking on the button would submit a form to the server where the request would be processed (possibly invoking an Indigo service) then the response would be returned as an entire new page to the browser. In this demo using Atlas, client-side javascript sends a request to the Indigo service directly and displays the returned data without reloading the entire page.
2. The format used for communicating between browser and Indigo service is a simple text format known as JavaScript Object Notation or JSON. This means no XML parser is required on the client side. It also means that Indigo is versatile enough to speak HTTP GET and extensible enough to understand non-XML formats.

Just to reiterate, the Web client Scott showed used just JavaScript and a simple text format to talk to Indigo. No SOAP and no XML. This client also works on non IE browsers and was shown running on the Mac in the keynote.
We've done some work on Indigo to support HTTPisms like query string and various HTTP verbs (e.g. GET). This feature (and the Indigo JASON encoder that Steve Maine wrote) is the corner stone of enabling Atlas to communicate with Indigo services. The Indigo PDC bits have this support built in so once you get your hands on them, you can try that feature out and send us feedback.

Bizarre afternoon in L.A.

This morning, I flew out to LA for PDC. The plane landed around 12:15PM, nice and early. The ride to the hotel was mostly uneventful until we got to downtown and I noticed all the traffic lights were flashing red and drivers were confused at every major intersection. After dodging collisions a couple of times, the driver dropped me off at the Westin. I go to check in and they tell me I can't because all of downtown LA has no power. I'm thinking "you've got to be kidding! Does this really happen?". So I go leave my bags with the bell captain and head over to the restaurant for a nice cold salad (no power means no hot food).
About an hour later, power returns and I go to check in. I run into Steve Swartz standing in the check in line and we chat about CommunicationObject. When I finally get to the counter, the lady informs me I was supposed to have arrived yesterday. Apparently, the reservation says I arrive on Sunday and leave the following Sunday but I actually arrived today (Monday) and leave on Friday. I must have entered the wrong dates when I made the reservation. Anyway, there are only smoking rooms now so I settle for one and go up planning to work on my demo.
I set the laptop down and pull the adapter out of the bag only to find it's missing the tip that connects into the laptop is missing (it's a universal adapter with interchangeable tips). I call the concierge and ask for a nearby electronics store she tells me there's one across the street. Great, things are starting to look up. So I walk down to California Computer Center. The helpful clerk there shows me three different Targa adapters only one of which has my laptop listed on the back (I have a Compaq Evo N800w). The clerk says "be sure you get the right one because we don't return them." So I buy it (relying on the list on its back) and walk back to the hotel room. I plug it in, it lights up but my laptop gets no power. I go online and lookup the number for Targa, I call them and they say it looks like a faulty unit so I should exchange it. So I walk back downstairs to CCC and ask for an exchange. They're out. So I argue with them for 15 minutes until I convince them to refund my money. I ask for the nearest Radio Shack which turns out to be 3 blocks away. I walk down there and find they have only one adapter left which happens to work with my laptop. I'm back in business!
Next, came my adventure in getting a Starbucks coffee. Never mind.
Well, despite the rocky start, I'm psyched about PDC and the talk that Kenny and I are giving about writing custom channels for WCF (a.k.a Indigo). I plan to hang out in the Communications Track lounge so stop by and say hi.