Esperpento

PDC05

The other day I mentioned some of the issues surrounding the appropriate scenarios for atomic transactions deployment. I also mentioned the compensating model for long-running activities. Choosing between these models inside distributed applications is a common dichotomy when building real-life systems.

Atomic transactions are a fundamental building block solution for a vast number of software problems. It is difficult to overstate the benefits provided to developers by the atomic transaction model, in terms of simplifying the reliability, concurrency and error handling of a transactional program. However, atomic transactions require strong assurances of trust and relative liveness between all participants. In scenarios where these assurances are not possible, a more loosely-coupled model must be used.

An example of such a scenario is the painfully real-life sequence of events and decisions with which anyone building a business application backend will be familiar. A business may wait for supplies ordered from different vendors, require human input to make decisions, await confirmations from customers, tolerate partial failures, resubmit orders, offer customers alternative products, etc. When such a sequence is cancelled, compensating actions are applied to undo every action that was taken as part of the sequence of events leading up to the cancellation. Both actions and compensating actions are likely to use atomic transactions in their internal implementations, but that is a detail at a lower level of granularity than the high-level business process.

While the end result of a compensating workflow may be a concrete outcome or result, it would be limiting to label the entire set of states, events and decisions a “transaction”. In fact, the use of that word in this context is so misleading that you should immediately slap anyone who uses “transaction” to refer to a long-running activity. An atomic transaction is an aggregate set of operations that either complete or fail in unison, usually in timeframes approaching a few milliseconds. Furthermore, an aborted transaction leaves no changes in its wake: the final state of the world is necessarily identical to the initial state. On the other hand, a cancelled workflow activity is likely to leave the world state in the most equivalent state possible to the initial state. The quality of that equivalence will depend on the nature of the actions and compensating actions taken during the workflow activity.

The greater complexity of the compensating model leads to an interesting conclusion. From the user’s perspective, an infrastructure that provides support for atomic transactions can provide a highly transparent interface. However, an infrastructure that provides support for compensating activities has no obvious transparent interface.

A good example of the former is the new System.Transactions namespace in .NET 2.0, which provides a simple and lightweight veneer on top of what is actually a highly complex system for propagating and coordinating distributed transactions. All of the complexity is buried underneath the veneer: in a very real sense, we implemented System.Transactions and MSDTC so that you didn’t have to.

On the other hand, a framework for long-running compensating activities is likely to be an exercise in delegation to user code: both the logic that makes forward progress and the logic that performs compensating operations fall within the domain of the user’s business needs. Consequently, a compensating infrastructure is far less able to isolate a user from his own system’s complexity.

With that said, the workflow modeling environment provided by a sister component of Indigo/WCF, the Windows Workflow Foundation, provides an excellent basis for designing applications that use the compensating model for long-running activities. The combination of WCF for distributed messaging and WWF for state machine and message exchange pattern (MEP) design, as well as general implementation modeling, is extraordinarily powerful.

As WCF and WWF evolve towards their eventual release next year, you should expect to see guidelines and best practices from us on how to build distributed compensating workflows using these components.

PDC05

I should point out that I enjoyed last night’s Birds of a Feather session. It was a lively and spirited discussion. I appreciated the opportunity both to correct some misconceptions and to hear some good feedback. Thank you for that, William.

A common theme that has come up repeatedly here at the PDC: there is some confusion around for WS-AtomicTransactions (WS-AT) and how it composes with the general web services architecture.

The strawman syllogism goes something like this:

Premise 1: WS-AT is a specification for atomic transactions over web services.
Premise 2: Web services are for reaching across trust domains and doing business on the internet.
Conclusion: Microsoft wants us to share our atomic transactions across trust domains.
Corollary: Microsoft is crazy!

This is not an accurate reflection of what we’re accomplishing with WS-AT.

Atomic transactions are not a new invention. They have been described in academia and used in industry for over thirty years. Every transaction monitoring system worthy of the name has provided some form of atomic transactions. Some have extended the concept to implement distributed atomic transactions, taking variations of the two-phase commit protocol and mapping them to (generally proprietary) protocol transports. For example, Microsoft’s Distributed Transaction Coordinator (DTC) has provided support for distributed atomic transactions over an RPC protocol (Ole Transactions) for about ten years.

Atomic transactions exist to provide a solution to a common business problem: how to ensure the ACID properties (most notably atomicity) across multiple stores or resources that are used inside a single unit of work. The nature of the two-phase commit (2PC) agreement protocol leads to an obvious conclusion about the applicability of an atomic transaction as a business solution: there is both an implicit trust and an implicitly tight coupling between all actors in an atomic transaction. Superior transaction coordinators must reserve local resources (log space) to ensure atomicity for subordinates. Subordinate transaction coordinator must rely on their superiors to both provide them with the correct outcome and to do it in a timely fashion. All resource managers involved with an atomic transaction are likely to be holding locks (on what might be essential business data) until the transaction’s outcome is known. A rogue or malfunctioning actor can therefore deny service very effectively: this is a feature of the 2PC protocol, not a hole in any implementation of it.

In short: it is absolutely true that atomic transactions are best suited for tightly-coupled scenarios where trust between actors is a given.

The problem with the strawman syllogism is actually in its second premise: we do not recommend that customers use WS-AT across trust domains. WS-AT was not designed to bring atomic transactions into the web services world; instead, it was designed to bring web services into the glass house.

Let me be categorical about this: the general topologies where atomic transactions have been used in the past are the same topologies that are appropriate for WS-AT. In other words, WS-AT does not represent a radical change in the scenarios and uses of atomic transactions. Instead, it provides evolutionary improvements in the following areas:

1.     Atomic Transaction interoperability. With WS-AT, we have significantly raised the bar for functional interoperability across vendors, systems, languages, runtimes and technologies. While there have been some attempts to accomplish this in the past (e.g. the mostly-ignored TIP specification), none have enjoyed the industry support that WS-AT has today.

2.     Web service ubiquity. While web services have traditionally been targeted at crossing trust domains and firewalls, there are no conceptual obstacles to deploying web service technologies appropriately inside those trust domains. In fact, the very availability of WS-AT makes it easier to create internal solutions that aren’t restricted to a single vendor’s technology stack.

This leaves an open question: if atomic transactions are not intended for long-running activities between businesses, then what should be used instead? The traditional answer has been to use a compensating model, in which business resources are durably committed during execution of a long-running activity, then un-done or compensated if the activity is ultimately cancelled. This model scales to the business needs of long-running activities far better than atomic transactions, and is clearly more suited to web service scenarios that cross trust, ownership, reliability and timing boundaries. However, it is significantly more difficult to leverage than the atomic model (which can be made virtually transparent to the user).

I’ll talk about the long-running activity model some more next time.

Those of you who pay attention to these things might have noticed that we recently published new versions of the WS-Transaction family of specifications.  These versions are dated August 2005 and supersede the previous versions from October 2004.

You can find the new specifications on MSDN:

WS-Coordination: http://msdn.microsoft.com/ws/2005/08/ws-coordination/
WS-AtomicTransaction: http://msdn.microsoft.com/ws/2005/08/ws-atomictransaction/
WS-BusinessActivity: http://msdn.microsoft.com/ws/2005/08/ws-businessactivity/

So what's new in these specifications, you might ask? Great question. We made a number of changes, all for good:

Authorship

We've added some new co-proposers, including representatives from Hitachi, IONA and Arjuna. These companies have been involved in our WS-Transaction interoperability events for a long time and have made significant contributions in this space.

Fault handling

We addressed one of the more common interoperability problems by defining specific actions for WS-C and WS-AT fault messages, instead of leveraging the WS-Addressing fault action. This allows implementations to easily distinguish faults generated by the actual protocols from faults generated by the infrastructure or other binding-specific elements.

We also made the fault message schemas more precise and clarified the meaning of each standard fault code.

Security composition

We clarified the use of WS-Trust in the WS-C security model for authorizing participants. Instead of leaving the precise details of this composition to the implementation, we specified the use of the <wst:IssuedTokens> header to propagate security tokens alongside coordination contexts. This should greatly simplify the interoperability of the model significantly.

Policy composition

Last, but far from least, we removed the old (and rather obsolete) WS-Policy sections from all three specifications, introducing a new policy model for WS-AT and WS-BA. It is now possible to create policy documents for applications that specify in an interoperable manner the requirements of a given web service with respect to coordination context flow. For example, by using the new policy assertions, one can now write a WS-AT-aware service that uses WS-Policy to tell clients whether they MUST, MAY or SHOULD NOT flow WS-AT transactions to the service. This is an area in which proprietary mechanisms and frameworks have been used forever, so having this policy in place is a highly significant step for the WS-* model and for web service interoperability in general.

One interesting fact: while we corrected some minor problems with the 2004/10 schemas, we did not make any changes that were significant enough to merit a namespace change. Consequently, the good old http://schemas.xmlsoap.org/ws/2004/10/wscoor/ namespace and friends are still current. Our intention here was to preserve the interoperability gains we made in our January 2005 interoperability event, preserving the possibility of continuing to interoperate with implementations that targeted the previous version of the specifications.

And yes, in case you were wondering, Indigo Windows Communication Framework has supported WS-AT since its first public releases and Beta 2 will be no exception. We have spent a lot of time working on transaction flow functionality, security and performance in these last few months, and that includes a fully secure and interoperable implementation of WS-AT. I was tempted to post what would apparently be the first snippet of code + configuration demonstrating how to enable WS-AT transaction flow between services, but I think I should probably wait until our public disclosure at the PDC. Let me just say that switching to WS-AT as your transaction flow protocol is a single line of configuration. Enabling a service for interoperable WS-AT transaction flow is as simple as selecting an interoperable standard binding and turning on transaction flow with another line of configuration.

Did I mention that I love the transaction flow features in WCF Beta 2? I'm sure I have. One of these days, I'll spend some time talking about the most interesting parts of WS-C and AT and why I think they're so important.

Speaking of the PDC, if you're going to be there and you'd like to talk to me about WCF transactions, or if you have questions about (or plans for !) WS-Transaction implementations or interoperability, drop me an email and let me know that you'll be looking for me there.

I suspect that I have managed to set a record for the longest silence after an initial blog post. Except, that is, for all the people posted once and never came back.

Like all poor students, I do have some excuses. I was on vacation in Alaska, an utterly excellent sojourn which I may talk about sometime if there’s nothing else to write down. Suffice to say that if you have a chance to drive the Denali Highway, just do it. Never mind those car rental agencies and their pesky girly-man warnings. We had the time of our lives out there. Just make sure you slow down for other cars and respect the ptarmigans as they slowly moonwalk across the road.

My other excuse is that our team has been working like ants to meet the dates for our next milestone. It has not been a true death march, although there were some minor casualties. Our mission? To boldly make Indigo Beta 2 safe for human use. Obviously nothing can be said about this until the Los Angeles PDC, but even so, I should warn you: the transactions support is really good. Much better than in Beta 1, in the same way that Tom Yum is a much better soup than Caldo Verde.

Speaking of the PDC, I will be attending this year. Not as a talker like some of my more illustrious co-workers, but as a free-lancer eager to see what other Microsoft teams are working on and what our customers make of it all. Maybe I’ll even see you there.

Hello, world. Be kind. Be fair.

I should introduce myself, just to get that part out of the way.

My name is Max Feingold I am a development lead on Microsoft’s Transactions team, which is a core component of the Indigo team. I work mostly on transaction protocols, most notably on the WS-Coordination and WS-AtomicTransaction specifications. I am also involved in a number of Indigo features, including Indigo’s COM+ integration support, the transaction flow model, the transactional programming model and so forth.

As the last developer in building 42 to begin a blog, I’m not entitled to give you the long explanation of what Indigo is. The short explanation is this: Indigo is not love; it’s a healthy long distance relationship. It’s also an interoperable messaging system, an application server, an enterprise framework and (in a mystical sort of way) a reincarnation of COM+.

So what is it that I do at work? I used to debug and write code. I still do that, but these days find myself slipping more and more into various roles requiring more maturity and savoire faire: designer, enabler, driver, writer, negotiator, process manager, people manager, general sinkhole of responsibility... Life was simpler when I could just play on the monkey bars. These days I have to worry about helping those crazy kids when they hurt themselves.

Next time, I’ll talk about the things I worked on before Indigo came along.