PatHelland's WebLog : Architectural

The Emissary Design Pattern and RIAs (Rich Internet Applications)

PatHelland — Sun, 10 Aug 2008 18:28:00 GMT

Here is a first draft of a new presentation. I gave it a couple of months ago just after TechEd and thought I would share it as I try to write up some of my thoughts on RIAs. I plan to rework this a bit more and present it again at TechEd Europe. The talk is titled: "The Emissary Design Pattern and RIAs (Rich Internet Applications)"

Abstract:

The Emissary design pattern was first described in 1999 in the old "Fiefdoms and Emissaries" talk. The concept of a "fiefdom" is very similar to what we today call a service in a Service Oriented Architecture. The fiefdom is a separate trust sphere and transactional boundary. An emissary is a prescriptive pattern for interacting with a service (or fiefdom) which leverages reference data and a deep understanding of the service to prepare requests for service and maximize the chance those requests will comply with the requirements of the service. An emissary may be richly interactive and anticipate the validation requirements of the service.

The emerging world of RIAs (Rich Internet Applications) is a fascinating blend of a classic smart client and a browser-based web application. In a RIA app, client code runs in the browser but still must comply with the browser enforced sand-boxing and not cause harm to the host client machine. Navigation, naming, linking, and much more are being defined in a fashion drawing from both the web style and the client style. Many of the design issues with RIAs are under discussion today as this support for these applications is emerging.

This talk examines both the emissary design pattern and the nascent space of Rich Internet Applications. It motivates how one can look to the workflow patterns contained in our parents' use of paper forms for workflow to understand the possibilities of implementing user-centric workflow as shared replicated data. The talk concludes with some preliminary concepts of a shared and declarative definition of the "paper form" model and its constraints and how these may someday be used in the automatic generation of emissary-based RIA clients.

More RIA stuff soon!

- Pat

Controllers and Doers

PatHelland — Sun, 06 Jul 2008 22:05:00 GMT

First of all, I've put off writing this for a while since I wanted a term for "controllers" that does not feel pejorative. I really mean this characteristic in a healthy and natural fashion!

So, it's my belief that the people working in IT fall into two broad categories (with occasional cross-over and hybrids). Folks generally are either:

Controllers -- These people struggle every day with the overwhelming number of applications, data sets, schemas, projects, and the chaotic interaction between them. They strive to bring order to the chaos and rationalize how all this crap can work together. They are fervently backing the enterprise-wide schema rationalization and struggling to get the architecture team to be successful in its attempt to unify many different aspects of the application install base. Part of this effort is to unify the logging, monitoring, and management of all the applications within the enterprise. As a first step in that, they support the ongoing project to create an accurate inventory of the various applications within the enterprise. It is always a work in progress!

Doers -- These folks see an important business imperative that requires application support. They WANT to work within the framework advocated by the Controllers but it seems very challenging to meet their business constraints while dealing with all the overhead (and apparent confusion) of doing things compliant with the Controllers' constraints. Sometimes, they comply. Other times, they say "Screw It" and just do their application changes the way they want to do them.

So, this dichotomy seems stable just like the American Two-Party System of Politics seems to be stable. The Democrats and Republicans always seem to squabble and that clarifies the argument. Over time, the center viewpoint migrates (and American political views are surprisingly homogeneous -- just ask any non-American). The existence of the debate (and the tension it causes) drives the system towards a new balance over time. Similarly, the Controllers and the Doers move the center of gravity for the IT organization -- in both its centralized capacity and its per-department capacity. The proclivity of the Controllers adds some small sense of consistency (which is of tremendous value) while the business demands of the Doers keeps a spark under the IT organization's butt.

Some observations:

When I was a kid, the Controllers had great big mainframes and multi-year waiting lists for application development. The Doers bought mini-computers using their departmental budget and really annoyed the Controllers.
Later, the Controllers were using minicomputers (usually running Unix) and the Doers were building PC applications! Some of these were even built in Excel and met the business needs just fine (even if the data wasn't properly "Controlled" which did pose challenges).
Now, the PC is "The Man" and has moved into the mainstream of the world of the Controller. The Doers are using lots of different tools including mash-ups!

So, when I look around Microsoft and other vendors, I always ask myself "Is this product for the Controllers or for the Doers?" Different divisions and groups within the various vendors build stuff targeting the two constituencies. Neither is wrong and neither is right. The tension and the balance seems to be stable while the manifestations of each groups efforts evolves forward relentlessly. What a hoot!

- Pat

PDF version of the slides for "The Irresistible Forces Meet the Movable Objects"

PatHelland — Sat, 12 Jan 2008 04:04:00 GMT

Some friends of mine told me that they don't have Powerpoint 2007 (gasp!) and so it was impossible for them to see this talk.

The original blog entry (with PPTX file and abstract) is http://blogs.msdn.com/pathelland/archive/2007/11/25/the-irresistible-forces-meet-the-movable-objects-closing-general-session-at-teched-emea-in-barcelona.aspx.

I took the presentation and made some modifications to allow it to be legible as a PDF (some animations are spread over a few slides; one complex animation that can't animate as a PDF has a description attached).

Hope this makes this more accessible.

- Pat

Durability Is in the Eye of the Beholder

PatHelland — Thu, 27 Dec 2007 23:47:03 GMT

So, I occasionally ponder the D in ACID transactions and wonder what it REALLY means.

Observation #1: Committed Subject To...

When I was working at Tandem in the 1980s, we had a complex (and fascinating) multiprocessor system with dual-ported disk drives, dual-ported IO controllers, multiple (2 to 16) processors connected via a message-passing bus and it was fault tolerant. I'm very proud of that part of my career when I worked on TMF (Transaction Monitoring Facility) which implemented the transaction logging and recovery spread across the Tandem Computer's NonStop system. While working there, I first realized the spectrum of commitment that happens over time. When a transaction would commit on TMF, it would get progressively more committed over time:

Committed subject to App process failure. When the call from the application processor to the OS on the local processor (one of the 2 to 16 in the multi-processor) is made, the application process can then fail and the transaction will still commit.
Committed subject to processor failure. One of the first things TMF does when processing a commit is to send a packet to its next neighbor indicating the intention to commit the transaction in question. Once that packet is received by the neighboring processor, the failure of the application's original processor would no longer result in the transaction being aborted.
Maybe committed subject to system (multi-processor) failure. When the commit record is written to the tail of the log, the mirrored writes are (or were back in my day) always done sequentially to ensure that the rewrite of the last block did not lose the previous contents of the last block. So, you went from neither mirror having the commit record to one mirror having the commit record... we haven't yet written the second mirror. A crash at this point would result in an attempt to read the tail of the log after restart. It was the luck of the dice as to which mirror was read but whichever tail was read, that was rewritten onto the other mirror to ensure a consistent opinion of the tail of the log. Hence, we've entered a window in which the transaction has a 50% chance of being committed if the entire system crashes and restarts. Is this transaction durable???
Committed subject to system (multi-processor) failure. When the commit record is written to the tail of the log on the second mirror, you had a (close to) 100% chance of having the transaction remain committed in the face of a system crash and restart.
Committed subject to the destruction of the data-center. This would occur when the log was shipped offsite either via tape or by squirreling it across the network to a backup site.
Committed subject to thermo-nuclear exchange. Now, THIS one was a tough one. You have to send the backup tape to sit under some mountain to ensure the transaction is REALLY committed if this happens... I'm not sure we achieved this...

So, when is the transaction durable? What about a transaction whose commitment is recorded on multiple machines but only in volatile memory (but with enough of them to have as many nines availability as you want)? Hmmm... this durable stuff is annoying (just like all the letters in ACID).

Observation #2: Commit Dependencies and the Event Horizon

I am reminded of IMS (IBM's Information Management System) and its special version called IMS Fast Path. What I'm about to say may be apocryphal but this is how I think it works based on occasional conversations with transactional old-farts...

As I understand it, there was this cool optimization called "internal commit".

IMS worked as a database management system integrated with 3270 block mode terminals. Work happened in three steps:

The user filled out the screen and pushed "enter" which caused the image of the changed fields to be sent to a terminal controller. The terminal controller interacted with IMS to ensure the input image of the screen was transactionally recorded. Only after the input was logged would the work of the transaction against the database be performed.
The incoming data was dequeued, the work of the transaction was performed (which likely modified data in the database), and the output screen image was enqueued.
The output screen image was flashed up for the user to see and a transaction performed to delete the output screen.

This has the characteristic that it is IMPOSSIBLE to see the contents of the database other then by looking at the effects of a committed transaction. Unlike most DBMS systems today, you could not begin a transaction, examine record-X, and display the results without committing the transaction that LOOKED at Record-X. Just the act of reading required a commitment to see the effects outside the system.

So, now what the heck is an internal commit? The idea is to count on the single log buffer in this mainframe based system. If transaction-T1 is committed and its updates placed into the transaction log in memory (but not flushed to disk), and transaction-T2 comes along while the system is still running, T2 cannot commit unless T1 commits. T1 will commit if the system remains alive long enough to flush the transaction log to disk. T2 is running on the same system and can only commit using the SAME log and assuming the system stays up long enough for T2 to get its changes written to the log on disk. If T2 succeeds in doing that, T1 most definitely will have committed! This is called a commit dependency --> T2 has a commit dependency on T1.

Leveraging these two concepts (the commit dependency and the fact you can't see any effects outside the system other than through a transaction commit), IMS-FastPath would play this cute trick. Say transaction-T1 modifies record-X and is on its way to committing. Once the commit record is in the log buffer in memory, then Record-X could be unlocked. This would be heretical in most systems because a crash might cause the loss of the new value for Record-X (remember, transaction-T1 is not yet durable when it is unlocked). This worked without anomalies because of what I call the event horizon.

An event horizon (my terminology) refers to the ever increasing scope of knowledge and our ability to leverage knowledge with some assumptions about its propagation. It is OK for transaction-T1's changes to record-X to be unlocked because no one can tell the difference! If you see the new value for record-X, you fate is lashed to the success of transaction-T1. You have a commit dependency on transaction-T1 (either because of the design constraints defined above OR because transaction-T1 is actually committed and durable on disk).

So, for transaction-T2 which is looking at transaction-T1, it appears durable when the changes are in the buffer in main memory due to the event horizon effect. It's like that spy movie: "I could tell you but then I'd have to kill you..." The knowledge doesn't matter if all the impacts of its use are eliminated from the system.

Observation #3: Dialog Semantics and Visibility of Failures

I spent a couple of years working on a feature of SQL Server (shipped in MS SQL Server 2005) called SQL Service Broker. Service Broker defined a notion of a dialog which implements transactionally-consistent, exactly-once, in-order messaging between to "services" which are reified by their state in the database. The notion of dialogs was that the messages would be delivered transactionally, exact-once, in-the-order-sent, within a timeout window OR a dialog-failure was delivered to the service. SQL Service Broker ONLY provides services whose state was represented in a SQL database and, hence, counted on the durability guarantees of SQL Server.

As you try to understand the semantics of message delivery guarantees, it is essential to think about WHO is being provided with the guarantee and what THEIR durability is. The more we thought about this, the more it was clear that the dialog failed precisely when it LOOKED like it failed. Consider Service-A in a dialog with Service-B. The dialog has a timeout. If Service-A cannot receive a message before the timeout, the Service Broker must give it a Dialog-Failure message. The guarantee of delivery of this message is null and void if Service-A is not around to receive the dialog failure...

While in Service Broker within MS SQ Server 2005 only fully durable services are supported, it is meaningful to support more permutations. Consider Service-A and Service-B, each of which may be either durable or in-memory. The durable flavor means that receiving and/or sending a message occurs only when the change to the service happens on disk in the database (just like changing a record in SQL Server). The in-memory flavor means that the state is just kept in memory, a system failure wipes out the state.

Consider four permutations:

Durable-Service-A has a dialog with Durable-Service-B. In this case, we don't have to worry about one end surviving and the other not [well... more discussion below].
Durable-Service-A has a dialog with In-Memory-Service-B. Now, if Service-B crashes, the dialog will time-out, causing a dialog failure delivered to Service-A. Service-A knows that the flaky In-Memory-Service-B hasn't communicated within the time-out window and it was do the appropriate thing based on the death on Service-B.
In-Memory-Service-A has a dialog with Durable-Service-B. Symmetric with case 2 above...
Both Service-A and Service-B are In-Memory and are connected with a dialog. Here, we have two interesting sub-cases. First, if Service-A and Service-B are both in the same failure unit, a failure which wipes out BOTH of them has no end-point visible semantics. Neither of them are around to see the damage! Second, if one is on Computer-A and the other on Computer-B, it is entirely possible that one lives and one dies... The time-out on the dialog provides the semantics of failure (and is delivered as a "dialog-failure" message transactionally delivered to the survivor(s) ). Indeed, the notion of transactionally delivering to an in-memory service is fascinating. Really, we mean delivered with a durability as great as the durability of the observer!

Wow... so this leads us to an interesting observation. In case-1 (durable-to-durable), what if one of the two services is in a triple-data-center redundant high-availability site and the other is on a laptop? It is possible (actually easy) to lose the laptop and have the fact that it is durable on disk be irrelevant. So, what are the semantics of the dialog? What you know is that the partner service (on the other side of the dialog) either responds in time or doesn't. If the partner DOES respond and complete its part of the work (and finish the dialog), the partner may be blown to smithereens and you think you have done some cooperative work. Yet, the partner (and its part of the work) are gone! In case-4 (both in-memory), also has the dilemma that you don't know ANYTHING about the partner except if has sent you the correct messages. Basically, the only thing you know in ALL the cases is if the partner responded in a timely fashion. Once the work is completed, you have no REAL guarantee that it has stuck and won't be blown up.

During the dialog, you have a relationship that can detect the failure and amnesia via the time-out and dialog-failure. After the dialog successfully completes, you are disconnected and cannot tell if the partner's state (and work) are lost in a failure. This is intractable and is really a spectrum of durability (from in-memory to "committed subject to thermo-nuclear exchange"). You have a programmatically visible relationship and then you don't!

Durability Is in the Eye of the Beholder

So, let's consider this proposition that "durability is in the eye of the beholder". Who cares if a transaction is durable?? Remember, I am not (in this blog entry) questioning Atomicity, Consistency, or Isolation. If you see ANY effect of the transaction, you see ALL the effects of the transaction. Why did we have this D thing in ACID, anyway??

Well, it turns out the old-farts (including me) doing the old-time transaction systems just kinda' assumed a special case for interacting with the human. We knew that we needed to tell the person using the system "OK... We did it!" but we didn't talk a lot about this being an example of a messaging relationship in which the human is one participant in a two-party messaging pattern. Also, as I am about to discuss joint failure and the destruction of state at both ends of the messaging relationship, it is slightly uncomfortable to talk about the human being obliterated in the same way as I am about to discuss the annihilation of a communicating service. I'm really nicer than that...

If we assume atomicity and/or a long-running relationship like a dialog, there is a window of time during which you can programmatically tell if the remote work is lost in a failure. Once you exit that window, your make assumptions about the remote work persisting (being durable) even when you are NOT keeping tabs on it. There are cases in which the loss of the remote work only occurs when YOU are wiped out... those are convenient because you aren't around to notice the problem. There are other cases in which we just ASSUME that the probability of loss of the remote work is low enough that we ignore the dilemma.

Basically, big, complex, and distributed system are big, complex, and distributed. We can't get perfect behavior out of them. Something needs to be durable only if I, the partner, am still around to notice the need for it to durable!

- Pat

"The Irresistible Forces Meet the Movable Objects" -- Closing General Session at TechEd EMEA in Barcelona

PatHelland — Mon, 26 Nov 2007 03:35:00 GMT

The big presentation for me at TechEd was the closing general session. I did an entirely new talk called "The Irresistible Forces Meet the Movable Objects".

This talk was a LOT of work and a LOT of fun!!! I had prepared a 30 minute version for the HPTS workshop and enjoyed the practice that this gave me. The closing general session for TechEd (for an estimated 3000 people) was planned as a 75 minute talk with no questions. That left a lot of material to develop to forge a full presentation. Tracking down how to explain the reasons why processor frequencies are not increasing much (due to power constraints) took me some part time effort over the coarse of a week or ten days... Finally, I was happy with the two slide progression summarizing the drive towards many-core. Content from Jim Gray, James Hamilton, Dave Patterson, Jan Gray, Dileep Bhandarkar, and Norm Jouppi helped tremendously as I tried to distill the forces bearing down on us in our development of software. It seemed like the couple of weeks before the conference were hectic with me working late on talk content while working mostly on the new project during work hours. I gave a dry run to my team at Redmond a few days before getting on the plane and that helped a lot when there were a few points of confusion that needed some extra slides to clarify.

About 20 minutes into the talk I ran a 3 minute video which shows a vision of "Software and Services". In this video, a construction contractor named Mike uses multiple devices to access his work and his data as he moves from his home to his car to his worksite. My favorite portion of the video shows his tablet-like PC being destroyed by accident and yet he seamlessly borrows someone else's device and gets at his data in the cloud.

The abstract and video are attached... It is SO great to be back presenting again!!! I want to thank the TechEd EMEA folks for trusting me with such a cool opportunity to present!

--- Pat

ABSTRACT

There are a vast array of economic forces being unleashed on our industry today that mean we need to change how we create applications. From many core processors, low end datacenters through to the rise of ‘Pervasive Intermittent Connectivity’ and the re-definition of the Client, this session provides a unique perspective of the changing landscape and asks the question...

How we can create applications that are approachable to implementers, compose-able in their deployments, and responsive to these economic and technical forces bearing down on us?

This talk is about a vision and not about any kind of product announcements...

Presentation of "Data on the Inside versus Data on the Outside" at TechEd EMEA at Barcelona

PatHelland — Mon, 26 Nov 2007 03:17:00 GMT

On Wednesday, November 7th, I presented "Data on the Inside versus Data on the Outside" at TechEd. While this talk is now over two years old, it is still one of my favorites in spite of the fact that I have crammed too many slides into the session (80 slides into a 75 minute talk). It was a lot of work to finish on time but I think the content was clear and people seemed to like it.

I sure did miss presenting and had a blast at TechEd!!!

--- Pat

ABSTRACT

Recently, a lot of interest has been shown in SOA (Service Oriented Architectures). In these systems, there are multiple services each with its own code and data, and ability to operate independently of its partners. In particular, atomic transactions with two-phase commit do not occur across multiple services because this necessitates holding locks while another service decides the outcome of the transaction. This talk proposes there are a number of seminal differences between data inside a service and data sent into the space outside of the service boundary. The act of unlocking data as a copy of it is sent in the message means the interpretation of the received message must include the understanding that this data in unlocked. This changes how the data can be used.

We then consider objects, SQL, and XML as different representations of data. Each of these models has strengths and weaknesses when applied to the inside and outside of the service boundary. The talk concludes that the strength of each of these models in one area is derived from essential characteristics underlying its weakness in the other area.

Presentation of "Life Beyond Distributed Transactions: an Apostate's Opinion" at TechEd EMEA at Barcelona.

PatHelland — Mon, 26 Nov 2007 03:08:00 GMT

On Tuesday, November 6th, I presented "Life Beyond Distributed Transactions: an Apostate's Opinion" at TechEd EMEA at Barcelona.

Unlike most of my presentations, this one had only 45 slides for a 75 minute session and so I went slower and we had time for some Q&A at the end. Perhaps, I should take a lesson from how enjoyable that was and not quite stuff a slide-per-minute everytime.

The abstract and slides are attached.

ABSTRACT

Many decades of work have been invested in the area of distributed transactions including protocols such as Two-Phase Commit, Paxos, and various approaches to quorum. These protocols provide the application programmer a façade of global serializability. Personally, I have invested a non-trivial portion of my career as a strong advocate for the implementation and use of platforms providing guarantees of global serializability.

My experience over the last decade has led me to liken these platforms to the Maginot Line. [The Maginot Line was a huge fortress that ran the length of the Franco-German border and was constructed at great expense between World War I and World War II. It successfully kept the German army from directly crossing the border between France and Germany. It was quickly bypassed by the Germans in 1940 who invaded through Belgium.

In general, application developers simply do not implement large scalable applications assuming distributed transactions. When they attempt to use distributed transactions, the projects founder because the performance costs and fragility make them impractical. Natural selection kicks in...Instead, applications are built using different techniques which do not provide the same transactional guarantees but still meet the needs of their businesses.

This talk explores and names some of the practical approaches used in the implementations of large-scale mission-critical applications in a world which rejects distributed transactions. We discuss the management of fine-grained pieces of application data which may be repartitioned over time as the application grows. We also discuss the design patterns used in sending messages between these repartitionable pieces of data. The talk is intended to provoke a different way of thinking about the direction of applications in a wildly-scalable world.

Presentation of "Metropolis: Interchangeability of Operations" at TechEd EMEA in Barcelona

PatHelland — Mon, 26 Nov 2007 02:52:00 GMT

On Monday, November 5th, I presented a talk I had written before leaving Microsoft. The first session ARC201 was entitled "Metropolis:Interchangeability of Operations" and the abstract and powerpoint deck are attached. Three more talks coming!

It was a LOT of fun and I enjoyed the crowd immensely.

- Pat

ABSTRACT
Through the 19th century, manufacturers invested in making machines to make the creation of machine parts faster and easier. In a separate effort largely by the armories producing small arms for the US War Department, significant investment was made to make parts so identical that they may be interchanged across two different guns. This work was surprisingly challenging over a forty year period. Simple machine tools created parts that were so inaccurate that they still required manual fitting and adjusting to make a working machine. It was only through the combination of machine tools and interchangeability that we arrived at the mass production techniques of manufacturing that have so changed our world.

If you consider the interaction of services as they send request for operational functions, we see the same challenges of interchangeability. Distrusting services won’t support classic distributed transactions and this necessitates the use of operational requests that may be subsequently canceled. For the service providing the functions, this is only practical to the extent that the tentative operations are interchangeable. Just as most people in the 19th century thought precision interchangeability of manufactured parts was a silly endeavor compared to fitting by highly skilled craftsmen, most programmers don’t recognize the importance of operations that are so equivalent that some may be canceled later without causing duress.

HPTS - High Performance Transaction Systems Workshop

PatHelland — Thu, 22 Nov 2007 12:50:00 GMT

Every two years since 1985, I have gone to the Asilomar Conference Center in Pacific Grove, California for a very special three day gathering. The HPTS Workshop was started by a number of old-time transaction processing folks including Dieter Gawlick, Andreas Reuter, and Jim Gray. In 1985, I was a wet behind the ears 29 year old kid who had been working at Tandem Computers for three years. My friend Jim Gray invited me to attend and this workshop has changed my life in very many ways.

HPTS is an invitation-only gathering comprising the leading workers in the field of Transaction Processing systems. These are primarily the implementers of large scale systems in industry with the occasional academic thrown into the mix. Sometimes, the owners of large applications are invited. Back in the 1980s, the big challenge was to take a huge mainframe and accomplish 1000 transactions per second. Through the years, this evolved into distributed systems, the Internet, large scale web solutions, high availability, workflow, and service oriented architectures. Basically, the workshop focuses on how to get huge amounts of work done flowing through computer systems and all the different aspects of making this happen. The companies involved has changed through the years starting out beginning with IBM, Tandem, and Amdahl dominating the event. This evolved to include DEC and then Microsoft. In the late 1990s, the web companies, Amazon, eBay, and others started to be represented. I was privileged to be chairman of the conference in 1989 and this last year, I was general chairman and was a major contributor to driving the agenda. The workshop ran from October 7th through 10th and, in addition to the work in .NET starting a new project, this kept me VERY busy in July, August, and September.

This workshop is VERY special to me. All of my job changes since the 1980s and many of the interesting connections I have made in the industry have been from this workshop. The spark behind HPTS has always been my friend and mentor, Jim Gray. I have attended every one of the 12 workshops. By waiting for two years between them, it is always a shock to see how much the industry has changed. It is always an invigorating and rejuvenating process for me. The disappearance of Jim at sea last January made this gathering especially poignant for so many of us who consider Jim a dear friend.

The HPTS workshop has left a broad impact on the industry and on me. It was the progenitor of an internal Microsoft event called WHIPS (which I helped start) and, in turn, at least three other internal Microsoft events. I used it when at Amazon as a foundation for an Amazon internal conference. The database community used HPTS as a basis for the CIDR (Conference on Innovative Database Research) which meets every two years at the same venue, Asilomar in January of the odd-numbered years. Indeed, it was at CIDR hanging out with Jim just three weeks before his disappearance that I really became determined to return to the public eye and to Microsoft. Again, my life directly impacted by HPTS and the wake it has left.

The event always runs from Sunday dinner through Wednesday lunch. We had a number of sessions on scalable platforms, service-based application platforms, and operational issues with wildly scalable environments. Tuesday, we shifted into database challenges, Rich Internet Applications (RIAs), and some miscellaneous fun talks like SecondLife, Erlang, and Millecomputing. Wednesday, we heard about the implications of Many-Core computers on our software platforms and wrapped up with some sessions on Service Oriented Architectures. I gave a shortened first try of my planned talk for TechEd Barcelona due on November 9th (I've given it by the time I'm blogging this but I will explain more about that in a subsequent blog -- trying to catch up on what's happened!).

Traditionally, on Monday nights we have poster sessions with 10 minute talks on whatever the speaker wants to cover. As usual, I tried to make mine silly and fun. Normally, Tuesday evening is dedicated to some raucous entertainment (with our colleagues providing the fun) but this year was different. We decided to hold a Tribute to Jim Gray and I had the responsibility and honor of hosting that event. This was a special honor and I have written an entire blog entry on the Tribute to Jim Gray at HPTS.

Jim has always taken the job of Local Arrangements Chair and "Nagging Conscience" for HPTS. We missed his support this year a LOT and, indeed, it was a learning experience for me (helped by my friend Shel Finkelstein now of SAP and Peter Gassner of Verticals on Demand) as we pushed forward to ensure the continuity of this important gathering. At the end of the gathering, my friends gave me the honor of attempting to follow in Jim's footsteps and assume this role. For the 2009 HPTS workshop, we have James Hamilton as General Chairman and Margo Seltzer of Harvard and Oracle as the Program Chair. This is assigned to a new pair of people each event but the Local Arrangements has been maintained by Jim and now that will be my responsibility.

The buildup to HPTS was a TON of work, especially while maintaining a day job starting up a project in .NET. This has been the reason I have not been blogging as much over the summer. The event was, in my opinion, very successful and very rewarding. I miss my friend and mentor, Jim, and am committed to continuing this workshop that he has always loved so much and that I love so much.

-- Pat

Every Noun Can Be Verbed...

PatHelland — Wed, 13 Jun 2007 04:33:00 GMT

As I watch the REST versus non-REST discussion, it occurs to me that there are a number of aspects to the debate. While I am not attempting to address all of these aspects, I think that part of the discussion revolves around DATA versus BEHAVIOR.

When people talk about CRUD (Create, Read, Update, and Delete) interfaces, on the surface, they are manipulating data. Sometimes, this implicitly causes behavior. This is a long-standing pattern that I certainly know has gone on for decades...

So, let's apply some nomenclature:

Nouns: These are data items which get manipulated via CRUD (or equivalent) interfaces.
Verbs: This is behavior which is implemented with a method call or some equivalent.

Now, of course, method calls get marshaled... Hmm... Is the marshaled method call a noun or a verb?

OK, let's speak about the programmatic abstraction visible to the caller... Is it a method call or is it a CRUD operation? What about the methods "Set" and "Get"? Those are obviously method calls. But their soul is to set and get property values and, hence, are really pretty CRUDdy! Gee, verbs can be nouned!

But let's look at verbing the nouns (rather than nouning the verbs).

Example-1 of Verbing a Noun: the Multi-Master Behavior-Turd

A common pattern I have seen when using multi-master database replication across occasionally connected systems is a hub-and-spoke model. The spoke systems will write a record in the database with a unique key (typically based on the spoke's system identity to ensure uniqueness). The semantic meaning of this little record (the "behavior-turd") is a request to perform work at the hub and ship down the answer in the replicated database.

The behavior-turd remains in the offline database until the system can reconnect. When the systems connect, the behavior-turd is replicated to the hub which fires a trigger. Work is done on the hub and the results of this work replicate their way back to the spoke (and all of the other spokes, too).

Important to this approach is the separation of the location in the database for the behavior-turds (basically the messaging mechanism) from the other data updated by the hub and read by the spokes. So the behavior-turds are nouns which act as verbs. Definitely a case of verbing a noun.

Example-2 of Verbing a Noun: CRUDing on the Purchase-Order

So, let's consider a purchase-order data structure. As the purchaser (or software representing the purchaser) adds line-items to the purchase-order, that will cause behavior in the back-end system. Is the purchase-order (or even the line-item) a noun or a verb? I would argue is it syntactically a noun but semantically a verb.

This happens ALL the time. A form is filled out and that implies work to do... Seems like behavior to me even if the localized action of the invoker is to manipulate some data!

Private (and Encapsulated) Data versus Public (Interfacey) Data

It is important to look at the purpose and use of the "data". When we abstract all data as being equivalent, we miss out on a lot of underlying complexity. The internal inventory balance (which may change every few seconds) is certainly not updatable by outside services. We can't assume that CRUD across boundaries carries the evil of CRUD (at a distance) over internal (i.e. private and encapsulated) data.

It has always been my inclination to model the encapsulated behavior as, well, behavior. There's a lot of evidence of successful applications that model the invocation of their behavior as CRUDdy manipulation of some externally updatable data. This is just the same as paper forms in the (before computers) normal business models. Fill out the piece of paper and hand it to the person behind the desk. Filling out the piece of paper is a CRUD operation on the data of the paper. Handing over the piece of paper causes the semantic invocation of some behavior provided by the business.

Natural Selection and Usage of Primitives

There seems to be a gravitation towards simple primitives even if the usage of these simple primitives creates a lot of complexity over time. OK... I can dig that. Back in the mid-90s, I was heavily arguing for early binding (as manifested in COM). The use of early binding drove crisp and clear semantics; goodness, light, and moral rectitude. IDispatch was clearly too fuzzy in its semantics for grown ups...

Well... I admit I was wrong! The ability to use late binding and tooling to adaptively "do the right thing" dominated the lack of crispness in the semantics. Indeed, the crispness wasn't really crispness but it was brittleness. People LIKED the late binding of IDispatch and made it work.

So, in my new state of late-life Zen, I'm not about to fuss about the use of NOUNS rather than VERBS... Just make sure you interpret the durn' things with the correct semantics and I'll loosen up over the syntax of your invocation.

In fact, I'll go so far as to say that the usage of data (NOUNS) to invoke distant behavior (VERBS) might just be a dandy thing. It may be easier to do transformations, filtering, and other mashup-esque whacking on the intended behavior.

It occurs to me that we are seeing both:

The natural selection of mechanisms used to knit together our services (or apps or systems), AND
The natural selection of the implementers who either:

adapt to the new world order with its rampant flexibility and composability, OR
retreat into a self-consistent and increasingly irrelevant view of the world.

In both cases, the forces of natural selection are driving towards malleability and "making it work" (even if the answer is approximate). This parallels the loosening of consistency I pointed out when discussing the CAP-conjecture.

It's a great time to be alive! I can't WAIT to see what else I am completely wrong about...

-- Pat

SOA and Newton's Universe

PatHelland — Sun, 20 May 2007 17:42:21 GMT

Bill de hOra and my old friend (from our Amazon days) Mike Dierken commented on my use of SOA versus "distributed systems". There was also an interest in my perspective on the CAP Conjecture. Let me spew forth some thoughts...

It may be a bit unusual, but my way of thinking of "distributed systems" was the 30+ year (and still continuing) effort to make many systems look like one. Distributed transactions, quorum algorithms, RPC, synchronous request-response, tightly-coupled schema, and similar efforts all try to mask the existence of independence from the application developer and from the user. In other words, make it look to the application like many systems are one system. While I have invested a significant portion of my career working in this effort, I have repented and believe that we are evolving away from this approach.

I wrote a paper for CIDR 2005 called "Data on the Outside versus Data on the Inside". In that paper, I explore the difference in the semantics of data when it is unlocked. Inside of a database, the meaning of the data can be interpreted with a clear and crisp sense of "now" provided by the current transaction. Nothing moves when you are in a transaction unless the currently running application that began the transaction changes the data. There is a strong sense of stillness and of now. Inside data is very much what we have historically programmed to.

In "distributed computing" (in my unusual usage... not in the commonly accepted vernacular), we are trying to extend this notion across multiple machine.

In SOA (again, how I think of it), we are acknowledging the existence of independent machines. This affects the transactional scope (we end up with different chunks of data which cannot be updated by the same transaction) and we end up with independently evolving schema and operations for the different systems. This is a seminally different concept than distributed systems (at least in the way I think of them).

If you try to impose a global ordering to the transactions across a LOT of systems, it is very much like the way Newton thought of the Universe with time marching forward uniformly everywhere. This is why I say that "distributed systems" are like Newton's Universe.

Now, let's consider SOA with independent scopes of serializability (i.e. the collection of computers has some different groups of data which are independent in their transactions... you cannot do a transaction across these different chunks of data). These are broken into independent systems (typically independent applications) which encapsulate their own data and communicate via messaging. When System-A sends a message to System-B, the data contained in the message will be unlocked before sending it. That means that the data is a historic artifact. System-B can only see what some of System-A's data used to look like. This is an essential aspect of these independent systems which do not share transactions. Because of this, I think of each system living in its on temporal domain. It is aware of its internal state and is aware of a subset of its partner's older state. This is just like looking into the night sky and seeing light from neighboring stars emitted years earlier. Each of these systems lives in its own time and has an independent view of time. To me this is like Einstein's Universe where time marches forward based on the perspective of the viewer.

So, the move from distributed systems (one transactional scope --> one notion of time) to SOA (independent transactional scopes --> time based on the perspective of the user) is like moving from Newton's Universe to Einstein's Universe.

-----> Now, let's rant for a while about the CAP Conjecture...

First, let's summarize what Eric Brewer said in 2000 in an invited keynote at the Principals of Distributed Computing. While I am not super familiar with all the literature on this, I believe I understand it well enough to spew forth. Eric said that if you consider CAP - Consistency, Availability, and Partition-tolerance, he offered a conjecture that it is impossible to achieve all three. I totally believe in this conjecture but want to offer some twists in how to think about it.

First, I noticed a long time ago that there is an intrinsic conflict between consistency and availability in the face of partitions. This has led to me growing to increasingly dislike distributed transactions (see "Life Beyond Distributed Transactions: an Apostate's Opinion"). The two-phase commit protocol (which I've spent big portions of my career working on) will ensure perfect consistency given infinite time. I say that because it will wait and wait and wait until the transaction is resolved and then provide perfect consistency. Of course, while partitioned and waiting, arbitrary swaths of the application's database may be locked up rendering the application unusable. For this reason, I've frequently referred to the two phase commit protocol as the "Anti-Availability Protocol". It is increasingly clear to me that this protocol is best used sparingly.

What I think is interesting is how real world applications modify the definition of Consistency and Availability to provide Partition-tolerance. Note that my observations about this do not invalidate the CAP conjecture (which I think is correct) but show how the pain is dramatically reduced by loosing up some age-old assumptions about distributed systems.

Classic database/transaction approaches to Consistency choose to emphasize read-write semantics. To preserve Read-Write-Consistency, you lock the data. We've been at this for over 30 years. What I see happening in loosely-coupled systems is identical to how businesses operated 150-200 years ago when messages were sent with couriers running across the city between businesses. You allocated (i.e. reserved) the ability to perform an operation and then later on you would take the confirming step to ensure the completion of the work. Today, you make a reservation at a hotel and then later on you show up to complete the operation. What is the definition of consistency in this world? It is the successful remembering of the reservation and then keeping a room for you. The reserved room count is not locked waiting for you to decide if you want the reservation, waiting for you to cancel, nor while waiting to see if you show up. The definition of consistency evolves to one that is explicitly including independence and loose-coupling.

A really good paper on this concept is: Isolation Support for Service-based Applications. In this paper, Paul Greenfield et al argue that predicates are the best expression of the isolation required while performing long-running work that spans loosely-coupled (SOA) systems. For example, my work may allocate $200 from your bank account because it is supposed to be mine if the cooperative work we are doing commits. Your bank account balance does not get locked, just $200 of it is encumbered pending the outcome of our cooperative work.

This bends the definition of both Consistency and Availability. Usually, they are considered as providing Read-Write semantics and so they are implemented with locking. If you provide operation semantics, this just changes the form of locking and changes the form of consistency and isolation. I first saw this from Pat O'Neil in The Escrow Transaction Method published in 1986. In this paper, Pat observes that the use of operations, and operation-logging can increase concurrency. At Tandem, we did this in the late 1980s to allow addition and subtraction to have special behavior to improve our TPC-B benchmark numbers. What we did was detect when a SQL operation was an addition or subtraction to a field of a record. We would log a "+30" or "-50" as the operation in the transaction's log. We would also keep a worst-case lower-bound and a worst-case upper-bound for the field's value based on it's committed value combined with the pending (not yet committed nor aborted) transactions. We would manage that the field would remain within the accepted bounds. If you performed a "+30" and then later on aborted the transaction, the act of aborting the operation would subtract 30. In this way, all the transactions would correctly perform their operations but we could run MANY transactions concurrently against a very hot-spot value. Of course, an attempt to read the value would shut everything down, wait for the pending transactions to settle, show the underlying result, and then let the chaos resume. This was a very successful technique for coping with highly-concurrent commutative operations (in this case addition and subtraction) against a hot-spot value.

Nowadays, we see this same technique applied at a different granularity. If you move away from thinking about locking data at distance and move towards ensuring the ability to perform a reserved operation, you think about this differently. This is why you can reserve a king-sized non-smoking room but not (typically) reserve room 301. What is being promised is a member of a fungible category of rooms. It is a different promise of consistency.

My recent post "Memories, Guesses, and Apologies" plays off of what I see happening in an increasing fashion. More and more, I see businesses being willing to loosen Consistency even more than what I was describing above. They are willing to occasionally give the wrong answer because it is more cost-effective.

In the presence of imperfect availability of knowledge, a business is forced to choose between closing down service (reducing availability of the service), over-booking, or over-provisioning. Indeed, if multiple systems (or humans) are extending commitments independently, they must choose between over-booking, over-provisioning, or some unknown balance between them. If I have 10,000 widgets to sell and 100 salespeople, I could allocate 100 to each sales-person and know that I have not over-booked if they go out and independently sell the widgets. To do this, though, I am almost certain to need extra inventory for the sales-people that don't sell all 100 of their widgets. Indeed, for most businesses, this is a ridiculously expensive proposition. So, an analysis is done on the statistics, a cost of over-booking is calculated, and allocations are given based on the expectations of selling the 100,000 widgets. The loosely-coupled algorithm explicitly allows for a probability of over-booking based on its cost-benefit. This is a relaxation of the definition of Consistency to cope with the realities of the CAP Conjecture.

It is essential to approach computing as a means to support business, not a religious fervor. I don't think that it is "wrong" to relax consistency, I think it is important to understand the business trade-offs and apply the technology realities to support the business effectively.

- Pat

Link to "Life Beyond Distributed Transactions: an Apostate's Opinion

PatHelland — Wed, 16 May 2007 20:35:01 GMT

My friend Harry Pierson points out that I am blogging-challenged and didn't include the links to my paper and talk.

This talk and paper were given at CIDR (Conference on Innovative Database Research) on Jan 8th, 2007.

Thanks, Harry!

- Pat

Memories, Guesses, and Apologies

PatHelland — Wed, 16 May 2007 03:56:09 GMT

Well, here I am blogging on the bus with my newly installed Windows Live Writer!!!

This blog is a text version of a five minute "Gong Show" presentation I did at CIDR (Conference on Innovative Database Research) on Jan 8,2007.

All computing can be considered as: "Memories, Guesses, and Apologies". This is a personal opinion about how computers suck. Furthermore, it offers additional opinions about how we can take advantage of their sucki-ness. Lets dig into this...

Newton and Einstein

It used to be that we thought of computing as one big-ass mainframe. The database folks only thought about the database. Transactions (and transactional serializability) offered a crisp and clear perspective of how time marches forward uniformly. When working on transaction T(i), any other transaction T(j) can be perceived as occurring before T(i) or after T(i). If T(i) and T(j) are concurrently processed, the transaction system ensures that either order is correct without modifying the semantics. This offers a crisp and clear perspective of now. Time marches forward like a clock exactly as Newton envisaged his universe.

Nowadays, we have lots and lots of computers. Big ones, small ones, connected, disconnected, occasionally connected, etc. These computers each have their own perspective of time. When you see data, it is unlocked and an artifact of the past. Time is subjective with many different notions of now. This is very much the way Einstein revamped our understanding of the universe.

Moving to SOA is like moving from Newton's Universe to Einstein's Universe.

Inventory and Forklifts

Even is your computer system is perfectly accurate, the data contained withe in it may be incorrect. Data is entered by people and/or sensors. You have the challenge of garbage-in-garbage-out.

Based on the knowledge contained with the computer, decisions are made.

"Jim wants to buy a widget."
"Hey, we have one in the warehouse in New York!"
"Ship it to Jim, should be there on Tuesday!"

Now this is just great... Then the forklift runs over the only widget you have in stock and you won't get any more widgets for a month!

Even if the computer is perfect, it is disconnected from the real world!!! Decisions made by the computer may not be possible to implement!

Guessing and Partial Knowledge

Computers always have partial knowledge for a couple of reasons. First, they will always be separated from the real world. Stuff that happens in the real world will, at best, be reflected after the fact in the computer system. Second, a computer system may have other replicas from which it is separated. A computer's knowledge of the world will be partial.

Computers do not make decisions... They try to make decisions.

The best a computer can do is make a guess. It might be a good guess. It might be a bad guess. Either way, there is no certainty, only guesses.

Memories and Sharing

It is really nice to remember your guesses... It makes it easier when you guessed correctly. It makes it easier to clean up the mess when you've guessed wrong. Usually, computers remember what they've guessed.

Furthermore, it is nice to share with other replicas. You get all the coolness of replicas,disaster protection, etc. Replication definitely helps with memories.

Increased fidelity of memories implies increased cost! The more money you spend on mirroring, replicas, disaster backup, and so forth, the better your memory. The more you spend on increased latencies while you ensure the replicas are in sync before proceeding, the more you reduce the chance of forgetfulness.

It is a business decision how much money, latency, and energy should be spent on reducing forgetfulness. To make this decision, the costs of the increased probability of remembering should be weighed against the costs of occasionally forgetting stuff.

Screw-Ups and Apologies

Consider the following slide from this mini-presentation:

#1 - The application has only a single replica and makes a "decision" to ship the widget on Wednesday. This "decision" is sent to the user.

#2 - The forklift pummels the widget to smithereens.

#3 - The application has no recourse but to apologize, informing the customer they can have another widget in one month (after the incoming shipment arrives).

#4 - Consider an alternate example with two replicas working independently. Replica-1 "decides" to ship the widget and sends that "decision" to User-1.

#5 - Independently, Replica-2 makes a "decision" to ship the last remaining widget to User-2.

#6 - Replica-2 informs Replica-1 of its "decision" to ship the last remaining widget to User-2.

#7 - Replica-1 realizes that they are in trouble... Bummer.

#8 - Replica-1 tells User-1 that he guessed wrong.

#9 - Note that the behavior experienced by the user in the first example is indistinguishable from the experience of user-1 in the second example.

Eventual Consistency and Crappy Computers

Business realities force apologies. To cope with these difficult realities, we need code and, frequently, we need human beings to apologize. It is essential that businesses have both code and people to manage these apologies.

Replication can force apologies. This is the same crap we have to deal with when the physical realities are out of sync with the computer's knowledge, [Please note: I am NOT saying that all replication errors are identical to the business realities. I AM saying that when you replicate business operations (rather than the back-end state of your internal database), the shape of being out-of-sync does, indeed, mimic the business realities.]

We try too hard as an industry. Frequently, we build big and expensive datacenters and deploy big and expensive computers.

In many cases, comparable behavior can be achieved with a lot of crappy machines which cost less than the big expensive one.

Consider the cost/benefit of building big-ass and expensive machines! Careful design of a collection of replicas may fill the business need at a better value!

Just a thought for the day...

- Pat

Why I hate the phrase "Long Running Transactions"...

PatHelland — Thu, 12 Aug 2004 18:44:00 GMT

I always talk about "Long Running Work" and steadfastly avoid "Long Running Transactions"... My preference is to use the word "transaction" to mean an atomic operation that occurs at a single service within a second or so. I mean the ACID (Atomic Consistent Isolated Durable) transactions that usually involve holding locks and that we transaction theorists love.

Whenever I hear about a "Long Running Transaction", I wonder about where it ends. Let's consider the following example:
1) I decide to take a trip to Europe so I book some airline and hotel reservations.
2) The hotel in London hits a threshold of occupancy and decides to increase staffing and food for the restaurant.
3) The hotel orders more food from the Green Grocer.
4) The Green Grocer hits a threshold and orders another delivery from its shipper.
5) The shipping company hits a thresold and orders more diesel fuel for its trucks...
6) Two weeks later, I cancel my trip.

So, if I believe in long-running-transactions, the shipping company doesn't need any more diesel fuel! I don't think so!

The long-running-transaction approach assumes that there is a cohesive set of work across loosely-coupled and independent systems that is atomic. Notice it is not ACID in that Isolation is clearly different and Consistency is clearly different. Hence, you would consider these "long-running-transactions" to exhibit Atomicity and Durability. I am arguing that Atomicity is not a property that flows across multiple services as the semantics of interaction across these services is not tightly associated.

My preference is to talk about "long-running-work" across two services. I can examine the semantic interaction of two services across a set of messages. This leads to the notions of reliable messaging (and multi-message dialogs) which is one of my favorite topics and something I will be posting more work to my website (www.pathelland.com) soon. It also leads to the notion of a contract across the messages that flow between two services. I love these concepts.

Another concept that I am strongly in favor of is the concept of a service being in the midst of multiple related dialogs. My attempt to book travel arrangements puts me in multiple interactions with different airlines and hotels. These interactions involve multiple messages and continue at least until the date of my travel (unless I cancel). When my incoming reservation causes the hotel to increase its food requirements, the hotel will use business logic to control the relationship between my incoming messages and the restaurant related messages. This allows it to consider the wisdom of canceling the food order when I cancel my reservation... very different than mindlessly assuming some coordinator associates these into a long-running-transaction.

Work flows across services and businesses in ways that do not magically "undo" even if you do master the use of compensations rather than the classic "undo" of replacing the before values. The behavior of our interconnected systems is more an integration of lots of disparate work into summaries. When I through a rock into the ocean by San Francisco, that can and does impact the waves that hit Tokyo but only in conjunction with countless other stimuli.

This is why I like to talk about long-running-work inside a single service or across exactly two services using a multi-message dialog. The phrase long-running-transaction is a chimera and leads to a false hope of simplicity. The reality is much richer and more complex.

Love, Pat

Upcoming talks at TechEd EMEA in Amsterdam...

PatHelland — Fri, 18 Jun 2004 18:58:00 GMT

Sorry about the long delay in blogging...

I was supposed to present at TechEd SanDiego a few weeks ago but my mother passed away on May 19th... We will all miss her, especially my 84 year old father who is quite lonely now...

I wanted to tell everyone that I am working on getting a website going that will contain all the ongoing (and much of the past) work that I am involved with. I have set a target of the beginning of July to get this going. Many of the topics I've wanted to blog on required so much context from my presentations and papers that I've been reticent to blog them. Hopefully, having the ability to link to this content will create a better dynamic.

I am committed to six sessions at TechEd EMEA starting on June 29th. I am VERY exceited about this but have a ton of powerpoint work! Some of these are done and others are still a mess. Here's the abstracts of what's coming! Hope you can join us in Amsterdam!

More blogging soon...

Love,
Pat

---------------------------

GNLARC – Metropolis: Independent and Interconnected

This is general session for the Architecture Track at TechEd EMEA.

In this talk, we examine the parallels between urban development and the development of IT shops. In the 1800s, railroads connected independent cities and stimulated changes in manufacturing, retail, and standardization. We argue that this very much like what is happening today as independent IT shops are connected by the Internet.

The forces let loose by this interconnection lead us to service orientation just as our economy has developed independent companies offering manufacturing and retail which work together. The metropolitan analogy offers some surprising predictions for the future of IT.

At the end of this presentation, I am excited that Don Box and David Chappell are going to help me with a surprise! It promises to be a lot of fun and very, very memorable. For those of you that can’t see it live, we will get the video up on the web by early July! Tons of work in this one! Planning, rehearsing… lots of work!

ARC302 – Metropolis: Building Applications in a Service Oriented Architecture

Some buildings adapt to change and continually renew themselves to take on a new life as their environment evolves. An examination of buildings shows that there are certain design characteristics that make buildings more adaptable. In the last few decades, there has been an increasing trend towards the creation of buildings which fit within a framework of use and all for many adaptations and reuse. Examples of this include strip malls for retail and industrial parks for light manufacturing and many other uses.

These trends can be seen in applications and how they are or are not designed to cope with a changing environment. This talk examines the parallels between buildings and applications and shows the re-emergence of these patterns in the virtualization of systems and the creation of service oriented architectures.

ARC 303 – Metropolis: Using Interchangeability

Through the 19^th century, manufacturers invested in making machines to make the creation of machine parts faster and easier. In a separate effort largely by the armories producing small arms for the US War Department, significant effort was made to make parts so identical that they may be interchanged across two different guns. This effort was surprisingly difficult over a forty year period. Simple machine tools created parts that were so inaccurate that they still required manual fitting and adjusting to make a working machine. It was only through the combination of machine tools and interchangeability that we arrived at the mass production techniques of manufacturing that have so changed our world.

If you consider the interaction of services as they send request for operational functions, we see the same challenges of interchangeability. Distrusting services won’t support classic distributed transactions and this necessitates the use of operational requests that may be subsequently canceled. For the service providing the functions, this is only practical to the extent that the tentative operations are interchangeable. Just as most people in the 19^th century thought precision interchangeability of manufactured parts was a silly endeavor compared to fitting by highly skilled craftsmen, most programmers don’t recognize the importance of operations that are so equivalent that some may be canceled later without causing duress.

This talk explores the history of manufacturing and highlights some of the important challenges we have in creating collections of interoperating services.

ARC402-- Thoughts on Data in Service Oriented Architecture

Services provide a formal boundary of computing. Inside a service we typically find data that is needed for the operation of the service. In between services, we see the flow of operational requests and the transmission of data. This talk explores the difference between data on the outside that flows between services and data on the inside that is maintained for the internal use of the service.

As data is transmitted between services, we need to recognize that it is immutable and, once it is written, can only be changed by creating a new version for transmission. Schema definition and extensibility take on new importance in the world outside of services. This talk explores these design issues with an emphasis on pragmatic design choices made in the architecting of services and the interaction between them.

Finally, we examine the characteristic of SQL, Objects, and XML. Each has its strengths and weaknesses. We observe that the characteristics that make SQL wonderful for representing internal data, Objects wonderful for software engineering, and XML wonderful for communicating across services are exactly the characteristics that are challenges in other usages. For SQL, Objects, and XML, their amazing strengths in one use are the essence of their weakness in another use. We need all three but we need each in its place of strength.

PLN005 – Metropolis: Implementation Advice for the Service-Oriented Enterprise

This is a panel discussion with: Pat Helland, Arvindra Sehmi, David Hill, Maarten Mullender, and Steve Cook.

ARC230 – The Nerd, the Suit, and the Fortune Teller

This is a skit and presentation working with Clemens Vasters and Rafal Lukawiecki. I am really excited about this session as we hope to motivate the perspectives of the Suit (the businessman), the Nerd (the technologist), and the Fortune Teller (the architectural visionary). I get to play the Fortune Teller!!

This promises to be a fun way to end the TechEd week by providing some thought provoking perspectives on the trends towards SOA.