Clemens Vasters.

4 Questions

clemensv — Thu, 01 Dec 2011 08:38:37 GMT

I answered 4 questions in Richard Seroter’s series of interviews with folks working on connect systems. See the Q&A here.

Achieving Transactional Behavior with Messaging

clemensv — Thu, 06 Oct 2011 12:10:14 GMT

Elastic and dynamic multitenant cloud environments have characteristics that make traditional failure management mechanisms using coordinated 2-phase transactions a suboptimal choice. The common 2-phase commit protocols depend on a number of parties enlisted into a transaction making hard promises on the expected outcome of their slice a transaction. Those promises are difficult to keep in an environment where systems may go down at any time with their local state vanishing, where not all party trust each other, where significant latency may be involved, and network connectivity cannot be assumed to be reliable. 2-phase-commit is also not a good choice for operations that take significant amounts of time and span a significant amount of resources, because such a coordinated transaction may adversely affect the availability of said resources, especially in cases where the solution is a high-density multitenant solution where virtualized, and conceptually isolated resources are collocated on the same base resources. In such a case, database locks and locks on other resources to satisfy coordinated transaction promises may easily break the isolation model of a multitenant system and have one tenant affect the other.

Therefore, failure management – and this is ultimately what transactions are about – requires a somewhat different approach in cloud environments and other scalable distributed systems with similar characteristics.

To find a suitable set of alternative approaches, let’s quickly dissect what goes on in a distributed transaction:

To start, two or more parties ‘enlist’ into a shared transaction scope performing some coordinated work that’s commonly motivated by a shared notion of a ‘job’ that needs to be executed. The goal of having a shared transaction scope is that the overall system will remain correct and consistent in both the success and the failure cases. Consistency in the success case is trivial. All participating parties could complete their slice of the job that had to be done. Consistency in the failure case is more interesting. If any party fails in doing their part of the job, the system will end up in a state that is not consistent. If you were trying to book a travel package and ticketing with the airline failed, you may end up with a hotel and a car, but no flight. In order to prevent that, a ‘classic’ distributed transaction asks the participants to make promises on the outcome of the transaction as the transaction is going on.

As all participating parties have tentatively completed but not finalized their work, the distributed transaction goes into a voting phase where every participant is asked whether it could tentatively complete its portion of the job and whether it can furthermore guarantee with a very high degree of certainty that it can finalize the job outcome and make it effective when asked to do so. Imagine a store clerk who puts an item on the counter that you’d like to purchase – you’ll show him your $10 and ask for a promise that he will hand you the item if you give him the money – and vice versa.

Finally, once all parties have made their promises and agreed that the job can be finalized, they are told to do so.

There are two big interesting things to observe about the 2-phase-commit (2PC) distributed transaction model that I just described: First, It’s incredibly simple from a developer’s perspective because the transaction outcome negotiation is externalized and happens as ‘magic’. Second, it’s not resembling anything that happens in real life and that should be somewhat suspicious. You may have noticed that there was no neutral escrow agent present when you bought the case of beverages at the store for $10 two paragraphs earlier.

The grand canonical example for 2PC transactions is a bank account transfer. You debit one account and credit another. These two operations need to succeed or fail together because otherwise you are either creating or destroying money (which is illegal, by the way). So that’s the example that’s very commonly used to illustrate 2PC transactions. The catch is – that’s not how it really works, at all. Getting money from one bank account to another bank account is a fairly complicated affair that touches a ton of other accounts. More importantly, it’s not a synchronous fail-together/success-together scenario. Instead, principles of accounting apply (surprise!). When a transfer is initiated, let’s say in online banking, the transfer is recorded in form of a message for submission into the accounting system and the debit is recorded in the account as a ‘pending’ transaction that affects the displayed balance. From the user’s perspective, the transaction is ’done’, but factually nothing has happened, yet. Eventually, the accounting system will get the message and start performing the transfer, which often causes a cascade of operations, many of them yielding further messages, including booking into clearing accounts and notifying the other bank of the transfer. The principle here is that all progress is forward. If an operation doesn’t work for some technical reason it can be retried once the technical reason is resolved. If operation fails for a business reason, the operation can be aborted – but not by annihilating previous work, but by doing the inverse of previous work. If an account was credited, that credit is annulled with a debit of the same amount. For some types of failed transactions, the ‘inverse’ operation may not be fully symmetric but may result in extra actions like imposing penalty fees. In fact, in accounting, annihilating any work is illegal – ‘delete’ and ‘update’ are a great way to end up in prison.

As all the operations occur that eventually lead to the completion or failure of the grand complex operation that is a bank transfer, the one thing we’ll be looking to avoid is to be in any kind of ‘doubt’ of the state of the system. All participants must be able to have a great degree of confidence in their knowledge about the success or failure of their respective action. No shots into the dark. There’s no maybe. Succeed or fail.

That said, “fail” is a funny thing is distributed systems because it happens quite a bit. In many cases “fail” isn’t something that a bit of patience can’t fix. Which means that teaching the system some patience and tenacity is probably a good idea instead of giving up too easily. So if an operation fails because it runs into a database deadlock or the database is offline or the network is down or the local machine’s network adapter just got electrocuted that’s all not necessarily a reason to fail the operation. That’s a reason to write an alert into a log and call for help for someone to fix the environment condition.

If we zoom into an ‘operation’ here, we might see a message that we retrieve from some sort of reliable queue or some other kind of message store and subsequently an update of system state based on message. Once the state has been successfully updated, which may mean that we’ve inserted a new database record, we can tell the message system that the message has been processed and that it can be discarded. That’s the happy case.

Let’s say we take the message and as the process wants to walk up the database the power shuts off. Click. Darkness. Not a problem. Assuming the messaging system supports a ‘peek/lock’ model that allows the process to first take the message and only remove it from the queue once processing has been completed, the message will reappear on the queue after the lock has expired and the operation can be retried, possibly on a different node. That model holds true for all failures of the operation through to and in the database. If the operation fails due to some transient condition (including the network card smoking out, see above), the message is either explicitly abandoned by the process or returns into the queue by ways of a lock timeout. If the operation fails because something is really logically wrong, like trying to ship a product out of the inventory that’s factually out of stock, we’ll have to take some forward action to deal with that. We’ll get to that in a bit.

Assuming the operation succeeded, the next tricky waypoint is failure after success, meaning that the database operation succeeded, but the message subsequently can’t be flagged as completed and thus can’t be removed from the queue. That situation would potentially lead to another delivery of the message even though the job has already been completed and therefore would cause the job to be executed again – which is only a problem if the system isn’t expecting that, or, in fancier terms, if it’s not ‘idempotent’. If the job is updating a record to absolute values and the particular process/module/procedure is the only avenue to perform that update (meaning there are no competing writers elsewhere), doing that update again and again and again is just fine. That’s natural idempotency. If the job is inserting a record, the job should contain enough information, such as a causality or case or logical transaction identifier that allows the process to figure out whether the desired record has already been inserted and if that’s the case it should do nothing, consider its own action a duplicate and just act as if it succeeded.

Checkpoint: With what I said in the last two paragraphs, you can establish pretty good confidence about failure or success of individual operations that are driven by messages. You fail and retry, you fail and take forward action, or you succeed and take steps to avoid retrying even if the system presents the same job again. There’s very little room for doubt. So that’s good.

The ‘forward action’ that results from failure is often referred to as ‘compensation’, but that’s a bit simplistic. The forward action resulting from running into the warehouse with the belief that there’s still product present while the shelf is factually empty isn’t to back out and cancel the order (unless you’re doing a firesale of a touch tablet your management just killed). Instead, you notify the customer of the shipping delay, flag a correction of the inventory levels, and put the item on backorder. For the most part, pure ‘compensation’ doesn’t really exist. With every action, the system ends up in a consistent state. It’s just that some states are more convenient than others and there are some state for which the system has a good answer and some states for which it doesn’t. If the system ends up in a dead end street and just wants to sit down and cry because nobody told it what to do now, it should phone home and ask for human intervention. That’s fine and likely a wise strategy in weird edge cases.

Initiating the ‘forward action’ and, really, any action in a system that’s using messaging as its lifeline and as a backplane for failure resilience as I’m describing it here is not entirely without failure risk in itself. It’s possible that you want to initiate an action and can’t reach the messaging system or sending the message fails for some other reason. Here again, patience and tenacity are a good idea. If we can’t send, our overall operation is considered failed and we won’t flag the initiating message as completed. That will cause the job to show up again, but since we’ve got idempotency in the database that operation will again succeed (even if by playing dead) or fail and we will have the same outcome allowing us to retry the send. If it looks like we can send but sending fails sometime during the operation, there might be doubt about whether we sent the message. Since doubt is a problem and we shouldn’t send the same message twice, duplicate detection in the messaging system can help suppressing a duplicate so that it never shows up at the receiver. That allows the sender to confidently resend if it’s in doubt about success in a prior incarnation of processing the same message.

Checkpoint: We now also can establish pretty good confidence about initiating forward action or any other action in the system given if the ‘current’ action is following the principles described above.

So far I’ve talked about individual actions and also about chains of actions, albeit just in the failure case. Obviously the same applies to success cases where you want to do something ‘next’ once you’re done with ‘this’.

Now let’s assume you want to do multiple things in parallel, like updating multiple stores as part of executing a single job – which gets us back to the distributed transaction scenario discussed earlier. What helps in these cases is if the messaging system supports ‘topics’ that allow dropping a message (the job) into the messaging system once and serve the message to each participant in the composite activity via their own subscription on the topic. Since the messaging system is internally transactional it will guarantee that each message that is successfully submitted will indeed appear on each subscription so it ensures the distribution. With that, the failure handling story for each slice of the composite job turns into the same model that I’ve been explaining above. Each participant can be patient and tenacious when it comes to transient error conditions. In hard failure cases, the forward action can be a notification to the initiator that will then have to decide how to progress forward, including annulling or otherwise invoking forward actions activities that have been executed in parallel. In the aforementioned case of a ticketing failure that means that the ticketing module throws its hands up and the module responsible for booking the travel package either decides to bubble the case to the customer or an operator leaving the remaining reservations intact or to cancel the reservations for the car and the hotel that have been made in parallel. Should two out of three or more participants’ operations fail and each report up to the initiator, the initiator can either keep track of whether it already took corrective forward action on the third participant or, in doubt, the idempotency rule should avoid doing the same thing twice.

The model described here is loosely based on the notion of ‘Sagas’, which were first described in a 1987 ACM paper by Hector Garcia-Molina and Kenneth Salem, so this isn’t grand news. However, the notion of such Sagas is only now really gaining momentum with long-running and far distributed transactions becoming more commonplace, so it’s well worth to drag the model further out into the limelight and give it coverage. The original paper on Sagas is still assuming that the individual steps can be encapsulated in a regular transaction, which may not even be the case in the cloud and with infrastructures that don’t have inherent transaction support. The role of the messaging system with the capabilities mentioned above is to help compensate for the absence of that support.

… to be continued …

Service Bus Relay Load Balancing–The Missing Feature (But Not For Much Longer!)

clemensv — Tue, 20 Sep 2011 14:54:10 GMT

Load Balancing on the Service Bus Relay is by far our #1 most requested feature now that we’ve got Queues and Topics finally in production. It’s reasonable expectation for us deliver that capability in one of the next production updates and the good news is that we will. I’m not going to promise any concrete ship dates here, but it’d be sorely disappointing if that wouldn’t happen while the calendar still says 2011.

I just completed writing the functional spec for the feature and it’s worth communicating how the feature will show up, since there is a tiny chance that the behavioral change may affect implementations that rely on a particular exception to drive the strategy of how to perform failover.

The gist of the Load Balancing spec is that the required changes in your code and config to get load balancing are zero. With either the NetTcpRelayBinding or any of the HTTP bindings (WebHttpRelayBinding, etc) as well as the underlying transport binding elements, you’ll just open up a second (and third and fourth … up to 25) listener on the same name and instead of getting an AddressAlreadyInUseException as you get today, you’ll just get automatic load balancing. When a request for your endpoints shows up at Service Bus, the system will roll the dice on which of the connected listeners to route the request or connection/session to and perform the necessary handshake to make that happen.

The bottom line is that we’re effectively making the AddressAlreadyInUseException go away for the most part. It’ll still be thrown when the listener’s policy settings don’t match up, i.e. when one listener wants to have Access Control enabled and the other one doesn’t, but otherwise you’ll just won’t see it anymore.

The only way this way of just lighting up the feature may get anyone in trouble is if your application were to rely on that exception in a situation where you’ve got an active listener on Service Bus on one node and a ‘standby’ listener on another node that keeps trying to open up a listener into the same address to create a hot/warm cluster failover scheme and if the two nodes were tripping over each other if they were getting traffic concurrently. That doesn’t seem too likely. If you have questions about this drop me a line here in the comments, by email to clemensv at microsoft.com or on Twitter @clemensv.

Securing Service Bus with the Access Control Service

clemensv — Mon, 19 Sep 2011 10:00:29 GMT

This session explains how to secure Service Bus using the Access Control Service. This is also an extension session for my session at BUILD, but watching the BUILD session is not a strict prerequisite.

Service Bus Topics and Queues – Advanced

clemensv — Sun, 18 Sep 2011 17:15:26 GMT

This session is a followup to the Service Bus session that I did at the build conference and explains advanced usage patterns:

Building loosely-coupled Apps with Windows Azure Service Bus Topics and Queues

clemensv — Fri, 16 Sep 2011 05:49:01 GMT

From //build in Anaheim

Understanding Windows Azure AppFabric Queues (and Topics)

clemensv — Fri, 10 Jun 2011 16:29:21 GMT

Our team’s Development Manager MK (Murali Krishnaprasad) and me were interviewed by Michael Washam on May 2011 CTP release of Windows Azure AppFabric. We discuss new technologies such as Topics, Queues, Subscriptions and how this relates to doing async development in the cloud.

Republished from Channel 9

Service Bus May 2011 CTP Client Bits now on NuGet

clemensv — Mon, 23 May 2011 04:42:39 GMT

(This post has been written by my coworker Eric Lam who pulled this all together – he wrote it as an email to the team – I’m just ripping that off)

NuGet (http://www.nuget.org/) is an open source package manager for .NET started by Microsoft. For some context there is a really good TechEd talk here.

To install:

1. Unless you have it (for instance as a side-effect of installing ASP.NET MVC3), get NuGet installed (it’s an VS extension) from here

2. Right click on the project you want to add Service bus support to and select “Add Library Package Reference”

3. The NuGet manager will show up, and search for “AppFabric”, which will show all our AppFabric packages available right now (note: Cache is there too!)

4. Click install, and that’s it! The package do a number of things

Copy Microsoft.ServiceBus.dll and Microsoft.ServiceBus.Messaging.dll to your local project directories
Add the references to your project
If you have a App.Config and/or Web.Config, it will add the necessary WCF bindings/extensions for project consumption.

Some things to note for the package:

The packages require the projects to use the .Net 4.0 Full Profile (Client profile doesn’t work).
For sample code you will need to install it on a C# Console Application project.
This is only available for the CTP right now, not yet for the production SDK.

Let us know if you have any feedback or comments.

Audio clips recorded on the road

clemensv — Fri, 20 May 2011 12:11:05 GMT

I’ve been starting to use Cinchcast to post little audio clips that I record while driving from and to work about things that come up with regards to Service Bus on the forums or on Twitter or elsewhere. The RSS feed is at http://www.cinchcast.com/clemensv.rss

Service Bus May 2011 CTP Resources

clemensv — Thu, 19 May 2011 07:33:30 GMT

Here’s a (incomplete) snapshot of what’s out there in terms of material for the new Service Bus CTP:

First read the release notes where we provide a summary of what’s new and what changed and also point out some areas of caution on parallel installs of the CTP and production SDKs.
You can get the SDK bits from here. Get the right set of binaries for your machine (x64 or x86) and the right set of samples (CS or VB) and definitely get the user guide. We will have a NuGet package shortly that will allow you integrating the Service bus assembly and all necessary config incantations straight into your apps without even having the SDK on your machine.
The reference docs are located here. This is a CTP and the documentation is likewise in CTP state, so there are some gaps that we try to fill.
My introduction to the CTP is on the new AppFabric blog here.
At the same location you’ll find David Ingham’s primer on Queues.
My TechEd talk on the new features is now posted on Channel 9.
We have a video series providing high-level overviews on Service Bus.
Neudesic’s Rick Garibay provides a community insider’s perspective on the new features. Matt Davey also likes what he sees.
The forums. Go there for questions or suggestions.

There’s more on the way. Let me know if you write a blog post about what you find out so I can link to it.

Introducing the Windows Azure AppFabric Service Bus May 2011 CTP

clemensv — Mon, 16 May 2011 08:31:09 GMT

A lot of partners and customers we talk to are telling us that they think of Service Bus as one of the key differentiators of the Windows Azure platform because it enables customers to build and interconnect applications that reflect the reality of where things stand with regard to moving workloads to cloud infrastructures: Today and for years to come, applications and solutions will straddle desktop and devices, customer-owned and operated servers and datacenters, and private and public cloud assets.

After a decade and more of application integration and process streamlining, no line-of-business application is and should ever again be an island.

If applications move to the cloud or if cloud-based SaaS solutions are to be integrated into enterprise solutions for individual customers, integration invariably requires capabilities like seamless access to services and secure, reliable message flow across network and trust boundaries. Also, as more and more applications are federated across trust boundaries and are built to work for multiple tenants, classic network federation technologies such as VPNs are often no longer adequate since they require a significant degree of mutual trust between parties as they permit arbitrary network traffic flow that needs to be managed.

We just released a new Community Technology Preview that shows that we’re hard at work and committed to expand Service Bus into a universal connectivity, messaging, and integration fabric for cloud-hosted and cloud-connected applications – and we invite you to take a look at our Windows Azure AppFabric SDK V2.0 CTP – May Update and accompanying samples.

Service Bus is already unique amongst platform-as-a-service offerings in providing a rich services relay capability that allows for global endpoint federation across network and trust boundaries.

As of today, we’re adding a brand-new set of cloud-based, message-oriented-middleware technologies to Service Bus that provide reliable message queuing and durable publish/subscribe messaging both over a simple and broadly interoperable REST-style HTTPS protocol with long-polling support and a throughput-optimized, connection-oriented, duplex TCP protocol.

The new messaging features, built by the same team that owns the MSMQ technology, but on top of a completely new technology foundation, are (of course) integrated with Service Bus’s naming and discovery capabilities and the familiar management protocol surface and allow federated access control via the latest release of the Windows Azure AppFabric Access Control service.

Queues

Service Bus Queues are based on a new messaging infrastructure backed by a replicated, durable store.

Each queue can hold up to 100MB of message content in this CTP, which is a quota we expect to expand by at least an order of magnitude as the service goes into production. Messages can have user-defined time-to-live periods with no enforced maximum lifetime.

The size of any individual message is limited to 256KB, but the session feature allows creating unlimited-size sequences of related messages whereby sessions are pinned to particular consumers and therefore enabling chunking of payloads of arbitrary sizes. The session state facility furthermore allows transactional recording of the progress a process makes as it consumes messages from a session and we also support session-based correlation, meaning that you can build multiplexed request/reply paths in a straightforward fashion.

Queues support reliable delivery patterns such as Peek/Lock both on the HTTP API and the .NET API that help ensuring processing integrity across trust boundaries where common mechanisms like distributed 2-phase transactions are challenging. Along with that, we have built-in detection of inbound message duplicates, allowing clients to re-send messages without adverse consequences if they’re ever in doubt whether a message has been logged in the queue due to intermittent network issues or an application crash.

In addition to a dead-letter facility for messages that cannot be processed or expire, Queues also allow deferring messages for later processing, for instance when messages are received out of the scheduled processing order and need to be safely put on the side while the process waits for a particular message to permit further progress.

Queues also support scheduled delivery – which means that you can hand a message to the queue infrastructure, but the message will only become available at a predetermined point in time, which is a very elegant way to build simple timers.

Topics

Service Bus Topics provide a set of new publish-and-subscribe capabilities and are based on the same backend infrastructure as Service Bus Queues – and have all the features I just outlined for Queues.

A Topic consists of a sequential message store just like a Queue, but allows for many (up to 2000 for the CTP) concurrent and durable Subscriptions that can independently yield copies of the published messages to consumers.

Each Subscription can define a set of rules with simple expressions that specify which messages from the published sequence are selected into the Subscription; a Subscription can select all messages or only messages whose user-or system defined properties have certain values or lie within certain value ranges. Rules can also include Actions, which allow modifying message properties as messages get selected; this allows, for instance, selecting messages by certain criteria and affinitizing those messages with sessions or to stamp messages with partitioning keys, amongst many other possible patterns.

The filtered message sequence represented by each Subscription functions like a virtual Queue, with all the features of Queues mentioned earlier. Thus, a Subscription may have a single consumer that gets all messages or a set of competing consumers that fetch messages on a first-come-first-served basis.

To name just a few examples, Topics are ideal for decoupled message fan-out to many consumers requiring the same information, can help with distribute work across partitioned pools of workers, and are a great foundation for event-driven architecture implementations.

Topics can always be used just like Queues by setting set up a single, unfiltered subscription and having multiple competing consumers pull messages from the subscription. The great advantage of Topics over Queues is that additional subscriptions can be added at any time to allow for additional taps on the message sequence for any purpose; audit taps that log pre-processing input messages into archives are a great example here.

Access Control Integration

This new CTP federates with the appfabriclabs.com version of the Access Control service, which is compatible with the Access Control “V2” service that is in available commercially since April. The current commercially available version of Service Bus federates with Access Control “V1”.

The Service Bus API to interact with Access Control for acquiring access tokens has not changed, but we are considering changes to better leverage the new federation capabilities of Access Control “V2”.

Customers who are setting up access control rules for Service Bus programmatically will find that there are significant differences between the management APIs of these two versions of the Access Control service. The current plan is to provide a staged migration for customers with custom access control rules on their Service Bus namespaces; migration will be an option for some period of time when we will operate the V1 and V2 versions of the Access Control Service side-by-side. We will publish concrete guidance for this migration over the next several months with initial details coming this week here on this blog.

What Changed and What’s Coming?

We believe that providing these capabilities in the cloud – paired with the features we already have available in Service Bus – will open up a whole new range of possibilities for cloud-hosted and cloud-enhanced applications. We have seen amazing business solutions built on Service Bus and based on customer feedback we’re convinced that the addition of a fully featured set of message-oriented middleware capabilities will enable even more powerful solutions to be built. Our intention is to make all capabilities contained in this preview commercially available in the second half of 2011.

The load balancing and traffic optimization features for the Relay capability of Service Bus that were added in the PDC’10 CTP of Service Bus have been postponed and are no longer available in this CTP. However, “postponed” does not mean “removed” and we are planning on getting these features back into a CTP release soon. We’ve traded these features for capabilities that we expect will be even more important for many customers: Full backwards compatibility between the current production release of Service Bus and the new version we’re presenting in this CTP, even though we have changed a very significant portion of the Service Bus backend. We are committed to provide full backwards compatibility for Service Bus when the capabilities of this CTP go into production, including backwards compatibility with the Microsoft.ServiceBus.dll that you already have deployed.

To help customers writing apps on platforms other than .NET we will also release Java and PHP samples for the new messaging capabilities in the next few weeks. These samples will be versions of the chat client implemented in the Silverlight and Windows Phone chat samples included in the SDK for this CTP release.

Lastly, and most importantly, the purpose of a Community Technology Preview is to collect feedback from the community. If you have suggestions, critique, praise, or questions, please let us know at http://social.msdn.microsoft.com/Forums/en-US/appfabricctp/

You can also Twitter me personally at @clemensv and I’ll let the team know what you have to say.

For the Service Bus Team,

Clemens Vasters
Principal Technical Lead
Windows Azure AppFabric Service Bus

A Bit Of Service Bus NetTcpRelayBinding Latency Math

clemensv — Thu, 12 May 2011 05:50:39 GMT

From: John Doe
Sent: Thursday, May 12, 2011 3:10 AM
To: Clemens Vasters
Subject: What is the average network latency for the AppFabric Service Bus scenario?
Importance: High

Hi Clemens,

A rough ballpark range in milliseconds per call will do. This is a very important metric for us to understand performance overhead.

Thanks,
John

From: Clemens Vasters
Sent: Thursday, May 12, 2011 7:47 AM
To: John Doe
Subject: RE: What is the average network latency for the AppFabric Service Bus scenario?

Hi John,

Service Bus latency depends mostly on network latency. The better you handle your connections, the lower the latency will be.

Let’s assume you have a client and a server, both on-premise somewhere. The server is 100ms avg roundtrip packet latency from the chosen Azure datacenter and the client is 70ms avg roundtrip packet latency from the chosen datacenter. Packet loss also matters because it gates your throughput, which further impacts payload latency. Since we’re sitting on a ton of dependencies it’s also worth telling that a ‘cold start’ with JIT impact is different from a ‘warm start’.

With that, I’ll discuss NetTcpRelayBinding:

There’s an existing listener on the service. The service has a persistent connection (control channel) into the relay that’s being kept alive under the covers.
The client connects to the relay to create a connection. The initial connection handshake (2) and TLS handshake (3) take about 5 roundtrips or 5*70ms = 350ms. With that you have a client socket.
Service Bus then relays the client’s desire to connect to the service down the control channel. That’s one roundtrip, or 100ms in our example; add 50ms for our internal lookups and routing.
The service then sets up a rendezvous socket with Service Bus at the machine where the client socket awaits connection. That’s just like case 2 and thus 5*100ms=500ms in our case. Now you have an end-to-end socket.
Having done that, we’re starting to pump the .NET Framing protocol between the two sites. The client is thus theoretically going to get its first answer after a further 135ms.

So the handshake takes a total of 1135ms in the example above. That’s excluding all client and service side processing and is obviously a theoretical number based on the latencies I picked here. You mileage can and will vary and the numbers I have here are the floor rather than the ceiling of relay handshake latency.

Important: Once you have a connection set up and are holding on to a channel all subsequent messages are impacted almost exclusively by the composite roundtrip network latency of 170ms with very minimal latency added by our pumps. So you want to make a channel and keep that alive as long as you can.

If you use the Hybrid mode for NetTcpRelayBinding and the algorithm succeeds establishing the direct socket, further traffic roundtrip time can be reduced to the common roundtrip latency between the two sites as the relay gets out of the way completely. However, the session setup time will always be there and the Hybrid handshake (which follows establishing a session and happens in parallel) may very well up to 10 seconds until the direct socket is available.

For HTTP the story is similar, with the client side socket (not the request; we’re routing the keepalive socket) with overlaid SSL/TLS triggering the rendezvous handshake.

I hope that helps,
Clemens

Reading ATAPI SMART Data from Drives using .NET; Temperature Anyone?

clemensv — Mon, 11 Apr 2011 07:21:16 GMT

I’ll admit this is an odd topic for me to write about since my job pretty far away from that part of the world, but our PM team at MS is building a set of demos for which we need some semi-random and fun input data that doesn’t change all that rapidly. So we thought that reading the temperature off hard drives would be a nice input. But how to get at it?

The solution is to use the WMI interface for ATAPI to get at the SMART data. Binging the subject you’ll find a ton of little snippets that have one thing in common: ‘magic’. Somehow, you get at the ‘VendorSpecific’ structure of the SMART data using WMI and then you believe that byte number 115 is the one that holds the temperature. Of course that’s not what someone who’s doing protocol in their day-job would ever settle for. So I’ve been digging around a little and found a description of the structure and grabbed the attribute value list from Wikipedia, shook it all up a little and out came the little program below.

The app grabs the vendor specific array from the ATAPI data, shreds it into a set of structures, and dumps it out. Code here, zip file at the bottom.

// (c) Microsoft Corporation

// Author: Clemens Vasters (clemensv@microsoft.com)

// Code subject to MS-PL: http://opensource.org/licenses/ms-pl.html

// SMART Attributes and Background: http://en.wikipedia.org/wiki/S.M.A.R.T.

// SMART Attributes Overview: http://www.t13.org/Documents/UploadedDocuments/docs2005/e05171r0-ACS-SMARTAttributes_Overview.pdf

namespace SmartDataApp

    using System;

    using System.Collections.Generic;

    using System.Management;

    using System.Runtime.InteropServices;

    public enum SmartAttributeType : byte

        ReadErrorRate = 0x01,

        ThroughputPerformance = 0x02,

        SpinUpTime = 0x03,

        StartStopCount = 0x04,

        ReallocatedSectorsCount = 0x05,

        ReadChannelMargin = 0x06,

        SeekErrorRate = 0x07,

        SeekTimePerformance = 0x08,

        PowerOnHoursPOH = 0x09,

        SpinRetryCount = 0x0A,

        CalibrationRetryCount = 0x0B,

        PowerCycleCount = 0x0C,

        SoftReadErrorRate = 0x0D,

        SATADownshiftErrorCount = 0xB7,

        EndtoEnderror = 0xB8,

        HeadStability = 0xB9,

        InducedOpVibrationDetection = 0xBA,

        ReportedUncorrectableErrors = 0xBB,

        CommandTimeout = 0xBC,

        HighFlyWrites = 0xBD,

        AirflowTemperatureWDC = 0xBE,

        TemperatureDifferencefrom100 = 0xBE,

        GSenseErrorRate = 0xBF,

        PoweroffRetractCount = 0xC0,

        LoadCycleCount = 0xC1,

        Temperature = 0xC2,

        HardwareECCRecovered = 0xC3,

        ReallocationEventCount = 0xC4,

        CurrentPendingSectorCount = 0xC5,

        UncorrectableSectorCount = 0xC6,

        UltraDMACRCErrorCount = 0xC7,

        MultiZoneErrorRate = 0xC8,

        WriteErrorRateFujitsu = 0xC8,

        OffTrackSoftReadErrorRate = 0xC9,

        DataAddressMarkerrors = 0xCA,

        RunOutCancel = 0xCB,

        SoftECCCorrection = 0xCC,

        ThermalAsperityRateTAR = 0xCD,

        FlyingHeight = 0xCE,

        SpinHighCurrent = 0xCF,

        SpinBuzz = 0xD0,

        OfflineSeekPerformance = 0xD1,

        VibrationDuringWrite = 0xD3,

        ShockDuringWrite = 0xD4,

        DiskShift = 0xDC,

        GSenseErrorRateAlt = 0xDD,

        LoadedHours = 0xDE,

        LoadUnloadRetryCount = 0xDF,

        LoadFriction = 0xE0,

        LoadUnloadCycleCount = 0xE1,

        LoadInTime = 0xE2,

        TorqueAmplificationCount = 0xE3,

        PowerOffRetractCycle = 0xE4,

        GMRHeadAmplitude = 0xE6,

        DriveTemperature = 0xE7,

        HeadFlyingHours = 0xF0,

        TransferErrorRateFujitsu = 0xF0,

        TotalLBAsWritten = 0xF1,

        TotalLBAsRead = 0xF2,

        ReadErrorRetryRate = 0xFA,

        FreeFallProtection = 0xFE,

    public class SmartData

        readonly Dictionary<SmartAttributeType, SmartAttribute> attributes;

        readonly ushort structureVersion;

        public SmartData(byte[] arrVendorSpecific)

            attributes = new Dictionary<SmartAttributeType, SmartAttribute>();

            for (int offset = 2; offset < arrVendorSpecific.Length; )

                var a = FromBytes<SmartAttribute>(arrVendorSpecific, ref offset, 12);

                // Attribute values 0x00, 0xfe, 0xff are invalid

                if (a.AttributeType != 0x00 && (byte)a.AttributeType != 0xfe && (byte)a.AttributeType != 0xff)

                    attributes[a.AttributeType] = a;

            structureVersion = (ushort)(arrVendorSpecific[0] * 256 + arrVendorSpecific[1]);

        public ushort StructureVersion

get

                return this.structureVersion;

        public SmartAttribute this[SmartAttributeType v]

get

                return this.attributes[v];

        public IEnumerable<SmartAttribute> Attributes

get

                return this.attributes.Values;

        static T FromBytes<T>(byte[] bytearray, ref int offset, int count)

            IntPtr ptr = IntPtr.Zero;

try

                ptr = Marshal.AllocHGlobal(count);

                Marshal.Copy(bytearray, offset, ptr, count);

                offset += count;

                return (T)Marshal.PtrToStructure(ptr, typeof(T));

            finally

                if (ptr != IntPtr.Zero)

                    Marshal.FreeHGlobal(ptr);

    [StructLayout(LayoutKind.Sequential)]

    public struct SmartAttribute

        public SmartAttributeType AttributeType;

        public ushort Flags;

        public byte Value;

        [MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]

        public byte[] VendorData;

        public bool Advisory

get

                return (Flags & 0x1) == 0x0; // Bit 0 unset?

        public bool FailureImminent

get

                return (Flags & 0x1) == 0x1; // Bit 0 set?

        public bool OnlineDataCollection

get

                return (Flags & 0x2) == 0x2; // Bit 0 set?

    public class Program

        public static void Main()

try

                var searcher = new ManagementObjectSearcher("root\\WMI", "SELECT * FROM MSStorageDriver_ATAPISmartData");

                foreach (ManagementObject queryObj in searcher.Get())

                    Console.WriteLine("-----------------------------------");

                    Console.WriteLine("MSStorageDriver_ATAPISmartData instance");

                    Console.WriteLine("-----------------------------------");

                    var arrVendorSpecific = (byte[])queryObj.GetPropertyValue("VendorSpecific");

                    // Create SMART data from 'vendor specific' array

                    var d = new SmartData(arrVendorSpecific);

                    foreach (var b in d.Attributes)

                        Console.Write("{0} :{1} : ", b.AttributeType, b.Value);

                        foreach (byte vendorByte in b.VendorData)

                            Console.Write("{0:x} ", vendorByte);

                        Console.WriteLine();

            catch (ManagementException e)

                Console.WriteLine("An error occurred while querying for WMI data: " + e.Message);

SmartDataProgram.zip (2.25 KB)

What I do at work – Cloud and Service Bus for Normal People

clemensv — Fri, 18 Mar 2011 21:06:43 GMT

Why would anyone ever use a (Message) Queue?

clemensv — Fri, 18 Mar 2011 21:04:39 GMT

For programmers writing distributed systems and are not using queues in them just yet. If you are a message-oriented middleware veteran - move along ;-)

Windows Azure AppFabric October 2010 CTP - Service Bus Samples from my PDC Talk (Part 1)

clemensv — Wed, 03 Nov 2010 07:05:27 GMT

In my PDC talk I’m showing a few little samples that are not in the SDK for the CTP release. I’ve bundled them up and they’re attached to this post.

The solution contains three samples:

SBMT is the ‘Service Bus Management Tool’, a small command line utility to create/view/delete ConnectionPoints and MessageBuffers using the new management model for Service Bus. I’ll explain the options of the tool below and will drill into the inner workings of the tool and the ideas behind the new management interface in the next post of this series.
EchoService/EchoClient are versions of the meanwhile probably familiar ‘Echo’ sample for Service Bus that illustrates the new load balancing capability with session affinity.
DurableMessageBufferSample is a variation of the SDK sample for the new durable message buffer that sends a sequence of 200 messages and fetches them back.

The SBMT project/tool is the probably most interesting one, because we’re looking to change the way how connection points are managed on Service Bus and that requires a tiny bit of tooling.

In the current version of Service Bus, creating a listener is an implicit operation. The service (listener) picks a name (or address) in the namespace, binds that name to an endpoint, and opens the ServiceHost holding the endpoint. That’s extremely convenient, but what we’ve found is that this kind of implicit management makes it hard for us to create consistent behavior for eventing (NetEventRelayBinding) and for one of our most-requested ‘missing’ features, which is load balancing and/or failover. For eventing, the majority of customers that we’ve heard feedback from is expecting the the fabric will absorb sent events with no error message even if no event subscribers are present. For the new load balancing capability we’d like to appropriately return an error message in the case that no listener is connected, but differentiate that from the case where the connection point isn’t known, at all.

The implicit model causes no connection point to exist when there are no listeners, so we can’t solve either of these puzzles without making a change. The good thing is that we’ve got a model that works quite well and we’re building on that: Our message buffers already need to be created using an explicit AtomPub operation on the namespace, so what we’re doing is to make connection points explicit by requiring that you create them beforehand just as you create message buffers. We believe that’s the right model long-term, especially also because the <MessageBuffer/> and <ConnectionPoint/> descriptions are going to have more siblings over time and we’d like to have a consistent model. If you take a deeper look at the slides for the PDC talk, you can already infer a few of the siblings.

In the talk I also give a few reasons for why we’re splitting off the management and runtime namespaces as we’re doing this. It turns out that our longer term plans for how to enable rich monitoring and control for connection points and message buffers and other messaging artifacts on Service Bus will require quite a bit space in terms of protocol surface and that’s difficult to create in a single namespace. If I have a connection point at sb://clemens.servicebus.appfabriclabs.com/foo, any right-side extensions of that URI belong to the listener. If we were now looking to give you a protocol surface to enumerate the current sessions that are active and suspend/terminate the individually or summarily, for instance, it’s difficult to get that slotted into the same place without resorting to stunts like special escape characters in URIs. Instead, we’d like to provide a straightforward mechanism where you can get at the session connection with https://clemens-mgmt.servicebus.appfabriclabs.com/Resources/ConnectionPoints(foo)/Sessions/.

If that begs the question how those two addresses are related – I explain it in the talk. The resource has a projection into the runtime namespace. Resources are organized by their type (there will be other taxonomy pivots in the future) and you can project them into the runtime address space as you see it fit.

The SBMT tool allows interacting with the management namespace to create connection points and message buffers. The call structure is:

sbmt.exe –n <namespace> –a <account-name>–s <account-secret> <type> <command> <resource-name> <uri> [<args> ..]</

-n <namespace> - the Service Bus namespace created via http://portal.appfabriclabs.com
-a <account-name>– the service account name that’s set up in Access Control (defaults to ‘owner’)
-s <account-secret>– the service account key set up in Access Control (the key you find on the portal)
<type>– ‘cp’ for connection points, ‘mb’ for message buffers
<command>– ‘create’, ‘get’, or ‘delete’ (see below)
<resource-name>– a unique, friendly name for the resource (eg. ‘crm-leads’)
<uri>– the URI projection into the runtime namespace (e.g. ‘sb://<NAMESPACE>.servicebus.appfabriclabs.com/crm/leads/’)
<args>– further arguments (see below)

Examples:

Create a connection point. ‘cp create’ requires a trailing argument after the URI that indicates the number of concurrent listeners (max. 20):
sbmt.exe –n clemens –a owner –s […] cp create crm-leads sb://clemens.servicebus.appfabriclabs.com/crm/leads/ 10
Delete a connection point:
sbmt.exe –n clemens –a owner –s […] cp delete crm-leads
Show a connection point:
sbmt.exe –n clemens –a owner –s […] cp get crm-leads
Enumerate all connection points:
sbmt.exe –n clemens –a owner –s […] cp get
Create a message buffer:
sbmt.exe –n clemens –a owner –s […] mb create jobs https://clemens.servicebus.appfabriclabs.com/jobs/ 10
Delete a message buffer:
sbmt.exe –n clemens –a owner –s […] mb delete jobs
Show a message buffer:
sbmt.exe –n clemens –a owner –s […] mb get jobs
Enumerate all message buffers:
sbmt.exe –n clemens –a owner –s […] mb get

The Echo sample requires that you create a connection point in your namespace like this:

sbmt.exe –n –a owner –s […] cp create cp1 sb://<namespace>.servicebus.appfabriclabs.com/services/echo/ 10

The Message Buffer sample requires this setup, whereby the friendly name is asked for by the sample itself (‘mb1’, so you can choose that freely)

sbmt.exe –n –a owner –s […] mb create mb1 https://<namespace>.servicebus.appfabriclabs.com/mybuf/

The good news here is that you do this exactly once for your namespace. There is no renewal. A connection point that’s created sticks around like an object in the file system.

In the next post I’ll drill into what happens at the protocol level.

Since this is a CTP, we’re keenly interested in your feedback on all the things we’re doing here – making connection points explicit, splitting the namespace, what’s the tooling experience you’d like. This tool here is obviously just something I wrote to get going on demos; you can probably see that having a durable namespace is begging for tooling and it’s one of the reasons we do it.

pdc10-servicebus-oct10-ctp.zip (24.65 KB)

PDC10 Windows Azure AppFabric - Service Bus Futures

clemensv — Fri, 29 Oct 2010 09:20:38 GMT

My PDC10 session is available online (it was pre-recorded). I talk about the new ‘Labs’ release that we released into the datacenter this week and about a range of future capabilities that we’re planning for Service Bus. Some of those future capabilities that are a bit further out are about bringing back some popular capabilities from back in the .NET Services incubation days (like Push and Service Orchestration), some are entirely new.

One important note about the new release at http://portal.appfabriclabs.com – for Service Bus, this is a focused release that provides mostly only new features and doesn’t provide the full capability scope of the production system and SDK. The goal here is to provide insight into an ongoing development process and opportunity for feedback as we’re continuing to evolve AppFabric. So don’t derive any implications from this release on what we’re going to do with the capabilities already in production.

Click here to go to the talk.

Windows Azure AppFabric Datacenter IP ranges

clemensv — Mon, 25 Oct 2010 08:11:08 GMT

We know that there’s a number of you out there who have outbound firewall rules in place on your corporate infrastructures that are based on IP address whitelisting. So if you want to make Service Bus or Access Control work, you need to know where our services reside. Below is the current list of where the services are deployed as of today, but be aware that it’s in the nature of cloud infrastructures that things can and will move over time.IP address whitelisting strategy isn’t really the right thing to do when the other side is a massively multi-tenant infrastructure such as Windows Azure (or any other public cloud platform, for that matter)

Asia (SouthEast): 207.46.48.0/20, 111.221.16.0/21, 111.221.80.0/20
Asia (East): 111.221.64.0/22, 65.52.160.0/19
Europe (West): 94.245.97.0/24, 65.52.128.0/19
Europe (North): 213.199.128.0/20, 213.199.160.0/20, 213.199.184.0/21, 94.245.112.0/20, 94.245.88.0/21, 94.245.104.0/21, 65.52.64.0/20, 65.52.224.0/19
US (North/Central): 207.46.192.0/20, 65.52.0.0/19, 65.52.48.0/20, 65.52.192.0/19, 209.240.220.0/23
US (South/Central): 65.55.80.0/20, 65.54.48.0/21, 65.55.64.0/20, 70.37.48.0/20, 70.37.64.0/18, 65.52.32.0/21, 70.37.160.0/21

Blobber– A trivial little tool for uploading/listing/deleting Windows Azure Blob Storage files

clemensv — Sat, 09 Oct 2010 17:56:06 GMT

There must be dozens of these things, but I didn’t find one online last week and I needed a tool like this to prep a part of my keynote demo in Poland last week – and thus I wrote one. It’s a simple file management utility that works with the Windows Azure Blob store. No whistles, no bells, 344 lines of code if you care to look, both exe and source downloads below, MS-PL license.

Examples:
   List all files from the 'images' container:
        Blobber.exe -o list -c images -a MyAcct -k <key>
    List all files matching *.jpg from the 'images' container:
        Blobber.exe -o list -c images -a MyAcct -k <key> *.jpg
    List all files matching *.jpg from the 'images' container (case-insensitive):
        Blobber.exe -o list -l -c images -a MyAcct -k <key> *.jpg
    Delete all files matching *.jpg from the 'images' container:
        Blobber.exe -o deletefile -l -c images -a MyAcct -k <key> *.jpg
    Delete 'images' container:
        Blobber.exe -o deletectr -c images -a MyAcct -k <key>
    Upload all files from the c:\pictures directory into 'images' container:
        Blobber.exe -o upload -c images -a MyAcct -k <key> c:\pictures\*.jpg
    Upload like above and include all subdirectories:
        Blobber.exe -o upload -s -c images -a MyAcct -k <key> c:\pictures\*.*
    Upload like above and convert all file names to lower case:
        Blobber.exe -o upload -l -s -c images -a MyAcct -k <key> c:\pictures\*.*
    Upload all files from the c:\pictures directory into 'images' container on dev storage:
        Blobber.exe -o upload -l -s -c images -d c:\pictures\*.*
Arguments:
    -o <operation> upload, deletectr, deletefile, list (optional, default:'list')
        -o list [options] -c <container> <relative-uri-suffix-pattern> (* and ? wildcards)
        -o upload [options] [-s] -c <container> <local-path-file-pattern>
        -o deletefile [options] -c <container> <relative-uri-suffix-pattern> (* and ? wildcards)
        -o deletectr [options] -c <container>
    -c <container> Container (optional, default:'files')
    -s Include local file subdirectories (optional, upload only)
    -p Make container public (optional)
    -l Convert all paths and file names to lower case (optional)
    -d Use the local Windows Azure SDK Developer Storage (optional)
    -b <baseUri> Base URI (optional override)
    -a <accountName> Account Name (optional if specified in config)
    -k <key> Account Key (optional if specified in config)

You can also specify your account infomation in blobber.exe.config and omit the -a/-k arguments.

Executable: Blobber-exe.zip (133.4 KB)
Source: Blobber-src.zip (5.41 KB)

[Update: I just find that I unintentionally used the same name as a similar utility from Codeplex: http://blobber.codeplex.com/. Sorry.]

Cloud Architecture: The Scheduler-Agent-Supervisor Pattern

clemensv — Mon, 27 Sep 2010 16:55:18 GMT

As our team was starting to transform our parts of the Azure Services Platform from a CTP ‘labs’ service exploring features into a full-on commercial service, it started to dawn on us that we had set ourselves up for writing a bunch of ‘enterprise apps’. The shiny parts of Service Bus and Access Control that we parade around are all about user-facing features, but if I look back at the work we had to go from a toy service to a commercial offering, I’d guess that 80%-90% of the effort went into aspects like infrastructure, deployment, upgradeability, billing, provisioning, throttling, quotas, security hardening, and service optimization. The lesson there was: when you’re boarding the train to shipping a V1, you don’t load new features on that train – you rather throw some off.

The most interesting challenge for these infrastructure apps sitting on the backend was that we didn’t have much solid ground to stand on. Remember – these were very early days, so we couldn’t use SQL Azure since the folks over in SQL were on a pretty heroic schedule themselves and didn’t want to take on any external dependencies even from close friends. We also couldn’t use any of the capabilities of our own bits because building infrastructure for your features on your features would just be plain dumb. And while we could use capabilities of the Windows Azure platform we were building on, a lot of those parts still had rough edges as those folks were going through a lot of the same that we went through. In those days, the table store would be very moody, the queue store would sometimes swallow or duplicate messages, the Azure fabric controller would occasionally go around and kill things. All normal – bugs.

So under those circumstances we had to figure out the architecture for some subsystems where we need to do a set of coordinated action across a distributed set of resources – a distributed transaction or saga of sorts. The architecture had a few simple goals: when we get an activation request, we must not fumble that request under any circumstance, we must run the job to completion for all resources and, at the same time, we need to minimize any potential for required operator intervention, i.e. if something goes wrong, the system better knows how to deal with it – at best it should self-heal.

My solution to that puzzle is a pattern I call “Scheduler-Agent-Supervisor Pattern” or, short, “Supervisor Pattern”. We keep finding applications for this pattern in different places, so I think it’s worth writing about it in generic terms – even without going into the details of our system.

The pattern foots on two seemingly odd and very related assumptions: ‘the system is perfect’ and ‘all error conditions are transient’. As a consequence, the architecture has some character traits of a toddler. It’s generally happily optimistic and gets very grumpy, very quickly when things go wrong – to the point that it will simply drop everything and run away screaming. It’s very precisely like that, in fact.

The first picture here shows all key pieces except the Supervisor that I’ll introduce later. At the core we have a Scheduler that manages a simple state machine made up of Jobs and those jobs have Steps. The steps may have a notion of interdependency or may be completely parallelizable. There is a Job Store that holds jobs and steps and there are Agents that execute operations on some resource. Each Agent is (usually) fronted by a queue and the Scheduler has a queue (or service endpoint) through which it receives reply messages from the Agents.

Steps are recorded in a durable storage table of some sort that has at least the following fields: Current State (say: Disabled, Active), Desired State (say: Disabled, Active), LockedUntil (Date/Time value), and Actor plus any step specific information you want to store and eventually submit with the job to the step agent.

When Things Go Right

The initial flow is as follows:

(1)a – Submit a new job into the Scheduler (and wait)
(2)a – The Scheduler creates a new job and steps with an initial current state (‘Disabled’) in the job store
(2)b – The Scheduler sets ‘desired state’ of the job and of all schedulable steps (dependencies?) to the target state (‘Active’) and sets the ‘locked until’ timeout of the step to a value in the near future, e.g. ‘Now’ + 2 minutes.
(1)b – Job submission request unblocks and returns

If all went well, we now have a job record and, here in this example, two step records in our store. They have a current state of ‘Disabled’ and a desired state of ‘Active’. If things didn’t go well, we’d have incomplete or partially wedged records or nothing in the job store, at all. The client would also know about it since we’ve held on to the reply until we have everything done – so the client is encouraged to retry. If we have nothing in the store and the client doesn’t retry – well, then the job probably wasn’t all that important, after all. But if we have at least a job record, we can make it all right later. We’re optimists, though; let’s assume it all went well.

For the next steps we assume that there’s a notion of dependencies between the steps and the second steps depends on the first. If that were not the case, the two actions would just be happening in parallel.

(3) – Place a step message into the queue for the actor for the first step; Agent 1 in this case. The message contains all the information about the step, including the current and desired state and also the LockedUntil that puts an ultimatum on the activity. The message may further contain an action indicator or arguments that are taken from the step record.
(4) – After the agent has done the work, it places a completion record into the reply queue of the Scheduler.
(5) – The Scheduler records the step as complete by setting the current state from ‘Disabled’ to ‘Active’; as a result the desired and the current state are now equal.
(6) – The Scheduler sets the next step’s desired state to the target state (‘Active’) and sets the LockedUntil timeout of the step to a value in the near future, e.g. ‘Now’ + 1 minute. The lock timeout value is an ultimatum for when the operation is expected to be complete and reported back as being complete in a worst-case success case. The actual value therefore depends on the common latency of operations in the system. If operations usually complete in milliseconds and at worst within a second, the lock timeout can be short – but not too short. We’ll discuss this value in more detail a bit later.
(7), (8), (9) are equivalent to (3), (4), (5).

Once the last step’s current state is equal to the current state, the job’s current state gets set to the desired state and we’re done. So that was the “99% of the time” happy path.

When Things Go Wrong

So what happens when anything goes wrong? Remember the principle ‘all errors are transient’. What we do in the error case – anywhere – is to log the error condition and then promptly drop everything and simply hope that time, a change in system conditions, human or divine intervention, or – at worst – a patch will heal matters. That’s what the second principle ‘the system is perfect’ is about; the system obviously isn’t really perfect, but if we construct it in a way that we can either wait for it to return from a wedged state into a functional state or where we enable someone to go in and apply a fix for a blocking bug while preserving the system state, we can consider the system ‘perfect’ in the sense that pretty much any conceivable job that’s already in the system can be driven to completion.

In the second picture, we have Agent 2 blowing up as it is processing the step it got handed in (7). If the agent just can’t get its work done since some external dependency isn’t available – maybe a database can’t be reached or a server it’s talking to spews out ‘server too busy’ errors – it may be able to back off for a moment and retry. However, it must not retry past the LockedUntil ultimatum that’s in the step record. When things fail and the agent is still breathing, it may, as a matter of courtesy, notify the scheduler of the fact and report that the step was completed with no result, i.e. the desired state and the achieved state don’t match. That notification may also include diagnostic information. Once the LockedUntil ultimatum has passed, the Agent no longer owns the job and must drop it. It must even not report failure state back to the Scheduler past that point.

If the agent keels over and dies as it is processing the step (or right before or right after), it is obviously no longer in a position to let the scheduler know about its fate. Thus, there won’t be any message flowing back to the scheduler and the job is stalled. But we expect that. In fact, we’re ok with any failure anywhere in the system. We could lose or fumble a queue message, we could get a duplicate message, we could have the scheduler die a fiery death (or just being recycled for patching at some unfortunate moment) – all of those conditions are fine since we’ve brought the doctor on board with us: the Supervisor.

The Supervisor

The Supervisor is a schedule driven process (or thread) of which one or a few instances may run occasionally. The frequency depends on much on the average duration of operations and the expected overall latency for completion of jobs.

The Supervisor’s job is to recover steps or jobs that have failed – and we’re assuming that failures are due to some transient condition. So if the system would expect a transient resource failure condition that prevented a job from completing just a second ago to be healed two seconds later, it’d depend on the kind of system and resource whether that’d be a good strategy. What’s described here is a pattern, not a solution, so it depends on the concrete scenario to get the timing right for when to try operations again once they fail.

This desired back-off time manifests in the LockedUntil value. When a step gets scheduled, the Scheduler needs to state how long it is willing to wait for that step to complete; this includes some back-off time padding. Once that ultimatum has passed and the step is still in an inconsistent state (desired state doesn’t equal the current state) the Supervisor can pick it up at any time and schedule it.

(1) – Supervisor queries the job store for any inconsistent steps whose LockedUntil value has expired.
(2) – The Supervisor schedules the step again by setting the LockedUntil value to a new timeout and submitting the step into the target actor’s queue
(3) – Once the step succeeds, the step is reported as complete on the regular path back to the Scheduler where it completes normally as in steps (8), (9) from the happy-path scenario above. If it fails, we simply drop it again. For failures that allow reporting an error back to the Scheduler it may make sense to introduce an error counter that round-trips with the step so that the system could detect poisonous steps that fail ‘forever’ and have the Supervisor ignore those after some threshold.

The Supervisor can pursue a range of strategies for recovery. It can just take a look at individual steps and recover them by rescheduling them – assuming the steps are implemented as idempotent operations. If it were a bit cleverer, it may consider error information that a cooperative (and breathing) agent has submitted back to the Scheduler and even go as far as to fire an alert to an operator if the error condition were to require intervention and then take the step out of the loop by marking it and setting the LockedUntil value to some longer timeout so it’s taken out of the loop and someone can take a look.

At the job-scope, the Supervisor may want to perform recovery such that it first schedules all previously executed steps to revert back to the initial state by performing compensation work (all resources that got set to active are getting disabled again here in our example) and then scheduling another attempt at getting to the desired state.

In step (2)b up above, we’ve been logging current and desired state at the job-scope and with that we can also always find inconsistent jobs where all steps are consistent and wouldn’t show up in the step-level recovery query. That situation can occur if the Scheduler were to crash between logging one step as complete and scheduling the next step. If we find inconsistent jobs with all-consistent steps, we just need to reschedule the next step in the dependency sequence whose desired state isn’t matching the desired state of the overall job.

To be thorough, we could now take a look at all the places where things can go wrong in the system. I expect that survey to yield that at as long we can successfully get past step (2)b from the first diagram, the Supervisor is always in a position to either detect that a job isn’t making progress and help with recovery or can at least call for help. The system always knows what its current intent is, i.e. which state transitions it wants to drive, and never forgets about that intent since that intent is logged in the job store at all times and all progress against that intent is logged as well. The submission request (1) depends on the outcome of (2)a/b to guard against failures while putting a job and its steps into the system so that a client can take corrective action. In fact, once the job record is marked as inconsistent in step (2)b, the scheduler could already report success back to the submitting party even before the first step is scheduled, because the Supervisor would pick up that inconsistency eventually.

The Magical Input Queue Of T (aka InputQueue)

clemensv — Sat, 11 Sep 2010 22:29:23 GMT

This post explains an essential class for asynchronous programming that lurks in the depths of the WCF samples: InputQueue<T>. If you need to write efficient server-side apps, you should consider reading through this and add InputQueue<T> to your arsenal.

Let me start with: This blog post is 4 years late. Sorry! – and with that out of the way:

The WCF samples ship with several copies of a class that’s marked as internal in the System.ServiceModel.dll assembly: InputQueue<T>. Why are these samples – mostly those implementing channel-model extensions – bringing local copies of this class with them? It’s an essential tool for implementing the asynchronous call paths of many aspects of channels correctly and efficiently.

If you look closely enough, the WCF channel infrastructure resembles the Berkeley Socket model quite a bit – especially on the server side. There’s a channel listener that’s constructed on the server side and when that is opened (usually under the covers of the WCF ServiceHost) that operation is largely equivalent to calling ‘listen’ on a socket – the network endpoint is ready for business. On sockets you’ll then call ‘accept’ to accept the next available socket connection from a client, in WCF you call ‘AcceptChannel’ to accept the next available (session-) channel. On sockets you then call ‘receive’ to obtain bytes, on a channel you call ’Receive’ to obtain a message.

Before and between calls to '’AcceptChannel’ made by the server-side logic, client-initiated connections – and thus channels – may be coming in and queue up for a bit before they handed out to the next caller of ‘AcceptChannel’, or the asynchronous equivalent ‘Begin/EndAcceptChannel’ method pair. The number of channels that may be pending is configured in WCF with the ‘ListenBacklog’ property that’s available on most bindings.

I wrote ‘queue up’ there since that’s precisely what happens – those newly created channels on top of freshly accepted sockets or HTTP request channels are enqueued into an InputQueue<T> instance and (Begin-)Accept is implemented as a dequeue operation on that queue. There are two particular challenges here that make the regular Queue<T> class from the System.Collections.Generic namespace unsuitable for use in the implementation of that mechanism: Firstly, the Dequeue method there is only available as a synchronous variant and also doesn’t allow for specifying a timeout. Secondly, the queue implementation doesn’t really help much with implementing the ListenBacklog quota where not only the length of the queue is limited to some configured number of entries, but accepting further connections/channels from the underlying network is also suspended for as long as the queue is at capacity and needs to resume as soon as the pressure is relieved, i.e. a caller takes a channel out of the queue.

To show that InputQueue<T> is a very useful general purpose class even outside of the context of the WCF channel infrastructure, I’ve lifted a version of it from one of the most recent WCF channel samples, made a small number of modifications that I’ll write about later, and created a little sample around it that I’ve attached to this post.

The sample I’ll discuss here is simulating parsing/reading IP addresses from a log-file and then performing a reverse DNS name resolution on those addresses – something that you’d do in a web-server log-analyzer or as the background task in a blog engine wile preparing statistics.

Reverse DNS name resolution is quite interesting since it’s embarrassingly easy to parallelize and each resolution commonly takes a really long time (4-5 seconds) –whereby all the work is done elsewhere. The process issuing the queries is mostly sitting around idle waiting for the response. Therefore, it’s a good idea to run a number of DNS requests in parallel, but it’s a terrible idea to have any of these requests execute as a blocking call and burning a thread. Since we’re assuming that we’re reading from a log file that requires some parsing, it would also be a spectacularly bad idea to have multiple concurrent threads compete for access to that file and get into each other’s way. And since it is a file and we need to lift things up from disk, we probably shouldn’t do that ‘just in time’ as a DNS resolution step is done, but there should rather be some data readily waiting for processing. InputQueue<T> is enormously helpful in such a scenario.

The key file of the sample code – the implementation of the queue itself aside – is obviously Program.cs. Here’s Main() :

static void Main(string[] args)

    int maxItemsInQueue = 10;

    InputQueue<IPAddress> logDataQueue = new InputQueue<IPAddress>();

    int numResolverLoops = 20;

    ManualResetEvent shutdownCompleteEvent = new ManualResetEvent(false);

    List<IPAddressResolverLoop> resolverLoops = new List<IPAddressResolverLoop>();

    Console.WriteLine("You can stop the program by pressing ENTER.");

We’re setting up a new InputQueue<IPAddress> here into which we’ll throw the parsed addresses from our acquisition loop that simulates reading from the log. The queue’s capacity will be limited to just 10 entries (maxItemsInQueue is the input value) and we will run 20 'resolver loops’, which are logical threads that process IP-to-hostname resolution steps.

    Console.WriteLine("You can stop the program by pressing ENTER.");

    // set up the loop termination callback

    WaitCallback loopTerminationCallback = o =>

        if (Interlocked.Decrement(ref numResolverLoops) == 0)

            shutdownCompleteEvent.Set();

};

    // set up the resolver loops

    for (int loop = 0; loop < numResolverLoops; loop++)

        // add the resolver loop 'i' and set the done flag when the

        // last of them terminates

        resolverLoops.Add(

            new IPAddressResolverLoop(

                logDataQueue, loop,

                loopTerminationCallback, null));

Next we’re kicking off the resolver loops – we’ll look at these in detail a bit later. We’ve got a ManualResetEvent lock object that guards the program’s exit until all these loops have completed and we’re going to set that to signaled once the last loop completes – that’s what the loopTerminationCallback anonymous method is for. We’re registering the method with each of the loops and as they complete the method gets called and the last call sets the event. Each loop gets a reference to the logDataQueue from where it gets its work.

   // set up the acquisition loop; the loop auto-starts

    using (LogDataAcquisitionLoop acquisitionLoop =

        new LogDataAcquisitionLoop(logDataQueue, maxItemsInQueue))

        // hang main thread waiting for ENTER

        Console.ReadLine();

        Console.WriteLine("*** Shutdown initiated.");

Finally we’re starting the acquisition loop that gets the data from the log file. The loop gets a reference to the logDataQueue where it places the acquired items and it’s passed the maxItemsInQueue quota that governs how many items may be read ahead into the queue. Once the user presses the ENTER key, the acquisition loop object is disposed by ways of exiting the using scope, which stops the loop.

    // shut down the queue; the resolvers will auto-close

    // as the queue drains. We don't need to close them here.

    logDataQueue.Shutdown();

    // wait for all work to complete

    shutdownCompleteEvent.WaitOne();

Lastly, the queue is shut down (by fittingly calling Shutdown). Shutdown closes the queue (all further enqueue operations are absorbed) and causes all pending readers for which no more entries are available on the queue to unblock immediately and return null. The resolver loops will complete their respective jobs and will terminate whenever they dequeue null from the queue. As they terminate, they call the registered termination callback (loopTerminationCallback from above) and that will eventually cause shutdownCompletedEvent to become signaled as discussed above.

The log-reader simulator isn’t particularly interesting for this sample, even though one of the goodies is that the simulation executes on an I/O completion port instead of a managed thread-pool thread – that’s another blog post. The two methods of interest are Begin/EndGetLogData – all that’s of interest here is that EndGetLogData returns an IPAddress that’s assumed to be parsed out of a log.

class IPAddressLogReaderSimulator

    public IAsyncResult BeginGetLogData(AsyncCallback callback, object data);

    public IPAddress EndGetLogData(IAsyncResult result);

The simulator is used internally by the LogDataAcquisitionLoop class – which we’ll drill into because it implements the throttling mechanism on the queue.

class LogDataAcquisitionLoop : IDisposable

    readonly IPAddressLogReaderSimulator ipAddressLogReaderSimulator;

    readonly InputQueue<IPAddress> logDataQueue;

    int maxItemsInQueue;

    int readingSuspended;

    bool shuttingDown;

    public LogDataAcquisitionLoop(InputQueue<IPAddress> logDataQueue, int maxItemsInQueue)

        this.logDataQueue = logDataQueue;

        this.maxItemsInQueue = maxItemsInQueue;

        this.shuttingDown = false;

        this.ipAddressLogReaderSimulator = new IPAddressLogReaderSimulator();

        this.ipAddressLogReaderSimulator.BeginGetLogData(this.LogDataAcquired, null);

The constructor sets up the shared state of the loop and kicks off the first read operation on the simulator. Once BeginGetLogData has acquired the first IPAddress (which will happy very quickly), the LogDataAcquired callback method will be invoked.

    void LogDataAcquired(IAsyncResult result)

        IPAddress address = this.ipAddressLogReaderSimulator.EndGetLogData(result);

        Console.WriteLine("-- added {0}", address);

        this.logDataQueue.EnqueueAndDispatch(address, this.LogDataItemDequeued);

        if (!this.shuttingDown && this.logDataQueue.PendingCount < this.maxItemsInQueue)

            this.ipAddressLogReaderSimulator.BeginGetLogData(this.LogDataAcquired, null);

        else

            // the queue will be at the defined capacity, thus abandon

            // the read loop - it'll be picked up by LogDataItemDequeued

            // as the queue pressure eases

            Interlocked.Exchange(ref this.readingSuspended, 1);

            Console.WriteLine("-- suspended reads");

The callback method gets the IPAddress and puts it into the queue – using the InputQueue<T>.EnqueueAndDispatch(T, Action) method. There are two aspects that are quite special about that method when compared to the regular Queue<T>.Enqueue(T) method. First, it does take a callback as the second argument alongside the item to be enqueued; second, the method name isn’t just Enqueue, it also says Dispatch.

When EnqueueAndDispatch() is called, the item and the callback get put into an internal item queue – that’s the ‘enqueue’ part. As we will see in context a bit later in this post, the ‘dequeue’ operation on the queue is the BeginDequeue/EndDequeue asynchronous method call pair. There can be any number of concurrent BeginDequeue requests pending on the queue. ‘Pending’ means that the calls – rather their async callbacks and async state – are registered in another queue internal to InputQueue<T> that preserves the call order. Thus, BeginDequeue always only puts the async callback and async state into that queue and returns afterwards. There is no thread spun or hung. That’s all it does.

As things go, the best opportunity to service a pending dequeue operation on a queue is when an item is being enqueued. Consequently, EnqueueAndDispatch() will first put the item into the internal queue and will then look whether there are registered waiters and/or readers – waiters are registered by ‘(Begin-)WaitForItem’, readers are registered by ‘(Begin-)Dequeue’. Since it’s known that there a new item in the queue now, the operation will iterate overall waiters and complete them – and does so by invoking their async callbacks, effectively lending the enqueue operation’s thread to the waiters. If there’s at least one pending reader, it’ll then pop a message from the head of the internal item queue and call the reader’s async callback, lending the enqueue operation’s thread to processing of the dequeue operation. If that just made your head spin – yes, the item may have been dequeued and processed as EnqueueAndDispatch returns.

There is an overload for EnqueueAndDispatch() that takes an extra boolean parameter that lets you cause the dispatch operation to happen on a different thread, and there is also a EnqueueWithoutDispatch() method that just won’t dispatch through and a standalone Dispatch() method.

The callback supplied to EnqueueAndDispatch(), here the LogDataItemDequeued method, is am Action delegate. The queue will call this callback as the item is being dequeued and, more precisely, when the item has been removed from the internal item queue, but just before it is returned to the caller. That turns out to be quite handy. If you take another look at the LogDataAcquired method you’ll notice that we’ve got two alternate code paths after EnqueueAndDispatch(). The first branch is called when the queue has not reached capacity and it’s not shutting down. When that’s so, we’re scheduling getting the next log item – otherwise we don’t. Instead, we set the readingSuspended flag and quit – effectively terminating and abandoning the loop. So how does that get restarted when the queue is no longer at capacity? The LogDataItemDequeued callback!

    void LogDataItemDequeued()

        // called whenever an item is dequeued. First we check

        // whether the queue is no longer full after this

        // operation and the we check whether we need to resume

        // the read loop.

        if (!this.shuttingDown &&

            this.logDataQueue.PendingCount < this.maxItemsInQueue &&

            Interlocked.CompareExchange(ref this.readingSuspended, 0, 1) == 1)

            Console.WriteLine("-- resuming reads");

            this.ipAddressLogReaderSimulator.BeginGetLogData(this.LogDataAcquired, null);

The callback gets called for each item that gets dequeued. Which means that we’ll get an opportunity to restart the loop when it’s been stalled because the queue reached capacity. So we’re checking here whether the queue isn’t shuttong down and whether it’s below capacity and if that’s so and the readingSuspended flag is set, we’re restarting the read loop. And that’s how the throttle works.

So now we’ve got the data from the log in the queue and we’re throttling nicely so that we don’t pull too much data into memory. How about taking a look at the DNS resolver loops that process the data?

class IPAddressResolverLoop : IDisposable

    readonly InputQueue<IPAddress> logDataQueue;

    readonly int loop;

    readonly WaitCallback loopCompleted;

    readonly object state;

    bool shutdown;

    public IPAddressResolverLoop(InputQueue<IPAddress> logDataQueue, int loop, WaitCallback loopCompleted, object state)

        this.logDataQueue = logDataQueue;

        this.loop = loop;

        this.loopCompleted = loopCompleted;

        this.state = state;

        this.logDataQueue.BeginDequeue(TimeSpan.MaxValue, this.IPAddressDequeued, null);

This loop is also implemented as a class and the fields hold shared that that’s initialized in the constructor. This loop also auto-starts and does so by calling BeginDequeue on the input queue. As stated above, BeginDequeue commonly just parks the callback and returns.

    void IPAddressDequeued(IAsyncResult ar)

        IPAddress address = this.logDataQueue.EndDequeue(ar);

        if (!this.shutdown && address != null)

            Console.WriteLine("-- took {0}", address);

            Dns.BeginGetHostEntry(address, this.IPAddressResolved, new object[] { Stopwatch.StartNew(), address });

        else

            this.loopCompleted(this.state);

As an IPAddress is becomes available on the queue, the callback is being invoked and that’s quite likely on a thread lent by EnqueueAndDispatch() and therefore sitting on the thread the log file generator is using to call back for completion of the BeginGetLogData method if you trace things back. If we get an address and the value isn’t null, we’ll then proceed to schedule the DNS lookup via Dns.BeginGetHostEntry. Otherwise we’ll terminate the loop and call the loopCompleted callback. In Main() that’s the anonymous method that counts down the loop counter and signals the event when it falls to zero.

    void IPAddressResolved(IAsyncResult ar)

        var args = ((object[])ar.AsyncState);

        var stopwatch = (Stopwatch)args[0];

        var address = (IPAddress)args[1];

        stopwatch.Stop();

        double msecs = stopwatch.ElapsedMilliseconds;

try

            IPHostEntry entry = Dns.EndGetHostEntry(ar);

            Console.WriteLine("{0}: {1} {2}ms", this.loop, entry.HostName, msecs);

        catch (SocketException)

            // couldn't resolve. print the literal address

            Console.WriteLine("{0}: {1} {2}ms", this.loop, address, msecs);

        // done with this entry, get the next

        this.logDataQueue.BeginDequeue(TimeSpan.MaxValue, this.IPAddressDequeued, null);

The IPAddressResolved method just deals with the mechanics of printing out the result of the lookup and then schedules another BeginDequeue call to start the next iteration.

Summary: The enabler for and the core piece of the implementation of this scenario is InputQueue<T> – the dequeue-callback enables implementing throttling effectively and the dispatch logic provides an efficient way to leverage threads in applications that leverage asynchronous programming patterns, especially in I/O driven situations as illustrated here.

And last but not least – here’s teh codez; project file is for VS2010, throw the files into a new console app for VS2008 and mark the project to allow unsafe code (for the I/O completion thread pool code).

UsingInputQueue.zip (13.85 KB)

or if you'd rather have a version of InputQueue that is using the regular thread pool, download the WCF samples and look for InputQueue.cs.

[The sample code posted here is subject to the Windows SDK sample code license]

Programming WCF Services, Third Edition

clemensv — Fri, 10 Sep 2010 18:10:05 GMT

Juval Löwy’s very successful WCF book is now available in its third edition – and Juval asked me to update the foreword this time around. It’s been over three years since I wrote the foreword to the first edition and thus it was time for an update since WCF has moved on quite a bit and the use of it in the customer landscape and inside of MS has deepened where we’re building a lot of very interesting products on top of the WCF technology across all businesses – not least of which is the Azure AppFabric Service Bus that I work on and that’s entirely based on WCF services.

You can take a peek into the latest edition at the O’Reilly website and read my foreword if you care. To be clear: It’s the least important part of the whole book :-)

Twitter

clemensv — Wed, 08 Sep 2010 16:20:17 GMT

For the (blog-) record - if you'd ever be looking for me on Twitter (where I'm sadly writing tons more than here), I'm @clemensv over there :-)

Windows Azure AppFabric Overview session from TechEd Australia 2010

clemensv — Wed, 08 Sep 2010 16:08:52 GMT

In case you need a refresher or update about the things me and our team work on at Microsoft, go here for a very recent and very good presentation by my PM colleague Maggie Myslinska from TechEd Australia 2010 about Windows Azure AppFabric with Service Bus demos and a demo of the new Access Control V2 CTP

TechEd: ASI204 Windows Azure Platform AppFabric Overview

clemensv — Tue, 01 Jun 2010 12:15:06 GMT

Room 398, Tuesday June 8
3:15pm-4:30pm

Session Type: Breakout Session
Track: Application Server & Infrastructure
Speaker(s): Maggie Myslinska
Level: 200 – Intermediate

Come learn how to use Windows Azure AppFabric (with Service Bus and Access Control) as building block services for Web-based and hosted applications, and how developers can leverage services to create applications in the cloud and connect them with on-premises systems.

If you are planning on seeing Juval’s and my talk ASI304 at TechEd and/or if you need to know more about how Windows Azure AppFabric enables federated cloud/on-premise applications and a range of other scenarios, you should definitely put Maggie’s talk onto your TechEd schedule as well.