This post has been moved to a Maestro-dedicated blog. Please direct all comments and questions to the new blog. Thanks!
Informative post, Joshua, thanks for this.
I'm interested to see where Maestro will go. Will it be like Spec#, where it starts out as a language but gets merged into a library available for all .NET languages?
Or will we have a future where we use Maestro in combination with our existing C# codebase?
Will be interesting to see where this goes. Keep the posts (and C9 vids) coming!
Re: Fortunately for us, a thread-safe queue can be implemented lock-free such that a single enqueue could translate to as little as a single interlocked operation
Just one question: are you going to maintain per-agent queue sizes in Maestro?
Judah,
Thanks for the questions. As an incubation project, Maestro could end up anywhere: as it's own language, as extensions to an existing language, or even on the cutting room floor. I'd recommend watching the Channel9 video. We talk a bit about the language vs. library thing. In fact, Maestro is really built on top of the CCR (a library for coordination and message-passing). Maestro is really about making concurrent program easier by making it safer and we feel that language provides us with the best tools for this, i.e. static verification that you're not breaking the no-implicit-dependencies model.
In it's current form, Maestro is a separate language that is object-aware but won't allow you to define classes. As such, it's nearly impossible to do anything non-trivial without using C# or another language to define your objects. Maestro is really about coordination -- we leave all the OO-stuff to the other languages.
Dmitriy,
I'm sorry, I'm not understanding the question. Can you explain what per-agent queue sizes are?
Thanks!
In message-passing system like Maestro inevitably must be message queues. Queues can be attached to agents/actors, or probably to Domains in Maestro. Basically, will these queues in Maestro contain 'GetLength()' member? Will it be possible to determine current queue length at run-time?
--
Dmitriy V'jukov
The main design guideline to simplify programming of multi-threading applications is to decouple the business logic (applicative programming) from the code that manage threads, locking, synchronization, ... (system programming)
So you can and use commercial, off-the-shelf components for the multi-threading.
Have you in plan to deliver components that implement the most common multi-threading service models (say Thread-per-request, Thread-per-Session, Thread-Pool, ...) ?
Thank you for submitting this cool story - Trackback from progg.ru
Maestro does have a series of interaction points (message queues) that have a Count property on them. In Maestro, you define channels which are a collection of ports that are transformed into these interaction points. Of course, in a concurrent application, the size of these ports do not have much use as the information can become stale as soon as you retrieve it. Just out of curiousity, how would you use the queue count?
Luca,
Maestro is a language that abstracts threads into agents. Agents, depending on their privileges, will either run serially or in parallel inside of a given domain. As such, while Maestro doesn't have explicit constructs that implement different multi-threading service models, it does make it easy to support such models. For example, say you had a service that simply communicated some sort of state and you wanted it to do so on a thread-per-request basis.
In super-simplified pseudo-Maestro-code, that would look something like this.
domain MyService
{
State _theState;
MyService()
IHost httpHost = GetHost("http");
httpHost.Host<StateGetterAgent>("http://myservice");
}
channel StateGetterChannel
input GetState : State;
reader agent StateGetterAgent : StateGetterChannel
while ( StillAlive() )
// get request
var request = receive(PrimaryChannel::GetState);
// send response
request::Reply <-- _theState
Everytime a client would access http://myservice, a new StateGetterAgent would be created inside the domain MyService. Since StateGetterAgent is a reader agent, it would execute in parallel with all other reader agents. I hope that helps.
Thanks both for the great questions!
Re: Just out of curiousity, how would you use the queue count?
I'm thinking about overload control. The idea is that it's impossible to create robust and efficient systems based on async message passing if there are no means to detect and control load at run-time.
Here is very demonstrative example:
http://blogtrader.net/page/dcaoyuan/entry/a_case_study_of_scalable
Count() method in queues provides a minimum needed to detect and control load - at least agent can check target agent's queue size and if it's too big spin while queue size will not decrease below low watermark.
What was caught my eyes is that you sad that enqueue operation is just one Interlocked operation. Are you using DWCAS (double-word CAS)? Or some more advanced techniques?
Btw, are you going to provide some built-in means for overload control? While Erlang provides ability to query process' queue size, it's neither efficient (user is unable to implement efficient load control mechanisms based only on Count() method), nor convenient (user will not implement any load control mechanisms until some bad things happen on client site). So my point is that async message passing library have to provide developed built-in means for load control. Here I've described some my thoughts on this (it's in Russian, but you can read only code - it's relatively self-explanatory):
http://groups.google.com/group/sobjectizer/tree/browse_frm/thread/b8b89bbd80cf8e7a/b6a0e1fdc1dcaff3
Please, also read the following article:
http://blog.rednael.com/2009/02/05/ParallelProgrammingUsingTheParallelFramework.aspx
It's an article about basic parallel programming. Examples in C# .Net included. Also, it describes a lightweight parallel framework to work with tasks. Opposed to some other frameworks, this one is very light and very easy to use.
After reading this article, you should be able to write code using parallelism.
Regards,
Martijn
Throttling is a bit of a double-edged sword. It is indeed important to create a robust system but on the other hand, blocking a producer when a consumer cannot accept data may halt progress and can poison the responsiveness Maestro strives to maintain.
As such, the ports defined on a channel can only be unbounded. However, as you'll hear about in the Channel 9 video, Maestro is built on top of a runtime that is in fact a modified version of the CCR. This runtime contains a number of interaction points that each behave a little differently. In fact, when you define a port on a channel, its transformed into a OrderedInteractionPoint<T> that is unbuffered. There is also an UnorderedInteractionPoint<T> a SingleItemInteractionPoint<T> and a WriteOnceInteractionPoint<T>. Since all of these are both sources and targets, you can send and receive to them just as you would a standard Maestro port. Additionally, the runtime also includes an interface (IInteractionPoint<T>) that you can use to build your own interaction point, including a bounded interaction point that supports throttling.
So while we don't have a blocking queue built in, throttling is supported via some other patterns, such as SingleItemInteractionPoint<T> and it will be very easy to roll your own BoundedInteractionPoint<T> for throttling producers.
Application that is swapped out to disk because of the memory pressure or already dead can't be very responsive too :)
And the problem here is that it's basically impossible to eliminate overloads "by design", asynchronous message-passing systems are just suspected to overloads.
Replacing unbounded queues with bounded blocking queues... well, I think I will not do this until I will want to show to someone what is a deadlock in a message-passing system :) Is queue interface general enough to allow developer to capture whole graph of blocked agents? Even if so, I will have periodically analyze the graph for cycles and unblock some agents... I don't think that it is a good advice for application developer...
What I am talking about is a much more flexible mechanism. Agent can choose one of the predefined policies: suspend sender, drop message, redirect message, enqueue message regardless of load. Also the sender can affect policy by setting "dont drop my messages" or "don't suspend me on overload" options. Default overload criterion is based on the high and low watermarks (can be set globally or on a per-agent basis). If default policies are not fit, developer is free to implement overload control policy manually, but still take advantage of automatic deadlock prevention mechanism, etc. System provides several places to hook in; for example, in the case of overload agent can periodically scan own queue and drop low-priority or outdated messages in order to release memory pressure.
You've mentioned web/internet as an example of async system, and network systems do have automatic overload detection and propagation. Sockets just become non-writable, thus overload conditions propagate through the system.
Isn't allowing a developer to implement an agent's message-queue how they see fit the most flexible option?
Or is your feedback more that it's inconvenient to have to do so? Again, some of these patterns are already supported by constructs that we have: for example, "enque message regardless of load" is the default, you can drop newer messages using a SingleItemInteractionPoint<T>, you can redirect message using networks.. The beauty of the interface is, you have the flexibility to implement a pattern that best supports your scenario. For example, some applications may want to drop a message and log the drop, some may not. Some applications may want to suspend the sender indefinitely, some may want to suspend the sender for only a small period. Providing the interface allows the implementer to choose what's best for the situation at hand.
Most importantly, we feel pretty stongly that these semantics should not be surfaced in the language, they should only be exposed in the library to keep the language simple. Also, to maintain loose coupling, we have to keep the contract between two agents very simple, i.e. the direction of the data, the type of the data, and the protocol of the communication. Just like in the web, a client doesn't care how a service deals with load, just that it does.
Another way of saying the same thing as Josh here, is that we want throttling behavior such as the various options Dimitriy describes to be a transport-layer decision, not baked into the language. In the most simple (and efficient) transport, i.e. the default in-process transport, there is no throttling.
Anyone can extend the underlying communication mechanism by writing a communication provider; for example, we have one that uses WCF for inter-process communication between Maestro agents. WCF has all kinds of support for configuration of behavior, and we think that's the layer such things belong at. At least, that's our current thinking.
I had expected that the first word on Maestro would come on this blog, but that's what happens when you
Josh, Niklas,
I think it's perfectly Ok to push overload control to transport layer (anyway it's impossible to solve all the problems simultaneously).
Nevertheless, it's still unclear to me whether interfaces will be flexible enough to support implementation of various overload control strategies. And I am talking NOT about implementation of queue, I am talking about system-wide overload control. For example, if I want to block/suspend a sender, will I be able to implement dead-lock prevention mechanism? Or will I be able to notify the sender that his message was dropped? Or will I be able to just notify the sender that he causes overload, so that he will be able to switch to special working mode? The latter case also requires detection of the fact that overload has dispersed, in order to notify senders, so that they will be able to switch back to normal working mode.
In general, these strategies require to be able to capture not only separate message, but the whole context - {message, sender, receiver, probably something else}. Is queue interface support this?
PingBack from http://blogs.techrepublic.com.com/programming-and-development/?p=891
Hi Dimitry,
Using just the interaction point you most certainly have references to the message, sender, and receiver. What's the something else?
PingBack from http://blogs.msdn.com/concurrently_speaking/archive/2009/02/23/maestro-and-isolation.aspx
Josh,
Thank you. Ok, I get the whole picture. I don't know that is something else yet :)
[1] Is it possible for messages within Maestro to survive a process failure/restart/crash since this is not currently supported in the CCR?
[2] If messages are persisted what is the persistence store (SQL Server, distributed memory cache etc)?
BajzT,
Great question. This is somewhat related to Dmitriy's throttling issue. To perform as well as possible, Maestro has a relatively basic message passing system. In shared-memory applications where failure is less likely, we want Maestro to be blazing fast, and persisting messages would deteriorate performance, perhaps unneccesarily for most applications. Also your second question highlights another good reason not to store messages by default: what would be the best way to store them?
As with Dmitriy's throttling problem, this can be solved by creating your own interaction point in Maestro (exactly as it can be solved in the CCR).
Also, keep in mind that just because a feature won't be in the language, doesn't mean we won't support it or even provide it. We're intentionally keeping the language as light as possible but we've yet to start really building up a framework around Maestro. No promises, but you might just see a BoundedInteractionPoint<T> or a PersistentInteractionPoint<T> in the future. :)
Joshua,
Persistence providers in the Maestro framework for SQL Server, Service Broker, MS Velocity, MSMQ etc would allow people to choose the right store for their problem domain. In my scenario coordination and state for example are more important than pure message throughput.
My next question might be a little premature but it seems it will be possible to run agents on multiple servers to scale out, load balance and failover message processing?
