Sehmi-Conscious Thoughts : CCR/DSS

ARCast.TV - Nice Example of CCR Adoption in the Enterprise

asehmi — Thu, 18 Sep 2008 15:52:14 GMT

Over on ARCast.TV, Bob Familiar interviews Stephen Tarmey of Tyco Software House on how he discovered the Concurrency and Coordination Runtime (CCR) in the Robotics Studio SDK and used it to implement a high-throughput asynchronous messaging I/O pattern. This is a very common problem scenario that a number of enterprises are discovering can be solved rather elegantly with CCR.

ARCast.TV - Stephen Tarmey of Tyco on adopting Robotics Studio CCR for High Performance Async IO

Also, see my earlier post on CCR/DSS Use Cases in the Enterprise for some ideas. Watch this space as we'll be publishing some public case studies soon.

Gorgeous Fractals with VPL and Distributed DSS Services

asehmi — Mon, 23 Jun 2008 08:43:21 GMT

I think Angel is fast becoming one of VPL's biggest fans. After discussing some ideas with him about simple Map-Reduce scenarios for his agent toolkit and a grid system that I'm building, he went ahead and did it. Check out his distributed Mandelbrot fractal generator. Gorgeous results. A bit over-engineered as a fractal calculator, but that wasn't the point really!

You can get Angel's blog, agent toolkit and sample applications here.

Using the Visual Programming Language (VPL) as a Deployment Tool

asehmi — Mon, 16 Jun 2008 11:43:48 GMT

When building applications using CCR, DSS and VPL one doesn't need to know a priori the precise order of data and message flows between services. Many of the VPL examples provided with the MRDS show services already 'wired' up which can give the impression that one needs to know this information and design one's set of services to be orchestrated as such up front. What if you don't know this information up front but still want to use the VPL? Well, there's the pub/sub pattern supported by DSS out of the box which is very handy and well-understood. Or one can define a 'host' service that acts as a message forwarding system based on its own service directory (perhaps using the one built into DSS). Or one can use an external configuration file that defines the orchestration itself (like a BPML specification, say). Additionally, one could use UPnP device (service) discovery.

But orchestration is not the only thing, what about deployment? Well, the VPL also supports service deployment and this works across multiple physical nodes too!

Angel Lopez has extended his Web Crawler sample and without knowing the orchestration a priori uses VPL simply to create multiple instances of his several service 'agents'. The actual 'orchestration' is emergent as a result of collaboration between his agents and the 'agent host' - in fact he can have several agent hosts in the same Web Crawler application. The agent hosts forward messages to service agents based on their capabilities inferred from a logical 'URI' which gets physically bound when the agents themselves register with their (machine) local hosts. Should more than one agent service offer the same capability, such as web crawling or page downloading, the agent host will automatically load balance across them.

An interesting benefit of using the new VPL (in the April 2008 CTP of MRDS) is its ability to deploy services on multiple DSS nodes which themselves can be hosted on physically different machines. Angel does this with his Web Crawler application and together with the application-level load balancing facility his sample offers a neat and simple outline of potentially powerful solutions that you can build using this approach. I particularly liked the lack of a priori orchestration and the use of VPL as a multi-node deployment tool.

Check out Angel's application here.

CCR/DSS Use Cases in the Enterprise

asehmi — Fri, 13 Jun 2008 00:40:18 GMT

I'm often asked what Enterprise customers are using CCR/DSS in serious, i.e. pretty significant, production deployments. I'm not at liberty to disclose the names of our customers without their permission - a few will be made public soon - but I can say there are quite a lot of application domains where this technology is being applied seriously. Here's a list of scenarios I mined from our internal list of CCR/DSS adoption customers in the Enterprise:

Very large scale web request load balancing & IO handling (online social networks)
Security systems (distributed sensor networks)
Package sorting system (real-time logistics monitoring and control)
Real time processing of seismic data in drilling control (industrial automation & control)
Supply Chain Modeling (simulation and control)
Law enforcement information intercept/inspection (large scale complex event processing)
Automated inventory inspection system in FMCG/retail scenarios (vision processing)
Financial trading systems (transaction flow routing and processing)
Business processes - there is an academic BPEL implementation on CCR
Scientific computing – MPI primitives concisely written as CCR patterns (exchange, shift, double shift)

Hopefully that's useful and piques your interest to look at this Microsoft Robotics (CCR/DSS) stuff deeper.

Web Crawler in VPL/DSS

asehmi — Mon, 26 May 2008 19:33:14 GMT

In another weekend project down in sunny Buenos Aires the prolific Angel "Java" Lopez has (re)written a web crawler sample app we had from another project so it can be used in the VPL in MRDS. Check it out here.

I had some minor Visual Studio version issues and in my case contract versioning issues getting it to work as is with MRDS April CTP and VS2008. The fixes are in Angel's blog comments.

Mexico Regional Architect Forum 2008

asehmi — Sat, 03 May 2008 18:27:32 GMT

Last week I had loads of fun delivering a CCR-DSS-Robotics presentation at the Mexico Regional Architects Forum (RAF) in Mexico City. There’s something wonderful about this technology that set many people abuzz and I had some great conversations and brainstorming with some customers over drinks at the evening cocktail party. I find there's a very strong connection to enterprise scenarios and I gave a number of examples of how CCR-DSS could be used in common enterprise situations. I only hope we can get permission to publish some of our internal case studies on how some big enterprise customers are using this stuff in pretty impressive ways.

Many thanks to Vlad Vinogradsky; Marc Mercuri; George Chrysanthakopoulos; Henrik Frystyk Nielsen and Charles Liptaak for their material which I mashed together! Vlad did a great job putting together a compendium of samples that illustrate the basics of programming against the CCR which was extremely useful. To gain the most out of it, you must read Jeffrey Richter's MSDN magazine article on the CCR.

I also did a keynote presentation on Business Architecture and explained some of the recent fine-tuning being made to the MSBA methodology and I was able to demo a prototype of the new SQL-backed MSBA Entity Modeling Tool and Repository which replaces the unwieldy Excel-based heat maps and repository. My thanks here go to Mike Lloyd of Carbonflame who came up with a pretty neat Flex Index which helps contrast Capability models to Organizational Hierarchies, Process Optimisation and IT Infrastructure models. (Mike has been a long time proponent of MSBA since the days it was referred to a Microsoft Motion.)

I want to say what a pleasure it was for me to be in Mexico City which I thought was fabulous even though I saw very little of it. The food was superb and quite different to the commercialised Mexican food I have experienced in Europe and USA. The people I met are so easy going and friendly. Juan Lozada Villas, the D&PE Director, treated the RAF speakers to an evening at an authentic and apparently famous Mariachi restaurant (I think it was called Tarana?) which was awesome! I had an excellent education in Tequila from Jesus Hernandez Sanchez and tasted a number wonderful varieties. My favourite was Jose Cuervo 1800 Anejo and the Reposados generally... I bought three bottles at the airport Duty Free!

One last thing... Feliz Cumpleaños to Eduardo Nava Malagon! Sorry we didn't manage to sing to you during the keynote!

RoboChamps

asehmi — Fri, 02 May 2008 16:37:10 GMT

RoboChamps has launched!

I've been on the road in Latin America so am late to make this announcement as I was really busy. But better late than never as they say. Robotics = Services + Orchestration so it could prove really interesting - intellectually at the very least - to those of you who have not considered this area as useful as traditional application development in terms of learning about software design and architecture. It's time to prove yourself wrong.

Very briefly RoboChamps is:

A new simulation-based robotics league
New 3-D environments will be released every 1-2 months
Based on MRDS 2008
Targeted at ALL software developers - professionals, students, academics and hobbyists alike
A community site providing links, videos, samples, training and forums
Participants can win REAL robots

The Web site is chock full of great learning material and I especially like the tutorial videos, starter kits and the most excellent Chapter 3 on the CCR of Kyle John's upcoming book. I must admit I love all the bling too. :-)

Marc Mercuri has done a really outstanding job with this project - Thanks Marc!

Fun with Microsoft Robotics Studio

asehmi — Sun, 20 Apr 2008 22:50:19 GMT

Not bad for a Saturday project down in sunny Buenos Aires! My friend and professional geek Angel "Java" Lopez has recently rekindled his love of distributed computing (and Agents) through MRDS and specifically CCR/DSS. We're going to have a lot more fun with this stuff in the coming weeks. In the mean time check out his great little experiment and download the source code here.

Angel concludes:

All these tools opened my brain. Writing DSS service component is a new way of thinking and programming, something I want to explore for years: how to write objects with one-way message processing, in a distributed and parallel way. I envision some system that model Minsky ideas about the human mind.

Well, it’s a new brave world. Dorothy: we are not in Kansas anymore…

I couldn't agree more!

Microsoft Robotics Developer Studio 2008 CTP April Available

asehmi — Thu, 10 Apr 2008 10:33:29 GMT

The high level set of changes is listed below. For me #2 and #3 are golden, but be sure to drill down to find out the details... like async streaming support!

Improved security model with support for Message Level Authorization using Role-based access control allowing to fine-grained control of who can access services and how.
Support for LINQ queries in DSSP subscriptions enabling applications to create complex queries that can be executed close to the data source instead of requiring moving data across the network
Support for creating applications that run on multiple DSS nodes using Visual Programming Language and DSS Manifest Editor. This makes it much simpler to create applications that run across nodes, either on the same machine or across the network. When an application containing multiple nodes is to be started, VPL creates individual deploy packages for each node and fires them up across the network.
Support for record and playback functionality in Visual Simulation Environment enabling it to record the position and orientation of each entity and save that information to a file which later can be replayed.
Support for storing simulation media resources such as texture maps, bitmaps, and mesh files directly as part of the CLR assemblies in the form of embedded resources rather than as external files.
New Simulation floor plan editor which allows the user to put together walls, doors, windows, etc. to represent the interior of a room, house, or other building.

Quoting from Marc Mercuri's blog:

At RoboBusiness today, Microsoft announced the release of Microsoft Robotics Developer Studio 2008 CTP1.

Get the free download here -

http://www.microsoft.com/downloads/details.aspx?FamilyId=EB00C558-2163-45A5-BEFE-531AD48BC525&displaylang=en

Mary-Jo Foley has an article on why you should care ('Why business users should grab a copy of Microsoft's new robotics' toolkit')

See that here - http://blogs.zdnet.com/microsoft/?p=1325

The folks who attended the conference also saw some videos from my new project. CNET references it in the article - "Microsoft's new gig: Virtual host to robot competitions".

See that here - http://www.news.com/8301-10784_3-9914959-7.html.

Outside of this, mum's the word until the clock at www.robochamps.com hit's 0's across the board.

By the way, in addition to RoboChamps, Marc also planned and built Tafiti – the guy is smart!

For further information, you can also check out:

1. MSDN Developer Centre

2. MRDS Team Blog

3. MRDS News Feed

4. MRDS Community Forum

Multi-Agent AI - I've Got the Bug Again...

asehmi — Tue, 25 Mar 2008 00:11:14 GMT

It's Easter holiday and it's raining, snowing, sleeting, windy and generally undecided. So the family stayed in just hanging about, doing some reading, eating well and enjoying the odd bottle or two of vino. In a recent post I talked about The Microsoft Robotics Studio and it's sub components, CCR and DSS. I have to confess one of the reasons I'm excited about this technology is because I cut my programming teeth many years ago in the (then largely academic) field of Multi-Agent Systems (and Decentralized Computing) which involved very similar thinking and discussed many of the issues and patterns that I see in the Robotics space. Recently, I also started working in the Embedded Systems area and I hope to post some articles on how Embedded Systems and CCR/DSS come together in interesting distributed computing scenarios that are highly relevant in the enterprise!

Anyway, before I get ahead of myself, I just wanted to mention that through my reading in this holiday came across Mat Buckland's excellent book, Programming Game AI by Example, which very briefly mentioned the use of AI in Halo. Mat's site eventually led me to the Halo section on Bungie.org where I found a really eye-opening presentation by Jaime Griesemer (Halo level designer) and Chris Butcher (Halo AI programmer), which brought me back full circle to my Multi-Agent Systems programming days. In some ways I'm also reminded of the 'Agile Machine' we built in FABRIQ. Wow, what a fun trip down memory lane I'm having this holiday!

Distributed Computing Gets Spruced-Up with CCR/DSS!

asehmi — Wed, 27 Feb 2008 03:00:40 GMT

The reality of distributed systems development today is the complexity that eventually if not immediately bogs it down. Instead of being fun, exciting and easy to develop, distributed systems are often hard and unwieldy to design and work with. We continue to face the same old issues time and again yet never appear to come up with game-changing simplifying solutions to our common issues of complexity which, amongst others, are:

The large variety of participants: consumers, providers, intermediaries, administrators, etc.
Highly heterogeneous systems and architecture, implementation and deployment practices
Highly autonomous and independent systems & services
Highly distributed information sources and services

The challenges developers, architects, testers, etc. face are many. In general we need to be able to:

Apply (re)configuration in a live running system
Coordinate many simultaneous asynchronous interactions
Support high throughputs (and afford low latencies)
Provide simple lifecycle services like starting and stopping components dynamically
Monitor, interact with, and debug a running (distributed) system
Ideally be able to develop the system in simulated conditions or a test environment
Span multiple physically or logically distributed compute units
Re-use components across multiple hardware platforms & devices

A typical solution to these challenges might be to centralise & standardise, but that's a kind of anti-pattern for many types of distributed systems scenarios today and simply serves to:

Reduce the variety of systems, i.e. seeking economies of scale
Remove the positive benefits of heterogeneity in a distributed system
Eliminate the autonomy
Put demands on 'perfect information' which is really hard to do consistently and reliably
Hardwire the connections between components of the system

A better solution is to simply accept diversity and decentralize!

Accept variety, i.e. seeking economies of innovation
Encourage autonomy
Accept heterogeneity
Understand that that custom, personalized, networked information is normal
Imagine an adaptable, open, distributed information system

Recently I've been energized by the power of the Concurrency and Coordination Runtime (CCR) + Decentralized Software Services (DSS) combined features, and dare I say, paradigm shifting concepts to fix a lot of the problems we have today in building distributed systems.

CCR provides concurrent (multi-core) programming support by way of some key pattern implementations and developer conveniences
DSS provides higher level support to create, host, and discover lightweight services
DSS services are oriented to have a state and react to incoming request and updates, and sends update and request to other services
DSS services state can be serialized and a service can migrate to another host
DSS services are naturally RESTful
The Visual Programming Language designer can be used to define, compose, monitor and manage DSS services
CCR/DSS can be hosted in a device and is capable of running under .NET Compact Framework (and .NET Micro Framework coming soon)

Side Note - How does CCR/DSS relate to WCF?

WCF is more about communications infrastructure – standards-based ‘plumbing’ – whereas CCR/DSS is a high level programming model for distributed service-based and resource-oriented systems (however initially targeted to Robotics applications). Some key CCR features such as ports, memory queues, dispatchers, arbiters, etc. are hard to implement and beyond the scope of WCF. A better comparison would be the combo of WCF+WF, but still there would be many features and patterns missing from both sides. Apples to Oranges - just avoid trying to force a comparison and understand the core use cases for each technology set. In my opinion the patterns supported in CCR are also of immense value to WCF+WF developers. I wonder if anyone has implemented CCR patterns in WCF and furthermore provided a DSSP transport protocol binding for WCF? That would be really cool!

Resources

The Microsoft Robotics Studio includes the Concurrency and Coordination Runtime (CCR), Decentralized Software Services (DSS), Visual Programming Language (VPL), and a virtual worlds (robotics) simulation environment! If you want to dive straight into CCR/DSS check out the Wiki, watch a video or two and install Microsoft Robotics Studio.

As part of the Microsoft Robotics Studio (1.5) release, Microsoft has placed the Decentralized Software Services Protocol (DSSP) under its Open Specification Promise. This means that the Microsoft patents for the services-oriented protocol specification may be used by anyone in the world, at no cost, and for any type of development including free software, open source, academic, or commercial, enabling interoperability among a wide variety of devices and applications. The XSD schema for DSSP is also available.

MSRS Blog	Main forum page	DSS forum	CCR forum
VPL forum	Hardware configuration and troubleshooting forum	Simulation forum	Community examples forum

The creators of CCR/DSS are George Chrysanthakopoulos and Henrik Frystyk Nielsen. Search on their names and you'll be very impressed.

CCR/DSS is not just for Robotics - it'll help you spruce-up your enterprise distributed systems architectures too! Like I said earlier, I'm pretty jazzed by this technology and hope to be writing about it again very soon.