Watcher of the skies

Distributed Data-Parallel Computation in the Cloud: Dryad - Now Available for Academic Use

LifeOnTitan — Thu, 16 Jul 2009 20:45:00 GMT

Microsoft Research recently announced the availability, under Academic Licensing, of Dryad, an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs. A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without knowing anything about concurrent programming.

That's quite a statement! Well, when you think about what happens when you submit a query to a general purpose search engine like Bing, for example, you can imagine that what happens on the other side of the fence (the distributed search infrastructure) happens so quickly and efficiently because of highly parallel computations across many servers. So, you don't have to understand the details of search computation occuring behind the scenes to input a search term and get a bunch of results... DryadLINQ is similar from a programming perspective -> you create a LINQ query which, on the surface, is just a set of sequential query commands (but only on the surface...). The DryadLINQ compiler takes the resulting AST and creates a Dryad vertex topographical map that gets handed over to the Dryad runtime. "The computation is structured as a directed graph: programs are graph vertices, while the channels are graph edges. A Dryad job is a graph generator which can synthesize any directed acyclic graph. These graphs can even change during execution, in response to important events in the computation. Dryad is quite expressive. It completely subsumes other computation frameworks, such as Google's map-reduce, or the relational algebra. Moreover, Dryad handles job creation and management, resource management, job monitoring and visualization, fault tolerance, re-execution, scheduling, and accounting", say the Dryad people.

DryadLINQ is the managed high level programming abstraction (think LINQ to DistributedDataParallelComputation :-)) used to compose Dryad vertex topology graphs that the Dryad infrastructure uses to partition, manage and schedule parallel computations.

In essence, Dryad and DryadLINQ enable a sequential programming experience over what will execute across potentially thousands of machines (depending upon the computational complexity of the program) concurrently. There's a good introductory piece on Dryad/DryadLINQ over on Channel 9 that covers the basics and provides a glimpse into some of the thinking behind the thinking...

In the near future, Channel 9 will present a Going Deep episode that digs into the architecture and composition of Dryad with one of the scientists who designed and implemented the system.

Collaboration in Vista

LifeOnTitan — Fri, 24 Feb 2006 05:00:00 GMT

Check out this interview on Channel 9 that covers the new Collaboration Technologies in Vista. Really cool stuff.

A Casual Conversation with Bill Gates

LifeOnTitan — Sat, 18 Feb 2006 04:53:00 GMT

I recently interviewed Bill Gates on Channel 9. He talks openly about IE, the future of software, and even where he surfs on the web! It was an honor to meet him. He's a really down to earth person and very nice. You'd never guess he's worth 48 billion dollars. Enjoy!

Deep Vista Audio Stack and API video interview

LifeOnTitan — Wed, 14 Dec 2005 05:49:00 GMT

I recently caught up with seasoned Niner (Channel 9 member), Larry Osterman, an SDE and 20 year Microsoft veteran, and Elliot H Omiya, a Software Architect and audio guru, to dig into the innerworkings of Vista's updated Audio Stack and new user mode API in the latest installment of the Going Deep series on Channel 9. Much of the guts of Windows audio have been moved up into the land of the user and this has consequences for both Windows audio developers at the API level and for Windows at the general programmability, reliability and stability levels.

Concurrency and Coordination Runtime (CCR)

LifeOnTitan — Wed, 07 Dec 2005 23:20:00 GMT

We just released a great video interview on Channel 9 that should be of tremendous interest to those of you who write managed threaded code: The Concurrency and Coordination Runtime (CCR) is a lightweight port-based concurrency library for C# 2.0 developed by George Chrysanthakopoulos in the Advanced Strategies group at Microsoft. Here, we have a deep discussion about CCR with George, a Software Architect, and Satnam Singh, Architect. You can get more info about CCR on the CCR Wiki. This is super cool stuff and represents a really innovative approach to making managed threaded programming more readily understandable and predictable.

Please check out the OOPSLA/SCOOL paper on the CCR.

Enjoy!

Singularity Revisited

LifeOnTitan — Tue, 06 Dec 2005 02:34:00 GMT

Recently, I had a discussion with four of the reseachers behind MSR's Singularity, a research kernel written predominately in safe managed code (where possible...) that represents a highly innovative approach to Reliability and Security in operating systems. For example, the notion of shared memory does not exist in Singularity, nor does the ability to inject code into a running process. The notion of Channels as means for messaging is also highly interesting. I will head over to MSR again in about 6 months to check in on the status of Singularity. It will be interesting to see if there's anything besides a DOS-like shell to look at (not that I'm expecting there to be a UI or anything, but what about an application or two, each running in a SIP and communicating via Channels?). Stay tuned. They are doing some amazing stuff over there...

It will be really fascinating to see how Singularity evolves over time and we hope to be able to document it in a user-friendly way on Channel 9.

Coming soon to C9 is a chat Scoble and I had with a few of the folks from the Windows Kernel Architecture team. We discuss the future of Windows and an interesting future she has... Great stuff.

Channel 9 Shows

LifeOnTitan — Fri, 02 Dec 2005 04:54:00 GMT

We just released a new "feature" on Channel 9: Shows. The Show concept is a way to categorize tightly-bound video interviews, podcasts, etc by creating a "content encapsulation structure" that contains Episodes. You know, the same thing they do on commercial TV. So far, we have Videos and Podcasts as show mediums, with 3 Shows, one episode each:

Going Deep: Singularity Revisited

WM_IN: Angela Mills: From UDDI to Indigo

ArcTalk: DSL and Software Factories

The number of episodes for Going Deep and WM_IN will increase tomorrow. More Shows are on the horizon.

One of the cool things about Shows is the Showroom: You don't have to leave where you are to watch all the episodes of a Show. Oh yeah, another use of that cutting edge, five-year old technology they're calling AJAX... I won't start ranting here.

What kind of Show would you like to see?

Life: Beyond Carbon and Water

LifeOnTitan — Sat, 15 May 2004 04:02:00 GMT

We generally make the assumption that carbon and water are fundamental ingredients required for the advent of living systems in the universe. Given that all the data we have to work with is based on the behavior of biologic systems operating on the surface of a single planet, this conclusion, from a purely scientific perspective, is based on insufficient understanding. If we define life as a pattern of physical behavior that is independent of its specific physical and chemical ingredients, then our current universal view of life’s required chemistry and supporting environment is truly suspect. One consequence of this change in perception will be seen in the Sagan-Drake “equation” for the estimation of intelligent life in a galaxy.

The estimation of the number of worlds harboring intelligent life would increase due to an increase in the number of planets capable of supporting biological systems (n_e; see below) if we remove the carbon and liquid water requirements (which implies specific temperature ranges) for the advent and evolution of life. There is no good reason why Carbon and water should be deemed universally necessary ingredients for the advent of living systems up to and including life forms that are capable of thinking in unusally abstract ways. After all, thinking is an electrochemical process (so we think, anyway). There is, however, one obvious reason to think Carbon and water are required components for life. It's what we have found to be the case here on Earth. So, better to rephrase what we are looking for out there: Life like ours. That focuses the question. It's always easier to look for things when you know what you're looking for. Still, if you want to look for something as broad as the notion of life in the universe you certainly can't base your search criteria on what works for a single planet. On to the Sagan-Drake equation (which is necessarily composed of some rather subjective variables).

The Sagan-Drake equation:

N = R^*f_pn_ef_lf_if_cL

Where

N is the number of intelligent communicating civilizations in the galaxy at present

R* is the average rate of star formation in our galaxy (stars/year)

f_p is the fraction of stars that have planetary companions

n_e is the number of planets per planet-bearing star that have suitable ecospheres (that is, environmental conditions necessary to support the chemical evolution of life)

f_l is the fraction of planets with suitable ecospheres on which life actually starts

f_i is the fraction of planetary life starts that eventually evolve to intelligent life-forms

f_c is the fraction of intelligent civilizations that attempt interstellar communication

L is the average lifetime (in years) of technically advanced civilizations

The notion that microbial life is abundant in the universe is certainly a compelling possibility ( requires a high value for n_e ) if life is in fact an endemic planetary surface property with an evolutionary pattern that is tightly coupled to that of its planet. Perhaps the degree of bio-environmental coupling is a significant factor in determining if the development of intelligent complex life is possible. Certainly, if a planet harbors substantial life (present globally like here on Earth) then the evolution of the planet’s surface will be strongly coupled to that of its biology and the planet will maintain a surface environment capable of supporting life for periods of geologic time. Time, and lots of it, is a critical ingredient in advanced biological evolution. Or is it? I just ranted about the inherent problems with requiring Carbon and water for the advent and evolution of biological systems. Why this geologic time business? That's a good question. I don't have the answer. For now, let's just say that, regardless of specific chemistry and physics, it takes a long time for life starts to blossom into thinking creatures. I agree that this restriction may be too harsh.

Perhaps Mars is an example of a rocky planet that had sparse microbial life (relative to Earth) and therefore Martian biology had little net effect on the evolution of the Martian surface and atmosphere leading to a relatively short geobiologic lifespan and therefore no chance for the advent of complex life. Clearly, this is wild conjecture, but in the next 5 or so years we will probably know the answers to these Martian questions.

For detecting extraterrestrial life we should not only focus on whether or not carbon/water-based life forms can be supported on a rocky planet (geologically active and rocky surface (not a gas giant), like Earth, Mars, Venus and Saturn's wildly interesting moon, Titan), but whether or not a planet possesses surface properties that demonstrate a predictable pattern of behavior over time ( for example, a substantial and dynamic atmosphere (like the consistent addition and removal of Methane from Earth's or Titan's atmosphere) ) which is independent of the specific chemistry and physics operating on the geobiologic level. Now, if places like Jupiter's Europa support biological systems, then throw this paticular solution out of the nearest window since there is no way using this technique to remotely detect the presence of life that lives beneath the surface of a moon with no atmosphere in an ocean of salty water. It's certain that just looking at geologic patterns on the surface of a place like Europa will not provide enough evidence for the existence of life.

New place to talk to MicrosoftPeople and your peers: Channel 9

LifeOnTitan — Mon, 26 Apr 2004 20:35:00 GMT

Tired of waiting around for your favorite Microsoft Bloggers to post topics to their blogs that may or may not interest you? Want to initiate conversation with your peers and Microsoft Developers (and Testers and Program Managers and Architects and even Vice Presidents...) about aspects of Microsoft technologies that are important to you? Then come on down to Channel 9 and let's talk.

Charles

A few things I'd like to see change in Windows

LifeOnTitan — Thu, 25 Mar 2004 04:57:00 GMT

There really is no good reason to not allow users running under restricted privlege to view the system clock and calendar in Windows XP.

In general, I think we need to get away from the IT mindset when it comes to our consumer operating systems like XP Home and Pro, etc. The average user just doesn't get Admin, Power User, etc. In fact, most users run as Admin since that is the default context. This is a security risk. C'mon.

Reboots after install/modification suck. End of story.

App hangs are still too frequent in XP. I have to have Task Manager running in my tray... Speaking of which:

The process viewer in Task Manager is not very useful to the average user. In some cases, even the most experienced users have a hard time determining what running processes named svchost.exe or mdm.exe, for example, actually are; they could be anything! And I don't like that from a privacy perspective. Hey, I'm paranoid. What can I say?

Windows and .NET: We're making it simpler

LifeOnTitan — Thu, 05 Feb 2004 03:13:00 GMT

Let's create a new Window in Win32:

HWND hwndMain = CreateWindowEx(
     0, "MainWClass", "Main Window",
     WS_OVERLAPPEDWINDOW | WS_HSCROLL | WS_VSCROLL,
     CW_USEDEFAULT, CW_USEDEFAULT,
     CW_USEDEFAULT, CW_USEDEFAULT,
     (HWND)NULL, (HMENU)NULL, hInstance, NULL );

ShowWindow( hwndMain, SW_SHOWDEFAULT );

UpdateWindow( hwndMain );

Now let's do the same thing in “.NET“:

Window w = new Window();
w.Text = "Main Window";
w.Show();

Managed code and the .NET Framework is about elegance in simplicity. We are trying to make our future managed Windows APIs hold true to the notion of enabling you to solve complex problems simply. That's the future.

Charles

The value of Microsoft Research

LifeOnTitan — Wed, 04 Feb 2004 01:18:00 GMT

As mentioned in one of Brumme's past entries, more visibility (and kudos) is needed for the Microsoft Research division. This large, university-like component of Microsoft consistently produces so many amazing technologies that eventually end up in our products and in our development tools. Check out the MS Research web site for some examples of the truly innovative technologies like new programming languages and tools. Many are even available for public use. Amazing stuff. MS Research can also act as a fantastic technical resource (though one should refrain from asking basic questions like how to do X or Y in C#...). Even the most crazy theoretical questions that I've sent to Research Land have been answered with enthusiasm from interested researchers. I love working at this company! Charles

Dying Extra-Solar Gas Giant Planet Leaks Carbon-Oxygen

LifeOnTitan — Wed, 04 Feb 2004 01:01:00 GMT

Interesting article on a recent discovery of Carbon and Oxygen in the atmosphere of the dying gas giant Osiris.

Interplanetary Internet

LifeOnTitan — Thu, 29 Jan 2004 02:34:00 GMT

Fascinating, captain.

What do you think of .NET's exception model?

LifeOnTitan — Thu, 29 Jan 2004 01:38:00 GMT

How much you do you really know about the CLR exception model?
Do you know how Windows SEH works? Really?

Please read this article:

http://blogs.msdn.com/cbrumme/archive/2003/10/01/51524.aspx

Now, what do you think of our current managed exception model? What would you like to see in the next version of the CLR with respect to exception handling?

Do you ever write code like this? :

try { Compute(); } catch( Exception ex ) { //boo! (ex happens to be an Access Violation exception)}
I didn't think so. ;-) But if you do, you should make a point of not ever doing so again as code like the above is very dangerous since it can expose stack-based buffer overrun exploits. You'd be surprised how many times I see the above pattern.

Wouldn't it be nice if C# just supported exception filtering (like VB.NET and MC++) so you could have more granular control over how your application deals with runtime exceptions? If you've found yourself wanting this feature (or any other feature for that matter), just let the C# design people know!

More later.