Harry Pierson's DevHawk Weblog

Blog Moved...Sort Of

DevHawk — Mon, 21 Aug 2006 20:13:00 GMT

I originally started and have always maintained DevHawk as my primary blog. However, back in the spring of 2004, I started cross posting to this MSDN blog due since I was being featured on the MSDN Architecture Center and I needed the blog up 24x7.

Fast forward two years and I have a new role on a new team. Keeping two blogs is more trouble than it's worth, so I'm going to stop blogging here. Regular readers - or at least readers who wish to be regular - should head over to DevHawk, or pick up the RSS feed directly.

New Teammates Blogging

DevHawk — Mon, 17 Jul 2006 23:43:39 GMT

I'm settling in to my new job. One way to tell, read Dale Churchward's blog. Dale's a teammate of mine. He only joined Microsoft a few months ago. Apparently, he used to blog at his old job, but either way we've now doubled the number of bloggers on my new team, with hopefully more to follow.

In addition to his opinions of political discource and the Seahawks chances next season, Dale's got some interesting posts on data integration and system diagrams. Check it out.

Gartner EA Summit Day One

DevHawk — Thu, 22 Jun 2006 09:14:16 GMT

I'm in sunny San Diego for the Gartner Enterprise Architecture Summit. I'm presenting a sponsor session and case study tomorrow (MSFT is a platinum sponsor for the event) but I came in yesterday so I could attend a few sessions, meet a few customers and work the MS booth in the Solution Showcase. It's my last event and deliverable for my old team before switching to the new role full time.

The first of two keynotes today was Richard Buchanan's session on The New Enterprise Architecture: Time for Leadership. Pretty decent session, though much of it was pretty obvious. He even said at one point that this was "Strategic Enterprise Planning 101". Not exactly the best way to kick of an EA summit, IMHO. However, he did make some interesting points:

It's hard to quantify the value of business effectiveness. Richard's quote on this was great: "What’s the dollar value of staying out of jail?"
He compared most of how IT is operated today as "looking in the mirror" (i.e. focusing on running what we already have). He suggested instead "looking out the window" (i.e. at the industry and the future).
I always say that architecture is the intersection of business and IT. Richard said “Architecture is a translation from business strategy to technology implementation. Architects must institutionalize this translation.” Close enough.
He also suggested that architects need to learn to speak the language of business. That's good advice.
Richard did do a good job capturing the dynamic aspect of enterprise and IT architecture. It's about change, not structure. He said "EA is not about the past or the present. It’s about the future." Couldn't agree more.

The second keynote was Werner Vogels talk Order in the Chaos: Building the Amazon.com Platform. This was a great talk. I know a little about how Amazon has evolved, but I had no idea that it powered websites like Target and Bebe. His talk was a little scattered - I'm guessing he's not as used to speaking at events like this than the Gartner folks. There's no way to do the talk justice without basically repeating it verbatim, but my key takeaways were:

Amazon naturally evolved from an application into a platform. This is fascinating and worthy of more study, esp. as I'm making the switch to MSFT's internal IT department. Microsoft knows a thing or two about platforms, but I'm not sure how it applies inside IT.
Amazon sees it's "secret sauce" as their ability to automate operations at scale. For example, handling over a million sellers in their system. That helps explain their moves into services like S3 and MTurk which at first glance seems at odds with their retail web site.
One of the key values to becoming a platform is being able to open it up to partners. Again, Microsoft knows a lot about opening a platform to partners, but I'm not sure how it applies inside IT.
Money Quote: "At Amazon, things are always failing. It’s not a bad thing, it’s a fact of life." I've started theorizing about this on my own, good to know where to start looking for people putting this into practice.
Towards that end, he made probably the most interesting observation of the day. At Amazon, there is no wall between development and operations. Combined with the secret sauce of automating operations at scale and there is a good recipe for how enterprises need to run their IT department.

The final session I went to today was Nick Gall on Architecture for the Agile Enterprise: Integrating EA & SOA. The use of the term "agile" in this context was unfortunate, as he had no discussion of agile principles. He primarily focused on what he called Web Oriented Architecture or WOA. His formula for WOA was 'WOA = SOA + WWW + REST" which seems redundant. Isn't REST an attempt to capture the architectural style of the WWW? Anyway, this session wasn't very good. He had about 15 minutes of really good content but you had to wade thru the other 45 minutes of crap to find it. For example, he spent about ten minutes talking about the value of using a small set common modular operations (i.e. the REST / WS-Transfer approach) before he used this great analogy:

Modularity can be open or closed. Closed modularity is like a jigsaw puzzle. There are lots of individual pieces, but they can only be put together one way. Open modularity is like a tangram puzzle. There are only seven pieces, but they can be put together in hundreds of different combinations.

That was a great analogy that really got the point across! Why not just start with that and skip the mumbo jumbo?

I missed the last session as I had to prep for booth duty. Even though this audience is very different from a typical MSFT event like TechEd, they still mobbed the booth for swag and a chance to win an Xbox 360. I had a few interesting architectural discussion, but mostly it was about the swag.

My session is tomorrow at 11am. I'm presenting Beyond SOA and a case study session on the Dell Integrated Desktop . Then there's two more hours of booth duty tomorrow, but I'm hoping it's more content and less swag this time as 1) I will have just presented so I'm hoping to get some questions and 2) everyone has already gotten their swag ration for the conference.

Alpha Release of Ruby.NET

DevHawk — Wed, 21 Jun 2006 22:37:03 GMT

I'm not sure what a "Preliminary Beta" is but QIT has released a one for their Ruby.NET compiler. They expect to achieve "full semantic compatibility" (can you tell this is an acidemic project?) by the end of the year. Thanks to David Ing for the link.

They claim to "pass all 871 tests in the samples/test.rb installation test suite of Ruby 1.8.2." which seems odd since later they say "We have a plan for how to deal with continuations but we have not yet implemented them." Doesn't the Ruby test suite test continuations? I wish they would provide more details on this plan, continuations might not be the most interesting thing in Ruby, but it's up there and it's probably the hardest thing to implement on top of the CLR.

BTW, there are two other projects @ QIT that Ruby.NET leverages that look interesting. The Gardens Point Parser Generator is essentially a YACC clone written in C# and making extensive use of generics. Personally, I'm more interested in Parsing Expression Grammars, but there's no C# implementation as of yet. QIT also has a library for reading and writing program executable files (i.e. EXEs and DLLs).

As a quick aside, I'm getting pretty tired of all the different euphemisms for "alpha". In the age of perpetual beta, isn't alpha the new beta? But everyone seems worried about calling their releases alpha as if it means "it might not cause your machine to explode, if you could actually get it to compile". So we end up with things like "Preliminary Beta" and "Community Tech Preview". We all KNOW what these terms mean, so lets just call an alpha and alpha, shall we?

Moving On...

DevHawk — Thu, 15 Jun 2006 20:19:21 GMT

It's not the biggest job change news this week (or the day), but after three years on Architecture Strategy and six years total as an evangelist, I'm moving on to a new role. After six years, I decided it was time for me to put my money where my mouth is as well as get my hands dirty building something more substantial than buzz.

I'll be moving over into Microsoft's IT division as a member of the Integration Center of Excellence Architecture Team. Integration, as you might guess, is a euphemism here for service-orientation. My team is tasked with architecting and delivering the shared service-oriented infrastructure for four of the biggest projects Microsoft IT will be delivering in the next year. Last time I changed jobs, I lamented that "With each job I take at MSFT, coding seems to become less a part of the job description." Happily, this is NOT the case this time.

About a year ago, Microsoft hired Stuart Scott to run the business apps side of IT as one of our two CIOs (our other CIO Ron Markezich oversees the IT infrastructure). Stuart was kind enough to spend about an hour with me last week explaining his vision for how he sees MSIT evolving under his leadership. Here's what he said in a recent interview:

PressPass: How do you see Microsoft IT evolving?

Scott: There is a broader role for IT to play at the front end of the development of products and services. Our IT organization knows a lot about the challenges that other IT organizations face because we build and maintain the IT backbone of a massive worldwide enterprise. IT must become future-thought leaders in the development of the product roadmap for our enterprise products.

By using our internal applications and experiences to build better products for our enterprise customers, we have the potential to solve the challenges that other IT organizations face. We’re heavily involved in dogfooding our products once they’ve been developed, but we also see a role closer to the front end of the product development cycle. Business Intelligence is one area where we will be partnering with the product groups and Finance as we build out our internal capability. I want to ensure that any product we develop to meet the needs of Microsoft, also meets the needs of the marketplace.

He was also very frank about the current state of affairs in MSIT relative to the vision. He was quoted in China Information World (no link, sorry) as saying that "The systems Microsoft now uses are already 14 years old and based on previous versions of windows, so from a systems capability perspective, they cannot support the needs of the growing business."

All in all, I was pretty impressed with what he's setting out to do and the opportunity not only from a business perspective from from an industry perspective as well. Hence the whole "going to work in his division" thing. Of course, "Thought Leadership" is one of the things Architecture Strategy works on very diligently, so in some ways this isn't as big a change as it might be. On the other hand, giving advice to people solving hard problems is a lot different than solving those hard problems yourself.

I'll be starting this new role pretty much immediately, so expect the less-than-usual blogging to continue for the time being. But my external visibility on my blog and presenting at conferences and executive briefings is one of the things they hired me for. So after I get my bearings things should be back to normal. At that point, I'll hopefully be able to talk in more specific terms about what we're tackling on my new team. I hope to shake things up quite a bit over there and deliver the play by play here on my blog.

Massive thanks to John deVadoss and the rest of the Architecture Strategy Team. Back when I started, I think Simon Guest was the only other blogger on the team. Now there are only three non-bloggers on the whole team. It's been a great three years and I've been a part of so many great accomplishments:

Making Microsoft the first (I think!) global sponsor for the International Association of Software Architects
Delivering the CRM Integration for Outlook Sample Application
Launching the Solution Architect and Infrastructure Architect MVP programs
Delivering the first-ever Architecture Track at Tech-Ed 2004
Taking Arvindra's amazing JOURNAL The Architecture Journal worldwide
Launching the Architecture Resource Center and the MSDN Solution Architecture Center websites
Presenting at PDC, worldwide Tech-Eds, p&p Summits and a host of other events
Meeting tons of customers in user group meetings, at executive briefings and in architecture design sessions

I always joke that if I ever left Microsoft, I wouldn't want to go work for another technical company. Now, I get the chance to go affect the business of a Fortune 50 business while not having to leave Microsoft. Pretty sweet.

See you on the other side.

June DSL CTP

DevHawk — Fri, 02 Jun 2006 19:56:11 GMT

Congrats to the team for their latest version of the DSL Toolkit, integrated into the June CTP of the VS SDK. According to the published product plans of the VS SDK, they're suppoesd to ship their next release - including the final DSL toolkit - next month. Looking forward to it.

Thinking About Object Models

DevHawk — Thu, 01 Jun 2006 10:54:05 GMT

I'm doing some experiments with Amazon's S3 Service. Very cool service, I might add. Anyway, the sample C# REST code basically wraps the network requests with a single connection class that has individual methods for each type of service interaction (list all my buckets, list all objects in a bucket, create a bucket, create an object, you get the idea).

However, S3's service is a natural hierarchy. The Service contains many Buckets, which in turn contain many Objects. So another way to wrap the service interaction is with a series of objects that are related to one another and only implement the service interactions relevant to that class. (Service would implement List My Buckets and perhaps Create Bucket. Bucket would implement List Objects and Delete Bucket. Again, you get the idea.)

For an interface as relatively simple as S3 (the SOAP interface has a grand total of 13 operations) it probably doesn't matter one way or the other. Furthermore, it's probably a question of personal preference. My question: What's your personal preference? A single object with many methods or a hierarchy of objects each with fewer methods?

Enterprise 2.0 ARCast

DevHawk — Thu, 01 Jun 2006 03:03:24 GMT

Ron just posted his latest ARCast featuring yours truly talking about Enterprise 2.0. Some of the same stuff I blogged about last month, but in a conversational style. Check it out.

Architect Connections Conference

DevHawk — Wed, 31 May 2006 02:56:55 GMT

For any readers in LA, I'm keynoting the MS Architect Connections conference in LA next week. They have a few spots still open, so if you're interested you can sign up here.

TechEd Iron Architect Contest

DevHawk — Fri, 12 May 2006 04:57:48 GMT

For a change, I'm not involved with TechEd at ALL this year. It's all Simon and Marty. Looks like they've got some cool stuff cooking, particularly the Iron Architect contest. Almost makes me wish I was going this year.

ChucK Audio Programming Language

DevHawk — Thu, 27 Apr 2006 04:13:55 GMT

I saw reference to ChucK on Prompt Criticality, a Second Life Blog. I wanted to blog it for my own reference if for no other reason. ChucK is "a new audio programming language for real-time synthesis, composition, and performance" that "presents a new time-based concurrent programming model, which supports a more precise and fundamental level of expressiveness, as well as multiple, simultaneous, dynamic control rates, a precise and straightforward concurrency, and the ability to add, remove, and modify code, on-the-fly, while the program is running, without stopping or restarting".

Sounds facinating, esp. given my recent discussion of internal vs. external DSLs w/ Neal Ford. Plus it deals w/ concurrency and is dynamic.

Sacred Code Cows

DevHawk — Tue, 25 Apr 2006 02:50:52 GMT

I wrote of my meeting w/ Neal Ford a few weeks ago. Today he has a long post on what he calls Eating Sacred Hamburger that goes into more detail of some of the things we discussed:

Software development cults tend to create sacred cows: habits and idioms that might have meant something at one time but only remain as baggage now. I tend to like to kill sacred cows and grill them up, with some nice lettuce, tomato, and a sesame seed bun. On my current project, we're actively killing some sacred cows.

The main sacred cow he's talking about slaughtering is Hungarian Notation for interfaces. You know, interfaces like IDbConnection and IHttpHandler.

Thankfully, Hungarian Notation has mostly been banished, except for one lingering, annoying location in the .NET world: the stupid "I" preface on interfaces. In fact, if you understand how interfaces should be used, this is exactly the opposite of what you want. In our application, every important semantic type is represented by an interface. Using interfaces like this makes it easier to do a whole host of things, including mocking out complex dependencies for testing. Why would you destroy the most important names in your application with Hungarian Notation telling you it's an interface? Ironically enough, that your semantic type is an interface is an implementation detail -- exactly the kind of detail you want to keep out of interfaces. I suspect this nasty habit developed in the .NET world because interfaces first came to the Microsoft world as COM (or, back when it started, OLE). It's a stupid cow now, and should be slaughtered.

I agree 100% with this, though I'm guessing the next time I write an interface, I'll have to go back and delete the stupid "I" because I'm so used to writing it.

The other convention he's looking to do away with camel and pascal casing, which is convention in .NET. He's a bigger fan of using underscores between words (which is big in the Ruby world). So far, he's only using underscores in his test methods since they tend to be longer, such as "Verify_end_to_end_security_connectivity_to_infrastructure". Frankly, I'm ambivalent on this one. I'm pretty good at reading camel and pascal casing and I would hope never to see a production method name like that.

So that's two down, but there's still an entire herd of sacred cows out there. What other ones do we need to get rid of?

ActiveRecord::Migration

DevHawk — Wed, 19 Apr 2006 03:22:14 GMT

When I wrote about the Dual Schema problem a few weeks ago, I specifically wrote that the Rails model is backwards because it derives the in-memory schema from the database schema. While I still believe that, Rails' ActiveRecord::Migration library does make it significantly easier to manage the database from Ruby code. For those not familiar, ActiveRecord::Migration is a series of Ruby script files that define the database schema. Inside each migration script is an up and down method, so you can migrate forward and backward in the history of your project. And it provides easy to use abstractions such as create_table and add_column so you don't have to geek out on SQL syntax (unless you want to). Once you have a collection of these scripts, simply calling "rake migrate" will bring your database instance up to the current schema (rake is Ruby's equivalent of make). Or, you can set your database to a specific version of the schema by running "rake migrate VERSION=X".

I wonder why the Rolling on Rails tutorial uses the database tools directly instead of ActiveRecord::Migrate? I'm thinking it wasn't available when the tutorial was written. Whatever the reason, they really should update the tutorial to reflect the current state of Rails.

Apparently, Microsoft Delivered on Enterprise 2.0 Three Years Ago

DevHawk — Wed, 19 Apr 2006 02:46:49 GMT

In the past few weeks, there’s been a major uptick in discussion about Web 2.0 / Enterprise convergence. Andrew McAfee has a new article on what he calls Enterprise 2.0. Dion’s got an entire blog on the subject, though he thinks it should be called Enterprise Web 2.0. Nicholas Carr is skeptical. Seems to me all this discussion about what might happen in this space is pretty silly since it’s happened already.

Unfortunately, Andrew’s Enterprise 2.0 isn’t freely available (you can buy a copy of the PDF for $6.50), but it primarily focuses on the growing frustration with email and the rise of collaborative Web 2.0 technologies such as blogs and wikis inside the enterprise. No big shock here – for collaboration, blogs and wikis are to email what word processors are to typewriters. Andrew also introduces a model he calls SLATES for describing the aspects of these technologies: Search, Links, Authorship, Tags, Extensions and Signals. So far, all good stuff.

The problem with the article is that he talks about these technologies in the future tense. For example, he writes: "As technologists build Enterprise 2.0 technologies that incorporate the SLATES components" which implies that these are coming down the pipe rather than here right now. Not only here right now, but available for going on three years. I’m talking about SharePoint 2003. 2003 as in "a year before Tim O’ Reilly coined the term Web 2.0".

SharePoint (I’m talking primarily about the free feature pack for Windows Server 2003 though about the portal server as well) supports Search, Links, Authorship and Signals – four of the six components of Andrew’s Enterprise 2.0 stack. (And frankly, I'm not sure where Andrew is going w/ his Extensions aspect so four out of five is probably more accurate.) More importantly, it’s specifically designed to support what Dion called the Democratization of Content. As of December 2004, Microsoft’s internal IT department was supporting "more than 60,000 users, 250 group and division portals, 50,000 team sites, and manages more than 3 terabytes of information." Personally, I use the coportate enterprise intranet portal, my division portal, a handful of team sites and my personal site on a pretty much daily basis. Only the enterprise and division portal are centrally managed. Given the explosion of SharePoint sites inside Microsoft, I'm obviously not alone.

Creating a new SharePoint team site inside Microsoft is totally self service and takes literally a few seconds. Once you have a site, you can configure it as you like, creating lists and setting permissions as you see fit. Again, it’s totally self service. Plus, it’s totally public unless you specifically lock it down (well, public inside the firewall at any rate). Of course, it could be easier and better, and that’s what next versions are for. SharePoint 2007 will have direct support for blogs, wikis and RSS. Check out the C9 video for more info.

Given the market momentum to date and the impending release of a new version, I find it very surprising to find Dion, Andrew and Nicholas discussing the potential ramifications of these technologies without even mentioning SharePoint. If these guys want to see the Enterprise 2.0 technology in action, all they need to do install SharePoint.

Why's (Poignant) Guide to Ruby

DevHawk — Sat, 15 Apr 2006 09:07:08 GMT

Even if you don't care a lick about Ruby, you should read Why’s (Poignant) Guide to Ruby. It is by far the most facinating (and freaky) technical book I've ever read.

Talking Dynamic Lanugages with Neal Ford

DevHawk — Fri, 14 Apr 2006 19:42:43 GMT

I spent a couple of hours chatting with Neal Ford from ThoughtWorks yesterday. Ted Neward had virtually introduced us a few months ago and he was in town for MTS, so he arranged a meeting. I had asked Ted to introduce me to some dynamic language folks for some research and public debate purposes, and Neal was one of the people he hooked me up with. Unfortunately, this was right before I changed roles and got real busy. Of course, dynamic languages in general and Ruby in particular plays a large role in Edge Architecture, so I’m thankful Neal took the time to drop me a line and meet with me.

Above all else, talking to Neal made me realize that I just don’t know enough about dynamic languages, which limits my ability to discuss them. To date, I’ve flirted with them, but haven’t made a real commitment. For example, I’ve played around with Instant Rails, but hadn’t actually installed Ruby yet. It was time to re-image my dev partition anyway, so I’m going to try using Ruby exclusively for a while.

Here’s a brain dump of some of what we talked about. Not sure what it all means yet, so I’ll try and refrain from making commentary.

Hungarian notation for interfaces (i.e. ISomething) is a big code smell. This has nothing really to do with Ruby or dynamic languages, but it’s an important point that I wanted to include here. Neal’s point is that the interface defines the semantics of the type and the concrete class is an “implementation detail”. In other words, contract-first isn’t just for web services. Apparently, ThoughtWorks doesn’t use ADO.NET directly primarily because the interfaces “aren’t pervasive enough” and are difficult to mock out. Also, they’re using Rhino Mocks which I wasn’t previously aware of.
For all the debate about static vs. dynamic languages, it seems like the value Ruby brings is in meta-programming rather than dynamic typing. Certainly, that’s one of the big differentiators for Ruby vs. other dynamic languages like Python. While Rails has pushed the popularity of Ruby thru the roof recently, Neal seems much more enamored with Ruby than Rails.
There is an even bigger gulf between dynamic and static typing proponents than I had thought. I brought up Singularity, which uses static typing exclusively to deliver a provably dependable system. Neal disagreed with that approach, pointing out that “tests are the best way of encoding the specification of the system” rather than compile time checking. Given my lack of expertise in this space, I’m withholding comment (for now) but I’m guessing the truth is somewhere in the middle.
However, while the dynamic vs. static typing gulf is big, meta-programming is potentially the bridge. I don’t believe meta-programming is exclusive to dynamic languages. Certainly, some of the new features in the “Orcas” versions of C# and VB bring more expressiveness to the languages while still remaining type safe.
All this meta-programming leads to domain specific languages. Ruby has strong support what Martin Fowler called “internal DSLs”, but Neal thought over time the focus would shift to external DSLs as they are more expressive and not constrained by the semantics of an existing language. Obviously, we’re pretty heavily focused on DSLs. However, Neal did think our focus on graphical DSLs is misplaced. He called them a “hangover” from CASE/UML tools. He rightfully pointed out that “business analysis speak English”.

All in all, it was time well spent. Neal, I hope we can pick up the conversation again sometime.

LINQ + WAP == Coolness

DevHawk — Fri, 07 Apr 2006 05:48:43 GMT

I never really missed Web Application Project support in VS05 until I realized what I could do with it. I've been experimenting with LINQ and wanted to be able to use it in a web application I'm prototyping. Under the file system model, there's no easy way to change the compiler used for dynamic compilation of web pages. With WAP, it's no problem.

If you've got both WAP and LINQ installed, you can enable LINQ support in your web apps by doing the folllowing.

Open your WebApp.csproj file in notepad.
Add references to System.Query, System.Data.DLinq and System.Xml.XLinq.
Change the Target Import element to import "$(ProgramFiles)\LINQ Preview\Misc\Linq.targets" instead of "$(MSBuildBinPath)\Microsoft.CSharp.targets"

That's it! Seems simple, but it let me bind a GridView to the following function:

public static IEnumerable<Titles> GetTitles()
{
var con = new SqlConnection(Properties.Settings.Default.PubsConString);
var db = new Pubs(con);
var q = from t in db.Titles select t;

return q;
}

It would be even cooler if I could simply write that query in the conext of the ObjectDataSource, but of course ObjectDataSource doesn't know about LINQ. I imagine a LinqDataSource would be a logical next step.

RubyNet Project

DevHawk — Thu, 06 Apr 2006 22:59:42 GMT

Having written about Ruby in the scope of the Compiler Dev Lab and the Dual Schema Problem, I was interested to come across the Ruby.NET project from Queensland University of Technology. From the Ruby.NET home page:

Our goal is to create a compiler for the Ruby language that targets the .NET CLR. We aim to support 100% of Ruby language semantics, including all dynamic constructs such as closures and continuations. We plan to generate 100% managed and verifiable CIL code.

Sweet!

The Dual Schema Problem

DevHawk — Wed, 29 Mar 2006 02:35:03 GMT

A few months ago, Ted Neward wrote a great article about the history of the Object Relational Impedance Mismatch problem and how LINQ is addressing it in a new way. Basically, LINQ is introducing new language abstractions and complementary libraries to enable queries as a first class concept within the language. However, I don't believe that O/R Impedance Mismatch is the whole problem. More specifically, it's a follow-on problem to what I would call the Dual Schema problem.

In a nutshell, the Dual Schema problem is that you have to design and implement two separate versions of your persistent entities. There's the in memory version, typically written in an OO language like C# or Java. Then there's the on disk version, typically written in SQL. Regardless of the difficulties translating between the two versions (i.e. the aforementioned impedance mismatch), you have to first deal with the complexity of keeping the two versions in sync. While LINQ does a great job eliminating much of the friction translating between on disk and in memory formats, it could go much farther by eliminating the need for translation in the first place.

A variety of solutions to the Dual Schema problem have evolved, primarily outside the hallowed halls of enterprise vendors (i.e. MS and others like us). One such solution is Ruby on Rails. In a Rails environment, I simply declare the existence of a given persistent entity:

class Person < ActiveRecord::Base
end

The ActiveRecord base class (a standard part of Rails) will dynamically create methods and attributes on the Person object at runtime, based on the schema of the People table in the database. (Rails is smart enough to understand English plurals, hence the automatic connection of Person and People.) So technically there are still two schemas, but the in-memory version is automatically derived of the on-disk version.

(Note, DLinq provides a conceptually similar tool - SqlMetal - that can generate the static types from a given database schema. However, as static types they have to be defined at compile time. So while SqlMetal reduces the effort to keep schemas in sync, it doesn't eliminate it the way Rails does.)

By slaving the object schema to the database schema, Rails essentially solves the Dual Schema problem. The problem with the Rails approach is that defining a database schema requires a significant amount of skill and effort. Defining classes is typically trivial in comparison.The fact Rails allows you to implement a persistent entity with almost no code doesn’t help you much if you have to write and maintain a ton of SQL code to define your database schema.

I believe the Rails model is actually backwards. It would be much better for the developer if they could define their persistent entity in code and slave the database schema to the object model instead of the other way around.

Of course, this approach isn't exactly news. In his article, Ted writes of the rise and fall of OO database management systems, which were supposed to solve the Dual Schema and Impedance Mismatch problems. I'm certainly not suggesting a return to the heyday of OODBMS. However, one of the reasons Ted points out OODBMS failed was because big companies were already wedded to RDBMS. But those big companies are the short head. As you move down the long tail of software, relational database as the primary storage paradigm makes less and less sense. For the vast majority of applications, relational databases are overkill.

Ted's other point about OODBMS is that loose coupling between the data store and the in memory representation is a feature, not a flaw. He's totally right. But can't we advance the state of the art in database typing to the level of modern day OO languages? How about eliminating anachronisms like fixed length strings? What if we derive the database schema from the object model - Rails in reverse if you will - but is still loosely coupled enough to allow for schema evolution?

An example of this code-centric model for data storage is Consus. It’s written by Konstantin Knizhnik, who has written a bunch of open source, object-oriented and object-relational databases across a wide variety of languages and execution environments, including CLR. Consus is actually written in Java, but he provides version compiled for .NET using Visual J#. Consus lets you to define your data either as tables or objects. So you can do this:

Statement st = db.createStatement();
st.executeUpdate(
    "create table Person (name string, address string, salary bigint)");
st.executeUpdate(
    "insert into Person values ('John Smith', '1 Guildhall St.', 75000)");
ResultSet rs = st.executeQuery(
    "select name, address, salary from Person where salary > 100000");

Or you can do this:

class Person {
    String name;
    String address;
    long salary;
    Person(String aName, long aSalary, String aAddress) {
        name = aName;
        salary = aSalary;
        address = aAddress;
    }
};

Person p = new Person("John Smith", 75000, "1 Guildhall St.");
ConsusStatement st = db.createStatement();
stmt.insert(p);
ConsusResultSet cursor = (ConsusResultSet)st.executeQuery(
"select from Person where salary > 100000");

Consus also handles OO concepts like derivation and containment. Of course, the embedded queries are ugly, but you could imagine DLinq style support for Consus. In fact, one of the primary issues with Consus is that it supports both object and tuple style queries. When you explicitly request tables (i.e. "select name, address salary from Person"), you’ve got a tuple style query. When you don’t (i.e. "select from Person”) you’ve got an object style query. Of course, the issues with tuple style queries are well documented in Ted’s article and is exactly the problem that LINQ is designed to solve.

(Konstantin, if you’re reading this, drop me a line and I’ll look into getting you hooked up with the LINQ folks if you’re interested in adding LINQ support to Consus.NET.)

The tradeoff between the Rails approach and the Consus approach is one of performance. I have a ton of respect for Konstantin and the work he’s done on Consus and other OO and OR databases available from his site. However, I sure the combined developer forces at major database vendors like Microsoft (and other DB companies) means SQL Server (and the like) will out perform Consus by a significant margin, especially on large scale databases. So if execution performance is your primary criteria, the Ruby on Rails approach is better (leaving aside discussion of the Ruby runtime itself). However, in the long run execution performance is much less important than developer productivity. So I believe that for all the current interest in Rails, I think a Consus-style model will become dominant.

CLR Everywhere

DevHawk — Wed, 22 Mar 2006 19:51:53 GMT

Big news from the Game Developers Conference this week is the XNA Framework. From the press release:

The XNA Framework contains a custom implementation of the Microsoft® .NET Framework and new game-development-specific libraries designed to help game developers more easily create cross-platform games on Windows® and Xbox 360 using the highly productive C# programming language. Using the XNA Framework, game developers will benefit from the ability to re-use code and game assets in developing multiplatform titles, without sacrificing performance or flexibility.

So now I can the Xbox to the list of personal technology I can program, joining my phone, my TV, my server and my laptop all with C# or VB (VB plug brought to you by Brian and Erik who think I should to switch). I wonder if XNA Framework will support WPF?

BTW, I still want an easy way to program my house and my car, but I'm sure those are coming.

Where else should the CLR live?

The Annoying C# 3.0 "Unsupported Version" Dialog

DevHawk — Sat, 18 Mar 2006 01:41:45 GMT

I am sick and tired of the warning dialog box for the C# 3.0 preview. Every time you launch VS it pops up a dialog reading "This is an unsupported version of Microsoft Visual C# 3.0. As such many features may not work as expected." You know, this isn't my first time working with alpha or beta quality software. I'm also running WinFX Feb CTP, VB 9.0 Jan CTP and the Expression Interactive Designer March CTP. None of them feel the need to constantly remind me that they're unsupported.

Who suddenly thought C# developers need hand holding?

The SQL Complexity Problem

DevHawk — Thu, 16 Mar 2006 11:51:32 GMT

I mentioned on the first day of the Compiler Dev Lab that Brian Beckman is a hoot. He's also wicked smart. He posted about his demo from Monday where he demonstrates building indexes for use in LINQ queries. In his words:

In the terminology of relational databases, a “join” is, semantically, like a nested loop over a pair of lists (or tables) of records, saving only those where some certain fields match. Unless we do something smart, this could be very expensive. Imagine searching a database of a million DNA profiles for the closest match to a sample that has 10,000 DNA features (I have no idea whether those are real numbers: I just made them up, but they sound ballpark to me). A dumb join would search all 1 million profiles for each of the 10,000 features, resulting in 10 billion match tests, almost all of which will fail – by design, of course. That’s going to hurt.

The “something smart” is to build an index and search through that. Your database doesn’t have to be large at all for this to pay off. In fact, even with just a few records, it’s cheaper to build an index, use it, and throw it away than it is to do a nested loop.

He goes on to prove out his point about building an index. For his full dataset (joining 3053 cities with 195 countries) it is literally 65x slower not to build a one-off index. Even for smaller datasets, the time difference is less dramatic but still significant. For example, with 89 cities instead of 3053, it's 3x slower not to build the index.

The reason I'm so interested in Brian's post is because of my experiments with Ning. As you might recall, in trying to build a .NET version of Partisan Hacks, I found ASP.NET 2.0 to be significantly simpler than PHP (which Ning uses). However, building even the trivial SQL Express database for Partisan Hacks was a non-trivial exercise. Sure, I've done it many times before, but it seems strange that ASP.NET makes it so easy to build a site while SQL Server makes it so complex to build a database. If I was a novice user, I would never be able to build a database for my web site.

Why is this? I think that the simple app or amateur developer is simply not the target audience for SQL Server (even SQL Express). If you don't know the difference between nvarchar(100) and varchar(max) you're pretty much out in the cold when it comes to SQL Server. Their target audience appears to be enterprise databases that are cared for by enterprise database administrators. Databases with scores of tables and millions of rows. Great for them, bad for novice users who just want to persist their data somewhere quickly and easily.

Why can't building my database be as simple as building my site?

Ning makes it easy to use their Content Store. You create an instance of a content object, you set properties (dynamic ones), you hit save. No fuss, no muss, no db schema. Sure is an easier model to understand and program to. In that regard, it blows away everything, even Ruby on Rails. RoR is pretty sweet, but it needs a real database schema on the back end in order to drive RoR's guiding principle of "convention over configuration". If there's no DB schema to discover, I think much of the RoR model would break down. (but that may just be my lack of RoR experience talking)

I not sure what a simpler database system would look like, but one idea of mine is to use a schemaless database. Much of the complexity comes from having to define both an in memory as well as perseistant schema, as well as the translation between them. If you just stored managed .NET objects, you would eliminate the redundant schema specification. It's not a fully fleshed out concept, but it is a start of an idea.

What other ideas would make persistant data significantly easier to work with?

Compiler Dev Lab - Scripting

DevHawk — Wed, 15 Mar 2006 22:25:22 GMT

Day Two of the Compiler Dev Lab was all about scripting. Iron Python was the primary focus of the day, but they also had Phalanger (Managed PHP) and Monad folks there as well.

I hadn't realized just how performant these dynamic languages are on the CLR when compared to their native versions. The original version of Iron Python was 1.7x faster than the standard C implementation back in the summer of '04. Now with CLR 2.0, that version is now 2x faster with out any code changes. The Phalanger folks said they are 2.5x faster than the native version of PHP (1.7x faster than PHP + the Zend Optimizer). That's pretty impressive performance.
The IronPython folks are heavy users of the new DynamicMethod class from .NET 2.0. Otherwise known as Lightweight Code Generation, DynamicMethod allows you emit a static function but have it get garbage collected when it's no longer needed. IP almost never generates new classes, since new types can't be garbage collected. The only times they generate actual classes are when you inherit from an existing .NET class or when you generate a new delegate type.
It's really hard to serve the dual masters of both the existing language community and the .NET community. Jim Hugunin used the example of String.Trim(). A .NET developer would expect String.Trim() to "just work". A Python developer would expect that to throw an AttributeError exception (the Python equivalent of Trim is strip). How do you handle this? In IP, it defaults to pure Python mode, but if you enter "import clr", you move into .NET hybrid mode.
One of the typical features of dynamic languages is the ability to change the base class of an object on the fly. Jim demoed this with WPF. He created a class that inherited from one type of panel and then set the __class__ property of the object to a different panel and the display changed immediately. Freaky, but cool.
Jim showed a demo of a WPF app that hosted Python for extensibility. One of the scripts in turn hosted Python to create an interactive console for the app. Having a scripting engine that can host itself is awesome.
The VSIP SDK CTP (reg required) includes an sample lanugage integration project for Iron Python. So you can get both the source into IP language itself as well as the source to the integration into Visual Studio.
I got an email yesterday from someone asking about the possibility of Visual Ruby.NET. I haven't heard anything about it, but it would be cool to see Ruby on Rails runing under CLR. John Lam is working on RubyCLR, but my understanding is that is a bridge between the CLR and the Ruby runtime, not a CLR implemenation of the Ruby runtime. (IP is a CLR implementation of the Python runtime.) I'm thinking that there are some similarities between Ruby and Python, so having the source of IronPython would be a huge help in building a Visual Ruby implementation. For example, both Ruby and Python have closures. IP has a FunctionEnvironment class which is used to lift stack variables onto the heap in a variety of scenarios, including closures. So if I was building Visual Ruby, having access to the FunctionEnvironment class would be a good start.
I said yesterday that I need to learn more about F#. They showed a video of an internal F# presentation, but I spent most of my time cracking jokes with Sam Gentile who's in town for an SC-BAT workshop.
I didn't pay enough attention to the Monad presentation. :(

Compiler Dev Lab - LINQ

DevHawk — Tue, 14 Mar 2006 22:33:14 GMT

Even though I haven't finished my ETech postings, I'm already onto another event. This week, thanks to an invite from Michael Lehman, I'm sitting in on a Compiler Lab discussing implementing other languages for CLR. The first day was about LINQ. Much of the info is rehashed from PDC or the docs up on MSDN. However, I have learned a few new things.

One of the standard features of LINQ is Extension Methods. That enables you to declare a static method like "static void Foo(this string source)" and then use it like "stringvar.Foo()". Apparently, they are considering adding other types of extension members including properties and fields. The idea of extension fields is somewhat scary but powerful.
LINQ uses something Anders called deferred query execution. The query isn't executed until the values are asked for (typically by calling foreach on the query). That means you can compose queries to your hearts content with no perf impact until you actually invoke the query.
Query Comprehensions in C# and VB is a pattern implementation in a similar vein to foreach. Foreach is relatively simple shorthand for iterating through an collection by calling IEnumerator.MoveNext until it returns false. While LINQ enables arbitrary composition of queries, there is obvious gravitational pull towards the SELECT / FROM / WHERE / ORDER BY / GROUP BY approach favored by SQL. So if you build your own query operator, you can include it in a LINQ query, but C# and VB won't be able to include it in the Query Comprehension syntax. Probably not a big deal, given the breadth of standard query operators as well as the deferred query execution, but it's good to understand how the abstraction works.
I want to know more about how DLinq is implemented. I've been refining my thinking about data since working with Ning's content store and I'm convinced of the need for a simplified datastore. SQL is designed for significantly complex database schemas, which means a significantly complex development environment.
I'm looking much more closely at VB, given the new features in VB 9.0. Not only the LINQ stuff from C# like type inference, extension methods and anonymous types but also VB specific stuff like XML Literals and Duck Typing. Combined with VB's existing support for late binding, there are compelling features to make VB attractive over C#.
I've been hanging out with Brian Beckman. He's a hoot.
I think I need to take a deeper look at F#.

ETech Day Three Quick Thoughts

DevHawk — Thu, 09 Mar 2006 07:42:20 GMT

After my marathon blogging session last night and taking notes all day, I'm a bit burnt out on writing. But here are a few quick thoughts. More details to follow.

I'm digging the Live.com home page and the integrated Live Search. Since I'm on a rented laptop, Live Toolbar will have to wait. Coolest new feature IMO is the Search Macros, though it's a tight race with the new image search interface.
Jon Udell and Michael Goldhaber spoke about attention economy today. I still don't get it, though Jon had some interesting ideas about metadata. I'll believe that attention is a currency when I can buy a car with it.
I liked the session on the Yahoo! Design Patterns, though the title and abstract of the session were awful. The title was "The Language of Attention: A Pattern Approach". The inclusion of attention just confused the issue. Why couldn't they just call it "A Pattern Language for User Experience"? Because it doesn't have the concept of attention shoehorned into it.
I really like Eventful, even though I'm on record as thinking their business model doesn't work. Their new demand feature is pretty cool, though it doesn't really help their business model any.
George Dyson's session on "Turing's Cathedral" was fascinating, though he tried to cover too much ground in the time alloted.
I'm not sure what the point of Joel Spolsky's Blue Chip Report Card was. Apparently the alien from Reddit is cute and Motorola newer cell phones (RAZR and PEBL) are taking Joel's advice on becoming "blue chip". This is somewhat related to points the folks from Adobe (previously Macromedia) made, except much more obtuse.
I have no idea what the point or business model of Plum is, even though it was featured as a keynote (a last minute promotion it appears from the conference guide). Seems too complex and centralized to actually work.
I wrote last night that Casting Words isn't really a business because nothing stops me from going directly to Mechanical Turk and getting the transcription services myself. Today, I found a Casting Words task on Mechanical Turk so I decided to figure out how much they're making. The task I found was to transcribe about 28 minute podcast and they were offering $5.41 for anyone willing to do it. That's about 19.5 cents per minute. Tack on Amazon's 10% charge brings the total to around 21.5 cents a minute that Casting Words is paying for transcription services. Given that they're charging 42 cents a minute, that's just under a 49% profit margin. Exactly what are they doing to earn that profit? What's their value add and is really worth a 100% markup?
Anyone want to start "Cheap Casting Words" with me? We'll pay 22 cents a minute (11% more than Casting Words) and charge 36 cents a minute (14% less than Casting Words) and keep the 12 cents a minute markup (a 33% markup). :)

UPDATE (8:45pm) - Added Quick Thoughts on Yahoo!, Eventful, George Dyson, Joel Spolsky and Plum. Added more detail about the attention economy sessions from today.