It's amazing how much information is there in our email archives. Now that we've shipped the thing, I thought I would share my summarized (still long), partial view of how the ADO.NET Data Services Framework ("Project Astoria") came to be. I left out a ton of partners, important events and features that came and went for the sake of brevity, but most key points in time are there.

Idea
Hack
Pitch
Prototype
Review
Announcement
Team
Design
Release
Future

Idea

September July 2006. We were having lunch at the building 35 cafeteria with Alex Barnett (back then our Community Program Manager) and he brought up that a bunch of people out there were doing REST-based APIs. The question on the table was if there was anything interesting around entities (as in EDM entities) and REST. I didn't know enough about REST to answer anything interesting, so I agreed to go do some reading.

September July 24th, 2006. Earliest date for what I have something documented. I found an email message I sent to Alex that basically said "I read about the REST thing...we could expose Entities as resources and might fit the REST model". My struggle at that point was whether there were valid scenarios for exposing a REST interface on top of entities without added semantics ("interceptors" came to the rescue later on).

Hack

August 24th, 2006. Finally found time to look at this. I wrote a quick hack overnight called, unimaginatively, "REST for Entities". It was a simple ASP.NET handler that would take data in EDM terms and give every entity on the system a URI, allow clients to retrieve the entities in multiple representations (XML and JSON, based on the HTTP accept header), and make changes to entities by using simple HTTP verbs (POST, PUT, DELETE). The use of URIs with added semantics driven by EDM schemas, support for multiple formats and the simple HTTP interface still remain core aspects of Astoria.

Pitch

September/October 2006. Alex pinged everyone and then some to see who'd hear us. He and I started to tour around Microsoft pitching the idea, testing it to see if it was interesting to folks, adjusting the pitch as we learned how to deliver the story. It was clear we needed some well-articulated content to move forward.

Late October 2006. Found the time to sit down and write a white paper, "Entities in the Cloud"; the paper focused on two observations: a) the raise of "application/service hybrids", typically consumer application that became development platforms, and the key role of their service interfaces, typically data-driven, and b) how simple the interfaces to these things are...just URLs and basic HTTP functionality. The paper described how Astoria greatly facilitated the creation of such application/service hybrids and had the potential to create a small ecosystem of libraries and tools to consume them. It also explored early ideas around data-independence in the HTTP/data interface, synchronization and offline support for data services and higher order semantics.

November and December 2006. We were busy with other things (I think I was working on the Entity Framework at that point, either on transactions support or on LINQ to Entities), but we kept working on the pitch in the background so we could show a "vision" to folks that would eventually decide to fund the effort. In addition to the paper above, we also wrote a paper for "ThinkWeek" with Britt Johnston (now the PUM of the data programmability tools team). This was probably the first time we explicitly painted the picture of Astoria not only as a framework and common HTTP interface, but also as an online structured storage service (fast-forward to the present: our collaboration effort with SQL Server Data Services will land us closer to this initial vision than I would have ever expected).

Prototype

February-May 2007. We had been talking about announcing the plans for Astoria at Mix 2007, which was planned for May 2007. We wanted bits in people's hands, not just slideware. In less than 3 months we built the "Mix CTP" of Astoria as a side-project. It wasn't just the code, but the setup, the samples, the client libraries, the huge documents, the online service and more. I wrote a large chunk of these, and several folks helped with specific areas, such as Elisa Flasko with reviewing the documents and creating websites, Tim Mallalieu with parts of the URL parser, and Asad Khan with parts of the client library. During these months I had a lot of fun building the system from scratch, but didn't exactly have a life. In retrospective, it was a great choice. Laser-focus on something and build it in a time-boxed manner.

First check-in was on February 16th, 2007.

A few key developments happened during the prototype building work:

  • Service definition: various folks independently influenced the way Astoria services are authored. At first it was declarative-only, which was falling short. During a few chats with Nikhil Kothari, the early forms of the current code-centric model came up. Also, during conversations with Anders Hejlsberg the thought of using LINQ as a layering mechanism came up.
  • WCF: While working on this, the WCF folks offered a better integration path. After a brief discussion Steve Maine, WCF wizard, got going into a week of non-stop hacking where we had many 2, 3 and 4 AM email threads. At the end, we had a nicely integrated story. Nothing like people that know their stuff and can get their hands dirty.
  • Client libraries: Another thing that happened here was the client library. It started as "let's cook up a couple of quick bindings for .NET and Javascript". I found an email from January 30th, 2007 that discusses the idea for the first time, and even mentions the possibility of supporting some form of LINQ-to-URL translation.
  • URL formats: with this work we introduced the second (but not last) URL format for Astoria. After a ton of community feedback we adjusted it to be what shipped in the final version later on.

Review

April 9th and 10th, 2007. You don't want to invest too much while locked into your office and then think you're solving a real problem. While we had ran our plans by a lot of internal folks, we wanted to get some external perspective before the general announcement. So on April 9 and 10 we held the first Astoria SDR (software design review, common practice at Microsoft).

The organization of this goes back to January and February, and Alex did a superb job organizing the event and in particular picking the right set of folks to give us feedback on the problem space, the plan and the pitch.

The feedback was good overall, which was a relief. Right before the event I started to wonder if they would just say "this? you brought us all the way here for this?". Happily, they were a bit more interested than that :)

Announcement

April 30th, 2007. We announced Project Astoria at Mix 2007 in Las Vegas. On that Monday I gave a talk called "Accessing Data Services in the Cloud", which described the problem space and announced the downloadable Astoria framework as well as the experimental online service that we had hosted for everyone to access (with Astoria front-ends for Northwind, AdventureWorks, Encarta and TagSpace). The talk went great and feedback was good.

Throughout that week at Mix we had a crazy amount of activity. We talked with a ton of folks, did labs, repeated the talk on Wednesday, and more. There was even some press follow up, which for a low profile project was a nice tough.

Team

The team formed sort of organically, so I don't have an exact timeline, but did find a few emails and old meetings in my schedule that are good reference points.

April 27th, 2007. The first actual interview for a member of the yet-to-exist Astoria Team was for Mike Flasko. Mike was working in the Windows networking group. He nailed the interview and became the Program Manager for Project Astoria and one of the driving forces that made the product possible. I couldn't think of a better PM for Astoria than Mike.

May 2007. For some reason, over the years whenever I'm working on cooking something Andy Conrad is also working on cooking something independently, and these things tend to be related. Around Mix 2007 Andy was working on Jasper, exploring dynamic languages and data access. Andy was the first internal member of the team, jumping-in as the Developer Lead for Astoria, and was very influential on how we built the system. Agile methodologies, morning scrum meetings and all that good stuff...

May-July 2007 and on. I won't go through each member of the team. You can read some of them in their own blogs like here and here, some in the Astoria team blog, and some don't write but you'll be using their code and relying in their tests whenever you use Astoria. We had an initial team with development, quality assurance and program management in place by early July, ready to start the official design. New folks continued to join the team throughout the project.

Design

July 5th-11th, 2007. To go from prototype mode to production-quality product mode, we started with a weeklong "design marathon" where we got all together and went from the problem statement and state of things on the web to a walk-through of that Astoria should look like and how it would work at a high level.

From there we set to build the product. While we kept the lessons learned in mind, we didn't use a single line of the prototype code in the real version of the product.

Design is an ongoing process, and we would hold design meetings two times a week, 2 hours each, through the whole product cycle.

Another thing we experimented with around design is how we communicate with the developer community. We tried, successfully in my opinion, to go with a "transparent design process".

Release

June 20th, 2008. We started to actually code stuff in late July 2007, and we worked on the product code, test code and functional specifications for around 9 months. The remaining time included bug fixing, performance work, standard overhead for packaging, etc. After almost a year of "official" (read non-prototype, non-side-project) work, on June 20th 2008 we got together to celebrate and claim victory over the ADO.NET Data Services Framework v1, aka Project Astoria.

August 11th, 2008. Service Pack 1 for Visual Studio 2008 and .NET Framework 3.5, containing a few new features including the Data Services Framework, becomes publicly available.

Future

Astoria v1 was an amazing ride, but we're far from done. What's in the future? In the short term we're working on closing and shipping the Silverlight version of our .NET client library and working together with the ASP.NET folks to refresh our AJAX libraries. As for the next release, some of us have been already working for a while on that...this deserves a post of its own, but the topics getting attention these days include support for synchronization and offline applications, extensions to Astoria to support the largest, busiest online services in the internet, changes for being a better Atom citizen and more.

-pablo