During the design of Data Services (Astoria) v1 we did the transparent design thing. We're quite happy with the result, we got a lot of feedback and were able to adjust many aspects of the project based on that.

Now that we're in full swing with v2 design work, we're going to be posting regularly again (hopefully :).

Andy got a new tiny camera and he had to use it for something...so he suggested that we do short clips that may help explain design points. Sometimes a short explanation is just much better than a bunch of writing.

The first post with the write-up + short video format discusses "Friendly Feeds" and is here. If you get a change to take a look, let us know what you think about the new style (and friendly feeds)!

-pablo

It's amazing how much information is there in our email archives. Now that we've shipped the thing, I thought I would share my summarized (still long), partial view of how the ADO.NET Data Services Framework ("Project Astoria") came to be. I left out a ton of partners, important events and features that came and went for the sake of brevity, but most key points in time are there.

Idea
Hack
Pitch
Prototype
Review
Announcement
Team
Design
Release
Future

Idea

September July 2006. We were having lunch at the building 35 cafeteria with Alex Barnett (back then our Community Program Manager) and he brought up that a bunch of people out there were doing REST-based APIs. The question on the table was if there was anything interesting around entities (as in EDM entities) and REST. I didn't know enough about REST to answer anything interesting, so I agreed to go do some reading.

September July 24th, 2006. Earliest date for what I have something documented. I found an email message I sent to Alex that basically said "I read about the REST thing...we could expose Entities as resources and might fit the REST model". My struggle at that point was whether there were valid scenarios for exposing a REST interface on top of entities without added semantics ("interceptors" came to the rescue later on).

Hack

August 24th, 2006. Finally found time to look at this. I wrote a quick hack overnight called, unimaginatively, "REST for Entities". It was a simple ASP.NET handler that would take data in EDM terms and give every entity on the system a URI, allow clients to retrieve the entities in multiple representations (XML and JSON, based on the HTTP accept header), and make changes to entities by using simple HTTP verbs (POST, PUT, DELETE). The use of URIs with added semantics driven by EDM schemas, support for multiple formats and the simple HTTP interface still remain core aspects of Astoria.

Pitch

September/October 2006. Alex pinged everyone and then some to see who'd hear us. He and I started to tour around Microsoft pitching the idea, testing it to see if it was interesting to folks, adjusting the pitch as we learned how to deliver the story. It was clear we needed some well-articulated content to move forward.

Late October 2006. Found the time to sit down and write a white paper, "Entities in the Cloud"; the paper focused on two observations: a) the raise of "application/service hybrids", typically consumer application that became development platforms, and the key role of their service interfaces, typically data-driven, and b) how simple the interfaces to these things are...just URLs and basic HTTP functionality. The paper described how Astoria greatly facilitated the creation of such application/service hybrids and had the potential to create a small ecosystem of libraries and tools to consume them. It also explored early ideas around data-independence in the HTTP/data interface, synchronization and offline support for data services and higher order semantics.

November and December 2006. We were busy with other things (I think I was working on the Entity Framework at that point, either on transactions support or on LINQ to Entities), but we kept working on the pitch in the background so we could show a "vision" to folks that would eventually decide to fund the effort. In addition to the paper above, we also wrote a paper for "ThinkWeek" with Britt Johnston (now the PUM of the data programmability tools team). This was probably the first time we explicitly painted the picture of Astoria not only as a framework and common HTTP interface, but also as an online structured storage service (fast-forward to the present: our collaboration effort with SQL Server Data Services will land us closer to this initial vision than I would have ever expected).

Prototype

February-May 2007. We had been talking about announcing the plans for Astoria at Mix 2007, which was planned for May 2007. We wanted bits in people's hands, not just slideware. In less than 3 months we built the "Mix CTP" of Astoria as a side-project. It wasn't just the code, but the setup, the samples, the client libraries, the huge documents, the online service and more. I wrote a large chunk of these, and several folks helped with specific areas, such as Elisa Flasko with reviewing the documents and creating websites, Tim Mallalieu with parts of the URL parser, and Asad Khan with parts of the client library. During these months I had a lot of fun building the system from scratch, but didn't exactly have a life. In retrospective, it was a great choice. Laser-focus on something and build it in a time-boxed manner.

First check-in was on February 16th, 2007.

A few key developments happened during the prototype building work:

• Service definition: various folks independently influenced the way Astoria services are authored. At first it was declarative-only, which was falling short. During a few chats with Nikhil Kothari, the early forms of the current code-centric model came up. Also, during conversations with Anders Hejlsberg the thought of using LINQ as a layering mechanism came up.
• WCF: While working on this, the WCF folks offered a better integration path. After a brief discussion Steve Maine, WCF wizard, got going into a week of non-stop hacking where we had many 2, 3 and 4 AM email threads. At the end, we had a nicely integrated story. Nothing like people that know their stuff and can get their hands dirty.
• Client libraries: Another thing that happened here was the client library. It started as "let's cook up a couple of quick bindings for .NET and Javascript". I found an email from January 30th, 2007 that discusses the idea for the first time, and even mentions the possibility of supporting some form of LINQ-to-URL translation.
• URL formats: with this work we introduced the second (but not last) URL format for Astoria. After a ton of community feedback we adjusted it to be what shipped in the final version later on.

Review

April 9th and 10th, 2007. You don't want to invest too much while locked into your office and then think you're solving a real problem. While we had ran our plans by a lot of internal folks, we wanted to get some external perspective before the general announcement. So on April 9 and 10 we held the first Astoria SDR (software design review, common practice at Microsoft).

The organization of this goes back to January and February, and Alex did a superb job organizing the event and in particular picking the right set of folks to give us feedback on the problem space, the plan and the pitch.

The feedback was good overall, which was a relief. Right before the event I started to wonder if they would just say "this? you brought us all the way here for this?". Happily, they were a bit more interested than that :)

Announcement

April 30th, 2007. We announced Project Astoria at Mix 2007 in Las Vegas. On that Monday I gave a talk called "Accessing Data Services in the Cloud", which described the problem space and announced the downloadable Astoria framework as well as the experimental online service that we had hosted for everyone to access (with Astoria front-ends for Northwind, AdventureWorks, Encarta and TagSpace). The talk went great and feedback was good.

Throughout that week at Mix we had a crazy amount of activity. We talked with a ton of folks, did labs, repeated the talk on Wednesday, and more. There was even some press follow up, which for a low profile project was a nice tough.

Team

The team formed sort of organically, so I don't have an exact timeline, but did find a few emails and old meetings in my schedule that are good reference points.

April 27th, 2007. The first actual interview for a member of the yet-to-exist Astoria Team was for Mike Flasko. Mike was working in the Windows networking group. He nailed the interview and became the Program Manager for Project Astoria and one of the driving forces that made the product possible. I couldn't think of a better PM for Astoria than Mike.

May 2007. For some reason, over the years whenever I'm working on cooking something Andy Conrad is also working on cooking something independently, and these things tend to be related. Around Mix 2007 Andy was working on Jasper, exploring dynamic languages and data access. Andy was the first internal member of the team, jumping-in as the Developer Lead for Astoria, and was very influential on how we built the system. Agile methodologies, morning scrum meetings and all that good stuff...

May-July 2007 and on. I won't go through each member of the team. You can read some of them in their own blogs like here and here, some in the Astoria team blog, and some don't write but you'll be using their code and relying in their tests whenever you use Astoria. We had an initial team with development, quality assurance and program management in place by early July, ready to start the official design. New folks continued to join the team throughout the project.

Design

July 5th-11th, 2007. To go from prototype mode to production-quality product mode, we started with a weeklong "design marathon" where we got all together and went from the problem statement and state of things on the web to a walk-through of that Astoria should look like and how it would work at a high level.

From there we set to build the product. While we kept the lessons learned in mind, we didn't use a single line of the prototype code in the real version of the product.

Design is an ongoing process, and we would hold design meetings two times a week, 2 hours each, through the whole product cycle.

Another thing we experimented with around design is how we communicate with the developer community. We tried, successfully in my opinion, to go with a "transparent design process".

Release

June 20th, 2008. We started to actually code stuff in late July 2007, and we worked on the product code, test code and functional specifications for around 9 months. The remaining time included bug fixing, performance work, standard overhead for packaging, etc. After almost a year of "official" (read non-prototype, non-side-project) work, on June 20th 2008 we got together to celebrate and claim victory over the ADO.NET Data Services Framework v1, aka Project Astoria.

August 11th, 2008. Service Pack 1 for Visual Studio 2008 and .NET Framework 3.5, containing a few new features including the Data Services Framework, becomes publicly available.

Future

Astoria v1 was an amazing ride, but we're far from done. What's in the future? In the short term we're working on closing and shipping the Silverlight version of our .NET client library and working together with the ASP.NET folks to refresh our AJAX libraries. As for the next release, some of us have been already working for a while on that...this deserves a post of its own, but the topics getting attention these days include support for synchronization and offline applications, extensions to Astoria to support the largest, busiest online services in the internet, changes for being a better Atom citizen and more.

-pablo

I've been sort of under a rock for a while, but I thought I'd come out for a minute to celebrate. Today we made available .NET 3.5 SP1 and Visual Studio 2008 SP1. There are two components in the release I spent a bunch of time on, which interestingly enough have very different origins and get to RTM through very different processes. One is the ADO.NET Entity Framework, which has been cooking for several years and survived controversies, comparisons with non-shipped previous attempts and other natural disasters; the other one is the ADO.NET Data Services Framework or Project Astoria, which was built, well...fast.

I won't go into details of the release, folks have discussed the Astoria, Entity Framework and general data-related features in the release already.

Why have I been under a rock? In the last few months I've been spending time working on various things related to Astoria, online services and data interfaces. Some I can discuss, some will need to wait a bit until the stakeholders are comfortable to talk about it publicly.

Moving Astoria as a framework forward: we were ready (modulo bug fixing and last minute tweaks) some time ago, and we've been thinking about the next steps for the Astoria framework. In Mix 08 we mentioned that we were working on "Astoria Offline" and showed a prototype. We've been working hard in that topic. There is also a bunch of features we want to take on for the next release. I'm sure we'll post something in the Astoria blog at some point about our thinking and give a change for folks to give feedback.

Online services: as you can imagine there is a number of things going on around online services these days, and a number of them involve Astoria one way or the other. I've been working with several of them, varying from providing guidance all the way to writing custom "v.next" versions of Astoria to experiment with their needs. An example of these efforts is the work we're doing to align SQL Server Data Services and the ADO.NET Data Services framework. We would like to see them as the "service" and the "framework" pieces, both using the same HTTP interface, same client interfaces, etc., so we've been spending a bunch of time exploring how to bring them together.

Anyway, there, a bit of a celebration.

-pablo

Roger just tagged me for this software development meme thing…it looks like Julia tagged him, Shawn tagged Julia, etc. so all the usual suspects have been down this path already. I’ll bite…

How old were you when you first started programming?
I got my first computer, a Commodore 64, when I was 10. I just had to figure out how things like games worked, so I made my way through Commodore BASIC then (later I figured that none of the games I used where actually written in BASIC…).

How did you get started in programming?
I kind of started twice…first when I got my first computer, I got a couple of books on BASIC. None of the programs I wrote back then did anything useful. I "started again" when I was in school studying industrial electronics, where I learned assembly (for the Motorola 6809) and then C for controlling microprocessors/microcontrollers. 

I loved their bottom-up training style in this place: we first learned to assemble by hand using the processor manual and literally enter hex number into the "kits" using an hex keyboard. Then we used an assembler. Then we learned this "simpler, shorter way" of writing programs (a subset of C), which we still were required to translate to assembly by hand on paper. Only at the very end we were allowed to use a C compiler.

What was your first language?
First language at all: BASIC. First language for a useful program: C. First language for a paid job: C++.

What was the first real program you wrote?
As part of a school project I designed the hardware and wrote the corresponding software for controlling direct-current electric motors. The software had feedback from the engine (electric current consumption, speed in RPM, temperature) and maintained constant speed on the motor and made sure it stayed within safe ranges of temperature and energy consumption. I also did some graphics of all the state information on the screen (a "Hercules" monochrome screen).

There is something interesting about writing software that can physically break things…

What languages have you used since you started programming?
I wrote commercial-grade software in 80x86 assembly, C, C++, Prolog, SQL, LotusScript, FoxPro, VB, Javascript, Java and  C#. I've also written/modified small programs in other languages such as assembly language for various Motorola processors (6510, 6809, HC11), Haskell/Gopher, COBOL, BASIC and Pascal.

What was your first professional programming gig?
I wrote a small messaging application for Kodak Argentina as a freelancer. They wanted the thing to be really lightweight and really fast, so they wanted the whole thing in C/C++ with no dependencies. In retrospective I did all the wrong things on this one, such as re-inventing the wheel several times (wrote a small database from scratch, a synchronization-over-cc:mail infrastructure, etc.).

If you knew then what you know now, would you have started programming?
Absolutely. I still find it amazing that somebody actually pays me to do what I do. I get to spend most of my days working with smart folks solving hard problems. Couldn't ask for more.

If there is one thing you learned along the way that you would tell new developers, what would it be?
Well…I'll go with two things: first, stay close to the code where things are real, don't go into high-level-limbo-land –at least not too fast. Second, even if you're the best coder ever, you can't lock yourself in an office…software development is a social activity as much as it is a technical discipline, and it takes good interaction skills among team members to build great software.

What's the most fun you've ever had ... programming?
So hard to pick one…
 
There was this time when I lead a project (and was one of the developers as well) to build a highly scalable rules-based expert system for credit risk analysis. Expert systems are CPU-bound, so making them scale (back then more than now) meant massive distribution. We built both the expert system from scratch (first in Prolog, then switched to Java and a custom inference engine we also built), and then the networking stack and client/server agents for job distribution and control. The result was a system where you could simple plug-in more computers and you'd get more inferences/minute; the system was self-balancing, automatically adjusted itself for various nodes with different computing power, automatically recovered from failed nodes, and would scale pretty much as much as the network could take the load.

I must say though that after I thought I had all the fun and that I "knew" how to build software, I jointed Microsoft and got a different perspective, both from the people perspective and from the projects scale perspective. It's not just "programming" so it may not fit here, but being involved in building something like SQL Server is just too good…

Who's next?
Well, I can't resist the temptation of going with a few of the Astoria team folks, Andy, Mike, Marcelo, Phani.

-pablo

The news are out. The ADO.NET Data Services Framework (Astoria) and the ADO.NET Entity Framework will be shipping as part of .NET 3.5 SP1, and the Beta 1 release is now available. All the official blogs discussed the details already, including the Astoria team blog, ADO.NET team blog, Scott's, and many others out there.

Folks out there trying Astoria and the EFx have been working on bits from last December for a while. Finally we have a newer release for everyone to take a look, try stuff and send feedback.

In addition to the release of the framework and VS, we also put out the Data Services AJAX library in codeplex.

This is the last beta before we're done, so give this release a shot and let us know what you find!

-pablo

There are fresh news about ADO.NET provider support here, and there is an official looking statement from last December with more details here.

The ADO.NET Entity Framework is designed so that the upper layers of the system are database-independent. There has been many attempts at this in the past, with varying degrees of success.

I really like the Entity Framework approach because it goes all the way. It's not just a bunch of interfaces to make the API itself generic, but it's also machinery to make things that need to be provider independent so. For example, the Entity SQL compiler and the LINQ to Entities translator sit high in the stack and provides the *same code* for query translation across all databases. That means guaranteed consistency in syntax, something that was somewhere between hard and impossible before.

Of course, at some point we hand out the query expression to the provider for database-specific handling, and behavioral differences may arise there, but they are much more contained.

Coming back to the topic at hand, I've been looking at all the buzz about adoption of the ADO.NET Entity Framework provider model and how it enables access to many databases. This is a big deal...we kind of slowed down on database independence with previous versions of ADO.NET. This new round restores the database independent capabilities.

This is great news both for the Entity Framework and for Data Services (Astoria). It means that developers writing middle-tier code against databases and those creating Data Services get immediate support for using many databases in addition to SQL Server.

In the case of Data Services, the system itself is designed for data source independent beyond databases. If you have an IQueryable implementation, you're ready to go for read-only services (and you can add IUpdatable for update support). For custom data sources this flexibility is great. However, when you're targeting relational databases there is no need to go through the process of writing your own IQueryable (which is far from an easy task); we included rich database support out of the box through integration with the Entity Framework.

-pablo

The Astoria team builds the ADO.NET Data Services Framework and works on creative projects in the data+web space. In my completely biased opinion, it's quite a special team at Microsoft; we're given a lot of freedom to innovate; we use agile methodologies for development, cross the traditional lines between software and services constantly as we build infrastructure and frameworks that span them, and are constantly looking for new applications and creative solutions to existing problems.

We are now growing the team as we get ready to take on new challenges. We are looking for folks for Development, Quality Assurance and Program Management positions. If you are interested feel free to drop me a line, ask questions or send your resume directly. You can contact me through this blog's contact link or email me directly.

-pablo

As part of the Astoria design process we scanned through many topics, some of them are straightforward, some are hard but mostly mechanical, but there are some that become interesting, fundamental aspects to address.

I found the problem of concurrency control over REST interfaces very interesting to explore. The problem is actually well addressed in HTTP so it wasn't that we had to invent a bunch of stuff. It was more about learning what was out there, reading about good/bad stories, and then mapping it to our model and see how it fitted.

In the end we landed on what I think is a quite nice place. If you're interested, check out this write up in the Astoria team blog on the topic.

-pablo

As David Treadwell announced yesterday, we are starting to align the Windows Live services interfaces to use the AtomPub protocol, and to have a uniform set of conventions that are shared across internet services and the Project Astoria bits.

What does that mean? It means that starting now (and more in the future) you can interact with many consumer and infrastructure services in the same way. It also means that those Microsoft services out there and your own services -online or on-premises- created with the ADO.NET Data Services Framework look and behave in highly compatible ways.

You won't need to learn yet another API every time a new service comes out. Furthermore, we (or you, or services providers out there) won't need to create new APIs for every service. Highly uniform interfaces bring the opportunity for extensive client library and tools re-use across services of broad nature.

We've been talking about this for a while...and it's exiting to see the first pieces start to come in place.

If you want to know more, see live demos of services and tools working together, and chat about the details, we'll be going through all of this at Mix 2008. There is a talk specifically about this called "Accessing Windows Live Services using AtomPub" where I'll be telling the whole story and showing the thing working live.

More details about our Mix sessions here.

-pablo

Mix is one of my favorite events. It's a different kind of conference, many perspectives all in one place. Since it's all about the web, the Astoria team couldn't miss Mix 2008.

Mike, Andy and myself will be there giving talks, making announcements and hanging out in the open space area. So if you want to chat about the ADO.NET Data Services Framework or about data and services on the web in general come find us there!

Check out session times, titles and abstracts in the Mix 2008 session list.

-pablo

Jon Udell wrote a brief piece on how data is locked on servers behind UIs that were not designed for data sharing. He views this as "data friction"...it's just the perfect way to describe the problem.

I couldn't agree more with Jon's take. I would even take it further: an operation-centric approach to interfaces is good for closed systems or systems where semantics are completely centered around behavior; however, the data behind those operations is still somewhat locked-in. Data-centric APIs that expose a uniform interface for clients to consume is how the class of scenarios that Jon talks about will come to life. We can't tell upfront how all those applications out there will use our data, it's just too hard to predict, so a "function centric" interface just won't do. Instead, those systems need put out the data plus a way to interact with it that enforces proper semantics while not getting in the way of how a consumer wants to explore that data.

We're hoping to help at least a bit in that space with Project Astoria...we'll see how it goes :)

 -pablo

Greg is a great interviewer, I'm sure you'll enjoy this conversation. In this occasion we did a bit of history around LINQ, talked about LINQ and the Entity Framework with a some of a DBA perspective, and then we discussed Astoria (the ADO.NET Data Services Framework) briefly. We also jumped topics here or there and touched on snowboarding and a few other random non-computer-related subjects.

http://sqlblog.com/blogs/greg_low/archive/2008/02/05/sql-down-under-show-30-pablo-castro-linq-entity-framework-ado-net-data-services.aspx

-pablo

A few folks sent me email asking about idempotence on Astoria operations over the HTTP/REST interface, motivated by this post. I completely agree that idempotence is an important characteristic of an interface as it allows you to make a bunch of assumptions on the consuming side. That said, there are practical limits to what we can do, and there are some assumtions that come backed on the protocol to some extent.

Here is what the Astoria interface does from the idempotence perspective:

Whether operations on the underlying data (from a database or otherwise) are effectively idempotent is beyond the control of Astoria. While on the Astoria runtime side we define whether *we* consider an operation could be idempotent, the underlying data source in the end is who'll determine whether the operation is effectively idempotent or not.

• GETs are side-effect-free, so they are out of the discussion.
• PUTs are idempotent from the Astoria perspective, but the underlying data source could do something that breaks that (e.g. a trigger in a database that counts the number of changes...or simple a timestamp value).
• DELETEs are the same as PUT. Idempotent from Astoria, but the datasource could break that effect.
• POSTs are definitely not expected to be idempotent, and the HTTP spec explicitly calls that out. We use POST as this interpretation described in the spec: " Extending a database through an append operation ". Such operation is non-idempotent by definition. In the case where the client generates all the record, including the keys, you can re-try safely because the second try will fail if the first went through...but that's more circumstantial idempotence than real.

Does this sound reasonable?

-pablo

There is a number of folks that have been writing about Astoria. Collectively they built a lot of reference material that is probably the best reference point for getting started and learning about practical aspects around creating and using data services with the Astoria framework.

Jonathan Carter is writing an impressive multi-part series that's covering pretty much every usage perspective on Astoria.

Guy Burstein is also working on a multi-part series that starts here.

Mike Taulty is going for the low-level details and analyzing Astoria clients and servers in and out.

Christian Weyer, who knows way, way more stuff about WCF than I do, is using that to pull out great tricks such as how to host Astoria in stand-alone processes and more.

There is much more out there, just highlighting a few. So if you were looking for some reading material for the holidays, there is plenty out there :)

-pablo

We just released the December 2007 CTP of Project Astoria, or I guess I should say the ADO.NET Data Services framework as Mike pointed out.

This is an important milestone for Astoria, as this is the first release that is based on the real, production code base and not on the initial prototype that we built to explore the space. Formats, APIs, URLs and other interface elements have been extensively revisited so I would recommend that you switch to the new CTP for any future work.

This CTP is compatible with the RTM version of Visual Studio 2008, so you can finally move off intermediate builds of it when working with Astoria.

We're looking forward to hearing your thoughts on this new CTP!

-pablo