Pablo Castro's blog : Astoria

Every SharePoint 2010 server is a Data Services server

pabloc — Tue, 20 Oct 2009 06:26:00 GMT

I haven't been writing much here, mostly because I've been way too busy but also because I couldn't discuss publicly many of the things I'm doing. Now that SharePoint 2010 has been announced and its feature set published everywhere, I can finally discuss one of the coolest things we've been up to lately.

SharePoint is a repository of resources (list items and documents in document libraries) that are collected and manipulated collaboratively. Resources have a bunch of security and business logic attached to them, such as who can see each item, who gets to change it, or whether a particular column in a list needs to conform to a particular validation formula.

When SharePoint folks said they wanted a RESTful interface this was great news...the system is just a perfect fit. Not only it's a perfect fit for RESTful services in general, but also for Astoria in particular. In the end SharePoint is very data-centric in nature, it already supports queries and business logic as part of the uniform interface.

So we're really excited to announce that as of SharePoint 2010, every SharePoint server is an Astoria server out of the box. No configuration required or anything, just make sure the proper version of ADO.NET Data Services is in the box. For SharePoint 2010 beta, the "right" version is ADO.NET Data Services v1.5 CTP2. We'll put details out there for future iterations as they come.

Official write up in the Astoria team blog.

The SharePoint Data Service head is not just a side integration deal, it's a full-on REST-over-HTTP head for SharePoint. It supports browsing as well as modifying data using regular HTTP verbs (GET, PUT, DELETE, etc.), it does ETags for concurrency control, enforces business logic as part of side-effecting methods, and it handles the full range of Data Services conventions for URLs, Atom and JSON payload formats, etc. It also exposes full metadata like any other Astoria service, allowing Visual Studio and any other metadata-driven client to give you a great experience on the client side. Now if you need to get or manipulate data in SharePoint from any platform in any language, all you need is an HTTP stack.

The other cool aspect is that the SharePoint folks where able to build this entirely on top of public bits, using our new fancy provider model for advanced data sources I discussed here some time ago.

SharePoint joins a growing family of Microsoft products that enable users to share data through the use of a simple RESTful interface that follows the Astoria conventions when needed (e.g. URLs, structured data in Atom). Another example on the server side include SQL Server Reporting Services, which in SQL Server 2008 R2 can now render any report as Atom feed that follows the Astoria conventions (examples here). On the client, in addition to developer-oriented options such as .NET, Silverlight, AJAX, PHP, Java and more, we now also have PowerPivot (f.k.a. Gemini) which can pull data from any data service and do fancy analysis and publishing over it, making it trivial to bring data from Data Services into Microsoft Excel 2010 among other things.

I find this to be extremely important. A simple, uniform way of sharing data at the protocol level, and in a way that truly enables the lowest possible bar of entry, is key to enable broad integration across products and break data silos that form around applications.

I'll be talking about this at the SharePoint conference 2009 in Las Vegas this week, and at the Professional Developer Conference (PDC 2009) in late November. If you're attending any of these and this sounds interesting, these sessions will drill into plenty of details.

-pablo

We're on a roll: Java client for Data Services announced

pabloc — Tue, 29 Sep 2009 04:14:00 GMT

Just a few weeks after announcing the PHP toolkit for Data Services we now are happy to announce a Java toolkit for consuming Data Services that follow the Astoria RESTful data services pattern. The library ships as an extension to the existing Java Restlet library. I'm really happy to see more and more clients and servers come up that can be used to share data between systems.

You can check out the announcement in the Interoperability's Team blog.

-pablo

ADO.NET Data Services v1.5 CTP2!

pabloc — Tue, 01 Sep 2009 21:27:00 GMT

Yesterday we announced that the CTP 2 of the ADO.NET Data Services framework (yeah, Astoria) is available for download. We put in a ton of work on this release, ranging from adding better support for high-end services to making it easier to write applications in Silverlight by having cross-domain and proper data-binding support.

Official announcement is here, and includes details on where to download from, what to watch for (it's still a CTP, comes with fine-print :) ), and a list of the new and updated features.

Also, many folks in the Astoria Team have been writing a bunch of content to cover the new features, so watch the Astoria Team blog as well as related blogs for all sorts of samples and details to come up.

-pablo

PHP toolkit for ADO.NET Data Services

pabloc — Fri, 21 Aug 2009 19:42:00 GMT

Folks in the interoperability team at Microsoft just announced something they've been cooking for a while, developed Persistent Systems: a client PHP toolkit for Astoria services. It follows more or less the model of the .NET client where you can run a tool during development to get code-gen based on a data service metadata, and then it has a runtime library that can be used to send queries, obtain responses as PHP objects, track and submit changes, etc. They implemented both the Atom and JSON formats, and even threw in batching support.

The best part is that it's all written in PHP (both code-gen tool and runtime), so it runs in any environment where PHP and the toolkit dependencies (XML, XSLT, CURL) are available.

For more details check out the post in the interoperability team's blog, the kick off Channel 9 video, and the PHP toolkit project in codeplex.

-pablo

Busy week: ADO.NET Data Services v1.5 CTP coming and "Astoria Offline" preview out

pabloc — Sun, 08 Mar 2009 02:57:00 GMT

We announced two releases this week, kind of usual but it worked out this way.

The first one is the first CTP of ADO.NET Data Services v1.5. This is the next version of "Astoria" or the ADO.NET Data Services framework, and it includes a number of enhancements that were requested both by the developer community and by some internal partners. More details about this upcoming release here.

The second one is the preview release of an exploration project we have been calling "Astoria Offline". This project sits at the intersection between Data Services, Sync Framework, SQL Express/Compact and the Entity Framework. It will be interesting to see what folks think about it. I'm personally very interested in this space so I'm happy to see this finally out. As the announcement says, this is not an "official" product or anything like that, but more like an early experiment to understand the problem space. More details about the release and pointers to the download page available in the announcement in the Data Services team blog.

Looking forward to hear feedback about both of these releases and the technologies behind them.

-pablo

Adding support for JSONP and URL-controlled format to ADO.NET Data Services

pabloc — Thu, 26 Feb 2009 06:29:33 GMT

JSONP is a common way of making data accessible in client-side mashups even when the requests need to be cross-domain.

While the current version of the ADO.NET Data Services framework does not support this, it’s possible to build it on top. There are a couple of ways of doing this. Here is what’s probably the simplest way. There is some downsides to this approach, but overall is the most straightforward path to get there.

The default transport layer for Data Services is WCF, which has a many extensibility points across the stack. For the case of JSONP support, IDispatchMessageInspector comes in handy.

There are two things needed to support JSONP properly:

The ability to control the response format. Data Services uses standard HTTP content type negotiation to select what representation of a given resource should be sent to the client (e.g. JSON, Atom). That requires that the caller can set the Accept request header, which is not possible when doing the JSONP trick (which basically just uses <script> tags). We need to add the ability to use the query string in the URL to select format. (e.g. /People(1)/Friends?$orderby=Name&$format=json).
A new option to wrap the response in a callback if such callback was provided in the request (also in the query string). For example /People(1)/Friends?$orderby=Name&$format=json&$callback=loaded.

What we’ll do is register a message inspector and adjust the request/response when we see these new options coming in.

In order to support the $format=json option we can intercept the message before it gets dispatched to the Astoria runtime, at the IDispatchMessageInspector.AfterReceivedRequest method. If we see the query string option then we’ll a) strip it out from the URL so Data Services does not generate an error and b) change the “Accept” header to “application/json”, so the rest of the system just thinks that the client asked for a JSON response in the first place.

For the second part, where we need to wrap the response into a Javascript call if the $callback option was used, we have the IDispatchMessageInspector.BeforeSendReply method which gives us the perfect spot to rewrite the response. One unfortunate side-effect of this is that the response will get buffered and re-encoded; that said, in many cases this won’t make any noticeable difference.

Finally, we need to register the interceptor with WCF’s dispatchers. For that we create an attribute that implements IServiceBehavior, so we get called during service initialization. When we get called we can register our message interceptor.

The net effect is that if you include this code in your project, you just need to add a single attribute to your Data Service to make it support JSONP:

[JSONPSupportBehavior]
public class SampleService : DataService<ContactsData>
{
// your service code here...
}

Once that's in place you can use JSONP by adding $format and $callback to URLs, for example:

http://<host>/SampleService.svc/People?$format=json&$callback=cb

Of course, you can still use all the other Data Services URL options in addition to these.

The implementation and a small sample service are available at MSDN code gallery, here:

http://code.msdn.microsoft.com/DataServicesJSONP

-pablo

ADO.NET Data Services in Windows Azure: pushing scalability to the next level

pabloc — Sun, 02 Nov 2008 05:40:40 GMT

The announcement of Windows Azure is a big milestone for us in the Astoria team. We got a chance to add our little contribution to the platform by providing data service interfaces for a couple of the Azure services.

Currently there are two services that use the ADO.NET Data Services runtime: the Windows Azure Tables Service, which was announced this week as part of the whole Windows Azure story, and SQL Data Services, which has been around for a while but got a new experimental Data Services interface this week to coincide with the PDC.

These services -and others that will come in the future also based on Data Services- share a common aspect: they have extreme scalability requirements.

In order to enable them to use our Data Services server runtime we had to extend the data service framework to make it scale in various new dimensions. In the rest of this post I'll summarize some of the walls we hit and the changes we made to the system to handle these scenarios.

Things that already scaled

The Data Services runtime already incorporates many design principles that help with scalability.

For example, the system does not keep any required state between requests (we do cache stuff, but we can throw it away at any time), so scale out of the front-end servers of the storage systems is relatively straightforward. This allows the existing runtime to handle an arbitrarily large number of requests by throwing more front-ends to the problem (as long as the back-end systems can take it, of course).

Also, we don't make any assumptions around the size of the data and provide mechanisms to push-down filters in requests to the data source, so effectively in principle there are no limits to the amount of data that a data service may be fronting.

Hitting the scalability wall

While some things scaled, there are certain aspects in which we ran into a scalability wall that required a number of changes in the system.

Using .NET types to represent the shape of the services is great in a single application, but not-so-great if you have millions of users with hundreds or thousands. We needed another way of describing the "shape of the data in the service", that is the metadata or schema of the service.

Since you can't practically create a distinct type for every user/application/table in the system, that means that the instances of objects that represent data flowing through the data services runtime cannot be of a specific type for each entity type. Instead, we needed independent of the flow format with respect of the declared types.

Metadata and service schema

The data services runtime needs to know the "schema" of each service it exposes. That is, the list of entity-sets, the entity-types of the instances living in those entity sets and the relationships between the various entities.

In a typical data service, the service exposes data for a given application or domain-specific service, so the schema of the service is known and static (within a given version at least) and all the front-end servers simple share the same schema.

The way a service author specifies the schema of a service in the shipping version of the Data Services runtime is by using .NET classes or an Entity Framework model (which in turn generates .NET classes). That works great for application developers, because .NET classes are a simple and natural way of defining the shape of your objects.

Now, if the requirement is to be able to handle millions of applications, each of which can have hundreds or thousands of tables, does that mean that we have to create a .NET type for each service and for each table, and the corresponding number of properties and such? And if so, since the front-end systems are stateless and potentially don't have any affinity to parts of the data, does that mean that any given system may end up having to load up millions of types in memory? To complicate things further, once you load an assembly (the only container in which .NET types can exist), you can't unload it unless you unload the AppDomain.

.NET types are a great solution for the scenarios where the schema is known and more or less bounded, and will continue to be the primary way of creating services in that context. However, we needed something else to handle the high-end side of the spectrum.

To address this need we introduced a new interface that data services can optionally implement. We already had the internals of the system organized more or less like this, but didn't expose it in the first release. The idea is that there is main split between the "upper half" of the runtime that deals with URL translation, LINQ expression tree generation, interceptors, policies and all aspects that make a Data Service look like a Data Service. The "bottom half" is the "data service provider", and is responsible for describing the shape of the service among other things. There are two built-in "data service providers", the Entity Framework provider which is what you use when you create a data service over an Entity Framework model, and the reflection-based provider which is what you use when creating a service on top of an arbitrary object graph. With the new change you can now create new implementations of these data service provider thingies that can obtain and manage metadata any way they want.

The way we interact with the provider is carefully designed to avoid requiring long term state state in the provider or the consumer of the provider in any way, while at the same time allowing the provider to do caching of metadata and control information if desired.

First, we never hold on to information returned by the provider beyond the scope of a single request. So for all we know the provider could be reloading all the metadata in every request. In practice, providers will probably cache this metadata in some way or another.

Second, we load metadata on demand and piecemeal. For example, during URI translation we do a small scale version of the usual binding and semantic analysis that any compiler does, and for that we need metadata. In those cases we don't load all the metadata, but only the pieces we need to do type checks, symbol lookups, etc.

Making metadata dynamic

Another aspects around metadata to consider is the fact that the shape of one of these data services can be altered at any time. For example, the Azure Table service has the concept of tables, and you can add and remove tables whenever you want.

The new scheme with custom data service providers make this possible because we don't remember anything at all across requests. So all the provider needs to do when the underlying shape of the data changes is report a different schema on the next request, and the data services runtime will happily take it.

With .NET types this would have meant creating and re-distributing new types (or creating them on demand on each node), and dealing with not being able to unload the old types from memory. Clearly not an option at this scale.

Flow format independence

With the addition of the "data service provider" interfaces we no longer have .NET types to use for the instances of each entity-type that flows through the system (e.g. from the data source to the runtime via the IEnumerables returned in LINQ queries, and from there to the serialization stack).

Another important change we made in the system is that we no longer assume anything about the shape of each CLR object returned by the query. We treat instances just as "object" all over the code base. When we need to access a member, we use methods in the data service provider interface to do that, imagine something like GetPropertyValue(object o, string name).

That means it's now possible to use some form of generic record type across the system. Not only this avoids the need for specific types, but also allows providers to piggyback control information in the instances themselves, avoid copies from the original format into CLR objects just to flow them through our runtime and a few more benefits.

Impact on LINQ expressions

While having flow format independence is great, it did complicate things for query formulation.

We typically translate URLs to expression trees, and since we have all the CLR types in the server that correspond to the entity types, all the expression trees are nice and clear.

When we're operating against unknown types we can't generated "typed" expression trees anymore. In those cases we still produce expression trees, but the member-access operations (and certain operators) are represented using custom calls to a well-known set of static members. The providers that enable this feature need to know about this and do proper translation of these expression trees.

Extension to the data model

We did one more major change that while it's not directly related to scalability it has a lot to do with the database/storage services in Windows Azure.

In the current version of Data Services types are "closed" in the sense that they have a structure that's final. You list a set of properties for each type and instances of that type cannot have properties added dynamically.

It turns out that the data services we have online have a more flexible model, where each entity has a fixed portion but also a dynamic portion. Typically the fixed portion includes a key or some sort and a version property. The dynamic portion is a property bag where you can add any name/typed-value pair.

We call these types that can be extended on a per-instance basis at runtime "open types". We introduced support for open types in the Data Services runtime such that you can mark a given entity type as "open" in metadata and that would cause the system to allow unknown properties to be set, as well as the use of unknown properties in queries (e.g. in filter predicates).

There is a lot of details around open types that I won't go into here, maybe the topic for another post, but I wanted to point out the change because it was a significant addition.

What do these changes mean for developers?

What does all this mean to current users of data services. Well...not much other than some background on how the system is evolving. Other than open types, services created with custom metadata/custom flow formats are indistinguishable from the ones created the "classic" way.

Furthermore, we will preserve the existing model where creating a service based on some .NET objects or an Entity Framework schema is really straightforward, and we consider that our primary scenario for developers.

At the same time, addressing the needs for the highest-end services out there is important, so many (if not all) of these changes will eventually make it into the shipping product so that other folks out there can use them if they chose to. Beware that these interfaces are not designed to be "nice", but rather optimized for control and efficiency, so it may not be exactly a fun experience, but you'll get all the scalability you'll need out of them.

-pablo

Now you know...it's Windows Azure

pabloc — Wed, 29 Oct 2008 04:02:29 GMT

Since we shipped ADO.NET Data Services v1 in .NET 3.5 SP1 (and actually before that as well) I've been working on a few things that I could share (such as offline/sync support for data services) and some that I couldn't discuss publicly until all the big plans where announced.

This week at PDC Microsoft announced Windows Azure. A lot has been written about it, so I won't go into the details.

On our side, in the data services team, we made our small contribution to the big picture.

The Windows Azure table service is a structured storage facility that's part of the core part of Azure. Access to the table service is done through a data-services compatible RESTful interface that uses the Astoria conventions over an HTTP binding. That means that you use either any client with an HTTP stack to talk to it, or you can use the ADO.NET Data Services client, which does a nice job exposing data as .NET objects, letting you write simple queries using LINQ instead of URLs, etc.

Another cool thing about the table service (and the blobs and queuing service for that matter) is that they are accessible both from the virtual compute environment and from anywhere in the Internet. In both cases, if you're using .NET, you can use the data services client to interact with it. In the case of code running in the Windows Azure hosting environment, the client is already present (the environment includes .NET 3.5 SP1) so you can use it without worrying about taking new dependencies.

You find out more about the table service you can watch Brad's PDC session for a discussion of the service itself, and this other session than Niranjan and I did together (or "will do" if you're reading this before Wed in the PDC week) for a drill down on how to program the Windows Azure table service. If you're not at PDC no worries, these talks are accessible to online.

On the next layer up from the core, the Windows Azure service layer, SQL Data Services also is making big announcements in this PDC. We're introducing more relational capabilities into the system, and also experimenting with a data services-compatible interface. This PDC talk from Patrick will discuss and demo the new interface, and you can follow how this effort goes here.

-pablo

Trying something: short videos in our design notes

pabloc — Mon, 29 Sep 2008 01:32:00 GMT

During the design of Data Services (Astoria) v1 we did the transparent design thing. We're quite happy with the result, we got a lot of feedback and were able to adjust many aspects of the project based on that.

Now that we're in full swing with v2 design work, we're going to be posting regularly again (hopefully :).

Andy got a new tiny camera and he had to use it for something...so he suggested that we do short clips that may help explain design points. Sometimes a short explanation is just much better than a bunch of writing.

The first post with the write-up + short video format discusses "Friendly Feeds" and is here. If you get a change to take a look, let us know what you think about the new style (and friendly feeds)!

-pablo

Coming from under the rocks just to celebrate for a bit

pabloc — Tue, 12 Aug 2008 05:46:00 GMT

I've been sort of under a rock for a while, but I thought I'd come out for a minute to celebrate. Today we made available .NET 3.5 SP1 and Visual Studio 2008 SP1. There are two components in the release I spent a bunch of time on, which interestingly enough have very different origins and get to RTM through very different processes. One is the ADO.NET Entity Framework, which has been cooking for several years and survived controversies, comparisons with non-shipped previous attempts and other natural disasters; the other one is the ADO.NET Data Services Framework or Project Astoria, which was built, well...fast.

I won't go into details of the release, folks have discussed the Astoria, Entity Framework and general data-related features in the release already.

Why have I been under a rock? In the last few months I've been spending time working on various things related to Astoria, online services and data interfaces. Some I can discuss, some will need to wait a bit until the stakeholders are comfortable to talk about it publicly.

Moving Astoria as a framework forward: we were ready (modulo bug fixing and last minute tweaks) some time ago, and we've been thinking about the next steps for the Astoria framework. In Mix 08 we mentioned that we were working on "Astoria Offline" and showed a prototype. We've been working hard in that topic. There is also a bunch of features we want to take on for the next release. I'm sure we'll post something in the Astoria blog at some point about our thinking and give a change for folks to give feedback.

Online services: as you can imagine there is a number of things going on around online services these days, and a number of them involve Astoria one way or the other. I've been working with several of them, varying from providing guidance all the way to writing custom "v.next" versions of Astoria to experiment with their needs. An example of these efforts is the work we're doing to align SQL Server Data Services and the ADO.NET Data Services framework. We would like to see them as the "service" and the "framework" pieces, both using the same HTTP interface, same client interfaces, etc., so we've been spending a bunch of time exploring how to bring them together.

Anyway, there, a bit of a celebration.

-pablo

Data Services and Entity Framework beta bits available

pabloc — Mon, 12 May 2008 20:48:00 GMT

The news are out. The ADO.NET Data Services Framework (Astoria) and the ADO.NET Entity Framework will be shipping as part of .NET 3.5 SP1, and the Beta 1 release is now available. All the official blogs discussed the details already, including the Astoria team blog, ADO.NET team blog, Scott's, and many others out there.

Folks out there trying Astoria and the EFx have been working on bits from last December for a while. Finally we have a newer release for everyone to take a look, try stuff and send feedback.

In addition to the release of the framework and VS, we also put out the Data Services AJAX library in codeplex.

This is the last beta before we're done, so give this release a shot and let us know what you find!

-pablo

Provider availability enables Entity Framework and Data Services over many database vendors

pabloc — Wed, 07 May 2008 20:14:00 GMT

There are fresh news about ADO.NET provider support here, and there is an official looking statement from last December with more details here.

The ADO.NET Entity Framework is designed so that the upper layers of the system are database-independent. There has been many attempts at this in the past, with varying degrees of success.

I really like the Entity Framework approach because it goes all the way. It's not just a bunch of interfaces to make the API itself generic, but it's also machinery to make things that need to be provider independent so. For example, the Entity SQL compiler and the LINQ to Entities translator sit high in the stack and provides the *same code* for query translation across all databases. That means guaranteed consistency in syntax, something that was somewhere between hard and impossible before.

Of course, at some point we hand out the query expression to the provider for database-specific handling, and behavioral differences may arise there, but they are much more contained.

Coming back to the topic at hand, I've been looking at all the buzz about adoption of the ADO.NET Entity Framework provider model and how it enables access to many databases. This is a big deal...we kind of slowed down on database independence with previous versions of ADO.NET. This new round restores the database independent capabilities.

This is great news both for the Entity Framework and for Data Services (Astoria). It means that developers writing middle-tier code against databases and those creating Data Services get immediate support for using many databases in addition to SQL Server.

In the case of Data Services, the system itself is designed for data source independent beyond databases. If you have an IQueryable implementation, you're ready to go for read-only services (and you can add IUpdatable for update support). For custom data sources this flexibility is great. However, when you're targeting relational databases there is no need to go through the process of writing your own IQueryable (which is far from an easy task); we included rich database support out of the box through integration with the Entity Framework.

-pablo

The Astoria team is hiring!

pabloc — Wed, 23 Apr 2008 09:15:00 GMT

The Astoria team builds the ADO.NET Data Services Framework and works on creative projects in the data+web space. In my completely biased opinion, it's quite a special team at Microsoft; we're given a lot of freedom to innovate; we use agile methodologies for development, cross the traditional lines between software and services constantly as we build infrastructure and frameworks that span them, and are constantly looking for new applications and creative solutions to existing problems.

We are now growing the team as we get ready to take on new challenges. We are looking for folks for Development, Quality Assurance and Program Management positions. If you are interested feel free to drop me a line, ask questions or send your resume directly. You can contact me through this blog's contact link or email me directly.

-pablo

REST and Concurrency Control

pabloc — Wed, 23 Apr 2008 04:13:00 GMT

As part of the Astoria design process we scanned through many topics, some of them are straightforward, some are hard but mostly mechanical, but there are some that become interesting, fundamental aspects to address.

I found the problem of concurrency control over REST interfaces very interesting to explore. The problem is actually well addressed in HTTP so it wasn't that we had to invent a bunch of stuff. It was more about learning what was out there, reading about good/bad stories, and then mapping it to our model and see how it fitted.

In the end we landed on what I think is a quite nice place. If you're interested, check out this write up in the Astoria team blog on the topic.

-pablo

Unifying service interfaces

pabloc — Sat, 01 Mar 2008 01:37:00 GMT

As David Treadwell announced yesterday, we are starting to align the Windows Live services interfaces to use the AtomPub protocol, and to have a uniform set of conventions that are shared across internet services and the Project Astoria bits.

What does that mean? It means that starting now (and more in the future) you can interact with many consumer and infrastructure services in the same way. It also means that those Microsoft services out there and your own services -online or on-premises- created with the ADO.NET Data Services Framework look and behave in highly compatible ways.

You won't need to learn yet another API every time a new service comes out. Furthermore, we (or you, or services providers out there) won't need to create new APIs for every service. Highly uniform interfaces bring the opportunity for extensive client library and tools re-use across services of broad nature.

We've been talking about this for a while...and it's exiting to see the first pieces start to come in place.

If you want to know more, see live demos of services and tools working together, and chat about the details, we'll be going through all of this at Mix 2008. There is a talk specifically about this called "Accessing Windows Live Services using AtomPub" where I'll be telling the whole story and showing the thing working live.

More details about our Mix sessions here.

-pablo