ADO.NET Data Services Team Blog : Design Notes

Design Notes: Row Count

dpblogs — Tue, 16 Dec 2008 00:18:34 GMT

One of the scenarios we have heard feedback around is the ability for the a client of a data service to determine the total number of entities in a set without having to retrieve them all. The following video discusses this scenario in more detail and some of our thoughts regarding how to make the experience better.

Astoria Design Walkthrough: Introducing $count as an URI query option

Additional design details:

- The count value would be calculated after $filter expressions are applied. For example /Customers?$filter=City eq 'London' would return a count value made up of only the customers living in London and not all the customers.

- If a query included $top, $skip or $orderby, the count value would be unaffected. For example /Customers?$top=1&$skip=2&$orderby=Name would return the total number of customers stored by the service.

- If a query includes a $expand, then the count applies to the outer entity set. For example /Customers?$expand=Orders would return the number of customers stored by the service

What do you think? Would this be useful in your data services applications?

Mike Flasko

ADO.NET Data Services, Program Manager

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Alpha preview of Project Codename "Astoria Offline" coming soon

dpblogs — Tue, 04 Nov 2008 02:01:58 GMT

As we already discussed in a previous blog post, one of the problem spaces related to data services we’d like to take on is synchronization-capable services, enabling offline scenarios. We’re still in the early stages of the project, where direction can be adjusted and good feedback can be very influential.

Giving good feedback without actually having something to look at it’s tricky, so we want to ship bits. Earlier at the PDC talk that showed this technology end-to-end for the first time we announced that we’ll be shipping a very early release of the bits that extend our entity platform, including the Entity Framework and ADO.NET Data Services, with synchronization capabilities through integration with the Microsoft Sync Framework and a nice toolset that makes development of offline-capable applications reasonably straightforward.

Here is a brief video with Waseem and myself discussing the release. Waseem was probably the most involved developer so far in this effort and has been dealing with a lot of the challenges that we hit as we put all the pieces together and build new ones on top.

Astoria Design Walkthrough: Alpha preview of Project Codename "Astoria Offline" coming soon

Just like the first release of “Astoria” almost two years ago, the goal if this alpha release is to paint the picture of what we want to build, what scenarios we consider interesting and how we want to approach the tooling around it. It’s definitely not a goal to ship something that’s usable in production or even something that can be used to start writing real applications now. Things will change substantially as we hear feedback from all of you.

You can expect the release to ship before the end of this year, and to include both runtime and tool components for the whole thing. If you care about synchronization and offline applications, giving this release a try and sending us your thoughts, comments and suggestions would help build the right technology in this space.

Pablo Castro
Software Architect
Microsoft Corporation

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Astoria futures: offline-enabled data services

dpblogs — Thu, 23 Oct 2008 04:32:00 GMT

We mentioned that we were doing some early thinking of “Astoria Offline” back in Mix 2008, where we even demo’ed an early proof of concept. Now we’ve been working on various design aspects of Data Services for its future versions, and synchronization/offline support is one of them. It’s still an experimental thing with no official home or release vehicle, so this is the best time to follow the design process if you find the scenario interesting, as this is when it’s easiest to influence the direction we’ll go for.

A short way of describing this can be: “imagine you can point Visual Studio to a data service and say ‘take it offline’, and things just happen”.

Of course, the real world is more complicated than that :-)

Astoria Design Walkthrough: Thinking of a future with sync & offline

In this first note I’ll just touch on the scenarios we want to hit and go over a few guiding principles. In future posts I’ll elaborate more on the details.

We have many scenarios in mind for this infrastructure. The ones we’re thinking of tackling first:

· Outlook type of apps: I’m sure there is a fancier way of saying this, but anyone that has used Microsoft Outlook knows what I mean. The application is basically a 1-tier app that interacts with a local (embedded) database. In the background -and independent from UI activity- 2-way synchronization with a data service (e.g. a Microsoft Exchange server) happens. Often sync’ing against a database is not quite what you want…”Astoria Offline” will let you sync against your data-service layer, where the usual business logic/validation/etc will run just like in the online path.

· The description above sort of implies that server and client are built in collaboration, perhaps as part of the same development team. That’s certainly an scenario and we can do some things easier when that’s the case. But the other scenario we want to tackle is when client and server in a synchronization relationship are independent from each other (e.g. sync a service that’s just available for sync on the web).

· Local replicas of cloud-stored data: as more online services offer structured storage capabilities, and more of them use the Data Services REST interface, it becomes more interesting to be able to synchronize that data locally either for latency reduction, offline operation or other reasons.

· Data consolidation: if you have multiple data services that expose data from a variety of sources (some databases, some online/”cloud” stores, some custom repositories), you may want to synchronize a slice of data of each store to a local database, and then work with the data locally.

A couple of guiding principles:

· We will stick to a simple and open interface. What that means is that while we will definitely build a nice end-to-end integrated story for Visual Studio, it will be on top of a well-documented underlying data exchange using just HTTP and known formats. Anybody with an HTTP client and enough knowledge of our sync strategy should be able to synchronize with a data service.

· Data independence will remain there for sync as it is already for online access. Today when you access a data service the interface is the same regardless of whether the service is backed by a database, a cloud store, some custom application or whatever. With sync, the same applies. If the data service is sync-enabled you can sync with it, no matter what backs it.

· We are targeting data services for structured stores and business applications. That implies certain level of sophistication in the shape of data, such as assuming cross-item dependencies, store-level and application-level constraints that dictate consistent states of data, the need for making partial progress during synchronization, etc. Such support does come with some extra complexity, but we think it’s the right target.

We’re just taking on this space, so any feedback you may have is good. Are our initial scenarios interesting? Do you need this thing at all? Does the initial direction we’re looking at sound reasonable?

btw – if you are going to be at PDC, we have a full talk on this at the event.

I hope this “short video” format that Andy wants to do for our design notes adds a good little twist and makes them more interesting.

Pablo Castro
Software Architect
Microsoft Corporation
http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Making Feeds Friendly

dpblogs — Mon, 29 Sep 2008 01:17:49 GMT

The goal of Astoria in V1 is to easily expose a data source using the REST approach to the web (with support for concurrency, versioning, auth, etc). In general, the way we think about serialization formats (ie. how we represent entites on the wire) is that they are a tool and an application will select the best tool (ie. serialization format) for the job. For example, AJAX developers will likely opt to interact with Astoria services using our JSON serialization format while .NET developers may choose AtomPub since our .NET client library uses that format under the covers. In V1, Astoria supports using the AtomPub and JSON formats as a general purpose data exchange representation and has fixed requirements regarding how an entity is to be represented using the formats. While this works well for a number of app scenarios, we’ve received feedback regarding use cases (one example being mashups) where an app/tool/etc is written to consume general Atom feeds and can then make assumptions about the data given (ie. what the title is, who was the author, etc). This area of thinking/exploration (uses of feeds in the wild and how to make Astoria’s feeds just work in certain scenarios) has been dubbed by our team as “Friendly Feeds”.

One of the specific scenarios we’ve discussed is a mapping site (ie. virtual earth, etc) which accepts feeds as an input and then displays the feed information on the appropriate spots on the map. In this scenario the mapping site would pull metadata from the well known Atom elements (title, author, content, etc), but also needs to locate geo information to be able to determine where on the map to place the data. For this we’ve seen some feeds start to embed micro formats in feeds such as georss. To make this more concrete, imagine being able to drive the example shown here via an Astoria feed.

We are still thinking about this space, but right now the high level goals we see as necessary for such a feature are listed below. What do you think?

Goals:

· Enable feeds to be easily customized such that a generic feed viewer can render an Astoria feed

· The feed customization approach could be applied to feed-based formats other than Atom. JSON serialization will not support this type of “Friendliness”

· Enable entity properties to be serialized as microformats within a feed

· Serialization customizations “just work” with V1 data service client libraries wherever possible. Changes that require a protocol version number increase should be explicit and off by default

· The ADO.NET Data Services client library (for .NET Fx and Silverlight) would be able to roundtrip a “Friendly Feed” in the same way the client roundtrips data in Astoria V1.

· We would also teach our code generation tools to understand such “Friendly Feeds” and generate the appropriate clients.

· It is NOT a goal to make ADO.NET Data Services into a blog server -- there are better domain-specific tools for that

Oh, I almost forgot……a nice benefit is that with this feature Astoria Atom feeds could likely be made directly viewable in IE. Folks who have inspected AtomPub-based feeds in web browsers today know what I’m referring to.

Below is a short video we recorded that discussed this area as well. Andy (Astoria Dev Lead) got a new camera and came up with the idea of adding these videos. Let us know what you think of having short video snippets such as this to accompany our design notes as we blog them....

Astoria Design Walkthrough: Friendly Feeds Part 1

In a future post, I'll include some specific examples to show how we will enable this in our server runtime and client libraries.

Cheers,
Mike Flasko

ADO.NET Data Services, Program Manager

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Merge vs. Replace Semantics for Update Operations

dpblogs — Tue, 20 May 2008 19:45:49 GMT

So far Astoria has used “merge” semantics for update. That is, an “update” operation (an HTTP PUT request) replaces the values of the properties for the target entity that are specified in the input payload; this applies both to properties and links. If a property or link is not present in a PUT operation, it means “leave it with its current value”. To null-out a property or link it has to be present in the PUT body and has to have a null marker (null attribute for properties, null href for links).

While supporting merge is important and will remain part of Astoria, there are scenarios where we need “replace” semantics. That is, an operation where the entity properties are entirely replaced by those in the request body, even if the request body is partial. In particular, the AtomPub protocol requires PUT to have replace semantics.

Operation semantics

In a “replace” operation, each property in the target entity either takes the value specified in the payload (if any) or its default value. The meaning of “default value” for a “replace” operation is server implementation-specific. The Astoria server will likely use the CLR default values for each value type and null for references.

For operations on primitive values (e.g. PUT /Customers(123)/CompanyName), there is no practical difference between “merge” and “replace”.

A “replace” operation, just like a “merge” operation, cannot specify different values in the key properties. That is, keys remain non-updatable in “replace”. Similarly, a “replace” operation is subject to the same requirements from the ETags perspective as a “merge” operation.

About links inside entity payloads: “replace” operations replace the properties in entities themselves, not its links. If you include a link in a “merge” or “replace” operation we’ll wire it up, but if you don’t it’ll maintain the existing links.

About directly-addressed links: links are currently atomic values (just the link itself), so there is no difference between replacing it and merging it when operating on link resources (e.g. /$links/…). Astoria servers should support both operations but keep identical semantics. The rest of this discussion won’t touch on links as stand-alone resources anymore.

Note that these operational semantics are the same regardless of the actual format (Atom, JSON, etc.) used to represent the resources being exchanged.

Finally, this in general does not apply to service operations, as the meaning of service operations is service-defined. It’s interesting to think about whether in the Astoria server library we introduce some mechanism to allow it to expose a MERGE-enabled URL, but that would still require the user-implementation to make sense of it.

HTTP interface

AtomPub specifically requires HTTP PUT to mean replace. So we adjusted the way Astoria interprets PUT to mean “replace”.

In order to request a “merge” operation we have two options:

1. Introduce a new HTTP method, “MERGE”, and rely on verb tunneling (POST + a header) for the cases where custom methods are not allowed. There has been talk about a PATCH verb in multiple circles including the AtomPub community, but it seems to be going in a somewhat different direction.

2. Introduce a new custom header, “DataServices-Merge” or something, that when set to “1” in a PUT request indicates that the server should merge the body with the server entity instead of replacing it.

While we’re not thrilled with the idea of introducing a new HTTP method, overloading PUT with an extra header seems to be very problematic. If anything else, a server that does not support “merge” through headers would see PUT as a regular “replace” request and perform an operation that’s not what the client expected. Also other things break. For example, if a server sees an actual MERGE request and cannot handle it then it can respond with 405 – method not supported.

So we’re leaning toward MERGE and tunneling (we already support tunneling for PUT/DELETE in Astoria servers and clients).

Responses to “merge” and “replace” requests are identical.

.NET client

Using “merge” from the client has several advantages:

1. The server does not know what a client considers a “whole” entity. The client may be using entity types that contain a subset of the properties of the server-side version, either due to versioning mismatches or because the client is not interested in all of the properties.

2. The client is pretty much required to use “merge” to make links work. Since an entity might have been brought down to the client without its related entities expanded, one or more links won’t be present, and if we used replace we’d lose information on the server.

Based on the above, it would seem that the client should do “merge”, which will effectively result in “replacing the subset the client knows about” because the client always sends all the fields.

The problem now is servers that don’t implement “merge” operation. One option is to require “merge” for the client to work, but that leaves too many interesting scenarios out. Since we already had a SaveChangesOptions enum that’s used as an argument to SaveChanges/BeginSaveChanges, we introduced SaveChangeOptions.UpdateAsReplace to indicate that you want the client to use PUT.

AJAX client

The AJAX client’s DataService.update method can be extended with a new argument “UpdateAsReplace” that enables use of replace when set to true. By default we would continue to do “merge” (this requires tweaking the library because currently DataService.update generates a PUT request).

If the new boolean adds one too many arguments for update(), alternatively we could add a knob to the DataService class, we still made this change.

Astoria runtime-data source interaction

The only change in the interaction between Astoria and the data source should be that for the resource being replaced the system will call IUpdatable.ReplaceResource instead of IUpdatable.GetResource.

Pablo Castro
Software Architect
Microsoft Corporation

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Optimistic Concurrency & Data Services

dpblogs — Wed, 23 Apr 2008 01:09:37 GMT

Different applications have different requirements around consistency and how concurrent modifications are handled. I’ll oversimplify and put all these applications in two buckets: either you care about controlling concurrent changes or you don’t.

If you’re creating a REST interface to your data and don’t care about concurrency (e.g. no deep consistency rules, or nice units of change that change in whole consistent ways), then you can use the basic HTTP methods to retrieve (GET) and manipulate (POST, PUT, DELETE) resources directly without any more context than the representations of your resources. You get “last one wins” semantics on updates in this case.

On the other hand, if you do care about concurrency in your REST interface, there are more aspects to take into consideration. If your resources are atomic (no further structure that’s interesting from the concurrent changes perspective than the resource as a whole), then you can have an out of band mechanism for creating a “version number” for each resource -typically a monotonically increasing number- and use HTTP’s existing mechanism to ensure you overwrite stuff that you know about. In HTTP you can stick an “entity tag” or ETag to your responses that contain an opaque value used to denote the version or state of a resource. Later on, when you want to modify a resource, you can use that value in a “if-match” request header to make sure that your knowledge about the state of the resource you’re modifying is still current. If it’s not the resource in the sever would have an ETag that won’t match the one you provided and you’d get back a 412 “Precondition failed” status code. All that is standard HTTP 1.1 stuff described in RFC 2616. (ETags are also used for caching and conditional gets in addition to the scenario I described here).

Now, REST data services that expose structured data have to deal with various challenges beyond the basics, which I’ll go into details below. While I discuss this in the context of the ADO.NET Data Services Framework (Astoria), I’m sure some of these problems apply to a broader set of applications.

Creating ETags: concurrency tokens

The data services framework has to deal with the fact that we don’t control the data sources that we expose through the REST interface. Sometimes each entity that we turn into a resource will have a nice clean property that’s a timestamp or similar and maps perfectly to ETag semantics (e.g. whenever we change the value in a significant way the value of this property changes). However, often the schema of the underlying data is not under the control of the service developer so we have to work with what we have. What that means in practice is that you can tell the data services framework which properties of each entity type are “concurrency tokens”. Changing those values means that you chanced the version of the resource.

The way you do that in the framework is by using an [ETag(props…)] attribute in your class or an annotation in your EDM schema. For types that don’t have any concurrency tokens we won’t generate ETags for the responses for those types, and they get “last one wins” update behavior.

Once you indicated which property or properties are your concurrency tokens we can produce ETags by using the values of those properties for the particular instance we’re returning.

During update the data services runtime works with the data source to determine whether the concurrency token values that were marshaled through ETags and if-match headers still match, and if so perform the update/delete operation. If they don’t match a 412 response is sent to the client.

Including ETags in headers and/or payloads

The HTTP spec describes the ETag response header to transport the entity tag for a given resource. That works great for us for cases were we respond with a single entity (e.g. an entry in Atom terms), but it doesn’t when we return a collection of entries from a URL (e.g. an Atom feed). For the latter scenario, we include the ETag as part of the resource representation (in the entry for Atom, in the “__metadata” property for JSON), for example:

</entry>

Validation during side-effecting operations

Concurrency tokens are validated whenever you perform an operation that affects the state of an existing resource. In the data services REST interface that means HTTP PUT and DELETE methods.

As I mentioned above, validation happens during update processing by extracting the “original” values from the ETag (which was sent back through the if-match header) and comparing them with the data in the data source. If they are the same, we consider the whole resource the same and proceed with the modification.

An interesting question is whether presenting an ETag in a if-match header should be mandatory for resources that have concurrency tokens. Put another way: should the decision of whether it’s ok to potentially overwrite changes based on state knowledge be up to the client or restricted by the server? The HTTP spec defines a special value of “*” for the if-match header that effectively means “any value will match”. The behavior that we are planning for is that if an entity type has concurrency tokens then we’ll always require an “if-match” header in modification operations. The header value can be an actual ETag obtained through a GET request or “*” meaning “I know this type supports concurrency control, but I’ll overwrite it anyway”.

Almost, but not quite, a perfect match

HTTP ETags and conditional operations are almost a perfect match to what we need to handle concurrent activity in RESTful data services. There are, however, a few glitches. This is where we get into the fine-print that’s not necessarily popular knowledge. Mike brought up many of these details I wasn’t aware of.

ETags can be “strong entity tags” or “weak entity tags”. Weak ETags are very similar to what happens when we have entities for which only some properties are designated concurrency tokens. From section 13.3.3 of the HTTP spec:

“However, there might be cases when a server prefers to change the

validator only on semantically significant changes, and not when

insignificant aspects of the entity change. A validator that does not

always change when the resource changes is a "weak validator." “

The problem is that weak ETags only apply to GET operations, they cannot be used for PUT/DELETE which is what we’re trying to do.

For cases where you own the data, the data services framework can expose a compliant interface by using constructs such as timestamps (if using a database as a data source), where any change in the entity will reflect in the ETag. You can also used a relaxed form of ETags where the entity might change but the ETag stay the same. It’s not completely HTTP compliant and may confuse intermediate systems, but it may be your only option in some scenarios.

As always, thoughts and feedback is welcome.

Pablo Castro
Software Architect
Microsoft Corporation

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

IUpdatable & ADO.NET Data Services Framework

dpblogs — Fri, 11 Apr 2008 00:40:00 GMT

Astoria service allows reading/querying of data via the already-established IQueryable interface – this helps in abstracting Astoria from the underlying data source. But there is no existing interface for the update operations (CUD – create, update, delete operations). Hence we came up with IUpdatable interface to support CUD operations and support read-write services.

One of the main design goals while designing the IUpdatable interface was to make it resource independent. In other words, the methods that return objects representing resources can return anything – for Astoria, the returned object is a opaque object that represents the resource being asked, and whenever we want to use the resource (reading/updating a value from the resource), we will pass the same opaque back to IUpdatable. The actual implementation of IUpdatable needs to track the mapping between this opaque object to the actual object it represents. The only time we need the actual clr instance of the resource is when we need to serialize the object and we call a specify method on IUpdatable (ResolveResource) for that.

Let take a quick look at IUpdatable interface

public interface IUpdatable

{

/// <summary>

/// Creates the resource of the given type and belonging to the given container

/// </summary>

/// <param name="containerName">container name to which the resource belongs</param>

/// <param name="fullTypeName">full type name i.e. Namespace qualified type name of the resource</param>

/// <returns>object representing a resource of given type and belonging to the given container</returns>

object CreateResource(string containerName, string fullTypeName);

/// <summary>

/// Gets the resource of the given type that the query points to

/// </summary>

/// <param name="query">query pointing to a particular resource</param>

/// <param name="fullTypeName">full type name i.e. Namespace qualified type name of the resource</param>

/// <returns>object representing a resource of given type and as referenced by the query</returns>

object GetResource(IQueryable query, string fullTypeName);

/// <summary>

/// Gets the resource of the given type that the query points to. The resource returned contains the default values,

/// and not the value as present in the server

/// </summary>

/// <param name="query">query pointing to a particular resource</param>

/// <param name="fullTypeName">full type name i.e. Namespace qualified type name of the resource</param>

/// <returns>object representing a resource of given type and belonging to the given container and containing default values</returns>

object ReplaceResource(IQueryable query, string fullTypeName);

/// <summary>

/// Sets the value of the given property on the target object

/// </summary>

/// <param name="targetResource">target object which defines the property</param>

/// <param name="propertyName">name of the property whose value needs to be updated</param>

/// <param name="propertyValue">value of the property</param>

void SetValue(object targetResource, string propertyName, object propertyValue);

/// <summary>

/// Gets the value of the given property on the target object

/// </summary>

/// <param name="targetResource">target object which defines the property</param>

/// <param name="propertyName">name of the property whose value needs to be updated</param>

/// <returns>the value of the property for the given target resource</returns>

object GetValue(object targetResource, string propertyName);

/// <summary>

/// Sets the value of the given reference property on the target object

/// </summary>

/// <param name="targetResource">target object which defines the property</param>

/// <param name="propertyName">name of the property whose value needs to be updated</param>

/// <param name="propertyValue">value of the property</param>

void SetReference(object targetResource, string propertyName, object propertyValue);

/// <summary>

/// Adds the given value to the collection

/// </summary>

/// <param name="targetResource">target object which defines the property</param>

/// <param name="propertyName">name of the property whose value needs to be updated</param>

/// <param name="resourceToBeAdded">value of the property which needs to be added</param>

void AddReferenceToCollection(object targetResource, string propertyName, object resourceToBeAdded);

/// <summary>

/// Removes the given value from the collection

/// </summary>

/// <param name="targetResource">target object which defines the property</param>

/// <param name="propertyName">name of the property whose value needs to be updated</param>

/// <param name="resourceToBeRemoved">value of the property which needs to be removed</param>

void RemoveReferenceFromCollection(object targetResource, string propertyName, object resourceToBeRemoved);

/// <summary>

/// Delete the given resource

/// </summary>

/// <param name="targetResource">resource that needs to be deleted</param>

void DeleteResource(object targetResource);

/// <summary>

/// Saves all the pending changes made till now

/// </summary>

void SaveChanges();

/// <summary>

/// Returns the actual instance of the resource represented by the given resource object

/// </summary>

/// <param name="resource">object representing the resource whose instance needs to be fetched</param>

/// <returns>Returns the actual instance of the resource represented by the given resource object</returns>

object ResolveResource(object resource);

}

Let’s go through each api one by one.

object CreateResource(string containerName, string fullTypeName) :– This is called when one tries to insert a new resource via the POST http method. The first parameter points to the container that the resource belongs to and the second parameter tells the namespace qualified name of the resource type that needs to be created. The second parameter might not be that useful when there is no inheritance, since from the container, one can easily figure out the type, but for inheritance cases, this is helpful. The return type, as said before, need not be the actual clr instance of the resource. It can be anything (a cookie) that only the IUpdatable implementor needs to understands.

object GetResource(IQueryable query, string fullTypeName) :- Get the given resource as resolved by the query and the namespace qualified name of the type that the query resolves to. In some cases, the full type name can be null. Look at the examples below to see the cases when the fullTypeName can be null. Again, the return type can be anything that represents the resource.

object ReplaceResource(IQueryable query, string fullTypeName) :- Very similar to the GetResource API, but this is used for replace-semantics and GetResource is used for merge-semantics. The implementation needs to return the resource referred by the query, but with default values for all non-key properties.

void SetValue(object targetResource, string propertyName, object propertyValue) :- Set the value of the property with the given name on the target resource to the given property value. The target resource is the opaque object returned by either CreateResource or GetResource. This method is called for scalar properties and complex properties only.

object GetValue(object targetResource, string propertyName) :- Gets the value of the property with the given name for the target resource. The target resource is the opaque object returned by either CreateResource or GetResource. This method is called for scalar properties or complex properties. If it’s a scalar property, we expect the returned object to be the actual value.

void SetReference(object targetResource, string propertyName, object propertyValue) :- This method is called for setting the value of navigation property with the given name on the target resource to the given propertyValue. The target resource and the propertyValue are opaque objects returned by GetResource or CreateResource API. This method is called for navigation properties that refer to a single resource – representing 0 or 0..1 side of a relationship.

void AddReferenceToCollection(object targetResource, string propertyName, object resourceToBeAdded):- This method is called for adding a resource to the collection navigation property with the given name on the target resource. Again, targetResource and resourceToBeAdded are opaque objects returned by GetResource or CreateResource API. The navigation property presents many side of a relationship. We generally call this operation as binding, which means you are binding the resourceToBeAdded to targetResource. In other words, you are setting up a relationship between the 2 resource objects.

void RemoveReferenceFromCollection(object targetResource, string propertyName, object resourceToBeRemoved):- Very similar to the above method, except it removes the given resource from the collection navigation property on the target object. We generally call this operation as unbinding, which means you are deleting the relationship between the two resource objects in question.

void DeleteResource(object targetResource):- This actually deletes the given resource. Again, the targetResource is the opaque object returned by GetResource or CreateResource API.

void SaveChanges():- This actually saves all the changes that has been made till now, using the above API’s. The IUpdatable implementation needs to track all changes until this API is called and then save all of them when this API is called. The IUpdatable implementation is expected to save all the changes or nothing

object ResolveResource(object resource):- This API is called whenever we want to resolve the opaque object returned by the CreateResource or GetResource API into the actual clr instance. This normally is called after SaveChanges, when we want to serialize out the resource (for POST methods). This method is also called if there are UpdateInterceptors, that needs to be invoked with the actual clr resource instances or the provider supports optimistic concurrency and the resource type has concurrency tokens (defined via etag properties in clr based provider).

This blog has already become bigger that I intended it to be. In my next blog, I will try and come up with some examples and the sequence of calls made on the IUpdatable interface for each of them. I have purposefully keep this post simple just so that people can get the basic idea of IUpdatable interface. There are few features like ETag, Update interceptors, etc due to which there might be additional calls on this interface. I will try and cover them in my coming blogs.

Pratik Patel

Developer, ADO.NET Data Services Framework

(Project Astoria)

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Batching Data Service Requests

dpblogs — Mon, 07 Apr 2008 08:24:30 GMT

We have received a fair amount of feedback regarding a number of use cases where it would be beneficial to enable a client of a data service to “batch” up a group of operations and send them to the data service in a single HTTP request. This reduces the number of roundtrips to the data service for apps that need to make numerous requests to the data service to perform a given action and allows a set of operations to be logically grouped together.

Below is the design we have landed on. Note: We will have something close to this design in our next CTP/Beta release of Astoria, but its not quite there yet.

Background

The base ADO.NET Data Services Framework semantics provide two mechanisms to query and send updates to a data service. From a high-level they are:

Query:

1) Send an HTTP GET request to a URI representing a resource (or set of resources) and receive in the response a representation of the resource (or set of resources). Example: a GET request to /Customers(1) returns a single customer entity in the response

2) Same as #1, but add the $expand query string operator to the request to request resources related to the resource(s) specified in the request URI be returned in the response as well. Example: a GET request to /Products(1)?expand=Category, Parts returns product #1 as well as the Parts and Category associated the product in the response

Update:

1) Insert / update / delete a single resource per HTTP request by sending POST, PUT or DELETE requests to a data service

2) Insert a new resource and related resources in a single request. Example: a POST to /Customers can insert a Customer and related Orders in a single request by inlining the related orders in the request body

Why do we need batching?

Now assume you have the following situation: Single “Save” button per page in my RIA line of business application: Contoso Solutions is building an online Silverlight-based order entry system for its salesforce. Any given sale requires a number of entities within the data service be inserted and/or updated. Some of the entities are associated via navigation properties while others have no relation to the other entities being acted on as part of processing the sale. The user experience contoso wants to enable is to paint the full order processing information on a single screen and include a “save changes” button at the bottom to persist all the updates made to create the order.

In this case, the $expand operation cannot be used to pull down all the data to paint the order entry screen in a single HTTP request. Also, the update operations cannot easily be persisted as an atomic set of operations to the underlying data store.

Batching Design

To support batching, ADO.NET Data Services has added a new $batch URI which will accept batch requests and return batch responses. Logically a batch request is a group of 0 or more QueryOperations and 0 or more ChangeSet operations. QueryOperations are analogous to a simple "non batch" query request. ChangeSet operations are just a group of unordered, atomic CUD (update,insert&delete) operations (ie. all operations succeed or none do).

Now that we have the logical model (Batch is a collection of ordered QueryOps and ChangeSets), we needed a wire representation. After a bit of exploring various ways to represent batches in ATOM, JSON , etc, Yaron Goland pointed out to us there is already a well defined way to represent multiple HTTP requests in a single request using multipart/mixed MIME messages and the mime type application/http. This turned out to be just what we needed and enables us to easily encapsulate binary and text based content in a request or response. Also, it looks like using multipart/mime for batching has been explored (with pretty positive feedback) a number of times in the blogosphere, so perhaps we'll all land on something generally applicable. Instead of describing this, lets just look at an example.

Example - Batch Request:

The example assumes the batch request is sent to a data service located at: http://foo.com/dataservice.svc

The Batch example contains the following operations (in order):

A Change Set which contains the following operations in order:
- POST operation
- PUT operation
A Query Operation
A Query Operation

Note:

Outer HTTP Request elements & batch boundaries are shown in blue
Query Operations are shown in green
Change Sets are shown in red

POST /dataservice.svc/$batch HTTP/1.1
Host: foo.com
Content-Type: multipart/mixed; boundary=batch(36522ad7-fc75-4b56-8c71-56071383e77b)

--batch(36522ad7-fc75-4b56-8c71-56071383e77b)
Content-Type: multipart/mixed; boundary=changeset(77162fcd-b8da-41ac-a9f8-9357efbbd621)
Content-Length: ###

--changeset(77162fcd-b8da-41ac-a9f8-9357efbbd621)
Content-Type: application/http
Content-Transfer-Encoding:binary

POST /dataservice.svc/Categories HTTP/1.1
Host: foo.com
Content-Type: application/atom+xml;type=entry
Content-Length: ###

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<entry xmlns:d="http://schemas.microsoft.com/ado/..."
       xmlns:m="http://schemas.microsoft.com/ado/.../metadata"
       xmlns="http://www.w3.org/2005/Atom">
…
<content type="application/xml">
    <d:CategoryName>Software</d:CategoryName>
    <d:Description d:null="true" />
    <d:Picture d:null="true" />
</content>
</entry>
--changeset(77162fcd-b8da-41ac-a9f8-9357efbbd621)
Content-Type: application/http
Content-Transfer-Encoding:binary

PUT /Categories(5) HTTP/1.1
Host: foo.com
Content-Type: application/atom+xml;type=entry
If-Match: xxxxx
Content-Length: ###

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<entry xmlns:d="http://schemas.microsoft.com/ado/..."
       xmlns:m="http://schemas.microsoft.com/ado/.../metadata"
       xmlns="http://www.w3.org/2005/Atom">
…
<content type="application/xml">
    <d:CategoryID>5</d:CategoryID>
    <d:CategoryName>UpdateCategoryName</d:CategoryName>
</content>
</entry>
--changeset(77162fcd-b8da-41ac-a9f8-9357efbbd621)--
--batch(36522ad7-fc75-4b56-8c71-56071383e77b)
Content-Type: application/http
Content-Transfer-Encoding:binary

GET /Categories(5) HTTP/1.1
Host: foo.com

--batch(36522ad7-fc75-4b56-8c71-56071383e77b)
Content-Type: application/http
Content-Transfer-Encoding:binary

Operation: GET /Categories(6)
Host: foo.com

--batch(36522ad7-fc75-4b56-8c71-56071383e77b)--

Batch Response

Now that we have seen what a request looks like, the response is pretty much the mirror image of the request (also uses multipart/mime), with a mime part containing the associated HTTP response for each operation in the batch request. The exception to this rule is for responses to ChangeSets. Since ChangeSets are atomic if an operation in the set fails, the response for the ChangeSet is a single HTTP response instead of a nested multipart/mixed collection of responses.

This is already getting a bit long, so I'll cut this off here. In a future post we'll walk through an end to end request + response and talk about how to cross reference operations within a batch request. What do you think so far? Are we overlooking/missing something?

Mike Flasko

ADO.NET Data Services, Program Manager

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Looking for feedback: query caching in data services

dpblogs — Mon, 31 Mar 2008 23:58:25 GMT

(sorry, tricky problem -> long write-up)

One of the few things pending in the server library of the ADO.NET Data Services Framework is query caching to help with performance. Here is a brief explanation of why we needed and a couple of design options. Feedback is welcome.

Query processing in Astoria

To give a little bit of context, let me first briefly go through the process that takes place when Astoria receives a URL and works its magic to turns it in query results ready for serialization in the HTTP response.

Data sources are hooked into Astoria services through LINQ expression trees. That means that the role of Astoria during query processing is to take a URL and translate it into an expression tree that then we can ask the data source to execute and give us results. The general flow is more or less like this:

URL -> Expression Tree -> [Data Source Execution and Materialization] -> Objects -> Serialization (Atom/JSON)

The thing in brackets is data source-specific. For example, if using the ADO.NET Entity Framework it would look like this:

URL -> Expression Tree -> Canonical Query Tree -> View expansion/Query simplification -> Canonical Query Tree -> SQL -> Rows (DataReader) -> Entities (DataReader) -> Objects -> Serialization (Atom/JSON)

This is not a cheap thing to do in every request, hence this discussion about query caching.

Why query caching

The processing pipeline that I showed above can be expensive. In particular, all that query translation between different tree types as well as all that analysis for view expansion and query simplification is an expensive, CPU-bound activity that we want to avoid as much as we can.

To help with this we are planning to introduce query caching, similar to what database systems do (e.g. the SQL Server “proc cache”). The idea of query caching is that for commonly requested URLs we’d bypass most of the processing required to setup the query for execution, and we’d go as directly as possible to the execution phase.

Since Astoria is a generic framework that works on many data sources, we have to enable this in a way that allows different data sources plug in their query caching capabilities if they have such thing.

Data source-independent query compilation

In order to implement caching, we need a way of doing our own work for translation, then tell the data source to do its own work on the expression trees, and then save that work to be used every time we have to respond to the “same” request (the definition of “same” is well…difficult, more about this later); that is, we need a way of compiling queries.

You can imagine an interface with two methods for this:

interface IQueryCompilationProvider

{

object CompileQuery(Expression<Func<T1, T2, …, Tn, TResult>> query);

IEnumerable ExecuteCompiledQuery(object compiledQuery);

}

The idea is that each data source can give whatever meaning it wants to “compiling” a query, and they can return us an opaque token (object) that we’d pass back along with parameter values in order to execute the query. Ideally you’d do as much work as possible. For example, there is a CompiledQuery class in LINQ to SQL and LINQ to Entities that can do the translation all the way to a SQL statement only once and then re-use it in all subsequent executions (re-binding parameter values).

Query caching design, part 1: for simple cases, a simple design does it

In the simpler cases where the data services does not have customizations the URL -> Expression Tree translation is 1:1. The only consideration in this case would be to parameterize the URLs so that we don’t fragment the query cache with URLs that are the same query with different constants (e.g. /Customers(1) vs /Customers(2), or /Customers?$filter=City eq ‘Seattle’ and /Customers?$filter=City eq ‘Las Vegas’).

In this case we can keep a simple map of parameterized URL to compiled query. If we don’t find a given URL, we go through the full translation process to produce an expression tree, and then –assuming the data source supports compilation- call CompileQuery to obtain a compiled query opaque token. On subsequent executions we look up the URL, find the compiled query token, bind the constants that we turned into parameters to parameter values and execute the query directly.

The rest is standard caching stuff…keep a map, have a limit and an eviction policy, make it “spike resistant”, etc.

Of course, life is rarely that simple…

Query interceptors

So far we’ve assumed that there is a 1:1 correspondence between (parameterized) URLs and expression trees. Astoria has a nice feature called “query interceptors” that causes that correspondence to break.

A query interceptor allows the service developer to introduce a custom filter predicate for each entity set that’s exposed through the service interface. For example, if you had a Customers table and a CustomerAccess table that indicates which user-ids can see which customers, you could implement entity-level security for customers by adding this interceptor:

[QueryInterceptor("Customers")]

Expression<Func<Customer, bool>> QueryCustomers()

{

// retrieve the user from the environment (e.g. the currently //logged-in user)

UserDescriptor u = GetUser();

if (u.IsAdmin)

return c => true; // can see all customers

else

return c => c.CustomerAccess.Any(ca => ca.UserID ==

u.UserID);

}

Interceptors are invoked whenever a URL involves the entity-set the interceptor is bound to, regardless of how. The system injects a Where operator with the predicate returned by the interceptor. This includes top level entity-set access (e.g. /Customers), access of subsets through link traversal (e.g. /SalesPeople(123)/AssignedCustomers), and inline expansion (e.g. /SalesPeople(123)?$expand=AssignedCustomers).

Note that now URLs and expression trees are no longer 1:1. There are two kinds of differences that might show up:

a) Same query but different parameters. For example, if user 1 and user 2, both not administrators, fetch the URL /Customers, then both requests will produce identical query trees, with the only difference that the value of “u.UserID” will be different. The difference with the parameterization of the URLs is that in this case we’re not the ones parsing the input.

b) Different query. In the example above, if one of the users accessing the URL /Customers is administrator and the other is not, the filter predicates used in each of them will be different.

Now the caching thing got complicated.

Side-note: I’ve excluded update interceptors in this discussion because they don’t participate in query composition. They are a key part of enforcing access control the way I described above though.

Query caching design, part 2

We really wanted to make query caching work without any API surface other than maybe a configuration knob for the size of the cache. That’s not looking great at this point, but we do have a few options on the table.

The essence of the problem is the fact that query interceptors need to be able to capture data from the execution environment at the time a given request is being processed. The main example of this is grabbing the user credentials for a given request (e.g. Thread.CurrentThread.CurrentPrincipal, or the user-id extracted from an encrypted cookie/custom HTTP header), but there can be other scenarios as well.

There are a couple of approaches that could do the trick, but then come with their trade-offs:

Option I: a bit of extra magic for a nicer API.

We could let users write interceptors just like I showed above. Those interceptors mix references to environment data and the description of the filter predicate in a single construct, a LINQ expression tree that has references to external variables in the reference closure. What the developer writes looks like this (copied from above):

[QueryInterceptor("Customers")]

Expression<Func<Customer, bool>> QueryCustomers()

{

// retrieve the user from the environment (e.g. the currently logged-in user)

UserDescriptor u = GetUser();

if (u.IsAdmin)

return c => true; // can see all customers

else

return c => c.CustomerAccess.Any(ca => ca.UserID ==

u.UserID);

}

To cache queries we would have to:

· On every request invoke all interceptors that are needed

· Extract all uncorrelated subexpressions from each interceptor filter predicate and turn them into constants (do “funcletization” in LINQ jargon), then replace the constants with generic parameters. Now we have a little tree for each interceptor

· Now use a parameterized URL + all the interceptor trees the caching key.

Option II: explicitness at the cost of API complexity

The other approach is to explicitly separate the definition of the filter predicate from the per-execution state that comes from the environment. The developer would write two pieces, statically-defined filter predicate and a method that sets-up per-request state, as follows:

[QueryInterceptor("Customers")]

static Expression<Func<Customer, CustomersDbContext, Dictionary<string, object>, bool>> QueryCustomers()

{

return (c, ctx, state) => (bool)state["IsAdmin"] ||

c.CustomerAccess.Any(ca => ca.UserID ==

(string)state["UserID"]);

}

void override OnStartRequest(RequestDescriptor descriptor, Dictionary<string, object> state)

{

// retrieve the user from the environment (e.g. the currently logged-in user)

UserDescriptor u = GetUser();

state[“IsAdmin”] = u.IsAdmin;

state[“UserID”] = u.UserID;

}

As you can see, the definition of the filter predicate gets a bit trickier (more parameters in the lambda expression in particular).

Of course, since at this point the interceptor is called only once and can’t use any request-bound context, we may as well just remove the idea of a method all together, and move the filter setup to the service initialization, where we already have APIs for configuring service policies. So the code would become:

static void InitializeService(IDataServiceConfiguration config)

{

// other policy initialization

// ...

// filters

config.SetEntitySetFilterPredicate<Customer>("Customers",

(c, ctx, state) => (bool)state["IsAdmin"] ||

c.CustomerAccess.Any(ca => ca.UserID ==

(string)state["UserID"]);

}

void override OnStartRequest(RequestDescriptor descriptor, Dictionary<string, object> state)

{

// retrieve the user from the environment (e.g. the currently logged-in user)

UserDescriptor u = GetUser();

state["IsAdmin"] = u.IsAdmin;

state["UserID"] = u.UserID;

}

The idea is that OnStartRequest (or whatever is a nice name for that) would explicitly capture the state any and all interceptors would use. Then interceptors become static constructs that are setup during initialization. An important detail is that now the predicate expression cannot refer to variables in the environment any more. Instead, for anything that’s request specific it needs to access the “state” object, which is then setup with data on a per-request basis.

This yields a very efficient system, because now we can do minimal work on repeated requests that result in the same query, even in the presence of interceptors. On the other hand, the code that developers have to write is pretty tricky…

Is it better to take a performance hit and leave it more usable? Is there a middle ground that we didn’t consider?

If you made it reading this far, you’re probably one of few J. In any case, feedback is very welcome.

Pablo Castro

Software Architect

Microsoft Corporation

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Related entries and feeds: links and link expansion

dpblogs — Tue, 19 Feb 2008 09:04:12 GMT

While going through application scenarios for the ADO.NET Data Services Framework (Project Astoria) one of the first things we noticed is that data-centric applications usually want to bring down graphs of related resources in each interaction with the server. For example, if you are retrieving a resource that represents an "Event", you may want to also bring in the set of related Contact resources that are invited or the "Venue" resource where the event will take place. This write up briefly describes how we model associations between resources as Atom links and proposes a usage pattern of the atom:link element to support retrieving resource graphs in a single response. We're looking for feedback on the approach and also to get folks thinking about inlined content and whether it should be considered an extension to Atom.

More context on Astoria support for Atom here:

http://blogs.msdn.com/astoriateam/archive/2008/02/13/atompub-support-in-the-ado-net-data-services-framework.aspx

1. Links for modeling associations between resources

Related resources can be seen at the instance level as "links" in Atom terms. Of course, from the data application development perspective, it's interesting to make this discoverable at the service description (schema) level. In Astoria data services the underlying model is the Entity Data Model (EDM), which describes data in terms of "Entities" (instances of Entity Types) and associations between entities. In the context of the Atom interface, Entities are mapped to entries and Associations to links. So by looking at the service description a developer can discover the links that will be present in an entry of a given type.

We model related entries or feeds using a link with a "rel" attribute of "related", and with a "type" of either "application/atom+xml;type=feed" or "application/atom+xml;type=entry" depending on the cardinality of the other end of the association.

One tricky aspect is that we need to indicate which association it is. At the model level we have a "navigation property" that identifies the starting "end" of the association (e.g. "Attendees", "Venue"). We currently put that name in the "title" attribute of the link. That solution is not perfect, as we try not to overload constructs that are for human-readable content. However, the alternative is to use a custom attribute, and we've been trying not to introduce custom attributes unless absolutely needed. Another option would be to use different "rel" values to specify the relationship, which feels natural but makes it much less likely that generic processors will be able to do something interesting with it.

Do these trade-offs sound reasonable? Is any of the other options more appropriate?

Continuing with the Events sample, this is what an entry (/Events(456)) with links looks like:

<id>http://localhost:81/EventsSample/Events(456)</id>

</author>

<d:EventID m:type="Int32">456</d:EventID>

<d:Name>Big Party</d:Name>

<d:NoteToAttendees>It's going to be a great party!</d:NoteToAttendees>

<d:DateAndTime m:type="DateTime">2008-03-05T06:00:00</d:DateAndTime>

</content>

</entry>

From the data modification perspective, links pointing to other resources in the service can be specified in the payload of POST and PUT operations, to establish links between the resource being manipulated and other existing resources.

2. Expanding links inline

As I summarized at the beginning of this note, we want to enable clients to request whole sub-graphs of data starting at some resource or set of resources. There are two aspects that need to be addressed: how does the client indicate that it wants one or more links expanded and how are the expanded links represented on the response.

How link expansion is requested is outside of the atom-syntax problem space, so I'll just briefly state what we currently do in case you have an opinion: data services support the query string option "$expand" to request link expansion. So you could say "/Events?$expand=Attendees " to retrieve all Events and all contacts that are attendees for each of them, or "/Events(456)?$expand=Attendees" to retrieve a single event (with key 456) and its attendees. Expand syntax allows for deep expands such as "Attendees/BestFriend" (expand Attendees, and on the expanded entry(es) expand BestFriend) and wide expands such as "Venue, Attendees/BestFriend" meaning expand two immediate links, and for the Attendees one further expand its BestFriend link.

For representing expanded links we put the expanded content inside the link element itself. According to section 4.2.7 of RFC 4287:

"The "atom:link" element defines a reference from an entry or feed to a Web resource. This specification assigns no meaning to the content (if any) of this element."

So it seems that adding content to the link element is not disallowed and at the same time it does not overlap with any existing semantics given to such construct. Based on that we thought it would be the perfect place for this information, as the link itself already contains the metadata about the link that we needed.

When a client indicates that the target of a link should be expanded, the server responds with the Atom representation of the resources pointed at by links wrapped in an <inline> element. For example, for "/Events(456)?$expand=Attendees,Venue" the response would be:

<id>http://localhost:81/EventsSample/Events(456)</id>

</author>

<m:inline>

<id>http://localhost:81/EventsSample/Venues(789)</id>

</author>

<d:VenueID m:type="Int32">789</d:VenueID>

<d:Name>The Cool Place</d:Name>

<d:Description>Great place for parties!</d:Description>

<d:Capacity m:type="Int32">1500</d:Capacity>

<d:Type>Nightclub</d:Type>

</content>

</entry>

</m:inline>

</link>

<m:inline>

<feed>

<title type="text">Attendees</title>

<id>http://localhost:81/EventsSample/Events(456)/Attendees</id>

<id>http://localhost:81/EventsSample/Contacts(123)</id>

</author>

<d:ContactID m:type="Int32">123</d:ContactID>

<d:FirstName>John123</d:FirstName>

<d:LastName>Doe123</d:LastName>

<d:EmailAddress>jd123@foo.com</d:EmailAddress>

<d:Phone>123-456-123</d:Phone>

<d:BirthDate m:type="Nullable`1[System.DateTime]">1990-04-01T00:00:00</d:BirthDate>

</content>

</entry>

<id>http://localhost:81/EventsSample/Contacts(124)</id>

</author>

<d:ContactID m:type="Int32">124</d:ContactID>

<d:FirstName>John124</d:FirstName>

<d:LastName>Doe124</d:LastName>

<d:EmailAddress>jd124@foo.com</d:EmailAddress>

<d:Phone>123-456-124</d:Phone>

<d:BirthDate m:type="Nullable`1[System.DateTime]">1990-05-01T00:00:00</d:BirthDate>

</content>

</entry>

</feed>

</m:inline>

</link>

<d:EventID m:type="Int32">456</d:EventID>

<d:Name>Big Party</d:Name>

<d:NoteToAttendees>It's going to be a great party!</d:NoteToAttendees>

<d:DateAndTime m:type="DateTime">2008-03-05T06:00:00</d:DateAndTime>

</content>

</entry>

I focused on the GET operations above. We think it would be better to stay away from attempting to support full modification operations on expanded graphs. In particular, we do not handle PUT on more than one entry at a time today. We do support POSTing an expanded graph, and we simply create all the nested entries and link them to the parent entry, creating the whole graph in a single operation.

Feedback in general about this approach would be greatly appreciated.

Pablo Castro
Technical Lead
Microsoft Corporation

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

AtomPub support in the ADO.NET Data Services Framework

dpblogs — Thu, 14 Feb 2008 10:07:48 GMT

We have been looking for the last few months at adding first-class support for AtomPub to Project Astoria (we briefly touched on it before here). We are at a point where we have some parts of the AtomPub story and their initial implementation running (and we'll share fresh experimental bits soon), some parts on the design board and other parts that we haven’t explored yet. I wanted to write a few words to explain why we think AtomPub support is important and enumerate the challenges we face. Guidance and general feedback on the reasoning, the approach and on the details is very much appreciated.

Why are we looking at AtomPub?

Astoria data services can work with different payload formats and to some level different user-level details of the protocol on top of HTTP. For example, we support a JSON payload format that should make the life of folks writing AJAX applications a bit easier. While we have a couple of these kind of ad-hoc formats, we wanted to support a pre-established format and protocol as our primary interface.

If you look at the underlying data model for Astoria, it boils down to two constructs: resources (addressable using URLs) and links between those resources. The resources are grouped into containers that are also addressable. The mapping to Atom entries, links and feeds is so straightforward that is hard to ignore. Of course, the devil is in the details and we'll get to that later on.

The interaction model in Astoria is just plain HTTP, using the usual methods for creating, updating, deleting and retrieving resources. Furthermore, we use other HTTP constructs such as "ETags" for concurrency checks, "location" to know where a POSTed resource lives, and so on. All of these also map naturally to AtomPub.

From our (Microsoft) perspective, you could imagine a world where our own consumer and infrastructure services in Windows Live could speak AtomPub with the same idioms as Astoria services, and thus could both have a standards-based interface and also use the same development tools and runtime components that work with any Astoria-based server. This would mean less clients/development tools for us to create and more opportunity for our partners in the libraries and tools ecosystem out there.

How are we approaching this?

We are simply mapping whatever we can to regular AtomPub elements. Sometimes that is trivial, sometimes we need to use extensions and sometimes we leave AtomPub alone and build an application-level feature on top. Here is an initial list of aspects we are dealing with in one way or the other. We’ll also post elaborations of each one of these to the appropriate Atom syntax|protocol mailing lists.

a) Mapping the data model: how do we map Astoria’s underlying data model, the Entity Data Model, to Atom constructs. This is quite straightforward but it deserves a look for completeness.

b) We use just the regular format/protocol whenever we can, we would be interested in validating our use with folks out there

c) Using AtomPub constructs and extensibility mechanisms to enable Astoria features:

· Inline expansion of links (“GET a given entry and all the entries related through this named link”, how we represent a request and the answer to such a request in Atom?).

· Properties for entries that are media link entries and thus cannot carry any more structured data in the <content> element

· HTTP methods acting on bindings between resources (links) in addition to resources themselves

· Optimistic concurrency over HTTP, use of ETags and in general guaranteeing consistency when required

· Request batching (e.g. how does a client send a set of PUT/POST/DELETE operations to the server in a single go?)

d) Astoria design patterns that are not AtomPub format/protocol concepts or extensions:

· Astoria gives semantics to URLs and has a specific syntax to construct them

· How metadata that describes the structure of a service end points is exposed. This goes from being to find out entry points (e.g. collections in service documents) to having a way of discovering the structure of entries that contain structured data

e) How do we deal with aspects that AtomPub does not handle by design or just because it has not been needed so far?

· What to do with fields that may not have a backing value in the input source (e.g. updated, author).

· Replace versus merge semantics during updates

f) High-level client libraries. How high-level can we make clients so they can consume AtomPub-based Astoria services but still feel that they are working against regular objects and have general integration with the development environment?

There are probably more, but I think this is a good starting list.

Where do we go from here?

The folks in the AtomPub community understand this the best, so we’ll take our questions to the atom-syntax and atom-protocols lists to hear opinions there. We’ll probably track posts and comments in the Astoria blog as well so people that follow it can keep track of what’s going on in this space.

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Pablo Castro

Technical Lead

Microsoft Corporation

Design Notes: URI Containment in Astoria

dpblogs — Sat, 22 Dec 2007 01:00:22 GMT

Problem statement

Today in Astoria if you have two entity types that are functionally in a containment relationship, you still see a top-level entity set for the instances of the contained type, and further the URLs for instances of the contained types that go through the parent type contain redundant information.

For example, assuming a schema where Order Lines (type “OrderLine”, entityset “OrderLines”) live within Orders(type “Order”, entityset “Orders”), two issues surface as shown below (points a & b). Additionally, assume that Orders have an “id” member that is the key, and that OrderLines has id and orderId (container id) members that form a composite key.

a) entities are always addressable through top-level entitysets, e.g. /OrderLines(1,2)

b) deep addressing through container has redundant information, e.g. /Orders(1)/OrderLines(1,2) (the container id, “1”, is repeated)

We have received feedback that this does not work for some scenarios in which people want to use Astoria. Continuing the example above where OrderLines are strictly contained within Orders, it does not feel right to have top-level access to it, even a composite key. On the other hand, containment where keys of the contained type repeat within different parents is something that the EDM currently does not model.

We need a solution that forces access to contained instances through its parents, does not require redundant specification of keys, and does not deviate from our data model.

Proposal

We define two new features to handle the general scenario of composition:

The first feature eliminates the need for redundant keys specifically for children that have a composite key where part of the key is the complete parent key. The model metadata needs to indicate that this is a parent-child relationship. For the case of the EDM metadata (entity framework), we’ll annotate the CSDL with attributes [1] and for the case of CLR object models we’ll use an attribute as in the following example:

class MyData

{

public IQueryable<Order> Orders;

[CanonicalAccessPath(Parent = “Orders”,

ParentNavigationProperty = “OrderLines”,

KeyMapping = { id, orderId},

]

public IQueryable<OrderLine> OrderLines;

}

Once an entity-set has been annotated like this, the keys of the parent are no longer required, so what was /Orders(1)/OrderLines(1,2) becomes /Orders(1)/OrderLines(2), where the key that goes in /OrderLines can repeat across different containers because it’s implicitly scoped to the other parts of the key that flow from the parent. URI construction rules to support this are:

· Child relationships can be chained to build multi-level containment arrangements.

· URIs with compound keys require the key to be specified using name/value pairs consisting of the property name of the key followed by its value

Example: /Orders(1)/Es(orderId=1,id=2)

· For non-compound keys (ex. /Customers(1) ) the key/value format is allowed, but is not the canonical form

Example: /Customer(1) is legal and canonical, /Customer(Key=1) is also legal, but not canonical

· The canonical form of containment URIs is the representation which does NOT include the parent key

Example: given /Orders(1)/OrderLines(orderId=1,id=2) & /Orders(1)/OrderLines(2) from above, the canonical URI is: /Orders(1)/OrderLines(2)

· While dropping the parent key results in the canonical URI, the “full form” (ie. /Orders(1)/OrderLines(orderId=1,id=2)) is allowed

From the Astoria runtime perspective, this means 2 things:

-the URL translator will know about this annotations in the sets, and whenever we’re translating a URL and hit an entity set with an access path marker we’ll simply flow the key values from the parent. This means that at the query expression tree level there will be no noticeable difference, which scopes the change to URL translation and metadata handling only.

-the metadata generation will include an annotation to indicate that a given entity set contains child resources.

The second feature is the ability to hide a top-level set and re-direct the canonical URL to an alternate path that goes through a parent. This is annotated with the CanonicalAccessPath attributes (or in CSDL via custom annotations[1]). Obviously you can have only one of these. Example (different from previous in green):

class MyData

{

public IQueryable<Order> Orders;

[CanonicalAccessPath(Parent = “Orders”,

ParentNavigationProperty = “OrderLines”,

KeyMapping = { id, orderId},

TopLevelAccess = false)]

public IQueryable<OrderLine> OrderLines;

}

The effect of applying this attribute is two-fold:

· the top-level entityset is no longer accessible (specified via TopLevelAccess=false), so something like /OrderLines(orderId=1,id=2) would result in a 404

· the presence of the CanonicalAccessPath attribute means the canonical URLs in the serializer are generated using the parent container, so the canonical form of /OrderLines(orderId=1,id=2) becomes /Orders(1)/OrderLines(2)

NOTE: An additional attribute named AccessPath (ie. no “Canonical” prefix) is defined which enables the specification of additional routes to a resource just as is done with the CanonicalAccessPath attribute; however, paths defined with the AccessPath attribute do NOT represent the canonical path to the resource.

Note that since the keys of the parent are always entirely contained within the child keys we never need additional queries to build the full nested URL.

A side-effect of this feature is that clients that are metadata-driven need to understand the special annotation to know that they need to generate URLs for an entity marked this way in a special way

Notes:

[1] Two annotations will be defined in CSDL which map to the information conveyed by the CanonicalAccessPath and AccessPath attributes noted in the document. The attributes will be:

· On an EntitySet element: TopLevelAccess=true|false

· One a NavigationProperty element: AccessPath=Canonical|NonCanonical

We look forward to hearing what you think of this approach...

Happy Holidays,

-Mike Flasko

Program Manager, ADO.NET Data Services ("Project Astoria")

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

ADO.NET Data Services Dec2007 CTP - Validation and Access Control

dpblogs — Tue, 11 Dec 2007 06:25:54 GMT

Note: This post refers to the CTP available here.

When building frameworks to expose and consume data on the web, access policy and data validation are always one of the first topics discussed during the design process, at conferences, etc. Given this, we thought the topic would be a good one to kick off a series of blog posts about the feature set of the recently announced Dec 07 CTP of the product. If you haven’t seen ADO.NET Data Services / “Project Astoria” before, I suggest checking out http://astoria.mslivelabs.com for an overview and “getting started” style walkthroughs of the product.

Authentication

Before getting into access control and validation, I need to put in a quick note about authentication to set the stage. Data services do not directly implement an authentication scheme. Instead, they rely on the authentication infrastructure of the technology (ex. WCF+ASP.NET, WCF, etc) hosting the data service. If you have an ASP.NET site that uses authentication (either one of the built-in authentication schemes or a custom one that sets the HTTP context principal appropriately), Astoria will simply leverage that mechanism to establish the current identity (principal) for a given request. Ok, now that we know how data services determines who is making a request, its’ time to look at how to implement access control and perform validation based on this information.

Service-wide Access Control

By default all entities, Service Operations and metadata describing any resources in a data service cannot be retrieved. Said another way, by default a data service is completely locked down with no read or write access. This is one of the significant differences from previous (prototype) releases of Astoria. In prior releases, a data service was fully open by default to enable easily creating and evaluating our ideas for what data services on the web might look like. Now that the project has moved out from incubation and into an official product release, we changed such that services are locked down out of the box to align with typical security requirements of production web infrastructures.

One of the first steps a data service developer needs to take is to open up access to the appropriate resources in the data service. One way to do this is to set service wide, access control policy by adding a data service initialization method. The code below shows what a typical data service created over the northwind sample database would look like if only the Customers, Orders and Products sets were exposed.

public class MyDataService : WebDataService<Northwind>

{

public static void InitializeService
(IWebDataServiceConfiguration configuration)
{

// URI ‘/Customers’ entities is enabled for all

//read and write operations

configuration.SetResourceContainerAccessRule

("Customers", ResourceContainerRights.All);

// URI ‘/Orders’ is disabled, but ‘/Orders(1)’

// is enabled for read only

configuration.SetResourceContainerAccessRule

("Orders", ResourceContainerRights.ReadSingle);

// Can insert and update, but not delete
// Products

configuration.SetResourceContainerAccessRule

("Products",

ResourceContainerRights.WriteInsert |

ResourceContainerRights.WriteUpdate);

}

The access policies in the snippet above are applied to all requests to the data service and are related to entity sets, not particular URIs. For example, the policy described by the call: ‘SetResourceContainerAccessRule("Customers",ResourceContainerRights.All)’ would apply to a request sent to ‘/Customers(‘ALFKI’)’ as well as ’/Orders(1)/Customers’. Basically, the rule is no matter how you address an entity or entity set, the rules in the init method apply equally. All other entities not explicitly enabled in the initialize service method will not be exposed by the data service. For example, the northwind db also includes Employee entities. Since the initialization method does not say anything about Employees, any request to retrieve an employee entity will result in an HTTP 404 (Not Found) response. In addition accessing the endpoint to retrieve the service’s metadata (ex. http://host/service.svc/$metadata) will return a document which only describes those resources where were explicitly made visible.

Per Request Access Control & Validation

Many data services will need to need to run validation logic when entities enter the data service (for inserts, updates or deletes) and/or restrict access to entities on a per request basis. For these scenarios Interceptors are used which enable a data service developer to plug in custom validation or access policy logic into the request/response processing pipeline of a data service. We decided to take the approach of providing infrastructure to build out custom per request access policy instead of defining an access policy model specific to ADO.NET Data Services because we recognize that each application, business, etc have different requirements in this space and that we could add value by providing comprehensive infrastructure elements for one to build custom access policy.

For example, assume you want to implement a policy which enabled customers to only retrieve their orders and not orders placed by other customers. The example below shows a query interceptor that implements this access policy. The important aspect to note about query interceptors is that they accept as a parameter an instance of IQueryable<T> which represents the query the system will push down to the underlying data store to retrieve the requested entity. As shown in the example, this approach enables access policy to be defined using query composition eliminating any added round trips to the data store to retrieve access control information. The example assumes the data service is hosted within a WCF+ASP.NET web application with authentication enabled as the ASP.Net HTTPContext object is used to retrieve the principle of the current request.

using System;

using System.Web;

using System.Collections.Generic;

using System.ServiceModel.Web;

using System.Linq;

using Microsoft.Data.Web;

using NorthwindModel;

namespace TestApplication

{

public class nw : WebDataService<NorthwindModel.NorthwindEntities>

{

public static void InitializeService(

IWebDataServiceConfiguration configuration)

{

configuration.SetResourceContainerAccessRule("Customers",

ResourceContainerRights.All);

configuration.SetResourceContainerAccessRule("Orders",

ResourceContainerRights.ReadSingle);

configuration.SetResourceContainerAccessRule("Products",

ResourceContainerRights.WriteInsert |

ResourceContainerRights.WriteUpdate);

}

[QueryInterceptor("Orders")]

public IQueryable<Orders> OnQueryOrders(IQueryable<Orders>

orderQuery)

{

return from o in orderQuery

where o.Customers.ContactName ==

HttpContext.Current.User.Identity.Name

select o;

}

To hook into update operations (insert, update & delete), change interceptors are used. The example below validates all the Product entities that are created are not discontinued. This would also be the place to implement per request access policy to restrict inserts, updates and/or deletes per request.

[ChangeInterceptor("Products")]

public void OnChangeCategories(Products p, ResourceActions action){

if(action == ReceiveEntityOperation.Insert) {

if(p.Discontinued) {

throw new WebDataServiceException(400,

"Cannot add discontinued products");

}

To summarize, services created using the ADO.NET Data Services framework are locked down by default and resources must be explicitly exposed. Once exposed, the service developer has a number of hooks at their disposal to implement per request access control and business rule validation.

What do you think of this approach? Is there something we should change/alter/add/etc to better meet your requirements? We look forward to hearing your feedback and continuing our open design process on this blog.

Mike Flasko

Program Manager

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Astoria data sources and system layering

dpblogs — Fri, 28 Sep 2007 04:02:12 GMT

The question is: what data sources should Astoria support as inputs? Should we allow ADO.NET Entity Framework only or open up more? If we open up more, how does the system need to be layered to enable the use of those alternate data sources?

Let me step back for a second first and establish why certain data sources are interesting.

Why does Astoria build on top of the EDM and the Entity Framework?

Putting an HTTP head on top of some set of data is easy enough that it’s tempting to do it against any data source out there. The problem is that then if the way data is modeled and surfaced has not been thought through, inconsistencies and awkward aspects start to show up in the HTTP interface.

The Entity Data Model (EDM) was a very nice candidate. EDM explicitly models all data as entities, which can be nicely mapped to resources. So the definition of a resource is crisp and it comes from the underlying data model. Furthermore, the EDM defines associations between those entities, which can be surfaced quite naturally as hyperlinks. Finally, semantics around query and update for EDM data are clearly defined and can be mapped easily to HTTP verbs.

Entity Framework also provides an important contribution: mapping. Very rarely you want to surface your database data as-is to the web, either to an application or as a service for external applications. You’ll want to adjust names, reshape data, merge tables, etc. The Entity Framework mapping layer can help with data, and with the VS integration we’ll include in the Entity Framework for EDM design and mapping, you’ll be able to do this in a nice visual experience.

Clearly, the world of data sources does not end with the Entity Framework…

Strictly from the data model perspective we clearly need to pick a single one for consistency, and EDM has the right characteristics for our needs. Now, from the perspective of the actual data sources that we can handle, it would not be reasonable to assume that every data source out there will be plugged into the Entity Framework. There are many scenarios where you want to surface data coming from other sources and you still want to use the Astoria URI and payload formats, along with the client libraries and tools that will be available for it.

So we want to support Entity Framework *and* other data sources. Now the trick is how to do this without basically writing Astoria twice :-), and maintain a sound, consistent data- and interaction-model.

What do we need from a data source?

Astoria has a few specific needs when interacting with a data source:

Surface data as reasonable units we can expose as resources. For example, we could say that each CLR object is a resource
Addressability: each resource (say CLR object) needs to be addressable. For us to create an address we need to be able to figure out what members are the “keys” on that resource
Query composition: Astoria URIs are a simple form of queries. We need to be able to formulate and execute queries against the data source without knowing the specifics of the target data source. We also need to be able to “compose” queries; e.g. to take a given query and say add sort order to it
Update: we need to be able to pull a bunch of resources (say, again, CLR objects), make some modifications, and then push the changes back into the data source.

Proposed approach

Trying to avoid inventing new stuff, we could try to tackle this problem using the technology that was introduced as part of LINQ. The IQueryable interface provides a mechanism by which you can build a query by applying operators to input IQueryable objects, obtaining a new IQueryable object that includes the applied operator (there is plenty of blog entries and such on IQueryable on the web, so I won’t elaborate on it here; Matt Warren’s series on IQueryable provide a great detailed reference for this.).

In this approach we can say that your data service takes as input any class that has a set of public properties of type IQueryable<T>. For example:

public class MyDataService : WebDataService<MyDataSource>

{

}

Where MyDataSource could be something like:

public class MyDataSource

{

public IQueryable<SomeType> SomeThings {…}

public IQueryable<OtherType> OtherThings {…}

}

public class SomeType {

public int ID { get; set; }

public string SomeData { get; set; }

public ICollection<OtherThings> RelatedThings { get; }

}

public class OtherType { ... }

That would result in a service with two top-level entity containers, /SomeThings and /OtherThings.

Translation to LINQ expression trees is for the most part fairly straightforward. Following the example, let’s say that you now request the URI /SomeThings!1, we would translate it to roughly:

myDataSourceInstance.SomeThings.Where(i => i.ID == 1)

(at least conceptually, we would build the expression trees directly and not go through the language form, of course).

A more elaborate example would be association traversal, for example /SomeThings!1/RelatedThings; the translation for that would be:

myDataSourceInstance.SomeThings.Where(i => i.ID == 1).SelectMany(i => i.RelatedThings)

So we are basically saying that the Astoria server runtime has 2 halves. The top-half is the Astoria runtime itself; this part is “fixed”, and it implements URI translation, the XML/JSON/etc wire formats, the interaction protocol, etc. It’s basically what it makes an Astoria service look like an Astoria service. The bottom half is the data-access layer and it’s pluggable. Communication between layers happens in terms of the IQueryable interface plus a set of conventions to map CLR graphs into the URI/payload patterns of Astoria.

Other options?

An alternate approach would be to come up with a new interface. Astoria needs only a few operations to be supported by data sources, and the IQueryable interface is certainly overkill for the job. However, the fact that IQueryable already exists and there will be various implementations for it has a lot of value itself…

Net result

The net result that we want, through IQueryable or some other alternative, is that we can surface various data sources through Astoria, so that clients can interact with data services across the way using a single, uniform mechanism. The exposed HTTP interface is still EDM-ish even when the source is not the Entity Framework, and the metadata is still expressed in EDM terms for consistency. That helps maintain the simple, straightforward semantics of the Astoria HTTP interface.

Now you can bring any data source you want and expose it through Astoria, provided that you have or can write an IQueryable implementation for it. Not only we’ll automatically hook it up in the Astoria pipeline, but also we’d push-down all the filters, sorting, and other operators, so the data source can efficiently implement them.

Querying over the Entity Framework

It turns out that we build first-class LINQ support into the Entity Framework, we call the thing LINQ to Entities. That means that while we have a different code-path for metadata, the query composition and execution code paths, as well as all of the serialization and other details, is common code across Entity Framework and any other LINQ implementation plugged into Astoria.

What’s missing

Metadata: one thing I did not discuss here is how we turn CLR object graphs obtained by executing IQueryable objects into something that works well with the Astoria HTTP interface. There is a set of conventions that we’ll use, and a few attributes to override conventions when they don’t work for you. In a future post we’ll discuss those in detail.

Update: while this model enables us to support querying over arbitrary .NET classes that expose containers of instances as IQueryable objects, it does not say how to update stuff. Discussing the update model will take a whole post (or several, most likely), but the short story is that we’ll define an interface, something like IUpdatable or whatever names works, that has the basic operations we need to perform in order to handle updates. The interface would have primitive operations for adding a new resource, remove an existing resource, applying modifications to resources and also handle linking/unlinking of resources.

Pablo Castro
Technical Lead
Microsoft Corporation
http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

URI Format - Part 1 - Addressing resources using URI path segments

dpblogs — Fri, 21 Sep 2007 19:25:00 GMT

Deciding on something that becomes a public interface of a developer-oriented technology is a tricky task. Not only does the resulting design need to be correct and complete, but also there are various aspects that are more around aesthetics and personal preference. The URI format used by Astoria will need to survive both sets of challenges…

The Astoria REST “protocol” is made up of a URI addressing scheme, HTTP-based interaction model and payload formats (Web3S/XML,JSON , ATOM/APP). In the interest of staying focused on the URI format, this write-up will only touch on the URI format used by Astoria and leave discussion of the interaction model to a future post. See this post for a discussion around payload formats used by Astoria.

In general, Astoria takes a conceptual model expressed in terms of entities in an EDM schema and surfaces data that follows that model over an HTTP interface, representing entities as resources and associations between entities as links. The URI interface needs to provide a rich yet simple way of addressing those resources.

The URI format in Astoria has a few specific goals:

a) Provide a mechanism to point to every resource or member of a resource in the system. That is, every piece of data is addressable, and the URI used to address it needs to be derivable from the service metadata which describes the conceptual model of the system

b) Allow for simple queries to be formulated. That is, instead of pointing to a particular resource, allow URIs that express filtered sets of resources satisfying certain criteria

c) Support manipulating the presentation of results. This includes things such as sorting resources, paging over them and expanding related resources.

This “part I” write up focuses on item a) above; pointing to resources and their members. We will discuss b) and c) in future blog posts

NOTE: the following descriptions use EDM terminology and constructs. Regardless of the underlying data access layer (Entity Framework, Custom LINQ provider, etc) an Astoria service is exposing, the service is described using an EDM schema, so this description applies equally to any data source. In addition, typical REST verbiage (as is done above) refers to items pointed to by URIs as resources. In the remainder of this write up the term ‘entity’ should be interpreted as a synonym for ‘resource’.

Starting from the root

At the root of the service we are thinking of carrying the behavior of the CTP forward and putting all of the resource sets, which are simply the list of entity sets we find in the EDM schema. These are addressed by name, separated by a forward-slash (“/”) from the service root URI. (e.g. …/northwind.svc/Customers, where “Customers” is a resource container).

A detail: In an EDM schema an entity set is contained within a single entity container and there may be multiple entity containers in the schema. If that’s the case, to access non-default containers, the names need to be container-qualified (e.g. “/NorthwindContainer.Customers”). The default one *cannot* be qualified, to avoid introducing a redundant way of getting to it.

Pointing to a particular entity

Every entity in an EDM schema has a key which consists of one or more of the properties in the entity. An entity key is unique within the containing entity set, so to identify an entity with a URI we need to include at least the entity-set and the key values.

--Location of keys

The key value could go before or after the question mark. That is, the URI could be built by adding a query parameter after the URI question mark as in:

…/northwind.svc/Customers?key=23

Or we could consider the entity-set-plus-value construct part of the URI namespace of the service and write it as

…/northwind.svc/Customers(23)

We prefer the second approach with the entity-sets and keys form a URI namespace and there is no query parameter required. One of the reasons for leaning towards the second approach is that it makes it explicit which entity set the key is associated with, especially when the URI path becomes quite long.

--a bit of syntax

Now, assuming we go with that approach, there is now the question of the syntax. The May 2007 CTP used values in square-brackets (e.g. “…/Customers[ALFKI]”). We got “generous” feedback saying that square-brackets were a bad choice.

The approach we are currently thinking to take is to attach the key directly after the entity set name and using the ‘!’ character as the separator (e.g. …/Customers!23 ). That said, as per our last “design” posting on formats, we are looking to support the Web3S format. Web3S has a more flexible data model in that it allows heterogeneous sets while an Astoria server supports homogenous sets. To enable interoperability between any servers implementing Web3S and an Astoria server a URI scheme flexible enough to address heterogeneous sets is required.   Therefore, we are thinking to expand on our current approach and allow a “full” form of URI and a “compressed” form, where the full form supports heterogeneous sets and the compressed form can be used as a shorthand notation when the set being addressed is on an Astoria server and is thus homogenous. For example:

“Full” form: …/Customers/Customer(123) would identify the instance 123 of type Customer within the Customers set.

“Compressed” form: …/Customers!123 would identify the same resource as above in Astoria because the ‘Customer’ type is implied since Astoria Servers support homogenous sets.

--composite keys

One option would be to encode name/value pairs, but that would result in verbose URIs and extra syntax to be invented. We are leaning toward a simple approach: we only use the values, separated by semicolon. The values are listed in the same order as they appear in the metadata document which describes the service. Metadata will be the topic of a future post, however, for now it’s enough to say in the typical case the description of a service will be available by making a GET request to …/$metadata. The following is an example of a URI which contains a composite key:

…/Customers!’ALFKI’,2

Some folks love this, some hate it. The main concern from folks who hate it is readability: you cannot interpret the URI without the schema. Is that an issue? The alternate option of using name-value pairs is more explicit in this sense. We could have:

…/Customers!CustomerID='ALFKI',Key=2

The single-key case would still not require the name, and given that most cases will be single-key cases, compactness won’t suffer too much. An aspect of this that’s both good and bad is the fact that by using names you can specify the values in different order. That’s “handy”, but it means that these URIs are not useful for comparison as strings when trying to determine identity.

-- literal forms

Using just literal values in a composite key doesn’t really work, because now you cannot tell whether that’s a 2-element key or a 1 element key that happens to have a comma in it. So we need to use proper literal forms for the values. We will need that when we want to express query expressions such as filter predicates anyway, so we may as well be consistent and use a single literal form everywhere.

Literals:

·         Strings: a string surrounded by single-quotes, (e.g. 'ALFKI')

·         Numbers: just the number, using US style (dot separates decimal digits)

·         Dates: quoted as strings. Inside the string, use format described in RFC3339

·         Guids: use the form “dddddddd-dddd-dddd-dddd-dddddddddddd”

·         Binary: “0x” followed by two hex digits per byte (e.g. 0x1AB4)

So the examples above would actually be, for single- and composite-key respectively:

…/Customers('ALFKI')

…/Customers('ALFKI',2)

Probably is a good idea to not allow spaces, as it would help making sure that URIs that mean the same thing are easily comparable.

Addressing members

To address a member of an entity, simply append the member to a URI that points to the entity, separated by a forward-slash. For example, if Customers have a CompanyName property:

…/Customers!'ALFKI'/CompanyName

Note that addressing a member like that would return the member appropriately wrapped to conform to the negotiated MIME type. For example, if using XML you’d get the value wrapped in an XML element and annotated with the required namespaces and such.

If you want just the value with no wrappers, you can use the /$value “magic member”. For example:

…/Customers('ALFKI')/CompanyName/$value

For a string, this would just return the string (text/plain) by default (same applies to all types but binary, which would return application/octet-stream). The developer can customize this by annotating the schema and indicating which MIME type a given value should be treated as. That would allow for example a text field to store HTML or a binary field to store an image, and HTTP responses would include the proper MIME type for them.

Association traversal

A special form of member access is when the member being accessed is actually a navigation property (a link in non EDM terms). Such a property can be considered a hard-link that resolves into the related entity (for associations that have a cardinality of 1 on the other end) or a set of entities (if the other end is “many”).

The syntax is the same as in regular members, independent of the cardinality of the other end. So, if a customer has sales orders, the URI to access the orders would be:

…/Customers!'ALFKI'/Orders

Keep drilling down

When the result of a given URI is a single object, the members of those objects can be accessed by adding the member name to the URI. For example, if a customer has a “Contact” navigation property that points to a single Person object, which has a Name property, the name can be retrieved directly by using this URI:

…/Customers!'ALFKI'/Contact/Name

For the case where traversing an association yields a set, you need to further scope the set by providing a key to point to a single element of the set in order to traverse further using the URI path. For example if a customer has a set of orders and each order has an order date property, one valid URI to access an order date would be:

…/Customers!'ALFKI'/Orders!123/OrderDate

A note on escaping

Quite a bit of escaping beyond basic URI encoding is necessary for the whole scheme to work. Things like “=” and “?” need to be carefully handled to not confuse URI translators and agents. Details go beyond this write up, but specific thoughts are welcome. Although the trickiest one deserves to be brought up: should we escape “/” inside a quoted string? The same question, asked more deeply, is “should we assume that consumers of Astoria URIs understand their syntax?”.

In general, we are leaning towards requiring characters of special meaning in a URI path (ex. ‘/’) to be escaped even when such a character is within a quoted string. If the character is not escaped we will treat it as per its predefined meaning in path segments in RFC 3986. We believe this would provide a consistent method for developers to craft and interpret URIs.

Wrapping Up...

The ideas presented above represent our current thinking in the space. As always, feedback and comments are most welcome. We look forward to hearing your thoughts. In follow up posts we will discuss the query string section of the URI and dig into addressing service operations.

Mike Flasko

Program Manager

http://blogs.msdn.com/mflasko

Pablo Castro

Tech Lead

http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

ADO.NET Data Services Team Blog : Design Notes

Design Notes: Row Count

Alpha preview of Project Codename &quot;Astoria Offline&quot; coming soon

Astoria futures: offline-enabled data services

Making Feeds Friendly

Merge vs. Replace Semantics for Update Operations

Optimistic Concurrency & Data Services

IUpdatable & ADO.NET Data Services Framework

Batching Data Service Requests

Looking for feedback: query caching in data services

Related entries and feeds: links and link expansion

AtomPub support in the ADO.NET Data Services Framework

Design Notes: URI Containment in Astoria

ADO.NET Data Services Dec2007 CTP - Validation and Access Control

Astoria data sources and system layering

URI Format - Part 1 - Addressing resources using URI path segments

Alpha preview of Project Codename "Astoria Offline" coming soon