ADO.NET Data Services Team Blog : Project Codename "Astoria"

Simplifying our n-tier development platform: making 3 things 1 thing

dpblogs — Tue, 17 Nov 2009 09:39:00 GMT

As you’ve probably observed, we’ve been working hard over the past year or so to grow our application stacks to better support the types of applications (Silverlight, rich desktop, AJAX, etc) and services (SOAP, REST, etc) that are required to build modern, robust solutions. At present, a few of the technologies we have to help in building services & n-tier applications are: Windows Communication Foundation, .NET RIA Services and ADO.NET Data Services.

We’ve been very pleased to see each of these stacks be well received in the community and, given that positive feedback, we’ve been eagerly working on expanding each based on your comments. While today these stacks target different application scenarios and/or levels of abstraction, we see opportunities to align their foundations by building the concepts shared in each stack (authentication, conventions for business logic, logging, configuration, etc) on a single foundation. Additionally, we’ve heard your feedback that traversing our offerings in this space is, at times, too complicated.

So, with the goal of simplifying our platform by aligning common components, we’d like to announce a few changes we’ll be making to achieve our goals….

Windows Communication Foundation (WCF) == your “one stop shop” for building services and n-tier applications

Since .NET Fx 3.0, WCF has been the place to go to in the .NET Framework to rapidly build service-oriented applications that communicate across the web and the enterprise. As we’ve developed the product roadmaps for .NET RIA Services and ADO.NET Data Services we’ve found they complement the core WCF stack quite well as components/extensions for WCF or as new top-level layers of abstraction.

To formalize our direction in aligning these technologies, we’re making a few name changes in the .NET Framework 4 timeframe. ADO.NET Data Services will change its name slightly to be WCF Data Services and .NET RIA Services will be known as WCF RIA Services. We’ll be talking about our alignment of these technologies starting at this PDC, so if you are attending, keep an eye out for sessions and information at the event.

We think of these name changes as a key first step in simplifying our offerings in this space. Starting at this PDC and continuing over the coming Silverlight & .NET Framework releases cycles, you’ll see us further bring together these applications stacks such that you can leverage key parts of each in one WCF-based application. As we progress along this path we’ll be sure to post our thinking to get your feedback.

Q&A

We thought this might generate a few questions, so here’s a couple Q&As on the topic that should help clarify this announcement…

Q: Now that the names are aligned, when will alignment occur in the products?

A: We’ll start aligning the technologies in the .NET Framework 4 and Silverlight 4 timeframes and, guided by your feedback, continue through subsequent release cycles as appropriate.

Q: Doesn’t ADO.NET Data Services and .NET RIA Services already use WCF?

A: They do, but we believe we can further their alignment & integration to provide a more seamless developer experience.

Q: Does this mean you are changing the direction of Data Services?

A: The vision we have for Data Services does not change with this announcement. We believe this announcement further solidifies our investment in the area of simple, standards-based communication on the web by (overtime) bringing support for the Data Services conventions directly into WCF. For further details, see this post on our future direction regarding data services and OData support: http://blogs.msdn.com/astoriateam/archive/2009/11/17/breaking-down-data-silos-the-open-data-protocol-odata.aspx

Q: How does this announcement affect the planned Data Services update for .NET Fx 3.5 SP1?

A: It doesn’t. The Data Services update for .NET Fx 3.5 SP1 will ship as planned this calendar year.

-Mike Flasko

Lead Program Manager, ~~ADO.NET~~ WCF Data Services

Astoria futures: offline-enabled data services

dpblogs — Thu, 23 Oct 2008 04:32:00 GMT

We mentioned that we were doing some early thinking of “Astoria Offline” back in Mix 2008, where we even demo’ed an early proof of concept. Now we’ve been working on various design aspects of Data Services for its future versions, and synchronization/offline support is one of them. It’s still an experimental thing with no official home or release vehicle, so this is the best time to follow the design process if you find the scenario interesting, as this is when it’s easiest to influence the direction we’ll go for.

A short way of describing this can be: “imagine you can point Visual Studio to a data service and say ‘take it offline’, and things just happen”.

Of course, the real world is more complicated than that :-)

Astoria Design Walkthrough: Thinking of a future with sync & offline

In this first note I’ll just touch on the scenarios we want to hit and go over a few guiding principles. In future posts I’ll elaborate more on the details.

We have many scenarios in mind for this infrastructure. The ones we’re thinking of tackling first:

· Outlook type of apps: I’m sure there is a fancier way of saying this, but anyone that has used Microsoft Outlook knows what I mean. The application is basically a 1-tier app that interacts with a local (embedded) database. In the background -and independent from UI activity- 2-way synchronization with a data service (e.g. a Microsoft Exchange server) happens. Often sync’ing against a database is not quite what you want…”Astoria Offline” will let you sync against your data-service layer, where the usual business logic/validation/etc will run just like in the online path.

· The description above sort of implies that server and client are built in collaboration, perhaps as part of the same development team. That’s certainly an scenario and we can do some things easier when that’s the case. But the other scenario we want to tackle is when client and server in a synchronization relationship are independent from each other (e.g. sync a service that’s just available for sync on the web).

· Local replicas of cloud-stored data: as more online services offer structured storage capabilities, and more of them use the Data Services REST interface, it becomes more interesting to be able to synchronize that data locally either for latency reduction, offline operation or other reasons.

· Data consolidation: if you have multiple data services that expose data from a variety of sources (some databases, some online/”cloud” stores, some custom repositories), you may want to synchronize a slice of data of each store to a local database, and then work with the data locally.

A couple of guiding principles:

· We will stick to a simple and open interface. What that means is that while we will definitely build a nice end-to-end integrated story for Visual Studio, it will be on top of a well-documented underlying data exchange using just HTTP and known formats. Anybody with an HTTP client and enough knowledge of our sync strategy should be able to synchronize with a data service.

· Data independence will remain there for sync as it is already for online access. Today when you access a data service the interface is the same regardless of whether the service is backed by a database, a cloud store, some custom application or whatever. With sync, the same applies. If the data service is sync-enabled you can sync with it, no matter what backs it.

· We are targeting data services for structured stores and business applications. That implies certain level of sophistication in the shape of data, such as assuming cross-item dependencies, store-level and application-level constraints that dictate consistent states of data, the need for making partial progress during synchronization, etc. Such support does come with some extra complexity, but we think it’s the right target.

We’re just taking on this space, so any feedback you may have is good. Are our initial scenarios interesting? Do you need this thing at all? Does the initial direction we’re looking at sound reasonable?

btw – if you are going to be at PDC, we have a full talk on this at the event.

I hope this “short video” format that Andy wants to do for our design notes adds a good little twist and makes them more interesting.

Pablo Castro
Software Architect
Microsoft Corporation
http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

URI Format - Part 1 - Addressing resources using URI path segments

dpblogs — Fri, 21 Sep 2007 19:25:00 GMT

Deciding on something that becomes a public interface of a developer-oriented technology is a tricky task. Not only does the resulting design need to be correct and complete, but also there are various aspects that are more around aesthetics and personal preference. The URI format used by Astoria will need to survive both sets of challenges…

The Astoria REST “protocol” is made up of a URI addressing scheme, HTTP-based interaction model and payload formats (Web3S/XML,JSON , ATOM/APP). In the interest of staying focused on the URI format, this write-up will only touch on the URI format used by Astoria and leave discussion of the interaction model to a future post. See this post for a discussion around payload formats used by Astoria.

In general, Astoria takes a conceptual model expressed in terms of entities in an EDM schema and surfaces data that follows that model over an HTTP interface, representing entities as resources and associations between entities as links. The URI interface needs to provide a rich yet simple way of addressing those resources.

The URI format in Astoria has a few specific goals:

a) Provide a mechanism to point to every resource or member of a resource in the system. That is, every piece of data is addressable, and the URI used to address it needs to be derivable from the service metadata which describes the conceptual model of the system

b) Allow for simple queries to be formulated. That is, instead of pointing to a particular resource, allow URIs that express filtered sets of resources satisfying certain criteria

c) Support manipulating the presentation of results. This includes things such as sorting resources, paging over them and expanding related resources.

This “part I” write up focuses on item a) above; pointing to resources and their members. We will discuss b) and c) in future blog posts

NOTE: the following descriptions use EDM terminology and constructs. Regardless of the underlying data access layer (Entity Framework, Custom LINQ provider, etc) an Astoria service is exposing, the service is described using an EDM schema, so this description applies equally to any data source. In addition, typical REST verbiage (as is done above) refers to items pointed to by URIs as resources. In the remainder of this write up the term ‘entity’ should be interpreted as a synonym for ‘resource’.

Starting from the root

At the root of the service we are thinking of carrying the behavior of the CTP forward and putting all of the resource sets, which are simply the list of entity sets we find in the EDM schema. These are addressed by name, separated by a forward-slash (“/”) from the service root URI. (e.g. …/northwind.svc/Customers, where “Customers” is a resource container).

A detail: In an EDM schema an entity set is contained within a single entity container and there may be multiple entity containers in the schema. If that’s the case, to access non-default containers, the names need to be container-qualified (e.g. “/NorthwindContainer.Customers”). The default one *cannot* be qualified, to avoid introducing a redundant way of getting to it.

Pointing to a particular entity

Every entity in an EDM schema has a key which consists of one or more of the properties in the entity. An entity key is unique within the containing entity set, so to identify an entity with a URI we need to include at least the entity-set and the key values.

--Location of keys

The key value could go before or after the question mark. That is, the URI could be built by adding a query parameter after the URI question mark as in:

…/northwind.svc/Customers?key=23

Or we could consider the entity-set-plus-value construct part of the URI namespace of the service and write it as

…/northwind.svc/Customers(23)

We prefer the second approach with the entity-sets and keys form a URI namespace and there is no query parameter required. One of the reasons for leaning towards the second approach is that it makes it explicit which entity set the key is associated with, especially when the URI path becomes quite long.

--a bit of syntax

Now, assuming we go with that approach, there is now the question of the syntax. The May 2007 CTP used values in square-brackets (e.g. “…/Customers[ALFKI]”). We got “generous” feedback saying that square-brackets were a bad choice.

The approach we are currently thinking to take is to attach the key directly after the entity set name and using the ‘!’ character as the separator (e.g. …/Customers!23 ). That said, as per our last “design” posting on formats, we are looking to support the Web3S format. Web3S has a more flexible data model in that it allows heterogeneous sets while an Astoria server supports homogenous sets. To enable interoperability between any servers implementing Web3S and an Astoria server a URI scheme flexible enough to address heterogeneous sets is required.   Therefore, we are thinking to expand on our current approach and allow a “full” form of URI and a “compressed” form, where the full form supports heterogeneous sets and the compressed form can be used as a shorthand notation when the set being addressed is on an Astoria server and is thus homogenous. For example:

“Full” form: …/Customers/Customer(123) would identify the instance 123 of type Customer within the Customers set.

“Compressed” form: …/Customers!123 would identify the same resource as above in Astoria because the ‘Customer’ type is implied since Astoria Servers support homogenous sets.

--composite keys

One option would be to encode name/value pairs, but that would result in verbose URIs and extra syntax to be invented. We are leaning toward a simple approach: we only use the values, separated by semicolon. The values are listed in the same order as they appear in the metadata document which describes the service. Metadata will be the topic of a future post, however, for now it’s enough to say in the typical case the description of a service will be available by making a GET request to …/$metadata. The following is an example of a URI which contains a composite key:

…/Customers!’ALFKI’,2

Some folks love this, some hate it. The main concern from folks who hate it is readability: you cannot interpret the URI without the schema. Is that an issue? The alternate option of using name-value pairs is more explicit in this sense. We could have:

…/Customers!CustomerID='ALFKI',Key=2

The single-key case would still not require the name, and given that most cases will be single-key cases, compactness won’t suffer too much. An aspect of this that’s both good and bad is the fact that by using names you can specify the values in different order. That’s “handy”, but it means that these URIs are not useful for comparison as strings when trying to determine identity.

-- literal forms

Using just literal values in a composite key doesn’t really work, because now you cannot tell whether that’s a 2-element key or a 1 element key that happens to have a comma in it. So we need to use proper literal forms for the values. We will need that when we want to express query expressions such as filter predicates anyway, so we may as well be consistent and use a single literal form everywhere.

Literals:

·         Strings: a string surrounded by single-quotes, (e.g. 'ALFKI')

·         Numbers: just the number, using US style (dot separates decimal digits)

·         Dates: quoted as strings. Inside the string, use format described in RFC3339

·         Guids: use the form “dddddddd-dddd-dddd-dddd-dddddddddddd”

·         Binary: “0x” followed by two hex digits per byte (e.g. 0x1AB4)

So the examples above would actually be, for single- and composite-key respectively:

…/Customers('ALFKI')

…/Customers('ALFKI',2)

Probably is a good idea to not allow spaces, as it would help making sure that URIs that mean the same thing are easily comparable.

Addressing members

To address a member of an entity, simply append the member to a URI that points to the entity, separated by a forward-slash. For example, if Customers have a CompanyName property:

…/Customers!'ALFKI'/CompanyName

Note that addressing a member like that would return the member appropriately wrapped to conform to the negotiated MIME type. For example, if using XML you’d get the value wrapped in an XML element and annotated with the required namespaces and such.

If you want just the value with no wrappers, you can use the /$value “magic member”. For example:

…/Customers('ALFKI')/CompanyName/$value

For a string, this would just return the string (text/plain) by default (same applies to all types but binary, which would return application/octet-stream). The developer can customize this by annotating the schema and indicating which MIME type a given value should be treated as. That would allow for example a text field to store HTML or a binary field to store an image, and HTTP responses would include the proper MIME type for them.

Association traversal

A special form of member access is when the member being accessed is actually a navigation property (a link in non EDM terms). Such a property can be considered a hard-link that resolves into the related entity (for associations that have a cardinality of 1 on the other end) or a set of entities (if the other end is “many”).

The syntax is the same as in regular members, independent of the cardinality of the other end. So, if a customer has sales orders, the URI to access the orders would be:

…/Customers!'ALFKI'/Orders

Keep drilling down

When the result of a given URI is a single object, the members of those objects can be accessed by adding the member name to the URI. For example, if a customer has a “Contact” navigation property that points to a single Person object, which has a Name property, the name can be retrieved directly by using this URI:

…/Customers!'ALFKI'/Contact/Name

For the case where traversing an association yields a set, you need to further scope the set by providing a key to point to a single element of the set in order to traverse further using the URI path. For example if a customer has a set of orders and each order has an order date property, one valid URI to access an order date would be:

…/Customers!'ALFKI'/Orders!123/OrderDate

A note on escaping

Quite a bit of escaping beyond basic URI encoding is necessary for the whole scheme to work. Things like “=” and “?” need to be carefully handled to not confuse URI translators and agents. Details go beyond this write up, but specific thoughts are welcome. Although the trickiest one deserves to be brought up: should we escape “/” inside a quoted string? The same question, asked more deeply, is “should we assume that consumers of Astoria URIs understand their syntax?”.

In general, we are leaning towards requiring characters of special meaning in a URI path (ex. ‘/’) to be escaped even when such a character is within a quoted string. If the character is not escaped we will treat it as per its predefined meaning in path segments in RFC 3986. We believe this would provide a consistent method for developers to craft and interpret URIs.

Wrapping Up...

The ideas presented above represent our current thinking in the space. As always, feedback and comments are most welcome. We look forward to hearing your thoughts. In follow up posts we will discuss the query string section of the URI and dig into addressing service operations.

Mike Flasko

Program Manager

http://blogs.msdn.com/mflasko

Pablo Castro

Tech Lead

http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Astoria Design: payload formats

dpblogs — Mon, 10 Sep 2007 07:49:00 GMT

The goal of Astoria is to make data available to loosely coupled systems for querying and manipulation. In order to do that we need to use protocols that define the interaction model between the producer and the consumer of that data, and of course we have to serialize the data in some form that all the involved parties understand. So protocols and formats are an important topic in our design process.

With that in mind, we’ve made sure that Astoria can handle a number of options from the system architecture perspective. Of course, each additional option has the potential of introducing added complexity to the system, and also adds to the overall development and testing time of the system, so we have to pick a small set to start with for the first release.

This entry focuses on formats. In general, introducing support for a format in Astoria means going through the exercise of mapping the data model used in Astoria to the different constructs of the target format. Astoria is largely an EDM-based system from the data model perspective, so one could see the problem as creating serialization forms for EDM-schematized data.

Key questions:

Is this the right list of formats? Should we do less, more, or a different set?
For each format, does the mapping to the underlying Astoria and EDM construct look reasonable?
Format experts in each area: feel free to pick on details and add random thoughts…none of the aspects discussed here are set on stone

NOTE: we’ve discussed these formats to a ridiculous level of detail during our design meetings. Here I’m including the most relevant aspects and those that will show up immediately when looking at a request or response. Once we have cleaner specs we can post all the details if there is demand for them.

Multiple formats, one protocol (almost)

For the most part there is a single “protocol”, and by that I mean the set of HTTP headers for requests and responses, as well as the overall interaction model. In certain cases in order to make a format really look natural to clients we do need to introduce a format-specific protocol element, but we try to keep those to a minimum.

Also, any added protocol elements on top of HTTP need to be done so that unsophisticated agents can ignore a lot of that, do “plain HTTP” and still get by for the most part.

A full discussion on protocol details needs its own write-up, and it’ll come later on. The short story is that we leverage as much as HTTP as we can; HTTP verbs are used for the basic operations, and negotiation such as content-type and charsets are done in the standard way.

Now, with the almost-single protocol in place, the question comes to which formats should we do. Right now we’re thinking ATOM/APP, Web3S, and JSON. We need to define the basic requirements for any format used in Astoria, and then map those to each format we want to support. That’s what comes next.

Astoria requirements for formats

For each format we want to support in Astoria we need to introduce representations for the following constructs:

A “top level” resource, which is returned when a client does a GET on the data service URI. We want to present a fully connected (hyperlinked) set of resources, so this resource is typically some sort of list of links to top-level resource containers.
A representation for an “Entity”, which is what resources are in Astoria. This is typically a property bag.
A representation for properties. Each property has a name and a value; the value can be a scalar, a structure (complex type in EDM parlance) or a link to a related entity or set of entities (an association in EDM).
A representation for a “set of entities” and a “bag of entities”. This will typically be the same, but it’s worth noting that sometimes we return sets and sometimes we return bags. Each top-level resource container returns one of these; other sources of sets/bags are URI queries and the result of traversing an association when the other end is a set of entities.

These formats are used, in somewhat different contexts, for all Astoria operations, including retrieval, update and query over resources. Formats are as symmetric as we can make them for retrieval and update (you get from, and send to the server the same thing, except if very rare cases).

In general Astoria services don’t embed a lot of typing information in the payloads. Some formats include some “light” typing (e.g. JSON), but if a client wants strict typing then the metadata resource should be accessed and the complete typing information extracted from it (more about metadata resources in a future post).

ATOM/APP

This format follows the specification described in APP and ATOM (and it goes beyond formats to include protocol-related aspects). I resisted APP at first. My question was “why do you want to use a feed format to represent data that is a temporal feed”. After pretty much everybody inside/outside Microsoft and within the Astoria team pushed for it, I realized that I was missing an obvious aspect: the fact that it’s a feed format is completely irrelevant. The important characteristic about ATOM/APP is that everybody speaks it, and it’s increasingly becoming kind of a lingua franca of the web when it comes to data exchange between unrelated agents.

It turns out that many of the ATOM and APP concepts map pretty well to the way Astoria exposes data.

Top-level resource: the “service document” defined by APP makes a great root resource for Astoria services, as it provides a clean, compliant way to list the links to each of the top-level resource containers in a given data service, presenting them as APP “collections”.
Entities: Entities or resources are serialized as “entry” elements in an ATOM feed.
Properties:

Each scalar/structure property is turned into a custom property in the item, carrying a custom XML namespace prefix to separate them from standard ATOM elements
Optionally we can map some of the properties to standard elements (e.g. title, author). I’m curious to know what folks think about this…should we do this at all? Should we do this by convention (e.g. by name) or by annotations in metadata?
Finally, ATOM defines the concept of links in entries, which not only have a target URI but also a “rel” (relation) attribute that carries added semantics for the link. This maps nicely to the EDM model that underlies Astoria, where entities are linked together by named associations. Associations are turned into links, and the “rel” attribute says which association it is.

Sets/bags of entities are mapped to APP “containers” which are represented as ATOM “feed” documents. This isn’t completely natural (they are not really feeds), but it works ok in practice.

From the perspective of manipulating data (i.e. adding, removing, modifying entries), APP follows HTTP very closely, which makes our life easy from the Astoria perspective, as the system is built from the ground up to map to HTTP operational semantics.

There are some caveats with mapping data to an ATOM feed, and to APP more in general to cover for updates, service documents and collections. Most of them have to do with certain required elements in ATOM that do not have a direct mapping in Astoria or EDM in general. For example “title” and “author” elements of “feed” are tricky to map. ATOM was the last protocol in our design list so it’s the least baked right now. Once we have all the rules for it we’ll write an ATOM/APP specific post to discuss the details.

The production code does not yet produce APP, so I can’t include an example right now. Once this is complete we’ll iterate again on the topic of APP and include samples.

JSON

The JSON (Javascript Object Notation) format has become popular lately mostly (my guess) because it makes it easy to consume results from services in Javascript environments, particularly inside web browsers. JSON is a simple format, with no prescription and no widely adopted standards on top of the base format. On one hand that means a fairly clean serialization form, on the other it means that we have to do a few tricks in order to accommodate our requirements.

In addition to be compliant, we tried to design the JSON format so that it’s friendly to their main consumers: javascript clients. What that means in practice is that whenever possible we structured the format so that the result of doing an “eval” of a given service response results in an object graph that is fairly natural and can be used directly from Javascript code without seeing any artifacts in it.

The JSON view of Astoria resources is fairly simple:

Top-level resource: we return an object with a member for each top-level resource container and their links as values.
Entities: each entity/resource is mapped to a JSON object (associative array)

In addition to the actual data, we usually want to include some instance-level metadata with each entity. At least we want the URI for the entity and its actual type. Since there are no namespaces in JSON to separate this from actual data, we use a property called “__metadata” that has these values in it.

Properties: we map them to JSON object properties.

If the property is a scalar, we use a JSON scalar. This is no easy task sometimes because of the small set of types and literal forms supported in JSON. For example, there is no Date/Time literal form, and there is no decimal (precise floating point) data type at all. For those that don’t map directly we generate their values as strings; the one exception is date/time, where we use the Microsoft ASP.NET AJAX (aka Atlas) literal form (see more about it here).
If the property is a structure (e.g. an expanded related object, or a value of a complex type) we simply nest the object underneath the outer object
If the property is a link we add an object with a single property called “__deferred”, indicating that there is deferred content for this property. The __deferred object has a single property called “uri” that indicates how to get the deferred data. We may generate links for associations and we’re also considering doing it for BLOBs so we don’t include them by default in the payload.

Sets/bags: we use JSON arrays for these. One caveat is that we cannot attach metadata to the array itself, sometimes resulting in redundant instance-level metadata in entities.

To show an example I’ll use the typical Northwind sample database, and pick a single “Customer” entity; this response is a set like the one that would be returned from a top-level container:

[

{

__metadata: {

uri: "Customers!\'ALFKI\'",

type: "NorthwindModel.Customer"

CustomerID: "ALFKI",

CompanyName: "Alfreds Futterkiste",

ContactName: "Maria Anders",

ContactTitle: "Sales Representative",

Address: "Obere Str. 57",

City: "Berlin",

Region: null,

PostalCode: "12209",

Country: "Germany",

Phone: "030-0074321",

Fax: "030-0076545",

Orders: {

__deferred: {

uri: "Customers!\'ALFKI\'/Orders"

}

CustomerDemographics: {

__deferred: {

uri: "Customers!\'ALFKI\'/CustomerDemographics"

}

]

Web3S

In the first CTP release of Astoria we had a “plain” XML format we informally called POX for plain-old XML. The goal of this format was to introduce something that did not added extra concepts on top of Astoria/EDM and just presented the data in XML for easy parsing.

It turns out that at the same time the folks at the Live organization in Microsoft were working on a protocol for data exchange that would be used uniformly across many different services. That protocol is called Web3S and its extensively documented here, and there is also a FAQ here.

The obvious question is: should we still roll our own custom XML format or try to unify with Web3S. Since in this kind of thing less is often more, we’ve been working hard trying to unify them. What that means in practice is that Astoria would speak Web3S, and we would make some tweaks to Web3S so that it work well with Astoria services and clients.

Making Astoria work with Web3S is a bit tricky…the main issue is that Web3S models data as a big tree, where as Astoria uses a record-oriented view of the world (entities and associations to be precise). The mapping goes more or less as follows:

Top-level resource: this one looked easy so we ended up postponing this a bit. We will list the set of top-level resource containers; what we haven’t decided yet is whether we’ll make it a Web3S element itself or not.
Entities: in Web3S there are “elements”, which are a very generic, tree like thingy. We map entities to elements, which actually works well because nobody says that we cannot only generate elements that are a single level deep (i.e. not trees). To be somewhat more specific, we map entities to elements in a multi-element container, so every element that represents an entity has an ID (which we form using the key(s) of the entity).
Properties:

Scalar properties become children of entity elements. They are elements themselves but have no children, only a scalar value.
Structures (EDM complex types) are represented as elements with nested elements.
Links (associations) are mapped to elements with “deferred content”. Since Web3S has a clean and simple URI composition rule, clients can easily come up with the URI of any element, including deferred ones.

Sets/bags: these are Web3S elements with multiple children element. In Web3S children of an element could be of any kind; Astoria does not use such generic construct and instead children of a given element are typically restricted to the ones that are of the type of the container or some subtype.

Here is an example of the typical Customer entity from the Northwind sample database in Web3S format; in this case it includes a wrapping Customers element, as in when the entity is returned as part of a set:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>

<web3s:ID>'ALFKI'</web3s:ID>

<web3s:Uri>Customers/Customer('ALFKI')</web3s:Uri>

<CustomerID>ALFKI</CustomerID>

<CompanyName>Alfreds Futterkiste</CompanyName>

<ContactName>Maria Anders</ContactName>

<ContactTitle>Sales Representative</ContactTitle>

<Address>Obere Str. 57</Address>

<City>Berlin</City>

<Country>Germany</Country>

<web3s:DeferredContent />

</Orders>

<web3s:DeferredContent />

</CustomerDemographics>

</Customer>

</Customers>

What happened with RDF?

The May 2007 CTP also included support for RDF. While we got positive comments about the fact we supported it, we didn’t see any early user actually using it and we haven’t seen a particular popular scenario where RDF was a must-have. So we are thinking that we may not include RDF as a format in the first release of Astoria, and focus on the other 3 formats (which are already a bunch from the development/testing perspective).

My personal take is that while I understand how RDF fits in the picture of the semantic web and related tools, the semantic web goes well beyond a particular format. The point is to have well-defined, derivable semantics from services. I believe that Astoria does this independently of the format being used. That, combined with the fact that we didn’t see a strong demand for it, put RDF lower in our priority lists for formats.

Retrieval, query and update semantics

This post did not elaborate a lot on how payloads are actually used in retrieval and update operations. This is already a long post as it is, so let’s call this “the formats initial write-up” and stop it here. Future posts will discuss these topics and also dig into specific details of each particular format.

Pablo Castro
Technical Lead
Microsoft Corporation
http://blogs.msdn.com/pablo

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Transparency in the design process

dpblogs — Fri, 20 Jul 2007 20:40:01 GMT

We have been looking at various options to ensure that the design of Astoria truly reflects the requirements of the day to day challenges that our developer community faces when building real-world web applications and services. We would like to start by being as transparent as possible in the design process. I wanted to briefly describe how we’ll go about it so you know exactly what you are looking at when reading one of our design-related posts.

How did we get here? We shipped the first Astoria CTP last May 2007. That CTP was the result of an incubation effort. During the incubation phase we typically focus on the vision, and the actual implementation details are secondary. We built something that was good enough to illustrate our thinking and get feedback from both internal and external sources. Now that we have gotten strong support from the development community we have decided to fund the Astoria project as a full-on project, not an incubation effort anymore.

Where are we now? The Astoria team is staffed and the engineering effort to build the real product has been kicked-off. What this means in practice is that we have a team with developers, testers, program managers, etc., that we have regular design meetings, and that we have a timeline (or more accurately, that we’re required to have a timeline but we’re still working on it :).

Transparency in the design? Over the years Microsoft has been opening up the engineering processes incrementally. Long ago there were only betas, and that was the only chance to see and give feedback about a product before it shipped. Then we started to do Community Tech Previews (CTPs). CTPs enabled us to provide bits more often and gather feedback frequently. The goal with increasing the transparency of design is to take this one step further: we would like to enable folks that are interested in Astoria to follow the design topics as we discuss them, and have the opportunity to provide feedback right during the time where we are actively discussing a certain aspect and haven’t made a decision yet.

What exactly would we make visible? In short, our design process. To be more concrete, I’m not talking about some fancy set of specifications. What I mean is that as we go through the detailed design of the various aspects of Astoria, we would post in this blog a) the meeting notes from our design meetings (the Astoria team has a design meeting twice a week, plus a lot of impromptu hallway chats), and b) deeper write-ups of specific design challenges where we’d like folks to understand how we’re seeing a problem and provide a channel for deep, detailed feedback.

How transparent is transparent? I want to be completely clear about the scope of the information we are sharing. One of the things we need to learn both from the Microsoft side and from the community side is whether the model works within a practical set of restrictions. We would post as much of our discussions as it is practically possible. However, we have to make sure we don’t compromise the interests of Microsoft as a company. There are certain things that can range from ideas to specific implementation details that we could consider trade secrets, high-value Microsoft intellectual property or something along those lines. It *will* happen that in some cases we will not discuss a topic publicly, either for a certain term (e.g. until a proper IP protection mechanism is in place) or until we ship or ever. This is nothing new, but I haven’t seen folks from large companies discuss this explicitly before, so I wanted to make sure it is clear here.

About your feedback: We would love to hear your thoughts, be it comments, suggestions, ideas or anything else. However, in the end we are designing a commercial Microsoft product. So we’ll happily take your feedback but you need to understand that by providing us feedback in any form you are agreeing that we may use it to develop our product, that others may use it in connection with the product and that you will not be compensated for any of these things. We may incorporate ideas or make changes based on comments you make, or we may make changes to the product that are indirectly influenced by discussions that we have with you and other folks in the community. Again, this is nothing new, but instead of having some fancy statement written in legal lingo I wanted to be upfront about this here in this first post on the topic. Of course our legal folks looked at this, and they were cool enough to understand that the informal nature of the process is what makes it work, and they let us get away with this statement in which I think we clearly delineate what will happen with whatever feedback you send our way.

So, what do folks think? Is this a good idea? Does it sound useful/practical? We will start posting design notes and challenges soon and tweak the process as we go.

Pablo Castro
Technical Lead
Microsoft Corporation

Welcome!

dpblogs — Wed, 18 Jul 2007 21:17:00 GMT

Welcome to the Project Astoria Team Blog! The Astoria team (part of the “Data Programmability” (DP) team within the SQL Server organization at Microsoft) is responsible for analyzing how current and next generation internet enabled applications use data on the web and to build solutions to facilitate such usage patterns. For those that have followed the Project Codename “Astoria” since its prototype stages, the project has moved out of the incubation phase and will become part of the suite of data access technologies created by the DP team. We look forward to sharing our thoughts and listening to your feedback via this blog as we progress through the project.

Project Astoria Overview

The goal of the Astoria project is to enable applications to expose data as a data service that can be consumed by web clients within corporate networks and across the internet. Such data services are reachable over regular HTTP requests using standard HTTP verbs such as GET, POST, PUT and DELETE to represent the operations against the service. The payload format for the data exchanged with the service can be controlled by the client and all options are simple, open formats such as plan XML and JSON. The use of web-friendly technologies make it ideal as a data back-end for AJAX-style applications, Rich Interactive Applications and other applications that need to operate against data that is across the web.

Further reading: Astoria Overview, Using Astoria

What is available now?

During the incubation phase of Project Astoria the team created an experimental Community Tech Preview (CTP) release to share our early thoughts of what we envision the Astoria product to look like. The CTP release has two components:

· we shipped bits you can download, install and run on your own systems

· we also made available an experimental online service that we hope will help us better understand the requirements and usage patterns of data services on the web

Additional Resources

Project Codename “Astoria” website: http://astoria.mslivelabs.com

Mix conference presentation announcing experimental CTP release of Astoria: http://sessions.visitmix.com/default.asp?event=1011&session=2011,2012&pid=XD006&disc=&id=1573&year=2007&search=XD006

Mike Flasko

Program Manager