Pablo Castro's blog : Web

Adding support for JSONP and URL-controlled format to ADO.NET Data Services

pabloc — Thu, 26 Feb 2009 06:29:33 GMT

JSONP is a common way of making data accessible in client-side mashups even when the requests need to be cross-domain.

While the current version of the ADO.NET Data Services framework does not support this, it’s possible to build it on top. There are a couple of ways of doing this. Here is what’s probably the simplest way. There is some downsides to this approach, but overall is the most straightforward path to get there.

The default transport layer for Data Services is WCF, which has a many extensibility points across the stack. For the case of JSONP support, IDispatchMessageInspector comes in handy.

There are two things needed to support JSONP properly:

The ability to control the response format. Data Services uses standard HTTP content type negotiation to select what representation of a given resource should be sent to the client (e.g. JSON, Atom). That requires that the caller can set the Accept request header, which is not possible when doing the JSONP trick (which basically just uses <script> tags). We need to add the ability to use the query string in the URL to select format. (e.g. /People(1)/Friends?$orderby=Name&$format=json).
A new option to wrap the response in a callback if such callback was provided in the request (also in the query string). For example /People(1)/Friends?$orderby=Name&$format=json&$callback=loaded.

What we’ll do is register a message inspector and adjust the request/response when we see these new options coming in.

In order to support the $format=json option we can intercept the message before it gets dispatched to the Astoria runtime, at the IDispatchMessageInspector.AfterReceivedRequest method. If we see the query string option then we’ll a) strip it out from the URL so Data Services does not generate an error and b) change the “Accept” header to “application/json”, so the rest of the system just thinks that the client asked for a JSON response in the first place.

For the second part, where we need to wrap the response into a Javascript call if the $callback option was used, we have the IDispatchMessageInspector.BeforeSendReply method which gives us the perfect spot to rewrite the response. One unfortunate side-effect of this is that the response will get buffered and re-encoded; that said, in many cases this won’t make any noticeable difference.

Finally, we need to register the interceptor with WCF’s dispatchers. For that we create an attribute that implements IServiceBehavior, so we get called during service initialization. When we get called we can register our message interceptor.

The net effect is that if you include this code in your project, you just need to add a single attribute to your Data Service to make it support JSONP:

[JSONPSupportBehavior]
public class SampleService : DataService<ContactsData>
{
// your service code here...
}

Once that's in place you can use JSONP by adding $format and $callback to URLs, for example:

http://<host>/SampleService.svc/People?$format=json&$callback=cb

Of course, you can still use all the other Data Services URL options in addition to these.

The implementation and a small sample service are available at MSDN code gallery, here:

http://code.msdn.microsoft.com/DataServicesJSONP

-pablo

Now you know...it's Windows Azure

pabloc — Wed, 29 Oct 2008 04:02:29 GMT

Since we shipped ADO.NET Data Services v1 in .NET 3.5 SP1 (and actually before that as well) I've been working on a few things that I could share (such as offline/sync support for data services) and some that I couldn't discuss publicly until all the big plans where announced.

This week at PDC Microsoft announced Windows Azure. A lot has been written about it, so I won't go into the details.

On our side, in the data services team, we made our small contribution to the big picture.

The Windows Azure table service is a structured storage facility that's part of the core part of Azure. Access to the table service is done through a data-services compatible RESTful interface that uses the Astoria conventions over an HTTP binding. That means that you use either any client with an HTTP stack to talk to it, or you can use the ADO.NET Data Services client, which does a nice job exposing data as .NET objects, letting you write simple queries using LINQ instead of URLs, etc.

Another cool thing about the table service (and the blobs and queuing service for that matter) is that they are accessible both from the virtual compute environment and from anywhere in the Internet. In both cases, if you're using .NET, you can use the data services client to interact with it. In the case of code running in the Windows Azure hosting environment, the client is already present (the environment includes .NET 3.5 SP1) so you can use it without worrying about taking new dependencies.

You find out more about the table service you can watch Brad's PDC session for a discussion of the service itself, and this other session than Niranjan and I did together (or "will do" if you're reading this before Wed in the PDC week) for a drill down on how to program the Windows Azure table service. If you're not at PDC no worries, these talks are accessible to online.

On the next layer up from the core, the Windows Azure service layer, SQL Data Services also is making big announcements in this PDC. We're introducing more relational capabilities into the system, and also experimenting with a data services-compatible interface. This PDC talk from Patrick will discuss and demo the new interface, and you can follow how this effort goes here.

-pablo

"Data Friction", spot-on

pabloc — Thu, 21 Feb 2008 03:22:00 GMT

Jon Udell wrote a brief piece on how data is locked on servers behind UIs that were not designed for data sharing. He views this as "data friction"...it's just the perfect way to describe the problem.

I couldn't agree more with Jon's take. I would even take it further: an operation-centric approach to interfaces is good for closed systems or systems where semantics are completely centered around behavior; however, the data behind those operations is still somewhat locked-in. Data-centric APIs that expose a uniform interface for clients to consume is how the class of scenarios that Jon talks about will come to life. We can't tell upfront how all those applications out there will use our data, it's just too hard to predict, so a "function centric" interface just won't do. Instead, those systems need put out the data plus a way to interact with it that enforces proper semantics while not getting in the way of how a consumer wants to explore that data.

We're hoping to help at least a bit in that space with Project Astoria...we'll see how it goes :)

-pablo

We'll host an experimental Astoria data service for you

pabloc — Sat, 04 Aug 2007 01:49:00 GMT

When we announced Project Astoria at Mix 2007 last May we made the toolkit available along with a few samples as an online service. The missing piece there was the ability to create your own service online in the experimental setting we had setup. Now Mike has just announced the availability of the new version of the experimental online service that allows you to describe the schema of the data you'd like for your data service, and we'll create and host it for you.

So go ahead, create your data service and let us know how it works for you. To access the online service you can use HTTP directly or the client library that was included in the May 2007 CTP.

-pablo

Astoria Client for Silverlight Alpha 1.1

pabloc — Thu, 12 Jul 2007 07:22:00 GMT

Last May when we shipped the first CTP of Microsoft Codename “Astoria” we included a client library in addition to the components for the server. The original client library can be used from .NET applications in cases where you do not want to develop directly against the HTTP interface, and rather you want to use a higher-level API that works in terms of .NET objects.

Back then, we were just releasing Microsoft Silverlight Alpha 1.1, which includes support for using the .NET Framework inside Silverlight applications. Unfortunately, timing just didn’t work out well enough for us to include a version of the Astoria client library that could run inside the Silverlight Alpha 1.1 environment, so the only way to access Astoria services from code running in Silverlight was to use HTTP directly.

Now we have fixed that :)

Today we made available an add-on for the Astoria May 2007 CTP that consists of a new client library for use in Silverlight applications. You can simply add a reference to Microsoft.Astoria.SilverlightClient.dll to your Silverlight application and use the API just like you would in a regular .NET application. The add-on is available for download here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=6d85055e-3549-48fc-8a2b-f678e6786e3a&displayLang=en

In addition to the existing API, we added one small extension. Since most Silverlight applications will run on the web where network latency tends to be high, we included asynchronous query execution support, so that you don’t block the UI thread when waiting for an Astoria server to respond.

Porting to Silverlight was painless for the most part. Mainly just another build script. We only needed a few additional #ifdef’s for one or two specific things that either weren’t present or behaved slightly different in the Silverlight libraries. It took a little longer than planned to release, since at the same time we have been heads down building Astoria to ship as soon as possible, not really because porting it to Silverlight was difficult. :)

If you are interested in Silverlight + Astoria go ahead, download the CTP add-on and give it a shot. If you build something cool with it, please let us know!

-pablo

Podcast interview with Jon Udell

pabloc — Tue, 03 Jul 2007 21:20:00 GMT

Last week I had a great conversation with Jon Udell about Astoria. Jon is a great interviewer and he knows how to explore a topic in his own style, so you get to hear about Astoria from a different perspective, without having to listen to my "standard pitch" again :)

The interview is posted in Jon's blog and in Channel 9.

If there was anything that came up on the interview that you'd like to know more about, please go ahead and post them here and I'll be happy to comment.

-pablo

Security in data services

pabloc — Tue, 22 May 2007 05:11:00 GMT

We still have more questions than answers on what is the appropriate security story for data services. The May CTP of Astoria had some ideas built into it, but it was clearly not enough to build real-world applications. There is quite a bit that needs to be explored in this space.

I wanted to take a few paragraphs to make a few problem statements and describe the way we see the problem. I’d be curious to hear what application scenarios you have and how they may affect the security-related requirements for a data service.

Authentication

When running as part of an ASP.NET application, Astoria data services just pick up whatever authentication scheme the application is using. All that matters from the Astoria perspective is that once the client-agent has been authenticated the current principal is properly set; Astoria will just use that principal later on during authorization.

For the online service version things are trickier. There are at least two different players here: the creator/owner of the application or web site, and the user that logs in, the viewer/owner of the data. One trivial approach would be to hand out a set of credentials to the creator of the application, and they could use that to log into the data service. However, there are two big issues with that:

That works well if the access to the data service happens from the server side, but what if you want to access the service from the client? You'd have to send the credentials to the client so that the code sitting on the client can log-in, probably not a good idea.
Even if you could protect the credentials, is that the right thing to do? The end-user is probably the owner of the data in the end, so it may not make sense to use the application creator’s credentials for accessing it.

Another approach that addresses these issues would be to say that the end-user has to authenticate against the data service separately from the application or web site. This way, the user owns the data and can secure it. This seems more natural, but has the issue of requiring the user to authenticate twice (with the application and with the data service).

You could take this further and have some sort of secure token service that all parties trust; then users could authenticate once and the client code could present the tokens as needed for both the application and the data service.

Authorization

The key question from the authorization perspective when it comes to data services is, in my opinion, what is the right granularity.

We are thinking of doing the usual principal/role-based authorization, but the thing is what is authorized.

In the May CTP of Astoria the story is super simple, and it wasn't really so you could build real applications, but more to get developers thinking whether that was the pattern that we should grow into a full-blown one or we needed to start from scratch. In that release the unit of authorization was a whole entity-set (e.g. "Customers"); you could say whether you could act on the entity-set if you are anonymous and if you are authenticated. "Act" means whether you can even see it, and separately whether you can write to it.

Moving forward we could extend that so that it is not just whether the client is authenticated, but you can also say which principals/roles are allowed. However, that may still not be flexible enough. Beyond that, I could imagine a couple of options:

Predicate-based authorization, where each principal/role can be bound to a given filter predicate for each entity-set, and you get to see/modify only those entities that match the criteria given by the predicate.
Instance-based (row-level) security. This is nice to have, but very tricky to implement if you want the system to scale arbitrarily (say "internet scale").

Maybe there are other strategies for authorization, in particular to describe the units of authorization so that they are useful for building actual applications.

As I said above, more questions than answers in this space for now. I'll keep posting as we make progress on the design in this area.

-pablo

Astoria in Channel9

pabloc — Tue, 08 May 2007 06:51:00 GMT

Charles from Channel9 came by my office a week or two ago and recorded this video where we discussed the motivations, target scenarios and some of the interface details of Astoria.

http://channel9.msdn.com/showpost.aspx?postid=305985

-pablo

Application models for Astoria

pabloc — Sat, 05 May 2007 02:27:00 GMT

When you combine technologies such as Silverlight or AJAX-style applications with Astoria there is an opportunity for building great interactive data-driven applications. However, this combination also results in a new ways of organizing the various pieces that make up the application, so the question on what is the right application architecture comes up.

I’ll tackle the question from the middle-tier/data service perspective, and how the interaction with the presentation tier works. I won’t touch on aspects related to building and managing the user-interface elements of the presentation tier.

One extreme: the "pure data" model

There is a class of applications that are just about data, with relatively or no semantics on top of it; or more likely, where semantics is deeply embedded into the data and doesn’t need a layer of behavior on top to surface or enforce it.

For those applications you can imagine pointing the Astoria service to the database, prepare a nice EDM schema that models types and associations appropriately, and leave the URI space open so client agents can freely use any part of it. This is great when the high-order bit is to share data with people or systems. The data in the store is readily available and with easy access; many tools can deal with HTTP and URIs; Astoria-aware tools and widgets can even make it trivial to display and manipulate data.

You would still setup policies for authorization and such, of course, but I see that as somewhat orthogonal to the semantics/behaviors that may go with the data. From the authorization perspective you would say who can read, who can write, when do you need to be authenticated to even see something and so on. It does happen often that some security aspects are derived from semantics (e.g. "my customers" implies that some components knows how to find customers that are assigned to me, and that is typically captured as data as well), in which case the "pure data" approach won't work in that particular scenario.

I’m not sure if there is a lot of applications that fall in this category, but I'm pretty sure its > 0.

The other extreme: the "pure RPC" model

Elaborate applications, such as those designed to support the operations of businesses out there, need more than "just data" pretty much always, at least for large portions of the data they operate on.

A way of approaching an application with these requirements is to build the core business logic on top of the data, and then provide an interface or façade that clients and other applications can consume. This is nothing new, and has evolved in what is currently called Service Oriented Architecture or SOA.

One of the key aspects of SOA is that contracts are clearly defined, and they are described in terms of operations; of course that the data used by those operations needs to be described as well, but the focus is on operations.

I think that there is a space where this is the right way of creating apps. It does introduce a bit of complexity, but you get in return the ability to build composite applications based on the metadata that describes individual components.

We (Microsoft) have a solid offering in this space through the Microsoft Windows Communication Foundation (WCF) stack. WCF handles the runtime and design-time aspects of this, supports various services and related technologies, and it even plays well with both the SOAP and the HTTP-only (should I say REST?) ways of doing things.

The main drawback of this approach, though, is that it's very hard to build generic frameworks/UI components on top. For example, if you had a "grid" widget, how would the widget be able to do paging and sorting (when the user clicks on a column heading) independently of the service operation being called? I am assuming, of course, that bringing all the data to the client and perform the operation there is not an option (as is the case in any reasonable sized data-set). I think there is an opportunity there.

A hybrid approach

Now that we went through the extremes, let me describe a hybrid approach that looks really promising in my opinion.

Typically in business applications you can find a mix when it comes to the data they consume and its requirements for business logic and enforcement of semantics on top of the raw data. I think that often in these applications you can find a part of the data that is still "plain", that doesn't need added business rules, it is just what it is; you also find the other part that does need business logic on top to make sense, or to say consistent, or both.

The relative sizes of the "pure data" and "data + behavior" parts depends on the nature of the application. I would expect, for example, that a public web site that is designed to share information will lean towards the "pure data" side, where as an internal business application would go the other way.

As I discussed in the Astoria Overview document, one of the key elements of the Astoria services is the use of a uniform URI format regardless of the particular data service you're hitting. This enables tools and frameworks to be built so that they leverage that.

These applications could be built by using two Astoria elements: flat access and "data aware" service operations.

For the parts of the data that are "just data", you could enable direct access to the data through regular Astoria URIs. This helps with development productivity, allows developers to use common widgets and libraries and reduces how much redundant code you write and maintain.

For the parts of the data that need business logic, you create the special form of WCF service operation that Astoria introduces support for, where you don't just return the final data that you want. Instead, you return "query objects" that represent the data you'd like to return. Since what is returned is a non-executed-yet query, the Astoria runtime can still compose query operations on top, such as sorting and paging, and then executing it in order to obtain the data to be sent to the client.

Let me use a brief example to illustrate this point. If you have a service operation that returns Customer entities for a given city, you would use a URI like this one to invoke it:

http://host/vdir/northwind.svc/CustomersByCity?city=London

Now, if this is a data aware service operation, the URI options that UI controls and such use to do their work would still work, so when the user clicks on the “CompanyName” column to sort it, the UI control could simply say:

http://host/vdir/northwind.svc/CustomersByCity?city=London&$orderby=CompanyName

in similar ways, using control arguments such as $skip and $top, the grid widget could support paging over any arbitrary data aware operation.

This offers a great middle-ground between plain open access to the store through URIs and fixed RPCs that do not support query composition. The service operation is still written as code in the middle tier that is controlled by the developer; that code can perform validations on arguments, adjust arguments based on the user or the context, inject predicates to queries based on various conditions and so on. In other words, they are the spot where you can invoke your business logic.

I focused on data retrieval so far. The hybrid approach can be used for updates as well. In those cases where it makes sense to allow direct updates to the data you can let the standard HTTP POST/PUT/DELETE operations do its work; for the other cases, where you do not want just an update but a more complex operation that may result in one or more updates, you can use a service operation.

A complementary tool: interceptors

A broad subset of the cases where you want custom business logic during updates, the requirement is to be able to perform validation that goes beyond the declarative validation that can be enforced by constraints in the EDM model or in the underlying database schema.

For those cases you can still use regular HTTP-style updates with POST/PUT/DELETE verbs, and register code to run in the server whenever an entity is being created, updated or deleted. In Astoria you can define an "interceptor" and bind it to a given entity-set and a direction (reading or writing); the interceptor is a regular .NET method defined in the service class that runs in the middle-tier (where the Astoria service runs). This method can do pretty much anything it wants (Astoria won't limit what it can do, although the runtime environment may). Astoria will even provide an already-established session to the database wrapped in an ADO.NET typed object context to provide easy access to the database in case you need to perform lookups in order to validate the operation.

BTW - if anybody can think of a better name than "interceptors", it'll be happy to take it :)

For more details and an example of this you can take a look at the "Intercepting Entities on Retrieval and Update" section of the Using Astoria document.

Summary

So, summing it up, I think that there is a good middle-ground between flat open access to data and pure RPC. I think that this middle-ground can enable frameworks and tools to help more and make it easier to build applications.

-pablo

Astoria FAQ from MIX

pabloc — Thu, 03 May 2007 22:55:00 GMT

We announced Microsoft Codename Astoria at Mix this week, and we did two sessions on the technology during the event. This was great because we got a chance to talk 1:1 with a lot of folks that were interested in the topic. We also got a ton of feedback online through blogs and emails.

Here are some of the most common questions we got since we announced Astoria. Again, this is a very early release, so not all is nice and baked and fully designed yet. Also, there are some topics that need some in-depth discussions, and I’ll tackle those separately in future posts in the next few days.

Can Astoria expose data from data sources other than databases?

Astoria is built on top of the Entity Framework. Currently the EFx only works on top of databases (or things that have ADO.NET providers and look at lot like databases), so Astoria has this restriction as well for its default data services. Astoria also has ability to expose service operations (which are just WCF service operations); those can gather data from anywhere (we won't look at the code inside the operation implementation) but you don't get the nice automatic URI mapping for entities and relationships, and you don't get automatic create/modify/delete support.

How tightly integrated is Astoria to SQL Server?

It is not. I do have an emotional attachment with SQL Server as I worked (still do) in the product for several years :), but Astoria hasn't really anything specific to SQL Server. As I mentioned in the previous answer, Astoria builds on top of the ADO.NET Entity Framework, so if you have an up-to-date ADO.NET provider for your database you should be able to use Astoria with it.

This looks really similar to SQLXML. What is the relationship between the two technologies?

They way I see it, these are very similar technologies that are designed to tackle quite different scenarios. The goal of SQLXML was to provide an XML view on top of a SQL Server database, preserving many (all?) aspects of the XML data model. The goal of Astoria is to provide a service for data that can be web facing and AJAX/RIA applications can use as their back-end.

This difference in goals surfaces in many forms. For example, SQLXML used XPath for queries, where in Astoria we’ve specifically stayed away from it. SQLXML was, well…about XML, where as Astoria is about data services and formats come second (Astoria currently does XML, RDF+XML and JSON, and I was given some great ideas for new formats at Mix). As a last example, Astoria has several mechanisms built-in for introducing business logic in the data service, so it results in a web-facing entry point that client agents can interact with without compromising the integrity of the application.

On a more practical note, I work in what would be the current form of the team that created SQLXML among other things. I regularly talk with the folks that built it, such as Andy Conrad and Michael Rys. We'll "reuse" as many good ideas as we can, and we’ll avoid things that didn’t turn out that well.

The documents say "semantics" all over the place, but not "semantic web", is there any relationship between Astoria and the semantic web?

The "semantic web" is a broad space of specifications, technologies and concepts. The goals of the semantic web are far-reaching.

Astoria does heavily rely on understanding more semantics on the data it handles than a flat "blob store". In Astoria we get semantics off of the Entity Data Model schema that a given data service uses. However, Astoria doesn’t not currently tackle all of the elements that would be required to say that it’s a tool to directly support semantic web constructs.

It does have some pieces though. The fact that everything is pointed at by a URI and looks like a resource is an important building block. The fact that it supports RDF+XML, with entities modeled as sets of triplets and associations modeled as resources, is also another step there.

Now, there are things such as not supporting RDF Schema or OWL ontologies that will quickly come up when you look at this with a semantic web perspective. You could imagine generating ontologies based on the EDM schema information (we have information about types, inheritance, associations, etc.). Astoria also does not support SPARQL or any other RDF query language.

Bottom line: I do think Astoria brings a small little contribution to the goal of adding semantics to the web, but it’s not a full on semantic web tool. Whether it will evolve into something closer to it or not it’s something that time will tell, and user feedback will influence.

What is the security model for Astoria?

I’m putting this here for completeness, but let me defer the answer to a future post.

No business logic? Data directly exposed? How do you architect applications on top of Astoria?

Same as above. I’ll do a write up soon.

If you have more questions feel free to send them, I'll do another round of FAQs once I get a few more together.

-pablo

Codename "Astoria": Data Services for the Web

pabloc — Mon, 30 Apr 2007 20:31:00 GMT

The "data programmability" team at Microsoft is responsible for the various technologies developers use in applications to access and manipulate data. One of the topics we have been looking at lately is how "new" internet applications use data in the web environment. Project codename "Astoria" reflects our current thinking on the topic. Instead of telling "what we are going to build", we decided to make the bits and infrastructure public for the development community to look at and give us feedback.

Today we are making a very early experimental release of Project Codename "Astoria". This Community Tech Preview (CTP) release has a dual nature; we are shipping both CTP bits you can download, install and run on your own systems; we are also making available an experimental online service that we hope will help better understand the requirements and use cases of data interfaces in the web.

So, what does it do?

Astoria exposes "data services" that enable applications to access and manipulate data over regular HTTP connections, using URIs to identify pieces of information within the data service, and simple payload formats such as XML and JSON to represent the data exchanged between the client and the server.

What is it useful for?

There are a number of scenarios that we think are interesting for this technology. Most of the scenarios are related to the way new web applications are built.

If you look at AJAX-based web applications, one interesting aspect of the way they are delivered to the client is that typically the presentation and client-side behavior is delivered on the initial hit to a given web page, and then the code in that page (typically Javascript) turns back and fetches data as the user interacts with the user interface. This brings a strong separation between presentation and data. What is more, this means that now we require a server-side piece that the javascript code can "talk to". Of course, you can always roll you own server-side entry points for data access; however, not only that is expensive but also it greatly reduces the chances to build tooling and user-interface controls that can work generically on any data access entry point into the server.

This separation is even stronger in rich internet applications built on technologies such as Silverlight and Flash. The code that is delivered to the client when a user hits the web application is pre-compiled and it contains no data at all other than any hardcoded information. Again, these applications talk back to their servers to retrieve data, manipulate it and push changes back into the service for storage.

Astoria data services are designed to address this space. They present data in a uniform way that can be consumed by any client that can connect over HTTP and parse XML or JSON. The uniform URI and payload format patterns mean that user-interface widgets can be built so they work against any particular shape of data (“schema” if you will) that is exposed through the service.

Where is it?

The main site is here: http://astoria.mslivelabs.com

This document introduces the concepts and motivations for Astoria.

You can also see and interact with several sample data-sets that are already available on the experimental online service. Click on the "Online Service" link in the link bar.

Looking forward to hearing your feedback.

-pablo