Man vs Code

As promised the slides & demos for my two ASP.NET Connections talks from 11/11/08:

    MDA01: Deep Dive: ADO.NET Data Services Framework—Application Patterns

    MDA04: LINQ to XML, SQL, Entities, DataSets and Co.: Data Access Technologies Explained  - AKA Linq to Anything

Update: 11/12/08 - download now includes all demos including Linq to Twitter, PetBook, and my LINQ samples from our LINQ to Anything (MDA04) talk.  Will update again with Shyam's demos from LINQ to Anything sometime in the next few days.

IUpdatable for Linq to Sql

I have had an implementation of IUpdateable for Linq to Sql about half done for quite some time now. Yesterday I decided to bite the bullet and finish it up.

Below I have included that code. I’ve done some light testing, but I am sure there are some bugs – and perhaps some perf fixes that can be made. My plan is to get this out on CodePlex as soon as I get a moment.

 Update - 11/06/08, I actually ended up putting it on CodeGallery here.

A couple of notes about this code:

1) I haven’t implemented ClearChanges() yet. As far as I can tell there is really no straight forward to do this with Linq to Sql. It isn’t that big of deal because ClearChanges is only required by Astoria when processing batches with multiple ChangeSets. Something that is not supported by the Astoria .NET client.

2) The easiest way to use this is to define a partial class for your DataContext type and provide the implementation there. i.e.

public partial class nwDataContext : IUpdatable

That way you don’t need to change your generated context class.

Next step - implementing IExpandProvider

    public partial class myDataContext : IUpdatable
    {
        /// <summary>
        /// Creates the resource of the given type and belonging to the given container
        /// </summary>
        /// <param name="containerName">container name to which the resource needs to be added</param>
        /// <param name="fullTypeName">full type name i.e. Namespace qualified type name of the resource</param>
        /// <returns>object representing a resource of given type and belonging to the given container</returns>
        public object CreateResource(string containerName, string fullTypeName)
        {
            Type t = Type.GetType(fullTypeName);
            Debug.Assert(t != null);  // assume can find type
            ITable table = (ITable)this.GetType().GetProperty(containerName).GetValue(this, null);
            object resource = Activator.CreateInstance(t);
            table.InsertOnSubmit(resource);
            return resource;
        }

        /// <summary>
        /// Gets the resource of the given type that the query points to
        /// </summary>
        /// <param name="query">query pointing to a particular resource</param>
        /// <param name="fullTypeName">full type name i.e. Namespace qualified type name of the resource</param>
        /// <returns>object representing a resource of given type and as referenced by the query</returns>
        public object GetResource(IQueryable query, string fullTypeName)
        {
            object resource = null;

            foreach (object o in query)
            {
                if (resource != null)
                {
                    throw new Exception("Expected a single response");
                }
                resource = o;
            }

            // fullTypeName can be null for deletes
            if (fullTypeName != null && resource.GetType() != Type.GetType(fullTypeName))
                throw new Exception("Unexpected type for resource");
            return resource;
        }


        /// <summary>
        /// Resets the value of the given resource to its default value
        /// </summary>
        /// <param name="resource">resource whose value needs to be reset</param>
        /// <returns>same resource with its value reset</returns>
        public object ResetResource(object resource)
        {
            Type t = resource.GetType();
            Debug.Assert(t != null);
            ITable table = this.GetTable(t);
            return table.GetOriginalEntityState(resource);
        }

        /// <summary>
        /// Sets the value of the given property on the target object
        /// </summary>
        /// <param name="targetResource">target object which defines the property</param>
        /// <param name="propertyName">name of the property whose value needs to be updated</param>
        /// <param name="propertyValue">value of the property</param>
        public void SetValue(object targetResource, string propertyName, object propertyValue)
        {
            PropertyInfo pi = targetResource.GetType().GetProperty(propertyName);
            if (pi == null)
                throw new Exception("Can't find property");
            pi.SetValue(targetResource, propertyValue, null);
        }

        /// <summary>
        /// Gets the value of the given property on the target object
        /// </summary>
        /// <param name="targetResource">target object which defines the property</param>
        /// <param name="propertyName">name of the property whose value needs to be updated</param>
        /// <returns>the value of the property for the given target resource</returns>
        public object GetValue(object targetResource, string propertyName)
        {
            PropertyInfo pi = targetResource.GetType().GetProperty(propertyName);
            if (pi == null)
                throw new Exception("Can't find property");
            return pi.GetValue(targetResource, null);
        }

        /// <summary>
        /// Sets the value of the given reference property on the target object
        /// </summary>
        /// <param name="targetResource">target object which defines the property</param>
        /// <param name="propertyName">name of the property whose value needs to be updated</param>
        /// <param name="propertyValue">value of the property</param>
        public void SetReference(object targetResource, string propertyName, object propertyValue)
        {
            this.SetValue(targetResource, propertyName, propertyValue);
        }

        /// <summary>
        /// Adds the given value to the collection
        /// </summary>
        /// <param name="targetResource">target object which defines the property</param>
        /// <param name="propertyName">name of the property whose value needs to be updated</param>
        /// <param name="resourceToBeAdded">value of the property which needs to be added</param>
        public void AddReferenceToCollection(object targetResource, string propertyName, object resourceToBeAdded)
        {
            PropertyInfo pi = targetResource.GetType().GetProperty(propertyName);
            if (pi == null)
                throw new Exception("Can't find property");
            IList collection = (IList) pi.GetValue(targetResource, null);
            collection.Add(resourceToBeAdded);
        }

        /// <summary>
        /// Removes the given value from the collection
        /// </summary>
        /// <param name="targetResource">target object which defines the property</param>
        /// <param name="propertyName">name of the property whose value needs to be updated</param>
        /// <param name="resourceToBeRemoved">value of the property which needs to be removed</param>
        public void RemoveReferenceFromCollection(object targetResource, string propertyName, object resourceToBeRemoved)
        {
            PropertyInfo pi = targetResource.GetType().GetProperty(propertyName);
            if (pi == null)
                throw new Exception("Can't find property");
            IList collection = (IList)pi.GetValue(targetResource, null);
            collection.Remove(resourceToBeRemoved);
        }

        /// <summary>
        /// Delete the given resource
        /// </summary>
        /// <param name="targetResource">resource that needs to be deleted</param>
        public void DeleteResource(object targetResource)
        {
            ITable table = this.GetTable(targetResource.GetType());
            table.DeleteOnSubmit(targetResource);
        }

        /// <summary>
        /// Saves all the pending changes made till now
        /// </summary>
        public void SaveChanges()
        {
            this.SubmitChanges();
        }

        /// <summary>
        /// Returns the actual instance of the resource represented by the given resource object
        /// </summary>
        /// <param name="resource">object representing the resource whose instance needs to be fetched</param>
        /// <returns>The actual instance of the resource represented by the given resource object</returns>
        public object ResolveResource(object resource)
        {
            return resource;
        }

        /// <summary>
        /// Revert all the pending changes.
        /// </summary>
        public void ClearChanges()
        {
        }
    }

Along with my manager Shyam Pather, I am giving the following talk at the upcoming Dev Connections conference:

MDA04: LINQ to XML, SQL, Entities, DataSets and Co.: Data Access Technologies Explained
With the release of the Microsoft .NET Framework 3.5, Microsoft introduced several new data access technologies based on its LINQ technology, such as LINQ to SQL and LINQ to DataSets. Shortly after, we expect Microsoft to release the ADO.NET Entity Framework which enables LINQ to Entities in addition to Entity SQL. Finally, we have all the existing ADO.NET data access patterns. This raises one important question: Which data access technology is best suited for which situation? In this session, we look at different scenarios and requirements that can inform your technology decision.

As part of this talk, we are trying to give as many Linq demos as possible.  So far we have:

Linq to Objects

Linq to Sql

Linq to Entities

Linq to DataSet

Linq to Xml

Linq to ADO.NET Data Services

Linq to Windows Azure Storage

Linq to Sql Data Services

Plus a few surprises. 

I am looking at something real esoteric to write a Linq provider.  So if anyone has any ideas, please leave a comment or email me.

The Astoria team is hiring!

We have positions open for Developers and Program Managers.  Currently, we are finishing up V1 and planning for V2, so it is a great time to join the team.  For V2, we are thinking about expanding the project into new areas (e.g. offline support, sync, dynamic language support) so there will be plenty of new and exciting work.

Please forward your resume to aconrad@microsoft.com if you’re interested.  The project covers a diverse set of technical areas (databases, Linq, O/R, REST, etc) so we value passion for the technology more than deep expertise in any particular area. The Astoria team uses an agile development model, so the project moves fast and we encourage individuals to take on a variety of roles and responsibilities. If you’re a developer who wants to be responsible for more than just coding (i.e. API design, specification writing, working with customers and partners) this team is the place for you!

And finally, team members must be willing to participate in the grueling Team Astoria off-site events.

Update 4/23/08 - As of the start of this week, we now have a block of additional new positions on the Astoria Team we need to fill.  These include development, program management, and quality assurance positions.  We are trying to fill these very quickly, so if you are interested please contact us soon.

The news is out! At Mix, Windows Live will be announcing several new Live Services with AtomPub endpoints.  These services will also be compatible with Astoria and therefor will work with the Astoria client library.  An excerpt from David Treadwell's post:

Microsoft is making a large investment in unifying our developer platform protocols for services on the open, standards-based Atom format (RFC 4287) and the Atom Publishing Protocol (RFC 5023). At MIX we are enabling several new Live services with AtomPub endpoints which enable any HTTP-aware application to easily consume Atom feeds of photos and for unstructured application storage (see below for more details). Or you can use any Atom-aware public tools or libraries, such as .NET WCF Syndication to read or write these cloud service-based feeds.

In addition, these same protocols and the same services are now ADO.NET Data Services (formerly known as “ Project Astoria”) compatible. This means we now support LINQ queries from .NET code directly against our service endpoints, leveraging a large amount of existing knowledge and tooling shared with on-premise SQL deployments.

The intent for these early, experimental releases are to gather valuable feedback from the community around our idiomatic and freely licensed extensions to AtomPub which deal with important service scenarios, such as URL formats, nested directories, image streams, and service metadata. You can read more about this on the Project Astoria team blog.

There will also be a few more surprises in this area announced at MIX. Stay tuned. Try them out and give us your feedback!

At Mix, we have a Project Astoria talk all about using the Astoria client library to consume Live services. 

RESTful Data Services with the ADO.NET Data Services Framework

Wednesday, March 5 4:30 PM - 5:45 PM, Lando 4204

Speaker(s): Pablo Castro

Audience(s): Technical

Session Type: Breakout

Learn how to use ADO.NET Data Services Framework to easily create and consume REST data services on the web. This session will cover the main concepts of the ADO.NET Data Services Framework (aka Project "Astoria"), show how to use it, and discuss how to use it with Microsoft's broader vision of a common interface for Windows Live and 3rd party services.

So if you are going to Mix, be sure to go to that talk.  The Mix talks are also available online 24-48 hours after they are given.

On a related note Pablo, Mike, and myself will all be at Mix next week.  We should be spending a lot of time in the hands on lab/ Open Spaces area.  If anyone has any questions about any of our projects, wants to see some Astoria demos, etc - drop  me a line and we can set something up.

Last Wednesday we had a very important offsite for the entire Project Astoria team:

Many key technical issues were discussed. In fact, the entire team was quite spent after a long, hard day. 

Michael Sync has written a sample using the new Astoria Silverlight client library.

David Hayden posted a nice walk through for getting an Data Service up and running over Linq To Sql.

Our own Macelo Ruiz talks about some tweaks we made to the Astoria URI syntax on his personal blog.

Matthew Wills ported my original C# CopyToDataTable<T> Linq operator to VB.  According to him, he used Instant VB to convert the code from C# to VB and then hand tweaked it.  Looks like a fairly cool tool that we could use to port our C# unit tests to VB.   (We try to run the same functionals in both VB and C# these days because of the differences in Linq and comprehension support between the two compilers).

The updated Astoria Silverlight client is now available.

This client is 95% compatible with the client library we released back in the December CTP (we hope to make this 100% source code compatible by RTM) + works with the latest Silverlight 1.1 Alpha.

When the next preview of Silverlight releases this spring, we will release an update of the Astoria Silverlight client.

A number of people have asked me for a VB version of the CopyToDataTable<T> sample I wrote a few months back.   Unfortunately, between getting some skiing in (lots of snow in Washington this year!) and getting Astoria ready for Mix, I have not had very little free time.  Hence, if anyone else wants to take a crack at this - or knows of an existing port of the code to VB, let me know.

Marcelo Lopez Ruiz, a developer on the Astoria team, has written a nice post describing how to deal with Astoria server errors returned via HTTP.

One thing I regret that we didn't get into the Astoria December CTP was the ability to show detailed Astoria server errors to remote clients.  Something like what ASP.NET supports.  That way the developer can see what errors occurred on the server via the web browser.  Currently, one has to have a debugger attached to the server process to get the equivalent information.  This is particularity interesting in cases where the Astoria server code is not the component throwing the exception - for example when the Queryable provider throws an exception.

In addition, we are also looking at adding a way for providers to map specific exceptions to HTTP errors which could be used by provider writers to specify runtime HTTP errors instead of the generic "Internal Server Error".  For example, if there was a database constraint error when inserting some data.  Hopefully, this feature will make it into the next beta.

Jonathan Carter is in the midst of writing a 11 Part (!) Training Series about ADO.NET Data Services.

Guy Burnstein has also started a “Get Started with ADO.Net Data Services” post series.

Late Sunday night we released the first public preview of ASP.NET 3.5 Extensions.  For information on the release, see ScottGu's announcement.

As part of that preview, we released the first CTP of the production version of Project Astoria.  For details, see public release announcement by Mike Flasko on the Astoria Team Blog.

Astoria actually has been given its official Microsoft name:  ADO.NET Data Services.  So, in this post I will use the name Astoria and ADO.net Data Services interchangeably.

For a complete overview of Astoria check out the information available on Microsoft Live Labs.

For the next few weeks, I will be writing a series of blog posts to demonstrate what I think is one of the most compelling features in Astoria, namely remote execution of Linq queries via the RESTFUL API exposed by Astoria Data Services.

For these posts, I will be covering:

1)  How to set up an Astoria Data Service which can be used for remote execution of Linq queries.

2)  Dive deep into the Astoria Client API's Linq support including a detailed description of how Linq queries are translated to URIs.

3)  How to make the Astoria Data Service updateable so the results of the Linq queries can be remotely modified and persisted.

So the goal of this initial post is to set up an Astoria Data Service over a Linq to Sql DataContext source and remotely execute a Linq query via the Astoria client library over HTTP.

(The following steps assume Visual Studio 2008 and ASP.NET 3.5 Extensions has been installed.  Also, this demo assumes access to the Sql Server sample Northwind database).

First, Create a standard ASP.NET Web Application:

Next, add an ADO.NET Data Service called Northwind.svc to the project via the Add New Item option:

Next, add a new set of Linq to Sql via the designer called Northwind.dbml.

From my Data Connections in the Server Explorer, I add the Categories, Products, and Supplier tables to Northwind.dbml from the Northwind database

Because the keys for the new types follow the Astoria convention of being named [EntityName]Id, I don't have to do anything further.  The Linq to Sql data classes and the strongly typed DataContext (NorthwindDataContext) is totally useable with Astoria without any modifications.

Next, I want to wire up my Astoria Data Service to use NorthwindDataContext as the data source.  To do this, go back to the Northwind.svc.cs file that was created when the Astoria Data Service was added to the project.  Look for the TODO's in the Northwind class:

    public class Northwind : WebDataService< /* TODO: put your data source class name here */ >
    {
        // This method is called once during service initialization to allow
        // service-specific policies to be set
        public static void InitializeService(IWebDataServiceConfiguration config)
        {
            // TODO: set rules to indicate which entity sets and service operations are
            // visible, updatable, etc.
            // (for testing purposes use "*" to indicate all entity sets/service
            // operations, but that option should NOT be used in production systems)

            // Example for entity sets (this example uses "AllRead" which allows reads but not writes)
            // config.SetResourceContainerAccessRule("MyEntityset", ResourceContainerRights.AllRead);

            // Example for service operations
            // config.SetServiceOperationAccessRule("MyServiceOperation", ServiceOperationRights.All);
        }

        // Query interceptors, change interceptors and service operations go here
    }

First, put the name of the DataContext in as the generic parameter for the Northwind class.

Next. add the following line to the InitializeService method:

config.SetResourceContainerAccessRule("*", ResourceContainerRights.AllRead);

By default, Astoria turns off all access to the data source.  By adding this line, I am giving read access to all public IQueryable<T> entry points on the DataContext. 

Note, the wildcard option (*) should only be used for testing purposes.  For actual application development, please explicitly grant access to each IQueryable<T> entry point by via the property name.

Now build the application and press F5 to fire it up.  If everything went ok, you should get back the following Atom service definition when you access the URI http://localhost:18752/Northwind.svc/ (I am using port 18752 with Cassini for debugging).

  <service xml:base="http://localhost:18752/Northwind.svc/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:app="http://www.w3.org/2007/app" xmlns="http://www.w3.org/2007/app">
    <workspace>
       <atom:title>Default</atom:title> 
       <collection href="Categories">
         <atom:title>Categories</atom:title> 
       </collection>
       <collection href="Products">
         <atom:title>Products</atom:title> 
       </collection>
       <collection href="Suppliers">
         <atom:title>Suppliers</atom:title> 
       </collection>
    </workspace>
  </service>

Now, try a few URIs to make sure everything it working:

(Note, Internet Explorer by default renders ATOM in a friendly format that makes Astoria feeds unreadable.  To fix this, turn off IE feed viewing by turning off the option :  Tools -> Internet Options -> Content ->  Feeds (settings) -> Turn on feed viewing)

http://localhost:18752/Northwind.svc/Products - gets all the Products

http://localhost:18752/Northwind.svc/Products(2) - gets the Product with ProductId == 2

http://localhost:18752/Northwind.svc/Products?$filter=UnitsInStock gt 100 - gets all the Products where UnitsInStock is greater than 100

http://localhost:18752/Northwind.svc/Products?$orderby=ProductName&$top=10 - orders the Products by ProductName and gets the first 10.

For more information about Astoria's URI Addressing Scheme see here.

Now, lets get a sample Linq query working.

First, add another project to the solution.  In this case, a simple console Application will work:

In this project, add a reference to the Astoria client library assembly called Microsoft.Data.WebClient.dll. It should be in your /Program Files/Reference Assemblies/Microsoft/Framework/ASP.NET 3.5 Extensions directory.

Now add make sure the namespace Microsoft.Data.WebClient is in scope, and add the following class to your project:

    [OpenObject("PropBag")]
    public class Product
    {
        private Dictionary<string, object> propBag = new Dictionary<string, object>();

        [Key]
        public int ProductID { get; set; }
        
        public string ProductName { get; set; }
        
        public int UnitsInStock { get; set; }

        public IDictionary<string, object> PropBag { get { return propBag; } }

    }

Finally, add the following to Program.Main:

        static void Main(string[] args)
        {
            WebDataContext context = new WebDataContext("http://localhost:18752/Northwind.svc");

            var query = from p in context.CreateQuery<Product>("Products")
                        where p.UnitsInStock > 100
                        select p;

            foreach (Product p in query)
            {
                Console.WriteLine(p.ProductName + " , UnitsInStock= " + p.UnitsInStock);
            }

        }

To get both projects to start up for debugging, select Solution-> Properties and select Multiple startup projects with the Action of Start.

Put a breakpoint on the foreach statement in the client app, and hit F5.

If you hover over the query variable, you will actually see the Astoria URI which the Linq query is translated into by the Astoria client library:

http://localhost:18752/Northwind.svc/Products?$filter=(UnitsInStock)%20gt%20(100)

So, there you go.  Linq to Astoria's RESTFUL API.  In other words, Linq to REST. 

In future posts, I will explain how that worked.  What all the Linq capabilities that are supported.  And what one can do with the objects once they are materialized.

A couple of months ago I became Development Lead for Project Astoria.   The goal of Project Astoria is to provide a data services for the web. AKA Project Astoria provides a RESTFUL head for any sort of "queryable" data, for example data stored in relational databases.  It also contains a O/R style client library (usable via a standard .Net or Silverlight client) with Linq support for accessing the Astoria Data Services.  For a much more in depth description of the project, check out the project overview document.

Over the last few months, the team has moved from prototype mode to working on the production code.  In an effort to elicit quicker feedback on our design, we have started publishing our design notes on the Astoria Team Blog.  Normally our design team meets twice a week for a couple of hours, so we expect to be able to publish these notes fairly regularly.  In most cases, we are going to be right in the middle of coding and testing the features described in the design notes when the notes are published.   Hence, based on feedback from the developer community we will change the design and implementation mid flight. 

Pablo has already published the first set of design notes on the Payload Formats.  Later this month we should have notes on the URI format and the service extensibility.

We are also looking to hire a few people in anyone is interested.  See here for more information or send me your resume directly to me if interested.

Tok Wee Hyong of the .Net Singapore user group has put together a nice Project Jasper sample available here if anyone is interested.