ADO.NET Data Services Team Blog : ADO.NET Data Services

Introducing the Microsoft Open Data Protocol Visualizer (CTP1)

dpblogs — Tue, 17 Nov 2009 19:36:00 GMT

1. Introduction

The Open Data Protocol Visualizer for Visual Studio CTP1 (“the visualizer”) is a extension that provides a read-only visualization of the types, properties, associations, and other objects in the Entity Data Model (EDM) returned from an Open Data Protocol’s metadata endpoint (this document assumes basic familiarity with the EDM).

For instructions on how to create a data service, see here.

This blog post uses a slightly modified version of a model reverse-engineered from part of the Northwind database. For general purposes, the AdventureWorks sample database, or any other database you choose, will work as well.

2. Walkthrough

The modifications made to the Northwind model were done in order to demonstrate additional features of the visualizer. Specifically:

1. The address fields in various types in the model have been replaced by the usage of a Complex Type called “Address”.
2. A subtype of Product called “DiscontinuedProduct” has been added.
    a. The “Discontinued” property has been removed from Product.
    b. Product is mapped to the Products table with condition Discontinued=false
    c. DiscontinuedProduct is mapped to the Products table with condition Discontinued=true
3. Two function imports have been added to the model for the “Employee_Sales_by_Country” and “Sales_by_Year” stored procedures.
4. For each stored procedure, a complex type has been created to hold its results, and a method added to the data context to enable it.

The XML for this model is available for download here.

2.1 Install

Launch the “Extension Manager” from the Visual Studio “Tools” menu. The visualizer is available from the online gallery, and you can find it quickly by entering “Open Data Protocol” into the search field:

The visualizer is available for download from the Visual Studio Gallery here:

http://visualstudiogallery.msdn.microsoft.com/en-us/f4ac856a-796e-4d78-9a3d-0120d8137722

2.2 Setup

To get up and running:

1. Create a solution containing a web application project and a console application.

2. Add a new, blank ADO.NET Entity Data Model to the web application.

3. Close the model.

4. Right-click the model file, select “Open With” and in the list, select “XML Editor”.

5. Paste the contents of the Modified Northwind Model into it.

6. Save and close the model file.

7. If you do not have it already set up, add the Northwind.MDF database to the web application’s App_Data folder.

8. Double-click the model file to open it with the designer.

9. Right click on the designer surface and select “Update Model from Database”.

10. Select the connection to Northwind.MDF, which will be automatically created for you when the MDF file is added to App_Data. If you are using your own database, you may need to create a new connection to it.

11. Proceed through the update wizard, accepting all the defaults, adding no new objects. This process will set up the database connection for you in web.config. It should look like this:

12. Add a new WCF Data Service (in Beta2 this will show up as ADO.NET Data Service) and configure it as shown below. Note the additional web methods and the exposure of all entity sets and all service operations using the “*” string. This lack of security means that this is not a data service that you would want to expose publicly.

using System;
using System.Collections.Generic;
using System.Data.Objects;
using System.Data.Services;
using System.Data.Services.Common;
using System.Linq;
using System.ServiceModel.Web;
using System.Web;

namespace NorthwindServices
{
    public class NorthwindService : DataService<NORTHWNDEntities>
    {
        // This method is called only once to initialize service-wide policies.
        public static void InitializeService(DataServiceConfiguration config)
        {
            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
            config.SetServiceOperationAccessRule("*", ServiceOperationRights.All);
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
        }

        [WebGet]
        public ObjectResult<Employee_Sales_by_Country_Result> GetEmployeeSalesByCountry(DateTime start, DateTime end)
        {
            NORTHWNDEntities entities = new NORTHWNDEntities();
            return entities.Employee_Sales_by_Country(start, end);
        }

        [WebGet]
        public ObjectResult<Sales_by_Year_Result> GetSalesByYear(DateTime start, DateTime end)
        {
            NORTHWNDEntities entities = new NORTHWNDEntities();
            return entities.Sales_by_Year(start, end);
        }
    }
}

13. Right click on the service and select “View in Browser”.

14. Copy the URL shown in the browser, it should look like this: http://localhost:11968/NorthwindService.svc

15. In the console application project add a new service reference.

16. Paste the URL into the “Address” field. Change the “Namespace” to “NorthwindServiceReference”, as shown below.

17. Click OK.

2.3 Visualizing the Model

The visualizer consists of two main components: The diagram, which can show all, or part of the model, as a two-dimensional graph, and the model browser, which is a Visual Studio tool window that presents the complete contents of the model as a hierarchical tree. We begin by examining the structure and capabilities of the diagram.

2.4 Launching the Visualizer

Once installed, the visualizer is launched by right-clicking any Service Reference that supports the Open Data Protocol and selecting “View in Diagram”:

This will bring up the overview screen in a new document window. This screen provides a number of quick links for rapidly creating a diagram:

Link

Behavior

Data services model browser

Will make the model browser tool window visible.

Add all namespaces

Adds all entity types and complex types, and by extension, all associations, to the model.

Add all entity types

Will add all entity types, and any associations between them, to the diagram.

Add all hubs

Will add the entity types whose number of associations is in the top 25% in the model to the diagram. This is intended to provide a useful starting point in making sense of models by adding only the most highly connected types.

Add all complex types

Will add all complex types to the diagram.

Add all associations

Adds all associations and their entity types to the diagram.

2.5 Working with a Diagram

1. Selecting “add all hubs” will result in the following diagram:

2. Deselect any selected objects (by clicking anywhere on the canvas) and right-click the “Employee” shape to bring up this context menu:

Select “Add Related Entities”.

3. Click on Employees again and select “Add Related Complex Types”.

4. Switch to the left-to-right view (Alt+Right Arrow.)
The diagram now shows all entity types related to Employee as well any complex types used by it:

5. Multi-select Order and Product and select “Add Related Entities”. At this point, most of the model will be visible in the diagram. Note that all newly added objects are now selected. This is intended to accomplish two goals: Allow easy iterative addition of related types to a model, and also make it easy to distinguish newly added types from previously added types.

2.5.1 Visualizing Relationships

1. We can further simplify the rendering of the model by changing the way shapes are rendered. Right click anywhere in the diagram and unmark the “Expanded Shapes” checkbox command. This will hide all properties, making it easier to focus on relationships between types:

2. Selecting an association will highlight the navigation properties that navigate it. For example, after expanding the shapes to show properties again, we select the association between Employee and Territory and see that navigation properties Employee.Territories and Territory. Employees, which traverse this association, become highlighted:

3 Using the Model Browser

The Model Browser provides an unfiltered view of the model. Its root contains a schema node for each namespace in a model, and a node the EntityContainer.

To show the model browser, go to the “View” menu item in Visual Studio and select “Data Services Model Browser”. You can also right-click in the visualizer and select “Show In Model Browser”, which will bring the model browser into view and select the object under the mouse.

3.1 Schema Nodes

Schema nodes show the entity types, complex types, and associations in a schema.

To add a type to the diagram, expand the Entity Types node and drag and drop it onto the diagram. You can also right-click the type and click on “Select in Diagram,” or double-click it. This will add the type to the diagram if it is not present, select it, and scroll the diagram to make the type shape visible.

These same actions also work for complex types and associations. In the latter case, adding an association to the diagram will add both of the association’s ends’ entity types as well.

1. Clear the diagram by selecting all objects using Ctrl+A and removing them using the Delete key.

2. Double-click the “Employee” entity type.

3. Drag and drop “Order” onto the diagram. Note that the location of the drop does not matter, as the layout in the diagram is always automatic.

4. Right-click the “Address” complex type and click on “Select In Diagram”. Finally, double-click on the “FK_Orders_Shippers” association. With a left to right layout, your diagram should look like this:

Note that the addition of the foreign key added the “Shipper” type and selected both it and “Order”, which make up both sides of the association, and also selected the two navigation properties, “Order.Shipper” and “Shipper.Orders”, which traverse this association.

3.1.1 Properties

While the diagram surface does not yet allow selection of properties, the model browser does. Expand the Entity Types node, and within it, expand the “Category” type. Right-click “CategoryID” and select “View Generated Code” to see the code generated for this property. Close the code, right-click “CategoryID” and select “Properties” to view the property sheet for it.

3.2 EntityContainer Nodes

Expand the Entity Sets node and right-click on “Categories”. Selecting “View Generated Code” will take you to the dataServiceQuery<Category> that was generated from this entity set. Next, select “Go to Entity Type”, which should select the “Category” type. Finally, select “View in Browser” which will bring up the browser and point it at http://[your server]/NorthwindDataService.svc/Categories. Note that selecting “View In Browser” on entity types, for example on “Category”, will bring up the same URL, as all such queries must reference a type’s entity set.

3.3 The Search Bar

At the top of the model browser is the search bar. Collapse down the model browser to its root node and click the search box labeled “Type here to search”. In the search box, type “cat” and hit enter or click on the icon. The search will cover all objects in the entity container and schema nodes, down to individual properties or association ends. The model browser will show the search results bar, expand all nodes with matches and highlight those matches, and provide tick marks on the scroll bar:

Click the up/down arrows to the right of the “Items matching…” message to move forward and back through results. You can also click on tick marks, which will scroll the corresponding match into view and select it. When you are done, click the button at the right of the search results bar to clear out the search. You can perform this search again any time by selecting the search from the search box’s drop-down search history:

4. Conclusion

This walkthrough does not touch on some additional capabilities of the visualization tool, such as the ability to display data services-specific metadata. We will cover these capabilities, and others, when we release the next update to this tool.

We hope you will find this tool useful and look forward to your feedback.

Noam Ben-Ami & Fabian Winternitz
Microsoft

Breaking Down ‘Data Silos’ – The Open Data Protocol (OData)

dpblogs — Tue, 17 Nov 2009 09:39:00 GMT

A few weeks ago we posted about steps we’ve been taking to enable a wide range of data sources (DB, SharePoint, Reports, Cloud storage services, etc) to expose their data for programmatic access from any platform (Java, PHP, Silverlight, .NET, AJAX, etc) using a simple HTTP/AtomPub based protocol as the means of communication. Our exploration in the area of data sharing has been driven by frustration we’ve observed in the broader community around the fact that data tends to exist in horizontal or vertical silos: it can be shared within a particular environment but often becomes difficult to use beyond its original intent. For example, one factor that can contribute to such a “silo effect”, is application domain/business logic that changes the shape of its data such that it is harder to share. As systems continue to become more interconnected, the need to reuse information in its raw form or exposed through layers of domain/business logic grows and, it probably goes without saying, such data becomes more valuable the more it can be shared. It is this need for data sharing, regardless of the type of data source, where the “silo effect” becomes increasingly painful.

Driven by the goal to enable simple data sharing across disparate data sources, today, at PDC, we are announcing the Open Data Protocol (OData). OData is an open protocol for sharing data that provides a way to break down data silos and increase the shared value of data by creating an ecosystem in which data consumers can interoperate with data producers in a way that is far more powerful than currently possible, enabling more applications to make sense of a broader set of data. Every producer and consumer of data that participates in this ecosystem increases its overall value. This is similar in many ways to Microsoft’s efforts with ODBC, although applied to the Web environment and supporting a variety of data sources including (but not limited to) relational databases, file systems, content management systems, and traditional web sites.

OData applies web technologies such as HTTP, Atom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores and will be published under the Open Specification Promise to allow anyone to freely interoperate with OData implementations. OData was previously known informally as the “data services protocol”, however, we’re renaming the specification to better express its intent. We are very interested in working with the web community around how to best align OData to future versions of AtomPub or other appropriate standards and welcome participation from anyone interesting in furthering the type of data sharing ecosystem OData strives to enable.

From a Microsoft standpoint, the Data Services framework was the first Microsoft technology to support the Open Data Protocol in Visual Studio 2008 SP1 and provides developers with client libraries for .NET, Silverlight, AJAX, PHP and Java. In addition to developing WCF Data Services, Microsoft currently supports the OData in SQL Server 2008 R2, Windows Azure Table Storage, Excel 2010 (through PowerPivot) and SharePoint 2010 with many other applications (such as Microsoft Mediaroom) in the works.

In addition to MSFT technologies, we’ve also already seen interest in the community to grow the OData ecosystem. For example, IBM’s WebSphere eXtreme Scale project implements the protocol to enable broad accessibility to their grid technologies.

We’re actively working to provide more public facing data and tools for OData. For now, be sure to check out www.odata.org. As we progress we’ll be sure to post our thoughts and ideas for your feedback. We look forward to your feedback and participation in this effort.

-Mike Flasko

Lead Program Manager, Microsoft

Simplifying our n-tier development platform: making 3 things 1 thing

dpblogs — Tue, 17 Nov 2009 09:39:00 GMT

As you’ve probably observed, we’ve been working hard over the past year or so to grow our application stacks to better support the types of applications (Silverlight, rich desktop, AJAX, etc) and services (SOAP, REST, etc) that are required to build modern, robust solutions. At present, a few of the technologies we have to help in building services & n-tier applications are: Windows Communication Foundation, .NET RIA Services and ADO.NET Data Services.

We’ve been very pleased to see each of these stacks be well received in the community and, given that positive feedback, we’ve been eagerly working on expanding each based on your comments. While today these stacks target different application scenarios and/or levels of abstraction, we see opportunities to align their foundations by building the concepts shared in each stack (authentication, conventions for business logic, logging, configuration, etc) on a single foundation. Additionally, we’ve heard your feedback that traversing our offerings in this space is, at times, too complicated.

So, with the goal of simplifying our platform by aligning common components, we’d like to announce a few changes we’ll be making to achieve our goals….

Windows Communication Foundation (WCF) == your “one stop shop” for building services and n-tier applications

Since .NET Fx 3.0, WCF has been the place to go to in the .NET Framework to rapidly build service-oriented applications that communicate across the web and the enterprise. As we’ve developed the product roadmaps for .NET RIA Services and ADO.NET Data Services we’ve found they complement the core WCF stack quite well as components/extensions for WCF or as new top-level layers of abstraction.

To formalize our direction in aligning these technologies, we’re making a few name changes in the .NET Framework 4 timeframe. ADO.NET Data Services will change its name slightly to be WCF Data Services and .NET RIA Services will be known as WCF RIA Services. We’ll be talking about our alignment of these technologies starting at this PDC, so if you are attending, keep an eye out for sessions and information at the event.

We think of these name changes as a key first step in simplifying our offerings in this space. Starting at this PDC and continuing over the coming Silverlight & .NET Framework releases cycles, you’ll see us further bring together these applications stacks such that you can leverage key parts of each in one WCF-based application. As we progress along this path we’ll be sure to post our thinking to get your feedback.

Q&A

We thought this might generate a few questions, so here’s a couple Q&As on the topic that should help clarify this announcement…

Q: Now that the names are aligned, when will alignment occur in the products?

A: We’ll start aligning the technologies in the .NET Framework 4 and Silverlight 4 timeframes and, guided by your feedback, continue through subsequent release cycles as appropriate.

Q: Doesn’t ADO.NET Data Services and .NET RIA Services already use WCF?

A: They do, but we believe we can further their alignment & integration to provide a more seamless developer experience.

Q: Does this mean you are changing the direction of Data Services?

A: The vision we have for Data Services does not change with this announcement. We believe this announcement further solidifies our investment in the area of simple, standards-based communication on the web by (overtime) bringing support for the Data Services conventions directly into WCF. For further details, see this post on our future direction regarding data services and OData support: http://blogs.msdn.com/astoriateam/archive/2009/11/17/breaking-down-data-silos-the-open-data-protocol-odata.aspx

Q: How does this announcement affect the planned Data Services update for .NET Fx 3.5 SP1?

A: It doesn’t. The Data Services update for .NET Fx 3.5 SP1 will ship as planned this calendar year.

-Mike Flasko

Lead Program Manager, ~~ADO.NET~~ WCF Data Services

Enabling X-Domain access to your Data Services

dpblogs — Fri, 06 Nov 2009 06:43:08 GMT

In the Astoria V1.5 CTP2 release, we introduced support for X-Domain and Out Of Browser access to Data Services in our Silverlight client library.

This blog post talks about how to enable Cross-Domain access to Data Services from the Silverlight client library.

I am a Service author, How do I enable X-Domain access to Data Services I host in my applications?

You would need to deploy a ClientAccessPolicy.xml file to enable X-Domain and Out Of Browser access to Data Services hosted by your applications.

The CAP.xml file should be deployed at the root of the Web Server hosting the Data Services and should not require authentication to access the file.

What is a ClientAccessPolicy.xml file?

A client access policy file defines the Services that client Silverlight applications can call
and the capabilities that these service calls can have.
Here is a sample client access policy.xml file :

<?xml version="1.0" encoding="utf-8"?>
<access-policy>
<cross-domain-access>
    <policy>
      <allow-from http-request-headers="*">
        <domain uri="http://ui.norhtwind.com"/>
      </allow-from>
      <grant-to>
        <resource path="/Northwind.svc" include-subpaths="true"/>
      </grant-to>
    </policy>
</cross-domain-access>
</access-policy>

Anatomy of a ClientAccessPolicy.xml file:

The CAP.xml file contains a lot of sections to allow fine tuning of client access.
For a Data Service author, the following are relevant sections of the file:

1. <allow-from>

Defines all the domains that are affected by this policy.

Defines the sites that are allowed to access resources in a certain policy.

By using the allow-from element for a particular policy, you implicitly deny access for all non-listed domains.

If allow-from is empty, then the policy file grants access to no sites.

The “http-request-headers” attribute of this node specifies the HTTP Request headers that client Service calls are allowed to pass to the Service.

For a Data Services Silverlight client application, the following headers are required

X-Http-Method
DataServiceVersion
Accept
Content-Type
Location
MimeType
MaxDataServiceVersion

2. <allow-from><domain/></allow-from>

The “uri” attribute of the “domain” node specifies the domain in which the client accessing the Service has to be hosted
to be able to call the Service.

3. <grant-to>

The “<grant-to>” node specifies the Services which are allowed to be accessed by clients.

4. <grant-to><resource/></grant-to>

Each service , access to which is governed by the CAP.xml file, is represented by a <resource> node in the <grant-to> section of the CAP.xml file .

The <resource> node contains two important attributes.

path : this attribute is the relative path of the Data Service
include-subpaths : this attribute specifies if paths based off of the resource path are allowed. This value should be set to true for a Data Service.

Additional references:

Using Data Services over SharePoint 2010 – Part 2 - CRUD

dpblogs — Thu, 29 Oct 2009 21:23:00 GMT

In part 1 you learned how to get SharePoint 2010 and Astoria working together, how to add a Service Reference to your Client Application and how to do a basic insert.

In this installment you will see full CRUD, so you’ll learn how to do queries, data binding, updates and deletes.

The Problem:

In part 1 we created a ‘Make a Suggestion’ feature for a department Line of Business application, by storing suggestions in SharePoint 2010.

Now we need a little Silverlight application for reviewing all those suggestions and making comments on them. So we need to display interesting suggestions, to allow the user to provide a comment, or make edits and deletes.

The Solution:

Step 1: Displaying suggestions

Assuming you’ve set-up your service reference (see part 1) the next step is to create XAML to display the Suggestions:

This XAML code, displays a ListBox (of suggestions) and an series of textboxes for editing the currently selected suggestion.

When running it should looks something like this:

To make this work we have a very simple class to encapsulate Data Services and the Data:

public class SuggestionContext
{
    TeamSiteDataContext _ctx;
    DataServiceCollection<SuggestionsItem> _suggestions;
    public TeamSiteDataContext DataContext
    {
        get
        {
            if (_ctx == null)
            {
                _ctx = new TeamSiteDataContext(
                    new Uri(
http://mflasko-dev/_vti_bin/listdata.svc,
                        UriKind.Absolute
                    )
                 );
            }
            return _ctx;
        }
    }
    public DataServiceCollection<SuggestionsItem> TrackedSuggestions
    {
        get
        {
            if (_suggestions == null)
                _suggestions = DataServiceCollection.CreateTracked<SuggestionsItem>(DataContext);
            return _suggestions;
        }
    }
}

This class has two properties, both evaluated on first access.

One gets the DataServiceContext and the other gets a DataServiceCollection which will hold, and track changes to, a collection of Suggestions.

The SuggestionContext is then exposed as a property of our Silverlight UserControl:

private SuggestionContext _currentContext;
public SuggestionContext CurrentContext {
    get {
        if (_currentContext == null)
            _currentContext = new SuggestionContext();
         return _currentContext;
    }
}

Next in the constructor of our UserControl we use this as the DataContext for our layout root, we call GetSuggestionsWithNoComments() to retrieve the interesting suggestions, and finally we hookup some event handlers for our buttons, so we can add a new suggestion and save any changes we make back to the database.

public MainPage()
{
    InitializeComponent();
    GetSuggestionsWithNoComments();
    LayoutRoot.DataContext = CurrentContext;
    this.btnSaveChanges.Click += new RoutedEventHandler(SaveChanges_Click);
    this.btnNewSuggestion.Click += new RoutedEventHandler(AddNewSuggestion);
}

The GetSuggestionsWithNoComments() method looks like this:

public void GetSuggestionsWithNoComments()
{
    var query = (
           from suggestion in CurrentContext.DataContext.Suggestions
           where suggestion.Comments == null
           select suggestion
    ) as DataServiceQuery<SuggestionsItem>;

    query.BeginExecute(
        (IAsyncResult asyncResult) => Dispatcher.BeginInvoke(() =>
        {
            DataServiceQuery<SuggestionsItem> queryResults = asyncResult.AsyncState as DataServiceQuery<SuggestionsItem>;
            if (queryResults != null)
            {
                CurrentContext.TrackedSuggestions.Clear(true);
                CurrentContext.TrackedSuggestions.Load(
                     queryResults.EndExecute(asyncResult)
                );
            }
        })
        , query
     );
}

It executes an asynchronous query to retrieve the interesting suggestions and load them into the CurrentContext.TrackedSuggestions collection. Which in turn causes the ListBox to fill, because of the data binding we’ve set up.

Step 2: Inserting, Editing and Deleting suggestions

Now in our XAML we’ve already setup two way data binding from our textboxes back to the suggestion selected in the ListBox, so as we make changes in the TextBoxes the Suggestion is modified, and those changes are tracked by the DataServiceCollection.

So that covers Editing.

Insert is handled by the ‘New Suggestion’ button. The event handler looks like this:

void AddNewSuggestion(object sender, RoutedEventArgs e)
{
    CurrentContext.TrackedSuggestions.Add(
        new SuggestionsItem {
            Title = "{Enter a title}",
            Description = "",
            Modified = DateTime.Now,
            Created = DateTime.Now
        }
    );
    ListBox.SelectedIndex = CurrentContext.TrackedSuggestions.Count - 1;
}

As you can see this creates a new Suggestion, adds it to the DataServiceCollection, and then tells the ListBox to select the new Suggestion so it can be modified via our TextBoxes.

In our XAML we added a Delete button to each ListBoxItem, this event is bound to those buttons:

private void DeleteSuggestion(object sender, RoutedEventArgs e)
{
    Button b = sender as Button;
    SuggestionsItem toDelete = b.Tag as SuggestionsItem;
    CurrentContext.TrackedSuggestions.Remove(toDelete);
}

This relies on the fact that when during DataBinding we put the Suggestion in the button’s Tag property, so all we do when the button is clicked, is retrieve the current Suggestion and remove the suggestion from the TrackedSuggestions collection. This is interpreted as a delete.

Step 3: Submitting your changes

Because we are using a DataServiceCollection, every Insert, Edit and Delete is automatically tracked for us.

But as yet they haven’t been committed to the server.

To do that you have to click the “Save Changes” button, which calls this code:

void SaveChanges_Click(object sender, RoutedEventArgs e)
{
    CurrentContext.DataContext.BeginSaveChanges(
        (IAsyncResult asyncResult) =>
            Dispatcher.BeginInvoke(
                () =>
                {
                    CurrentContext.DataContext.EndSaveChanges(asyncResult);
                    GetSuggestionsWithNoComments();
                }
            )
        , CurrentContext
    );
}

As per usual the request is invoked asynchronously, in this case by calling BeginSaveChanges.

One thing to note is that once we’ve actually saved the changes we refresh the list of Suggestions by calling GetSuggestionsWithNoComments() again.

The idea here is that once we’ve edited some suggestions they may not longer be interesting for us, or there maybe some new suggestions on the server.

Summary:

As you can see it is really easy to create a Silverlight application that uses Data Services to query and manipulate list data.

In Part 3 we will cover how to interact with blobs (i.e. Documents, Images etc), and look out for an upcoming blog post covering X-Domain Policy which is important if your Silverlight application isn’t hosted on the same web-server as Sharepoint.

As always don’t hesitate to leave a comment.

Alex James
Program Manager Microsoft

Using Data Services over SharePoint 2010 – Part 1 – Getting Started

dpblogs — Wed, 21 Oct 2009 18:21:00 GMT

ADO.NET Data Services 1.5 and SharePoint 2010 allow developers to write applications against SharePoint Lists using the familiar Data Services Client programming model.

Setup

The SharePoint 2010 beta will become generally available in November.

In order to use Data Services with SharePoint 2010 Beta, ADO.NET Data Services 1.5 must be installed on your server, ideally *before* you install SharePoint 2010.

If however SharePoint 2010 is already on your box when you install Data Services, you will need an iisreset to make the Data Service Endpoint show-up.

And then to program against a Data Services enabled SharePoint box you need either:

VS 2008 SP1 with the ADO.NET Data Services 1.5 installed (CTP 2 or higher)

or
VS 2010 (Beta 2 or higher)

Where is your Data Service Endpoint?

Assuming you’ve got everything setup and working the first thing you need to know is where your Data Service endpoint is installed.

If your SharePoint site is http://mflasko-dev/ (thanks Mike) then your Data Services endpoint will be http://mflasko-dev/_vti_bin/listdata.svc.

If you open one of these endpoints up in Internet Explorer you’ll see something like this:

As you can see this is a standard Data Services Atom feed, listing all the Lists on the SharePoint site.

Cool.

Once you know where your Data Service is located, the next step is to write your Data Services Client application.

Create the Sharepoint Application:

But first to demonstrate how easy this is, I’ve used SharePoint to create the simplest of all Suggestion Tracking applications.

You can follow along by following these simple steps:

Create a ‘Suggestions’ list which is based on an ‘Issues’ list.
Finished!

The resulting list looks something like this:

The idea is that this list will capture all the suggestions users have for making a departmental line of business (LOB) application better.

But rather than making people go to the SharePoint site to make the suggestion, we want to allow the users to do it right in the LOB application.

Talking to Sharepoint using Data Services:

The first step as always to add a service reference, using the location of the Data Service that exposes SharePoint data (i.e. http://mflasko-dev/_vti_bin/listdata.svc)

Once you’ve done this you will have CLR classes representing the types in each list, and you will have a strongly typed DataServiceContext called TeamSiteDataContext that allows you to access the data:

Rather than focusing on how the user enters the suggest in your LOB application, lets focus on how to POST that suggestion to Sharepoint.

To do that you need to write a method something like this:

public void PostSuggestion(string title, string description)
{
   TeamSiteDataContext ctx = new TeamSiteDataContext(
        new Uri(
            http://mflasko-dev/_vti_bin/listdata.svc,
            UriKind.Absolute
        )
   );
   ctx.AddToSuggestions(
      new SuggestionsItem{
          Title = title,
          Description = description
      }
   );
   ctx.SaveChanges();
}

As you can see this is very easy.

In an environment like Silverlight it is a little more complicated because whenever you cross a network boundary have to deal with the Async and Threading issues. Which means you can’t use SaveChanges() directly.

You have to use the standard BeginXXX() and EndXXX() async pattern and you need to use the Dispatcher to ensure the results are marshaled back on the UI thread:

ctx.BeginSaveChanges(
      (IAsyncResult result) => Dispatcher.BeginInvoke(
          () => ctx.EndSaveChanges(result)),
       ctx
   );

In future blog posts we will dig into this in more detail, covering topic like Query / Update / Data Binding, Blob Handling and security policies over Sharepoint.

As always we are keen to hear your comments.

Alex James
Program Manager
Microsoft

Share your data across data sources (Sharepoint, SQL Server, Azure, Reporting Services, etc) & applications (.NET, Silverlight, Excel, etc) using Data Services

dpblogs — Tue, 20 Oct 2009 03:32:00 GMT

This week at the SharePoint conference, Pablo Castro will be announcing that the next version of SharePoint natively integrates with data services such that all SharePoint lists will be exposed as an ADO.NET Data Service. This means you can now program against SharePoint list data using all the VS tools and runtime client libraries (.NET, Silverlight, AJAX, Java, PHP, etc…) which support ADO.NET Data Services. Also, since the Data Service interface is just plain HTTP and XML/Atom or JSON, pretty much any environment with an HTTP stack can now browse, update and interact with list and document data in SharePoint. We are very excited to be able to talk about SharePoint’s data service integration (more code-centric blogs to come on this topic) as this has been an ask we hear a lot each time we present data services.

This integration is also a good example of a step towards our larger vision for data services and, more generally, open data access on the web. The remainder of this post outlines our roadmap around data services and what parts of that roadmap we are announcing this week.

First, a bit of context…

Successful applications collect lots of valuable data. Often it is relatively easy to get data into applications but hard to get it out or to have multiple applications to use one another’s data. Part of the problem is lack of a simple, uniform way of exchanging data across systems. Server side applications such as SharePoint server are examples where it would be extremely valuable to have a general purpose data service interface. Another example is SQL Server Reporting Services, as reports frequently become the official entry point for application data. Excel and the analysis services clients in SQL Server are examples of client side applications that would greatly benefit from having readily access to data managed by applications of all kinds, from SharePoint sites to Reporting Services reports in existing applications to custom applications that expose data as services. Client mash-ups also consolidate data from multiple sources to provide valuable cross-application perspective, provided that they can actually access data from all systems.

Background on the approach

Services, instead of APIs, is what is becoming common ground for interaction across tiers or across applications, both on the web and inside organizations. Microsoft has been working on data-centric service interfaces that enable simple, effective data sharing within and across applications. The first materialization of this effort is the ADO.NET Data Services framework that shipped last year as part of .NET 3.5 SP1. The approach taken is to use a RESTful data service interface built on top of standard, broadly available concepts and technologies such as HTTP and AtomPub. Conventions are defined on top such that things such as metadata discovery is possible

What we are announcing

A number of products and developer tools are all now aligned around this single RESTful data service interface. More specifically:

Developer tools:

We updated the protocol itself as well as our frameworks and developer tools in Visual Studio 2010, .NET 4.0 (starting with Beta 2) and Silverlight 4. We’ll also release an update to .NET 3.5 SP1.

SharePoint:

Starting with SharePoint 2010 (beta release to come in November) all SharePoint sites are automatically exposed as RESTful data services that follow the ADO.NET Data Services convention. Now any client with an HTTP stack can read and write to SharePoint by just using simple HTTP methods and Atom (XML) or JSON formatted data. All the rich clients that work with Data Services will work with SharePoint out of the box.

Reporting Services:

Starting with SQL Server 2008 R2, all reports created with SQL Server Reporting Services will expose an option to render as an Atom feed that follows the Data Services convention as well. This makes it possible for developers and information workers to see the data “underneath” a report (whether it displays as tables, charts, etc.) and use it in their own applications.

Excel and Microsoft SQL Server PowerPivot for Excel 2010 (aka “Project Gemini”):

Also starting with SQL Server 2008 R2, the PowerPivot plug-in for Excel allows information workers to bring data from all sources for analysis will natively understand and load data from any Data Service. Now information workers can obtain data from reports, SharePoint sites, public sites that expose reference data and even custom applications and load it directly into the data analysis environments.

Cross-platform clients:

in addition to the .NET, Silverlight and AJAX clients that were already available, Microsoft sponsored open-source projects to create a PHP and a Java client for Data Services. Both are already publicly available.

All these products introduce first class support for producing or consuming the Data Services protocol, enabling data sharing scenarios for both developers and information workers. They join other already-existing services using the Data Services protocol such as the Windows Azure Table Store.

Stay tuned, more to come at the PDC & TechEd conferences later this year.

Mike Flasko

ADO.NET Data Services, Lead Program Manager

Announcing the PHP Toolkit for ADO.NET Data Services

dpblogs — Fri, 21 Aug 2009 18:57:00 GMT

This morning the Microsoft Interoperability team announced the release of a new project that bridges PHP and.NET: the PHP Toolkit for ADO.NET Data Services. The toolkit makes it easier for PHP developers to connect to and take advantage of services built using ADO.NET Data Services. The PHP Toolkit for ADO.NET Data Services is an open source project funded by Microsoft and developed by Persistent Systems and is available today on Codeplex at phpdataservices.codeplex.com

You can see an overview and quick demo of the toolkit in the Channel9 video with Pablo Castro and Claudio Caldato, Senior Program Manager with the Interoperability Technical Strategy team.

For more information on the toolkit check out the Interoperability Teams blog post and phpdataservices.codeplex.com.

- Elisa Flasko
Program Manager, Data Programmability

ADO.NET Data Services v1.5 CTP1 - Row Count

dpblogs — Wed, 18 Mar 2009 23:21:22 GMT

Peter Qian, a developer on the data services team, has posted a nice write up describing the row count feature in the ADO.NET Data Services v1.5 CTP1 release we made available for download yesterday.

You can check out Peter's Row Count write up here.

Also, we have created a new forum dedicated to providing feedback on "pre-release" versions of data services such as this CTP. Please direct all questions and comments about this feature to this new forum which is available at: (http://social.msdn.microsoft.com/Forums/en-US/dataservices/threads).

We look forward to hearing your feedback.

-Mike Flasko

ADO.NET Data Services, Program Manager

Announcing ADO.NET Data Services v1.5 CTP1

dpblogs — Mon, 02 Mar 2009 02:24:00 GMT

It's been a little while since we've written about the future direction of data services, so this post is intended to layout our thinking towards our next release. The release described below isn't yet available, but should be in the coming months, so stay tuned :). The rest of this post will discuss the specifics of the release being announced and what to expect in terms of features in CTP1.

ADO.NET Data Services v1.5 CTP1 is a tech preview of the next release of ADO.NET Data Services. This release (v1.5) will target the .NET Framework 3.5 SP1 & Silverlight 2 platforms and provide new client and server side features for data service developers.

Why is v1.5 in the name?

V1 = The release of the ADO.NET Data Services runtime and client libraries included in the .NET Framework 3.5 SP1 and Silverlight 2

V1.5 (this release) = A standalone web release which includes a superset of the functionality shipped in V1.

How does this release relate to ADO.NET Data Services as shipped in the .NET Framework 3.5 SP1?

This release (v1.5 CTP1) is a standalone release and represents our in progress work towards the next version of the ADO.NET Data Service technology we shipped in the .NET Framework 3.5 SP1. Since this is a standalone release, it will NOT be an in-place update of the v1 assemblies (System.Data.Services.*.dll) in the .NET Framework 3.5 SP1. Instead, this release will coexist with v1, allowing you the choice of building applications against either release. Like v1, this release also targets the .NET Framework 3.5 SP1 so we can quickly iterate on a feature set which makes it easier to use data services in cloud services (e.g. Windows Azure Table Storage, etc) and to address the needs of web-focused applications which iterate on a relatively short cycle.

How does this release relate to Silverlight 2?

This release (v1.5 CTP1) will include an updated client library for consuming data services from Silverlight 2. The additional features included are described in the feature list below.

How does this release relate to the next release of the .NET Framework (ie. .NET Framework 4.0)?

As noted above, ADO.NET Data Services v1.5 will initially target the .NET Framework 3.5 SP1 and Silverlight 2 platforms and will likely be released prior to the .NET Framework 4.0 platform. A future version of this standalone release (v1.5) will be created to support the .NET Framework 4.0 platform.

When will v1.5 CTP1 ship?

We aren't yet ready to announce a release date, but we are in the close down phase of the feature set for CTP1, so we are hoping to have more to say here in the not too distant future :).

What's coming in CTP1?

ADO.NET Data Services v1.5 CTP1 will include the following enhancements and features:

Data Binding: The data services client library for the .NET Framework 3.5 SP1 and Silverlight2 has been extended to support two-way data binding for WPF and Silverlight based applications.

Row Count: One scenario we heard a ton of feedback on after shipping V1 of ADO.NET Data Services in the .NET Framework 3.5SP1 is the ability for the a client of a data service to determine the total number of entities in a set without having to retrieve them all. To address this need, we have extended the data services addressing scheme to allow a client to obtain this type of information without having to download all the entities in a set.

Feed Customization (aka "Web Friendly Feeds"): A common ask we have received is to provide the ability to customize how entities are mapped into the various elements of an AtomPub feed. This feature does just that by providing a data service author declarative control over how the data service runtime maps the properties of an entity (e.g. a Customer, Order, etc) to the elements of a feed.

Server Driven Paging: This one is best described by example. If you had a data service that exposes photos, you likely want to limit the total number of photos a single request to the service can retrieve because the total collection of photos may be very large. This feature allows a service author to set per collection limits on the total number of entities returned for each request. In addition to limiting the number of photos returned per request, the server provides the client a "next link" which is simply a URI specifying how to continue retrieving the rest of the entities in the collection not returned by the first request. For those familiar with AtomPub, this feature adds support for AtomPub <next ..> elements to the data service runtime. For CTP1 we'll include server side support for this feature only. Client library support will likely come in a future CTP.

Enhanced BLOB Support: This feature enhances the BLOB support provided in V1 to enable data services to stream arbitrarily large BLOBs, store binary content separate from its metadata, easily defer the loading of BLOB content when its metadata is requested, etc. For CTP1 we'll include server side support for this feature only. Client library support will likely come in a future CTP.

New "Data Service Provider" Interface for Custom Provider Writers: As the data services runtime has evolved, so has the number of ways people want to plug data into the data service framework. In V1, two methods (Entity Framework and arbitrary .NET classes) were supported to enable a data service to interact with various diverse data sources. To address another class of environments and data sources we have introduced a way to write a "custom" provider for those cases when the previous two provider models don't meet your needs.

Bug Fixes: This release builds on our existing (v1) code base and will incorporate all fixes we have made to this point.

We've already discussed some of the features above in our transparent design process notes on this blog (example: here and here). This CTP release represents our take at addressing these areas, so when the release becomes available, please give these features a spin and let us know what you think.

As we get closer to having the CTP1 bits ready to make available on the download center, we'll post additional documentation, samples, etc. as "getting started" material for the features.

-Mike Flasko

ADO.NET Data Services Program Manager

Office Business Applications & Data Services

dpblogs — Fri, 20 Feb 2009 11:57:00 GMT

Beth Massi has been busy creating a number of great posts describing how to build add-ins to Excel, Outlook, Word, etc to work with data from an ADO.NET Data Service. The articles can be found here:

ADO.NET Data Services - Building an Excel Client

OBA Part 1 - Exposing Line-of-Business Data

OBA Part 2 - Building an Outlook Client against LOB Data

OBA Part 3 - Storing and Reading Data in Word Documents

Also, Beth graciously offered to convert the code samples from the current "Using ADO.NET Data Services" whitepaper to VB as they are currently only available in C#. We're working on getting the VB snippets integrated into the document, but for now they are also available here.

-Mike Flasko

ADO.NET Data Services, Program Manager

Timeout Workaround

dpblogs — Tue, 13 Jan 2009 20:03:00 GMT

Believe it or not, but we did ship ADO.NET Data Services with a few bugs. One issue in particular is fairly nasty. However the code developers might be tempted to write to work around the bug could cause far worse problems down the road when we fix the initial bug. Hence I wanted to give a quick write up to describe the issue and give a quick sample of how to write code that will work now and in the future when we fix the issue.

The problem exists in the ADO.NET Data Services client (System.Data.Services.Client.dll) we shipped with .NET 3.5 sp1 and Silverlight 2.0 SDK. In particular, we do not throw the correct exception when a HTTP request has problems while being sent to the server. I.e. a request timeouts, there is a bad server name, etc. In other words, any problem that would normally raise a System.Net.WebException. Unfortunately, in V1 of ADO.NET Data Services a NullReferenceException is raised instead of a WebException and even worse, the exception message is lost:

DataServiceContext context =

     new DataServiceContext(

         new Uri("http://BadServer/Northwind.svc"));
// Set small timeout amount to intentionally timeout request

context.Timeout = 1;

DataServiceQuery<Customer> query =

     context.CreateQuery<Customer>("Customers");

try
{
    foreach (Customer c in query) { }
}
catch (NullReferenceException nre)
{
    // in V1, NullReferenceException is

    //thrown instead of WebException.
    // Also, no useful description
    Console.WriteLine("Something bad happened!");
}

We are planning on fixing this bug in a future version of ADO.NET Data Services client (both in .NET and Silverlight) so if your code relies exclusively on catching the NullReference exception, it will be broken when we ship the fix. Hence, to make your code robust – have your code explicitly catch both exception types:

DataServiceContext context =

    new DataServiceContext(

        new Uri("http://BadServer/Northwind.svc"));
// Set small timeout amount to intentionally timeout request

context.Timeout = 1;

DataServiceQuery<Customer> query =

    context.CreateQuery<Customer>("Customers");

try
{
    foreach (Customer c in query) { }
}
catch (WebException e)
{
    // When fix is shippped, will ADO.NET Data Services

    // will correctly throw webException
    Console.WriteLine(e.Message);
}
catch (NullReferenceException nre)
{
    // in V1, NullReferenceException is thrown

    // instead of WebException.
    // Also, no useful description
    Console.WriteLine("Something bad happened!");
}

We will update the team blog when we have a better idea of when the fix will be available. All I can say now is we are looking into trying to get it out sooner than later, but we hope to have more information on this in the near future.

Andrew Conrad

ADO.NET Data Services Development Lead.