Part 1 – Consuming a Basic Open Feed

In a previous post we discussed the Publishing capabilities of Microsoft Codename “Data Explorer”. After mashing up your data and generating interesting results, you can share your findings literally with the world at large. One of the formats available to publish data is as an OData feed, which can be used to allow programmatic access to the results.

In this post we cover a basic example of consuming a published feed in Visual Studio (VS) using a simple console application along with some pointers to set it up. We are actively working on improving the ease of access to published results in “Data Explorer” from VS and other tools, so bear with us as we iron out the wrinkles and please reach out to us if you run into any issues. Our goal is to reduce the need for function calls by adding extra user interface support, but in the meantime, for those of you who like to tinker, here are some roll-back-your-sleeves-and-get-your-hands-dirty instructions.

1. Ensuring the published mashup is set up to be consumable

Currently, in order to consume a published OData feed using VS, resources need to have primary keys and include only supported data types (more on this below). For the purposes of this example, the mashup also needs to be published as visible to everyone, including unauthenticated users (meaning, that no authentication mechanism is requested when trying to access the feed). We will cover the explanation for how to access feeds secured via a feed key in a subsequent blog post.

1.a. Verifying the mashup resources can be externally consumed as a feed

First and foremost, all published resources that are to be consumed by VS need to have a primary key. You can ensure that a given table has a primary key by using the Table.AddKey() function. We are working on an “add key” button to the user interface to facilitate this, but in the meantime you can manually add a primary key using the formula bar.

For example, let’s start with a very simple table of IDs, First and Last names, created by directly inputting the text into “Data Explorer” and converting to a table:

• Create a new mashup (call it, say, “Basic publish to VS”) and add data by inputting text.

• Use “To Table” after pressing “Done” to convert the free-form text into a table.

• Take the default options (using spaces as delimiters and the first row as headers) to create the table.

With a simple table in place, we can now start the process to make it consumable.

The first step is to add a primary key. To invoke the Table.AddKey() function using the formula bar, click on the fx icon next to the Edit button and select “New Task”:

In the new task, call the function on the TableFromText reference by typing it in as seen in the figure. This function call makes the “ID” column the primary key (since primary keys must be unique, remember that multiple columns can be used to create a key, thus ensuring uniqueness). Note that “TableFromText” happens to be the name of the previous task.

Once sure that the table has a primary key, there are a couple of caveats regarding the data that will be exposed in the feed:

• Currently, the set of supported data types between VS and “Data Explorer” is different, so in order to ensure VS consumption, the data published in “Data Explorer” (which supports a larger set of data types) needs to be adjusted to be consumed in VS. Among the data types not yet recognized in VS are Time, Duration, DateTimeOffset, and Geospatial. Attempting to import a feed with these data types will cause the entire VS import operation to fail, so make sure that the feed you wish to import into VS does not have any of these data types exposed.
• Similarly, an exposed service operation that returns a keyed-table type will import correctly into VS, but the service operations will be ignored.
• Exposed service operations that return tables without keys will fail the VS import, though, as will service operations that have parameters.
• Lastly, for now tables with navigation properties will also fail the VS import, but we are actively working on a fix that should address this in the near future.

With the above in mind, and with a mashup that can be consumed by VS via feed, we are ready to publish and consume the data.

1.b. Publishing the mashup

When publishing the mashup, for this example we will make the published results visible to everyone, including unauthenticated users, to allow programmatic access (we will cover the more complex case of consuming authenticated feeds in a future post).

Click on Publish and follow the link to access the publish page. From there, copy the full web address under “Data Feed” for use in programmatic access.

This link to the feed is what will allow VS to access the data.

2. Consuming a feed in Visual Studio

Once the feed has been created in a consumable format, there are a few things to be aware of when programmatically accessing the feed via VS.

2.a. Installing the Microsoft WCF Data Services October 2011 CTP

In order to successfully add a service reference using the steps outlined below, be sure to install the community technical preview that included the updated version of the WCF Data Services library; click here for the Download Center page.

2.b. Adding a service reference in Visual Studio

In your VS project, right-click on References or on Service References to “Add Service Reference”.

Enter the feed URL from the Data Explorer publish page into the Address box and click Go. Make sure the URL you enter matches the published feed URL exactly (for example, verify there are no trailing spaces). You should be able to see the published resources and click OK to add the service.

2.c. Using the service reference in a console application

Here is a sample console program to consume the above feed:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ConsumingDEpublishedDataInVSPart1

{

    class Program

    {

        static void Main(string[] args)

        {

            // Note: Make sure you replace the URI 

            // with the feed URL used when adding a service reference.

            EntityCatalog catalog = new EntityCatalog(new 

            Uri("https://ws32842467.dataexplorer.sqlazurelabs.com/

 Published/Basic%20publish%20to%20VS%20test/Feed"));

            foreach (EntityType0 element in catalog.Text1) 

            {   //Note: ‘Text1’ is the name of your mashup resource

                Console.WriteLine(element.ID + " " + element.FirstName 

                                                 + " " + element.LastName + " ");

            }

            Console.WriteLine("Press any key to continue...");

            Console.ReadKey();

        }

    }

}

As you can see in the figure below, the two “EntityCatalog” references are unresolved (and there are a couple of other interesting service-specific items as well). Note that the resource name in our mashup (“Text1”) is used when accessing the elements in the feed. Note also that the column names (“ID”, “FirstName”, and “LastName”) are also used when accessing each element.

Before this code can compile, we need to resolve the “EntityCatalog” references via a “using” statement.

This resolves both the “EntityCatalog” and “EntityType0” references because they are names generated by the “Data Explorer” feed.

As a quick aside, those names and their relationship to the mashup resources can be seen in the $metadata description of the “Data Explorer” feed.

The above code, when compiled and run, will output the data in the table from our simple mashup.

This pattern can of course be extended to extract data from more meaningful or complex tables. The metadata for the feed (which is accessible by adding “/$metadata” to the feed URL and opening it up in a browser) can be used to tailor the queries over the exposed feed, accessing data from different resources.

We are still working hard to improve the experience of consuming “Data Explorer” feeds using VS but hopefully the above example can unlock some interesting new ways to work with your data.

What do you think? Let us know your thoughts and stay tuned as we further integrate and simplify the user interface to more easily publish and consume your data with Microsoft Codename “Data Explorer”.

3. Some troubleshooting FAQs

While support for programmatic consumption using VS is still a work in progress, following the above steps should allow you to create a service reference and pull data from your feed. Here are some common issues with setting the reference up; please have a look to see if they help, or ask a question in our forum with more specific issues you find.

3.a. I encounter an Add Service Reference Error. What’s up with that?

Depending on the specific details, this Add Service Reference Error may mean that an incorrect feed URL was used. For instance, one common mistake may be to use the link to the “Data Explorer” publish page rather than the full feed URL (which is the publish page link plus “/Feed”). Double check that the address you use is the full feed link that is presented in the publish page, and not the publish page’s URL.

3.b. Why does the service require me to enter a user name and password?

This is most likely due to having selected “Make visible to workspace consumers only” instead of “Make visible to everyone, including unauthenticated users” when publishing the mashup. For this example, try going back to “Data Explorer”, and republish by making visible to everyone. We will explain how to consume feeds protected by a feed key in a future blog post.

3.c. Why do I get an “Unrecognized version 3.0” error?

This error crops up if the WCF Data Services October 2011 CTP is not installed. Please see above and verify that this CTP is installed.

4. Resources used in this post

* The publicly available publish page for the basic feed used in this post can be found at https://ws32842467.dataexplorer.sqlazurelabs.com/Published/Basic%20publish%20to%20VS%20test

* The feed URL used for this post is available at https://ws32842467.dataexplorer.sqlazurelabs.com/Published/Basic%20publish%20to%20VS%20test/Feed

* The sample VS project used for this post can be downloaded here.