Previous Posts to this post
 
Previous Blog Posts Link
Supporting Billions of entities/rows for Mobile – Android Series - Part 1 - Why Scale Matters http://blogs.msdn.com/b/brunoterkaly/archive/2011/09/27/supporting-billions-of-entities-rows-for-mobile-android-series-part-1-why-scale-matters.aspx
Supporting Billions of entities/rows for Mobile – Android Series -
Part 2 - What are some high level cloud offerings?
http://blogs.msdn.com/b/brunoterkaly/archive/2011/09/27/supporting-billions-of-entities-rows-for-mobile-android-series-part-2-what-are-some-high-level-cloud-offerings.aspx
Supporting Billions of entities/rows for Mobile – Android Series - Part 3–Architecture and Data Options http://blogs.msdn.com/b/brunoterkaly/archive/2011/09/28/supporting-billions-of-entities-rows-for-mobile-android-series-part-3-architecture-and-data-options.aspx
You will need to download the free Azure trial.
   
 
What this post will cover
  There are two parts:
Part 1 Setting up your hardware with software
Part 2 Writing the RESTful Service that will support Android clients
2rxvbim5
 
Part 1 - Needed Software and Hardware
 
Software

To have a valid copy of Visual Studio 2010 Professional, Premium, Ultimate ,or Test Professional, including SQL Server 2008 Express.

Preferably Windows 7 Professional, Enterprise, or Ultimate. I haven’t tested Vista or Visual Studio 2008.
The Azure SDK - link at top of blog.

I am running Windows 7 Ultimate – 64-bit

Hardware

1.6 GHz or faster non-Mac CPU
2+ GB RAM
20 GB Hard Disk

 
Part 1 - Setup instructions for the Azure SDK
  Details for this instructions can be found here:
hyperlink2  

See section: Standalone Windows Azure SDK

Instructions for installing Azure SDK can be found here


wiq54sak
 
Part 1 – Verifying your setup
 
hyperlink2  

Video that will help you verify your setup


dwuf5eic
 
Part 2 – Why REST?
  Usually the debate is whether to use REST or SOAP. Here are some images to help you understand the differences.

3vslgtj5

Some more differences:

posccdpr

Here are some other points to remember:

REST is an architecture style An architectural style is a set of constraints that can be applied when building something.
Architecture can guide you Architecture describes the features that can be used to guide the implementation of a software system.
Clients request resources from endpoints REST is an architectural style that can be used to build software in which clients (user agents) can make requests of services (endpoints).
REST is client-server REST is one way to implement a client-server architectural style—in fact, REST explicitly builds on the client-server architectural style.
Roy Fielding REST is based on a dissertation found here
hyperlink2  

Roy's Dissertation



 
Part 2 – New Cloud Project
 
Step Task
1 Create a new cloud project with Visual Studio as Administrator.
   

Running as Administrator, Select File, New Project. The project type is Windows Azure Project.

ofeb3ry3


At this point, you should be selecting the Cloud template and Windows Azure Project Visual C#. For any of this to work, you need to have installed the Azure SDK as stated above. Notice I have also provided a Name, Location, and Solution Name.

image

We are about to add a Web Role to our project. Think of the Web Role as the container for our REST-based service. We will add an .svc file to our Web Role. The .svc file will be what our Android application communicates with to retrieve data from our service. 

mfqc2rk5

Visual Studio makes adding a Web Role easy. There are several types of Web Roles. For this demo we will select ASP.NET Web Role.  Go ahead and select ASP.NET Web Role so that your screen looks like this:

pqvc1fu5

Rename your WebRole to something more appropriate. I renamed it to FastMotorcycleList_WebRole.

ddbzxcn2

Here is what Solution Explorer should look like:

kdyvrda0

When you create a new Windows Azure Project, it is actually composed of two separate projects. There are a couple of things to notice above about the Configuration Project and the RESTful Service Project:

Configuration Project This is the where you specify the runtime settings for your project when it runs. For example, you specify the number of instances you would like. Compute instance is a VM dedicated to your service. This is how you control scale - more users means you should allocate more compute instances. As with other cloud-computing platforms the primary scalability mechanism is horizontal scaling through scaling out the number of compute instances (or VMs) on which the service runs. When you increase the instance count to 2, your service will be running on 2 VMs.
REST Service Project This is the project that will contain our RESTful service. Notice that there are a lot of files that have been given to us by the project creation wizard in Visual Studio. We will add some files as well as removing files. Most of the work in this post will take place here.
 
Quick Note
  It is possible to programmatically change the instance count using external applications. Be aware that there is a REST based Service Management API that lets you change the instance count programmatically. You can download an example of this from CodePlex.

we5wnmoz
 
Part 2 – Eliminating Files We Don’t Need and Testing our Configuration of Visual Studio
  It turns out this will be an extremely simple Web Role. We will eliminate many of the files that aren’t needed. To keep this demo as simple as possible, we are not authenticating or authorizing users. Remove the Account folder and the About.aspx file. We’ll keep Default.aspx around simply to be able to run our blank project to make sure everything is setup correctly.

Before

MyImage
After

image

At this point we are ready to run our project. It is by no means finished. We should just run to make sure our everything is setup correctly. We cannot move forward until we validate that the storage and compute emulators work correctly.

Right mouse click on the solution as follows:

MyImage

Assuming you do not get compiler errors (which you shouldn’t), you are ready to run. Go to the Debug menu and select Start Debugging.

image

You should be able to see the storage and compute emulators. Before you see default.aspx pop up, the emulation environment needs to start. The emulation environment is very useful, because it allows you to run your Azure project locally, without having to deploy the application to the Azure Data centers. Typically, you can do all your testing and debugging without the hassles of deployment. There are exceptions to testing everything locally.

Windows Azure compute emulator A tool that locally emulates the environment in which hosted services run.
Windows Azure storage emulator A tool that locally emulates the services of Windows Azure storage.


MyImage
 
Understanding Emulators
  You can start the emulation environment manually – if you wish. This link explains how to start the emulation environments outside of starting Visual Studio.
hyperlink2  

Azure SDK Tools


Your running app should like this:

1bv3g3z5
 
Step 2 – Adding startup code
  We will need to write some startup code that kicks off once our RESTful service starts. This startup code needs to prepare our Azure Table store. We talked about Azure Tables in a previous post.

This startup code typically goes in global.asax.cs.

d2ctf3ir_thumb1
 
global.asax.cs
  This is your startup code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Security;
using System.Web.SessionState;

// These 3 using statements are needed for the Application_Start() method
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.StorageClient;

namespace FastMotorcycleList_WebRole
{
    public class Global : System.Web.HttpApplication
    {

        protected void Application_Start(object sender, EventArgs e)
        {
            // Grab the configuration string
            string configValue = RoleEnvironment.GetConfigurationSettingValue("DataConnectionString");
            // The account key will map to: 
            //   (1) development storage 
            //   (2) our storage account in a data center
            var account = CloudStorageAccount.Parse(configValue);
            // Create 
            CloudTableClient tableClient = account.CreateCloudTableClient();
            tableClient.CreateTableIfNotExist("lists");
        }
    }
}

 
Comment
  The code above is needed to setup the table objects that will be used to store data.

CloudTableClient is part of the Azure SDK. It provides a client for accessing the Windows Azure Table service. It allows you to create tables, delete tables, list tables, etc.

 
Step 2 – Adding a DataConnectionString
  The point here is to setup a DataConnectionString, which will be used to either direct our application to run in a Microsoft Data Center or run in our local development fabric.


Cloud-based in a Microsoft Data Center

This is a future step. We will deploy our application in a Microsoft Data Center when we are finished writing, debugging and testing our application.






Storage Emulator/Compute Emulator

When developing our application, it is possible to run in a local simulation environment. This is very convenient. It saves us the step of deploying our application to test it. We can run our web role and levage our table objects all from the comfort of our local development computer.

In the system tray you can see the Compute Emulator.
hdfvishd_thumb[1]

Start by selecting properties as seen below.

ruxtmm40_thumb

Click in a step-by-step fashion as seen below. The end result is that we want a setting called DataConnectionString that has a value of UseDevelpmentStorage=true.

eymqzlu1_thumb[1]


 
Step 2 – Adding diagnostic code
  We need to add code to WebRole.cs.

0x01c030_thumb1

DiagnosticMonitorConfiguration is part of the Azure SDK that allows you to track diagnostic information as the application runs. It represents set of data buffers for logging. You can find more here.

hyperlink22  

DiagnosticMonitorConfiguration


 
WebRole.cs
  Add the following code to WebRole.cs
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;

namespace FastMotorcycleList_WebRole
{
    public class WebRole : RoleEntryPoint
    {
        public override bool OnStart()
        {
            DiagnosticMonitorConfiguration diagnosticConfig = 
                DiagnosticMonitor.GetDefaultInitialConfiguration();
            diagnosticConfig.Directories.ScheduledTransferPeriod = 
                TimeSpan.FromMinutes(1);
            diagnosticConfig.Directories.DataSources.Add(
                AzureLocalStorageTraceListener.GetLogDirectory());
            return base.OnStart();
        }
    }
}

 
Comments
  The main task left is to add AzureLocalStorageTraceListener, which is a custom class that defines aspects of the trace file, such as:

1. The name of the trace file
2. The size of the trace file
3. The name of the container of the trace file
4. The path of the trace file
 
AzureLocalStorageTraceListener
  This is a custom class that we are adding. It provides details about the tracing information that we are tracking.

MyImage_thumb3
Call the file AzureLocalStorageTraceListener.cs.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

// These using statements need to be added
// based on the code below
using System.Diagnostics;
using System.IO;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;

namespace FastMotorcycleList_WebRole
{
    public class AzureLocalStorageTraceListener : XmlWriterTraceListener
    {
        public AzureLocalStorageTraceListener()
            : base(Path.Combine(AzureLocalStorageTraceListener.GetLogDirectory().Path, 
                "FastMotorcycleList.Web.svclog"))
        {
        }

        public static DirectoryConfiguration GetLogDirectory()
        {
            DirectoryConfiguration directory = new DirectoryConfiguration();
            directory.Container = "mobile-tracefiles";
            directory.DirectoryQuotaInMB = 10;
            directory.Path = RoleEnvironment.GetLocalResource("FastMotorcycleList.Web.svclog").RootPath;
            return directory;
        }
    }

}

 
Step 2 – Specifying local storage for FastMotorcycleList.Web.svclog
  You will need to setup Project Properties to finish this task.

gzqwu0nj_thumb1

You will need to click on Local Storage, followed by Add Local Storage. Next you will enter:

Filename FastMotorcyleList.Web.svclog
Size 10

Here is what the finished work looks like:

z1w5mivl_thumb1

 
Comments
  1. Used by code in WebRole.cs
2. Sets up our trace file to help us with diagnostics
3. The name of the trace file is FastMotorcycleList.Web.svclog
4. Inherits from XMLWriteTraceListener to format the trace entires in XML
 
Step 2 - Review
  This is just a quick recap.

srqfeuf2_thumb

Now we are ready to start the main RESTful service aspect of our project. This next section is very important.
 
Step 2 – Adding our RESTful service
  We will now add the code that allows Android clients (or any client for that matter) to connect and perform CRUD operations (create, read, update, delete) against our table objects.

To start things off, let’s add a data provider,which is a class that abstracts away the details CRUD operations and details of the data store.

The following 3 classes will be contained inside of FastMotorcycleListDataProvider.cs file.

In Solution Explorer, add a new class by right mouse clicking on FastMotorcycleList_WebRole and call it FastMotorcycleListDataProvider.cs.

MyImage_thumb1

Call the file FastMotorcycleListDataProvider.cs. Your Solution Explorer should look like this:

3aj3koj1_thumb1
 
FastMotorcycleListDataProvider.cs
  You will now paste in the code for this file (FastMotorcycleListDataProvider.cs).

There are 3 main classes here, although the class FastMotorcycleListDataProvider is the only one that is used by the RESTful service.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

// Added using statements
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;
using System.Data.Services.Client;

namespace FastMotorcycleList_WebRole
{
    // ---------------------------------------------------------
    // Purpose of Class:   Provides basic create, read, delete operations to the service.
    //                     Abstracts away details of interacting with Azure Tables.
    // Notes:              Uses FastMotorcycleList as persistant list
    public class FastMotorcycleListDataProvider
    {
        private FastMotorcycleList _list;


        // ---------------------------------------------------------
        // Purpose of Method:   Connects to Azure Table Service and retrieves data
        // Notes:               Uses web.config to read account credentials.
        //                      Developer needs to edit web.config to either point
        //                      to Azure Tables in local development environment
        //                      or up in cloud.
        public FastMotorcycleListDataProvider()
        {
            string configValue = RoleEnvironment.GetConfigurationSettingValue("DataConnectionString");
            var account = CloudStorageAccount.Parse(configValue);

            _list = new FastMotorcycleList(account.TableEndpoint.ToString(),
                                     account.Credentials);
        }

        // ---------------------------------------------------------
        // Purpose of Method:   Retrieves list from Azure Table Service
        // Notes:               Uses "owner" for lookup
        //                      Owner is part of partitionkey so retrieval is fast
        public List<string> GetItems(string owner)
        {
            var results = from entity in _list.Items
                          where entity.PartitionKey == owner
                          select entity;

            var list = new List<string>();

            foreach (var item in results)
            {
                list.Add(item.RowKey);
            }

            return list;
        }

        // ---------------------------------------------------------
        // Purpose of Method:   Adds an entity to Azure Table Service
        // Notes:               Each entity is associated with an owner
        public void AddItem(string owner, string item)
        {
            _list.AddObject("lists", new FastMotorcycleListItem(owner, item));
            _list.SaveChanges();
        }

        // ---------------------------------------------------------
        // Purpose of Method:   Deletes an entity to Azure Table Service
        // Notes:               Each entity is associated with an owner
        // ---------------------------------------------------------
        public void DeleteItem(string owner, string item)
        {
            var entity = (from i in _list.Items
                          where i.PartitionKey == owner
                          && i.RowKey == item
                          select i).Single();

            _list.DeleteObject(entity);
            _list.SaveChanges();
        }
    }


    // ---------------------------------------------------------
    // Purpose of Class:   Acts as the blueprint for an entity
    // Notes:              The properties you add here define the
    //                     name value pairs that make up the table object
    public class FastMotorcycleListItem : TableServiceEntity
    {
        // ---------------------------------------------------------
        // Purpose of Method:   Default constructor
        public FastMotorcycleListItem()
        {
        }

        // ---------------------------------------------------------
        // Purpose of Method:   Represents the entity of 2 columns 
        //                      (owner and list item)
        // Notes:               No other properties defined because 
        //                      only a two column table
        public FastMotorcycleListItem(string partitionKey, string rowKey)
            : base(partitionKey, rowKey)
        {
        }
    }

    // ---------------------------------------------------------
    // Purpose of Class:   Abstracts away the fact that our Azure table
    //                     is in the cloud. TableServiceContext allows
    //                     us to code the entity as though it were
    //                     local
    public class FastMotorcycleList : TableServiceContext
    {
        // ---------------------------------------------------------
        // Purpose of Method:   Connect to Azure Table Service
        //                      using storage credentials
        public FastMotorcycleList(string baseAddress,
                            StorageCredentials storageCredentials)
            : base(baseAddress, storageCredentials)
        {
        }

        // ---------------------------------------------------------
        // Purpose of Method:   Our list of entities is called "lists"
        public DataServiceQuery<FastMotorcycleListItem> Items
        {
            get
            {
                return this.CreateQuery<FastMotorcycleListItem>("lists");
            }
        }
    }

}
 
Comments
  1. There are 3 classes but only FastMotorcycleListDataProvider is used outside this class.
2. Together these 3 classes allow us to communicate with the Azure Table Service.
3. Row key and partition key is very important and determines how your data gets scaled across multiple storage nodes. Read the note below.
 
Partition Key and Row Key
  This is a very important concept. This is an important read for those wanting more details:

0xyzge01_thumb1
The important points relate to partitionkey and rowkey.

gpcwhpr2_thumb

Notice that there are 2 partitions. Each partition is defined by the partitionkey and rowkey. Partition 1 and Partition 2 are 2 separate storage nodes, on separate physical machines.

The storage system achieves good scalability by distributing the partitions across many storage nodes.

The system monitors the usage patterns of the partitions, and automatically balances these partitions across all the storage nodes.


 
Step 2 – Explanation of the 3 classes
  Here are the 3 main classes inside of FastMotorcycleListDataProvider.
 
Class Subclasses Description
class FastMotorcycleListDataProvider none This is the main class used by the rest service to perform REST operations. Contains:
1. GetItems()
2. AddItem()
3. DeleteItem()
class FastMotorcycleListItem : TableServiceEntity TableServiceEntity Represents one item in the list of items tracked in the Azure tables.

Defines the partition and row key and any other data elements we wish to persist inside of our table objects.

TableServiceEntity represents a set of public properties among which must be the PartitionKey and RowKey, which are the primary key of the Azure Table.

Read more here.
class FastMotorcycleList : TableServiceContext TableServiceContext

Represents an Azure table, composed of multiple FastMotorCycleListItems.

At a lower level, represents a DataServiceContext object and allows the developer to use object oriented techniques to communicate with a table inside the Windows Azure Table service.

DataServiceContext to serve as a data access context class.

DataServiceContext supports functionality such as AddObject() to add an entity, UpdateObject() to update an entity, SaveChanges() to save changes to an entity, and CreateQuery<T> to create a query against the dataset.

The core functionality is supported in both synchronous and asynchronous versions.

DataServiceContext also supports batch operations.

     

Methods for FastMotorcycleListDataProvider

public List<string> GetItems(string owner) Retrieves a list of fast motorcycles, given an owner.
public void AddItem(string owner, string item) Adds a fast motorcycle to the list.
public void DeleteItem(string owner, string item) Deletes a fast motorcycle to the list.
 
Step 2 – Adding references
  Adding a reference to System.Data.Services.Client.

tn2rflhu_thumb1

Selecting the reference.

p0sp1w1m_thumb1
 
Step 2 – Adding RESTful Code
  At this point we are ready to start the main piece of the RESTful service.

fb2okl0q_thumb[1]

The Android Application will make a all into FastMotorcycleListService.svc. From there calls will be made into the classes inside of FastMotorcycleListDataProvider.

The actual interaction with the underlying table is abstracted away in FastMotorcycleListDataProvider, which makes use of two helper classes:

FastMotorcycleList Represents our list of fast motorcycles in Azure tables
FastMotorcycleListItem Represents an individual entity of a fast motorcycle in Azure tables


lbzr3nas_thumb[1]

The next step here is to finish the RESTful Service. There are two files to edit here. The .svc file is edited by writing XML while the second file (.cs) is edited using the built-in C# code editor.


tdh5ijyf_thumb[1]

FastMotorcycleListService.svc will be edited to use a Web Service Host Factory. Using this approach a host instance is created dynamically in response to incoming messages. The managed hosting environments that support dynamic activation are Internet Information Services (IIS) and Windows Process Activation Service (WAS), which is supported in Windows Azure.

fgbpx21o_thumb[1]

 
FastMotorcycleListService.svc
  The following markup will automatically instantiate an object of type FastMotorcycleListService. It is this code that will begin to service requests from mobile devices.
<%@ ServiceHost  Debug="true" Factory="System.ServiceModel.Activation.WebServiceHostFactory"
Service="FastMotorcycleList_WebRole.FastMotorcycleListService" %>
 
Step 2 – Writing the code-behind
  Let’s turn our attention to FastMotorcycleListService.svc.cs.
t1pexdtn_thumb[1]
 
FastMotorcycleListService.svc.cs
  The following markup will automatically instantiate an object of type FastMotorcycleListService. It is this code that will begin to service requests from mobile devices.

One important thing to notice here is that the data coming back is in JSON format. You could also use XML format. Read more here:
e2qklmwt
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

using System.ServiceModel;
using System.ServiceModel.Web;


namespace FastMotorcycleList_WebRole
{
    [ServiceContract]
    // ---------------------------------------------------------
    // Purpose of Class:   The main class to service mobile clients RESTfully
    //                     
    // Notes:              Uses FastMotorcycleListDataProvider
    public class FastMotorcycleListService
    {
        // ---------------------------------------------------------
        // Purpose:  Helper class to abstract away the underlying
        // details of the Azure Table Service.
        private FastMotorcycleListDataProvider _data;

        // ---------------------------------------------------------
        // Purpose: Constructor for abstracted Azure Table Service
        public FastMotorcycleListService()
        {
            _data = new FastMotorcycleListDataProvider();
        }


        // ---------------------------------------------------------
        // Purpose:  Retrieve a list of Fast Motorcycles based on list 
        //           owner
        [OperationContract]
        [WebGet(UriTemplate = "/list/{owner}", ResponseFormat = WebMessageFormat.Json)]
        public List<string> GetItems(string owner)
        {
            return _data.GetItems(owner);
        }

        // ---------------------------------------------------------
        // Purpose:  Add a new Fast Motorcycle based on owner
        [OperationContract]
        [WebInvoke(UriTemplate = "/list/{owner}", Method = "POST",
                   RequestFormat = WebMessageFormat.Json)]
        public void AddItem(string owner, string item)
        {
            _data.AddItem(owner, item);
        }

        // ---------------------------------------------------------
        // Purpose:  Delete a Fast Motorcycle based on Owner
        [OperationContract]
        [WebInvoke(UriTemplate = "/list/{owner}/{item}", Method = "DELETE")]
        public void DeleteItem(string owner, string item)
        {
            _data.DeleteItem(owner, item);
        }

    }
}
 
Step 2 – Redefining Web.Config
  we need to change the Web.config that was generated when we first created our cloud-based project.

We will simplify this file.

We will erase the existing web.config file and replace it with new configuration code. This is essentially a configure-less configuration file. Instead, we are using Web Serivce Host Factory, as explained earlier.

Start by replace web.config with this code:
<?xml version="1.0"?>
<configuration>
  <configSections></configSections>
  <!--To collect diagnostic traces, uncomment the section below.  
  To persist the traces to storage, update the DiagnosticsConnectionString setting with your storage credentials.
  To avoid performance degradation, remember to disable tracing on production deployments.
  <system.diagnostics>     
  <sharedListeners>
  <add name="AzureLocalStorage" type="AzureShoppingList.Web.AzureLocalStorageTraceListener, AzureShoppingList.Web"/>
  </sharedListeners>
  <sources>
  <source name="System.ServiceModel" switchValue="Verbose, ActivityTracing">
  <listeners>
  <add name="AzureLocalStorage"/>
  </listeners>
  </source>
  <source name="System.ServiceModel.MessageLogging" switchValue="Verbose">
  <listeners>
  <add name="AzureLocalStorage"/>
  </listeners>
  </source>
  </sources> 
  </system.diagnostics> -->
  <system.diagnostics>
    <trace>
      <listeners>
        <add
          type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
          name="AzureDiagnostics">
          <filter type=""/>
        </add>
      </listeners>
    </trace>
  </system.diagnostics>
  <system.web>
    <compilation
      debug="true"
      targetFramework="4.0"/>
  </system.web>
  <system.webServer>
    <modules runAllManagedModulesForAllRequests="true"/>
  </system.webServer>
</configuration>

 
Step 2 – Completed
  We should now be able to compile the project successfully. The output window should be error free.
21a1mq5y_thumb[1]
 
Next Step
  Before constructing our RESTful client (either Android, iOS, Windows Phone 7), we will test with a browser. The whole point of REST is that it is built upon http.

The next blog post will explore using Fiddler to test our RESTful service.

lahxjo2d