“Cloud Numerics” Example: Analyzing Demographics Data from Windows Azure Marketplace

Imagine your sales business is ‘booming’ in cities X, Y, and Z, and you are looking to expand. Given that demographics provide regional indicators of sales potential, how do you find other cities with similar demographics? How do you sift though large sets of data to identify new cities and new expansion opportunities?

In this application example, we use “Cloud Numerics” to analyze demographics data such as average household size and median age between different postal (ZIP) codes in the United States.

This blog post demonstrates the following process for analyzing demographics data:

We go through the steps of subscribing to the dataset in the Windows Azure Marketplace.
We create a Cloud Numerics C# application for reading in data and computing correlation between different ZIP codes. We partition the application into two projects: one for reading in the data, and another one for performing the computation.
Finally, we step through the writing of results to Windows Azure blob storage.

Before You Begin

Create a Windows Live ID for accessing Windows Azure Marketplace
Create a Windows Azure subscription for deploying and running the application (if do not have one already)
Install “Cloud Numerics” on your local computer where you develop your C# applications
Run the example application as detailed in the “Cloud Numerics” Lab Getting Started wiki page to validate your installation

Because of the memory usage, do not attempt to run this application on your local development machine. Additionally, you will need to run it on at least two compute nodes on your Windows Azure cluster.

Note!

You specify how many compute nodes are allocated when you use the Cloud Numerics Deployment Utility to configure your Windows Azure cluster. For details, see this section in the Getting Started guide.

Step 1: Subscribe to the Demographics Dataset at Windows Azure Marketplace

For this demonstration, we use the dataset “2010 Key US Demographics by ZIP Code, Place and County (Trial)”

To subscribe to the dataset:

Sign up to Windows Azure Marketplace using your Windows Live ID at: https://datamarket.azure.com/
Navigate to the page for the dataset: https://datamarket.azure.com/dataset/c7154924-7cab-47ac-97fb-7671376ff656
Select Sign Up. Then click the Details tab of the dataset, and take note of the Service Root URI https://api.datamarket.azure.com/Esri/KeyUSDemographicsTrial/ . This URI will be used in creating the service reference in the next step.
Finally, go to My Account, Account Keys and take note of the long Value string. You will need this string in the next step to access the dataset from the C# application.

Step 2: Set Up the Cloud Numerics Project

To set up your Cloud Numerics Visual Studio project:

Start Microsoft Visual Studio and create a solution from the Cloud Numerics project template. Let’s call the solution “CloudDemographics.”
Rename the “MSCloudNumerics” project with more descriptive language. For the purposes of this blog post, we will name it “Demographics.”
Next, add a new project to the solution as a C# class library. This project will have the code for reading data. let’s call the project “DemographicsReader.”

We now need to add project dependency from Demographics to DemographicsReader.

In Solution Explorer, under the Demographics project, right-click References and select Add References.
Click the Projects tab, and browse and add DemographicsReader as a reference.
Also add a .NET reference for Microsoft.WindowsAzure.StorageClient. We’ll need it later on when it’s time to write results to blob storage.

Finally, go to the DemographicsReader project, and add references to the following managed assemblies. In the Add References window, click Browse and find the assemblies, typically found at C:\Program Files\Microsoft Numerics\v0.1\Bin

Microsoft.Numerics.DenseArrays
Microsoft.Numerics.Distributed.IO
Microsoft.Numerics.DistributedDenseArrays
Microsoft.Numerics.ManagedArrayImpl
Microsoft.Numerics.Runtime
Microsoft.Numerics.TCDistArrayHandle

Step 3: Add Service Reference to Dataset

To get data into the application, we need to add a service reference to the dataset.

In Solution Explorer, right-click the DemographicsReader project and select Add Service Reference.
In the dialog window, enter https://api.datamarket.azure.com/Esri/KeyUSDemographicsTrial/ as the address. Also, specify “USDemographics” as the namespace. Adding the reference provides auto-generated flexible query methods that reside in the DemographicsReader.USDemographics namespace and allow us to access the dataset.

Note!

For an example of how to use a flexible query, see the MSDN Library topic titled Create a Flexible Query Application.

3. Add the following “using” statements to the DemographicsReader project’s Class1.cs file, and declare a namespace “DemographicsReader” that will contain the classes for accessing the demographics data.

using System;         
using System.Collections.Generic;         
using System.Linq;         
using msnl = Microsoft.Numerics.Local;         
using DemographicsReader.USDemographics;         
using System.Net;         
        
namespace DemographicsReader         
{         
        
}

4. Add a class named “DemographicsReference” to this namespace with the following private members and a constructor.

public class DemographicsReference<T>         
{         
    private Uri serviceURI;         
    private EsriKeyUSDemographicsTrialContainer context;         
    public DemographicsReference()         
    {         
        serviceURI = new Uri(@"https://api.datamarket.azure.com/Esri/KeyUSDemographicsTrial/");         
        context = new EsriKeyUSDemographicsTrialContainer(serviceURI);         
        // Change these to your Live ID and Azure Marketplace key         
        context.Credentials = new NetworkCredential(@"myemail@live.com",         
                                                    @"myAzureMarketplaceKey");         
    }         
}

5. Replace "myemail@live.com” with your Live ID and “myAzureMarketplaceKey” with the key from Step 1.

Step 4: Create a Serial Reader for Windows Azure Marketplace Data

Next, let’s build the LINQ query operation for reading in the data. A single transaction returns a maximum of 100 rows. However, we want to read in much more data. Therefore, we create a list where we append and consolidate the individual query results. We structure the query so that we can start from a given row, read in a block of rows, move forward, read in another block, and so forth.

The query, by default, returns objects of type demog1. This is a class that holds the demographics data as fields. However, we want to be able to select an arbitrary subset of columns. We accomplished this by supplying Func<demog1, T> selector function into the query as an argument.

Add the following method to the DemographicsReference<T> class

// Method to get demographics data         
public IList<T> GetDemographicsData(int startIndex, int blocks, int blockSize, Func<demog1, T> selector)         
{         
    // Read in blocks of data and append the results to a list            
    List<T> queryResult = new List<T>();         
    for (int i = 0; i < blocks; i++)         
    {         
        // Try reading each block few times in case reading fails                
        int retries = 0;         
        int maxRetries = 10;         
        bool success = false;         
        while (success == false)         
        {         
            try         
            {         
                // Select only geographies that have ZIP codes                        
                // Order the results by ZIP code to ensure consistent ordering                        
                // Skip to specific starting index                        
                // Finally select subset of members and convert to list                        
                queryResult.AddRange(context.demog1.         
                    Where(demographic => demographic.GeographyType == "ZIP Code").         
                    OrderBy(demographic => demographic.GeographyId).         
                    Skip(startIndex + i * blockSize).Take(blockSize).         
                    Select(selector).ToList());         
                success = true;         
            }         
            catch (Exception ex)         
            {         
                Console.WriteLine("Error: {0}", ex.Message);         
                retries++;         
                if (retries > maxRetries) { throw ex; }         
            }         
        }         
    }         
    return queryResult;         
}

Note!

To increase robustness, the method attempts to read in a block a few times in case the query fails initially.

Step 5: Parallelize Reader Using IParallelReader Interface

Now, we have the basic framework in place to read in data. Next, we implement the Cloud Numerics IParallelReader interface to read in the data to a distributed array in parallel fashion.

Add the following class “DemographicsData” to the DemographicsReader namespace.

[Serializable]         
public class DemographicsData : Microsoft.Numerics.Distributed.IO.IParallelReader<double>         
{         
    private int samples;         
    private int blockSize = 100; // Maximum number of rows from query is a block of 100 rows         
        
    public DemographicsData(int samples)         
    {         
        if (samples % blockSize != 0)         
        {         
            throw new ArgumentException("Number of samples must be a multiple of " + this.blockSize);         
        }         
        this.samples = samples;         
         
    }         
        
    // Concatenate blocks of rows to form a distributed array         
    public int DistributedDimension         
    {         
        get { return 0; }         
        set { }         
    }         
        
    // Simple helper struct to keep track of block dimensions         
    [Serializable]         
    public struct BlockInfo         
    {         
        public int startIndex;         
        public int blocks;         
    }         
        
    // Attempt to divide blocks between ranks evenly         
    public Object[] ComputeAssignment(int ranks)         
    {         
        Object[] blockInfo = new Object[ranks];         
        BlockInfo thisBlockInfo;         
        int blocks = samples / blockSize;         
        int blocksPerRank = (int)Math.Ceiling((double)blocks / (double)ranks);         
        int remainingBlocks = blocks;         
        int startIndex = 0;         
        for (int i = 0; i < ranks; i++)         
        {         
            // Compute block info         
            thisBlockInfo.startIndex = startIndex;         
            thisBlockInfo.blocks = System.Math.Max(System.Math.Min(blocksPerRank, remainingBlocks),0);         
            startIndex = startIndex + thisBlockInfo.blocks * blockSize;         
            remainingBlocks = remainingBlocks - thisBlockInfo.blocks;         
        
            // Assingn thisBlockInfo into blockInfo         
            blockInfo[i] = thisBlockInfo;         
        }         
        return blockInfo;         
    }         
        
    // Method that queries the data and converts results to numeric array         
    public msnl.NumericDenseArray<double> ReadWorker(Object blockInfoAsObject)         
    {         
        var demographicsRef = new DemographicsReference<demog1>();         
        
        // Select all data from each row         
        Func<demog1, demog1> selector = demographic => demographic;         
        
        // Get information about blocks to read         
        BlockInfo blockInfo = (BlockInfo)blockInfoAsObject;         
        int blocks = blockInfo.blocks;         
        int startIndex = blockInfo.startIndex;         
        
        long columns = 10;         
        msnl.NumericDenseArray<double> data;         
        if (blocks == 0)         
        {         
            /// If number of blocks is 0 return empty array         
            data = msnl.NumericDenseArrayFactory.Create<double>(new long[] { 0, columns });         
        }         
        else         
        {         
            // Execute query         
            var dmList = demographicsRef.GetDemographicsData(startIndex, blocks, blockSize, selector);         
            var rows = dmList.Count();         
            data = msnl.NumericDenseArrayFactory.Zeros<double>(new long[] { rows, columns });         
        
            // Copy a subset of data to array columns         
            for (int i = 0; i < rows; i++)         
            {         
                data[i, 0] = (double)dmList[i].AverageHouseholdSize2010;         
                data[i, 1] = (double)dmList[i].HouseholdGrowth2010To2015Annual;         
                data[i, 2] = (double)dmList[i].MedianAge2010;         
                data[i, 3] = (double)dmList[i].MedianHomeValue2010;         
                data[i, 4] = (double)dmList[i].PctBachelorsDegOrHigher2010;         
                data[i, 5] = (double)dmList[i].PctOwnerOccupiedHousing2010;         
                data[i, 6] = (double)dmList[i].PctRenterOccupiedHousing2010;         
                data[i, 7] = (double)dmList[i].PctVacantHousing2010;         
                data[i, 8] = (double)dmList[i].PerCapitaIncome2010;         
                data[i, 9] = (double)dmList[i].PerCapitaIncomeGrowth2010To2015Annual;         
            }         
        }         
        return data;         
    }         
}

The class constructor expects the number of dataset rows (“samples”) as input. The method “ComputeAssignment” divides the total number of rows into groups (referred to here as “blocks”) and attempts to distribute these blocks evenly between ReadWorkers. The “ReadWorker” method is executed in parallel fashion: there is one parallel call in each rank. The method expects as input a start index and a specified number of blocks to read. For example, the first worker might read rows 0-999, the second one rows 1000-1999 and so on. Then, each ReadWorker converts the demographics indicators from the query results into array columns. The arrays from each ReadWorker are then concatenated to form a large distributed array.

See the blog post titled “Cloud Numerics” Example: Using the IParallelReader Interface for more details on how to use the IParallelReader interface.

Step 6: Add Method for Getting Mapping from Rows to Geographies

Because the resulting distributed array is all numeric, we need an auxiliary list to map array rows to geographies. We accomplish this by adding a serial reader named “GetRowMapping.” We also add a struct “ZipCodeAndLocation” that holds the ZIP Code, city name, and state as well as the row index. The GetRowMapping then returns a list of such structs.

1. First, add the following struct to the top-level of the DemographicsReader namespace

// A class to contain zip code and location information, as well as an index         
public class ZipCodeAndLocation         
{         
    string _zipCode, _location, _state;         
    int _idx;         
    public string zipCode { get { return _zipCode; } }         
    public string location { get { return _location; } }         
    public string state { get { return _state; } }         
    public int idx         
    {         
        get { return _idx; }         
        set { _idx = value; }         
    }         
     
    public ZipCodeAndLocation(string zipCode, string location, string state)         
    {         
        _zipCode = zipCode;         
        _location = location;         
        _state = state;         
        _idx = 0;         
    }         
}

2. Next, add a method GetRowMapping to the DemographicsData class from the previous step.

public IList<ZipCodeAndLocation> GetRowMapping()         
{         
    var demographicsRef = new DemographicsReference<ZipCodeAndLocation>();         
        
    // select geography ID, name, state abbreviation and index         
    Func<demog1, ZipCodeAndLocation> selector =         
        demographic => new ZipCodeAndLocation(demographic.GeographyId, demographic.GeographyName, demographic.StateAbbreviation);         
        
    // Execute the query         
    var rowMapping = demographicsRef.GetDemographicsData(0, this.samples / this.blockSize, this.blockSize, selector);         
        
    // Construct mapping         
    for (int i = 0; i < rowMapping.Count; i++)         
    {         
        rowMapping[i].idx = i;         
    }         
    return rowMapping;         
}

Here we use a selector that only picks out the GeographyID (the ZIP code), GeographyName (the city name), and StateAbbreviation from the full query. Then, because the results are in the same sorted order as those from the earlier demographics query, we can use a simple for-loop to add an index.

Step 7: Compute Correlation Matrix Using Distributed Data

Next, we’ll move to the Demographics project. We add a static method that expects a distributed array as input, normalizes the rows and columns, and then computes a correlation matrix. The columns have to be normalized because otherwise they would be on vastly different scales: for comparing per capita income and median age, for example. We do this by subtracting the mean of each column, and dividing by the standard deviation. Note how the KroneckerProduct method is used to tile vectors into an array that matches the data array size. The rows are then normalized to obtain the correlation coefficients between –1 and 1.

To get the correlation matrix between all ZIP codes, we need to compute correlation coefficients between each pair of rows. An efficient way to perform this computation is to multiply the data with its own transpose. This operation is at the heart of the application. Because it produces a very large matrix (30000-by-30000), performing the computation on a distributed cluster yields benefits.

Delete the default template code from Demographics project source code, and add the following:

using System;         
using System.Collections.Generic;         
using System.Linq;         
using System.Text;         
using msnl = Microsoft.Numerics.Local;         
using msnd = Microsoft.Numerics.Distributed;         
using Microsoft.Numerics.LinearAlgebra;         
using Microsoft.Numerics.Statistics;         
using Microsoft.Numerics.Mathematics;         
using Microsoft.WindowsAzure;         
using Microsoft.WindowsAzure.StorageClient;         
using DemographicsReader;         
        
        
namespace Demographics         
{         
    public class Program         
    {         
        static msnd.NumericDenseArray<double> Correlate(msnd.NumericDenseArray<double> data)         
        {         
            // Compute correlation matrix         
        
            // Helper vectors         
            long rows = data.Shape[0];         
            long columns = data.Shape[1];         
            msnd.NumericDenseArray<double> columnOfOnes = new msnd.NumericDenseArray<double>(new long[] { rows, 1 });         
            columnOfOnes.Fill(1);         
            msnd.NumericDenseArray<double> rowOfOnes = new msnd.NumericDenseArray<double>(new long[] { 1, columns });         
            rowOfOnes.Fill(1);         
        
            // Normalize each column to be between -1 and 1         
            // Subtract mean         
            var columnMean = Descriptive.Mean(data, 0);         
            data = data - Operations.KroneckerProduct(columnOfOnes, columnMean);         
            // Divide by standard deviation         
            var columnStandardDeviation = BasicMath.Sqrt(rows * Descriptive.Mean(data * data, 0));         
            data = data / Operations.KroneckerProduct(columnOfOnes, columnStandardDeviation);         
        
            // Next normalize rows         
            var rowMean = Descriptive.Mean(data, 1);         
            data = data - Operations.KroneckerProduct(rowMean, rowOfOnes);         
            var rowStandardDeviation = BasicMath.Sqrt(columns * Descriptive.Mean(data * data, 1));         
            data = data / Operations.KroneckerProduct(rowOfOnes, rowStandardDeviation);         
        
            // Compute correlation matrix by multiplying         
            var correlationMatrix = Operations.MatrixMultiply(data, data.Transpose());         
            return correlationMatrix;         
        }         
    }         
}

In addition to the Correlate method, this code includes the necessary “using” statements for Cloud Numerics namespaces.

Step 8: Select Interesting Subset of Cities and Write the Correlations into a Blob

We store results to Windows Azure blob storage as a blob resembling a .csv formatted file. To save time and memory, we only write a small subset of the 30000-by-30000 correlation matrix to a blob. Add the following method to the Demographics Program class. Also, change “myAccountKey” and “myAccountName” to the key and name of your own storage account.This method takes a list of ZipCodeAndLocation objects we’re interested in, finds the corresponding rows and columns from the correlation matrix, and writes the results to a string. Then it creates a container “demographicsresult” and a blob, “demographicsresult.csv”, makes the blob publicly readable, and uploads the string to http://<myAccountName>.blob.core.net/demographicsresult/demographicsresult.csv .

static void WriteOutput(msnd.NumericDenseArray<double> corr, int[] rowArray, IList<ZipCodeAndLocation> rowLocation)         
{         
    var nRows = rowArray.Count();         
    StringBuilder output = new StringBuilder();         
        
    // Write header rows         
    output.Append(" , ");         
    for (int i = 0; i < nRows; i++)         
    {         
        output.Append(", " + rowLocation[rowArray[i]].location);         
    }         
    output.AppendLine();         
        
    output.Append(" , ");         
    for (int i = 0; i < nRows; i++)         
    {         
        output.Append(", ZIP " + rowLocation[rowArray[i]].zipCode);         
    }         
        
    // Write rows         
    for (int i = 0; i < nRows; i++)         
    {         
        output.AppendLine();         
        output.Append(rowLocation[rowArray[i]].location + ", ZIP " + rowLocation[rowArray[i]].zipCode);         
        for (int j = 0; j < nRows; j++)         
        {         
            output.Append(", " + corr[rowArray[i], rowArray[j]]);         
        }         
    }         
        
    // Write to blob storage         
    // Replace "myAccountKey" and "myAccountName" by your own storage account key and name         
    string accountKey = "myAccountKey";         
    string accountName = "myAccountName";         
    // Result blob and container name         
    string containerName = "demographicsresult";         
    string blobName = "demographicsresult.csv";         
        
    // Create result container and blob         
    var storageAccountCredential = new StorageCredentialsAccountAndKey(accountName, accountKey);         
    var storageAccount = new CloudStorageAccount(storageAccountCredential, true);         
    var blobClient = storageAccount.CreateCloudBlobClient();         
    var resultContainer = blobClient.GetContainerReference(containerName);         
    resultContainer.CreateIfNotExist();         
    var resultBlob = resultContainer.GetBlobReference(blobName);         
        
    // Make result blob publicly readable,         
    var resultPermissions = new BlobContainerPermissions();         
    resultPermissions.PublicAccess = BlobContainerPublicAccessType.Blob;         
    resultContainer.SetPermissions(resultPermissions);         
        
    // Upload result to blob         
    resultBlob.UploadText(output.ToString());         
}

Step 9: Putting it All Together

Finally, we add a main method that instantiates the reader, reads in data, computes correlations, and selects the interesting subset of cities from the row mapping. For example, let’s select Cambridge, MA, Redmond, WA, and Palo Alto, CA. Add the following main method to the Demographics Program class.

static public void Main()         
{         
    // Initialize distributed runtime         
    Microsoft.Numerics.NumericsRuntime.Initialize();         
        
    // Read in demographics data from Azure marketplace for different ZIP codes         
    var demographics = new DemographicsData(30000);         
        
    var data = Microsoft.Numerics.Distributed.IO.Loader.LoadData<double>(demographics);         
        
    Console.WriteLine("Got {0} rows", data.Shape[0]);         
        
    // Compute correlation matrix         
    Console.WriteLine("Computing correlation matrix");         
    var corr = Correlate(data);         
        
    //Get mapping between ZIP codes, city names and matrix rows         
    Console.WriteLine("Selecting rows");         
    var rowLocation = demographics.GetRowMapping();         
        
    // Select some pairs of cities to demostrate result         
    var rowSelection = rowLocation.Where(row =>         
        (row.location == "Cambridge" && row.state == "MA" ) ||         
        (row.location == "Redmond" && row.state == "WA" )||         
        (row.location == "Palo Alto" && row.state == "CA" )).         
        Select(row => row.idx).ToArray();         
        
    // Write output         
    Console.WriteLine("Writing output");         
    WriteOutput(corr, rowSelection, rowLocation);         
        
    // Shut down runtime         
    Microsoft.Numerics.NumericsRuntime.Shutdown();         
}

Now, the application is ready to be deployed. Set AppConfigure as StartUp project, build and deploy using the Deployment Utility. The application will run for a few minutes. The results can be viewed, for example by opening http://<myAccountName>.blob.core.net/demographicsresult/demographicsresult.csv as a spreadsheet

Here the correlation coefficients above 0.9 between different ZIP codes are highlighted.

Source Code

For your convenience, this section of the blog post contains the entire source code for you to copy. Remember to change the “myAccountKey” and “myAccountName” strings to your storage account key and name, and change “myemail@live.com” and “myAzureMarketplaceKey” to your Live ID and key from Step 1.

Demographics Program.cs

using System;         
using System.Collections.Generic;         
using System.Linq;         
using System.Text;         
using msnl = Microsoft.Numerics.Local;         
using msnd = Microsoft.Numerics.Distributed;         
using Microsoft.Numerics.LinearAlgebra;         
using Microsoft.Numerics.Statistics;         
using Microsoft.Numerics.Mathematics;         
using Microsoft.WindowsAzure;         
using Microsoft.WindowsAzure.StorageClient;         
using DemographicsReader;         
        
        
namespace Demographics         
{         
    public class Program         
    {         
        static msnd.NumericDenseArray<double> Correlate(msnd.NumericDenseArray<double> data)         
        {         
            // Compute correlation matrix         
        
            // Helper vectors         
            long rows = data.Shape[0];         
            long columns = data.Shape[1];         
            msnd.NumericDenseArray<double> columnOfOnes = new msnd.NumericDenseArray<double>(new long[] { rows, 1 });         
            columnOfOnes.Fill(1);         
            msnd.NumericDenseArray<double> rowOfOnes = new msnd.NumericDenseArray<double>(new long[] { 1, columns });         
            rowOfOnes.Fill(1);         
        
            // Normalize each column to be between -1 and 1         
            // Subtract mean         
            var columnMean = Descriptive.Mean(data, 0);         
            data = data - Operations.KroneckerProduct(columnOfOnes, columnMean);         
            // Divide by standard deviation         
            var columnStandardDeviation = BasicMath.Sqrt(rows * Descriptive.Mean(data * data, 0));         
            data = data / Operations.KroneckerProduct(columnOfOnes, columnStandardDeviation);         
        
            // Next normalize rows         
            var rowMean = Descriptive.Mean(data, 1);         
            data = data - Operations.KroneckerProduct(rowMean, rowOfOnes);         
            var rowStandardDeviation = BasicMath.Sqrt(columns * Descriptive.Mean(data * data, 1));         
            data = data / Operations.KroneckerProduct(rowOfOnes, rowStandardDeviation);         
        
            // Compute correlation matrix by multiplying         
            var correlationMatrix = Operations.MatrixMultiply(data, data.Transpose());         
            return correlationMatrix;         
        }         
        
        // Output writer method         
        static void WriteOutput(msnd.NumericDenseArray<double> corr, int[] rowArray, IList<ZipCodeAndLocation> rowLocation)         
        {         
            var nRows = rowArray.Count();         
            StringBuilder output = new StringBuilder();         
        
            // Write header rows         
            output.Append(" , ");         
            for (int i = 0; i < nRows; i++)         
            {         
                output.Append(", " + rowLocation[rowArray[i]].location);         
            }         
            output.AppendLine();         
        
            output.Append(" , ");         
            for (int i = 0; i < nRows; i++)         
            {         
                output.Append(", ZIP " + rowLocation[rowArray[i]].zipCode);         
            }         
        
            // Write rows         
            for (int i = 0; i < nRows; i++)         
            {         
                output.AppendLine();         
                output.Append(rowLocation[rowArray[i]].location + ", ZIP " + rowLocation[rowArray[i]].zipCode);         
                for (int j = 0; j < nRows; j++)         
                {         
                    output.Append(", " + corr[rowArray[i], rowArray[j]]);         
                }         
            }         
        
            // Write to blob storage         
            // Replace "myAccountKey" and "myAccountName" by your own storage account key and name         
            string accountKey = "myAccountKey";         
            string accountName = "myAccountName";         
            // Result blob and container name         
            string containerName = "demographicsresult";         
            string blobName = "demographicsresult.csv";         
        
            // Create result container and blob         
            var storageAccountCredential = new StorageCredentialsAccountAndKey(accountName, accountKey);         
            var storageAccount = new CloudStorageAccount(storageAccountCredential, true);         
            var blobClient = storageAccount.CreateCloudBlobClient();         
            var resultContainer = blobClient.GetContainerReference(containerName);         
            resultContainer.CreateIfNotExist();         
            var resultBlob = resultContainer.GetBlobReference(blobName);         
        
            // Make result blob publicly readable,         
            var resultPermissions = new BlobContainerPermissions();         
            resultPermissions.PublicAccess = BlobContainerPublicAccessType.Blob;         
            resultContainer.SetPermissions(resultPermissions);         
        
            // Upload result to blob         
            resultBlob.UploadText(output.ToString());         
        }         
        
        static public void Main()         
        {         
            // Initialize distributed runtime         
            Microsoft.Numerics.NumericsRuntime.Initialize();         
        
            // Read in demographics data from Azure marketplace for different ZIP codes         
            var demographics = new DemographicsData(30000);         
        
            var data = Microsoft.Numerics.Distributed.IO.Loader.LoadData<double>(demographics);         
        
            Console.WriteLine("Got {0} rows", data.Shape[0]);         
        
            // Compute correlation matrix         
            Console.WriteLine("Computing correlation matrix");         
            var corr = Correlate(data);         
        
            //Get mapping between ZIP codes, city names and matrix rows         
            Console.WriteLine("Selecting rows");         
            var rowLocation = demographics.GetRowMapping();         
        
            // Select some pairs of cities to demostrate result         
            var rowSelection = rowLocation.Where(row =>         
                (row.location == "Cambridge" && row.state == "MA") ||         
                (row.location == "Redmond" && row.state == "WA") ||         
                (row.location == "Palo Alto" && row.state == "CA")).         
                Select(row => row.idx).ToArray();         
        
            // Write output         
            Console.WriteLine("Writing output");         
            WriteOutput(corr, rowSelection, rowLocation);         
        
            // Shut down runtime         
            Microsoft.Numerics.NumericsRuntime.Shutdown();         
        }         
    }         
}

DemographicsReader Class1.cs

using System;         
using System.Collections.Generic;         
using System.Linq;         
using msnl = Microsoft.Numerics.Local;         
using DemographicsReader.USDemographics;         
using System.Net;         
        
namespace DemographicsReader         
{         
    // A class to contain zip code and location information, as well as an index         
    public class ZipCodeAndLocation         
    {         
        string _zipCode, _location, _state;         
        int _idx;         
        public string zipCode { get { return _zipCode; } }         
        public string location { get { return _location; } }         
        public string state { get { return _state; } }         
        public int idx         
        {         
            get { return _idx; }         
            set { _idx = value; }         
        }         
        
        public ZipCodeAndLocation(string zipCode, string location, string state)         
        {         
            _zipCode = zipCode;         
            _location = location;         
            _state = state;         
            _idx = 0;         
        }         
    }         
        
    public class DemographicsReference<T>         
    {         
        private Uri serviceURI;         
        private EsriKeyUSDemographicsTrialContainer context;         
        public DemographicsReference()         
        {         
            serviceURI = new Uri(@"https://api.datamarket.azure.com/Esri/KeyUSDemographicsTrial/");         
            context = new EsriKeyUSDemographicsTrialContainer(serviceURI);         
            // Change these to your Live ID and Azure Marketplace key         
            context.Credentials = new NetworkCredential(@"myemail@live.com",         
                                                        @"myAzureMarketplaceKey");         
        }         
        
        // Method to get demographics data         
        public IList<T> GetDemographicsData(int startIndex, int blocks, int blockSize, Func<demog1, T> selector)         
        {         
            // Read in blocks of data and append the results to a list            
            List<T> queryResult = new List<T>();         
            for (int i = 0; i < blocks; i++)         
            {         
                // Try reading each block few times in case reading fails                
                int retries = 0;         
                int maxRetries = 10;         
                bool success = false;         
                while (success == false)         
                {         
                    try         
                    {         
                        // Select only geographies that have ZIP codes                        
                        // Order the results by ZIP code to ensure consistent ordering                        
                        // Skip to specific starting index                        
                        // Finally select subset of members and convert to list                        
                        queryResult.AddRange(context.demog1.         
                            Where(demographic => demographic.GeographyType == "ZIP Code").         
                            OrderBy(demographic => demographic.GeographyId).         
                            Skip(startIndex + i * blockSize).Take(blockSize).         
                            Select(selector).ToList());         
                        success = true;         
                    }         
                    catch (Exception ex)         
                    {         
                        Console.WriteLine("Error: {0}", ex.Message);         
                        retries++;         
                        if (retries > maxRetries) { throw ex; }         
                    }         
                }         
            }         
            return queryResult;         
        }         
    }         
        
    [Serializable]         
    public class DemographicsData : Microsoft.Numerics.Distributed.IO.IParallelReader<double>         
    {         
        private int samples;         
        private int blockSize = 100; // Maximum number of rows from query is a block of 100 rows         
        
        public DemographicsData(int samples)         
        {         
            if (samples % blockSize != 0)         
            {         
                throw new ArgumentException("Number of samples must be a multiple of " + this.blockSize);         
            }         
            this.samples = samples;         
             
        }         
        
        // Concatenate blocks of rows to form a distributed array         
        public int DistributedDimension         
        {         
            get { return 0; }         
            set { }         
        }         
        
        // Simple helper struct to keep track of block dimensions         
        [Serializable]         
        public struct BlockInfo         
        {         
            public int startIndex;         
            public int blocks;         
        }         
        
        // Attempt to divide blocks between ranks evenly         
        public Object[] ComputeAssignment(int ranks)         
        {         
            Object[] blockInfo = new Object[ranks];         
            BlockInfo thisBlockInfo;         
            int blocks = samples / blockSize;         
            int blocksPerRank = (int)Math.Ceiling((double)blocks / (double)ranks);         
            int remainingBlocks = blocks;         
            int startIndex = 0;         
            for (int i = 0; i < ranks; i++)         
            {         
                // Compute block info         
                thisBlockInfo.startIndex = startIndex;         
                thisBlockInfo.blocks = System.Math.Max(System.Math.Min(blocksPerRank, remainingBlocks),0);         
                startIndex = startIndex + thisBlockInfo.blocks * blockSize;         
                remainingBlocks = remainingBlocks - thisBlockInfo.blocks;         
        
                // Assingn thisBlockInfo into blockInfo         
                blockInfo[i] = thisBlockInfo;         
            }         
            return blockInfo;         
        }         
        
        // Method that queries the data and converts results to numeric array         
        public msnl.NumericDenseArray<double> ReadWorker(Object blockInfoAsObject)         
        {         
            var demographicsRef = new DemographicsReference<demog1>();         
        
            // Select all data from each row         
            Func<demog1, demog1> selector = demographic => demographic;         
        
            // Get information about blocks to read         
            BlockInfo blockInfo = (BlockInfo)blockInfoAsObject;         
            int blocks = blockInfo.blocks;         
            int startIndex = blockInfo.startIndex;         
        
            long columns = 10;         
            msnl.NumericDenseArray<double> data;         
            if (blocks == 0)         
            {         
                /// If number of blocks is 0 return empty array         
                data = msnl.NumericDenseArrayFactory.Create<double>(new long[] { 0, columns });         
            }         
            else         
            {         
                // Execute query         
                var dmList = demographicsRef.GetDemographicsData(startIndex, blocks, blockSize, selector);         
                var rows = dmList.Count();         
                data = msnl.NumericDenseArrayFactory.Zeros<double>(new long[] { rows, columns });         
        
                // Copy a subset of data to array columns         
                for (int i = 0; i < rows; i++)         
                {         
                    data[i, 0] = (double)dmList[i].AverageHouseholdSize2010;         
                    data[i, 1] = (double)dmList[i].HouseholdGrowth2010To2015Annual;         
                    data[i, 2] = (double)dmList[i].MedianAge2010;         
                    data[i, 3] = (double)dmList[i].MedianHomeValue2010;         
                    data[i, 4] = (double)dmList[i].PctBachelorsDegOrHigher2010;         
                    data[i, 5] = (double)dmList[i].PctOwnerOccupiedHousing2010;         
                    data[i, 6] = (double)dmList[i].PctRenterOccupiedHousing2010;         
                    data[i, 7] = (double)dmList[i].PctVacantHousing2010;         
                    data[i, 8] = (double)dmList[i].PerCapitaIncome2010;         
                    data[i, 9] = (double)dmList[i].PerCapitaIncomeGrowth2010To2015Annual;         
                }         
            }         
            return data;         
        }         
        
        // Serial method to query the data to get a mapping from array rows to geographies         
        public IList<ZipCodeAndLocation> GetRowMapping()         
        {         
            var demographicsRef = new DemographicsReference<ZipCodeAndLocation>();         
        
            // select geography ID, name, state abbreviation and index         
            Func<demog1, ZipCodeAndLocation> selector =         
                demographic => new ZipCodeAndLocation(demographic.GeographyId, demographic.GeographyName, demographic.StateAbbreviation);         
        
            // Execute the query         
            var rowMapping = demographicsRef.GetDemographicsData(0, this.samples / this.blockSize, this.blockSize, selector);         
        
            // Construct mapping         
            for (int i = 0; i < rowMapping.Count; i++)         
            {         
                rowMapping[i].idx = i;         
            }         
            return rowMapping;         
        }         
    }         
}

Microsoft Codename "Cloud Numerics"

Getting Started

Sign Up for the Lab

Lab Home

SQL Azure Labs Home