Olav Tollefsen's Blog

Using Windows Azure Blob Storage to store aerial images

2009-03-12T15:57:04Z

The scenario

I have an ASP.NET web application, which is used to search and navigate over a very large number of aerial images. Currently there are nearly 400.000 images consuming 2 TB of storage + backups.

I thought it would be a good idea to see how the Windows Azure Blob storage could be used to store the aerial photos for my application. I wanted to do this without moving the entire web application to Windows Azure (taking one step at a time). Since Windows Azure currently is in beta, I only have about 50GB of cloud storage to play with. Therefore, I will start by moving all the lower resolution images to the Windows Azure Blob Storage.

Create a new Storage Project in Windows Azure

Before we can start to upload the images to the Windows Azure Blob Storage, we need to create a new Storage Account in the Azure Services Developer Portal:

When you go through the process for creating a new Storage Account, you will get the required Endpoints and Access Keys for working with the Blob Storage from your code.

Uploading the Images to Windows Azure Blob Storage

To upload the images from my local storage to the Windows Azure Blob Storage in the Microsoft cloud datacenter, I decided to write a small and simple console program. This would iterate through all the image files in a given directory and upload each image. From my datacenter in Norway, this process would typically upload between 4000 and 6000 images per hour for small 50 kB thumbnail images.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Samples.ServiceHosting.StorageClient;
using System.Collections.Specialized;
using Microsoft.ServiceHosting.ServiceRuntime;
using System.IO;
using MapaidLibrary.BusinessLogic;
using MapaidLibrary.Entities;

namespace UploadPhotos
{
    class Program
    {
        static string sourceDirectory = @"\\leopard\mapaid\photos\l1";
        static string targetContainer = "photosl1";

static BlobContainer _Container = null;

        static void Main(string[] args)
        {
            // Get the configuration from the cscfg file
            StorageAccountInfo accountInfo = StorageAccountInfo.GetDefaultBlobStorageAccountFromConfiguration();

            // Container names have the same restrictions as DNS names
            BlobStorage blobStorage = BlobStorage.Create(accountInfo);
            _Container = blobStorage.GetBlobContainer("photosl1");

// Make the container public so that we can hit the URLs from the web
_Container.CreateContainer(new NameValueCollection(), ContainerAccessControl.Public);

var files = from file in Directory.GetFiles(sourceDirectory) select file;

int i = 0;

            DateTime startTime = DateTime.Now;
            foreach (string file in files)
            {
                UploadPhoto(file);
                i++;
            }

}

        static void UploadPhoto(string fromPath)
        {
            try
            {
                FileStream fs = File.OpenRead(fromPath);

                byte[] data = new byte[fs.Length];
                fs.Read(data, 0, data.Length);
                fs.Close();

string blobFilename = Path.GetFileName(fromPath);
BlobProperties properties = new BlobProperties(blobFilename);

                // Create metadata to be associated with the blob
                NameValueCollection metadata = new NameValueCollection();
                metadata["FileName"] = blobFilename;

properties.Metadata = metadata;
properties.ContentType = "image/jpeg";

                //// Create the blob
                BlobContents fileBlob = new BlobContents(data);
                bool ok = _Container.CreateBlob(properties, fileBlob, true);
            }
            catch (FileNotFoundException ex)
            {
                Console.WriteLine("File not found: " + fromPath);
            }

}
}
}

This needs to be added to the App.config file:

Using the images from the Windows Azure Blob Storage

After uploading all the images, I would now be able to access the images by a simple URL like this:

http://youraccount.blob.core.windows.net/photosl1/50389339.jpg

In my ASP.NET web application, I simply had to change the URL of an ASP.NET Image tag like this:

<asp:Image runat="server" Width="193" Height="128" ImageUrl='<%# Eval("PhotoId", "http://youraccount.blob.core.windows.net/photosl1/{0}.jpg") %>'

Now my aerial images are served directly from a Windows Azure datacenter!

I have been using the Windows Azure Blob Storage for a few months now (since the PCD 2008) and even if the service is in beta, I have not experienced any issues accessing the images from my web application.

How to create and Publish your first Windows Azure application

2008-10-30T02:35:05Z

Signup for a Windows Azure account

First you need to signup for a Windows Azure account and get an invitation code to setup the server.

Visit http://www.azure.com for that.

Setup the development environment

Download and install the following from http://www.microsoft.com/azure/sdk.mspx

Windows Azure SDK

Windows Azure Tools for Microsoft Visual Studio

Create a new Windows Azure "Web Cloud Service" project in Visual Studio 2008

Now write some simple code:

Build the solution:

Now we are ready to package it for publishing our application to Windows Azure:

The above step creates a couple of files for us:

Create a new project in the online Azure Services Developer Portal

Click on New Project and select Hosted Services:

Enter a name for the project

Enter a name for your hosted service:

Deploy our application to the Windows Azure Cloud

Now we are ready to upload the files created by the Publish step from Visual Studio 208 earlier:

Select the application package file:

And then the Configuration file:

Our application is now uploaded to the Windows Azure Cloud:

The Application is first deployed to a Staging environment:

Start the Application in the Staging Environment

We can now Run the Application:

The Application is now ready to be tested in the Staging environment:

Test the Application in the Staging Environment

This is the Application running from the Windows Azure Cloud Staging environment:

Wow! The application is available on the Internet and is working!

Deploy the application into Production

Now click on the image with the arrow to deploy the Application into the Production environment:

The application is now up and running into the Windows Azure Cloud production environment!

Windows Azure announced

2008-10-27T20:51:00Z

Today at the PDC 2008 keynote Windows Azure and the Azure Services plattform was announced by Ray Ozzie.

What is Azure?

Windows Azure is a new operating system and platform for building and running applications and services, that will be hosted on a cloud-based infrastructure. Windows Azure will maker it easier to write applications and deploy them in a manner that will scale as needed on the Internet wouthout needing to go into the very complex task of setting up and manage the required hardware, data centers and other infrastructure. Windows Azure Services will also make it easier to handle authentication that spans across users from different organizations that could be using many different authentication systems. Windows Azure build on familiar development tools and frameworks like Visual Studio and the .NET ramework.

For more information see http://www.azure.com

What is the PDC 2008 all about?

2008-09-23T12:22:00Z

Have you heard that PDC2008 is the place to see the coolest new technology? That's right, at PDC2008, you'll discover what is in store for the future of ASP.NET. Have some fun with XNA and game development, or go in-depth with IronRuby. Other topics include: Live Mesh, Office Communications Server, SQL Server 2008, FAST, Silverlight, Oslo, Visual Studio Team System, Virtualization, SharePoint, Dynamics, Windows 7 and more! You can also network with top developers hang out with your friends and colleagues, and have fun in the L.A. sun!

Take a look at these videos for more insight into what the PDC 2008 is about:

Countdown to PDC 2008: This is the Software + Services PDC, Plus a Hard Drive Chock Full o’Bits is a PDC Attendee’s Dream Come True!

Countdown to PDC2008: Amitabh Srivastava, Corporate Vice President of Cloud, Infrastructure, & Services Has the Longest Title Ever!

Countdown to PDC2008: Extending the Data Platform to the Cloud

Countdown to PDC2008: By Developers, for Developers: Don Box and Chris Anderson

Countdown to PDC2008: Technical Fellow John Shewchuk Talks About His Talk at PDC2008

Using LINQ to Objects to simplify procedural code

2007-08-30T22:09:00Z

How many times haven't you written procedural code to loop through lists to do sorting and filtering of items like this?

List<Position> positions = GetPositions();

// Sort the list and filter

SortedList<string, Position> openPositions = new SortedList<string,Position>();

foreach (Position position in positions)

{

if (position.Quantity > 0.0)

openPositions.Add(position.Paper, position);

}

The above sample is of course very simple, but by using LINQ to Objects you are able to do it even simpler like this:

List<Position> positions = GetPositions();
var openPositions = from p in positions where p.Quantity > 0.0 orderby p.Paper select p;

That was nice! I find that using LINQ to Objects makes my code more clean and even easier to read and understand.

How does LINQ to SQL affect the architecture of your application?

2007-08-30T16:13:00Z

During the past week, I have been converting a few applications to using Visual Studio 2008 and LINQ. It’s nice to be able to remove all of the SQL query strings from my code and replace them with nice and clean LINQ syntax. In the beginning, I was a bit confused on how to structure LINQ into the various layers of my applications, but after a few iterations, I think I have found a reasonably good structure.

Before using LINQ

Traditionally I have structured my application back-end into a Business Entities layer, a Data Access layer and Business Logic layer.

Business Entities

In the Business Entities layer, I create classes of simple objects that are used by many different parts of the application. Examples of such simple classes could be Product, Order and Shopping Cart. These classes typically contain simple constructors, some properties with setters and getters and maybe some functions to do some calculations on the data contained in the object.

An example:

public class Product

{

public int ProductId { get; set; } // New VS 2008 syntax!

public double Listprice { get; set; }

public double Discount { get; set; }

public double Price

{

get { return (Listprice - Discount); }

}

Data Access Layer

The Data Access layer contains a lot of functions accessing the database. There would be functions for doing create, read, update and delete of records in the database. A lot of strings containing embedded SQL statements would be contained in this layer.

An example:

public Product GetProduct(int productId)

{

Product product = null;

string sqlCommand = @"

SELECT ProductId, Listprice, Discount

FROM Products

WHERE (ProductId = ?)";

OdbcConnection myConnection = new OdbcConnection(connectionString);

OdbcCommand myCommand = new OdbcCommand(sqlCommand, myConnection);

myCommand.Parameters.Add("ProductId", OdbcType.Int).Value = productId;

OdbcDataAdapter myDataAdapter = new OdbcDataAdapter(myCommand);

DataSet ds = new DataSet();

myDataAdapter.Fill(ds, "Products");

if (ds.Tables["Products"].Rows.Count == 1)

{

product = new Product();

product.ProductId = (int) ds.Tables["Products"].Rows[0]["ProductId"];

product.Listprice = (double)ds.Tables["Products"].Rows[0]["Listprice"];

product.Discount = (double)ds.Tables["Products"].Rows[0]["Discount"];

}

return product;

}

Business Logic Layer

In the Business Logic layer there are functions that perform higher level business processing and will often invoke several operations across multiple functions from the Data Access layer.

An example:

public static Order CreateOrder(ShoppingCart shoppingCart, CommerceUser user)

{

Customer customer = CustomerDAL.Find(shoppingCart.BillingAddress.Name, user.Email);

if (customer == null)

{

customer = CustomerDAL.Create(

shoppingCart.BillingAddress.Name,

user.Email);

}

Order order = new Order(

shoppingCart.UserGuid,

customer.CustomerId,

customer.Email,

shoppingCart.BillingAddress.Copy(),

shoppingCart.ShippingAddress.Copy(),

shoppingCart.Comments);

foreach (ShoppingCartItem item in shoppingCart.Items)

{

OrderItem orderItem = new OrderItem(

(OrderItem.OrderItemType)item.ItemType,

item.CatalogId,

item.SKU,

item.ProductVariantName,

item.Quantity,

item.Price,

item.Discount,

item.TaxRate);

order.AddItem(orderItem);

}

// Check that everything is in stock

foreach (OrderItem orderItem in order.OrderItems)

{

if (orderItem.ItemType == (int)OrderItem.OrderItemType.Product)

{

int quantityInStock = InventoryDAL.GetQuantityInStock(orderItem.CatalogId, orderItem.SKU);

if (orderItem.Quantity > quantityInStock)

throw new OutOfStockException("Item out of stock");

}

OrderDAL.CreateOrder(order);

// Reserve all the items in the order from the inventory

foreach (OrderItem orderItem in order.OrderItems)

{

if (orderItem.ItemType == (int)OrderItem.OrderItemType.Product)

{

InventoryDAL.ReserveInventory(

order.OrderId,

orderItem.CatalogId,

orderItem.SKU,

orderItem.Quantity);

}

OrderEvent orderEvent = new OrderEvent(

DateTime.Now,

OrderEvent.EventCode.OrderReceived,

"");

ExecuteOrderReceived(order, orderEvent);

return order;

}

Using LINQ

How does the use of LINQ affect the above structure of the application? When you work with LINQ, the first thing you want to do is to make sure that your database is structured as well as possible. That is if you have the flexibility to change the schema at this point in time. LINQ auto-generates entity classes based on the database schema and the generated classes will be a lot easier to work with when the data model is nice and clean.

Where to add the “LINQ to SQL Classes” item?

When you want to use LINQ to SQL in your application from Visual Studio, the first thing you do is to add a “LINQ to SQL Classes” item in your project. To which layer does this item belong? My initial thought was to add this item to the Data Access layer, but after a little experimenting, I have decided that this item belongs to the Business Entity layer of the application. The reason behind this is that “LINQ to SQL Classes” auto-generates a lot of classes is very similar to the classes you traditionally write in the Business Entities layer. If you look closer, you will see that the classes generated are partial classes. This makes it possible to split the declaration of a single class into multiple source files. It’s therefore very easy to add functionality to the auto-generated classes.

Let’s take the Product class we started with as an example. The Product class that we originally created by hand contained four properties: ProductId, Listprice, Discount and GetPrice. If we look at a Products table in the database, we could expect ProductId, Listprice and Discount to be present as columns in the table, while the GetPrice would maybe be computed dynamically at run-time. LINQ to SQL would auto-generate a Product class for us with all the properties corresponding to the columns found in the Products table. We could then extend the auto-generated Product class like this in another source file (Products.cs):

public partial class Product

{

public double Price

{

get { return (Listprice - Discount); }

}

To be able to extend a class using partial classes, you need the two classes to be declared within the same namespace. Having the “LINQ to SQL Classes” item (with the auto-generated clases) in the Business Entities layer together with the extended classes you write by hand therefor makes a lot of sense.

How was our Business Entities layer affected by using LINQ? Well, a lot of code-lines that we had to write by hand before using LINQ could be removed and we were only left with the extensions to the partial classes that wee still need. All the interfaces could still be exactly the same as before and we would not have to change other parts of our application.

Rewrite of the Data Access Layer using LINQ

Now let’s take a look at how our Data Access layer is affected by using LINQ. Here is the new version of the GetProduct function:

static public Product GetProduct(int productId)

{

CommerceDataContext db = new CommerceDataContext();

Product product = (from p in db.Products where p.ProductId == productId select p).SingleOrDefault();

return product;

}

Wow! That was a LOT simpler than the previous version not using LINQ. In addition the LINQ query above is type safe and compile-time checked. Again, the interface to our Data Access Layer classes / functions remains exactly the same and there is no need to rewrite other parts of the application. I can even convert the data access classes one by one and have the old SQL queries in some classes and LINQ queries in others while I’m rewriting the code.

Business Logic Layer

How is the Business Logic layer affected by using LINQ? In my case, the Business Logic layer was not affected at all. All the changes made have been internal to each class used. This is also a great benefit.

What do you think?

Howto enable Windows Search Service in Windows Server 2008

2007-05-07T11:15:00Z

I found that the Windows Search Service in Windows Server 2008 is a bit hard to discover. The Windows Search Service is actually burried as a Role Service under the File Services Role.

These are the steps to add and enable the Windows Search Service in Windows Server 2008:

1) Start Server Manager

2) Click on Roles in the left navigation pane

3) Select Add Role in the Roles Summary pane to the right

4) Select the File Services role and click Next

5) Select the Windows Search role service

Finish the wizard and now you should have the Windows Search service up and running.

Windows Presentation Foundation (Avalon) Databinding to objects

2005-09-30T21:44:00Z

Databinding to an object datasource is really simple in Windows Presentation Foundation. I didn't find any good samples around, so I'm posting some code here to help you save some time.

My scenarion is really simple, I would like to create an WPF application to visualize my DVD collection. The code also shows how to bind an Image control i WPF to an JPEG image stored inside the SQL Server database (not just a pointer to a file). The code is simplified to only illustrate the main concepts and I have not worked with the design part yet.

First I start with a class to represent the information about a DVD title:

public class Dvd
{

        int _dvdId;
        private string _title;
        private byte[] _frontCover;

public int Id { get { return _dvdId; } }

public string Title { get { return _title; } }

        public BitmapFrame FrontCover
        {
            get
            {
                BitmapFrame frontCover = System.Windows.Media.Imaging.BitmapFrame.Create(new System.IO.MemoryStream(_frontCover));

                return frontCover;
            }
        }

        public Dvd(
            int id,
            string title,
            byte[] frontCover)
        {
            _dvdId = id;
            _title = title;
            _frontCover = frontCover;
        }

}

The Data Access Layer contains a method to return a list DVD's from the database:

        public static List<Dvd> GetTitles()
        {
            OdbcConnection connection = new OdbcConnection(m_connectionString);

            string selectCommand = @"
                SELECT     DvdId, Title, FrontCover
                FROM         Titles";

OdbcCommand command = new OdbcCommand(selectCommand, connection);

List<Dvd> titles = new List<Dvd>();

            connection.Open();
            OdbcDataReader reader = command.ExecuteReader();
            while (reader.Read())
            {
                Dvd dvd = new Dvd(
                    Convert.ToInt32(reader["DvdId"]),
                    Convert.ToString(reader["Title"]),
                    (byte[])reader["FrontCover"]);

titles.Add(dvd);

}

return titles;
}

Then we have the DvdCollection class which includes a method to return a list of Dvd objects:

using System;
using System.Collections.Generic;
using System.Text;
using DvdLibrary.BusinessEntities;
using DvdLibrary.DataAccessLayer;

namespace DvdLibrary.BusinessLogicLayer
{
    public class DvdCollection
    {
        public static List<Dvd> GetTitles()
        {
            return TitlesDAL.GetTitles();
        }

}
}

Then in the XAML code-behind, I have this code to retrieve a a list of DVD titles from the business logic layer:

private void OnInit(object sender, EventArgs e)
{
List<Dvd> titles = DvdCollection.GetTitles();

DvdList.DataContext = titles;
}

Finally we have the XAML file showing how to do the Databind to the list of DVD titles using a ListBox:

<?Mapping XmlNamespace="DvdLibrary_BusinessLogicLayer" ClrNamespace="DvdLibrary.BusinessLogicLayer" Assembly="DvdLibrary"?>
<Grid
xmlns="http://schemas.microsoft.com/winfx/avalon/2005"
xmlns:x="http://schemas.microsoft.com/winfx/xaml/2005"
xmlns:c="http://schemas.microsoft.com/winfx/markup-compatibility/2005"
xmlns:d="http://schemas.microsoft.com/expression/interactivedesigner/2005"
c:Ignorable="d"
Background="#FFFFFFFF"
x:Name="DocumentRoot"
Width="640"
Height="480"
x:Class="DvdBrowser.Scene1"
Loaded="OnInit"
>

<Grid.Resources>
<ResourceDictionary>
<Storyboard x:Key="OnLoaded" FillBehavior="HoldEnd" BeginTime="{x:Null}" />

<Grid.Triggers>
  <EventTrigger RoutedEvent="FrameworkElement.Loaded">
   <EventTrigger.Actions>
    <BeginStoryboard x:Name="OnLoaded_BeginStoryboard" Storyboard="{DynamicResource OnLoaded}"/>
   </EventTrigger.Actions>
  </EventTrigger>
</Grid.Triggers>

<ColumnDefinition/>
<RowDefinition/>
<ListBox
  IsSynchronizedWithCurrentItem="True"
  x:Name="DvdList"
  Margin="19,23,41,16"
  Width="Auto"
  Height="Auto"
  RenderTransformOrigin="0.5,0.5"
  ItemsSource="{Binding}"
  ItemTemplate="{StaticResource DvdItemTemplate}"
  />
</Grid>