David L's Blog

Goodbye Microsoft

davidlem — Wed, 08 Sep 2010 06:48:24 GMT

Today I resigned from Microsoft. While I’m leaving to take a big step in a new direction, I have to say, it’s with very mixed emotions.

Five and a half years ago, I started a journey that changed me forever. Not only was it the realization of a teenage dream (I still remember unwrapping the Visual Basic 3.0 box as an intern and thinking, one day I’ll ship a product for Microsoft), but it was also a graduation of sorts into the realm of industrial software engineering.

My time at Microsoft has been completely magical. Not only have I been fortunate to work with some of the most amazing professionals in my career (colleagues, partners, influences and students!), I’ve also had the opportunity to travel around the world, move from Australia to the United States, and fulfill a life-long dream and ambition, to ship a major product for Microsoft Corporation.

But, it’s time to move on.

Where am I going? I’m joining PricewaterhouseCoopers.

What am I going to do there? I’m taking a role as Director of Cloud Computing, focusing on how customers transform their businesses to take advantage of the cloud.

Why? Cloud is my passion, and while building one of the biggest and best at Microsoft was amazing from a geek and engineer perspective, I’m yearning to focus on the other aspects of what it takes for customers to truly use the cloud as a business transformer.

What will happen to my blog? I have a new one here.

So to all my Microsoft friends and family, thanks for five and a half years of being awesome!

Vale, lacerte!

Getting Started with Parallel Programming in .NET 4

davidlem — Fri, 14 May 2010 16:51:06 GMT

So one of the features of .NET 4 that I’ve been eagerly awaiting is the new Parallel Programming “stuff”.

Now, anyone who has worked in concurrent environments knows, it’s daunting to learn a new set of technologies, constructs, tools, etc. For those that haven’t worked in concurrent environments, the entry into concurrency can be downright intimidating. Either way, adopting concurrent practices in your code is not a trivial process.

So to get started, I took a simple approach, which I’ve detailed in this post. The goals of my approach were:

Understand at a high level, the full surface area of what is now possible with the Parallel Programming “stuff” (I really don’t know what to refer to it as")
Get my hands dirty building some simple examples that map to real problems I need to solve in my day to day job
Discover some new patterns that are made easier with the new PP “stuff”

First stop was to read this guide by Stephen Toub from the Parallel Extensions team. This is a fantastic guide, written at the perfect level for achieving goal 1. I read it top to bottom to glean the high level points, then went back over it with a more detailed eye to try and map the concepts Stephen talks about to real scenarios I have/have been experienc[ed][ing].

The second stop was to download the sample code from MSDN Code Gallery. There are a ton of samples here, and just starting with something like the Acme Pizza sample is very enlightening.

The third stop was to download the articles on Parallel Programming with the .NET Framework 4. I only got through the Coding Guidelines one, but will slowly grind through the others as part of my journey.

Finally, I have started to work through the blog posts on the team blog. So much gold here, I mean, check out this article as a start!

Now, there is a natural path that this kind of stuff leads to, because when you take the fact that Parallel Programming “stuff” is rolled into .NET 4, and Windows Azure will have .NET 4 support very soon, then the opportunity to build Parallel Programming Windows Azure “stuff” is pretty strong. Just putting it out there!

Enjoy! :)

Technorati Tags: Parallel Programming,.NET 4,Windows Azure

WCF, REST and URL Rewriting with Windows Azure!

davidlem — Sun, 25 Apr 2010 22:48:24 GMT

So there is nothing I find nicer than a well formatted, REST based URL.

Take for example:

http://myhost/customer/2/orders

As a way to retrieve all orders for customer number 2.

There is also nothing I like more than WCF! The flexibility to declaratively control the behavior of my web service is perfect for situations where I need to respond quickly to heterogeneous client calling requirements (REST, SOAP, etc).

And finally, I love Windows Azure, nothing more really to say there.

So how do you build a RESTful WCF Service in Windows Azure?

Start with a new Cloud project:

Then add a new WCF Service Web Role:

Now, this is enough to get a WCF service going in Windows Azure, so just hit F5 and you’ll get this:

If you don’t get this, then something is not right. First thing to check is that you’ve switched on WCF activation:

If this doesn’t help matters, then you’re out of luck my furry friend. Time to hit the power button and try again.

Next we want to change our service’s behavior to be more REST’y, so we start by changing the way the page is hosted. To do this, we set the Factory property of our ServiceHost by editing the markup of our service to use the WebServiceHostFactory:

Now, the first time you run this, you’re going to hit a 400:

Simply go into your web.config and remove the system.serviceModel section, as the WebServiceHostFactory will take care of your services behavior. (Some more helpful info on RESTful services with WCF here)

This should get your service going, but you’ll notice that the WebServiceHostFactory will not have any endpoints enabled.

All good, onto our next part. To make sure we can invoke our service through a HTTP verb, and the response can be returned in basic XML, we need to add some attributes to our OperationContract:

So now, you can simply invoke your service like such:

Getting closer.

But I don’t like having the .svc extension in the URL. One of the great things about REST, well, web services in total, is you don’t have to expose the implementation details to the caller. So let’s get rid of the .svc part. To do this, we use a fantastic feature available in Windows Azure called the URL Rewrite module. So I write a crude and dirty little rule in my web.config that maps any calls to myservice to Service1.svc.

Now we’re almost there:

Now, to make URL Rewriting work on your local machine, remember to install the appropriate version of the URL Rewrite Module on your machine, per these instructions.

Now, not happy to leave well enough alone, I want to pass in my value as part of the URI, rather than part of the query string, so I’m going to set the UriTemplate property of the WebGet attribute to reflect my formatting requirements:

You’ll also notice I had to change the input type of my method to a string, as the UriTemplate requires that parameters, in this example, {value}, are string types.

Now, everything is as it should be, and I can call my service straight from a HTTP verb, in this case, GET, using a nicely formatted URI:

This all runs nicely in Windows Azure, and more importantly, is callable from any client capable of constructing a HTTP payload, so the interop is awesome!

Here is the basic Windows Azure project in VS2010 format.

Enjoy!

Technorati Tags: Windows Azure,WCF,REST,URL Rewrite Module

Build Your Own Pivot Server in Windows Azure!

davidlem — Thu, 15 Apr 2010 16:07:48 GMT

So one thing I’ve been wanting to do since moving from the Windows Azure team to the Pivot team, is build a Pivot Dynamic Server in Windows Azure.

Well, tonight I decided to do just that, allow me to expatiate! ;0

Let’s start with a basic overview of what Pivot requires to load a collection.

First it needs a collection file, affectionately referred to as “the cxml”. This lays out the details of your collection, things like the facet categories that provide the faceted exploration capabilities, and the details about the item in your collection. Within the cxml, there is a reference to another file, known as “the dzc”, or the DeepZoom Collection. This file describes all the smaller “dzi” files, or DeepZoom Images, that make up the whole collections set of images.

Now, this is all pretty high level, and to get a real sense of what this all means, a little time spent reading through our site is time well spent. For those that have, and are simply asking the question, “How do I create a collection on the fly, and serve it up from Windows Azure?”, then I hope this blog post satisfies.

So given the basics above, that Pivot requires a cxml that points to a dzc of dzi’s, we’re ready to move to the Windows Azure part.

So Windows Azure has some awesome capabilities, that makes it possible to build something as powerful as a Pivot Dynamic Server in no more than a night.

To start with, I set some goals.

1. Be able to serve a completely dynamic collection from Windows Azure that would load in Pivot.

2. To not use Windows Azure Storage (yet!). For my demo, I’m assuming a single web role instance, with all files being served from the role itself, old skool style. I’ll rev this sample/demo to use blobs in Windows Azure Storage in a follow up post.

3. To generate the collection from a web source, that is, use something like Flickr or Twitter as the source of my collection and transform it on the fly (I picked Flickr).

4. Auto-generate my facets. This way, there is no need for “authoring” per se, you just point and click.

Now on to the code.

I started with a standard Windows Azure web role.

I then added a Generic Handler (.ashx) to the project, to handle the requests coming from Pivot. Because Pivot expects a file that has a CXML extension and contains XML, and because goal 1 requires that we are able to handle dynamic requests, we need some way to map a virtual request like http://foo/bar.cxml?id=n to something that will return the dynamically generated cxml.

The second thing I needed to do was setup a URL rewrite rule so that all request to my web role are directed to my handler, where it can make the appropriate changes to the request.context.

Now the basics of my service are ready.

Next I need to serve up some basic cxml. To do this, I created a simple class that supports XmlSerialization that provides the basic infrastructure to create a collection. This way, I can build my collection in code, then simply return the serialized version of the collection through my handler.

OK, next I needed to populate my items. For this I simply load a public Flickr RSS URI, use a LINQ to convert the Flickr RSS format to my new collection format, preserving key information such as the href’s of the images, as we’ll need that later.

Now, onto the facets. One of the most powerful attributes of Pivot is the ability to search and discover information from your collection using facets. For my Flickr collection, the most meaningful facets are those that exist in the names of the photos I’ve uploaded, so I created a very simple keyword extractor, to build my facets.

Now finally, I need to convert the images to a set of DeepZoom Images, and all of the DeepZoom Images into a DeepZoom Collection. To this, I need to get a hold of the DeepZoomTools.dll that comes with the Deep Zoom Composer. Once I have that, I simply iterate through my collection, and build the DZI’s and DZC from the details I have about my images.

And I’m done!

Now, the last thing to discuss is the way I serve up the files that I’ve built in my Windows Azure web role, under the LocalResource. To do this, I have a simple function in my handler which says:

If the request is for a cxml, then build a cxml
If the request is not for a cxml, then it must be a request from Pivot for either the DZC or a DZI, so simply serve up the file.

Voila! Give it a shot for yourself:

http://davidlempivot.cloudapp.net/FlickrCollection.cxml?id=85482401@N00

Simply put your FlickRss ID in place of mine, and plug the URL into Pivot like I have below, and you’re Windows Azure Dynamic Flickr Collection is served up!

Sample code is here, usual disclaimers apply in terms of lack of error handling, general crud’iness, terrible camel casing! ;)

Enjoy!

Technorati Tags: Windows Azure,Live Labs,Pivot,Dynamic Server

New Gig!

davidlem — Tue, 02 Mar 2010 00:52:38 GMT

One of the things I love about working for Microsoft is the amazing amounts of opportunity to follow your passions. I’ve been very fortunate over my five years (yep, I celebrate 5 years at Microsoft this month) to have explored a number of great roles, and today I start off on another great adventure.

So where am I going?

Microsoft Live Labs! More specifically, to go work on the Pivot project.

Why am I going?

To springboard off the amazing experience I had at Windows Azure, and at a more fundamental level, the journey I started when I joined Microsoft. Being able to traverse the disciplines of evangelism, product management and engineering at the level Microsoft performs at is very rare, especially for someone who has spent the majority of their time in software engineering. Prior to joining Microsoft, I only ever worked in technical roles, so being able to contribute in so many ways is not only very unique to Microsoft but also hugely rewarding.

So wish me luck! And stay tuned for much Pivot goodness! :)

Technorati Tags: Microsoft,Live Labs,Pivot

Australia, here I come… again!?

davidlem — Wed, 27 Jan 2010 23:45:05 GMT

I’m so excited, I just can’t hide it!

And neither could Coatesy!

http://blogs.msdn.com/acoat/archive/2010/01/27/windows-azure-user-group-briefings.aspx

That’s right, I’m being crated up and shipped back home to Terra Australis for a week of Windows Azure goodness from the 22nd of Feb to the 25th of Feb.

Please check-out Coatesy’s blog for all the deets and I look forward to seeing everyone very soon!

Woo hoo! :)

Technorati Tags: Windows Azure,Launch,Australia

Expression Blend 3 and SketchFlow! Wow!

davidlem — Wed, 16 Dec 2009 23:42:22 GMT

So I’m doing some prototyping at the moment and one of the most frustrating aspects of getting started with any prototype is capturing the:

Flow
Main Screens
Specification

I think in mind maps, so one of my biggest problems when starting a new project is setting up the high-level map of how everything is going to connect together.

The second issue I have is I want to capture the essence and linkage of each screen, but not in painful detail, just enough so someone can infer the underlying data model and application behavior.

So when I heard about Sketchflow (jeez, I sound like one of the guys of those late night infomercials, I promise, no blanket that doubles as clothing) I was immediately intrigued.

Four things that are awesome about Sketchflow with respect to my needs are:

1. The Sketchflow Map

This wonderful little feature lets you define new screens by simply adding them to the map, then connect those to other screens. So building your high level flow is super simple and fast!

2. Sketch Tools

So building the above page, which was created by simply creating a new screen on the map was also dead easy. There is a wonderful little toolbox of controls that I can use to build out these pages quickly:

3. The Run-time Experience

So I just hit F5 and the project gets spun-up as a little web project in the player! Awesome, I can then try out the ergonomics of the sketch, provide feedback on what I like/don’t like, and navigate from page to page.

4. Export to Word!

So this is awesome, I’ve built my flow, played around with the ergo and feel, and I’m ready to send it to my boss in a form he can consume so the cash flows out of his pocket into my “build it” budget!

Now, there are a whole bunch of things Expression Blend 3 + Sketchflow does in addition to what I’ve captured above, but what is the most important aspect is how easy it is to use and how much you can achieve in a very short period of time. This is the key to me using for my initial needs, but also what will keep me using it to learn more about what else it does.

Enjoy :)

Technorati Tags: Product Development,Expression Blend,Sketchflow

Remote Command Service for Windows Azure!

davidlem — Sat, 12 Dec 2009 00:56:48 GMT

So a while back I blogged a little sample code on how to get some info about what’s happening inside your instance using a simple process within a web page.

Looking Inside Windows Azure!

Well, I decided to refresh this code sample using one of my favorite features, InputEndpoints!

So let’s take a quick tour of the sample. At a high level, this is what’s going on:

Let’s dig deeper. The client is a simple console app, and the service is a worker role that exposes a TCP InputEndpoint, connected to a socket. The socket accepts a single inbound client connection, and then executes commands on the local instance and returns the results to the client.

So, I can fire up my client and connect to my service like so:

And execute commands such as DIR or NETSTAT (or others you could try), against the local VM. This is very handy when trying to debug local instance issues.

You can download the code here. It contains the code for the VS2010 Worker Role cloud service project and the simple console client.

Also, you’ll need to update the diagnostics connection string before deploying it to the cloud, and I’d suggest deploying the worker role with a single instance to avoid any load balanced strangeness.

Enjoy :)

Technorati Tags: Windows Azure

My PDC09 Session – Code and Clip!

davidlem — Thu, 10 Dec 2009 23:20:00 GMT

So one thing I’ve been terribly slack about since returning from PDC09 has been posting my session clip and code snippets for folks to download/reference, so my sincerest apologies, and without further ado:

1. My session

2. My slide deck

3. My code

4. My banner (picture care of JLD)!

Let me know if you have any issues, otherwise, enjoy! :)

Technorati Tags: PDC09,Windows Azure

Guy Shahine from the Windows Azure Dev Team!

davidlem — Tue, 24 Nov 2009 02:44:28 GMT

Guy has started blogging!

http://blogs.msdn.com/gshahine/

Go check out his first post, and stay tuned for more information from the belly of the beast!

Technorati Tags: Windows Azure,Dev Team

Windows Azure, MSBuild, and Lotsa Files!

davidlem — Wed, 04 Nov 2009 00:10:46 GMT

So I’ve been working on a project, and I came across a little challenge that I thought folks may be interested in the solution to.

So, the problem is this, I have a dependency on a bunch of files that I need access to from my worker role in the cloud and dev fabric. I have all these files in a folder, and the folder has a sub-directory structure that I want preserve inside my project.

Question! How do I add this folder to my project, with the directory structure preserved, and have all the files “Copy To Output Directory” property set to something, like in my example, “Copy Always”, so it gets taken up to the cloud with my package? Easy… well, kinda.

First step, from Windows Explorer, drag said folder into your role project:

This will add the folder into your project, with the folder structure in place, however the properties of the files will not be set, and therefore, when you publish your project, the files won’t be picked up in your package.

So, second step is, exclude the folder from your project. Don’t delete it! This will remove it from the disk, you just want to remove the reference of it from the csproj file.

Next, you need to unload your project.

Then edit the csproj file.

Inside the csproj, scroll down to the area where you find all the ItemGroup elements:

Add a new ItemGroup element, say above the app.config one (it doesn’t matter where), and set it to include your folder like so:

What you’re telling VS to do is, include the "Melbourne Photos” folder, recursively, to the project, with the CopyToOutputDirectory property set to Always.

When you reload the project:

You will notice all the files have been included in the project, and the Copy to Output Directory property has been set to Copy always for all of them!

When you publish your role, all these files will be taken up with your project, and accessible by your code!

Big thanks to Guy and Dan for helping with this!

Enjoy! :)

Documentation Driven Development!

davidlem — Wed, 23 Sep 2009 20:03:04 GMT

So I was having lunch with our Programmer Writer, Tamra Myers the other day, and we got onto the conversation of documentation. I’m a huge Huge HUGE fan of documentation, in fact, when I coded in anger, I was known for my extensive, well written, downright Elizabethan doco.

I was then having lunch with Simon Guest on a different day and we got onto the topic of doco too, in fact, we were talking about some research he had done into how much of MSDN doco actually had code samples! You might start seeing a trend emerge… yes, I love to eat lunch with passionate doco people!

So what has this got to do with the price of Tzatziki in Redmond? Well, we all strive (in software engineering anyway) to build technology that people can use, or at least, work out how to use. If you build software, especially developer platform software, it’s not always easy to build highly intuitive APIs and components. So what’s left is the doco. The doco is the first port of call for the user of developer goo, in there, they can orientate themselves around the nuances and intricacies of a component and the bend it to their will. However, if you don’t provide doco, well then, you’re putting the developer user in a world of pain, and forcing them to make a choice. They can either try and hack away, using coincidental discovery to work out how your thing works, or try and find a blog post from someone who has. If they choose the former, they could waste a bunch of time and never hit paydirt, if the they choose the latter, there is always the chance that the code they are using is not optimal, or inline with the way the component/API was designed.

So why is there such poor doco? Well first of all, the engineering process if fundamentally broken. It starts with requirements, then specs, then code, then test, then release, then somewhere along the way, doco. Now, it’s like running a marathon, they always tell you to stretch and cool down afterwards, but for most of us, when you hit the finish line, you just want to pass out. Coding is the same, when you’ve squashed that last bug, and checked in your final build, the last thing you want to do is write your coding memoirs. However, herein lies the travesty! Because most of us don’t summon the last drops of energy to write good doco, all our hard work is wasted because know one knows where to find the secret red button that can transport us through space and time. Instead they are left to prod around the facade and try and find the switch.

How do you solve the problem? Start with doco first! Just like the industry realized that by putting testing after development, you’re putting the cart before the horse, putting doco before testing makes sure you stay true to what the customer is meant to have and what it’s meant to do. And more important, you can see how easy or usable your stuff is before you code it. I mean, how many of us have written a component, then given it to someone, had them ask us for a quick sample, then as you develop the sample you realize, “Damn, this is hard to use” or “I could have added a method that would have reduced this from 10 lines to 2 lines of code”? By starting with how should someone use the component, what it does, what are the inputs/outputs, exceptions, expected behavior, you have more than just the functional spec to work against. And, most importantly, when you’re ready to release, you have the best doco on earth, better than the code! To the customer anyway ;)

By the way, if you want to read more on this topic, check out the following paper:

Documentation Driven Development for Complex Real-Time Systems

Now go forth and doco!

Technorati Tags: Engineering

What’s Your Cloud Address?

davidlem — Tue, 01 Sep 2009 19:20:16 GMT

One of the many challenges to building a cloud is your addressing strategy.

Now, addressing is an issue for any public facing Internal application, but for clouds, the issues increase because of the dynamic nature of deploying applications that require publicly addressable endpoints at huge scale.

Let’s start by thinking about a traditional web application, whether it be large or small. At the early design phase, you are able to identify the front end roles that will require public IP space. You can then start capacity planning to identify your peak load forecasts, how many front end nodes you will need to support max load, then make the appropriate request and allocation. As the process for allocating IP space from the public range can take many months, you want to conclude this process early in the design phase.

Secondly is the scarcity of resources. If you rely on IPv4, your going to have to be very sparing of your use of public space, as IPv4 is not only very scarce, but it’s expensive. IPv6 on the other hand is abundant, simplifies your network strategy (as you don’t have to necessarily NAT addresses etc), and is easy to acquire. On the flip side, you need to do more planning and up front design to ensure your system, both hardware and software, can accommodate IPv6 addressing.

And now, finally onto the cloud part. In a cloud environment, the tenant, that is, the application being deployed, requires a public IP address. The may have many back end nodes, but for the most part, these will sit behind a load balanced front end node, and therefore, a public IP space. You could try and virtualize that whole layer, and instead, have a primary “head” IP address, where all traffic is routed through, then split based on host headers, but that isn’t going to scale very well. So instead, each customer should have a public IP address for each publicly exposed endpoint, but then, you don’t know how many customer exposed endpoints your going to have during design, or deployment for that fact. You could have 100, you could have 1,000,000.

Using strategies like just-in-time allocation, based on historical usage and prediction is a good approach, but requires a level of sophistication, or you can acquire lots of space up front, but this can be costly and wasteful. Ideally you want to use a mixture of both, establishing a small unit of allocation, something like a /20 or higher space, then constantly measuring the watermark and determining when you need to add more, based on prior use patterns. This way, you know how early you need to make the request, and you don’t over allocate.

Food for thought! :)

Technorati Tags: Building a Cloud

Cloud Apprenticeship!

davidlem — Wed, 26 Aug 2009 19:54:53 GMT

So I was talking to an old friend yesterday about building software, and we both came to the conclusion that there are some things you need to learn:

From books
From others
From doing

We then got onto the topic of cloud computing, he works in the same space, so it was a great context to place on our conversation.

Taking the thinking further, we both realized that when we decomposed our jobs into buckets, we arrived at:

Working with customers – This is the most important element, and the only way to sharpen your skills are to engage real customers, books are pretty much useless in this area, and while you can learn some tricks from others, nothing beats logging hours in the field.
Core engineering – This is bread and butter stuff, and doesn’t really change depending on what your working on. And it definitely get’s ticks in all three categories from above.
Working in a team – Again, this doesn’t change all that much, working with your own team and other teams is not necessarily specific to cloud, but when you think of it in terms of the list above, you could learn a little about it in books, but nothing compares to learning from others and from doing.

We then got to the unique parts when working in the cloud, and this ended up being more about trading lessons learned, and while it definitely leverages the core concepts above, there are some aspects which I’ve never encountered in any other type of project.

Think big! Very few projects work at the scale of cloud computing projects. When you take into account the sheer size of the data center deployments, the hardware, facilities, connectivity, nothing can prepare you for the types of problems you will need to solve in terms of core engineering and software development. How do you deploy, manage, test, recover, fail-over, charge, audit, a massive, geographically dispersed, mega-distributed system? You don’t only have to build new software, you need to build new ways to test, deploy, manage and support it.
Shipping! The cloud world is not like the packaged product world, it’s not even like the online services world. Not only do you need to think in small bite sized chunks, to reduce the integration and dependency workload and risk, but you also need to plan for how you get that chunk out to thousands of live servers running live customer apps, in a completely passive and non-intrusive/non-disruptive way. You also have the incumbent hardware that comes with cloud, so not only are you shipping software, your rolling out hardware, and lots of it, to places you never knew about before, I liken it to landing jumbo jets on small islands.
Simplicity! If you can’t describe how you plan to define, develop, test, deploy, test, deploy, test, deploy again, and finally test once more, troubleshoot, maintain, update, and finally decommission the feature you want to build, to a layperson, then think harder or don’t do it. When you’re working in an environment where every bug is a potential go-home bug (the kind of bug that wipes every users data out or something else so catastrophic that you can never recover, you just go home and find a new job), you need to raise the bar on all aspects, and complexity is usually the root of all evil
Test! I have a saying that I only started using when I started working in the cloud, “Give me a tester before you give me a dev”. The reality of the cloud is you cannot and will not ever test enough, the best you can do is test lots. And the types of tests are so diverse, starting from a one box environment for the developer, to check-in tests for unit completeness, integration tests for fit and flow, perf/scale/load tests, environment tests for datacenter compatibility, regression testing, pen testing both at the edge and at the platform boundaries. Key take away is that quality matters so much more in the cloud because the potential impact of a bug is much larger and unpredictable that most other environments. You also want to build self-testing into your code, you want your code/feature to be cognizant of where it is, so it can ask questions of itself, and react accordingly. Think of it like the safety switch on a power point, make sure your code doesn’t need to touch the iron to know it’s hot!

Anyhoo, just some thoughts from my time as an apprentice of the cloud.

Enjoy :)

Technorati Tags: Thoughts on the Cloud

Windows Azure and Live ID Auth!

davidlem — Tue, 25 Aug 2009 19:43:25 GMT

OK, so MattT (thanks for the feedback on my blog too Matt :)) pinged me and asked if something had changed in the way Live ID auth worked in Windows Azure.

He was looking around Bing for some samples, and came across this post but couldn’t get it working. You’re right Matt, there was a change.

In late May, 2009, we made a change to the way developers access Live ID, see this post:

Windows Azure - Changes to Alerts, Analytics, and Windows Live ID

So how do you code it up?

Step 1. Download the appropriate SDK (I downloaded the CS one)

Step 2. Go to http://windows.azure.com and create a new Live Services project, make sure you select Live Services: Existing APIs as your project type

Step 3. Enter the details of your Web Role project (you should have created this already, it’s your http://xxxx.cloudapp.net url). The domain should be the straight “xxxx.cloudapp.net” part, in the returnUrl field, enter http://xxxx.cloudapp.net:81/webauth-handler.aspx just while you do local machine testing.

Step 4. Now, while I tested it locally, I had to do a quick update to my hosts file to point my url (crmintegration.cloudapp.net) to 127.0.0.1 so I could test it locally. Just create a pointer to your url in your hosts file to point to 127.0.0.1 just while you test locally. This will enable the above http://xxxx.cloudapp.net:81/webauth-handler.aspx returnUrl to work.

Step 5. Create a Web Role project, and import the WindowsLiveLogin.cs, webauth-handler.aspx and webauth-handler.aspx.cs files from the Windows Live SDK sample directory. You’ll need these to process the return details from Live ID.

Step 6. Run and test, and make sure it’s connecting to Live ID and returning a Id.

Step 7. Deploy to the cloud, remove the local pointer from your hosts file, and update your Live Services returnUrl to not include the port 81 reference. (it should look like http://xxxx.cloudapp.net/webauth-handler.aspx)

To get everyone started, here is a quick sample project that has all the bits in there. You just need to update the web.config with your Application ID and secret from the Live Services page for your project in the Windows Azure portal.

Ping me with questions/issues.

Enjoy :)

Technorati Tags: Windows Azure,Live ID SDK

Tracing Windows Azure Apps to Storage

davidlem — Mon, 24 Aug 2009 23:05:10 GMT

So one of the things I like to do, all of the time, is trace. It started when I was a little child, and the rest of the kids could draw really good and stuff, and I couldn’t, so I would get pictures of horses and trace around them…

Now that I’m older, I still can’t draw, let alone code, very well, so I still trace.

In my Windows Azure apps, I like to trace to my own log files, it’s not the RoleManager.WriteToLog isn’t good, but I have code from other apps that already uses the trace infrastructure and pragma concepts, so having calls to Trace.WriteLine simply work as normal is a big attraction.

So how do you do this?

First, simply create your own custom trace listener and have it write to Windows Azure Storage.

Then load it into your role.

And now when you call the Trace.WriteLine for example, it will call your custom trace handler too.

Also, the sample below also has a little UI web role so you can see what’s being logged!

You can find the zip archive of the project here. I’ve scrubbed the .cscfg file so you can add your own links into it for your storage account and test URL for the pic it downloads.

Enjoy :)

Save Your Word Docs to Windows Azure Blobs!

davidlem — Sun, 19 Jul 2009 22:10:40 GMT

So Mohit and I were at WPC last week, and while we were hanging in the booth, a customer came up with a very interesting problem.

Essentially, the customer needed to be able to have their Word documents saved to Windows Azure, in a format that could be opened in Word, but also consumed by a legacy Java application.

I promised I would see how hard it would be, and so while waiting for Yil to cook Sunday lunch, I had a quick crack at it.

First step, I created my own ribbon in Office, using the new Office Add-in projects in VS 2010.

Added in a ribbon visual designer.

Which ended up looking like this.

The code was very easy, essentially on the click event, I saved the contents of the doc to a blob.

The key part is to call the WordOpenXML method on the ActiveDocument.Content property, to get the OpenXML version of the doc.

A quick test.

And… you little bewdy. Now, I’ve set the content-type to the docx content type for dramatic effect, as otherwise the XML would just load in my browser, and not let me save it. When I save it to my desktop, I save it as xxx.xml, and when I double click on it, it loads in Word.

The next step for this would be to create a Web Role to list the documents that have been saved to the blob store, to make it nicer to view, but in terms of proving this little bit of functionality, I’m pretty happy.

Here is a link to the code in case you’re interested.

Enjoy :)

Technorati Tags: Windows Azure,Blobs,Office 2007,VSTO

WPC09 – My Windows Azure Session

davidlem — Fri, 17 Jul 2009 04:50:00 GMT

Here is my session from the WPC09 in New Orleans:

Lap around Windows Azure, Business Edition

Big thanks to Eilert Hanoa from Mamut ASA for his help with the session, and to everyone who attended. :)

Hope everyone enjoyed WPC! :)

Technorati Tags: WPC09,Windows Azure

Windows Azure Project Attire!

davidlem — Thu, 09 Jul 2009 19:24:19 GMT

So one of my favorite parts of working in the Windows Azure team is the camaraderie that grows when you take a bunch of really smart people and apply them to a big problem. I’m very fortunate to be part of three extremely interesting Windows Azure v-teams (teams assembled for a particular project or work stream); our TAP team (I’ll explain a little more about that in a later post), our billing team, and our deployments team.

One thing that I have noticed during the course of this year as we work towards PDC2009, is that each team has it’s own dynamic, it’s own challenges and issues, and most of all, it’s own in-jokes. Along the ship path, as the pressure increases, it’s easy to forget to have fun, so the other day, I jumped onto Zazzle, and captured some of the more interesting moments on our road to PDC09 so far.

Filing Bugs Is Free!

My first one comes from what we call a ship room, which essentially is a regular meeting (we hold ours twice a week for an hour) where everyone involved in shipping the next release (in our case, billing) attends to triage bugs from the previous release, and discuss blocking issues for the upcoming release. Everyone attends; developers, testers, program managers, release managers, everyone. As the billing team is relatively new, not everyone had worked together. Now, as every dev/test knows, filing bugs is not a trivial task, there is a tension between overfiling (carelessly filing bugs which adds work to the triage team and can cause wasted effort) and underfiling (not wanting to file bugs because it may make the developer look bad, however has the effect of giving an incorrect representation of how solid the codebase is). It’s a tension that exists in every project, and there is no right or wrong balance, you have to take a position that suits your objective. Entering billing ship room, there was definitely a tendency to underfile, but given the importance of billing, we decided that it was better to overfile, wear the cost of triage, to ensure we didn’t miss any bugs, than underfile to optimize for speed of delivery. Hence, the shirt below!

My Rack’s Too Fat for Singapore!

The second one comes from my experience in our infrastructure deployments team. This team is responsible for deploying all new Windows Azure capacity to support the cloud, which includes selecting hardware/network gear, designing the deployment topology/config, rolling out and setting up the physical assets, and wiring it all up to the Windows Azure grid, both from a wire and software point of view. As you can imagine, it’s a pretty fast paced environment, and I can only imagine, is akin to landing planes; you need to get lots of iron and copper on the ground, in different countries, very very quickly, in quick succession. The other aspect is that we don’t use your standard enterprise configurations, which means are racks tend to be tall, heavy, and many. So it some times happens that a datacenter location may not be able to support our deployments, which was the case with a location we were looking at in Singapore. I couldn’t resist capturing that little moment, in a very literal way in the shirt I’m wearing today!

So keep this in mind, next time your sitting with your project crew, toiling through the hurdles and issues, and that little moment pops up that makes everyone have a laugh and relax, capture it in some way, shape or form. It’s a great way to remind everyone that we do this stuff because we love it, and it also helps keep those moments alive down the path, where trust me, you’ll need them! ;)

Enjoy! :)

Technorati Tags: Work Stuff

Looking Inside Windows Azure!

davidlem — Mon, 06 Jul 2009 09:59:42 GMT

So I’ve been working on a proof of concept this past week, and one of the biggest issues I’ve had has been trying to see what has been going on with my “container” when I deploy my app to Windows Azure. It’s not that the logs aren’t good enough, but sometimes I need to try interactive commands to see the state of the environment or test something out.

First, let’s recap a little Windows Azure 101; when you deploy your application, we spin up a VM and deploy your app into that VM. That VM has some policies set, which restrict what you’re able to do compared to your local machine or dev server.

So how do you see what your VM is doing? Well, since we don’t really provide any option to TS (remote admin) into your VM instances, I had to come up with a slightly different approach. The first step was to flick on a little known setting, but very powerful one:

Enabling Native Code Execution turns off the partial trust policy in Windows Azure, allowing you to run a bunch of functions that you were previously able to run in the old Nov CTP 2008 days.

My next task was to code up a little page that would allow me to run commands in the VM environment, kind of like a remote console, so I could see what was happening when I was trying to run my app.

The main bit of code is quite simple, and uses the System.Diagnostics.Process to create a new command window to run my commands, and capture the output back into my ASP.NET page. Here is the code to create the command process and get the return data:

newProc.StartInfo.UseShellExecute = false;
newProc.StartInfo.RedirectStandardOutput = true;
newProc.StartInfo.FileName = "cmd";
newProc.StartInfo.Arguments = "/c " + txtCommand.Text;
newProc.EnableRaisingEvents = false;
newProc.Start();
sr = newProc.StandardOutput;
txtStatus.Text += String.Format("{0}\r\n", sr.ReadToEnd());
newProc.Close();

So I have a simple page now, where I can enter commands, and see what is returned when run within the VM:

Oh the things one finds out about their VM! :) For example, I can run commands like:

SET

NETSTAT

TASKLIST

This type of insight is really important, for example, in my task, I was trying to start a new process (Apache Tomcat), but couldn’t get a response from the default address that Tomcat binds to (http://127.0.0.1:8080), and couldn’t work out why it worked on my local dev fabric and not in the cloud. I then ran NETSTAT and saw that the 8080 port was already bound to a virtual address, hence why it wouldn’t work for me.

Anyway, there is a wealth of information that can help you as a developer, when trying to debug/troubleshoot an issue, that can only be gleaned from the live environment, so tuck in!

Enjoy! :)

Technorati Tags: Windows Azure

Process Improvement Starts with a Blank Page!

davidlem — Thu, 30 Apr 2009 21:08:02 GMT

So anytime you’re doing something more than once, there is an opportunity to do it better every consecutive time. Over my career, I’ve worked in a number of different project teams; software teams, hardware teams, software/hardware teams, even a marketing team!

One thing that has been consistent about my experience in teams though, is that if you are trying to get many people to work efficiently on one project, you need a process. Now, I know there are those who don’t favor process, and I agree with them that a bad process is just as destructive as no process, but a good process, that’s a different story. And also, sometimes, there isn’t any process in place whatsoever, and I don’t mean there isn’t any processes in existence, it just that the team and project you’re working on hasn’t adopted one.

So where do you start? Well, I have some principles I’ve always used to guide me thinking.

First, start with a blank sheet of paper. I have a philosophy that by even having a blank sheet of paper, that represents your process, you’re already off to a good start, because you now have a process to improve on. But if you don’t even have a blank piece of paper, you haven’t got anything to improve on, so you’re doomed to keep starting from scratch and feeling your way through every project, painful ad-hoc step by painful ad-hoc step.

Second, think logically about the stages/phases of your process, and the functional groups involved. For example, let’s say your trying to build a new product, you can start simply by saying, what are the big steps we will go through as a team, and what are the functional roles that the team will play. This is important if you want to avoid having everyone involved in every part of the project. This kind of high contact leads to burnout and failure, as everyone is always on, even when they don’t need to be, and your project ends up looking like a rave party than a slow waltz. So an example process sheet would look like (I like to use Visio Cross Functional Flowcharts):

Next, you put a simple box in each intersection where a process activity needs to occur. Now, I never put more than a single box in an intersecting square. The purpose of the chart above is to highlight a work interaction between a functional team and a phase, you can drill down into the actual deliverables and steps later. So my process might look like this.

So for example, in the Requirements phase, the business team are responsible for building the customer requirements. To start with, you may not have any steps, so at the project kickoff, you would start by defining the steps for this project to satisfy that part of the process. Next time around though, you could look back at the last project and say, “Wow, out of the 3 things we did, the focus groups worked really well, so let’s add that to the best practices list for Customer Requirements tasks”. Again, it’s not about forcing people to follow anything more than the high level process, but you also want to capture the prior learning, and express it in a way where someone can look over the task list for a process stage and say, “This was used to get this outcome, and I think I need that outcome for my project, so I’ll use that task”, but also enable people to add new tasks to the catalog.

The next step is you have to have a basic communications plan. In too many projects, emails fly around between project members and other connected people, with everything from FYI’s to work items and change requests. It’s dangerous because the wrong people can affect the outcome of your project, and critical items get lost in the email explosion. Set up some basic email aliases for your project, at a basic level, have 3:

1. A basic project comms alias for general FYI comms

2. A change request alias where requests can be managed

3. An oversight/stakeholder alias where critical decisions that need stakeholder approval/review can be sent

You should also attach a contract to each alias, for example, for the oversight alias, you might say that all the people on that alias commit to responding to any email sent to that list within 24 hours. This way, the alias doesn’t just become an vacuum or black hole, and people don’t lose faith in the alias and start emailing people directly.

And finally, you should have a clear accountability matrix. This ensures that the right people are involved in the right parts of the process, and more importantly, the wrong people are not involved. You matrix should map to your functional groups from your process chart, and should follow a model such as OARPi (Owner, Approver, Reviewer, Participant, Informed). This way, everyone knows when they are involved, and in what capacity. And also, they know who is not involved, so they can filter those people from the process. It really improves the signal:noise ratio for a project.

And that’s pretty much it. Each time you runt he process for a project, take the time at the end of the project to reflect, then update the body of knowledge for the process, highlighting tasks that worked really well in a phase for a functional group, and also tasks that didn't’, and include enough information for the next person so they know what you were trying to achieve, and what the outcome was.

Process is good, process is your friend! :)

Technorati Tags: Stuff

Build the right feature, build the feature right!

davidlem — Sun, 26 Apr 2009 09:56:14 GMT

So one of the key responsibilities of a Program Manager at Microsoft is to drive features. Features are the same as functions in some software nomenclature, and essentially, they map to something the software performs to satisfy a customer need.

Now, over the course of my career as a software engineer, I’ve seen two major disconnects in product teams. The first is where product teams never seem to be able to pick the right features to build. They have great talent, amazingly smart people, lots of resources, but just no direction, so they end up with awesome software that no one wants. I’ve also seen the opposite, teams with great feature lists, but a lack of diligence and professionalism required to ship quality features, so they build a product everyone wants but no one can use or will pay money for. This has led me to distill a little saying which has always held me in good stead: build the right feature, build the feature right. What does that even mean?

Well, knowing what features to build is an art in itself. When building a global product, it’s not easy to simply ask your customers what features you should build. Instead you have to use different techniques to ascertain what features are valuable to them, and which are not. You need to take into account multiple inputs, such as focus groups, customer visits, user feedback, market movements and pressures, and insights into areas of possible innovation. By doing this, you make sure that all the hard work the developers will do to produce your feature, is worthwhile not only to the customer, but to the business too.

Now once you’ve worked out the right features, you need to work out how to build those features right. See, this is where I’ve seen many a software group come unstuck. Picking a good feature is only half the battle, delivering it with quality and style is another. For example, in order to ship a great feature, many factors need to be considered, such as how the customer intends to use the feature, how the customer would like to use the feature, the best way to design the feature so the customer can be efficient and comfortable in using it, and can leverage existing skills and experience to be productive. It’s not good enough to simply cut some code, there needs to be a quality process from design through to deployment to ensure the feature is rock solid and appealing to the customer.

Shipping great software is about embracing both of these aspects, not just one of the other.

Technorati Tags: Building Products

What is Software + Services?

davidlem — Fri, 10 Apr 2009 19:02:01 GMT

I came across this video today and really found it simple yet powerful.

For me, the best part of this video is it puts a real context around the whole S+S story, which to date, has been pretty hard to decipher.

The main takeaway as I see it, is that you can’t achieve everything you need from just desktop/server software or the cloud, you need a little bit of both. Why not have your cake and eat it too I say! ;)

Technorati Tags: S+S

Design for Failure, Growth and Distribution!

davidlem — Tue, 24 Mar 2009 17:06:41 GMT

When designing any piece of software, there are always going to be design tenants that you practice based on the type of software being developed.

For service software that is designed for a global, business critical audience, there are three key tenets in my opinion. In fact, there not so much tenets as they are assumptions.

So when you start designing your service(s), you need to design with the assumption that your service is going to do the following, and do it on a regular basis:

Fail
- This does not only mean your service will fail during runtime execution, but also during upgrades, deployment of new functionality, bug fixes, configuration changes to the environment
- Your operating environment will fail. Power will go out, network appliances will die, hard disks will crash, CPU and Memory will fault, assume it will all happen, and probably all at once
- In a standard data center deployment, you need to work out what your minimum unit of failure is. Is it a rack? Is it a switch? The data center? Knowing this helps you work out how to distribute your service to counter failures.
Grow

Your application is going to need to serve lots and lots of requests, so think about the scale of your application. Not only scale out (as in, the ability to serve lots of incoming requests simultaneously) but also scale up (make sure your application can dole out capacity)
Bandwidth and Ingress/Egress is also a concern, pipes in and out of your service are constrained by physical limitations, so you’re going to have to think about how to deal with heavy load on your network and how your application will deal with this during peak and off-peak periods
At some point, you service is going to exceed the place its located, as in, there just won’t be any more machines to deploy your service to, and there won’t be any more space to deploy more machines, so you’re going to have to design your service with the goal that you can deploy it across many servers across many data centers

Distribute
- Your customers invariably will want your service closest to them, and they will invariably never be located closest to your primary deployment data center, so design your service in a way that it can serve your customers at the point that closest to them. It reduces latency and improves service quality and experience

Also, designing a service involves thinking through the whole application lifecycle. Traditionally, we think about issues and bugs at the runtime level, but with services, because they are always on, living, breathing systems, you have to design them in a way that they can be changed (upgraded, configured differently, etc) at any time without interrupting its availability (the ability for customers to connect and use your service) and its consistency (the state maintained by the system must always be consistent and correct).

Now, this is by no means the magna carta of service design, but as you start thinking about designing your always available, highly scalable, geo-distributable service, the tenets above should be good food for thought.

Enjoy :)

Technorati Tags: Design,Services

Windows Azure: March 2009 CTP!!!

davidlem — Thu, 19 Mar 2009 23:41:45 GMT

OK folks, so all the announcements have been made for Mix, so it’s time to get cranking!

We have a new release of tools and SDK bits.

First, you’ll need to install some pre-reqs, but only if you are going to exploit some of the new capabilities:

Hotfix: Native Debugging Improvements (Tools only; and only if you intend to do native debugging)

Hotfix: Support for FastCGI on the Development Fabric (SDK and Tools; but only if you intend to use the FastCGI capabilities)

IIS7.0 URL Rewrite Module (SDK Only; and only if you are using URL rewrite rules)

If you don’t plan to use native code, FastCGI or URL rewriting, then don’t worry about the pre-reqs above.

Then, the tools and/or SDK. One big thing to note about the March release is we now have two install options:

If you don’t use Visual Studio, then you can download the standalone Windows Azure SDK here, or
If you use Visual Studio, then you can now get an all up VS Tools for Windows Azure and Windows Azure SDK installer here.

Enjoy!

Technorati Tags: Windows Azure,SDK