Ben Riga's Deep Shift

In this episode of “Azure Lessons Learned” Rob Fraser from RiskMetrics talks about the work they’ve done with Windows Azure to scale some of their heavy computational workloads out to thousands of nodes on Windows Azure.

RiskMetrics specializes in helping to manage risk for financial institutions and government services.  The solution they built on Windows Azure is primarily for calculating financial risk for their clients.  Calculating the risk on portfolios of financial assets is an incredibly compute-intensive problem to solve (Monte Carlo simulations on top of Monte Carlo simulations).  There is an ongoing and increasing demand for this type of computation.  RiskMetrics calculations require enormous computational power but the need for that power tends to come in peaks.  That means the required hardware is idle for much of the time.  Windows Azure solves this problem by allowing RiskMetrics to quickly acquire the very large number of required processors, use them for a short time and then release them.

Channel 9: Windows Azure Lessons Learned: RiskMetrics

To give you a sense of the scale RiskMetrics is talking about, the initial target is to use 10,000 worker roles on Windows Azure.  And that’s just a beginning as Rob thinks they could eventually be using as many as 30,000.

While using Windows Azure may help control costs, the real motivation is having the kind of compute power they need to build analytic services for their clients that they just wouldn’t otherwise be able to do easily.

Rob goes in to some depth on the architectural pattern they devised to ensure the efficient flow of work packets from their data center into the cloud for processing and then back again with the results.  The architecture is an interesting hybrid of on-premises and cloud computing.

Rob (along with his colleague Phil Jacob) also presented some of this in a PDC session.

I’m building a Windows Media Center for a home theater setup I’m planning and in the process was installing a new hard drive into the PC.  Before I dropped it into the new machine I was showing it to my son and pulled out an old drive I had for comparison.  I was a little taken aback at the contrast of the two drives below.  The one on the left is a 10 megabyte (MB) Seagate ST-412.  The one on the right is a 1 terabyte (TB) Western Digital Caviar.  In case you’re having trouble doing the math in your head, let me help you out.  1TB is roughly equal to 1,000,000MB. So the drive on the right holds roughly about 100,000 times more data than the one on the left.

Click to see the full size photo

That Seagate ST-412 on the left is pretty old.  I’ve had it since forever and just use it as a bookend.  For those interested in the history of that drive, that was the first hard drive used in the original IBM PC XT back in 1981.

Good times, good times…  <grumble grumble>

In this episode of Lessons Learned I chat with Rex Hansen of ESRI.  Rex works on MapIt; a product for visualizing enterprise data on maps.  This was recently released as a on-premises product that enables developers to work with the tabular and spatial data in SQL Server 2008 and integrate that data with maps on ArcGIS online and Bing Maps. 

ESRI has been working to extend that functionality to Windows Azure and SQL Azure.  MapIt takes advantage of SQL Azure to consume location-based data.  The MapIt spatial data service can be deployed as a role on Windows Azure and provides spatial data capabilities to applications using SQL Azure.  This provides a valuable service to folks that miss the spatial data types they were used to using in SQL Server.

Channel 9: SQL Azure Lessons Learned: ESRI 

Rex walked me through building a Silverlight application with their Silverlight Control Toolkit in Expression Blend.  As Rex mentions in the video ESRI has released the source code for their Silverlight toolkit controls.  You can find those on CodePlex here: http://ESRISilverlight.codeplex.com/

It sounds like ESRI has some big plans for where they want to take the MapIt product to provide even better integration with SQL Azure.

You can find out more about the ESRI MapIt product here.

Telerik is one of the more popular component vendors building good stuff for ASP.NET, Silverlight, and WPF.  I was intrigued by the interest they have taken in the Windows Azure Platform.  Naturally all the UI components just work on the platform with no changes required.  Telerik has gone further in looking at how they can adapt their Object Relational Mapper (ORM) and Content Management System (CMS) technologies to take advantage of the benefits of Azure.

Stephen Forte is their Chief Strategy Officer (love that title :)) responsible for thinking about their new technology directions.  In this episode of Lessons Learned, Stephen shows me how they’re now able to go from SQL Azure tables to persistent classes when developing applications using their OpenAccess ORM product.

Channel 9: SQL Azure Lessons Learned: Telerik

The process of implementing a SQL Azure provider for OpenAccess was fairly straightforward.  Telerik ran unit tests of their existing SQL Server provider against SQL Azure to find the differences.  They found the few minor issues and were able to get a working version running in three weeks or so.

The biggest difference they found between SQL Server and SQL Azure was the data types.  SQL Azure supports most of the SQL Server data types but some (like spatial, text, etc) are not supported.  Another difference was multiple active result sets (MARS).  This one was a little trickier and with some work they were able to get the same end-result using different t-sql.

The OpenAccess ORM product is available now.  Telerik is also dogfooding it internally using another of their products: Sitefinity CMS.  The next release of Sitefinity will then support SQL Azure as a data store.

A few links mentioned in this episode:

• Telerik team blogs are here
• Stephen’s blog is here
• OpenAccess ORM product page
• Sitefinity CMS product page

In this episode of Azure Lessons Learned I chat with Dmitry Sotnikov, new product research manager with Quest Software.  Quest, as you probably know, is a huge global ISV focused primarily on Systems Management software.

Quest is an early adopter of the Windows Azure platform.  They’ve been working on a new offering for their various management business.  They’ve built out an extensive services framework as well as a few service offerings on that framework. 

Channel 9: Windows Azure Lessons Learned: Quest Software

Quest has hundreds of solutions for the enterprise.  These are your typical on-premises that would normally require hardware and people to install and maintain those solutions.  The Quest OnDemand project Dmitry is working would extend those offerings to the small and medium business by making many of the solutions available as subscription services.  The first 3 offerings are Recovery Manager OnDemand for Active Directory, InTrust OnDemand event log management, and Site Administrator Reports OnDemand for SharePoint.  These are now available in beta.

Dmitry walks us through the experience of setting up Recovery Manager OnDemand to provide backup and recovery of AD.  The solution running in Windows Azure does most of the work in the cloud while sending requests to the local infrastructure via agents that are running locally.  There is no need to set up servers or install anything beyond that simple local agent service.

Dmitry was very open about the process they went through to adapt their existing solutions to this new subscription model.  He mentions that they were able to re-use about 50% of the code for one of their solutions (AD Recovery Manager).

Since Quest OnDemand is meant to be on solution with many different component offerings it was important that there be a consistent framework across all those offerings (including login authentication/access control, portal, subscription/payment/billing).  Even within Quest I think they were a little surprised as to how fast they could port applications over once that framework was in place and available to re-use.

Dmitry also goes into detail about the architecture they have built out for this set of services including the portal, Windows Identity Foundation (formerly Geneva) Security Token Service (STS), and the individual services.  He also is very open about what they learned along the way of this development.  It’s interesting to note that the bulk of the Quest products are on-premises so although they had many large installations there had never been a call for super-scalable super-secure multi-customer/multi-tenant deployments like the one required by Quest OnDemand.

I love that Windows Azure played such a big role in helping Quest to adapt the way that they do business.

Have a look at the Quest OnDemand Service here:

• http://www.quest.com/ondemand/

I’ve been thinking a lot about Windows Mobile lately (more on that, a lot more, later).  As I looked at the device I’ve been carrying for the past year or so (Samsung Blackjack II) I’ve found a few things annoying so I recently set about fixing things.  I’m sharing the solutions here for others who might want to fix the same issues. 

Caution: These are fairly advanced customizations to be making.  If you’re uncomfortable with that sort of thing then just put the phone down and slowly back away as you’re liable to brick your phone.

BTW, I used PHM Registry Editor to makes these fixes.  Also, your phone needs to be “Application Unlocked.”  I used the SDA Application Unlocker to do that.  Again, both of these are advanced tools so be careful.

Leaving on a Jet Plane

The single most annoying thing about the Samsung Blackjack II are the Startup and Shutdown sounds.  I can’t imagine what the product manager was thinking when this feature was put in: “Hey, how about if it made a loud sound like, maybe a jet engine, whenever you turn it on or off?  Yeah! Users will love that.”  I find it incredibly annoying in many situations.  Apparently I’m not alone as there are numerous posts about this on the intertubes.

Here are the instructions to fix this:

1. Use your registry editor and get to:  HKey_Local_Machine>system>
2. under system there are two entries: Shutdown and Startup
3. under each one you will find an entry named 1
4. under 1 there is a value name of Dll
5. the default value for both entries is OemAnimationDll.dll
6. change the default value data of each of those files (startup and shutdown) to this: *none* (include the astericks...this helps the OS distinguish that it is not a specific file)
7. after changeing the value to *none*, back out to the main menu - wait a couple seconds and shut down and then restart, and you will find the annoying shutdown and startup screens and audio are gone.

Can I get a menu over here?

Another annoyance is more of a personal preference.  I’ve been using what used to be called the “smartphone” edition of Windows Mobile.  It’s now called Windows Mobile Standard.  The big difference is that the screen is not a touch screen.  I’ve always had a preference for a phone that I could use one-handed and had good physical keyboard so I’ve had a non-touch device since 2004.  In the previous editions the menus all had numbers (aka accelerators) next to them to make selection easier.  On the Blackjack the menus are selected by characters which, for me, is a little more difficult. 

To change that option here are the instructions:

1. Use your registry editor and get to:  HKEY_CURRENT_USER\Software\Microsoft\Shell
2. Change the value "HasKeyboard"
   • '0' will show 1-9 accelerators, and '1' will show letter-based accelerators.

Holy Crap!

The final annoyance is one inflicted by AT&T.  Like many organizations their margins are squeezed and so look for any opportunity to sell their value-add services.  AT&T has a whole slew of these that they’d love to sell me (GPS, shopping, music, video, etc) but that I have never used and do not intend to.  Unfortunately they have hardcoded the 6 services near the top of the “All Programs” menu so I have to scroll past them every time I’m looking for a program. 

Here’s how to remove those options:

1. Connect the device it to your Windows PC using ActiveSync.
2. Browse to your Blackjack 2 using Windows Explorer
3. Navigate to your Blackjack 2’s /Windows/Start Menu/
4. You will see a file listing of all the Folders and Shortcuts displayed in your Start Menu.
5. Select the Folders and Shortcuts you wish to remove and delete.
6. Your Blackjack 2’s Start Menu is now crapware free

In this episode of Lessons Learned we talk with Jones Pavan and Gurleen Randhawa of CCH about tax and accounting!  No wait stick around, that stuff can be exciting too.  Yes, really!  :)

The good folks at CCH (a Wolters Kluwer company) have built an interesting service on Windows Azure.  The solution we discuss here is a sales tax calculation service which they offer to other accounting firms.  This is an existing on-premises product that they are now moving to the cloud.

Channel 9: Windows Azure Lessons Learned: CCH

The existing product was a stateless web service that was designed to live behind the firewall.  The service is meant to be called directly via a plug-in in an accounting firms ERP system (for example, Dynamics AX).  To move that to the cloud CCH wrapped the web services in Windows Communication Foundation (WCF).

They had been using another third party RAD development tool called CA Plex.  The Plex runtime was added to the project and copied out to the cloud.  One of the things they quickly learned is that the nature of the cloud app is to be stateless and that required special consideration when moving on-premises apps (for example the Plex tool was caching db connections behind the scenes).

Anther important consideration was security.  They were not ready to move to ACS so for the initial release they used X.509 certs, ADFS and message-based security to establish trust relationships with the server.

BTW, the Windows Azure marketing folks have already published a case study on the CCH solution (available here).

There are not many solutions that can claim to have saved lives.  In this episode of Lessons Learned I chat with Richard Prodger of Active Web Solutions about the Windows Azure project they’ve been working on that tracks fishermen in real time.  It monitors not only their location but also their status so as to immediately raise the alarm if help is needed (e.g. fallen off the side of a boat or pressed a panic button).  This solution is already credited with saving the lives of 9 fishermen.

Channel 9: Windows Azure Lessons Learned: Active Web Solutions

Electronics on the fishing vessels communicate directly via satellite to the Windows Azure solution.  Those messages are processed via Windows Azure worker roles and routed using the Windows Azure AppFabric Service Bus to various on-premises systems for review and action.  The desktop client overlay marine charts onto Bing maps so that the coast guard gets a visual representation of the exact location of boats that have raised alarms.

The good folks at Active Web Solutions have published some of the source code that they developed to “automatically bridge arbitrary TCP endpoints, handling any intermediate firewall traversal.”  The code is available on CodePlex as the SocketShifter project:

http://socketshifter.codeplex.com/

If this is interesting, you should also have a look at Port Bridge published by Clemens Vasters on his blog.  Clemens describes it as “Socketshifter’s older brother”

http://blogs.msdn.com/clemensv/archive/2009/11/18/port-bridge.aspx

One question that is often asked is how hosters can benefit from the Windows Azure Platform.  While the platform can be used to deploy many types of web apps we expect many partners including hosters to develop on top of the Windows Azure platform infrastructure.  In this episode of Azure Lessons Learned I chat with Paul Lappas, VP Engineering at GoGrid and Mehul Shah and Madhavrao Pachupate from Blue Star Infotech.  GoGrid has been working on a hybrid solution that builds on the GoGrid infrastructure to assist in development and load testing of Windows Azure applications.

For more information on the GoGrid solution for Windows Azure have a look here: http://www.gogrid.com/azure/

In this episode of Windows Azure Lessons Learned I chat with Paul Forney, System Architect for Invensys and Aleksey Savateyev, Senior Architect in Microsoft’s Global ISV group working with Invensys.  Invensys is well known for industrial automation and control systems.  They’ve been working to develop a system for the power industry to manage the large network of smart meters that will be used to build out smart grids delivering electricity from suppliers to consumers.  To do this Invensys is using Windows Azure AppFabric (formerly called “.NET Services”).  The AppFabric Service Bus is the magic that allows this type of application.  It allows those meters not only to connect across the cloud to on-premises systems but also does it a way that can scale to the millions of homes and businesses that will form the smart grids.

Channel 9: Windows Azure Lessons Learned: Invensys

Database tooling is important for many developers and DBAs as they manage numerous databases across the enterprise and the cloud.  In this episode of Azure Lessons Learned I chat with Scott Walz, Sr. Director Product Management at Embarcadero Technologies responsible for the DBArtisan product.

Scott walks us through the DBArtisan product to show how SQL Azure integrates seamlessly into this cross-DBMS product.  It was interesting to hear how quickly the effort to add SQL Azure went.  I think that bodes well for other tooling in general for SQL Azure.  Since SQL Azure is so very close to SQL Server it should be relatively simple for ISVs to add SQL Azure support to products that support SQL Server today.

Channel 9: Azure Lessons Learned: Embarcadero

Scott mentions a DBArtisan download for SQL Azure.  You can grab that here.

As always let me know your comments below.

I’m back with another episode of Azure Lessons Learned.  In this one I discuss building apps that need to be able to scale very highly very quickly.  Jim Zimmerman, the CTO and Lead Developer for Thuzi, walks me through the Facebook application his team built for Outback Steakhouse.

Thuzi specializes in building social media applications for large organizations to connect with their customers on sites like Facebook.  The concern with these types of applications is the possibility that the offer will go viral as it gets passed from customer to customer and potentially swamp the servers to the point that they are not responsive and ultimately disappoint customers.  Thuzi chose to use Windows Azure since the Windows Azure platform has ample capacity and they could scale up or down the solution based on the current demand of the market.  The Outback Steakhouse offer in this case was a free Bloomin’ Onion at any of the thousands of Outback Steakhouse restaurants.

Facebook apps don’t actually run on Facebook.  They are embedded using an iframe into a Facebook page.  The app that is running in that iframe must be hosted somewhere else.  In this case, Thuzi hosts that in Windows Azure. They actively monitor the campaign and turn on or off web and worker role instances as required.  In order to scale they used Windows Azure table storage and queues.  They also use SQL Azure to perform reporting and analytics on the results of the campaign.

Channel 9: Azure Lessons Learned: Outback Steakhouse Facebook App

In this episode of Azure Lessons Learned I chat with Stephan Friedl, Chief Architect at Quark Software.  Quark (of Quark XPress fame) has been built a new business called Quark Promote for small and medium business to design and print high-quality collateral (brochures, business cards, postcards etc) to promote their business. 

In and of itself this is an interesting Software + Services solution built with a compelling WPF design client and a high performance ASP.NET server.  Quark chose to deploy this solution using Windows Azure.  In that way they could build out their business to handle the numerous relationships they’ve setup neighborhood printers.  The architecture is service-based specifically so they could handle these type of relationships and host the solution on the partner’s site and in fact host on multiple sites from that same single multi-tenant solution running on the Windows Azure platform.

Channel 9: Azure Lessons Learned: Quark Software

As I mentioned, over the past few months I’ve been working on a number of activities related to the Windows Azure Platform.  In particular, I’ve been working with several partners as we prepared for the PDC’09 conference.  While we were preparing for the conference we welcomed a few partners to a deep-dive event in Redmond where they did some architectural reviews and met with various members of the product team in the final sprint to releasing solutions.  While they were in Redmond I took advantage to record a few videos for Channel 9.

Kelley Blue Book stands out as they were featured on the main stage during Bob Muglia’s keynote on day 1 of PDC (Andy comes on at about 1:27).  In this video with Andy Lapin, Director of Enterprise Architecture spent a few minutes showing off the site and then discussing some of the lessons the KBB team learned as they ported their site from a hosted facility to the Windows Azure Platform.

Channel 9: Azure Lessons Learned: Kelley Blue Book

I had been maintaining a list of Dynamics CRM-related blogs that I tried to read on a regular basis.  I thought I’d make that list available to folks in the community as I’m sure others will find that interesting.  I haven’t been maintaining this since I moved out of the Dynamics world a few months ago.  These are primarily developer-focused.  By that I mean they are mostly blogs that discuss developer CRM/xRM topics as opposed to end-user CRM topics. 

I’ve divided these into the following categories:

• Microsoft Employee Blogs
• Community Blogs
• Non-English blogs

I’ve put this into an OPML file so that you can import it into pretty much any modern RSS feed reader.  You can grab the OPML file here: