Accelerating Things

Dryad Beta Now Available

[MSFT] Jeremy Espenshade — Wed, 04 May 2011 03:03:22 GMT

Dryad, the distributed data computation framework originally developed by MSR, has now been integrated with HPC Server. You can now use the LINQ programming model in C# to develop applications that scale out across a cluster and handle large amounts of distributed data.

Beta bits, documentation, and samples are now available on the Connect site.

AppFabric Caching and HPC

[MSFT] Jeremy Espenshade — Fri, 11 Feb 2011 18:52:08 GMT

I recently published a white paper about how to integrate Windows Server AppFabric Caching with Windows HPC Server 2008 R2 to provide an in-memory cache capable of supporting the data demands of many distributed applications. The paper walks through the setup and configuration of an AppFabric Caching subsystem, the development of an application leveraging HPC's Service Oriented Architecture (SOA) development model, and the performance measurements motivating the use of AppFabric Caching. In addition, a sample code package provides a reference solution which can be extended by developers.

You can download both the white paper and sample code via the above links or get the whole package by downloading the "CommonDataCaching.Sample.zip" from the Windows HPC SDK page.

HPC Services for Excel Video

[MSFT] Jeremy Espenshade — Fri, 01 Oct 2010 23:19:09 GMT

We've put together a really slick video which goes through what the HPC Services for Excel does and why it's useful. All in less than 5 minutes! The production values are great and it introduces the feature in a way that's interesting and easy to follow.

If you're even casually interested in how Excel can work with an HPC cluster, this video is worth watching. Find it from the HPC site or access the video link directly.

HPC Launches on Wall Street

[MSFT] Jeremy Espenshade — Mon, 20 Sep 2010 17:49:00 GMT

Several of my HPC colleagues are in NYC this week for the official launch of Windows HPC Server 2008 R2, which includes the new HPC Services for Excel among other great new features. Check out the latest and greatest from HPC and Microsoft's new Technical Computing initiative here.

Instant Search is Easy

[MSFT] Jeremy Espenshade — Fri, 17 Sep 2010 17:15:01 GMT

Apparently some people thought Google's new "Instant Search" feature, where searches are performed as you type, was pretty slick. Then they wrote their own using the public Bing APIs.

You can play with it at binginstant.com.

It's always funny when features touted as revolutionary turn out to be really simple to throw together.

Loading Excel Add-Ins at Runtime

[MSFT] Jeremy Espenshade — Thu, 16 Sep 2010 21:37:00 GMT

When automating Excel, the Excel.exe process should be started using the '/automation' switch. This is done automatically when creating a new Microsoft.Interop.Excel.Application object, however if you want to keep track of the Excel processes created you may have to create the Excel process and then get a reference to the application associated with that process. In this case, the /automation switch must be applied manually.

Starting Excel with the /automation switch tells Excel, among other things, not to load any of the files in the XLStart directory or open any Excel Add-Ins. COM add-ins installed for all users are still loaded.

This streamlines the launching of Excel and makes sure that no unnecessary code is running during automation. There are important cases where you may want to have a specific Excel Add-In like an .XLL or .XLAM loaded however. For example, you may have some library functions you want to leverage defined as UDFs in an XLL or VBA functions in an XLAM.

The following options are available if you want to do this.

1.) Application.RegisterXLL
a. This is a method which can be called from VBA which loads an XLL at a specific location and registers the functions and commands contained in the XLL.
2.) AddIns.Add
a. This is a method which can be called from VBA which loads any type of add-in (XLL, XLA or XLAM). After loading the add-in, perform step 3 to open it.
3.) AddIn.Installed = true
a. Once you have a reference to a loaded add-in, set AddIn.Installed = true to cause the add-in to be opened.
b. Note that add-ins that are known when Excel is started with the /automation switch will already be marked as "Installed", but they are not opened. In this case, Set Installed = false before setting Installed = true

Private Sub Workbook_Open()
    Dim success As Boolean
    Dim myAddIn As AddIn
    
    ' Load XLL
    success = Application.RegisterXLL("c:\myaddins\myxll.xll")
    
    ' Load and install new XLAM
    Set myAddIn = Application.AddIns.Add("c:\myaddins\myxlam.xlam")
    myAddIn.Installed = True
    
    ' Load known XLAM
    For Each myAddIn In AddIns
        If myAddIn.Name = "myknownaddin.xlam" Then
            myAddIn.Installed = False
            myAddIn.Installed = True
        End If
    Next
End Sub

HPC Services for Excel

The above strategies are effective for using add-ins on cluster compute nodes and workstation nodes with the HPC Services for Excel in Windows HPC Server 2008 R2.

Excel on HPC Resources

[MSFT] Jeremy Espenshade — Tue, 31 Aug 2010 01:40:15 GMT

We have a new one-stop shop for all the HPC Services for Excel documentation and training. The Windows HPC and Microsoft Excel Survival Guide has everything from high-level overview and training to how-to guides and code samples. Check it out!

Workbook Offloading Versioning Plan

[MSFT] Jeremy Espenshade — Tue, 31 Aug 2010 01:29:00 GMT

In Microsoft HPC Server 2008 R2, we offer HPC Services for Excel 2010. As part of this feature, we provide a framework for running Excel workbooks on a cluster through the ExcelClient component. As this is a developer framework which we expect to support in future versions of HPC Server, we have developed a comprehensive versioning plan to ensure that your development investment is sustained across future versions of our product.

To adhere to the versioning guidelines we expect to follow in the future, the only thing we require is for VBA developers to implement the HPC_GetVersion macro in their workbooks as described in the existing documentation here. Since this release represents the first version of the HPC Services for Excel, "1.0" must be returned by the HPC_GetVersion function. With this information, the ExcelClient component will know that the workbook has implemented version 1.0 of the macro framework specification which supports the partition/execute/merge pattern for distributed computation of the Excel workbook, and thereby allow future versions of the ExcelClient component to provide expected behavior even if features are added or changed.

Future Versioning Plans

In future releases, one or more of the following features may undergo revisions:

.NET Interfaces

The .NET interfaces will all follow strict versioning guidlines and adhere to Microsoft guidelines for obselescence of any public members.

COM Interface

The COM interface, IExcelClient, will always be strictly versioned so that code written targeting the current interface will never be broken. For each revision of this interface, IExcelClient will be assigned a new GUID and an IExcelClientVx will be created with the previously assigned GUID where x is the previous version. So in the next revision of this interface, we will provide both an IExcelClient and IExcelClientV1 interface.

Developers are encouraged only to use the IExcelClient interface, as Excel's VBA environment uses name-based type resolution and this will better allow workbooks written targetting the next version of the interface to work on computers which only have the current version installed. The IExcelClientV1 interface will be included to allow other COM clients which cache GUIDs to be compiled with the current binaries and continue to work on future versions without recompilation.

If VBA developers would like to design their workbooks such that they make use of new APIs only provided in future versions of the IExcelClient interface but also would like to work when used on a client with the current version of the product, they must check the IExcelClient.Version property at runtime and only call APIs provided in version X when version X or higher is returned. This will allow developers to explicitly fall back into a compatibility mode where only APIs provided in the current release, 1.0, are used, or produce a useful message to the user stating the requirements to run the workbook.

If a feature is added to the IExcelClient that is exposes a feature that is not available in the current release of the HPC Server product and a cluster running the current version is targeted, the user will receive an error at runtime which explains any incompatibility.

Macro Specification

Separately from the IExcelClient interface, the macro specification may change. In this case, a new version of the macro specification will be published. Workbooks that wish to use the new framework will have to return the specified version in HPC_GetVersion ("2.0" rather than "1.0" for example). Workbooks that wish to support multiple macro specifications will be able to return a list of versions (either as a string[] or variant[]). If such a workbook is run on a computer with the current release installed, the "1.0" macros will be used, while a computer with the latest release installed will use the latest macro specification.

If the workbook does not implement version "1.0" of the macro specification and is used on either a client or a cluster with the current release installed, the user will receive a runtime error stating that the workbook is not supported.

SOA Service

The current version of the ExcelService exposes a service contract which will follow strict versioning guidelines so client code written targetting the current service will work on future versions of this service. If additional functionality is added to this contract, it will only be used by future versions of the ExcelClient component. In the case that some added functionality in a future version of ExcelClient requires features not available in the current release of the ExcelService, a runtime error will be generated explaining any incompatibility.

Why can't I open two workbooks with the same name??

[MSFT] Jeremy Espenshade — Tue, 31 Aug 2010 00:57:00 GMT

This is a pop-up that's sparked multi-minute-long rants which always result in a profanity-laden diatribe against seemingly arbitrary rules in general and Microsoft in particular. When I joined the company, it was with a mandate to figure out why this is the case and fix it. I'm happy to report that I've determined the answer to the first part and very good reasons why neither I nor anyone else is going to do the second.

It all has to do with naming qualification and ambiguity resolution. Let's say you have two workbooks, BookA.xlsm and BookB.xlsm. If BookA wants to reference a cell in BookA, it can simply set one cell equal to another using the familiar =[cellref] syntax as shown below:

What if you want to reference a cell in BookB? Well that's easy, just qualify it with the name of the workbook and worksheet where Excel should go to find the cell in question as shown below:

This is a great feature of Excel which has enabled people to build up tremendously complicated systems of workbooks with simple cross-references, multi-workbook calculation chains, and encapsulated functional areas. You can even follow standard software development practices like Model-View-Controller patterns by having a Model workbook encapsulating the business logic and data, a View workbook encapsulating the UI and results display, and a Controller workbook managing the interactions. As you jump into VBA programming, ties between workbooks will only get more complicated and qualification more necessary.

But wait!! What happens if there are two workbooks named BookB.xlsm open in the same Excel process? Which workbook is meant by BookB.xlsm? The answer is that you can't know. The duplicated use of the same name has made this case strictly ambiguous.

What can be done about that? You have two options 1.) Redesign the qualification scheme such that it is completely unique, or 2.) Disallow multiple workbooks with the same name to be opened by a single Excel process. Obviously the later option was chosen by the Excel team way back in the early days of Excel.

This is why multiple workbooks with the same name cannot be opened by Excel within a single process.

If that sounds unfortunate, consider option 1. To come up with a qualification scheme that's unique, you'd need to detach the workbook name from the qualifier. Let's say you choose a GUID-based system where each GUID is associated with a particular workbook file and that you know about all the workbooks you'll need and all their GUIDs. Now you write a big complicated workbook system with tons of GUIDs all over the place. What happens if someone makes some changes to an existing workbook and saves a copy of it. Either these two copies now have different GUIDs or the same GUID, both of which would introduce problems down the line.

If they have different GUIDs, then the newly saved workbook cannot be used with any old code that accesses it's entries based on the GUID. This would essentially destroy the idea of backups, as you'd be changing the workbook when you saved a copy.
If they have the same GUID and they contain different logic or data, then how can the workbook developer choose between these two workbooks? Well, you'd need a qualification scheme... Yikes! You can see where this cycle would go.

The Workaround

The only reason that multiple workbooks can reference each other is that Excel supports a Multi-Document Interface (MDI) with multiple workbooks managed by a single process and UI. So the natural cure to the resulting reference ambiguity is to disallow it by opening each workbook in a separate instance of Excel. This can easily be done by opening Excel from the commandline or start menu and then opening workbooks from within that interface.

That's all on this topic. Hopefully this at least helps you to feel informed the next time you forget and try to open a second workbook with the same name and get that lovely warning.

Testing Access to Excel Workbook

[MSFT] Jeremy Espenshade — Thu, 12 Aug 2010 19:37:00 GMT

One of the common problems people may run into while offloading a workbook across a cluster is making sure that all the compute nodes can see the workbook. When the workbook is simply inaccessible (not deployed or shared), this is very clear from client error messages. However, if only a subset of the compute nodes cannot see the workbook, then determining which nodes don't have a copy can be more difficult. You end up having to do a bit of trial and error or write a script that tests if the workbook can be accessed from each node. Even this isn't foolproof in the case of corrupt workbooks which exist where they're supposed to but still cannot be opened.

To address this issue, I've put together a powershell script which can be run from any cluster client computer (cluster nodes included) that uses the HPC Services for Excel binaries to open the workbook from each cluster node. To modify this for use with you're workbook, simply replace [WorkbookPath] with the path where you expect to find the workbook (eg \\mycomputer\share\myworkbook.xlsm or c:\temp\myclusterworkbook.xlsm, etc).

    $script = {  
                $workbookPath = "[WorkbookPath]" 
                $asm = [Reflection.Assembly]::Load("Microsoft.Hpc.Excel,Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35");  
                $driver = new-object Microsoft.Hpc.Excel.ExcelDriver;  
                try 
                { 
                    $driver.OpenWorkbook($workbookPath);  
                } 
                catch 
                { 
                    $msg = "Unable to open workbook: " 
                    $msg += $workbookPath 
                    return $msg         
                } 
                finally 
                { 
                    $driver.Dispose() 
                }}       
                 
    $encodedscript = [System.Convert]::ToBase64String([System.Text.Encoding]::UNICODE.GetBytes($script)) 
     
    clusrun powershell -EncodedCommand $encodedscript

This script builds a series of powershell commands and then encodes it such that it can be invoked from CMD. This gets around the inability to invoke complex powershell commands from CMD and difficulty remotely running powershell scripts without lowering security settings. The one downside to this approach is that you have to explicitly write out the workbook path rather than passing it as an argument. This is due to the encoding process which encodes exactly the script you give it without evaluating any contained code. Regardless, the above script works nicely and will give you a rundown of your cluster nodes and whether or not they can open the workbook specified. An example output is below:

Excel Workbook Offloading Beta Transition Guide

[MSFT] Jeremy Espenshade — Wed, 07 Jul 2010 01:58:00 GMT

Excel Workbook Offloading Beta Transition Guide

The Beta 2 release of Windows HPC 2008 R2 that was made available this past spring included the first public release of a feature allowing developers to take a computationally intensive workbook, add some VBA code, and run it on a cluster entirely within the workbook. By implementing a set of pre-defined VBA functions to parallelize and coordinate computation, existing business logic can used across a set of workbooks running in parallel on a cluster.

This post is for those developers and partners that have already started using HPC Beta 2 to offload workbook computation or received training on the API as it was defined in the Beta release. Based on customer and partner feedback, we made a number of breaking changes for the release candidate (RC) to ensure that the first official release met the needs of our customers in an intuitive and robust manner. This guide will walk through those changes, why they were made, and how to update your existing Beta workbooks to use the new APIs. For those of you who are unfamiliar with this feature, I highly suggest reading the HPC Services for Excel white papers (links available here ) and encourage you to think about what workbooks you use that may benefit from parallelization (see also Developer resources for Excel).

This article addresses the following changes from Beta 2 to RC:

Macro Specification

For workbook offloading, the workbook must now include a new macro called HPC_GetVersion().

ExcelClient COM API

ExcelClientV1 and IExcelClientV1 have changed to ExcelClient and IExcelClient, respectively.
To select a resource type, use SessionUnitType instead of ResourceUnitType.
The workbook path is now a parameter for the ExcelClient.OpenSession method instead of for the ExcelClient.Initialize method.
The ExcelClient.OpenSession method includes new parameters to specify the workbook path, the service name, and a job template.
The ExcelClient.DeployWorkbook method was removed.

Macro Specification

As part of the Beta 2 release, we provided a specification for macros that must be implemented to allow a workbook computation to be divided into pieces (HPC_Partition), calculated in parallel (HPC_Execute), and aggregated and displayed to the user (HPC_Merge). In addition to these and the other optional macros defined in Beta 2 (HPC_Initialize, HPC_Finalize, and HPC_ExecutionError), a final macro, HPC_GetVersion is now required. It is expected to be implemented as follows.

Function HPC_GetVersion() As String
{
HPC_GetVersion = "1.0"
}

This macro is necessary for future versioning support in the eventuality that we change the macro specification to support additional features in future releases. By implementing HPC_GetVersion now, you will be ensuring that your workbook will run correctly across future releases of HPC Services for Excel. This is a required function and ExcelClient.Initialize will throw if it is not implemented or returns an incompatible version ("1.0" is the only compatible version for this release).

ExcelClient COM API

Type Names

To initiate calculation and request cluster resources, a COM-visible library is available called Microsoft_HPC_Excel. In the Beta 2 release, we exposed two types: ExcelClientV1 and IExcelClientV1, the class and interface respectively. In the RC, the names of these types have changed to ExcelClient and IExcelClient, dropping the V1 suffix. As our future versioning plans solidified we realized that this naming scheme will better enable us to seamlessly support forward/backward compatibility among workbooks written targeting multiple versions of the HPC Services for Excel libraries.

SessionUnitType Enumeration

In Beta2, developers were instructed to use the ResourceUnitType enumeration to select the type of resource (core, node, or socket) requested in ExcelClient.OpenSession. This type was defined in Microsoft.Hpc.Scheduler.Properties. In an effort to increase the modularity and independence of our components, we now expose a separate enumeration over COM called SessionUnitType in Microsoft.Hpc.Excel. This enum contains the same possible selections of core, socket, or node allocation.

ExcelClient.Initialize Method

In Beta 2, the ExcelClient.Initialize method signature was defined as follows.

ExcelClient.Initialize (Microsoft.Office.Interop.Excel.Application app, string remoteWorkbookPath)

app - Currently running Excel application
remoteWorkbookPath - path where the compute nodes in the cluster can find the workbook

The first thing we realized was that specifying the remoteWorkbookPath in ExcelClient.Initialize was unintuitive. Initialize may only be called once per ExcelClient instance, so changing the workbook path between calculations was not possible without disposing and recreating an ExcelClient instance. In addition, this path is not required if calculating locally and it doesn't make sense to require a parameter that is never used. To remedy both of these issues, remoteWorkbookPath is now a parameter to the ExcelClient.OpenSession method.

The remaining parameter to ExcelClient.Initialize, the Excel application, was also changed. It is possible to have multiple workbooks open within a single Excel instance, but when offloading workbook computations, all of the interaction is with the workbook, so it is less ambiguous to pass a reference to the workbook that implements the partition/execute/merge macro specification rather than the application which has that workbook (and potentially other workbooks) open.

With those changes, ExcelClient.Initialize now has the signature:

ExcelClient.Initialize (Microsoft.Office.Interop.Excel.Workbook workbook)

workbook - Excel workbook that contains the functions that implement the macro specification

ExcelClient.OpenSession Method

In Beta 2, the ExcelClient.OpenSession method signature was defined as follows.

ExcelClient.OpenSession(string headNode, int minResources, int maxResources, ResourceUnitType resourceType)

headNode - name of the head node of the cluster to use
minResources - minimum number of resources required
maxResources - maximum number of resources required
resourceType - type of resource requested. Defined by Microsoft_Hpc_Scheduler_Properties.ResourceUnitType. Core, Socket, or Node types available.

As discussed in the ExcelClient.Initialize Method section above, the remoteWorkbookPath parameter specifying where cluster compute nodes can find the workbook is now included in the parameter list for ExcelClient.OpenSession. ResourceUnitType was replaced by SessionUnitType as discussed above as well. Several additional parameters were also added while making a number of parameters optional.

The additional parameters added are a job template and a service name. The job template parameter allows a cluster administrator to create a template that defines things like job priority and node groups. This is particularly helpful for Excel workbook offloading because many clusters may only have Excel installed on a subset of compute nodes, and creating an Excel-specific job template could allow jobs to be run only on those nodes that have Excel installed.

The default service name is Microsoft.Hpc.Excel.ExcelService and a configuration file called Microsoft.Hpc.Excel.ExcelService_1.0.config is installed to the service registration directory. Unfortunately, this implies that the same configuration can be used for all Excel workbook offloading jobs. In some cases (extremely long running HPC_Execute macros, for example), it may be important to provide different configurations for different workbooks. This can be accomplished by creating a separate configuration file with a distinct name, say LongRunningExcel.config, and passing the service name, in this case "LongRunningExcel", to ExcelClient.OpenSession.

Finally, with the exception of the headNode parameter which selects a cluster and the remoteWorkbookPath parameter which specifies where compute nodes can find the workbook, all parameters were made optional and product-wide defaults will be used for each unspecified parameter.

After these changes, the method signature is as follows.

ExcelClient.OpenSession(string headNode, string remoteWorkbookPath, [int minResources], [int maxResources], [SessionUnitType resourceType], [string jobTemplate], [string serviceName])

headNode - name of the head node of the cluster to use
remoteWorkbookPath - path where the compute nodes in the cluster can find the workbook
minResources - optional parameter specifying the minimum number of resources required
maxResources - optional parameter specifying the maximum number of resources required
resourceType - optional parameter specifying the type of resource requested. Defined by Microsoft_Hpc_Excel.SessionUnitType. Core, Socket, or Node types available.
jobTemplate - optional parameter specifying a job template to use
serviceName - optional parameter specifying a service name to use

ExcelClient.DeployWorkbook Method

In an effort to clarify what functionality we provide make use of existing functionality where already provided, we have removed the DeployWorkbook method. This method essentially saved a copy of the workbook in a specified directory, which is functionality provided by the ThisWorkbook.SaveCopyAs method provided in Excel's API.

Example Transition

Beta 2 API usage

Final API usage after transition

Additional Changes to Notice

Similar changes were made in the ExcelClient .NET API, and using this guide should make .NET code transition easy as well.
ExcelClient .NET API now uses an event property rather than a parameter in OpenSession to specify a delegate for handling errors returned by the cluster.
Event Viewer Tracing has moved from [Applications and Services Logs]\[Microsoft HPC Excel Pack] to [Applications and Services Logs]\[Microsoft]\[HPC]\[Excel] where it is organized into the traditional admin/operational/debug events. Significant expansion in tracing is also apparent and local tracing of has been added to help diagnose client problems.
Performance and robustness improvements.