As an HPC cross platform developer, I have spent a good part of my 20s working on scientific codes under different operating systems. Windows, Mac, LINUX, HPUX, AIX, IRIX, Solaris. Building /compiling code is a full time job, not to mention porting and making changes due to the API differences. This is what I call the #ifdef jungle.
In any case, I found a group of 3 musketeers, that shared my pain, and we would like to save you time when it comes to porting your applications to Windows, where almost a billion users can run YOUR code. The wonderful folks at Interopsystems agreed to host our project for us on their website. http://www.interopcommunity.com/dictionary/index.php
This dictionary is organized by frequently used HPC Unix calls. Each entry show the Unix function, its Windows equivalent (if it exists), required header files, an example of use, and related discussion. As you (or your compiler) discovers functions that are platform-specific or otherwise need modification, you can look up the offending function to find out what changes should be done. Information not specific to a function as well as overall suggestions and strategies can be found in the Porting Tips.
We hope this will save you time, but also please provide us with feedbacks on the site. Feel free to discuss and provide your personal porting tips, and we would love to add them to the collection.
Quite a few folks from the Microsoft HPC team will be heading down to Portland next week to attend the 2009 super computing conference. I will be distributing a new version of the memory stick containing reference materials. (download in one zip)
Feel free to find me and get a copy at the Microsoft Booth.
This year Microsoft and IBM are sponsoring ASU students (the cluster devils team) going to the sc09 cluster challenge. This year’s challenge includes running:
1. CHOMBO 2. WRF (binary here). 3. HPCC 4. VisIt 5. NWCHEM These codes cover finite element/structure, weather simulation, benchmarking, visualization of data output, and computational chemistry. The students will be measured on performance/power consumption, efficiency, when running these codes. I wish them the best of luck!
China has apparently become the third country to build a petaflop supercomputer. Xinhua news agency reported on Thursday that the country has unveiled "Tianhe," a 1.206 peak petaflop machine powered by a combination of 6,144 Intel CPUs and 5,120 AMD GPUs, and it runs linpack at 563.1 teraflops. Amazingly, the price tag was a mere $88.24 million. The system is installed at the National University of Defense Technology (NUDT) in Changsha, capital of central China's Hunan Province. Link This is about 3 times as fast as the previous record which Microsoft helped to power. See (Microsoft Breaks Into Top 10 of World's Largest Supercomputers!)
Kyril Faenov, the General Manager, HPC and Parallel Computing Technologies, announced last week that Microsoft purchased Interactive supercomputing. [link] This is pretty exciting news for the Microsoft HPC group, as Star-P functionalities will be integrated into future Windows HPC products making HPC mainstream for the Windows platform.
I have explained what SUA is and how it differs from Cygwin in previous blogs. Basically, SUA stands for Sub system for Unix Applications, and it is an alternative to Cygwin if you have windows server or Windows Client (7, vista) Ultimate installed. Remember, SUA sits on the kernel as a peer to the windows 32 sub system. Since Microsoft's MPI stack runs on the windows 32 subsystem, you would not want to build your program by linking against SUA libraries or use SUA's GCC compiler for performance, and availability reasons. MS does not make an MPI module for SUA sub system.
However, if you would like to use the Unix tools to assist in building windows binaries, that's perfectly fine. Most typical setup is that people use SUA or cygwin for their unix tools, and windows 32/64 native compiler for actual code generation. Please see my previous blog.
To install SUA:
Goto Control Pannel, Programs, and choose Turn Windows features on or off.
Select Subsystem for UNIX-based Applications
Click OK.
Next, goto http://suacommunity.com and follow instructions there to download Utilities and SDK for SUA.
Installing the package requires a bit Unix knowledge.
Make sure everything other than SVR-5 utilities are selected.
All check boxes in this dialog should be checked.
You are done.
Launch the KornShell or BASH from SUA menu.
There are additional add-ons that include 100s of recently built Unix utilities from Interops Website. Download and install them.
More detailed version can be found at: http://www.suacommunity.com/SUA_Tools_Env_Start.htm
The MPI Cluster Debugger (with C/C++ project templates) is now available via the Microsoft Download Center. Search for 'hpc debugger' or go directly to http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=d3993532-bdf8-4024-b164-db2ee8a851f5
This package integrates the 'F5' experience of launching the Visual Studio 2008 MPI Cluster Debugger and your Windows HPC Server 2008 environment, for a better and more integrated debugging experience. This package provides a new tool window from which you can specify on which nodes the debugging will take place on the cluster. Once your break points are set and the debugger settings are specified you only have to press 'F5' to launch the debugger. Pressing 'F5' deploys your application binaries and the remote debugger, and submits a job to the scheduler. When the job is scheduled the MPI processes start under the control of the remote debugger.
Let's see how it works.
1. First download the package from the link above. Double click to install, then Launch visual studio. I assume you already have your Microsoft HPC 2008 SDK installed.
2. Click on new project, and navigate down to project type: Visual C++ , you will see an MPI Project listed in the template selection. Select it, and click on OK.
2. It will then create a default project for you with source code. Put a break point on the line where it says MPI_Init.
3. Configure the debugger by going to Tools Menu then select Cluster Debugger Configuration. You will now see the configuration details on the left side of the IDE window.
4. Next, select head node of the cluster that you want to run it on. Then, the number of processors, and make any changes to the advanced setting if necessary.
5. Edit the nodes you would like to use for debugging.
6. A dialog pops up allowing you customizing which nodes you would like to use. The CPU usages are displayed with live information.
7. Now, the moment of truth... Press F5 to start debugging. Launch your HPC management console to observe whether the job is started on the cluster.
8. Back to visual studio window, you may select processes to attach to on local and remote nodes, by right clicking on the window, it brings up the "attach to process" window. Selecting a remote process is easy, simply type in the domain credential, and the node. Visual studio debugging processes are all named msvmon.exe.
9. Click on Play, and you can see how the break point gets stopped at each launched MPI process that the debugger attaches to(total of 5), and run the job to completion.
This release of the cluster debugger real simplifies and streamlines the user experience in debugging MPI programs. Thanks for reading my blog.
MSMPI is Microsoft’s implementation of MPI (Message passing interface) library. I’ve explained it a few times before. MSMPI Explained, MSMPI SDK patch.
Many people have asked me for examples that they can understand and relate to how MPI is used on Unix. Here’s the information you’ve been looking for!
Compilers supported: MingW’s GCC, microsoft CL, PGI, Intel…. but not GCC under cygwin or SUA.
Caution: For 32 bit, Microsoft MPI library uses __STDCALL, which means one must use MPIAPI macro in function declaration. Here’s an example:
PetscMPIInt PETSC_DLLEXPORT MPIAPI Petsc_DelTag
(MPI_Comm comm,PetscMPIInt keyval,void* attr_val,void* extra_state);
You really just need to make sure mpi.h can be found in the include path, and msmpi.lib can be found in the Library path. Here are a few examples:
Visual Studio:
Include dir in “Additional Include directories” in your project property.
If you are building a 32 bit application, make sure i386 path is used.
64 bit library path.
Finally, make sure msmpi.lib is part of the linker input.
MINGW native windows GCC compiler:
g++ -O3 -g -c Main.cpp -I/c/Program\ Files/Microsoft\ HPC\ Pack\ 2008\ SDK/Include/
g++ -O3 -g Main.o /c/Program\ Files/Microsoft\ HPC\ Pack\ 2008\ SDK/Lib/i386/msmpi.lib -o mympi.exe
Intel compiler under cygwin Unix environment:
Using include path example:
-I/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15 -I/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/bmake/win32_intel -I/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/include -I /cygdrive/c/Program\ Files/Microsoft\ HPC\ Pack\ 2008\ SDK/Include
Library path example:
-L/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/lib/win32_intel -L/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/lib/win32_intel -lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc /cygdrive/c/Program\ Files/Microsoft\ HPC\ Pack\ 2008\ SDK/Lib/amd64/msmpi.lib -L/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/externalpackages/f2cblaslapack/win32_intel -L/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/externalpackages/f2cblaslapack/win32_intel -lf2clapack -L/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/externalpackages/f2cblaslapack/win32_intel -L/cygdrive/c/Users/wenmingy/petsc-intel/petsc-2.3.3-p15/externalpackages/f2cblaslapack/win32_intel -lf2cblas Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib
resourcekit.windowshpc.net
Finally version 2 is updated and online.
Changes include:
New Parallel computing content section in developers.
Updated many older documents, presentations.
New hpc webcasts from partners.
New marketing white papers.
We will be distributing limited quantities at techED. If you would like to get one of these, please find us at the Windows HPC booth.
The 10th LCI (Linux Cluster Institute) international Conference on High Performance Clustered Computing was hosted at National Center for Atmospheric Research (NCAR) in Boulder, Colorado on 3/9 - 3/12. This year's focus was on large scale system design that address balance between inter-and intra-node parallelism. LCI is a small conference that had roughly 150 attendees. Microsoft was invited not only as a vendor sponsor but also to give a tutorial on Windows HPC Server 2008.
Here's a list of abstracts, technical papers, and program schedule. Microsoft is relatively new to the field of HPC. We were there to listen to HPC users, educators, and developers, and update them about what we are doing to make their lives easier with latest tools, and best practices from the software industry.
Developers are starting to see more user demand for their HPC applications on the Windows HPC Server platform. Like any software projects, due to budget and time constraint, the platform that the developer is not developing on may not get a build. After discussing with several HPC developers at the conference, I have a few ideas as a result, to share with everyone in terms of sustainable cross platform development specific to many HPC apps.
First, there is no silver bullet in getting your HPC code running on the Windows platform, but it is generally not difficult, since most HPC apps are console applications. There are also numerous Unix environments on the Windows platform including SUA, Cygwin, MingW/MSYS. These environments allow you to build your applications without changing your build system too much. The free Windows HPC SDK, enables you to build your HPC application against our MSMPI library, which is industry standard.
Some of the more complex platform issues include dealing with Unix System calls, Pthreads, on Windows. Developers don't necessarily know how to code to specific OS for some of these calls. There are articles, guides and books on them. Of course, for the hyper-impatient developers, I could always offer the advice of hiring a professional Windows programmer to resolve these issues.
As a software engineering practices, it is always a great idea to develop your Linux HPC software with portability in mind. In many cases, by going to a new platform, your code is exercised more, thus revealing some of the bugs not discovered using a single platform. One thing that is a big plus on the Windows platform is that Microsoft has invested billons in Visual studio to help developer write faster, better code. Today's complex software projects demand an IDE for productivity, why not take advantage of the best tool in the industry.
During the discussions with developers, we have concluded that the best way to keep a quality build of your HPC project on Windows is to have at least one of project developer using Visual studio to write the code, that way there is no more last minute scrambling to get a Windows binary built. If the project can spare more cycles, then an automated regression test that checks out and builds the project in a regular basis would also make the build on Windows platform better. This, however may require expertise in maintaining the original Unix build system with the windows platform in mind. The PETSC project does a great job of making things transparent; their python build system runs great under cygwin, thus no additional automated build system is needed on Windows.
References and tools on the Windows platform:
HPC Resource kit. HPC white papers, presentations, tutorials.
Sys Internal Tools. List of tools Utilities.
Windows SDK with free command line compilers, libraries.
When it comes to HPC development, linear algebra libraries are the essential building blocks. You often hear about BLAS/LAPACK, so what exactly are they and how do they relate to the top 500 run benchmark called LINPACK?
BLAS stands for basic linear algebra subprograms, it contains routines that does basic linear algebra functions like vectors and matrix multiplications. It has existed for nearly 30 years. LAPACK, you guessed it, is an abbreviation for linear algebra package. It is a larger set that include BLAS. Heavily utilized in high performance computing, BLAS/LAPACK have been implemented/optimized, re-implemented many times over for many platforms. LINPACK, though the official benchmark for HPC, relies on a specific subroutine DGEMM in BLAS to solve dense matrix in order to obtain measurement of FLOPS (floating point operations per second). LINKPACK, however, has little practical use, as it has been replaced by LAPACK. There are many commercial and non-commercial versions of BLAS/LAPACK. I’ll mention a few that works on windows.
refblas: The official reference implementation from netlib. C and Fortran implantations.
Intel MKL: The Intel Math Kernel Library.
ACML: The AMD Core Math Library.
ATLAS: Automatically Tuned Linear Algebra Software. (uses cygwin as build env)
CUDA SDK: The NVIDIA CUDA SDK includes BLAS functionality for writing C programs that runs on GeForce 8 Series graphics cards.
Goto BLAS: Kazushige
Goto's implementation of BLAS.
uBLAS: A generic C++ template class library providing BLAS functionality. Part of the Boost library. Note that, unlike other implementations, uBLAS focuses on correctness of the algorithms using advanced C++ features, rather than high performance.
GSL: The GNU Scientific Library Contains a multi-platform implementation in C. windows.
BLAS/LAPACk is the very basics of HPC numerical libraries. Many solver libraries such as PETSC and TRILINOS build on top of it. Here are three links back to my previous blogs that explain, Linear, non-linear, and Time-stepping problems.
Math under the hood (linear solvers)
5 min Intro to nonlinear solvers
Explicit and Implicit (time stepping, or PDE solvers)
Overview
This update fixes an issue where the HPC Pack 2008 SDK was attempting to use a function that was not available on olders versions of Windows, such as 32-bit Windows XP. If you encounter the error "The procedure entry point GetProcessIdOfThread could not be located in the dynamic link library KERNEL32.dll" when attempting to run an application that you've built using the HPC Pack 2008 SDK then you should apply this fix and recompile your application.
http://www.microsoft.com/downloads/details.aspx?FamilyID=6415c650-78a3-4b6a-8264-b855173b14d8&DisplayLang=en
To start the download, click the Download button and then:
1) Click Save to copy the download to your computer. 2) Run the download on the machine that you build your applications using the HPC Pack 2008 SDK. 3) Recompile your application with the updated SDK.
High Performance Computing (HPC) Clusters: Reduce Complexity with IBM and Microsoft
Please join us for an informative and exciting webcast with IBM and Microsoft. No matter where you are located in the world, IBM has an HPC cluster solution that is easy to deploy. Learn how IBM can help you reduce the risk and manage growth more easily with the pre-tested, easy-to-deply, easy-to-manage IBM Cluster 1350 solution, and how when combined with Windows HPC Server 2008 you can leverage your current Windows server expertise to accelerate your time to insight on computational analysis. Whether you are a financial analyst or engineer our Windows based HPC cluster will meet the demand of your computational needs. Register today:
Windows HPC Server 2008 environment with System Center Operations Manager 2007: A Dynamic Integration For HPC Fans: Discover how to integrate your Windows HPC Server 2008 environment with System Center Ops Mgr 2007 (SC0M 2007), to leverage a new way of dynamic management and reporting. You'll learn what can be done by using the regular Compute Cluster Server Management Pack to monitor a cluster and how to leverage the Windows 2003 Management Pack and Job Scheduler to understand why a job failed. Learn how to build custom reports with SCOM 2007 and Windows HPC Server 2008 and see how the integration works. Finally, get a sneak peek into what will be done in the Windows HPC Server 2008 Management Pack. Join us today for this informative and highly productive webcast!
Windows HPC Server 2008 - New and Advanced Features
Look-- no slides just kick the tires! Want to find out what's new in Windows HPC Server 2008 and what exactly is "High Productivity Computing"? Join us for an exciting look into Windows HPC Server 2008. We'll take you on a tour of this new version of Windows Compute Cluster Server, highlighting the major advancements in deployment, management, interoperability and scheduling. We promise no slides and tons of live demonstrations plus a lively discussion on how Windows HPC Server 20008 will increase your cluster administrators' productivity.
Scaling up Analysis using Windows HPC Server 2008 presented by Ansys
Join Microsoft and Ansys for an informative HPC webcast. Moving from desktop computing to a cluster can expand the scope of simulation, allowing more simulation studies and reducing the turnaround time for larger models. This presentation will review the performance of simulation software from ANSYS on Windows HPC Server 2008 and highlight customer experiences with cluster deployment based on Windows.
Leveraging Computational Features of Windows HPC Server 2008 using the IMSL® Numerical Libraries with Visual Numerics & Microsoft
Windows HPC Server 2008 is a key component in supporting the development of high performance computational applications within a rich development environment. New capabilities support the development of high performance computing (HPC) applications in a variety of languages such as C/C++, Fortran, and .NET for both multi-threaded and multi-node computing. This presentation will focus on the computation algorithm support for developing HPC applications using the IMSL Numerical Libraries with Windows HPC Server 2008. An example for leveraging a multi-tiered computing environment will be included.
It's the last day of 2008, and I am uploading some of the web casts that we have done for HPC in the last year.
All available at: http://channel9.msdn.com/shows/The+HPC+Show/
(may take a day to fully show up, the the links below should work)
HPC SERVER 2008 - 2008 PARTNER AND MS WEBCASTS:
Bringing speed and visibility to software builds using Windows high-performance clustering with Electric Cloud and Microsoft
Improving the speed of the software build/test/deploy cycle is one of the most effective ways of increasing developer productivity. In order to break the build bottleneck, all limitations to performance must be overcome: dependencies, accuracy, and infrastructure speed. ElectricAccelerator solves the problem of hidden and implicit dependencies, guaranteeing accuracy while enabling parallel execution, and Microsoft Windows HPC Server 2008 provides the most effective cluster infrastructure to support the ElectricAccelerator technology.
Dramatic Acceleration of Excel-based Trading Simulations with Platform and Microsoft
Tired of waiting hours for simulation results? In this solution overview, Platform will review the challenges of running Excel models in a distributed environment, and describe the benefits and details of how to deploy models without wholesale code/macro changes. Learn how to enhance application performance and empower quants and developers to deploy distributed applications quickly without modification, using Platform Symphony's Excel Connector with Windows HPC Server 2008 and Microsoft® Office 2007 Excel. You'll discover how the world's top financial firms can realize new revenue opportunities ahead of the competition by leveraging this joint solution.
High Performance Computing for the Masses with Intel
High Performance Computing has its roots in non-commercial, academic, national lab and governmental environments where very large, compute-intensive workloads and applications are executed. According to recent data from IDC HPC, economically-attractive clusters have become the common denominator across most HPC deployments. In parallel, the continuing maturation of HPC ISV commercial business software, the refinement of operating system & cluster software, and the consistent availability of more computing power at lower cost have opened up the inherent performance advantages of HPC clusters to a much wider community of users. Please join Intel for an overview of the current HPC marketplace and the potential alternatives of utilizing High Performance Computing clusters for both increased productivity and competitive advantage.
HPC in Life Insurance Modeling - Leveraging Milliman's MG-ALFA on the Cray CX1 Supercomputer with Windows HPC Server 2008 to Meet Emerging Insurance Requirements and Deliver Advanced Analytics
MG-ALFA from Milliman is available on the new Cray CX1 deskside supercomputer and is fully integrated with the Windows HPC Server 2008. Join us for this informative webcast and learn about the new Cray CX1 system, get the latest information on the recent developments in version 6.7 of MG-ALFA and explain the benefits of MG-ALFA on the latest version of Windows HPC technology, Windows HPC server 2008. Topics covered will include: Introduction to the recently-introduced Cray CX1 supercomputer; Performance of MG-ALFA on Windows HPCS, Scaling performance for very large system counts running MG-ALFA on Windows HPCS, Technical requirements for the implementation of MG-ALFA on Windows HPCS, Enhancements to Windows HPCS over the prior version of this software, Windows Compute Cluster Server. The Webinar is of interest to anyone evaluating an implementation of the Principles based approach to Capital.
Windows HPC Server Development, the MPI Application Model
Windows HPC Server 2008 introduces innovative features for traditional High Performance Computing solutions. Significant improvements have been implemented across the entire HPC stack. Learn about the new NetworkDirect RDMA technology and how it enables very fast, high-bandwidth, distributed shared memory computing. MS MPI is now integrated with Event Tracing for Windows easing the task of profiling, analysis, and visualization of HPC solution performance. A new Job Scheduler architecture delivers impressive performance while also enabling intelligent scheduling policies and resource management. Traditional HPC solution developers will appreciate the elegance of Microsoft's latest platform technologies while those new to the parallel computing problem domain will want to consider Windows HPC based solutions
Improving Electronic System Design Productivity using Synopsys System Studio and Saber on Windows HPC Server 2008
Synopsys System Studio and Saber products are now available on Windows HPC Server 2008. Join us for this informative webcast and get the latest information on how System Studio further improves performance and signal processing design productivity by taking full advantage of Windows HPC Server 2008 as a stable, extensible environment on high-performance CPUs. Learn how Saber helps automotive and aerospace supply chains meet stringent reliability and safety requirements for mechatronic systems in harsh environments. System Studio is Synopsys' model-based algorithm design and analysis offering providing a unique dataflow simulation engine with the highest performance for exploring, verifying and optimizing digital signal processing algorithms. Saber is Synopsys' technology-leading mechatronic design and analysis software, advancing Robust Design and Design for Six Sigma (DFSS) methodologies into today's automotive, aerospace, and commercial design.
The Development of a Next Generation Networking Interface - Network Direct
Mellanox and Microsoft bring you an informative and informational webcast on Network Direct. The ability of scientists to access the right data and apply it the right way is critical for many scientific disciplines. "The life sciences with genomic and proteonomic data, it's an explosion of information that people around the world should be able to benefit from the results that other people are gathering together," Bill Gates, Supercomputing 2005. Three years after, Microsoft has introduced a new networking interface called Network Direct, which enables high performance applications not only to be able to execute faster and handle large information databases, but also to provide efficient parallelism with effective remote data management. In the future, Network Direct might be the common interface not only for MPI based application but also for sockets and file systems. The session will provide an overview of Network Direct development and its potential benefits, a discussion of the role of Windows for MPI applications, and industry case studies.
HPC in Oil and Gas: Reservoir Simulation using Roxar's Tempest on Windows Compute Cluster Server
Join us for an exciting webcast specific for the Oil and Gas Industry! Predicting the performance of an oil reservoir requires a large amount of computing muscle. This is now available at a reasonable cost using cluster computers with typically 8 - 16 processors. Roxar's Tempest simulator can take full advantage of these clusters using Windows Compute Cluster Server to simplify job submission and administration. This webcast will show you how. Presented by Robert Frost, Tempest Development Manager and Martin Parker Systems Integration Manager, Roxar Corporation.
Taking the Guesswork out of High-performance Computing for Financial Services
Learn how insurers can use Towers Perrin's MoSesT V6.2 software with Microsoft Windows Compute Cluster Server 2003 to run sophisticated Financial models in less time. Changes in the financial services industry are requiring firms to manage risks and capital better. To meet this challenge, insurers need robust financial modeling software integrated with a cost-effective high-performance computing platform, which can extend existing IT investments. Find out how MoSes and WCCS can help save time and increase productivity.
Windows HPC Server Development, the WCF Application Model
Windows HPC Server 2008 introduces a new interactive programming model for High Performance Computing solutions. Learn how to host Windows Communication Foundation (WCF) components within a compute cluster and how to leverage the cluster in various service-oriented client application scenarios. An innovative departure from traditional clustered computing, the HPC+WCF Services model enables ease of solution development, deployment, and management. Early adopters include financial services enterprises with near real-time interactive compute requirements.
I am at Microsoft PDC this week, demonstrating the Microsoft HPCS Server 2008. Please come by the Parallel computing booth and talk to us.
I will be distributing the HPC technical resource kit on a thumb drive at the booth and for everyone that attends the HPC talk on Tuesday right after Lunch.
