Project Trident: A Scientific Workflow Workbench available for download

eScience — Wed, 15 Jul 2009 02:58:03 GMT

Project Trident CTP is now available for download. Project Trident is a scientific workflow workbench MSR External Research has been working on for the past few years, which allows scientists to analyze large, diverse datasets. It’s built on Windows Workflow and utilizes SQL Server (Express or Server). Download it and try it out…

Project Trident

Built on the Windows Workflow Foundation, this scientific workflow workbench allows users to:

Automate analysis and then visualize and explore data

Compose, run, and catalog experiments as workflows

Capture provenance for each experiment

Create a domain-specific workflow library to extend the functionality of the workflow workbench

Use existing services, such as provenance and fault tolerance, or add new services

Schedule workflows over HPC clusters or cloud computing resources

Project Trident: A Scientific Workflow Workbench - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Scientific Discovery through Advanced Computing Program (SciDAC) tutorials on Friday, July 18th

eScience — Sat, 12 Jul 2008 02:08:35 GMT

Microsoft Research is hosting the DOE 2008 Scientific Discovery through Advanced Computing Program (SciDAC) tutorials on Friday, July 18^th. If interested in attending, please register at the SciDAC Tutorials Registration.

SciDAC Tutorial Agenda

July 18 – Microsoft Research, Bldg 99.

9:00 - 9:30

Reception, checkin and orientation

9:30 - 12:00

First session

Visualization and Data Analysis with VisTrails

Introduction to Scientific Workflow Management and the Kepler System

Porting and Scaling Applications on BlueGene/P

Advanced CompuTational Software (ACTS) Collection Tutorial

12:00 - 1:00

Lunch in bldg. 99 Atrium
(attendees in bldg. 117 walk to bldg. 99 next door)

1:00 - 3:30

Second session

Advanced Visualization and Data Analysis with the VisIt Visualization System

ADIOS The ADaptible IO System

An Introduction to the Cray XT4 for Application Scientists

Advanced CompuTational Software (ACTS) Collection Tutorial

Tutorial Abstracts

<></>

Title:

Introduction to Scientific Workflow Management and the Kepler System

Contact:

Ilkay Altintas

Partners:

The Scientific Data Management Center (SDM)

Presenters: Ilkay Altintas, Scott Klasky, Norbert Podhorszki, Mladen Vouk

A scientific workflow combines data and processes into a configurable, structured set of steps that implement semi- automated computational solutions of a scientific problem. Scientific workflow systems provide a framework and often graphical user interfaces to combine different technologies along with efficient methods for using them. This reduces overhead and increases the efficiency of the scientists towards a scientific discovery. SciDAC SDM Center's Scientific Process Automation group develops scientific workflow tools (called Kepler) for automation of scientific data management processes in a reusable, robust, tractable, and recoverable fashion to enhance scientific exploration. This tutorial provides an introduction to scientific workflow construction and management using the Kepler system. It is intended for the SciDAC scientists and students with a computational science background. It will cover principles and foundations of scientific workflows, Kepler environment installation, workflow construction out of the available Kepler library components, and workflow execution management that uses Kepler-based facilities to provide process and data monitoring, provenance information, portal access, and high speed data movement solutions. This tutorial includes hands-on sessions and application examples from different scientific disciplines.

<></> <></>

Title:

ADIOS: The ADaptible IO System

Contact:

Chen Jin

Partners:

The Scientific Data Management Center (SDM) and The Center for Plasma Edge Simulation and Oak Ridge National Lab

Presenters: Chen Jin, Scott Klasky, Steve Hodson: ORNL, Hasan Abbasi, Jay Lofstead, and Matthew Wolf: Georgia Tech, and Manish Parashar Rutgers

The ADaptible IO System (Adios) is a componentization of the IO layer for high performance scientific computations. Currently, we have integrated ADIOS into Fusion, Combustion and other pioneering scientific applications and achieved excellent performance results. ADIOS provides a simple to use, common programming interface for different transport layer methods. The design goals are set to allow the application scientist to program a very easy to use IO system, which can choose between different IO implementations at runtime. This allows IO experts to tune their implementations on different systems without changing the IO implementation. We currently support MPI-IO, collective MPI-IO, POSIX, asynchronous MPI-IO, the Georgia Tech DataTap system based on RDMA, and the Rutgers DART implementation also based on RDMA. ADIOS separates out the metadata and methods, by using an external XML file. This allows users to add annotations outside of their F90/C/C++ codes and to change the implementation outside of their code. ADIOS 1.0 will be released before 12/1/2008 and will provide very fast writes of data. In this tutorial, we will first present an introduction of ADIOS, along with the current supported methods, and file converters. We will then provide a hands-on session which will allow users to work with both a F90 and C code, and change the IO to ADIOS, and change this to use both MPI-IO and Posix IO. We will then show them how to read in the data, and convert the output to HDF5, NetCDF, and ASCII.

<></> <></>

Title:

Porting and Scaling Applications on ALCF’s BlueGene/P

Contact:

Kalyan Kumaran, Argonne Leadership Computing Facility

Partners:

The Argonne Leadership Computing Facility

Presenters: Vitali Morozov and Ray Loy

The Argonne Leadership Computing Facility houses a half a petaflop BlueGene/P supercomputer. A number of petascale applications, from a variety of engineering domains, are run every day on this highly scalable architecture via the Department of Energy's INCITE (Innovative and Novel Computational Impact on Theory and Experiment) program. Nevertheless, porting and scaling applications onto petascale computers remains a challenging task. This tutorial will focus on the BlueGene/P architecture and present an overview of compilers, libraries, performance tools, debuggers available for porting and scaling applications.

<></> <></>

Title:

Advanced Visualization and Data Analysis with the VisIt Visualization System

Contact:

Hank Childs

Partners:

The Visualization and Analytics Center for Enabling Technology (VACET)

Presenters: Hank Childs and Sean Ahern

Understanding scientific datasets generated at DOE's high-performance computing facilities is becoming increasingly difficult as dataset sizes and complexity grow beyond the scale that is approachable by traditional analysis techniques. VACET is delivering scalable solutions to real-world visualization and analysis problems through the VisIt visualization system. VisIt is a turnkey application for data exploration, visualization, code assessment, and quantitative analysis suitable for use on SciDAC datasets of any size. This tutorial is targeted at all levels of users, from first timers on up to VisIt experts. The tutorial will present the basics of using VisIt on up to advanced visualization techniques.

<></> <></>

Title:

Visualization and Data Analysis with VisTrails

Contact:

Claudio Silva

Partners:

The Visualization and Analytics Center for Enabling Technology (VACET)

Presenters: Claudio Silva and Carlos Scheidegger

The demand for the construction of complex visualizations is growing in many disciplines of science, as scientists are faced with ever increasing volumes of data to analyze. We give an overview of VisTrails, an open-source provenance management system that provides infrastructure for data exploration and visualization. VisTrails transparently records detailed provenance (history) information for exploratory computational tasks, both for the derived data products and for the pipelines used to derive them. Besides enabling the reproducibility of results, this provenance information can be used to simplify the process of data exploration through visualization. We will present several mechanisms and intuitive interfaces provided by VisTrails that allow flexible re-use of pipelines; exploration of large parameter spaces; comparison of visualizations and their respective pipelines; and the creation and refinement of visualizations by analogy.

This tutorial is targeted to users with different levels of expertise, from novice to expert. We will cover a number of specific examples that reflect typical visualization needs of DOE applications.

<></> <></>

Title:

An Introduction to the Cray XT4 for Application Scientists

Contact:

Richard Gerber

Partners:

NERSC and ORNL

Presenters: Richard Gerber and Rebecca Hartman-Baker

This tutorial provides the basics to get up and running on the Cray XT4. Topics covered include compilers, libraries, and job management with a focus on parallel scaling and performance optimization of real world applications. The presenters are HPC consultant staff with hands on experience with the XT4 systems at ORNL and NERSC. This tutorial is targeted at researchers transitioning to the XT4 from serial or other parallel computing environments.

<></> <></>

Title:

Advanced CompuTational Software (ACTS) Collection Tutorial

Contact:

Tony Drummond

Partners:

ACTS and TAU

Presenters: Tony Drummond

This short course will be an introduction to a set of advanced computational software tools to leverage the development of high performance applications. The lectures will focus on the selection, installation and use of scalable and robust software tools. Functionalities implemented in these software tools include; numerical algorithms for the solution of large computational problems, performance monitoring and profiling, and automatic tuning. Participants should expect to learn about techniques used to solve common computational problems and monitor their performance. Participants are encouraged to bring laptop computers and follow live demonstrations through hand-on experiences. The software presented here is freely available and widely used by the computational sciences international community.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

eScience @ Microsoft : Workflow

Project Trident: A Scientific Workflow Workbench available for download

Project Trident

Scientific Discovery through Advanced Computing Program (SciDAC) tutorials on Friday, July 18th