Dan on eScience & Technical Computing @ Microsoft : Research

Cray Brings Windows 7 to HPC

Dan Fay — Fri, 13 Nov 2009 03:41:09 GMT

Now what computational scientist wouldn’t want one of these under their desk. I like the tagline, “putting ‘work’ back into workstation”.

Cray Brings Windows 7 to HPC

by Michael Feldman, HPCwire Editor

If you thought Windows 7 was just for mere mortals, think again. Microsoft's latest OS is about to show up in Cray's newest CX1 deskside supercomputer that puts a Windows workstation and a Windows HPC Server cluster into a single box. Called the CX1-iWS (for integrated workstation), the machines are to be sold exclusively through Dell and will range in price from $39K to $55K.

The idea behind the iWS is to retain the interactive experience of a personal workstation, but extend its computational power to that of a small HPC cluster. It's generally aimed at technical computing users who have simply run out of compute headroom on their two-socket machines, but are loathe to give up the intimacy of the workstation environment. A generic CX1 can be configured to provide the equivalent capabilities, but the iWS is preconfigured to deliver this experience right out of the box.

Since Windows 7 and Windows HPC Server 2008 form a natural client-server relationship, cluster administration and job management becomes relatively seamless. In addition, since there is disk storage shared between the workstation and the cluster, data management becomes much more straightforward. As long as your data set fits in 4 TB, no data transfers back and forth between client and server will be necessary.

HPCwire: Cray Brings Windows 7 to HPC

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

Dan Fay — Fri, 16 Oct 2009 17:39:17 GMT

Today Microsoft Research announced the availability of the book - The Fourth Paradigm: Data-Intensive Scientific Discovery. The book focuses on the change of all sciences moving from observational, to theoretical, to computational and now to the 4th Paradigm – Data-Intensive Scientific Discovery. This is based on Jim Gray’s insights captured via his final public talk to the National Research Council on Jan 11, 2007. This is truly a legacy of his work.

The book is available from the web and is released under a Creative Commons license.

Part 1: Earth and Environment	Part 3: Scientific Infrastructure
Part 2: Health and Wellbeing	Part 4: Scholarly Communication

I feel fortune to have been able to contribute the introduction to the Earth and Environment section - I had many discussions with Jim on need for balance between data and computations, and the need to make scientific exploration through the use of computing technologies much easier for scientists. I had also “borrowed” many of Jim’s slides to discuss the change to the upcoming fourth paradigm, he made the points so succinctly – there was no need for marketing fluff.

The Fourth Paradigm: Data-Intensive Scientific Discovery

Presenting the first broad look at the rapidly emerging field of data-intensive science

Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets.

The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud computing technologies.

In The Fourth Paradigm: Data-Intensive Scientific Discovery, the collection of essays expands on the vision of pioneering computer scientist Jim Gray for a new, fourth paradigm of discovery based on data-intensive science and offers insights into how it can be fully realized.

Praise for The Fourth Paradigm

“The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science."

— Bill Gates, Chairman, Microsoft Corporation

The Fourth Paradigm: Data-Intensive Scientific Discovery - Microsoft Research

Amazon Web Services support Sloan Digital Sky Survey DR6 Subset

Dan Fay — Wed, 30 Sep 2009 00:11:45 GMT

Now that AWS is supporting SDSS, there is probably useful integration between the Worldwide Telescope and the SDSS datasets that can be leveraged, beyond what is currently supported - such as seeing the all the known galaxies in a 3D view – seeing the lattice structure of the universe.

Search and Browse data from the Virtual Observatory and plot data over imagery: WorldWide Telescope delivers one-click contextual access to distributed Web information and data sources and Interoperates through SAMP and other popular tools like TopCat, Aladin, SAOImage DS9 and many more.

Data derived from researchers using the AWS datasets can integrate imagery and catalog data directly into WorldWide Telescope using the WWT Developer Kit.

Professional features

Virtual Observatory Cone search/registry look up and SIMBAD search
Load and Adjust basic FITS images and AVM files
SIAP with footprint preview
Connect your Telescope to WWT
Multi-monitor cluster rendering
Visualization of large scale structure
SAMP Inter-application communication
Full dome projection

New Public Data Set: Sloan Digital Sky Survey DR6 Subset

The Sloan Digital Sky Survey, or SDSS, is now available as a Public Data Set.

Weighing in at 180 GB, the SDSS is the most ambitious astronomical survey ever undertaken. The researchers have used a 2.5 meter, 120 megapixel telescope located in Apache Point, New Mexico to capture images of over one quarter of the sky, or about 230 million celestial objects. They have also created 3-dimensional maps containing more than 930,000 galaxies and 120,000 quasars.

Amazon Web Services Blog: New Public Data Set: Sloan Digital Sky Survey DR6 Subset

Science Analytics – look to use Project “Gemini”

Dan Fay — Fri, 21 Aug 2009 19:34:37 GMT

When I first saw and heard details about Project “Gemini”, I was blown away by the technology and innovation created by SQL and Excel teams and that held up when I was able to test it out on my own. It will be especially useful for scientists that want to not only analyze large amounts of data in Excel, but also aggregate different datasets. This upcoming Excel 2010 add-in removes the storage limits of Excel by adding the in-memory database and brings the power of SQL Server and SQL Analysis Services into the hands of mere mortals. Scientists that utilize Excel for viewing/analyzing data will find this add-in extremely helpful.

Project Gemini Blog – Check out the videos -

Project "Gemini": Build powerful analytical applications

Need to make timely business decisions without having to use complicated and sluggish analytical applications? Love to use Excel? Project Gemini is an Excel 2010 add-in that allows you to create powerful analyses by quickly manipulating millions of rows of data into a single Excel workbook and utilize Microsoft Office 2010 to share and collaborate on your insights with your team.

You can combine native Excel 2010 functionality with Gemini’s in-memory engine to allow users to interactively explore and perform calculations on large data sets. In addition, you can easily streamline the process of integrating data from multiple sources – including corporate databases, spreadsheets, reports, and data feeds.

Share and collaborate with confidence by easily publishing your analysis to SharePoint 2010 and have other users enjoy the same slicer and fast-query capabilities when working on your Excel Services reports.

Are you part of Office 2010 Tech Preview? Register and then Download and learn about Project Gemini now! (Note: You need to have Office 2010 before you can use Gemini.)

Introducing Microsoft Office 2010 for Business

SciScope app and code available for download

Dan Fay — Thu, 16 Jul 2009 02:47:08 GMT

Earlier today the code behind the SciScope site was made available at SciScope.CodePlex.com. This enables others to make their datasets/repositories available and allow others to discover, download and utilize their data in a simple to use website. Also the semantic support is quite useful in finding related data.

SciScope Project Description

SciScope (see it live) is a prototype web application that allows data discovery from across multiple distributed heterogeneous data repositories. It leverages Bing Maps (formerly Microsoft Virtual Earth) and Microsoft SQL Server 2008 to support queries involving spatial, temporal and thematic constraints over an index of sensors operated by agencies such as USGS, EPA and NOAA as well as user provided data. SciScope leverages taxonomies stored as triples in SQL Server to provide search suggestions and for dealing with semantic heterogeneity between different data repositories.

_{SciScope screenshots (discovering/downloading insecticide data, browsing ecoregions left to right) for video tutorials click here}
This CodePlex release includes some desktop tools to simplify data publishing and content crawling for SciScope namely Catalog Publisher and Catalog Updater.

SciScope - Home

Project Tuva: Richard Feynman is now available to all.

Dan Fay — Wed, 15 Jul 2009 21:15:31 GMT

Project Tuva, an enhanced video player showcasing Richard Feynman’s “Messenger” lectures is available for all to try out. It’s the way I’d like to view talks and related information – check it out. Not only does it allow for web links, images, but it also integrates with the WorldWide Telescope control to help augment the example Feynman uses in the gravitational talk. Currently only the first lecture in the series “Law of Gravitation – an Example of Physical Law” utilizes all the annotations/links, but the do all have the transcripts, so you can search on something like “particles” and see where it is mentioned in all the different videos, and then jump directly to the location.

Microsoft Research and Bill Gates Bring Historic Physics Lectures to Web

Lecture series by celebrated physics professor Richard Feynman is now available to all.

REDMOND, Wash. — July 14, 2009 — Microsoft Research, in collaboration with Microsoft Corp. Chairman Bill Gates, today launched a Web site that makes an acclaimed lecture series by the iconic physicist Richard Feynman freely available to the general public for the first time. The lectures, which Feynman originally delivered at Cornell University in 1964, have been hugely influential for many people, including Gates. Gates privately purchased the rights to the seven lectures in the series, called “The Character of Physical Law,” to make them widely available to the public for free with the hope that they will help get kids excited about physics and science.

The historic lectures and related content can be seen at http://research.microsoft.com/tuva. The name “Tuva” was chosen because of Feynman’s lifelong fascination with the small Russian republic of Tuva, located in the heart of Asia.

Feynman was one of the most popular scientists of the 20th century, equally regarded for his scientific insights as well as his ability to convey his enthusiasm for science through his lectures and writings. He shared the Nobel Prize in Physics in 1965 and was also known for his quirky sense of humor and eccentric and wide-ranging interests.

“No one was more adept at making science fun and interesting than Richard Feynman,” said Gates. “More than 20 years after first seeing them, these are still some of the best science lectures I’ve heard. Feynman worked hard during his life to popularize science, so I’m sure he’d be thrilled that now anyone, anywhere in the world, can just click a button and experience his lectures.”

Curtis Wong, a principal researcher with Microsoft Research, enhanced the experience of viewing the lectures by integrating the historic video with a Microsoft Silverlight-based video player that allows viewers to search the lectures for references to particular subjects, take notes that are synchronized to the video, and click on hyperlinks to related Web content, among other customized operations.

“There is a lot of public interest in building innovative educational resources online,” Wong said. “This is an opportunity to take some existing educational content and utilize software and the wealth of resources available on the Web to create a richer learning experience. And because people can annotate the lectures with their own comments and links to related resources, I expect this experience to become richer and richer over time.”

Microsoft Research has been exploring video annotation for many years and chose to publish the Feynman “Messenger” lectures with a new enhanced video player. Neither Microsoft nor the Bill and Melinda Gates Foundation were involved in the acquisition of the rights to the lectures.

Microsoft Research and Bill Gates Bring Historic Physics Lectures to Web: Lecture series by celebrated physics professor Richard Feynman is now available to all.

Project Trident: A Scientific Workflow Workbench available for download

Dan Fay — Wed, 15 Jul 2009 02:58:03 GMT

Project Trident CTP is now available for download. Project Trident is a scientific workflow workbench MSR External Research has been working on for the past few years, which allows scientists to analyze large, diverse datasets. It’s built on Windows Workflow and utilizes SQL Server (Express or Server). Download it and try it out…

Project Trident

Built on the Windows Workflow Foundation, this scientific workflow workbench allows users to:

Automate analysis and then visualize and explore data

Compose, run, and catalog experiments as workflows

Capture provenance for each experiment

Create a domain-specific workflow library to extend the functionality of the workflow workbench

Use existing services, such as provenance and fault tolerance, or add new services

Schedule workflows over HPC clusters or cloud computing resources

Project Trident: A Scientific Workflow Workbench - Microsoft Research

eScience Workshop 2009 – Call for Papers

Dan Fay — Mon, 06 Jul 2009 20:48:57 GMT

The 2009 eScience Workshop will be held at the Gates Center for Computer Science at Carnegie Mellon University in Pittsburgh, PA, October 15-17, 2009. The call for papers closes on July 31, 2009.

Call for Papers

We invite contributions from all areas of eScience and e-Research including:

Computational support for scientific research in life sciences, biomedical computing, environment, energy, and other scientific grand challenges
Knowledge discovery and merging datasets
Large-scale scientific data analysis, mining, and visualization
High-performance computing applied to solving problems in a variety of scientific disciplines
Dissemination of scientific literature/results and the discovery, curation, and sharing of data
Scientific sensors, data-gathering tools and technologies
Collaboration/workflow tools and technologies
Data-intensive science
Emerging multidisciplinary fields such as Digital Heritage and eEconomy
Research implications of computational thinking
How strategies for semantics and ontology formulation enable scientific discovery

eScience Workshop 2009

The eScience Workshop, to be held October 15-17, 2009, will provide a unique opportunity to share experiences, learn new techniques, and influence the domain of scientific computing. Scientists and researchers will explore the evolution, challenges and potential of computing in scientific research, including how the latest tools, Web services and database technologies are being applied to scientific computing.

Workshop Theme
Facilitating Scientific Discovery through Data-Intensive Computing

Hosting and Location
Co-hosted by Microsoft Research and Carnegie Mellon University, this workshop will take place in the Gates Center for Computer Science at Carnegie Mellon University in Pittsburgh, PA.

eScience Workshop 2009 - Microsoft Research

Adding quarters to the innovation machine

Dan Fay — Tue, 09 Jun 2009 21:12:15 GMT

I really enjoyed Dan Reed’s latest posting - HPC: Making a Small Fortune - and the need for real innovation in this space, especially with the rise of new technologies – that’s part of the reason we looked at at combining HPC and Databases via the GrayWulf Project…

HPC: Making a Small Fortune

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

There is an old joke in the high-performance computing community that begins with a question, "How do you make a small fortune in high-performance computing?" There are several variations on the joke, but they all end with the same punch line, "Start with a large fortune and ship at least one generation of product. You will be left with a small fortune." Forty years of experience, with companies large and small, has confirmed the sad truth of this statement.

Reed's Ruminations: A Blog by Dan Reed: HPC: Making a Small Fortune

Graywulf Takes Byte Out of Data Overload

Dan Fay — Fri, 08 May 2009 20:46:17 GMT

Graywulf is the natural evolution of Beowulf Clusters – it brings together HPC clusters and databases to do efficient processing and data management. It’s name and design also pays homage to Jim Gray – who helped champion the use of relational databases in the scientific projects.

At it’s simplest form Graywulf is having a database installed on each of the HPC compute nodes – this brings the data to the computation – one of the points Jim made quite often and utilizes the power of databases (queries, stored procedures, etc). Since it’s a generic architecture Graywulf clusters can be built using any OS and any database…the ones in the case study below implemented them using Windows HPC Server and SQL Server and the motivation was to be more efficient in doing the science – it’s always great to have innovative folks using technologies to do good work.

“To put it simply, a scientist needs to be able to live within the data,” says Alexander Szalay, a cosmologist-turned-computer-scientist at The Johns Hopkins University (JHU) in Baltimore, Maryland. The power of information, Szalay says, is determined not by its quantity so much as how easy it is to access, manipulate and analyze.
“It’s not just about doing the numerical calculations,” adds Andrew Simms, a biomedical health informatics graduate student working on protein structure analysis in Valerie Daggett’s bioengineering laboratory at the University of Washington (UW) in Seattle. “It’s also about assembling the data so we can run calculations while performing analyses and ad hoc explorations and then feed it all back into the data warehouse.”

Graywulf Takes Byte Out of Data Overload

Astronomers at The Johns Hopkins University and protein scientists at the University of Washington are using inexpensive computer hardware combined with powerful computing and database software to help manage and analyze a growing volume of scientific data.

For details, read the Graywulf case study.

Project Principals

Alexander Szalay, Alumni Centennial Professor, Department of Physics and Astronomy, The Johns Hopkins University

Valerie Daggett, Professor of Bioengineering, University of Washington

Graywulf Takes Byte Out of Data Overload - Microsoft Research

WWT at TechFest on NYTimes.com

Dan Fay — Mon, 02 Mar 2009 19:49:24 GMT

The NYTimes.com article by Ashlee Vance, included a great picture by Stuart Isett for The New York Times showing the dome that was put together for TechFest to demonstrate the planetarium projection mode of WWT as well as the gesture interaction from Andy Wilson and Hrvoje Benko.

Microsoft Maps Course to a Jetsons-Style Future

Stuart Isett for The New York Times

Hrvoje Benko demonstrating a Microsoft projection system that lets people manipulate large video images with their hands.

Microsoft Maps Course to a Jetsons-Style Future - NYTimes.com

Microsoft Research TechFest 2009

Dan Fay — Tue, 24 Feb 2009 21:31:26 GMT

Today is the first day of the MSR TechFest 2009 – a showcase of MSR technologies – you can see pictures from TechFest. One of the demos I’m loosely tied to is Interactions with an Omni-Directional Projector – which utilizes WorldWide Telescope projected on a dome with gesture interaction. The neat part about it is that it brings together the great interaction (hands) work from Andy Wilson and Hrvoje Benko from the Adaptive Systems and Interaction and Jonathan Fay (no relation)

TechFest is much like a Science Fair – booths to demonstrate new algorithms, discoveries, etc. Great place to wander and dream what the magic of software can do….

TechFest 2009

TechFest is an annual event that brings researchers from Microsoft Research’s labs around the world to Redmond to share their latest work with Microsoft product teams. Attendees experience some of the freshest, most innovative technologies emerging from Microsoft’s research efforts. The event provides a forum in which product teams and researchers can discuss the novel work occurring in the labs, thereby encouraging effective technology transfer into Microsoft products.

TechFest 2009 - Microsoft Research

Using Flickr for Astronomy – and viewing in WWT

Dan Fay — Sat, 21 Feb 2009 01:55:17 GMT

The use of online services such as Flickr to help scientists is in its infancy and applications utilizing commodity based solutions will continue to pick up momentum. I especially like the integration and the ease of use – science should be about discovery and exploration – not about the technology. Of course the ability to view those analyzed images in WorldWide Telescope completes the circle and allows you to view the image in context.

Check out the Orion Nebula.
After it opens up – click on the thumbnail at the top.

You can also add your own – check out Dinoj’s post on the WWT Data Blog - Sticking images on the sky with WWT. For fun you can see the crown for the Corona Borealis overlaid on the sky

See the article written by Frederic Lardinois on ReadWriteWeb.

The Great Gig in the Sky: Using Flickr for Astronomy

Flickr hosts a wide range of beautiful images, but a new project built on top of Flickr's API only focuses on photos of the night sky from amateur astronomers. The Astrometry.net project constantly scans the Astrometry Flickr group for new images to catalog and to add to its open-source sky survey. At the same time, this project also provides a more direct service to the amateur astronomers, as it also analyzes each image and returns a high-quality description of the photo's contents.

The Astrometry group currently has over 400 members, and as Christoper Stumm, a member of the Astrometry.net team, told the Flickr Code blog, the back-end software uses geometric hashing to exactly pinpoint and describe the objects in the images. When you submit an image to the Flickr pool, the robot will not just respond with a comment that contains an exact description of what you see in the image, but it will also annotate the image automatically.

While a lot of members of the Astrometry group use high-end telescopes and cameras, the Astrometry.net solver can also analyze images from consumer-level digital cameras.

While just being able to automatically analyze and catalog these images is pretty cool already, every description also contains a link that displays the image in Microsoft's WordWide Telescope.

Astronomy is one of those few scientific disciplines where dedicated amateurs can still make major discoveries and this is definitely one of the cooler applications of Flickr's API that we have seen in a long time.

The Great Gig in the Sky: Using Flickr for Astronomy - ReadWriteWeb

New Tools Mobilize Local Data to Study Global Environmental Issues from Berkeley Lab

Dan Fay — Thu, 05 Feb 2009 07:06:12 GMT

Here’s a really good article from the folks at Lawrence Berkeley National Laboratory on the collaboration MSR has ongoing between LBL and the Berkeley Water Center. It highlights the use of databases for scientific information as Catharine mentions…

“One of the greatest challenges of the next century will be developing cyber-architectures that allow scientists to easily navigate their digital assets. Today, the internet has given environmental researchers instant access to a wealth of field data. Now, they need a scientific ‘safety deposit box’ system that will not only store this information, but also organize it so it is searchable and ready for analysis,” says van Ingen.

New Tools Mobilize Local Data to Study Global Environmental Issues

Guarding water supplies, protecting endangered species and curbing greenhouse gases is going high-tech. Environmental scientists are turning to innovative cyber-infrastructures and data-mining tools.

As they strive to develop effective strategies for guarding water supplies, protecting endangered species and curbing greenhouse gases, environmental scientists are turning to innovative cyber-infrastructures and data-mining tools developed by an ongoing collaboration between researchers at Lawrence Berkeley National Laboratory, Microsoft Research, and the University of California, Berkeley.

The Microsoft eScience program is the primary funder of this project, which is one of numerous ventures cultivated by the Berkeley Water Center (BWC). Launched approximately three years ago by researchers from the Berkeley Lab and UC Berkeley’s Colleges of Engineering and Natural Resources, the BWC marshals expertise from public institutions and the private sector in support of projects that enable science and public policy researchers to more easily access and work with water and environmental datasets.

“The most cost-efficient way to impact issues like global climate change and water management is to develop cyber-architectures that organize data and foster scientific collaboration,” says Susan Hubbard, staff scientist in the Berkeley Lab’s Earth Sciences Division and associate director of the BWC.

Environmental scientists typically collect data on a project-by-project basis, in campaigns targeted at very specific topics. One study may use NASA satellites to track annual rainfall of deserts around the globe, while another project sponsored by the National Science Foundation (NSF) might measure the annual water tables of the Sahara desert with commercial sensors. The data are then typically stored in local archive systems and accessed by researchers associated with that particular project. These sites are scattered across the country, tend to be aligned with specific campaigns, and are funded by a variety of organizations.

Rest of the article at: New Tools Mobilize Local Data to Study Global Environmental Issues « Berkeley Lab News Center

So you don’t think you can Sing?

Dan Fay — Thu, 08 Jan 2009 21:43:03 GMT

Now you can have your own musical accompaniment to match your voice without having to worry about artistic differences. I’ll have to play with Songsmith and see if can help even my voice sound decent :-) But you won’t see me posting the songs or videos online.

Just think – you can add a musical soundtrack to your everyday tasks…doing the dishes, walking the dog, or the one I like – kids ~~saying~~ singing “Science is Cool”….neat to see this product coming out of Microsoft Research.

What is Songsmith?
Songsmith generates musical accompaniment to match a singer’s voice. Just choose a musical style, sing into your PC’s microphone, and Songsmith will create backing music for you. Then share your songs with your friends and family, post your songs online, or create your own music videos.
Where can I get it?
A free trial download is available on our download page.