eScience @ Microsoft : Article

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

eScience — Fri, 16 Oct 2009 17:39:17 GMT

Today Microsoft Research announced the availability of the book - The Fourth Paradigm: Data-Intensive Scientific Discovery. The book focuses on the change of all sciences moving from observational, to theoretical, to computational and now to the 4th Paradigm – Data-Intensive Scientific Discovery. This is based on Jim Gray’s insights captured via his final public talk to the National Research Council on Jan 11, 2007. This is truly a legacy of his work.

The book is available from the web and is released under a Creative Commons license.

Part 1: Earth and Environment	Part 3: Scientific Infrastructure
Part 2: Health and Wellbeing	Part 4: Scholarly Communication

I feel fortune to have been able to contribute the introduction to the Earth and Environment section - I had many discussions with Jim on need for balance between data and computations, and the need to make scientific exploration through the use of computing technologies much easier for scientists. I had also “borrowed” many of Jim’s slides to discuss the change to the upcoming fourth paradigm, he made the points so succinctly – there was no need for marketing fluff.

The Fourth Paradigm: Data-Intensive Scientific Discovery

Presenting the first broad look at the rapidly emerging field of data-intensive science

Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets.

The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud computing technologies.

In The Fourth Paradigm: Data-Intensive Scientific Discovery, the collection of essays expands on the vision of pioneering computer scientist Jim Gray for a new, fourth paradigm of discovery based on data-intensive science and offers insights into how it can be fully realized.

Praise for The Fourth Paradigm

“The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science."

— Bill Gates, Chairman, Microsoft Corporation

The Fourth Paradigm: Data-Intensive Scientific Discovery - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

TIME - WorldWideTelescope one of 50 Best Websites 2009

eScience — Mon, 24 Aug 2009 23:29:15 GMT

TIME magazine has the WorldWide Telescope WebClient (Silverlight) on their list of 50 Best Websites for 2009. That puts WWT in with sites like Flickr, Twitter, Skype, YouTube, Amazon, and even PhotoSynth – not bad for the very small team we have on it :-)

WorldWideTelescope

Like Google Earth for the heavens, WWT aggregates terabytes of astronomical data from the world's biggest telescopes to create a single virtual scope that anyone can look through. WWT is not a model of the known universe, but rather a centralized repository for just about everything known about the universe. The idea is to democratize the science of astronomy with a single tool that can be used by students and scientists. Who knows, when everyone has access to the same data, maybe the next big discovery in astronomy will be made by an amateur? There are hundreds of terabytes of digitized sky — enough data for everyone

WorldWideTelescope - 50 Best Websites 2009 - TIME

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Adding quarters to the innovation machine

eScience — Tue, 09 Jun 2009 21:12:15 GMT

I really enjoyed Dan Reed’s latest posting - HPC: Making a Small Fortune - and the need for real innovation in this space, especially with the rise of new technologies – that’s part of the reason we looked at at combining HPC and Databases via the GrayWulf Project…

HPC: Making a Small Fortune

N.B. I also write for the Communications of the ACM (CACM). The following essay recently appeared on the CACM blog.

There is an old joke in the high-performance computing community that begins with a question, "How do you make a small fortune in high-performance computing?" There are several variations on the joke, but they all end with the same punch line, "Start with a large fortune and ship at least one generation of product. You will be left with a small fortune." Forty years of experience, with companies large and small, has confirmed the sad truth of this statement.

Reed's Ruminations: A Blog by Dan Reed: HPC: Making a Small Fortune

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

New Tools Mobilize Local Data to Study Global Environmental Issues from Berkeley Lab

eScience — Thu, 05 Feb 2009 07:06:12 GMT

Here’s a really good article from the folks at Lawrence Berkeley National Laboratory on the collaboration MSR has ongoing between LBL and the Berkeley Water Center. It highlights the use of databases for scientific information as Catharine mentions…

“One of the greatest challenges of the next century will be developing cyber-architectures that allow scientists to easily navigate their digital assets. Today, the internet has given environmental researchers instant access to a wealth of field data. Now, they need a scientific ‘safety deposit box’ system that will not only store this information, but also organize it so it is searchable and ready for analysis,” says van Ingen.

New Tools Mobilize Local Data to Study Global Environmental Issues

Guarding water supplies, protecting endangered species and curbing greenhouse gases is going high-tech. Environmental scientists are turning to innovative cyber-infrastructures and data-mining tools.

As they strive to develop effective strategies for guarding water supplies, protecting endangered species and curbing greenhouse gases, environmental scientists are turning to innovative cyber-infrastructures and data-mining tools developed by an ongoing collaboration between researchers at Lawrence Berkeley National Laboratory, Microsoft Research, and the University of California, Berkeley.

The Microsoft eScience program is the primary funder of this project, which is one of numerous ventures cultivated by the Berkeley Water Center (BWC). Launched approximately three years ago by researchers from the Berkeley Lab and UC Berkeley’s Colleges of Engineering and Natural Resources, the BWC marshals expertise from public institutions and the private sector in support of projects that enable science and public policy researchers to more easily access and work with water and environmental datasets.

“The most cost-efficient way to impact issues like global climate change and water management is to develop cyber-architectures that organize data and foster scientific collaboration,” says Susan Hubbard, staff scientist in the Berkeley Lab’s Earth Sciences Division and associate director of the BWC.

Environmental scientists typically collect data on a project-by-project basis, in campaigns targeted at very specific topics. One study may use NASA satellites to track annual rainfall of deserts around the globe, while another project sponsored by the National Science Foundation (NSF) might measure the annual water tables of the Sahara desert with commercial sensors. The data are then typically stored in local archive systems and accessed by researchers associated with that particular project. These sites are scattered across the country, tend to be aligned with specific campaigns, and are funded by a variety of organizations.

Rest of the article at: New Tools Mobilize Local Data to Study Global Environmental Issues « Berkeley Lab News Center

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

HPC - MPI.NET 1.0 Is Now Released...!

eScience — Wed, 08 Oct 2008 21:12:04 GMT

PhilPen has posted that the folks at Indiana University have released MPI.NET: High Performance C# library for Message Passing. The runtime and source code are available for download as well as a Tutorial. This should make it much easier for folks to use any .Net language to write MPI apps – I’m interested in seeing ones written with F# and even PowerShell.

MPI.NET 1.0 Is Now Released...!

MPI.NET is a high-performance, easy-to-use implementation of the Message Passing Interface (MPI) for Microsoft's .NET environment. MPI is the de facto standard for writing parallel programs running on a distributed memory system, such as a compute cluster, and is widely implemented. Most MPI implementations provide support for writing MPI programs in C, C++, and Fortran. MPI.NET provides support for all of the .NET languages (especially C#), and includes significant extensions (such as automatic serialization of objects) that make it far easier to build parallel programs that run on clusters.

MPI.NET has been developed by the research staff at Indiana University in collaboration with Microsoft. Developers leverage the "Windows HPC Server 2008 SDK" in tandem with the MPI.NET SDK to build MPI.NET applications. An MPI.NET runtime component must be installed onto Windows HPC Server 2008 based clusters to host MPI.NET applications.

Regarding Windows Server : MPI.NET 1.0 Is Now Released...!

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Science News to justify half a bar of chocolate per week - lowers risk of heart disease

eScience — Wed, 24 Sep 2008 07:45:39 GMT

I only wish the study coming out of the Moli-sani Project being run by Research Laboratories of the Catholic University in Campobasso recommended milk chocolate instead dark chocolate.

Sorry Snickers - Half A Bar Of Chocolate Per Week, But Just The Dark Kind, Lowers Risk Of Heart Disease | Scientific Blogging

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Accessible Research blog

eScience — Thu, 28 Aug 2008 01:42:59 GMT

Sam Stokes has spun up a blog to make research more accessible to students and though's of us that are learners for life. His latest post is looking at a project Sensor touch: Fun in reaching out using Xbox controllers…sounds like fun – would also be interesting to see if there is a way to use the Xbox Messenger Kit – maybe to text to IM or twitter or even a remote LCD panel.

Sam Stokes on Research in your life and studies

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

"How to blog, get tenure and prosper"

eScience — Thu, 24 Jul 2008 19:57:20 GMT

One of the discussions that frequently happens at the eScience/Science 2.0 type meetings is how does academia incorporate the Web 2.0 technologies (Blogging, etc) into the tenure based reward system - these are technologies that change how quickly/accessible information can be and are not connected to the current papers to journal process. So I found it really interesting to read john hawks blog (Univ of Wisconsin - Madison) posting on getting tenure and blogging. The advice he gives is great...

Personally, I think that maturity as a scientist comes with the ability to explain your work to your parents.

The advice is not only good for blogging but also in communicating the research cross domains (ie. eScience confs). For talks - knowing the audience is a good thing and resist the need to show off your mental muscle.

Since it is a four-part series - check back for the other posts. I'm looking forward to the final installment - how john quantified "my blog's role as a service to the field and to the public."

How to blog, get tenure and prosper

Last month, the University of Wisconsin officially granted me tenure. So, I can say without any doubt (if other examples had not been sufficient), it is absolutely possible to write a daily, high-profile blog and still be recognized by your colleagues as a scholar. In fact, it is possible to blog, do good research, and earn tenure at a Research I university.

<...>

Happily things have changed. With the rise of science blogging, people have become much more aware of the ways that a blog can contribute to a career in science. If you establish a readership, the chances are your colleagues will find out about your blog themselves, instead of looking at you in befuddlement. Blogs are not research, but in some fields they have become an important part of the process of networking and critical commentary. A well-written blog is far from a liability to a scientific career, and may be a real boon.

How to blog, get tenure and prosper: Starting the blog | john hawks weblog

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Making Sense of Data Overload: An Innovative Approach to Progressive Data Analysis

eScience — Thu, 24 Jul 2008 01:14:18 GMT

It's really good to see that the datamining work (ProDA) that Cyrus Shahabi of USC is getting more visibility - the use of wavelet compression is a really neat way to deal with large amounts of data and make it easy to see the trends...you can see why folks like Chevron were interested in it.

Making Sense of Data Overload: An Innovative Approach to Progressive Data Analysis

When Professor Cyrus Shahabi of the University of Southern California decided to tackle the problem of complex data analysis, he was confronted by the limitations of current software. Realizing what an impediment this was for businesses and the scientific community, he began to explore alternative forms of analysis. When he came across signal processing and wavelet compression, he knew he was onto something, and ProDA was born. Since creating ProDA, NASA’s JPL and Chevron have had major successes using the program to manage their huge datasets. With the help of Microsoft Research’s Smart Client initiative, Shahabi was able to bring ProDA to the next level by making it more compatible with XML, Microsoft Excel, text files, and many more formats. All these changes have made ProDA more accessible and user friendly.

April 2008 article in IEEE Computer - ProDA: An End-to-End Wavelet-Based OLAP System for Massive Datasets by Cyrus Shahabi (USC)

Abstract

ProDA employs wavelets to support exact, approximate, and progressive OLAP queries on large multidimensional datasets, while keeping update costs relatively low. ProDA not only supports online execution of ad hoc analytical queries on massive datasets, but also extends the set of supported analytical queries to include the entire family of polynomial aggregate queries as well as the new class of plot queries.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

SCIENCE Article: An Earth Systems Science Agency?

eScience — Thu, 10 Jul 2008 23:41:19 GMT

Very interesting article in current issue of Science...

The United States faces unprecedented environmental and economic challenges in the decades ahead. Foremost among them will be climate
change, sea-level rise, altered weather patterns, declines in freshwater availability and quality, and loss of biodiversity. Addressing these challenges will require well-conceived, science-based, simultaneous responses on
multiple scales, from global and national, to regional and local. The executive and legislative branches of the federal government and of the states will have to transcend bureaucratic boundaries and become much more innovative in developing and implementing policy responses. We strongly believe organizational changes must be made at the federal level to align our public institutional infrastructure to address these challenges. The most pressing organizational change that is required is the establishment of an independent Earth Systems Science Agency formed by merging the National Oceanic and Atmospheric Administration (NOAA) and the U.S. Geological Survey (USGS).

An Earth Systems Science Agency
Mark Schaefer,^* D. James Baker, John H. Gibbons, Charles G. Groat, Donald Kennedy, Charles F. Kennel, David Rejeski
Addressing serious environmental and economic challenges in the United States will require organizational changes at the federal level.

An Earth Systems Science Agency -- Schaefer et al. 321 (5885): 44 -- Science

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

The Petabyte Problem: Scrubbing, Curating and Publishing Big Data

eScience — Fri, 04 Jul 2008 00:38:14 GMT

Carol Minton Morris has a good entry on the HatCheck Newsletter on Alex Szalay's keynote, “Scientific Publishing in the Era of Pedabyte Data,” at JCDL on June 19, 2008. I always enjoy listening to Alex and hearing his perspective, especially since he gets his hands dirty with the eScience work and has put together a really good team. and of course there is always the fact that he plays the lead guitar :-).

The Petabyte Problem: Scrubbing, Curating and Publishing Big Data

He suggests that the there is a science project pyramid–single lab at the base, multi-campus in the center, and international consortia on top. Often a scientific discipline will recognize the need for a major “giga” initiatives such as supercomputing research that is highly collaborative and distributed. The output from these efforts at every scale contain:

–Literature

–Derived and re-combined data

–Raw data

Szalay would like to see a continous feedback loop among these three aspects where data and analysis are always updating.

To answer the question, “How can you publish data so that others might recreate your results in 100 yrs.,” he referred to Gray’s laws of Data Engineering: scientific computing revolves around data; scale-out the solution for analysis; take the analysis to the data; start with 20 queries, and; go from working to working.

HatCheck Newsletter » Blog Archive » The Petabyte Problem: Scrubbing, Curating and Publishing Big Data

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Environmental Science related activities

eScience — Thu, 03 Jul 2008 03:09:42 GMT

Here are some good articles on Environmental Science related activities that our researchers at Microsoft Research are involved with.

All of these are from the Microsoft Enivronment site – also check out the Software Enabled Earth Blog

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

WorldWide Telescope is a powerful educational tool — a way of telling compelling stories about the Universe.

eScience — Mon, 30 Jun 2008 23:04:52 GMT

An On the Issues Essay came out last week featuring Alyssa Goodman, Professor of Astronomy, Harvard University. Highlights how WWT can be used to not only browse and view the Universe – but can be a powerful tool to allow Astronomers to get to data and make discoveries.

A Virtual Telescope: Microsoft’s WorldWide Telescope is a powerful educational tool — a way of telling compelling stories about the Universe.

Seven years ago, a graduate student and I were analyzing an unusual image of the gas jettisoned by a forming star, named PV Ceph, when we realized the image could best be explained if the young star were speeding across the sky ten times faster than normal. But confirming our hypothesis required us to spend two years accumulating, overlaying and analyzing many more images made using ground- and spacebased radio, infrared and optical telescopes.

Today, a project of this kind would be much easier thanks to the WorldWide Telescope, a rich, Web-based software application that anyone can download from www.worldwidetelescope.org. Released last month by Microsoft Research, the WorldWide Telescope stitches together images from the world’s best ground- and space-based telescopes to enable a seamless exploration of the Universe.

<…>

A Virtual Telescope: Microsoft’s WorldWide Telescope is a powerful educational tool — a way of telling compelling stories about the Universe.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

eScience @ Microsoft : Article

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

Praise for The Fourth Paradigm

TIME - WorldWideTelescope one of 50 Best Websites 2009

Adding quarters to the innovation machine

New Tools Mobilize Local Data to Study Global Environmental Issues from Berkeley Lab

HPC - MPI.NET 1.0 Is Now Released...!

Science News to justify half a bar of chocolate per week - lowers risk of heart disease

Accessible Research blog

"How to blog, get tenure and prosper"

Making Sense of Data Overload: An Innovative Approach to Progressive Data Analysis

SCIENCE Article: An Earth Systems Science Agency?

The Petabyte Problem: Scrubbing, Curating and Publishing Big Data

Environmental Science related activities

WorldWide Telescope is a powerful educational tool — a way of telling compelling stories about the Universe.