Today Microsoft Research announced the availability of the book - The Fourth Paradigm: Data-Intensive Scientific Discovery.  The book focuses on the change of all sciences moving from observational, to theoretical, to computational and now to the 4th Paradigm – Data-Intensive Scientific Discovery.  This is based on Jim Gray’s insights captured via his final public talk to the National Research Council on Jan 11, 2007. This is truly a legacy of his work.

The book is available from the web and is released under a Creative Commons license.

Part 1: Earth and Environment	Part 3: Scientific Infrastructure
Part 2: Health and Wellbeing	Part 4: Scholarly Communication

I feel fortune to have been able to contribute the introduction to the Earth and Environment section -  I had many discussions with Jim on need for balance between data and computations, and the need to make scientific exploration through the use of computing technologies much easier for scientists.  I had also “borrowed” many of Jim’s slides to discuss the change to the upcoming fourth paradigm, he made the points so succinctly – there was no need for marketing fluff.

The Fourth Paradigm: Data-Intensive Scientific Discovery

Presenting the first broad look at the rapidly emerging field of data-intensive science

Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets.

The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud computing technologies.

In The Fourth Paradigm: Data-Intensive Scientific Discovery, the collection of essays expands on the vision of pioneering computer scientist Jim Gray for a new, fourth paradigm of discovery based on data-intensive science and offers insights into how it can be fully realized.

Praise for The Fourth Paradigm

“The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science."

— Bill Gates, Chairman, Microsoft Corporation

The Fourth Paradigm: Data-Intensive Scientific Discovery - Microsoft Research

Now that AWS is supporting SDSS, there is probably useful integration between the Worldwide Telescope and the SDSS datasets that can be leveraged, beyond what is currently supported - such as seeing the all the known galaxies in a 3D view – seeing the lattice structure of the universe. 

Search and Browse data from the Virtual Observatory and plot data over imagery: WorldWide Telescope delivers one-click contextual access to distributed Web information and data sources and Interoperates through SAMP and other popular tools like TopCat, Aladin, SAOImage DS9 and many more.

Data derived from researchers using the AWS datasets can integrate imagery and catalog data directly into WorldWide Telescope using the WWT Developer Kit.  

Professional features

• Virtual Observatory Cone search/registry look up and SIMBAD search
• Load and Adjust basic FITS images and AVM files
• SIAP with footprint preview
• Connect your Telescope to WWT
• Multi-monitor cluster rendering
• Visualization of large scale structure
• SAMP Inter-application communication
• Full dome projection
  

New Public Data Set: Sloan Digital Sky Survey DR6 Subset

The Sloan Digital Sky Survey, or SDSS, is now available as a Public Data Set.

Weighing in at 180 GB, the SDSS is the most ambitious astronomical survey ever undertaken. The researchers have used a 2.5 meter, 120 megapixel telescope located in Apache Point, New Mexico to capture images of over one quarter of the sky, or about 230 million celestial objects. They have also created 3-dimensional maps containing more than 930,000 galaxies and 120,000 quasars.

Amazon Web Services Blog: New Public Data Set: Sloan Digital Sky Survey DR6 Subset

Bill Gail has written a thoroughly thought provoking piece Achieving Climate Sustainability in the Bulletin of the American Meteorological Society.  One of the things I really like about Bill is that he understands the challenge of climate sustainability and the intersection of the Earth system’s three fundamental components: physical, ecosystem, and human. Right now most definitions of sustainability only applies to climate as a physical system and thus overlooks the interactions of the physical system with both ecosystems and humans. 

Achieving Climate Sustainability

William B. Gail

It is often assumed that climate change policies, including the Kyoto Protocol and the follow-on Copenhagen agreement now being negotiated, align well with sustainability's tenets. A closer look reveals this is not the case. First, they treat climate change as a one-time problem - anthropogenic greenhouse gases - with a one-time solution. In contrast, research has begun to reveal that human-caused climate change is far from monolithic. Moreover, the clear trend is for societal climate influence to increase and diversify, not decline and simplify. Second, they fail to address the impact of natural climate change on ecosystems and society, an area that is less-well understood than the public commonly believes. A sustainable framework that guides human interaction with Earth's climate system must encompass the broader aspects of climate change and reconcile the reality of ongoing human influence. This includes the highly-controversial use of overt human influence to benefit society and ecosystems.

Achieving climate sustainability will be far from straightforward, if we even choose to proceed. The concept unearths deeply-held philosophical and religious conflicts, stretches our scientific capabilities, and forces us to address a considerable spectrum of practical concerns. Should we not choose to embrace it, we will find that our policies become less and less effective with time as climate problems expand beyond society's ability to avoid or eliminate them individually. This article elaborates on the need to include sustainability within the climate dialogue and explores the complex considerations that will quickly become part of the public debate.

CAPSULE SUMMARY

As humans and nature become increasingly interconnected, there is a need for public dialogue about sustainability as a framework for addressing climate change.

AMS Online Journals - Achieving Climate Sustainability

I previously posted about using Project Gemini as a tool for scientific analysis – as a quick way to learn about them and see them in action, take a look at the one minute Gemini videos by Donald Farmer.  One of the latest videos is about using Reports as data sources and highlights the orange button for getting data sets.  This is a perfect way for scientists to publish data and easily make the feeds available for others to consume.  It could even be included in papers to easily enable research reproducibility. 

Thanks to Robert Bruckner's Advanced Reporting Services Blog : Reports As Data Feeds for Gemini

TIME magazine has the WorldWide Telescope WebClient (Silverlight) on their list of 50 Best Websites for 2009.  That puts WWT in with sites like Flickr, Twitter, Skype, YouTube, Amazon, and even PhotoSynth – not bad for the very small team we have on it :-)

WorldWideTelescope

Like Google Earth for the heavens, WWT aggregates terabytes of astronomical data from the world's biggest telescopes to create a single virtual scope that anyone can look through. WWT is not a model of the known universe, but rather a centralized repository for just about everything known about the universe. The idea is to democratize the science of astronomy with a single tool that can be used by students and scientists. Who knows, when everyone has access to the same data, maybe the next big discovery in astronomy will be made by an amateur? There are hundreds of terabytes of digitized sky — enough data for everyone

WorldWideTelescope - 50 Best Websites 2009 - TIME

When I first saw and heard details about Project “Gemini”, I was blown away by the technology and innovation created by SQL and Excel teams and that held up when I was able to test it out on my own.  It will be especially useful for scientists that want to not only analyze large amounts of data in Excel, but also aggregate different datasets. This upcoming Excel 2010 add-in removes the storage limits of Excel by adding the in-memory database and brings the power of SQL Server and SQL Analysis Services into the hands of mere mortals.  Scientists that utilize Excel for viewing/analyzing data will find this add-in extremely helpful.

Project Gemini Blog – Check out the videos -

Project "Gemini": Build powerful analytical applications

Need to make timely business decisions without having to use complicated and sluggish analytical applications? Love to use Excel? Project Gemini is an Excel 2010 add-in that allows you to create powerful analyses by quickly manipulating millions of rows of data into a single Excel workbook and utilize Microsoft Office 2010 to share and collaborate on your insights with your team.

You can combine native Excel 2010 functionality with Gemini’s in-memory engine to allow users to interactively explore and perform calculations on large data sets. In addition, you can easily streamline the process of integrating data from multiple sources – including corporate databases, spreadsheets, reports, and data feeds.

Share and collaborate with confidence by easily publishing your analysis to SharePoint 2010 and have other users enjoy the same slicer and fast-query capabilities when working on your Excel Services reports.

Are you part of Office 2010 Tech Preview? Register and then Download and learn about Project Gemini now! (Note: You need to have Office 2010 before you can use Gemini.)

Introducing Microsoft Office 2010 for Business

Deep in one of the garages on campus – I ran across this project being built…a modern Stonehenge? maybe the spinal tap version of Stonehenge?

View Full Album

If you’d like to see instant results while you’re typing in your search query try out “The Real Live Search” from Long Zheng from istartedsomething – it uses the fast Bing AJAX APIs and the JSON results to give you information as you type.  It’s really neat to see – here’s an example doing a WorldWide Telescope search  The image doesn’t do it justice – try it out yourself.

I wonder if it could be integrated with the Windows 7 via the Federated Search support – much like Long was able to do with the Flickr Search Connector for Windows 7

The Real Live Search – Bing API experiment - istartedsomething

Earlier today the code behind the SciScope site was made available at SciScope.CodePlex.com.  This enables others to make their datasets/repositories available and allow others to discover, download and utilize their data in a simple to use website.     Also the semantic support is quite useful in finding related data.

SciScope Project Description

SciScope (see it live) is a prototype web application that allows data discovery from across multiple distributed heterogeneous data repositories. It leverages Bing Maps (formerly Microsoft Virtual Earth) and Microsoft SQL Server 2008 to support queries involving spatial, temporal and thematic constraints over an index of sensors operated by agencies such as USGS, EPA and NOAA as well as user provided data. SciScope leverages taxonomies stored as triples in SQL Server to provide search suggestions and for dealing with semantic heterogeneity between different data repositories.

_{SciScope screenshots (discovering/downloading insecticide data, browsing ecoregions left to right) for video tutorials click here}
This CodePlex release includes some desktop tools to simplify data publishing and content crawling for SciScope namely Catalog Publisher and Catalog Updater.

SciScope - Home

Project Tuva, an enhanced video player showcasing Richard Feynman’s “Messenger” lectures is available for all to try out.  It’s the way I’d like to view talks and related information – check it out.  Not only does it allow for web links, images, but it also integrates with the WorldWide Telescope control to help augment the example Feynman uses in the gravitational talk.  Currently only the first lecture in the series “Law of Gravitation – an Example of Physical Law” utilizes all the annotations/links, but the do all have the transcripts, so you can search on something like “particles” and see where it is mentioned in all the different videos, and then jump directly to the location.

Microsoft Research and Bill Gates Bring Historic Physics Lectures to Web

Lecture series by celebrated physics professor Richard Feynman is now available to all.

REDMOND, Wash. — July 14, 2009 — Microsoft Research, in collaboration with Microsoft Corp. Chairman Bill Gates, today launched a Web site that makes an acclaimed lecture series by the iconic physicist Richard Feynman freely available to the general public for the first time. The lectures, which Feynman originally delivered at Cornell University in 1964, have been hugely influential for many people, including Gates. Gates privately purchased the rights to the seven lectures in the series, called “The Character of Physical Law,” to make them widely available to the public for free with the hope that they will help get kids excited about physics and science.

The historic lectures and related content can be seen at http://research.microsoft.com/tuva. The name “Tuva” was chosen because of Feynman’s lifelong fascination with the small Russian republic of Tuva, located in the heart of Asia.

Feynman was one of the most popular scientists of the 20th century, equally regarded for his scientific insights as well as his ability to convey his enthusiasm for science through his lectures and writings. He shared the Nobel Prize in Physics in 1965 and was also known for his quirky sense of humor and eccentric and wide-ranging interests.

“No one was more adept at making science fun and interesting than Richard Feynman,” said Gates. “More than 20 years after first seeing them, these are still some of the best science lectures I’ve heard. Feynman worked hard during his life to popularize science, so I’m sure he’d be thrilled that now anyone, anywhere in the world, can just click a button and experience his lectures.”

Curtis Wong, a principal researcher with Microsoft Research, enhanced the experience of viewing the lectures by integrating the historic video with a Microsoft Silverlight-based video player that allows viewers to search the lectures for references to particular subjects, take notes that are synchronized to the video, and click on hyperlinks to related Web content, among other customized operations.

“There is a lot of public interest in building innovative educational resources online,” Wong said. “This is an opportunity to take some existing educational content and utilize software and the wealth of resources available on the Web to create a richer learning experience. And because people can annotate the lectures with their own comments and links to related resources, I expect this experience to become richer and richer over time.”

Microsoft Research has been exploring video annotation for many years and chose to publish the Feynman “Messenger” lectures with a new enhanced video player. Neither Microsoft nor the Bill and Melinda Gates Foundation were involved in the acquisition of the rights to the lectures.

Microsoft Research and Bill Gates Bring Historic Physics Lectures to Web: Lecture series by celebrated physics professor Richard Feynman is now available to all.

Project Trident CTP is now available for download.  Project Trident is a scientific workflow workbench MSR External Research has been working on for the past few years, which allows scientists to analyze large, diverse datasets.  It’s built on Windows Workflow and utilizes SQL Server (Express or Server).  Download it and try it out…

Project Trident

Built on the Windows Workflow Foundation, this scientific workflow workbench allows users to:

• Automate analysis and then visualize and explore data
• Compose, run, and catalog experiments as workflows
• Capture provenance for each experiment
• Create a domain-specific workflow library to extend the functionality of the workflow workbench
• Use existing services, such as provenance and fault tolerance, or add new services
• Schedule workflows over HPC clusters or cloud computing resources

 

Project Trident: A Scientific Workflow Workbench - Microsoft Research

For the past two days at the Microsoft Research Faculty Summit, we hare hosted a Environment and Energy Workshop to look at areas where computing and IT can help solve some of these grand challenges.  Output from the workshop will be posted on the site in the next few weeks.

Faculty Summit 2009 - Microsoft Research

Just heard that the Michael Jackson Memorial concert will be broadcast live in HD over the Web.  They will be using IIS Smooth Streaming and Silverlight, the technology that delivered on-demand video for the 2008 Sumer Olympics on NBCOlympics.com.

Michael Jackson Memorial Concert | Sympatico / MSN inMusic

On July 22nd there will be a total eclipse of the Sun that will last for over 6 mins that is visible from within a narrow corridor that traverses half of Earth.  One of the best regions to see it will be China and the organizers have published a WorldWide Telescope tour previewing the Eclipse.

See the trailer (below) for the project, "Multi-site Federated Live Broadcast of Solar Eclipse on July 22, International Year of Astronomy 2009"

The 2009 eScience Workshop will be held at the Gates Center for Computer Science at Carnegie Mellon University in Pittsburgh, PA,  October 15-17, 2009.  The call for papers closes on July 31, 2009.

Call for Papers

We invite contributions from all areas of eScience and e-Research including:

• Computational support for scientific research in life sciences, biomedical computing, environment, energy, and other scientific grand challenges
• Knowledge discovery and merging datasets
• Large-scale scientific data analysis, mining, and visualization
• High-performance computing applied to solving problems in a variety of scientific disciplines
• Dissemination of scientific literature/results and the discovery, curation, and sharing of data
• Scientific sensors, data-gathering tools and technologies
• Collaboration/workflow tools and technologies
• Data-intensive science
• Emerging multidisciplinary fields such as Digital Heritage and eEconomy
• Research implications of computational thinking
• How strategies for semantics and ontology formulation enable scientific discovery

eScience Workshop 2009

The eScience Workshop, to be held October 15-17, 2009, will provide a unique opportunity to share experiences, learn new techniques, and influence the domain of scientific computing. Scientists and researchers will explore the evolution, challenges and potential of computing in scientific research, including how the latest tools, Web services and database technologies are being applied to scientific computing.

Workshop Theme
Facilitating Scientific Discovery through Data-Intensive Computing

Hosting and Location
Co-hosted by Microsoft Research and Carnegie Mellon University, this workshop will take place in the Gates Center for Computer Science at Carnegie Mellon University in Pittsburgh, PA.

eScience Workshop 2009 - Microsoft Research