eScience @ Microsoft : Environment

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

eScience — Fri, 16 Oct 2009 17:39:17 GMT

Today Microsoft Research announced the availability of the book - The Fourth Paradigm: Data-Intensive Scientific Discovery. The book focuses on the change of all sciences moving from observational, to theoretical, to computational and now to the 4th Paradigm – Data-Intensive Scientific Discovery. This is based on Jim Gray’s insights captured via his final public talk to the National Research Council on Jan 11, 2007. This is truly a legacy of his work.

The book is available from the web and is released under a Creative Commons license.

Part 1: Earth and Environment	Part 3: Scientific Infrastructure
Part 2: Health and Wellbeing	Part 4: Scholarly Communication

I feel fortune to have been able to contribute the introduction to the Earth and Environment section - I had many discussions with Jim on need for balance between data and computations, and the need to make scientific exploration through the use of computing technologies much easier for scientists. I had also “borrowed” many of Jim’s slides to discuss the change to the upcoming fourth paradigm, he made the points so succinctly – there was no need for marketing fluff.

The Fourth Paradigm: Data-Intensive Scientific Discovery

Presenting the first broad look at the rapidly emerging field of data-intensive science

Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets.

The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud computing technologies.

In The Fourth Paradigm: Data-Intensive Scientific Discovery, the collection of essays expands on the vision of pioneering computer scientist Jim Gray for a new, fourth paradigm of discovery based on data-intensive science and offers insights into how it can be fully realized.

Praise for The Fourth Paradigm

“The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science."

— Bill Gates, Chairman, Microsoft Corporation

The Fourth Paradigm: Data-Intensive Scientific Discovery - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Achieving Climate Sustainability – Article in American Meteorological Society

eScience — Fri, 25 Sep 2009 20:29:02 GMT

Bill Gail has written a thoroughly thought provoking piece Achieving Climate Sustainability in the Bulletin of the American Meteorological Society. One of the things I really like about Bill is that he understands the challenge of climate sustainability and the intersection of the Earth system’s three fundamental components: physical, ecosystem, and human. Right now most definitions of sustainability only applies to climate as a physical system and thus overlooks the interactions of the physical system with both ecosystems and humans.

Achieving Climate Sustainability

William B. Gail

It is often assumed that climate change policies, including the Kyoto Protocol and the follow-on Copenhagen agreement now being negotiated, align well with sustainability's tenets. A closer look reveals this is not the case. First, they treat climate change as a one-time problem - anthropogenic greenhouse gases - with a one-time solution. In contrast, research has begun to reveal that human-caused climate change is far from monolithic. Moreover, the clear trend is for societal climate influence to increase and diversify, not decline and simplify. Second, they fail to address the impact of natural climate change on ecosystems and society, an area that is less-well understood than the public commonly believes. A sustainable framework that guides human interaction with Earth's climate system must encompass the broader aspects of climate change and reconcile the reality of ongoing human influence. This includes the highly-controversial use of overt human influence to benefit society and ecosystems.

Achieving climate sustainability will be far from straightforward, if we even choose to proceed. The concept unearths deeply-held philosophical and religious conflicts, stretches our scientific capabilities, and forces us to address a considerable spectrum of practical concerns. Should we not choose to embrace it, we will find that our policies become less and less effective with time as climate problems expand beyond society's ability to avoid or eliminate them individually. This article elaborates on the need to include sustainability within the climate dialogue and explores the complex considerations that will quickly become part of the public debate.

CAPSULE SUMMARY

As humans and nature become increasingly interconnected, there is a need for public dialogue about sustainability as a framework for addressing climate change.

AMS Online Journals - Achieving Climate Sustainability

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Environment and Energy Workshop at Faculty Summit 2009

eScience — Wed, 15 Jul 2009 02:35:32 GMT

For the past two days at the Microsoft Research Faculty Summit, we hare hosted a Environment and Energy Workshop to look at areas where computing and IT can help solve some of these grand challenges. Output from the workshop will be posted on the site in the next few weeks.

Faculty Summit 2009 - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Virtual Earth in full view

eScience — Tue, 28 Oct 2008 23:01:25 GMT

Now this is a way to view Virtual Earth – talk about an immersive experience. I would like to see how WorldWide Telescope would look on this display…

PDC2008 ShowOff Entry: Multi-channel Virtual Earth
Multi-channel Virtual Earth
Description
This video shows a multi-channel version of the Virtual Earth control running on a custom curved screen that provides a 180 degree horizontal field of view. The screen is created using eight high-end full 1080p projectors with a professional warping and blending system. The code is a modified version of a sample project with a custom camera class to properly adjust the FOV and camera offset for each projector and some code to synchronize the camera position across the network. The system is being controlled with a wireless Xbox 360 controller.

PDC2008 ShowOff Entry: Multi-channel Virtual Earth | briankel | Channel 9

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Data Mining Services in the Cloud – Mine your Data, Any Place, Any Time

eScience — Thu, 04 Sep 2008 22:24:38 GMT

This is great news - Software-plus-Services that any scientist/researcher could use. The SQL Server Data Mining folks have a Data Mining Service in the cloud they are testing out…I posted previously [OLAP and Scientific Data & Data Mining Addins for Office 2007 (Excel & Visio)] about the Excel addins that allow anyone with Excel to do Data Mining on Excel tables. Now the team is testing out SQL Server Data Mining Services – from which you can do the data mining directly from Excel 2007 or even upload a csv file.

So for fun – I downloaded a csv file of a stream gauge near Redmond into Excel and ran the “Highlight Exceptions” tool to find outliers in the dataset – it read the table, uploaded it to the service and in seconds returned the results - which included the number of outliers - in this case air temperature and it also highlighted in the table the rows. It was so easy. I can see it being used for many scientific datasets - even to clean them before doing other analysis, charting, graphs, uploads, etc.

The Table Analysis Tools included are:

Analyze Key Influencers
Detect Categories
Fill from Example
Forecast
Highlight Exceptions
Scenario Analysis
Prediction Calculator
Shopping Basket Analysis

Microsoft SQL Server Data Mining Services

Mine your Data, Any Place, Any Time

The SQL Server Data Mining team is working to extend the power and ease of use of SQL Server Data Mining to the Cloud. Our goal is provide services that allow you to build rich, predictive applications without worrying about server infrastructure, and showcase these services with cool applications that give you a glimpse of what’s possible. We bring you a technology preview of our work below. Enjoy!

Current Projects

Table Analysis Tools for the Cloud

Build powerful predictive reports on your data with just a few clicks!
- No data mining expertise required
- No server installation required
- All you need is your Internet connection

http://www.sqlserverdatamining.com/cloud/

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

NOAA – 2009 Climate Program Announcement

eScience — Fri, 18 Jul 2008 18:47:25 GMT

Just received the Sectoral Applications Research Program announcement from Nancy Beller-Simms at the NOAA Climate Program Office. The due date for the proposal is 5:00 P.M. Eastern Time October 9, 2008.

The overall NOAA 2009 Climate Program Announcement is also available – competitions in the follow areas:

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

SCIENCE Article: An Earth Systems Science Agency?

eScience — Thu, 10 Jul 2008 23:41:19 GMT

Very interesting article in current issue of Science...

The United States faces unprecedented environmental and economic challenges in the decades ahead. Foremost among them will be climate
change, sea-level rise, altered weather patterns, declines in freshwater availability and quality, and loss of biodiversity. Addressing these challenges will require well-conceived, science-based, simultaneous responses on
multiple scales, from global and national, to regional and local. The executive and legislative branches of the federal government and of the states will have to transcend bureaucratic boundaries and become much more innovative in developing and implementing policy responses. We strongly believe organizational changes must be made at the federal level to align our public institutional infrastructure to address these challenges. The most pressing organizational change that is required is the establishment of an independent Earth Systems Science Agency formed by merging the National Oceanic and Atmospheric Administration (NOAA) and the U.S. Geological Survey (USGS).

An Earth Systems Science Agency
Mark Schaefer,^* D. James Baker, John H. Gibbons, Charles G. Groat, Donald Kennedy, Charles F. Kennel, David Rejeski
Addressing serious environmental and economic challenges in the United States will require organizational changes at the federal level.

An Earth Systems Science Agency -- Schaefer et al. 321 (5885): 44 -- Science

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

NGO - Nonprofit Connection

eScience — Thu, 10 Jul 2008 22:48:28 GMT

Just ran across the NGO Connection site – this is the place for non-profit orgs in the scientific space to look into….

NGO Connection is an online resource for non-profit groups around the world that provides resources to help advance the causes of the NGOs. See http://www.microsoft.com/ngo for more information about

Getting free and discounted software

Free access to Online software such as Windows Live, Microsoft Office Live and other online applications

Software grants through the Microsoft Software Donation program

Access to low cost personal computers and other hardware products

Refurbished Computers from the Community Microsoft Authorized Refurbisher (MAR) program

donated and discounted hardware from TechSoup for Refurbished computers, networks accessories, switches and routers

learning about software support resources and other grant benefits

Microsoft Support Resources

Software Grant Support

getting training material and information about obtaining certification

NGO Connection

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Environmental Science related activities

eScience — Thu, 03 Jul 2008 03:09:42 GMT

Here are some good articles on Environmental Science related activities that our researchers at Microsoft Research are involved with.

All of these are from the Microsoft Enivronment site – also check out the Software Enabled Earth Blog

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Fluxnet move to Silverlight v2.0 Beta 1

eScience — Mon, 28 Apr 2008 21:25:47 GMT

Savas has been coding again :-) - he's updated the FLUXNET Data set visualization to Silverlight v2.

Fluxnet move to Silverlight v2.0 Beta 1

It's been a while since I touched the Fluxnet code. Given that the original release was built on Silverlight v1.1, I put some time today in re-implementing it in Silverlight v2.0 Beta 1.

Enjoy.

Fluxnet move to Silverlight v2.0 Beta 1

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Autonomous Monitoring of Vulnerable Habitats

eScience — Fri, 04 Apr 2008 02:35:20 GMT

There is a new article on the MSR Cambridge project on Skomer Island -Microsoft scientists unveil new technology to give early warning of sensitive species’ response to changing environments. Also interesting is the use of the ScatterWeb .NET SDK from the Freie Universität Berlin.

ScatterWeb .NET SDK is a new approach to working with wireless sensor networks. It hides the complexity of embedded programming and makes it easy to handle objects representing wireless sensors.

Autonomous Monitoring of Vulnerable Habitats
In this project, we are developing a number of technologies that we hope will be instrumental in enabling this goal. We are currently using this system on Skomer Island to monitor the behaviour of the Manx Shearwater.
In close collaboration with Oxford University and the Freie Universitat Berlin, we have developed an innovative system that can autonomously monitor animals and their habitats. Using a combination of wireless sensor networks and innovative software we are able to intelligently gather data from remote locations, relaying it back for local storage and processing and allow remote reconfiguration of the network by research scientists.

Autonomous Monitoring of Vulnerable Habitats

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Environmental Scenario Search Engine Research Project Portal

eScience — Tue, 22 Jan 2008 18:36:14 GMT

Here's a link to a Russian based project - where they are mining environmental data archives. It even has NASA WorldWind visualization plugins....there is also a web based test site.

Environmental Scenario Search Engine Research Project Portal

The main idea behind ESSE is a flexible, efficient and easy to use search engine for data mining in environmental data archives. What makes it so different from conventional text-based search engines is that it actually searches inside the numeric datasets. With ESSE scientists will be able to find specific parameter values, conditions, and scenarios among the huge amount of available environmental data. ESSE will help you find useful data even if you don't know exactly what you are looking for.

By using its fuzzy logic capabilities you can construct complex queries using vague linguistic terms, such as "very large" or "average", or set fuzzy numeric thresholds (for example, "about 100"). If you are an experienced researcher you can still use exact parameter values in your queries. ESSE uses Common Data Model for internal data representation, thus acting as a bridge between the user and the multiple heterogeneous data sources.

Environmental Scenario Search Engine Research Project Portal

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

eScience @ Microsoft : Environment

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

The Fourth Paradigm: Data-Intensive Scientific Discovery

Praise for The Fourth Paradigm

Achieving Climate Sustainability – Article in American Meteorological Society

Achieving Climate Sustainability

Environment and Energy Workshop at Faculty Summit 2009

Virtual Earth in full view

Data Mining Services in the Cloud – Mine your Data, Any Place, Any Time

Microsoft SQL Server Data Mining Services

NOAA – 2009 Climate Program Announcement

SCIENCE Article: An Earth Systems Science Agency?

An Earth Systems Science Agency

NGO - Nonprofit Connection

Environmental Science related activities

Fluxnet move to Silverlight v2.0 Beta 1

Autonomous Monitoring of Vulnerable Habitats

Environmental Scenario Search Engine Research Project Portal