eScience @ Microsoft : eScience

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

eScience — Fri, 16 Oct 2009 17:39:17 GMT

Today Microsoft Research announced the availability of the book - The Fourth Paradigm: Data-Intensive Scientific Discovery. The book focuses on the change of all sciences moving from observational, to theoretical, to computational and now to the 4th Paradigm – Data-Intensive Scientific Discovery. This is based on Jim Gray’s insights captured via his final public talk to the National Research Council on Jan 11, 2007. This is truly a legacy of his work.

The book is available from the web and is released under a Creative Commons license.

Part 1: Earth and Environment	Part 3: Scientific Infrastructure
Part 2: Health and Wellbeing	Part 4: Scholarly Communication

I feel fortune to have been able to contribute the introduction to the Earth and Environment section - I had many discussions with Jim on need for balance between data and computations, and the need to make scientific exploration through the use of computing technologies much easier for scientists. I had also “borrowed” many of Jim’s slides to discuss the change to the upcoming fourth paradigm, he made the points so succinctly – there was no need for marketing fluff.

The Fourth Paradigm: Data-Intensive Scientific Discovery

Presenting the first broad look at the rapidly emerging field of data-intensive science

Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets.

The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud computing technologies.

In The Fourth Paradigm: Data-Intensive Scientific Discovery, the collection of essays expands on the vision of pioneering computer scientist Jim Gray for a new, fourth paradigm of discovery based on data-intensive science and offers insights into how it can be fully realized.

Praise for The Fourth Paradigm

“The impact of Jim Gray’s thinking is continuing to get people to think in a new way about how data and software are redefining what it means to do science."

— Bill Gates, Chairman, Microsoft Corporation

The Fourth Paradigm: Data-Intensive Scientific Discovery - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Project Gemini – Videos available

eScience — Wed, 26 Aug 2009 17:35:05 GMT

I previously posted about using Project Gemini as a tool for scientific analysis – as a quick way to learn about them and see them in action, take a look at the one minute Gemini videos by Donald Farmer. One of the latest videos is about using Reports as data sources and highlights the orange button for getting data sets. This is a perfect way for scientists to publish data and easily make the feeds available for others to consume. It could even be included in papers to easily enable research reproducibility.

Thanks to Robert Bruckner's Advanced Reporting Services Blog : Reports As Data Feeds for Gemini

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Science Analytics – look to use Project “Gemini”

eScience — Fri, 21 Aug 2009 19:34:37 GMT

When I first saw and heard details about Project “Gemini”, I was blown away by the technology and innovation created by SQL and Excel teams and that held up when I was able to test it out on my own. It will be especially useful for scientists that want to not only analyze large amounts of data in Excel, but also aggregate different datasets. This upcoming Excel 2010 add-in removes the storage limits of Excel by adding the in-memory database and brings the power of SQL Server and SQL Analysis Services into the hands of mere mortals. Scientists that utilize Excel for viewing/analyzing data will find this add-in extremely helpful.

Project Gemini Blog – Check out the videos -

Project "Gemini": Build powerful analytical applications

Need to make timely business decisions without having to use complicated and sluggish analytical applications? Love to use Excel? Project Gemini is an Excel 2010 add-in that allows you to create powerful analyses by quickly manipulating millions of rows of data into a single Excel workbook and utilize Microsoft Office 2010 to share and collaborate on your insights with your team.

You can combine native Excel 2010 functionality with Gemini’s in-memory engine to allow users to interactively explore and perform calculations on large data sets. In addition, you can easily streamline the process of integrating data from multiple sources – including corporate databases, spreadsheets, reports, and data feeds.

Share and collaborate with confidence by easily publishing your analysis to SharePoint 2010 and have other users enjoy the same slicer and fast-query capabilities when working on your Excel Services reports.

Are you part of Office 2010 Tech Preview? Register and then Download and learn about Project Gemini now! (Note: You need to have Office 2010 before you can use Gemini.)

Introducing Microsoft Office 2010 for Business

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

eScience Workshop 2009 – Call for Papers

eScience — Mon, 06 Jul 2009 20:48:57 GMT

The 2009 eScience Workshop will be held at the Gates Center for Computer Science at Carnegie Mellon University in Pittsburgh, PA, October 15-17, 2009. The call for papers closes on July 31, 2009.

Call for Papers

We invite contributions from all areas of eScience and e-Research including:

Computational support for scientific research in life sciences, biomedical computing, environment, energy, and other scientific grand challenges
Knowledge discovery and merging datasets
Large-scale scientific data analysis, mining, and visualization
High-performance computing applied to solving problems in a variety of scientific disciplines
Dissemination of scientific literature/results and the discovery, curation, and sharing of data
Scientific sensors, data-gathering tools and technologies
Collaboration/workflow tools and technologies
Data-intensive science
Emerging multidisciplinary fields such as Digital Heritage and eEconomy
Research implications of computational thinking
How strategies for semantics and ontology formulation enable scientific discovery

eScience Workshop 2009

The eScience Workshop, to be held October 15-17, 2009, will provide a unique opportunity to share experiences, learn new techniques, and influence the domain of scientific computing. Scientists and researchers will explore the evolution, challenges and potential of computing in scientific research, including how the latest tools, Web services and database technologies are being applied to scientific computing.

Workshop Theme
Facilitating Scientific Discovery through Data-Intensive Computing

Hosting and Location
Co-hosted by Microsoft Research and Carnegie Mellon University, this workshop will take place in the Gates Center for Computer Science at Carnegie Mellon University in Pittsburgh, PA.

eScience Workshop 2009 - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

WorldWide Telescope – Busy couple of weeks – NASA and SilverLight

eScience — Wed, 25 Mar 2009 23:53:06 GMT

Yesterday the joint press release went out on the collaboration and Space Act Agreement we’re been working on with NASA for sometime. We’re really excited about working with NASA to process many datasets like the Mars Reconnaissance Orbiter (MRO) and make them available in WorldWide Telescope. Having these images available in the TOAST (tesselated octahedral adaptive subdivision transform) projection format will not only benefit WWT but any viewer supporting that format. The benefit of using TOAST as Jonathan Fay one mentioned “It creates a 360-degree wraparound view that’s either a planet surface or the infinite sphere of the sky, and lets you represent it using a 3D graphics accelerator, very rapidly and efficiently. So we can have an image pyramid the way Deep Zoom does, and TerraServer before it, but we don’t have to give up the poles.”

NASA and Microsoft to Make Universe of Data Available to the Public

There is a really good article out talking about some of the background behind Curtis Wong and Jonathan Fay’s labor of love. WorldWide Telescope Puts Wonders of Space on a PC

The other big news is that at Mix09 we put out a alpha release of the worldwide telescope web client built using SilverLight. Now all the folks running Macs can see what all the buzz was behind Scoble’s post What made me cry: Microsoft’s World Wide Telescope from last year.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

MSR Open Tools to Enhance Scientific Research Efforts Building on Science Commons Ontologies

eScience — Wed, 11 Mar 2009 20:50:46 GMT

At ETech 2009 today, the announcement went out that Science Commons in conjunction with MSR External Research released the source code to two Word 2007 add-ins to allow scientists to markup terms and phrases in their documents/papers with ontologies, such as the ones at http://neurocommons.org. Some of this is based on previous collaboration with Phil Bourne from UCSD.

“Microsoft’s openness in working with the Science Commons has significant implications for the scientific research community because it will make it easy for authors to link their documents straight into the semantic Web of science — making that research, data and material easier to find and use,” said Philip E. Bourne, Ph.D.

You can seem more about the add-ins at Pablo’s Blog - Ontology Add-in for Word 2007

Microsoft Releases Open Tools to Enhance Scientific Research Efforts Building on Science Commons Ontologies

REDMOND, Wash., and SAN JOSE, Calif. — March 11, 2009 — The nuggets of information necessary for science to progress are often hard to find, submerged deep within the Web, or within databases that can’t be easily accessed or integrated. As a result, many scientists today work in relative isolation, follow blind alleys and unnecessarily duplicate existing research.

Addressing this critical challenge for researchers, Microsoft Corp. and Creative Commons announced today, before an industry panel at the O’Reilly Emerging Technology Conference (ETech 2009, http://en.oreilly.com/et2009 ), the release of the Ontology Add-in for Microsoft Office Word 2007 that will enable authors to easily add scientific hyperlinks as semantic annotations, drawn from ontologies, to their documents and research papers. Ontologies are shared vocabularies created and maintained by different academic domains to model their fields of study.

This Add-in will make it easier for scientists to link their documents to the Web in a meaningful way. Deployed on a wide scale, ontology-enabled scientific publishing will provide a Web boost to scientific discovery.

Science Commons, a division of Creative Commons, is incubating the adoption of semantic scientific publishing through creation of a robust database of ontologies (http://neurocommons.org) and development of supporting technical standards and code. Microsoft Research has built a technology bridge to enable the link between Microsoft Office Word 2007 and these ontologies.

[more]

Microsoft Releases Open Tools to Enhance Scientific Research Efforts Building on Science Commons Ontologies: Breakthrough collaboration helps researchers make easier connections on the Web.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

WWT at TechFest on NYTimes.com

eScience — Mon, 02 Mar 2009 19:49:24 GMT

The NYTimes.com article by Ashlee Vance, included a great picture by Stuart Isett for The New York Times showing the dome that was put together for TechFest to demonstrate the planetarium projection mode of WWT as well as the gesture interaction from Andy Wilson and Hrvoje Benko.

Microsoft Maps Course to a Jetsons-Style Future

Stuart Isett for The New York Times

Hrvoje Benko demonstrating a Microsoft projection system that lets people manipulate large video images with their hands.

Microsoft Maps Course to a Jetsons-Style Future - NYTimes.com

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

New Tools Mobilize Local Data to Study Global Environmental Issues from Berkeley Lab

eScience — Thu, 05 Feb 2009 07:06:12 GMT

Here’s a really good article from the folks at Lawrence Berkeley National Laboratory on the collaboration MSR has ongoing between LBL and the Berkeley Water Center. It highlights the use of databases for scientific information as Catharine mentions…

“One of the greatest challenges of the next century will be developing cyber-architectures that allow scientists to easily navigate their digital assets. Today, the internet has given environmental researchers instant access to a wealth of field data. Now, they need a scientific ‘safety deposit box’ system that will not only store this information, but also organize it so it is searchable and ready for analysis,” says van Ingen.

New Tools Mobilize Local Data to Study Global Environmental Issues

Guarding water supplies, protecting endangered species and curbing greenhouse gases is going high-tech. Environmental scientists are turning to innovative cyber-infrastructures and data-mining tools.

As they strive to develop effective strategies for guarding water supplies, protecting endangered species and curbing greenhouse gases, environmental scientists are turning to innovative cyber-infrastructures and data-mining tools developed by an ongoing collaboration between researchers at Lawrence Berkeley National Laboratory, Microsoft Research, and the University of California, Berkeley.

The Microsoft eScience program is the primary funder of this project, which is one of numerous ventures cultivated by the Berkeley Water Center (BWC). Launched approximately three years ago by researchers from the Berkeley Lab and UC Berkeley’s Colleges of Engineering and Natural Resources, the BWC marshals expertise from public institutions and the private sector in support of projects that enable science and public policy researchers to more easily access and work with water and environmental datasets.

“The most cost-efficient way to impact issues like global climate change and water management is to develop cyber-architectures that organize data and foster scientific collaboration,” says Susan Hubbard, staff scientist in the Berkeley Lab’s Earth Sciences Division and associate director of the BWC.

Environmental scientists typically collect data on a project-by-project basis, in campaigns targeted at very specific topics. One study may use NASA satellites to track annual rainfall of deserts around the globe, while another project sponsored by the National Science Foundation (NSF) might measure the annual water tables of the Sahara desert with commercial sensors. The data are then typically stored in local archive systems and accessed by researchers associated with that particular project. These sites are scattered across the country, tend to be aligned with specific campaigns, and are funded by a variety of organizations.

Rest of the article at: New Tools Mobilize Local Data to Study Global Environmental Issues « Berkeley Lab News Center

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

WorldWide Telescope Academic Development Kit Release -Microsoft Research

eScience — Thu, 22 Jan 2009 01:21:39 GMT

With this ADK, users can convert their own astronomical images/data to the format that can be read by WWT and share with other WWT users. Can’t wait to see more images/datasets made available.

WorldWide Telescope Academic Development Kit, January 2009 Release

The WorldWide Telescope (WWT) Academic Development Kit, January 2009 release contains two utilities that enable people to convert their astronomical images, panoramas, sky surveys, and planetary textures to a format that can be read by WWT and shared with other WWT users. It produces image pyramids of the photographs, thumbnails, and WTML files. WTML files are XML files in the WWT format that point to the images on the Internet and store details of how they are to be displayed in WWT and metadata such as image title and credits. The WWT SphereToaster Tool enables users to provide images in an equirectangular format that covers all or part of the inside or outside of a sphere. This includes, for example, cylindrical projections of panoramas and all-sky surveys. SphereToaster converts these to a different projection system—the TOAST system, currently unique to WWT—and then stores an image pyramid of the resulting TOAST-projected image. The tool also produces thumbnails and WTML files. The WWT StudyChopper Tool enables users to provide photographs of small parts of the sky, such as a high-resolution image of the Crab Nebula, and enter appropriate coordinate information and metadata. It creates image pyramids of the photographs, thumbnails, and WTML files. Once the output image pyramids and thumbnails are hosted by the user's servers and the WTML files are made available to others, anyone with access to the WTML files will be able to browse the images in WWT.

WorldWide Telescope Academic Development Kit, January 2009 Release - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Science Images in Photosynth

eScience — Tue, 20 Jan 2009 23:44:24 GMT

Here are a few really good synths of scientific images in Photosynth. You can find others using Microscopes or Biology. And there is even a Dissected Cat if you have a strong stomach.

Obelia, polyps, golangia by ppberk

Micrographs of powder coating material by SynthSets

Frog Kidney by ppberk

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

GrayWulf (on SQL Server) wins SC’08 Storage Challenge

eScience — Wed, 26 Nov 2008 09:52:32 GMT

Congratulations to Alex Szalay and his amazing team at JHU for winning the SC’08 Storage Challenge – with the entry GrayWulf:Scalable Clustered Architecture for Data Intensive Computing. GrayWulf – is implemented with SQL Server 2008

Abstract:

Data intensive computing presents a significant challenge for traditional supercomputing architectures that maximize FLOPS since CPU speed has surpassed IO capabilities of HPC systems and BeoWulf clusters. We present the architecture for a three tier commodity component cluster designed for a range of data intensive computations operating on petascale data sets. The design goal is a balanced system in terms of IO performance and memory size, according to Amdahl’s Laws. GrayWulf pays tribute to Jim Gray who stimulated the system and its design. The hardware currently installed at JHU exceeds one petabyte of storage and has 0.5 bytes/sec of I/O and 1 byte of memory for each CPU cycle. The GrayWulf provides almost an order of magnitude better balance than existing systems. Our benchmarks are based on date from the petascale Pan-STARRS project, building the largest sky survey to date. The benchmarks involve sequential searches over hundreds of terabytes.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Virtual Earth in full view

eScience — Tue, 28 Oct 2008 23:01:25 GMT

Now this is a way to view Virtual Earth – talk about an immersive experience. I would like to see how WorldWide Telescope would look on this display…

PDC2008 ShowOff Entry: Multi-channel Virtual Earth
Multi-channel Virtual Earth
Description
This video shows a multi-channel version of the Virtual Earth control running on a custom curved screen that provides a 180 degree horizontal field of view. The screen is created using eight high-end full 1080p projectors with a professional warping and blending system. The code is a modified version of a sample project with a custom camera class to properly adjust the FOV and camera offset for each projector and some code to synchronize the camera position across the network. The system is being controlled with a wireless Xbox 360 controller.

PDC2008 ShowOff Entry: Multi-channel Virtual Earth | briankel | Channel 9

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Data Mining Services in the Cloud – Mine your Data, Any Place, Any Time

eScience — Thu, 04 Sep 2008 22:24:38 GMT

This is great news - Software-plus-Services that any scientist/researcher could use. The SQL Server Data Mining folks have a Data Mining Service in the cloud they are testing out…I posted previously [OLAP and Scientific Data & Data Mining Addins for Office 2007 (Excel & Visio)] about the Excel addins that allow anyone with Excel to do Data Mining on Excel tables. Now the team is testing out SQL Server Data Mining Services – from which you can do the data mining directly from Excel 2007 or even upload a csv file.

So for fun – I downloaded a csv file of a stream gauge near Redmond into Excel and ran the “Highlight Exceptions” tool to find outliers in the dataset – it read the table, uploaded it to the service and in seconds returned the results - which included the number of outliers - in this case air temperature and it also highlighted in the table the rows. It was so easy. I can see it being used for many scientific datasets - even to clean them before doing other analysis, charting, graphs, uploads, etc.

The Table Analysis Tools included are:

Analyze Key Influencers
Detect Categories
Fill from Example
Forecast
Highlight Exceptions
Scenario Analysis
Prediction Calculator
Shopping Basket Analysis

Microsoft SQL Server Data Mining Services

Mine your Data, Any Place, Any Time

The SQL Server Data Mining team is working to extend the power and ease of use of SQL Server Data Mining to the Cloud. Our goal is provide services that allow you to build rich, predictive applications without worrying about server infrastructure, and showcase these services with cool applications that give you a glimpse of what’s possible. We bring you a technology preview of our work below. Enjoy!

Current Projects

Table Analysis Tools for the Cloud

Build powerful predictive reports on your data with just a few clicks!
- No data mining expertise required
- No server installation required
- All you need is your Internet connection

http://www.sqlserverdatamining.com/cloud/

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

F# – Sept CTP available and units of measure checking/inference

eScience — Sat, 30 Aug 2008 20:07:22 GMT

The September 2008 CTP of F# is now available for download. F# is a functional programming language for the .NET Framework and really should be looked at by scientists/researchers. Also check out the F# Developer Center on MSDN for more info and resources.

There are a lot of new features in this release – here’s a sampling:

Broadly improved Visual Studio 2008 integration, which allows F# users to scale from scripting and explorative development, up to large-scale component and application design, all within Visual Studio.
Improvements to the F# language and libraries to make them simpler and more regular.
An exciting new language feature, Units of Measure, which extends F#’s inference and strong typing to floating point data.

The Units of Measure checking and interference feature is very exciting feature and potentially most scientifically revolutionary programming language features around - scientists and engineers to check out. This is because the F# compiler knows the rules of units

When values of floating-point type are multiplied, the units are multiplied too; when they are divided, the units are divided too, and when taking square roots, the same is done to the units. So by the rule for multiplication, the expression inside sqrt above must have units m^2/s^2, and therefore the units of speedOfImpact must be m/s.

Take a look at the SolarSystem sample - A Solar System simulation application, taking advantage of Units of Measure in F# to do physics simulation. Andrew Kennedy, who researched, architected and implemented this feature has all the details.

Other F# resources:

Microsoft F# Developer Center

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

OLAP and Scientific Data

eScience — Thu, 21 Aug 2008 01:57:52 GMT

While I’ve been pushing the ideas of using OLAP data cubes to evaluate scientific data for awhile, I thought it might be a good time to pull together some relevant papers and links. I believe OLAP is ideal to help analyze large quantities of data including time series information...making it easier for the scientist/researcher to explore the data in real-time and from tools they know like Excel. For example the data served up on FluxData site is done by creating OLAP cubes using SQL Server Analysis Services.

A couple of tools/links that might be of interest as well:

The Data Mining Add-ins for Office 2007 - very useful since you can do much of the data mining directly from Excel. Microsoft SQL Server 2008 Data Mining Add-ins for Microsoft Office 2007
http://www.sqlserverdatamining.com
SQL Server 2008 Analysis Services
Microsoft Computational Biology Web Tools
Jamie’s blog

Here are a couple of papers that reference the use of OLAP for different types of scientific data.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

eScience @ Microsoft : eScience

The Fourth Paradigm: Data-Intensive Scientific Discovery – Book Released

The Fourth Paradigm: Data-Intensive Scientific Discovery

Praise for The Fourth Paradigm

Project Gemini – Videos available

Science Analytics – look to use Project “Gemini”

Project "Gemini": Build powerful analytical applications

eScience Workshop 2009 – Call for Papers

eScience Workshop 2009

WorldWide Telescope – Busy couple of weeks – NASA and SilverLight

MSR Open Tools to Enhance Scientific Research Efforts Building on Science Commons Ontologies

Microsoft Releases Open Tools to Enhance Scientific Research Efforts Building on Science Commons Ontologies

WWT at TechFest on NYTimes.com

Microsoft Maps Course to a Jetsons-Style Future

New Tools Mobilize Local Data to Study Global Environmental Issues from Berkeley Lab

New Tools Mobilize Local Data to Study Global Environmental Issues

WorldWide Telescope Academic Development Kit Release -Microsoft Research

WorldWide Telescope Academic Development Kit, January 2009 Release

Science Images in Photosynth

GrayWulf (on SQL Server) wins SC’08 Storage Challenge

Virtual Earth in full view

Data Mining Services in the Cloud – Mine your Data, Any Place, Any Time

Microsoft SQL Server Data Mining Services

F# – Sept CTP available and units of measure checking/inference

OLAP and Scientific Data