Dan on eScience & Technical Computing @ Microsoft : Data Analysis

Project Gemini – Videos available

Dan Fay — Wed, 26 Aug 2009 17:35:05 GMT

I previously posted about using Project Gemini as a tool for scientific analysis – as a quick way to learn about them and see them in action, take a look at the one minute Gemini videos by Donald Farmer. One of the latest videos is about using Reports as data sources and highlights the orange button for getting data sets. This is a perfect way for scientists to publish data and easily make the feeds available for others to consume. It could even be included in papers to easily enable research reproducibility.

Thanks to Robert Bruckner's Advanced Reporting Services Blog : Reports As Data Feeds for Gemini

Graywulf Takes Byte Out of Data Overload

Dan Fay — Fri, 08 May 2009 20:46:17 GMT

Graywulf is the natural evolution of Beowulf Clusters – it brings together HPC clusters and databases to do efficient processing and data management. It’s name and design also pays homage to Jim Gray – who helped champion the use of relational databases in the scientific projects.

At it’s simplest form Graywulf is having a database installed on each of the HPC compute nodes – this brings the data to the computation – one of the points Jim made quite often and utilizes the power of databases (queries, stored procedures, etc). Since it’s a generic architecture Graywulf clusters can be built using any OS and any database…the ones in the case study below implemented them using Windows HPC Server and SQL Server and the motivation was to be more efficient in doing the science – it’s always great to have innovative folks using technologies to do good work.

“To put it simply, a scientist needs to be able to live within the data,” says Alexander Szalay, a cosmologist-turned-computer-scientist at The Johns Hopkins University (JHU) in Baltimore, Maryland. The power of information, Szalay says, is determined not by its quantity so much as how easy it is to access, manipulate and analyze.
“It’s not just about doing the numerical calculations,” adds Andrew Simms, a biomedical health informatics graduate student working on protein structure analysis in Valerie Daggett’s bioengineering laboratory at the University of Washington (UW) in Seattle. “It’s also about assembling the data so we can run calculations while performing analyses and ad hoc explorations and then feed it all back into the data warehouse.”

Graywulf Takes Byte Out of Data Overload

Astronomers at The Johns Hopkins University and protein scientists at the University of Washington are using inexpensive computer hardware combined with powerful computing and database software to help manage and analyze a growing volume of scientific data.

For details, read the Graywulf case study.

Project Principals

Alexander Szalay, Alumni Centennial Professor, Department of Physics and Astronomy, The Johns Hopkins University

Valerie Daggett, Professor of Bioengineering, University of Washington

Graywulf Takes Byte Out of Data Overload - Microsoft Research

new CloudApp(): Azure™ Developer Challenge

Dan Fay — Tue, 05 May 2009 19:53:51 GMT

The Azure folks have setup the new CloudApp () challenge – I’d like to see some Science apps created on Azure and see how it changes the way to work with different datasets.

new CloudApp() - The Azure™ Services Platform Developer Challenge

What Is It?

new CloudApp() is a US-based developer challenge for .NET & PHP developers creating cloud applications or services (hereafter "application") on the Azure™ Services Platform. Have your application judged by industry leaders Om Malik and Michael Cote and share your cloud coding skills with other developers. Grand Prize Winners will be announced on stage at Structure 09 and featured on azure.com.

new CloudApp(): The Azure™ Services Platform Developer Challenge - Home

Using Flickr for Astronomy – and viewing in WWT

Dan Fay — Sat, 21 Feb 2009 01:55:17 GMT

The use of online services such as Flickr to help scientists is in its infancy and applications utilizing commodity based solutions will continue to pick up momentum. I especially like the integration and the ease of use – science should be about discovery and exploration – not about the technology. Of course the ability to view those analyzed images in WorldWide Telescope completes the circle and allows you to view the image in context.

Check out the Orion Nebula.
After it opens up – click on the thumbnail at the top.

You can also add your own – check out Dinoj’s post on the WWT Data Blog - Sticking images on the sky with WWT. For fun you can see the crown for the Corona Borealis overlaid on the sky

See the article written by Frederic Lardinois on ReadWriteWeb.

The Great Gig in the Sky: Using Flickr for Astronomy

Flickr hosts a wide range of beautiful images, but a new project built on top of Flickr's API only focuses on photos of the night sky from amateur astronomers. The Astrometry.net project constantly scans the Astrometry Flickr group for new images to catalog and to add to its open-source sky survey. At the same time, this project also provides a more direct service to the amateur astronomers, as it also analyzes each image and returns a high-quality description of the photo's contents.

The Astrometry group currently has over 400 members, and as Christoper Stumm, a member of the Astrometry.net team, told the Flickr Code blog, the back-end software uses geometric hashing to exactly pinpoint and describe the objects in the images. When you submit an image to the Flickr pool, the robot will not just respond with a comment that contains an exact description of what you see in the image, but it will also annotate the image automatically.

While a lot of members of the Astrometry group use high-end telescopes and cameras, the Astrometry.net solver can also analyze images from consumer-level digital cameras.

While just being able to automatically analyze and catalog these images is pretty cool already, every description also contains a link that displays the image in Microsoft's WordWide Telescope.

Astronomy is one of those few scientific disciplines where dedicated amateurs can still make major discoveries and this is definitely one of the cooler applications of Flickr's API that we have seen in a long time.

The Great Gig in the Sky: Using Flickr for Astronomy - ReadWriteWeb

New Tools Mobilize Local Data to Study Global Environmental Issues from Berkeley Lab

Dan Fay — Thu, 05 Feb 2009 07:06:12 GMT

Here’s a really good article from the folks at Lawrence Berkeley National Laboratory on the collaboration MSR has ongoing between LBL and the Berkeley Water Center. It highlights the use of databases for scientific information as Catharine mentions…

“One of the greatest challenges of the next century will be developing cyber-architectures that allow scientists to easily navigate their digital assets. Today, the internet has given environmental researchers instant access to a wealth of field data. Now, they need a scientific ‘safety deposit box’ system that will not only store this information, but also organize it so it is searchable and ready for analysis,” says van Ingen.

New Tools Mobilize Local Data to Study Global Environmental Issues

Guarding water supplies, protecting endangered species and curbing greenhouse gases is going high-tech. Environmental scientists are turning to innovative cyber-infrastructures and data-mining tools.

As they strive to develop effective strategies for guarding water supplies, protecting endangered species and curbing greenhouse gases, environmental scientists are turning to innovative cyber-infrastructures and data-mining tools developed by an ongoing collaboration between researchers at Lawrence Berkeley National Laboratory, Microsoft Research, and the University of California, Berkeley.

The Microsoft eScience program is the primary funder of this project, which is one of numerous ventures cultivated by the Berkeley Water Center (BWC). Launched approximately three years ago by researchers from the Berkeley Lab and UC Berkeley’s Colleges of Engineering and Natural Resources, the BWC marshals expertise from public institutions and the private sector in support of projects that enable science and public policy researchers to more easily access and work with water and environmental datasets.

“The most cost-efficient way to impact issues like global climate change and water management is to develop cyber-architectures that organize data and foster scientific collaboration,” says Susan Hubbard, staff scientist in the Berkeley Lab’s Earth Sciences Division and associate director of the BWC.

Environmental scientists typically collect data on a project-by-project basis, in campaigns targeted at very specific topics. One study may use NASA satellites to track annual rainfall of deserts around the globe, while another project sponsored by the National Science Foundation (NSF) might measure the annual water tables of the Sahara desert with commercial sensors. The data are then typically stored in local archive systems and accessed by researchers associated with that particular project. These sites are scattered across the country, tend to be aligned with specific campaigns, and are funded by a variety of organizations.

Rest of the article at: New Tools Mobilize Local Data to Study Global Environmental Issues « Berkeley Lab News Center

Data Mining Services in the Cloud – Mine your Data, Any Place, Any Time

Dan Fay — Thu, 04 Sep 2008 22:24:38 GMT

This is great news - Software-plus-Services that any scientist/researcher could use. The SQL Server Data Mining folks have a Data Mining Service in the cloud they are testing out…I posted previously [OLAP and Scientific Data & Data Mining Addins for Office 2007 (Excel & Visio)] about the Excel addins that allow anyone with Excel to do Data Mining on Excel tables. Now the team is testing out SQL Server Data Mining Services – from which you can do the data mining directly from Excel 2007 or even upload a csv file.

So for fun – I downloaded a csv file of a stream gauge near Redmond into Excel and ran the “Highlight Exceptions” tool to find outliers in the dataset – it read the table, uploaded it to the service and in seconds returned the results - which included the number of outliers - in this case air temperature and it also highlighted in the table the rows. It was so easy. I can see it being used for many scientific datasets - even to clean them before doing other analysis, charting, graphs, uploads, etc.

The Table Analysis Tools included are:

Analyze Key Influencers
Detect Categories
Fill from Example
Forecast
Highlight Exceptions
Scenario Analysis
Prediction Calculator
Shopping Basket Analysis

Microsoft SQL Server Data Mining Services

Mine your Data, Any Place, Any Time

The SQL Server Data Mining team is working to extend the power and ease of use of SQL Server Data Mining to the Cloud. Our goal is provide services that allow you to build rich, predictive applications without worrying about server infrastructure, and showcase these services with cool applications that give you a glimpse of what’s possible. We bring you a technology preview of our work below. Enjoy!

Current Projects

Table Analysis Tools for the Cloud

Build powerful predictive reports on your data with just a few clicks!
- No data mining expertise required
- No server installation required
- All you need is your Internet connection

http://www.sqlserverdatamining.com/cloud/

F# – Sept CTP available and units of measure checking/inference

Dan Fay — Sat, 30 Aug 2008 20:07:22 GMT

The September 2008 CTP of F# is now available for download. F# is a functional programming language for the .NET Framework and really should be looked at by scientists/researchers. Also check out the F# Developer Center on MSDN for more info and resources.

There are a lot of new features in this release – here’s a sampling:

Broadly improved Visual Studio 2008 integration, which allows F# users to scale from scripting and explorative development, up to large-scale component and application design, all within Visual Studio.
Improvements to the F# language and libraries to make them simpler and more regular.
An exciting new language feature, Units of Measure, which extends F#’s inference and strong typing to floating point data.

The Units of Measure checking and interference feature is very exciting feature and potentially most scientifically revolutionary programming language features around - scientists and engineers to check out. This is because the F# compiler knows the rules of units

When values of floating-point type are multiplied, the units are multiplied too; when they are divided, the units are divided too, and when taking square roots, the same is done to the units. So by the rule for multiplication, the expression inside sqrt above must have units m^2/s^2, and therefore the units of speedOfImpact must be m/s.

Take a look at the SolarSystem sample - A Solar System simulation application, taking advantage of Units of Measure in F# to do physics simulation. Andrew Kennedy, who researched, architected and implemented this feature has all the details.

Other F# resources:

Microsoft F# Developer Center

OLAP and Scientific Data

Dan Fay — Thu, 21 Aug 2008 01:57:52 GMT

While I’ve been pushing the ideas of using OLAP data cubes to evaluate scientific data for awhile, I thought it might be a good time to pull together some relevant papers and links. I believe OLAP is ideal to help analyze large quantities of data including time series information...making it easier for the scientist/researcher to explore the data in real-time and from tools they know like Excel. For example the data served up on FluxData site is done by creating OLAP cubes using SQL Server Analysis Services.

A couple of tools/links that might be of interest as well:

The Data Mining Add-ins for Office 2007 - very useful since you can do much of the data mining directly from Excel. Microsoft SQL Server 2008 Data Mining Add-ins for Microsoft Office 2007
http://www.sqlserverdatamining.com
SQL Server 2008 Analysis Services
Microsoft Computational Biology Web Tools
Jamie’s blog

Here are a couple of papers that reference the use of OLAP for different types of scientific data.