Dan on eScience & Technical Computing @ Microsoft : SQL Server

Graywulf Takes Byte Out of Data Overload

Dan Fay — Fri, 08 May 2009 17:46:17 GMT

Graywulf is the natural evolution of Beowulf Clusters – it brings together HPC clusters and databases to do efficient processing and data management. It’s name and design also pays homage to Jim Gray – who helped champion the use of relational databases in the scientific projects.

At it’s simplest form Graywulf is having a database installed on each of the HPC compute nodes – this brings the data to the computation – one of the points Jim made quite often and utilizes the power of databases (queries, stored procedures, etc). Since it’s a generic architecture Graywulf clusters can be built using any OS and any database…the ones in the case study below implemented them using Windows HPC Server and SQL Server and the motivation was to be more efficient in doing the science – it’s always great to have innovative folks using technologies to do good work.

“To put it simply, a scientist needs to be able to live within the data,” says Alexander Szalay, a cosmologist-turned-computer-scientist at The Johns Hopkins University (JHU) in Baltimore, Maryland. The power of information, Szalay says, is determined not by its quantity so much as how easy it is to access, manipulate and analyze.
“It’s not just about doing the numerical calculations,” adds Andrew Simms, a biomedical health informatics graduate student working on protein structure analysis in Valerie Daggett’s bioengineering laboratory at the University of Washington (UW) in Seattle. “It’s also about assembling the data so we can run calculations while performing analyses and ad hoc explorations and then feed it all back into the data warehouse.”

Graywulf Takes Byte Out of Data Overload

Astronomers at The Johns Hopkins University and protein scientists at the University of Washington are using inexpensive computer hardware combined with powerful computing and database software to help manage and analyze a growing volume of scientific data.

For details, read the Graywulf case study.

Project Principals

Alexander Szalay, Alumni Centennial Professor, Department of Physics and Astronomy, The Johns Hopkins University

Valerie Daggett, Professor of Bioengineering, University of Washington

Graywulf Takes Byte Out of Data Overload - Microsoft Research

OLAP and Scientific Data

Dan Fay — Wed, 20 Aug 2008 22:57:52 GMT

While I’ve been pushing the ideas of using OLAP data cubes to evaluate scientific data for awhile, I thought it might be a good time to pull together some relevant papers and links. I believe OLAP is ideal to help analyze large quantities of data including time series information...making it easier for the scientist/researcher to explore the data in real-time and from tools they know like Excel. For example the data served up on FluxData site is done by creating OLAP cubes using SQL Server Analysis Services.

A couple of tools/links that might be of interest as well:

The Data Mining Add-ins for Office 2007 - very useful since you can do much of the data mining directly from Excel. Microsoft SQL Server 2008 Data Mining Add-ins for Microsoft Office 2007
http://www.sqlserverdatamining.com
SQL Server 2008 Analysis Services
Microsoft Computational Biology Web Tools
Jamie’s blog

Here are a couple of papers that reference the use of OLAP for different types of scientific data.

SQL Server 2005 Driver for PHP - Released

Dan Fay — Tue, 29 Jul 2008 20:42:54 GMT

PHP developers can now integrate data from SQL Server now that the driver is available for download. Also glad to see that it works with SQL Server Express edition.

It is my pleasure to announce that version 1.0 of the Microsoft SQL Server 2005 Driver for PHP has released! This release marks another step in Microsoft’s continued commitment to interoperability. To keep up with our announcements and customer feedback, please check out our team blog.

SQL Server 2005 Driver for PHP Team Blog

Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet

Dan Fay — Thu, 17 Jul 2008 23:19:11 GMT

Always looking for interesting papers that educate you on domains - really enjoyed this paper...

Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet

SQL Server Technical Article

Writers: Isaac Kunen

Summary: This paper is an introduction to Earth-oriented coordinate systems, projections, models, and mapping. While not specific to any technology, this information provides valuable background for those who will use spatial data in SQL Server.

Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet

[Papers] Supporting Finite Element Analysis with a Relational Database Backend; There is Life beyond Files

Dan Fay — Tue, 08 Apr 2008 18:00:53 GMT

In a discussion I had today around ways to advance a scientific problem I was reminded of Jim Gray and Gerd Heber's trilogy - Supporting Finite Element Analysis with a Relational Database Backend. The three papers are really a good resource for understanding how databases can be used in scientific challenges.

Part I: There is Life beyond Files

We show how to use a Relational Database Management System in support of Finite Element Analysis. We believe it is a new way of thinking about data management in well-understood applications to prepare them for two major challenges, - size and integration (globalization). Neither extreme size nor integration (with other applications over the Web) was a design concern 30 years ago when the paradigm for FEA implementation first was formed. On the other hand, database technology has come a long way since its inception and it is past time to highlight its usefulness to the field of scientific computing and computer based engineering. This series aims to widen the list of applications for database designers and for FEA users and application developers to reap some of the benefits of database development.

Part II: Database Design and Access

This is Part II of a three articles on using databases for Finite Element Analysis (FEA). It discusses (1) db design, (2) data loading, (3) typical use cases during grid building, (4) typical use cases during simulation (get and put), (5) typical use cases during analysis (also done in Part III) and some performance measures of these cases. It argues that using a database is simpler to implement than custom data schemas, has better performance because it can use data parallelism, and better supports FEA modularity and tool evolution because database schema evolution, data independence, and self-defining data.

Part III: OpenDX – Where the Numbers Come Alive

In this report, we show a unified visualization and data analysis approach to Finite Element Analysis. The example application is visualization of 3D models of (metallic) polycrystals. Our solution combines a mature, general purpose, rapid-prototyping visualization tool, OpenDX (formerly known as IBM Visualization Data Explorer) [1,2], with an enterprise-class relational database management system, Microsoft SQL Server [3]. Substantial progress can be made with established off-the-shelf technologies. This approach certainly has its limits and we point out some of the shortcomings which require more innovative products for visualization, data-, and knowledge management. But, overall, the approach is a substantial improvement in the FEA lifecycle, and probably will work for other data-intensive sciences wanting to visualize and analyze massive simulation or measurement datasets.

SQL Server 2008 info

Dan Fay — Fri, 27 Jul 2007 22:47:42 GMT

The latest version of TechNet Flash has some good pointers to details on SQL Server 2008 and the CTP available for download. Some of the things scientists might find interesting are:

Language Integrated Query (LINQ) enables developers to issue queries against data by using a managed programming language such as C# or Visual Basic.NET, instead of SQL statements.
ADO.NET Entity Framework enables developers to be more productive by working with logical data entities that align with business requirements instead of programming directly with tables and columns.
SQL Server 2008 provides new data types that enable developers and administrators to efficiently store and manage unstructured data such as documents and images.
Support for advanced geospatial data has been added

Organizations today thrive on data. Understanding data trends and having a vision of the requirements of your data-driven applications is crucial in meeting the needs of these applications going forward. For a view of the Microsoft data platform vision, and to see how Microsoft SQL Server 2008 will deliver on that vision, the Microsoft SQL Server 2008 Product Overview white paper is a great place to start. And Microsoft recently released a Community Technology Preview (CTP) version of SQL Server 2008 for you to download and provide your feedback to the SQL Server development team. For a quick look at the key improvement pillars in the SQL Server 2008 CTP, click over to the SQL Server 2008 June CTP Highlights.
Over the next month, we are also featuring a series of SQL Server 2008 TechNet Webcasts including the SQL Server 2008 Data Management Overview on July 26. Check out the Webcasts section below for the entire series.

Source: http://www.microsoft.com/technet/abouttn/subscriptions/flash/archive/07-25-07.htm

Data Mining Virtual Hands-on Lab Now Available!

Dan Fay — Mon, 23 Jul 2007 23:37:42 GMT

I wrote about the Data mining Addins previously - Data Mining for the rest of us... and now you can test it out via the Virtual hands-ons lab...glad to see this post by JamieMac...

Data Mining Virtual Hands-on Lab Now Available!

For those of you who have been dying to try out the data mining addins we released earlier this year - here's your chance! Ok, scratch that, if you're reading my blog you've been there done that, but if there's that someone special who you know would love the addins and hadn't had the chance to get everything configured to give them a go, have I got a deal for you!
Since the Data Mining Addins hands-on lab was ranked in the top 10 of all labs at TechEd (it was, in fact, #2), it earned a special distinction of being made available as an online hands-on lab where you can reserve 90 minutes of time to remotely operate machine to walk through the exercises. Try it out today!

JamieMac's WebLog : Data Mining Virtual Hands-on Lab Now Available! Also, in other news, Donald in a fishbowl!

Galaxy Zoo - Help classify a million galaxies

Dan Fay — Thu, 12 Jul 2007 04:11:32 GMT

Thanks to Alex Szalay (JHU) for pointing me to GalaxyZoo - now you can help classify galaxies. This really neat...

Welcome to GalaxyZoo , the project which harnesses the power of the internet - and your brain - to classify a million galaxies. By taking part, you'll not only be contributing to scientific research, but you'll view parts of the Universe that literally no-one has ever seen before and get a sense of the glorious diversity of galaxies that pepper the sky.
Why do we need you?
The simple answer is that the human brain is much better at recognising patterns than a computer can ever be. Any computer program we write to sort our galaxies into categories would do a reasonable job, but it would also inevitably throw out the unusual, the weird and the wonderful. To rescue these interesting systems which have a story to tell, we need you.

Source: Galaxy Zoo

Reporting@Home: Delivering Dynamic Graphical Feedback in Community Computing Projects

Dan Fay — Mon, 09 Jul 2007 23:18:52 GMT

I meant to do an entry on this paper by Stuart Ozer (MSR) and David Kim & David Baker (Rosetta@Home) months ago...it's a great way to integrate SQL Reporting services w/ something like Rosetta@Home, and provide really great service for not only the community users - but also for the researchers using the system. Below is the architecture diagram...

Reporting@Home: Delivering Dynamic Graphical Feedback to Participants and Researchers in Community Computing Projects

Stuart Ozer; David Kim; David Baker
February 2007
Available Documents:Word 638 Kb PDF 482 Kb
A new generation of computationally intensive scientific research projects relies on volunteers from around the world contributing idle computer time to calculate mathematical models. Many of these projects utilize a common architecture to manage the scheduling and distribution of calculations and collection of results from participants. User engagement is critical to the success of these projects, and feedback to participants illustrating their role in the project’s progress is known to increase interest and strengthen the community. This article describes how one project -- University of Washington’s Rosetta@Home, which predicts and designs the folded conformations of proteins and protein complexes -- created a web-based, on-demand reporting system that graphically illustrates a user or team’s contributions to the project. The reporting service is also useful to the project scientists in assessing the utility of alternative models and computational techniques. The system relies on a comprehensive database platform that includes tools for data integration, data management, querying and web-based reporting. The reporting components integrate seamlessly with the rest of the project’s data and web infrastructure, and the report pages have proven to be popular among both participants and lab members.

Source: Reporting@Home: Delivering Dynamic Graphical Feedback to Participants and Researchers in Community Computing Projects

Virtual Labs - giving away a Camera

Dan Fay — Wed, 16 May 2007 21:31:50 GMT

Just saw that the Virtual Labs folks are giving away a camera for using the labs...

I'd recommend the

Win a Digital SLR Camera!

Take part in any TechNet or MSDN virtual lab or labcast through June 30, 2007, and you could win* a digital SLR camera package.

Source: Welcome to the TechNet Virtual Labs

Dynameomics - Protein folds by Molecular Dynamics Simulation

Dan Fay — Fri, 11 May 2007 04:54:54 GMT

Valerie Daggett and her team at UW have put up a site with their simulations of over 300 proteins for a combined simulation time of more than 35 microseconds. This site contains information for their top 30 targets. The other interesting part is that they have built this using SQL Server and use SQL Server Analysis Services to create cubes of the molecule locations.

Dynameomics

Dynameomics is a continuing project in the Daggett group to characterize the native state dynamics and the folding / unfolding pathway of representatives from all known protein folds by molecular dynamics simulation.
This effort began with the creation of a consensus fold list. This was done by cross-referencing the fold definitions used in SCOP, CATH, and the Dali Domain Dictionary as described in the Origin of the Fold List page. Next, targets were selected from the consensus fold list. A target refers to a specific protein structure from the PDB that has been chosen to represent a given fold (see the example on the left). The specifics of this choice are give on the Target Selection page. The complete list of consensus folds, their populations and targets are provided in the fold and target pages.
At this time, we are continuing to simulate targets from the fold list, generally in order of decreasing fold population. The simulation protocols, software, and analyses are described on the methods page.

Source: Dynameomics - Home Page

Data Mining Addins for Office 2007 (Excel & Visio)

Dan Fay — Thu, 01 Mar 2007 01:08:00 GMT

I think of the Data Mining Addins for Excel 2007, as data mining tools for common man - they allow anyone to use SQL Server Analysis Services without having to know how to program the backend. It allows you to look for Key Influencers, Detect Categories, Highlight Exceptions, etc. This can be a real benefit for scientists looking to do things like data cleaning directly from Excel.

SQL Server Starter Data Schema

Dan Fay — Fri, 19 Jan 2007 06:14:01 GMT

Very cool - it would be great to see some scientific reserach schemas added to the list...

SQL Server 2005 Express Data Schemas
Deploying SQL Server Express in real-world environments
This data schema series is designed for new database administrators and enthusiasts to quickly get started learning the basics of managing and administering SQL Server 2005 Express Edition in real-world environments. These fully-functional starter data schemas cover a wide-range of scenarios that are designed to help you quickly and easily deploy SQL Server 2005 Express Edition within your organization.

Source: Starter Data Schema

SQL Server Data Mining Add-ins for Office 2007 CTP

Dan Fay — Wed, 20 Dec 2006 03:15:46 GMT

So glad to see the CTP of Data Mining Add-ins available - I see this is as Data Mining for the rest of us...

Microsoft SQL Server 2005 Data Mining Add-ins for Office 2007 (Data Mining Add-ins) allow you take advantage of SQL Server 2005 predictive analytics in Office Excel 2007 and Office Visio 2007. The download includes the following components:

Table Analysis Tools for Excel: This add-in provides you with easy-to-use tasks that leverage SQL Server 2005 Data Mining under the covers to perform powerful analytics on your spreadsheet data.
Data Mining Client for Excel: This add-in allows you to go through the full data mining model development lifecycle within Excel 2007 using either your spreadsheet data or external data accessible through your SQL Server 2005 Analysis Services instance.
Data Mining Templates for Visio: This add-in allows you to render and share your mining models as annotatable Visio 2007 drawings.

Source: Download details: SQL Server Data Mining Add-ins for Office 2007 CTP

Mundie: Science as a Web Service

Dan Fay — Fri, 08 Dec 2006 22:47:23 GMT

I realized I forgot to post about Craig Mundie's article in the March/April Technology Review -

Science as a Web Service

XML can supercharge research.
<...>
XML is also one of the enabling technologies for grid computing and Web services, which will revolutionize the scientific community in the coming decade by enabling the free exchange of information across distributed systems. Remote computation will be directly accessible from any desktop, and sensors and instruments will have their own Internet addresses.
The immediate challenge for the scientific and engineering community is to take advantage of available data management and data analysis tools. The larger and longer-term challenge is for the leaders in academic research to leverage software and Web services technologies to standardize the way they present and track their data.