eScience @ Microsoft : SQL Server

Science Analytics – look to use Project “Gemini”

eScience — Fri, 21 Aug 2009 19:34:37 GMT

When I first saw and heard details about Project “Gemini”, I was blown away by the technology and innovation created by SQL and Excel teams and that held up when I was able to test it out on my own. It will be especially useful for scientists that want to not only analyze large amounts of data in Excel, but also aggregate different datasets. This upcoming Excel 2010 add-in removes the storage limits of Excel by adding the in-memory database and brings the power of SQL Server and SQL Analysis Services into the hands of mere mortals. Scientists that utilize Excel for viewing/analyzing data will find this add-in extremely helpful.

Project Gemini Blog – Check out the videos -

Project "Gemini": Build powerful analytical applications

Need to make timely business decisions without having to use complicated and sluggish analytical applications? Love to use Excel? Project Gemini is an Excel 2010 add-in that allows you to create powerful analyses by quickly manipulating millions of rows of data into a single Excel workbook and utilize Microsoft Office 2010 to share and collaborate on your insights with your team.

You can combine native Excel 2010 functionality with Gemini’s in-memory engine to allow users to interactively explore and perform calculations on large data sets. In addition, you can easily streamline the process of integrating data from multiple sources – including corporate databases, spreadsheets, reports, and data feeds.

Share and collaborate with confidence by easily publishing your analysis to SharePoint 2010 and have other users enjoy the same slicer and fast-query capabilities when working on your Excel Services reports.

Are you part of Office 2010 Tech Preview? Register and then Download and learn about Project Gemini now! (Note: You need to have Office 2010 before you can use Gemini.)

Introducing Microsoft Office 2010 for Business

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Project Trident: A Scientific Workflow Workbench available for download

eScience — Wed, 15 Jul 2009 02:58:03 GMT

Project Trident CTP is now available for download. Project Trident is a scientific workflow workbench MSR External Research has been working on for the past few years, which allows scientists to analyze large, diverse datasets. It’s built on Windows Workflow and utilizes SQL Server (Express or Server). Download it and try it out…

Project Trident

Built on the Windows Workflow Foundation, this scientific workflow workbench allows users to:

Automate analysis and then visualize and explore data

Compose, run, and catalog experiments as workflows

Capture provenance for each experiment

Create a domain-specific workflow library to extend the functionality of the workflow workbench

Use existing services, such as provenance and fault tolerance, or add new services

Schedule workflows over HPC clusters or cloud computing resources

Project Trident: A Scientific Workflow Workbench - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Graywulf Takes Byte Out of Data Overload

eScience — Fri, 08 May 2009 20:46:17 GMT

Graywulf is the natural evolution of Beowulf Clusters – it brings together HPC clusters and databases to do efficient processing and data management. It’s name and design also pays homage to Jim Gray – who helped champion the use of relational databases in the scientific projects.

At it’s simplest form Graywulf is having a database installed on each of the HPC compute nodes – this brings the data to the computation – one of the points Jim made quite often and utilizes the power of databases (queries, stored procedures, etc). Since it’s a generic architecture Graywulf clusters can be built using any OS and any database…the ones in the case study below implemented them using Windows HPC Server and SQL Server and the motivation was to be more efficient in doing the science – it’s always great to have innovative folks using technologies to do good work.

“To put it simply, a scientist needs to be able to live within the data,” says Alexander Szalay, a cosmologist-turned-computer-scientist at The Johns Hopkins University (JHU) in Baltimore, Maryland. The power of information, Szalay says, is determined not by its quantity so much as how easy it is to access, manipulate and analyze.
“It’s not just about doing the numerical calculations,” adds Andrew Simms, a biomedical health informatics graduate student working on protein structure analysis in Valerie Daggett’s bioengineering laboratory at the University of Washington (UW) in Seattle. “It’s also about assembling the data so we can run calculations while performing analyses and ad hoc explorations and then feed it all back into the data warehouse.”

Graywulf Takes Byte Out of Data Overload

Astronomers at The Johns Hopkins University and protein scientists at the University of Washington are using inexpensive computer hardware combined with powerful computing and database software to help manage and analyze a growing volume of scientific data.

For details, read the Graywulf case study.

Project Principals

Alexander Szalay, Alumni Centennial Professor, Department of Physics and Astronomy, The Johns Hopkins University

Valerie Daggett, Professor of Bioengineering, University of Washington

Graywulf Takes Byte Out of Data Overload - Microsoft Research

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

OLAP and Scientific Data

eScience — Thu, 21 Aug 2008 01:57:52 GMT

While I’ve been pushing the ideas of using OLAP data cubes to evaluate scientific data for awhile, I thought it might be a good time to pull together some relevant papers and links. I believe OLAP is ideal to help analyze large quantities of data including time series information...making it easier for the scientist/researcher to explore the data in real-time and from tools they know like Excel. For example the data served up on FluxData site is done by creating OLAP cubes using SQL Server Analysis Services.

A couple of tools/links that might be of interest as well:

The Data Mining Add-ins for Office 2007 - very useful since you can do much of the data mining directly from Excel. Microsoft SQL Server 2008 Data Mining Add-ins for Microsoft Office 2007
http://www.sqlserverdatamining.com
SQL Server 2008 Analysis Services
Microsoft Computational Biology Web Tools
Jamie’s blog

Here are a couple of papers that reference the use of OLAP for different types of scientific data.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

SQL Server 2005 Driver for PHP - Released

eScience — Tue, 29 Jul 2008 23:42:54 GMT

PHP developers can now integrate data from SQL Server now that the driver is available for download. Also glad to see that it works with SQL Server Express edition.

It is my pleasure to announce that version 1.0 of the Microsoft SQL Server 2005 Driver for PHP has released! This release marks another step in Microsoft’s continued commitment to interoperability. To keep up with our announcements and customer feedback, please check out our team blog.

SQL Server 2005 Driver for PHP Team Blog

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet

eScience — Fri, 18 Jul 2008 02:19:11 GMT

Always looking for interesting papers that educate you on domains - really enjoyed this paper...

Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet

SQL Server Technical Article

Writers: Isaac Kunen

Summary: This paper is an introduction to Earth-oriented coordinate systems, projections, models, and mapping. While not specific to any technology, this information provides valuable background for those who will use spatial data in SQL Server.

Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

[Papers] Supporting Finite Element Analysis with a Relational Database Backend; There is Life beyond Files

eScience — Tue, 08 Apr 2008 21:00:53 GMT

In a discussion I had today around ways to advance a scientific problem I was reminded of Jim Gray and Gerd Heber's trilogy - Supporting Finite Element Analysis with a Relational Database Backend. The three papers are really a good resource for understanding how databases can be used in scientific challenges.

Part I: There is Life beyond Files

We show how to use a Relational Database Management System in support of Finite Element Analysis. We believe it is a new way of thinking about data management in well-understood applications to prepare them for two major challenges, - size and integration (globalization). Neither extreme size nor integration (with other applications over the Web) was a design concern 30 years ago when the paradigm for FEA implementation first was formed. On the other hand, database technology has come a long way since its inception and it is past time to highlight its usefulness to the field of scientific computing and computer based engineering. This series aims to widen the list of applications for database designers and for FEA users and application developers to reap some of the benefits of database development.

Part II: Database Design and Access

This is Part II of a three articles on using databases for Finite Element Analysis (FEA). It discusses (1) db design, (2) data loading, (3) typical use cases during grid building, (4) typical use cases during simulation (get and put), (5) typical use cases during analysis (also done in Part III) and some performance measures of these cases. It argues that using a database is simpler to implement than custom data schemas, has better performance because it can use data parallelism, and better supports FEA modularity and tool evolution because database schema evolution, data independence, and self-defining data.

Part III: OpenDX – Where the Numbers Come Alive

In this report, we show a unified visualization and data analysis approach to Finite Element Analysis. The example application is visualization of 3D models of (metallic) polycrystals. Our solution combines a mature, general purpose, rapid-prototyping visualization tool, OpenDX (formerly known as IBM Visualization Data Explorer) [1,2], with an enterprise-class relational database management system, Microsoft SQL Server [3]. Substantial progress can be made with established off-the-shelf technologies. This approach certainly has its limits and we point out some of the shortcomings which require more innovative products for visualization, data-, and knowledge management. But, overall, the approach is a substantial improvement in the FEA lifecycle, and probably will work for other data-intensive sciences wanting to visualize and analyze massive simulation or measurement datasets.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

SQL Server 2008 info

eScience — Sat, 28 Jul 2007 01:47:42 GMT

The latest version of TechNet Flash has some good pointers to details on SQL Server 2008 and the CTP available for download. Some of the things scientists might find interesting are:

Language Integrated Query (LINQ) enables developers to issue queries against data by using a managed programming language such as C# or Visual Basic.NET, instead of SQL statements.
ADO.NET Entity Framework enables developers to be more productive by working with logical data entities that align with business requirements instead of programming directly with tables and columns.
SQL Server 2008 provides new data types that enable developers and administrators to efficiently store and manage unstructured data such as documents and images.
Support for advanced geospatial data has been added

Organizations today thrive on data. Understanding data trends and having a vision of the requirements of your data-driven applications is crucial in meeting the needs of these applications going forward. For a view of the Microsoft data platform vision, and to see how Microsoft SQL Server 2008 will deliver on that vision, the Microsoft SQL Server 2008 Product Overview white paper is a great place to start. And Microsoft recently released a Community Technology Preview (CTP) version of SQL Server 2008 for you to download and provide your feedback to the SQL Server development team. For a quick look at the key improvement pillars in the SQL Server 2008 CTP, click over to the SQL Server 2008 June CTP Highlights.
Over the next month, we are also featuring a series of SQL Server 2008 TechNet Webcasts including the SQL Server 2008 Data Management Overview on July 26. Check out the Webcasts section below for the entire series.

Source: http://www.microsoft.com/technet/abouttn/subscriptions/flash/archive/07-25-07.htm

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Data Mining Virtual Hands-on Lab Now Available!

eScience — Tue, 24 Jul 2007 02:37:42 GMT

I wrote about the Data mining Addins previously - Data Mining for the rest of us... and now you can test it out via the Virtual hands-ons lab...glad to see this post by JamieMac...

Data Mining Virtual Hands-on Lab Now Available!

For those of you who have been dying to try out the data mining addins we released earlier this year - here's your chance! Ok, scratch that, if you're reading my blog you've been there done that, but if there's that someone special who you know would love the addins and hadn't had the chance to get everything configured to give them a go, have I got a deal for you!
Since the Data Mining Addins hands-on lab was ranked in the top 10 of all labs at TechEd (it was, in fact, #2), it earned a special distinction of being made available as an online hands-on lab where you can reserve 90 minutes of time to remotely operate machine to walk through the exercises. Try it out today!

JamieMac's WebLog : Data Mining Virtual Hands-on Lab Now Available! Also, in other news, Donald in a fishbowl!

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Galaxy Zoo - Help classify a million galaxies

eScience — Thu, 12 Jul 2007 07:11:32 GMT

Thanks to Alex Szalay (JHU) for pointing me to GalaxyZoo - now you can help classify galaxies. This really neat...

Welcome to GalaxyZoo , the project which harnesses the power of the internet - and your brain - to classify a million galaxies. By taking part, you'll not only be contributing to scientific research, but you'll view parts of the Universe that literally no-one has ever seen before and get a sense of the glorious diversity of galaxies that pepper the sky.
Why do we need you?
The simple answer is that the human brain is much better at recognising patterns than a computer can ever be. Any computer program we write to sort our galaxies into categories would do a reasonable job, but it would also inevitably throw out the unusual, the weird and the wonderful. To rescue these interesting systems which have a story to tell, we need you.

Source: Galaxy Zoo

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Reporting@Home: Delivering Dynamic Graphical Feedback in Community Computing Projects

eScience — Tue, 10 Jul 2007 02:18:52 GMT

I meant to do an entry on this paper by Stuart Ozer (MSR) and David Kim & David Baker (Rosetta@Home) months ago...it's a great way to integrate SQL Reporting services w/ something like Rosetta@Home, and provide really great service for not only the community users - but also for the researchers using the system. Below is the architecture diagram...

Reporting@Home: Delivering Dynamic Graphical Feedback to Participants and Researchers in Community Computing Projects

Stuart Ozer; David Kim; David Baker
February 2007
Available Documents:Word 638 Kb PDF 482 Kb
A new generation of computationally intensive scientific research projects relies on volunteers from around the world contributing idle computer time to calculate mathematical models. Many of these projects utilize a common architecture to manage the scheduling and distribution of calculations and collection of results from participants. User engagement is critical to the success of these projects, and feedback to participants illustrating their role in the project’s progress is known to increase interest and strengthen the community. This article describes how one project -- University of Washington’s Rosetta@Home, which predicts and designs the folded conformations of proteins and protein complexes -- created a web-based, on-demand reporting system that graphically illustrates a user or team’s contributions to the project. The reporting service is also useful to the project scientists in assessing the utility of alternative models and computational techniques. The system relies on a comprehensive database platform that includes tools for data integration, data management, querying and web-based reporting. The reporting components integrate seamlessly with the rest of the project’s data and web infrastructure, and the report pages have proven to be popular among both participants and lab members.

Source: Reporting@Home: Delivering Dynamic Graphical Feedback to Participants and Researchers in Community Computing Projects

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Virtual Labs - giving away a Camera

eScience — Thu, 17 May 2007 00:31:50 GMT

Just saw that the Virtual Labs folks are giving away a camera for using the labs...

I'd recommend the

Win a Digital SLR Camera!

Take part in any TechNet or MSDN virtual lab or labcast through June 30, 2007, and you could win* a digital SLR camera package.

Source: Welcome to the TechNet Virtual Labs

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Dynameomics - Protein folds by Molecular Dynamics Simulation

eScience — Fri, 11 May 2007 07:54:54 GMT

Valerie Daggett and her team at UW have put up a site with their simulations of over 300 proteins for a combined simulation time of more than 35 microseconds. This site contains information for their top 30 targets. The other interesting part is that they have built this using SQL Server and use SQL Server Analysis Services to create cubes of the molecule locations.

Dynameomics

Dynameomics is a continuing project in the Daggett group to characterize the native state dynamics and the folding / unfolding pathway of representatives from all known protein folds by molecular dynamics simulation.
This effort began with the creation of a consensus fold list. This was done by cross-referencing the fold definitions used in SCOP, CATH, and the Dali Domain Dictionary as described in the Origin of the Fold List page. Next, targets were selected from the consensus fold list. A target refers to a specific protein structure from the PDB that has been chosen to represent a given fold (see the example on the left). The specifics of this choice are give on the Target Selection page. The complete list of consensus folds, their populations and targets are provided in the fold and target pages.
At this time, we are continuing to simulate targets from the fold list, generally in order of decreasing fold population. The simulation protocols, software, and analyses are described on the methods page.

Source: Dynameomics - Home Page

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

Data Mining Addins for Office 2007 (Excel & Visio)

eScience — Thu, 01 Mar 2007 04:08:00 GMT

I think of the Data Mining Addins for Excel 2007, as data mining tools for common man - they allow anyone to use SQL Server Analysis Services without having to know how to program the backend. It allows you to look for Key Influencers, Detect Categories, Highlight Exceptions, etc. This can be a real benefit for scientists looking to do things like data cleaning directly from Excel.

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)

SQL Server Starter Data Schema

eScience — Fri, 19 Jan 2007 09:14:01 GMT

Very cool - it would be great to see some scientific reserach schemas added to the list...

SQL Server 2005 Express Data Schemas
Deploying SQL Server Express in real-world environments
This data schema series is designed for new database administrators and enthusiasts to quickly get started learning the basics of managing and administering SQL Server 2005 Express Edition in real-world environments. These fully-functional starter data schemas cover a wide-range of scenarios that are designed to help you quickly and easily deploy SQL Server 2005 Express Edition within your organization.

Source: Starter Data Schema

Cross Posted from Dan Fay's Blog (http://blogs.msdn.com/dan_fay)