eScience & Technical Computing - Web Services and Scientific Research
One of the discussions that frequently happens at the eScience/Science 2.0 type meetings is how does academia incorporate the Web 2.0 technologies (Blogging, etc) into the tenure based reward system - these are technologies that change how quickly/accessible information can be and are not connected to the current papers to journal process. So I found it really interesting to read john hawks blog (Univ of Wisconsin - Madison) posting on getting tenure and blogging. The advice he gives is great...
Personally, I think that maturity as a scientist comes with the ability to explain your work to your parents.
The advice is not only good for blogging but also in communicating the research cross domains (ie. eScience confs). For talks - knowing the audience is a good thing and resist the need to show off your mental muscle.
Since it is a four-part series - check back for the other posts. I'm looking forward to the final installment - how john quantified "my blog's role as a service to the field and to the public."
How to blog, get tenure and prosper Last month, the University of Wisconsin officially granted me tenure. So, I can say without any doubt (if other examples had not been sufficient), it is absolutely possible to write a daily, high-profile blog and still be recognized by your colleagues as a scholar. In fact, it is possible to blog, do good research, and earn tenure at a Research I university. <...> Happily things have changed. With the rise of science blogging, people have become much more aware of the ways that a blog can contribute to a career in science. If you establish a readership, the chances are your colleagues will find out about your blog themselves, instead of looking at you in befuddlement. Blogs are not research, but in some fields they have become an important part of the process of networking and critical commentary. A well-written blog is far from a liability to a scientific career, and may be a real boon.
Last month, the University of Wisconsin officially granted me tenure. So, I can say without any doubt (if other examples had not been sufficient), it is absolutely possible to write a daily, high-profile blog and still be recognized by your colleagues as a scholar. In fact, it is possible to blog, do good research, and earn tenure at a Research I university.
<...>
Happily things have changed. With the rise of science blogging, people have become much more aware of the ways that a blog can contribute to a career in science. If you establish a readership, the chances are your colleagues will find out about your blog themselves, instead of looking at you in befuddlement. Blogs are not research, but in some fields they have become an important part of the process of networking and critical commentary. A well-written blog is far from a liability to a scientific career, and may be a real boon.
How to blog, get tenure and prosper: Starting the blog | john hawks weblog
Jon Udell gets the details from Jonathan Fay (no relation) on how WWT works. How the WorldWide Telescope works
It's really good to see that the datamining work (ProDA) that Cyrus Shahabi of USC is getting more visibility - the use of wavelet compression is a really neat way to deal with large amounts of data and make it easy to see the trends...you can see why folks like Chevron were interested in it.
Making Sense of Data Overload: An Innovative Approach to Progressive Data Analysis
When Professor Cyrus Shahabi of the University of Southern California decided to tackle the problem of complex data analysis, he was confronted by the limitations of current software. Realizing what an impediment this was for businesses and the scientific community, he began to explore alternative forms of analysis. When he came across signal processing and wavelet compression, he knew he was onto something, and ProDA was born. Since creating ProDA, NASA’s JPL and Chevron have had major successes using the program to manage their huge datasets. With the help of Microsoft Research’s Smart Client initiative, Shahabi was able to bring ProDA to the next level by making it more compatible with XML, Microsoft Excel, text files, and many more formats. All these changes have made ProDA more accessible and user friendly.
April 2008 article in IEEE Computer - ProDA: An End-to-End Wavelet-Based OLAP System for Massive Datasets by Cyrus Shahabi (USC)
Abstract ProDA employs wavelets to support exact, approximate, and progressive OLAP queries on large multidimensional datasets, while keeping update costs relatively low. ProDA not only supports online execution of ad hoc analytical queries on massive datasets, but also extends the set of supported analytical queries to include the entire family of polynomial aggregate queries as well as the new class of plot queries.
Abstract
ProDA employs wavelets to support exact, approximate, and progressive OLAP queries on large multidimensional datasets, while keeping update costs relatively low. ProDA not only supports online execution of ad hoc analytical queries on massive datasets, but also extends the set of supported analytical queries to include the entire family of polynomial aggregate queries as well as the new class of plot queries.
.NetMap an Excel plugin and dll is now available (Binaries and Source) on CodePlex – This brings the ability to visualize different types of networks – ie. Social networks (facebook, etc) and even things like Protein interaction network (see image). What’s great is that you can do this directly from within Excel – which makes it easy to use for anyone. The other neat piece of the add-in is that it will analyze your email (Outlook, Outlook Express, etc) and then you can use the NetMap chart to see who sends you a ton of email . You can also embed the class libraries in other apps to visualize your graphs…I’d like to hear about other possible Science applications of graph visualizations…and who will be the first to create a SilverLight app using the libraries.
NetMap is a pair of applications for viewing network graphs, along with a set of .NET Framework 2.0 class libraries that can be used to add network graphs to custom applications. A network graph is a series of vertices (sometimes called nodes) connected by edges. See this Wikipedia article for an overview of network graphs. .NetMap was created by Marc Smith's team at Microsoft Research. Here is a sample network graph created by .NetMap. It shows an individual's network of friends within Facebook.com:
.NetMap - on CodePlex
Wish I was creative enough to create a Xbox LIVE Community Games and not only get fame - but fortune :-) I'd be interested in seeing some Science or environment related games up there - might be a way to educate as well as have fun.
Once the game is reviewed and the price point set, you’re done. The game is listed on Xbox LIVE Marketplace, and you’ll get a check every quarter, for up to 70% of the game’s total revenue in your own currency. Depending on your game’s success, you may even have your game advertised on Xbox 360 and other Microsoft online properties. Just imagine - your game in the hands of millions of Xbox 360 gamers around the world: that’s the power of Community Games.
Once the game is reviewed and the price point set, you’re done. The game is listed on Xbox LIVE Marketplace, and you’ll get a check every quarter, for up to 70% of the game’s total revenue in your own currency. Depending on your game’s success, you may even have your game advertised on Xbox 360 and other Microsoft online properties.
Just imagine - your game in the hands of millions of Xbox 360 gamers around the world: that’s the power of Community Games.
XNA Creators Club Online - join the community: make games, make money
Just received the Sectoral Applications Research Program announcement from Nancy Beller-Simms at the NOAA Climate Program Office. The due date for the proposal is 5:00 P.M. Eastern Time October 9, 2008.
The overall NOAA 2009 Climate Program Announcement is also available – competitions in the follow areas:
Always looking for interesting papers that educate you on domains - really enjoyed this paper...
Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet SQL Server Technical Article Writers: Isaac Kunen Summary: This paper is an introduction to Earth-oriented coordinate systems, projections, models, and mapping. While not specific to any technology, this information provides valuable background for those who will use spatial data in SQL Server.
SQL Server Technical Article
Writers: Isaac Kunen
Summary: This paper is an introduction to Earth-oriented coordinate systems, projections, models, and mapping. While not specific to any technology, this information provides valuable background for those who will use spatial data in SQL Server.
Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet
Microsoft Research is hosting the DOE 2008 Scientific Discovery through Advanced Computing Program (SciDAC) tutorials on Friday, July 18th. If interested in attending, please register at the SciDAC Tutorials Registration.
SciDAC Tutorial Agenda
July 18 – Microsoft Research, Bldg 99.
9:00 - 9:30
Reception, checkin and orientation
9:30 - 12:00
First session
Visualization and Data Analysis with VisTrails
Introduction to Scientific Workflow Management and the Kepler System
Porting and Scaling Applications on BlueGene/P
Advanced CompuTational Software (ACTS) Collection Tutorial
12:00 - 1:00
Lunch in bldg. 99 Atrium (attendees in bldg. 117 walk to bldg. 99 next door)
1:00 - 3:30
Second session
Advanced Visualization and Data Analysis with the VisIt Visualization System
ADIOS The ADaptible IO System
An Introduction to the Cray XT4 for Application Scientists
Tutorial Abstracts
Title:
<></>
Contact:
Ilkay Altintas
Partners:
The Scientific Data Management Center (SDM)
Presenters: Ilkay Altintas, Scott Klasky, Norbert Podhorszki, Mladen Vouk
A scientific workflow combines data and processes into a configurable, structured set of steps that implement semi- automated computational solutions of a scientific problem. Scientific workflow systems provide a framework and often graphical user interfaces to combine different technologies along with efficient methods for using them. This reduces overhead and increases the efficiency of the scientists towards a scientific discovery. SciDAC SDM Center's Scientific Process Automation group develops scientific workflow tools (called Kepler) for automation of scientific data management processes in a reusable, robust, tractable, and recoverable fashion to enhance scientific exploration. This tutorial provides an introduction to scientific workflow construction and management using the Kepler system. It is intended for the SciDAC scientists and students with a computational science background. It will cover principles and foundations of scientific workflows, Kepler environment installation, workflow construction out of the available Kepler library components, and workflow execution management that uses Kepler-based facilities to provide process and data monitoring, provenance information, portal access, and high speed data movement solutions. This tutorial includes hands-on sessions and application examples from different scientific disciplines.
ADIOS: The ADaptible IO System
Chen Jin
The Scientific Data Management Center (SDM) and The Center for Plasma Edge Simulation and Oak Ridge National Lab
Presenters: Chen Jin, Scott Klasky, Steve Hodson: ORNL, Hasan Abbasi, Jay Lofstead, and Matthew Wolf: Georgia Tech, and Manish Parashar Rutgers
The ADaptible IO System (Adios) is a componentization of the IO layer for high performance scientific computations. Currently, we have integrated ADIOS into Fusion, Combustion and other pioneering scientific applications and achieved excellent performance results. ADIOS provides a simple to use, common programming interface for different transport layer methods. The design goals are set to allow the application scientist to program a very easy to use IO system, which can choose between different IO implementations at runtime. This allows IO experts to tune their implementations on different systems without changing the IO implementation. We currently support MPI-IO, collective MPI-IO, POSIX, asynchronous MPI-IO, the Georgia Tech DataTap system based on RDMA, and the Rutgers DART implementation also based on RDMA. ADIOS separates out the metadata and methods, by using an external XML file. This allows users to add annotations outside of their F90/C/C++ codes and to change the implementation outside of their code. ADIOS 1.0 will be released before 12/1/2008 and will provide very fast writes of data. In this tutorial, we will first present an introduction of ADIOS, along with the current supported methods, and file converters. We will then provide a hands-on session which will allow users to work with both a F90 and C code, and change the IO to ADIOS, and change this to use both MPI-IO and Posix IO. We will then show them how to read in the data, and convert the output to HDF5, NetCDF, and ASCII.
Porting and Scaling Applications on ALCF’s BlueGene/P
Kalyan Kumaran, Argonne Leadership Computing Facility
The Argonne Leadership Computing Facility
Presenters: Vitali Morozov and Ray Loy
The Argonne Leadership Computing Facility houses a half a petaflop BlueGene/P supercomputer. A number of petascale applications, from a variety of engineering domains, are run every day on this highly scalable architecture via the Department of Energy's INCITE (Innovative and Novel Computational Impact on Theory and Experiment) program. Nevertheless, porting and scaling applications onto petascale computers remains a challenging task. This tutorial will focus on the BlueGene/P architecture and present an overview of compilers, libraries, performance tools, debuggers available for porting and scaling applications.
Hank Childs
The Visualization and Analytics Center for Enabling Technology (VACET)
Presenters: Hank Childs and Sean Ahern
Understanding scientific datasets generated at DOE's high-performance computing facilities is becoming increasingly difficult as dataset sizes and complexity grow beyond the scale that is approachable by traditional analysis techniques. VACET is delivering scalable solutions to real-world visualization and analysis problems through the VisIt visualization system. VisIt is a turnkey application for data exploration, visualization, code assessment, and quantitative analysis suitable for use on SciDAC datasets of any size. This tutorial is targeted at all levels of users, from first timers on up to VisIt experts. The tutorial will present the basics of using VisIt on up to advanced visualization techniques.
Claudio Silva
Presenters: Claudio Silva and Carlos Scheidegger
The demand for the construction of complex visualizations is growing in many disciplines of science, as scientists are faced with ever increasing volumes of data to analyze. We give an overview of VisTrails, an open-source provenance management system that provides infrastructure for data exploration and visualization. VisTrails transparently records detailed provenance (history) information for exploratory computational tasks, both for the derived data products and for the pipelines used to derive them. Besides enabling the reproducibility of results, this provenance information can be used to simplify the process of data exploration through visualization. We will present several mechanisms and intuitive interfaces provided by VisTrails that allow flexible re-use of pipelines; exploration of large parameter spaces; comparison of visualizations and their respective pipelines; and the creation and refinement of visualizations by analogy.
This tutorial is targeted to users with different levels of expertise, from novice to expert. We will cover a number of specific examples that reflect typical visualization needs of DOE applications.
Richard Gerber
NERSC and ORNL
Presenters: Richard Gerber and Rebecca Hartman-Baker
This tutorial provides the basics to get up and running on the Cray XT4. Topics covered include compilers, libraries, and job management with a focus on parallel scaling and performance optimization of real world applications. The presenters are HPC consultant staff with hands on experience with the XT4 systems at ORNL and NERSC. This tutorial is targeted at researchers transitioning to the XT4 from serial or other parallel computing environments.
Tony Drummond
ACTS and TAU
Presenters: Tony Drummond
This short course will be an introduction to a set of advanced computational software tools to leverage the development of high performance applications. The lectures will focus on the selection, installation and use of scalable and robust software tools. Functionalities implemented in these software tools include; numerical algorithms for the solution of large computational problems, performance monitoring and profiling, and automatic tuning. Participants should expect to learn about techniques used to solve common computational problems and monitor their performance. Participants are encouraged to bring laptop computers and follow live demonstrations through hand-on experiences. The software presented here is freely available and widely used by the computational sciences international community.
Very interesting article in current issue of Science...
The United States faces unprecedented environmental and economic challenges in the decades ahead. Foremost among them will be climate change, sea-level rise, altered weather patterns, declines in freshwater availability and quality, and loss of biodiversity. Addressing these challenges will require well-conceived, science-based, simultaneous responses on multiple scales, from global and national, to regional and local. The executive and legislative branches of the federal government and of the states will have to transcend bureaucratic boundaries and become much more innovative in developing and implementing policy responses. We strongly believe organizational changes must be made at the federal level to align our public institutional infrastructure to address these challenges. The most pressing organizational change that is required is the establishment of an independent Earth Systems Science Agency formed by merging the National Oceanic and Atmospheric Administration (NOAA) and the U.S. Geological Survey (USGS).
An Earth Systems Science Agency Mark Schaefer,* D. James Baker, John H. Gibbons, Charles G. Groat, Donald Kennedy, Charles F. Kennel, David Rejeski Addressing serious environmental and economic challenges in the United States will require organizational changes at the federal level.
Addressing serious environmental and economic challenges in the United States will require organizational changes at the federal level.
An Earth Systems Science Agency -- Schaefer et al. 321 (5885): 44 -- Science
Just ran across the NGO Connection site – this is the place for non-profit orgs in the scientific space to look into….
NGO Connection is an online resource for non-profit groups around the world that provides resources to help advance the causes of the NGOs. See http://www.microsoft.com/ngo for more information about Getting free and discounted software Free access to Online software such as Windows Live, Microsoft Office Live and other online applications Software grants through the Microsoft Software Donation program Access to low cost personal computers and other hardware products Refurbished Computers from the Community Microsoft Authorized Refurbisher (MAR) program donated and discounted hardware from TechSoup for Refurbished computers, networks accessories, switches and routers learning about software support resources and other grant benefits Microsoft Support Resources Software Grant Support getting training material and information about obtaining certification
NGO Connection is an online resource for non-profit groups around the world that provides resources to help advance the causes of the NGOs. See http://www.microsoft.com/ngo for more information about
NGO Connection
Really neat demo by Michael Cohen (MSR) of Deep Photo at the Pro Photo Summit – uses Virtual Earth and the 3D geometric information to create haze models and clear up images…wish I had that for many of my photos.
Video: Virtual Earth Demo at Microsoft Pro Photo Summit
[Thanks – LiveSide]
Also check out the more in depth MSR lecture by Johannes Kopf.
Carol Minton Morris has a good entry on the HatCheck Newsletter on Alex Szalay's keynote, “Scientific Publishing in the Era of Pedabyte Data,” at JCDL on June 19, 2008. I always enjoy listening to Alex and hearing his perspective, especially since he gets his hands dirty with the eScience work and has put together a really good team. and of course there is always the fact that he plays the lead guitar :-).
The Petabyte Problem: Scrubbing, Curating and Publishing Big Data He suggests that the there is a science project pyramid–single lab at the base, multi-campus in the center, and international consortia on top. Often a scientific discipline will recognize the need for a major “giga” initiatives such as supercomputing research that is highly collaborative and distributed. The output from these efforts at every scale contain: –Literature –Derived and re-combined data –Raw data Szalay would like to see a continous feedback loop among these three aspects where data and analysis are always updating. To answer the question, “How can you publish data so that others might recreate your results in 100 yrs.,” he referred to Gray’s laws of Data Engineering: scientific computing revolves around data; scale-out the solution for analysis; take the analysis to the data; start with 20 queries, and; go from working to working.
The Petabyte Problem: Scrubbing, Curating and Publishing Big Data
He suggests that the there is a science project pyramid–single lab at the base, multi-campus in the center, and international consortia on top. Often a scientific discipline will recognize the need for a major “giga” initiatives such as supercomputing research that is highly collaborative and distributed. The output from these efforts at every scale contain:
–Literature
–Derived and re-combined data
–Raw data
Szalay would like to see a continous feedback loop among these three aspects where data and analysis are always updating.
To answer the question, “How can you publish data so that others might recreate your results in 100 yrs.,” he referred to Gray’s laws of Data Engineering: scientific computing revolves around data; scale-out the solution for analysis; take the analysis to the data; start with 20 queries, and; go from working to working.
HatCheck Newsletter » Blog Archive » The Petabyte Problem: Scrubbing, Curating and Publishing Big Data
Here are some good articles on Environmental Science related activities that our researchers at Microsoft Research are involved with.
All of these are from the Microsoft Enivronment site – also check out the Software Enabled Earth Blog
An On the Issues Essay came out last week featuring Alyssa Goodman, Professor of Astronomy, Harvard University. Highlights how WWT can be used to not only browse and view the Universe – but can be a powerful tool to allow Astronomers to get to data and make discoveries.
A Virtual Telescope: Microsoft’s WorldWide Telescope is a powerful educational tool — a way of telling compelling stories about the Universe. Seven years ago, a graduate student and I were analyzing an unusual image of the gas jettisoned by a forming star, named PV Ceph, when we realized the image could best be explained if the young star were speeding across the sky ten times faster than normal. But confirming our hypothesis required us to spend two years accumulating, overlaying and analyzing many more images made using ground- and spacebased radio, infrared and optical telescopes. Today, a project of this kind would be much easier thanks to the WorldWide Telescope, a rich, Web-based software application that anyone can download from www.worldwidetelescope.org. Released last month by Microsoft Research, the WorldWide Telescope stitches together images from the world’s best ground- and space-based telescopes to enable a seamless exploration of the Universe. <…>
Seven years ago, a graduate student and I were analyzing an unusual image of the gas jettisoned by a forming star, named PV Ceph, when we realized the image could best be explained if the young star were speeding across the sky ten times faster than normal. But confirming our hypothesis required us to spend two years accumulating, overlaying and analyzing many more images made using ground- and spacebased radio, infrared and optical telescopes.
Today, a project of this kind would be much easier thanks to the WorldWide Telescope, a rich, Web-based software application that anyone can download from www.worldwidetelescope.org. Released last month by Microsoft Research, the WorldWide Telescope stitches together images from the world’s best ground- and space-based telescopes to enable a seamless exploration of the Universe.
<…>
A Virtual Telescope: Microsoft’s WorldWide Telescope is a powerful educational tool — a way of telling compelling stories about the Universe.
