Wenming's Big Data and Big Compute Blog

In the first two blogs of this series, we installed the IPython notebook using the minimum requirement.    Creating a Linux VM on Windows Azure (1/4) Installing IPython notebook (2/4) The third blog post will walk you through some of the...

In the previous blog, we demonstrated how to create a Windows Azure Linux VM in detail . We will continue the installation process for the IPython notebook and related packages. Python 2.7 or 3.3 One of the discussions that happened at the Python in Finance...

I just returned from the Python in Finance Conference in New York, I would like to thank Bank of America and Andrew Shepped organizing the event.  It was not difficult to see the popularity of Python in the financial community; the event was quickly...

Running the Python example step by step: We explained the basic idea behind LSA or latent semantic analysis in the first part of this blog . We built a matrix by word counting for each document. The set of document vectors are then sorted by...

Data Science is compute and labor intensive In the previous blogs , we showed you how to find a dataset, clean it and run simple mapReduce, sort on the dataset. It was meant to give you a flavor of what data science is all about, and I also wanted...

Mammoth the NullSQL tool Most of us by now understand the properties of big data. Many of us are already working with big data tools, or NoSQL tools such as Hadoop. I've spent a bit of my spare time in the last 2 months working on prototypes of a new...

Our previous blog, MapReduce on 27,000 books using multiple storage accounts and HDInsight showed you how to run the Java version of the MapReduce code against the Gutenberg dataset we uploaded to the blog storage. We also explained how you can add multiple...

In our previous blog, Preparing and uploading datasets for HDInsight , we showed you some of the important utilities that are used on the Unix platform for data processing. That includes Gnu Parallel, Find, Split, and AzCopy for uploading large amounts...

In the previous blog  http://blogs.msdn.com/b/hpctrekker/archive/2013/03/30/finding-and-pre-processing-datasets-for-use-with-hdinsight.aspx   we went over how to get English only documents from the Gutenberg DVD.  We showed you the Cygwin...

  Free datasets There are many difficult aspects associated with Big Data, getting a good, clean, well tagged dataset is the first barrier.  After all, you can not really do much data processing without data!  Many companies are yet to...

Windows Azure HDInsight Service, formerly known as Hadoop on Windows Azure, is now available inside the Windows Azure Preview portal. Hadoop-based big data tools are what I call the WMD(P), or Weapons of Mass Data Processing. (You heard it here first...

In a previous blog, I explained Big Compute and the newly released HPC Pack 2012 from Microsoft.  I also announced a blog tutorial series on installation, deployment and running HPC applications using the latest version of the software.  This...

While most of you that have been using the Microsoft HPC Server product know the history and background of HPC, I still find that many Windows Azure developers new to this product. HPC, or high performance computing is a small (14 billion dollar) industry...

About 9 months ago, I deployed a Weather forecast demo at an internal Microsoft event, Techfest. The demo uses real data from NOAA and predicts high resolution weather forecast up to the next 3 days running a HPC modeling code called WRF . Since then...

The Super Computing conference attracts some of biggest names in the industry, academia, and government institutions. This year’s attendance was down from 11,000 to about 8,000. The main floor was completely full even without some of the largest...

HadoopOnAzure allows a user to run Hadoop on Microsoft Windows Azure as a service. It is currently in private CTP with limited capacity, and by invitation only. We did add more capacity today, you may attempt to sign up this free service at https://connect...

Last month my colleague Don Pattee at the HPC team announced the availability of the Windows Azure HPC Scheduler and HPC pack 2008 R2 Service pack 3 releases. The title is long, but the story is simple, while we updated our HPC on-premise solution with...

  If you have CPU intensive jobs that require 100s of Cores on Azure or on HPC on-premise cluster, I’ve just uploaded five easy-to-learn samples with lab instructions. (See figure 3) These samples have been added to the same location of the white...

  I will update here with more information as they become available for this exciting distribution which covers MPI on Azure, Azure VM roles and other exciting new and essential features. TechNet Library: · What's New in Windows HPC Server 2008 R2...

I will be covering some of the new, and more complete Azure burst features on my blog, please stay tuned! The Microsoft HPC Pack 2008 R2 software, and the Windows HPC Server 2008 R2 Suite, enables cluster based supercomputing based on x64 versions...

One of the interesting things living in Boulder Colorado for almost 20 years is that I bump into NCAR (National Center for Atmospheric Research), NOAA(National Oceanic and Atmospheric Administration), and CU Boulder researchers on a daily basis. I have...

  My previous blog showed you how to make a movie using ray tracing software and the powerful HPC scheduler on Azure with little programming.  Today, we’ll look at a more complex example in a series of 3 blog posts in the coming months. ...

  For the last two years, I’ve been using my 6 core 3.3 ghz Intel Nehalem X 980 processor courtesy of Intel (thank you!), it slices through just about any workload like butter, for example, encoding a 1 hour live meeting video takes a few minutes...

  We all know that Scientists and Engineers run Computationally intensive jobs. Many of us often wish we had an extra computer or a dozen to run the applications that seem to take forever on our desktop.  We wait for the code to compile, the...

I attended and gave a tutorial at the 11th LCI International conference last year at the Pittsburgh Super Computing Center.  There, I had the honor to meet several leading quantum chemistry HPC code researchers. One of them, Dr. Wang Yang. ...