What is HDInsight? HDinsight is the product name for Microsoft installation of Hadoop and Hadoop on azure service. HDInsight is Microsoft’s 100% Apache compatible Hadoop distribution, supported by Microsoft. HDInsight, available both on Windows Server or as an Windows Azure service, empowers...

For each Mapper and Reducer you can get Task attempt id and Task ID both. This can be done when you set up your map using the Context object. You may also know that the when setting a Reducer an unique reduce ID is used inside reducer class setup method. You can get this ID as well. There are multiple...

When submitting a Map/Reduce job in Hadoop cluster, you can provide number of map task for the jobs and the number of reducers are created depend on the Mappers input and the Hadoop cluster capacity. Or you can push the job and Map/Reduce framework will adjust it per cluster configuration. So setting...

While working with Map/Reduce jobs in Hadoop, it is very much possible that you have got “sorted data” stored in HDFS. As you may know the “Sort function” exists not only after map process in map task but also with merge process during reduce task, so having sorted data to sort...

Sometimes it is required to run multiple Map/Reduce jobs in same Hadoop cluster however opening several Hadoop command shell or (Hadoop terminal) could be trouble. Note that depend on your Hadoop cluster size and configuration, you can run limited amount of Map/Reduce jobs in parallel however if you...

When you have jobs running in Hadoop, you can use the map/reduce web view to list the current running jobs however what if you would need to kill any current running job because the submitted jobs started malfunctioning or in worst case scenario, the job is stuck in infinite loops. I have seen several...

When writing MapReduce programs you definitely going to hit problems in your programs such as infinite loops, crash in MapReduce, Incomplete jobs etc. Here are a few things which will help you to isolate these problems: Map/Reduce Logs Files: All MapReduce jobs activities are logged by default...

When running MapReduce jobs it is possible to have several MapReduce steps with overall job scenarios means the last reduce output will be used as input for the next map job. Map1 -> Reduce1 -> Map2 -> Reduce2 -> Map3... While searching for an answer to my MapReduce job, I stumbled...

If you format only Namenode, it will remove the metadata stored by the Namenode, however all the temporary storage and Datanode blocks will still be there. To remove temporary storage and all the Datanode blocks you would need to delete the main Hadoop storage directory from every node. This directory...

Once you have access enabled to Hadoop on Windows Azure you can run any mahout sample on head node. I am just trying to run original Apache Mahout ( http://mahout.apache.org/ ) sample which is derived from the clustering sample on Mahout's website ( https://cwiki.apache.org/confluence/display/MAHOUT...

Apache Hadoop Primary Namenode and secondary Namenode architecture is designed as below: Namenode Master: The conf/masters file defines the master nodes of any single or multimode cluster. On master, conf/masters that it looks like this: ———————-...

Apache Hadoop is designed to have Master Slave architecture. Master: Namenode, JobTracker Slave: {DataNode, TaskTraker}, ….. {DataNode, TaskTraker} HDFS is one primary components of Hadoop cluster and HDFS is designed to have Master-slave architecture. Master: NameNode Slave...

Map Task (HDFS data localization): The unit of input for a map task is an HDFS data block of the input file. The map task functions most efficiently if the data block it has to process is available locally on the node on which the task is scheduled. This approach is called HDFS data localization....

As you may have already know that Hadoop Cluster is network and disk, IO intensive. Recently I was trying to run a test scenario where I decided to change SATA hard disk to a high performance SSD Disk while keeping the cluster hardware the same. I was running the terra sort test to validate if having...

I was recently asked to show that Pig scripts are actually MapReduce jobs so to explain it in very simple way I have created the following example: Read a text file using Pig Script Dump the content of the file As you can see below that when “dump” command was used a...

Technically they both will do the job, you are looking from "either hive or Pig" perspective, means you don't know what you are doing yet. However if you first define the data source, scope and the result representation and then look for which one to choose between Hive or Pig, you will find they are...

In this article we will learn how to configure the same Hadoop cluster to do some customization. To learn about creating your own Hadoop cluster on Windows Azure by using your own Windows Azure Subscription account – Click here To add more worker nodes to Hadoop cluster on Windows Azure we...

[ As of now this functionality is not available with Hadoop on Windows Azure. These instruction are not applicable as of now and when things will change i will add more info here.. Thanks for your support ] Apache Hadoop distribution (currently in CTP) allows you to setup your own Hadoop cluster...

To get your Amazon S3 account setup with Apache Hadoop cluster on Windows Azure you just need you AWS security credentials which pretty much look like as below: After you completed creating Hadoop cluster in Windows Azure, you can log into your Hadoop Portal. In the portal, you can select “Manage...

When you run Map/Reduce job in Hadoop cluster on Windows Azure you will get an aggregated progress and log directly on portal, so you can see what is happening with your job. This log is different the what see when you do check individual job status in the datanode instead this log gives you cumulative...

After you log into your Hadoop Portal and configured your cluster, you can select “Manage Data” tile as below: On the next screen you can select: “Set up ASV” to set your Windows Azure Blob Storage as data source “Set up S3” to set your Amazon S3...

Microsoft Distribution of Apache Hadoop comes with Pig Support along with an Interactive JavaScript shell where users can run their Pig queries immediately without adding specific configuration. The Apache distribution running on Windows Azure has built in support to Apache Pig. Apache Pig is a platform...

Microsoft Distribution of Apache Hadoop comes with Hive Support along with an Interactive Hive shell where users can run their Hive queries immediately without adding specific configuration. The Apache distribution running on Windows Azure has built in support to Hive. What is Hive? Hive...

Microsoft distribution to Apache Hadoop comes by direct connectivity to cloud storage i.e. Windows Azure Blob storage or Amazon S3. Here we will learn how to connect your Windows Azure Storage directly from your Hadoop Cluster. To learn how to connect you Hadoop Cluster to Windows Azure Storage...

Microsoft distribution to Apache Hadoop comes by direct connectivity to cloud storage i.e. Windows Azure Blob storage or Amazon S3. Here we will learn how to connect your Windows Azure Storage directly from your Hadoop Cluster. As you know Windows Azure Storage access needed following two things...