Carl's Blog - All Comments

re: Hadoop .Net HDFS File Access

Oleg Subachev — Fri, 17 May 2013 12:28:37 GMT

When I used the library from Windows Azure HDInsight Service CTP all was OK. I could copy files from HDInsight headnode to HDFS and then see through Interactive JavaScript console that the files are there in HDFS.

But now under Windows Azure HDInsight preview something goes wrong: when I run the same test application on headnode all seems OK - diagnostic messages and email are OK, but when I look in HDFS through Interactive JavaScript console there are no copied files there. So no errors, but no files ???

What's wrong ?

re: Hadoop .Net HDFS File Access

Oleg Subachev — Fri, 17 May 2013 12:15:00 GMT

The problem was solved by connecting not to local loopback address (127.0.0.1), but to normal IP address of the local computer.

re: Hadoop Binary Streaming and PDF File Inclusion

Carl Nolan — Mon, 25 Mar 2013 11:33:01 GMT

Hi Sagar you can use Hadoop for document processing in this fashion, provided you have sufficient volume. If you know Java you can use the Java binary reader that comes with this code as the reader for submitting a MR job written just in Java.

re: Hadoop Binary Streaming and PDF File Inclusion

Sagar Nikam — Thu, 21 Mar 2013 14:32:13 GMT

Sir,Your idea is great,as I want same with some modification.

The theme is,I have thousand of files pdf,txt,docx in a folder.I want to extracts most occuring top 10 words for each file using Hadoop/any software which gives quick results

I totally don't know C# & .NET,I try to understand the code,but I can't.

I know little bit of Java.Can u tell me how to modify it into Java Program?

I will be thankful,if u convert it completely into MapReduce form as many peoples are using Java for Hadoop programming

You can mail me also - sagarnikam123@gmail.com

re: Hadoop .Net HDFS File Access

Rui — Tue, 19 Mar 2013 20:05:15 GMT

Unable to connect!! getting INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 0 time(s).

I have the HDInsught running on my box (http://localhost:8085/ works fine) but I cant run your test "WinHdfsManagedTest"

re: Hadoop .Net HDFS File Access

Imaya — Tue, 12 Mar 2013 07:05:04 GMT

Good one.

re: Hive and XML File Processing

Carl Nolan — Mon, 04 Mar 2013 09:41:46 GMT

The error relating to java.lang.UnsupportedClassVersionError: org/apache/mahout/classifier/bayes/XmlElementStreamingInputFormat : Unsupported major.minor version 51.0 could be down to the current code being compiled with the Oracle SDK. Are you running this on Azure. If so you may need to recompile the Java classes. There is a script to do this, just point it to the correct javac exe.

re: Hive and XML File Processing

Gordon — Fri, 22 Feb 2013 18:21:50 GMT

I hits error when execute below statement:

CREATE EXTERNAL TABLE StoresXml (storexml string)

STORED AS INPUTFORMAT 'org.apache.mahout.classifier.bayes.XmlElementStreamingInputFormat'

OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

LOCATION '/stores/demographics';

Error:

java.lang.UnsupportedClassVersionError: org/apache/mahout/classifier/bayes/XmlEl

ementStreamingInputFormat : Unsupported major.minor version 51.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)

at java.lang.ClassLoader.defineClass(ClassLoader.java:615)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14

What wrong? How to fix?

re: Hadoop .Net HDFS File Access

Carl Nolan — Wed, 20 Feb 2013 11:54:41 GMT

What account are you trying to connect to the server as? Have you tried running under the Isotope user or from the HDInsight command rpompt?

re: Hadoop .Net HDFS File Access

Oleg Subachev — Mon, 18 Feb 2013 11:58:53 GMT

The problem was with debug version of my test app. There are no debug versions of C++ runtime on Azure HDInsight cluster to load debug version of 'WinHdfsManaged.dll'. Release version does not throw the exception.

But there is another problem:

Feb 18, 2013 10:05:23 AM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure

INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 0 time(s).

Feb 18, 2013 10:05:25 AM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure

INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 1 time(s).

And so on.