Hadoop .Net HDFS File Access

re: Hadoop .Net HDFS File Access

Oleg Subachev — Fri, 17 May 2013 12:28:37 GMT

When I used the library from Windows Azure HDInsight Service CTP all was OK. I could copy files from HDInsight headnode to HDFS and then see through Interactive JavaScript console that the files are there in HDFS.

But now under Windows Azure HDInsight preview something goes wrong: when I run the same test application on headnode all seems OK - diagnostic messages and email are OK, but when I look in HDFS through Interactive JavaScript console there are no copied files there. So no errors, but no files ???

What's wrong ?

re: Hadoop .Net HDFS File Access

Oleg Subachev — Fri, 17 May 2013 12:15:00 GMT

The problem was solved by connecting not to local loopback address (127.0.0.1), but to normal IP address of the local computer.

re: Hadoop .Net HDFS File Access

Rui — Tue, 19 Mar 2013 20:05:15 GMT

Unable to connect!! getting INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 0 time(s).

I have the HDInsught running on my box (http://localhost:8085/ works fine) but I cant run your test "WinHdfsManagedTest"

re: Hadoop .Net HDFS File Access

Imaya — Tue, 12 Mar 2013 07:05:04 GMT

Good one.

re: Hadoop .Net HDFS File Access

Carl Nolan — Wed, 20 Feb 2013 11:54:41 GMT

What account are you trying to connect to the server as? Have you tried running under the Isotope user or from the HDInsight command rpompt?

re: Hadoop .Net HDFS File Access

Oleg Subachev — Mon, 18 Feb 2013 11:58:53 GMT

The problem was with debug version of my test app. There are no debug versions of C++ runtime on Azure HDInsight cluster to load debug version of 'WinHdfsManaged.dll'. Release version does not throw the exception.

But there is another problem:

Feb 18, 2013 10:05:23 AM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure

INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 0 time(s).

Feb 18, 2013 10:05:25 AM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure

INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 1 time(s).

And so on.

re: Hadoop .Net HDFS File Access

Oleg Subachev — Fri, 15 Feb 2013 13:24:43 GMT

I've compiled WinHdfsManaged.dll and then used it in a very simple test app, but when run the test app on Azure HDInsight cluster I get the exception:

System.IO.FileNotFoundException: Could not load file or assembly 'WinHdfsManaged.dll' or one of its dependencies. The specified module could not be found.

There is no such exception when run the same app locally.

WinHdfsManaged.dll assembly is there, so some dependency is missing.

What's wrong ?

Oleg Subachev