Carl Nolan’s ramblings on development
Provided with the Microsoft Distribution of Hadoop, HDInsight, is a C library for HDFS file access. This code extends this library through a Managed C++ solution. This solution enables one to consume HDFS files from within a .Net environment. The purpose of this post is first to ensure folks know about the new Windows HDFS Managed library (WinHdfsManaged) provided alongside the native C library, and secondly to give a few samples of its usage from C#.
The complete code libraries can be downloaded from here: http://code.msdn.microsoft.com/Hadoop-Net-HDFS-File-Access-18e9bbee
Let’s start with a simple class diagram of the Win HDFS Managed library:
The main premise is that the HdfsFileSystem is your starting point, from which one can acquire a HdfsFileStream or a HdfsFileHandle. From the HdfsFileStream you can perform operations one would normally expect when working with .Net Streams. From the HdfsFileHandle you can perform operations analogous to normal HDFS file operations.
For brevity I have excluded samples using the HdfsFileHandle. So let’s run through some sample file operations.
As in all operations one firstly needs to get a connection to the HDFS cluster. This is achieved by calling a Connect() method and specifying the host, name or IP address, and access port:
Once one has the connection one can then easily perform a directory traversal to enquire into the files and directories:
Here is a sample output created from the test application:
In addition to getting directory information one can also query on a file or directory directly:
The HdfsFileSystem class also supports other operations such as copying and moving files, file renaming, deleting files, modifying security, checking a file exists, etc. The copy and move operations support copying and moving these files between systems.
So now onto creating and reading files.
Processing HDFS files is not that dissimilar from normal .Net file operations. Once one has opened a file for reading, operations are available for operations such as reading a byte, line, or block of bytes:
The OpenFile operations support parameter overrides for the file block size and replication factors, whereas a value of zero implies the default values will be used.
If one wants to read the full contents of a file into a second Stream, the HdfsFileStream makes this a simple process:
There are other options available for reading the full contents of a file. The first option is to perform a ReadLine() until a null is returned, processed using a StreamReader:
Alternatively, for more efficient reading of files, one can read the blocks of data into a byte array:
Other operations that are supported are PositionalReadByte(), PositionalReadBytes(), and Seek(). These operations allow reading the contents of a file from specific positions.
One final sample worth noting is copying a HDFS file to a local file using byte reads:
The reason a chunk size is specified in this case is to sync the size being used for HDFS file access to the byte array used for writing the local file.
If one has a Stream reference one can also get the associated file information:
Also one can modify the file properties:
So now onto writing files.
As in the case for reading, writing operations are supported for writing a byte, line, and block of bytes:
The chunk size when opening a file is set to correspond to the size of the buffer used for writing the data.
As in the reading case, if one wants to copy a file from the local file system to an HDFS file one would write:
All one has to do is read, in byte chunks, data from the local file and write the corresponding bytes to the HDFS file.
Of course one can also use the CopyTo operation:
A quick word is warranted on appending to a file. Although the API currently supports open files for Append, this is only supported in Hadoop version 1.0.0 and above.
The download not only consists of the compiled libraries but also the full source code and sample C# application that this post is based upon. The source supports both x86 and x64 compilations. However one has to remember that if one does a 32-bit compilation then a 32-bit version of the JRE will also be required.
In the Native library the include path is defined as:
If your environment variable is not correctly defined for the Java include directory this will have to be modified.
Happy HDFS file access!
I've compiled WinHdfsManaged.dll and then used it in a very simple test app, but when run the test app on Azure HDInsight cluster I get the exception:
System.IO.FileNotFoundException: Could not load file or assembly 'WinHdfsManaged.dll' or one of its dependencies. The specified module could not be found.
There is no such exception when run the same app locally.
WinHdfsManaged.dll assembly is there, so some dependency is missing.
What's wrong ?
The problem was with debug version of my test app. There are no debug versions of C++ runtime on Azure HDInsight cluster to load debug version of 'WinHdfsManaged.dll'. Release version does not throw the exception.
But there is another problem:
Feb 18, 2013 10:05:23 AM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure
INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 0 time(s).
Feb 18, 2013 10:05:25 AM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure
INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 1 time(s).
And so on.
What account are you trying to connect to the server as? Have you tried running under the Isotope user or from the HDInsight command rpompt?
Unable to connect!! getting INFO: Retrying connect to server: 127.0.0.1/127.0.0.1:9000. Already tried 0 time(s).
I have the HDInsught running on my box (http://localhost:8085/ works fine) but I cant run your test "WinHdfsManagedTest"
The problem was solved by connecting not to local loopback address (127.0.0.1), but to normal IP address of the local computer.
Can this library use in Hadoop Sandbox? Or it must be running in Environment Windows?