Eric White's Blog

LINQ to Text Files

(Sept 30, 2008 - I've changed my approach for querying text files.  The new approach is detailed in LINQ to TEXT and LINQ to CSV.) 

Lazy evaluation is an important technique in functional programming. There is a entertaining article on functional programming here.

I was thinking about lazy evaluation the other day, and the issues of processing huge text files using streaming techniques, and I realized that LINQ could do really cool things if I implemented an extension method for StreamReader that iterated over the lines of a text file. Because of the lazy evaluation, provided you construct only certain queries, this technique doesn't read the entire file into memory. Each line is read, processed by both LINQ query expressions, output to the console, and then the next line is read. However, be aware that certain LINQ operators such as orderby can force the entire file to be read into memory.

The following example is somewhat artificial; it could be implemented with a single query expression, but I wanted to use two query expressions to demonstrate the laziness of processing. No lines are read from the text file until the program iterates over the results of the second query expression.

I just gotta say, this is the expressiveness that I REALLY wanted many years ago when I was writing Unix shell scripts to manipulate text files.

The implementation is trivial. The following listing contains both the extension method as well as the code to use the extension method:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Linq;

namespace LinqToText
{
  public static class StreamReaderSequence
  {
    public static IEnumerable<string> Lines(this StreamReader source)
    {
      String line;

      if (source == null)
          throw new ArgumentNullException("source");
      while ((line = source.ReadLine()) != null)
      {
        yield return line;
      }
    }
  }

  class Program
  {
    static void Main(string[] args)
    {
      StreamReader sr = new StreamReader("TextFile.txt");

      var t1 =
        from line in sr.Lines()
        let items = line.Split(',')
        where ! line.StartsWith("#")
        select String.Format("{0}{1}{2}",
            items[1].PadRight(16),
            items[2].PadRight(16),
            items[3].PadRight(16));

      var t2 =
        from line in t1
        select line.ToUpper();

      foreach (var t in t2)
        Console.WriteLine(t);

      sr.Close();
    }
  }
}

If you run this example with the following text file:

#This is a comment
1,Eric,White,Writer
2,Bob,Jones,Programmer
3,Orville,Wright,Inventor
4,Thomas,Jefferson,Statesman
5,George,Washington,President

It produces the following output.

ERIC            WHITE           WRITER
BOB             JONES           PROGRAMMER
ORVILLE         WRIGHT          INVENTOR
THOMAS          JEFFERSON       STATESMAN
GEORGE          WASHINGTON      PRESIDENT