Fabulous Adventures In Coding : Scrabble

Five-Dollar Words For Programmers, Part Four: Boustrophedonic

Eric Lippert — Thu, 26 Mar 2009 16:39:00 GMT

Here’s an almost useless but thoroughly delightful five-dollar word. English of course is read left-to-right. Hebrew and Arabic are read right-to-left. A text is boustrophedonic if it reads left-to-right and right-to-left, alternating.

It’s from the Greek βουστροφηδόν meaning “as the ox turns”; you’d plow a field with an ox right to left and then left to right, obviously.

There are a number of ancient languages which were written boustrophedonically, which I’m sure has given members of the Unicode committee many sleepless nights. The example here is a rare early Latin text written boustrophedonically.

What’s the relevance to computer people not on the Unicode committee, given that odds are slim to none that Word will ever support boustrophedonic editing? That’s how most modern dot-matrix and inkjet printers print. The head goes left-to-right, then prints the image “backwards” right-to-left, and so on.

I discovered this word several years ago when grepping through the Scrabble Tournament Word List post-game to see if HEDONIC was in fact a legal bingo, or if I had played a phony. (It is legal.) But I had partial text matching on, so I hit BOUSTROPHEDONIC first and was intrigued, so I looked it up.

At fifteen letters long, it would run the entire width or height of the board. If OUST, HE and ON were all on the board already in the right place along an edge, you could play the remaining seven letters, get the triple-triple-triple word score plus the bingo bonus, and score 725 points. That would almost double the world record for highest scoring play (CAZIQUES, 392 points).

This seems unlikely, but you never know. Might come in handy.

Santalic tailfans, part two

Eric Lippert — Fri, 06 Feb 2009 21:19:31 GMT

As I have said before many times, there is only one sensible way to make a performant application. (As an aside: perfectly good word, performant, deal with it!) That is:

Set meaningful, measurable, customer-focused goals.
Write the code to be as clear and correct as possible.
Carefully measure your performance against your goals.
Did you meet your goal? Great! Don't waste any time on performance analysis. Spend your valuable time on features, documentation, bug fixing, robustness, security, whatever.
If you did not meet your goal, use tools to discover what the worst-performing fixable thing is, and fix it.

Iterate on that process, updating your goals if it proves necessary, until you either have something that meets your goals or you give up.

My explicit goal for my little dictionary search program was that it give results fast enough that I would not be sitting there impatiently waiting for more than a few seconds. That's a very customer-focused goal, me being the only customer of this program. With only a very small amount of tweaking my program met that goal right away, so why would I spend any more time on it? The program takes just under two seconds to report the results for a typical query, which is faster than I can read.

But suppose that we did want to improve the performance of this program for some reason. How? Well, let's go down the list. We have goals, we have very clear code, we can measure the performance easily enough. Suppose we didn't meet the goal. The last thing on the list is "use tools to discover what the slowest fixable thing is".

A commenter conjectured that the performance bottleneck of this program was in the disk I/O. As you can see, every time we do a query we re-read the two megabyte dictionary file dozens of times. This has the benefit of using very little memory; we never have more than one line of the dictionary in memory at a time, instead of the whole four megs it would take to store the dictionary (remember, the dictionary is in ASCII on disk but two-byte Unicode if in strings in memory, so the in-memory size will double.)

That's a conjecture -- a reasonable conjecture -- but nevertheless, it's just a guess. If I've learned one thing about performance analysis it's that my guesses about where the bottleneck is are often wrong. I'll tell you right now that yes, the disk-hitting performance is bad, but it is not the worst thing in this program, not by far. Where's the real performance bottleneck? Any guesses? Could you know without using tools?

Here's the result of a timing profile run of a seven-letter queries with one blank, repeated 20 times:

43%: Contains
21%: ReadLine
14%: Sort
7%: ToUpperInvariant
15%: everything else

Holy goodness! The call to the Contains extension method in the query to test whether the dictionary word is in the rack set array is almost half the cost of this program!

Which makes sense, once you stop to think about it. The "Contains" method is by its highly general nature necessarily very naive. When given an array, 99.9% of the time it has to look through the entire 26-item array because 99.9% of the time, the word is not actually going to match any of the possible racks. It cannot take advantage of any "early outs" like you could do if you were doing a linear search on a sorted list. And each time through the array it has to do a full-on string comparison; there's no fancy-pants checks in there that take advantage of string immutability or hash codes or any such thing.

We have a data structure that is designed to rapidly tell you whether a member is contained in a set. And even better, it already does the "distinct" logic. When I replace

var racks = (from rack in ReplaceQuestionMarks(originalRack)
select Canonicalize(rack)).Distinct().ToArray();

with

    var racks = new HashSet<string>(
                from rack in ReplaceQuestionMarks(originalRack)
                select Canonicalize(rack));

suddenly performance improves massively. The "Contains" drops down to 3% of the total cost, and of course, the total cost is now half of what it was before.

Another subtle point here: notice how when I changed the type of the variable "racks" from "array of string" to "set of string", I didn't have to redundantly change the type thanks to implicit typing of local variables. I want to emphasize the semantics here, not the storage mechanism. If I felt that communicating the storage mechanism was an important part of this code -- because it has such a strong effect on performance -- perhaps I would choose to emphasize the storage by eschewing the "var".

With this change, the program performance improves to about one second per query and the profile now looks like this:

39%: ReadLine
23%: Sort
11%: ToUpperInvariant
7%: the iterator block goo in FileLines
5%: ToArray (called by Join)
15%: everything else

Now the bottleneck is clearly the combination of repeated file line reading (48%) and the string canonicalization of every dictionary line over and over again (37%).

With this information, we now have data with which to make sensible investments of time and effort. We could cache portions of the dictionary in memory to avoid the repeated disk cost. Is the potential increase in speed worth the potentially massive increase in memory usage? We could be smart about it and, say, only cache the seven- and eight-letter words in memory.

We could also attack the canonicalization performance problem. Should we perhaps precompute an index for the dictionary where every string is already in canonical form? This in effect trades increased disk space, increased program complexity and increased redundancy for decreased time. Should we use a different canonicalization algorithm entirely?

All of these decisions are driven by the fact that I have already exceeded my performance goal, so the answer is "no". Good enough is, by definition, good enough. If I were using this algorithm to actually build a game AI, it would not be good enough anymore and I'd go with some more clever solution. But I'm not.

A nasality talisman for the sultana analyst

Eric Lippert — Thu, 05 Feb 2009 00:52:48 GMT

The other day my charming wife Leah and I were playing Scrabble Brand Crossword Game (a registered trademark of Hasbro and Mattel) as is our wont. I went first, drawing the Q and a bunch of vowels. Knowing that the Q is death to hold onto, I immediately opened with QI for 22 points. I silently thanked Miriam-Webster for adding QI, KI and ZA to the OSPD 4th edition.

My thankfulness was short-lived, as Leah thought for a few moments and then played ANALYST, for the fifty point "bingo" bonus, making QIS for good measure along the way. She then went on to thoroughly cream me. I am not very good at Scrabble.

We were wondering afterwards what all the possible seven and eight letter "bingo" words were that she could have made with the rack AALNST?. (A blank is conventionally written as "?".) I stared at it for a few moments and found SULTANA and SEALANT, but I was suspicious that there were a lot more. So I wrote a program to find out, which I shall share with you now.

(An interesting historical note: solving this problem efficiently was one of the questions I was given during my interviews when I first applied for full-time work at Microsoft; if you want to get a job here, you might need to know this!)

The core of the program is a method SearchDictionary which takes a rack string and returns a sequence containing every word in a dictionary file which can be formed using all the letters in that rack. In this case I wanted to know not just what all the words matching the given rack were, but what all the words matching the given rack that used an existing letter on the board were. In this case the only two letters on the board were Q and I, but let's assume that it could have been any letter on the board.

The main loop of my little console program looks like this:

public static void Main()
{
    while (true)
    {
        Console.Write("Enter rack (use '?' for blank): ");
        string rack = Console.ReadLine();
        if (rack == "")
            break;
        Console.WriteLine("{0} : {1}", rack, SearchDictionary(rack).Join());
        foreach (char c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ")
            Console.WriteLine("{0}+{1} : {2}", rack, c, SearchDictionary(rack + c).Join());
    }
}

Pretty straightforward so far. Of course, this is certainly not production quality code. This is a little utility that I hacked together for myself. A production-quality version would have a lot more error handling, for one thing.

The Join method is just a handy helper function that sticks a sequence of strings together:

private static string Join(this IEnumerable<string> strs)
{
return string.Join(" ", strs.ToArray());
}

The trick to efficiently searching for anagrams in a dictionary is to realize that all anagrams have the same letters, just in different order. If you "canonicalize" each word so that its letters are uppercase and in alphabetical order, then checking whether one word is an anagram of another is as simple as comparing their canonical forms:

private static string Canonicalize(string s)
{
    char[] chars = s.ToUpperInvariant().ToCharArray();
    Array.Sort(chars);
    return new string(chars);
}

I had a performance goal for this project; I wanted to be able to use the ~2MB 2006 Tournament Word List, searching for racks with up to two blanks, in reasonable human scale time, but not necessarily appearing to be instantaneous. My first naive implementation did not meet this goal so I made a few tweaks to the algorithm until it did, and then I stopped. (It is interesting to think about how this could be made much faster, but that's a subject for another day.)

private static IEnumerable<string> SearchDictionary(string originalRack)
{
const string dictionary = @"d:\twl06.txt";

    // Calculate all the possible distinct values for the rack.
    // As an optimization, stuff the resulting racks in an array so
    // that we do not recalculate them during the query.

var racks = (from rack in ReplaceQuestionMarks(originalRack)
select Canonicalize(rack)).Distinct().ToArray();

    // Check every line in the dictionary to see if it matches
    // any possible rack. As an optimization, do an early
    // out if the line length does not match the query length.

    return from line in FileLines(dictionary)
           where line.Length == originalRack.Length
           where racks.Contains(Canonicalize(line))
           select line;
}

LINQ queries are awesome. I love how the code reads like a description of what I'm trying to do, rather than how I'm doing it.

We need a way to turn a rack that might contain blanks into a sequence of the 26 or 26x26 possible racks without blanks. Here's a handy recursive method that does so:

private static IEnumerable<string> ReplaceQuestionMarks(string s)
{
    int index = s.IndexOf('?');
    if (index == -1)
    {
        yield return s;
        yield break;
    }
    foreach (char c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ")
    {
        string s2 = s.Substring(0, index) + c.ToString() + s.Substring(index + 1);
        foreach (string result in ReplaceQuestionMarks(s2))
            yield return result;
    }
}

And of course, the code to extract the lines from the dictionary is our old friend:

private static IEnumerable<string> FileLines(string filename)
{
    using (var sr = File.OpenText(filename))
    {
        while (true)
        {
            string line = sr.ReadLine();
            if (line == null)
                yield break;
            yield return line;
        }
    }
}

And there you go. Six very brief little methods that tell you that Leah could have made the seven letter bingos ANALYST CANTALS LATINAS PLATANS SALTANT SALTPAN SEALANT and SULTANA or by going through the "I", the eight letter bingos ALATIONS ANNALIST FANTAILS LANITALS NASALITY PLATINAS SANTALIC STAMINAL TAILFANS TALISMAN and VALIANTS.