Rick Byers : Science

How is good software like good science?

rmbyers — Tue, 18 Sep 2007 08:19:00 GMT

I'm not one who believes mainstream large-scale software development really deserves the title of "computer science" (or "software engineering" for that matter). However, I have been thinking lately that there is an interesting analogy between good software development and good scientific theories. Here are some examples:

	Software program	Scientific theory
What is it?	A description of some desired computer behavior	A postulated description of reality
Who creates it?	Programmer	Theorist
Who first validates it?	Tester	Experimentalist
Provability	Usually can't ever be proven 100% correct. The longer we go without finding any bugs, the more we suspect it may be bug-free.	Can never be proven correct, just proven wrong. The more it resists being proven wrong, the more we believe it is probably 100% correct.
Testability	Good software is designed from the ground up to be tested effectively. If you build a large system first, and then later start thinking about how you might test it, you're likely to end up with a lot of bugs that are hard to find.	For something to qualify as a "scientific theory" it must be "falsifiable" - that is, there needs to be a way in theory you could prove it doesn't work. In general, the more ways you might be able to show that a theory is wrong, the better the theory is. For example, many people argue that most forms of string theory are not sufficiently testable to be seriously considered as science.
Predictive power	If you write a program precisely to pass a certain set of tests, you don't really know if it's correct until you test it against some new test cases.	Theories are generally designed to fit known experimental results. Properly predicting the results of experiments that have already been done is important, but the real test comes when the theory predicts results which aren't yet known.
Reproducibility	No matter how much testing you do, you ultimately need to get some real customers to try your software and see if they confirm your belief of correctness. Often the best tests come from users which are most critical of you. For example, seeing positive comments on slashdot about Silverlight has been a confirmation to me that my team is making some good decisions.	No matter how much one group's experiments agree with theory, little weight is attributed to the experiments unless other independent groups are able to reproduce the results. Often the best tests of a theory come from people who believe it to be false. The history of quantum mechanics is full of scientists who disbelieved it, but whose arguments and experiments ultimately strengthened it (such as Einstein and the EPR paradox).
Simplicity	Correct software tends to have a simpler, smaller, and/or more elegant implementation. This doesn't mean it's behavior needs to be simple.	All things being equal, simpler theories tend to be the correct ones. This is known as Occam's razor. There are many examples in science of theories which are conceptually and mathematically very simple, but whose implications are very complex and non-obvious. A classic example is an explanation for the complex movements of the planets in the sky. Ptolemy's model with the earth at the center was very complex, but Copernicus presented a much simpler model with the sun at the center.
Believability	Good software tends to be well structured and documented so that a person can reason about it's correctness.	Good theories tend to have a logical and believable explanation for why they should be correct.
Specificity	Good software tends to have a concrete specification for what is considered correct behavior. We try to avoid the temptation to build something, write some tests for it, and call the results of those tests "correct". When the tests are first executed, there is ideally only one possible correct output.	Good theories tend to have fewer "free variables", which are parameters determined by experiment. For example, each planet in Ptolemy's model of the solar system had a number of concentric rings with various sizes associated with it. In the Newtonian model, the movement it determined entirely by each planet's mass, orbital radii (major and minor), and the universal gravitational constant. When new experiments are performed, there is ideally just one result that would be consistent with the theory.
Generality	The field of software (and often individual large systems) advances by replacing special-purpose components with general-purpose frameworks. Operating systems and managed runtimes like the CLR are obvious examples here.	Good theories often supercede many previous (apparently unrelated) results, encompassing them all under one larger umbrella. A great example here is the realization that electricity, magnetism, radio waves, and light were all properties of the same electromagnetic force (and eventually even just one aspect of the electroweak force).
Reusability	Good software tends to build on previous successes, re-using components that are known to be of high quality, but avoiding dependencies on high-risk pieces. It's extremely difficult (and wasteful) to build a completely new large system from scratch. Personally, I believe this is an area we could do better on the CLR.	The progress of science has obviously only been possible by building on previous successes.

So what can we learn from the study of good science (which has had a lot longer to mature) about how we should approach software? Here are some ideas:

Be evidence-based - try to rely as much as possible on concrete data, it helps avoid the inevitable temptation to deceive yourself.
Have a culture of humility - accept that it's a lot easier to be wrong than it is to be right, and that your work should be assumed to be incorrect until there is enough independent evidence to suggest otherwise. Recognize that certainty and black-and-white positions are usually overly simplistic and damaging.
Extraordinary claims require extraordinary evidence - there is no silver bullet, trying to chase it can leave you running in circles.
Be willing to accept a paradigm shift when necessary - it's sometimes necessary to abandon a long-held and cherished philosophy and accept well-justified radical new ideas in order to keep making progress.
Strive for simplicity - adding more total lines of code (like more special cases in your theory) should be considered more of a last resort, than as the normal process of growth. You can only continue to tack on new ad-hoc solutions to problems for so long before the maintenance costs become stifling. Removing code is more important for the quality of your software than writing new code.
Be self-critical - it's human nature for a group of like-minded intelligent people to be blinded to the truth by their arrogance. Recognize this and seek out opportunities to prove yourself wrong.
Re-architect when necessary - it's sometimes advantageous to combine a group of previously independent things into a new component which replaces them all.
Study the past - learn from the patterns of past mistakes and successes, and recognize how to predict the most likely avenues for success. Often negative results (understanding why a theory or piece of software failed to be successful) are more valuable than positive results.
Know when to start over - sometimes we have to be willing to let go and give up on an idea or piece of software and start from scratch. Clinging to the past can be very destructive in the long-run.

I'd love to hear your comments about where this analogy works and where it doesn't. I keep thinking about it whenever I read something about "good science", but I'm not yet sure whether it's just the natural tendency to make connections between things you know well, or whether there is some deeper underlying principle here connecting these two ideas.

Using LINQ for Computational Genomics

rmbyers — Tue, 08 May 2007 08:23:00 GMT

I’ve been playing around a bit lately with computational genomics (I’m doing a project for my parallel computation class). I wanted to write some simple algorithms that operate on potentially large amounts of DNA data without using a ton of RAM. For example, the entire human genome is 3 billion base pairs – reading it all into memory is out of the question (at least on my home PC). It occurred to me that this was a perfect opportunity to spend some more time using LINQ. Computations like this are inherently stream based, and LINQ allows you to express and compose operations on streams very effectively. Here’s a simple example that I hope will help demonstrate the power of LINQ.

One of the simplest forms of statistical analysis done on a DNA sequence is producing a graph of it’s “GC density”. To a computer scientist, a DNA sequence is just a string of A, C, T, and G characters. The GC density with window size ‘w’ for a sequence is another (similarly sized) sequence of numbers between 0 and 1, each of which represents the ratio of Gs and Cs in the previous ‘w’ characters. For example, for the sequence “ATGCAG”, the GC density with window size 2 is “0, 0.5, 1, 0.5, 0.5”.

So, in order to calculate this, I first started with a simple function which, given a nucleotide sequence, would return an identical-length sequence of the number of Gs or Cs seen so far:

/// <summary>

/// Generate a sequence that is the prefix sum of Gs and Cs in the input sequence.

/// </summary>

/// <param name="sequence">The input nucleotide sequence</param>

/// <returns>A sequence of the same length for which each element indicates the

/// count of Gs and Cs between the beginning and corresponding position in

/// the input sequence.</returns>

public static IEnumerable<int> GenerateGCCount(IEnumerable<Nucleotide> sequence)

{

int count = 0;

foreach (Nucleotide n in sequence)

{

if (n == Nucleotide.G || n == Nucleotide.C)

count++;

yield return count;

}

Given this sequence (call it s), the density for window size w at some position i could (conceptually) be calculated with the formula d[i] = (s[i+w]-s[i])/w. However, since we might want a large window size, we don’t even necessarily want to have w nucleotides in memory at once, so we can’t use such a simple formula. However, the stream-based version is almost as simple:

/// <summary>

/// Compute the GC density of the specified nucleotide sequence.

/// </summary>

/// <param name="sequence">The sequence of nucleotides</param>

/// <param name="windowSize">Size of the window for each density calculation</param>

/// <returns>A sequence for which each element indicates the ratio of Gs and Cs to

/// total nucleotides over the past windowSize elements</returns>

public static IEnumerable<float> ComputeGCDensity(

this IEnumerable<Nucleotide> sequence,

int windowSize)

{

if (windowSize < 1)

throw new ArgumentOutOfRangeException("windowSize");

// First compute a sequence of the running count of G or C nucleotides

var gcCounts = GenerateGCCount(sequence);

// Combine the sequence with a copy of itself offset by the window size

// The elements of the sequence are pairs of counts, with the first element

// of the pair being the GC count at the beginning of the window, and the

// second element of the pair being the GC count at the end of the window.

var gcWindows = gcCounts.Zip(gcCounts.Skip(windowSize - 1));

// Now for each window, compute the gc density and return it

return gcWindows.Select(

p => (float)(p.Second - p.First) / windowSize);

}

This is pretty much it. If you remove the comments this is only a couple lines of code (or a single long query line if you prefer). Zip is just a helper function I wrote (based on list-processing functions in other functional languages like standard ML’s ListPair.zip or OCaml’s List.Combine) that returns a sequence of pairs from the two input sequences:

/// <summary>

/// A pair (2-tuple)

/// </summary>

/// <typeparam name="T1">Type of the first element</typeparam>

/// <typeparam name="T2">Type of the second element</typeparam>

public struct Tuple<T1,T2>

{

public Tuple(T1 first, T2 second)

{

First = first;

Second = second;

}

public readonly T1 First;

public readonly T2 Second;

}

/// <summary>

/// Combine two sequences into a sequence of pairs

/// </summary>

/// <typeparam name="T1">Type of the elements of the first sequence</typeparam>

/// <typeparam name="T2">Type of the elements of the second sequence</typeparam>

/// <param name="source1">The first sequence</param>

/// <param name="source2">The second sequence</param>

/// <returns>A sequence that is as long as the shorter of source1 and source2.

/// Each element is a pair of values - one from each of the input sequences

/// at the same position. If the input sequences are of unequal length, any extra

/// data in the longer list is not used.</returns>

public static IEnumerable<Tuple<T1, T2>> Zip<T1, T2>(

this IEnumerable<T1> source1,

IEnumerable<T2> source2)

{

// Conceptually we just want to foreach over both sequences, but we must

// write the code manually since foreach works for only a single sequence

var enum1 = source1.GetEnumerator();

var enum2 = source2.GetEnumerator();

while (enum1.MoveNext() && enum2.MoveNext())

{

yield return new Tuple<T1,T2>(enum1.Current, enum2.Current);

}

Together I think this is pretty simple and elegant. I create two instances of the same sequence, offset one by the window size, and then use their difference to compute the density. All of this is done on demand each time a density value is required. I’m not keeping any information about any position other than the current one in memory at any one time. One drawback is that for small window sizes, it may actually be pretty silly (from a performance perspective) to read each byte of the sequence from the file twice. But I consider this an optimization problem with the operating system file system cache should ideally solve for me. In practice execution still appears to be CPU bound, so I think the OS is doing a good job – the extra I/O isn’t a concern.

If you’re new to this style of programming, it can sometimes get a little confusing trying to keep track of the timing in which things execute. C#’s iterator method syntax and the IEnumerable pattern in general can make the use of delayed computation a little subtle. This is a big benefit of using a functional programming style like I have where no shared state is modified and the functions have no side-effects. The order in which things execute is unimportant.

To test that this actually worked on large data sources without using a lot of RAM, I ran it on the entire Human Chromosome 1 sequence (the largest one - almost 300 MB uncompressed) with a window size of 100000. For kicks, I also ran it on the Chimpanzee chromosome 1. Both runs took about 10 minutes on my relatively slow home PC, and used only a steady 10MB of RAM. Here are the resulting graphs if you’re interested.