Rick Byers : Languages

Using LINQ for Computational Genomics

rmbyers — Tue, 08 May 2007 08:23:00 GMT

I’ve been playing around a bit lately with computational genomics (I’m doing a project for my parallel computation class). I wanted to write some simple algorithms that operate on potentially large amounts of DNA data without using a ton of RAM. For example, the entire human genome is 3 billion base pairs – reading it all into memory is out of the question (at least on my home PC). It occurred to me that this was a perfect opportunity to spend some more time using LINQ. Computations like this are inherently stream based, and LINQ allows you to express and compose operations on streams very effectively. Here’s a simple example that I hope will help demonstrate the power of LINQ.

One of the simplest forms of statistical analysis done on a DNA sequence is producing a graph of it’s “GC density”. To a computer scientist, a DNA sequence is just a string of A, C, T, and G characters. The GC density with window size ‘w’ for a sequence is another (similarly sized) sequence of numbers between 0 and 1, each of which represents the ratio of Gs and Cs in the previous ‘w’ characters. For example, for the sequence “ATGCAG”, the GC density with window size 2 is “0, 0.5, 1, 0.5, 0.5”.

So, in order to calculate this, I first started with a simple function which, given a nucleotide sequence, would return an identical-length sequence of the number of Gs or Cs seen so far:

/// <summary>

/// Generate a sequence that is the prefix sum of Gs and Cs in the input sequence.

/// </summary>

/// <param name="sequence">The input nucleotide sequence</param>

/// <returns>A sequence of the same length for which each element indicates the

/// count of Gs and Cs between the beginning and corresponding position in

/// the input sequence.</returns>

public static IEnumerable<int> GenerateGCCount(IEnumerable<Nucleotide> sequence)

{

int count = 0;

foreach (Nucleotide n in sequence)

{

if (n == Nucleotide.G || n == Nucleotide.C)

count++;

yield return count;

}

Given this sequence (call it s), the density for window size w at some position i could (conceptually) be calculated with the formula d[i] = (s[i+w]-s[i])/w. However, since we might want a large window size, we don’t even necessarily want to have w nucleotides in memory at once, so we can’t use such a simple formula. However, the stream-based version is almost as simple:

/// <summary>

/// Compute the GC density of the specified nucleotide sequence.

/// </summary>

/// <param name="sequence">The sequence of nucleotides</param>

/// <param name="windowSize">Size of the window for each density calculation</param>

/// <returns>A sequence for which each element indicates the ratio of Gs and Cs to

/// total nucleotides over the past windowSize elements</returns>

public static IEnumerable<float> ComputeGCDensity(

this IEnumerable<Nucleotide> sequence,

int windowSize)

{

if (windowSize < 1)

throw new ArgumentOutOfRangeException("windowSize");

// First compute a sequence of the running count of G or C nucleotides

var gcCounts = GenerateGCCount(sequence);

// Combine the sequence with a copy of itself offset by the window size

// The elements of the sequence are pairs of counts, with the first element

// of the pair being the GC count at the beginning of the window, and the

// second element of the pair being the GC count at the end of the window.

var gcWindows = gcCounts.Zip(gcCounts.Skip(windowSize - 1));

// Now for each window, compute the gc density and return it

return gcWindows.Select(

p => (float)(p.Second - p.First) / windowSize);

}

This is pretty much it. If you remove the comments this is only a couple lines of code (or a single long query line if you prefer). Zip is just a helper function I wrote (based on list-processing functions in other functional languages like standard ML’s ListPair.zip or OCaml’s List.Combine) that returns a sequence of pairs from the two input sequences:

/// <summary>

/// A pair (2-tuple)

/// </summary>

/// <typeparam name="T1">Type of the first element</typeparam>

/// <typeparam name="T2">Type of the second element</typeparam>

public struct Tuple<T1,T2>

{

public Tuple(T1 first, T2 second)

{

First = first;

Second = second;

}

public readonly T1 First;

public readonly T2 Second;

}

/// <summary>

/// Combine two sequences into a sequence of pairs

/// </summary>

/// <typeparam name="T1">Type of the elements of the first sequence</typeparam>

/// <typeparam name="T2">Type of the elements of the second sequence</typeparam>

/// <param name="source1">The first sequence</param>

/// <param name="source2">The second sequence</param>

/// <returns>A sequence that is as long as the shorter of source1 and source2.

/// Each element is a pair of values - one from each of the input sequences

/// at the same position. If the input sequences are of unequal length, any extra

/// data in the longer list is not used.</returns>

public static IEnumerable<Tuple<T1, T2>> Zip<T1, T2>(

this IEnumerable<T1> source1,

IEnumerable<T2> source2)

{

// Conceptually we just want to foreach over both sequences, but we must

// write the code manually since foreach works for only a single sequence

var enum1 = source1.GetEnumerator();

var enum2 = source2.GetEnumerator();

while (enum1.MoveNext() && enum2.MoveNext())

{

yield return new Tuple<T1,T2>(enum1.Current, enum2.Current);

}

Together I think this is pretty simple and elegant. I create two instances of the same sequence, offset one by the window size, and then use their difference to compute the density. All of this is done on demand each time a density value is required. I’m not keeping any information about any position other than the current one in memory at any one time. One drawback is that for small window sizes, it may actually be pretty silly (from a performance perspective) to read each byte of the sequence from the file twice. But I consider this an optimization problem with the operating system file system cache should ideally solve for me. In practice execution still appears to be CPU bound, so I think the OS is doing a good job – the extra I/O isn’t a concern.

If you’re new to this style of programming, it can sometimes get a little confusing trying to keep track of the timing in which things execute. C#’s iterator method syntax and the IEnumerable pattern in general can make the use of delayed computation a little subtle. This is a big benefit of using a functional programming style like I have where no shared state is modified and the functions have no side-effects. The order in which things execute is unimportant.

To test that this actually worked on large data sources without using a lot of RAM, I ran it on the entire Human Chromosome 1 sequence (the largest one - almost 300 MB uncompressed) with a window size of 100000. For kicks, I also ran it on the Chimpanzee chromosome 1. Both runs took about 10 minutes on my relatively slow home PC, and used only a steady 10MB of RAM. Here are the resulting graphs if you’re interested.

More on generic variance

rmbyers — Fri, 02 Jun 2006 04:44:00 GMT

In my entry on generic variance in the CLR, I said that you can’t convert a List<String> to a List<Object>, or even an IEnumerable<String> to IEnumerable<Object>. I should point out however that the real-world scenarios where you’d want to do this usually involve passing an object of a more specific type to an API that (for abstraction reasons) takes a less specific type. For example, say you have a class hierarchy like this:

abstract class Shape

{

public abstract double ComputeArea();

}

class Square : Shape

{

public Square(double height)

{

m_height = height;

}

public override double ComputeArea()

{

return m_height * m_height;

}

private double m_height;

}

class Circle : Shape

{

public Circle(double radius)

{

m_radius = radius;

}

public override double ComputeArea()

{

return Math.PI * m_radius * m_radius;

}

private double m_radius;

}

You’d like to be able to use it uniformly like this:

class Program

{

static void Main(string[] args)

{

List<Square> ls = new List<Square>();

ls.Add(new Square(2));

ls.Add(new Square(3));

double totS = GetTotalArea(ls); // won’t compile

List<Circle> lc = new List<Circle>();

lc.Add(new Circle(2));

lc.Add(new Circle(3));

double totC = GetTotalArea(lc); // won’t compile

}

public static double GetTotalArea( IEnumerable<Shape> l)

{

double total = 0;

foreach (Shape s in l)

{

total += s.ComputeArea();

}

return total;

}

This would only work if C# supported generic variance and IEnumerable was defined as IEnumerable<+T>. However, there is another option in this case. You could make GetTotalArea a generic method and rely on a base-type constraint:

public static double GetTotalArea<T>(IEnumerable<T> l)

where T : Shape

{

double total = 0;

foreach (Shape s in l)

{

total += s.ComputeArea();

}

return total;

}

Now the above Main method will work perfectly, you don’t even have to modify it to specify the type parameters (C#’s type inference can figure them out automatically). So although you're not really converting an IEnumerable<Square> to an IEnumerable<Shape>, you are able to use it that way.

This is certainly great. And in practice, most .NET developers will find the support in .NET 2.0 for generics to be more than powerful enough. However, if you’re trying to do some serious “generic programming” [3], or if you’re excited by programming language theory like I am, then this does still leave something to be desired. Most notably, we can specify an upper-bound using a constraint like this (rather than require our language to support covariant type parameters), but the CLR and C# don’t have support for lower bounds (“supertype constraints”) so you can’t use this technique in place of contravariant type parameters (eg. my IComparer<-T> example). See section 4.4 (“Comparison with Parametric Methods”) of [1] for more details.

Java 5 addresses these scenarios with “Wildcard types” [2]. Eg. you would use a “List<? extends Shape>” in Java to implement my example above, and there is also a syntax “? super T” for lower-bounds. It’s interesting to note that the authors of the paper on wildcards indicate that the main advantage of wildcard types over using normal generic methods (as I’ve done above) is that wildcard types don’t require exact type information (see section 4.4 of [2]). The big difference between generics in .NET and Java is that in .NET, the CLR supports generics to the core and so we have exact type information everywhere (which is why you can get information about generic types using Reflection at run-time). In Java, generics are “erased” by the compiler, and so at run-time, the information about generic instantiations are lost. Using erasure has the benefit of better compatibility with old code and a much simpler implementation, but extending the VM (like we did for the CLR) has several important benefits including avoiding boxing for instantiations at value types and therefore better performance (eg. a List<int> can be very efficient), and the ability to use exact type information at run-time.

The big difference between wildcards and generic variance in the CLR is that wildcard types are an example of “usage-site variance” where MSIL uses “definition-site variance” (meaning it’s the type definition that specifies the variance annotation, not the user of the type). I was reading about a cool academic programming language called Scala recently, and was pleased to see that after some experience with usage-site variance, they decided to switch to definition-site variance because they found it easier to use correctly (see “Comparison with wildcards” in [4]). Scala is a very cool language (my programming languages professor recently mentioned it as a great example of a language on the forefront of modern academic language design), and can target .NET. Unfortunately they haven’t built support for the V2.0 CLR yet so they aren’t actually making use of the definition-site variance support in the CLR (for the moment <grin>).

Anyway, I think that’s about all I have to say about generic variance. I’ve got a programming languages exam on Tuesday (I’m working on my masters in computer science) so I think I better stop procrastinating and study for it <grin>.

References

[1] On variance-based subtyping for parametric types

[2] Adding Wildcards to the Java Programming Language

[3] A Comparative Study of Language Support for Generic Programming

[4] An Overview of the Scala Programming Language (2. Edition)

Linq and the cost of additional language complexity

rmbyers — Sun, 25 Sep 2005 23:10:00 GMT

Uwe Keim posted a thought provoking comment in response to my entry about Linq. Here is an excerpt:

I do see the benefits, but I also have a big déjà vû: The C#-language seems to go the C++-way, where I, even after 10 years of programming, don't know all of the features and sometimes still wonder "why does this thing behave this way?". So the complexity is raised from version to version. Does everyone think, this is a good way? I think it would be better to NOT enhance the language/compiler from version to version, but to enhance functionality by enhancing the library instead.

I think the added complexity concern is a valid one which deserves careful analysis. I'll just touch on my thoughts here.

One important difference between C++ and C#/VB is that C++ is designed by committee, where C# and VB each have a chief architect (Anders Hejlsberg and Paul Vick respectively) who doesn't have to try and satisfy everyone's favorite language request. I know the complexity concern is something Anders and his team have taken very seriously (as have the VB folks). I've heard Anders talk about the fact that they have to be extremely careful about this, and that he'd rather err on the side of caution. The C# team has rejected several of my favorite language features (such as generic variance, C++ style const, etc.) because they felt the benefits for the average user didn't significantly outweigh the extra complexity (even through there were compelling benefits for SOME users). This is where a lot of the "art" of language design comes into play. Personally I have a lot of respect for Anders and Paul and trust their judgment and the customer research their teams have done which concluded that in this case the extra complexity was justified. As much as I'd like to believe otherwise, there is probably a good reason here why Anders is the chief C# architect and you and I are not <grin>.

Of course they could be wrong, and it's important that Microsoft and our customers evaluate this risk seriously. Uwe suggested a compiler plug-in model where you could enable different features on demand. I'd love a public compiler plug-in model - there are so many cool features I'd like to add and use in my code. I've even asked Anders about this (in the context of the fact that expression trees means that some errors that could normally be detected a compile time now become run-time errors), and he indicated that to maintain simplicity and comprehensibility they are avoiding any such dynamic / customizable compiler behavior. I think Anders has a point here. In this case, just using the v1.1 or v2.0 compiler to target the latest CLR has similar benefits except that the possible feature combinations are fixed to well known sets, which avoids the confusion and complexity of the plug-in or optional feature model.

The other thing to keep in mind is that a lot of effort went into reducing complexity by adding a few powerful general language concepts, instead of many feature-specific concepts. I think Linq did even better than Cω in this respect. For example, unlike the Cω compiler, the C# 3.0 compiler has no knowledge of databases, SQL or XML, it's all in the DLinq and XLinq libraries. Personally, I find the new language features in the C# 3.0 spec to be quite simple and elegant, but then I'm a (functional) programming languages nerd so I'm not exactly the target audience.

Ultimately it will be up to our customers. If, after seriously experimenting and evaluating Linq, you think the extra complexity isn't worth the benefit, then it's important that you give us your feedback (we're getting better and better at listening to this sort of thing). On the other hand, the response from the PDC was overwhelmingly positive, so it would be likely to be an uphill battle <grin>.

Comparison of a simple select statement in DLinq (C# 3.0) vs. ADO.Net

rmbyers — Sun, 25 Sep 2005 07:59:00 GMT

Six months ago I posted a comparison of a simple select statement in C-omega vs. ADO.Net which some people found very exciting. Now that Linq has been officially unveiled, I figured I should update my comparison using C# 3.0 and DLinq. Although Linq and C-omega have some significant differences, everything I said in that post about the benefits of C-omega applies equally well to Linq (in fact, I like Linq even better, primarily due to expression trees) [Update: Erik Meijer e-mailed me to say that C-omega also has indirect support for expression trees by exposing the parse tree to compiler plug-ins.]

Here is the example ADO.Net code using strongly-typed data sets:

SqlDataAdapter da = new SqlDataAdapter(

"SELECT * FROM Employees WHERE City= @city", nwindConn );

SqlParameter cityParam = da.SelectCommand.Parameters.Add("@city", SqlDbType.VarChar, 80);

cityParam.Value = city;

NorthwindDataSet ds = new NorthwindDataSet();

da.Fill(ds, ds.Employees.TableName );

foreach (NorthwindDataSet.EmployeesRow dr in ds.Employees.Rows)

{

string name = dr.LastName;

int id = dr.EmployeeID;

Console.WriteLine( id + ": " + name);

}

And here is the equivalent C# 3.0 code:

var rows = from e in db.Employees where e.City == city select e;
foreach( var e in rows )
{
   string name = e.LastName;
   int id = e.EmployeeID;
   Console.WriteLine( id + ": " + name );
}

Or even more concisely:

foreach( var e in db.Employees.Where( e => e.City == city ) )
Console.WriteLine( e.EmployeeID + ": " + e.LastName );

There is already tons of information available about Linq (more so than there ever was for C-omega), and there are some great blogs by plenty of people more authoritative than myself (eg Matt, Dinesh, Paul, Dan, Scott, etc.), so I probably won't say much more about it myself. But isn't it awesome that this is almost certainly going to become part of mainstream programming in a couple years? Are we finally on the road to eliminating SQL injection attacks (not to mention all the other great benefits)? It's advances like this that make me proud to work for Microsoft!

Interested in C-omega? LINQ finally announced!

rmbyers — Wed, 14 Sep 2005 01:07:00 GMT

Ever since I started planning for my users-group talk (and wrote this blog entry) about data access with C-omega, I've been dying to tell everyone about the plans to add similar functionality to C#. For those of you at my talk, you'll remember this video of Anders which alluded to the work that was being done here. Well, today at the PDC, project LINQ (formerly "Clarity", etc.) was publicly announced. In a nutshell, C# 3.0 (and VB 9.0) will add fundamental support for seamlessly working with data, which includes several core language enhancements that are generally useful (such as type inference). Functional programming language fans (such as myself) should find the design especially appealing. See the LINQ website for details.

This is definitely the one thing I'm most excited about in the Orcas (3.0) release. Not only is it cool for programming-language theory fans, but it also has the potential to significantly improve the productivity of a typical enterprise developer.

Comega talk

rmbyers — Tue, 05 Apr 2005 06:48:00 GMT

On Thursday I gave a .NET users group talk on Comega to somewhere around 100 .NET developers. Overall I think it went pretty well. I was nervous at first, but once I got into talking about the cool stuff I like I forgot about the pressure and had a good time. If you were there, please let me know what you thought about both my presentation style and the technical content.

I asked the audience how many of them wrote a lot of code for interacting with databases or XML data, and almost every hand went up. My premise was that there was a huge potential to improve the productivity, quality and security of a lot of mainstream software development through data-oriented language features like those in Comega, and that if customers agreed, it was their responsibility to demand these features from the industry. In my opinion, we're far too content as programmers to go about doing things the way we've always done them rather than push for (and pay for) better ways of building systems (obviously finding the right balance here is hard). Perhaps this reflects a typically short-term-oriented reward system of the software industry (shipping the next new feature sooner), or perhaps it just reflects a cynicism of a market full of half-baked "improve productivity quickly" products that lack empirical data and research (which development shops often aren't eager to share).

One thing that surprised me was that when I showed the naive ADO.Net query code and asked the audience what the problems were with it, it wasn't until the 3rd response a minute or so into the discussion that someone said it was vulnerable to a SQL injection attack. This may have just been typical developer shyness to answer obvious questions, but I had expected that to be the most obvious response. Regardless, it is scary how easy it is to fall into this trap without noticing right away that constructing a SQL query based on user input is a bad idea. It's also unfortunate how many more keys you need to press (~128 in my example) to avoid the problem.

There was some great discussion around how Comega might or might not be able to also help with the encapsulation typically provided by business objects. After playing with some options and thinking about this a bit more, I think the two issues (accessing back-end data, and exposing that data through business objects) are largely orthogonal. If you want to add a layer of abstraction that provides encapsulation, data hiding, client-side validation, etc. then you're still going to have to wrap the data in some custom objects that define the interface and logic you want. However, Comega can make it much easier to implement those objects, especially due to the simple syntax for converting data to different shapes.

Anyway, for those of you that asked me after the presentation to post my slide deck, here it is. Sorry I didn't post sooner, but my whirlwind vacation in Ontario was pretty hectic. Here also are some links to papers and relevant video:

Unifying Tables, Objects and Documents
Programming with Rectangles, Triangles, and Circles
The essence of data access in Cω
Gavin Bierman (MSR) on Comega
Anders Hejlsberg on the future of C# and other things (it was question #17 that I played during the talk)
Anders Hejlsberg - Programming data in C# 3.0

Comparison of a simple select statement in C-omega vs. ADO.Net

rmbyers — Mon, 21 Mar 2005 04:30:00 GMT

In a couple weeks, I'm doing a talk at a .NET users group in Ontario about Cω. Cω is a cool research language from Microsoft Research that extends C# with direct support for data access (SQL and XML) and concurrency abstractions. I’ve been planning on writing a number of blog entries about Cω, but I haven’t yet been able to make the time. I was just working on the slides for my talk and thought I’d at least post one of my examples comparing simple database access with ADO.Net and Cω. One important concept in efficiently building reliable and maintainable software is “discover errors as early (and cheaply) as possible”. With that in mind, let’s look at some code (disclaimer: I’m by no means an ADO.Net expert).

Say we just want to get the names of all the employees who live in a specific city (in the sample Northwind database that comes with SQL Server). The simplest approach has a number of well-known problems:

SqlDataAdapter da = new SqlDataAdapter(

"SELECT * FROM Employees WHERE City='"+city+"'", nwindConn );

DataSet ds = new DataSet();

da.Fill(ds,"Employees");

foreach (DataRow dr in ds.Tables["Employees"].Rows)

{

string name = dr["LastName"].ToString();

int id = (int)dr["EmployeeID"];

Console.WriteLine( id + ": " + name);

}

The argument ‘city’ is subject to a SQL injection attack
The data is weakly typed, so we have to cast to get what we want, which could fail at run-time
Several string literals (table and column names) are used which could result in run-time errors
The query is stored as a string providing no opportunity for validation before run-time

We prevent the injection attack by using SqlParameters, and use strongly typed data sets to prevent the problems with string literals and casting:

SqlDataAdapter da = new SqlDataAdapter(

"SELECT * FROM Employees WHERE City= @city", nwindConn );

SqlParameter cityParam = da.SelectCommand.Parameters.Add("@city", SqlDbType.VarChar, 80);

cityParam.Value = city;

NorthwindDataSet ds = new NorthwindDataSet();

da.Fill(ds, ds.Employees.TableName );

foreach (NorthwindDataSet.EmployeesRow dr in ds.Employees.Rows)

{

string name = dr.LastName;

int id = dr.EmployeeID;

Console.WriteLine( id + ": " + name);

}

This is a lot better (although people are often too lazy to write the extra code to use SqlParameters). However, the SQL command is still stored as a string meaning we won’t find any errors in it until run-time, and the relationship between the shape of its output and the NorthwindDataSet is implicit and brittle. We might want to put the SQL into a stored procedure:

CREATE PROCEDURE EmployeesForCity

@City nvarchar(80)

SELECT EmployeeID, LastName FROM Employees WHERE City = @City

And then write our code without explicit knowledge of the structure of the underlying database:

SqlCommand cmd = new SqlCommand( "dbo.EmployeesForCity", nwindConn );

cmd.CommandType = CommandType.StoredProcedure;

SqlParameter cityParam = cmd.Parameters.Add("@city", SqlDbType.VarChar, 80);

cityParam.Value = city;

SqlDataAdapter da = new SqlDataAdapter( cmd );

NorthwindDataSet ds = new NorthwindDataSet();

da.Fill(ds, ds.EmployeesForCity.TableName );

foreach (NorthwindDataSet.EmployeesForCityRow dr in ds.EmployeesForCity.Rows)

{

string name = dr.LastName;

int id = dr.EmployeeID;

Console.WriteLine( id + ": " + name);

}

This is better; at least now our SQL statement can be checked before we run our app. However, we still have some literal strings in our code, and we’re still going to get a run-time error if the stored procedure changes in some way (I don’t see any easy way to rebuild the XSD from the database schema in VS.Net 2003). Plus, we’ve had to write significantly more code here. At best, it feels like our code has a weak connection to the underlying database. For small programs like this, it’s no big deal. But for large data-intensive programs, people complain a lot about spending a lot of the time messing with plumbing code.

Here’s the equivalent code in Cω:

rows = select * from DB.Employees where City == city;

foreach( row in rows )

{

string name = row.LastName.Value;

int id = row.EmployeeID.Value;

Console.WriteLine( id.ToString() + ": " + name);

}

Note the following:

There is no knowledge embedded in string literals
The select statement is strongly typed, meaning that we know exactly at compile-time what the shape of the data will look like (intellisense even shows us), and what valid statements are.
We can embed local variables like “city” directly into our sql without worrying about injection attacks.
We can rely on type inference so we don’t have to explicitly specify the type of “rows” and “row”, even though it is strongly typed.
We can configure our build such that we’ll know at compile time if the database has changed in any way that would make the code fail. Of course the database could still be changed underneath our program resulting in a run-time error if we don’t recompile.

And if you are willing to abandon SQL syntax all together, you could equivalently write:

DB.Employees [City==city].{

Console.WriteLine( it.EmployeeID + ": " + it.LastName );

};

How do you like that for concisely expressing what you mean?

[Update: Added entry with C# 3.0 / DLinq syntax for this comparison]

Generic type parameter variance in the CLR

rmbyers — Thu, 17 Feb 2005 10:00:00 GMT

When people start using C# generics for the first time, they are sometimes surprised that they can’t convert between related generic instances. For example, since you can convert a string to an object, shouldn’t you also be able to convert a List<string> to a List<object>? After all, you can convert a string[] to an object[], why should List<T> be any different?

More formally, in C# v2.0 if T is a subtype of U, then T[] is a subtype of U[], but G<T> is not a subtype of G<U> (where G is any generic type). In type-theory terminology, we describe this behavior by saying that C# array types are “covariant” and generic types are “invariant”.

There is actually a reason why you might consider generic type invariance to be a good thing. Consider the following code:

List<string> ls = new List<string>();

ls.Add("test");

List<object> lo = ls; // Can't do this in C#

object o1 = lo[0]; // ok – converting string to object

lo[0] = new object(); // ERROR – can’t convert object to string

If this were allowed, the last line would have to result in a run-time type-check (to preserve type safety), which could throw an exception (eg. InvalidCastException). This wouldn’t be the end of the world, but it would be unfortunate. Part of the benefit of generic types is that it helps to avoid the run-time checking and error handling overhead involved in converting back and forth from a base type (object in C#). It simplifies things greatly (both from a developer and run-time perspective) to be able to say that a variable of type List<object> can for sure hold any object.

So what about arrays, don’t we have the exact problem there? Yep. This was a hotly debated aspect to C#/CLR design. Java also has covariant array types, and whether or not this is a language flaw has been debated for ages. Jim Miller has an interesting annotation in his CLI book (pg 59) where he says "The decision to support covariant arrays was primarily to allow Java to run on the VES. The covariant design is not thought to be the best design in general, but it was chosen in the interest of broad reach". [Update: I've heard that Bill Joy, one of the original Java designers, has since said that he tried to remove array covariance in 1995 but wasn't able to do it in time, and has regretted having it in Java ever since]

Let’s get back to our original question. What if you really did want covariant generic types in a .NET language? Does the CLR really prevent that? Well, you may recall that I alluded to the fact that Eiffel makes use of additional features in the CLR generic support, that C# and VB do not. In Eiffel, generic types are always covariant (and so have the same sort of run-time checks as arrays). If you were to look at the draft v2 ECMA specification, you would see the following:

In addition, CLI supports covariant and contravariant generic parameters, with the following characteristics:

· It is type-safe (based on purely static checking)

· Simplicity: in particular, variance is only permitted on generic interfaces and generic delegates (not classes or value-types)

· Languages not wishing to support variance can ignore the feature, and treat all generic types as non-variant.

· Enable implementation of more complex covariance scheme as used in some languages, e.g. Eiffel.

Contravariance is the opposite of covariance – it means that the subtype relationship of the generic type varies inversely with the relationship of the type parameter. Or formally, G<T> is a subtype of G<U> if and only if U is a subtype of T. This is a pretty cool approach in my opinion. It allows a compiler to mark generic type parameters on an interface as being nonvariant, covariant or contravariant, and then enforces that those types are used only in places where they are safe (i.e. don’t require a run-time type check). Specifically, it’s always safe to use a covariant type as an output (return type or out parameter), and a contravariant type as an input. In IL, covariant type parameters are indicated by a ‘+’, and contravariant type parameters are indicated by a ‘-‘ (non-variant type parameters are the default, and can be used anywhere). Consider this simple example (using a theoretical extension of C#) from the draft ECMA spec [Update: switched to the newly announced C# 4.0 syntax]:

// Covariant parameters can be used as result types

interface IEnumerator<out T> {

T Current { get; }

bool MoveNext();

}

// Covariant parameters can be used in covariant result types

interface IEnumerable<out T> {

IEnumerator<T> GetEnumerator();

}

// Contravariant parameters can be used as argument types

interface IComparer<in T> {

bool Compare(T x, T y);

}

This would mean we could write code like the following:

IEnumerable<string> stringCollection = ...;

IEnumerable<object> objectCollection = stringCollection;

foreach( object o in objectCollection ) { ... }

IComparer<object> objectComparer = ...;

IComparer<string> stringComparer = objectComparer;

bool b = stringComparer.Compare( "x", "y" );

But we couldn’t do the opposite – try and convert an IEnumerable<object> to an IEnumerable<string>, or try and convert an IComparer<string> to an IComparer<object>. In those cases, we’d get a compile time error telling us such a conversion wouldn’t be type-safe. Of course, languages (like Eiffel) are free to build their own run-time type checking on top of the strict CLR support to relax the rules where they see fit.

In my opinion this sort of language feature would occasionally be very useful (although the real scenarios would, of course, be much more involved than the above simplistic examples). There have been a couple of times when writing real-world C# code when I knew I could write simpler code if the CLR generic variance support was exposed in C#. Generic variance support in C# has been discussed on the MSDN product feedback center, and I don’t think the C# team has ruled it out for a future addition to the language (but of course I don’t know their plans any better than you do). Although I think this is a cool feature, I don’t think I’m going to start writing any applications in IL just so I can take advantage of it. However, it would probably be a fun project to write or extend and existing free C# compiler (like Mike Stall’s Blue) with support for generic variance. If you're interested in more details (including a thorough description of assignment compatibility in the face of generic variance) see the draft v2 ECMA specification.

[Update: Added details about covariant arrays from Jim Miller's CLI book, and pointer to the draft v2 ECMA spec above.]

[Update: See my post "More on generic variance" for more details and discussion]

[Update: Here's an MSDN article that discusses this as well: http://msdn2.microsoft.com/en-us/library/ms228359(vs.80).aspx]

[Update: Anders has finally announced that C# 4.0 and the BCL will support generic variance with "in" and "out" keywords. This has been in the works for awhile, and of course I'm super excited about it.]

"Hello world" quiz answers

rmbyers — Wed, 09 Feb 2005 07:07:00 GMT

Matthew Cosier was the first person to post correct answers to all my Hello, World quiz questions, good work Matthew! Here are the answers with some details and links:

System.Console.WriteLine(“Hello World!”)
Visual Basic .NET - note the lack of a semi-colon <grin>
write('Hello world'),nl.
Prolog - an older (but cool) logic programming language. Prolog makes it incredibly easy to solve certain classes of logic problems (especially constraint satisfaction problems). Using a domain-specific language like Prolog for a problem it is good at really makes you recognize the value of being a multi-lingual developer. I’ve heard of one project (scheduling) that took a week or so to build in C++, and a few hours to re-implement in Prolog. Unfortunately, I’m not aware of any commercial-grade .NET implementation, but there is an academic Prolog to C# translator called P#.
std::cout << “Hello World!” << std::endl;
C++, where Microsoft Visual C++ with managed extensions is of course the .NET implementation.
." Hello, world!" CR
Forth – an older (but not cool) low-level programming language. I was forced to write an assignment in Forth in University – an experienced that helped me to truly appreciate the benefits if higher-level languages <grin>. Detla Forth .NET is a shareware .NET implementation.
Ada.Text_IO.Put_Line ("Hello, world!");
Ada (obviously) – an interesting language which was engineered in the 70s from the ground up (by the US department of defence) for building highly-reliable embedded and real-time systems. Ada still has a strong following, and I think today’s mainstream languages could benefit a lot from some of the ideas (strong type safety, built-in concurrency support, etc.), championed by Ada. A# is an academic project to run Ada code on the CLR – although (despite my pride in our product), I wouldn’t suggest you go moving your nuclear missile silo control applications over to running on the CLR <grin>.
ldstr "Hello World!" call void [mscorlib]System.Console::WriteLine(string)
Common Intermediate Language (CIL/MSIL) – the input to the CLR, and yes IlAsm.exe doesn’t seem to care whether I put a newline in there.
io.put_string("Hello, world!%N")
Eiffel – a modern object-oriented language with an emphasis on reliability through the “design by contract” development philosophy. In my opinion, there are some very important ideas here (I’m reading Bertrand Meyer’s classic book “Object Oriented Software Construction” now). Eiffel is one of the CLR’s most important language partners (some aspects of Generics in Whidbey were added largely for the sake of Eiffel).
System.Console.WriteLine(“Hello World!”);
C# (or J# etc.)
string* s = {yield return "Hello"; yield return "World!";}; s.{Console.WriteLine(it);};
Cω (Comega) – An extension to C# from Microsoft Research (based on Xen, X# and polyphonic C#). Cω adds powerful support for data access (including integrating SQL and XML) and interesting concurrency abstractions. Expect to see a number of future blog entries here about Cω.
(display "Hello, world!") (newline)
Scheme – an older but very simple and elegant functional language. Every computer-science student should have to write some programs in Scheme (or other variant of Lisp) and read the classic textbook “The Structure and Interpretation of Computer Programs” (sadly, I was permitted to graduate without ever reading this elegant text, but I’ve recently corrected that error in my education). The Hotdog scheme compiler can target .NET.

If you’ve read this far, then you might be interested in this catalog of “Hello World” programs. Sadly, I didn’t find it until after I had written most of my quiz <grin>. I also like this Programming Languages Timeline, and here is a list of .NET-based language implementations. Anyone else have any good links like this?

Disruptive Programming Language Technologies

rmbyers — Mon, 07 Feb 2005 04:24:00 GMT

"Disruptive Programming Language Technologies" (video, slides) is one of my favorite talks on the future of programming languages. The talk is by Todd Proebsting, who is a senior researcher at the Microsoft Center for Software Excellence (formerly the Programmer Productivity Research Center). His point applies particularly well to the success of managed environments like .NET. Managed (type-safe, garbage collected) environments have existed in the same basic form for decades, but due to their disadvantages (eg. performance) the mainstream industry largely ignored them. Eventually the industry viewpoint shifted radically, and relatively quickly, to embrace this technology. I find it amusing looking back at my own viewpoint on this subject. I’m ashamed to say that it wasn't that long ago that I thought managed languages like Java and VB were "toy" languages that no real serious programmer would choose to use. Over the past 5 years, my attitude changed radically, even though the technology really didn’t. I've committed myself to not being so narrow-minded about whatever the next big changes are <grin>.

Here's is Todd's summary of this talk:

For the past few decades, programming language design and implementation research has concentrated heavily in a few notable areas: type theory, functional programming, object-oriented programming, and, of course, optimization techniques. Yet the recent commercially successful languages (e.g., Perl, Python, Visual Basic, Java) are not particularly interesting when judged in these domains. What happened? Each represented a "disruptive technology" that allowed it to capture programmer mindshare while everybody else was looking. In this talk, I will present what I think makes a programming language technology disruptive, and I will propose possible future disruptive programming language technologies.

Hello, World!

rmbyers — Mon, 07 Feb 2005 03:10:00 GMT

Hi, my name is Rick Byers. I’m a developer on the Common Language Runtime (CLR) team at Microsoft. I work on the debugger services team where our main deliverable is the ICorDebug API which debuggers like Visual Studio use to debug managed code.

Mike Stall already has an awesome ICorDebug blog, so I’m not about to try to compete with him here <grin>. Instead, I plan on using this space to blab about whatever ideas from the software industry I happen to find exciting at the moment. I’ll also try and share some information about the CLR that you might find interesting.

Mainly, I’m excited about improvements to programmer productivity in the software industry. Building large and complex pieces of software which are also correct, reliable, secure, robust, maintainable and understandable is incredibly difficult. In my opinion, figuring out how to do it better (“engineering” anyone?) is incredibly important. I don’t pretend to have any great insights here, but I do try and read and learn what I can in this area, so I’ll often pass on tidbits of information and research which I think are exciting and important. My favourite areas are the design of programming languages and developer tools.

Of course, the CLR (and managed type-safe environments in general) is all about improving developer productivity (which is primarily why I joined this team). The CLR also makes it easier to experiment with and deploy new programming languages. On that note, let’s get things started off with a little quiz. Which language is each of the following code snippets written in? Here’s a hint, there are .NET implementations for all of them.

System.Console.WriteLine(“Hello World!”)
write('Hello world'),nl.
std::cout << “Hello World!” << std::endl;
." Hello, world!" CR
Ada.Text_IO.Put_Line ("Hello, world!");
ldstr "Hello World!" call void [mscorlib]System.Console::WriteLine(string)
io.put_string("Hello, world!%N")
System.Console.WriteLine(“Hello World!”);
string* s = {yield return "Hello"; yield return "World!";}; s.{Console.WriteLine(it);};
(display "Hello, world!") (newline)