Krzysztof Cwalina : System.Collections

Simulated Covariance for .NET Generics

kcwalina — Wed, 02 Apr 2008 18:00:00 GMT

I just wrote this pattern, but I am not sure if I should add it officially to the Framework Design Guidelines. It seems like a bit of a corner case scenario, though I do get questions about it from time to time. Anyway, let me know what you think.

Different constructed types don’t have a common root type. For example, there would not be a common representation of IEnumerable<string> and IEnumerable<object> if not for a pattern implemented by IEnumerable<T> called Simulated Covariance. This post describes the details of the pattern.

Generics is a very powerful type system feature added to the .NET Framework 2.0. It allows creation of so called parameterized types. For example, List<T> is such a type and it represents a list of objects of type T. The T is specified at the time when the instance of the list is created.

List<string> names = new List<string>();

names.Add(“John Smith”);

names.Add(“Mary Johnson”);

Such Generic data structures have many benefits over their non-Generic counterparts. But they also have some, sometimes surprising, limitations. For example, some users expect that a List<string> can be cast to List<object>, just as a String can be cast to Object. But unfortunately, the following code won’t even compile.

List<string> names = new List<string>();

List<object> objects = names; // this won’t compile

There is a very good reason for this limitation, and that is to allow for full strong typing. For example, if you could cast List<string> to a List<object> the following incorrect code would compile, but the program would fail at runtime.

static void Main(){

List<string> names = new List<string>();

// this of course does not compile, but if it did

// the whole program would compile, but would be incorrect as it

// attempts to add arbitrary objects to a list of strings.

AddObjects((List<object>)names);

string name = names[0]; // how could this work?

}

// this would (and does) compile just fine.

static void AddObjects(List<object> list){

list.Add(new object()); // it’s a list of strings, really. Should we throw?

list.Add(new Button());

}

Unfortunately this limitation can also be undesired in some scenarios. For example, there is nothing wrong with casting a List<string> to IEnumerable<object>, like in the following example.

List<string> names = new List<string>();

IEnumerable<object> objects = names; // this won’t compile

foreach(object obj in objects){

Console.WriteLine(obj.ToString());

}

In general, having a way to represent “any list” (or in general “any instance of this generic type”) is very useful.

// what type should ??? be?

static void PrintItems(??? anyList){

foreach(object obj in anyList){

Console.WriteLine(obj.ToString());

}

Unfortunately, unless List<T> implemented a pattern that will be described in a moment, the only common representation of all List<T> instances would be System.Object. But System.Object is too limiting and would not allow PrintItems method to enumerate items in the list.

The reason that casting to IEnumerable<object> is just fine, but casting to List<object> can cause all sorts of problems is that in case of IEnumerable<object>, the object appears only in the output position (the return type of GetEnumerator is IEnumerator<object>). In case of List<object>, the object represents both output and input types. For example, object is the type of the input to the Add method.

// T does not appear as input to any members or dependencies of this interface

public interface IEnumerable<T> {

IEnumerator<T> GetEnumerator();

}

public interface IEnumerator<T> {

T Current { get; }

bool MoveNext();

}

// T does appear as input to members of List<T>

public class List<T> {

public void Add(T item); // T is an input here

public T this[int index]{

get;

set; // T is actually an input here

}

In other words, we say that in IEnumerable<T>, the T is at covariant positions (outputs). In List<T>, the T is at covariant and contravariant (inputs) positions.

To solve the problem of not having a common type representing the root of all constructions of a generic type, you can implement what’s called the Simulated Covariance Pattern.

Given a generic type (class or interface) and its dependencies

public class Foo<T> {

public T Property1 { get; }

public T Property2 { set; }

public T Property3 { get; set; }

public void Method1(T arg1);

public T Method2();

public T Method3(T arg);

public Type1<T> GetMethod1();

public Type2<T> GetMethod2();

}

public class Type1<T> {

public T Property { get; }

}

public class Type2<T> {

public T Property { get; set; }

}

Create a new interface (Root Type) with all members containing Ts at contravariant positions removed. In addition, feel free to remove all members that might not make sense in the context of the trimmed down type.

public interface IFoo<T> {

T Property1 { get; }

T Property3 { get; } // setter removed

T Method2();

Type1<T> GetMethod1();

IType2<T> GetMethod2(); // note that the return type changed

}

public interface IType2<T> {

T Property { get; } // setter removed

}

The generic type should then implement the interface explicitly and “add back” the strongly typed members (using T instead of object) to its public API surface.

public class Foo<T> : IFoo<object> {

public T Property1 { get; }

public T Property2 { set; }

public T Property3 { get; set;}

public void Method1(T arg1);

public T Method2();

public T Method3(T arg);

public Type1<T> GetMethod1();

public Type2<T> GetMethod2();

object IFoo<object>.Property1 { get; }

object IFoo<object>.Property3 { get; }

object IFoo<object>.Method2() { return null; }

Type1<object> IFoo<object>.GetMethod1();

IType2<object> IFoo<object>.GetMethod2();

}

public class Type2<T> : IType2<object> {

public T Property { get; set; }

object IType2<object>.Property { get; }

}

Now, all constructed instantiation of Foo<T> have a common root type IFoo<object>.

var foos = new List<IFoo<object>>();

foos.Add(new Foo<int>());

foos.Add(new Foo<string>());

…

foreach(IFoo<object> foo in foos){

Console.WriteLine(foo.Property1);

Console.WriteLine(foo.GetMethod2().Property);

}

þ CONSIDER using the Simulated Covariance Pattern, if there is a need to have a representation for all instantiations of a generic type.

The pattern should not be used frivolously as it results in additional types in the library and can makes the existing types more complex.

þ DO ensure that the implementation of the root’s members is equivalent to the implementation of the corresponding generic type members.

There should not be an observable difference between calling a member on the root type and calling the corresponding member on the generic type. In many cases the members of the root are implemented by calling members on the generic type.

public class Foo<T> : IFoo<object> {

public T Property3 { get { ... } set { ... } }

object IFoo<object>.Property3 { get { return Property3; } }

…

}

þ CONSIDER using an abstract class instead of an interface to represent the root.

This might sometimes be a better option as interfaces are more difficult to evolve (see section X). On the other hand there are some problem with using abstract classes for the root. Abstract class members cannot be implemented explicitly and the subtypes need to use the new modifier. This makes it tricky to implement the root’s members by delegating to the generic type members.

þ CONSIDER using a non-generic root type, if such type is already available.

For example, List<T> implements IEnumerable for the purpose of simulating covariance.

Synchronization at the Right Level

kcwalina — Fri, 05 May 2006 21:24:00 GMT

I get a lot of questions about why the new generic collection interfaces (IList<T> for example) do not support synchronization. There are several reasons for it. One, probably the main, is that in practice synchronizing on the level of a single data structure is rarely useful. Rico explains this in a series of posts on locking. In particular see the post on putting your synchronization at the correct level.

C5 Collections

kcwalina — Mon, 20 Feb 2006 21:58:00 GMT

Peter Sestoft and his team just released a library of collections for the CLI called C5 Collections. Unlike the Power Collections, this library introduces many new abstractions that are not present in the .NET Framework. The benefit of such approach is that it allows for some powerful scenarios that are not possible if you simply build on top of the existing abstractions (riches compile time checking). The drawback is that it might not integrate with code that uses the standard abstractions as seamlessly as the Power Collections.

These two libraries represent two distinct philosophies of extending the Framework/CLI. They make different tradeoffs between the degree of innovation and interoperability. If you have used any of these libraries, I would appreciate you comments on how these philosophies work in practice and in general you observations on using 3rd party libraries. Do they integrate well with the Framework? How are they in terms of usability? Do they provide support for main functionality you are/were missing in the Framework?

Disposable Collection

kcwalina — Sat, 11 Feb 2006 00:38:00 GMT

Somebody asked me today, if we have a collection that is disposable and calls dispose on all its items when the collections.Dispose method is called. You can implement such collection by inheriting from List<T> and adding Dispose logic. If you add one additional requirement that Dispose is called on the items before they are removed from the collection when the collection is cleared, you need to use Collection<T> which can be customized more than List<T>. Here is a quick custom collection that does what I am talking about.

public class DisposableCollection<T> : Collection<T>, IDisposable where T: IDisposable {
   protected virtual void Dispose(bool disposing){
      if(disposing){
         foreach(T item in this){
          item.Dispose();
         }
      }
   }
   public void Dispose(){
      Dispose(true);
   }

   protected override void ClearItems(){
      this.Dispose();
      base.ClearItems();
   }
}

There are two problems with this collection though: a) what happens when an item is removed from the collection by calling Remove. b) what should the user do if they hold a reference to an item residing in the collection and then call Clear, which causes the item pointed to by the reference to be disposed:

var col = DisposableCollection<Foo>();
col.Add(new Foo());
Foo f = col[0];
col.Clear();
// now f points to a disposed object

… maybe Clear should not be disposing the items. What do you think?

Immutable Collections

kcwalina — Tue, 22 Nov 2005 21:28:00 GMT

I just saw an interesting post that talks about a general pattern of immutable reference types. Good read. I highly recommend it!

At the end of the post the author suggests that we add “value object” generic collection to the BCL. It’s an interesting suggestion, but unfortunately not as easy as it may seem. The difficulty lays in the fact that the items in such collection also need to be immutable. String is an immutable collection of characters, which are immutable themselves.

One way to design this would be to add IImmutable marker interface, implement it on immutable types, and constrain the proposed ReadOnlyList<TImmutable> to TImmutable : IImuttable. Now, I am not sure the value of doing this outweighs the added complexity, but we will take a closer look.

Why we don’t recommend using List in public APIs

kcwalina — Mon, 26 Sep 2005 19:35:00 GMT

We don’t recommend using List<T> in public APIs for two reasons.

List<T> is not designed to be extended. i.e. you cannot override any members. This for example means that an object returning List<T> from a property won’t be able to get notified when the collection is modified. Collection<T> lets you overrides SetItem protected member to get “notified” when a new items is added or an existing item is changed.
List<T> has lots of members that are not relevant in many scenarios. We say that List<T> is too “busy” for public object models. Imagine ListView.Items property returning List<T> with all its richness. Now, look at the actual ListView.Items return type; it’s way simpler and similar to Collection<T> or ReadOnlyCollection<T>.

System.Collections vs. System.Collection.Generic and System.Collections.ObjectModel

kcwalina — Sat, 24 Sep 2005 03:10:00 GMT

Many people ask how the new.NET Framework 2.0 generic collections relate to the non-generic collections we shipped before. So, here you go. The most important types are in bold font.

List<T> is basically a better ArrayList. It is optimized for speed, size, and power. Use it for majority of internal implementations whenever you need to store items in a container. Do not use it in public APIs.

Dictionary<TKey,TValue> is just a strongly typed Hashtable. Some changes were made to the hashing algorithm. The details are described here.

Collection<T> is a much better CollectionBase. Use it to expose read/write collection output from public APIs. It’s in System.Collections.ObjectModel namespace. Why in a separate namespace and why such strange name? See here.

ReadOnlyCollection<T> is a much better ReadOnlyCollectionBase. It’s in System.Collections.ObjectModel namespace.

Queue<T> and Stack<T> are equivalent to Queue and Stack.

SortedList<TKey,TValue> is basically a generic version of SortedList.

SortedDictionary<TKey,TValue> does not have a corresponding non-generic collection. It’s similar to SortedList and SortedList<TKey,TValue>, but it’s implemented as a balanced tree (red-black tree). Insertions are on average much faster, but some lookups are slightly slower and memory consumption is larger.

IEnumerable<T> is just like IEnumerable, but strongly typed. One difference is that IEnumerable<T> extends IDisposable. The reasons are described here.

ICollection<T> seems like ICollection, but it’s actually a very different abstraction. We found that ICollection was not very useful. At the same time, we did not have an abstraction that represented an read/write non-indexed collection. ICollection<T> is such abstraction and you could say that ICollection does not have an exact corresponding peer in the generic world; IEnumerable<T> is the closest.

IList<T> is just strongly typed IList. We removed the notion of IsFixedSize and the reasons are described here.

IDictionary<TKey,TValue> is roughly equivalent to IDictionary.

DictionaryBase does not have a corresponding generic type. Simply implement IDictionary<TKey,TValue> if you need a custom dictionary.

Also, we added KeyedCollection<TKey,TItem>. It’s a new collection which allows items to be indexed by both a key and an index. Use it to expose collections of items that have natural “names” (keys) from public APIs. You need to inherit from the collection to use it.

And finally many people asked for linked-list. LinkedList<T> was added to .NET Framework 2.0.

So here is the summary:

Non-Generic                        Similar Generic Type
ArrayList              List<T>
Hashtable              Dictionary<TKey,TValue>
SortedList             SortedList<TKey,TValue>
Queue                  Queue<T>
Stack                  Stack<T>
IEnumerable            IEnumerable<T>
ICollection            N/A (use IEnumerable<T> anything that extends it)
N/A                    ICollection<T>
IList                  IList<T>
CollectionBase         Collection<T>
ReadOnlyCollectionBase ReadOnlyCollection<T>
DictionaryBase         N/A (just implement IDictionary<TKey,TValue>
N/A                    SortedDictionary<TKey,TValue>
N/A                    KeyedCollection<TKey,TItem>
N/A                    LinkedList<T>

Let me know if you would like me to blog more information about any of these types.

List vs. ArrayList Performance

kcwalina — Fri, 26 Aug 2005 00:25:00 GMT

Rico just posted a new performance quiz where he compares performance of List<T> and ArrayList in some simple scenarios. List<T> is my Whidbey baby, so of course I had to answer that List<T> is faster :-)

PowerCollections Beta Avaliable

kcwalina — Sat, 04 Jun 2005 02:04:00 GMT

The beta release of the PowerCollections library has been posted today. See http://wintellect.com/WEBLOGS/pgolde/archive/2005/06/03/1304.aspx.

Generic interfaces, IsReadOnly, IsFixedSize, and array

kcwalina — Wed, 18 May 2005 17:32:00 GMT

Peter Golde recently posted about the IsReadOnly and IsFixedSize change we made to the generic collection interfaces.There are also several other discussions around the net on this topic. I thought I write up something to explain the motivation and our thinking on this.

We decided that there are really not that many fixed size data structures that are not read only. In fact, we only have one – the array. We think that having a property on a very commonly implemented interface that is useful in only one case is not the right thing to do. So we decided to remove IsFixedSize. If we see a need for this in the future, we will add an interface called something like IIndexerSettable and implement it on fixed size structures.

We also decided that arrays will return true from IsReadOnly, but will actually allow setting values using the indexer (which we are trying to fix in RTM as it does not work today). In other words, the IsFixedSize and IsReadOnly properties are now changed to IsReadOnly and “is Array”, like in the following example:
void EitherAddOrSet(IList<int> list, int newValue){
if(!list.IsReadOnly) list.Add(newValue);
else if(list is int[]) list[0] = newValue;
}

… oh, the IIndexerSettable interface mentioned above would work the following (and would make the EitherAddOrSet routine more general):

void EitherAddOrSet(IList<int> list, int newValue){
if(!list.IsReadOnly) list.Add(newValue);
else if(list is IIndexerSettable) list[0] = newValue;
}

The reason why Collection, ReadOnlyCollection, and KeyedCollection were moved to System.Collections.ObjectModel namespace

kcwalina — Tue, 15 Mar 2005 21:55:00 GMT

Several people asked me why Collection<T>, ReadOnlyCollection<T>, and KeyedCollection<TKey,TValue> were moved to System.Collections.ObjectModel namespace. Here are the two main reasons:

1. Microsoft.VisualBasic namespace contains a non-generic type called Collection. The namespace is imported by default by the VB project template and we were afraid that it would confuse developers who import System.Collections.Generic namespace as well and would see two types called ‘Collection’ in their intellisense (given one of them generic, but still). We were faced with either renaming Collection<T> to something else or moving it to another namespace. We really did not want to rename it because Collection<T> is just the best name for the type (with not many good alternatives) and it will show in many APIs.

2. We also thought that not many people will need to declare variables of thses types. Most developer who will directly use (instantiate) the collections are framework (object model) designers. Developers writing applications (majority) would simply call Directory.Files.Count for example (and not even realize it’s a Collection<FilerInfo>) or they will/should use List<T> instead. Having said that, recently I started to be concerned that some people will have to declare Collection<X> variables to cache collection instances they get from APIs, but I have not received any substantial feedback on this. BTW, this will be mitigated by many APIs actually returning subclasses of Collection<X> instead of the base class.

Some people also commented that the name “ObjectModel” is a bit “kooky” (direct quote). I have received some feedback (not much but some) about it from customers, but I think the main issue is that the term is so overloaded and many people mean many different things when they say “object model”. Anyway, we would probably try to come up with some less controversial name if we were in the design phase today, but it’s kind of late in the product cycle now to make such changes.

The reason why IEnumerator extens IDisposable

kcwalina — Sat, 08 Jan 2005 02:35:00 GMT

Lot’s of people asked me why IEnumerator<T> extends IDisposable. We did this to support some obscure, yet important scenarios where an enumerator enumerates database rows, or files in a directory, etc. In such cases, the enumerator usually opens some connection or a handle, which then needs to be closed at the end of the enumeration.

Now, we could solve the problem in two ways.

1) We could do what we did. Now, foreach loop can simply call Dispose when the loop terminates.

2) We could implement foreach loop to do a dynamic cast to IDisposable at the end of the loop and if the enumerator happened to implement IDisposable, we would call it.

We chose #1 because it’s slightly faster and the cost of implementing an empty Dispose method, for those who don’t need it, is relatively low.

… besides, have you heard about the new C# yield statement?

Power Collections on GotDotNet Featured Sites

kcwalina — Thu, 04 Nov 2004 00:46:00 GMT

I just noticed that the Power Collections project made it to the “Featured Sites” on GotDotNet. See http://www.gotdotnet.com/content/featuredsite/powercollections/default.aspx

For those who don’t know, Power Collections is a community open source library of advanced collection classes and algorithms. The project is co-sponsored by Microsoft. It has generated lots of good APIs (you can download a working library here) and lots of feedback for the BCL team on how to we should be designing our collections.

I would encourage anybody with a passion for collection libraries to get involved. Power Collections will influence the .NET Framework collections and so it would be great if Power Collections were influenced by the broader community.

You can find the specs of the APIs at http://www.wintellect.com/powercollections/spec.aspx

… and provide feedback at the forum http://www.wintellect.com/forum/main.asp?CAT_ID=7

IComaprer & IComparable Refactoring Proposal

kcwalina — Thu, 28 Oct 2004 04:01:00 GMT

We are exploring the possiblility of changing the design of IComparer<T> and IComparable<T> interfaces that will ship with Whidbey. This post describes some more detail on the issues we are trying to address with the design change and we are looking for feedback on the proposal.

Background

The first version of the .NET Framework had three main comparison related-interfaces: IComparable, IComparer, and IHashCodeProvider. One of the main issues with the interfaces was that there was no single interface that could be used by Hashtable to override the default identity of the keys (change which keys compare to equal and which not). You needed to use IHashCodeProvider to provide custom hash codes for the keys and IComparer to provide custom equality (via Compare==0). This resulted in a common usage mistake (a bug) where a user would pass incompatible pair of IHashCodeProvider and IComparer to the Hashtable constructor. For example, a case sensitive hash code provider and a case insensitive comparer. This would result in a bug where two strings that do compare equal could have different hash codes, and so the Hashtable indexer could not find the item with the given key.

We tried to fix this problem in Whidbey generic APIa and decided to put all comparison related methods from IComarer and IHashCodeProvider into a single interface (IComparer<T>), so users could not pass two incompatible implementations of the interfaces to the Hashtable constructor. We have always seen the new design as a compromise between a very clean OO design and trying to avoid explosion of the number of interfaces.

Well, we have received feedback from people using the new interfaces (IComparer<T> and IComparable<T>) that made us thinking that may be we compromised a bit too much J Here are the relevant links describing the issues:

Peter Golde (who creates power collection library) gave as his feedback here: http://wintellect.com/WEBLOGS/pgolde/archive/2004/07/06/180.aspx.
There is also a MSDN feedback thread on this at http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=3c698098-9290-4443-a924-97e173a6e37d

We decided to give it one more shot to fix the concerns. You can find the proposal for the new design below. I think the change proposal addresses most of the concerns, but at the same time it has the following drawbacks:

More interfaces to understand. Makes the Framework less approachable.
Custom comparers can implement one interface (let’s say IEqualityProvider<T>) but not the other (IComparer<T>). Switching from one collection which uses IEqualityProvider<T> to another which happens to use IComparer<T> may be difficult (as you will need a different custom comparer).

I am looking for feedback on the proposal from people who have experimented with the interfaces. The change is something that we would do only if there is an overwhelming positive feedback from the community, not if the improvement seems only marginal.

Thanks!

Current Design

public interface IComparable<T> {

bool Equals(T other);

int CompareTo(T other);

}

public interface IComparer<T> {

bool Equals(T x, T y);

int Compare(T x, T y);

int GetHashCode(T obj);

}

public interface IKeyComparer : IComparer, IHashCodeProvider {

bool Equals(object x, object y);

}

Proposed New Design

IComparable<T>

· Break into two interfaces

public interface IEquateable<T> {

bool Equals(T other);

}

public interface IComparable<T> {

int CompareTo(T other);

}

IComparer<T>

· Break into two interfaces

public interface IEqualityProvider<T> {

bool Equals(T x, T y);

int GetHashCode(T obj);

}

public interface IComparer<T> {

int Compare(T x, T y);

}

IKeyComparer

· Rename from IKeyComparer to IHashComparer

· Remove IComparer requirement (base type)

public interface IEqualityProvider : IHashCodeProvider {

bool Equals(object x, object y);

}

System.Collections.Generic Dictionary Capacity, Hashing, and Collisions.

kcwalina — Sat, 07 Aug 2004 03:59:00 GMT

Somebody asked me a question how to set the initial capacity of System.Collections.Generic.Dictionary<K,V> if one knows that the Dictionary will contain 1000 entries.

The short answer

Dictionary<int,string> numbers = new Dictionary<int,string>(1000);

The long answer

Dictionary<K,V> is a generic type almost identical to the non-generic System.Collections.Hashtable. Both types are implemented as hashtables, but they are slightly different implementations. In particular, they use different algorithms to deal with collisions. Hashtable uses probing and Dictionary<K,V> uses chaining.

Probing means that when a collision is detected, the algorithm tries to store the new entry in the next free bucket. When a hashtable gets close to full (saturated), finding the next free bucket becomes difficult. For this reason, hashtables that use probing grow their size way before they get completely saturated to minimize collisions. In our Hashtable, the threshold of acceptable saturation is controlled by the loadFactor parameter the constructor, but this is a topic for another time. Hashtables using probing always have more buckets allocated than the number of entries they store.

Now, the interesting question is whether the capacity specifies the number of buckets we allocate or the algorithm is actually smarter. The good news is that we take the loadFactor into account when allocating the buckets. So, if you specify capacity of let’s say 1000, we actually allocate more buckets. In fact we allocate enough buckets to ensure that we don’t have to grow the number of buckets before more entries are added than the specified capacity.

Chaining means that all entries with the same hash code are stored in a list associated with one bucket corresponding to the hash code. Therefore the Dictionary<K,V> can be completely saturated, we don’t use loadFactor, and we simply take the next prime number after the capacity to determine how many buckets to allocate.