Why is covariance of value-typed arrays inconsistent?

Why is covariance of value-typed arrays inconsistent?

Rate This
  • Comments 22

Another interesting question from StackOverflow:

uint[] foo = new uint[10];
object bar = foo;
Console.WriteLine("{0} {1} {2} {3}",       
  foo is uint[], // True
  foo is int[],  // False
  bar is uint[], // True
  bar is int[]); // True

What the heck is going on here?

This program fragment illustrates an interesting and unfortunate inconsistency between the CLI type system and the C# type system.

The CLI has the concept of "assignment compatibility". If a value x of known data type S is "assignment compatible" with a particular storage location y of known data type T, then you can store x in y. If not, then doing so is not verifiable code and the verifier will disallow it.

The CLI type system says, for instance, that subtypes of reference type are assignment compatible with supertypes of reference type. If you have a string, you can store it in a variable of type object, because both are reference types and string is a subtype of object. But the opposite is not true; supertypes are not assignment compatible with subtypes. You can't stick something only known to be object into a variable of type string without first casting it.

Basically "assignment compatible" means "it makes sense to stick these exact bits into this variable". The assignment from source value to target variable has to be "representation preserving".

One of the rules of the CLI is "if X is assignment compatible with Y then X[] is assignment compatible with Y[]".

That is, arrays are covariant with respect to assignment compatibility. As I've discussed already, this is actually a broken kind of covariance.

That is not a rule of C#. C#'s array covariance rule is "if X is a reference type implicitly convertible to reference type Y (via a reference or identity conversion) then X[] is implicitly convertible to Y[]". That is a subtly different rule!

In the CLI, uint and int are assignment compatible; therefore uint[] and int[] are too. But in C#, the conversion between int and uint is explicit, not implicit, and these are value types, not reference types. So in C# it is not legal to convert an int[] to a uint[]. But it is legal in the CLI. So now we are faced with a choice.

1) Implement "is" so that when the compiler cannot determine the answer statically, it actually calls a method which checks all the C# rules for identity-preserving convertibility. This is slow, and 99.9% of the time matches what the CLR rules are. But we take the performance hit so as to be 100% compliant with the rules of C#.

2) Implement "is" so that when the compiler cannot determine the answer statically, it does the incredibly fast CLR assignment compatibility check, and live with the fact that this says that a uint[] is an int[], even though that would not actually be legal in C#.

We chose the latter. It is unfortunate that C# and the CLI specifications disagree on this minor point but we are willing to live with the inconsistency.

So what's going on here is that in the "foo" cases, the compiler can determine statically what the answer is going to be according to the rules of C#, and generates code to produce "True" and "False". But in the "bar" case, the compiler no longer knows what exact type is in bar, so it generates code to make the CLR answer the question, and the CLR gives a different opinion.

 

  • Pavel,

    See the following link for an example of what happens when you assume that a valid use case does not exist just because you can't think of one:

    https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=361539

    The CLR team knew about this breaking change before the release of .NET 3.5 SP1, but decided that the chances that it would affect anyone were so small that they didn't have to worry about it (that is not an assumption; I heard it directly from a PM on the CLR team). What actually happened is that my entire organization had to delay installing SP1 for a year until we could dedicate time to rewriting a portion of an internal application that was broken by the change. That also meant that we had to delay installing other applications which required SP1. There were other reports in the bug of commercial software companies whose applications were broken when their customers installed SP1, which they of course could not control (unfortunately those reports were wiped in the recent Connect upgrade).

    The bottom line is that when you expose functionality, especially on the scale that the .NET Framework operates, you have to assume that someone is going to use it. Choosing to expose this kind of inconsistency rather than accept (what I imagine, without any hard evidence, was) an incredibly small performance impact has almost certainly caused some developers time, money, and bugs, along with the resulting loss of boss/colleague/customer/user satisfaction. This is especially true since, like the breaking change I referenced above, the inconsistency is apparently not documented anywhere.

  • @David, I know very well about that particular issue (I upvoted it immediately when it first appeared, and all comments posted under the alias "int19h" there are mine). The problem with it, however, is that it is a _breaking change_ - as in, it quietly changes the behavior of code that was legal on the same implementation (and according to specification) before. It was made more restrictive than it was, which is why it is such a big deal.

    In addition, it actually affects some very valid use cases for generics. I'm very surprised that someone in MS actually assessed the change as "not significant", because to me it clearly is, even before a customer comes in with a concrete example of broken code.

    In contrast, array variance has been working the same way on all .NET releases, starting from the very first one. Furthermore, it is more permissive than what the spec allows, rather than more restritive. In practice, this means that it's only a breaking change if a code somewhere relies on either the result of an "is" or "as" check being null, or on a cast throwing InvalidCastException, and not being able to correctly handle an array of different-signedness values received. I simply cannot think of any case where it would matter. Can you?

    In the end, the fact that there's no bug report for this, even though the inconsistency (or the language extension, whichever is your take on it) has been there for 7 years now, seem to prove the point better than anything else.

  • @Pavel,

    I knew that you are int19h. I am not sure if you were aware that I am CommonGenius, and I reported the bug initially :)

    A "feature" which does not match expectations AND is undocumented has the same affect as a breaking change. You write code that looks like it should work, but it doesn't. Especially on a corner case like this, unless you have exceptionally good test coverage, there is a very good chance that you won't catch it during development. Then one day something unusual happens and your production application starts crashing. And good luck finding the problem, since everything you can see in the code looks correct.

    Yes, the fact that it is a corner case makes it less likely to show up as a real world problem. But it also means that, when it does show up, it will be vastly more difficult to find and fix. Especially since it is apparently not documented anywhere (except this blog post and the stackoverflow question).

    I cannot think of a specific case where this bug (and I do consider it a bug, Eric's explanation notwithstanding) would cause problems for me, mostly because I rarely use unsigned types. But absence of proof is not proof of absence. Despite having significant programming experience and a very strong knowledge of .NET, I am certain that I can only imagine a very small percentage of the possible use cases of the platform. Even the .NET team with all of their combined knowledge and experience cannot possibly anticipate every possible use case. That is precisely why general purpose languages exist: to provide a foundation on which specialized functionality can be built, even when that specialized functionality cannot be foreseen. But when the foundation is shaky, the structure is in constant danger of collapsing.

  • @David, I agree with everything in your post, but I believe that calling .NET "shaky" is a serious overstatement. In the 37+ years I have been developing software I have not encountered a single compiler/platform that was "perfect"; neither have I encountered something as wide reading as the .Net framework and assocuated languages.

    "Pound for Pound" I do believe it is by far the best I have seen (albiet not perfect) in all these years.

  • Just another good reason to avoid raw arrays.

  • @TheCPUWizard

    I agree with you. I switched to .NET from C++ 6 years ago and have never looked back. My few forays into Java are dark periods of my life that I would rather forget. I have staked my career on the viability of the .NET platform as a way to quickly deliver quality applications that provide value to end users.

    However, as you said, no platform is perfect, and I think it is perfectly legitimate to describe the way in which those imperfections hurt the platform as a whole, in the hopes that similar mistakes are not made in the future. In this case, it was not my intention to describe the entire .NET framework as "shaky", but rather to point out that any application which is built believing that this particular case works as expected has a shaky foundation that could collapse at any time without warning and without any explanation. Allowing that kind of unseen danger into a general purpose platform for the sake of premature performance optimization was, IMHO, a bad design decision.

  • > if X is a reference type implicitly convertible to reference type Y then X[] is implicitly convertible to Y[]

    That statement is a bit misleading. "implicitly convertible" usually includes user defined conversion. Array conversion obviously can't include user defined conversions since we're basically talking about reinterpret_casts here.

    Looking array variance up in the specification(ECMA 4.0) it's more restrictive than your statement(and also matches my experimental results)

    > For any two reference-types A and B, if an implicit reference conversion (§13.1.4) or explicit reference

    conversion (§13.2.3) exists from A to B, then the same reference conversion also exists from the array type

    A[R] to the array type B[R],

    With the notable difference that it's talking about an implicit *reference* conversion not just any implicit conversion.

    ----------------------------------------------------------------------------------------------------------------

    Example program:

    void Main()

    {

    Y y=new X();  // X is implicitly convertible to Y

    Y[] ys=new X[1];  // X[] isn't implicitly convertible to Y[]

    }

    class Y

    {

    }

    class X

    {

        public static implicit operator Y(X a)

    {

    return null;

    }

    }

Page 2 of 2 (22 items) 12