BigArray<T>, getting around the 2GB array size limit

re: BigArray, getting around the 2GB array size limit

CN — Thu, 11 Aug 2005 05:02:18 GMT

Just one note: On a 32-bit system, I think a single object reference is size 8 (the real pointer and some reserved data for the CLR/GC). On a 64-bit system, I imagine this is 12 or even 16 bytes. Wouldn't this mean that the practical maximum index in a reference type array is actually *lower* on a 64-bit machine, compared to 32 bits? Allocating a contig chunk of memory larger than 1 GB on a 32-bit machine might be possible, and that would be enough to beat the 64-bit implementation. Of course, most references have to be null references, but it's *possible*.

Or am I missing something here?

re: BigArray, getting around the 2GB array size limit

joshwil — Thu, 11 Aug 2005 05:19:01 GMT

A single object reference is just a pointer into the GC heap, 4 bytes on 32-bit and 8-bytes on 64-bit. You are correct in noticing that this means you can have 2x the number of slots in a maximally sized object reference array on 32-bit as you can get on 64-bit (see my previous entry on generics on 64-bit: http://blogs.msdn.com/joshwil/archive/2004/04/13/112598.aspx).

However, in real world scenarios, given that the minimum object size on 32-bit is 12 bytes if you have a unique reference for every element in your array you are looking at 16 bytes per object and therefore limited to 1/2 the number of objects as you would get on 64-bit before you run our of virtual address space.

It does make one biased twoards using arrays of primitive data types where you can instead of boxed references (e.g. ArrayList) to avoid the significantly heavier penalties that go with that practice on the 64-bit platform.

re: BigArray, getting around the 2GB array size limit

Oleg Mihailik — Thu, 11 Aug 2005 08:28:54 GMT

Oh, there is other ugly side of 2Gb limit!

List<T> class is limited too, because it based on array internally.

It seems fairly impractical. Why on Earth 64-bitness is supported in Framework, if BCL does not support it itself?

I believe List<T> should follow your BigArray pattern and do workaround on 64-bit platform to support high range of indices.

This also should be applied to other array-like classes. BitArray, for example (bit, not big). If no, support of 64-bitness stays nominative and not full-featured.

re: BigArray, getting around the 2GB array size limit

Willy Denoyette — Thu, 11 Aug 2005 22:57:39 GMT

Josh,

Are you sure you meant 2GByte, I assume the largest array can contain 2Giga elements (the array length field being a 32 bit entity holding an int), if these elements are longs that would mean a 16 GByte array, or 8 GByte for int's, 4 GB for short's an right... 2GB for bytes. Or am I missing something here.

Willy.

re: BigArray, getting around the 2GB array size limit

joshwil — Thu, 11 Aug 2005 23:08:36 GMT

Willy --

I meant 2GB = 2 GigaBytes.

The maximum object size in the managed heap in the 2.0 CLR (on all platforms) is 2GB. Managed arrays have to fit into that. Given this you can only fit 2Giga elements in an array where each element is 1 byte in size (e.g. an array of bytes or booleans) (however in reality the maximum is just under 2 Giga elements as there is some CLR book keeping that eats a few bytes).

re: BigArray, getting around the 2GB array size limit

joshwil — Thu, 11 Aug 2005 23:12:47 GMT

Oleg --

I've pinged the PM who owns collections for the CLR and they will be responding re: big collections. There is always the option to implement big collections yourself or push for some crew (like the powercollections crew) to do it. Our deciding not to do it in the 2.0 CLR release came down to a number of tradeoffs that we needed to consider including
- 64-bit performance
- 32-bit performance
- ease of porting code from 32-bit to 64-bit
- trying not to bloat the BCL more than it already is.

I would be willing to bet that most applications won't find themselves coming up against this wall for the number of elements in a particular collection and will instead find themselves overwhelmingly happy about the huge amounts of virtual address space opened up by the 64-bit platform.

However, we are currently seriously considering what to do with the next version of the runtime re: big objects and all input is good input!

re: BigArray, getting around the 2GB array size limit

Willy Denoyette — Thu, 11 Aug 2005 23:13:26 GMT

Ok, after more carefully reading I see that the size of the object is restricted to 2GByte contiguous memory space in the managed heap (the Large Object Heap is assume), irrespective the type of object.

Willy.

re: BigArray, getting around the 2GB array size limit

mihailik — Fri, 12 Aug 2005 10:01:57 GMT

Thank you, Josh for description.

For me, it seems 64-bitness in CLR v2 is early implementation. Probably, it collect real field performance data, scenarios and all kind of problems to analyze it and to resolve in next version. I hope I will see improvements in Vista version of CLR :-)

And thanks that PM too. I guess his name is Kit :-)

re: BigArray, getting around the 2GB array size limit

joshwil — Fri, 12 Aug 2005 17:38:02 GMT

mihailik -

I am sure that the 64-bit runtime will get better over time, however it's current implementation is pretty vetted, we've been working on it in one form or another for > 5 years.

-josh

BigArray, getting around the 2GB array size limit

Mon, 26 Nov 2007 09:19:01 GMT

PingBack from http://feeds.maxblog.eu/item_720001.html

przemkovv’s world » Blog Archive » Zarz??dzanie pami??ci?? pod Windowsem x64

Fri, 29 Aug 2008 02:29:24 GMT

PingBack from http://przemkovv.net/2008/08/29/zarzadzanie-pamiecia-pod-windowsem-x64/

Initialize the size of array but face OutOfmemoryException | keyongtech

Initialize the size of array but face OutOfmemoryException | keyongtech — Sun, 18 Jan 2009 19:49:21 GMT

PingBack from http://www.keyongtech.com/1194272-initialize-the-size-of-array

Large MemoryStream > 2gb | keyongtech

Large MemoryStream > 2gb | keyongtech — Thu, 22 Jan 2009 06:29:36 GMT

PingBack from http://www.keyongtech.com/432376-large-memorystream-2gb

java vs .NET - egzystencjalnie :) - Strona 3 | hilpers

java vs .NET - egzystencjalnie :) - Strona 3 | hilpers — Thu, 22 Jan 2009 23:47:28 GMT

PingBack from http://www.hilpers.pl/373505-java-vs-net-egzystencjalnie/3