The XNA Framework and .NET Command Language Runtime (CLR) work on XBox 360 is clicking along. We're head long into feature work, bug fixing, and performance analysis. We've made great progress on planning and implementation since we have returned from GDC.  I can attest first hand, the XNA Framework team is progressing and will have some pretty cool stuff for all y'all as we march closer to the day when we can release bits.

One of the major differences we have between Xbox 360 and any of the other platforms that the .NET CLR has shipped on (Windows, Windows Mobile, and other Windows CE platforms) is the endianness of the system.  The following example should illustrate.

    Consider 67305985 in base 10 = 04030201 in base 16.
   
    This value would be stored in a 4 byte integer as follows
        Little Endian: 0x01 0x02 0x03 0x04
           Big Endian: 0x04 0x03 0x02 0x01

Microsoft has predominantly shipped on Little Endian systems which includes the AMD and Intel family of processors.  The Power PC chip that powers the XBox 360 can actually be configured to run in either little or big endian mode; the version in the XBox 360 is big endian. We actually put a LOT of effort into trying to get most of this correct in spite of only shipping on little endian systems.  But as any coder will tell you, it is near impossible to get this correct for ALL components in a large and complex system like the CLR.  We simply did not have any platform to do this validation on.

While compilers will do the right thing for general computation, but we potentially run into trouble when reading data from storage, over the network, from devices, etc. We need to map the source data's endianness to that of our system. Over the last several months, our stellar QA team has been working hard getting test infrastructure functional and executing an increasing number of the army of test we have for other platforms up and running against the XNA Framework. 

As you can imagine, several of the tests find endian related bugs.  I've squashed enough of them that I can almost smell their stench as I examine source code.

If you're a coder, read on, especially aspiring cross platform XNA Framework game devs.

I'd like to share with you one genre of endian bugs related to the System.BitConverter class that could potentially affect you.

BitConverter is comprised of a set of static functions that do exactly as the name implies - convert bits.  More specifically, they convert from all of the integral value types to byte[] and vice-versa. Most applications of the BitConverter class involves symmetry such as the following

int i = BitConverter.ToInt32(byte[] inputdata);
(manipulate i)
return BitConverter.GetBytes(i);

In the case where the data never leaves the confines of your application, this actually works fine. However, the other typical use of BitConverter involves symmetry across serialization.

file.Write( BitConverter.GetBytes( inputdata ), 0, 4 );

(then later on)

byte[] inputdata = new byte[4];
file.Read( inputdata, 0, 4 )
return BitConverter.ToInt32(inputdata);

This is a potential endian bug.  If the endianess of either end of the serialization does not match, the above code will fail.  All methods of BitConverter accept or return byte arrays in SYSTEM endian order.  The burden is on the application developer to conceive a scheme to maintain consistency across symmetric applications.

How do you know if your usage of BitConverter can run into potential endian issues?

  1. If your data never leaves the confines of your program, then you're safe.  Sort of.  We all know how feature creep, new/unforeseen requirements,  and redesign of our applications utilize code in ways that were never intended.   So unless you have a very good reason (performance would be one if correctly justified) I would not rely on this "safety".
     
  2. If you know the source/dest data format will ALWAYS match your system endianness then you are safe. This is often true when handing data down to devices and similar operating system calls.
     
  3. In my opinion, one should always error on the side of assuming that you will NOT be safe unless you code the safety in.

So how do you write endian safe BitConverter code?  There are two things you must take note of.

  1. You must know the endianness of your source data. Whether reading bytes from a file, slurping bytes off of a socket, or some other source, you must know what endianness the data is in.  If you're processing a well known format (JPG, ZIP, TAR, etc.) refer to the respective formats specification to determine the correct endianness.  If you control generation and consumption of the data, you must choose a consistent endianness and document in your code and specifications.
     
  2. You must honor the little known (and used) BitConverter.IsLittleEndian flag to determine the underlying system endianness.

Here's the corrected file write/read sample from above.

// The file format store integral types in little endian order
if (!BitConverter.IsLittleEndian)
    Array.Reverse( inputdata, 0 );
file.Write( BitConverter.GetBytes( inputdata ), 0, 4 );

(then later on)

byte[] inputdata = new byte[4];
file.Read( inputdata, 0, 4 )
// The file format store integral types in little endian order
if (!BitConverter.IsLittleEndian)
    Array.Reverse( inputdata, 0 );
return BitConverter.ToInt32(inputdata);

All right.  Admission time.  This isn't a very nice solution.  BitConverter is a bit unwieldy in this case.  Okay.  It kinda sucks.

One solution is as to code a EndianBitConverter class that accepts a flag on construction to control the reversing of the byte array data. You could then abstract construction of the endian aware bit converter class into a static function based on the source data.

// The file format store integral types in little endian order
EndianBitConverter littlebc = EndianBitConverter.CreateForLittleEndianData();
file.Write( littlebc.GetBytes( inputdata ), 0, 4 );

(then later on)

EndianBitConverter littlebc= EndianBitConverter.CreateForLittleEndianData();
byte[] inputdata = new byte[4];
file.Read( inputdata, 0, 4 )
return littlebc.ToInt32(inputdata);

The static method above could be implemented as follows

public class EndianBitConverter
{
    public static EndianBitConverter CreateForLittleEndian()
    {
        return new EndianBitConverter( !BitConverter.IsLittleEndian );
    }

    public static EndianBitConverter CreateForBigEndian()
    {
        return new EndianBitConverter( BitConverter.IsLittleEndian );
    }

    bool swap;
    private EndianBitConverter(bool swapBytes)
    {
        swap = swapBytes;
    }

    public Int32 ToInt32(byte[] data)
    {
        byte[] corrected;
        if (swap)
        {
            corrected = data.Clone();
            Array.Reverse( corrected, 0, 4 );
        }
        else
        {
            corrected = data;
        }
        int i = BitConverter.ToInt32( corrected );
    }

    // And similar methods for GetBytes(Int32) and all other types needed.
}

Of course, if you need more perf, you could ditch the array cloning and reversing and do the bit manipulations yourself.  Just be careful for sign extension issues.

I've fixed several bugs in our source code base that do not honor the BitConverter.IsLittleEndian flag. Hopefully this post will help prevent you from falling into the same pitfalls.