Shawn Hargreaves Blog
One of the new features in Game Studio 3.1 is automatic serialization for .xnb files. This is a feature I have wanted ever since we first designed the Content Pipeline, so it makes me very happy that we finally found time to implement it!
You know those ContentTypeWriter and ContentTypeReader thingamyhickies? You don't need them any more! Delete them, and the Content Pipeline will automatically serialize your data using reflection.
Note: you can still write a ContentTypeWriter and ContentTypeReader by hand if you want. You just don't have to any more.
If the Content Pipeline encounters a type that has no matching ContentTypeWriter, it will dynamically create one for you. This works in a similar way to the IntermediateSerializer, but it reads and writes binary .xnb files instead of XML.
The trick to successfully using the Content Pipeline is to understand which code runs at build time versus runtime. Even though the serialization is now automatic, you can still get in a muddle if you have things like cyclic references where your game tries to use a type that is defined inside itself while building itself!
Here's an example of using the new serializer:
Create a new Windows Game project. Let's call this MyGame.
Right-click on the Solution node, Add / New Project, and choose the Windows Game Library template (not Content Pipeline Extension Library, because we want to use this at runtime as well as build time). Call it MyDataTypes.
Add this class to the MyDataTypes project:
public class CatData
public string Name;
public float Weight;
public int Lives;
We're going to use this type to build some custom content, so we need to reference it during the Content Pipeline build process. Right-click on the Content project that is nested inside MyGame, choose Add Reference, and select MyDataTypes from the Projects tab.
Now we can add this file (let's call it cats.xml) to our Content project:
<?xml version="1.0" encoding="utf-8" ?>
Hit F5, and the content will build. Look in the bin directory, and you will see that it created a cats.xnb file. If you build in Release configuration, this file will be compressed. With the above example, my 326 byte XML file becomes 270 bytes in .xnb format, but the compression ratio will improve as your files get bigger.
Before we can load this data into our game, we must reference the MyDataTypes project directly from MyGame as well from its Content sub-project, in order to use our custom type at runtime as well as build time. Once we've done that, we can load the custom content:
CatData cats = Content.Load<CatData>("cats");
If you create a copy of this project for Xbox 360, you will notice that although the Xbox version of MyGame references the Xbox version of MyDataTypes, its Content project still uses the Windows version of MyDataTypes, even though it is building content for Xbox. This is an important point to understand. Because our custom type is used both at build time (on Windows) and at runtime (on Xbox), we must provide both Windows and Xbox versions of this type.
Remember how some types are not the same at build time versus runtime? For instance a processor might output a Texture2DContent, but when you load this into your game it becomes a Texture2D.
The .xnb serializer understands such situations, as long as you give it a little help. For instance if I used this type in MyGame:
public class Cat
public string Name;
public Texture2D Texture;
I could declare a corresponding build time type in a Content Pipeline extension project:
public class CatContent
public string Name;
public Texture2DContent Texture;
Note how both types have basically the same fields, and in the same order, but the runtime Cat class uses Texture2D where the build time version has Texture2DContent. Also note how the build time CatContent class is decorated with a ContentSerializerRuntimeType attribute. This allows me to use CatContent objects in my Content Pipeline code, but load the resulting .xnb file as type Cat when I call ContentManager.Load.
Every silver lining has a cloud, right?
The automatic .xnb serialization mechanism uses reflection. Reflection is slow, and causes a lot of boxing, which can result in a lot of garbage collections.
Fortunately, this is not as bad in practice as it sounds on paper. Many custom data types are small, or at least the custom part of them tends to contain just a few larger objects of types that have built-in ContentTypeWriter implementations (textures, vertex buffers, arrays or dictionaries of primitive types, XNA Framework math types, etc). In such cases the performance overhead will be low. When I converted a bunch of existing Content Pipeline samples to use the new serializer, it did not significantly affect their load times.
But if you have a collection holding many thousands of custom types, you may see poor load performance. In such cases, you can provide a ContentTypeWriter and ContentTypeReader for just the specific types that are slowing you down. The new system is easier to use, but it can be more efficient to do things the old manual way.
This will come in very handy, for sure. I could have used it a month ago. I guess now I'll be one of those old guys who looks back and says "I remember when I had to write my own ContentTypeWriter. Kids nowadays have it easy".
Ok, I was waiting for this post :) It's good to be finally be rid of writing those writers/readers.
Now, a question:
For the last example, the one with the Texture2DContent: what is the corresponding input XML for this example? Can the Texture2DContent be simply written in the XML (how?), or does it have to be initialized inside a ConentProcessor (because this is where you have access to the context)?
Shawn, with this new feature is it possible to serialize to xnb during runtime?
barkers crest: no. Writing .xnb files is done as part of the Content Pipeline build process, same as before.
Catalin: in theory you could embed the texture data directly into XML (TextureContent can be serialized by IntermediateSerializer) but I'm not sure how useful that would be!
More likely you would want to just store the texture filename in your XML, then go off and read that file in during the build process, which would require some processor code.
Here's an example from one of our unit tests of what a TextureContent looks like in IntermediateSerializer XML:
<XnaContent xmlns:Graphics="Microsoft.Xna.Framework.Content.Pipeline.Graphics" xmlns:PackedVector="Microsoft.Xna.Framework.Graphics.PackedVector">
Ok, so I still need a Processor to solve this in an elegant (and resource-cheap) way, since textures will be shared between different assets (which means embedding them isn't too good).
I would most likely do this in a processor, yes. The auto serialization doesn't remove the need to write processor code if you want to process data or trigger other actions (like going off and building a texture) at runtime.
There are a couple of other options, though:
- you could just embed a string asset name in in the XML, then call ContentManager.Load on that name at runtime after you load the .xnb
- or you could embed an ExternalReference instance, in which case the .xnb deserializer will automatically follow that reference and load the asset it points to in place of the original reference
These options only work if the referenced asset is already being built through some other mechanism, though, so you know what name the reference should point to. They would be an option if you have a bunch of textures that are already added to your Content project, and can just put the name of these textures in this other XML asset, but unlike using a custom processor, they will not automatically go off and build that texture for you if the XML references a texture that is not otherwise part of your Content project.
And unfortunately, I'd have no errors during build telling me that some textures are missing.
Just amazing !!!
Thank you really !
I work with XNA 3.0, and it couldn't auto serialize my XML objects, and I'm so happy to see this post !
It's a very good new. I earn a lot of time.
I see that runtime serialization to .xnb will not be availble which makes sense because that is part of the build process.
Does that mean that this new fancy reflection based auto serialization will be useless for necessary things such as saving/loading game states, or can you use the framework to seriliaze to something other than .xnb at run time?
Kevin: this new serializer is part of the Content Pipeline, so the writing part is only available at build time.
For runtime serialization of things like save games, I would probably start with XmlSerializer, or if you need something more efficient, you could use BinaryWriter/BinaryReader to create a more compact binary format.
So I've finally gotten around to trying out the content pipeline. Until now, I've been loading/saving all my game data using XML serialization, even on the 360, and it works fine - but I want to start deploying my content as packed XNB files for better load performance.
As a basic implementation, I wrote a raw importer to read in my XML files as byte and then a processor to convert those to my runtime data structures. This seems to work, but for some reason, the XNB files generated by this automatic reflection-based serializer are around 10 times larger than my XML files.
Is this behavior intended, or am I using the content pipeline incorrectly? Will the overhead remain fixed as my files grow larger? It's hard to tell whether or not it's fixed overhead from looking at the XNB files in a hex editor.
Kevin: this really depends on what your data looks like. XNB files are usually much smaller than XML (especially if you enable compression) but it's certainly possible to construct objects where this would not be the case.
I would recommend the forums on creators.xna.com if you want to follow up with more details, as they're somewhat nicer than these comment boxes for having a discussion :-)
Luckily it turns out it was a bit of a false alarm :) The size went down significantly once I fiddled with some of the serialization attributes - there's still some overhead, but from testing it appears to be fixed - most of it comes from the fact that the ReflectiveWriter has to write out the fully qualified names of all the types it uses, which isn't the case in my source XML.
The actual binary data for the content seems to be quite compact, which is what I was hoping for.
I currently have several different (fairly large ~12KB) xml files that I copy to the output directory, and load when my game starts up. They seem to take a long time to load on the Xbox.
Would I expect to see a performance improvement by moving these files into content pipeline? (I would write a ContentImporter that deserializes from xml, and then uses this new automatic serialization functionality).
There would be some improvement from the smaller file sizes... but is the deserialization functionality generally faster than what I have now (the standard Xml deserialization)?