Effect compilation and Content Pipeline automation in XNA Game Studio 4.0

Effect compilation and Content Pipeline automation in XNA Game Studio 4.0

  • Comments 28

Previous XNA versions included an Effect.CompileEffectFromSource method. We do not support runtime shader compilation on Xbox, so this was only available in the Windows framework, which could compile shaders targeting either Windows or Xbox. This factoring was problematic because:

  • The API looked like part of the runtime framework, but was not available on all platforms.

  • Our Windows redistributable had to include the Xbox shader compiler, which was wasted space for people shipping Windows games.

  • As we add more platforms, we increasingly find ourselves having to ship pieces of our product out-of-band for specific platforms. Not such a great versioning story if changing the shader compiler for unrelated platforms requires us to update the Windows framework DLL!

Game Studio 4.0 moves the shader compiler from the Windows framework to the Content Pipeline. Effect.CompileEffectFromSource is replaced by EffectProcessor.

There are several ways you can invoke the Content Pipeline to compile an effect:

  • Add .fx file to the Content folder in Visual Studio, then hit F5. Easy, but not especially flexible.

  • Add .fx file to an MSBuild project (either using Visual Studio, or hand-editing XML), then build this from the command line using msbuild.exe. This approach is powerful and IMHO unfairly neglected.

  • Use the MSBuild API to programmatically create and build a project in memory. Great for big projects like level editors, but a hassle to set up.

  • New in Game Studio 4.0: call directly into an importer or processor from your C# code, bypassing MSBuild entirely. The rest of this article describes this new approach.

To call into the Content Pipeline, you must add a reference to Microsoft.Xna.Framework.Content.Pipeline.dll. But when you open the Add Reference dialog, this will most likely not be listed. What gives?

.NET Framework 4 introduces the concept of a client profile, which is a subset of the full .NET Framework optimized for client applications (as opposed to server or developer tools), and which therefore has a smaller download size. The XNA Framework runtime works with the client profile, and our Windows Game project templates target it by default, but the Content Pipeline requires the full .NET framework. To reference the Content Pipeline assembly:

  • Open your Visual Studio project properties
  • Focus the Application tab
  • Change 'Target framework' from '.NET Framework 4 Client Profile' to '.NET Framework 4'
  • Choose Add Reference
  • Focus the .NET tab
  • Add Microsoft.Xna.Framework.Content.Pipeline, plus whatever importers you want to use (let's pick Microsoft.Xna.Framework.Content.Pipeline.TextureImporter as well)

We're going to import a texture. First some using statements:

    using Microsoft.Xna.Framework.Content.Pipeline;
    using Microsoft.Xna.Framework.Content.Pipeline.Graphics;
    using Microsoft.Xna.Framework.Content.Pipeline.Processors;

Now we make a custom logger class. This version just throws away any warnings or messages, but a real app might want to display these somewhere:

    class MyLogger : ContentBuildLogger
    {
        public override void LogMessage(string message, params object[] messageArgs) { }
        public override void LogImportantMessage(string message, params object[] messageArgs) { }
        public override void LogWarning(string helpLink, ContentIdentity contentIdentity, string message, params object[] messageArgs) { }
    }

Finally we create a custom importer context, which allows the importer to communicate with whoever is hosting it:

    class MyImporterContext : ContentImporterContext
    {
        public override string IntermediateDirectory { get { return string.Empty; } }
        public override string OutputDirectory { get { return string.Empty; } }

        public override ContentBuildLogger Logger { get { return logger; } }
        ContentBuildLogger logger = new MyLogger();

        public override void AddDependency(string filename) { }
    }

Armed with these helper classes, it is easy to call into the TextureImporter:

    TextureImporter importer = new TextureImporter();

    TextureContent texture = importer.Import("cat.tga", new MyImporterContext());

Calling a processor is similar, except we need a custom processor context, which is more complex than the importer context we used before:

    class MyProcessorContext : ContentProcessorContext
    {
        public override TargetPlatform TargetPlatform { get { return TargetPlatform.Windows; } }
        public override GraphicsProfile TargetProfile { get { return GraphicsProfile.Reach; } }
        public override string BuildConfiguration { get { return string.Empty; } }
        public override string IntermediateDirectory { get { return string.Empty; } }
        public override string OutputDirectory { get { return string.Empty; } }
        public override string OutputFilename { get { return string.Empty; } }
        
        public override OpaqueDataDictionary Parameters { get { return parameters; } }
        OpaqueDataDictionary parameters = new OpaqueDataDictionary();
        
        public override ContentBuildLogger Logger { get { return logger; } }
        ContentBuildLogger logger = new MyLogger();

        public override void AddDependency(string filename) { }
        public override void AddOutputFile(string filename) { }

        public override TOutput Convert<TInput, TOutput>(TInput input, string processorName, OpaqueDataDictionary processorParameters) { throw new NotImplementedException(); }
        public override TOutput BuildAndLoadAsset<TInput, TOutput>(ExternalReference<TInput> sourceAsset, string processorName, OpaqueDataDictionary processorParameters, string importerName) { throw new NotImplementedException(); }
        public override ExternalReference<TOutput> BuildAsset<TInput, TOutput>(ExternalReference<TInput> sourceAsset, string processorName, OpaqueDataDictionary processorParameters, string importerName, string assetName) { throw new NotImplementedException(); }
    }

To build for a different platform, or for HiDef as opposed to Reach, change the TargetPlatform and TargetProfile properties.

You can use the AddDependency method to track additional files that are read by the importer or processor. For instance this is called any time the effect compiler encounters a #include statement in the effect source code. This information is useful if you want to implement incremental rebuild, so you can handle things like recompiling effects because their #include files have changed, even when the main .fx has not.

This simple example does not bother to implement the Convert, BuildAndLoadAsset, and BuildAsset methods. These are not neccessary when running simple standalone processors, but you will need to hook them up if you want to support more complex things like ModelProcessor, which uses them to call into the MaterialProcessor, EffectProcessor, and TextureProcessor.

Armed with our custom processor context, we can create an EffectContent object, set its EffectCode property to a string containing our HLSL source, then compile it with EffectProcessor:

    EffectContent effectSource = new EffectContent
    {
        Identity = new ContentIdentity { SourceFilename = "myshader.fx" },

        EffectCode =
        @"
            float4 MakeItPink() : COLOR0
            {
                return float4(1, 0, 1, 1);
            }

            technique Technique1
            {
                pass Pass1
                {
                    PixelShader = compile ps_2_0 MakeItPink();
                }
            }
        ",
    };

    EffectProcessor processor = new EffectProcessor();

    CompiledEffectContent compiledEffect = processor.Process(effectSource, new MyProcessorContext());

We can use processor parameters to adjust the compilation behavior:

    EffectProcessor processor = new EffectProcessor();

    processor.Defines = "ENABLE_FOG;NUM_LIGHTS=3";
    processor.DebugMode = EffectProcessorDebugMode.Optimize;

    CompiledEffectContent compiledEffect = processor.Process(effectSource, new MyProcessorContext());

Note that the Content Pipeline is not part of the XNA Framework redistributable, so this code will only work on computers that have a full Game Studio install.

  • This is looks great awesome, except for the full game studio install requirement...

    I wonder, would it be possible for there to be an intermediate install, which doesn't require the whole of visual studio, but supports building content?

    It would be good for non-programmers.

  • I'm not sure how I feel about this - personally, I find the content processor fairly unhelpful and only suited to extremely small games without user created content.

    I was really hoping for the *opposite* in XNA 4 - the ability to use all types without a dependency on the content pipeline. Previously about half of the content types (fonts, audio, etc) could only be constructed via the content system. Now there are more - incredibly frustrating!

    I really wish XNA was treated more like a true API with the content processor/pipeline as an isolated, optional component. Of *all* the things in a game, generally content has the most specific requirements in terms of storage, layout, etc.

  • Whats the advantage of doing things this way, rather than using the simpler(in most cases) ContentProcessorContext.Convert<>() ?

    If it is performance, then why doesnt XNA do that anyway?

  • Never mind... I think I see way, to allow people who dont genuainly want to use the content pipeline to hack around the fact that the shader functions are missing(perhaps rightly missing).

    Still wouldnt a helper function in the content pipeline assembly have been better, rather than embedding the compilation into the processor?

    (or even an independant assembly distributed with the content pipeline)

  • Finally we can build content without MSBuild,  great improve!

  • > Whats the advantage of doing things this way, rather than using the simpler(in most cases) ContentProcessorContext.Convert<>() ?

    The simple answer is that to call Convert, you need a ContentProcessorContext instance, which is provided to you if you are running inside the Content Pipeline / MSBuild environment, but if you are running somewhere else, you need to create your own context object, in which case you are responsible for implementing Convert yourself.

    To implement the Convert method, you typically end up writing code that looks very much like the above: find the requested processor type, instantiate it, then call into it.

    The reason these methods all go through a context object, and why you can't just use our build-in context if you are running in some different environment to the standard MSBuild setup, is that these methods require a lot of policy to work correctly:

    - They need to know how to locate importer and processor types, which depends on how these assemblies are loaded (for instance our MSBuild implementation loads everything into a worker app domain, marshalling build operations back and forth, in order to be able to selectively reload modified processor assemblies between one build and the next).

    - They need to understand how to track dependencies, which has different requirements depending on if/how you are implementing incremental rebuild.

    - The BuildAsset method needs to be able to kick off other nested build requests, generate ExternalReferences pointing to the final build output locations, etc.

    All these things require intimate knowledge of the environment in which the build is running. We use MSBuild to implement our standard Content Pipeline build environment, and provide a standard context object which implements these things by hooking up to MSBuild.

    If you run somewhere else where there is no MSBuild, you have to make your own choices about how to locate and load pipeline assemblies, how to track dependencies, etc, so you must provide your own context object, and therefore can't just rely on our built-in implementation of the Convert<> helper.

  • Hmm.. Sounds like a Problem for me. In my Game, Postprocessing Shaders are generated dynamicly, so that i only need One extra Rendertarget for this. But with this changes I can Not longer compile the shader on the client maschiene...

    Am I missing something, or is there a workaround?

  • I'm currently relying quite heavily on Effect.Disassemble(), I'm guessing this has gone too?

    I expect I'll end up calling into fxc directly.

  • I also have always compiled some of my shaders on the client machine. Is there a workaround as having visual studio installed on all the machines isn't really an option.

    This seems like the first actual loss in 4.0 however, everything else seems to be an improvement.

  • Nice ,, this what is was looking for for my editor, so i can make the physics work from inside my editor,, insted of recompile the physics all the time

    i have uploaded a nice physics wideo with the new stuff ,,,

    Please shawn , write more about this and put up a code sample at creators club "windform sample 3"

    best regards

    Michael

    link to physics content pipeline new stuff direct from shawn cool code..

    http://www.youtube.com/watch?v=ffLwJyV1HJU

    i now work in my editor , i am happy

  • This seems like a change that is going in the wrong direction. I am working on a game that generates shaders dynamically at run-time and this is not good for me.

  • The XNA policy of always pre-compiling shaders does seem a bit odd.

    If you look at many native engines the approach seems to often be a mix of pre-compiling a shader cache and dynamically compiling shaders if there is a cache miss(eg out of date cache).

    Why not allow this approach with XNA?

  • I agree that this change is backwards and frustrating... that is until we finally get a content pipeline redist!!. When can we finally write tools that utilise the power of the XNA FX but do not require a full XNA GS install?

  • In my last game, I had some code in an #if DEBUG #if WINDOWS block that gave my game a hotkey that I could use to recompile shaders within the running game, during development. (Very, very useful!)

    I'd like to do the same thing in XNA 4.0. But I cannot figure out how to reference the content pipeline assembly for a debug build, but not a release build. Thoughts?

  • I wonder, is there any policy/EULA similar to the DirectX redist that prevents the content pipeline DLLs from being included with the binaries? If so, could this perhaps be waived for internal/development use so we can do this by the book?

    I personally was planning on sneakily doing this anyway for some internal content tools, to make the deployment easier on the artists. These guys are hard enough to find and I don't want to frustrate them with cumbersome installation processes. I'll already frustrate them enough with my art requirements :)

Page 1 of 2 (28 items) 12
Leave a Comment
  • Please add 1 and 2 and type the answer here:
  • Post