Introduction
I spent some time this week thinking about how to test a code generator that generates C# code and is itself written in C# (so it's once again all about managed code). At first I thought about how to verify the generator output directly. Since the output is C# code - which ultimately is just a bunch of text - one approach is to create a collection of base line outputs and tests that run the generator with different inputs and compare the outputs to the base lines which have been verified when then the tests were written. There are two major flaws in this idea: Verifying the base lines in this scenario will be a manual task and even small changes to the generator output will cause tests to fail even if the new code behaves exactly the same way then before. Another approach would be to actually analyze the generator output but at the very least this would require parsing C# and I'll leave that to people who enjoy (and know how to) build compilers.
And? How would you test it?
So, coming to the conclusion that verifying the output was not an option I started thinking about the next best thing: Writing tests against the generated code and thus implicitly testing the generator. Makes sense; however, the question remains how to do that. One could run the generator manually for all interesting inputs, add the outputs to a test project and then write tests for them. This is actually worse than the approach with base lines since now there are a bunch of source files to update every time the generator output changes but the tests won’t automatically break when that happens. What I really wanted was running the code generator with every test pass. That said - and without further ado - let's look at some code. Consider a C# class library project containing the following "generator":
public static class CodeGenerator
{
public static string Generate(string name)
return @"
public static class PersonalizedHelloWorld
public static string Name { get { return " + (name != null ? "@\"" + name.Replace("\"", "\"\"") + "\"" : "null") + @"; } }
public static string GetHelloWorldMessage()
if (Name == null)
return ""Hello World! Hello anonymous user!"";
else
return ""Hello World! Hello "" + Name + ""!"";
}
}";
I say "generator" because obviously it doesn't do anything useful and I just wrote it for demonstrating my approach to testing generators which is also why I'm not using the CodeDOM. Now, let's think about the output we get for the string John "\/\/" Doe as the input. What I would normally do to test that code is create a new test project and write a test class like this one:
[TestClass]
public class NonemptyNameTests
[TestMethod]
public void NameProperty()
Assert.AreEqual<string>(@"John ""\/\/"" Doe",
PersonalizedHelloWorld.Name);
public void GetHelloWorldMessageMethod()
Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",
PersonalizedHelloWorld.GetHelloWorldMessage());
Apparently I can't do that because the class this code calls into does not exist when the test assembly is built. So instead I add a new source file to the test project with the following code and set its build action to Embedded Resource.
public static class NonemptyNameTests
public static void NameProperty()
public static void GetHelloWorldMessageMethod()
The idea is to compile this code together with the compiler output for the input John "\/\/" Doe during test execution. In order to do this I introduce a little helper method - we need to run and test the generator multiple times with different inputs after all.
public static class TestCompiler
private static string unitTestFrameworkAssemblyPath = typeof(Assert).Assembly.Location;
public static Assembly Compile(string generatedCode,
string generatedCodeFileName,
string dynamicTestCodeResourceName,
string dynamicTestCodeFileName,
string assemblyFileName)
string dynamicTestCode;
using (StreamReader reader = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(dynamicTestCodeResourceName)))
dynamicTestCode = reader.ReadToEnd();
File.WriteAllText(generatedCodeFileName, generatedCode);
File.WriteAllText(dynamicTestCodeFileName, dynamicTestCode);
CompilerParameters parameters = new CompilerParameters();
parameters.GenerateExecutable = false;
parameters.GenerateInMemory = false;
parameters.OutputAssembly = assemblyFileName;
parameters.ReferencedAssemblies.Add(unitTestFrameworkAssemblyPath);
#if DEBUG
parameters.IncludeDebugInformation = true;
#else
parameters.IncludeDebugInformation = false;
#endif
CSharpCodeProvider codeProvider = new CSharpCodeProvider();
CompilerResults results = codeProvider.CompileAssemblyFromFile(parameters,
generatedCodeFileName,
dynamicTestCodeFileName);
foreach (string message in results.Output)
Debug.WriteLine(message);
return results.CompiledAssembly;
All Compile() does is take a couple of names and generated code and return a new assembly. A key characteristic of this implementation is that the source code as well as the assembly are written to the file system which allows further analysis in case the code does not compile or test execution fails (the code provider does write code passed in as strings to the file system automatically - but it writes it to a temp directory and it is very quick to delete those temporary files as soon as the compiler finishes execution). Now we can add what VSTT/MSTest.exe will consider to be the test class:
public const string AssemblyFileName = "NonemptyNameTests.dll";
public const string GeneratedCodeFileName = "NonemptyNameTests.target.cs";
public const string DynamicTestCodeFileName = "NonemptyNameTests.dynamic.cs";
public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NonemptyNameTests.dynamic.cs";
private static Type dynamicTestType;
[ClassInitialize]
public static void ClassInitialize(TestContext context)
string generatedCode = CodeGenerator.Generate(@"John ""\/\/"" Doe");
Assembly assembly = TestCompiler.Compile(generatedCode,
GeneratedCodeFileName,
DynamicTestCodeResourceName,
DynamicTestCodeFileName,
AssemblyFileName);
dynamicTestType = assembly.GetType("NonemptyNameTests");
dynamicTestType.GetMethod("NameProperty").Invoke(null, null);
dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);
Note that the only real work this class does is run the code generator, build another test assembly with the generator output and the test code we embedded as a resource and store a Type instance for the new test class. Even more important, it does so in the class initialization method meaning that if an exception is thrown during the generation or the compilation step it will cause all tests in this class to fail. Furthermore, for every test to be added you need to add two methods: One in the class with the actual test logic which gets compiled at runtime and one in the class above that just calls the first one using reflection. And that's actually all we need to make this work. What's left for this post is a non-exhaustive list with the advantages and disadvantages of this approach.
Advantages
Disadvantages
Full annotated sample source code
/**********************************
* Project: CodeGenerator *
* File Name: CodeGenerator.cs *
* Build Action: Compile *
**********************************/
// Useless code generator method for
// demonstration purposes only.
//
// DO NOT USE IN PRODUCTION ENVIRONMENTS!
/************************************
* Project: CodeGeneratorTests *
* File Name: TestCompiler.cs *
************************************/
using Microsoft.CSharp;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Reflection;
// Use by test class initializers to compile the generated
// code together with actual test code into an assembly
// during the test pass.
// Reads the actual test code from a resource
// stream in the static test assembly.
// Writes the generated code as well as the actual
// test code from the resource stream to the file
// system in order to allow debugging the actual test code.
// Parameters for the code provider telling it to generate
// a DLL and write it to the file system.
// The paths to/names of all assemblies containing types
// referenced in the code to compile need to be added to
// the CompilerParameters object ReferencedAssemblies collection.
// Need to make debugging easy in case a class initializer or
// test fails by having the compiler include debug information
// and create a PDB file.
// Creating code provider instance and invoking C# compiler.
// Writing full compiler output to trace listeners in debug
// builds to preserve all errors in case the generated/dynamic
// test code does not compile.
// Returning new assembly to caller.
/**************************************
* File Name: NonemptyNameTests.cs *
**************************************/
using System;
// Test class verifying the code generator
// by testing its compiled output.
// File and resource name constants for this test class.
// Running the code generator we want to test
// with one set of the inputs we need to cover.
// Compiling the generated code together with the test code
// for the chosen generator input into a new assembly. Note
// that if the generator produced code with syntax errors
// all tests in this test class will fail because of an
// exception in the class initializer.
// Storing a Type instance for the test class type in the new
// assembly to be used by the test methods in this class.
// The following two test methods do not contain actual
// verification code. Instead they call methods in the new
// assembly using reflection.
/**********************************************
* File Name: NonemptyNameTests.dynamic.cs *
* Build Action: Embedded Resource *
**********************************************/
// Class containing the actual verification logic for a specific set
// of code generator input. This code is not compiled into
// CodeGeneratorTests.dll but embedded as a resource. It is compiled
// together with the generator output and called by methods marked
// with the TestMethod attribute using reflection during the test pass.
* File Name: NameNullTests.cs *
public class NameNullTests
public const string AssemblyFileName = "NameNullTests.dll";
public const string GeneratedCodeFileName = "NameNullTests.target.cs";
public const string DynamicTestCodeFileName = "NameNullTests.dynamic.cs";
public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NameNullTests.dynamic.cs";
string generatedCode = CodeGenerator.Generate(null);
dynamicTestType = assembly.GetType("NameNullTests");
/******************************************
* File Name: NameNullTests.dynamic.cs *
******************************************/
public static class NameNullTests
Assert.IsNull(PersonalizedHelloWorld.Name);
Assert.AreEqual<string>("Hello World! Hello anonymous user!",
This posting is provided "AS IS" with no warranties, and confers no rights.