There have been a lot of problems with compatibility of custom builds when upgrading from TFS2010 to TFS2012. This blog post is my attempt to summarize what you have to do and how to fix some issues you might run into.

1. The very first step is to upgrade your Server. Your build machines should match the version of your server.
2. To upgrade your build machine to TFS2012 follow the instructions on
   http://msdn.microsoft.com/en-us/library/jj620928.aspx
3. If you need Visual Studio on the build machine, install VS 2012 on the build machine. You will probably want to keep VS2010 on the build machine as well. This shouldn't cause any problems.
4. The next step is to recompile any custom activity assemblies that this build machine uses to target the 4.5 .net framework and the version 11.0.0.0 Team Foundation assemblies.
1. Open the solutions for your custom activities
2. Open Properties on each project and change the target framework to 4.5
3. Remove all TeamFoundation references and replace them with the ones that ship with Visual Studio 2012 (the version number should be 11.0.0.0 - NOT 10.0.0.0).
4. Recompile the solutions and check in the binaries to the custom assembly location
5. ALL assemblies in the custom assembly location for a 2012 controller should be compiled against the 11.0.0.0 versions of the TeamFoundation assemblies.
5. The final step before testing your builds is to "clean up" any versioned namespaces in your Build Process Templates (XAML files)
1. locate all of your XAML files - these are usually in $/TeamProject/BuildProcessTemplates
2. for each XAML file: check it out, clean it by removing versioned namespaces, and check it back in (see http://blogs.msdn.com/b/jpricket/archive/2012/07/17/tfs-2012-cleaning-up-workflow-xaml-files-aka-removing-versioned-namespaces.aspx)
6. Finally, you should be able to test your build definitions.

Limitations

• A 2010 Build machine will NOT work with a 2012 Team Foundation Server
• A 2012 Build machine will NOT work with a 2010 Team Foundation Server
• A 2012 Build machine cannot build a 2010 template that contains versioned namespaces that reference 10.0.0.0 assemblies
• A 2012 Build machine cannot build a 2010 template that calls out to custom activities that are compiled against 10.0.0.0 Team Foundation Assemblies
• Build Definitions that reference custom types may not be editable in both Visual Studio 2010 and 2012
• It's hard to identify all the combinations here, but if loading your custom types causes Team Foundation assemblies of the wrong version to be loaded then the editor will not work as desired.
• Note that in Visual Studio 2012, you should be able to edit other parts of the definition, just not the process parameters that cause errors.

Problems you might face and workarounds:

• After upgrading your templates and custom assemblies as described above, you will only be able to edit them in Visual Studio 2012.
• If you install both Visual Studio 2010 and 2012 on the same machine, make sure you install them in that order: 2010 and then 2012.
• Opening a template in the VS 2010 Worflow Designer on this machine will not work without changes to the template. It will attempt to load the 2012 Team Foundation assemblies as well as the 2010 versions which will cause errors when it tries to evaluate any types. There's really nothing wrong with the actual template.
• If you have to have the designer working in 2010 on a machine with 2012 installed, you can change the namespaces in the XAML file to use the full assembly name including the version information. But I wouldn't recommend this, since it will have to be removed for the template to work in 2012.
• The 2012 Workflow Designer should work fine if you followed the steps above to "clean" the XAML
• If you get the following error when building your definitions against a recently upgraded build machine, you should restart your build machine service from the TF Admin Console on the build machine.
• TF215097: An error occurred while initializing a build for build definition \Project1 (Dev10)\Simple - Manual: The values provided for the root activity's arguments did not satisfy the root activity's requirements:
  'DynamicActivity': Expected an input parameter value of type 'Microsoft.TeamFoundation.Build.Workflow.Activities.BuildSettings' for parameter named 'BuildSettings'.
• I am not sure what caused this error when I did it, but the fix was pretty simple once I figured it out.
• UPDATE: Several customers have continued to get this error after upgrading even though they restarted their build service. It turns out that these customers had fixed most of their custom assemblies but not all. If you continue to see problems like this one (even intermittently), it means that there is still a build definition that uses assemblies/activities that are compiled against the 10.0.0.0 assemblies. Once the error starts, all subsequent builds fail because the old assemblies are loaded into the build services memory. Only restarting will resolve this issue.
• If you get the following error, you didn't update ALL your custom assemblies to reference the 11.0.0.0 versions of the TF assemblies.
• TF215097: An error occurred while initializing a build for build definition \Project1 (Dev10)\Advanced - Manual: The root element of the build process template found at $/Project1 (Dev10)/MyActivities/MyActivities/AdvancedBuildProcessTemplate.xaml (version C42) is not valid. (The build process failed validation. Details:
  Validation Error: The private implementation of activity '1: DynamicActivity' has the following validation error:   Compiler error(s) encountered processing expression "DropBuild AndAlso BuildDetail.Reason = Microsoft.TeamFoundation.Build.Client.BuildReason.ValidateShelveset".
  Type 'IBuildDetail' is not defined.
• If you get any errors like these, the XAML file probably still contains some versioned namespaces.
• TF215097: An error occurred while initializing a build for build definition \Project1 (Dev10)\Moderate - Manual: The root element of the build process template found at $/Project1 (Dev10)/BuildProcessTemplates/ModerateBuildProcessTemplate.xaml (version C31) is not valid. (The build process failed validation. Details:
  Validation Error: Compiler error(s) encountered processing expression "New Microsoft.TeamFoundation.Build.Workflow.Activities.BuildSettings()".
  'BuildSettings' is ambiguous in the namespace 'Microsoft.TeamFoundation.Build.Workflow.Activities'.
  Validation Error: Compiler error(s) encountered processing expression "Microsoft.TeamFoundation.Build.Workflow.Activities.CleanWorkspaceOption.All".
  'CleanWorkspaceOption' is ambiguous in the namespace 'Microsoft.TeamFoundation.Build.Workflow.Activities'.