Incorrect solution build ordering when using MSBuild.exe

Incorrect solution build ordering when using MSBuild.exe

  • Comments 20

UPDATE: This issue has been fixed in Visual Studio 11/NETFX45. As always, feedback is welcome! Please leave your comments in this blog post and report any bugs on Microsoft Connect.

 

We've had a few reports of cases where Visual Studio, and previous versions of MSBuild, will build the projects in the solution in the correct order, but the 4.0 version of MSBuild.exe gets the order wrong.

Here's a full description of what's going on, why it began in 4.0, and the fix we recommend to your projects to solve the problem. If you're not interested in the "why", skip ahead to the workaround.

Archetypical case exhibiting the problem

dep1b

This diagram shows a solution file containing three projects, A, B, and C. Let's say they are C# projects.

A has a regular project reference to B, so it will invoke B, then when B comes back done, it will build itself. At the same time in the solution file, there is a manually specified dependency: B depends on C.

I've shown regular project references with solid lines, and the information in the solution with dotted lines.

This manually specified dependency was set up in the solution by right clicking on the solution and choosing Project Dependencies…, then checking a box. Below I've shown the context menu and what you see in the dialog when you have this setup.

The build ordering you expect here is C, then B, then A, and Visual Studio shows that correctly in the Build Order tab, as you see below.

Of course a real case would have more projects in it, but it would boil down to this case.

 

image

image image

Why does this happen (skip ahead if you just want the "fix")

Essentially the problem is that MSBuild doesn't know anything about the project files until it starts to build them.

Solution files, as you know, are not in MSBuild format (yet). On the command line, MSBuild.exe is on its own, so it parses them and generates one or more in-memory MSBuild format files that are essentially a translation. If you want to see these ugly files, set an environment variable MSBUILDEMITSOLUTION=1 then build the solution. You'll see a .sln.metaproj file emitted next to your solution file, and possibly one or more files with an extension like .csproj.metaproj next to some of your projects.

The .metaproj generated for B.csproj is how MSBuild makes sure that the solution dependency is respected -- at least, it's created so that the solution itself does not invoke B until C is built. It does this by invoking the B metaproj instead of B directly, and in the B metaproj, it builds C before B. This is exactly equivalent to someone going into B and adding a project reference to C, instead of a solution dependency, but it means that we don't have to edit the B project directly.

Here's what it looks like after this translation:

dep2

Why doesn't that work in this case? In short, the problem is the project reference from A to B. Here's what happens: the solution invokes A.csproj, B.metaproj, and C.csproj concurrently (or at least, in undefined order), which would normally be fine. B.metaproj invokes C.csproj, and waits, then invokes B.csproj. However in the meantime, A.csproj was invoked, and because it has a project reference to B.csproj, it invokes B.csproj -- it "goes around the back". C.csproj hasn't necessarily built yet, so the build breaks.

Why did this work in previous versions of MSBuild?

In previous versions, we loaded and scanned every project file listed in the solution, and any they referenced, in order to draw a complete graph. Then we used the graph to create the MSBuild format equivalent of the solution file. The reason we did all this scanning was not actually to address this problem, it was to make interop with old-style non-MSBuild VC projects (".vcproj") work correctly. It was also slow, especially for large solutions.

In VS2010, VC projects converted to MSBuild, so in 4.0 we took out this complex interop code. After making .metaproj's to express any dependencies stored in the solution file, we could now simply invoke all the projects in the solution and the build would order itself. That was potentially much faster, because we didn't need to load any projects (potentially hundreds) to scan them before building anything. (Of course, when MSBuild 4.0 is fed a VS2005 or VS2008 solution file, it still calls into the old code in the old assembly to do it the old way, since they may contain .vcproj's. So those guys don't have this problem.) The oversight was this case -- where a project reference "goes behind the back" of a solution-expressed dependency.

To fix this we would have to revert to loading and scanning, which slows things down -- the correct approach is to use project references instead of solution dependencies, as I explain below.

How to fix this

Follow this principle: do not use dependencies expressed in the solution file at all! Better to express dependencies in the file that has the dependency: put a project reference in the project, instead. In our example, that would be a project reference from B to C.

You may not have done that before because you didn't want to reference the target of the project reference, but merely order the build. However, in 4.0 you can create a project reference that only orders the build without adding a reference. It would look like this - note the metadata element, and all this is inside an <ItemGroup> tag of course:

<ProjectReference Include="... foo.csproj">
    <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>

Note that you have to add the child element with a text editor -- Visual Studio can add a project reference, but doesn't expose UI for this metadata.

I can tidy up by removing the dependency in the solution file as well - removing now-unnecessary lines like this -- your GUID will be different, but use the VS dialog and it will do the job.

    ProjectSection(ProjectDependencies) = postProject
        {B79CE0B0-565B-4BC5-8D28-8463A05F0EDC} = {B79CE0B0-565B-4BC5-8D28-8463A05F0EDC}

    EndProjectSection

If you're using C++ projects, you are less likely to have this problem, because in the upgrade process that converts .vcproj's to .vcxproj's, it moves any solution dependencies relating to them to project references for you. However, if you do, there's a similar fix. For C++/CLI project references to other managed projects, use project references like the one above. For the equivalent situation with a project reference to a static lib, where you want a project reference without linking in the referenced lib, the metadata you want is

<ProjectReference Include="... lib.vcxproj">
    <LinkLibraryDependencies>false</LinkLibraryDependencies>
</ProjectReference>

Summary

Although it's tiresome to have to edit your projects in this way to make the bug go away, it's a best practice to use project references instead and consider the solution file merely a "view", and you'll end up with projects that if you want can be built without a solution file.

Post Script

I know of one other, more obscure and completely different case, where MSBuild 4.0 does not order correctly but Visual Studio does. This can happen if you have web application projects, AND you build with 64 bit MSBuild (which is the default, in Team Build 2010). I won't go into the tedious details but the fix is to do one of these things: (1) set a property or environment variable named MSBuildExtensionsPath to C:\program files (x86)\msbuild or (2) Build with 32 bit MSBuild, which I recommend in general for other reasons or (3) copy the web application targets files under the 64 bit program files MSBuild folder to the equivalent location in the 32 bit program files MSBuild folder.

If you find any other case where the solution is not ordering correctly yet this workaround does not work, that's interesting. Please make a minimal repro like the one in this bug, and send it to Dan at msbuild@microsoft.com.

Dan

Update (12/25)

Luc C points out below that sometimes removing a project reference in favor of a new solution dependency reference is an alternative solution. That assumes you're only using the project reference for ordering , or you replace it with a file reference. Still, I do recommend project references in general, on the principle of "express the dependency in the place it applies" – you can look at the project and see that the dependency is correct, and you can include the projects in more than one solution easily.

Luc also points out the case of a managed project depending on a non-CLR native project (presumably for PInvoke). In my experiments, VS will let you add a project reference, albeit with an ugly bang, and it will do the ordering correctly, as will msbuild.exe.

Leave a Comment
  • Please add 3 and 7 and type the answer here:
  • Post
  • What about the issues with batch building which does not respect Debug and Release configuration for builds? E.g. when batch building in some cases Debug dlls may end up in the release output directory!

  • I think that "what" is always important.

  • Dear Dan,

    We tried this approach (with <ReferenceOutputAssembly>false</ReferenceOutputAssembly>) few months ago.

    It works well with C++ projects, but in case of C# projects VS shows exclamation mark on the project reference. And, AFAIR, we had some strange compilation behavior periodically.  

  • Project dependencies often weren't respected correctly in MsBuild versions before 4.0 either.

    social.msdn.microsoft.com/.../80cc6447-b720-4806-8395-7c257b207613

  • There is a problem with your proposed solution of adding a project reference directly in the project as far as I can see.

    The one place I have been using the dependecies set in the solution file a lot is when C# code is P/Invoking into my own native DLLs - both the the C# project and the native code (C++ project) are projects in the solution, but the C++ DLL is not an assembly, just a plain old DLL, so cannot be referenced in C#. The only way to add this dependency (as far as I can see) is to add the dependency in the solution.

    In my scenario the native DLL (a C++ project) is the bubble 'C' in your diagram, and the C# library using it is bubble 'B'. I bumped into the problem described into this post when building 'A': an application using the libray 'B' and therefor indirectly also 'C'. The way I fixed this was by adding a dependency of 'A' on 'C' in the solution.

  • @Harry -- batch build has a bug (I believe this has been true for a while). It's a VS bug, unrelated to this. The workaround I recommend is to build on the command line with either MSBuild or devenv.exe /build using a batch file. That has the advantage of not tying up VS while it runs, as well.

    @Sergey D -- the bang is a visual issue only, as far as I know. I'm not aware of the 'strange compilation issue' but if you can narrow it down please open a Connect issue.

    @Luc C -- (1) I just tried this in VS2010 and it does let me add a reference to a non-CLR VC project - albeit with the bang - and does respect it both in VS and with msbuild.exe (as an ordering-only reference). (2) Good point -- I'll update the post to point out that sometimes adding more solution-based dependencies is an alternative.

  • Dan, you mention a "batch build" bug, but there is also a "clean" bug that might be related:

    connect.microsoft.com/.../rebuild-solution-should-do-all-cleaning-before-any-building

  • Dan,

    Is there a workaround for a Silverlight project that requires a non-Silverlight project to be built first?

  • @Matt, Yes there is. I can't seem to get the blog to let me edit it right now but here it is:

    Put this in the projct just inside the </Project> tag

    <Target Name="MyProjectReference">

    <ItemGroup>

      <ProjectReference Include="…what you had here..."/>

    </ItemGroup>

    </Target>

    Then, edit the root <Project> tag to add an attribute InitialTargets="MyProjectReference"

    This makes it still work, but hides it from VS. I hope we can fix this in the next version.

    Dan

  • @Matt, Yes there is. I can't seem to get the blog to let me edit it right now but here it is:

    Put this in the projct just inside the </Project> tag

    <Target Name="MyProjectReference">

    <ItemGroup>

      <ProjectReference Include="…what you had here..."/>

    </ItemGroup>

    </Target>

    Then, edit the root <Project> tag to add an attribute InitialTargets="MyProjectReference"

    This makes it still work, but hides it from VS. I hope we can fix this in the next version.

    Dan

  • The problem I have with this setup is that when you use a partitioned solution approach, you need to add all reference assemblies/projects to the satellite solutions.

    We prefer to use assembly references for the satellite solutions and one encompassing solution that contains all the project files from all the solutions.

    Currently this solution breaks on the buildserver and works great from within Visual Studio.

  • @Dan, I could only add such references to 2 class library projects, but couldn’t add it to the main pair of projects: Silverlight application and hosting ASP.NET Web Application. The reference broke the build with the message

    error MSB4006: There is a circular dependency in the target dependency graph involving target "CopySilverlightApplications".

    Although my solution now builds fine on the build server, the workaround doesn’t fit in all cases.

  • You might want to ensure you keep the reference output assembly set to false or it will try to bring it in and fail on other references missing:

    <Target Name="NonSilverlightProjectReferences">

     <ItemGroup>

       <ProjectReference Include="..\ CsharpProject.Server\CsharpProject.Server.csproj">

         <ReferenceOutputAssembly>false</ReferenceOutputAssembly>

       </ProjectReference>

     </ItemGroup>

    </Target>

  • I believe I've fixed this for the Beta release of 4.5. Essentially during the build of each project we will synthesize any missing necessary non-referencing project references.

    When you get the Beta, please consider giving this a try with your original project files, in order to confirm the fix is good.

    Thanks, Dan

  • Somewhat related to verifying the fix, but is there any way with devenv on the cmdline (or msbuild if not) to show the dependency graph but not actually execute them?  I'm using VS11 Beta if that makes a difference. :)

    Especially when trying to resolve things like circular dependencies, that would be very useful! :)

Page 1 of 2 (20 items) 12