Parallelizing Visual Studio solutions

msbuild — Thu, 10 Nov 2005 01:50:00 GMT

One of the MSBuild Team’s scenarios for multi-proc builds is the automatic parallelization of Visual Studio solutions. For this scenario, we want Visual Studio to automatically parallelize all “independent” projects, and serialize all “dependent” ones.

The key here is the separation of projects into the “independent” and “dependent” buckets. The separation is entirely based on project-to-project references and the ordering of projects in the “Project Dependencies” dialog. If the dependencies are not set up correctly, Visual Studio may end up building some dependent projects in parallel and thus break the build.

Visual C++ (VC) projects in Visual Studio 2005 have already adopted the “parallel by default” model. For any solution that contains VC projects, Visual Studio automatically builds the independent projects in parallel. For consistency with VC projects, we are thinking of adopting the same model for managed projects (C#, VB and J#), and reusing the same Visual Studio mechanisms for parallelism.

It is not clear to us, however, how many solutions this will negatively affect. If VS 2005 solutions that contain managed projects do not have their project dependencies set up correctly, when these solutions are opened in the next version of VS, their builds will break when they are automatically built in parallel. While this would not be ideal, the fix, of course, would be to update the project dependencies.

There are two ways to add project dependencies to the solution: project-to-project references (http://msdn2.microsoft.com/en-us/library/7314433t.aspx), or project build dependencies (http://msdn2.microsoft.com/en-us/library/et61xzb3). Setting up a project build dependency ensures the dependent project is built after the projects it depends on. Adding a project-to-project reference additionally passes the output assembly of the referenced project as a reference to the compiler while building the referencing project. (Note that VS 2005 allows adding a project-to-project reference to a project with an exe output.)

We are planning to use more formal means to gather data on real-world solutions, but in the meantime, we would appreciate your thoughts on this issue. You can add a comment to this blog entry, or send email to msbuild@microsoft.com.

[ Author: Sumedh Kanetkar ]

Parallel builds scenarios and implementation ideas

msbuild — Fri, 14 Oct 2005 21:19:00 GMT

The MSBuild Team has started thinking about adding multi-proc support to the MSBuild engine. Currently MSBuild is single-threaded and does not take advantage of any opportunities for parallel processing during a build. However, most builds inherently have chunks of work that can be done concurrently. By parallelizing these independent chunks of work we aim to reduce the total build time for all builds, large and small.

To enable parallel builds, we believe we need to support the following scenarios:

Ability to build a large tree of projects in parallel – we call this the “build lab” scenario. In this scenario multiple projects are built concurrently.
Ability to build a Visual Studio solution in parallel. Solutions in Visual Studio can contain multiple projects, and even for simple solutions we should be able to support parallel builds. In this scenario, the independent projects are built concurrently, while the dependent ones are serialized.
Ability to build multiple files in a project concurrently. Some projects have lots of files that need to be operated on by a single tool one-by-one. In this scenario, the tool is invoked concurrently on all the files.

We currently believe that scenarios (1) and (2) would provide the “biggest bang for the buck” in build performance. However, MSBuild is a general build orchestration engine and we know some of our customers build things other than source code. For many of these customers, supporting (3) would offer them significant benefits. This is particularly the case if the build calls into older tools that don’t support multi-processor machines natively. As a result, we believe we also need to look into supporting the parallelization of tasks within a build.

In addition to thinking about scenarios, we're working on implementation designs (as you can see from the above picture!). On the implementation side the build mechanism we are considering is completely declarative. The MSBuild engine will not make any attempt to determine what can be parallelized. Instead, we will instead introduce parallelization “constructs” in the file format that will allow a project or targets file author to indicate what can and cannot be parallelized.

For Visual Studio solutions, the parallelization will be automatic in a sense. The target files we will ship will allow all independent projects in a solution to be parallelized. As long as the dependencies are correctly expressed through project-to-project references, or through the “Project Dependencies” dialog, the build will parallelize correctly.

Now is your chance to give us feedback! What do your builds look like? How would the above three points of parallelization improve your build performance? What do you think about our plans to introduce new construcuts? Let us know by replying to this post, or send us email at msbuild@microsoft.com.

[ Author: Sumedh Kanetkar ]

MSBuild Team Blog : Sumedh Kanetkar

Parallelizing Visual Studio solutions

Parallel builds scenarios and implementation ideas