MSBuild Team Blog : multiproc

How would you spend $100 on MSBuild?

msbuild — Sat, 17 Nov 2007 21:43:00 GMT

We're currently planning for our next version (aka, "Dev10" - no code name this time) and subsequent releases. In that spirit, I'd like to do a quick poll of MSBuild aficionados to help us keep our "vision" for MSBuild aligned with yours, our customers.

First, a caveat - while ideally I would like to rank the results and work down the list, the reality is that we have other constraints: although we are a separate build platform, our biggest customer by far is Visual Studio and its customers. There's a lot more Visual Studio users than direct MSBuild users, so when Visual Studio needs to add a feature that requires work on MSBuild, we have to help. Then there's other constraints, like team resources. There's no guarantee which of these will appear in what version, and some of them may not be worth any investment. Some of them I just made up.

OK. Here’s some brainstormed features. Some of them are properly Visual Studio features, but they overlap closely with build:

1) Higher performance multiprocessor support. For example, we suspect there is plenty of room to improve the scheduling we do, and find speedups elsewhere in the code. As more and more of us have multicore machines, this might be a good place to invest.

2) VC support. This means converting VC projects (.vcproj) to MSBuild format, a customizable and extensible build process entirely defined by MSBuild .targets files rather than in makefiles or built into the VC project system, reuseable tasks for native code tools just like we have today for managed code, changing the VS project system for VC projects to sit on top of MSBuild format projects and build process, and replacing use of vcbuild.exe with msbuild.exe.

3) Support for other Microsoft project types that aren't yet in MSBuild format: Deployment/MSI (.vdproj), SQL Reporting (.rptproj), Biztalk (.btproj), Speech server (.prproj) etc -- whether currently supported by a VS project system or not. This is essentially a process of internal evangelization and encouragement for us.

4) Conversion of Visual Studio solution files (.sln files) and their (rudimentary) build process to MSBuild format and the VS support for reading and writing these, opening the way to create a targets file useful for traversing a tree of projects, and to let VS cleanly support n-level project hierarchies.

5) Extensible up-to-date checking: a way to plug in up-to-date checker extensions that you could use on selected Targets as an alternative to the simple timestamp checking you are currently restricted to. Perhaps including a ready-made extension that would by some means automatically support transitive dependencies - header files are an example of these; or to compare public interfaces for significant changes.

6) Distributed Build. (Like multiprocessor build we are now shipping, but building spread over a set of machines that you have pre-provisioned; possibly opening the way for future Team Build support for it too.)

7) Extensible reuseable inline tasks. This means the ability to create extensions that consume the language or description format of your choice and to create a task that you could use without explicit compilation or deployment. For example, a Powershell extension that you could use to create tasks implemented in Powershell. Think of putting script underneath a <UsingTask> element. These inline tasks would be easy to create and share online. Might include ready-made extensions for Powershell, and/or a data-description format for tool switches, so you could quickly create a task wrapping a command line tool without any compilation.

8) Typing and scoping for items and properties. For example, declaring an item to contain a path; letting a property fall out of scope at the end of a target.

9) Extensible functions. Today we have only 'exists(..)' and it's stepchild 'hastrailingslash(...)'. This would be an extension allowing new functions to be created for use in conditional expressions, and possibly elsewhere, and some functions like perhaps combinepath($(root),$(file)). Perhaps more built-in metadata to go along with %(Filename) and such.

11) An MSBuild debugger with full Visual Studio support for stepping, inspecting locals, and breakpoints. Today you have to use <Message> tasks and gaze at diagnostic logging output.

12) Visualization for project and target dependencies, mining of what files are consumed by what projects, impact analysis for changes to particular files, possibly leading to support for build refactoring at the project/tree level. This is essentially richer logging and datamining and a high level on a huge source tree.

So now here's the poll. If you had only $100 to spend, how would you distribute it amongst them?

Dan

(Edit: gave Distributed build a unique number :-) )

Enabling multiprocessor support in an MSBuild host

msbuild — Mon, 22 Oct 2007 02:29:00 GMT

As you know, MSBuild in .NET 3.5 adds support for building projects concurrently. MSBuild.exe exposes this support with the new /m switch, and because Team Build uses MSBuild to build projects, it will get a speed up as well. In this release, Visual Studio doesn't use this to build managed projects concurrently, but you can write your own host that can build in parallel.

There are some limitations. In 3.5, you can't build in-memory projects concurrently, only project files. Also, I don't like the API much. Msbuild.exe was the only host we were focusing on in 3.5 and we didn't make it pretty. That will change.

Anyway, here's an example. It assumes you have a sleep.exe in the same directory as the projects. So "sleep 3" will pause for 3 seconds. Our objective here is to run two sleeps concurrently :-)

using System;
using Microsoft.Build.BuildEngine;
using Microsoft.Build.Framework;
using System.Reflection;

namespace HostMsBuildExeTest
{
    class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            // We need to tell MSBuild where msbuild.exe is, so it can launch child nodes
            string parameters = @"MSBUILDLOCATION=" + System.Environment.GetFolderPath(System.Environment.SpecialFolder.System) + @"\..\Microsoft.NET\Framework\v3.5";

            // We need to tell MSBuild whether nodes should hang around for 60 seconds after the build is done in case they are needed again
            bool nodeReuse = true; // e.g.
            if (!nodeReuse)
            {
                parameters += ";NODEREUSE=false";
            }

// We need to tell MSBuild the maximum number of nodes to use. It is usually fastest to pick about the same number as you have CPU cores
int maxNodeCount = 3; // e.g.

// Create the engine with this information
Engine buildEngine = new Engine(null, ToolsetDefinitionLocations.Registry | ToolsetDefinitionLocations.ConfigurationFile, maxNodeCount, parameters);

            // Create a file logger with a matching forwarding logger, e.g.
            FileLogger fileLogger = new FileLogger();
            fileLogger.Verbosity = LoggerVerbosity.Detailed;
            Assembly engineAssembly = Assembly.GetAssembly(typeof(Engine));
            string loggerAssemblyName = engineAssembly.GetName().FullName;
            LoggerDescription fileLoggerForwardingLoggerDescription = new LoggerDescription("Microsoft.Build.BuildEngine.ConfigurableForwardingLogger", loggerAssemblyName, null, String.Empty, LoggerVerbosity.Detailed);

            // Create a regular console logger too, e.g.
            ConsoleLogger logger = new ConsoleLogger();
            logger.Verbosity = LoggerVerbosity.Normal;

            // Register all of these loggers
            buildEngine.RegisterDistributedLogger(fileLogger, fileLoggerForwardingLoggerDescription);
            buildEngine.RegisterLogger(logger);

// Do a build
buildEngine.BuildProjectFile("root.proj");

            // Finish cleanly
            buildEngine.Shutdown();
        }
    }
}

Now for testing purposes, I created three projects like this:

root.proj
Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
<Target Name="t">
    <Message Importance="high" Text="## in root building children ##"/>
    <MSBuild Projects="1.csproj;2.csproj" BuildInParallel="true"/>
    <Message Importance="high" Text="## in root done building ##"/>
</Target>
</Project>

1.csproj

2.csproj

Notice that the root project uses the <MSBuild> task with the parameter BuildInParallel="true". This is where the parallelism starts: you still only invoke the build on a single root project. Without BuildInParallel="true", the <MSBuild> task will build serially. In a tree of projects, you would want every node to build its children with BuildInParallel="true".

Here's the output I get:

c:\test>test.exe
Build started 10/21/2007 5:02:55 PM.
     1>Project "c:\test\projects\root.proj" on node 0 (defaul
       t targets).
         ## in root building children ##
     1>Project "c:\test\projects\root.proj" (1) is building "
       c:\test\projects\1.csproj" (2) on node 1 (default tar
       gets).
         ## starting 1 ##
     1>Project "c:\test\projects\root.proj" (1) is building "
       c:\test\projects\2.csproj" (3) on node 2 (default tar
       gets).
         ## starting 2 ##
     1>t:
         ## in root done building ##
     1>Done Building Project "c:\test\projects\root.proj" (de
       fault targets).
     3>t:
         ## finishing 2 ##
     3>Done Building Project "c:\test\projects\2.proj" (de
       fault targets).
     2>t:
         ## finishing 1 ##
     2>Done Building Project "c:\test\projects\1.proj" (de
       fault targets).

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:03.21

As you can see, the two 3-second sleeps ran concurrently, so the build took 3.21 seconds overall.

Just to prove it I changed the "3" to "1" and got this:

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:06.11

There's lots more to say about multiproc support, especially to explain the types of loggers I attached here, and that will be the subject of future posts. Even if you don't host MSBuild, you may want to create "traversal" projects (something like solution files, but nestable) that build their children in parallel. I'll explain that too in due course.

Just to be clear though, for most of us, it isn't necessary to know any of this. We can just point msbuild.exe at our solution file and throw the "/m" switch and our build will be faster.

Dan