MSBuild Team Blog : Futures

Recursion, and ItemGroups inside Targets

msbuild — Fri, 13 Apr 2007 21:22:00 GMT

Greetings MSBuilders!

An internal team (Office Live, I think) asked us how to make their build start up faster. Their traversal project at the root of their tree started like this:

...

The way they had this set up, was their developers to run a shared batch file at any point in the tree. It would set an environment variable BuildPath to the current directory, and then build this shared traversal project. The traversal project would find every project below the current directory, and build it. This allowed them to avoid putting traversal projects at every point in the tree and remember to add new projects to them. But the wildcard evaluation was slow.

They changed it to this

<Target Name="FindProjects">
      <ItemGroup>
       <DotNetProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.csproj'" />
       <NativeProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.cppproj'" />
       <WebProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.webproj'" />
       <WixProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.wixproj'" />
       <LocProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.locproj'" />
      </ItemGroup>
   </Target>

   ...

With the old approach, this took 12.6s to list all projects in the tree to the console. With the new approach target it takes 3.1s to list all projects in the tree. Of course, most of the time, the build starts lower in the tree, so the time taken is smaller. I wonder if they could have reduced the ExcludedDirectory line to just one line to make it faster still?

This example illustrates a new feature you'll find in Orcas Beta 1 -- ItemGroup and PropertyGroup are now allowed inside Targets. (In MSBuild v2.0 you would have to use CreateItem and CreateProperty instead, which were ugly and inconsistent.) I moved the ItemGroup inside the Target here, so that I could use batching to exclude specific extensions from each list.

I'll be blogging in detail about this new syntax in a future post.

Have fun!
[ Author: Dan, MSBuild developer ]

Update: They couldn't have a single ExcludedDirectory line as I suggested above, because one directory in that tree (Common\Platform) needed to be included. So they now do this:

Which takes it down from 11 to 3 traversals.

Multi-Targeting : How does it work?

msbuild — Wed, 15 Nov 2006 04:30:00 GMT

So in my last post, I described the multi-targeting feature at a very high level. I discussed how there will be three frameworks that you can build "for", and how there will be two toolsets - i.e. .NET Framework 2.0 / MSBuild 2.0 toolset, and the .NET Framework 3.5 / MSBuild 3.5 toolset.

Recall that the capability we have added to MSBuild as a part of this feature is the ability for you to take a project, and build it using either toolset. MSBuild 3.5 supports a new command line parameter known as ToolsVersion that allows you to specify which toolset you want to build using, and is invoked as shown:

msbuild WindowsApplication1.csproj /ToolsVersion:3.5

ToolsVersion is the parameter that you use to force a project to build using a specific toolset. There are only two toolsets available out of the box, but you can also set up "custom" toolsets depending on your needs, and MSBuild can be used to build projects using those that you have defined. More on that later.

So, the next question obviously is, what happens when you build a project using a specific ToolsVersion, and what are the mechanics in action behind the scenes?

To understand this, we have to know a bit about how our project files are described. A standard Visual Studio managed project includes an import statement like the one shown below that is responsible for pulling in all the MSBuild / managed code build process into the project:

In MSBuild 2.0 (Visual Studio 2005 release), $(MSBuildBinPath) evaluated to the location of MSBuild 2.0, i.e. the .NET Framework 2.0 install path. In order to import the new version of Microsoft.CSharp.targets, naturally this needs to change to location of Microsoft.CSharp.targets that ship along with MSBuild 3.5. So in essence, specifying a ToolsVersion value at the command line is essentially equivalent to choosing from a predefined set of valid values for $(MSBuildBinPath). So specifying a ToolsVersion of 2.0 will cause MSBuildBinPath to evaluate to the .NET Framework 2.0 install path, and 3.5 will cause it to evaluate to the .NET Framework 3.5 install path, etc. Additionally, the targets that ship in MSBuild 3.5 include enhancements to the build process that will cause it to use the new set of compilers and other tasks.

MSBuildBinPath has also been "deprecated" in favor of MSBuildToolsPath - new projects created will use MSBuildToolsPath since that is a better name for what the property represents.

Allowing you to specify which toolset to use is only half the story. Multi-Targeting is also about building your code "targeting" a specific version of the framework - for instance, you might want to build your app only so that it relies on 2.0 assemblies and not on any 3.0 or 3.5 assemblies. This is done using another lever that is available via your project file. You can include a property in your project file known as TargetFrameworkVersion that specifies which target framework you are building your application for. TargetFrameworkVersion can either be v2.0, v3.0 or v3.5 - this means that if you are referencing assemblies that are not in a given TargetFrameworkVersion, then you will get errors/warnings at build time that will tell you so. This prevents you from taking on dependencies that will surprise you when you later turn around and try to deploy your app in a more tightly controlled environment that might only have a smaller subset of the three frameworks installed. Since TargetFrameworkVersion is a new concept, you need to be using ToolsVersion 3.5 in order to succefully build projects while honoring the TargetFrameworkVersion property.

This is a confusing topic, so please feel free to ask questions, and provide feedback so that we can help clarify anything on the subject.

[ Author : Faisal Mohamood ]

New Feature Feedback Request: /IgnoreProjectExtensions - A new command-line switch

msbuild — Fri, 24 Mar 2006 04:53:00 GMT

I've been in a cave just getting things done toward our Orcas release. But there's not much a Program Manager can do without going back to their customers :) So here I am.

So here's the scoop. We've been debating internally about a new command line switch on msbuild.exe for sometime now. I am not certain that we've come to the right answer ourselves - but customers give us the final word - so tell us what you think about the following:

Today, when you invoke msbuild.exe from the command line and don't specify any project files as arguments, then we do some auto inferral and scanning and decide if we should build anything. If we find either a msbuild project (anything that has an extension of *proj) or a solution file (.sln), we will build either the project or the solution as long as there is only one solution or one project in the directory. If there is a solution and a project, we will give preference to the solution. If there's more than one project or more than one solution, we issue an error message because we can't decide which one to build.

Now, what if you wanted to pick the project instead when a solution is encountered, but didn't want to specify the project name as an argument? Old habits die hard - when building Visual Studio code, we are used to issuing a single command to build the current project instead of having to specify the project name that we are specifying. So the only way to get this to work was to think of supporting a /IgnoreProjectExtensions switch that has the following usage model:

msbuild /IgnoreProjectExtensions:vcproj;sln

Now how exactly does this simplify our lives, you ask? We would use a .rsp file that sits next to msbuild.exe that we will add in our environment - after which, simply issuing msbuild will ignore what we ask it to ignore in the solution file.

What do you think? Something you'd find useful? Or would you rather prefer the "clean" command line interface that we have today, and don't think this switch is worth it? The internal debate is about the fact that we don't want to add command line switch after command line switch to msbuild.exe as we please unless there is a specific customer scenario that we are trying to enable. But DevDiv build is also one of our biggest internal customers :) So we look to you to tell us!

Cheers.

[Author: Faisal Mohamood]

Sprint 12 review

msbuild — Thu, 02 Feb 2006 01:31:00 GMT

This morning we had our review of work done during Sprint 12. As you hopefully recall we run two sprints simultaneously: one focused on our conversion effort and one focused on platform investments.

Sumedh got stuck in traffic on 520, so Faisal stepped in to drive the proceedings. Faisal handled it well, applause applause. And now, on to the review details...

Converting Visual Studio Builds (conversion): We converted three more trees, with one more currently under code review (we can't call the one under code review done because for us "done" means checked-in). At this point every team in Visual Studio is either done or in the process of converting except one.

Tool for Infering Project-to-Project References (conversion): As part of our conversion effort we want to ensure that all dependencies between trees are explicitly stated through project-to-project (P2P) references. Dan worked on a tool that will go and automatically add P2P references to converted projects. He gave a demo during the review, and it was super sweet!

Multi-Proc Development Work (platform): Vlad and Sumedh spent most of the sprint working on implementing the beginnings of multi-processor support in the engine. They did a demo of what they have working so far, and it's so cool to see the beginnings showing up. We're a long way from having something that's useful outside of concocted projects, but progress is being made. Here's a picture of what the demo looked like:

Fixing Bugs: This wasn't part of either sprint, but was something we had to do as part of a larger effort by Visual Studio. We need to keep our bugs at or near zero, and since we've been receiving new bugs through the MSDN Product Feedback Center we had some work to do.

[ Author: Neil Enns ]

Platform Sprint 11

msbuild — Thu, 05 Jan 2006 22:29:00 GMT

The new year got off to a good start for us with the MSBuild Platform Sprint 11 Backlog selection meeting. Platform Sprint 10 was focused mostly on getting our Compatibility story in place. All of the work from the last sprint in terms of compatibility has been checked in and is now in use.

This means that we can finally get to the good stuff in Platform Sprint 11. Our sprint goal for this sprint is to deliver a working version of multi-proc build capability. Needless to say, everyone on the sprint is excited about this and we are looking forward to the end of the month to see where we stand. I believe that we will have good parts of the multi-proc functionality working at the end of the sprint, and here's why:

Over the past sprints, we have been thinking a lot about the design of the system - like we discussed here and here. It is time now to start thinking in terms of real implementation milestones - and to that end we decided to think strictly in terms of "User Stories". So, everything that has to do with multi-proc that went on our backlog were real stories that would make sense to you from a customer standpoint, as opposed to items that are more closely aligned with the design or the implementation of the system. For instance, the simplest user story in our case was "Build two simple, independent projects in parallel, with no logging support, no disk-bound tasks". You can think of the simple projects as those that have a single target, and a single Message task. That really is our first customer story for multi-proc. We build successively on individual stories till we eventually get to a fully featured multi-proc build functionality.

Of course there are a lot of other activities going on in this sprint as well. We are working on handful of bugs that we could not get to during the tail end of Whidbey. We are also investing in QA work so that we are able to deliver at an even higher level of quality during Orcas.

I couldn't have asked for a better start with the new year. I am highly optimistic of everything that is to come in the following months. In the meantime, I encourage you to log any issues you run into with MSBuild via the MSDN Product Feedback Center so that we can consider them as bug fixes during the current Platform Sprint 11, or the subsequent one.

[ Author: Faisal Mohamood ]

Sprint 10 Review

msbuild — Fri, 09 Dec 2005 20:55:00 GMT

Today we're holding our Sprint 10 review. If you recall we do sprints for both dogfooding support and platform work. Here's a recap of what we accomplished in the last month.

Dogfooding

Six more teams within Visual Studio completed converting to MSBuild
We completed our first integration of all the converted branches into the parent branch of our build process. If this sounds messy, that's because it is *grin*.
Prefix support is checked in!
Our dogfood builds now support IDL to TLB to meta assembly generation
Proper support for build phases natively with MSBuild .targets files (this was all done with pair programming)
Cleaned up build traversal and clean logic
Documentation for Managed C++ builds
Unit tests are now in place for dogfood .targets files

Platform

All compatibility tests are written! This includes targets, shipped tasks and loggers, source code/APIs, command-line build, and solution builds. Since this is all we set out to do for platform in this sprint, I'd say we did pretty well.
We started to engage with other teams such as FxCop, VB, and TeamBuild so we can understand our compatibility testing requirements there.

Sprint 11 will start in January when everyone is back from vacation.

[ Author: Neil Enns ]

Sprint 9 Review

msbuild — Fri, 11 Nov 2005 22:43:00 GMT

It’s been about a month since we last talked about Sprint 9, and it’s time for an update. If you recall, the MSBuild team was planning on two parallel sprints: one for platform investigations and one to support internal dogfooding efforts. Last Friday marked the end of the sprints, and on Monday we held our sprint review. Here’s a recap of what we accomplished.

Dogfooding Sprint

Managed C++ Targets: We completed the authoring of .targets files for building Managed C++ code that makes up Visual Studio, including support for managed resources and asmmeta generation.

Conversion Support: A large part of the sprint was supporting a very active internal conversion alias. With so many teams making forward progress on converting their builds to use MSBuild we wound up answering a lot of questions. We now have approximately 40% of Visual Studio building with MSBuild.

Platform Sprint

Multi-proc scenarios: We agreed on our key scenarios for enabling multi-processor support.

Core design docs: Design documents were written and reviewed for many of the core pieces of our multi-processor support.

Dogfooding roadmap: Since dogfooding MSBuild technology is critical to what we do, we’ve already prepared a plan for how we’ll roll out multi-processor support to our build lab once it’s usable.

Partner engagement: We met with a wide range of teams to discuss our multi-processor work, including teams that want the functionality as well as teams that have technologies that could help us in our implementation. As part of this engagement we also posted to our blog about some of our Visual Studio integration concerns.

Compatibility tests: We successfully reviewed and signed off on our compatibility test plan, and wrote compatibility tests for project, binary, and add-in compatibility verification. We also moved numerous test suites from our old test system to live directly in our developer code branches. This makes our tests vastly more portable between machines, and simplifies running them in conjunction with unit tests.

That’s a quick wrap-up of what happened in the last month. Our next sprints will also run in parallel, and planning starts today. The sprints start next Monday, the 14th, and we’ll post the backlog as soon as we have it.

[ Author: Neil Enns ]

Parallelizing Visual Studio solutions

msbuild — Thu, 10 Nov 2005 01:50:00 GMT

One of the MSBuild Team’s scenarios for multi-proc builds is the automatic parallelization of Visual Studio solutions. For this scenario, we want Visual Studio to automatically parallelize all “independent” projects, and serialize all “dependent” ones.

The key here is the separation of projects into the “independent” and “dependent” buckets. The separation is entirely based on project-to-project references and the ordering of projects in the “Project Dependencies” dialog. If the dependencies are not set up correctly, Visual Studio may end up building some dependent projects in parallel and thus break the build.

Visual C++ (VC) projects in Visual Studio 2005 have already adopted the “parallel by default” model. For any solution that contains VC projects, Visual Studio automatically builds the independent projects in parallel. For consistency with VC projects, we are thinking of adopting the same model for managed projects (C#, VB and J#), and reusing the same Visual Studio mechanisms for parallelism.

It is not clear to us, however, how many solutions this will negatively affect. If VS 2005 solutions that contain managed projects do not have their project dependencies set up correctly, when these solutions are opened in the next version of VS, their builds will break when they are automatically built in parallel. While this would not be ideal, the fix, of course, would be to update the project dependencies.

There are two ways to add project dependencies to the solution: project-to-project references (http://msdn2.microsoft.com/en-us/library/7314433t.aspx), or project build dependencies (http://msdn2.microsoft.com/en-us/library/et61xzb3). Setting up a project build dependency ensures the dependent project is built after the projects it depends on. Adding a project-to-project reference additionally passes the output assembly of the referenced project as a reference to the compiler while building the referencing project. (Note that VS 2005 allows adding a project-to-project reference to a project with an exe output.)

We are planning to use more formal means to gather data on real-world solutions, but in the meantime, we would appreciate your thoughts on this issue. You can add a comment to this blog entry, or send email to msbuild@microsoft.com.

[ Author: Sumedh Kanetkar ]

Parallel builds scenarios and implementation ideas

msbuild — Fri, 14 Oct 2005 21:19:00 GMT

The MSBuild Team has started thinking about adding multi-proc support to the MSBuild engine. Currently MSBuild is single-threaded and does not take advantage of any opportunities for parallel processing during a build. However, most builds inherently have chunks of work that can be done concurrently. By parallelizing these independent chunks of work we aim to reduce the total build time for all builds, large and small.

To enable parallel builds, we believe we need to support the following scenarios:

Ability to build a large tree of projects in parallel – we call this the “build lab” scenario. In this scenario multiple projects are built concurrently.
Ability to build a Visual Studio solution in parallel. Solutions in Visual Studio can contain multiple projects, and even for simple solutions we should be able to support parallel builds. In this scenario, the independent projects are built concurrently, while the dependent ones are serialized.
Ability to build multiple files in a project concurrently. Some projects have lots of files that need to be operated on by a single tool one-by-one. In this scenario, the tool is invoked concurrently on all the files.

We currently believe that scenarios (1) and (2) would provide the “biggest bang for the buck” in build performance. However, MSBuild is a general build orchestration engine and we know some of our customers build things other than source code. For many of these customers, supporting (3) would offer them significant benefits. This is particularly the case if the build calls into older tools that don’t support multi-processor machines natively. As a result, we believe we also need to look into supporting the parallelization of tasks within a build.

In addition to thinking about scenarios, we're working on implementation designs (as you can see from the above picture!). On the implementation side the build mechanism we are considering is completely declarative. The MSBuild engine will not make any attempt to determine what can be parallelized. Instead, we will instead introduce parallelization “constructs” in the file format that will allow a project or targets file author to indicate what can and cannot be parallelized.

For Visual Studio solutions, the parallelization will be automatic in a sense. The target files we will ship will allow all independent projects in a solution to be parallelized. As long as the dependencies are correctly expressed through project-to-project references, or through the “Project Dependencies” dialog, the build will parallelize correctly.

Now is your chance to give us feedback! What do your builds look like? How would the above three points of parallelization improve your build performance? What do you think about our plans to introduce new construcuts? Let us know by replying to this post, or send us email at msbuild@microsoft.com.

[ Author: Sumedh Kanetkar ]