jeffcal's blogland

rem koolhaas couldn't build better than us!

dependency analysis in msbuild

when people first get their hands on msbuild, especially those familiar with other build systems, one of the first things they notice is msbuild's dependency analysis mechanism.  the msbuild engine offers basic dependency checking based on explicitly declared target inputs and outputs.  currently, the analysis is pretty simple, although helpful in many cases:  msbuild checks output timestamps and input timestamps and if some of the outputs are older than some of the inputs (i'm glossing over details here), the engine will execute the target.

users of other build systems have pointed out a fairly obvious limitation of this mechanism:  dependency analysis often must be more complex than timestamp checking.  i couldn't agree more.

however, this limitation of msbuild's dependency checking mechanism should not be mistaken for a limitation of msbuild itself.  reason:  the author of a project file is not strictly limited to using the msbuild engine for her dependency checking, nor is she required to allow the engine to perform any dependency analysis at all!  furthermore, and most importantly, this mechanism (whether opting in or out) in no way precludes tasks from performing their own dependency analysis at any level of precision they'd like, exactly as many (N)Ant tasks do today.  the resgen task our team has developed does just this.  we found in some cases simple filestamp checking would lead to skipped calls to resgen when they were actually required, so we simply moved the dependency analysis from the target level to the task level and voila! problem solved.

all this still leaves room for the engine to do more complex analysis while still exposing more control over it to the user, and we're thinking hard about how to make that happen.

jeff.

 

Published Friday, May 28, 2004 9:58 PM by jeffcal
Filed under:

Comments

 

Stefan Bodewig said:

You are correct that nothing precludes task writers from using smarter strategies than the built-in one, see the last paragraph in my article you link to: "Like I previously said, you can create smart tasks in MSBuild as well."

The two major problem I see is that most task writers will use the target level checks intially and it will be hard to migrate away from that later.

How would I get the overwrite="true" functionality in the Copy task without rolling a Copy task of my own? It's quite possible that the task shipping with MSBuild already supports this (my docs are very old), but take this as an example.
June 1, 2004 4:45 AM
 

jeff said:

You did mention it before but I was worried some of what I'd read there and other places mischaracterized msbuild a bit. The main point which I wanted to make is that this absolutely is not a limitation of msbuild - in fact it's a free (albeit simple) dependency checking mechanism provided *in addition* to what tasks provide.

Hopefully this will be clear to msbuild task authors because you are right that is they who know best how dependency analysis should work in their domains. The difficulty in relying first on the engine's analysis and later discovering it's not enough doesn't lie in the project files (the change there is dead simple: remove the inputs and outputs from the target declaration) but in the code for the task.

Then the important thing for us, I suppose, is to communicate this to prospective task authors so they'll be sure to think about dependency analysis despite the engine's abilities. We've authored our shipping tasks with this thought in mind, and we need to suggest everyone else does too.

PS - For what it's worth, our Copy task does offer a SkipUnchangedFiles parameter today, and will soon support the OverwriteReadonlyFiles functionality you mentioned :)
June 1, 2004 11:42 AM
 

Stefan Bodewig said:

We are pretty much on the same page, Jeff.

Some early publications on MSBuild colored the MSBuild way of built-in dependency checking as superior, which may explain the length of my post (it's more than six months old by now 8-). Note that Ant offers the same timestamp checking logic (and more) to task writers via a Java API, so the service is available to task writers as a built-in service, they just have to use it explicitly as opposed to implicitly using the MSBuild approach.

One more thing to note is a backwards compatibility issue. Say you had shipped your resgen task without internal dependency checking in the first release of MSBuild. Later you discover that MSBuild's services are not enough and you change the task for the second release. This means that users will have to write version specific build files, right?

It may not be a big issue if you always think of MSBuild as a tool bundled with VS.NET and you can expect people to have access to a build-file migration wizard of some kind. But my understanding is, that MSBuild may be used and maybe even released stand-alone.

Ant has had its share of it-is-breaking-my-builds-with-every-new-release flamewars, so I've become very sensitive to this problem. 8-)
June 1, 2004 11:49 PM
 

Steve Loughran said:

Dependency logic is trouble. In some ways, declaring ins and outs is good. Especially in those cases where a task wrapper around 3rd party code doesn't know what the outputs will be.

examples, Ant's <ildasm>, <axis-wsdl2java>; both of these generate outputs that depend on the data processed, and that data isnt communicated up to the build engine.

Explicit declaration is a useful workaround, but what you really want is tools that tell the runtime what got updated.

The other troublespot is file system granularity -until FAT is turned off, Ant has to rely on 2s granularity on a DOS-derived box (I know, we could probe the filesystem and work it out on a directory by directory basis). That can cause confusion -especially as Zip files have the same problem.

The best dependency logic in Ant I know is the <cc> task set, which tracks all the inputs, compiler options and #define values for a build. That way, your classes get rebuild when the meta-dependencies -build options- change too.
July 9, 2004 12:00 PM
 

jeffcal s blogland dependency analysis in msbuild | Outdoor Ceiling Fans said:

May 31, 2009 5:55 AM
Anonymous comments are disabled

© 2009 Microsoft Corporation. All rights reserved. Terms of Use  |  Trademarks  |  Privacy Statement
Microsoft
Page view tracker