Setup projects (.vdproj) do not have a project system format that can be understood by MSBuild. Hence MSBuild cannot build these projects directly and consequently neither can Team Build. What we can do however is use the devenv /build command to build the vdproj files and generate msi installers. As I set out to build setup projects and install them on the build machine in order to run tests on those, I followed the sequence of steps listed below:

Override the target AfterCompile in the TFSBuild.proj file. Include an Exec task in the target to build the setup project separately using devenv /build.

<Target Name="AfterCompile">
    <Exec Command="&quot;$(VSINSTALLDIR)\Common7\IDE\devenv&quot; ..\Sources\MySolution\MySetup\MySetup.vdproj /Build"/>
  </Target>

Now, if you kick off a build, the msi is built, but is dropped into the bin folder under the Sources folder in the build directory. But in order to get it into the drop location, you need to edit the "Output file name" property in the setup project properties to ..\..\..\Sources\ since you cannot pass the output directory to the command line. Now, when you kick off a build, you see the msi file in the Binaries folder on the build machine and consequently the drop location. The caveat is that you will need VS installed on the build machine if you want to build using devenv.

Invariably the problem arises that you would want to build the setup project in different flavors now. The solution to that one is slightly tricky. Remember, you cannot pass in the output directory on the command line, that is why you have to build the setup project in the Binaries folder and then include a task to copy the dlls to the appropriate flavor's folder. Then again, you can build the setup project in a different flavor and re copy the dlls into the appropriate folder. You can use the /ProjectConfig switch to specify the flavor in which the setup project is to be built.

Now that you have built setup projects, you want to do something else. Run tests on the installed product to pronounce it good or bad. Consider a solution that has a console application, a setup project and a test project. The setup project is built into an msi that dumps the created exe into a folder called say "D:\Program Files\MySetup". Now, the test project has a generic test that calls the console application's exe from the same folder. If the install has happened successfully on a machine, the test calls the console application exe and based on the behaviour may either pass or fail. But, in case the product is not even installed, then the test aborts saying that the exe is not found. So, now after the AfterCompile custom task builds the setup project, you put in another custom target BeforeTest. Include a task "Exec" in this target that will call the setup.msi file with the /quiet switch to install the product without launching the UI. Right after this target, tests are run in the Test target and the generic test will not abort if the install has succeeded.

<Target Name="BeforeTest">
    <Exec Command="..\Binaries\Setup1.msi /quiet"/>
</Target>

While writing this task, I hit an error in install. This was due to the build service account not being an admin on the build machine and install rights being denied to the account. After adding the build service account as admin on the machine, remember to restart the build service. Now, the install succeeds and the tests run on the installed product.

Feel free to post comments if you need help with any other related scenarios.