Random Musings of Jeremy Jameson : TFS

Building SharePoint WSPs with Team Foundation Build

Jeremy Jameson — Wed, 18 Nov 2009 13:45:00 GMT

As I noted in my previous post, I recently discovered that my approach for building Web Solution Packages (WSPs) in Microsoft Office SharePoint Server (MOSS) 2007 isn't compatible with Team Foundation Build.

I'm actually a little embarrassed to say this, but when I created the original "DR.DADA" approach for MOSS 2007 development back on the Agilent Technologies project, we were using Visual SourceSafe -- not Team Foundation Server (TFS) -- and a "manual" build process.

I'd used VSS and automated builds on other projects before (using NAnt), but never got around to automating our MOSS 2007 builds on the Agilent project because, honestly, there were just too many other higher priority items. Besides, each build only required a couple of minutes of actual human effort because most of the build was scripted.

Still, an automated daily build (and deployment to DEV) is a really, really good thing to have.

I've been fortunate to be on a few projects since then that have leveraged TFS.

However, up until about a month ago, I hadn't used Team Foundation Build (outside of the Jameson Datacenter, of course) due to the fact that we are leveraging the extranet TFS instance hosted by Microsoft.

Note that Microsoft IT makes it very easy for us to provision new TFS projects on either the extranet or one of several internal TFS instances. Configuring builds using Team Foundation Build on one of the intranet TFS instances is very easy (from what I hear), but I strongly prefer working off the extranet TFS instance because then I don't have to VPN into CorpNet in order to have access to source control.

However, choosing the extranet TFS instance also means we can't configure builds using the out-of-the-box functionality in Team Foundation Server (at least not without setting up a build server on the extranet). Fortunately, I've found a way to schedule "manual" builds that look a lot like automated builds performed using Team Foundation Build.

So, if you are building SharePoint WSPs -- regardless of whether you use the real Team Foundation Build or my "imitation Team Foundation Build" -- you need a way to build the WSPs without referring to referenced assemblies using relative paths.

As I first mentioned in my previous post, relative paths work just fine when compiling from within Visual Studio or using MSBuild from the command line. However, they don't work at all when queuing the builds through Team Foundation Build.

The problem with relative paths is that Team Foundation Build uses a different folder structure when compiling your projects. Specifically, it changes the output folder for all compiled items to be under a new Binaries folder -- not the location specified in the project settings within Visual Studio.

In other words, if you refer to a referenced assembly using something like:

..\..\..\CoreServices\bin\%BUILD_CONFIGURATION%\Fabrikam.Demo.CoreServices.dll

then you will find that this works just fine when building through Visual Studio -- or even when compiling using TFSBuild.proj from the command line (a.k.a. a "Desktop Build"). However, if you then queue the build through Team Foundation Server, you'll find your build fails because the referenced assembly was actually output to a different folder.

If you dive into the log file for the build, you will find that Team Foundation Build modifies the OutDir variable and sets it to something like:

C:\Users\svc-build\AppData\Local\Temp\Demo\Daily Build - Main\Binaries\Debug\

So the trick to building WSPs with Team Foundation Build is to leverage the OutDir variable instead of relying on relative paths to referenced assemblies.

Here is the updated DDF file based on my earlier sample:

;
; This ddf specifies the structure of the .wsp solution cab file.
;
; HACK: OPTION EXPLICIT cannot be used when specifying a variable with the /D option,
; otherwise MakeCAB aborts with an error similar to the following:
;
;        ERROR: Option Explicit and variable not defined: OUT_DIR
;
;.OPTION EXPLICIT    ; Generate errors for undefined variables

.Set CabinetNameTemplate=Fabrikam.Demo.Publishing.wsp

; The following variable must be set when calling MakeCAB (using the /D option)
;.Define OUT_DIR=

.Set DiskDirectoryTemplate=CDROM    ; All cabinets go in a single directory
.Set CompressionType=MSZIP            ; All files are compressed in cabinet files
.Set UniqueFiles=ON
.Set Cabinet=ON
.Set DiskDirectory1=%OUT_DIR%\Package

DeploymentFiles\PackageFiles\manifest.xml

.Set SourceDir=%OUT_DIR%            ; Copy assemblies from %OUT_DIR% folder

Fabrikam.Demo.Publishing.dll
Fabrikam.Demo.CoreServices.dll

.Set SourceDir=                    ; Copy files relative to project folder

.Set DestinationDir=Fabrikam.Demo.Publishing.DefaultSiteConfiguration
DefaultSiteConfiguration\FeatureFiles\Feature.xml

.Set DestinationDir=Fabrikam.Demo.Publishing.Layouts
Layouts\FeatureFiles\Feature.xml
Layouts\FeatureFiles\ProvisionedFiles.xml

.Set DestinationDir=Fabrikam.Demo.Publishing.Layouts\MasterPages
Layouts\MasterPages\FabrikamMinimal.master

;.Set DestinationDir=Fabrikam.Demo.Publishing.Layouts\PageLayouts
;Layouts\PageLayouts\MinimalPage.aspx

.Set DestinationDir=Fabrikam.Demo.Publishing.Layouts\Images
Layouts\Images\FabrikamLogo_32x32.png

.Set DestinationDir=Fabrikam.Demo.Publishing.Layouts\Themes\Theme1
Layouts\Themes\Theme1\BreadcrumbBullet.gif
Layouts\Themes\Theme1\FauxColumn-Fixed-2Col.png
Layouts\Themes\Theme1\FauxColumn-Fixed-3Col.png
Layouts\Themes\Theme1\Fabrikam-Basic.css
Layouts\Themes\Theme1\Fabrikam-Core.css
Layouts\Themes\Theme1\Fabrikam-FixedLayout.css
Layouts\Themes\Theme1\Fabrikam-IE.css
Layouts\Themes\Theme1\Fabrikam-IE6.css
Layouts\Themes\Theme1\Fabrikam-QuirksMode.css
Layouts\Themes\Theme1\Tab-LeftSide.jpg
Layouts\Themes\Theme1\Tab-RightSide.jpg

.Set DestinationDir=ControlTemplates\Fabrikam\Demo\Publishing\Layouts
Layouts\Web\UI\WebControls\GlobalNavigation.ascx
Layouts\Web\UI\WebControls\StyleDeclarations.ascx

.Set DestinationDir=Layouts\Fabrikam
Layouts\MasterPages\FabrikamMinimal.master

Note how I've replaced the BUILD_CONFIGURATION variable with the OUT_DIR variable. Not surprisingly, the OUT_DIR variable in the DDF is specified similar to how BUILD_CONFIGURATION was previously specified when calling makecab.exe. However, unlike the build configuration the OutDir variable will likely contain spaces as well as a trailing slash (which makecab.exe apparently doesn't like). Therefore we must quote the OutDir variable and append with "." if a trailing slash is found.

Here is the corresponding update to the project file:

  <PropertyGroup>
    <BuildDependsOn>
      $(BuildDependsOn);
      CreateSharePointSolutionPackage
    </BuildDependsOn>
    <QuotedOutDir>"$(OutDir)"</QuotedOutDir>
    <QuotedOutDir Condition="HasTrailingSlash($(OutDir))">"$(OutDir)."</QuotedOutDir>
  </PropertyGroup>
  <Target Name="CreateSharePointSolutionPackage" Inputs="@(None);@(Content);$(OutDir)$(TargetFileName);" Outputs="$(ProjectDir)$(OutDir)Package\Fabrikam.Demo.Publishing.wsp">
    <Message Text="Creating SharePoint solution package..." />
    <Exec Command="makecab /D OUT_DIR=$(QuotedOutDir) /F &quot;$(ProjectDir)DeploymentFiles\PackageFiles\wsp_structure.ddf&quot;" />
  </Target>

With these changes, the SharePoint WSP is successfully built regardless of whether it is compiled through Visual Studio, from the command line using MSBuild and the TFSBuild.proj file, or as an automated build using a Team Foundation Build server.

Compiling C++ Projects with Team Foundation Build

Jeremy Jameson — Sat, 07 Nov 2009 16:24:00 GMT

As I mentioned in my previous post, this week I incorporated Password Minder into my "Toolbox" Visual Studio solution that is scheduled to build daily through Team Foundation Server (TFS).

It's not that I really need daily builds of Password Minder; rather it's just been something on my "TO DO" list for a long time and I finally got around to doing it. Unfortunately, it wasn't without issue.

In case you are not familiar with Password Minder, it is mostly written in C#, but it includes one C++ project (for the NativeHelpers.dll).

Thus when I added the Password Minder projects to my "Toolbox" solution, I woke up the next morning to a notification from Team Foundation Server that my build failed. [No, I don't have some sort of alarm that goes off when one of my builds break. By "notification" I am referring to the e-mail message that I found in my inbox when I sat down at the computer.]

Clicking the build log referenced in the e-mail message, I quickly discovered the following:

Using "VCBuild" task from assembly "Microsoft.Build.Tasks.v3.5, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a". Task "VCBuild" Locating vcbuild.exe: not found at "c:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\..\..\vc\vcpackages\vcbuild.exe". Locating vcbuild.exe: Visual C++ Express is not installed on this computer. Locating vcbuild.exe: falling back to the system PATH variable. C:\Users\svc-build\AppData\Local\Temp\Toolbox\Automated Build - Main\Sources\Source\Toolbox.sln : error MSB3411: Could not load the Visual C++ component "VCBuild.exe". If the component is not installed, either 1) install the Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5, or 2) install Microsoft Visual Studio 2008.

That certainly is one of the best error messages I've seen in a long time. It told me exactly what I needed to do to fix the problem. Well, almost...

Note that DAZZLER (a VM in the "Jameson Datacenter" that is my dedicated build server) does not have the full installation of Visual Studio 2008. Rather it only has the Team Foundation Build install. In keeping with best practices, I try to keep the build server as "clean" as possible. That means no Visual Studio, no SharePoint, etc.

Following option #1 from the log file, I proceeded to install the Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5 -- but, of course, I didn't bother to review the ReadMe file that comes with it (at least not at first).

Consequently, I was greeted with the following error on the next build attempt:

c:\Windows\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets : warning MSB3428: Could not load the Visual C++ component "VCProjectEngine.dll". To fix this, 1) install the Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5, 2) install Microsoft Visual Studio 2008 or 3) add the location of the component to the system path if it is installed elsewhere. System error code: 126. c:\Windows\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets : warning MSB3425: Could not resolve VC project reference "..\NativeHelpers\NativeHelpers.vcproj".

That's when I discovered the following from the release notes for the SDK:

5.1.1 VCBuild fails to compile or upgrade projects

In order for VCBuild to run properly, vcprojectengine.dll needs to be registered. If vcprojectengine.dll is not registered, VCBuild.exe will fail with errors such as:

On compile: warning MSB3422: Failed to retrieve VC project information through the VC project engine object model. System error code: 127.

On upgrade: Failed to upgrade project file ‘foo.vcproj'. Please make sure the file exists and is not write-protected.

To workaround this issue, vcprojectengine.dll must be manually registered. From a Windows SDK command line window (as administrator in Vista:

On an X86 machine, run:
cd %mssdk%\VC\bin
regsvr32 vcprojectengine.dll
On an X64 machine, run:
cd %mssdk%\VC\bin\X64
regsvr32 vcprojectengine.dll

Unfortunately, these instructions aren't quite right -- or at least they didn't work verbatim in my environment. The workaround stated above makes you think there's an environment variable (%mssdk%) that refers to the path where the SDK is installed. However, this wasn't configured on DAZZLER.

Also, I didn't find my copy of VCProjectEngine.dll in an x64 folder, but rather in an amd64 folder:

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\VCProjectEngine.dll

I know most of you out there are not doing much (if any) C++ work anymore, but I thought I should share this just in case you need it at some point.

Recommended Check-In Policies for Team Foundation Server

Jeremy Jameson — Sat, 31 Oct 2009 12:14:00 GMT

I love using Team Foundation Server (TFS). There's just an amazing amount of "goodness" for software development that comes out-of-the-box; and there's even more available from Microsoft and other sources in the form of add-ons (many of which are free).

From a source control perspective, one of my favorite features is check-in policies. Anything that improves the quality of software development with minimal effort is really a "no-brainer" in my opinion.

The check-in policies that I prefer to have configured on all TFS projects are listed in the following table.

Recommended TFS Check-In Policies
Policy Type	Description
Builds	This policy requires that any code changes have been compiled and the last build was successful.
Work Items	This policy requires that one or more work items be associated with every check-in.
Changeset Comments Policy	Reminds users to add meaningful comments to their check-ins.
Code Analysis	This policy requires that Code Analysis is run with a defined set of rules before check-in.
Testing Policy	Ensures that tests from specific test lists are successfully executed before checking in.

The first two check-in policies are virtually painless and can be applied to any existing project at any time. These policies ship with Visual Studio Team System 2008, and assuming your developers have good software development discipline already, there's absolutely no reason not to enable these policies.

After all, why would any developer want to check-in code that hasn't been verified to at least compile? Similarly, why would a developer make a code change that isn't specifically related to a work item (even if that work item is simply something like "Refactor X" or "Code cleanup (M5)")?

However, the last three policies in the table above require a little more effort.

The Changeset Comments Policy is included in the Team Foundation Server Power Tools, which means that every member of the Development team will need to download and install this in order to use it. (Note that if you don't have the Power Tools installed, you can still override the check-in policy in order to avoid being blocked.)

The Code Analysis policy can definitely cause a little heartburn for your Development team, depending on how many code analysis rules you enable and whether or not you treat these warnings as errors. I'll talk more about Code Analysis in a separate post.

The Testing Policy is not one that I consider essential for all projects. For projects in which you have a good set of Build Verification Tests (BVTs) that give you a signficant amount of code coverage and don't take very long to execute, then enabling this policy just makes sense. However, keep in mind that you should definitely be running BVTs as part of your automated build, so if these tests require a substantial amount of time to execute (for example, they require signficant setup or teardown), then forcing developers to run these tests before every check-in could definitely impede their productivity.

Of course, you can always separate your tests into "quick tests" and "long running tests" like I've mentioned in a previous post, and consequently only require the "quick tests" to be executed before check-in.

If you are fortunate enough to be using TFS on your project, I hope you are reaping the benefits of check-in policies.

Essential Add-Ins for Team Foundation Server

Jeremy Jameson — Sun, 25 Oct 2009 15:32:00 GMT

In a previous post, I mentioned how I use SourceGear's DiffMerge instead of the out-of-the-box tool that comes with Team Foundation Server (which is also called DiffMerge). If you haven't at least evaluated the SourceGear alternative, I definitely advise you to take a look.

The other "add-in" for TFS that I consider to be essential is the TFS Quick Search Plugin.

If you have Web access to your TFS instance (via Team System Web Access), then quickly locating work items by keywords is actually pretty easy. However if you either don't have access to Team System Web Access or you would simply rather prefer to search for a work item without leaving Visual Studio, then the TFS Quick Search Plugin is definitely the way to go.

Note that it doesn't search all of the fields in all work items, but rather searches the work items currently displayed within Visual Studio. However, if you know a keyword that appears in the title of a work item, this can make it very easy to locate the desired work item.

I tend to use the TFS Quick Search Plugin a lot for the scenario where I am trying to find a duplicate bug and I remember a couple of keywords in the title of the previous bug. It is certainly much faster than scrolling through a few hundred work items in order to find something, or trying to build a query on-the-fly in order to locate a work item.

I am really interested to see the improvements in Visual Studio 2010 and Team Foundation Server 2010, but honestly it's probably going to be a while before I'm able to use that in my day-to-day work. Nevertheless, I hope this kind of work item "quick search" comes out-of-the-box in the next release (as well as a better DiffMerge tool). [I did do some hands-on labs back in February that cover the new branching visualization features in TFS 2010, and I have to say that I absolutely love what I've seen so far.]

Best Practices for SCM and the Daily Build Process

Jeremy Jameson — Sat, 26 Sep 2009 19:12:00 GMT

In a previous post, I briefly discussed a simple branching strategy for Team Foundation Server (TFS). This was somewhat of a follow-up to another post in which I briefly referenced a great article titled The Importance of Branching Models in SCM. If you haven't read this article, I highly recommend it.

However, while branching is certainly an important aspect of Software Configuration Management (SCM) -- and perhaps one of the most important aspects -- it's certainly not the only one that warrants discussion. In this post, I want to cover some other best practices for SCM as well as the daily build process.

For example, if you are using TFS, you know that each TFS project typically has a source control repository as well as a project team site in SharePoint (a.k.a. the "Project Portal"). You also probably know that all SharePoint document libraries can be configured to support versioning similar to TFS source control, and thus allow you to, for example, view an old version of an installation guide stored on the Project Portal. However, just because you can do this doesn't mean that it is recommended as a best practice.

Note that I'm not saying you should not store any documents on the Project Portal, nor am I saying that you should never enable versioning on document libraries in the Project Portal.

On the contrary, the Project Portal provides a great place to store documents that various project stakeholders need to access from time to time (without going through Visual Studio or TFS Web Access), and I almost always recommend enabling versioning on all document libraries in your SharePoint sites (provided you have plenty of storage on the backend SQL Server, of course).

Enabling versions on document libraries provides the ability to go back and see who changed a document and when (which is sort of like an "audit trail" on a document). While you could use the SharePoint versioning feature to go "back in time" in order to view the state of a document (e.g. an installation guide) for a particular version or build of your solution, this would obviously involve a good deal of considerable effort based on timestamps.

A much easier way of correlating a version of a particular document to a particular build of your solution is to actually store the document in source control within TFS. This allows you to quickly retrieve the document version using a changeset number or, more generally, a build label.

Build Labels

Back in the days of using Visual SourceSafe (VSS) on customer projects, at the beginning of the build process, a label would always be applied to the project in VSS. Since the build label was applied at the top-level project within a branch, it applied to all files in the solution – including the source code, setup files, and automated tests. This ensures that the installation and tests could be repeated for any particular build.

Note

While VSS certainly provided the ability to retrieve a project based on a particular timestamp, it definitely wasn't easy (the only way that I was ever to do it was through the SS command line utility). Thus build labels provided a quick way of getting a snapshot of the code for a particular build.

With the concept of changesets in TFS -- and the ability to quickly get the code for a specific changeset -- build labels are obviously not as important as they were in VSS. However, regardless of which particular source control system you are using, build labels provide an easy way to identify and retrieve important builds (such as Beta 1, Beta 2, RC1, and v1.0) or simply specific versions of the solution (e.g. 1.0.57.0).

Source Code

Applying the build label to the entire solution allows the Development team to obtain all of the source code for a particular build, thereby enabling them to step through the code to debug an issue that may not be reproducible in the latest version of the code.

Setup Files

In addition to the source code, the files supporting the installation of the solution (such as the install scripts and the installation guide – but not the compiled setup packages) should also stored in the source control system. In this way, the changes to the installation can be tracked from one version to another.

Automated Tests

Automated tests change as features are added to the solution. Consequently the tests must be matched to a specific build and therefore need to be checked into the source control system (and therefore labeled as part of the correponding build). Automated tests typically include unit tests as well as Build Verification Tests (BVTs).

Daily Build Process

The build process is a fundamental activity within any software project. In order to ensure success, the build and deployment process needs to conform to a number of basic requirements. If the solution cannot be built directly from its source code then it is not possible to integrate many of the key processes required for successful delivery.

The following sections describe the steps in the build process and who is responsible for each part of the process:

Check-Ins

All code included in a build must be checked into source control before the build process is initiated. All checked-in code must compile and it is the responsibility of the developer who checks in the code to ensure that the solution builds and all files that are needed to build the solution are checked in.

Important

The source control must not be left in a broken state at any time. If a build breaks, resolving the problem becomes the highest priority.

Tip

You can use the Builds check-in policy for TFS to ensure the solution compiles before a developer is allowed to check-in a changeset.

Each member of the Development team is responsible for ensuring the following:

Check-ins must not break functionality in the solution. If a developer checks in a change in their area of the code, he or she must not expect another developer to resolve errors caused by the check-in; either the person checking in the code must fix the dependant code when they check-in or the developer must work with the other developer(s) to coordinate the check-in.
The installation process must also be kept in a functional state at all times. Whenever a developer adds a feature, he or she is responsible for making sure that any setup changes are completed as well. Feature owners are ultimately responsible for the installation and configuration of their features.
The developer is responsible for writing any custom installation actions needed to deploy their feature, for knowing what files need to be deployed on which server, and what permissions the feature needs to run.
The installation of the feature should be completed when the code is checked into source control. The development of any custom installation actions needs to be done in conjunction with the team lead(s), and should not be done in isolation.
Changes to the installation should be checked in to give the team sufficient time to test the changes before the next scheduled build.
Unit tests must pass in the local developer’s environment (LOCAL) before a check-in is performed. If the unit tests are broken because of changes to the feature’s functionality, the developer must resolve the failures -- or reach agreement with the rest of the Development team to disable certain tests until they can be fixed (which should be a rare occurrence).

Tip

You can use the Testing Policy check-in policy for TFS to ensure that specific unit tests pass before a developer is allowed to check-in a changeset.

If any of the above conditions cannot be met, the code should not be checked into source control.

Automated Build Process

The automated build process should:

Automatically increment the Build Number or Revision portion of the assembly version (depending on which branch is being built)
Apply a build label (if appropriate)
Retrieve the code from the source control system
Compile the solution
Run any associated unit tests
Copy the build to the Release Server

If an error occurs during the build -- including execution of the unit tests -- the entire process stops. In this event, an email detailing the error should be mailed to the appropriate distribution list. Once the error is fixed, the build can be restarted. Note that the build/revision number should be incremented for the restarted build.

Build Verification Tests

Build Verification Tests (BVTs) are a quick, automated test run on the solution to catch defects in the code. The BVTs should cover the major functionality of the solution but are not meant to be a full regression of all the functionality in the solution.

The time required to run the entire set of BVTs should be on the order of minutes -- not hours. Otherwise the tests become too cumbersome to run on a regular basis. [Note that this guideline obviously depends on the size of your solution and the resources involved. For example, I believe the BVTs used by the Windows team actually run for several hours, due to the size of the code base. However, it is still short enough to repeat on a daily basis.]

BVTs are owned by the Test team in conjunction with the Development team to verify the build has completed successfully and to catch any regressions that are introduced as a result of changes to the solution. BVTs should be launched immediately after the automated build and deployment process. The BVTs must return an error to indicate if they have succeeded or failed. Only after the BVTs have passed should the build be considered ready for additional testing by the Test team.

As noted before, BVTs are owned by Test. The Test team should work closely with Development to keep the BVTs in sync with the changes in the solution. By having the BVTs owned by Test, it ensures that there is a knowledge transfer between Development and Test for any changes to the behavior or architecture of the solution.

Smoke Tests

In order to successfully deploy the solution, automated tests should be written to survey the environments before and after the application has been deployed. The smoke tests should check that the correct components have been installed, the configuration of the solution has been completed correctly, and that the network connectivity between machines is functioning correctly. The smoke tests must result in a clear pass or fail.

Smoke tests are generally considered to be more exhaustive than BVTs. The key differentiator is that BVTs are intended to identify whether a build is worthy of additional testing. Ideally, a substantial portion of the testing performed by the Test team is automated -- not manual. So whether you refer to your automated tests as BVTs and consider them part of a "smoke test" is really just a matter of semantics.

Investigating Failures

When a failure is found with the BVTs or automated smoke tests, the Test team should be the first point of contact to ensure that the tests are correct and the failure is valid.

Build Hand-off

The Release Management team should always lead the deployment of the solution, with Test and Development provided support as necessary.

Note that the Development team owns the DEV environment. Consequently, developers will often resort to “tweaking” the environment in order to get the solution deployed and operational. However, in order for the solution to be deployed to TEST, the configuration changes made in DEV must either automated as part of the installation or thoroughly documented as part of the installation guide.

Troubleshooting Guide

When problems are discovered with the deployment of the solution, a troubleshooting guide should be created (or updated) to capture the experience learned in resolving problems in the solution. The troubleshooting guide is not meant to be exhaustive in terms of troubleshooting all aspects of the solution; however it should provide a baseline for developing the operational support documents for maintaining the solution in the Production environment.

The troubleshooting guide should capture the process and tools used for investigating the problems with the solution and where to look for logs, events, etc. in the system. It should also capture patterns that have been seen in the solution that point toward a particular fault in the system.

Tip

A SharePoint site -- or even just a simple SharePoint list -- provides an excellent alternative to a Troubleshooting Guide document. Think of this site --or list -- as a simple "Knowledge Base" for your solution.

Comparing Source Code Branches

Jeremy Jameson — Wed, 16 Sep 2009 15:54:00 GMT

During the more than three years I spent helping Agilent Technologies migrate their Internet site from their legacy, proprietary platform to Microsoft Office SharePoint Server (MOSS) 2007, we unfortunately never used Team Foundation Server (TFS). Instead, we used Visual SourceSafe (VSS) in combination with a "Work Items" list in SharePoint that I've described in previous posts.

While I certainly prefer TFS over VSS, sometimes you simply have to concede that you can't have everything you would like on a customer project and move on to actually getting the work done.

However, just because we used VSS doesn't mean we didn't follow good Software Configuration Management (SCM) principles. For example, as I've described in the past, I insist on using branching right from the start. Thus when I setup the Visual Studio solution for the Agilent project, I created a Main branch and subsequently created branches for the various releases (e.g. v1.0, v2.0, and v3.0).

The particular branch that a developer uses would thus depend on whether the changes are for the next major release or a QFE (hotfix) for the version running in Production. For example, after deploying the Technical Support site (i.e. v2.0), we began working on the "General Site" (i.e. v3.0). [Note that in Agilent's terminology, the "General Site" essentially refers to everything outside of Technical Support, the Literature Library, and the Online Store (i.e. the "Buy" tab).]

However, since we didn't create the v3.0 branch until shortly before the v3.0 release, all v3 ("General Site") development was initially done on the Main branch -- just like v2 (Tech Support) development was initially done in Main prior to creating the v2.0 branch.

This branching strategy works really well, regardless of which particular SCM system you actually use. The key thing to remember is that all of the changes should eventually make it into the Main branch (since that branch will eventually be used to create another branch for the next major release).

The problem with VSS is that while it certainly supports branching, the merging features, um...well, let's just say that they leave a lot to be desired ;-)

Personally speaking, I've never felt comfortable using the out-of-the-box merging features in VSS, and instead always insisted on merging the changes from one branch to another manually. TFS is obviously years ahead of VSS in terms of branching and merging, but as I said before, sometimes you simply have to deal with what you've got.

Fortunately, long before the Agilent project, I had previously created my own process that takes a great deal of the "pain" out of manually merging source code. Here is what I came up with.

In my Toolbox, I have two simple scripts: DiffBranches.cmd and CopyBranch.cmd.

Here are the contents of DiffBranches.cmd:

@echo off

setlocal

REM set DIFFTOOL=Windiff.exe
set DIFFTOOL=C:\NotBackedUp\Public\Toolbox\DiffMerge\DiffMerge.exe

set BRANCH1=%1
set BRANCH2=%2

if ("%BRANCH1%") == ("") set BRANCH1=Main

if ("%BRANCH2%") == ("") set BRANCH2=v3.0

call CopyBranch.cmd "%BRANCH1%" "%BRANCH1%_tmp"

call CopyBranch.cmd "%BRANCH2%" "%BRANCH2%_tmp"

"%DIFFTOOL%" "%BRANCH1%_tmp" "%BRANCH2%_tmp"

As you can see, there's not much to it. I simply make temporary copies of the two branches (i.e. by copying the branch folder into a new folder appended with "_tmp") and then use my "Diff Tool" to compare the two folders. Originally, I used WinDiff, but once I discovered DiffMerge, I quickly switched to using it exclusively for all of my "diff'ing" activities.

The real "magic" lies in CopyBranch.cmd:

@echo off

setlocal

set BRANCH1=%1
set BRANCH2=%2

if ("%BRANCH1%") == ("") set BRANCH1=Main

if ("%BRANCH2%") == ("") set BRANCH2="%BRANCH1%_tmp"

robocopy "%BRANCH1%" "%BRANCH2%" /E /MIR /XD bin obj TestResults /XF *.scc *.suo *.user *.vspscc

Note that I put the word magic in quotes because it's really not magic at all. When copying a source code branch, I simply use robocopy.exe and specify the following options:

robocoby.exe command-line options used in CopyBranch.cmd
Command-Line Option	Comments
/E	Copy subdirectories, including empty ones
/MIR	Mirror the directory tree, thus ensuring that when I subsequently run CopyBranch.cmd, any deleted files from the original branch are removed from the "temporary" copy of the branch)
/XD bin obj TestResults	Exclude directories matching the given names/paths, thus skipping the compiled output (i.e. bin and obj) as well as the test results generated by running unit tests from within Visual Studio
/XF .scc .suo .user .vspscc	Exclude files matching the given names/paths/wildcards, thus skipping source code control files (.scc and .vspscc), and Visual Studio solution/project user-specific options (i.e. .suo and .user)

For example, let's suppose that I've checked in some changes to the v3.0 branch that need to be propagated to the Main branch. I would open a command prompt and run the following:

C:\NotBackedUp\Agilent>DiffBranches.cmd v3.0 Main

After the two branches are copied to their respective temporary folders, DiffMerge.exe is launched to compare the two branches. Since I exclude many of the directories and files that are expected to differ between the two branches, I can quickly view only the differences that I am interested in. I can even use DiffMerge.exe to apply the changes interactively (checking out the files to be updated beforehand as necessary). This greatly reduces the effort involved in manually merging changes from one branch into another.

Note that it takes a little bit of time to copy a branch the first time, but this is lightning fast on subsequent runs. Hence why I don't delete the "_tmp" folders after comparing two branches.

I occasionally use this process even on projects where we've used Team Foundation Server. For example, I sometimes create a new workspace in TFS to represent a "temporary branch" where I want to experiment with some substantial changes to the source code (while I still have pending changes in my main workspace, that I may have shelved but don't want to check-in just yet). Using a little robocopy.exe and DiffMerge.exe, I can more easily compare two branches or workspaces.

Expression Web, My MSDN Blog, and (Now) Team Foundation Server

Jeremy Jameson — Sat, 12 Sep 2009 17:17:00 GMT

In case you haven't picked it up from some of my previous posts, I became somewhat of a "Web standards zealot" back in 2006 while creating a "community site" for a local organization of mental health professionals and attorneys that help children and their parents through divorce. After reading books like The Zen of CSS Design, Web Standards Solutions, and Bulletproof Web Design, I realized the fundamental concept of Web standards design closely aligns with my preference of keeping things as simple as possible. Last year, I got a copy of Transcending CSS and I highly recommend this book as well for anyone responsible for creating Web sites.

Although it may strike people as somewhat odd, I've been using Microsoft Expression Web for the last couple of years to manage the content for my MSDN blog. While the underlying platform for MSDN blogs (i.e. Community Server) certainly provides rich editing functionality, I personally prefer to have very tight control over the HTML whenever I am creating content for the Web. There are also a few things that really bother me about the editing functionality in Community Server (or at least the version of Community Server that Microsoft is currently using).

Before finally choosing Expression Web, I looked at various other tools for authoring blog posts, including Windows Live Writer and even Microsoft Word 2007. However, I found that by using Expression Web and a simple "template" that I created, I could quickly and easily create blog posts with just the kind of "minimal markup" that I was looking for.

My template simply contains a variety of markup that I commonly use when creating blog posts, such as:

<blockquote class="directQuote errorMessage">
    [Direct quote, error message]</blockquote>

Whenever I need to create a new blog post, I simply copy my template (i.e. default.htm) into the corresponding Archive\{year}\{month}\{day} folder, rename the file to something like sharepoint-2010-sneak-peek.aspx, and then start replacing the placeholder text with the corresponding content. Note that I have defined a few custom CSS rules in Community Server to change the style of my blog content, such as:

.directQuote {
    font-style: italic;
}
.errorMessage {
    color: red;
}

I define these styles in a CSS file (i.e. Themes\MSDN\MSDN.css) that is referenced by each page, thus giving me a WYSIWYG (What You See Is What You Get) experience when authoring blog content. This is one of the biggest things lacking from the editing functionality provided by Community Server, meaning that it doesn't apply your custom CSS rules when previewing a blog post before publishing.

Once I have finished writing a blog post in Expression Web, I simply copy the HTML within the <body> element, browse to my MSDN blog, click the Write a Blog Post link, click the Edit HTML Source toolbar button in the Community Server Control Panel, and then paste my content into the HTML source view.

I freely admit this approach takes a little longer than simply authoring posts directly on the site, but I believe it is well worth it. Even more so, now that my blog is stored in Team Foundation Server (TFS)...

Back in August while I was in Redmond for SharePoint 2010 Training, I took the opportunity of being on the Microsoft campus to rebuild my laptop with Windows 7. Prior to that rebuild, I had been running Windows Server 2008 on my laptop so that I could utilize Hyper-V regardless of whether I was working at a customer site or from home (with access to the "Jameson Datacenter" and all of my various VMs).

As part of this latest rebuild, I installed the latest version of the Microsoft Expression suite. One of the new features I discovered in Expression Web 3 is the ability to integrate with Team Foundation Server for source control. Note that this integration isn't completely out-of-the-box (i.e. you have to download a QFE to enable it).

This morning, I finally found some time to move the "offline copy" of my MSDN content from my Documents folder into Team Foundation Server. I simply needed to create a new team project in TFS and then configure my workspace for source control. Once this was done, the TFS integration features of Expression Web 3 simply just worked, as shown below:

Figure 1: Expression Web - my MSDN blog

See full-sized image.

Notice how items that are checked into source control have a lock icon and new items that are pending have a "plus" icon next to them. In other words, the Folder List in Expression Web (which essentially shows your Web site "solution") now behaves a lot like the Solution Explorer window in Visual Studio (at least from the perspective of source control).

Even though the offline content of my MSDN blog content was previously backed up on one of the servers in the Jameson Datacenter, now that it is stored in TFS, it is much easier for me to author blog posts from multiple computers (by simply doing a Get Latest from TFS). I also like the fact that my posts now have versioning -- so in the rare case where I actually go back and annotate a blog post, I can now easily "diff" my changes whenever I want.

I personally love Expression Web, and now that it integrates with TFS, I love it even more! [Although I'm still getting used to the new "dark theme" introduced in v3 -- which was carried over from the previous version of Expression Blend.]

Linked Files in Visual Studio Solutions

Jeremy Jameson — Thu, 02 Apr 2009 16:20:00 GMT

A couple of years ago, I wrote a post introducing my system for structuring Visual Studio solutions. However, I apparently forgot to post a follow-up providing additional details, such as configuring assembly versioning and what I like to call "shared assembly information."

Before I can cover these details, I need to first ensure that you are familiar with the concept of linking files in Visual Studio solutions, why this is a powerful feature, and when to use it.

If you ever used Visual SourceSafe (VSS), you likely used its feature for sharing files across multiple projects. For example, you could check-in a file at the root of your solution, and then drag-and-drop it into other projects (i.e. "subfolders") in your solution. Thus, whenever a change was made to the file (regardless of which particular VSS project the change was made in), the next time you "got latest" the change would be reflected in all locations. This was a common way of, for example, having all of your .NET assemblies in the same Visual Studio solution specify the same assembly version.

Note that Team Foundation Server (TFS) does not provide an equivalent "share file" feature. Fortunately, however, you no longer need such a feature.

Back in the days of the original Visual Studio .NET and the following version, Visual Studio .NET 2003, whenever you added an existing file to a project, it copied the file into the corresponding location in the project.

However, in Visual Studio 2005, the Add Existing Item feature provided the ability to choose to either Add the item or Add As Link (via the little down arrow on the button in the dialog box).

In other words, once you upgraded to Visual Studio 2005, it was no longer necessary to rely on any "sharing" features of your source control system in order to have multiple projects always reference the latest version of a file.

Note that when you Add an item in Visual Studio 2005 or Visual Studio 2008, the behavior is the same as earlier versions (meaning the file is copied into the corresponding location within the project). When you choose to Add As Link, however, you simply reference the file in-place. [Don't worry, relative paths ensure that everyone on your team is free to choose whatever root folder they wish for their individual workspaces.]

To illustrate this concept, I quickly built out a "demo" solution, as shown below.

Figure 1: Linked files in a Visual Studio solution

See full-sized image.

The corresponding folder structure on disk resembles the following:

Fabrikam
- Demo
  - Dev (branch)
    - Lab1 (branch)
      - AssemblyVersionInfo.cs
      - CustomDictionary.xml
      - Fabrikam.Demo.sln
      - Fabrikam.Demo.snk
      - SharedAssemblyInfo.cs
      - AdminConsole
        
        AdminConsole.csproj
        
        Program.cs
        
        Properties
        
        AssemblyInfo.cs
      - CoreServices
        
        CoreServices.csproj
        
        Logging
        
        Logger.cs
        
        Properties
        
        AssemblyInfo.cs

Note that AssemblyVersionInfo.cs, CustomDictionary.xml, Fabrikam.Demo.snk, and SharedAssemblyInfo.cs reside in the same folder as the Visual Studio solution file and are subsequently "linked into" the two Visual C# projects. Thus whenever a change is made to one of these files, the next build of each project will reflect that change.

With this foundation in place, I'll explain some other recommended best practices over a series of follow-up posts, including:

Stay tuned!

DiffMerge - A Better Differencing Tool

Jeremy Jameson — Tue, 24 Mar 2009 17:01:00 GMT

Last summer, I added DiffMerge to my Toolbox and I haven't used WinDiff since.

DiffMerge can do everything WinDiff can, plus a whole lot more -- like intra-line highlighting, merging, and comparing files using configurable rulesets (although you'll likely never need any more than those that come "out-of-the-box" with DiffMerge).

And since the folks at SourceGear don't mind giving away DiffMerge for free, there's no reason not to use it!

Heck, I even configure Visual Studio to use DiffMerge as my comparison tool instead of the default tool that comes with Team Foundation Server (TFS).

To customize your TFS comparison tool in Visual Studio 2008:

On the Tools menu, click Options...
In the Options window, expand Source Control, select Visual Studio Team Foundation Server, and then click Configure User Tools...
In the Configure User Tools window, click the Add... or Modify... button as necessary to configure the following:

Extension: .*
Operation: Compare
Command: C:\NotBackedUp\Public\Toolbox\DiffMerge\DiffMerge.exe
Arguments: %1 %2

Obviously, you may need to adjust the path to DiffMerge.exe as necessary for your environment.

In the past, I tried other tools like BeyondCompare -- but I preferred the simplicity of WinDiff. Then, a couple of years ago, one of my peers pointed me to an internal tool called "Odd" that would do intra-line highlighting. However, I couldn't get over the fact that this was an internal tool only -- and thus not something I could recommend to customers.

When I stumbled across DiffMerge, I was delighted and I haven't looked back since.

Perhaps Visual Studio 2010 will ship with a vastly improved differencing tool, but in the meantime -- or just in case it doesn't -- DiffMerge is my tool of choice.

Branching Strategy in Team Foundation Server

Jeremy Jameson — Tue, 10 Feb 2009 21:34:00 GMT

While attending TechReady (an internal Microsoft training conference) last week, I learned a lot -- not only about future versions of our products, but also numerous tips and tricks for current versions. One of the most valuable insights I gained was from a session on branching in Team Foundation Server (TFS). If you've read my previous post on structuring Visual Studio solutions, then you know that I'm a proponent of branching from the start in order to easily support parallel development and periodically stabilizing for a release.

I had previously read the TFS Branching Guidance on CodePlex, although I admit it has been a rather long time. Nevertheless, I still understood the commonly used Main, Dev, and Release branches. However, I discovered during the session last week that I had a fundamental misunderstanding about baseless merges in TFS. Prior to last week, I thought a baseless merge occurred when the source and target do not share any ancestry in the branching tree. In other words, I thought that as long as the target resides on a branch that can be traced through some lineage to the source branch, then TFS has sufficient "knowledge" (i.e. algorithms) to merge changes from the source to the target.

As I stated before, this was a misunderstanding on my part. When I asked a question about trying to delay the branching for stabilization for as long as possible, the presenters of the session -- James Pickell and Mario Rodriguez -- pointed out that a baseless merge occurs whenever the target is not in a "first level" branch from the source. In other words, you can automerge from parent-to-child (or child-to-parent), but you cannot automerge from grandchild-to-grandparent.

So, what does all of the mean in practical terms?

It all comes down to how -- and when -- you actually branch your code.

In the interest of not reinventing the wheel, allow me to pilfer one of the images from the updated TFS Branching Guide 2.0:

Figure 1: Basic branch plan

See full-sized image.

Note that this is a basic branch plan, but it is sufficient to demonstrate the concept. You may very well have a more complex branching strategy (e.g. a large organization like Microsoft in order to support "feature teams"). However, the essential pattern still applies.

The question that I asked during the session was essentially why not postpone branching for as long as possible? In other words, why not branch Release first, and then branch for hotfixes and service packs as necessary? In my mind, this alternative approach reduces the merge effort by only branching at the point in time when we actually need a branch (e.g. to stabilize for a release). At some later point in time after the release when we actually need a hotfix or service pack branch, then we'll create it from the Release branch. In response to my question, Mario termed this "reactive branching" -- which seems like a very good way to describe it. To help visualize the difference, I mocked up the following figure.

Figure 2: "Reactive" branch plan

See full-sized image.

There are a couple of significant problems with reactive branching:

In order to merge code changes from a QFE (a.k.a. hotfix) or service pack back into Main (and remember, we almost always want the fixes in QFEs to be incorporated into Main), we would either need to first automerge the changes into the Release branch and then merge the changes into Main -- or else perform a baseless merge (which, generally speaking, we want to avoid). If we merge the changes into the Release branch, then that code no longer reflects the actual Release build (e.g. v1.0) but rather the Release build plus some number of changes. Yes, we could still use labels to "snapshot" the source code for the Release version, but as James pointed out, labels can be manipulated and therefore may not be sufficiently "bulletproof" to meet your requirements.
We could still use reactive branching, but avoid merging through Release by renaming a branch (e.g. the Release branch is renamed to Service Pack, and a new Release branch is created and "locked down" from a permissions perspective). However, this requires significantly more work with each release -- work that can be avoided altogether by using the branching strategy shown in Figure 1.

So, in summary, while it may seem a little backwards to create your Service Pack branch before your Release branch, there are compelling reasons to do so. As for me, I now consider my "reactive branching" strategy to be a thing of the past.

Thanks to James and Mario for setting me straight!

By the way, if you haven't read the branching guidance created by the VSTS Rangers, I highly recommend it. Very good stuff.