It's been a while since I've blogged here, but that doesn't mean we haven't been busy in the migration space.  Today Brian blogged about the upcoming TFS 2008 SP1 release, and called out the improvements we've made in this release.  In short, we've fixed a ton of bugs in the release, greatly improving the stability and fidelity of a migration from VSS to TFS. 

If you've been waiting on migrating from VSS, or are blocked by issues with the last release of the VSS Converter, then this updated version is for you.  The release will be available in the near future, and we're going to be looking for feedback from customers.  The more feedback we have during the beta period means more bugs fixed in the final version, so please help out if you're preparing to finally migrate to TFS.  Stay tuned for details on the release!

Matt

For those people interested in the Migration and Sync Toolkit or for people that just want to be able to move source code and work items between TFS servers/projects, check out the TFS to TFS Migration Tool.  Currently, it is available only as a prerelease while we continue to test with customers.  You can learn more on the CodePlex project page:

http://www.codeplex.com/tfstotfsmigration

This tool was built using the Migration and Sync Toolkit, so it should prove to be useful for anyone that is writing a custom migration tool.  We have received some comments in the past about the WSS to TFS sample tools containing too much WSS specific code that makes it hard to understand the toolkit.  Although this tool has plenty of TFS specific code, it will probably more closely resemble a tool built to integrate with another version control or work item tracking system.  There is also a GUI built on top of the tool to make configuration easier, which also serves as a good example of how to build a configuration layer for a custom migration tool.

One of the migration tools that has been around as long as TFS is the VSSConverter utility.  Some time after the release, when TFS adoption started to grow, and VSSConverter usage started to grow, a strange issue began to be reported by customers: some versions of their files were empty or missing after migration. 

After extensively researching the issue, we noticed some common patterns - files with missing versions typically were either renamed, moved, or were under a parent folder that was renamed or moved.  Basically, if the namespace of a file changed at some point in time, some versions were not being migrated correctly.  This issue became affectionately known as "The Rename Problem".

This problem is known to exist in both the Whidbey and Orcas versions of the VSSConverter.  In both releases, only the historical versions of files are affected - the latest version in TFS will match the latest version in VSS. 

To get some context into the what is happening, and what the workaround is, keep reading.  Fixing this issue altogether is something we're planning for a post-Orcas release, but until then, the following workaround should be useful to many users.

Events such as renames, archive and restore, sharing, branching, and move can create namespace conflicts in the history of VSS.  What is a namespace conflict?  Imagine there is a file foo.cs under a folder Bar.  At some point, Bar is renamed to Baz, so the namespace of foo.cs is "Baz/foo.cs".  Later, another folder named Bar is added to VSS, and under it resides a file named foo.cs.  This new foo.cs has a namespace of "Bar/foo.cs", which used to be the namespace of the original foo.cs.  We would say that these files have a namespace conflict, because at they shared the same namespace, albeit at different points in time. 

These namespace conflicts result in files and folders not being migrated correctly by the VSSConverter.  This problem can be identified if there are missing versions of converted files, or missing files from the conversion.  In the log file and/or migration report, you will see the following errors: 

"The item already exists."

OR

"The item could not be found in workspace." 

In some cases, changing the mappings in the configuration file can help to migrate files correctly.  If the missing versions are located under a folder named "ProjA", the mappings can be changed to a subfolder of "ProjA".  For example, the mappings:

<ProjectMap>
<Project Source="$/Projects/ProjA" Destination="$/TFSProjects/ProjA" />
</ProjectMap>

Can be changed to:

<ProjectMap>
<Project Source="$/Projects/ProjA/SubFolder1" Destination="$/TFSProjects/ProjA/SubFolder1" />
<Project Source="$/Projects/ProjA/SubFolder2" Destination="$/TFSProjects/ProjA/SubFolder2" />
<Project Source="$/Projects/ProjA/SubFolder3" Destination="$/TFSProjects/ProjA/SubFolder3" />
</ProjectMap>

Once these mappings are changed in the config file, the VSSConverter can be run again.  If some of the files are still not migrated correctly, the mappings may be further scoped to another subfolder.

I just found out about Windows Live Writer (http://get.live.com/betas/writer_betas), so I'm giving it a test drive.  Hopefully it will do a better job at managing 5 blogs than me.

So far it has appeared to work fine on one of my SharePoint blogs, lets see how this one fares...

-Matt

After working with several customers on their migrations from ClearCase to TFS, it was evident that one of the big challenges of migrating between such vastly different version control systems was determining an appropriate branching structure.  This was the result of both the varied branching models of TFS and ClearCase, as well as the best practices encouraged and best adapted to each system.  Below are some high level explanations of the branching models of each system, how the aforementioned migration tool migrates branches, and recommendations on how to choose the correct set of branches to migrate from ClearCase to TFS.

ClearCase Branching

ClearCase users are familiar with just-in-time branching in version space - that is, new versions of individual files and folders are created when the file is edited on a branch.  This means that only when a particular file is checked-out on a branch is it even created on the branch.  Until then it lives on the parent branch that is determined by the config spec.  The config spec also defines how the right versions of files are selected from branches, and which versions should be defaulted to if a file doesn't exist on a branch.

TFS Branching

In TFS, branches are created early, and branch in path space.  This means that as soon as a branch operation is checked-in, all of the items under the branched path now exist on that branch.  Files from the branch are edited and checked-in to the branch, and files that are never edited on a branch also reside on the branch.  Since the contents of the branch are determined at creation time, there is no need for users to have config specs that pull together the correct items to form a usable/buildable branch.

(For details on TFS branches and branching strategies, visit the TFS Branching Guidance page on CodePlex: http://www.codeplex.com/BranchingGuidance)

Migrating Branches from ClearCase to TFS

The disparity between ClearCase and TFS branching leads to some challenges when migrating branches.  Clearly, config specs are integral to determining the correct files to select for a branch in ClearCase.  For this reason, the TFS to ClearCase migration tool requires a config spec for each branch to be migrated.  Using this config spec, the tool is able to create TFS style branches out of the ClearCase branches.  What that means, is that files that never technically existed on a ClearCase branch will now exist on the TFS branch. 

Although this strategy is not truly preserving the data as it existed in ClearCase, it is a necessary transformation to ensure the usefulness of branches in TFS.  For example, without all of the correct files on the branch in TFS, the files will no longer build.  Merging files back into parent branches is also impossible without all of the correct files in TFS, essentially defeating the reason to branch in the first place.  For these reasons, the tool takes the liberty of adding the files to the branch, as specified in the config spec.

For more details on how the TFS to ClearCase migration tool migrates branches, see  the documentation that is installed with the tool.

Which branches should I migrate?

In some cases, ClearCase VOBs can contain a very large number of branches.  This is often because many users will create many short lived branches used to fix bugs or add new features.  In TFS, the branching guidance is to keep a smaller number of branches, typically, Main, Dev, Production, and the feature branches off of Main (see the TFS Branching Guidance for details).

In order to keep the time to migrate files at a reasonable level, and to avoid a complex branching structure in TFS, the guidance for migrating branches is to migrate those ClearCase branches that are synonymous to Main, Dev, and Production.  It may also be desirable to migrate some specific feature, release, or bugfix branches, but the rule of thumb should be to minimize the number of branches migrated to TFS.

At last, the migration tool for Rational® ClearCase® is available for free download.  This tool provides the ability to migrate ClearCase data to TFS, and it also provides a service that will keep items in sync between TFS and ClearCase.  Together, these features will make the process of migrating from ClearCase to TFS much simpler, and far more automated. 

The tool and documentation (installed with the tool) can be downloaded from the new Team Foundation Server Migration and Integration Solutions page.

A quick summary of the features of the tool:

• Supports migration of base ClearCase VOBs
• Migrate a snapshot of source control to TFS
• Migrate files to Team Foundation Server while preserving history
• Migrate branches to Team Foundation Server retaining the branching structure/hierarchy
• Bidirectional synchronization of data between TFS and ClearCase

 For help with this tool please use the Team Foundation Server - Version Control forum on MSDN.  Be sure to indicate the question is specific to the migration tool in the title of the post so we can identify questions more quickly.

The first complete version of the Migration and Synchronization Toolkit has been released on CodePlex!  To get the latest source code, including the complete reference implementations, please visit the CodePlex page for the Toolkit: http://www.codeplex.com/MigrationSyncToolkit

In addition to updating the source code and reference implementations, a significant amount of new documentation has been added.  From the home page of the CodePlex project, there are links to pages to help users getting started, define the features, and provide guidance of building a tool, just to name a few.  Over the next few weeks, we will be adding to the content on the wikis, including some user feedback from our pilot programs for the toolkit.

Please also remember to give us feedback!  Any comments on the usage of the toolkit, questions about writing tools, questions on using the tools, and comments on the documentation are all welcomed.  All the feedback that we can get will ultimately help to improve the Toolkit, so please speak up if you have any comments.

Since the prerelease version, we've seen many requests for various tools to help migrate and synchronize data between version control and work item/bug tracking systems.  Part of the motivation behind this toolkit was to encourage community development of tools for common systems used by TFS users and potential TFS users.  If you are interested in starting an open/shared source project to develop a tool, please let us know, and we can help to get a project started on CodePlex.

I've heard plenty of requests for tools to migrate data from competitor products to TFS, and I'm trying to find out which tools have the highest demand.  If you need a tool to integrate Subversion and TFS, for example, please reply to this post and let us know.  Or better yet, create a new item in the issue tracker so that others can vote on the tools they want to see developed.

I'm also thinking about how to encourage the community to work together to develop migration and synchronization tools.  Surely there is more than one person with a need for a SVN tool that is willing to develop it using the toolkit, so why not bring those heads together? 

I'll start it off and open an issue for a SVN to TFS migration tool, and see where that goes.  Hopefully others will follow.

http://www.codeplex.com/MigrationSyncToolkit/WorkItem/List.aspx

A prerelease of the Migration and Synchronization Toolkit has been released on CodePlex!  If you are interested in seeing the progress that we have made to date, and are eager to try out the sample tools, please visit the project site on CodePlex at http://www.codeplex.com/MigrationSyncToolkit

All of the source code and documentation for the toolkit is available on the project site.  The source code can be downloaded from the .zip archive on the Releases tab.  From the home page, the Getting Started page should provide all of the info to setup and build the toolkit, and the Release notes contains the info pertinent to this prerelease.

Please note that this is a prerelease version of the Migration and Synchronization Toolkit. Several features are not complete and as testing has not completed, the code has the potential to contain bugs and overwrite data stored in TFS. Please be careful to use this only in a testing environment and not on live production data.

As a part of the effort to get feedback on the specs for VS Orcas release, we have published our in-progress spec for the TFS Migration and Synchronization Toolkit.  It is available, along with the other TFS and VS Orcas feature specs on MSDN.

http://msdn2.microsoft.com/en-us/vstudio/aa948851.aspx

To get the feedback to our team you can use the feedback link on the MSDN page, or feel free to leave feedback here.  We look forward to your comments!

If you are interested in our upcoming strategies and plans for TFS Migration, check out this interview on Channel 9 (shameless plug):

http://channel9.msdn.com/Showpost.aspx?postid=283218

If you have any feedback on the interview or the topics discussed please share them here.  We would be happy to see what others are thinking about migration and TFS.

In my previous posts, it was not made clear that the migration toolkit will support bi-directional synchronization as well as one-way conversion.  This enables many useful scenarios including the ability to migrate portions of a project over time.

Synchronization does bring up some complex scenarios, especially for non-versioned items.  We have come across a few questions concerning how labels in a version control system should be migrated/synchronized with TFS.  Any feedback on this is welcome.

1. What level of support for migrating labels is optimal?
1. Should all labels be migrated during the initial conversion of data,
2. or should users be able to choose specific labels to migrate,
3. or should no labels be migrated (they can be added in TFS after migrating)?

2. For labels that exist in both systems (the original system and TFS) after initial conversion:
1. Is it useful to synchronize labels across systems?
2. Are there specific labels that would be useful to synchronize between systems (i.e. a recent build label)?
3. Should labels only be synchronized if they are manually chosen to be synchronized?

As expected, the first questions about the Migration Toolkit concern the tools that will be developed on top of it.  Right now, we are planning to release two different tools (1 version control, 1 WIT) along with the toolkit itself.  These first tools will serve as examples on how to write toolkit based tools, and will provide full migration functionality for the respective systems.

For version control, our plan is to develop a tool to migrate files from WSS to TFS.  At this time, WSS lists will not be migrated into work-items, but we do recognize this as desirable tool for WIT.  The first tool are planning to develop for WIT will migrate data between an open source bug tracking system and TFS.  Although we considered developing the WSS list tool, bug tracking systems are typically more complex than the WSS list feature, and providing a tool to convert WSS lists would not illustrate many of the interesting parts of a WIT migration tool.

Are there any suggestions for migration tools that we should be targeting for the future?

Over the past few months we've been working on the development of a toolkit to facilitate the creation of tools to migrate data into TFS.  Using this toolkit, the task of creating a tool to convert data from a given version control/work-item tracking system will be greatly simplified.  The toolkit will provide many of the features that are standard in any data conversion tool and will abstract away the TFS OM.  This means that tool authors only need to write a plug-in specific to the system of their choice.

Some of the features that we are providing as part of the toolkit include:

• Specification of settings through a well defined XML configuration file
• Ability to configure several migration sessions in a single location
• Robust analysis and conversion engine
• Conflict resolution mechanisms
• Real-time and post migration reporting

 

We are also planning to release this toolkit via the web in a shared-source manner.  This would allow anyone to download, modify, and re-distribute the toolkit to meet their specific needs.  As we get nearer to the release, I'll be sure to post more details. 

For those teams that are planning to migrate from other systems to TFS, how does this list of features line up with what you would need to successfully migrate to TFS?  Besides version control and work-item tracking, what other areas of software development could benefit from a migration tool?

This isn't by any means breaking news, but it's related to migrations to TFS so I figure I should mention it.  Anyone converting from VSS to TFS using the VSSConverter command line tool might be interested in the GUI developed by EPocalipse Software. 

Check out their blog at: http://www.epocalipse.com/blog/2006/04/06/vssconverter-gui/

(Note: This software has not been developed/tested/endorsed by Microsoft.  Use at your own discretion.)