Robert Horvick's Weblog : TFS Migration

Migration Toolkit Pre-release nears 100 downloads ...

RobertHorvick — Thu, 26 Apr 2007 19:39:00 GMT

At this time 98 people have downloaded the TFS Migration and Synchronization Toolkit. Ok - some of you are saying "100? What's the big deal? A picture of a rabbit with a pancake on it's head gets more downloads every hour!"

And they would be right.

But think big picture - what if 50% of the downloaders actually open the zip file?

And what it 50% of them actually compile and run the product?

And what if 50% of them start looking into the source code and trying to write their own converter?

And what if 50% of them actually succeed?

And what if just one of those people decides that their converter is worth sharing with the world? How many people will that help? Hunderds? Thousands? More? Or maybe they decide to use the tool for consulting engagements - how many successful careers could be made using that tool?

In the migration space it does not require big numbers to make a big impact.

We've already impacted 100 people. That is exciting to me.

How many more will download it tomorrow? And the next day?

Have you downloaded it yet?

Migration Toolkit Pre-Release published on CodePlex

RobertHorvick — Sat, 21 Apr 2007 02:30:00 GMT

Interested in writing a migration tool targeting (or synchronizing with) Team Foundation Server version control or workitem tracking?

Then check out the pre-release of the TFS Migration and Synchronization Toolkit on CodePlex!

http://www.codeplex.com/MigrationSyncToolkit

This drop is the complete source code for the migration toolkit and sample converters. The VC sample is an application that will synchronize WSS document libraries with TFS version control. Developers can check in doc changes with their code and managers can see the updated docs on the WSS server after the mirroring is done (and it works the other way too).

Oh - and this is still under active development (and has known issues) so if you have any feedback it's still early enough for it to be considered for inclusion in the final drop.

And to quote Matt ...

"Please note that this is a prerelease version of the Migration and Synchronization Toolkit. Several features are not complete and as testing has not completed, the code has the potential to contain bugs and overwrite data stored in TFS. Please be careful to use this only in a testing environment and not on live production data."

TFS Migration Toolkit Spec released for public review

RobertHorvick — Fri, 02 Mar 2007 23:42:00 GMT

As Matt noted on the migration blog a working copy of the toolkit specification has been released. You can download it and leave feedback from this link:

http://msdn2.microsoft.com/en-us/vstudio/aa948851.aspx

If you have feedback on migration scenarios for version control, work item tracking or other products (test, office, web, file systems, etc) please check it out and let us know. We are still actively developing the toolkit and your feedback is very important to us!

How to handle VSSConverter UserMap.xml files for users who no longer exist

RobertHorvick — Thu, 18 Jan 2007 18:24:00 GMT

When running the VSS to TFS converter (VSSConverter.exe) one of the most common questions I hear is how to create a UserMap.xml file for users who are no longer with the company or have had their accounts renamed.

My first suggestion is that any customer who has to deal with this issue (or any user mapping issues – or users in a workgroup configuration) should start by getting the hotfix provided in KB 928828. This can be gotten through your CSS (support) contact.

This hotfix changes the way that user authentication is done and is an important fix. It is compatible with both RTM and SP1 however it was not included in SP1 (it was made after the SP1 cut-off date and does include all the VSSconverter fixes already in SP1).

That said - what I’m going to describe is the same in both the RTM and SP1 versions as well as in KB 928828.

The correct way to deal with the users may be as simple as doing nothing. Let me give you an example.

To test this out I created a sample VSS repository and added two users “User1” and “User2”. These were in addition to the default users of “rhorvick” (me) and “Admin”.

I connected as “User1” and created some content. Then as “User2” and created more. And finally as “rhorvick” and created even more.

I then ran the converter in analyze mode. This created the following UserMap.xml file:

<?xml version="1.0" encoding="utf-8"?>

<!--

This file is automatically created by VSS Converter. You can optionally use the file to map

a VSS user to a Team Foundation user. For example, <UserMap From="Jane" To="MyDomain\Janep"></UserMap>

This mapping causes all actions logged by VSS user “Jane” to be changed to Team

Foundation user “ MyDomain\Janep ” during migration.

-->

</UserMappings>

At this point I just ran the conversion – I did not change the mapping file in any way. Please note that User1 and User2 are NOT existing accounts on my TFS server.

With the QFE the conversion created the following history in TFS:

Changeset User Date Comment

--------- ------------- ---------- --------------------------------------------

10837 rhorvick 1/4/2007 {1/4/2007 3:47:26 PM}added by rhorvick---- D

10836 rhorvick 1/4/2007 {1/4/2007 3:47:22 PM}added by rhorvick

10835 rhorvick 1/4/2007 {1/4/2007 3:47:14 PM}

10834 User2 1/4/2007 {1/4/2007 3:47:00 PM}added by user2

10833 User2 1/4/2007 {1/4/2007 3:46:18 PM}

10832 User1 1/4/2007 {1/4/2007 3:46:00 PM}user 1

10831 User1 1/4/2007 {1/4/2007 3:45:12 PM}added by user1

10830 User1 1/4/2007 {1/4/2007 3:43:52 PM}created by user1

10829 rhorvick 1/4/2007 ----------------VSSConverter Project Mapping

So the migration did properly migrate the changes and attributed them to the proper authors. I don’t know that I’m explaining this correctly – perhaps someone from Admin/Ops can explains it better – but when a checkin occurs and the author the change is being attributed to (which is not necessarily the account making the checkin call) is a non-existing user and the account is not a domain name or a TFS account/group name then a special identity is created which can only be used for the purpose of attributing changesets to (i.e. an audit-only account, not one that can be logged in as).

Now – had I wanted to attribute the changes to a real user I could have altered the mapping file like so:

<?xml version="1.0" encoding="utf-8"?>

<!--

This file is automatically created by VSS Converter. You can optionally use the file to map

a VSS user to a Team Foundation user. For example, <UserMap From="Jane" To="MyDomain\Janep"></UserMap>

This mapping causes all actions logged by VSS user “Jane” to be changed to Team

Foundation user “ MyDomain\Janep ” during migration.

-->

</UserMappings>

And this time when I migrated I see the following history:

Changeset User Date Comment

--------- ------------- ---------- --------------------------------------------

10846 rhorvick 1/4/2007 {1/4/2007 3:47:26 PM}added by rhorvick---- D

10845 rhorvick 1/4/2007 {1/4/2007 3:47:22 PM}added by rhorvick

10844 rhorvick 1/4/2007 {1/4/2007 3:47:14 PM}

10843 rhorvick 1/4/2007 {1/4/2007 3:47:00 PM}added by user2

10842 rhorvick 1/4/2007 {1/4/2007 3:46:18 PM}

10841 rhorvick 1/4/2007 {1/4/2007 3:46:00 PM}user 1

10840 rhorvick 1/4/2007 {1/4/2007 3:45:12 PM}added by user1

10839 rhorvick 1/4/2007 {1/4/2007 3:43:52 PM}created by user1

10838 rhorvick 1/4/2007 ----------------VSSConverter Project Mapping

WSS Rant – Linking to the latest version of a sharepoint document considered harmful. (Lessons 6 and 7)

RobertHorvick — Thu, 04 Jan 2007 17:56:00 GMT

SharePoint does not have persistent hyperlinks for all document revisions.

More specifically – SharePoint has two methods for linking to documents: the canonical path and the revision paths.

The canonical path is the latest copy of the document at the time of the download request. Every time a new revision is created the contents of the canonical path change. Canonical paths look like this:

http://<servername>/Shared%20Documents/document.doc

Revision paths are paths to historical revisions of the document. More specifically they are paths to the non-tip (latest) version of the document. The revision path exists under the _vti_history location and have the version number in the path. They look like this:

http://<servername>/_vti_history/7/Shared%20Documents/document.doc

That link is for revision 7 of the document.

When I first saw this I assumed that all documents were accessible via revision paths, and the canonical path was just an abstraction over the latest revision document. I was incorrect.

You can prove this to yourself by taking any document in a WSS repository and trying to request the latest version by the revision path that would contain its content. For example if you add a new file it has only one revision. So its revision path (if it could have one) would be:

http://<servername>/_vti_history/1/Shared%20Documents/document.doc

Requesting this fails with a HTTP 400 response (HTTP BAD REQUEST). Requesting the canonical path works as expected. If a new revision of the document is created the revision path for version 1 becomes valid and the canonical path now refers to version 2 (and as before – the revision path for version 2 is not valid).

Why is this a problem?

Enumeration of the document library is slow. The time between when we analyze the changes to migrate and the point where we actually perform the migration could be seconds, minutes or hours apart. During that time the document may have been updated. When that happens the version we cared about is demoted from the canonical path down to a revision path. If we downloaded the canonical path we would end up getting the wrong content.

At first my plan was to store the canonical path and then to check during migration to make sure that canonical path is still the version we want before downloading. But that just narrows the window on the race condition – it does not eliminate it.

I ended up doing the following:

When analyzing an item we know if we are looking at the latest version (canonical path) or a historical revision (revision path). We store this fact as a boolean along with the path necessary to download the document (the canonical or revision path) and the version number of the item. During migration we download the item from the stored download path. Next, if the document was the latest version (i.e. a canonical path) we now double check that the canonical path refers to the document version we needed. If it does we are done. If it does not then we enumerate the revision history to find the revision we want. If it exists we download the specific revision and continue. If it does not then the revision was deleted and we handle that condition.

I believe this approach guarantees that the content we end up is always the proper content and does so in the minimum number of round-trips to the server.

All because canonical paths are not just an abstraction over revision history.

Lesson 6: The conceptually simplest operation can end up being incredibly complex to perform correctly.

Lesson 7: When representing data (such as document revisions) make all the data available through a uniform mechanism. Shortcuts to the latest revision of the data should be an abstraction over that mechanism.

WSS Rant - finding out what changed is much more difficult then it should be. (Lessons 4 and 5)

RobertHorvick — Thu, 04 Jan 2007 00:50:00 GMT

In my previous rant I made this statement (quoted out of context):

Migrating WSS content to TFS? Piece of cake. Just enumerate each WSS change and replay the action in TFS. If you are doing an incremental migration then just enumerate those changes that have occurred since the last time you checked for changes.

There are two base actions we need to support:

1) Enumerating all revision history for a document library

2) Enumerating all revision history for a document library after a specific point in time

At the end of the day these boil down to the same requirement (#2) with a default start time that precedes the first revision in the document library.

This should be easy, right?

Now in my mind I think that this should not be a big deal. All of this data is stored in SQL. There is a single SQL query that can be crafted to find this information very efficiently. Even if the document library had millions of items and was located on a slow and unreliable connection … this should be able to run quickly and without a lot of fuss.

Let’s find out!

So let’s go back to the WSS Web Service documentation and start looking for something that looks promising. “Document Workspace” (DWS) is the obvious starting point. When you visit the DWS front page it is sparse to say the least. It includes 5 words. The title “Document Workspace Web Service” and a link “Methods”. No summary of usage or sample code. Not an auspicious start.

So I click on “Methods” and this page has one-fifth the content of the previous page. It simply reads “Methods”.

So here is the first rant.

What the $*#@??!!?? “Methods”? That’s it? Someone decided to publish this? Is this a joke? I’m sure the answer has something to do with automatically generated content and blah-blah-blah.

So I used the search box. I entered “Document Workspace Web Service” into the MSDN search box and the first result in the search engine was this. This looks more promising. We’re up to nearly 20 words now. And that hyperlink looks good.

Lesson 4: MSDN has multiple active pages of documentation for the same logical thing. Some pages are horrible. Some are reasonable. Some are great. When in doubt, search.

Anyway – long (even by my standards) story short … the DWS web service was not what I wanted. It includes methods for creating, deleting and modifying document workspaces. I want to enumerate the contents of one.

Eventually I discovered that what I want to do is use the Site Data web service which has the SiteData class that has a EnumerateFolder method. With the results of all that I could use the Versions web service which exposes a Versions class whose GetVersions method returned what I wanted.

It didn’t take a lot of time to whip up a sample that enumerated the contents of a WSS service. The tedious part was parsing the XML responses (it turns out that parsing responses is a reoccuring theme in this project).

So now I’m sitting here with a code snippet that allows me to enumerate each revision of each item in a WSS document library. But this isn’t really what I wanted. I wanted to know every revision after a specific point in time. I spent a few hours searching the web, reading documentation and talking with folks more knowledgeable than I about WSS, and what I discovered was this…

There is no way to do this in WSS 2.0. WSS 2.0 does not include the search web service (which apparently can be jury-rigged into doing this using query constraints – but I never researched it to prove that). So my options were limited.

"I still haven't found what I'm looking for."

- U2 (The Joshua Tree)

Every time I want to identify changes to migrate I have to enumerate the entire tree structure and prune those items that are older than the range start time.

So while we’re enumerating things let’s enumerate a few problems with this approach:

1) Enumerating is a somewhat expensive operation that will result in something like 2*N round-trips to the server each time the migration process begins (where N is the number of versioned items in the WSS repository).

2) Enumerating is a slow operation (at least we're doing the expensive thing slowly ... right? Or are we just doing the expensive thing for a long time? You decide.).

3) The contents of the server are changing during our enumeration. Remember that we want to identify everything that has occurred since the last migration time. What if the enumeration process takes 8 minutes – should we consider documents added during those 8 minutes? It turns out we should not. If we do, we run the risk of missing or repeating revisions. If you don’t get why – leave me a comment and I’ll explain it in more detail. So really we need to filter based on items that occurred after the previous run and no later than the start of this run.

4) WSS uses one minute timestamp granularity. Why is this important? If three documents were added during the same 1 minute window we need to make sure we either record none of them or all of them. We won’t know how many to capture until the entire minute is over. This means that when we enumerate documents we don’t filter the based on the start time through to the current time – we filter based on the start time through to one minute prior to the current time.

Let 3 and 4 sink in for a minute.

They are a pretty big deal.

We can never know when we are done. The act of looking for a change takes so long that another change may have occurred during that time and we could have missed it. So we need to look again. Which takes a long time. Lather well. Rinse and repeat as necessary.

In summary…

So let’s review our original goal:

“Enumerating all revision history for a document library after a specific point in time”

Which has now become:

“Enumerating all revision history for a document library since a after point in time but before one minute prior to the start of the enumeration period.”

Lesson 5: Just because something can be done efficiently in the native storage system does not mean that an efficient means of doing it will be exposed to the user.

This is a good lesson for the TFS APIs too. We have several areas where it is more difficult than it should be to do things that seem like they should be trivial (searching changeset comments and annotations are two that come to mind quickly).

Is it just me?

When I’m writing code that feels wrong I tend to believe that I am the problem. That I haven’t invested enough time into learning how to use the API effectively. That may well be the case. I’m flying by the seat of my pants here. Am I missing the boat on this? Is there a “EnumerateAllChangeSince” method I didn’t see in the wsdl? Is there a better way to figure out the contents that are interesting for migration? And most importantly – is the criteria I’m using to figure out the interesting items correct?

The SharePoint client object model, chevy pickups and three important lessons.

RobertHorvick — Sat, 30 Dec 2006 07:26:00 GMT

If I could do it all over again I would take auto shop class in high school. I didn’t know it at the time but I really enjoy automotive technology. I have bought several text books used in automotive tech classes and read them cover to cover several times. I have a small library of books on the repair and restoration of Chevy trucks and engines, and Chilton manuals for vehicles I'll never own.

The Chevy Advanced Design (’47-’54) trucks are my hands-down favorite (with the ’51 being my favorite of the mix) – but I’d “settle for” a Judge if the opportunity presented itself.

A few years ago I went and bought a 73 Chevy K10 pickup. It was a good truck. A manly truck. A lime green truck. I intended to get the vanity plate "SOYLENT" but the plate would have been worth more than the truck and nobody seemed to get the joke anyway. I even finished a few projects on it (these should serve as evidence that I’m not a very good mechanic). But when we moved from the Fargo area to North Carolina I sold the truck since moving it was not practical.

For a few years I had been reading all about how to repair a truck. Before buying this truck I mentally rebuilt the Chevy 350 hundreds of times. I had taken countless “end of chapter” tests on identifying and resolving automotive problems. The only thing missing was the truck to prove how much I'd taught myself.

Got an electrical problem? Well that’s simple … consult the wiring diagram to identify the relevant wires and fuses. Check the fuses and connection ends, make sure you are getting power and have a good ground, and aren’t shorting out. If all of that is good then start looking at the component that is not working (e.g. the wiper motor).

But the books never mentioned that the previous owner may have re-wired the system and not color coded the wires properly. Or maybe the ground was cut when the previous owners kid tried to install a new stereo. Or perhaps there is a build-up of rust that is preventing the body from grounding to the frame properly. Or that 30 year old wires fall apart when you touch them.

It seemed so simple on paper. It always does.

So what is the method to enumerate all of the items in WSS? Hmm … wait … how do I even talk to WSS?

And so begins my first rant on WSS.

The WSS converter has a few simple requirements:

1) It will be able to run on a machine other than the TFS application or data tier and it will be able to run on a machine other than the WSS server (which is usually the TFS application tier).

2) It will use the WSS provided APIs to interact with the document libraries.

3) It will not access the WSS database directly.

Prior to this experience what I know about programming SharePoint could have been summed up in one word: “nothing”. So I grabbed a book and hit MSDN. I learned all about the SPWeb class and how to enumerate SPFile objects and man-oh-man life was good. This was going to be easy.

I setup a WSS server on my development box and started writing code. The tests were working great. I was just about to go grab my PM to show him a quick demo of my progress when I decided that it would be much cooler to demonstrate it by grabbing content from the production WSS site. I updated the configuration to point to the remote machine and hit "F5"...

All of the sudden my code, my beautiful code, which had worked so well just moments earlier was dead in the water. The WSS site couldn’t be found. The document libraries didn’t exist. The files were all missing. How could this be? I was pointing at the right machine. The document library existed. I had the proper access. Why wouldn't this work?

I spent almost a full day debugging this before I found this web page on MSDN – http://msdn2.microsoft.com/en-us/library/ms954094.aspx. Please note these two really important phrases:

“You can use the Windows SharePoint Services object model in code running on the server to access and work with list or Web site data and templates, to customize Web Part views of list data, or to manage a top-level site or a subsite.”

“The Windows SharePoint Services Web Service provides interfaces for communicating with the server running Windows SharePoint Services from a remote computer to work with list and site data.”

Remember requirement #1?

I literally yelled. In the event that children are present I won’t say what I yelled. But I’ve heard that it is still hanging in space over Lake Michigan.

A week of learning and I was back to square one.

I took a walk (this was a twelve thousand step day), had a Diet Pepsi and start reading all about the SharePoint web services.

Three important lessons were reinforced from this experience.

Lesson #1: If you need to support running on remote machines then do your testing on a remote machine from the beginning.

Lesson #2: Read the API overview pages on MSDN very carefully before investing time in reading the API.

Lesson #3: When in doubt, double-check that you have a good ground.

I'll be talking about migrating WSS document libraries to TFS source control for a while

RobertHorvick — Sat, 30 Dec 2006 06:20:00 GMT

As Matt has mentioned on the TFS Migration blog we’re working on a migration toolkit and, specifically, a sample converter tool to keep WSS document libraries in sync with a location in the TFS source control server.

One of the most common questions I get when people hear this is “Why a WSS converter? Are people really asking for that? Why not CVS or PVCS or something?”

Well – there are a few reasons. Some good. Some not so much.

1) This is a sample application to demonstrate how to write a converter on the toolkit. It is not necessarily meant to be useful to a broad group of people but rather useful as a code sample.

2) We did not want to create a converter that relied on technology that not every TFS customer would have access to or that would require them to install additional software. We can be pretty well assured that all TFS customers have access to TFS and WSS. This reason gets the response … “then why not TFS to TFS? That would be useful to many people!” … to which I say:

3) Because it is a sample application we needed to pick something that was approachable. WSS doesn’t have concepts like branch and merge (and it barely has rename) so that lowered the bar significantly (or I thought, anyway).

4) There is actually a somewhat cool scenario that this converter opens up – you can check in your document changes at the same time as your source code (to TFS) and it will be mirrored to WSS. Then your PM can make document updates in WSS and the change will be mirrored to TFS. That’s kind of cool … right? I think so anyway.

5) Writing a more complex converter would add months onto the delivery time.

6) I already had a WSS->TFS prototype so the bar seemed like it would be lower than any other option.

Ultimately it was the decision we made. Sorry if you’re not happy. Life isn’t always butterflies and unicorns. Sometimes it’s a WSS converter. I think it was the right decision.

Why am I sharing all this?

Over the next few blog posts I’m going to be sharing some of the problems I’ve run into while working with WSS. This isn’t just to rant. Don’t get me wrong. I fully intend to have these posts included in the Rant category … but I have ulterior motives.

I would love to hear feedback on the issues I describe. Is there a better solution? Did I do something horribly wrong? Stuff that would be great to know BEFORE releasing the code.

Also as I share the problems it is inevitable that I will share some of the design (I’d like to share it all – we’re trying to figure out how to do that). The design isn’t in stone yet. If I’ve done something dumb you can yell at me and I can fix it. More of that “stuff that would be great to know before releasing”.

hexadecimal value 0x07, is an invalid character

RobertHorvick — Mon, 13 Nov 2006 21:49:00 GMT

Every once in a while I get a message that at first baffles me but once I understand the problem it's hand-smackingly obvious. This error message was one of those.

"hexadecimal value 0x07, is an invalid character"

By now I’ve run into this problem a few times and with a few different converters that target the TFS version control server. Each time the fix has been the same so I thought this time I would share the problem and solution.

The problem in two words is “fancy quotes.”

Each time the issue has been that someone did a copy/paste from Excel or Word into the checkin or label comment box in the source version control system. In doing so they brought over “fancy quotes” from the editor they were copying from.

The reason this is a problem is that the TFS client APIs do not escape the data for the caller (and the server does not un-escape it) so we need to handle it on our own (aside – I think this is a TFS bug and I’m writing it up).

Basically before you send free-text to TFS (comments, usernames, etc) you need to make sure that the text only contains data that conforms to the XML recommendation at the W3C.

http://www.w3.org/TR/2004/REC-xml-20040204/#charsets

Easy enough!

--- (more details for our inner-geek and search engine mojo) ---

The following is the relevant portion of a callstack and the complete error message that I’m talking about:

Microsoft.TeamFoundation.VersionControl.Client.VersionControlException: The value contains characters that are not allowed (control characters, 0xFFFE, or 0xFFFF). Please remove those characters. ---> System.ArgumentException: '

', hexadecimal value 0x07, is an invalid character.

at System.Xml.XmlConvert.VerifyCharData(String data, ExceptionType exceptionType)

at System.Xml.XmlCharCheckingWriter.WriteString(String text)

at System.Xml.XmlWriter.WriteElementString(String localName, String ns, String value)

at System.Xml.XmlWriter.WriteElementString(String localName, String value)

at Microsoft.TeamFoundation.VersionControl.Client.Helper.ToXml(XmlWriter writer, String element, String str)

--- End of inner exception stack trace ---

at Microsoft.TeamFoundation.VersionControl.Client.Helper.ToXml(XmlWriter writer, String element, String str)

at Microsoft.TeamFoundation.VersionControl.Client.Changeset.ToXml(XmlWriter writer, String element)

at Microsoft.TeamFoundation.VersionControl.Client.Repository.CheckIn(String workspaceName, String ownerName, String[] serverItems, Changeset info, CheckinNotificationInfo checkinNotificationInfo, CheckinOptions checkinOptions, Failure[]& failures)

at Microsoft.TeamFoundation.VersionControl.Client.Workspace.CheckInInternal(String author, PendingChange[] changes, String comment, CheckinNote checkinNote, WorkItemCheckinInfo[] workItemChanges, PolicyOverrideInfo policyOverride, CheckinOptions checkinOptions)

at Microsoft.TeamFoundation.VersionControl.Client.Workspace.CheckIn(PendingChange[] changes, String author, String comment, CheckinNote checkinNote, WorkItemCheckinInfo[] workItemChanges, PolicyOverrideInfo policyOverride, CheckinOptions checkinOptions)

How long should a version control migration to TFS take?

RobertHorvick — Thu, 09 Nov 2006 23:50:00 GMT

I'm curious what you think.

Imagine you have an installation of Generic Version Control System with 500,000 items you intend to migrate to TFS. Each item has, on average, 3 revisions of history. So you have 1.5 million revisions. And let's say that this was done over 200,000 changsets. Also, if you want, let's say that all 1.5 million revisions amount to 6GB of file content.

If you were going to migrate that data, including all history, how long (in minutes, hours or days) would you expect that to take?

Visual Studio Team System Chat - Wed, Nov 8th

RobertHorvick — Tue, 07 Nov 2006 22:29:00 GMT

There is an upcoming Team System chat on the 8th. I will be involved to field questions on version control migration and the new QueryMergesWithDetails API that became available in SP1.

Join members of the Visual Studio Team System product group to discuss features available in Visual Studio Team Foundation Server, Team Editions for Architects, Developers, Database Pros, and Testers. In addition, discuss what's new in the latest Community Technology Preview (CTP).

Join the chat on Wednesday, November 8^th, 2006 from 10:00am - 11:00am Pacific Time.

Add to Calendar

Additional Time Zones

Choosing a host process type for a version control migration tool

RobertHorvick — Thu, 02 Nov 2006 17:42:00 GMT

Last time I mentioned that we’d look at more architectural decisions for a migration tool – specifically what type of process it should be hosted in.

Some common arguments I hear for the various approaches are:

1) It should be written in a scripting language so the hosting process is the interpreter. This way I can make changes as necessary.

2) It should be a console application so it can run with needing user interaction. This way I can automate it in a script and have it run several migrations sequentially.

3) It should be a Windows application because I don’t want to have to deal with hand editing the configuration and so that I can see a UI that lets me know what is happening.

4) It should be a Windows service because I want to be able to manage it remotely and let it run without having to be logged on to a machine.

Each idea has merits.

Being able to easily change the tool to fit your needs is important. We will design tools to solve the problems we recognize and understand – but our customers may be facing slightly, or even vastly different, problems. Also a migration tool’s target audience is likely developers so it is not unlikely that they would be in a position to fix any bugs they encounter or extend the tool to meet their needs.

Being able to fully automate the migration is critical. A migration process can take days to finish. If we can make that faster by enabling a script to kick off the migration at 2am instead of needing to wait until a person can come in to click “Next” in the morning that is a really big deal.

Not having to hand-edit configuration is also important. It reduces the chance of typos. During migration a typo may mean the difference between being able to use the system you just spent tens of thousands of dollars on or having to send an email letting everyone know that a week of work has been lost.

Reporting and progress information is also very important. Especially when you are doing a multi-day migration and people want to know when they can start using the system.

Remote management and unattended execution are also very important. Leaving a machine running, even locked, may not be feasible in some organizations and remote management can be a great time and resource saver.

So the plan is to try and make everyone happy without creating a monster application that is everything to everyone and has 12 modes of operation.

I propose it should be a service application that has a console entry point (to aide in debugging and provide a simple command line interface for small migrations). Then release the source code, and document the configuration format extremely well. Finally have a real-time reporting system available to let people know exactly what has been done, what is being done, and what is left to do.

This obviously appeases the groups that want a console or service application. It should help the script-minded folks sleep better by sharing the source code (and makes the developer’s job a bit easier by allowing them to work in their development environment and language of choice) and while it does not directly help the folks who don’t want to hand-edit the configuration it does enable the creation of tools to do this at a later time. We’ve already seen that there are folks out there willing to take on this challenge for the benefit of everyone.

And of course by designing the system to decouple the hosting process from the rest of the system, and by releasing the source code and having good docs, if someone wants to change the process host to a Windows application later on they can.

Nothing Earth shattering today.

But I wanted to share the thought process that went into the decision.

TFS Migration has a blog now...

RobertHorvick — Thu, 02 Nov 2006 17:04:00 GMT

A quick pointer to the TFS migration blog: http://blogs.msdn.com/TFS_Migration/

I'm going to keep my thoughts on VC migration issues here but that blog is where more formal announcements regarding the future of TFS migration will be made.

Time travel, 20 questions and the power of free soda.

RobertHorvick — Fri, 27 Oct 2006 20:31:00 GMT

Yesterday I presented 20 questions that I think are a good starting point when designing or evaluating a version control migration product. The point is basically to get folks thinking about what should be migrated, how it should be migrated and what would happen if that data were not there.

Last night I conjured up myself from the past. May 1999 to be specific. The European world was still in shock from the introduction of the Euro, geeks everywhere were still in awe of The Matrix, financial planners were shouting about the benefits of the Roth IRA and I was sitting in my cubical trying to figure out how to rationalize many different version control systems, build processes and development teams into a single (or at least fewer) process resembling something sane. Life was much easier just a few months earlier when I was still in college. The answers were no longer in the back of the book.

I checked.

In 1999 I'm responsible for several large VSS databases (4-8GB), a PVCS system and a custom version control system. All told there is something on the order of 10 years of history on a 22 million LOC project spread throughout multiple systems and we are researching moving to new, single, VC solution. I handed my 1999 self the list of 20 questions and one of the free sodas from a Microsoft cafe.

My 1999 self still has to pay for his caffeine so this seemed like a fair deal.

45 minutes later this is what he gave back… next time I'll share what my 2006 self thought of all this.

1) What versioned items do you absolutely have to have in the target system?

The system must be able to migrate any type of versioned asset (Source code, images, projects, documents, etc) that is stored in the source system as a retrievable directory or file.

Specifically we need to be able to migrate the projects we are currently working on but we might want to back-fill old projects later so that we can refer to the code in the new system.

2) Do you need history for those items?

a. If so – how much?

We do want history for these items. We would like to be able to migrate the full history for the items we choose to migrate. We would also like the ability to start migration a specific point in time or a label. This is because we have one branch “WidgetToolbox” that was completely flattened (everything deleted) at one point because of a botched integration. Since we know that the actions prior to the correction of the branch are all bogus we don’t want to spend time migrating hundreds of thousands of items that are just going to be deleted anyway.

3) What would happen if you did not have history for those items?

Not having history would mean that we would need to keep the source system around for a period of time for reference. This would mean maintaining the hardware and software as well as keeping at least a limited number of licenses current. This will result in over $10,000 per year in licensing costs as well as about $15,000 in maintenance time and one-time hardware costs (since we can’t reuse the existing hardware we will need to buy new hardware for the new system).

Further we anticipate that the average developer will lose 5 hours of time over the next 12 months due to lost history. With 200 developers on staff and a development “burn rate” (the cost of an unproductive developer) of $150 per hour this will result in $150,000 in lost productivity.

4) Do you have internal processes that rely on the ability to query against well-known labels?

a. If so – can the process be changed easily?

We have several processes that rely on version control labels. Some of them will be easy to fix an others not. The most important is a code promotion process where labels are applied at various stages in the development process to indicate code that is ready for test or production. We are researching replacing this with a branching model but we have not finished this research yet and have not yet converted any tools to that new process.

Also we label the source code after each build in order to reproduce builds at a later date. Having these labels is important as it allows developers to recreate an old build when reproducing a bug or debugging an old build.

5) Can those labels be recreated manually in the target system?

They could but there is some risk since we would need to know we did it right. I estimate it would take several days to implement this and test it properly. But the automation tools to do the promotions would still need to be re-written to support the new version control system. We are hesitant to do that when we are also rewriting the tools to support a new promotion model based around branching.

We could live without all of the build labels but there are some key ones we would like – such as publically shipped betas and final releases. If we could select 5 or 10 labels and have only those migrated it would save us a ton of time and make us more confident that we would be able to produce bug fixes in the new system.

6) What would happen if those labels were not available?

If this process were to stop working we would be unable to easily promote code from dev to test to production. This would cause an immediate drop in productivity and create an inability to create new production builds which could breach our customer support contracts.

Not being able to recreate an old build would mean that developers would need to go back to the old system frequently during the first few months since we are still getting frequent feedback from our customers that need to be investigated. Over time the need would probably ease off – but frankly that means that the cost of each incident will probably rise since we wouldn’t be as prepared for using both systems.

7) Do you have a need to have complete and accurate integration history in the target system (i.e. if there was a branch of Foo to Bar in the source system can the new system just have Foo and Bar as unrelated versioned items or does the integration link need to exist?)

It would be nice to have this but I don’t think that it is strictly necessary. Since our promotion model is based around labels right now we would not take a major hit because of this.

8) What would happen if that integration history were lost?

We would have duplicated content in the VC system which could result in a larger then necessary VC database. Since our dataset is not huge (under 100GB) even a doubling is only an additional few thousand in SAN drive costs. However we would need to know ahead of time how much SAN space to allocate to avoid running out of space during the migration.

9) Can you recreate it manually using a baseless merge?

I think this would work. It would allow us to do merges later on but it would not show the merge relationship in the UI which will require us to do merging via the command line going forward.

10) Does your source system make use of complex workspace or branch mappings in order to allow sparse branches or workspaces in a manner that would incompatible with your target system? (e.g. ClearCase’s branching model is fundamentally different than TFS and the Perforce client mapping syntax is more robust than the TFS)

No.

11) Do the source and target systems have operational parity?

a. If not – how will you address migrating those? Examples include:

i. Rename? (TFS does, Perforce does not)

ii. Destroy? (ClearCase does, TFS does not)

iii. Sharing? (VSS does, TFS does not)

iv. Perforce and VSS support keyword expansion, TFS does not

We don’t know but we do expect this to work. We know that keyword expansion will be lost but we expect that content to still migrate properly (i.e. don’t skip the files)

12) Are there naming convention rules that need to be addressed between the source and target system?

a. For example TFS cannot have items that have a path segment that begin with ‘$’

b. Perforce reserves ‘#’ and ‘@’ for revision identifiers

c. TFS has a 260 character path limit (other systems have more, others less)

d. TFS is not case sensitive (E.txt is the same as e.TXT) whereas solutions with unix clients often are case sensitive (E.txt is not the same as e.TXT).

We have identified some files that start path segments with ‘$’ but we can rename these files prior to migration. If we try to migrate the history of these files they will still have that old name. Ideally we’d like to be able to map the names to something else so that when they are encountered they are automatically handled.

13) Will users, security groups and permissions be migrated?

This would be nice but is not a must have. We use security groups for permissions so adding a few manually is not going to be a big deal. Also we don’t have many security policies in place. Basically you can either read all of the source code, or you have R/W access or you have no access.

14) Will items that are locked in the source system remain locked in the target?

We don’t use item locking.

15) Will items that have an encoding associated with them retain that encoding in the target system?

This is very important to us. We have many files that are in specific encodings that we need to retain that encoding in the new system. Additionally we have some files that appear to be text files but which are typed as binary files. So we need to actually encoding, not the auto-detected encoding, to be migrated.

16) Will workspaces be migrated?

Our developers have indicated that this would not be something they need. The amount of time they spend setting up their workspace pales in comparison to the amount of time the download of the workspace content takes. Also they have many unused workspaces on the source system and migrating them over would just bring over additional baggage.

There is one fellow with a really complex workspace who would like to know if a single workspace could be migrated.

17) Will links between items be migrated?

This would be great. Our current system allows linking workitems and changesets together to show that a checking was due to a specific bug. We use this data when generating reports and when reviewing change history.

18) Will changeset metadata be retained?

a. Will timestamps, comments and the original author be retained?

b. If the source system allows custom metadata (e.g. ClearCase) but the target system does not will that metadata simply be lost or captured somehow?

We definitely want the checkin comments retained. We would like the timestamps to be retained but we understand that TFS does not support this.

Having the original author retained would be very useful but we’ve had hundreds of people come and go. We don’t want to have to manage a large mapping file to map non-existent users to valid accounts. We’d like some sort of fall-back mechanism that defaults invalid users to a single account.

19) How does the migration tool respond to failure mid-way through the process?

a. Can it recover and pick up where it left off?

b. Does it require manual intervention to get going again?

c. Is the tool able to consolidate the logs for multiple runs into a single view?

The tool has to be able to recover from a crash and pick up where it left off. Our migration will be rather lengthy and starting over is not an option we would be happy with. We are ok with it requiring a minimal amount of intervention – for example restarting a process. However we would like to not have to do anything if possible.

Log consolidation is very important. We need to be able to report a single status at the end of the migration. Having many logs to sift through would make this nearly impossible. Also we can’t risk having a new log over-write an old log file since that would cause a loss of history that we would not be able to recover from.

20) How long will the migration take (or how will I know how much time is left)?

We aren’t as concerned with how long migration will take as we are with knowing what has been done, what is left to do and with not having to babysit the process continually. 4 days of computer time is practically free if we don’t need to have someone sitting at the machine watching the screen. If we need to be continually checking the system to watch for dialog boxes or command line prompts then we are throwing money away. If there is user interaction needed there should be a notification system that can let us know to look instead of making us check.

It’s Time for the Version Control Migration Feud!

RobertHorvick — Thu, 26 Oct 2006 23:32:00 GMT

Introducing on Team Foundation Server family! We have Work Item Tracking, Team Build, Version Control and the Data Warehouse!” And the “Other System” family! With Proprietary Bug Tracking, Informal Build Process and Your Version Control System of Choice!

On Your Marks! Let’s start the…

VC MIGRATION FEUD!

“We asked 100 people to name the version control artifact they most expected a migration utility to migrate to the target system.”

You’ve just rung in. You have 10 seconds to decide. What gets migrated during a version control migration? The clock is ticking. Can you get the number one answer? Will you even be on the board?

Whether you are writing a VC migration tool or trying to identify the right migration tool to use, this is a fundamentally important question.

What gets migrated?

The easy answer is “migrate the source” – or more generally migrate all of the versioned items in the system. But the easy answer is ambiguous. Does that mean to migrate just the latest versions or to migrate history as well?

Everyone loves history, right? Baseball, apple pie and version control history.

But how much history is enough? Is partial history sufficient? What exactly is partial history? Does it mean history back to a certain point in time or does it mean all points in time but not every versioned item? Both? Something else?

Does it have to be done in a single big-bang migration or can many small migrations be done over time?

So we’ve started to think about how to migrate versioned items but is that all there?

What about labels? Change metadata? Encodings? Integration history? Users? Workspaces? Item locks? Checked out items? Links to external systems (i.e. links that associate a changeset with work items)? Destoryed items (what if both systems don’t support a notion of destroy?)?

There are many questions that need to be answered in order to create or evaluate of a migration tool – my (current) top 20 are:

1) What versioned items do you absolutely have to have in the target system?

2) Do you need history for those items?

a. If so – how much?

3) What would happen if you did not have history for those items?

4) Do you have internal processes that depend on the ability to query against well-known labels?

a. If so – can the process be changed easily?

5) Can those labels be recreated manually in the target system?

6) What would happen if those labels were not available?

7) Do you have a need to have complete and accurate integration history in the target system (i.e. if there was a branch of Foo to Bar in the source system can the new system just have Foo and Bar as unrelated versioned items or does the integration link need to exist?)

8) What would happen if that integration history were lost?

9) Can you recreate it manually using a baseless merge?

10) Does your development environment depend of complex workspace or branch mappings in order to allow sparse branches or workspaces in a manner that would incompatible with your target system? (e.g. ClearCase’s branching model is fundamentally different than TFS and the Perforce client mapping syntax is more robust than the TFS)

11) Do the source and target systems have operational parity?

a. If not – how will you address migrating those? Examples include:

i. Rename? (TFS does, Perforce does not)

ii. Destory? (ClearCase does, TFS does not)

iii. Sharing? (VSS does, TFS does not)

iv. Perforce and VSS support keyword expansion, TFS does not

12) Are there naming convention rules that need to be addressed between the source and target system?

a. For example TFS cannot have items that have a path segment that begin with ‘$’ whereas Perforce reserves ‘#’ and ‘@’ for revision identifiers

b. TFS has a 260 character path limit (other systems have more, others less)

c. TFS is not case sensitive (E.txt is the same as e.TXT) whereas solutions with unix clients often are case sensitive (E.txt is not the same as e.TXT).

d. What would happen if those files were not migrated because of this?

13) Will users, security groups and permissions be migrated?

14) Will items that are locked in the source system remain locked in the target?

15) Will items that have an encoding associated with them retain that encoding in the target system?

16) Will workspaces be migrated?

17) Will links between items be migrated?

18) Will changeset metadata be retained?

a. Will timestamps, comments and the original author be retained?

b. If the source system allows custom metadata (e.g. ClearCase) but the target system does not will that metadata simply be lost or captured somehow?

19) How does the migration tool respond to failure mid-way through the process?

a. Can it recover and pick up where it left off?

b. Does it require manual intervention to get going again?

c. Is the tool able to consolidate the logs for multiple runs into a single view?

20) How long will the migration take (or how will I know how much time is left)?

Start with those. Spend a day thinking about them and see where it goes.

And one last question … What questions am I missing?