Robert Horvick's Weblog : TFS Version Control

Migration Toolkit Pre-Release published on CodePlex

RobertHorvick — Sat, 21 Apr 2007 02:30:00 GMT

Interested in writing a migration tool targeting (or synchronizing with) Team Foundation Server version control or workitem tracking?

Then check out the pre-release of the TFS Migration and Synchronization Toolkit on CodePlex!

http://www.codeplex.com/MigrationSyncToolkit

This drop is the complete source code for the migration toolkit and sample converters. The VC sample is an application that will synchronize WSS document libraries with TFS version control. Developers can check in doc changes with their code and managers can see the updated docs on the WSS server after the mirroring is done (and it works the other way too).

Oh - and this is still under active development (and has known issues) so if you have any feedback it's still early enough for it to be considered for inclusion in the final drop.

And to quote Matt ...

"Please note that this is a prerelease version of the Migration and Synchronization Toolkit. Several features are not complete and as testing has not completed, the code has the potential to contain bugs and overwrite data stored in TFS. Please be careful to use this only in a testing environment and not on live production data."

How to handle VSSConverter UserMap.xml files for users who no longer exist

RobertHorvick — Thu, 18 Jan 2007 18:24:00 GMT

When running the VSS to TFS converter (VSSConverter.exe) one of the most common questions I hear is how to create a UserMap.xml file for users who are no longer with the company or have had their accounts renamed.

My first suggestion is that any customer who has to deal with this issue (or any user mapping issues – or users in a workgroup configuration) should start by getting the hotfix provided in KB 928828. This can be gotten through your CSS (support) contact.

This hotfix changes the way that user authentication is done and is an important fix. It is compatible with both RTM and SP1 however it was not included in SP1 (it was made after the SP1 cut-off date and does include all the VSSconverter fixes already in SP1).

That said - what I’m going to describe is the same in both the RTM and SP1 versions as well as in KB 928828.

The correct way to deal with the users may be as simple as doing nothing. Let me give you an example.

To test this out I created a sample VSS repository and added two users “User1” and “User2”. These were in addition to the default users of “rhorvick” (me) and “Admin”.

I connected as “User1” and created some content. Then as “User2” and created more. And finally as “rhorvick” and created even more.

I then ran the converter in analyze mode. This created the following UserMap.xml file:

<?xml version="1.0" encoding="utf-8"?>

<!--

This file is automatically created by VSS Converter. You can optionally use the file to map

a VSS user to a Team Foundation user. For example, <UserMap From="Jane" To="MyDomain\Janep"></UserMap>

This mapping causes all actions logged by VSS user “Jane” to be changed to Team

Foundation user “ MyDomain\Janep ” during migration.

-->

</UserMappings>

At this point I just ran the conversion – I did not change the mapping file in any way. Please note that User1 and User2 are NOT existing accounts on my TFS server.

With the QFE the conversion created the following history in TFS:

Changeset User Date Comment

--------- ------------- ---------- --------------------------------------------

10837 rhorvick 1/4/2007 {1/4/2007 3:47:26 PM}added by rhorvick---- D

10836 rhorvick 1/4/2007 {1/4/2007 3:47:22 PM}added by rhorvick

10835 rhorvick 1/4/2007 {1/4/2007 3:47:14 PM}

10834 User2 1/4/2007 {1/4/2007 3:47:00 PM}added by user2

10833 User2 1/4/2007 {1/4/2007 3:46:18 PM}

10832 User1 1/4/2007 {1/4/2007 3:46:00 PM}user 1

10831 User1 1/4/2007 {1/4/2007 3:45:12 PM}added by user1

10830 User1 1/4/2007 {1/4/2007 3:43:52 PM}created by user1

10829 rhorvick 1/4/2007 ----------------VSSConverter Project Mapping

So the migration did properly migrate the changes and attributed them to the proper authors. I don’t know that I’m explaining this correctly – perhaps someone from Admin/Ops can explains it better – but when a checkin occurs and the author the change is being attributed to (which is not necessarily the account making the checkin call) is a non-existing user and the account is not a domain name or a TFS account/group name then a special identity is created which can only be used for the purpose of attributing changesets to (i.e. an audit-only account, not one that can be logged in as).

Now – had I wanted to attribute the changes to a real user I could have altered the mapping file like so:

<?xml version="1.0" encoding="utf-8"?>

<!--

This file is automatically created by VSS Converter. You can optionally use the file to map

a VSS user to a Team Foundation user. For example, <UserMap From="Jane" To="MyDomain\Janep"></UserMap>

This mapping causes all actions logged by VSS user “Jane” to be changed to Team

Foundation user “ MyDomain\Janep ” during migration.

-->

</UserMappings>

And this time when I migrated I see the following history:

Changeset User Date Comment

--------- ------------- ---------- --------------------------------------------

10846 rhorvick 1/4/2007 {1/4/2007 3:47:26 PM}added by rhorvick---- D

10845 rhorvick 1/4/2007 {1/4/2007 3:47:22 PM}added by rhorvick

10844 rhorvick 1/4/2007 {1/4/2007 3:47:14 PM}

10843 rhorvick 1/4/2007 {1/4/2007 3:47:00 PM}added by user2

10842 rhorvick 1/4/2007 {1/4/2007 3:46:18 PM}

10841 rhorvick 1/4/2007 {1/4/2007 3:46:00 PM}user 1

10840 rhorvick 1/4/2007 {1/4/2007 3:45:12 PM}added by user1

10839 rhorvick 1/4/2007 {1/4/2007 3:43:52 PM}created by user1

10838 rhorvick 1/4/2007 ----------------VSSConverter Project Mapping

WSS Rant - finding out what changed is much more difficult then it should be. (Lessons 4 and 5)

RobertHorvick — Thu, 04 Jan 2007 00:50:00 GMT

In my previous rant I made this statement (quoted out of context):

Migrating WSS content to TFS? Piece of cake. Just enumerate each WSS change and replay the action in TFS. If you are doing an incremental migration then just enumerate those changes that have occurred since the last time you checked for changes.

There are two base actions we need to support:

1) Enumerating all revision history for a document library

2) Enumerating all revision history for a document library after a specific point in time

At the end of the day these boil down to the same requirement (#2) with a default start time that precedes the first revision in the document library.

This should be easy, right?

Now in my mind I think that this should not be a big deal. All of this data is stored in SQL. There is a single SQL query that can be crafted to find this information very efficiently. Even if the document library had millions of items and was located on a slow and unreliable connection … this should be able to run quickly and without a lot of fuss.

Let’s find out!

So let’s go back to the WSS Web Service documentation and start looking for something that looks promising. “Document Workspace” (DWS) is the obvious starting point. When you visit the DWS front page it is sparse to say the least. It includes 5 words. The title “Document Workspace Web Service” and a link “Methods”. No summary of usage or sample code. Not an auspicious start.

So I click on “Methods” and this page has one-fifth the content of the previous page. It simply reads “Methods”.

So here is the first rant.

What the $*#@??!!?? “Methods”? That’s it? Someone decided to publish this? Is this a joke? I’m sure the answer has something to do with automatically generated content and blah-blah-blah.

So I used the search box. I entered “Document Workspace Web Service” into the MSDN search box and the first result in the search engine was this. This looks more promising. We’re up to nearly 20 words now. And that hyperlink looks good.

Lesson 4: MSDN has multiple active pages of documentation for the same logical thing. Some pages are horrible. Some are reasonable. Some are great. When in doubt, search.

Anyway – long (even by my standards) story short … the DWS web service was not what I wanted. It includes methods for creating, deleting and modifying document workspaces. I want to enumerate the contents of one.

Eventually I discovered that what I want to do is use the Site Data web service which has the SiteData class that has a EnumerateFolder method. With the results of all that I could use the Versions web service which exposes a Versions class whose GetVersions method returned what I wanted.

It didn’t take a lot of time to whip up a sample that enumerated the contents of a WSS service. The tedious part was parsing the XML responses (it turns out that parsing responses is a reoccuring theme in this project).

So now I’m sitting here with a code snippet that allows me to enumerate each revision of each item in a WSS document library. But this isn’t really what I wanted. I wanted to know every revision after a specific point in time. I spent a few hours searching the web, reading documentation and talking with folks more knowledgeable than I about WSS, and what I discovered was this…

There is no way to do this in WSS 2.0. WSS 2.0 does not include the search web service (which apparently can be jury-rigged into doing this using query constraints – but I never researched it to prove that). So my options were limited.

"I still haven't found what I'm looking for."

- U2 (The Joshua Tree)

Every time I want to identify changes to migrate I have to enumerate the entire tree structure and prune those items that are older than the range start time.

So while we’re enumerating things let’s enumerate a few problems with this approach:

1) Enumerating is a somewhat expensive operation that will result in something like 2*N round-trips to the server each time the migration process begins (where N is the number of versioned items in the WSS repository).

2) Enumerating is a slow operation (at least we're doing the expensive thing slowly ... right? Or are we just doing the expensive thing for a long time? You decide.).

3) The contents of the server are changing during our enumeration. Remember that we want to identify everything that has occurred since the last migration time. What if the enumeration process takes 8 minutes – should we consider documents added during those 8 minutes? It turns out we should not. If we do, we run the risk of missing or repeating revisions. If you don’t get why – leave me a comment and I’ll explain it in more detail. So really we need to filter based on items that occurred after the previous run and no later than the start of this run.

4) WSS uses one minute timestamp granularity. Why is this important? If three documents were added during the same 1 minute window we need to make sure we either record none of them or all of them. We won’t know how many to capture until the entire minute is over. This means that when we enumerate documents we don’t filter the based on the start time through to the current time – we filter based on the start time through to one minute prior to the current time.

Let 3 and 4 sink in for a minute.

They are a pretty big deal.

We can never know when we are done. The act of looking for a change takes so long that another change may have occurred during that time and we could have missed it. So we need to look again. Which takes a long time. Lather well. Rinse and repeat as necessary.

In summary…

So let’s review our original goal:

“Enumerating all revision history for a document library after a specific point in time”

Which has now become:

“Enumerating all revision history for a document library since a after point in time but before one minute prior to the start of the enumeration period.”

Lesson 5: Just because something can be done efficiently in the native storage system does not mean that an efficient means of doing it will be exposed to the user.

This is a good lesson for the TFS APIs too. We have several areas where it is more difficult than it should be to do things that seem like they should be trivial (searching changeset comments and annotations are two that come to mind quickly).

Is it just me?

When I’m writing code that feels wrong I tend to believe that I am the problem. That I haven’t invested enough time into learning how to use the API effectively. That may well be the case. I’m flying by the seat of my pants here. Am I missing the boat on this? Is there a “EnumerateAllChangeSince” method I didn’t see in the wsdl? Is there a better way to figure out the contents that are interesting for migration? And most importantly – is the criteria I’m using to figure out the interesting items correct?

hexadecimal value 0x07, is an invalid character

RobertHorvick — Mon, 13 Nov 2006 21:49:00 GMT

Every once in a while I get a message that at first baffles me but once I understand the problem it's hand-smackingly obvious. This error message was one of those.

"hexadecimal value 0x07, is an invalid character"

By now I’ve run into this problem a few times and with a few different converters that target the TFS version control server. Each time the fix has been the same so I thought this time I would share the problem and solution.

The problem in two words is “fancy quotes.”

Each time the issue has been that someone did a copy/paste from Excel or Word into the checkin or label comment box in the source version control system. In doing so they brought over “fancy quotes” from the editor they were copying from.

The reason this is a problem is that the TFS client APIs do not escape the data for the caller (and the server does not un-escape it) so we need to handle it on our own (aside – I think this is a TFS bug and I’m writing it up).

Basically before you send free-text to TFS (comments, usernames, etc) you need to make sure that the text only contains data that conforms to the XML recommendation at the W3C.

http://www.w3.org/TR/2004/REC-xml-20040204/#charsets

Easy enough!

--- (more details for our inner-geek and search engine mojo) ---

The following is the relevant portion of a callstack and the complete error message that I’m talking about:

Microsoft.TeamFoundation.VersionControl.Client.VersionControlException: The value contains characters that are not allowed (control characters, 0xFFFE, or 0xFFFF). Please remove those characters. ---> System.ArgumentException: '

', hexadecimal value 0x07, is an invalid character.

at System.Xml.XmlConvert.VerifyCharData(String data, ExceptionType exceptionType)

at System.Xml.XmlCharCheckingWriter.WriteString(String text)

at System.Xml.XmlWriter.WriteElementString(String localName, String ns, String value)

at System.Xml.XmlWriter.WriteElementString(String localName, String value)

at Microsoft.TeamFoundation.VersionControl.Client.Helper.ToXml(XmlWriter writer, String element, String str)

--- End of inner exception stack trace ---

at Microsoft.TeamFoundation.VersionControl.Client.Helper.ToXml(XmlWriter writer, String element, String str)

at Microsoft.TeamFoundation.VersionControl.Client.Changeset.ToXml(XmlWriter writer, String element)

at Microsoft.TeamFoundation.VersionControl.Client.Repository.CheckIn(String workspaceName, String ownerName, String[] serverItems, Changeset info, CheckinNotificationInfo checkinNotificationInfo, CheckinOptions checkinOptions, Failure[]& failures)

at Microsoft.TeamFoundation.VersionControl.Client.Workspace.CheckInInternal(String author, PendingChange[] changes, String comment, CheckinNote checkinNote, WorkItemCheckinInfo[] workItemChanges, PolicyOverrideInfo policyOverride, CheckinOptions checkinOptions)

at Microsoft.TeamFoundation.VersionControl.Client.Workspace.CheckIn(PendingChange[] changes, String author, String comment, CheckinNote checkinNote, WorkItemCheckinInfo[] workItemChanges, PolicyOverrideInfo policyOverride, CheckinOptions checkinOptions)

How long should a version control migration to TFS take?

RobertHorvick — Thu, 09 Nov 2006 23:50:00 GMT

I'm curious what you think.

Imagine you have an installation of Generic Version Control System with 500,000 items you intend to migrate to TFS. Each item has, on average, 3 revisions of history. So you have 1.5 million revisions. And let's say that this was done over 200,000 changsets. Also, if you want, let's say that all 1.5 million revisions amount to 6GB of file content.

If you were going to migrate that data, including all history, how long (in minutes, hours or days) would you expect that to take?

Visual Studio Team System Chat - Wed, Nov 8th

RobertHorvick — Tue, 07 Nov 2006 22:29:00 GMT

There is an upcoming Team System chat on the 8th. I will be involved to field questions on version control migration and the new QueryMergesWithDetails API that became available in SP1.

Join members of the Visual Studio Team System product group to discuss features available in Visual Studio Team Foundation Server, Team Editions for Architects, Developers, Database Pros, and Testers. In addition, discuss what's new in the latest Community Technology Preview (CTP).

Join the chat on Wednesday, November 8^th, 2006 from 10:00am - 11:00am Pacific Time.

Add to Calendar

Additional Time Zones

Time travel, 20 questions and the power of free soda.

RobertHorvick — Fri, 27 Oct 2006 20:31:00 GMT

Yesterday I presented 20 questions that I think are a good starting point when designing or evaluating a version control migration product. The point is basically to get folks thinking about what should be migrated, how it should be migrated and what would happen if that data were not there.

Last night I conjured up myself from the past. May 1999 to be specific. The European world was still in shock from the introduction of the Euro, geeks everywhere were still in awe of The Matrix, financial planners were shouting about the benefits of the Roth IRA and I was sitting in my cubical trying to figure out how to rationalize many different version control systems, build processes and development teams into a single (or at least fewer) process resembling something sane. Life was much easier just a few months earlier when I was still in college. The answers were no longer in the back of the book.

I checked.

In 1999 I'm responsible for several large VSS databases (4-8GB), a PVCS system and a custom version control system. All told there is something on the order of 10 years of history on a 22 million LOC project spread throughout multiple systems and we are researching moving to new, single, VC solution. I handed my 1999 self the list of 20 questions and one of the free sodas from a Microsoft cafe.

My 1999 self still has to pay for his caffeine so this seemed like a fair deal.

45 minutes later this is what he gave back… next time I'll share what my 2006 self thought of all this.

1) What versioned items do you absolutely have to have in the target system?

The system must be able to migrate any type of versioned asset (Source code, images, projects, documents, etc) that is stored in the source system as a retrievable directory or file.

Specifically we need to be able to migrate the projects we are currently working on but we might want to back-fill old projects later so that we can refer to the code in the new system.

2) Do you need history for those items?

a. If so – how much?

We do want history for these items. We would like to be able to migrate the full history for the items we choose to migrate. We would also like the ability to start migration a specific point in time or a label. This is because we have one branch “WidgetToolbox” that was completely flattened (everything deleted) at one point because of a botched integration. Since we know that the actions prior to the correction of the branch are all bogus we don’t want to spend time migrating hundreds of thousands of items that are just going to be deleted anyway.

3) What would happen if you did not have history for those items?

Not having history would mean that we would need to keep the source system around for a period of time for reference. This would mean maintaining the hardware and software as well as keeping at least a limited number of licenses current. This will result in over $10,000 per year in licensing costs as well as about $15,000 in maintenance time and one-time hardware costs (since we can’t reuse the existing hardware we will need to buy new hardware for the new system).

Further we anticipate that the average developer will lose 5 hours of time over the next 12 months due to lost history. With 200 developers on staff and a development “burn rate” (the cost of an unproductive developer) of $150 per hour this will result in $150,000 in lost productivity.

4) Do you have internal processes that rely on the ability to query against well-known labels?

a. If so – can the process be changed easily?

We have several processes that rely on version control labels. Some of them will be easy to fix an others not. The most important is a code promotion process where labels are applied at various stages in the development process to indicate code that is ready for test or production. We are researching replacing this with a branching model but we have not finished this research yet and have not yet converted any tools to that new process.

Also we label the source code after each build in order to reproduce builds at a later date. Having these labels is important as it allows developers to recreate an old build when reproducing a bug or debugging an old build.

5) Can those labels be recreated manually in the target system?

They could but there is some risk since we would need to know we did it right. I estimate it would take several days to implement this and test it properly. But the automation tools to do the promotions would still need to be re-written to support the new version control system. We are hesitant to do that when we are also rewriting the tools to support a new promotion model based around branching.

We could live without all of the build labels but there are some key ones we would like – such as publically shipped betas and final releases. If we could select 5 or 10 labels and have only those migrated it would save us a ton of time and make us more confident that we would be able to produce bug fixes in the new system.

6) What would happen if those labels were not available?

If this process were to stop working we would be unable to easily promote code from dev to test to production. This would cause an immediate drop in productivity and create an inability to create new production builds which could breach our customer support contracts.

Not being able to recreate an old build would mean that developers would need to go back to the old system frequently during the first few months since we are still getting frequent feedback from our customers that need to be investigated. Over time the need would probably ease off – but frankly that means that the cost of each incident will probably rise since we wouldn’t be as prepared for using both systems.

7) Do you have a need to have complete and accurate integration history in the target system (i.e. if there was a branch of Foo to Bar in the source system can the new system just have Foo and Bar as unrelated versioned items or does the integration link need to exist?)

It would be nice to have this but I don’t think that it is strictly necessary. Since our promotion model is based around labels right now we would not take a major hit because of this.

8) What would happen if that integration history were lost?

We would have duplicated content in the VC system which could result in a larger then necessary VC database. Since our dataset is not huge (under 100GB) even a doubling is only an additional few thousand in SAN drive costs. However we would need to know ahead of time how much SAN space to allocate to avoid running out of space during the migration.

9) Can you recreate it manually using a baseless merge?

I think this would work. It would allow us to do merges later on but it would not show the merge relationship in the UI which will require us to do merging via the command line going forward.

10) Does your source system make use of complex workspace or branch mappings in order to allow sparse branches or workspaces in a manner that would incompatible with your target system? (e.g. ClearCase’s branching model is fundamentally different than TFS and the Perforce client mapping syntax is more robust than the TFS)

No.

11) Do the source and target systems have operational parity?

a. If not – how will you address migrating those? Examples include:

i. Rename? (TFS does, Perforce does not)

ii. Destroy? (ClearCase does, TFS does not)

iii. Sharing? (VSS does, TFS does not)

iv. Perforce and VSS support keyword expansion, TFS does not

We don’t know but we do expect this to work. We know that keyword expansion will be lost but we expect that content to still migrate properly (i.e. don’t skip the files)

12) Are there naming convention rules that need to be addressed between the source and target system?

a. For example TFS cannot have items that have a path segment that begin with ‘$’

b. Perforce reserves ‘#’ and ‘@’ for revision identifiers

c. TFS has a 260 character path limit (other systems have more, others less)

d. TFS is not case sensitive (E.txt is the same as e.TXT) whereas solutions with unix clients often are case sensitive (E.txt is not the same as e.TXT).

We have identified some files that start path segments with ‘$’ but we can rename these files prior to migration. If we try to migrate the history of these files they will still have that old name. Ideally we’d like to be able to map the names to something else so that when they are encountered they are automatically handled.

13) Will users, security groups and permissions be migrated?

This would be nice but is not a must have. We use security groups for permissions so adding a few manually is not going to be a big deal. Also we don’t have many security policies in place. Basically you can either read all of the source code, or you have R/W access or you have no access.

14) Will items that are locked in the source system remain locked in the target?

We don’t use item locking.

15) Will items that have an encoding associated with them retain that encoding in the target system?

This is very important to us. We have many files that are in specific encodings that we need to retain that encoding in the new system. Additionally we have some files that appear to be text files but which are typed as binary files. So we need to actually encoding, not the auto-detected encoding, to be migrated.

16) Will workspaces be migrated?

Our developers have indicated that this would not be something they need. The amount of time they spend setting up their workspace pales in comparison to the amount of time the download of the workspace content takes. Also they have many unused workspaces on the source system and migrating them over would just bring over additional baggage.

There is one fellow with a really complex workspace who would like to know if a single workspace could be migrated.

17) Will links between items be migrated?

This would be great. Our current system allows linking workitems and changesets together to show that a checking was due to a specific bug. We use this data when generating reports and when reviewing change history.

18) Will changeset metadata be retained?

a. Will timestamps, comments and the original author be retained?

b. If the source system allows custom metadata (e.g. ClearCase) but the target system does not will that metadata simply be lost or captured somehow?

We definitely want the checkin comments retained. We would like the timestamps to be retained but we understand that TFS does not support this.

Having the original author retained would be very useful but we’ve had hundreds of people come and go. We don’t want to have to manage a large mapping file to map non-existent users to valid accounts. We’d like some sort of fall-back mechanism that defaults invalid users to a single account.

19) How does the migration tool respond to failure mid-way through the process?

a. Can it recover and pick up where it left off?

b. Does it require manual intervention to get going again?

c. Is the tool able to consolidate the logs for multiple runs into a single view?

The tool has to be able to recover from a crash and pick up where it left off. Our migration will be rather lengthy and starting over is not an option we would be happy with. We are ok with it requiring a minimal amount of intervention – for example restarting a process. However we would like to not have to do anything if possible.

Log consolidation is very important. We need to be able to report a single status at the end of the migration. Having many logs to sift through would make this nearly impossible. Also we can’t risk having a new log over-write an old log file since that would cause a loss of history that we would not be able to recover from.

20) How long will the migration take (or how will I know how much time is left)?

We aren’t as concerned with how long migration will take as we are with knowing what has been done, what is left to do and with not having to babysit the process continually. 4 days of computer time is practically free if we don’t need to have someone sitting at the machine watching the screen. If we need to be continually checking the system to watch for dialog boxes or command line prompts then we are throwing money away. If there is user interaction needed there should be a notification system that can let us know to look instead of making us check.

It’s Time for the Version Control Migration Feud!

RobertHorvick — Thu, 26 Oct 2006 23:32:00 GMT

Introducing on Team Foundation Server family! We have Work Item Tracking, Team Build, Version Control and the Data Warehouse!” And the “Other System” family! With Proprietary Bug Tracking, Informal Build Process and Your Version Control System of Choice!

On Your Marks! Let’s start the…

VC MIGRATION FEUD!

“We asked 100 people to name the version control artifact they most expected a migration utility to migrate to the target system.”

You’ve just rung in. You have 10 seconds to decide. What gets migrated during a version control migration? The clock is ticking. Can you get the number one answer? Will you even be on the board?

Whether you are writing a VC migration tool or trying to identify the right migration tool to use, this is a fundamentally important question.

What gets migrated?

The easy answer is “migrate the source” – or more generally migrate all of the versioned items in the system. But the easy answer is ambiguous. Does that mean to migrate just the latest versions or to migrate history as well?

Everyone loves history, right? Baseball, apple pie and version control history.

But how much history is enough? Is partial history sufficient? What exactly is partial history? Does it mean history back to a certain point in time or does it mean all points in time but not every versioned item? Both? Something else?

Does it have to be done in a single big-bang migration or can many small migrations be done over time?

So we’ve started to think about how to migrate versioned items but is that all there?

What about labels? Change metadata? Encodings? Integration history? Users? Workspaces? Item locks? Checked out items? Links to external systems (i.e. links that associate a changeset with work items)? Destoryed items (what if both systems don’t support a notion of destroy?)?

There are many questions that need to be answered in order to create or evaluate of a migration tool – my (current) top 20 are:

1) What versioned items do you absolutely have to have in the target system?

2) Do you need history for those items?

a. If so – how much?

3) What would happen if you did not have history for those items?

4) Do you have internal processes that depend on the ability to query against well-known labels?

a. If so – can the process be changed easily?

5) Can those labels be recreated manually in the target system?

6) What would happen if those labels were not available?

7) Do you have a need to have complete and accurate integration history in the target system (i.e. if there was a branch of Foo to Bar in the source system can the new system just have Foo and Bar as unrelated versioned items or does the integration link need to exist?)

8) What would happen if that integration history were lost?

9) Can you recreate it manually using a baseless merge?

10) Does your development environment depend of complex workspace or branch mappings in order to allow sparse branches or workspaces in a manner that would incompatible with your target system? (e.g. ClearCase’s branching model is fundamentally different than TFS and the Perforce client mapping syntax is more robust than the TFS)

11) Do the source and target systems have operational parity?

a. If not – how will you address migrating those? Examples include:

i. Rename? (TFS does, Perforce does not)

ii. Destory? (ClearCase does, TFS does not)

iii. Sharing? (VSS does, TFS does not)

iv. Perforce and VSS support keyword expansion, TFS does not

12) Are there naming convention rules that need to be addressed between the source and target system?

a. For example TFS cannot have items that have a path segment that begin with ‘$’ whereas Perforce reserves ‘#’ and ‘@’ for revision identifiers

b. TFS has a 260 character path limit (other systems have more, others less)

c. TFS is not case sensitive (E.txt is the same as e.TXT) whereas solutions with unix clients often are case sensitive (E.txt is not the same as e.TXT).

d. What would happen if those files were not migrated because of this?

13) Will users, security groups and permissions be migrated?

14) Will items that are locked in the source system remain locked in the target?

15) Will items that have an encoding associated with them retain that encoding in the target system?

16) Will workspaces be migrated?

17) Will links between items be migrated?

18) Will changeset metadata be retained?

a. Will timestamps, comments and the original author be retained?

b. If the source system allows custom metadata (e.g. ClearCase) but the target system does not will that metadata simply be lost or captured somehow?

19) How does the migration tool respond to failure mid-way through the process?

a. Can it recover and pick up where it left off?

b. Does it require manual intervention to get going again?

c. Is the tool able to consolidate the logs for multiple runs into a single view?

20) How long will the migration take (or how will I know how much time is left)?

Start with those. Spend a day thinking about them and see where it goes.

And one last question … What questions am I missing?

Writing Version Control Migration Tools - Handling Namespace Cycles

RobertHorvick — Wed, 25 Oct 2006 23:10:00 GMT

Last time I discussed the issues surrounding namespace conflicts when designing a version control migration tool. The basic lesson was that there are times when operations must be executed in a specific order for them to migrate properly. But what if there are namespace collisions for which there is no order that resolves the collision?

Then we must be talking about operational cycles.

Consider the following scenario.

There exist two items in the VC system:

$/Project/Item1.txt

$/Project/Item2.txt

On these two items the following operations occurred:

> tf rename $/Project/Item1.txt $/Project/temp.txt

> tf rename $/Project/Item2.txt $/Project/Item1.txt

> tf rename $/Project/temp.txt $/Project/Item2.txt

> tf checkin /i

When this is checked in there are two pending changes and the end result is:

rename $/Project/Item1.txt (was $/Project/Item2.txt)

rename $/Project/Item2.txt (was $/Project/Item1.txt)

Think back on the algorithm we looked at in the previous post. We look at each change item and attempt to migrate it in place. In this case we would attempt to pend a rename from item1 to item2 but that would fail because item2 already exists.

So what do we do?

We need to identify and resolve the cycle.

Identifying The Cycle

Identifying a cycle means you need to look at every item’s source and target name (i.e. Item1.txt and Item2.txt). For simple actions like add and rename they will be the same – but if the action is complex such as a rename or merge they will differ.

So for each item you need to compare if its target name is the same as some other actions source item. If it is then the action whose source name conflicts needs to happen before the action whose target name conflicts.

So using the example from the previous post:

rename foo bar

add foo

“foo” is the target action of the add and the source action of the rename. So the rename must happen first. Since the add has the same source and target name it cannot conflict and is therefore the end of the conflict chain.

So identifying a cycle means that you need to figure out which items are involved in the cycle and which ones can break it. If none can break it then you need to add intermediate actions to do the work for you.

This picture represents the 2 item cycle of renames described earlier (Item1 and Item2):

Identifying this cycle is pretty straight forward. First we look at Item1 and see that its source name is “Item1” and its target name is “Item2”. Then we look at Item2 and see that its source name is “Item2” and its target name is “Item1”. So we have identified a two item cycle.

Breaking the Cycle

Resolving a 2 item cycle is straight forward. You simply inject a third operation between the two that collide.

The following image demonstrates this:

It also demonstrates my decidedly poor artistic talent (and my ability to (ab)use Power Point).

But don’t go thinking that every cycle is a two item cycle. Any cycle can be N items long – here is an example of a 4 item cycle

As I used up the extent of my artistic talent on the first three pictures I don’t have an image showing the breaking of a four item cycle. But it’s sufficient to say that breaking the four item cycle is just a matter of identifying multiple two item cycles and breaking each one.

So what’s the bottom line here?

1) Designing to handle cycles must happen early in the migration design process as it is fundamental to success.

2) If you can identify and break cycle conflicts you can identify and resolve non-cyclic conflicts using the same algorithm.

3) Cycles can have N nodes long so be prepared for the worst (but rarely are cycles more than 2 nodes long).

4) Breaking an N node cycle is just the repeated process of identifying and breaking 2 node cycles.

5) Breaking cycles means you must be able to pend actions on the target system that aren’t in the formal record of the source system (i.e. two recorded rename operations become three executed rename operations on the target system).

Writing Version Control Migration Tools - Handling Namespace Conflicts

RobertHorvick — Tue, 24 Oct 2006 21:30:00 GMT

Migrating from one version control system to another is tough. I don’t care what the internet forums are saying or what Joe from down the hall told you. It’s hard. Very hard. Deceptively hard.

The obvious algorithm looks trivial:

FOR EACH Changeset CS in History DO

FOR EACH Change C in Changeset CS DO

SWITCH C.Action

CASE ADD:

DownloadFromSource(C)

PendAddOnTarget(C)

BREAK

CASE EDIT:

DownloadFromSource(C)

PendEditOnTarget(C)

BREAK

... and so on ...

Piece of cake. I’ll have it ready by noon.

The problem is that this algorithm falls over pretty quickly when presented with even relatively trivial changesets.

Over the next few posts I’ll present a few examples of changes that cause that algorithm to fall over and what some of the options are. These are not pathological cases that never happen in the real world. These are real examples that I see “in the wild” very frequently.

We’ll start with a simple change that has a namespace collision (i.e. the same namespace, “foo” in this case, is involved in two different operations in a single changeset). Foo is renamed to “bar” and then a new item named “foo” is added.

rename foo bar

add foo

In the above presented algorithm there is no guarantee that the rename of foo to bar would occur before the add of foo. If it does not occur first then the add will fail since there is already an item in that namespace.

Further the target system needs to support this. Imagine you were mirroring TFS to Perforce and had that sequence of operations. The Perforce equivalent would be:

integrate foo bar

delete foo

add foo

See the problem? Perforce doesn’t allow pending an add where you already have a delete. So you need to start making decisions. Do you do something like:

integrate foo bar

edit foo

Well – that’s nice only now you have an integration relationship between two otherwise unrelated items. How about this:

add bar

edit foo

Ok – that works. Only now you’ve lost the history that says that bar used to be foo. Maybe the answer is multiple changesets:

integrate foo bar

delete foo

submit

add foo

submit

Alright – but now you’ve split a single source operation into two target operations. Is that acceptable? If it is – what if the process crashed after the first submit. Will the migration tool have enough context when it is restarted to finish the second change? What if a length of time passed between the two submits and another foo was added? What should the checkin comment for the first and second changes be? Should the first indicate that there will be a second? Should the second indicate that it is part of a multiple checkin sequence? What if there are hundreds of files like this? Does each get its own checkin? Can you group them?

So what is the bottom line here? Two things:

1) When migrating there may be an implied order to the operations that must be satisfied for the change to work.

2) System feature parity makes migrating much more complex.

Next time: cycles – what they are, how to identify them and what you can do about them.

Team Foundation Power Toys release - Tree diff and annotate among the list!

RobertHorvick — Fri, 08 Sep 2006 14:29:00 GMT

The updated Team Foundation Power Toy (TFPT) released on 9/7. It includes 12 powertoys - they are:

Unshelve – adds merge capability to the existing Unshelve command
Rollback – allows you to undo a checkin of a changeset
Online – allows syncing of files that you’ve edited while not connected to a server
GetCS - gets all the items listed in a specific changeset
UU – undo unchanged. Undo checkout with a check for what’s actually unchanged
Review – code review tool, review files in any order you choose
History – display version history of a file, folder or branch
Workitem – create, view or update a work item from the command line
Query – run a work item query and display the results
Bind – convert VSS-bound solutions into TFS-bound solutions
TreeDiff – Compare folders and reconcile differences
Annotate – show who last changed a section of code

TreeDiff, annotate, workitem and query, and bind are four new power toys that I am really excited about. Tree diff is a very frequently requested feature and annotate just makes life easier. Workitem and query make accessing workitems from the command line easier. Bind will convert VSS bindings to TFS bindings after a VSS conversion to TFS.

The best part is that it's a stand-alone download. No longer will you have to install a 300MB package to get tfpt.

Check out the download page:

http://www.microsoft.com/downloads/details.aspx?FamilyId=3F21144A-4E98-4CE0-830F-D1F3E8AC9D67&displaylang=en

See Rob's and Brian's blogs for additional details.

Mario is blogging about TFS Version Control

RobertHorvick — Tue, 15 Aug 2006 16:40:00 GMT

Mario has begun a blog about TFS Version Control. If this is a topic that interests you I would highly recomend checking it out.

http://blogs.msdn.com/mrod/

Reporting VSSConverter Issues

RobertHorvick — Mon, 14 Aug 2006 21:06:00 GMT

Have you encountered an issue when using the VSSConveter (used to convert VSS source history to TFS)? Need a little help working past a blocking issue? The purpose of this post is to give you a few ideas about where to go for additional information and the best way to report the issue if you are unable to work through it.

First the obvious – make sure you have read the latest documentation:

http://msdn2.microsoft.com/en-us/library/ms253060.aspx

Next you should try searching the Team Foundation Server – Version Control forums. Chances are someone else has already had this issue and the answer may be there:

http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=478&SiteID=1

Search for “VSSConverter” and include your specific error message for the best results.

Still stumped? Before you call support try posting the issue on the forums.

The remainder of this post explains a simple three-step process you can use to generate additional information about the error (and will save time in assisting in diagnosing the issue).

1) Enable verbose logging

Verbose logging can be enabled by opening the VSSConverter.exe.config file (located next to VSSConverter.exe) and changing the diagnostic logging switchs to have level "4" logging (it is level 2 by default) and to ensure that a trace listener is configured (it should be by default).

The following is a configuration file that demonstrates this:

<system.diagnostics>
    <switches>
      <add name="MinLevelForAllSource" value="4" />
      <add name="Common" value="4" />
      <add name="VersionControl" value="4" />
      <add name="VSS" value="4" />
    </switches>
    <trace autoflush="true" indentsize="2">
      <listeners>
        <add name="VSSConverterTraceListener"
             type="Microsoft.TeamFoundation.Converters.Utility.LogListener,VSSConverter"
             initializeData="VSSConverter.log" />
      </listeners>
    </trace>
</system.diagnostics>
</configuration>

2) Reproduce the issue

Whatever you did to encounter the error … do it again. With verbose logging enabled there should be a good record of what occurred. In the sample configuration file above the log file will be named VSSConverter.log.

3) Report the issue on the MSDN forums (which I check 2-3x daily for migration issues).

I check the forums for migration issues a few times every business day. Post in the Version Control forum (link above) and include the word “VSSConverter” in your subject line and it will get my attention quickly.

When reporting the issue be sure to include either the entire log file (if it is small enough), a link to the log file or the relevant portions of the log file (if you can isolate the issue).

Using the forums does require having a Passport account to login. If you do not have one please take this opportunity to create one.

By following these three simple steps you will be making it easier to help diagnose your issue and suggest a resolution (otherwise the first thing I’ll ask for is the verbose log), and you will be helping improve the converter quality by letting us know what issues people are encountering.

Left/Right Only Diffing using recursive node compression

RobertHorvick — Mon, 17 Apr 2006 17:07:00 GMT

While we’re talking about diffing … now I needed to be able to diff two local directories with a few requirements …

1) determine left/right only items

2) determine the optimal recursive compression of those items

3) print some status one in a while

4) distinguish between compressed and non-compressed items in the result set

What is optimal recursive compression? Basically it says this:

If I have:

left-only: \path\foo\bar\file1.txt

left-only: \path\foo\bar\baz\file2.txt

left-only: \path\foo\bar

All I need to track is:

left-only: \path\foo\bar

Since it is left-only anything under it is implicitly left-only.

Make sense?

This is pretty easy with a simple tree structure. There are two types of nodes in this tree – terminal and non-terminal.

In the example above the following is the tree structure before pruning:

<root>

Path

Foo

Bar [t]

/ \

File1.txt [t] Baz

File2.txt [t]

It should be clear that the top-most terminal node is Bar so everything beneath it can be cut from the tree. This pruning can be done while the tree is being built or afterwards. The implementation I choose is to do it during. Obviously input order can’t be an issue (it should not matter if the terminal node comes in before or after non-terminal children) so you prune on the fly [which can mean deleting children when a terminal parent is found or discarding children of a terminal parent).

Ok – you can see the code. It’s the PruningTreeNode<T> and PruningTree<T> class in the example below.

Also I decided not to reinvent the wheel and used Eric Gunnerson’s DirectoryWalker class from his book “A Programmer’s Introduction to C#”

Oh – there is also the question of “Why do you want recursive node compression?”

Simple … TFS is designed to handle recursive cases. It’s cheaper to operate on a directory with 1000 children then to operate on all 1000 children and the parent directory individually.

On to the code (normal caveats about this being untested and how it works for me but I make no promises about what it might do on your machine)…

using System;

using System.Collections.Generic;

using System.Text;

using System.IO;

namespace locallocaldiff

{

class Program

{

static void Main(string[] args)

{

if (args.Length != 2)

{

Console.Error.WriteLine("Usage: locallocaldiff <path> <path>");

Environment.Exit(1);

}

Console.WriteLine("comment: start time: {0}", DateTime.Now);

System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();

sw.Start();

try

{

ProcessDiffs(

args[0].TrimEnd(new char[] { '\\', '/' }),

args[1].TrimEnd(new char[] { '\\', '/' }));

}

catch (LocalLocalDiffException e)

{

Console.Error.WriteLine(e.Message);

Environment.Exit(1);

}

Console.WriteLine("comment: Elapsed time: {0}", sw.Elapsed);

}

private static void ProcessDiffs(string lhsPath, string rhsPath)

{

if (!Directory.Exists(lhsPath))

{

throw new LocalLocalDiffException(string.Format("directory not found: {0}", rhsPath));

}

if (!Directory.Exists(rhsPath))

{

throw new LocalLocalDiffException(string.Format("directory not found: {0}", rhsPath));

}

Console.WriteLine("comment: Diffing item sets");

DiffItemSets(lhsPath, rhsPath);

}

private static Dictionary<string, bool> CreateLocalHash(string localPath)

{

Dictionary<string, bool> resultHash = new Dictionary<string, bool>();

DirectoryWalker dw = new DirectoryWalker(

new DirectoryWalker.ProcessDirCallback(delegate(DirectoryInfo d, int level, object obj)

{

printStatus(d.FullName);

Dictionary<string, bool> results = (Dictionary<string, bool>)obj;

results[getNormalizedLocalPath(localPath, d.FullName)] = true;

}),

new DirectoryWalker.ProcessFileCallback(delegate(FileInfo f, int level, object obj)

{

printStatus(f.FullName);

Dictionary<string, bool> results = (Dictionary<string, bool>)obj;

results[getNormalizedLocalPath(localPath, f.FullName)] = false;

})

);

dw.Walk(localPath, resultHash);

return resultHash;

}

private static string getNormalizedLocalPath(string rootPath, string filePath)

{

return filePath.Substring(rootPath.Length).Replace('\\', '/');

}

static DateTime lastMessage = DateTime.Now;

private static void printStatus(string msg)

{

if (DateTime.Now.Subtract(lastMessage).Seconds > 30)

{

Console.WriteLine("comment: {0}", msg);

lastMessage = DateTime.Now;

}

private static void DiffItemSets(string lhsRootPath, string rhsRootPath)

{

int same = 0;

int lonly = 0;

int ronly = 0;

Console.WriteLine("comment: loading {0}", lhsRootPath);

Dictionary<string, bool> lhsHash = CreateLocalHash(lhsRootPath);

Console.WriteLine("comment: loading {0}", rhsRootPath);

Dictionary<string, bool> rhsHash = CreateLocalHash(rhsRootPath);

PruningTree<Pair<string, bool>> lhsOnlyTree = new PruningTree<Pair<string, bool>>();

Console.WriteLine("comment: running diff algorithm");

foreach (string lPath in lhsHash.Keys)

{

printStatus(lPath);

bool lhsIsDir = lhsHash[lPath];

bool rhsIsDir;

if (rhsHash.TryGetValue(lPath, out rhsIsDir))

{

if (lhsIsDir == rhsIsDir)

{

same++;

}

rhsHash.Remove(lPath);

}

else

{

lhsOnlyTree.Add(

lPath.Split(new char[] { '\\' }, StringSplitOptions.RemoveEmptyEntries),

new Pair<string,bool>(lPath, lhsIsDir));

lonly++;

}

foreach (Pair<string, bool> lhOnlyitem in lhsOnlyTree)

{

string verb = (lhOnlyitem.Second) ? "left-only-r" : "left-only";

Console.WriteLine("{0}: {1}", verb, lhOnlyitem.First);

}

PruningTree<Pair<string, bool>> rhsOnlyTree = new PruningTree<Pair<string, bool>>();

foreach (string rPath in rhsHash.Keys)

{

rhsOnlyTree.Add(

rPath.Split(new char[] { '\\', '/' }, StringSplitOptions.RemoveEmptyEntries),

new Pair<string, bool>(rPath, rhsHash[rPath]));

ronly++;

}

foreach (Pair<string, bool> rhOnlyitem in rhsOnlyTree)

{

string verb = (rhOnlyitem.Second) ? "right-only-r" : "right-only";

Console.WriteLine("{0}: {1}", verb, rhOnlyitem.First);

}

Console.WriteLine("summary: Compared {0} items", ronly + lonly + same);

Console.WriteLine("summary: Same: {0}", same);

Console.WriteLine("summary: left only: {0}", lonly);

Console.WriteLine("summary: right only: {0}", ronly);

}

class PruningTreeNode<T>

{

bool terminal;

string nodeValue;

T tagValue;

List<PruningTreeNode<T>> children = null;

public PruningTreeNode(string value, bool isTerminal, T tag)

{

nodeValue = value;

terminal = isTerminal;

tagValue = tag;

}

public T Tag

{

get

{

return tagValue;

}

set

{

tagValue = value;

}

public string Value

{

get

{

return nodeValue;

}

public bool Terminal

{

get

{

return terminal;

}

set

{

terminal = value;

}

public List<PruningTreeNode<T>> Children

{

get

{

if (children == null)

{

children = new List<PruningTreeNode<T>>();

}

return children;

}

public bool HasChildren

{

get

{

return (!terminal && children != null && children.Count > 0);

}

public bool TryGetChild(string childValue, out PruningTreeNode<T> item)

{

bool has = false;

item = null;

if (!terminal && children != null)

{

foreach (PruningTreeNode<T> node in children)

{

if (node.Value == childValue)

{

item = node;

has = true;

break;

}

return has;

}

public override string ToString()

{

if (terminal)

{

return tagValue.ToString();

}

else

{

return string.Format("Non-terminal: {0}", nodeValue);

}

class PruningTree<T> : IEnumerable<T>

{

PruningTreeNode<T> root;

public PruningTree()

{

root = new PruningTreeNode<T>(null, false, default(T));

}

public void Add(string[] paths, T tag)

{

int depth = paths.Length;

PruningTreeNode<T> current = root;

if (depth > 0)

{

int i = 0;

for (; i < depth; i++)

{

if (current.Terminal)

{

// we hit a terminal node so we don't have to keep looking

break;

}

PruningTreeNode<T> tempNode;

if (current.TryGetChild(paths[i], out tempNode))

{

current = tempNode;

if (i == depth - 1)

{

// PRUNING children ...

// we are at the terminal node of the list and the node is

// in the tree - so either there are children and this is a

// terminal parent (prune) or there is a duplicate in the list

// (favor the last one seen)

current.Children.Clear();

current.Terminal = true;

current.Tag = tag;

}

else

{

// Adding a new node - may or may not be terminal.

bool terminal = (i == depth - 1);

T tagTemp = (terminal) ? tag : default(T);

tempNode = new PruningTreeNode<T>(paths[i], terminal, tagTemp);

current.Children.Add(tempNode);

current = tempNode;

}

private void flattenTree_internal(PruningTreeNode<T> current, List<PruningTreeNode<T>> nodes)

{

if (current.Terminal)

{

nodes.Add(current);

}

else if (current.HasChildren)

{

foreach (PruningTreeNode<T> child in current.Children)

{

flattenTree_internal(child, nodes);

}

private List<PruningTreeNode<T>> flattenTree()

{

List<PruningTreeNode<T>> nodes = new List<PruningTreeNode<T>>();

flattenTree_internal(root, nodes);

return nodes;

}

#region IEnumerable<T> Members

public IEnumerator<T> GetEnumerator()

{

List<PruningTreeNode<T>> nodes = flattenTree();

foreach (PruningTreeNode<T> node in nodes)

{

yield return node.Tag;

}

#endregion

#region IEnumerable Members

System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()

{

return GetEnumerator();

}

#endregion

}

internal class LocalLocalDiffException : Exception

{

public LocalLocalDiffException(string msg)

: base(msg)

{ }

}

internal class Pair<T1, T2>

{

public T1 First;

public T2 Second;

public Pair(T1 first, T2 second)

{

First = first;

Second = second;

}

A Programmer's Introduction to C# (Second Edition)

by Eric Gunnerson

Publisher: Apress L.P.

ISBN: 1-893115-62-3

// 32 - .NET Frameworks Overview\InputOutput\Traversing Directories

internal class DirectoryWalker

{

public delegate void ProcessDirCallback(DirectoryInfo dir, int level, object obj);

public delegate void ProcessFileCallback(FileInfo file, int level, object obj);

public DirectoryWalker(ProcessDirCallback dirCallback,

ProcessFileCallback fileCallback)

{

this.dirCallback = dirCallback;

this.fileCallback = fileCallback;

}

public void Walk(string rootDir, object obj)

{

DoWalk(new DirectoryInfo(rootDir), 0, obj);

}

void DoWalk(DirectoryInfo dir, int level, object obj)

{

foreach (FileInfo f in dir.GetFiles())

{

if (fileCallback != null)

fileCallback(f, level, obj);

}

foreach (DirectoryInfo d in dir.GetDirectories())

{

if (dirCallback != null)

dirCallback(d, level, obj);

DoWalk(d, level + 1, obj);

}

ProcessDirCallback dirCallback;

ProcessFileCallback fileCallback;

}

Diffing two trees in TFS version control

RobertHorvick — Fri, 07 Apr 2006 22:33:00 GMT

I often need to answer some seemingly simple questions like:

What has changed between these two builds?

How do the contents of these two branches deviate?

What % of the files in a child branch have deltas from the parent branch?

To answer these I’ve traditionally had to map both paths into a workspace, get everything and use a diffing tool like Beyond Compare to diff the tree structures. The history command does not give me the level of detail I want (it gives changeset detail, not file level detail) and the recursive diff command is painfully slow.

Yesterday Buck demonstrated a project diffing tool useful for comparing a tree at two different points in time. This was really close to what I needed but did not operate on arbitrary (and possibly unrelated) locations.

My goal was to extend that sample into something that worked as I needed and also integrated into a scripted environment well. E.g. Every line is prefixed with the change type or a comment/summary indicator:

left-only: <item>

right-only: <item>

diff: “<item>” – “<item>”

comment: <free text>

summary: <free text>

All normal output is sent to stdout. All error output is sent to stderr. The existence of error output means that the contents of stdout can not be considered complete.

The following example diffs a project at a specific changeset with a branch of that project at the tip. It shows that 5 files have been changed but none added or deleted.

> treediff.exe "$/MyTeamProject/Project1;C30006" "$/MyTeamProject/Project1Branch;T"

comment: Downloading items for path $/MyTeamProject/Project1

comment: Downloading items for path $/MyTeamProject/Project1Branch

comment: Diffing item sets

diff: "$/MyTeamProject/Project1/Src/Core/Client.cs" - "$/MyTeamProject/Project1Branch/Src/Core/Client.cs"

diff: "$/MyTeamProject/Project1/Src/Core/Command.cs" - "$/MyTeamProject/Project1Branch/Src/Core/Command.cs"

diff: "$/MyTeamProject/Project1/Src/Results/Changes.cs" - "$/MyTeamProject/Project1Branch/Src/Results/Changes.cs"

diff: "$/MyTeamProject/Project1/Src/Results/Describe.cs" - "$/MyTeamProject/Project1Branch/Src/Results/Describe.cs"

diff: "$/MyTeamProject/Project1/Src/Results/Integrated.cs" - "$/MyTeamProject/Project1Branch/Src/Results/Integrated.cs"

summary: Compared 50 items

summary: Same: 45

summary: Diff: 5

summary: Right Only: 0

summary: Left Only: 0

Items left as an exercise to the reader is support for detecting encoding changes and case-only rename changes (also more specialized rename support could be added) … and the usual disclaimers apply – YMMV.

The code…

using System;

using System.Collections.Generic;

using Microsoft.TeamFoundation;

using Microsoft.TeamFoundation.Client;

using Microsoft.TeamFoundation.VersionControl.Client;

using Microsoft.TeamFoundation.VersionControl.Common;

namespace treediff

{

class Program

{

static void Main(string[] args)

{

if (args.Length != 2)

{

Console.Error.WriteLine("Usage: treediff <itemspec> <itemspec>");

Environment.Exit(1);

}

try

{

ProcessDiffs(args[0], args[1]);

}

catch (TeamFoundationServerException e)

{

Console.Error.WriteLine(e.Message);

Environment.Exit(1);

}

catch (TreeDiffException e)

{

Console.Error.WriteLine(e.Message);

Environment.Exit(1);

}

private static void ProcessDiffs(string lhsSpecStr, string rhsSpecStr)

{

VersionControlServer tfsClient = GetVersionControlServer();

ItemSpec lhsSpec, rhsSpec;

VersionSpec lhsVersion, rhsVersion;

string lhsRoot = LoadItemAndVersionSpec(lhsSpecStr, out lhsSpec, out lhsVersion);

string rhsRoot = LoadItemAndVersionSpec(rhsSpecStr, out rhsSpec, out rhsVersion);

Console.WriteLine("comment: Downloading items for path {0}", lhsRoot);

ItemSet lhsItems = tfsClient.GetItems(lhsSpec, lhsVersion, DeletedState.NonDeleted, ItemType.Any, false);

Console.WriteLine("comment: Downloading items for path {0}", rhsRoot);

ItemSet rhsItems = tfsClient.GetItems(rhsSpec, rhsVersion, DeletedState.NonDeleted, ItemType.Any, false);

Console.WriteLine("comment: Diffing item sets");

DiffItemSets(lhsItems, rhsItems, lhsRoot, rhsRoot);

}

private static VersionControlServer GetVersionControlServer()

{

// Figure out the workspace information based on the local cache.

WorkspaceInfo wsInfo = Workstation.Current.GetLocalWorkspaceInfo(Environment.CurrentDirectory);

if (wsInfo == null)

{

throw new TreeDiffException("The current directory is not mapped.");

}

// Now we can get to the workspace.

TeamFoundationServer tfs = new TeamFoundationServer(wsInfo.ServerUri.AbsoluteUri);

return (VersionControlServer)tfs.GetService(typeof(VersionControlServer));

}

private static string LoadItemAndVersionSpec(string spec, out ItemSpec itemSpec, out VersionSpec versionSpec)

{

string fileName;

VersionSpec[] versions;

VersionSpec.ParseVersionedFileSpec(spec, null, out fileName, out versions);

switch (versions.Length)

{

case 0:

versionSpec = VersionSpec.Latest;

break;

case 1:

versionSpec = versions[0];

break;

default:

throw new TreeDiffException(string.Format("Expected 0 or 1 version spec - found {0}", versions.Length));

}

if (!string.IsNullOrEmpty(fileName))

{

itemSpec = new ItemSpec(fileName, RecursionType.Full);

}

else

{

throw new TreeDiffException("Item name was null or empty");

}

return fileName;

}

private static void DiffItemSets(ItemSet lhs, ItemSet rhs, string lhsRoot, string rhsRoot)

{

int same = 0;

int lonly = 0;

int ronly = 0;

int diff = 0;

Dictionary<string, Item> lhsItemHash = CreateHash(lhs, lhsRoot);

Dictionary<string, Item> rhsItemHash = CreateHash(rhs, rhsRoot);

foreach (string lhsPath in lhsItemHash.Keys)

{

Item lhsItem = lhsItemHash[lhsPath];

Item rhsItem;

if (rhsItemHash.TryGetValue(lhsPath, out rhsItem))

{

// verify that the file types are the same and if it is a file that the contents are identical

// this could be extended to also verify the case of the item name has not changed and that the

// encoding has not changed.

if (lhsItem.ItemType == rhsItem.ItemType &&

((lhsItem.ItemType == ItemType.File && EqualFileContents(lhsItem, rhsItem)) ||

lhsItem.ItemType == ItemType.Folder)

)

{

same++;

}

else

{

Console.WriteLine("diff: \"{0}\" - \"{1}\"", lhsItem.ServerItem, rhsItem.ServerItem);

diff++;

}

// by removing the RHS items when we are done in this loop the rhsItem collection

// will contain the RHS orphans.

rhsItemHash.Remove(lhsPath);

}

else

{

Console.WriteLine("left-only: {0}", lhsItem.ServerItem);

lonly++;

}

foreach (Item rhsItem in rhsItemHash.Values)

{

Console.WriteLine("right-only: {0}", rhsItem.ServerItem);

ronly++;

}

Console.WriteLine("Compared {0} items", ronly + lonly + diff + same);

Console.WriteLine("Same: {0}", same);

Console.WriteLine("Diff: {0}", diff);

Console.WriteLine("Right Only: {0}", ronly);

Console.WriteLine("Left Only: {0}", lonly);

}

// taken from Buck's post http://blogs.msdn.com/buckh/archive/2006/04/06/project_diff.aspx

static bool EqualFileContents(Item item1, Item item2)

{

if (item1.ContentLength != item2.ContentLength)

{

return false;

}

// If the two hash values have different lengths or both have a length of zero,

// the files are not the same. The only time this would happen would be for

// files uploaded by clients that have FIPS enforcement enabled (rare).

// Those clients can't compute the MD5 hash, so it has a length of zero in that

// case. To do this right with FIPS, the code would need to compare file

// contents (call item.DownloadFile()).

// For information on FIPS enforcement and MD5, see the following link.

// http://blogs.msdn.com/shawnfa/archive/2005/05/16/417975.aspx

if (item1.HashValue.Length != item2.HashValue.Length ||

item1.HashValue.Length == 0)

{

return false;

}

for (int i = 0; i < item1.HashValue.Length; i++)

{

if (item1.HashValue[i] != item2.HashValue[i])

{

return false;

}

return true;

}

private static Dictionary<string, Item> CreateHash(ItemSet items, string root)

{

Dictionary<string, Item> itemHash =

new Dictionary<string, Item>(items.Items.Length, StringComparer.OrdinalIgnoreCase);

foreach (Item item in items.Items)

{

itemHash.Add(item.ServerItem.Substring(root.Length), item);

}

return itemHash;

}

internal class TreeDiffException : Exception

{

public TreeDiffException(string msg)

: base(msg)

{

}