Edward Thomson's Blog

Locking and git-tf and... danger?

2012-08-28T14:56:00Z

If you haven't yet seen or heard about the new git-tf tool from Microsoft, this blog post probably won't make any sense. So... go check it out over at gittf.codeplex.com. It's okay, I'll wait.

Okay, got it? Yeah, I know, it's cool, right? Well, it is, but like any software project, there are a few problems. One of them is in the whole git-tf checkin process and how it needs to take a lock on your tree before it can continue. I mean, locks are annoying, right? If git-tf requires a lock to do a checkin, then that's going to fail if somebody else has already taken a lock somewhere in that folder. And, of course, the default in TFS is to take a lock on binaries, so if you've got some of those in your tree then git-tf is going to complain all the time when you try to checkin.

It's even worse when you run into this the first time and decide that we can't have been so obnoxious as force you to take a lock. Surely those developers put in some no lock option, maybe even called --no-lock, that will avoid locking. And we did! And with our appropriately scary warning in the checkin help:

--no-lock Does not take a lock on the server path before committing (dangerous)

...you can see why this is a problem. I mean, it's dangerous?! Who would want to use that!?

Okay, let's back up and explain why we decide to take this lock.

Team Foundation Server's version control model has good concurrency. Let's assume that you and I are both working on the latest version and we both edit some files. If we both hit the checkin button at the same time, this is a classic race condition, but one that's harmless. Our checkins get serialized and - assuming we haven't edited any of the same files to produce a conflict - it really doesn't matter all that much which one of us wins this race and which one loses. Both of our changesets got checked in and the next time we do a get latest, we get the other person's changes, too. Things get a little bit more complicated if we edit the same file, but not much. The winner of this race gets their changes checked in - the loser has to resolve a conflict and then checkin again. This is pretty classic version control behavior and a well-understood problem.

git, of course, also has good concurrency. Let's assume that you and I are both on HEAD and we both edit some files. If we both were to commit to our local git repository, now we both have new commits that share a common parent. At some point, we'll need to merge this branch in the graph, but this is no big deal. Again, a race condition that doesn't matter.

git-tf, on the other hand, can't cope well with this race condition. (It might be a bit much to have called it dangerous, but that little word scared you enough to read this blog post, so I think we'll stick with that term for a little while longer.) Here's how git-tf checkin works, in a nutshell:

Finds the latest TFS changeset that you've fetched and merged into your graph. This is the "high water mark".
Takes a lock on the TFS path you're bridging. Unless, of course, you're living dangerously and using that crazy --no-lock flag.
Ensures that this high water mark is actually the latest TFS changeset. If not, you will be told to fetch that changeset and merge.
Pends the changes to TFS that are representative of the changes in your git commit and checks them in.
Updates the aforementioned "high water mark" to the changeset that I just checked in.

Let's take another look at this race condition once git-tf gets in the picture. Imagine that I'm using git-tf and you're using some other TFS client like Visual Studio or the TFS plug-in for Eclipse. The latest version is changeset 3, and I've made a git commit on top of that. My git repository looks like this:

I'm ready to git-tf checkin these changes, and I'll do so without a lock. Meanwhile, you've also made some changes against TFS and you're going to check in at the same time. If you happen to sneak your changeset in between steps 3 and 4 (above), then I've updated my high-water mark to my changeset... but I never got your changeset.

In a perfect world, I would pull your changeset down as a git commit such that it has a parent of 3, and my git commit 4fac65... has your changeset as a parent. This keeps the linear model of TFS changesets that git-tf strives so hard for. Except that if I do that, my commit can't be 4fac65... anymore, it gets a completely new ID because those are computed based on their parents. Ouch.

The way it is, I'll never even see your changes until there's a new changeset and I can fetch that, since git-tf's high water mark already includes the latest changeset on the server. So it turns out that not locking can be dangerous after all.

Of course, most people don't have that much churn in their code tree that this is an issue. So you could just check the history in TFS to make sure that nobody else snuck with a new changeset in those few seconds. And if they did, of course, you could just stick a new dummy changeset in TFS so that you could pull the latest version that has the other changes, too. But this isn't exactly an ideal workflow.

So the question of course becomes: can we avoid taking a lock, but stay out of danger? We think so. We spun up a conversation over on our codeplex site about this - so if you're interested in this topic, we'd love your feedback.

Team Foundation Server and Xcode projects with git-tf

2012-08-28T14:15:00Z

Brian Harry announced the availability of git-tf last Monday, a new tool from Microsoft that allows you to create a local git repository from a path in Team Foundation Server, then check the changes in your repository back in to TFS. This tool is another piece in our cross-platform support, aimed at users of IDEs that don't have built-in TFS integration.

We hope that Xcode users will find this particularly useful, as Xcode sadly lacks a plug-in model that would allow us to build TFS support natively. Xcode does have git support, however, so once you've created your git repository with git-tf, you can use Xcode's source code integration directly.

This means that if you're writing mobile applications, you can now use Team Foundation Server as your ALM solution for the entire development team. Imagine you have your shared business logic in C, you have a Windows Phone 8 version being developed in Visual Studio, an Android version being developed in Eclipse and an iOS version being developed in Xcode. Now all your developers can take advantage of TFS Version Control - your Visual Studio users with Team Explorer, your Eclipse users with the TFS Plug-in for Eclipse and your Xcode users with git-tf.

I'm too hopelessly old-school to be hip to the new mobile apps, so I'm going to show how I work on a cross-platform command-line application that plays blackjack. I've got my source in TFS, with an Xcode project side-by-side with my Visual Studio solution:

Once I have Xcode and git-tf installed on my Mac, I can download this project easily:

ethomson@mbp:~/Demo% git tf clone --deep http://tfs2012:8080/tfs/DefaultCollection $/DefaultAgile/blackjack
Username: ethomson
Password: 
Connecting to TFS...
Cloning $/DefaultAgile/blackjack into /Users/ethomson/Demo/blackjack: 100%, done.                   
Cloned 14 changesets. Cloned last changeset 19 as fb65c36
ethomson@mbp:~/Demo%

Now I have a git repository that represents my TFS changesets, with full history. (That was the --deep option, without that I would have a single git commit that represents my latest TFS changeset. This is quicker to compute and useful if you don't need full history.) You can see that my history in Visual Studio:

is the same as the history in my local git repository:

ethomson@mbp:~/Demo/blackjack% git log
commit f33b75c0190d76c417fd5d724236a9c3f731b185
Author: tfsuser <tfs2012\tfsuser>
Date:   Tue Aug 14 15:20:20 2012 -0500

    This is a commit from Xcode.

commit a88b3887f1998fac74beb8ee32b4d91e06d8e625
Author: ethomson <tfs2012\ethomson>
Date:   Tue Aug 14 15:16:16 2012 -0500

    This is a change from Visual Studio.

...etc...

When I open up this project in Xcode, you'll notice that when I start making changes to files, they're automatically added to my git index, as denoted by the small M next to the filename in Project Navigator. Below, I've made some changes to my project's assertion logic, changing Assert.cpp and Assert.h.

When Xcode's source control integration, I can commit these changes to my local git repository from right within my IDE. When committing, it will show a handy diff viewer for each file in my commit.

You can see the two commits I've made in my Xcode project in my git log.

ethomson@mbp:~/Demo/blackjack% git log
commit 6f9bbafa5482c77de5b97b2548320b78de6b82fe
Author: Edward Thomson <ethomson@edwardthomson.com>
Date:   Thu Aug 16 15:58:43 2012 -0500

    Fixed some typos in the comments.

commit 8dd92848b4ea0f6549d6f88d28e47c3e1201cd7c
Author: Edward Thomson <ethomson@edwardthomson.com>
Date:   Thu Aug 16 15:55:01 2012 -0500

    A couple of minor changes in Xcode

...etc...

Once I'm done making changes locally and I want to integrate with the rest of the team, I can check them in to my TFS server using git-tf checkin. Just like when I cloned my project, I can use the --deep option to preserve history.

ethomson@mbp:~/Demo/blackjack% git tf checkin --deep
Username: ethomson
Password: 
Connecting to TFS...
Checking in to $/DefaultAgile/blackjack: 100%, done.                            
Checked in 2 changesets, HEAD is changeset 21

Back in Visual Studio, you can see both of these changesets in my TFS history.

And, of course, since I'm doing Continuous Integration builds for this project, as soon as I checkin from git-tf, TFS queues up a build for me. In this case, I get two - one for each changeset, since I did a deep checkin from my two git commits.

So you can see that now I can easily use Xcode to take advantage of the ALM functionality of Team Foundation Server, providing every developer on a team with intuitive access to TFS Version Control.

git-tf is available now on CodePlex, Microsoft's Project Hosting for Open Source Software and we're making git-tf available open source under the MIT license. We think this is a very useful tool right now, and we're looking forward to making it even better.

TFS Austin User Group

2011-04-05T04:37:00Z

A very special thanks to the TFS Austin User Group for providing me time today to present on TFS for Heterogeneous Environments. My slide deck from the presentation is available here:

(Please visit the site to view this file)