Progressive Development

Zany Adventures in Software Engineering with Maven and Motley

Motley says: “Branches are for trees, not source code”

Motley says: “Branches are for trees, not source code”

  • Comments 12

Summary

     

    Motley:  Branches are too complicated. The last thing we need is a copy of the code that has to be maintained in two or more places!

     

    Maven: Branches are source code copies with a tie back to a mainline set of code that allow for easy integration between copy and original set (and vice versa). Branches bring a level of isolation from mainline as well as keep the quality of mainline (and the daily builds) high.

    ______________________________

     

     [Context: The company has been having lots of trouble with build breaks in mainline of late. General quality of the product mid-way through the development cycle is lower than expected]

     

    Motley: Somebody is going to get crazy glue stuck to their cup! The quality of our daily builds has been crap lately. Too many people checking in lousy code. Too may conflicts. Not enough time to get our mainline build stable and of high quality before the next check-in goes in.  Something has to be done.

     

    Maven: Crazy glue? Ouch. If some moron did that to me I would pay them back big time - when they least expect it. Anyway, I agree with you - having a daily build of poor quality slows down everyone. The key is to isolate all the feature teams from one another giving them freedom to check-in yet keep the main build high quality.

     

    Motley: Thanks for stating the obvious, Mave. Your haircut sucks. Now I stated the obvious too.

     

    Maven: I thought I fixed that crooked sideburn! Ugh. There is a good way to accomplish this isolation. If you have a good source control system (like we do) such as Perforce or CVS, you can leverage a branch.

     

    Motley: Branches are for trees. What do they have to do with source code? Should I start working outside in the shade?

     

    Maven: Think of a branch as a more-involved copy of a collection of source code.

     

    Motley: Why would I want to make a copy? Copies are fraught with problems, such as keeping multiple copies synchronized. And to think, I was expecting a good idea from you. Although pigs don't fly quite yet.

     

    Maven: The digs keep coming! A branch is more than a copy. When you branch a source tree, you still maintain the tie to the original source code. Think of it like the trunk that a tree branch is connected to. You can bring forward changes in mainline to your branch, and bring back changes in your branch to mainline relatively easily. You can even work on the same source files in both places and merge the changes together in either direction.

     

    Motley: And how would I do that? Keep them tied together, I mean.

     

    Maven: You bring code forward to your branch from mainline and back to mainline from your branch through an integration, or more specifically, a forward integration (FI) and reverse integration (RI) respectively. Here is an example of a tree structure I used in a previous role.

     

    clip_image001

     

    Mainline: The main source code depot where daily builds take place from

    Feature Branches: As many of these as necessary to support isolated feature teams

    Feature Sub-branches: Use only as necessary (e.g. a developer making major changes as part of a feature team)

     

    Motley: That looks more complicated your morning routine! Do the benefits of a structure like that really outweigh the costs of doing all those integrations?

     

    Maven: As the consultant always says, "it depends". For a small overall team where not too many people are checking in to mainline, perhaps the structure is too much overhead. However, for larger teams, branches bring a level of isolation allowing features teams the freedom to do what they need to get the job done. In addition, a central build team could maintain mainline and have a very high quality bar for check-ins to mainline. Teams are only allowed to reverse integrate into mainline when they meet that bar.

     

    Motley: And what does that bar typically involve?

     

    Maven: It depends on the team. For us, we may have a quality gate in place that mandates the following when an RI is done:

    • All changes must have been code reviewed
    • All changes must have been gone through a test pass
    • Test automation is in place for new features (if applicable)
    • New features have been stress and performance tested
    • No memory leaks have been detected by our tools
    • FxCop has been run and is violation-free on all changed code
    • Unit tests have been executed and pass at 100%

     

    You can basically go as thorough as you like. As you move up the branch hierarchy, the criteria to check-in gets lighter and lighter so as not to slow down the team. RI operations get stricter and stricter as you get closer to mainline.

     

    Motley: Should every team have its own branch?

     

    Maven: Teams making changes to an isolated part of the code base with relatively few check-ins may not need a branch. Teams working on radical and risky changes or in common parts of the code base would likely benefit from a branch. Keep in mind, though, that the more levels of branching you have, the more overhead there is for integration operations. I recommend trying to stick to 1-2 levels of branching at most.

     

    Motley: What about builds? We lose the advantage of having a central build team generate a daily build?

     

    Maven: Yes, that can be true. Hopefully you can take their infrastructure on a couple of machines that your feature team owns and duplicate the build there so that you have your own daily build and verification going for your feature. For small features, this may not be necessary. For large features, it is a huge benefit as you nail down problems quicker. If a build break happens due to a bad check-in, you catch it long before you hit mainline. As a result, you block fewer people. Everyone wins!

     

    Motley: Ok, Einstein. I have a you now. What if feature team A is working on some changes that feature team B needs to make further progress? You cannot go across branches. Hah!

     

    Maven: You have several options:

    • Feature team A can pack up the changes and pass them to Feature team B. Most source control systems support packing up a change list for distribution.
    • You can do what's called a baseless merge. I don't recommend doing this as it loses your source history. Essentially you do an integration across branches but lose the tie to mainline. Yuck.
    • You can RI the changes of note to mainline with a test pass and then FI them up into Feature team B's branch. This doesn't work as well if the changes are in progress and not fully baked.

     

    Motley: I can think of another problem - my work in my branch can get out-of-date with what is going on with other teams. That may cause problems later.

     

    Maven: True. The onus is on the feature team to ensure it FIs on a regular basis to keep up. Additionally, teams should not "go dark" from mainline for too long. Regular RIs to mainline should be happening, say, at the end of each sprint. For feature teams practicing Scrum, the code at the end of each sprint should be of shippable quality and can be RId back to mainline.

     

    Motley: Also, merging in conflicting changes could be a real pain!

     

    Maven: Yes, but not if you have the right tools. A good merge tool automatically merges isolated changes in the same file (the majority case), and allows you to easily resolve conflicting changes.

     

    Motley: You seem to have an answer for everything. Let me see if I understand this branching thing:

    • Branching helps keep mainline quality high
    • A feature teams considers a branch to isolate its changes
    • Branching provides more freedom for check-ins
    • Quality gates are more strict the closer your branch is to mainline
    • Branching ensures you don't lose ties to the original source code via integrations
    • A good tool makes integrations and merges relatively painless

     

    Maven: Nice summary. Why don't you go approach our beloved development manager and propose that our team undertake a branching model?

     

    Motley: I think I'll do just that.

    ______________________________

     

    James' Pointer:  I am always amazed at how little discussion there is around source code branching. Branches are an incredibly useful software tool employed all over Microsoft for the reasons discussed above. Branches are particularly useful when you are an agile team in a large waterfall-based organization. They give you that level of isolation needed to follow your own effective processes yet still merge your changes in with mainline on a regular basis. Our team would be much less efficient without branches.

     

    Maven's Resources: 

  • PingBack from http://www.clickandsolve.com/?p=17185

  • I must disagree with you (first time for me). For me branches represent pure waste. I try to avoid them as much as I can. I think that raising up standards is a better strategy to avoid main line build issue. (every check in must pass the "quality gate")

    I've posted about this some time ago (http://imistaken.blogspot.com/2008/10/source-branches.html)

  • Actually, Lior, I don't think we entirely disagree with each other. Branches are a great solution when used pragmatically. For a small team with a good quality gate definition, I likely wouldn't branch at all. However, for large team development, they can be a good solution.

    I work in an organization that delivers software using the waterfall model (whether I like it or not and whether they admit it or not). Our team, however, follows the agile model and has to fit within the waterfall milestones. To isolate ourselves from the really hefty requirements of checking into mainline, we branch and abide by our own (sometimes stricter) check-in requirements.

    Our requirements may be stricter but not as heavy. Let me explain. The overall organization leverages a continuous integration tool that could add hours of transportation time to get into mainline. We isolate ourselves from this overhead and only do it once when we reverse integrate.

    Additionally, I want management to hold us accountable for results as opposed to process. Branching allows us to put our own processes in place and abide by their process only periodically at RI time. Every team is different, so I can see why they put these hard mainline check-in requirements in place.

    For individual feature work, we want to maintain quality but not necessarily do fully automated test passes. The test team modifies their test suite during feature development, and the branch allows them to isolate their changes from the rest of the team until ready.

    I agree with you that branches CAN represent waste (in terms of effort) when used incorrectly. If you have good reasons to branch, it is a useful tool. However, don't apply this hammer generically to solve all your problems.

    Thanks for commenting, and for a thought-provoking blog posting!

  • Hi James,

    Interesting post. It reminds me of a post I read at Agile Development Thoughts that takes this to the next level.

    http://damonpoole.blogspot.com/2008/01/advanced-multi-stage-continous.html

    I have linked to the vendor's web site above. Some very interestint concepts to consider when doing continuous integration or frequent builds.

    Mark

  • The key is the tools that are being used for branching.  The better the tool the less resistance it is to branch and integrate.  If a team uses a source control system that uses a poor merge tool and can't be configured to use Araxis or BeyondCompare then branching isn't going to be embraced by developers and testers.  A better use of time is to find a better source control system.

  • The key is the tools that are being used for merging the branches.  The better the tool the less resistance it is to branch and integrate.  If a team uses a source control system that uses a poor merge tool and can't be configured to use Araxis or BeyondCompare then branching isn't going to be embraced by developers and testers.  A better use of time is to find a better source control system.

  • Hey Marvin,

    Agree! A good merge tool makes a lot of difference. I use Beyond Compare myself because we have a license, but have used Araxis Merge in the past and it's great as well.

    Ideally you set up an automated tool that can do most of the automated resolve for you on a daily (or whatever) basis. Then a dev has to step in only to resolve conflicting changes.

    As other posters pointed out (including Mark), continuous integration is a key practice for branching (or otherwise). If you have a good CI system and a small team, branches may not even be necessary. In the event they are used, minimize the levels. One of the article links above went overboard in depth IMHO.

  • MergeMagician is focused on automated merging.

    http://www.mergemagician.com

    <a href="http://www.mergemagician.com">MergeMagician</a>

  • Looks like an interesting tool, Kevin! Thanks for the reference. I'll have to thank them for the blog link too!

    Any idea what source control systems it supports? It wasn't obvious on my quick look at the page and I only say "existing SCM systems".

  • The first version will support Subversion and Team Foundation Server.  The next ones after that will probably be Perforce and ClearCase.  There is a plug-in architecture planned so that any SCM tool will be able to be supported.

  • I'm on my third month on a project with my current client.  I have three branches already (replacing ASP's TreeView control with the CSS Friendly Control Adaptor, switching from class.name to class.id for key in a name-value collection, and a large refactoring in behaviour to a central class).  

    I came to this article while trying to decide if a fourth branch was a good idea.  My environment is poorly organised, so I never know when my lead will walk over and request a build.  Anything that's going to take more than a few hours makes me want to branch, so I can give him what he wants without undue delay.  Having said that, my trunk is already 200Mb (I store all libraries, tools etcetera) so branches aren't free.

    I am the only agile practitioner in my team (I'm on contract), and I'm a neophyte at best.  I use CC.NET to call NAnt, and because my .build file is in my trunk, it's replicated in each branch -- I can do builds of each branch, mitigating an earlier concern about losing the benefits of CI when branching.  

    I use VisualSVN as the server, and TortoiseSVN as my client.  I've used the TortoiseMerge and WinMerge for my tools, and I've settled on WinMerge though I'm not sure I use it to extreme, but I'll confess I'm at a loss to understand the comments about SVN making merges a pain.  To date every conflict I have seen made sense to me and was easily resolved, but so far I have been merging trunk-to-branch (bug-fixes) pretty frequently, usually after each bug-fix.

    Anyway, it has been a while since you posted.  Any updates?  You have a new subscriber to your blog...  :)

  • Hi Duncan,

    Yes, the blog has been a little quiet lately due to another project that is, actually, related to the blog. I may have more to announce soon. With that project and my day job, doesn't leave much time for anything else. I hope to get back to blog posting shortly, and thanks for the query.

    On branching, I always recommend keeping branches to a minimum. The overhead of forward and reverse integrates can be large, but it looks like you are doing them frequently in small chunks, which helps. Depends how big your team is.

    For merging, I have used Beyond Compare in the past (and still do) and I have heard Araxis Merge is very good. I am not familiar with the tools you mention.

    Keep us posted on how things go! I always check and try to respond to comments.

    James.

Page 1 of 1 (12 items)
Leave a Comment
  • Please add 1 and 1 and type the answer here:
  • Post