Progressive Development : build

Motley says: “Slow down to go fast? Do I look like a tortoise?”

James Waletzky — Wed, 22 Apr 2009 08:23:00 GMT

Summary

	Motley: There is no harm shortcutting a few steps of development to meet a deadline. Some up-front check-in steps just take too much time.
	Maven: Slow down to go fast. Obey check-in checklists or you will pay for it later in the development cycle. Pre-check-in tasks such as code reviews, unit testing, static analysis, and buddy builds of various flavors must not be avoided.

______________________________

[Context: The code check-in deadline is approaching. Motley has code that is in progress and he is hurrying to get his change in prior to the build snap]

Maven: Hey, Mot. How's it-

Motley: Quiet! I have to get this code checked-in by the end of the day to meet the deadline. I don't have time to chat and need to rush to get this code in.

Maven: Whoa - maybe we should chat. Please just give me a few minutes here - it won't affect anything in the bigger scheme of things. What do you mean you have to "rush"?

Motley: The daily build starts at 6pm and it's 5:35pm. I only have 25 minutes to get the code checked-in so I have to take a few shortcuts.

Maven: I cannot stand the word "shortcut". Care to clarify?

Motley: It doesn't really matter what you can and cannot stand. But since you asked, I am going to skip writing unit tests for this last class, I am going to forgo code review because my code is always perfect anyway, and I am going to skip running FxCop just this once.

Maven: Have you ever heard the phrase, "Pay me now, or pay me more later"? What you are doing is akin to racking up credit card debt. On the outside it looks like a $5000 purchase, but in the long run it is going to cost you a lot more in interest if you do not pay off the debt as soon as the bill comes due.

Motley: Oh, no. More Maven analogies. I can do without those. And by the way, Boyz II Men called - they want their shirt back.

Maven: Always insulting my clothes! This blue and pink sweater combo isn't that bad, is it? Anyway, taking shortcuts now when the code is cheap to fix pays off in the long run and leads to quicker ship cycles and shorter bug-fix cycles later in the development cycle.

Motley: How so? Bugs are bugs, I need to fix them anyway.

Maven: It's much cheaper to fix bugs now than later. Fixing them later usually involves someone else creating and maintaining bugs in a bug database, fixing them when the code is no longer in context and fresh in your memory, and the tendency to bolt-on fixes and shortcut fixes. Additionally, if the test team does not find an issue, the cost of fixing it post-release is large due to the overhead in working with the customer, the bug getting back to the product group, deploying the fix, the negative press, and other factors. If you find an issue that modifies the design later, you may have to change a lot of code possibly destabilizing the product.

Motley: Yeah, yeah - the cost of change curve. I know it.

Maven: Then you know that you shouldn't rush your development. Would you rather fix bugs later when they are more expensive or now when they are cheap without the overhead of a post-check-in bug? Leaving bugs until later forces long "stabilization" periods on the team, which, in my opinion, are evil and can be avoided by a diligent team.

It is well worth pushing back on the deadline to ensure that you can adhere to the following code check-in checklist:

Unit tests. We have talked about unit testing at length in other discussions. This is the number one quality-driven developer technique. You must have a set of unit tests in place that exercise all of your new code and you can check-in when all the tests pass. This items should be bolded on a checklist.
Code coverage. Hand-in-hand with unit testing is code coverage. Aim for 70%+ code coverage for all of your code. More important than the coverage number is the feedback you receive from the coverage tool regarding source lines that were missed in testing so that you can add new tests.
Debug and retail builds. A debug build is generally different than a retail build, particularly when you make use of debug-time primitives such as assertions. You need to build both flavors to ensure the build succeeds either way. Broken builds are not tolerable as they block the entire team. In fact, do a clean build (vs. incremental build) to be safe.
Static analysis. A static analysis tool like FxCop finds bugs for you. Per the cost of change discussion, why not fix them now instead of later when it is more expensive? Why risk security defects escaping your defenses and being released into the wild where hackers and customers will find them?
Reliability tools. Running a tool like AppVerifier is more relevant to C++ applications. Running this tool is a must prior to check-in as it will help identify heap corruptions, improper use of handles, and misuse of critical sections. Additionally, for native code it will point out memory leaks.
Warning level. Crank up the compiler to generate warnings at level 4 (or whatever is highest for your compiler) to ensure that you have addressed all possible compiler-identified issues. Get into the habit of perpetually building at the highest warning level. Get clean and stay clean.
Other tools. Run any other organization-wide tools such as a copyright header checker, XML documentation checker, or coding style checker (e.g. Source Analysis) prior to check-in. You don't want to be the grunt fixing a ton of these issues later. Trust me on this one.
Code reviews. All code should be at least peer-reviewed, with larger changes (e.g. new features) being put through a code inspection. Having at least one other pair of eyes look at your code is likely to find many issues before even the test team gets their hands on the code.
Change list description. Include complete descriptions for check-in change lists, including how you tested the code, why the change was made, how you built, possible regressions, and other risks.
Buddy build. To ensure you did not forget to add a file to your change list and to protect against other bad assumptions, package up your change and pass it to a buddy for her to build. Most source control systems allow you to package up a change list for distribution. If you have a continuous integration system that builds changes before check-in, this is less necessary.

Motley: You cannot possibly be serious! If I do all that now there is no way I would ever make the deadline, let alone make a check-in.

Maven: All of the above is necessary, particularly in a large, quality-focused organization. We want to exterminate bugs before they are even checked-in. The checklist above helps with that.

Motley: Can you say "Sloooooooooooooowwwwww"? I knew you could.

Maven: Get in touch with your inner tortoise. I heard that in a TED talk on Slowing Down in a World Built for Speed by Carl Honore. Going a little slower up front will save time in the overall development cycle, and I would claim that you ship sooner as a result. Of course, that was not the specific theme of the talk, but the lesson can be applied here. Also, be pragmatic. Perhaps you avoid a step or two for small bug fixes.

Motley: One thing I do not quite buy is the lack of long stabilization periods at the end of the development cycle. We still need to put all the pieces together!

Maven: Ultimately we want to avoid long stabilization cycles and instead be stable when you check-in. A tail of the development cycle needs to be dedicated to integration and overall acceptance test validation, but is not the same thing as "stabilization". We talked about this before in our discussion on developer testing.

Motley: So you are telling me that I need to pay my boss a visit as soon as possible to tell him that I am going to miss the deadline? He is going to kill me.

Maven: Well, seeing as you only have 5 minutes to the build snap-

Motley: <POW>. You deserved that shot in the nose for wasting all my time and missing the build snap!

Maven: Ouch! Okay, I deserved that. But seriously, your boss is a reasonable person and will understand. Firstly, you are communicating bad news as soon as possible (well, not really, but let's pretend), which all managers appreciate. Secondly, present it such that you are focused on quality and want to avoid taking shortcuts, and that you feel this course of action is best for the product. I will bet you lunch that he understands.

Motley: There are some days where I wish I stayed in bed...

______________________________

James' Pointer: The checklist items I mention above are from a real team in at Microsoft in Windows Mobile. Quality is our #1 value as a team. We want solid check-ins that adhere to the checklist to ensure we don't have a long tail of bugs later. Yes, it is a lot of work but ultimately saves time in the long run. Slow down. Don't rush. Nail the checklist on every check-in. You will thank yourself later. Okay, maybe not, but you should.

James' Double Pointer Indirection: A good developer knows how much work goes into each and every check-in and factors these things into their estimates. These things take time.

James' Triple Pointer Indirection: "Slow down to go fast" is real advice in car racing circles. Intuitively it does not make a lot of sense. However, if I go into the corner too fast I have to brake to avoid the wall and then take a lot of time and gas to come back up to speed exiting the corner. If I slow down on entry, roll through the corner and smoothly apply the accelerator on exit, I actually go faster even though I slowed down entering the corner.

Maven's Resources:

TED (http://www.ted.com): A great web site of some of the world's best speakers. There are talks on a multitude of topics. You are guaranteed to find something of interest.
AppVerifier (http://msdn.microsoft.com/en-us/library/ms220948.aspx): A mandatory tool for C++ developers on the Microsoft Windows platform.
Source Analysis (http://code.msdn.microsoft.com/sourceanalysis): A customizable tool that validates coding standards and is a good complement to FxCop.

Motley says: “Branches are for trees, not source code”

James Waletzky — Tue, 03 Mar 2009 19:08:00 GMT

Summary

	Motley: Branches are too complicated. The last thing we need is a copy of the code that has to be maintained in two or more places!
	Maven: Branches are source code copies with a tie back to a mainline set of code that allow for easy integration between copy and original set (and vice versa). Branches bring a level of isolation from mainline as well as keep the quality of mainline (and the daily builds) high.

______________________________

[Context: The company has been having lots of trouble with build breaks in mainline of late. General quality of the product mid-way through the development cycle is lower than expected]

Motley: Somebody is going to get crazy glue stuck to their cup! The quality of our daily builds has been crap lately. Too many people checking in lousy code. Too may conflicts. Not enough time to get our mainline build stable and of high quality before the next check-in goes in. Something has to be done.

Maven: Crazy glue? Ouch. If some moron did that to me I would pay them back big time - when they least expect it. Anyway, I agree with you - having a daily build of poor quality slows down everyone. The key is to isolate all the feature teams from one another giving them freedom to check-in yet keep the main build high quality.

Motley: Thanks for stating the obvious, Mave. Your haircut sucks. Now I stated the obvious too.

Maven: I thought I fixed that crooked sideburn! Ugh. There is a good way to accomplish this isolation. If you have a good source control system (like we do) such as Perforce or CVS, you can leverage a branch.

Motley: Branches are for trees. What do they have to do with source code? Should I start working outside in the shade?

Maven: Think of a branch as a more-involved copy of a collection of source code.

Motley: Why would I want to make a copy? Copies are fraught with problems, such as keeping multiple copies synchronized. And to think, I was expecting a good idea from you. Although pigs don't fly quite yet.

Maven: The digs keep coming! A branch is more than a copy. When you branch a source tree, you still maintain the tie to the original source code. Think of it like the trunk that a tree branch is connected to. You can bring forward changes in mainline to your branch, and bring back changes in your branch to mainline relatively easily. You can even work on the same source files in both places and merge the changes together in either direction.

Motley: And how would I do that? Keep them tied together, I mean.

Maven: You bring code forward to your branch from mainline and back to mainline from your branch through an integration, or more specifically, a forward integration (FI) and reverse integration (RI) respectively. Here is an example of a tree structure I used in a previous role.

Mainline: The main source code depot where daily builds take place from

Feature Branches: As many of these as necessary to support isolated feature teams

Feature Sub-branches: Use only as necessary (e.g. a developer making major changes as part of a feature team)

Motley: That looks more complicated your morning routine! Do the benefits of a structure like that really outweigh the costs of doing all those integrations?

Maven: As the consultant always says, "it depends". For a small overall team where not too many people are checking in to mainline, perhaps the structure is too much overhead. However, for larger teams, branches bring a level of isolation allowing features teams the freedom to do what they need to get the job done. In addition, a central build team could maintain mainline and have a very high quality bar for check-ins to mainline. Teams are only allowed to reverse integrate into mainline when they meet that bar.

Motley: And what does that bar typically involve?

Maven: It depends on the team. For us, we may have a quality gate in place that mandates the following when an RI is done:

All changes must have been code reviewed
All changes must have been gone through a test pass
Test automation is in place for new features (if applicable)
New features have been stress and performance tested
No memory leaks have been detected by our tools
FxCop has been run and is violation-free on all changed code
Unit tests have been executed and pass at 100%

You can basically go as thorough as you like. As you move up the branch hierarchy, the criteria to check-in gets lighter and lighter so as not to slow down the team. RI operations get stricter and stricter as you get closer to mainline.

Motley: Should every team have its own branch?

Maven: Teams making changes to an isolated part of the code base with relatively few check-ins may not need a branch. Teams working on radical and risky changes or in common parts of the code base would likely benefit from a branch. Keep in mind, though, that the more levels of branching you have, the more overhead there is for integration operations. I recommend trying to stick to 1-2 levels of branching at most.

Motley: What about builds? We lose the advantage of having a central build team generate a daily build?

Maven: Yes, that can be true. Hopefully you can take their infrastructure on a couple of machines that your feature team owns and duplicate the build there so that you have your own daily build and verification going for your feature. For small features, this may not be necessary. For large features, it is a huge benefit as you nail down problems quicker. If a build break happens due to a bad check-in, you catch it long before you hit mainline. As a result, you block fewer people. Everyone wins!

Motley: Ok, Einstein. I have a you now. What if feature team A is working on some changes that feature team B needs to make further progress? You cannot go across branches. Hah!

Maven: You have several options:

Feature team A can pack up the changes and pass them to Feature team B. Most source control systems support packing up a change list for distribution.
You can do what's called a baseless merge. I don't recommend doing this as it loses your source history. Essentially you do an integration across branches but lose the tie to mainline. Yuck.
You can RI the changes of note to mainline with a test pass and then FI them up into Feature team B's branch. This doesn't work as well if the changes are in progress and not fully baked.

Motley: I can think of another problem - my work in my branch can get out-of-date with what is going on with other teams. That may cause problems later.

Maven: True. The onus is on the feature team to ensure it FIs on a regular basis to keep up. Additionally, teams should not "go dark" from mainline for too long. Regular RIs to mainline should be happening, say, at the end of each sprint. For feature teams practicing Scrum, the code at the end of each sprint should be of shippable quality and can be RId back to mainline.

Motley: Also, merging in conflicting changes could be a real pain!

Maven: Yes, but not if you have the right tools. A good merge tool automatically merges isolated changes in the same file (the majority case), and allows you to easily resolve conflicting changes.

Motley: You seem to have an answer for everything. Let me see if I understand this branching thing:

Branching helps keep mainline quality high
A feature teams considers a branch to isolate its changes
Branching provides more freedom for check-ins
Quality gates are more strict the closer your branch is to mainline
Branching ensures you don't lose ties to the original source code via integrations
A good tool makes integrations and merges relatively painless

Maven: Nice summary. Why don't you go approach our beloved development manager and propose that our team undertake a branching model?

Motley: I think I'll do just that.

______________________________

James' Pointer: I am always amazed at how little discussion there is around source code branching. Branches are an incredibly useful software tool employed all over Microsoft for the reasons discussed above. Branches are particularly useful when you are an agile team in a large waterfall-based organization. They give you that level of isolation needed to follow your own effective processes yet still merge your changes in with mainline on a regular basis. Our team would be much less efficient without branches.

Maven's Resources:

Wikipedia definition of branching: http://en.wikipedia.org/wiki/Branching_(software)
(Tool) Beyond Compare: http://www.scootersoftware.com/moreinfo.php
(Tool) Araxis Merge: http://www.araxis.com/merge/
(Paper) Streamed Lines: Branching Patterns for Parallel Software Development, by Appleton et al, 1998, http://www.hillside.net/plop/plop98/final_submissions/P37.pdf

This paper has many good branching tips embedded within it.

Motley says: "The best way to control build breaks is to slap around the responsible developer"

James Waletzky — Tue, 11 Sep 2007 19:03:00 GMT

Summary

Motley: The best way to control build breaks is to slap around the responsible developer.

Maven: Continuous integration is an agile technique that minimizes daily build failures and minimizes integration problems in real-time.

______________________________

[Context: Motley is expressing some frustration that his team keeps breaking the daily build]

Motley: Damn! Yet another broken build last night. What is going on with this team? Lately we've had some really bad check-ins to source control causing issues with our daily build. The daily build is our lifeline here - if a build is broken that means a good portion of the team is blocked. ARRRRGGGHHH!

Maven: Calm down, bud. Why are the builds breaking?

Motley: Combination of things. Sometimes people are sloppy and forget to add a file to their source control change list. The build works locally on their machine, but once another machine syncs and builds, all hell breaks loose. Other times, it is obvious the developer has not run the unit test suite prior to check-in and there is a functionality break. A further problem is conflicting check-ins between developers causing havoc when the central build runs. In addition, there are times when the developer obviously has not run static analysis tools, which could have found a few problems even before check-in.

Maven: All common problems in many software companies. And all should be preventable.

Motley: I agree! It's just sloppiness and I am getting tired of it. You trust devs to do the right thing and it doesn't happen. Time for the hammer to come down.

Maven: Don't jump to conclusions just yet. Devs get put under lots of pressure to meet deadlines and periodically things get overlooked. What we need to do is put a system in place that helps them limit their mistakes without much in the proactive action relying on memory.

Motley: Me slapping them upside the head ought to be enough for them to remember!

Maven: The only time violence solves any problems is, well, I can't think of a scenario. Anyway, have you heard of "continuous integration"?

Motley: I'm not quite ready to live with my girlfriend, thank you very much.

Maven: Always the comedian! One recommended practice in agile development circles, most notably Extreme Programming, is continuous integration. The idea is that developers should be integrating their code together at regular intervals to head off build breaks before the daily build takes place. Leaving integration until the daily build happens or worse, until the end of a coding cycle, is a proven recipe for disaster.

Motley: I agree, in principle. But, how do you do it? Do I have to walk around and make sure syncs and builds happen on a regular basis?

Maven: Definitely not - tools to the rescue! One popular application in the .NET community is a free tool called CruiseControl.NET. You set it up on a server somewhere and have it monitor your source control system for check-ins. It works with many different types of source control systems, including Perforce, CVS, SourceSafe and others. Once a check-in happens, the server takes over and performs whatever actions you want it to - most notably, a sync and build of the complete system. You can even set it up to run other actions like a unit test suite, static analysis, or others.

Motley: Sweet! You mean once the server is set up it will do all of this on its own without manual intervention??

Maven: You bet! Individual developers can monitor the results by using either (a) a web site that they hit for integration status; (b) a Windows tray application that always indicates current build status (red icon = failure; green icon = successful run); or (c) an e-mail notification. It will track historical runs and tell you what change list triggered the run.

Motley: Oh, I must say, that is way cool! Now we don't have to babysit developers, and the code is being integrated constantly. That should definitely cut down on build breaks. How long does it take to set up?

Maven: All you need is a machine capable of performing builds, the CC.NET installation package, and an installation of IIS and ASP.NET if you want to use the web portal. Give it an hour or so to set up and you'll be off and running.

Motley: I am definitely going to try this out. It would be even cooler if Visual Studio had this kind of functionality built in.

Maven: Your wish is Microsoft's command! Visual Studio 2008 ("Orcas") Team Foundation Server supports continuous integration. You can specify a trigger for a build when you create a new build definition or modify an existing one. You can use on-demand builds, rolling builds, and continuous integration where each check-in starts a build. This all integrates with Visual Studio source control support.

Motley: Not bad, Mave. Now give me some more information on how I can get you out of my hair for a couple of hours so I can get this stuff implemented and communicated to the team.

______________________________

Maven's Pointer: Continuous integration (CI) is quickly becoming a mandatory practice for all development shops. Risk of breaking the software is substantially decreased, and as a result, the team becomes more efficient by minimizing blockages due to build breaks. In addition, you integrate as you go and avoid the "big bang" integration at the end. Also, since build breaks flagged by a CI tool are immediate and generally tied to one or two changes to the source, it is easy to narrow down the problem and fix it.

Maven's Resources:

CruiseControl.NET - free server-based continuous integration for .NET projects
Blog summary describing changes in Visual Studio 2008 Team System for builds
Extreme Programming practice description for continuous integration
Martin Fowler's description of continuous integration (great write-up)