Stop the *Nightly Build* madness (but keep Daily Builds)
I think it is pretty much common knowledge in the software world today that ‘Daily Builds’ are a good and necessary practice. I do not think that daily builds was that common a practice five years ago. I love this movement to building your product every day.
What I do not like is that some people (including some Microsofties) start their daily builds at *night* after everyone has gone home. I plead to all Microsoft writers to stop recommending that builds be run at night and instead recommend that automated tests run overnight. Builds should be run during the day – hence the name “Daily Builds’ - The main reason being that build breaks will be fixed a lot faster when people are in the office and not at home asleep. And sorry to break the news to you but build breaks will happen. It is just a fact of life. No matter how good your software or build process is, you will have build breaks - except maybe in the Windows Main Lab builds but then that's another complex story. There are ways to minimize breaks and the impact but that'll have to be another blog entry.
Many years ago in the NT Build lab we tried the nightly or overnight builds thinking that we would save a lot of time utilizing the high-power machines at night. The reality was that someone (aka: hero) on the build team ended up coming in very early in the morning to fix a build break that happened at night. Our overnight build success rate was around 15% which is totally unacceptable in any group, especially the NT Group. What we ended up doing was spending the whole morning fixing the build break and then restarting the build. The whole overnight effort actually slowed us down rather than helped us because it promoted bad development behaviors (such as no one took the build serious since it failed most of the time).
When I was in the BackOffice team we went through the same mistake; just ask my wife, she loved it when I would get a phone call at 4 or 5 in the morning (from the Build Verification Team) that something was broke in the build and they needed help getting it fixed asap. To put an end to this madness we just switched to building during the day. Sometimes we would be at the office until 11pm getting that daily build out but at least we could sleep in (until 9am) the next day because everyone was able to pick up a solid build (from the day before).
Here is roughly how this would look on a Daily basis (and pretty much how many teams build at MS):
9-10am Bug triage meetings (talk about check-ins that will be built today)
11-2pm Check-in window open (only allow check-ins into main source line during this time)
2-5pm Build product (times may vary)
5-6pm Release product to test shares
6pm-9am Run automated tests at night
As part of the daily build process, a regular build schedule should be published. This should include the last check-in time, the time that the build will be completed, and the time that initial Build Verification Tests (BVT) tests will be completed. When any part of this process is broken, the person and module that broke the build should be published via build intranet page and/or e-mail.
If you need more details or examples, wait for my book <grin>
