Automated Continuous Integration and the Ambient Orb™
How often have you or one of your fellow developers checked code into your source control system that causes a build failure for someone else? And how many times have you heard that “it builds just fine on my machine?” More importantly, how much time does it take to backtrack through all of the changes to uncover the root cause of the failure? If the last good check-in was two days ago, it can be difficult to figure out not only what caused the failure, but whose check-in did it.
Integration problems like this happen every day on software development projects. Back in 1996, Steve McConnell published an article in IEEE Software magazine about the benefits of performing a Daily Build and Smoke Test that addressed this specific issue. Since then, the concept of a daily build has become more accepted in the development community, but it still amazes me how many projects don’t take advantage of this simple, yet powerful technique.
Well, if we agree that integrating and building a project once a day is good, are there any benefits to continually integrating and building many times a day? How about every 15 minutes?
Continuous Integration
Continuous integration is a technique that encourages developers to check-in their code more frequently, have it compiled by an automated process, run a suite of unit tests, and report on the status. The idea is to tighten the feedback loop so that the effect of an integrated change is communicated back to the developer as soon as possible. By reducing the time between check-in and build status, developers find it much easier to identify faulty code.
A continuous integration server is responsible for monitoring the source code repository for changes that are made during check-in. When a change is detected, the server automatically:
- Performs a full check-out
- Cleans the build output folder
- Forces a complete rebuild of the entire project
- Executes all unit tests (optional)
- Reports on the build and unit testing status
Of course, for this to work, all source code needs to be stored in a central location that is shared by the development team. Fortunately, source control tools like Visual Studio 2005 Team System, Visual SourceSafe, CVS, Perforce, Subversion, Rational ClearCase, SourceGear Vault, etc. are found on many projects and provide a single repository.
For the continuous integration server to build the project, a build script needs to be created. NAnt is a popular and free Open Source project that is supported by both CruiseControl.NET and Draco.NET, two popular continuous integration systems for .NET development. Before configuring the continuous integration server, it is worth spending the time to create a functional build script. Although NAnt is not difficult to use, plan to invest a little time reading through the documentation before generating your first script.
Last, although unit tests are optional, they are highly recommended. On the NxOpinion project, we have over 600 individual NUnit tests that run during every build. The success of this suite of tests tells us that a majority of the basic functionality that is provided by our system is working as-expected. We consider the failure of a single unit test to be equal to a full build failure, and we work to immediately resolve any issues.
With CruiseControl.NET, when a build or test failure is detected, an e-mail message is sent to the entire development team notifying them of recent check-ins along with the specific failure. When a check-in is successful, an e-mail is sent only to those developers who checked-in code for that build. This way, everyone is always kept in-the-loop on build status. CruiseControl.NET also provides a web site and system tray application to monitor the status of the continuous integration server.
For our current project, we have configured the server to poll for changes every 15 minutes. If there have been no check-ins since the last build, the system sleeps for another 15 minutes. This cycle continues until new changes are detected. When this happens, the entire system is completely rebuilt. Because of the relatively short polling period, developers only have to wait a few minutes before learning the status of their most recent check-in.
CruiseControl.NET can also automate FxCop. FxCop is a code analysis tool that uses reflection and IL parsing to check managed assemblies for conformance to the Design Guidelines for Class Library Developers. Think of it as an "after-the-fact code review" that doesn't depend on the original source code. The current version of FxCop evaluates over 200 individual rules, and custom rules can be created. FxCop is very comprehensive, and it is very rare to receive a completely clean report. You may need to ignore rules that don't apply to your project. Although we run FxCop as part of our build, we do not currently have it configured to cause a build failure if any violations are found.
For a very thorough look at continuous integration, I recommend reading Continuous Integration with CruiseControl.NET and Draco.NET by Justin Gehtland.
The Ambient Orb™
Webster defines ambient as an adjective that means “existing or present on all sides.” When used as a noun, it means “an encompassing atmosphere.” Ambient science is an evolving field that attempts to convey low-bandwidth information by embedding it into our surrounding environment. The idea is that certain information isn’t worthy of interruption, so instead, it should available at a glance.
Ambient Devices is a company that manufactures products that convey ambient information. The Ambient Orb is a $150 sphere of frosted glass that glows a nearly unlimited variety of colors. The Orb is capable of some simple color “animations” like pulsing and an effect called “crescendo” that slowly brightens a color, then immediately dims. Ambient Devices also makes a Beacon that is similar in concept to the Orb, and a recently-available Dashboard.
The Orb is commonly sold as a device to monitor the stock market, local weather, or any of a number of other free channels of information. Fortunately, for the true geek, a premium account is available for around $20 a quarter that allows you to send your own custom information to the Orb. By calling a specific URL at Ambient’s web site, you can control both the color and animation of your own Orb.
You might ask: “why do I need to call a URL to change the color of something that’s sitting on my desk?” Good question…this is where the geek magic steps in. You see, each Ambient device has wireless pager electronics built-in that allow it to receive signals over the wireless network (I’m not talking about 802.11x wireless…I’m talking about a wireless pager like the kind you’d wear on your belt to monitor those servers that like to go down in the middle of the night). This means that you only have to plug the Orb into a power outlet, register its device ID with Ambient Devices, and from that point on, the Orb will change colors and animations when its information changes. Pretty cool.
The Orb is imminently touchable, and visitors always stop to feel its warm glow. It’s definitely a conversation piece. After they ask if you bought it at Target for $19.95, you can proceed to twist their brain by explaining how you’ve configured your system to send information over the Internet that the Orb wirelessly receives through the pager network. The look on their face is similar to the look you’d get if you said: “I’m an alien from Alpha Centauri and that’s my invisible spaceship parked out there on the lawn.” Priceless.
Raising Build Visibility
So I had this idea that we could configure an Ambient Orb to reflect the current status of our NxOpinion continuous integration build. A slowly pulsing green would mean that the build is currently okay, and a quickly pulsing red would indicate a build failure. I planned to put the Orb in the middle of our project team so that everyone would be aware of the build status. I hoped that by raising its visibility, everyone on the project team (including the customer) would be more aware of the project “health.”
Now, when the build breaks and the Orb pulses red, it’s like a fire alarm around here. The first question out of everyone’s mouth is “who broke it?” After appropriate developer guilt has been piled on by the development team (all in good fun, of course), it’s usually a relatively trivial matter to discover and fix the problem. Because we continuously integrate our code and the automated build potentially runs every 15 minutes, determining what caused the failure is as simple as looking at what has been checked-in since the last successful build. Fortunately, CruiseControl.NET includes this information (along with check-in comments) in its e-mail and web page summaries.
To-date, our solutions contain approximately 175,000 lines of C# code and over 600 unit tests. Since we consider the failure of a single unit test to be a failure of the entire build, if one test fails, the Orb pulses red. As you’d guess, CruiseControl.NET also includes unit test results in its e-mail and web page summaries which makes it easy to identify the problem.
Although we haven't done it on the NxOpinion project, it would be trivial to configure multiple Orbs to reflect the build health at any location within wireless pager range. You could have an Orb in every development office. And even one at home. Okay, maybe I'm going overboard with that last suggestion, but you know what I mean.
Configuration
To configure the automated process to send build status information to the Ambient Orb, we need to add some properties and targets to the NAnt build script that CruiseControl.NET uses. NAnt has two built-in properties that can be leveraged to execute a task on build success and failure. The properties are called nant.onsuccess and nant.onfailure, and they need to be set to point to valid target elements in the build file. In our case, we define targets called OnSuccess and OnFailure, although any valid names will work just fine.
To send information to the Ambient Orb, query string parameters are passed to an Ambient JavaServer Page. The format of the request is as follows:
http://myambient.com:8080/java/my_devices/submitdata.jsp?
devID=###-###-###&anim=#&color=#&comment=Comment+here
Where:
| devID | The device ID (serial number) of your Ambient Orb |
| color | A number representing the color (0-36) |
| anim | A number representing the animation style (0-9) |
| comment | A short comment that is logged at the Ambient web site |
For more information on the available colors, animation styles, and formatting requirements, see the Ambient Orb WDK documentation.
We use the NAnt <get> task to send our build status. The <get> task queries a URL and copies the response to a specified file. In our case, we copy the response to a file in the temporary folder, then immediately delete it with the next task. This isn’t our entire NAnt script, but it does contain enough for you to figure out how to incorporate this into your own process:
<?xml version="1.0" ?>
<project name="Example">
<!-- Load environment variables -->
<sysinfo />
<!-- Define targets for build status -->
<property name="nant.onsuccess" value="OnSuccess" />
<property name="nant.onfailure" value="OnFailure" />
<!-- Set Ambient Orb for successful build -->
<target name="OnSuccess" description="Build success">
<get src="http://myambient.com:8080/java/my_devices/submitdata.jsp?
devID=###-###-###&anim=4&color=12&comment=Build+success"
dest="${sys.os.folder.temp}\delete.me" failonerror="false"/>
<delete file="${sys.os.folder.temp}\delete.me" failonerror="false"/>
</target>
<!-- Set Ambient Orb for failed build -->
<target name="OnFailure" description="Build failure">
<get src="http://myambient.com:8080/java/my_devices/submitdata.jsp?
devID=###-###-###&anim=6&color=0&comment=Build+failure"
dest="${sys.os.folder.temp}\delete.me" failonerror="false"/>
<delete file="${sys.os.folder.temp}\delete.me" failonerror="false"/>
</target>
</project>
Conclusion
We’ve been using CruiseControl.NET for automated continuous integration for the past year-and-a-half, and it has been a fantastic addition to the project. Although continuous integration is typically associated with the Agile development community, it is a technique that can provide major benefits to teams using any project methodology (even if the developers aren’t writing unit tests).
It is rare that our build is in a failed status for more than 30 minutes, because CruiseControl.NET makes it so easy to determine what has changed since the last successful build. Plus, because our Ambient Orb is in a highly visible location within the project team, it is easy to see the health of our source code with just a quick glance. We can go home in the evening confident that our offshore team is working with a healthy build, and we have that same expectation when we come in to work the next morning.
I firmly believe that once you’ve worked on a project with automated continuous integration, you won’t want to work on a project without it. To get started, download either CruiseControl.NET or Draco.NET. It might take a bit of effort to create your first build script, but once you have it up and running, you'll find that it requires relatively little care and feeding to maintain.
Oh, and don’t forget to let me know about your experience!
Resources
Daily Build and Smoke Test by Steve McConnell: http://www.stevemcconnell.com/bp04.htm
Martin Fowler's introduction to Continuous Integration: http://www.martinfowler.com/articles/continuousIntegration.html
Continuous Integration with CruiseControl.NET and Draco.NET by Justin Gehtland: http://www.theserverside.net/articles/showarticle.tss?id=ContinuousIntegration
CruiseControl.NET: http://ccnet.thoughtworks.com/
Draco.NET: http://draconet.sourceforge.net/
NAnt: http://nant.sourceforge.net/
NUnit: http://www.nunit.org/
FxCop: http://www.gotdotnet.com/team/fxcop/
Ambient Orb: http://www.ambientdevices.com/cat/orb/
Ambient Orb WDK: http://www.ambientdevices.com/developer/OrbWDK.pdf