On the VB team, we don't checkin to our source control system (Source Depot or sd) ourselves like mere mortals, we use the automated checkin system called Gauntlet. Gauntlet runs as an .hta app that collects your unsubmitted changelist, asks for some information for the checkin email it will send if things go well, and the bugs it should close in our issue tracker (Product Studio or PS). You run the app from inside our build environment so it has all the right environment variables and stuff it needs to collect you change. After it gets all this data, it uploads it as XML to a server where you wait in the queue.

Once your job gets picked up, it is passed to a build machine which builds your fix and outputs it to a known path on that machine. The controller machine then picks up those changes and puts them on a public share and instructs the test machines to begin. These machines pick up your private fixes and run a pre-defined set of tests you picked when you submitted the job to Gauntlet (you pick what sub-team you work on and it runs all of those tests). If the tests pass, your code is checked into the depot, the bugs are resolved, and the checkin mail goes out (woohoo!). If the tests fail, you get mail indicating which suites failed, what build flavor (debug or release, checked and retail in our language), and links to the logs output by the tests.

When I got in this morning I got a bunch of suite failure mails for 2 checkins that I started over the weekend. Shortly after I parsed through all of them, I got a mail from our Gauntlet admin letting me know we had some machine issues and that's why all the suites failed. Thankfully, it's easy to re-activate a failed job through Gauntlet's web UI. Unfortunatly, I'll be at the back of a long line :P