It's on the whiteboard

Dave Cutler

orcmid — Thu, 13 Oct 2005 02:42:27 GMT

I met Dave Cutler at the NT announcement event in San Francisco in the Summer of 1992. What stood out for me based on lobby conversations and chatting was how he showed a very distinctive trait of system architects. He was very clear on the invariants that he would hold onto no matter what, confident that anything else would be fixable.

It is no surprise as development processed moved to shipping that Dave would be involved in the operational end, finding the key thing to keep tied down.

That's a great story. Thanks.

re: It's on the whiteboard

Norman Diamond — Thu, 13 Oct 2005 06:55:19 GMT

> when the whiteboard was full, the product
> had lots of bugs, when it was clear, we were
> close to being done. And when it was empty,
> we had shipped the product.

When the whiteboard was full, you were far from shipping. When it was clear, you were close to shipping. And when it was empty, you had shipped the product. The number of bugs fluctuates independently of that status.

re: It's on the whiteboard

LarryOsterman — Thu, 13 Oct 2005 08:30:25 GMT

Norman, every software product ever shipped has shipped with known bugs.

Every single one of them. The only question is if the bugs are sufficiently bad to justify holding up the shipment.

So as long as there are bugs bad enough to justify holding the product up, there will be fixes for those bugs (as we find the bugs, we fix them).

re: It's on the whiteboard

Matt — Thu, 13 Oct 2005 10:10:18 GMT

Nice posting, very informative. Please blog more of these stories

re: It's on the whiteboard

Manip — Thu, 13 Oct 2005 12:52:38 GMT

hmm, well I guess I should be paying attention in /Software Engineering/ then... However much I happen to hate it.

re: It's on the whiteboard

Tim Smith — Thu, 13 Oct 2005 17:01:34 GMT

Manip:

I'm not exactly sure what your "Software Engineering" course might teach, but a quick scan of Google makes me think you REALLY need to pay attention to that class.

My biggest day to day issues that I have with my level 1 newbie programmers all the way up to my level 4 Greek God programmers are failings in software engineering. Requirements gathering, testing, playing nice with others, maintainability are things I see day to day.

We all tend to come out of school thinking we are the smartest and best programmers who will magically produce the best code that the user could ever hope to have. It never works out that way. My job as a lead is to nicely (will, sometimes not so nicely) beat that out of people. My goal isn't to teach my guys how to produce software that is the coolest software ever created. My goal is to teach my guys how to deliver functional and maintainable software that meets the needs of the user so we can all sit on the lawn at 1:30 drinking beer because the user is happy and the software "just works".

re: It's on the whiteboard

Dan McCarty — Thu, 13 Oct 2005 17:04:33 GMT

"you had to explain why you screwed up to Dave directly, and it was FAR more effective"

That was a much better explanation than G. Pascal Zachary was able to manage in 20 pages of his Showstopper! book. Thanks for the information and history.

I have to give in to a bit of pedantry: Cutler wasn't a distinguished engineer back then, was he?

re: It's on the whiteboard

Jeff Parker — Thu, 13 Oct 2005 17:45:46 GMT

I have to sit there and laugh, actually it is amazing you could do that with 20 people. Let alone thousands. We still have problems sometimes with a 5 man team tripping over each other. Also as far as shipping software with bugs. Yep I have had to do the same myself. Sometimes it has to happen

re: It's on the whiteboard

Ben Bryant — Thu, 13 Oct 2005 18:40:23 GMT

I'd love to hear how you got past the whiteboard and scaled up to hundreds of developers. In other words, it sounds like at one time it was very dependent on the Dave Cutler personality, so who was instrumental in moving beyond that to a more scalable build arrangement? Great story, thanks.

re: It's on the whiteboard

Mike — Thu, 13 Oct 2005 21:56:03 GMT

The vbl system isn't all skittles and beer... The multiple layers of branch heirarchy and then buerocracy involved in getting fixes integrated between them means it can literally take months for a fix to propagate through the system. Even so it's better than what we had before where you counted yourself lucky to get a set of source that didn't have a build break.

re: It's on the whiteboard

Norman Diamond — Fri, 14 Oct 2005 05:13:51 GMT

Thursday, October 13, 2005 1:30 AM by LarryOsterman
> Norman, every software product ever shipped
> has shipped with known bugs.

There are a few that ship only with unknown bugs. I think we can agree by deleting one word from your sentence: every software product ever shipped has shipped with bugs.

> The only question is if the bugs are
> sufficiently bad to justify holding up the
> shipment.

Sure, but we have pretty big differences of opinion over what kind of bug is sufficiently bad. I think that a bug which destroys all files in a disk partition is an example of sufficiently bad.

But get this: even when Windows 95 did that to me, even after I got it tracked down, I felt somewhat understanding because I know that every product ships with bugs. What made the difference was your company's reneging on warranties, refusal to allow bug reports to be submitted without the victim paying a fee, and then when your company accidentally allowed a contact which led to discussion of this bug, your company told lies denying it, and then switched to lies saying that it wasn't serious because the number of victims was low and the product was old. (The number of victims was not low, only the number of victims who understood it was low. And Windows 95 was still being shipped to corporate customers.)

So we agree on the fact of bugs, but we have widely differing opinions on what kind of bug is serious and whether fixes should be delivered to customers.

re: It's on the whiteboard

Monica — Fri, 14 Oct 2005 06:27:41 GMT

Perhaps the whiteboard could be scaled using a "draft" repository and a "real" one. Checking a change into the former queues up a whiteboard-style entry on an integration list; the integration team pulls in the changes at their discretion, as they did with the whiteboard, and anything that breaks the build gets dumped back in the lap of the engineer who did the check-in, at which point he has to provide a replacement or followup.

I've never worked with that large an organization, so this is purely speculation.

re: It's on the whiteboard

Funnygroan — Fri, 14 Oct 2005 06:43:22 GMT

Of course, this procedure must surely have prompted the question, "What if Dave Cutler was hit by a bus?".

There has been a similar question asked in the GNU/Linux world for years. It was finally answered by the following experiment:

http://web.archive.org/web/20000422005356/http://segfault.org/story.phtml?mode=2&id=38b40d78-087dd360
[via geekz.co.uk]

Were similar studies performed at M$?

re: It's on the whiteboard

LarryOsterman — Fri, 14 Oct 2005 06:53:25 GMT

Monica, in fact, for NT 3.51 to somewhere during the Win2K process, the whiteboard was replaced with three Exchange public folders that comprised the "electronic whiteboard". You submitted (using a custom form) your change to the "pending" queue, the build lab picked it up and put it in the "processing" queue, when the change was committed, it went to the "completed" queue and mail was automatically sent out.

That worked for NT 4, but by the time W2K came out, it was unworkable. For Win2K, Mark Lucovsky and several others redesigned a totally different system. He went into some detail in his usenix talk at: http://www.usenix.org/events/usenix-win2000/invitedtalks/lucovsky_html/Lucovsky.ppt

For Win2K, there were about 1400 developers working on Windows. There are significantly more developers than that contributing to Vista, that's why the new structure is so critical.

Mike's also right - the branching architecture adds reliability at a cost of bug fix latency. Fixes can live in a build lab for three or four weeks before they hit the main tree, which can be a big deal. On the other hand, if a particular vbl is seeing a particular problem, it's usually pretty easy to make the fix in their branch and wait for it to propogate into your branch.

On the other hand, winmain builds are known to be of VERY high quality, which helps a LOT.