Notes on comments.
Welcome to our blog dedicated to the engineering of Microsoft Windows 7
Aka: A developers view of the Windows 7 Engineering process
This post is by Larry Osterman. Larry is one of the most “experienced” developers on the Windows team and has been at Microsoft since the mid 1980’s. There are only three other folks who have worked at Microsoft longer on the entire Windows team! Personally, I remember knowing about Larry when I started at Microsoft back in 1989—I remember he worked on “multimedia” (back when we used to host the Microsoft CD-ROM Conference) and he was one of those people that stood up and received a “5 Year” award from Bill Gates at the first company meeting I went to—that seemed amazing back then! For Windows 7, Larry is a developer on the Devices and Media team which is where we work on audio, video, bluetooth, and all sorts of cool features for connecting up devices to Windows.
Larry wrote this post without any prodding and given his experience on so many Windows releases these thoughts seemed really worthwhile in terms of sharing with folks. This post goes into “how” we work as a team, which for anyone part of a software team might prove pretty interesting. While this is compared and contrasted with Vista, everyone knows that there is no perfect way to do things and this is just a little well-informed perspective.
So thank you Larry! --Steven
Thanks to Steven and Jon for letting me borrow their soapbox :-).
I wanted to discuss my experiences working on building Windows 7 (as opposed to the other technical stuff that you’ve read on this blog so far), and to contrast that with my experiences building Windows Vista. Please note that these are MY experiences. Others will have had different experiences; hopefully they will also share their stories here.
The experience of building Windows 7 is dramatically different from the experience of building Vista. The rough outlines of the product development process haven’t changed, but organizationally, the Windows 7 process is dramatically better.
For Windows Vista, I was a part of the WAVE (Windows Audio Video Excellence) group. The group was led by a general manager who was ultimately responsible for the deliverables. There was a test lead, a development lead and a program management lead who reported to the general manager. The process of building a feature roughly worked like this: the lead program managers decided (based on criteria which aren’t relevant to the post) which features would be built for Windows and which program managers would be responsible for which feature. The development leads decided which developers on the team would be responsible for the feature. The program manager for the feature wrote a functional specification (which described the feature and how it should work) in conjunction with development. Note that the testers weren’t necessarily involved in this part of the process. The developer(s) responsible for the feature wrote the design specification (which described how the feature was going to be implemented). The testers associated with the feature then wrote a test plan which described how to test the feature. The program manager or the developer also wrote the threat model for the feature.
The developer then went off to code the feature, the PM spent their time making sure that the feature was on track, and when the developer was done, the tester started writing test cases.
Once the feature was coded and checked into the source tree, it moved its way up to the “winmain” branch. Aside: The Windows source code has been arranged into “branches” – the root is “winmain”, which is the code base that would ultimately become Windows Vista. Each developer works in what are called “feature branches”, which merge changes into “aggregation branches”, the aggregation branches move into winmain.
After the feature was coded, the testers tested, the developers fixed bugs and the program managers managed the program :-). As the product moved further along, it got harder and harder to get bug fixes checked into winmain (every bug fix carries with it a chance that the fix will introduce a regression, so the risk associated with each bug fix needs to be measured and the tolerance for risk decreases incrementally). The team responsible for managing this process met in the “ship room” where they made decisions every single day about which changes went into the product and which ones were left out. There could be a huge amount of acrimony associated with that – often times there were debates that lasted for hours as the various teams responsible for quality discussed the merits associated with a particular fix.
All-in-all, this wasn’t too different from the way that features have been developed at Microsoft for decades (and is basically consistent with what I was taught back in my software engineering class back in college).
For Windows 7, management decided to alter the engineering structure of the Windows organization, especially in the WEX [Windows Experience] division where I work. Instead of being fairly hierarchical, Steven has 3 direct reports, each representing a particular discipline: Development, Test and Program Management. Under each of the discipline leads, there are 6 development/test/program management managers, one for each of the major groups in WEX. Those 2nd level managers in turn have a half a dozen or so leads, each one with between 5 and 15 direct reports. This reporting structure has been somewhat controversial, but so far IMHO it’s been remarkably successful.
The other major change is the introduction of the concept of a “triad”. A “triad” is a collection of representatives from each of the disciplines – Dev, Test and PM. Essentially all work is now organized by triads. If there’s ever a need for a group to concentrate on a particular area, a triad is spun off to manage that process. That means that all three disciplines provide input into the process. Every level of management is represented by a triad – there’s a triad at the top of each of the major groups in WEX, each of the second level leads forms a triad, etc. So in my group (Devices and Media) there’s a triad at the top (known as DKCW for the initials of the various managers). Within the sound team (where I work), there’s another triad (known as SNN for the initials of the various leads). There are also triads for security, performance, appcompat, etc.
Similar to Windows Vista, the leads of all three disciplines get together and decide a set of features that go in each release. They then created “feature crews” to implement each of the features. Typically a feature crew consists of one or two developers, a program manager and one or two testers.
This is where one of the big differences between Vista and Windows 7 occurs: In Windows 7, the feature crew is responsible for the entire feature. The crew together works on the design, the program manager(s) then writes down the functional specification, the developer(s) write the design specification and the tester(s) write the test specification. The feature crew collaborates together on the threat model and other random documents. Unlike Windows Vista where senior management continually gave “input” to the feature crew, for Windows 7, management has pretty much kept their hands off of the development process. When the feature crew decided that it was ready to start coding (and had signed off on the 3 main documents), the feature crew met with the second level triad (in my case with DKCW) to sanity check the feature – this part of the process is critical because the second level triad gets an opportunity to provide detailed feedback to the feature crew about the viability of their plans.
And then the crew finally gets to start coding. Sort-of. There are still additional reviews that need to be done before the crew can be considered “ready”. For instance, the feature’s threat model needs to be reviewed by one of the members of the security triad. There are other parts of the document that need to be reviewed by other triads as well.
A feature is not permitted to be checked into the winmain branch until it is complete. And I do mean complete: the feature has to be capable of being shipped before it hits winmain – the UI has to be finished, the feature has to be fully functional, etc. In addition, when a feature team takes a dependency on another Windows 7 feature, the feature teams for the two features MUST sign a service level agreement to ensure that each team knows about the inter-dependencies. This SLA is especially critical because it ensures that teams know about their dependants – that way when they change the design or have to cut parts of the feature, the dependent teams aren’t surprised (they may be disappointed but they’re not surprised). It also helps to ensure tighter integration between the components – because one team knows the other team, they can ensure that both teams are more closely in alignment.
Back in the Vista day, it was not uncommon for feature development to be spread over multiple milestones – stuff was checked into the tree that really didn’t work completely. During Win7, the feature crews were forced to produce coherent features that were functionally complete – we were told to operate under the assumption that each milestone was the last milestone in the product and not schedule work to be done later on. That meant that teams had to focus on ensuring that their features could actually be implemented within the milestone as opposed to pushing them out.
For the nuts and bolts, The Windows 7 development process is scheduled over several 3-month long milestones. Each milestone allowed for 6 weeks of development and 6 weeks of integration – essentially time to fine-tune the feature and ensure that most of the interoperability problems were shaken out.
Ok, that’s enough background (it’s bad when over half a post on Windows 7 is actually about Windows Vista, but a baseline needed to be established). As I said at the beginning, this post is intended to describe my experiences as a developer on Windows 7. During Windows 7, I worked on three separate feature crews. The first crew delivered two features, the second crew delivered about 8 different features all relatively minor and the third crew delivered three major features and a couple of minor features. I also worked as the development part of the WEX Devices and Media security team (which is where my series of post on Threat Modeling came from – I wrote them while I was working with the members of D&M on threat modeling). And I worked as the development part of an end-to-end scenario triad that was charged with ensuring that scenarios that the Sound team defined at the start of the Windows 7 planning process were actually delivered in a coherent and discoverable way.
In addition, because the test team was brought into the planning process very early on, the test team provided valuable input and we were able to ensure that we built features that were not only code complete but also test complete by the end of the milestone (something that didn’t always happen in Vista). And it ensured that the features we built were actually testable (it sounds stupid I know, but you’d be surprised at how hard it can be to test some features). As a concrete example, we realized during the planning process that some aspect of one of the features we were working on in M2 couldn’t be completed during the milestone. So before the milestone was completed, we ripped the feature out (to be more accurate, we changed the system so that the new code was no longer being built as a part of the product). During the next milestone, after the test team had finished writing their tests, we re-enabled the feature. But we remained true to the design philosophy – at the end of the milestone everything that was checked into the “main” branch was complete – it was code AND test complete, so that even if we had to ship Windows 7 without M3 there was no test work that was not complete. This is a massive change from Vista – in Vista, since the code was complete we’d have simply checked in the code and let the test team deal with the fallout. By integrating the test teams into the planning process at the beginning we were able to ensure that we never put the test organization into that bind. This in turn helped to ensure that the development process never spiraled out of control. Please note that features can and do stretch across multiple milestones. In fact one of the features on the Sound team is scheduled to be delivered across three milestones – the feature crews involved in that feature carefully scheduled the work to ensure that they would have something worth delivering whenever Windows 7 development was complete.
Each of the feature crews I’ve worked on so far has had dramatically different focuses – some of the features I worked on were focused on core audio infrastructure, some were focused almost entirely on UX (user experience) changes, and some features involved much higher level components. Because each of the milestones was separate, I was able to work on a series of dramatically different pieces of the system, something I’ve really never had a chance to do before.
In Windows 7, senior management has been extremely supportive of the various development teams that have had to make the hard decisions to scale back features that were not going to be able to make the quality bar associated with a Windows release – and there absolutely are major features that have gone all the way through planning only to discover that there was too much work associated with the feature to complete it in the time available. In Vista it would have been much harder to convince senior management to abandon features. In Win7 senior management has stood behind the feature teams when they’ve had to make the tough decisions. One of the messages that management has consistently driven home to the teams is “cutting is shipping”, and they’re right. If a feature isn’t coming together, it’s usually far better to decide NOT to deliver a particular feature then to have that feature jeopardize the ability to ship the whole system. In a typical Windows release there are thousands of features and it would be a real shame if one or two of those features ended up delaying the entire system because they really weren’t ready.
The process of building 7 has also been dramatically more transparent – even sitting at the bottom of the stack, I feel that I’ve got a good idea about how decisions are being made. And that increased transparency in turn means that as an individual contributor I’m able to make better decisions about scheduling. This transparency is actually a direct fallout of management’s decision to let the various feature teams make their own decisions – by letting the feature teams deeper inside the planning process, the teams naturally make better decisions.
Of course that transparency works both ways. Not only were teams allowed to see more about what was happening in the planning process, but because management introduced standardized reporting mechanisms across the product, the leads at every level of the hierarchy were able to track progress against plan at a level that we’ve never had before. From an individual developer’s standpoint, the overhead wasn’t too onerous – basically once a week, you were asked to update your progress against plan on each of your work items. That status was then rolled up into a series of spreadsheets and web pages that allowed each manager to track all the teams’ progress against plan. This allowed management to easily and quickly identify which teams were having issues and take appropriate action to ensure that the schedules were met (either by simplifying designs, assigning more developers, or whatever).
In general, it’s been a total blast building 7. We’ve built some truly awesome features into the operating system and we’ve managed to keep the system remarkably stable during that entire process.
Just as Messenger, and soon Movie Maker got cut, we will see more and more cuts, but the features will still be out there. Way to go. The biggest feature of Windows 7 will have to be the code shrinkage. Even if the requirements are the same, and tons of new features added, I expect the code won't grow much if at all, if not shrink a little. While that isn't the holy grail, it will still be very interesting.
Also, I have to say that Seven is also a great name, and if I don't see a Bond (i.e. 007) joke before this thing hits SP1, I will have to give up on Microsoft's sense of humor or marketing.
Yes I love It UMPC MT+ Windows 7 = Surface consumer :D
but see this
Go Micosoft GO!!!
The phrase "delivered in a coherent and discoverable way" caught my attention. As said, there are thousands of features implemented in Windows releases. The problem is that a lot of them are not easily discovered.
The Welcome Center in Vista makes an attempt at helping to discover features but falls short, in my mind at least, of being truly useful. The "What's new in Windows Vista" entry talks about Searching And Organizing, for example, but never gets to any meat. It says you can search for stuff and points out the search box, but never talks about the cool points of how to use it. It links to a help article "Demo: Working with files and folders", which doesn't even touch on search.
I wonder how many people know that you can search from the start menu for emails in outlook. How many "power users" wanted to turn off UAC and had trouble finding the switch when they could have simply used the search box in the Control Panel window to search for "uac" and be led right to it? How many know that you can use natural language for searches to narrow down searches, like "email from SomeGuy@hotmail.com" or "Songs by SomeArtist"? For those that do, how many are really familiar with the extent of that usage? I'm certainly not. I just know that it can be done, and I haven't taken the time to find a listing of the possibilities.
This leaves a lot a room for improvement. Of course it's probobly a bit late in the development lifecycle for Win7 feature suggestions, but here's one anyway, maybe for the next release or a service pack or something. A mechanism to help in discovery of prominent features by tracking which have been hit. Privacy concerns abound with any tracking so it'd have to be opt-in or whatever, but I'd certainly make use of it if it did a good job.
Say there's a set of features that you want to make sure that users discover. Have a checklist somewhere of what points of what features are hit by the user. Let the user opt in, and track which features have been touched on, and how well they're being utilized. Monitor certain clues indicative of a place they could work faster or easier.
Take search, for example. If a person is always going into All Programs in the start menu, prompt them in a non-intrusive way and ask if they know they can search for programs in the start menu. If they're exploring a lot of Control Panels but never doing anything, suggest they try searching for what they want. If they search for *.mp3 suggest they use "songs" instead, and offer a link for an in-depth guide to really using the search functionality.
Different levels of pushiness might be desired, ranging from on-demand bubbles to a daily or weekly suggestion dialog to turning the feature off completely. The point is to actually help people discover the features that make the OS great.
There are so many things where it seems you have to know to look for them, like search semantics or keyboard shortcuts for frequently performed tasks (win-char combinations, anyone?), that a feature to help discover other features would really be useful for a lot of people. Power users can turn them off or leave it on for discovery. Grandparents can almost always use the extra tips on the things that make the system easier to use.
Overall, I think a lot more could be done to showcase the things that make each new version of windows better. It hurts me inside, but Apple really does a good job at that. The user manual for the iPhone, from what I hear, is basically a guide for how to demo the machine for your friends. Everybody I knew that got it when it first came out seemed to say "Well, all I know how to do is look at this picture, turn the phone sideways and pinch it to zoom, but doesn't it look neat?"
Apple has a great following for a phone that never impressed me because they did a great job at making a couple things work well and look pretty (and let's face it, displaying images isn't very special by itself), and showing their customers how to show their friends.
I mean, there were too many people saying "man, my older-than-dust printer doesn't have updated drivers" or "Vista doesn't work with my video card". How great would it have been if they were saying something like "Isn't this cool? I can search for any program right here", "Watch this, I can find all my favorite music by typing 'songs with more than 4 stars' and THEN I can save the search and have it right there ready for me", or "Check this out, these popups make sure I know something's messing with the system, so I can stop malware and viruses before they do anything serious." If we taught people how to use all the amazing features that get put into the releases, rather than letting people focus on and get frustrated about things like the frequent UAC prompts, the overall attitude would be so different.
The old way of building windows kinda sounds like a huge mess to me... the new one sounds much better.
Maybe i misunderstood a part of the post but did the old way really force some features to be merged into the winmain branch even though they were not ready/tested completely yet? O_O
I hope that win7 will be much better than Vista & XP and will use less space on the HDD compared to Vista. As I understand it, Vista is taking so much space because all the drivers/language support packages etc get copied to the HDD so the user doesn't have to insert the DVD every time a component is installed. Please, please make it an option to NOT do this. I prefer to insert the windows DVD over having all the components i'll possibly never need occupy the space i could use for other things.
Hey I thought you'd talk in general about audio instead of this, although this was an interesting read. (Milestones 2 and 3 did ship with under-the-hood features!!!, only thing ppl could do is post UI screenshots). Coming back to audio, I just hope your teams build something new for MIDI and USB interfaces and audio pros in Windows 7. (The pro audio area has been somewhat neglected by Microsoft and although Vista introduced WaveRT, it only supports PCI devices). And something new with HTPC and gaming audio as well. (Make XAudio 2 part of the OS).
Great to see engineers participating in this blog(go Larry, go!!) It's just awesome that senior Windows executives are so engaged here (and Sinofsky's writings are excellently written, informative and unusually candid.)
More devs on Engineering Windows please!
Thankyou for some insight into the process of developing Win7.
The current development process is very similar to the one that I am familiar with in my limited experiences as a developer. I must say that I have found that environment much less stressful and more productive then some of the other models.
What happens though with features that are not ready by the launch? Do they just disappear? Are they released later as an update or as part of a service pack?
This is excellent stuff, and I'll bet Windows 7 is a great OS, just like Vista before it.
Thanks for keeping us informed, these posts have really got me excited for what you'll be posting in the future on specific technologies.
MS is truly responsible for the dream of 'a PC in every house' coming true, I hope when Windows 7 is done and the press is falling over themselves to praise it, you guys take a nice, well deserved vacation. :)
Another great post, thanks!
It sounds like the building of Windows 7 is much more dynamic than previously, and that sounds like a good thing. I'm really pleased to hear that it could essentially be shipped today and all features would be complete and tested, even if the OS was not as complete as everyone might want.
I love Vista, but there's an excitement around 7 that I just can't escape...
See a lot of talk about features which is nice but what about core OS? are we getting vista with different and more features? Hopefully this time they are actually useful!
I have been myself on the short milestone model for a while and it is a model that I don't feel it can work for the long term. There are important changes that seem impossible to fit in, and they never get done. Then each discipline starts looking for workarounds when the other disciplines are busy and not looking (devs implementing "stealth" rearchitecture, PMs steamrolling tasks in the schedule, etc). In the end you have a team that works like a person with multiple personality disorder where the disparity in perception of reality (and tensions because of it) just keeps growing. I'm not sure how it will blow up but I'm almost sure it will. Of course, I may be wrong and a reorg will save the day when someone believing in a completely different model gets put at the helm and a new process model (that probably doesn't work for the long term either) gets implemented
back in the old Win 3.0 days Windows was innovative, with a capital 'i'. It made computers easier to use and more powerful simply by being there. Then 3.1, WfWg, Win '95, Win '98, and Win 2000 all showed vast measurable improvements over their predecessor. People _WANTED_ to upgrade, because it meant "more". XP was only slightly disappointing, but still had enough eye candy, hardware support, and '9x compatibility to get a general thumbs up.
But then... along came ".Net". What was already working well just _HAD_ to be re-invented, and in a bass-ackwards way that made everything that ran with it INEFFICIENT out of the gate (by comparison to what it would have been like). Never mind that senior developers such as myself were _NOW_ faced with the prospect of having to 'learn it all again' (so that we could become JUNIOR developers again, thanks a lot). It was, in my view, the beginning of a disaster waiting to happen.
Products generally go through a life cycle. In the beginning there is REVOLUTIONARY change, where early adopters get on board but that's about it. Then there's INNOVATION, which puts the product into the hands of the power users and early adopters. Then there's the "mass market" which puts the technology into everyone's hands. But without another development cycle of revolutionary change and TRUE innovation, you get _DECLINE_. This is what Vista is doing.
Lessons to be learned:
1. TRUE INNOVATION. Why can't I speak to my computer like they do on Star Trek? Why must I mousie-clickie _everything_ at a rate that is sometimes 10 times slower than it would be by just typing it (like MS-DOS)?
2. FASTER and BETTER (not SLOWER and MORE CUMBERSOME)
3. Eye candy is for SALES/MARKETING execs (regular folk just think it's gaudy and unnecessary). Eye candy doesn't sell OSs and Software.
4. What about the "Killer App" concept? Us developers were _SUPPOSED_ to be the reason Windows was successful! Back to the '.Net' gripe on THIS one...
5. Computer users aren't idiots. They don't need condescending menus and terms. 'Nuff said.
Bob, well said. I think we need to stop innovating and just focus on what we know best.
"BigBadBombasticBob: 1. TRUE INNOVATION. Why can't I speak to my computer like they do on Star Trek? Why must I mousie-clickie _everything_ at a rate that is sometimes 10 times slower than it would be by just typing it (like MS-DOS)?"
You can control Vista using by using your voice
You have broken DX7 badly in Vista. Compare the start transition with WinXP. I can work with you to fix it. Do not say port to DX9, there's good reasons for 7.
Mail me for details. Thank you.