Notes on comments.
Welcome to our blog dedicated to the engineering of Microsoft Windows 7
Just about every email we receive and every comment we get comes with feedback—something to change, something to do more of, something to do less of, and so on. As we’ve talked about in this blog, acting on each one in an affirmative manner is easier said than done. What we can say for certain, is that we are listening to each and every comment, blog post, news story, MS Connect report, Send Feedback item, and of course all the data and telemetry. This post kicks off the discussion of changes made to the product with an overview of the feedback process. We'll get into specific changes shortly and we'll continue to return to the theme of changes in the Release Candidate (RC) over the next weeks. Yesterday on the IE Blog, you saw that we'll be updating IE 8 on Windows 7, and there we also talked about the feedback process in general.
Feedback about Windows 7 of course starts before we've written any code, and by the time we've got running code thousands of people outside of Microsoft have provided input and influenced the feature set and design of Windows 7. As we've seen, the input from even a small set of customers can often represent a wide variety of choices--often in alignment, but just as often in opposition. As we're developing the features for Windows 7 we work closely with PC makers, enterprise customers, and all types of customers across small business, education, enthusiasts, product reviewers and industry "thought leaders", and so on. We shape the overall "blueprint" of the release based on this wide variety of input. As we have design prototypes or code running, we have much more targeted and specific feedback by using tools such as usability tests, concept tests, benchmark studies, and other techniques to validate the implementation of this blueprint. Our goal with this level of feedback is for it to be representative of the broad set of Windows customers, even if we don't have a 1:1 interaction with each and every customer. Hopefully this post will offer some insights into this process overall--the tools and techniques, and the scope of feedback.
In the first few weeks of the Windows 7 beta we had over one million people install and use Windows 7. That's an astounding number for any beta test and while we know it has been fun for many folks, it has been a lot of work for us--but work that helps to raise the quality of Windows 7. When you use the beta you are automatically enrolled in our Customer Experience Improvement Program (anonymous feedback and telemetry, which is voluntary and opt-in in the RTM release). Just by using Windows 7 as a beta tester you are helping to improve the product--you are providing feedback that we are acting on in a systematic manner. Here is a sense of the scale of feedback we are talking about:
We have a variety of tools we draw on to help inform the decision making process. A key element that we have focused on quite a bit in Windows 7 is the role of data in making decisions. Everything we do is a judgment call as ultimately product development is about deciding what to get done from an infinite set of possibilities, but the role of data is essential and is something that has become far more routine and critical. It is important to be super clear—data is not a substitute for good judgment or an excuse to make a decision one way or another, but it most definitely informs the decision. This is especially true in an era where the data is not only a survey or focus group, but often includes a “sampling” of millions of people using Windows over the course of an extended time period.
A quick story from years ago working on Office, many years ago before the development of telemetry and the internet deciding what features to put in a release of Office could really be best described as a battle. The battle took place in conference rooms where people would basically debate until one or more parties gave up from fatigue (mental or otherwise)—essentially adrenaline-based product development. The last person standing, the one with the most endurance, or the one who pulled an all-nighter to write the code pretty much determined how features ended up or what features ended up in a product. Sort of like turning feature design over to a Survivor-like process. I’m sure many of you are familiar with this sort of process. The challenges with this approach are numerous, but inevitably features do not hold together well (in terms of scenarios or architecture), the product lacks coherency, and most importantly unless you happen to have a good match between the “winner” and the target customers, features will often miss the mark.
In the early 1990’s we started instrumenting Word and learning about how people actually used the software (this was before the internet so this was a special version of the product we solicited volunteers to run and then we would collect the data via lots of floppies). We would compile data and learn about which features people used and how much people used them. We learned things such as how much more people used tables than we thought, but for things very different than tables. We learned that a very significant amount of time the first suggestion in the spelling dictionary was the right correction (hence autocorrect). We learned that no one ever read the tip of the day (“Don’t run with scissors”). This data enabled us to make real decisions about what to fix, the impact of changes, and then when looked at the goals (the resulting documents) what direction to take word processing.
Fast forward to the development of Windows 7 and we’re focused on using data to help inform decisions we make. This data takes many forms and helps in many ways. I know a lot of folks have questions about the data – is it representative, how does it help fix things people should be using but don’t, what about doing new things, and so on. Data is an important element of making decisions, but not a substitute for clear product goals, meaningful customer engagement, and working across the ecosystem to bring Windows 7 to customers.
Let’s talk a bit about “bugs”. Up front it is worth making sure we’re on the same page when we use the much overloaded term bug. For us a bug is any time the software does something that someone one wasn’t expecting it to do. A bug can be a cosmetic issue, a consistency issue, a crash, a hang, a failure to succeed, a confusing user experience, a compatibility issue, a missing feature, or any one of dozens of different ways that the software can behave in a way that isn’t expected. A bug for us is not an emotional term, but just shorthand for an entry in our database representing feedback on the product. Bugs can be reported by a human or by the various forms of telemetry built into Windows 7. This broad definition allows us to track and catalog everything experienced in the product and do so in a uniform manner.
Briefly, it is worth considering a few types of data that help to inform decisions as some examples.
This type of feedback all represents structured feedback in that the data is collected based on a systematic study and usually has a hypothesis associated with it. We also have the unstructured feedback which represents the vast array of bug reports, comments, questions, and points of view expressed in blogs, newsgroups, and the Send Feedback button—these are unstructured because these are not collected in a systematic manner, but aggressively collected by any and all means. A special form of this input is the bug reporting done through the Connect program—the technical beta—which represents bug reports, feature suggestions, and comments from this set of participants.
The Windows 7 beta represents a new level of feedback in this regard in terms of the overall volume as we talked about above. If you go back and consider the size of the development team and the time it would take to just read the reports you can imagine just digesting (categorizing, understanding, flagging) issues let alone responding to them is a massive undertaking (about 40 Send Feedback reports per developer during that one week, though as you can imagine they are not evenly distributed across teams).
The challenge of how to incorporate all the feedback at this stage in the cycle is significant. It is emotional for us at Microsoft and the source of both considerable pride and also some consternation. We often say “no matter what happens, someone always said it would.” By that we mean, on any given issue you can be assured that all sides will be represented by passionate and informed views of how to resolve it, often in direct opposition to each other plus every view in the middle. That means for the vast majority of issues there is no right or wrong in an absolute sense, only a good decision within the context of a given situation. We see this quite a bit in the debates about how features should work—multiple solutions proposed and debate takes place in comments on a blog (people even do whole blogs about how things should work). But ultimately on the Windows development team we have to make a call as we’re seeing a lot of people are looking forward to us finishing Windows 7, which means we need to stop changing the product and ship it. We might not always make the right call and we’ll admit if we don’t make the right call, even if we find changing the behavior is not possible.
Making these decisions is the job of program management (PM). PMs don’t lock themselves in their offices and issue opinions, but more realistically they gather all the facts, data, points of view, and work to synthesize the best approach for a given situation. Program management’s role is making sure all the voices are heard, including beta testers, development, testing, sales, marketing, design, customer support, other teams, ISVs, IHVs, and on and on. Their role is to synthesize and represent these points of view systematically.
There are many factors that go into understanding a given choice:
These are just a few of the factors that go into considering a product change. As you can see, this is not something that we take lightly and a lot goes into each and every change. We consider all the inputs we have and consider all the data we can gather. In some ways it is easy to freeze thinking about the decisions we must make to release Windows 7—if you think too hard about a decision because you might start to worry about a billion people relying on something and it gets very tricky. So we use data to keep ourselves objective and to keep the decision process informed and repeatable. We are always humbled by the responsibility we have.
While writing this post, I received a “bug report” email with the explicit statement “is Microsoft going to side step this issue despite the magnitude of the problem” along with the inevitable “Microsoft never listens to feedback”. Receiving mail like this is tough—we’re in the doghouse before we even start. The sender has decided that this report is symbolic of Microsoft’s inability or lack of desire to incorporate critical feedback and to fix must fix bugs during development. Microsoft is too focused on shipping to do the right thing. I feel like I’m stuck because the only answer being looked for is the fix and anything less is a problem or further proof of our failure. And in the back of my mind is the reality that this is just one person with one issue I just happen to be talking to in email. There over a couple of million people using the beta and if each one, or for that matter just one out of 10, have some unique change, bug fix, or must do work item we would have literally years of work just to make our way through that list. And if you think about the numbers and consider that we might easily get 1,000,000 submitted new “work items” for a product cycle, even if we do 100,000 of them it means we have 900,000 folks who feel we don’t listen compared to the 100,000 folks who feel listened to. Perhaps that puts the challenge in context.
With this post we tried to look at some of the ways we think about the feedback we’re getting and how we evaluate feedback in the course of developing Windows 7. No area is more complex than balancing the needs (and desires) of such a large and diverse population—end-users, developers, IT professionals, hardware makers, PC manufacturers, silicon partners, software vendors, PC enthusiasts, sysadmins, and so on. A key reason we augment our approach with data and studies that deliberately select for representative groups of “users” is that it is important to avoid “tyranny of the majority” or “rule by the crowd”. In a sense, the lesson we learned from adrenaline -based development was that being systematic, representative, and as scientific as possible in the use of data.
The work of acting on feedback responsibly and managing the development of Windows through all phases of the process is something we are very sincere about. Internally we’ve talked a lot about being a learning organization and how we’re always learning how to do a better job, improve the work we do, and in the process work to make Windows even better. We take this approach as individuals and how we view building Windows. We know we will continue to have tough choices to make as everyone who builds products understands and what you have is our commitment to continue to use all the tools available to make sure we are building the best Windows 7 we can build.
@solaris: "File Previews, File Visualisation (Coverflow), Gadgets, Style (the superbar looks like the dock in tiger)"
I do think Windows could use some Mac-like features, like a more consistent user interface, very good bundled apps, or details like making it easy to create screenshots. But if I was to decide, at the moment I'd focus Windows development on speed and reliability, more than fancy previews.
This is post good. But I would like to remind you few obvious things (really without trolling):
You're speaking about complexity, dependencies and similar things and problems connected with it.
Please note, that you made it.
You don't have clear barriers among applications and system, applications can change Registry keys belong to other one, etc. etc.
The faster you will change it, the faster you will be able to decrease number of problems & reports and the faster you will be able to react on market needs. You can't start from symptoms, but you should rather start from roots.
Your system would be built using new rules (they don't have to be totally different - NT architecture good implemented is OK) and it will use some kind of virtualization for existing win32 apps (sandboxing or other form). You can say - it will cost and need time. OK, but - how many USD you're paying employees for resolving current issues ? Many can be avoided.
Of course, some of your apps will have less meaning then. IE for example. But currently - you need to pay employees for creating "N" version and lawyers for depense against other companies.
Maybe it will possible to implement less DRM then too. But once again - many customers are afraid of protections put into WIndows 6.x, many are notifying, that they affect them somehow even when shouldn't
Maybe it's time to change it ?
Of course, you will have your own opinion and I respect it. But once again: your company want to earn money and you should go rather way, which allowed to create windows xp than way, which allowed to created windows vista.
@wtroost -- While no doubt the beta is heavy on tech enthusiasts, with the sample size we have it is more clearly representative of a broad set of customers. We have customers in 4 major languages, almost every country, and all walks to life.
I'm visiting some colleges on a recruiting trip and the flight attendant asked me about my PC (you know which one that is!) and she said she got a new "netbook" and just installed the WIndows 7 beta. The Air Force officer in the seat next to me chimed in and said he has the WIndows 7 beta as well. So we feel pretty good about the breadth of usage and many anecdotes seem to confirm this :-)
Our telemetry measures sequences of feature usage as well and we often spend considerable time understanding the starting point/end point of sequences. So not just quantity.
@marcinw -- it would be really great if you didn't randomly toss in a comment about DRM in every post. It does not add very much to our dialog.
I really try to forget about it and don't write about it, but when I use Vista/Seven : these systems remind me about it too often even I don't anything connected with playing music or watching videos. And my words are words from really annoyed customer (in this topic).
Additionally I know many people, who said - we will not switch from Win 5.x to 6.x because of it (and they're not making something illegal too).
Maybe it will be good to start dialog about it too ?
Please note, that many other companies are resigning from it...
I can't say anything but the best about Windows 7 even tho I am Linux user for past decade. As I always was saying. Windows 7 is an upgrade from Vista just like from ME to XP (remember ME? that thing was I think waist of money).
Steven, you write a lot! Took me so long to read it ;) but the more I read about it the better explanation I got. Sorry I am use to shot blog posts.
You guys have a great year and I hope I'll get the copy of Windows 7 by end of the year.
You know where to find me :)
e stop troll'n roll
@jestempies: I agree with you that reliability and performance should be prioritized. And I also agree that the things I've mentioned aren't the most important aspects of an operating system.
But Microsoft wants to establish Windows as a consumer product, as an exciting product, as something that evokes product lust and that people should buy instead of a mac.
Now imagine an everyday normal customer in a store comparing computers. Why shouldn't he buy a mac? His mind will probably tell him Windows but his emotions tell him to buy the computer that runs the fancy operating system.
I mean it's all there. Gadgets, Previews, Thumbnails, the superbar ... why not make it look great?
Wow. A fantastic post Steven.
I work in Software development so understand the points you're making. The numbers very much say it all.
If there is something wrong in my words, please show it - I will be more than happy, if you will demonstrate mistakes there.
I have never seen a part of your , one post of appreciation.
you do not look only at what you want ,
Windows is a product for all ,
and I can tell you that this beta is greater than the quality any other commercial operating system and not.
If you want a dress made to measure, you must hire a personal tailor.
Great blog, enjoyed reading it.
I'll start this post with word "DRM" since it apparently gets a rise out of you. :) I think it is hard to be in your position but in the business of supporting end users a "bug" is typically a feature that existed in previous versions that was taken away in a newer version. I am not even talking BIG FEATURES. For instance, a security best practice is to logon as a normal user and then elevate your permissions using RunAs but with VISTA and Windows 7 that feature is GONE! However, I can download code from MS and install it but that should not be the case. Also, many XP administrators used to be able to right-clicking Network Connections and then be able to manage network adapters. Sure we can run ncpa.cpl from the RUN command but that is one more thing that must be done to support this new OS. Also your enterprise customers were left in the wind with the decision to remove GINA integration at system startup. I cannot tell you how many complaints I've had now that the Cisco VPN client or even Juniper GINA client cannot be fired up before desktop logon. The answer is NOT "use the Windows VPN client"....I'm sure you get over a million things to "fix" or new "features" but this is a pre-existing feature that NO LONGER EXISTS! I'm not talking about bringing back the classic toolbar, etc, etc. or even eliminating DRM because we all know that is a political and strategic decision Microsoft has lots of capital invested into. Your technical customers are expecting "small issues" to be resolved before this product goes RTM. I'll use your words against you on this one because your goal of Windows 7 was to make access to items they use daily EASIER. Bring back my RUNAS and Manage Network Adapters!!!! Windows 7 has significant improvements over Windows VISTA and I'm actually running it daily without any major issues. I'm sure you've heard the "bugs" with IE8 but that itself is minimal. I think you are doing a great job and while my complaints are minimal I want them to fit in the 10 percent. :) As happy as I am with Windows 7 beta, even after your post it just feels that you all are just fixing broken items without any input regarding small features that bring added value. A good example was your fight regarding UAC and it took serious negative publicity for you to actually fix the feature. Maybe I'm wrong but that is how your community feels...or at least people I work with in the community.
Build 7000 seems to be between XP and Vista, today I have even put it into state (by installing patches or disabling services), where each UAC required action is blocked (you can't click Manage on Computer, install software, you can't change UAC level, etc.) - report about it was sent using "Send feedback".
I would like to remind you, that market seems to adopt Vista slower than previous Windows and there were reasons of it. I'm trying to show sometimes, what and how was done wrong (when there is possibility, that it will be repeated). I'm very sorry, but you haven't showed, what exactly was wrong in my words (we could discuss it then). And I must notify, that I'm not the only one person writing about some disadvantages and areas to improve in current solutions.
As summary, I will say - I like new toys :) When this one will be great, I will write only good words about it :)
I believe one of Apples biggest issues is they communicate HORRIBLY!! They pushed out 200+ MB's worth of patches almost three days apart back in August of 2008 and caused nightmares for security professionals in the industry. If Microsoft did that they would have been BLASTED!!!! Apple is like the "Barack Obama <--just an example" of the IT industry right now and can do no wrong. However, over time and the increase of Apple devices on the market, this too will fade and the hype will be gone. Microsoft is very deliberate and communicates greatly with security and IT professionals via their channels very very well. Every patch with Microsoft is communicated, tested and they work with their community well before it gets released. Apple does NOT do this at all. I have so many examples why Apple is horrible to their user community because they use them as their TEST LAB. With that said Microsoft could learn that they should have saved some "features" for a "wow" factor during release. This is one thing Steve Jobs does well. Until release he has everything very tightly controlled. Also, Microsoft could learn to have a consistent UI and I think W7 is a step in the right direction on that front. But hey lets not forget...telemetry drives Microsoft innovation nowadays. :)