Larry Osterman's WebLog

Confessions of an Old Fogey
Blog - Title

What's the big deal with Vista Betas?

What's the big deal with Vista Betas?

  • Comments 16
In this somewhat silly Channel9 post about unsubstantiated rumors of Microsoft canceling Vista Beta 2, "Sabot" wrote the following question:

What's all the fuss about a Beta? I don't actually get it? It comes when it's ready and I'm happy with that.

I'm one of those guys that's not really all that fussed about knowing the release date of any product, I always architect systems using existing technology, it's a school-boy error to do otherwise because the plain truth is, there is a 50/50 risk it won't be released by the time you need it in the project plan, and that in terms of risk is to high a percentage to be acceptable

As a consumer of Microsoft's operating systems, I'm 100% behind him.  I don't get the hype (although I think seeing the purported screen shots is cool).

As a developer who is working on Microsoft's operating systems, I have a rather different view.

You see, the betas of Microsoft's operating systems are really the only opportunity that Microsoft's developers have to see the "real world" outside the ivory towers of Microsoft.

Within Microsoft, you see, we all tend to have more-or-less the same types of machines.  They come from maybe a dozen different manufacturers, and they all have reasonably different capabilities (for example, many don't have any audio).  But they tend to be pretty homogenous - 1.5 to 3 GHz machines with between 256M and 2G of RAM, an "ok" video card, and an on-the-motherboard audio and network solutions.  The thing is, they don't even come close to representing the diversity of systems that our customers have.  We typically don't have the $200 low-end computers in our office, neither do we have $7,000 watercooled ultra high-end gaming rigs.

We also don't have all the myriad applications that our customers have.  We've got an amazingly huge library of apps to test (I know, I've been looking at appcompat failures), but we still get reports of problems in apps that we've never heard of.  We also don't have the applications that haven't yet shipped, so we have no ability to test them.

As a result, the OS developers within Microsoft LOVE betas.  We love it when real customers get a chance to use our stuff, because their environments are so much more diverse than any we could possibly have.  In reality, we do a pretty darned good job of weeding out the bugs before we ship (for instance, we've only found a handful of new customer reported audio bugs since Beta1), but there's no substitute for honest-to-goodness in-the-field tests.

It's kinda interesting - Platform software is very different from just about every other engineering endeavor, because the compatibility matrix is SO complicated.  You can build an automobile totally in isolation - you can solve every engineering problem you're likely to ever find before it leaves the plant, because you can pretty accurately gauge how your customers are going to use your product.  But you can't do that with a software platform, because the number of potential use scenarios is essentially limitless.  So for software platforms, betas are indispensable tools that provide critical feedback for the developers of the platform.

  • Very interesting article. I just have to say that your automotive analogy is not quire right. From outsite the industry, it may look that "simple," however, consider that you can go down to the dealership and order a pickup truck with whatever options you want, and that causes the factory to have to deal with > 10,000 possible combinations - all of which need to be designed to work well together.

    But it is true, however, that there is no responsibility on the part of the automotive OEM to support in-field updates/upgrades to the vehicle configuration, so from that perspective, it is simpler than what Windows faces, but it still overall is not "simple."
  • How has virtualization impacted the feedback loop? I don't install anything beta on my 'real hardware' anymore. It is all VPC's. Has this skewed the amount/quality of compatibility feedback?
  • I love new toys to play around with, and even if I'm not convinced Vista will be right for me I still have to try it to be sure.

    I don't like the toys as a developer for the most part, but just as a day to day user that spends way too much time on their computer. :)
  • If Microsoft paid a nominal bug bounty, perhaps betas would be more effective?
  • I agree with Manip, actually. As a developer at work, we only ever consider installing new software after it's been out for a few months at least (even better is after the first service pack comes out :p~)

    But on my laptop at home, it's got betas of anything I can lay my hands on! It's always good to stay ahead of the curve :)
  • Maurits:
    Can you imagine how much of a mess that would be?

    Consider:
    Customer 1 might complain that component ABC fails when they frobnicate the controls in some window. Customer 2 might complain that component XYZ fails when they click three times too fast on the start button.

    From a developers perspective, the root cause of the problem may come down to one bug, but from both customer perspectives it's two different bugs. Who gets the bounty? The first one that reports a symptom? Both?

    Now multiple customer 1 and customer 2 by 100,000 reports. It's a recipe for a lot of pissed off beta testers or a huge cash hemorrhage.

  • It just proves my argument that building software is a very hard beast to contain. It is my hypothesis that software building is hard is because we as humans are a visual animal, by that I mean that if we are building a physical structure, be it a house, bridge etc, during this build if something is going belly up, most times we have ability to correct this before it is completed. (Having said this, there has been some disasters in the past and shall continue in the future.)

    Now, when it comes to software building, we can only see the results (ie does the software/system match the requirements that was intended) once the application has been built – That is why it I agree in releasing Beta’s.

    What would be ideal is during the application/system build process a “build monitor” is positioned between the compiler and the requirements that check what is getting built matches to what has been specified in the requirement parameters for the application/system.
  • Dazhel: I was thinking the bounty amount should be at the developer's discretion, actually
  • I think the car analogy is a bit flawed because MS has ninety-odd percent of the operating system market, and ships a new OS version only every few years. There are a lot of commercially significant car companies, issuing in total dozens of new models every year, some of which people simply don't buy. Or buy and then hate. What MS has to do is more like packing three years' worth of new models of Accords, Corvettes, BMWs, and Prion^H^H^H^H^HPriuses into a single release. But they've probably got a couple of orders of magnitude more people on the project, too.

    The car companies' problem isn't trivial; if it were, GM would still be viable.
  • "What would be ideal is during the application/system build process a “build monitor” is positioned between the compiler and the requirements that check what is getting built matches to what has been specified in the requirement parameters for the application/system."

    Robert: Isn't that what software testers do?

    You're talking about an extremely complex problem for computers to solve. If you're going to write a program that eliminates software testers by checking a build against the requirements, why not just go the whole nine yards and write a program eliminates software developers as well by writing a program that takes the requirements and creates the actual program? :)
  • Maurits:
    Speaking as a developer myself, we already work our rear ends off prepping a product for release. I think I'd speak for a lot of developers when I say that the last thing I want to have draining on my time is mediating disputes over who gets what bug bounty.
  • > You see, the betas of Microsoft's operating
    > systems are really the only opportunity that
    > Microsoft's developers have to see the "real
    > world" outside the ivory towers of Microsoft.

    Which means that Microsoft's developers STILL never have an opportunity to see the real world outside the slightly extended ivory towers of Microsoft plus beta testers.

    Microsoft doesn't need to pay a bounty for bugs. Microsoft only needs to stop CHARGING a bounty, accept bug reports from the real world, and allow victims to get the fixes.
  • Dazhel,

    I think you meant, Richard. :)

    Yes I agree, that's what software testers do. I think you missed what I wrote. What I was attempting to get across was to provide a QA step before sending the build to testers, thus increasing the hit rate of a successful system/application the first time. Have you ever heard of “measure twice cut once”?

    However, I think it would be good to have this feature as an additional step. What's wrong with this? It was not my intention to say to get rid of software testers/ users and programmers on the contrary, all of these positions are vital in shipping working software.

    Also, the vast majority of time, I have limited resources in terms of testers, they aren’t dedicated testers, they have to fit in testing around their other duties. I work as a contractor and my clients don’t have the vast resources as the likes of Microsoft and other companies that have pools of cash.

    Yes, for this tool to be developed would be quite a challenge to build and to implement. But what’s wrong proposing this idea? After all, this would be a weapon in the programmer’s arsenal. :)

    Richard
  • Larry,

    I totally agree, but I have a few critical points. How can Microsoft ever expect to get "real world" testing on a diversity of machines unless they stop being so tight with the beta test. Once upon a time you had to practically sign over your first born to get to be an os beta tester. Obviously, things have gotten much better, but still I think the hurdles required to get in skew the demographics heavily in favor of OEMs/Coporate/IT professionals. Given the ubiquity of Windows operating systems these days, it is just not an accurate reflection of the userbase to still operate in this manner. You know, there are individuals who can't even drive yet who are bigger Windows enthusiasts then some people I know who've been doing IT for decades. I remember I was only 14 when Chicago betas rolled out (it seems like it was only yesterday). I had a friend who got a hold of one because his dad had whatever was equivilant to MSDN universal back then, and boy was I jealous!

    Anywho, I think what I'm trying to say is that you should actively allow anyone who is genuinely interested the ability to participate, regardless of their "place" in the food chain. I want to applaud Microsoft for the openess you exhibited with the Whidbey betas, something which seemed to be very beneficial for all sides involved. I hope that some consideration might be given to applying this same model to your other mainstream products. I would only encourage you to be more bold, and not just restrict the weekly bleeding-edge drops to the big-bucks MSDN customers. There are pleanty of us out there who can't afford to buy MSDN universal subscriptions who nonetheless would like to participate on the bleeding edge and provide useful feedback.
  • Nick, that happens eventually. But we've learned from VERY hard experience that you can't open wide betas early - look at the number of people who put the Longhorn Alpha code on their day-to-day machines.
Page 1 of 2 (16 items) 12