Larry Osterman's WebLog

Confessions of an Old Fogey
Blog - Title

Wait, that was MY bug? Ouch!

Wait, that was MY bug? Ouch!

Over the weekend, the wires were full with reports of a speech recognition demo at the Microsoft's Financial Analysts Meeting here in Seattle that went horribly wrong. 

Slashdot had it, Neowin had it,  Digg had it, Reuters had it.  It was everywhere.

And it was all my fault.

 

Well, mostly.  Rob Chambers on the speech team has already written about this, here's the same problem from my side of the fence.

About a month ago (more-or-less), we got some reports from an IHV that sometimes when they set the volume on a capture stream the actual volume would go crazy (crazy, for those that don't know, is a technical term).  Since volume is one of the areas in the audio subsystem that I own, the bug landed on my plate.  At the time, I was overloaded with bugs, so another of the developers on the audio team took over the investigation and root caused the bug fairly quickly.  The annoying thing about it was that the bug wasn't reproducible - every time he stepped through the code in the debugger, it worked perfectly, but it kept failing when run without any traces.

 

If you've worked with analog audio, it's pretty clear what's happening here - there's a timing issue that is causing a positive feedback loop that resulted from a signal being fed back into an amplifier.

It turns out that one of the common causes of feedback loops in software is a concurrency issue with notifications - a notification is received with new data, which updates a value, updating the value causes a new notification to be generated, which updates a value, updating the value causes a new notification, and so-on...

The code actually handled most of the feedback cases involving notifications, but there were two lower level bugs that complicated things.  The first bug was that there was an incorrect calculation that occurred when handling one of the values in the notification, and the second was that there was a concurrency issue - a member variable that should have been protected wasn't (I'm simplifying what actually happened, but this suffices). 

 

As a consequence of these two very subtle low level bugs, the speech recognition engine wasn't able to correctly control the gain on the microphone, when it did, it hit the notification feedback loop, which caused the microphone to clip, which meant that the samples being received by the speech recognition engine weren't accurate.

There were other contributing factors to the problem (the bug was fixed on more recent Vista builds than the one they were using for the demo, there were some issues with way the speech recognition engine had been "trained", etc), but it doesn't matter - the problem wouldn't have been nearly as significant.

Mea Culpa.

  • Don't you think you should retire or something? I think there are many skilled 20 years old programmers who could make that work in 5 minutes?

    Man you are getting too old..
  • When next public demonstration takes place, I hope the presenter starts "Dear aunt.."
  • someone forgot some unittest cases at a lower level :-)
  • furroy, no, the unit tests didn't catch this one, because that's not what unit tests are supposed to catch.

    It WAS caught by the FVT (feature verification tests), but it was intermittent even for them.
  • Well, sometime it also sounds like it was done intentionally? just to catch more customer attention. Microsoft was sure it will be hosted on videos.google.com.! Viral Marketing.!

    Wait - till they do a demo again - with everything - up and running. ! It will have more positive affect!

    Ah! BTW - I am not Microsoft fan, But I am indeed a marketing fan!
  • bmbm
  • I know that bug since 3 day but I did want say.
    If you cong that error really stupid, protent any conference.
    I do'nt love Microsoft but I use really Microsoft Productions.
    We are wait big antle Vista, if we'll wait a long time, dont working at the Vista.

    If you are ask my cong, you are very houldying stupid.
    I's speech properties, wait vista at the properties.

    Microsoft is wur00z.
    I kiss you everybody.
  • And now it is on the UserFriendly comic strip:

    http://ars.userfriendly.org/cartoons/?id=20060805&mode=classic

    Keep going Larry, and don't get intimidated by some of the negative comments. I am sure those guys haven't worked on any big projects.
  • >> The annoying thing about it was that the bug wasn't
    >> reproducible

    > So just mark it as resolved, not reproducible, and close it.  
    > That's what Microsoft does with lots of my Vista beta bug reports.

    When I was a tester at Microsoft (Yay, MacBU!), a bug of this severity would have been call a 'Pri 2' - the app didn't actually crash, but the user couldn't precede with their planned tasks. In most cases, Pri 2 bugs are accorded 'Must Fix' status unless the repro steps are deemed quite unlikely.

    I can accept that the test and/or dev teams might not have had time to actually chase down a known bug prior to the demo, but I do have to wonder why the demo team went ahead with that bug lurking in the background.

    Better to forego a demo and risk talk of slow development than to have a demo fail in an embarassing manner and create discussion of quality issues.

    For those who've suggested that Microsoft adopt the 'Apple Way' for demos, here are the two main reasons one rarely sees a glitch in Apple demos:

    1) Since Apple never reveals its development plans ahead of time, there's no pressure to demo half-baked software (except to Steve Jobs himself).

    2) Conversely, product teams are under intense pressure to clear up high-priority bugs as early as possible in the development cycle so that their app gets OK'd to be publicly demo'd.

    Microsoft could do with a bit *less* talk about its future plans and a good deal more focus on planning for excellence at Version 1 instead of at Version 3.
  • こんな悲惨なデモを、僕は見たことがありません。 これは、マイクロソフトの社員が、Windows Vistaの音声認識のデモを行ったときの様子です。 間違えまくっています。ひとつとして正しく認識できていません。 やればやるほど事態がひどくなっていきます。一方的に殴られ続けるボクサーのようです。 このひどいデモを生み出したエンジニアのサイトには、善意のひとびとからなる大量の情報が送り込まれ、大変なことになっているそうです。 Fun with Vista's speech recognition [Guardian
  • PingBack from http://www.istartedsomething.com/20060807/msnbc-deceived-the-public-vistas-speech-recognition/
  • Saturday, August 05, 2006 7:09 PM by Will Parker
    > When I was a tester at Microsoft (Yay, MacBU!), a bug of
    > this severity would have been call a 'Pri 2' - the app didn't
    > actually crash, but the user couldn't precede with their
    > planned tasks. In most cases, Pri 2 bugs are accorded 'Must
    > Fix' status unless the repro steps are deemed quite unlikely.

    Wow, Microsoft's ratings of "must fix" vs. "won't fix" are still astounding.  A bug in the DDK compiler's code generation a "won't release a hotfix", a different and older bug of severity that destroys entire hard disk partitions is a "won't fix", but a bug in speech recognition is a "must fix".  When a DDK Even my previous level of cynicism was inadequate.

    When the DDK's compiler causes a BSOD every app crashes and the user can't proceed with their tasks.  When an entire hard disk partition gets corrupted the user can proceed with their tasks a day later by restoring all files from another backup, unless the backup was on a partition created by the same FDISK program.  Therefore these rank far below such a critical app as speech recognition.

    Thank you very much for this information though.  Sad as the facts are, it's better to know them than not to know them.
  • You've do doubt heard about the Microsoft Financial Analyst Meeting two weeks ago where a demonstration...
Page 4 of 7 (100 items) «23456»