Larry Osterman's WebLog

Confessions of an Old Fogey
Blog - Title

Wait, that was MY bug? Ouch!

Wait, that was MY bug? Ouch!

Over the weekend, the wires were full with reports of a speech recognition demo at the Microsoft's Financial Analysts Meeting here in Seattle that went horribly wrong. 

Slashdot had it, Neowin had it,  Digg had it, Reuters had it.  It was everywhere.

And it was all my fault.

 

Well, mostly.  Rob Chambers on the speech team has already written about this, here's the same problem from my side of the fence.

About a month ago (more-or-less), we got some reports from an IHV that sometimes when they set the volume on a capture stream the actual volume would go crazy (crazy, for those that don't know, is a technical term).  Since volume is one of the areas in the audio subsystem that I own, the bug landed on my plate.  At the time, I was overloaded with bugs, so another of the developers on the audio team took over the investigation and root caused the bug fairly quickly.  The annoying thing about it was that the bug wasn't reproducible - every time he stepped through the code in the debugger, it worked perfectly, but it kept failing when run without any traces.

 

If you've worked with analog audio, it's pretty clear what's happening here - there's a timing issue that is causing a positive feedback loop that resulted from a signal being fed back into an amplifier.

It turns out that one of the common causes of feedback loops in software is a concurrency issue with notifications - a notification is received with new data, which updates a value, updating the value causes a new notification to be generated, which updates a value, updating the value causes a new notification, and so-on...

The code actually handled most of the feedback cases involving notifications, but there were two lower level bugs that complicated things.  The first bug was that there was an incorrect calculation that occurred when handling one of the values in the notification, and the second was that there was a concurrency issue - a member variable that should have been protected wasn't (I'm simplifying what actually happened, but this suffices). 

 

As a consequence of these two very subtle low level bugs, the speech recognition engine wasn't able to correctly control the gain on the microphone, when it did, it hit the notification feedback loop, which caused the microphone to clip, which meant that the samples being received by the speech recognition engine weren't accurate.

There were other contributing factors to the problem (the bug was fixed on more recent Vista builds than the one they were using for the demo, there were some issues with way the speech recognition engine had been "trained", etc), but it doesn't matter - the problem wouldn't have been nearly as significant.

Mea Culpa.

  • Norman "Mommy Never Loved Me" Diamond said:
    >> and root caused the bug fairly quickly.
    >That's what you get for running as root.  If you ran as a limited user then you wouldn't cause bugs.

    Norm, you are a douche.
  • Speech is not that bad in Vista.  My big issue is that when typing documents in MSFT Word, the Office 2003 Speech does a better job.

    Larry thankyou for keeping us up to date.
  • Your candor is refreshing and as a non-microsoftie investor I appreciate it very much. Just take extra care before you ship the product out. Good luck.
  • Larry, I think you're allowed to hold your hand up and say that one was mine. Most people wouldn't have the courage to do that. But it's not you fault some apeth decides to demo it to investors.
  • PingBack from http://www.carnatic.com/mokshore/2006/08/02/links-for-2006-08-02/
  • Tuesday, August 01, 2006 10:52 AM by Lon Ingram
    > Norman "Mommy Never Loved Me" Diamond said:
    >> and root caused the bug fairly quickly.
    >That's what you get for running as root.  If you ran as a
    >limited user then you wouldn't cause bugs.

    Close.  It's "grammar never loved me".  Sorry my pun had too many layers for you.

    Interesting that you have no comment on the way Microsoft treats more widespread bugs.  Though oddly, after I posted that comment, it seems Microsoft decided that they really will let me see some of my bug reports and their requests for further information.  I hope to try it out this weekend.  Of course even this much is only being allowed because Vista's bugs aren't yet being sold as a product.
  • I was present at the meeting and I didn't really think the 'glitch' was as bad as the media have led everyone to believe. I made a blog post about it <a href="http://blogs.3sharp.com/Blog/dougv/archive/2006/08/01/1672.aspx">here</a>.
  • Larry and Rob blog postings about the Vista Speech Recognition bug is a great indicator on how far Microsoft has come from the ‘’Borg’’ days.
    As a partner dealing with Microsoft, it gives me a lot of ...
  • This is a joke. Microsoft speech team lost all of its stars.
  • PingBack from http://www.cortell.net/2006/08/02/siete-anos-despues-microsoft-sigue-sin-conseguirlo/
  • The tipical excuse is say that the bug was solved in a early version but not in the demo version. To solve this there is a very good free product that is SUBVERSION and it's works very well, simply with tagging one revision
  • I'm glad to see individuals at MSFT own up to problems and suggest the approp. measures to address them.  Larry, thanks for your candid commentary on the issue and best of luck during the next demo!
  • Thank you larry for your explanation. You are clear and sincere as ever. Is your blog that made me dreaming of working on OSes, and even post like this one are a push forward to achieve my goal in the future, and study and work even more hard. Errors and bugs happens even to the greatest ones, the important thing is to not be disheartened and go forward!
  • > The annoying thing about it was that the
    > bug wasn't reproducible - every time he
    > stepped through the code in the debugger,
    > it worked perfectly, but it kept failing when
    > run without any traces.

    In the unlikely event you didn't know, the technical term for this is a 'Heisenbug'. See http://en.wikipedia.org/wiki/Heisenbug#Heisenbugs.

  • PingBack from http://www.werbeblogger.de/2006/08/02/mein-pc-hort-aufs-wort/
Page 3 of 7 (100 items) 12345»