Larry Osterman's WebLog

Confessions of an Old Fogey
Blog - Title

Wait, that was MY bug? Ouch!

Wait, that was MY bug? Ouch!

Over the weekend, the wires were full with reports of a speech recognition demo at the Microsoft's Financial Analysts Meeting here in Seattle that went horribly wrong. 

Slashdot had it, Neowin had it,  Digg had it, Reuters had it.  It was everywhere.

And it was all my fault.

 

Well, mostly.  Rob Chambers on the speech team has already written about this, here's the same problem from my side of the fence.

About a month ago (more-or-less), we got some reports from an IHV that sometimes when they set the volume on a capture stream the actual volume would go crazy (crazy, for those that don't know, is a technical term).  Since volume is one of the areas in the audio subsystem that I own, the bug landed on my plate.  At the time, I was overloaded with bugs, so another of the developers on the audio team took over the investigation and root caused the bug fairly quickly.  The annoying thing about it was that the bug wasn't reproducible - every time he stepped through the code in the debugger, it worked perfectly, but it kept failing when run without any traces.

 

If you've worked with analog audio, it's pretty clear what's happening here - there's a timing issue that is causing a positive feedback loop that resulted from a signal being fed back into an amplifier.

It turns out that one of the common causes of feedback loops in software is a concurrency issue with notifications - a notification is received with new data, which updates a value, updating the value causes a new notification to be generated, which updates a value, updating the value causes a new notification, and so-on...

The code actually handled most of the feedback cases involving notifications, but there were two lower level bugs that complicated things.  The first bug was that there was an incorrect calculation that occurred when handling one of the values in the notification, and the second was that there was a concurrency issue - a member variable that should have been protected wasn't (I'm simplifying what actually happened, but this suffices). 

 

As a consequence of these two very subtle low level bugs, the speech recognition engine wasn't able to correctly control the gain on the microphone, when it did, it hit the notification feedback loop, which caused the microphone to clip, which meant that the samples being received by the speech recognition engine weren't accurate.

There were other contributing factors to the problem (the bug was fixed on more recent Vista builds than the one they were using for the demo, there were some issues with way the speech recognition engine had been "trained", etc), but it doesn't matter - the problem wouldn't have been nearly as significant.

Mea Culpa.

  • Larry, a long time Microsoft blogger, and key member of the audio team, speaks out about the root cause...
  • May I repeat a never answered question I put in Larry's blog:

    Note that the "delete" and "select all" commands are perfectly recognized and written down (but not executed as wanted). If "all the audio data that was being received by WSR was being clipped, and thus was incredibly distorted", how does it come that easy words are distorted while longer ones are perfectly understood? And what about commands not being executed but written down, is it also an audio gain issue???
  • notquitesure: Beats me, it's tied up in DSP and how those algorithms work.  The SPL for words like "delete" and "select all" may be higher than the ones for the other words.
  • PingBack from http://blogs.reuters.com/2006/07/28/when-good-demos-go-very-very-bad/
  • Why did the presenter not rehearse his demo ten times before he went up to make sure everything was working?  Even if you have identified the bug now, I don't see how a presenter could not have tested the software before he went up.
  • Bob, see Rob Chamber's post - they did try it multiple times.  This problem is subtle and timing related, it's hideously unfortunate that the one time it showed up was the one time it mattered the most.

    Murphy's law at its finest.
  • I'm surprised there's no indication or warning that oversteering/clipping is occurring. If it has such detrimental effects, especially with analysts and TV crews present :), surely it'd be useful to notify users about it?
  • We've all been there.  Nothing like watching your app crash and burn on the public stage.  You seem to be taking it well though.  Keep your chin up.

    -Chris
  • PingBack from http://crazyfactor.com/2006/07/31/larry-says-%e2%80%9cit%e2%80%99s-my-bug%e2%80%9d/
  • Thanks for the good laugh.

    I wouldn't worry too much about it. It has happened to Windows, too, and it's still the most popular OS.
  • You dog, I got some advice for you...

    Get a Mac! They got some ill voice recognition that I use at the crib in Oak-town. I walk in an say "Bitch, fix me a chicken pot pie!" and the lights come up and the stereo starts playing my Wu-Tang.

    That shit is straight baller son. Believe that!
  • PingBack from http://blogs.reuters.com/2006/07/31/microsoft-recognizes-speech-software-bug/
  • Via Scoble: Wait, that was MY bug? Ouch! Microsoft's Larry Osterman steps up and claims responsibility for the code that botched the demo I posted on Saturday. Here's something new and interesting from Microsoft. Someone stepping up and saying that the
  • Here's the correct digg link (not that it matters, though)...

    http://digg.com/tech_news/When_good_demos_go_very_very_bad
  • Let’s set Wagged PR response: “ambient noise”, so double delete the killer bug reports, select all “silent room”, double the.
Page 1 of 7 (100 items) 12345»