Working the Spoken Word : Speech-to-text

What a year for speech recognition at Microsoft

Stephen Potter — Mon, 31 Dec 2007 23:45:00 GMT

Yeah, yeah, the year in review, what a crushingly unoriginal idea for a post. But wait - this is worth it. 2007 was a huge year for speech recognition products at Microsoft. I think we'll look back on it as a real turning point. Here's how it shaped up.

(Going into the year, Exchange Server 2007 had just shipped with Unified Messaging, including Outlook Voice Access that gives you access over the phone to email, calendar and other useful features. It's a significant integration of speech technology into the heart of a high-volume server product. Huge posters had been up on campus for months, and inside Exchange, they called it 'the sizzle on the steak'.

Meanwhile the teams in Windows, Speech Server, automotive and core technology are hard at work... )

January 2007

Windows Vista ships with Windows Speech Recognition built into the operating system in eight different languages. Now this is a significant investment in the voice user interface as a means of commanding and dictation for desktop users. The entire desktop is speech-enabled under the 'say what you see' metaphor; correction and selection are easy; and the system adapts to your voice and your typical word usage as time goes on. Since the release of WSR, many media reviews have been overwhelmingly positive - check out Rob Chambers' blog (one of the driving forces behind speech in Vista) for links and discussions.

March 2007

Microsoft announces intent to acquire Tellme Networks. Steve Ballmer says it all:

“Speech is universal, simple and holds incredible promise as a key interface for computing. Tellme brings to Microsoft the talent, technology and proven experience in speech that will enable us to deliver a new wave of products and revolutionize human-computer interaction.”

(Incidentally, CNET has a nice inside look at the discussions in Building 34 on Superbowl day between Steve Ballmer and Mike McCue, Tellme CEO, that led up to the deal.)

Also in March - Microsoft Response Point is launched out of Microsoft Research. Response Point is a new way for small business to manage their phone systems - inexpensive, easy to set up and easy to use. All thanks to VoIP and the speech technology that underlies the user interface.

May 2007

The acquisition of TellMe closes.

September 2007

In the mobile space, Tellme announces a deal with Sprint to incorporate Tellme's voice search technology with Live Search into certain phones.

Meanwhile, the first Ford cars hit the market in the USA with Sync - hands-free speech technology for voice dialing, messaging and media control within the car.

October 2007

Office Communications Server 2007 is released as the flagship of Microsoft's Unified Communications strategy. Bundled with OCS 2007 is the latest version of Microsoft Speech Server - now called Office Communications Server 2007 Speech Server (oh yes). It's a significant upgrade from Speech Server 2004, including native VoIP support, graphical dialog editing, conversational grammars, and rich data mining and tuning tools.

And - what a month - Live Search for Windows Mobile goes live with speech recognition. The speech team blog has more details of the kinds of searches possible. And you don't even need a mobile phone to make free 411 calls using the Live Search speech technology. Insider details from Long Zheng's interview with Program Manager Oliver Scholz.

So what's to come in 2008?

Let me say only that we have not been sitting around (well, actually, that's not quite true, I have been sitting around for the last month, since I was out on paternity leave. Only it wasn't really sitting around, there was a lot to do in terms of coping with the newborn's data streams and all that, but I wasn't building software, that's what I meant, now let me rescue my point) - all the teams behind these releases have been planning and executing on the next waves since even before the dates above, so huge momentum has already built in a number of areas, old and new, and we'll start to see evidence of this as the year progresses.

And - did I mention that we're hiring in a number of speech technology-related areas? Please contact me for details if you're interested.

Bored medical student impressed by speech recognition

Stephen Potter — Thu, 16 Aug 2007 04:02:00 GMT

Sullen student, bored by x-rays of terrible chest diseases, is mesmerised by speech recognition, mutters that's so cool.

Can't argue with that.

(And can't resist real stories with a whiff of The Onion.)

Thinking aloud

Stephen Potter — Thu, 02 Aug 2007 04:50:00 GMT

Speech recognition in Vista works well for me. Dictation accuracy is very high - especially since I flicked the switch to train it on my emails and documents - and the correction experience is smooth and efficient. But I hardly use it. I find it very difficult to dictate to my computer.

WTF? Typing is easier than speaking?

Yes. I can't speak the way I write. If I'm typing, I'll begin a sentence without knowing how it's going to end, future phrases will form as I finish typing previous ones. I'll pause, go back, select text, delete it, write again. It feels almost as if hitting the keys is a direct extension of the thought process.

If I'm dictating, there's no such flow. I'll begin a sentence (having got over the minor panic triggered by the Listening microphone UI over a blank document) and the phrase will appear - correctly - on my page, but saying it out loud has corrupted the thoughts that would have helped complete it. I have to step back and think hard about what should come next. Every phrase forces a little mental reboot.

I think what's going on - in addition to my inability to think in complete sentences - is that my thought processes while typing have habituated themselves to the gated speeds of my motor operations. The extra time is valuable and they use it to do the work of sounding out the current phrase in context, thinking up the next phrase, and so on. So typing actually greases the wheels of forming the right words and sentences. That's missing when I dictate, and I find it quite paralyzing.

Surely many keyboard users, even very slow typists, will be reluctant to move to speech recognition systems - even high accuracy systems - because of the advantages of the thinking on-the-side that we seem to do while hitting keys. The standard words-per-minute (wpm) metric for text input is usually measured over copying pre-existing texts. The text creation part is assumed to be equal to each. But my wpm drops dramatically when I dictate, because the creation processes that work with typing are unavailable to me. But what does the wpm look like when you do have them available?

Many users of speech recognition (and of transcription-taking secretaries) have obviously overcome this. Most visibly, Richard Powers, a novelist who wrote the 2006 National Book Award winner using speech recognition on his TabletPC, trained himself after years of typing:

I needed weeks to get over the oddness of auditioning myself in an empty room, to trust to the flow of speech, to learn to hear myself think all over again.

He broke through - and argues now that typing is the obstacle to the thought process:

What could be less conducive to thought’s cadences than stopping every time your short-term memory fills to pass those large-scale musical phrases through your fingers, one tedious letter at a time? You’d be hard-pressed to invent a greater barrier to cognitive flow.

Is the grass really greener over there? I'd love to know. If anyone has done it, please send a comment. I'm also going to try it out for myself. Over the next few weeks, I'll be training myself to think aloud.

The novelist's method

Stephen Potter — Fri, 19 Jan 2007 05:20:00 GMT

IBM engineer and blogger Michael Tolva contacted Richard Powers in disbelief about the dictation of his recent work (see my last post). Here's Mr Powers' reply about his correction UI:

...the combination of stylus and speech is far more fluid and powerful for me than a keyboard ever was. Two taps are usually all it takes to fix a misheard word. Occasionally, I highlight a phrase and speak again. When a word isn’t in the speech lexicon, I handwrite it in. Using handwriting, I can easily change individual letters faster than it would take me to navigate and correct with the arrow keys, backspace, and letter keys. On rare occasions when it’s needed, I can also drop into spelling mode and speak the spelling of a word out loud. As a last resort, for short acronyms, foreign words, etc., I can peck things in with the on-screen keyboard.

Poetry in motion?

Writing with speech

Stephen Potter — Wed, 17 Jan 2007 06:45:00 GMT

Speaking in last Sunday's New York Times, novelist Richard Powers tells how he has used speech recognition on his TabletPC to compose all his writings of recent years.

It's a fine article - you could quote him almost anywhere on the obstacles of the keyboard, on SR speed and accuracy, on getting accustomed to talking to a machine, or on "speakos and mondegreens"... And it says so much about speech interface technology - it would have been inconceivable only 5 years ago.

One factor behind this is the productivity of the TabletPC correction UI:

I speak untethered, without a headset, into the slate’s microphone array. The words appear as fast as I can speak, or they wait out my long pauses. I touch them up with a stylus, scribbling or re-speaking as needed. Whole phrases die and revive, as quickly as I could have hit the backspace.

Stylus vs. Keyboard. Editing is more direct and efficient with the stylus than with mouse and keyboard, given the word-on-word nature of transcript editing. (And assuming Mr Powers is using TabletPC's built-in speech recognition UI - hats off to Rob and team who put it there!)

Having read, years ago, Galatea 2.2, I almost wish this article had gone behind the lines of the human interface and into the SR technology itself - the hum of the decoders, the scatter-charts of the acoustic models, the forced couplings of the words in the language models... If anyone can surface the poetry of a speech recognition engine, it's the writer who brought to life the neural networks of a fictional machine that is taught to listen and (heartbreakingly) to speak, in that book.