TSF Aware : dictation

What’s new in Windows Speech Recognition?

Eric Brown — Thu, 29 Jan 2009 18:00:00 GMT

Now that the Beta of Windows 7 is out, it’s time to talk about the improvements and new features in Windows Speech Recognition.

For Windows 7, we focused primarily on improving the user experience and removing the “rough spots” that we did not have time to fix in Vista.

First and foremost, we focused on performance.

We rewrote the logic that builds the “say what you see” grammar to use the new native UI Automation API (instead of the MSAA IAccessible API). This dramatically reduces the number of cross-process COM calls (by an order of magnitude), and speeds up the grammar generation by about 5-6 times.
The document harvester also has substantial performance improvements;
Building the “start application” grammar also runs much faster, as well.

Second, we focused on usability.

Dictation into TSF-unaware applications works much better than it did before. Now, when you dictate into an unaware application, the dictation scratchpad appears.
- You can use the scratchpad as a temporary document, and it is even voice, mouse, and keyboard-enabled; you can type, use the arrow keys for navigation, or use the mouse or voice commands to select and correct text before inserting the finished text into the unaware application.
- If you don’t like the scratchpad, you can turn it off, and your dictations will be directly inserted into the unaware application.
Sleep mode works much better than it did before; false recognitions of “start listening” have been greatly reduced.
We simplified the transitions between OFF and SLEEP mode; for security reasons, we now default to OFF after “stop listening”; although users can change the default to SLEEP mode. (We call this “voice activation” in the Control Panel and First Time User Experience.)

Third, we looked at accuracy.

The SR engine now uses the WASAPI audio stack, so we support array microphones and echo cancellation; this vastly improves WSR’s accuracy when used without a headset.
Document harvesting runs periodically, rather than just at startup; this lets the harvester pick up new documents as you create them, rather than having to wait for you to reload speech.
You can upload your training data to Microsoft, so that we can improve the recognizers in the future. (You have to initiate this, incidentally; we will not upload any data without your explicit consent.)
The Chinese recognizer has substantial accuracy improvements as well.

Lastly, we did a few tweaks to the recognizer. In Vista, 3rd party applications couldn’t tell whether the shared recognizer was ON or SLEEPing. For Win7, there are new APIs that expose SLEEP mode.

Inline Dictation commands

Eric Brown — Wed, 27 Aug 2008 02:15:20 GMT

I’ve heard from a number of sources that there isn’t any good documentation about the ‘inline’ dictation commands. These commands can be uttered in the middle of a dictation stream (in other words, you don’t have to stop speaking to use these commands), and are used to guide the Speech Recognition Engine to produce the desired results.

Command	Description
tab	Inserts a <tab> character.
new-line	Inserts a new line character and forces the next word to be capitalized.
new-paragraph	Inserts two new line characters and forces the next word to be capitalized.
caps <word>	Capitalizes the first letter of <word>.
no caps <word>	Lowercases <word>.
all caps <word>	Uppercases <word>.
no space	Does not insert a space before the next word.
literal <argument>	Does not perform any inverse text normalization on <argument>.
numeral <argument>	Forces <argument> into numeric form, if possible.

Simple Examples

Tab, new-line, new-paragraph, caps, no caps, and all caps are pretty straightforward:

User Input	Recognition Result
king tab county	king<tab>county
this is a test new-line this is another test	This is a test This is another test
this is a test new-paragraph this is another test	This is a test This is another test
no caps C I A	cia
C I A	CIA
I have caps in my closet	I have In my closet
I have all caps nothing	I have NOTHING
I have no space available	I haveavailable

Examples using <literal>

In order to understand what <literal> does, you need to know what Inverse Text Normalization does. Inverse Text Normalization is the process of converting spoken forms into a preferred textual representation. Some examples are:

User Input	Recognition Result
doctor smith	Dr. Smith
period	.
united states of america	United States of America
three hundred and five	305

The <literal> command prevents that conversion from occurring, and forces the text to be input exactly as spoken.

Examples using <numeral>

The best way to describe the <numeral> command is to give some examples:

User Input	Recognition Result
numeral two	2
two	two
numeral twelve	12
twelve	12
numeral four eight seven six two three	487623
four eight seven six two three	four eight seven six two three
numeral one and a half million	1,500,000
numeral one point five million	1.5 million
numeral one point four five	1.45

If the phrase cannot be interpreted as a number, then the numeral command does nothing:

User Input	recognition result
numeral I want to go to the store	I want to go to the store

Note that this phrase has a possible number (to –> two –> 2), but the intervening words deactivate the command.

When Dictation doesn't Work

Eric Brown — Fri, 23 May 2008 02:17:40 GMT

WSR Dictation should always work in Wordpad. If you're having problems with dictation, make sure it works in Wordpad.

If dictation doesn’t work in WordPad:

1) Start regedit, go to the key HKEY_CURRENT_USER\Software\Microsoft\Speech\Preferences\en-us\ and remove any value named DictationEnabled.

If there is no registry key named DictationEnabled, then

2) Check %windir%\ime\sptip.dll - it should exist, be 128KB in size, and have a file version of 6.0.6001.17128 and a product version of 6.0.6000.16386.

3) The file %windir%\ime\en-us\sptip.dll.mui should also exist.

4) Make sure ctfmon.exe is running. It should start at login (there is a scheduled task that starts it).

If dictation works in WordPad, but doesn't work in your application:

Your application probably doesn't support TSF. Complain to your vendor. You can enable a (very limited) version of dictation by checking 'Enable Dictation Everywhere' in the Windows Speech Recognition context menu.

Oops!

Eric Brown — Wed, 14 Nov 2007 21:01:48 GMT

If you've tried to use the modified version of Scintilla that I described in my MSDN article, you will find that the zipped sources don't actually have the changes that I made. That was my fault; when I was packaging the sources, I had two versions of ScintillaWin.cxx around, and I picked the newer one, which (sigh) was the wrong one. Anyway, here's a link to the correct version of ScintillaWin.cxx with TSF support.

The usual path

Eric Brown — Mon, 08 Oct 2007 17:00:01 GMT

Text Services Framework assumes that your text service follows a particular processing path. If your text service doesn't conform to these assumptions, then your programming job will be more complicated. (Not impossible, just more complex.) The text service samples on MSDN also follow these assumptions, but they aren't explicitly stated anywhere (that I know of). I've mentioned some of these assumptions in previous articles, but I thought I'd bring them together in one post.

Text Services makes the following assumptions:

Your service must perform all changes to a context or range object within an edit session. Text Services Framework enforces this assumption through the use of edit cookies.
Your service should not assume that it is possible to request a synchronous edit session. (I discussed this here.)
Your service should track focus changes between applications and between controls within an application. This means that your text service must install event sinks for ITfThreadFocusSink and ITfThreadMgrEventSink.
Your text service should use compositions to handle partially formed input.

This last assumption is the big one. It can cause problems for text services that aren't keyboard-related (speech, for example).

The problem is that TSF handles the (admittedly, very difficult) job of interacting with non-TSF aware applications entirely through compositions. Once you close the composition, TSF assumes that you're completely finished with that piece of input.

Unfortunately, it's hard to tell beforehand when you're done with a piece of dictation. SAPI will tell you when it's recognized a piece of text, obviously, but, ideally, once you've dictated some text, you would like to be able to correct it. That requires that you leave the composition open.

In an application that isn't TSF-aware, though, you need to close that composition as soon as you can (it's bad form to have large open compositions; most IMEs have compositions that are a few characters in size).

So there's a tradeoff here. Dictation in Windows Vista currently closes the composition as soon as the text is recognized. (In fact, it doesn't use compositions at all.) That works fine for TSF-aware applications, but causes problems with TSF-unaware applications. In particular, once you've dictated some text, you can't correct it by voice. That's why Windows Speech recognition makes you confirm every dictation into a TSF-unaware application.

How do I use RichEdit 4.1?

Eric Brown — Thu, 14 Jun 2007 21:18:56 GMT

Another thing I didn't talk about in my article was how to make sure your rich text edit controls are based on RichEdit 4.1 (which has TSF support).

You need to do two things:

1) change your window class name from RICHEDIT_CLASS to MSFTEDIT_CLASS, and

2) LoadLibrary("msftedit.dll") instead of LoadLibrary("richedit.dll").

That's it!

Easy Dictation support for Windowless RichEdit controls

Eric Brown — Tue, 12 Jun 2007 17:00:00 GMT

In my article in MSDN, I mention that there are some easy ways to enable dictation support in controls that don't normally support dictation. All the methods I described assume that the control has a window. There is a way to enable dictation support for windowless rich text edit controls, assuming that you are using the RichEdit 4.1 implementation of ITextServices.

I did not include this method in my article because I didn't want to include all the details about ITextServices, how you get an instance, and so forth. I assume that if you already have an existing windowless rich edit control that uses ITextServices, you already know all that. (If you don't, there's a quick primer available at CodeProject; note that I haven't actually used this code, and cannot vouch for it.)

Anyway, if you have an ITextServices interface pointer, then you can enable TSF support (and, by extension, dictation support), by calling ITextServices::TxSendMessage(EM_SETEDITSTYLE, SES_USECTF, SES_USECTF).

I hope that more richedit users will enable TSF support, so that users will have an easier time using dictation.

Generating Candidates from an Application

Eric Brown — Fri, 25 May 2007 21:43:00 GMT

Kirby left a comment to my post on candidates asking when a text service should create an ITfCandidateList.

The answer is that when the text service wants to show candidates (via a preserved key or other mechanism), it should show its modal UI, and quite possibly may wish to push a new context so that the modal UI can be keyboard enabled (the sample does this).

The thing to note here is that ITfCandidateList is not involved here.

So when does the text service use ITfCandidateList? When the application wants to show candidates.

How does this work? Well, it's pretty complicated, but I'll try to outline the steps here. (Get ready, it's going to get a little wild...)

The big blocks

1) Application gets ITfFnReconversion interface.

2) Application gets target range (ITextStoreACPServices::CreateRange comes in very handy here).

3) Application calls ITfFnReconversion::QueryRange to transform the target range.

4) Application calls ITfFnReconversion::GetReconversion to get the list of candidates.

5) Application displays candidate list (e.g., in context menu, dialog, what have you).

6) Application calls ITfCandidateList::SetResult() when the selection is selected.

7) Application releases candidate list, reconversion interface, and does other cleanup.

Alternatively, if the application wants to let the text service manage the UI, it can call ITfFnReconversion::Reconvert at step 4 and step out of the way. (Note that not all text services implement this, as this function must not wait for the UI to be dismissed before returning.)

An excellent way to test this is with Microsoft Word. Word supports reconversion, and will show the candidates in the context menu when you right-click on a word.

Getting ITfFnReconversion

This is a two-step process. First, you get the system function provider, and then you get the reconversion pointer (error checking is omitted):

ITfFunctionProvider* ipProvider;

hr = ITfThreadMgr::GetFunctionProvider(GUID_SYSTEM_FUNCTIONPROVIDER, &ipProvider);

IUnknown* ipConv;

hr = ipProvider->GetFunction(GUID_NULL, IID_ITfFnReconversion, &ipConv);

ITfFnReconversion *ipConversion;

hr = ipConv->QueryInterface(IID_ITfFnReconversion, &ipConversion);

What should QueryRange do?

ITfFnReconversion::QueryRange needs to modify the range to cover the entire range of text contained by the conversion token. An example is useful here, as that last sentence didn't even make sense to me. The Speech text service does correction by words, and all words must be part of a single recognition.

Therefore, the Speech text services's implementation of QueryRange alters the range of text to contain entire words that are part of a single recognition.

If the range starts in the middle of a word with recognition data, QueryRange shifts the start of the range to the start of a word. If the range starts in text that doesn't have recognition data associated with it, it moves the start point to a point where it does have recognition data, or to the end of the initial range. If it runs off the end, it sets pfConvertable to FALSE, and returns NULL.

If it does find some recognition data for the start of the range, it adjusts the end of the range in the same manner.

But wait, you say. How can QueryRange do all that when it doesn't have an edit cookie? Well, it gets one in the usual manner - it (say it with me) schedules an edit session to do all that work. OK, you say, how can I get my data back from an asynchronous edit session? Well, you're in luck - you're guaranteed to be able to schedule a synchronous edit session from ITfFnReconversion::QueryRange.

What should GetReconversion do?

ITfFnReconversion::GetReconversion needs to compute the candidates from a target range (that should be aligned with a conversion token). Like QueryRange, most of the work will need to be done from an edit session, and also like QueryRange, you are guaranteed to be able to schedule a synchronous edit session from within GetReconversion. How you go about computing the candidates is discussed in my next post.