Rob's Rhapsody : Development

Speech Server 2007 Resources

robch — Wed, 08 Oct 2008 17:30:17 GMT

If you want to start developing on Speech Server 2007, you should probably start off using the Speech Server 2007 Developer Edition – a free, yet fully functional version of the retail product. You can download it here. Please be sure to follow the steps in the Installation Notes exactly, otherwise the installation may fail.

The Developer Edition is only free for developing your application. You'll still need to purchase Office Communications Server 2007 licenses once you're ready to deploy your application in a non-test deployment. You can read more about Speech Server Licensing here.

Speech Server 2007 works with Visual Studio 2005 not Visual Studio 2008. After installing Speech Server 2007, you'll also need to install a patch for a bug that occurs when you have .NET 3.0 SP1 or Vista SP1. You can download that patch here.

For support, you can use the Speech Server 2007 Community Support forums here. For bugs, please file the bug in the ECS-UC-Dev queue, or under OCS 2007.

Documentation is found here. White papers can be found here under the technical articles section. For example, here is a white paper on capacity planning.

The Speech Server (2007) MOM pack can be found here.

Books:

Pro Microsoft Speech Server 2007: Developing Speech Enabled Applications with .NET - walks intermediate to advanced developers through the basics of speech and telephony technology
Office Communication Server 2007 Resource Kit - reference for deploying, configuring, and supporting Office Communications Server 2007

Additional web resources:

http://gotspeech.net - useful site for Speech Server application development
http://gotspeech.net/elearning - The official Speech Server training

Speech Macro of the Day: Speech Dictionary

robch — Sun, 03 Aug 2008 01:06:47 GMT

I get feedback from people, from time to time, that they'd like a more efficient way to add items to their speech dictionary. Although, there is a facility in Windows Speech Recognition already to do this, it's one word at a time, and it only allows you to record the pronunciation, not specify it yourself.

So ... What do I do when I have request like this? I make a new macro of the day. Thus, today's speech macro of the day: Speech Dictionary.wsrMac

First, if we're going to be messing around with the speech dictionary, it might be nice to see what's already in it... To do that, I made a command where I can say "Export the speech dictionary", and it'll export all the words/phrases that have been customized into a text file, and then launch that text file for me to take a look at. Here's the command:

<command>
    <listenFor>Export ?the speech dictionary</listenFor>
    <script language="VBScript">
      <![CDATA[
        fileName = "dictionary.txt"
        Set lexToken = CreateObject("SAPI.SpObjectToken")
        lexToken.SetId("HKEY_CURRENT_USER\SOFTWARE\Microsoft\Speech\CurrentUserLexicon")
        Set lex = lexToken.CreateInstance()
        Set words = lex.GetWords(1)
        Set fso = CreateObject("Scripting.FileSystemObject")
        Set file = fso.CreateTextFile(fileName, 1)

        For Each word in words
          If (word.LangId = 1033) Then

            Set prons = word.Pronunciations
            If prons.Count = 0 Then
              file.Write word.Word & vbCrLf
            Else
              For Each pron in prons
                file.Write word.Word & "/"
                If pron.PartOfSpeech = 61440 Then
                  file.Write "BLOCKED" & vbCrLf
                Else
                  file.Write pron.Symbolic & vbCrLf
                End If
              Next
            End If

          End If
        Next
        file.Close
        Application.Run(fileName)
      ]]>
    </script>
</command>

As you can see, it uses a bunch of speech APIs that are already in Vista, that any application can take advantage of. The first 7 lines opens up the speech dictionary (aka User Lexicon), and also opens up a file (dictionary.txt) to stick the words in in a more human readable format.

Then, for each word that it finds, it checks the language, and if it's 1033 (which means US English), it'll output the word into the file. But to do that, it needs to see how many pronunciations are available for each word. If there are zero, it'll just output the word. If there are more than one, it'll output one line per pronunciation.

There's also a special case, where if the part of speech is "61440", it outputs the word "BLOCKED". "61440" is a special kind of part of speech that the underlying speech platform uses to tell the underlying speech engine, this word should be blocked and not recognized at any time. The "BLOCKED" convention is just one I made up for this macro.

After looping thru all the words, it'll close the file, and launch the text file that was created.

Here's a sample of what my speech dictionary looks like when I say "Export the speech dictionary":

Ima/BLOCKED
im/BLOCKED
Visual Studio/v ih zh uw l s t uw d iy ow
antidisestablishmentarianism/ae n t ay d ih s ih s t ae b l ih sh m ax n t eh r iy ax n ih z ax m
Rob Chambers
Itamar/ih d ae m aa r
Itamar/ih t ae m aa r
Zac Chambers
Nic Chambers
Bec Chambers
Jac Chambers

The first few I've pasted here are words that the system has learned thru adaptation that I don't actually want it to use. Sometimes when I say "I am a GPM at Microsoft", it thinks I'm saying Ima or im. Thus... The first time I saw that happen, I selected Ima, and blocked it from my dictionary. More on that in a second...

Then, you can see for Visual Studio, I've actually got a full pronunciation listed. I probably don't have to, but it's here to show you that you can either have pronunciations listed, or not. Having it added as a single unit, ensures that when I say it, it'll always be cased properly.

The next word, "antidisestablishmentarianism", is one of the longest words in the English language, but it's not included in the speech dictionary by default. My son, Zac, loves this word, so of course I have to have it in my speech dictionary.

Next, you can see my name is listed with no pronunciation. Since both my first and last name are also common words, I've added my name here as a single unit, so when I say "Rob Chambers", I again get the proper casing.

Next up, is Itamar, pronounced two different ways. One with a "d" sound and one with a "t" sound. This way, no matter how I end up saying it in a hurry, I can speak Itamar's name properly in email communication with Itamar. BTW ... If you don't know Itamar, you should check out his ms-speech forum on Yahoo! Groups. It's a great place to learn more about Microsoft Speech and speech recognition in general.

Next up are the names of my kids. I have 4 boys, and they have short forms of more traditional names for their first names, so that their initials actually are the same as their first names. I played a bit too many video games as a kid, and RLC wasn't that cool for initials, compared with other kids in my neighborhood. My kids initials are the same as their names. It throws their teachers for a loop at first, but ... Well ... What can I say... I like it. So do they.

OK, now that I've described the lines, let's talk about the format of the pronunciations for a minute. These pronunciations are an attempt at human readable form, but using the exact same form as the underlying speech platform. That brings me to the next command, "Show phonemes":

<command>
<listenFor>Show phonemes</listenFor>
<run command="http://msdn.microsoft.com/en-us/library/ms717239(VS.85).aspx"/>
</command>

Say it, and it'll take you to the page on MSDN that describes what the American English Phoneme Representation is for the Speech API.

OK, now we're getting to the fun part. Now, let's say you wanted to add a new word. The command I'm about to show you, will let you say "Add that to the speech dictionary", and it'll copy whatever word is selected in your document (using the Windows clipboard), and add it to the speech dictionary with no specific pronunciation.

When I originally wrote this set of commands, I had 4 different commands. One for adding words, one for removing words, one for blocking them, and one for unblocking them. I quickly saw that they were all identical, so I made one command that can do any one of those 4 operations. Here's what it looks like with it's helper listenForList:

<command>
<listenFor>[operationPhrase] ?for that ?from ?to the speech dictionary</listenFor>
<setTextFeedback>Speech Dictionary: {[operationPhrase]}</setTextFeedback>
<script language="VBScript">
    <![CDATA[
      ' Get the "that" text from the curent application...
      Application.SendKeys("{250 WAIT}{{CTRL}}c{250 WAIT}")
      that = Application.clipboardData.GetData("text")

      ' Determine if we're adding prons, adding phrases, remove phrases, or blocking phrases
      operation = "{[operation]}"
      ' If we're adding a pron, we'll need to use the recognizer, otherwise we'll just need the lexicon
      If operation = "addpron" Then
        Set recognizer = CreateObject("SAPI.SpSharedRecognizer")
      Else
        Set lexToken = CreateObject("SAPI.SpObjectToken")
        lexToken.SetId("HKEY_CURRENT_USER\SOFTWARE\Microsoft\Speech\CurrentUserLexicon")

        Set lex = lexToken.CreateInstance()
      End If
      ' Keep track of how many words/phrases we added, and loop thru the lines in the "that"...
      cWords = 0
      lineStartPos = 1
      Do

        ' Find the next line break
        lineSeperatorPos = InStr(lineStartPos, that, Chr(10))
        if (lineSeperatorPos = 0) Then lineSeperatorPos = Len(that)

        ' Find the text for that line
        thisLine = Mid(that, lineStartPos, lineSeperatorPos - lineStartPos + 1)
        lineStartPos = lineStartPos + Len(thisLine)

        ' Trim off the CR/LF
        if (Right(thisLine, 1) = Chr(10)) Then thisLine = Left(thisLine, Len(thisLine) - 1)
        if (Right(thisLine, 1) = Chr(13)) Then thisLine = Left(thisLine, Len(thisLine) - 1)

        ' If we have something to operate on
        If (Len(Trim(thisLine)) > 0) Then

          ' Determine if there's a pronuncation included
          pronSeperatorPos = InStr(thisLine, "/")
          If (pronSeperatorPos = 0) Then
            ' Perform the operation with no pronuncation
            If operation = "addpron" Then Call recognizer.DisplayUI(65552, thisLine, "AddRemoveWord", thisLine)
            If operation = "add" Then Call lex.AddPronunciation(thisLine, 1033, 0)
            If operation = "remove" Then Call lex.RemovePronunciation(thisLine, 1033, 0)
            If operation = "block" Then Call lex.AddPronunciation(thisLine, 1033, 61440)
          Else
            ' Find the pronuncation and collapse it
            word = Left(thisLine, pronSeperatorPos - 1)
            pron = Right(thisLine, Len(thisLine) - pronSeperatorPos)
            pron = CollapsePron(pron)
            ' Special case the "BLOCKED" pronuncation
            partOfSpeech = 0
            If pron="BLOCKED" Then
              partOfSpeech = 61440
              pron = ""
            End If
            ' Perform the operation with the pronuncation (and just continue if there's an error)
            On Error Resume Next
            If operation = "addpron" Then Call recognizer.DisplayUI(65552, word, "AddRemoveWord", word)
            If operation = "add" Then Call lex.AddPronunciation(word, 1033, partOfSpeech, pron)
            If operation = "remove" Then Call lex.RemovePronunciation(word, 1033, partOfSpeech, pron)
            If operation = "block" Then Call lex.AddPronunciation(word, 1033, 61440, pron)
            On Error Goto 0
          End If

          cWords = cWords + 1

        End If

      Loop while lineStartPos < Len(that)

      ' Tell the user what we did...
      If (cWords = 1) Then
        If operation = "addpron" Then Call Application.Alert("Added pronunciation for " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
        If operation = "add" Then Call Application.Alert("Added " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
        If operation = "remove" Then Call Application.Alert("Removed " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
        If operation = "block" Then Call Application.Alert("Blocked " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
      Else
        If operation = "addpron" Then Call Application.Alert("Added pronunciations for " & cWords & " words/phrases!", "Speech Dictionary", 1)
        If operation = "add" Then Call Application.Alert("Added " & cWords & " words/phrases!", "Speech Dictionary", 1)
        If operation = "remove" Then Call Application.Alert("Removed " & cWords & " words/phrases!", "Speech Dictionary", 1)
        If operation = "block" Then Call Application.Alert("Blocked " & cWords & " words/phrases!", "Speech Dictionary", 1)
      End If

      Function CollapsePron(pron)

        ret = ""
        insideBrackets = vbFalse
        For i = 1 to Len(pron)
          If (Not insideBrackets) Then
            If (Mid(pron, i, 1) = "[") Then
              insideBrackets = vbTrue
            ElseIf (Mid(pron, i, 1) <> "/") Then
              ret = ret & Mid(pron, i, 1)
            End If
          ElseIf (Mid(pron, i, 1) = "]") Then
            insideBrackets = vbFalse
          End If
        Next
        CollapsePron = ret
      End Function
    ]]>
</script>
</command>

and:

<listenForList name="operationPhrase" propname="operation">
<item propval="addpron">Add ?a pronunciation</item>
<item propval="addpron">Add ?a pron</item>
<item propval="add">Add</item>
<item propval="remove">Remove</item>
<item propval="block">Block</item>
<item propval="remove">Unblock</item>
</listenForList>

I'll leave the details on the specifics for an exercise for the readers. As a user of the macro, though, you can now say things like:

"Add a pronunciation for that from the speech dictionary",
"Add that to the speech dictionary",
"Remove that from the speech dictionary",
"Block that from the speech dictionary", and
"Unblock that from the speech dictionary"

Your selection will have to be a single word/phrase, or multiple words/phrases separated by line breaks. The word/phrases can also have a trailing pronunciation, similar in form to what you see in the output from "Export the speech dictionary".

"OK, but how can I generate those pronunciations myself?" Good question!

Use this command:

<command>
<listenFor>Sounds like [...]</listenFor>
<listenFor>Insert sounds like [...]</listenFor>
<script language="VBScript">
    <![CDATA[

      Application.SetTextFeedback("Sounds like...")
      Set pc = CreateObject("SAPI.SpPhoneConverter")
      pc.LanguageId = 1033

      pron = "/"
      firstElement = Result.PhraseInfo.Properties.Item(0).FirstElement
      numberOfElements = Result.PhraseInfo.Properties.Item(0).NumberOfElements

      For i = 1 To numberOfElements
        Set elem = Result.PhraseInfo.Elements.Item(firstElement + i - 1)
        pron = pron & "[" & elem.LexicalForm & "]" & "/" & pc.IdToPhone(elem.Pronunciation) & " "
      Next
      Application.Wait(0.25)
      Application.SetTextFeedback("Sounds like: " & pron)
      Application.InsertText(pron)

    ]]>
</script>
</command>

This will use dictation to allow you to say "Sounds like Visual Studio", and it'll output /[visual]/v ih zh uw l /[studio]/s t uw d iy ow. So, if you have a word that you're trying to add, you can use the built in pronunciations of other words that WSR already knows about, to cut and paste together your own pronunciation.

Another way to do it would be to select the word of phrase you wanted to build a pronunciation for, and saying "What's that sound like?", which is the final command we'll put into this macro:

<command>
<listenFor>What's that sound like</listenFor>
<listenFor>What does that sound like</listenFor>
<script language="VBScript">
    <![CDATA[

      Application.SendKeys("{250 WAIT}{{CTRL}}c{250 WAIT}")
      that = Application.clipboardData.GetData("text")

      Application.EmulateRecognition("Go after that")
      Application.EmulateRecognition("Insert sounds like " & that)

    ]]>
</script>
</command>

That will copy the selection, move right after it, and then pretend you actually said it. For many words/phrases, this will work even if Windows Speech Recognition doesn't really know how to pronounce the word/phrase, because the system will make it's best guess on how to pronounce it just like it would if you were trying to click on that word on a web page with your voice.

OK ... Now here's another command that will make your phrases a little shorter if you're actually using the commands inside Notepad.exe with the dictionary.txt file open:

<command>
<appIsInForeground processName="notepad.exe" windowTitleContains="dictionary.txt"/>
<listenFor>[operationPhrase] ?for that</listenFor>
<emulateRecognition>{[operationPhrase]} that the speech dictionary</emulateRecognition>
</command>

This basically only works when notepad is in focus, and it's editing dictionary.txt (as it would be when you've just said "Export the speech dictionary". This will enable you to say simpler commands like:

"Add a pronunciation for that",
"Add that",
"Remove that",
"Block that", and
"Unblock that"

Here's the macro in complete form:

<speechMacros>


<command>
    <listenFor>[operationPhrase] ?for that ?from ?to the speech dictionary</listenFor>
    <setTextFeedback>Speech Dictionary: {[operationPhrase]}</setTextFeedback>
    <script language="VBScript">
      <![CDATA[
        ' Get the "that" text from the curent application...
        Application.SendKeys("{250 WAIT}{{CTRL}}c{250 WAIT}")
        that = Application.clipboardData.GetData("text")

        ' Determine if we're adding prons, adding phrases, remove phrases, or blocking phrases
        operation = "{[operation]}"
        ' If we're adding a pron, we'll need to use the recognizer, otherwise we'll just need the lexicon
        If operation = "addpron" Then
          Set recognizer = CreateObject("SAPI.SpSharedRecognizer")
        Else
          Set lexToken = CreateObject("SAPI.SpObjectToken")
          lexToken.SetId("HKEY_CURRENT_USER\SOFTWARE\Microsoft\Speech\CurrentUserLexicon")

          Set lex = lexToken.CreateInstance()
        End If
        ' Keep track of how many words/phrases we added, and loop thru the lines in the "that"...
        cWords = 0
        lineStartPos = 1
        Do

          ' Find the next line break
          lineSeperatorPos = InStr(lineStartPos, that, Chr(10))
          if (lineSeperatorPos = 0) Then lineSeperatorPos = Len(that)

          ' Find the text for that line
          thisLine = Mid(that, lineStartPos, lineSeperatorPos - lineStartPos + 1)
          lineStartPos = lineStartPos + Len(thisLine)

          ' Trim off the CR/LF
          if (Right(thisLine, 1) = Chr(10)) Then thisLine = Left(thisLine, Len(thisLine) - 1)
          if (Right(thisLine, 1) = Chr(13)) Then thisLine = Left(thisLine, Len(thisLine) - 1)

          ' If we have something to operate on
          If (Len(Trim(thisLine)) > 0) Then

            ' Determine if there's a pronuncation included
            pronSeperatorPos = InStr(thisLine, "/")
            If (pronSeperatorPos = 0) Then
              ' Perform the operation with no pronuncation
              If operation = "addpron" Then Call recognizer.DisplayUI(65552, thisLine, "AddRemoveWord", thisLine)
              If operation = "add" Then Call lex.AddPronunciation(thisLine, 1033, 0)
              If operation = "remove" Then Call lex.RemovePronunciation(thisLine, 1033, 0)
              If operation = "block" Then Call lex.AddPronunciation(thisLine, 1033, 61440)
            Else
              ' Find the pronuncation and collapse it
              word = Left(thisLine, pronSeperatorPos - 1)
              pron = Right(thisLine, Len(thisLine) - pronSeperatorPos)
              pron = CollapsePron(pron)
              ' Special case the "BLOCKED" pronuncation
              partOfSpeech = 0
              If pron="BLOCKED" Then
                partOfSpeech = 61440
                pron = ""
              End If
              ' Perform the operation with the pronuncation (and just continue if there's an error)
              On Error Resume Next
              If operation = "addpron" Then Call recognizer.DisplayUI(65552, word, "AddRemoveWord", word)
              If operation = "add" Then Call lex.AddPronunciation(word, 1033, partOfSpeech, pron)
              If operation = "remove" Then Call lex.RemovePronunciation(word, 1033, partOfSpeech, pron)
              If operation = "block" Then Call lex.AddPronunciation(word, 1033, 61440, pron)
              On Error Goto 0
            End If

            cWords = cWords + 1

          End If

        Loop while lineStartPos < Len(that)

        ' Tell the user what we did...
        If (cWords = 1) Then
          If operation = "addpron" Then Call Application.Alert("Added pronunciation for " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
          If operation = "add" Then Call Application.Alert("Added " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
          If operation = "remove" Then Call Application.Alert("Removed " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
          If operation = "block" Then Call Application.Alert("Blocked " & Chr(34) & thisLine & Chr(34) & "!", "Speech Dictionary", 2)
        Else
          If operation = "addpron" Then Call Application.Alert("Added pronunciations for " & cWords & " words/phrases!", "Speech Dictionary", 1)
          If operation = "add" Then Call Application.Alert("Added " & cWords & " words/phrases!", "Speech Dictionary", 1)
          If operation = "remove" Then Call Application.Alert("Removed " & cWords & " words/phrases!", "Speech Dictionary", 1)
          If operation = "block" Then Call Application.Alert("Blocked " & cWords & " words/phrases!", "Speech Dictionary", 1)
        End If

        Function CollapsePron(pron)

          ret = ""
          insideBrackets = vbFalse
          For i = 1 to Len(pron)
            If (Not insideBrackets) Then
              If (Mid(pron, i, 1) = "[") Then
                insideBrackets = vbTrue
              ElseIf (Mid(pron, i, 1) <> "/") Then
                ret = ret & Mid(pron, i, 1)
              End If
            ElseIf (Mid(pron, i, 1) = "]") Then
              insideBrackets = vbFalse
            End If
          Next
          CollapsePron = ret
        End Function
      ]]>
    </script>
</command>

<command>
    <listenFor>Export ?the speech dictionary</listenFor>
    <script language="VBScript">
      <![CDATA[
        fileName = "dictionary.txt"
        Set lexToken = CreateObject("SAPI.SpObjectToken")
        lexToken.SetId("HKEY_CURRENT_USER\SOFTWARE\Microsoft\Speech\CurrentUserLexicon")
        Set lex = lexToken.CreateInstance()
        Set words = lex.GetWords(1)
        Set fso = CreateObject("Scripting.FileSystemObject")
        Set file = fso.CreateTextFile(fileName, 1)

        For Each word in words
          If (word.LangId = 1033) Then

            Set prons = word.Pronunciations
            If prons.Count = 0 Then
              file.Write word.Word & vbCrLf
            Else
              For Each pron in prons
                file.Write word.Word & "/"
                If pron.PartOfSpeech = 61440 Then
                  file.Write "BLOCKED" & vbCrLf
                Else
                  file.Write pron.Symbolic & vbCrLf
                End If
              Next
            End If

          End If
        Next
        file.Close
        Application.Run(fileName)
      ]]>
    </script>
</command>

<command>
    <listenFor>Sounds like [...]</listenFor>
    <listenFor>Insert sounds like [...]</listenFor>
    <script language="VBScript">
      <![CDATA[

        Application.SetTextFeedback("Sounds like...")
        Set pc = CreateObject("SAPI.SpPhoneConverter")
        pc.LanguageId = 1033

        pron = "/"
        firstElement = Result.PhraseInfo.Properties.Item(0).FirstElement
        numberOfElements = Result.PhraseInfo.Properties.Item(0).NumberOfElements

        For i = 1 To numberOfElements
          Set elem = Result.PhraseInfo.Elements.Item(firstElement + i - 1)
          pron = pron & "[" & elem.LexicalForm & "]" & "/" & pc.IdToPhone(elem.Pronunciation) & " "
        Next
        Application.Wait(0.25)
        Application.SetTextFeedback("Sounds like: " & pron)
        Application.InsertText(pron)

      ]]>
    </script>
</command>

<command>
    <listenFor>What's that sound like</listenFor>
    <listenFor>What does that sound like</listenFor>
    <script language="VBScript">
      <![CDATA[

        Application.SendKeys("{250 WAIT}{{CTRL}}c{250 WAIT}")
        that = Application.clipboardData.GetData("text")

        Application.EmulateRecognition("Go after that")
        Application.EmulateRecognition("Insert sounds like " & that)

      ]]>
    </script>
</command>

<command>
    <listenFor>Show phonemes</listenFor>
    <run command="http://msdn.microsoft.com/en-us/library/ms717239(VS.85).aspx"/>
</command>

<command>
    <appIsInForeground processName="notepad.exe" windowTitleContains="dictionary.txt"/>
    <listenFor>[operationPhrase] ?for that</listenFor>
    <emulateRecognition>{[operationPhrase]} that the speech dictionary</emulateRecognition>
</command>

<listenForList name="operationPhrase" propname="operation">
    <item propval="addpron">Add ?a pronunciation</item>
    <item propval="addpron">Add ?a pron</item>
    <item propval="add">Add</item>
    <item propval="remove">Remove</item>
    <item propval="block">Block</item>
    <item propval="remove">Unblock</item>
</listenForList>

</speechMacros>

That's it! I know this is a lot of script to digest, but if you don't really want to, don't! Just use the macro as is. Questions? Comments? Let us know!

Windows Speech Recognition - ExactMatchOverPartialMatch

robch — Mon, 07 May 2007 17:57:27 GMT

A user wrote in over the weekend with a question about how to automatically choose an item when prompted with our clarification user interface in Windows Speech Recognition (WSR). But ... Before I answer that question, I have to give a little background.

What is our clarification user interface?

When the recognizer hears the user say something ambiguous and we're not sure what to do, the system will ask the user for clarification (or confirmation). For example, if the user says "File" in an application that both has a "File" menu and a button somewhere that has the word "File" in it (like a toolbar button that is for creating a "New File"), little numbers will fly out and hover over the two items on the screen. The user can then choose, "1", or "2", and then the user can say "OK".

This is related to the feature that David Pogue wrote up as one of 2006's top 10 features of the year. That feature treats everything as ambiguous when the user says "Show Numbers".

There is another form of the clarification user interface, as well, that we use for items that don't have an on screen representation, like applications to start. In this example, if the user says "Start Internet Explorer", the system sees that there are two possible matches, "Internet Explorer" and "Internet Explorer (no add-ins)". Since we're not 100% sure which one the user wanted, we'll ask the user to clarify.

How do I make it automatically pick?

Well, in general, we can't automatically pick the right one because we still haven't perfected "brain wave recognition" yet. :-) But ... Seriously, though ... You can make it better.

There's a feature called ExactMatchOverPartialMatch. It's turned off by default, but you can turn it on by changing a value in the registry. If you're not comfortable changing the registry, don't do this. But if you are, here you go:

key = HKEY_CURRENT_USER\Software\Microsoft\Speech\Preferences
value name = ExactMatchOverPartialMatch
value type = DWORD
value = 1 (default is 0)

After changing this registry setting, you'll have to restart speech. Once that's done, now in the two examples I gave earlier ("File", and "Start Internet Explorer") no clarification user interface will be displayed, because there will be one and only one exact match. "File" == "File", and "Internet Explorer"=="Internet Explorer".

ExactMatchOverPartialMatch wont' solve all your problems, but it will make the system better for a lot of people. Try it out. Let us know what you think.

Introducing a new speech recognition blogger

robch — Fri, 04 May 2007 20:15:13 GMT

If you've used Windows Speech Recognition in Windows Vista you've undoubtedly used Eric Brown's work. He's a software developer that's worked on speech tirelessly for several years. He's mostly focused on the dilatation system we have in Windows Speech Recognition, but he's also done several other cool things as well like our "How do I <blah blah blah>" help system integration.

If you want to know more about how dictation in Windows Speech Recognition works (more from a developer perspective) or how other applications can be dictation enabled you should head over to Eric's new blog: TSFAware.

SpeechSaver -- kinda like a screen saver, but for speech recognition's listening state

robch — Sat, 28 Apr 2007 20:33:00 GMT

I was talking to a customer recently, Greg Duncan, and one of the things that he came up was an idea for a utility to automatically turn the speech recognition engine's state off after a certain amount of idle "speech" time.

A couple emails later, this morning, Greg had already posted the full source code and a link to an installer for SpeechSaver on his blog. I downloaded it, installed it, and checked it out. It does exactly what it's supposed to.

Nice job, Greg.

If you want a solution that automatically turns the microphone off after a set period of time of inactivity, you might want to check it out.

SpeechTEK 2007 in NY

robch — Tue, 24 Apr 2007 17:36:48 GMT

SpeechTEK is always an interesting convention for those that are interested in speech technologies. I often go to see what the industry at large is up to, and to see if I can meet some customers.

This year, in addition to just attending, I'll also be teaching a session at SpeechTEK University, which is held on site at the same time. I'll be teaching STKU-6: Developing Applications Using System.Speech and SAPI on Windows Vista.

Here's the description of the course from the SpeechTEK Program Guide:

Developing Applications Using System.Speech and SAPI on Windows Vista

Rob Chambers, Principal Development Lead and Architect, Microsoft)

Thursday, August 23rd, 2007, 1:30 p.m. – 4:30 p.m.

Speech recognition and speech synthesis have come a long way at Microsoft in the past 10 years. The evolution of the developer APIs, the included TTS and SR engines, and the integrated user experiences in Windows Vista will have a profound impact on the availability of Windows applications in the future. In this course, you will learn about both SAPI and System.Speech in a hands-on lab setting (bring your own laptop). Attendees with either COM and C++ experience or managed code developers with C# experience will experience Microsoft speech offerings for the desktop firsthand. (Attendees will receive hardware specs and resources to download in advance.)

If you've ever wanted to learn first hand how to use the developer APIs included for free with Windows Vista, feel free to sign up to SpeechTEK University this year.

Windows Speech Recognition in Vista: Dictation Everywhere

robch — Mon, 23 Apr 2007 17:53:00 GMT

Here's a question from a reader about the Dictation Everywhere feature in Windows Speech Recognition in Windows Vista:

This feature is thus very important to me and my future as a PC user.
I have found that in many form fields (such as the one I'm using to send this message) dictation is impossible unless I switch to "Enable dictation everywhere" (otherwise I get "what was that?" messages). There is no short command for switching, and even in that mode I have to confirm every fragment of text I dictate with "one, ok." And then I have to switch out of that mode again for navigation.
It is very important to me that I find ways around this, or at least that a way around it will soon exist. Would you be so kind as to let me know how I can stay up to date with Vista SR improvements, and whom I can contact about its development?
Thank you very much for your time; I really appreciate your efforts to communicate to the public about Vista SR.

Now, for those of you that don't already know, Windows Speech Recognition allows users to dictate into any text field that supports a few standard platform APIs. These APIs (Application Programming Interfaces) are used by the speech recognition system, the handwriting recognition system, and input method editors for foreign languages.

However, some text fields in custom applications don't support what's needed. In those cases Windows Speech Recognition has a fall back input method. It's called Dictation Everywhere. You can turn it on by doing this:

Say, "Show Speech Options"
Say, "Options"
Say, "Dictation Everywhere"

That will toggle it on. Doing those three steps again will turn it off.

When Dictation Everywhere is turned on, we'll listen for the user to speak dictated text, even when they're not in an text field that doesn't support those APIs I mentioned. But ... Because the field doesn't support the right APIs, instead of just sticking the text in there, we're going to have a miniature correction experience with the user first. Otherwise, the user wouldn't be able to correct the text if there was a recognition error.

So ... If you're in one of those fields, use the three steps above to turn on dictation, then you can say, "Hello <period> This is a test <period>". The correction dialog will pop up and allow you to pick an alternative from the list. If you don't see what you really said, simple say it again. If you still don't see what you said, you can say "Spell it", and spell what you wanted.

This is the feature that the user was asking about. They'd like to be able to turn the command on and off easily by using a single voice command.

Unfortunately, since we don't have an end user feature either included directly into Vista at this time, nor do we offer one for download (yet!) for creating macros, end users can't really simulate the same impact of turning this feature on and off with a single voice command.

At least not easily...

If you're a programmer (or don't mind dabbling), you certainly could though. In fact, you can create a simple shell script that does this like this:

set Recognizer = CreateObject("SAPI.SpSharedRecognizer")
Recognizer.EmulateRecognition ("show speech options")
WScript.Sleep(1000)
Recognizer.EmulateRecognition ("options")
WScript.Sleep(1000)
Recognizer.EmulateRecognition ("dictation everywhere")
set Recognizer = Nothing

When you run this it will connect to the shared recognizer that Windows Speech Recognition uses, it will pretend that the user spoke "show speech options", then wait for 1 second, then pretend that the user said "options", again wait for another 1 second, then again, pretend the user said "dictation everywhere".

In fact, you can even save this text as a file called "Dictation Everywhere Toggle.vbs" in your start folder (e.g. "c:\documents and settings\{your user name goes here}\start menu\dictation everywhere toggle.vbs") and you'll be able to say to Windows Speech Recognition, "Start Dictation Everywhere Toggle".

Unfortunately, for all this to work, you actually have to turn Access Control (UAC) off. Otherwise, the shell script can't communicate with the shared recognizer.

In the future, though, we'll have a true end to end macro facility to deal with this in a secure way. Stay tuned for more info on that front...

Keeping speech grammars up-to-date

robch — Thu, 29 Mar 2007 20:24:00 GMT

I've been having a good conversation with some guys that are working on making it easier to program by voice using Python. Their project is called "VoiceCode" and they've been working on it for quite some time. Currently it only supports Dragon NaturallySpeaking as a speech recognizer, but I'm helping them to understand how easy it would be to port to Vista.

One issue that came up recently was: How can they keep their grammars up to date if their grammars change based on what application is running, or what the title is in any given application. I was trying to describe to them in email how they could use the SetWinEventHook method, but I wasn't being as successful describing it to them as I wanted to... So ... I just wrote about 500 lines of code for them to show them.

Then, I thought ... Hey, other people might want to see how to do this too. So I posted the code here. Let me know if you have any questions!

Visual Studio 2005 Service Pack 1 Update for Windows Vista

robch — Wed, 07 Mar 2007 19:20:00 GMT

I know a lot of developers that have been waiting for this one. Yesterday, Microsoft announced the availability of Visual Studio 2005 Service Pack 1 Update for Windows Vista.

I've been using a prerelease version for some time and it's great. If you are a developer and you haven't downloaded it already, you might want to stop what you're doing and do it now!

Installing additional speech recognizer languages...

robch — Tue, 20 Feb 2007 18:57:00 GMT

Here's a question that I get from time to time...

Q: How do I install speech recognition for a language other than the language Windows Vista is running in?

A: By installing a language pack.

Language packs are the Windows way of installing additional language support for features included in the OS. By default, when you install Windows Vista (or purchase a PC with Windows Vista already on it), it comes with a single language.

In Windows Vista Enterprise and Windows Vista Ultimate, you can go to the control panel, search for "display language", and then click on the "Install or uninstall display languages" link.

Follow the on-screen instructions, and in no time you'll have additional recognizer languages as long as it's a Windows Speech Recognition supported language. In Windows Vista we support US English, UK English, German, French, Spanish, Japanese, Simplified Chinese, and Traditional Chinese.

Transfering profiles programatically

robch — Tue, 20 Feb 2007 08:28:00 GMT

Last week I wrote about how to transfer profiles using the Easy Transfer Wizard. That generated some email and some questions in the community about how to do this programmatically.

That's a good question. But unfortunately, the answer isn't that straight forward currently.

In the SAPI documentation you can read about how some of this works, but there's not a single overview of how all this works. So ... I'll try to summarize how it works here.

Here goes ...

In SAPI, recognition profiles are accessed by the engine (and by the application) using the Resource interfaces of SAPI. Specifically, the ISpObjectTokenCategory, IEnumSpObjectTokens, ISpObjectToken, and ISpDataKey interfaces.

To get a list of recognition profiles configured by the user you can use the SpEnumTokens helper function (defined in sphelper.h). You'll need to specify the category you want to enumerate. That's the SPCAT_RECOPROFILES category. This will give you an IEnumSpObjectTokens interface, which is a standard derivative of the IEnum* class of interfaces. Using this interface, you can iterate thru the individual "tokens" by calling IEnumSpObjectTokens::Next -- each of the ISpObjectTokens you get back will represent a single recognition profile.

Given a specific token you can QueryInterface for the ISpObjectToken interface, which derives from ISpDataKey. Then you can ask for the name of the reco profile, by calling ISpDataKey::GetStringValue(NULL, &pszCoTaskMemAllocedProfileName). Be sure and call CoTaskFree on the string returned when you're done with it.

To back up the profile, there are no special helpers, so you'll just have to do a few things the hard way.

First, you should make a backup of the ISpDataKey settings. The easiest way to do this is to use the token's ID to open up the registry location directly. You see, typically the object token is simply a wrapper on top of registry access functions. The token ID is typically actually just nothing more than a path into the registry. For example, for my reco profile that I'm using right now, the token id (retrieved by calling ISpObjectToken::GetID) is: "HKEY_CURRENT_USER\Software\Microsoft\Speech\RecoProfiles\Tokens\{90E13428-A7A5-419B-8034-EA6453E31906}"

So, you'll have to write your own code to iterate thru the registry recursively and back up all the keys and values (with their names) that you find.

There are a few special things that you have to take into consideration when doing this:

1.) If the token id doesn't live directly under "HKEY_CURRENT_USER\Software\Microsoft\Speech\RecoProfiles\Tokens\", you shouldn't assume that the procedure I'm describing here will actually work to back up the recognition profile. If you see a token ID outside of this area, you should abort the backup.

2.) While backing up the profile, you will encounter one or more key names called "Files". Under keys named "Files", the values represent "pointers" to files on disk. You'll have to also back up the files that these "file names" point to.

3.) These files may start with character sequences that look like "%1c%\". If they do, you can replace the values between the percent signs with the text as returned from SHGetFolderPath. For example, one of that data files in my registry under a "Files" key has a "filename" of "%1c%\Microsoft\Speech\Files\TrainingAudio\SP_E18DB2EFB76A432B97889A3DB7210406.wav". The "%1c%" corresponds to a "real" filename of "C:\Documents and Settings\Rob\Local Settings\Application Data\Microsoft\Speech\Files\TrainingAudio\SP_E18DB2EFB76A432B97889A3DB7210406.wav". Again, you can use SHGetFolderPath with the hex value of 0x1c to get the path of "C:\Documents and Settings\Rob\Local Settings\Application Data" and you can put all this together.

4.) When you restore the profile, you should take special care to restore files with different new filenames. If you don't, there could be some collisions. You can get a new filename by calling the ISpObjectToken::GetStorageFileName method.

That's basically it.

Yeah, I know... It's not necessarily the easiest thing you'll ever do, but it is possible, and this is basically how you do it. Hopefully, we'll be able to release an actual example that shows how a real end to end backup and restore works for profiles in the future. But until then, hopefully this post will illuminate some of the magic behind profile management in SAPI.

Windows Speech Recognition -- Where's the API documentation?!

robch — Tue, 13 Feb 2007 22:19:00 GMT

I get a lot of email from customers every day. And some days I end up answering the same questions again and again. Don't get me wrong, I love hearing from customers, and I love enabling them to do great things with Microsoft products.

The real problem isn't that customers ask questions ... it's that the information that they need isn't always easily found. Well ... I'm hopeful that all this blogging is really going to help users. So I'm going to try an experiment and see how it works.

I'm going to try and start answering customer questions by sending them a link to my blog. If the answer is not already on my blog, I'll put it there, and then send them the freshly coined link... Sound good?

OK... So what's the first question that I get asked frequently? Well, it's probably not the most common question, but it's one near and dear to my heart as a platforms kind of guy...

Q: Where can I find Speech API documentation?

A: On MSDN of course. Try these different locations depending on how you want to call into the speech APIs:

C# and managed code: http://msdn2.microsoft.com/en-us/library/system.speech.recognition.aspx
VB and automation clients: http://msdn2.microsoft.com/en-us/library/ms720592.aspx
C++ and native code: http://msdn2.microsoft.com/en-us/library/ms723627.aspx

Do you have a question? Send it in and I'll try to get it answered...