Robert Lucero's Testing Blog

Thoughts on Testing - Don’t Duplicate Code!

Robert Lucero — Fri, 18 May 2012 17:16:44 GMT

Here at Microsoft, the majority of our engineering groups are broken down into test development teams and product development teams. Generally the test teams develop test frameworks and test cases to validate functionality of products. Developers are usually responsible for unit and method based tests. Often times, these teams end up writing very similar test frameworks and to some extent duplicate functionality in code. The old adage, Great Minds Think Alike, applies here but unfortunately teams are unknowingly duplicating each other’s work. Here’s an approach to unifying the work and getting both teams on the same page.

Step 1 - Talk to the Testers and Talk to the Developers!

Talk about what you’re testing. If you’re a developer, meet with the test team to share your test design strategy. Get the unit test plan and unit test tools reviewed. If you’re a tester, get the test framework reviewed by the development team. Highlight points where you can share code and how you plan on using the product code.

By talking, teams get on the same page for what work is being done and what is already covered. The last thing engineering teams need to do is waste time running duplicate tests or changing duplicate code during crunch mode!

Step 2 - Draw the Boundaries

Meet as an engineering team and make sure that each group knows what test features they’re developing. If the development team is building a system of mock objects for their unit tests, the test team should use them. If the test team is developing an in-depth logger, the developers should use it instead of some other solution. If it is clear who is developing each component and there’s a dependency the amount of duplication should be minimized. Test code is often thought of after the fact, especially during busy development cycles. Teams need to get together and work together so that the test infrastructure that is being developed solves both dev and test needs.

This also applies to what tests engineers are writing. If the developers are writing more functional tests, testers can write more edge case or scenario tests. When the ownership is clear, teams can quickly determine the test coverage of their product and similar tests aren’t being simultaneously being written and executed.

Step 3 - Work Together!

Remember to that the end goals of the engineering team is to ship and awesome product. People work really hard and focus on the actual product code, but remembering to work together on the test infrastructure will help teams meet their goal. Again, talk about test development as an important part of the development cycle. Get test code and test projects treated like the product. By talking about the test code, working on the test code and running tests as a team you can minimize the amount of duplication that happens.

Summary

Duplicating test code can be costly and unnecessary. By working together and getting people vested in the test infrastructure, teams will have more reliable test frameworks and hopefully higher quality code!

Quick Aside on XML - The Wrong Way to Use XML

Robert Lucero — Wed, 11 Apr 2012 23:35:29 GMT

XML is a great way to collect and author data in a way that can be consumed by any number of applications. There are some exceptionally powerful tools that can consume XML formats and enforce levels of data validation, but this is only useful if the implementation make sense. After working on some legacy tools and trying to design some new test systems, I’ve realized that there’s a right way to use XML and a horribly wrong way to use XML.

Wrong Way #1: Parsing XML With XElement

Don’t get me wrong, XELement/XPath searching is a great way to look through XML when you’re just trying to retrieve specific values from generic XML files.
For Example:
I want to find every instance within 10 types of XML files where server value is IIS6.0 and change it IIS8.0.

XElement and XPath can quickly search through the XML files regardless of their structure (provided that they are valid XML files) and change those values. But, the limitations of this solution are that your searching through the files generically and you’re treating XML as just a structured input type. In addition, it just doesn’t scale. Even with fancy regex/XPath queries you can only do so much.

Wrong Way #2: Manually Translating XML to Objects

Another quick fix solution I’ve seen implemented a number of times is using Wrong Way #1 to build up an object by hand. First you create the object that you’re trying to build, then you populate the values of the object by reading values and setting them. At this point, any way of extracting information from a file is just as good. The one slight advantage of XML is that the APIs available can help query values.

This pattern’s translation layer can also lend itself to maintenance challenges because of written out step-by-step translation from XML document to object. It also makes it hard to enforce an object-XML document mapping when the translation occurs. The differences between the XML format or object changes are obscured by the translation layer:

Figure 1: Don’t Do This.

Wrong Way #3: Never Using a Schema

If you’ve taken the time to plan out your XML document layout and designing an object model for how you’re going to use all that sweet XML data, you’ve probably also created a schema. So, use it! Schemas that are loaded into editors, like Visual Studio, can make manual authoring of XML much faster. In addition, if you enforce your schema at document load time, you can ensure that you’re only loading documents that the system can handle.

XML Done Right

First, design the object model first. The important logic is going to work with objects not with XML files or random values pulled from the file. Make it strongly typed and serializable. The people consuming your data will appreciate you for it.

Second, let the XmlSerializer do the translation work for you. It won’t mess up with misspelled node names or incorrect value translations (most of the time). The benefit of this model is that when you want to save the object state all you have to do is deserialize the object out to a file. Done. An XML file is saved and available for later.

Third, with the object model in hand you can quickly create a schema. Again, with this schema you can enforce validation at load time and use the schema to get XML Intellisense when hand editing XML files.

Hopefully by planning your XML/Object model with these tips in hand, you can save yourself a lot of trouble in maintenance and helpful in feature design.

Figure 2: Do This

Quick Aside on C# - Reflection’s Power and Pitfalls

Robert Lucero — Thu, 08 Mar 2012 22:27:15 GMT

I am currently developing a couple of internal test tools to help with test case development and test case data analysis. I have designed the tool to be generic, meaning that anybody that can create and serialize objects for me I can edit. Here are some of the interesting issues I’ve run into so far…

C# Reflection Tip 1: Reflecting Over Enumerable Properties

Suppose someone gives me this object to reflect through:

public class Demo
{
    public int Number = 1;
    public Foo[] FooArray
    {
        get;
        set;
    }
}

I want to loop through and display all of the objects contained within the nested array. If you take the standard approach of reflecting through the Demo object’s types and collecting those types’ properties you’ll see something interesting. On the first pass of displaying the object, the type property for the Foo[] array is the strongly typed array object. Everything is fine when looping through the strongly typed elements of the array.

Example:

Type objType = testObject.GetType();
List<PropertyInfo> objProperties = new List<PropertyInfo>(objType.GetProperties());
          
foreach (PropertyInfo property in objProperties)
 {
 …

But, if the original object has been modified, even with data binding, it seems that there is a high probability that following the same method yields different results. Instead of the type property of the Foo[] array being strongly typed, you’ll see all of the weakly typed Array properties like SyncRoot, LongLength, IsSynchronized, etc. Now instead of displaying a clean list of Foo objects, I instead was displaying the base Array properties in the tool.

The way to fix this is by sort of 'preempting’ the fact that this is an array or enumerable collection. If I could recognize that this was an enumerable object, I would skip right ahead to looping through the nested objects like so:

foreach (Object obj in (item.GenericObject as IEnumerable))

Now, I can follow the same reflection probing patterns to show all of the nested object.

Visual Studio makes this issue hard to track down. While debugging, I put watches on the objects that I was expecting to loop through. The problem is that Visual Studio is too smart some times. What the debugger shows is the strongly typed object, even though its ethereal/intermediate form might be recognized by the runtime as a weakly typed generic array.

C# Reflection Tip 2: Don’t Forget That Strings Are Enumerable

Once I solved my generic array problem, I found that there were instances where I was displaying lots of garbage UI data that looked like it was coming from an array or collection. Again, after further investigation I found out that strings were tripping up my IsEnumerable alarm.

Here’s my check for determining if an object’s inherited interface contains IEnumerable as well as screening out strings:

Type[] interfaces = testObject.GetType().GetInterfaces();
foreach (Type i in interfaces)
{
        if (i.IsGenericType && i.GetGenericTypeDefinition().Equals(typeof(IEnumerable<>)) && i != typeof(string))
        {
              return true;
        }
}

This can probably be done better, but it works for now.

C# Reflection Tip 3: Use The Visual Studio Debugger

The Visual Studio Debugger and Immediate Window are invaluable tools for any sort of reflection based project. Putting watches on the objects and running GetProperty() and GetType() checks in the immediate window can help you quickly determine where you’re code’s gone wrong. On occasion, it will lead you astray (See Tip #1), but for the most part the debugging environment will expose other internals and values that will help push the project along.

Scott Guthrie’s blog post on debugging tips is a great primer to using the debugger for these sorts of tasks.

Thoughts on Testing - Breaking Down the Test Plan

Robert Lucero — Wed, 22 Feb 2012 16:05:36 GMT

Test plans, or test specs, are designed to share with a product team exactly what a tester plans on covering to ensure that their feature is ready to ship to customers.

Problem 1: Who Is the Test Plan For?

Thinking about who the test plan is for can greatly influence the results of what comes out of the actual document. There are usually a bunch of vested parties such as developers, program managers, peer testers, test managers, test vendors, sustained engineering teams and security experts just to name a few. Each one is looking for something different out of the test plan document.

Here’s a breakdown of the lifetime of a test spec and who cares about at each milestone:

Milestone

Who Cares

What They Care About

Design Phase

Developers

Program Managers

Peer Testers

Test Managers

Testability of a feature

Design of a feature

Manual and automated test coverage

Impacts to other features

How long testing will take

Test Strategy

Implementation Phase

Developers

Program Managers

Test Managers

Test Vendors

Security Experts

Test Coverage

Checklist of signoff items

Security Scenarios

Test Strategy

Stabilization Phase

Test Managers

Program Managers

Peer Testers

Test Vendors

Sustained Engineering

Test Coverage

Checklist of signoff items

Test Strategy

Post Release

Sustained Engineering

Security Experts

Peer testers

Test Coverage

Test Strategy

Security Scenarios
Checklist of signoff items

The two topics that are cared about in every milestone are test coverage and test strategy. Put simply, it is what is covered and why. Most of the time, test specs and test plans focus more on items in the design phase because this is when test plans are usually reviewed and iterated on. The people in the test plan reviews chose to focus on those items.

Problem 2: What Is the Test Plan Used For?

It isn’t always clear what a test plan is used for once it has been written. Often times, there are notes of work items and costs, lists of tests that need to be run or automated and perhaps specific performance and behavior goals to reference but it usually isn’t looked at again until sustained engineering handoff or something’s gone wrong.

Making test plans something that are consistently referenced, reviewed and used should be a goal for any test team. Otherwise, the exercise of writing the test plan is simply to think about how someone would test the feature and highlight areas of concern in a developer’s design.

Let’s go back to a subset test plan users:

Users	Needs
Feature Tester	Design Phase Testability of a feature Manual and automated test coverage Test Strategy Implementation Phase Test coverage Test work items Test cases Security scenarios Stabilization Phase Test Coverage Test Cases Checklist of signoff items
Test Managers Feature Team	Design Phase How long testing will take Test strategy Test coverage Implementation Phase Test coverage Test work items Test Cases Stabilization Phase Test Coverage Checklist of signoff items
Sustained Engineering Security Experts Future Feature Testers	Stabilization Phase Test Strategy Test Cases Test Coverage Checklist of signoff items

Problem 3: Keeping the Test Plan Up-to-Date

If developers are following more traditional design then implement models, the test plan will usually be signed off prior to a developer starting implementation work. What happens to the test plan now?

There are three scenarios to consider:

Scenario	Test Plan Needs Changes
Developer Design and Spec do not change, feature is implemented exactly as described	No
User requirements change and minor updates or design changes are needed. Feature goals modestly updated and changed	Yes
Major changes to requirements and design cause implementation to drastically change. Feature is implemented completely differently from original design.	Yes

But what usually happens in these situations? If deadlines or milestones are tight, the actual test plan might not be updated. Or, more often than not, the test cases and scenarios are updated on a case by case basis in automation but not necessarily captured in the original document.

Test Plan of the Future™: Features and Requirements

So, after all of that what does the Test Plan of the Future™ need? Here are some of the requirements that I thought of that will address the needs of the primary users:

Test Items

Test Cases

Manual
Automated
Security Specific Tests

Test Coverage

Code Coverage
Scenario Coverage

Checklist for Signing off

Items discussed among feature/product owners

General Test Strategy Notes

Feature Spec/Dev Items

Features
Prioritized Scenarios

Easy to Update
Reflects the current state of the product

This list of requirements satisfies a majority of the needs laid out above. If test plans can reflect the current state of the product, highlight feature and scenario coverage all while being easy to update, it can become a really valuable tester resource.

Test Plans: Looking Ahead…

As things progress, I’m going to try investigating and implementing different types of Test Plans of the Future™. Stay tuned for future posts on this topic…

Speech Recognition - Using Multiple Grammars to Improve Recognition

Robert Lucero — Fri, 03 Feb 2012 17:22:14 GMT

A difficult problem both users and developers face is recognizing words that are similar sounding, but wrong for the current context. An example of this would be the words “yellow” and “hello”.

Using the simple WPF app from the previous Exploring Grammar Based Recognition post, I will show an example of this confusion and a simple way to improve recognition based on a defined context. Specifically, a button to enable and disable grammars will be added to simulate context switching.

Check 2, 3… Check…

This is a continuation of the previous Exploring Grammar Based Recognition post. Please make sure that you’ve installed the Windows SDK as a prerequisite to both of these tutorials.

Step 1: Identifying Recognition Confusion

Using the Simple Speech Recognizer, add the word “hello” to the list of words to be recognized. Then repeat saying “hello” and “yellow” with various inflections. Depending on how I said it, I was able to get the wrong word recognized.

By throwing all the words into the same grammar rule mix, the recognition engine will do its best at guessing what word is spoken. Unfortunately, a bad user experience may result of a misrecognized word.

Step 2: Improving Recognition Through Multiple Grammars

This example of “hello” and “yellow” confusion highlights an interesting problem, but there are some ways to improve this experience.

I’m going to first separate out the two recognition word lists into different grammars. First, move rename the testGrammar to colorGrammar and make it available for other methods to use. I moved it to the MainWindowClass.

Second, create another Choices for the greetings grammar. Populate that with “hello” and any other test words to try out. To highlight where the recognized word is coming from, I also named the grammars and updated what is written out when the recognition event triggers.

Here’s what things should look like:

SpeechRecognizer sr;
List<String> colorList;
List<String> greetingsList;
Choices colors;
Choices greetings;
Grammar colorGrammar;
Grammar greetingsGrammar;

Updates to LoadGrammar():

      private void LoadGrammar()
         {
             //Load up the Choices object with the contents of the Color list, populate the GrammarBuilder,
            //create a Grammar with the Grammar builder helper and load it up into the SpeechRecognizer
            colors.Add(colorList.ToArray());
             greetings.Add(greetingsList.ToArray());
             GrammarBuilder colorGrammarBuilder = new GrammarBuilder(colors);
             GrammarBuilder greetingsGrammarBuilder = new GrammarBuilder(greetings);

             colorGrammar = new Grammar(colorGrammarBuilder);
             colorGrammar.Name = "colorGrammar";
             greetingsGrammar = new Grammar(greetingsGrammarBuilder);
             greetingsGrammar.Name = "greetingsGrammar";

             sr.LoadGrammar(greetingsGrammar);
             sr.LoadGrammar(colorGrammar);

         }

And the Speech Recognized Event:

        void sr_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
         {
             textBox1.Text = textBox1.Text + e.Result.Text + " - \t" + e.Result.Grammar.Name+ "\r\n";
         }

If you run the app now, both “hello” and “yellow” will be recognized and confused. So, I will add one button to toggle the greetingsGrammar.Enabled flag. Here’s what my button click method looks like:

        private void button2_Click(object sender, RoutedEventArgs e)
         {
             sr.Grammars[sr.Grammars.IndexOf(greetingsGrammar)].Enabled = !sr.Grammars[sr.Grammars.IndexOf(greetingsGrammar)].Enabled;

             if (sr.Grammars[sr.Grammars.IndexOf(greetingsGrammar)].Enabled == true)
             {
                 button2.Content = "Disable Greetings";
             }
             else
             {
                 button2.Content = "Enable Greetings";
             }
         }

Now, when you click the button words in the greetings grammar won’t be listened for.

What Was Improved?

In this case, pressing the button changes the words that the Speech Recognition engine is listening for. If the grouping inside of grammar rules or grammars are clever, developers can enable and disable scenarios when the system moves into a specific state. It can give context and, in some cases, better accuracy for the words the system is listening for.

However, it doesn’t improve the more basic problem of confusion if someone says a word that sounds very similar to a word the engine is listening for. This process primarily helps by focusing or broadening the words available for recognition.

Summary

By dynamically enabling and disabling grammars, apps have another tool to help improve the recognition scenarios. Contexts that are provided and acted upon can make for a better recognition experience.

Here are some posts I found useful:

For more ideas or for more background on this post check out my previous post: Exploring Grammar Based Recognition. As always, if you have feedback or questions feel free to leave a comment or contact me through the MSDN blog dashboard tools!

Once again, thanks go out to Steve Meyer for providing awesome feedback regarding this post!

Speech 101, Part 3 - Text To Speech: Getting the Computer to Carry the Conversation

Robert Lucero — Fri, 27 Jan 2012 19:51:18 GMT

In my previous posts I covered a number of scenarios around user speech recognition. System.Speech also includes the capability to synthesize text into sounds which can be used to create unique conversational experiences.

There are a number of great examples on writing Text-to-Speech apps and plugins. I will highlight getting up and running quickly with some examples.

Check 1,2…

The Windows SDK is a requirement for the following example. It needs to be installed to follow along.

TTS Part 1 - Giving the Computer Something to Say

There are two ways to get up and running with System.Speech’s Speech Synthesizer. Simplest way is by creating an instance of the SpeechSynthesizer object and passing a string to the Speak method.

SpeechSynthesizer synth = new SpeechSynthesizer();
synth.Speak("Text to Speak");

That’s it. This is the fastest way to get computers talking to users. Developers can incorporate this into audio prompts or notifications, general audio feedback from the system or whatever they can think of.

TTS Part 2 - Sometimes Hints Help

The simple method that is described above will cause the Speech engine to give its best guess at what the string is when it goes to synthesize. This works well with simple phrases, but not always. Try passing test strings and see what is working by default and what might need to be spelled out.

While working on this blog post I noticed that dates, times, and basic addresses seem to work out of the box.

TTS Part 3 - This Time With Some Feeling

One of the really fun things to do with the SpeechSynthesizer is to adjust the voice, rate and emphasis of what the computer is saying. Properly configured, it can give applications a little more personality.

Important Note: This only works if there are additional voices installed on users systems and if the voices are compatible with the changes that are described. If for some reason, the selected voice cannot be configured with the specific settings the Synthesizer will default to base operating system settings. This includes changing the voice.

PromptBuilder is a great way to modify how the computer sounds in code. In this example I will show how use both PromptBuilder and PromptStyle to slow down the computer voice using a global shared PromptBuilder.

promptBuilder.ClearContent();
PromptStyle style = new PromptStyle();
style.Rate = PromptRate.ExtraSlow;

promptBuilder.StartStyle(style);
promptBuilder.AppendText(phraseTextBox.Text);
promptBuilder.EndStyle();

speechSynth.SpeakAsync(promptBuilder);

There are a number of other configurable options within both PromptBuilder and PromptStyle to give the computer a personality, but as the note describes it is contingent on the voices that are available to the operating system.

TTS Part 3 - Interrupting What the Computer Has to Say

Depending on how much the computer has to say users might reach a point where they think to themselves “Too much talking! How do I make this stop!?”. This example will cover wiring up a basic cancel button action.

First, create a global Prompt object and register two of the main Speech Synthesizer events, SpeakStarted and SpeakCompleted:

speechSynth.SpeakStarted += new EventHandler<SpeakStartedEventArgs>(speechSynth_SpeakStarted);
speechSynth.SpeakCompleted += new EventHandler<SpeakCompletedEventArgs>(speechSynth_SpeakCompleted);

void speechSynth_SpeakCompleted(object sender, SpeakCompletedEventArgs e)
{
cancelButton.IsEnabled = false;
}

void speechSynth_SpeakStarted(object sender, SpeakStartedEventArgs e)
{
cancelButton.IsEnabled = true;
}

Next, define the cancel button behavior to stop the global prompt midsentence:

private void cancelButton_Click(object sender, RoutedEventArgs e)
{
speechSynth.SpeakAsyncCancel(cancelPrompt);
}

Finally, set the global cancel prompt when calling SpeakAsync from the Synthesizer:

private void speakButton_Click(object sender, RoutedEventArgs e)
{
cancelPrompt = speechSynth.SpeakAsync(phraseTextBox.Text);
}

Now users can interrupt the computer if it becomes too verbose.

Summary

Text-to-Speech can really enable some cool user scenarios and give applications another way to convey information. There are all sorts of additional TTS features and possibilities to explore. Below you will fine some additional resources and posts that I found useful:

Also, check out my previous posts on Speech Recognition:

Speech Recognition - Exploring Grammar Based Recognition

Robert Lucero — Tue, 17 Jan 2012 17:11:53 GMT

In my two previous posts, I covered how to create speech recognition engines and use them to parse through WAV files containing a sample “Hello World” recording.

This post will focus on two things. First, simple real-time recognition with a simple hardcoded grammar and second, a way to let you dynamically improve the system by updating the grammar.

Configuring Windows to Enable Speech Recognition

The core Windows operating system has had built in speech recognition since Vista. You must enable the OS speech recognition in order for this tutorial to work.

On Windows 7:

Follow the instructions here: http://windows.microsoft.com/en-US/windows7/Set-up-your-microphone-for-Speech-Recognition

Important: Don’t forget what was chosen to enable speech recognition!

Once the wizard is complete, you can enable and disable speech using the method specified or with the UI that shows up:

Try it out by following the User Experience tutorial.

Listen Up: Writing a Basic Real-Time Speech Recognition WPF App

This is a simple modification on the great MSDN tutorial for the System.Speech namespace. It is an application that will listen for colors that are predefined in a Speech Choices object and will write the output to a text box if the color was correctly recognized. Inside of this app I will show how to:

Initialize a simple grammar based on a string array
Initialize a Speech Recognizer object, which enables basic real-time speech recognition
Wiring up a way to update the current grammar

And as always, the Windows SDK is a requirement. Please install it to follow along with this tutorial.

Step 1: Initialize the Simple Grammar and Speech Recognizer

First, launch Visual Studio and start with a blank WPF app. In the designer, add two text boxes, three labels and one button like so:

Set the larger text box to read only to get the best results from this tutorial. Once the UI objects have been created and laid out open up the window’s code behind file.

Next, add System.Speech to the project references:

Now, add following code to the Window constructor:

SpeechRecognizer sr;
List<String> colorList;
Choices colors;
public MainWindow()
{
      InitializeComponent();

      sr = new SpeechRecognizer();
      colors = new Choices();
      colorList = new List<string>() { "red", "yellow", "green", "blue" };
      InitializeSpeechRecogonition();
}

Once that’s done, it is time to create the InitializeSpeechRecognizer and the LoadGrammar helper methods:

private void InitializeSpeechRecogonition()
{
        //First, load the grammar then wire up the Speech Recognition events
       LoadGrammar();
       sr.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(sr_SpeechRecognized);
}

private void LoadGrammar()
{
      //Load up the Choices object with the contents of the Color list, populate the GrammarBuilder,
     //create a Grammar with the Grammar builder helper and load it up into the SpeechRecognizer
     colors.Add(colorList.ToArray());
      GrammarBuilder grammarBuilder = new GrammarBuilder(colors);
      Grammar testGrammar = new Grammar(grammarBuilder);
      sr.LoadGrammar(testGrammar);

}

There are three objects of interest. The SpeechRecognizer object, Choices, and the Grammar. SpeechRecognizer hooks into the operating system’s shared recognizer but initializes its own engine to handle recognition events. The benefit of using SpeechRecognizer is that it makes it doesn’t require the developer to worry about the audio input, but it doesn’t allow for more advanced recognition scenarios.

Choices is a basic way to create simple grammars without having to define a detailed external grammar file. For this tutorial and basic scenarios requiring only simple recognition this is a great way to get started.

The Grammar object is instantiated from the GrammarBuilder, which will create a root rule for the string list that is contained within the Choices object. This is *essential* for the Speech Recognition engine. Without a grammar, there is nothing that the recognition engine can use to determine what was said.

Now that all the Speech Recognition objects have been wired up, set the Speech Recognized handler to write the output to the output text box:

void sr_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
{
textBox1.Text = textBox1.Text + e.Result.Text + "\r\n";
}

Once all of this is completed, the app will listen for whatever colors you defined in the colorList. In this example those colors are red, yellow, green and blue.

Step 2: Getting the App to Listen and Listen Better

Launch the application and enable Windows Speech Recognition. Make sure that if you have to click outside of your application to enable recognition, you click back into the test app.

Now try it out! Speak into your microphone and say one of the words that exist in the list. A few things will happen:

The word that was said wasn’t recognized. Either it was the correct word and the engine couldn’t figure it out or a word that wasn’t in the list was said.
The word was recognized and showed up in the text box. Awesome!
A different word was said, but a word in the list was recognized instead.

This is an example of ‘bad’ recognition happening to me:

In this case of bad recognition, apparently the word I said and they way I said it was enough to recognize a word that was in the grammar.

Using the text box and button that were also added to the app, I am going to enable the user to improve the recognition experience by adding words to the list and reloading the grammar.

All you need to do is wire up the button’s click event and add the following code:

private void button1_Click(object sender, RoutedEventArgs e)
{
     if (!String.IsNullOrEmpty(textBox2.Text))
     {
         colorList.Add(textBox2.Text);
         LoadGrammar();
         textBlock1.Text = "Added: " + textBox2.Text;
         textBox2.Text = string.Empty;
      }
}

Now, when users add a word and click the Add button, the Speech Recognizer’s grammar will be reloaded with the new word.

The grammar is updated and new words are recognized based on user input.

Summary

Try out other grammar scenarios. My previous two posts use a grammar file that can be modified and loaded up externally.

There are some useful tutorials and code examples over on MSDN. You can check them out here:

Getting Started With Speech Recognition: http://msdn.microsoft.com/en-us/library/hh361683.aspx
Exploring Speech Recognition from MSDN magazine (note - this was written in the Vista timeframe, so there are updates that apply only to Win7): http://msdn.microsoft.com/en-us/magazine/cc163663.aspx

Check out my other blog posts on the Speech Recognition Engine:

Speech 101 - Using C++: http://blogs.msdn.com/b/rlucero/archive/2011/12/12/speech-101-getting-the-computer-to-recognize-hello-world.aspx
Speech 101 - Using C#: http://blogs.msdn.com/b/rlucero/archive/2012/01/10/speech-101-part-2-using-c-to-recognize-hello-world.aspx

Special thanks to Steve Meyer for reviewing this post and suggesting the blog title!

Speech 101, Part 2 - Using C# to Recognize “Hello World”

Robert Lucero — Tue, 10 Jan 2012 17:09:00 GMT

In my last post I covered how to perform basic speech recognition with a simple grammar and a “Hello World” WAV recording in C++. This post will cover how to accomplish what was in the previous post, but this time using the managed APIs.

Sound Check, Round 2

The support files for this example can be found here. Both HelloWorld.xml and HelloWorld.wav are required.

Again, in addition to the helper files the Windows SDK is also required.

Hello, Hello? Is this C#?

The most basic Speech recognition work flow is this:

Create a Speech Recognition Engine
Load a Grammar
Set the Engine’s input
Handle Recognition Events
Start Recognition

First, launch Visual Studio and create a new Visual C# Console Application. Copy the two helper files to the project root directory, add them to the project and set their properties to “Copy on Build”.

Next, it is time to create a SpeechRecognitionEngine and Grammar:

SpeechRecognitionEngine srEngine = new SpeechRecognitionEngine();

Grammar gram = new Grammar("helloworld.xml");

Now, load the grammar into the engine and set the engine’s input to the sample wave file:

srEngine.LoadGrammar(gram);
srEngine.SetInputToWaveFile("helloworld.wav");

Last thing to do is handle the recognition events and start recognition:

srEngine.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(srEngine_SpeechRecognized);
srEngine.Recognize();

Here’s an easy example for the Recognition Handler:

static void srEngine_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
{
Console.WriteLine("Detected: " + e.Result.Text);
}

That’s it. When the application runs, “Detected: Hello world” should be written to the console window.

Now you have a simple recognition example in just 10 lines of code.

What Next?

Try out some of the SAPI experiments listed in my first Speech 101 post. Also, look at the different recognition events and results.

Resources

MSDN’s Documentation on the Speech Recognition Engine:
http://msdn.microsoft.com/en-us/library/hh361636.aspx

MSDN’s Starting Point for Speech Recognition:
http://msdn.microsoft.com/en-us/library/hh361633.aspx

Speech 101 - Getting the Computer to Recognize “Hello World”

Robert Lucero — Mon, 12 Dec 2011 16:09:00 GMT

I recently started working on the Speech at Microsoft core runtime team. As part of the ramp up process, I needed to write up a simple C++ app using the Speech APIs (SAPI). Below you’ll find a simple walk through in what it takes to write up a native app to analyze input wave files. This sample can be the basis for a simple speech recognition feature in your projects.

Sound Check

What you’ll need for this example is a single wave recording of your voice saying “Hello World”. I’ve made all of the samples available here. The two main things you’ll need are “helloworld.wav” and the corresponding grammar file, “helloworld.xml”. The source is also included as a reference.

In addition, you’ll need the Speech Development APIs which can be found in the Windows SDK. The SDK includes both the managed and native APIs.

Crash Course in Speech Recognition

Before I jump into the example let’s look at what happens Speech Recognition from a very basic level:

The SAPI recognition engine takes a loaded WAV file and with the help of a custom grammar file, can determine if the input WAV file is actually recognizable speech. If the engine detects there is a phrase within the WAV, an event is fired and the resulting string is contained within. Developers need to give some guidance to the SAPI recognition engine through the use of a custom grammar file. To learn more about that in complete detail, W3C’s Grammar RFC contains all of the information.

Hello Computer, Hello World

Once the Windows SDK has been installed, launch Visual Studio and create a C++ Win32 console application project. Copy the two helper files to the project root directory, add them to the project and set the property to “Copy on build”.

First code part - enable COM in the project’s stdfax.h:

#define _ATL_APARTMENT_THREADED
#include <atlbase.h>

extern CComModule _Module;
#include <atlcom.h>

Next, in the main cpp file add the references to SAPI.h and some basic speech objects:

#include <sapi.h>
#include <sphelper.h>

int _tmain(int argc, _TCHAR* argv[])
{
     LPCWSTR MY_WAVE_AUDIO_FILENAME = LPCWSTR(L"helloworld.wav\0");
     LPCWSTR TEST_CFG = LPCWSTR(L"helloworld.xml\0");

     CComPtr<ISpStream> cpInputStream;
     CComPtr<ISpRecognizer> cpRecognizer;
     CComPtr<ISpRecoContext> cpRecoContext;
     CComPtr<ISpRecoGrammar> cpRecoGrammar;
     CComPtr<ISpRecoResult> cpResult;

If you haven’t already, call CoInitialize(NULL):

if(FAILED(::CoInitialize(NULL)))
{
printf("CoInitialize failed... somehow...");
return FALSE;
}

Now, it is time to initialize the input stream and bind it to a file:

    CSpStreamFormat sInputFormat;

     hr = sInputFormat.AssignFormat(SPSF_22kHz16BitStereo);
     if(FAILED(hr))
     {
         printf("input formatting failed");
         return FALSE;
     }

     //bind input stream to a file
    hr = cpInputStream->BindToFile(MY_WAVE_AUDIO_FILENAME,
                                     SPFM_OPEN_READONLY,
                                     &sInputFormat.FormatId(),
                                     sInputFormat.WaveFormatExPtr(),
                                     SPFEI_ALL_EVENTS);
     //hr check
    if(FAILED(hr))
     {
         printf("binding to file failed. check file value, etc... ");
         return FALSE;
     }

Once the input stream is set, initialize the Recognizer and configure its input:

    //Initialize the recognition Engine
    hr = cpRecognizer.CoCreateInstance(CLSID_SpInprocRecognizer);

     //hook up wav input to the recognizer
    hr = cpRecognizer->SetInput(cpInputStream, TRUE);

     //hr check
    if(FAILED(hr))
     {
         printf("Linking WAV to recognizer failed. ");
         return FALSE;
     }

Finish up configuring the SR objects by creating the grammar, setting interest level and wiring up the SR Win32 event:

    //Create recognition context
    hr = cpRecognizer->CreateRecoContext(&cpRecoContext);

     hr = cpRecoContext->CreateGrammar(NULL, &cpRecoGrammar);
     hr = cpRecoGrammar->LoadDictation(NULL, SPLO_STATIC);

     //Check for things that are recognized and the end of the stream
    hr = cpRecoContext->SetInterest(SPFEI_ALL_SR_EVENTS | SPFEI(SPEI_END_SR_STREAM),
                                     SPFEI_ALL_SR_EVENTS | SPFEI(SPEI_END_SR_STREAM));

     //hook up the Win32 event
    hr = cpRecoContext->SetNotifyWin32Event();
     //hr check
    if(FAILED(hr))
     {
         printf("Wiring up win32 Event failed");
         return FALSE;
     }

     //Activate Dictation
    hr = cpRecoGrammar->SetDictationState(SPRS_ACTIVE);

     bool fEndStreamReached = FALSE;

Specifically, the interest level in this example is to “All SR Events”. This means that a SR event will fire for everything that the recognizer thinks it can parse accurately.

Here’s the basic loop for handling the SR Event logic:

    //Do stuff, like analyze the wave or something...
    while(!fEndStreamReached && S_OK == cpRecoContext->WaitForNotifyEvent(10000))
     {
         CSpEvent spEvent;

         //extract queued events from reco context's event queue!
        while(!fEndStreamReached && S_OK == spEvent.GetFrom(cpRecoContext))
         {
             //Figure out the event type
            switch (spEvent.eEventId)
             {
                 //recognized something
                case SPEI_RECOGNITION:
                     //DO SOMETHING!
                    fFoundSomething = TRUE;
                     LPWSTR buffer;

                     cpResult = spEvent.RecoResult();
                     cpResult->GetText(SP_GETWHOLEPHRASE, SP_GETWHOLEPHRASE, TRUE, &buffer, NULL);

                     break;
                 case SPEI_END_SR_STREAM:
                     fEndStreamReached = TRUE;
                     break;
             }

             //Clear the event
            spEvent.Clear();
         }
     }

If everything went well, the buffer in the loop will have “Hello World” when the recognition event is fired. To double check, set a breakpoint on "fFoundSomething” and step through at that point to see if it recognized correctly.

To do things right, here’s the clean up code:

    //CLEAN UP TIME
    hr = cpRecoGrammar->SetDictationState(SPRS_INACTIVE);
     hr = cpRecoGrammar->UnloadDictation();
     hr = cpInputStream->Close();

     return TRUE;

SAPI Experiments To Try

There are bunch of things to try out with this example to try to get a better understanding on how the SR system works. Here’s a quick list of things to try out:

Record your own voice saying “Hello World” and see if it the app correctly recognizes you.
Add more grammar and recognition rules to expand what the engine can find
Comment out the grammar section and see the engine’s best guesses at different WAV inputs.

Resources and Thanks!

You can find the MSDN example, which is what most of this blog post is based on here.

Special thanks goes out my new test colleagues who have helped me learn and guided me through this example!

WebMatrix 2 Beta - Using IIS Remote Manager to Enhance Your Site

Robert Lucero — Tue, 08 Nov 2011 02:46:00 GMT

If your web hosting provider uses is an IIS shared hosting provider chances are you can use IIS Manager for Remote Administration to additionally configure your site. This post will walk you through a simple WebMatrix publish example, connect to the remote...(read more)

WebMatrix 2 Beta - Remote Site Editing

Robert Lucero — Mon, 31 Oct 2011 20:03:52 GMT

Remote site editing is integrated in to WebMatrix 2. This allows for users to quickly edit content from both the original computer where they published from or a completely different computer. Life After Publishing Once you’ve published a site,...(read more)

WebMatrix 2 Beta - NuGet Gallery

Robert Lucero — Thu, 20 Oct 2011 16:36:27 GMT

NuGet is a versatile library package system that allows users to extend their web applications with some really cool tools. WebMatrix V1 users who are familiar with NuGet remember that they had to access the NuGet Package manager through the...(read more)

WebMatrix 2 Beta - Integrating with IIS Developer Express

Robert Lucero — Fri, 14 Oct 2011 16:36:17 GMT

I want to take the time to describe how WebMatrix works with IIS Express, the development web server. I hope that by the end of this article, users will better understand what IIS Express is capable of and how it works with WebMatrix. What...(read more)

WebMatrix 2 Beta - All About Extensions

Robert Lucero — Mon, 26 Sep 2011 17:22:00 GMT

WebMatrix 2 has been rebuilt with extensibility in mind. We have designed it to allow developers to write MEF based extensions that integrate into the WebMatrix experience. Members of the WebMatrix team have individually published some initial extensions...(read more)

WebMatrix 2 Beta–Top 5 Tips and Tricks

Robert Lucero — Tue, 20 Sep 2011 23:40:58 GMT

You’ve probably seen all of the cool high level features, but there are a number of cool features that are worth checking out. Like always, let us know what you think on our forums ! 1. Set Your Default Workspace If you find that you’re always...(read more)

WebMatrix 2 Beta–Released!

Robert Lucero — Fri, 16 Sep 2011 19:01:51 GMT

Today we’re releasing the first public Microsoft WebMatrix 2 Beta! It contains a number of new features and improvements from version 1. Try it out today! Here is just a short list of what’s new and improved: New Features: Remote Site Access...(read more)

WebMatrix Beta – Beta 3 Released!

Robert Lucero — Tue, 09 Nov 2010 17:15:40 GMT

Just a few short weeks ago, we shipped WebMatrix Beta 2 . We fixed a lot of bugs and stabilized the product overall in that release. Beta 3 is different. There are all sorts of new features to try out! Download WebMatrix Beta 3...(read more)

WebMatrix Beta - Beta 2 Released!

Robert Lucero — Thu, 07 Oct 2010 15:11:08 GMT

Yesterday, we released a refreshed Beta 2 for WebMatrix. It contains a number of bug fixes and feature improvements over Beta 1. Highlights include: New and updated Razor templates Feedback and bug fixes as suggested by the community on Microsoft...(read more)

WebMatrix Beta – Database Migration and Live Database Monitoring

Robert Lucero — Mon, 19 Jul 2010 17:04:01 GMT

One of the cool WebMatrix features is the ability to manage other databases from within the WebMatrix Databases Workspace. This post describes a scenario where you can take an existing SQL Compact Edition database from a template site, migrate it...(read more)

WebMatrix Beta – Testing With ‘Real’ Certificates

Robert Lucero — Fri, 16 Jul 2010 16:14:05 GMT

In my last post, one of the tips I described was how to configure a WebMatrix site to use the built-in test SSL certificate. If you enable SSL for a site within WebMatrix and browse to it using the link in the UI, you’ll probably receive a glaring...(read more)

WebMatrix Beta - Tips and Tricks

Robert Lucero — Fri, 09 Jul 2010 16:24:00 GMT

Here’s a short list of really cool features to try out in the new WebMatrix : Use Web Platform Installer v3.0 Beta to install Apps to both WebMatrix and IIS Manage and Maintain Other Databases from WebMatrix Perform SEO Analysis and Review Reports on...(read more)

Client Certificates, Now with a UI!

Robert Lucero — Tue, 13 Oct 2009 16:42:35 GMT

Dave , a test lead at IIS, sent me a link to Puneet Gupta’s blog post about the Client Certificates UI that he developed. One of the more common requests we get on the IIS 7 UI team is for a Client Certificates UI Module. Puneet took advantage of...(read more)

IIS Database Manager – Version 1.0 Released!

Robert Lucero — Fri, 25 Sep 2009 20:04:00 GMT

Today, the IIS team is releasing version 1.0 of the IIS Database Manager. It allows you to administer website and web application databases with the existing connection string or a custom client connection with an IIS UI extension. Major Features: Viewing...(read more)

Certificate Renewals in IIS 7

Robert Lucero — Thu, 28 May 2009 20:53:10 GMT

Certificate renewals have changed from IIS 6 to IIS 7. Specifically, IIS 7 generates a different type of certificate renewal. The way it was… IIS 6 creates a PKCS #10 type of renewal. This is essentially just a new request with the information...(read more)