Kinect Toolbox 1.1 : Template based posture detector and Voice Commander

David Catuhe

2 Aug 2011 6:23 AM

Comments 44

In a previous article I introduced the Kinect Toolbox : http://blogs.msdn.com/b/eternalcoding/archive/2011/07/04/gestures-and-tools-for-kinect.aspx.

Kinect Toolbox v1.1 is now out and this new version adds support for some cool features:

Templated posture detector
Voice Commander
NuGet package

You can find the toolbox here : http://kinecttoolbox.codeplex.com or you can grad it using NuGet : http://nuget.org/List/Packages/KinectToolbox

Templated posture detector

Using the same algorithm as TemplatedGestureDetector, you can now use a learning machine and a matching system to detect postures. In the sample attached with the toolbox I detect the ”T” posture (i.e. when you body is like the T letter):

To do that, I developed a new class : TemplatedPostureDetector which uses an internal learning machine (like the gesture detector) :

public class TemplatedPostureDetector : PostureDetector
 {
     const float Epsilon = 0.02f;
     const float MinimalScore = 0.95f;
     const float MinimalSize = 0.1f;
     readonly LearningMachine learningMachine;
     readonly string postureName;
 
     public LearningMachine LearningMachine
     {
         get { return learningMachine; }
     }
 
     public TemplatedPostureDetector(string postureName, Stream kbStream) : base(4)
     {
         this.postureName = postureName;
         learningMachine = new LearningMachine(kbStream);
     }
 
     public override void TrackPostures(ReplaySkeletonData skeleton)
     {
         if (LearningMachine.Match(skeleton.Joints.ToListOfVector2(), Epsilon, MinimalScore, MinimalSize))
             RaisePostureDetected(postureName);
     }
 
     public void AddTemplate(ReplaySkeletonData skeleton)
     {
         RecordedPath recordedPath = new RecordedPath(skeleton.Joints.Count);
 
         recordedPath.Points.AddRange(skeleton.Joints.ToListOfVector2());
 
         LearningMachine.AddPath(recordedPath);
     }
 
     public void SaveState(Stream kbStream)
     {
         LearningMachine.Persist(kbStream);
     }
 }

To use this class, we only need to instantiate it and give it some templates (using the [Capture T] button or using a previously saved file). After that, the class can track postures for each skeleton it receives:

Stream recordStream = File.Open(letterT_KBPath, FileMode.OpenOrCreate);
 templatePostureDetector = new TemplatedPostureDetector("T", recordStream);
 templatePostureDetector.PostureDetected += templatePostureDetector_PostureDetected;

templatePostureDetector.TrackPostures(skeleton);

void templatePostureDetector_PostureDetected(string posture)
 {
     MessageBox.Show("Give me a......." + posture);
 }

Voice Commander

One thing worth noting when you develop with Kinect is that you will spend your time getting up and sitting down Sourire . In the previous article, I introduced the replay system which is very useful to record a Kinect session.

But when you are alone, even the recording is painful because you cannot be at the same time in front of the sensor and in front of your keyboard to start/stop the record.

So here enters the Voice Commander (tadam!!). This class can use a list of words and raise an event when it detect one of them (using the microphone array of the sensor). So for example, you can use “record” and “stop” orders to launch and stop the recording session while you stay in front of the sensor!

The code is really simple (thanks to Kinect for Windows SDK and Microsoft Speech Platform SDK):

public class VoiceCommander
 {
     const string RecognizerId = "SR_MS_en-US_Kinect_10.0";
     Thread workingThread;
     readonly Choices choices;
     bool isRunning;
 
     public event Action<string> OrderDetected;
 
     public VoiceCommander(params string[] orders)
     {
         choices = new Choices();
         choices.Add(orders);
     }
 
     public void Start()
     {
         workingThread = new Thread(Record);
         workingThread.IsBackground = true;
         workingThread.SetApartmentState(ApartmentState.MTA);
         workingThread.Start();  
     }
 
     void Record()
     {
         using (KinectAudioSource source = new KinectAudioSource
         {
             FeatureMode = true,
             AutomaticGainControl = false,
             SystemMode = SystemMode.OptibeamArrayOnly
         })
         {
             RecognizerInfo recognizerInfo = SpeechRecognitionEngine.InstalledRecognizers().Where(r => r.Id == RecognizerId).FirstOrDefault();
 
             if (recognizerInfo == null)
                 return;
 
             SpeechRecognitionEngine speechRecognitionEngine = new SpeechRecognitionEngine(recognizerInfo.Id);
 
             var gb = new GrammarBuilder {Culture = recognizerInfo.Culture};
             gb.Append(choices);
 
             var grammar = new Grammar(gb);
 
             speechRecognitionEngine.LoadGrammar(grammar);
             using (Stream sourceStream = source.Start())
             {
                 speechRecognitionEngine.SetInputToAudioStream(sourceStream, new SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, null));
 
                 isRunning = true;
                 while (isRunning)
                 {
                     RecognitionResult result = speechRecognitionEngine.Recognize();
 
                     if (result != null && OrderDetected != null && result.Confidence > 0.7)
                         OrderDetected(result.Text);
                 }
             }
         }
     }
 
     public void Stop()
     {
         isRunning = false;
     }
 }

Using this class is really simple:

voiceCommander = new VoiceCommander("record", "stop");
 voiceCommander.OrderDetected += voiceCommander_OrderDetected;
 
 voiceCommander.Start();

void voiceCommander_OrderDetected(string order)
 {
     Dispatcher.Invoke(new Action(() =>
     {
         if (audioControl.IsChecked == false)
             return;
 
         switch (order)
         {
             case "record":
                 DirectRecord(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "kinectRecord" + Guid.NewGuid() + ".replay"));
                 break;
             case "stop":
                 StopRecord();
                 break;
         }
     }));
 }

Conclusion

With Kinect Toolbox 1.1, you have a set of tools to help you develop fun and powerful applications with Kinect for Windows SDK!

Kinect, Gestures