Kinect Toolbox 1.1 : Template based posture detector and Voice Commander - Eternal Coding - HTML5 / Windows / Kinect / 3D development - Site Home - MSDN Blogs

Kinect Toolbox 1.1 : Template based posture detector and Voice Commander


 

Kinect Toolbox 1.1 : Template based posture detector and Voice Commander

  • Comments 44

In a previous article I introduced the Kinect Toolbox : http://blogs.msdn.com/b/eternalcoding/archive/2011/07/04/gestures-and-tools-for-kinect.aspx.

image

Kinect Toolbox v1.1 is now out and this new version adds support for some cool features:

  • Templated posture detector
  • Voice Commander
  • NuGet package

You can find the toolbox here : http://kinecttoolbox.codeplex.com or you can grad it using NuGet : http://nuget.org/List/Packages/KinectToolbox

Templated posture detector

Using the same algorithm as TemplatedGestureDetector, you can now use a learning machine and a matching system to detect postures. In the sample attached with the toolbox I detect the ”T” posture (i.e. when you body is like the T letter):

image

To do that, I developed a new class : TemplatedPostureDetector which uses an internal learning machine (like the gesture detector) :

public class TemplatedPostureDetector : PostureDetector
{
    const float Epsilon = 0.02f;
    const float MinimalScore = 0.95f;
    const float MinimalSize = 0.1f;
    readonly LearningMachine learningMachine;
    readonly string postureName;

    public LearningMachine LearningMachine
    {
        get { return learningMachine; }
    }

    public TemplatedPostureDetector(string postureName, Stream kbStream) : base(4)
    {
        this.postureName = postureName;
        learningMachine = new LearningMachine(kbStream);
    }

    public override void TrackPostures(ReplaySkeletonData skeleton)
    {
        if (LearningMachine.Match(skeleton.Joints.ToListOfVector2(), Epsilon, MinimalScore, MinimalSize))
            RaisePostureDetected(postureName);
    }

    public void AddTemplate(ReplaySkeletonData skeleton)
    {
        RecordedPath recordedPath = new RecordedPath(skeleton.Joints.Count);

        recordedPath.Points.AddRange(skeleton.Joints.ToListOfVector2());

        LearningMachine.AddPath(recordedPath);
    }

    public void SaveState(Stream kbStream)
    {
        LearningMachine.Persist(kbStream);
    }
}

To use this class, we only need to instantiate it and give it some templates (using the [Capture T] button or using a previously saved file). After that, the class can track postures for each skeleton it receives:

Stream recordStream = File.Open(letterT_KBPath, FileMode.OpenOrCreate);
templatePostureDetector = new TemplatedPostureDetector("T", recordStream);
templatePostureDetector.PostureDetected += templatePostureDetector_PostureDetected;
templatePostureDetector.TrackPostures(skeleton);
void templatePostureDetector_PostureDetected(string posture)
{
    MessageBox.Show("Give me a......." + posture);
}

Voice Commander

One thing worth noting when you develop with Kinect is that you will spend your time getting up and sitting down Sourire. In the previous article, I introduced the replay system which is very useful to record a Kinect session.

But when you are alone, even the recording is painful because you cannot be at the same time in front of the sensor and in front of your keyboard to start/stop the record.

So here enters the Voice Commander (tadam!!). This class can use a list of words and raise an event when it detect one of them (using the microphone array of the sensor). So for example, you can use “record” and “stop” orders to launch and stop the recording session while you stay in front of the sensor!

The code is really simple (thanks to Kinect for Windows SDK and Microsoft Speech Platform SDK):

public class VoiceCommander
{
    const string RecognizerId = "SR_MS_en-US_Kinect_10.0";
    Thread workingThread;
    readonly Choices choices;
    bool isRunning;

    public event Action<string> OrderDetected;

    public VoiceCommander(params string[] orders)
    {
        choices = new Choices();
        choices.Add(orders);
    }

    public void Start()
    {
        workingThread = new Thread(Record);
        workingThread.IsBackground = true;
        workingThread.SetApartmentState(ApartmentState.MTA);
        workingThread.Start();  
    }

    void Record()
    {
        using (KinectAudioSource source = new KinectAudioSource
        {
            FeatureMode = true,
            AutomaticGainControl = false,
            SystemMode = SystemMode.OptibeamArrayOnly
        })
        {
            RecognizerInfo recognizerInfo = SpeechRecognitionEngine.InstalledRecognizers().Where(r => r.Id == RecognizerId).FirstOrDefault();

            if (recognizerInfo == null)
                return;

            SpeechRecognitionEngine speechRecognitionEngine = new SpeechRecognitionEngine(recognizerInfo.Id);

            var gb = new GrammarBuilder {Culture = recognizerInfo.Culture};
            gb.Append(choices);

            var grammar = new Grammar(gb);

            speechRecognitionEngine.LoadGrammar(grammar);
            using (Stream sourceStream = source.Start())
            {
                speechRecognitionEngine.SetInputToAudioStream(sourceStream, new SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, null));

                isRunning = true;
                while (isRunning)
                {
                    RecognitionResult result = speechRecognitionEngine.Recognize();

                    if (result != null && OrderDetected != null && result.Confidence > 0.7)
                        OrderDetected(result.Text);
                }
            }
        }
    }

    public void Stop()
    {
        isRunning = false;
    }
}

Using this class is really simple:

voiceCommander = new VoiceCommander("record", "stop");
voiceCommander.OrderDetected += voiceCommander_OrderDetected;

voiceCommander.Start();
void voiceCommander_OrderDetected(string order)
{
    Dispatcher.Invoke(new Action(() =>
    {
        if (audioControl.IsChecked == false)
            return;

        switch (order)
        {
            case "record":
                DirectRecord(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "kinectRecord" + Guid.NewGuid() + ".replay"));
                break;
            case "stop":
                StopRecord();
                break;
        }
    }));
}

Conclusion

With Kinect Toolbox 1.1, you have a set of tools to help you develop fun and powerful applications with Kinect for Windows SDK!

Leave a Comment
  • Please add 2 and 1 and type the answer here:
  • Post
  • Other than the error about INuiInstanceHelper (which is just a pop-up without a line number), it's the first error.

  • I just tried installing the DirectX SDK and runtime, as well as all Windows Updates on the 32-bit computer... No dice. :(

  • I believe I've discovered the solution to the INuiInstanceHelper problem... I just deleted the files in the Data directory and everything's working now... I guess it makes sense that it was looking for an old version of the assemblies when it tried to restore the serialized data...

    However, I still get a crash now when I click "Stop Recording", after having successfully started by clicking "Capture Circle":

    System.ArgumentOutOfRangeException was unhandled

     Message=Der Index lag außerhalb des Bereichs. Er muss nicht negativ und kleiner als die Auflistung sein.

    Parametername: index

     Source=mscorlib

     ParamName=index

     StackTrace:

          bei System.ThrowHelper.ThrowArgumentOutOfRangeException()

          bei System.Collections.Generic.List`1.get_Item(Int32 index)

          bei Kinect.Toolbox.Gestures.Learning_Machine.GoldenSection.ProjectListToDefinedCount(List`1 positions, Int32 n) in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\Learning Machine\GoldenSection.cs:Zeile 52.

          bei Kinect.Toolbox.Gestures.Learning_Machine.GoldenSection.Pack(List`1 positions, Int32 samplesCount) in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\Learning Machine\GoldenSection.cs:Zeile 109.

          bei Kinect.Toolbox.RecordedPath.CloseAndPrepare() in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\Learning Machine\RecordedPath.cs:Zeile 78.

          bei Kinect.Toolbox.LearningMachine.AddPath(RecordedPath path) in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\Learning Machine\LearningMachine.cs:Zeile 46.

          bei Kinect.Toolbox.TemplatedGestureDetector.EndRecordTemplate() in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\Gestures\TemplatedGestureDetector.cs:Zeile 56.

          bei GesturesViewer.MainWindow.recordCircle_Click(Object sender, RoutedEventArgs e) in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\GesturesViewer\MainWindow.Gestures.cs:Zeile 25.

          bei System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)

          bei System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)

          bei System.Windows.UIElement.RaiseEventImpl(DependencyObject sender, RoutedEventArgs args)

          bei System.Windows.UIElement.RaiseEvent(RoutedEventArgs e)

          bei System.Windows.Controls.Primitives.ButtonBase.OnClick()

          bei System.Windows.Controls.Button.OnClick()

          bei System.Windows.Controls.Primitives.ButtonBase.OnMouseLeftButtonUp(MouseButtonEventArgs e)

          bei System.Windows.UIElement.OnMouseLeftButtonUpThunk(Object sender, MouseButtonEventArgs e)

          bei System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(Delegate genericHandler, Object genericTarget)

          bei System.Windows.RoutedEventArgs.InvokeHandler(Delegate handler, Object target)

          bei System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)

          bei System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)

          bei System.Windows.UIElement.ReRaiseEventAs(DependencyObject sender, RoutedEventArgs args, RoutedEvent newEvent)

          bei System.Windows.UIElement.OnMouseUpThunk(Object sender, MouseButtonEventArgs e)

          bei System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(Delegate genericHandler, Object genericTarget)

          bei System.Windows.RoutedEventArgs.InvokeHandler(Delegate handler, Object target)

          bei System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)

          bei System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)

          bei System.Windows.UIElement.RaiseEventImpl(DependencyObject sender, RoutedEventArgs args)

          bei System.Windows.UIElement.RaiseTrustedEvent(RoutedEventArgs args)

          bei System.Windows.UIElement.RaiseEvent(RoutedEventArgs args, Boolean trusted)

          bei System.Windows.Input.InputManager.ProcessStagingArea()

          bei System.Windows.Input.InputManager.ProcessInput(InputEventArgs input)

          bei System.Windows.Input.InputProviderSite.ReportInput(InputReport inputReport)

          bei System.Windows.Interop.HwndMouseInputProvider.ReportInput(IntPtr hwnd, InputMode mode, Int32 timestamp, RawMouseActions actions, Int32 x, Int32 y, Int32 wheel)

          bei System.Windows.Interop.HwndMouseInputProvider.FilterMessage(IntPtr hwnd, WindowMessage msg, IntPtr wParam, IntPtr lParam, Boolean& handled)

          bei System.Windows.Interop.HwndSource.InputFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)

          bei MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)

          bei MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)

          bei System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)

          bei MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(Object source, Delegate method, Object args, Int32 numArgs, Delegate catchHandler)

          bei System.Windows.Threading.Dispatcher.InvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)

          bei MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)

          bei MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)

          bei System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)

          bei System.Windows.Threading.Dispatcher.PushFrame(DispatcherFrame frame)

          bei System.Windows.Application.RunDispatcher(Object ignore)

          bei System.Windows.Application.RunInternal(Window window)

          bei System.Windows.Application.Run(Window window)

          bei System.Windows.Application.Run()

          bei GesturesViewer.App.Main() in C:\Users\klein\Downloads\Kinect\kinecttoolbox-91786\GesturesViewer\obj\x86\Debug\App.g.cs:Zeile 0.

          bei System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

          bei System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

          bei Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

          bei System.Threading.ThreadHelper.ThreadStart_Context(Object state)

          bei System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

          bei System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

          bei System.Threading.ThreadHelper.ThreadStart()

     InnerException:

  • For your info, even though I had installed all pre-requisite software I was receiving error messages when lunching your GestureViewer app. The error I was getting is the following: "Unable to find assembly 'Kinect.Toolkit, Version=1.0.1.0 ...", the application window would still launch but I would then get nullReferenceExceptions when starting to move in front of the kinect.

    I checked the AssemblyInfo.cs of the Kinect.Toolbox project and there I found the following reference: assembly: AssemblyVersion("1.1.0.1") which was different from the error message I got. I then thought that it was most likely coming from something being obviously created in an earlier version of the assembly and by reading at Mike's post above, I got an idea and decided to remove all Speech-Related code from the projects and that solved my problem.

    Hope this can help others.

    By the way. Thanks David C. for the great work !

  • sorry.. correction.. I did not removed all Speech-Related code.. but the recording files (.save) in the various "data" folders.

  • Yes I think some old references lie in the .save files:)

  • Hi David,

    I looked around but couldn't find the answer to the following question: Is it possible to use the Microsoft Speech Platform SDK in French ? Meaning I would like to talk to my kinect in French. Is it possible ? I googled SR_MS_fr-FR_Kinect_10.0 without success.

    Thanks for your time.

    Regards, David

  • Hi you can find all that you need here:

    www.microsoft.com/.../details.aspx

  • Hi

    I have download this Kinect toolbox and will be using it wit my Kinect SDK for windows beta2

    im trying to familiarize with the codes, i have downloaded the sources but where could i get the samples?

    I need to see some samples to learn

    Please help

    Thanks

    Irfan

  • On the kinecttoolbox.codeplex.com site you can download a sample called Gestures Viewer available in the samples link:

    kinecttoolbox.codeplex.com/.../76297

  • Hi David,

    I am trying to implement new gestures using two hands. You said in the previous article that "you just have to plot the positions of each hand in the template."

    Could you tell me how you would do that more precisely please?

    Thank you

  • You can instanciate 2 TemplateGestureDetectors and synchronize them to detect on combined gestures.

  • HI david,

    I actually dont understand what to do?

    Here is what i have done, I have build a software for the kinect, when the program starts, the mousecursor of the dekstop is in my control.

    But what i want tto do is, implement ur "T" posture, so when the T is recognized the left hand should be able to utilse the program. Basically that is done. What i dont understand is how to build ur code with my code?

        private void Window_Loaded(object sender, RoutedEventArgs e)

           {

               try

               {

                   nui.Initialize(RuntimeOptions.UseSkeletalTracking);

               }

               catch (Exception ex)

               {

                   MessageBox.Show("Could not initialize Kinect device: " + ex.Message);

               }

               #region Transformsmooth

                        nui.SkeletonEngine.TransformSmooth = true;

                        var parameters = new TransformSmoothParameters

                       {

                                 Smoothing = 0.10f,

                                 Correction = 0.90f,

                                 Prediction = 0.10f,

                                 JitterRadius = 1.00f,

                                 MaxDeviationRadius = 0.5f

                        };

                       nui.SkeletonEngine.SmoothParameters = parameters;

               #endregion

               // event to receive data

               nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

           }

           void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)

           {

               SkeletonFrame allSkeletons = e.SkeletonFrame;

               foreach (SkeletonData s in allSkeletons.Skeletons)

               {

                   if (s.TrackingState == SkeletonTrackingState.Tracked)

                   {

                       foreach (Joint joint in s.Joints)

                       {

                           Joint scaledHandRight = s.Joints[JointID.HandRight].ScaleTo((int)SystemParameters.PrimaryScreenWidth, (int)SystemParameters.PrimaryScreenHeight, SkeletonMaxX, SkeletonMaxY);

                           Joint scaledHandLeft = s.Joints[JointID.HandLeft].ScaleTo((int)SystemParameters.PrimaryScreenWidth, (int)SystemParameters.PrimaryScreenHeight, SkeletonMaxX, SkeletonMaxY);

                           int xCoordinate = (int)scaledHandRight.Position.X;

                           int yCoordinate = (int)scaledHandRight.Position.Y;

                           bool clicked;

                          MouseHook.MoveMouse(new Point(scaledHandRight.Position.X, scaledHandRight.Position.Y));

                       }

                   }

               }

           }

    I hope u can help me out

    thx

  • I don't understand your problem

    Have you try to use the gesture viewer sample? (in the codeplex site)

  • When i start my application,  the skeletal tracker tracks my left and right hand, automatically the left hand can move the mouse cursor.

    I would like to begin the program like this:

            The kinect sees the persons infront of the camera, but it selects the right hand of the person, that is Waving or making the "T" posture.

             So after the kinect sees the person who is waving in this example, can move the mouse cursor and can navigate in the application.

    I have tried the gestureviewer, but i dont understand how to build this.

Page 2 of 3 (44 items) 123