The Kinect for Windows SDK has finally arrived, and we couldn’t wait to get our hands on it!

After downloading, installing, and actually getting a Kinect with a USB connector for our PC’s, we were good to go…

Step 1 – What does the SDK give us?

When we started, we didn’t really know what to expect from the SDK.

What you get are a set of 20 points, each corresponding to a specific joint in the body. These points come in standard ‘x, y, z’ coordinates, with an extra coordinate ‘w’, which we haven’t got round to figuring out what it is there for just yet.

Here comes problem one. The points that are given are within the range -1 and 1, going from left to right for x, and bottom to top with y. Therefore, you have to first convert these to work with WPF applications so that if fits within the coordinates of a standard page.

This part was just a formality and had a very simple fix with a couple of converters.

One for X:

   1: /// <summary>
   2: /// Joint converter
   3: /// </summary>
   4: public class KinectValueToScreenCoOrdinatesConverterX : IValueConverter
   5: {
   6:     /// <summary>
   7:     /// Converts a value.
   8:     /// </summary>
   9:     /// <param name="value">The value produced by the binding source.</param>
  10:     /// <param name="targetType">The type of the binding target property.</param>
  11:     /// <param name="parameter">The converter parameter to use.</param>
  12:     /// <param name="culture">The culture to use in the converter.</param>
  13:     /// <returns>
  14:     /// A converted value. If the method returns null, the valid null value is used.
  15:     /// </returns>
  16:     public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
  17:     {
  18:         double multiplier = Application.Current.MainWindow.Width;
  19:  
  20:         double translation = 0;
  21:         try
  22:         {
  23:             if (parameter != null)
  24:             {
  25:                 translation = double.Parse(parameter as string);
  26:             }
  27:  
  28:             float fl = (float)value;
  29:             return translation + (multiplier + (fl * multiplier));
  30:        }
  31:        catch (InvalidCastException)
  32:        {
  33:             return 0;
  34:         }
  35:     }
  36:  
  37:     /// <summary>
  38:     /// Converts a value.
  39:     /// </summary>
  40:     /// <param name="value">The value that is produced by the binding target.</param>
  41:     /// <param name="targetType">The type to convert to.</param>
  42:     /// <param name="parameter">The converter parameter to use.</param>
  43:     /// <param name="culture">The culture to use in the converter.</param>
  44:     /// <returns>
  45:     /// A converted value. If the method returns null, the valid null value is used.
  46:     /// </returns>
  47:     public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
  48:     {
  49:         throw new NotImplementedException();
  50:     }
  51: }

And one for Y:

   1: /// <summary>
   2: /// Joint converter Y
   3: /// </summary>
   4: public class KinectValueToScreenCoOrdinatesConverterY : IValueConverter
   5: {
   6:     /// <summary>
   7:     /// Converts a value.
   8:     /// </summary>
   9:     /// <param name="value">The value produced by the binding source.</param>
  10:     /// <param name="targetType">The type of the binding target property.</param>
  11:     /// <param name="parameter">The converter parameter to use.</param>
  12:     /// <param name="culture">The culture to use in the converter.</param>
  13:     /// <returns>
  14:     /// A converted value. If the method returns null, the valid null value is used.
  15:     /// </returns>
  16:     public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
  17:     {
  18:         double multiplier = Application.Current.MainWindow.Height;
  19:  
  20:             double translation = 0;
  21:             try
  22:             {
  23:                 if (parameter != null)
  24:                 {
  25:                     translation = double.Parse(parameter as string);
  26:                 }
  27:  
  28:                 float fl = (float)value;
  29:                 return translation + (multiplier + (-fl * multiplier));
  30:             }
  31:             catch (InvalidCastException)
  32:             {
  33:                 return 0;
  34:             }
  35:         }
  36:  
  37:         /// <summary>
  38:         /// Converts a value.
  39:         /// </summary>
  40:         /// <param name="value">The value that is produced by the binding target.</param>
  41:         /// <param name="targetType">The type to convert to.</param>
  42:         /// <param name="parameter">The converter parameter to use.</param>
  43:         /// <param name="culture">The culture to use in the converter.</param>
  44:         /// <returns>
  45:         /// A converted value. If the method returns null, the valid null value is used.
  46:         /// </returns>
  47:         public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
  48:         {
  49:             throw new NotImplementedException();
  50:         }
  51:     }

A converter is assigned to every part of the skeleton that is displayed on the UI. Since Kinect returns a series of skeletons (each one representing a person) with the points already mapped to each joint, we decided to drive the MVVM pattern directly off of the SkeletonData that Kinect returns.

Step 2 – Putting everything into the MVVM pattern

To simplify things and make it easier to create multiple skeletons on screen at any one time, we decided to use the MVVM pattern. There are four main parts to this:

1. The Kinect connection

2. The Main View Model

3. The Skeleton View Model

4. The UI

Kinect Connection

The Kinect connection is an intermediate class between our view models and the Kinect sensor. It fires Skeleton ready events for each skeleton that is returned and skeleton frame complete events when an entire frame has been processed. As it processes the skeletons in a frame, it keeps track of their ID value so we can track all the skeletons that were in a frame during the Skeleton Frame Complete event.

One of the main parts of this class is the InitializeNui() method:

   1: /// <summary>
   2:         /// Initializes the Kinect sensor.
   3:         /// </summary>
   4:         /// <returns>bool value true if the sensor initialised correctly</returns>
   5:         private bool InitializeNui()
   6:         {
   7:             if (this.kinectRunTime == null)
   8:             {
   9:                 return false;
  10:             }
  11:  
  12:             try
  13:             {
  14:                 this.kinectRunTime.Initialize(RuntimeOptions.UseDepth | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);
  15:             }
  16:             catch (Exception exception)
  17:             {
  18:                 Console.WriteLine(exception.ToString());
  19:                 return false;
  20:             }
  21:  
  22:             this.kinectRunTime.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
  23:  
  24:             this.kinectRunTime.SkeletonEngine.TransformSmooth = true;
  25:  
  26:             var parameters = new TransformSmoothParameters
  27:             {
  28:                 Smoothing = 0.75f,
  29:                 Correction = 0.0f,
  30:                 Prediction = 0.0f,
  31:                 JitterRadius = 0.05f,
  32:                 MaxDeviationRadius = 0.04f
  33:             };
  34:  
  35:             this.kinectRunTime.SkeletonEngine.SmoothParameters = parameters;
  36:  
  37:             return true;
  38:         }
  39:     }

This method initialises the Kinect sensor with a set of pre define arguments. The actual initialization of the sensor happens in the line :

   1: this.kinectRunTime.Initialize(RuntimeOptions.UseDepth | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

The runtime options specify what you want the Kinect sensor to look for. In our example we were looking for skeleton frames and images from the camera.

Main View Model

The main view model controls the allocation of skeletons to the skeleton view models. It contains a list of Skeleton View models and a dictionary of skeleton view models with the skeleton id as the key. Each time a skeleton ready event is fired in the Kinect connection it looks up the skeleton ID value in the dictionary and assigns the new skeleton data to the skeleton in that view model.

Originally the Main view model was also going to sort out the removal of skeletons that are no longer active. However the Kinect sensor does not always return all of the active skeletons in each of the Skelton frame ready events that the Kinect SDK produce. This means that a simple removal of any skeletons not in the current frame will fail and the view model associated with a skeleton will be continuously deleted and re-created. To get around this we placed a timer in each of the skeleton view models. If the view model is not updated within a set period of time the view model raises an event and the main view model then catches it and deleted the skeleton.

The Skeleton View Model

The skeleton view model is essentially a way of exposing the properties of a SkeletonData object so that the UI can bind to them. When the skeleton property is changed a NotifyPropertyChanged event is raised on all of the exposed properties which updates all the bindings on the UI.

The skeleton view model also controls the Gesture Service which we will discuss in our next blog post. This is again updated when the skeleton property changes.

The full code for the skeleton view model can be found in the source code below:

   1: /// <summary>
   2:     /// The main view model
   3:     /// </summary>
   4:     public class SkeletonViewModel : INotifyPropertyChanged
   5:     {
   6:         /// <summary>
   7:         /// The gesture controler for this skeleton
   8:         /// </summary>
   9:         private GestureControler gestures = new GestureControler();
  10:  
  11:         #region fields
  12:  
  13:         /// <summary>
  14:         /// backing field for the gesture text
  15:         /// </summary>
  16:         private string gestureText;
  17:  
  18:         /// <summary>
  19:         /// backing field for the gesture property
  20:         /// </summary>
  21:         private bool gestureDetected;
  22:  
  23:         /// <summary>
  24:         /// The skeleton data
  25:         /// This is used as the backing field for all properties
  26:         /// </summary>
  27:         private SkeletonData skeleton;
  28:  
  29:         /// <summary>
  30:         /// backing field for the joint color
  31:         /// </summary>
  32:         private Color jointColor;
  33:  
  34:         /// <summary>
  35:         /// Occurs when [delete skeleton].
  36:         /// </summary>
  37:         public event EventHandler DeleteSkeleton;
  38:         
  39:         /// <summary>
  40:         /// the timer for the on screen gesture text
  41:         /// </summary>
  42:         private DispatcherTimer textTimer = new DispatcherTimer() { Interval = TimeSpan.FromSeconds(3) };
  43:  
  44:         #endregion
  45:  
  46:         #region constructors
  47:  
  48:         /// <summary>
  49:         /// Initializes a new instance of the <see cref="SkeletonViewModel"/> class.
  50:         /// </summary>
  51:         public SkeletonViewModel()
  52:         {
  53:             this.ChangeTimer = new DispatcherTimer();
  54:             this.ChangeTimer.Interval = TimeSpan.FromSeconds(0.5);
  55:             this.ChangeTimer.Tick += new EventHandler(this.ChangeTimer_Tick);
  56:             this.textTimer.Tick += new EventHandler(this.TextTimer_Tick);
  57:             ChangeTimer.Start();
  58:             DefineGestures();
  59:             this.gestures.GestureRecognised += new EventHandler<GestureEventArgs>(this.Gestures_GestureRecognised);
  60:         }
  61:  
  62:         #endregion
  63:  
  64:         #region events
  65:  
  66:         /// <summary>
  67:         /// Occurs when a property value changes.
  68:         /// </summary>
  69:         public event PropertyChangedEventHandler PropertyChanged;
  70:  
  71:         #endregion
  72:  
  73:         #region properties
  74:  
  75:         /// <summary>
  76:         /// Gets or sets the change timer.
  77:         /// </summary>
  78:         /// <value>
  79:         /// The change timer.
  80:         /// </value>
  81:         public DispatcherTimer ChangeTimer
  82:         {
  83:             get;
  84:             set;
  85:         }
  86:  
  87:         /// <summary>
  88:         /// Gets or sets the gesture text.
  89:         /// </summary>
  90:         /// <value>
  91:         /// The gesture text.
  92:         /// </value>
  93:         public string GestureText
  94:         {
  95:             get
  96:             {
  97:                 return this.gestureText;
  98:             }
  99:  
 100:             set
 101:             {
 102:                 if (this.gestureText != value)
 103:                 {
 104:                     this.gestureText = value;
 105:                     this.NotifyPropertyChanged("GestureText");
 106:                 }
 107:             }
 108:         }
 109:  
 110:         /// <summary>
 111:         /// Gets or sets a value indicating whether this <see cref="SkeletonViewModel"/> is waved.
 112:         /// </summary>
 113:         /// <value>
 114:         ///   <c>true</c> if waved; otherwise, <c>false</c>.
 115:         /// </value>
 116:         public bool GestureDetected
 117:         {
 118:             get
 119:             {
 120:                 return this.gestureDetected;
 121:             }
 122:  
 123:             set
 124:             {
 125:                 if (this.gestureDetected != value)
 126:                 {
 127:                     this.gestureDetected = value;
 128:                     this.NotifyPropertyChanged("GestureDetected");
 129:                 }
 130:             }
 131:         }
 132:  
 133:         /// <summary>
 134:         /// Gets or sets the color of the joint.
 135:         /// </summary>
 136:         /// <value>
 137:         /// The color of the joint.
 138:         /// </value>
 139:         public Color JointColor
 140:         {
 141:             get
 142:             {
 143:                 return this.jointColor;
 144:             }
 145:  
 146:             set
 147:             {
 148:                 if (this.jointColor != value)
 149:                 {
 150:                     this.jointColor = value;
 151:                     this.NotifyPropertyChanged("JointColor");
 152:                 }
 153:             }
 154:         }
 155:  
 156:         /// <summary>
 157:         /// Gets or sets the skeleton.
 158:         /// </summary>
 159:         /// <value>
 160:         /// The skeleton.
 161:         /// </value>
 162:         public SkeletonData Skeleton
 163:         {
 164:             get
 165:             {
 166:                 return this.skeleton;
 167:             }
 168:  
 169:             set
 170:             {
 171:                 if (this.skeleton != value)
 172:                 {
 173:                     this.skeleton = value;
 174:                     this.NotifyPropertyChanged("Skeleton");
 175:                     this.NotifyAllChange();
 176:                     ResetTimer();
 177:                     this.gestures.UpdateAllGestures(this.skeleton);
 178:                 }
 179:             }
 180:         }
 181:  
 182:         #region KinectBodyParts
 183:  
 184:         /// <summary>
 185:         /// Gets the head.
 186:         /// </summary>
 187:         public Joint Head
 188:         {
 189:             get
 190:             {
 191:                 if (this.Skeleton != null)
 192:                 {
 193:                     return this.Skeleton.Joints[JointID.Head];
 194:                 }
 195:                 else
 196:                 {
 197:                     return new Joint();
 198:                 }
 199:             }
 200:         }
 201:  
 202:         /// <summary>
 203:         /// Gets the left hand.
 204:         /// </summary>
 205:         public Joint LeftHand
 206:         {
 207:             get
 208:             {
 209:                 if (this.Skeleton != null)
 210:                 {
 211:                     return this.Skeleton.Joints[JointID.HandLeft];
 212:                 }
 213:                 else
 214:                 {
 215:                     return new Joint();
 216:                 }
 217:             }
 218:         }
 219:  
 220:         /// <summary>
 221:         /// Gets the right hand.
 222:         /// </summary>
 223:         public Joint RightHand
 224:         {
 225:             get
 226:             {
 227:                 if (this.Skeleton != null)
 228:                 {
 229:                     return this.Skeleton.Joints[JointID.HandRight];
 230:                 }
 231:                 else
 232:                 {
 233:                     return new Joint();
 234:                 }
 235:             }
 236:         }
 237:  
 238:         /// <summary>
 239:         /// Gets the left wrist.
 240:         /// </summary>
 241:         public Joint LeftWrist
 242:         {
 243:             get
 244:             {
 245:                 if (this.Skeleton != null)
 246:                 {
 247:                     return this.Skeleton.Joints[JointID.WristLeft];
 248:                 }
 249:                 else
 250:                 {
 251:                     return new Joint();
 252:                 }
 253:             }
 254:         }
 255:  
 256:         /// <summary>
 257:         /// Gets the right wrist.
 258:         /// </summary>
 259:         public Joint RightWrist
 260:         {
 261:             get
 262:             {
 263:                 if (this.Skeleton != null)
 264:                 {
 265:                     return this.Skeleton.Joints[JointID.WristRight];
 266:                 }
 267:                 else
 268:                 {
 269:                     return new Joint();
 270:                 }
 271:             }
 272:         }
 273:  
 274:         /// <summary>
 275:         /// Gets the left elbow.
 276:         /// </summary>
 277:         public Joint LeftElbow
 278:         {
 279:             get
 280:             {
 281:                 if (this.Skeleton != null)
 282:                 {
 283:                     return this.Skeleton.Joints[JointID.ElbowLeft];
 284:                 }
 285:                 else
 286:                 {
 287:                     return new Joint();
 288:                 }
 289:             }
 290:         }
 291:  
 292:         /// <summary>
 293:         /// Gets the right elbow.
 294:         /// </summary>
 295:         public Joint RightElbow
 296:         {
 297:             get
 298:             {
 299:                 if (this.Skeleton != null)
 300:                 {
 301:                     return this.Skeleton.Joints[JointID.ElbowRight];
 302:                 }
 303:                 else
 304:                 {
 305:                     return new Joint();
 306:                 }
 307:             }
 308:         }
 309:  
 310:         /// <summary>
 311:         /// Gets the shoulder center.
 312:         /// </summary>
 313:         public Joint ShoulderCenter
 314:         {
 315:             get
 316:             {
 317:                 if (this.Skeleton != null)
 318:                 {
 319:                     return this.Skeleton.Joints[JointID.ShoulderCenter];
 320:                 }
 321:                 else
 322:                 {
 323:                     return new Joint();
 324:                 }
 325:             }
 326:         }
 327:  
 328:         /// <summary>
 329:         /// Gets the spine.
 330:         /// </summary>
 331:         public Joint Spine
 332:         {
 333:             get
 334:             {
 335:                 if (this.Skeleton != null)
 336:                 {
 337:                     return this.Skeleton.Joints[JointID.Spine];
 338:                 }
 339:                 else
 340:                 {
 341:                     return new Joint();
 342:                 }
 343:             }
 344:         }
 345:  
 346:         /// <summary>
 347:         /// Gets the hip center.
 348:         /// </summary>
 349:         public Joint HipCenter
 350:         {
 351:             get
 352:             {
 353:                 if (this.Skeleton != null)
 354:                 {
 355:                     return this.Skeleton.Joints[JointID.HipCenter];
 356:                 }
 357:                 else
 358:                 {
 359:                     return new Joint();
 360:                 }
 361:             }
 362:         }
 363:  
 364:         /// <summary>
 365:         /// Gets the left knee.
 366:         /// </summary>
 367:         public Joint LeftKnee
 368:         {
 369:             get
 370:             {
 371:                 if (this.Skeleton != null)
 372:                 {
 373:                     return this.Skeleton.Joints[JointID.KneeLeft];
 374:                 }
 375:                 else
 376:                 {
 377:                     return new Joint();
 378:                 }
 379:             }
 380:         }
 381:  
 382:         /// <summary>
 383:         /// Gets the right knee.
 384:         /// </summary>
 385:         public Joint RightKnee
 386:         {
 387:             get
 388:             {
 389:                 if (this.Skeleton != null)
 390:                 {
 391:                     return this.Skeleton.Joints[JointID.KneeRight];
 392:                 }
 393:                 else
 394:                 {
 395:                     return new Joint();
 396:                 }
 397:             }
 398:         }
 399:  
 400:         /// <summary>
 401:         /// Gets the left ankle.
 402:         /// </summary>
 403:         public Joint LeftAnkle
 404:         {
 405:             get
 406:             {
 407:                 if (this.Skeleton != null)
 408:                 {
 409:                     return this.Skeleton.Joints[JointID.AnkleLeft];
 410:                 }
 411:                 else
 412:                 {
 413:                     return new Joint();
 414:                 }
 415:             }
 416:         }
 417:  
 418:         /// <summary>
 419:         /// Gets the right ankle.
 420:         /// </summary>
 421:         public Joint RightAnkle
 422:         {
 423:             get
 424:             {
 425:                 if (this.Skeleton != null)
 426:                 {
 427:                     return this.Skeleton.Joints[JointID.AnkleRight];
 428:                 }
 429:                 else
 430:                 {
 431:                     return new Joint();
 432:                 }
 433:             }
 434:         }
 435:  
 436:         /// <summary>
 437:         /// Gets the left foot.
 438:         /// </summary>
 439:         public Joint LeftFoot
 440:         {
 441:             get
 442:             {
 443:                 if (this.Skeleton != null)
 444:                 {
 445:                     return this.Skeleton.Joints[JointID.FootLeft];
 446:                 }
 447:                 else
 448:                 {
 449:                     return new Joint();
 450:                 }
 451:             }
 452:         }
 453:  
 454:         /// <summary>
 455:         /// Gets the right foot.
 456:         /// </summary>
 457:         public Joint RightFoot
 458:         {
 459:             get
 460:             {
 461:                 if (this.Skeleton != null)
 462:                 {
 463:                     return this.Skeleton.Joints[JointID.FootRight];
 464:                 }
 465:                 else
 466:                 {
 467:                     return new Joint();
 468:                 }
 469:             }
 470:         }
 471:  
 472:         /// <summary>
 473:         /// Gets the Left Hip
 474:         /// </summary>
 475:         public Joint RightHip
 476:         {
 477:             get
 478:             {
 479:                 if (this.Skeleton != null)
 480:                 {
 481:                     return this.Skeleton.Joints[JointID.HipRight];
 482:                 }
 483:                 else
 484:                 {
 485:                     return new Joint();
 486:                 }
 487:             }
 488:         }
 489:  
 490:         /// <summary>
 491:         /// Gets the Left Hip
 492:         /// </summary>
 493:         public Joint LeftHip
 494:         {
 495:             get
 496:             {
 497:                 if (this.Skeleton != null)
 498:                 {
 499:                     return this.Skeleton.Joints[JointID.HipLeft];
 500:                 }
 501:                 else
 502:                 {
 503:                     return new Joint();
 504:                 }
 505:             }
 506:         }
 507:  
 508:         /// <summary>
 509:         /// Gets the Right Shoulder
 510:         /// </summary>
 511:         public Joint RightShoulder
 512:         {
 513:             get
 514:             {
 515:                 if (this.Skeleton != null)
 516:                 {
 517:                     return this.Skeleton.Joints[JointID.ShoulderRight];
 518:                 }
 519:                 else
 520:                 {
 521:                     return new Joint();
 522:                 }
 523:             }
 524:         }
 525:  
 526:         /// <summary>
 527:         /// Gets the Left Shoulder
 528:         /// </summary>
 529:         public Joint LeftShoulder
 530:         {
 531:             get
 532:             {
 533:                 if (this.Skeleton != null)
 534:                 {
 535:                     return this.Skeleton.Joints[JointID.ShoulderLeft];
 536:                 }
 537:                 else
 538:                 {
 539:                     return new Joint();
 540:                 }
 541:             }
 542:         }
 543:         #endregion
 544:  
 545:         #endregion
 546:  
 547:         #region methods
 548:  
 549:         /// <summary>
 550:         /// Handles the Tick event of the textTimer control.
 551:         /// </summary>
 552:         /// <param name="sender">The source of the event.</param>
 553:         /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
 554:         private void TextTimer_Tick(object sender, EventArgs e)
 555:         {
 556:             this.GestureDetected = false;
 557:             this.textTimer.Stop();
 558:         }
 559:  
 560:         /// <summary>
 561:         /// Handles the GestureRecognised event of the Gestures control.
 562:         /// </summary>
 563:         /// <param name="sender">The source of the event.</param>
 564:         /// <param name="e">The <see cref="KinectSkeltonTracker.GestureEventArgs"/> instance containing the event data.</param>
 565:         private void Gestures_GestureRecognised(object sender, GestureEventArgs e)
 566:         {
 567:             if (e.GestureType == GestureType.WaveRight)
 568:             {
 569:                 this.GestureDetected = true;
 570:                 this.GestureText = "Waved with right hand";
 571:                 this.textTimer.Start();
 572:             }
 573:             else if (e.GestureType == GestureType.WaveLeft)
 574:             {
 575:                 this.GestureDetected = true;
 576:                 this.GestureText = "Waved with left hand";
 577:                 this.textTimer.Start();
 578:             }
 579:             else if (e.GestureType == GestureType.LeftSwipe)
 580:             {
 581:                 this.GestureDetected = true;
 582:                 this.GestureText = "Swiped left";
 583:                 this.textTimer.Start();
 584:             }
 585:             else if (e.GestureType == GestureType.RightSwipe)
 586:             {
 587:                 this.GestureDetected = true;
 588:                 this.GestureText = "Swiped right";
 589:                 this.textTimer.Start();
 590:             }
 591:             else if (e.GestureType == GestureType.Menu)
 592:             {
 593:                 this.GestureDetected = true;
 594:                 this.GestureText = "Menu";
 595:                 this.textTimer.Start();
 596:             }
 597:         }
 598:  
 599:         /// <summary>
 600:         /// Defines the gestures.
 601:         /// </summary>
 602:         private void DefineGestures()
 603:         {
 604:             IRelativeGestureSegment[] waveRightSegments = new IRelativeGestureSegment[6];
 605:             WaveRightSegment1 waveRightSegment1 = new WaveRightSegment1();
 606:             WaveRightSegment2 waveRightSegment2 = new WaveRightSegment2();
 607:             waveRightSegments[0] = waveRightSegment1;
 608:             waveRightSegments[1] = waveRightSegment2;
 609:             waveRightSegments[2] = waveRightSegment1;
 610:             waveRightSegments[3] = waveRightSegment2;
 611:             waveRightSegments[4] = waveRightSegment1;
 612:             waveRightSegments[5] = waveRightSegment2;
 613:             this.gestures.AddGesture(GestureType.WaveRight, waveRightSegments);
 614:  
 615:             IRelativeGestureSegment[] waveLeftSegments = new IRelativeGestureSegment[6];
 616:             WaveLeftSegment1 waveLeftSegment1 = new WaveLeftSegment1();
 617:             WaveLeftSegment2 waveLeftSegment2 = new WaveLeftSegment2();
 618:             waveLeftSegments[0] = waveLeftSegment1;
 619:             waveLeftSegments[1] = waveLeftSegment2;
 620:             waveLeftSegments[2] = waveLeftSegment1;
 621:             waveLeftSegments[3] = waveLeftSegment2;
 622:             waveLeftSegments[4] = waveLeftSegment1;
 623:             waveLeftSegments[5] = waveLeftSegment2;
 624:             this.gestures.AddGesture(GestureType.WaveLeft, waveLeftSegments);
 625:  
 626:             IRelativeGestureSegment[] swipeleftSegments = new IRelativeGestureSegment[3];
 627:             swipeleftSegments[0] = new SwipeLeftSegment1();
 628:             swipeleftSegments[1] = new SwipeLeftSegment2();
 629:             swipeleftSegments[2] = new SwipeLeftSegment3();
 630:             this.gestures.AddGesture(GestureType.LeftSwipe, swipeleftSegments);
 631:  
 632:             IRelativeGestureSegment[] swiperightSegments = new IRelativeGestureSegment[3];
 633:             swiperightSegments[0] = new SwipeRightSegment1();
 634:             swiperightSegments[1] = new SwipeRightSegment2();
 635:             swiperightSegments[2] = new SwipeRightSegment3();
 636:             this.gestures.AddGesture(GestureType.RightSwipe, swiperightSegments);
 637:  
 638:             IRelativeGestureSegment[] menuSegments = new IRelativeGestureSegment[20];
 639:             MenuSegments1 menuSegment = new MenuSegments1();
 640:             for (int i = 0; i < 20; i++)
 641:             {
 642:                 //gesture consists of the same thing 20 times 
 643:                 menuSegments[i] = menuSegment;
 644:             }
 645:  
 646:             this.gestures.AddGesture(GestureType.Menu, menuSegments);
 647:         }
 648:  
 649:         /// <summary>
 650:         /// Handles the Tick event of the ChangeTimer control.
 651:         /// </summary>
 652:         /// <param name="sender">The source of the event.</param>
 653:         /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
 654:         void ChangeTimer_Tick(object sender, EventArgs e)
 655:         {
 656:             if (this.DeleteSkeleton != null)
 657:             {
 658:                 this.DeleteSkeleton(this, null);
 659:             }
 660:         }
 661:  
 662:         /// <summary>
 663:         /// Resets the timer.
 664:         /// </summary>
 665:         public void ResetTimer()
 666:         {
 667:             this.ChangeTimer.Stop();
 668:             this.ChangeTimer.Start();
 669:         }
 670:  
 671:         /// <summary>
 672:         /// Notifies all change.
 673:         /// </summary>
 674:         private void NotifyAllChange()
 675:         {
 676:             this.NotifyPropertyChanged("Head");
 677:             this.NotifyPropertyChanged("LeftHand");
 678:             this.NotifyPropertyChanged("LeftHandX");
 679:             this.NotifyPropertyChanged("LeftHandY");
 680:             this.NotifyPropertyChanged("RightHand");
 681:             this.NotifyPropertyChanged("LeftWrist");
 682:             this.NotifyPropertyChanged("RightWrist");
 683:             this.NotifyPropertyChanged("LeftElbow");
 684:             this.NotifyPropertyChanged("RightElbow");
 685:             this.NotifyPropertyChanged("ShoulderCenter");
 686:             this.NotifyPropertyChanged("HipCenter");
 687:             this.NotifyPropertyChanged("LeftKnee");
 688:             this.NotifyPropertyChanged("RightKnee");
 689:             this.NotifyPropertyChanged("LeftAnkle");
 690:             this.NotifyPropertyChanged("RightAnkle");
 691:             this.NotifyPropertyChanged("LeftFoot");
 692:             this.NotifyPropertyChanged("RightFoot");
 693:             this.NotifyPropertyChanged("Spine");
 694:         }
 695:  
 696:         /// <summary>
 697:         /// Notifies the property changed.
 698:         /// </summary>
 699:         /// <param name="propertyName">Name of the property.</param>
 700:         private void NotifyPropertyChanged(string propertyName)
 701:         {
 702:             if (this.PropertyChanged != null)
 703:             {
 704:                 this.PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
 705:             }
 706:         }
 707:  
 708:         #endregion
 709:     }
 710: }
Step 3 – Getting some sort of visual representation.

Before we do anything with the points that are coming through, the first thing to do is to simply display the camera feed from the Kinect. This is very easy to do.

In the XAML, add an image control (i.e. called “cameraFeed”). In the code, when the main window is created, set up an event handler for the Kinect’s ImageFrameReady property; and it will be called once the SDK has picked up an image.

Within the handler, create a ‘PlanarImage’ from the event args, and then set the image source of the image control to a new BitmapSource as seen below:

   1: public MainWindow()
   2: {
   3:     InitializeComponent();
   4:     //other stuff here        
   5:     model.Kinect.ImageFrameReady += new EventHandler<Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs>(kinect_ImageFrameReady);
   6: }
   7:  
   8: void kinect_ImageFrameReady(object sender, Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs e)
   9: {
  10:     PlanarImage image = e.ImageFrame.Image;
  11:     cameraFeed.Source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null, image.Bits, image.Width * image.BytesPerPixel);
  12: }

Now let’s do something with the actual points we have obtained.

What we wanted to do here was to create a very simple “image” of a skeleton on the screen.

One thing we did here was to disregard some points from the skeleton. We found it irrelevant to display a few points (i.e. shoulder blades, and sides of the hips etc.), as it didn’t really make the skeleton look or perform any differently, and by excluding them, improved the performance of the entire application.

So, with what remained, we used a combination of ellipses and lines to represent the skeleton.

Simple enough to start off with, add an items control to the page, and within its Data Template, place the 16 ellipses (representing each joint), and a line to connect to each set of relevant ellipses (sort of like the song…”the wrist bone is connected to the elbow”, and so forth).

Each ellipse was created as per below:

   1: <Ellipse Height="14" Name="LeftHand" Stroke="Black" Fill="{Binding JointColor, Converter={StaticResource ColorToSolidColorBrushConverter}}" Width="14" Visibility="{Binding LeftHand, Converter={StaticResource JointToVisibilityConverter}}">
   2: <Ellipse.RenderTransform>
   3: <TranslateTransform X="{Binding LeftHand.Position.X, Converter={StaticResource                KinectValueToScreenCoOrdinatesConverterX}, ConverterParameter=0}" Y="{Binding LeftHand.Position.Y, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterY}, ConverterParameter=0}" />
   4:        </Ellipse.RenderTransform>
   5: </Ellipse>

The lines were also very simple:

   1: <Line StrokeThickness="3" Stroke="Black" StrokeEndLineCap="Round" 
   2: X1="{Binding LeftHand.Position.X, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterX}, ConverterParameter=7}" 
   3: Y1="{Binding LeftHand.Position.Y, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterY}, ConverterParameter=7}" 
   4: X2="{Binding LeftWrist.Position.X, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterX}, ConverterParameter=7}" 
   5: Y2="{Binding LeftWrist.Position.Y, Converter={StaticResource KinectValueToScreenCoOrdinatesConverterY}, ConverterParameter=7}" 
   6: Fill="Black" />

By binding the ellipses to each relevant point in the Main View Model, and using the converter that we created earlier, you should obtain something similar to this:-

Untitled

The Kinect sensor has picked up the skeletons of the two people standing in front of it. By then updating the collection of skeletons and updating the bindings of the items control source, the skeletons should be displayed correctly on the screen.

Now, as the frames come in, and the bindings update, the ellipses will seamlessly move around the screen as you move, and there you have it, you’re very own living skeleton!

In our next post, we will talk about how to write a gesture service with the SDK, which will recognize the gestures the skeletons on screen are making.

Written by Michael Tsikkos and James Glading