KISS : Use Kinect for Windows SDK to protect your privacy - Eternal Coding - HTML5 / Windows / Kinect / 3D development - Site Home - MSDN Blogs

KISS : Use Kinect for Windows SDK to protect your privacy


 

KISS : Use Kinect for Windows SDK to protect your privacy

  • Comments 4
Fotolia_27381572_S

I have the incredible chance to work with a very fun and motivating team. But one drawback of this situation is the risk to send unwanted emails when you forget to lock your computer when you go out.

And they can be really creative with this so French and sophisticated humor when they use your Outlook. So to protect myself I decided to write a small program using one of my favorite technology : Kinect for Windows SDK.

The main goal of KISS (which stands for Kinect Intelligent Security System) is to track the user in front of the sensor and detect when he goes out (to lock the computer for example).

You will discover how to develop such a program during this article.

The final solution is available to download there: http://www.catuhe.com/msdn/kiss.zip

 

Initializing the sensor

First of all, you have to reference the Kinect for Windows SDK assembly (C:\Program Files\Microsoft SDKs\Kinect\v1.0\Assemblies\Microsoft.Kinect.dll) and instantiate a new KinectSensor. To do so, the SDK helps you with a static helper class called KinectSensor which contains a list of already detected sensors : KinectSensor.KinectSensors. It also provides an event KinectSensor.KinectSensors.StatusChanged which will be raised when something happens to one of the attached Kinect sensors.

  1. public KinectIntelligentSecuritySystem()
  2. {
  3.     try
  4.     {
  5.         //listen to any status change for Kinects
  6.         KinectSensor.KinectSensors.StatusChanged += Kinects_StatusChanged;
  7.  
  8.         //loop through all the Kinects attached to this PC, and start the first that is connected without an error.
  9.         foreach (KinectSensor kinect in KinectSensor.KinectSensors)
  10.         {
  11.             if (kinect.Status == KinectStatus.Connected)
  12.             {
  13.                 kinectSensor = kinect;
  14.                 break;
  15.             }
  16.         }
  17.  
  18.         if (kinectSensor == null)
  19.             MessageBox.Show("No Kinect found");
  20.         else
  21.             Initialize();
  22.  
  23.     }
  24.     catch (Exception ex)
  25.     {
  26.         MessageBox.Show(ex.Message);
  27.     }
  28. }

 

  1. void Kinects_StatusChanged(object sender, StatusChangedEventArgs e)
  2. {
  3.     switch (e.Status)
  4.     {
  5.         case KinectStatus.Connected:
  6.             if (kinectSensor == null)
  7.             {
  8.                 kinectSensor = e.Sensor;
  9.                 Initialize();
  10.             }
  11.             break;
  12.         case KinectStatus.Disconnected:
  13.             if (kinectSensor == e.Sensor)
  14.             {
  15.                 Clean();
  16.                 MessageBox.Show("Kinect was disconnected");
  17.             }
  18.             break;
  19.         case KinectStatus.NotPowered:
  20.             if (kinectSensor == e.Sensor)
  21.             {
  22.                 Clean();
  23.                 MessageBox.Show("Kinect is no more powered");
  24.             }
  25.             break;
  26.     }
  27. }

 

As you can see, it is pretty simple as you just have to track the StatusChanged event and call Clean or Initialize accordingly:

  1. private void Initialize()
  2. {
  3.     if (kinectSensor == null)
  4.         return;
  5.  
  6.     kinectSensor.Start();
  7.  
  8.     kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution80x60Fps30);
  9.     kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;
  10.     kinectSensor.DepthStream.Range = DepthRange.Near;
  11. }
  12.  
  13. void Clean()
  14. {
  15.     kinectSensor.Stop();
  16.     kinectSensor = null;
  17. }

A thing worth noting is the activation of the near mode:

  1. kinectSensor.DepthStream.Range = DepthRange.Near;

With this mode activated, skeletons tracking is disabled but the depth values can be retrieved from a near distance (about 40cm).

 

Displaying the depth stream

Now, the sensor is instantiated and ready to use. You have then to connect to the depth stream.

When you develop with Kinect, it is important to provide a visual feedback of what the sensor sees. An event is already handled in the Initialize method to do get the depth frames:

  1. kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;

So you have to create a method that can build a bitmap from the depth stream:

  1. void DisplayData(DepthImageFrame frame)
  2. {
  3.     for (int i16 = 0, i32 = 0; i16 < data.Length && i32 < data32.Length; i16++, i32++)
  4.     {
  5.         int realDepth = (data[i16] >> 3);
  6.         byte intensity = (byte)(255 - (255 * realDepth / 3000.0f));
  7.  
  8.         data32[i32] = (intensity / 2) + ((intensity / 2) << 16) + ((intensity / 2) << 8) + (255 << 24);
  9.     }
  10.  
  11.     if (DepthBitmap == null)
  12.     {
  13.         DepthBitmap = new WriteableBitmap(frame.Width, frame.Height, 96, 96, PixelFormats.Bgra32, null);
  14.     }
  15.  
  16.     DepthBitmap.Lock();
  17.  
  18.     int stride = DepthBitmap.PixelWidth * DepthBitmap.Format.BitsPerPixel / 8;
  19.     Int32Rect dirtyRect = new Int32Rect(0, 0, DepthBitmap.PixelWidth, DepthBitmap.PixelHeight);
  20.     DepthBitmap.WritePixels(dirtyRect, data32, stride, 0);
  21.  
  22.     DepthBitmap.AddDirtyRect(dirtyRect);
  23.     DepthBitmap.Unlock();
  24.  
  25.     if (PropertyChanged != null)
  26.     {
  27.         PropertyChanged(this, new PropertyChangedEventArgs("DepthBitmap"));
  28.     }
  29. }

DisplayData uses a DepthFrame (provided by the event) to fill a WriteableBitmap with grayed values (from white (near) to black (far)).

The KinectIntelligentSecuritySystem class implements INotifyPropertyChanged and exposes a property called DepthBitmap. This property is used by the main program to fill a WPF Image control:

  1. <Window x:Class="KISS.MainWindow"
  2.         xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  3.         xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  4.         Title="Kinect Intelligent Security System" Height="350" Width="525">
  5.     <Grid>
  6.         <Image x:Name="depthImage" Source="{Binding DepthBitmap}" />
  7.     </Grid>
  8. </Window>

 

  1. depthImage.DataContext = kiss;

So every time DisplayData computes a new image, the PropertyChanged event is raised and the WPF Image control updates itself.

This code is directly inspired by the Kinect Toolbox that you can grab there : http://kinecttoolbox.codeplex.com

 

Detecting user and locking the computer

Finally the main point is here: using the depth stream to compute the average depth of what is in front of the sensor and to detect any big variation of this average:

  1. // Computing depth average
  2. var avg = data.Average(pixel => pixel);
  3.  
  4. previousValues.Add(avg);
  5.  
  6. var currentAvg = previousValues.Average(value => value);
  7.  
  8. if (previousValues.Count > 60)
  9.     previousValues.RemoveAt(0);
  10.  
  11. if (previousValues.Count == 60 && (Math.Abs(currentAvg - avg) > 1500))
  12. {
  13.     if (OnMovement != null)
  14.         OnMovement();
  15.  
  16.     previousValues.Clear();
  17. }

You have to save a given number (60 in this example) of previous computed average depths in the previousValues list. With this list, every time a new frame is available, you have to compare the current average with the average of values stored in the list and if the difference is bigger than a given threshold, an event is raised.

In response to this event, I choose to lock my computer with this simple interop code:

  1. readonly KinectIntelligentSecuritySystem kiss = new KinectIntelligentSecuritySystem();
  2.  
  3. [DllImport("user32.dll")]
  4. public static extern void LockWorkStation();
  5.  
  6. public MainWindow()
  7. {
  8.     InitializeComponent();
  9.     kiss.OnMovement += kiss_OnMovement;
  10.  
  11.     depthImage.DataContext = kiss;
  12. }
  13.  
  14. void kiss_OnMovement()
  15. {
  16.     LockWorkStation();
  17. }

And voila! You are now safe to get out without locking your computer because KISS watches over you Rire.

The complete code for KISS is the following:

  1. using System;
  2. using System.Linq;
  3. using System.Windows;
  4. using System.Windows.Media;
  5. using System.Windows.Media.Imaging;
  6. using Microsoft.Kinect;
  7. using System.Collections.Generic;
  8. using System.ComponentModel;
  9.  
  10. namespace KISS
  11. {
  12.     class KinectIntelligentSecuritySystem : INotifyPropertyChanged
  13.     {
  14.         public event Action OnMovement;
  15.         public event PropertyChangedEventHandler PropertyChanged;
  16.  
  17.         public WriteableBitmap DepthBitmap { get; private set; }
  18.  
  19.         private KinectSensor kinectSensor;
  20.         readonly List<double> previousValues = new List<double>();
  21.         short[] data;
  22.         int[] data32;
  23.  
  24.         void Kinects_StatusChanged(object sender, StatusChangedEventArgs e)
  25.         {
  26.             switch (e.Status)
  27.             {
  28.                 case KinectStatus.Connected:
  29.                     if (kinectSensor == null)
  30.                     {
  31.                         kinectSensor = e.Sensor;
  32.                         Initialize();
  33.                     }
  34.                     break;
  35.                 case KinectStatus.Disconnected:
  36.                     if (kinectSensor == e.Sensor)
  37.                     {
  38.                         Clean();
  39.                         MessageBox.Show("Kinect was disconnected");
  40.                     }
  41.                     break;
  42.                 case KinectStatus.NotPowered:
  43.                     if (kinectSensor == e.Sensor)
  44.                     {
  45.                         Clean();
  46.                         MessageBox.Show("Kinect is no more powered");
  47.                     }
  48.                     break;
  49.             }
  50.         }
  51.  
  52.         public KinectIntelligentSecuritySystem()
  53.         {
  54.             try
  55.             {
  56.                 //listen to any status change for Kinects
  57.                 KinectSensor.KinectSensors.StatusChanged += Kinects_StatusChanged;
  58.  
  59.                 //loop through all the Kinects attached to this PC, and start the first that is connected without an error.
  60.                 foreach (KinectSensor kinect in KinectSensor.KinectSensors)
  61.                 {
  62.                     if (kinect.Status == KinectStatus.Connected)
  63.                     {
  64.                         kinectSensor = kinect;
  65.                         break;
  66.                     }
  67.                 }
  68.  
  69.                 if (kinectSensor == null)
  70.                     MessageBox.Show("No Kinect found");
  71.                 else
  72.                     Initialize();
  73.  
  74.             }
  75.             catch (Exception ex)
  76.             {
  77.                 MessageBox.Show(ex.Message);
  78.             }
  79.         }
  80.  
  81.         private void Initialize()
  82.         {
  83.             if (kinectSensor == null)
  84.                 return;
  85.  
  86.             kinectSensor.Start();
  87.  
  88.             kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution80x60Fps30);
  89.             kinectSensor.DepthFrameReady += kinectSensor_DepthFrameReady;
  90.             kinectSensor.DepthStream.Range = DepthRange.Near;
  91.         }
  92.  
  93.         void Clean()
  94.         {
  95.             kinectSensor.Stop();
  96.             kinectSensor = null;
  97.         }
  98.  
  99.         void DisplayData(DepthImageFrame frame)
  100.         {
  101.             for (int i16 = 0, i32 = 0; i16 < data.Length && i32 < data32.Length; i16++, i32++)
  102.             {
  103.                 int realDepth = (data[i16] >> 3);
  104.                 byte intensity = (byte)(255 - (255 * realDepth / 3000.0f));
  105.  
  106.                 data32[i32] = (intensity / 2) + ((intensity / 2) << 16) + ((intensity / 2) << 8) + (255 << 24);
  107.             }
  108.  
  109.             if (DepthBitmap == null)
  110.             {
  111.                 DepthBitmap = new WriteableBitmap(frame.Width, frame.Height, 96, 96, PixelFormats.Bgra32, null);
  112.             }
  113.  
  114.             DepthBitmap.Lock();
  115.  
  116.             int stride = DepthBitmap.PixelWidth * DepthBitmap.Format.BitsPerPixel / 8;
  117.             Int32Rect dirtyRect = new Int32Rect(0, 0, DepthBitmap.PixelWidth, DepthBitmap.PixelHeight);
  118.             DepthBitmap.WritePixels(dirtyRect, data32, stride, 0);
  119.  
  120.             DepthBitmap.AddDirtyRect(dirtyRect);
  121.             DepthBitmap.Unlock();
  122.  
  123.             if (PropertyChanged != null)
  124.             {
  125.                 PropertyChanged(this, new PropertyChangedEventArgs("DepthBitmap"));
  126.             }
  127.         }
  128.  
  129.         void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
  130.         {
  131.             var frame = e.OpenDepthImageFrame();
  132.  
  133.             if (frame == null)
  134.                 return;
  135.  
  136.             if (data == null)
  137.             {
  138.                 data = new short[frame.PixelDataLength];
  139.                 data32 = new int[frame.PixelDataLength];
  140.             }
  141.  
  142.             frame.CopyPixelDataTo(data);
  143.             
  144.             // Displaying frame
  145.             DisplayData(frame);
  146.  
  147.             frame.Dispose();
  148.             
  149.             // Computing depth average
  150.             var avg = data.Average(pixel => pixel);
  151.  
  152.             previousValues.Add(avg);
  153.  
  154.             var currentAvg = previousValues.Average(value => value);
  155.  
  156.             if (previousValues.Count > 60)
  157.                 previousValues.RemoveAt(0);
  158.  
  159.             if (previousValues.Count == 60 && (Math.Abs(currentAvg - avg) > 1500))
  160.             {
  161.                 if (OnMovement != null)
  162.                     OnMovement();
  163.  
  164.                 previousValues.Clear();
  165.             }
  166.         }
  167.  
  168.     }
  169. }
Leave a Comment
  • Please add 7 and 5 and type the answer here:
  • Post
  • Hehe plus moyen de mettre une photo de poireaux en fond d'écran alors ?  ;-)

  • is face recogniton for auto log in/out in the next version?

  • Well, for the moment, the Windows and L keys are my favourite ones. Too bad I can't have Kinect at work (would be awesome), or else I wouldn't have had to bring the breakfast for all my team last monday !

  • very good tutorial that help me in the process of setup Kinect sensor on Windows. Also, I add a link to this tutorial into an article with many more tutorial related to Kinect sensor www.intorobotics.com/working-with-kinect-3d-sensor-in-robotics-setup-tutorials-applications

Page 1 of 1 (4 items)