Building a simple executive toy using Kinect for Windows v2 and Unity3D

Building a simple executive toy using Kinect for Windows v2 and Unity3D

  • Comments 11

 

Untitled-1

Abstract

This article will look into how to leverage the new Kinect for Windows 2.0 device to create a simple executive toy using Unity and the Unity plugin. The aim is to demonstrate how easy it is to leverage Kinect in your Unity projects and how to build some thing simple quickly. This is a step by step guide on how to great the Kinect Wall app and how it works.

Keywords: Unity3D, Kinect, Shaders, Cg, Geometry Shaders, Vertex Shaders, Fragment Shaders, Textures
Introduction
Research Method

I am taking a tutorial approach for this article where I will show you step by step how to build create the application. The application source code will be available on Codeplex free to use in anyway you like.

Requirements
  • Unity 3d Pro – yes unfortunately because we are accessing unsafe code we need Pro
  • Kinect SDK
  • Kinect Unity Plugin
Tutorial

The first step is to start Unity and create a new Project. Give it a name and click create.

01.-Create-new-Project.

The next step is to do some minor admin in my project. The main thing I do here is create the following folders so I can manage my project a little neater:

  • Materials
  • Scripts
  • Shaders
  • Scenes
  • Textures

Now we begin to setup some of the items we need to create to make this all work. We will need the following:

  • Empty Material
  • Scene – Main one with our camera, lights, etc
  • Textures – These are textures we are going to use for our bricks
  • Shader – This does all the depth and body index process as well as all the geometry generation
  • Main Script – This spins up the Kinect device and passes information through to the Shader for processing.

We now need to add the Kinect for Windows v2 assets by going to the Assets –> Import Package –> Custom Package option and selecting the Kinect v2 Unity package. This will import all the required assets into the project so we can access the sensor. I need to enable unsafe code, the way that I do this is by adding a file called smcs.rsp to my Assets folder. This file is a simple text file with the single line inside of –unsafe.

Next we create the mainscript file and retrieve all the information from the sensor and pass it to the Shader. Inside the MainScript there are a number of default methods. Start which is used for initialisation and Update which is called as part of the update thread. Inside the Start function we are going to initialise the sensor and some of the storage structure for the stream of information. Firstly I create some local variables to store the sensor states:

   1: private KinectSensor sensor;
   2: private DepthFrameReader depthreader;
   3: private BodyIndexFrameReader bodyreader;
   4: private byte[] depthdata;
   5: private byte[] bodydata;

The KinectSensor object gives us access to the various data streams from the device. Because we want the outline of the player and the depth information we need to access to the DepthFrame and BodyIndex information.

   1: depthreader = sensor.DepthFrameSource.OpenReader();
   2: bodyreader = sensor.BodyIndexFrameSource.OpenReader();

We get the information from the reader in the update loop in the following manner

   1: var depthframe = depthreader.AcquireLatestFrame();
   2:  
   3: if (depthframe != null)
   4: {
   5:     fixed (byte* pData = depthdata)
   6:     {
   7:         depthframe.CopyFrameDataToIntPtr(new System.IntPtr(pData), (uint)depthdata.Length);
   8:     }
   9:     depthframe.Dispose();
  10:     depthframe = null;
  11:  
  12:     DepthTexture.LoadRawTextureData(depthdata);
  13:     DepthTexture.Apply();
  14: }

This will read the information into a byte array that I can then send to the Shader either through a Texture of via a ComputeBuffer. To kick start the shader I need to set all the parameters and then tell it to start drawing using the following:

   1: ShaderMaterial.SetTexture("_MainTex", MainTexture);
   2: ShaderMaterial.SetTexture("_DepthTex", DepthTexture);
   3: ShaderMaterial.SetFloat("_DepthWidth", DepthWidth);
   4: ShaderMaterial.SetFloat("_DepthHeight", DepthHeight);
   5: ShaderMaterial.SetBuffer("_BodyIndexBuffer", bodyIndexBuffer);
   6: ShaderMaterial.SetPass(0);
   7:     
   8: Graphics.DrawProcedural(MeshTopology.Points, WallWidth * WallHeight, 1);

We now have everything in the MainScript we need, to see the complete source code with both methods of getting data to a Shader as well as getting bodyindex information have a look at the codeplex source where the complete project is available.

Remember to save your scene scene into the scene folder. I then add some items to my scene the first is a few lights to provide a clean effect for the wall. The next is an empty game object (GameObject –> Create Empty /Ctrl-Shift-N). I now add my main script into my Scripts folder and select the mainScriptObject and drag the script onto it.

What I need to do now is create a shader and a material that will use this shader. Creating the shader first and then the material I then drag the material on to the mainScriptObject shader material property. Now we start looking at the shader. The shader will handle all the tile generation for my scene. I will be leveraging a geometry shader to perform all the object handling. The reason for this is mainly performance and processing for the textures will take too long outside of the shader.

The shader itself is below and I have commented as much of the code as possible to make it easy to understand. Currently the depth from the depth buffer isn't functioning as I had hoped and will post an update as soon as I have tweaked it.

   1: Shader "Custom/WallShader" {
   2:     Properties {
   3:         _MainTex ("Main Brick Texture", 2D) = "white" {}
   4:         _DepthTex ("Kinect Depth Image", 2D) = "white" {}
   5:         _DepthImageWidth("Depth Image Width",Float) = 100
   6:         _DepthImageHeight("Depth Image Height",Float) = 50          
   7:     }
   8:     SubShader {
   9:         Tags { "LightMode" = "ForwardBase" }            
  10:         Pass
  11:         {    
  12:             Cull Off            
  13:             LOD 200
  14:     
  15:             CGPROGRAM        
  16:                 #pragma vertex VS_Main
  17:                 #pragma geometry GS_Main            
  18:                 #pragma fragment FS_Main
  19:                 #include "UnityCG.cginc"                                
  20:                 
  21:                 uniform fixed4 _LightColor0;
  22:                 #define TAM 36
  23:  
  24:                 float MinDepthMM = 500.0;
  25:                 float MaxDepthMM = 8000.0;
  26:  
  27:                 float _DepthImageWidth;
  28:                 float _DepthImageHeight;                
  29:                 
  30:                 float4    _MainTex_ST;
  31:                 sampler2D _MainTex;
  32:  
  33:                 StructuredBuffer<float> _BodyIndexBuffer;
  34:                 StructuredBuffer<float> _DepthBuffer;
  35:                         
  36:                 // Structs for passing information from item to item
  37:                 struct EMPTY_INPUT
  38:                 {
  39:                 };
  40:  
  41:                 struct POSCOLOR_INPUT
  42:                 {
  43:                     float4    pos            : POSITION;
  44:                     float4  color        : COLOR;
  45:                     float2    uv_MainTex    : TEXCOORD0;
  46:                     float3    normal    : NORMAL;
  47:                 };
  48:                                 
  49:                 // Depth Helper function 
  50:                 float DepthFromPacked4444(float4 packedDepth)
  51:                 {
  52:                     // convert from [0,1] to [0,15]
  53:                     packedDepth *= 15.01f;
  54:     
  55:                     // truncate to an int
  56:                     int4 rounded = (int4)packedDepth;                
  57:     
  58:                     return rounded.w * 4096 + rounded.x * 256 + rounded.y * 16 + rounded.z;                
  59:                 }
  60:  
  61:                 
  62:                 EMPTY_INPUT VS_Main() 
  63:                 {
  64:                     return (EMPTY_INPUT)0;
  65:                 }
  66:  
  67:                  
  68:                 [maxvertexcount(TAM)]
  69:                 void GS_Main(point EMPTY_INPUT p[1], uint primID : SV_PrimitiveID, inout TriangleStream<POSCOLOR_INPUT> triStream)
  70:                 {                                                    
  71:                     float4 currentCoordinates = float4(primID % 100, (primID / 100), 0, 1.0);    
  72:                     
  73:                     float4 offset = float4(0,0,0,0);
  74:                     float4 scale = float4(1,1,1,0);
  75:                     float4 curcolor = float4(1,1,1,1);    
  76:  
  77:                     float depth = 0;
  78:                                         
  79:                     //TODO - Figure out best way to handle this // I suspect a compute shader would work                                                        
  80:                     float4 textureCoordinates = float4(currentCoordinates.x/100 * 512/4,currentCoordinates.y/50 * 424,0,0);
  81:  
  82:                     int index = (int)(textureCoordinates.x) + ((int)textureCoordinates.y * 512/4);
  83:                     int depthindex = (int)(textureCoordinates.x) + ((int)textureCoordinates.y * 512);
  84:                     float player = _BodyIndexBuffer[index];
  85:                     uint depthinfo = (uint)_DepthBuffer[depthindex] >> 3;
  86:                     
  87:                     // Get more depth information
  88:                     if(player > 0 && player < 1) 
  89:                     {                                                
  90:                         depth = -3;
  91:                         depth -= depthinfo * 3;                        
  92:                     }                            
  93:                                                             
  94:                      
  95:                     float f = 1.0;
  96:                     //Construct a cube
  97:  
  98:                     const float4 vc[TAM] = { 
  99:                         float4( -f,  f,  f, 1.0f), float4(  f,  f,  f, 1.0f), float4(  f,  f, -f, 1.0f),   //Top                                
 100:                         float4(  f,  f, -f, 1.0f), float4( -f,  f, -f, 1.0f), float4( -f,  f,  f, 1.0f),   //Top
 101:  
 102:                         float4(  f,  f, -f, 1.0f), float4(  f,  f,  f, 1.0f), float4(  f, -f,  f, 1.0f),   //Right
 103:                         float4(  f, -f,  f, 1.0f), float4(  f, -f, -f, 1.0f), float4(  f,  f, -f, 1.0f),   //Right
 104:  
 105:                         float4( -f,  f, -f, 1.0f), float4(  f,  f, -f, 1.0f), float4(  f, -f, -f, 1.0f),   //Front
 106:                         float4(  f, -f, -f, 1.0f), float4( -f, -f, -f, 1.0f), float4( -f,  f, -f, 1.0f),   //Front
 107:  
 108:                         float4( -f, -f, -f, 1.0f), float4(  f, -f, -f, 1.0f), float4(  f, -f,  f, 1.0f),   //Bottom                                         
 109:                         float4(  f, -f,  f, 1.0f), float4( -f, -f,  f, 1.0f), float4( -f, -f, -f, 1.0f),   //Bottom
 110:  
 111:                         float4( -f,  f,  f, 1.0f), float4( -f,  f, -f, 1.0f), float4( -f, -f, -f, 1.0f),   //Left
 112:                         float4( -f,  -f, -f, 1.0f), float4( -f, -f, f, 1.0f), float4( -f,  f,  f, 1.0f),   //Left
 113:  
 114:                         float4(  -f,  f,  f, 1.0f), float4( -f, f,  f, 1.0f), float4(  -f, -f,  f, 1.0f),   //Back
 115:                         float4(  f, -f,  f, 1.0f), float4(  -f, -f,  f, 1.0f), float4( -f,  f,  f, 1.0f)    //Back
 116:                     };
 117:  
 118:  
 119:                     const float2 UV1[TAM] = { 
 120:                         float2( 0.0f,    1.0f ), float2( 1.0f,   1.0f ), float2( 1.0f,   0.0f ),         
 121:                         float2( 1.0f,    0.0f ), float2( 0.0f,   0.0f ), float2( 0.0f,   1.0f ),         
 122:  
 123:                         float2( 0.0f,    1.0f ), float2( 1.0f,   1.0f ), float2( 1.0f,   0.0f ),         
 124:                         float2( 1.0f,    0.0f ), float2( 0.0f,   0.0f ), float2( 0.0f,   1.0f ), 
 125:  
 126:                         float2( 0.0f,    1.0f ), float2( 1.0f,   1.0f ), float2( 1.0f,   0.0f ),         
 127:                         float2( 1.0f,    0.0f ), float2( 0.0f,   0.0f ), float2( 0.0f,   1.0f ), 
 128:  
 129:                         float2( 0.0f,    1.0f ), float2( 1.0f,   1.0f ), float2( 1.0f,   0.0f ),         
 130:                         float2( 1.0f,    0.0f ), float2( 0.0f,   0.0f ), float2( 0.0f,   1.0f ), 
 131:  
 132:                         float2( 0.0f,    1.0f ), float2( 1.0f,   1.0f ), float2( 1.0f,   0.0f ),         
 133:                         float2( 1.0f,    0.0f ), float2( 0.0f,   0.0f ), float2( 0.0f,   1.0f ), 
 134:                         
 135:                         float2( 0.0f,    1.0f ), float2( 1.0f,   1.0f ), float2( 1.0f,   0.0f ),         
 136:                         float2( 1.0f,    0.0f ), float2( 0.0f,   0.0f ), float2( 0.0f,   1.0f )                        
 137:                         
 138:                     };                        
 139:  
 140:                     const int TRI_STRIP[TAM]  = {  0, 1, 2,  3, 4, 5,
 141:                         6, 7, 8,  9,10,11,
 142:                         12,13,14, 15,16,17,
 143:                         18,19,20, 21,22,23,
 144:                         24,25,26, 27,28,29,
 145:                         30,31,32, 33,34,35  
 146:                     }; 
 147:  
 148:  
 149:                     POSCOLOR_INPUT v[TAM];
 150:                     int i = 0;    
 151:  
 152:                     currentCoordinates.z = depth;
 153:  
 154:                     float3 lightdirection = normalize(_WorldSpaceLightPos0.xyz);
 155:                     float3 lightcolor = _LightColor0;
 156:                     // Assign new vertices positions 
 157:                     for (i=0;i<TAM;i++) {         
 158:                         float4 npos = float4(currentCoordinates.x, 50 - currentCoordinates.y,depth,1.0) * scale + offset;
 159:                         v[i].pos = npos + vc[i]; 
 160:                         v[i].color = curcolor;   
 161:                         v[i].uv_MainTex = TRANSFORM_TEX(UV1[i],_MainTex);                        
 162:                         v[i].normal = float3(0,0,0);
 163:                         v[i].pos = mul(UNITY_MATRIX_MVP, v[i].pos);
 164:                     }                                    
 165:                         
 166:                     // Build the cube tile by submitting triangle strip vertices
 167:                     for (i=0;i<TAM/3;i++)                    
 168:                     {                     
 169:                         //Calculate the normal of the triangle
 170:                         float4 U = v[TRI_STRIP[i*3+1]].pos - v[TRI_STRIP[i*3+0]].pos;
 171:                         float4 V = v[TRI_STRIP[i*3+2]].pos - v[TRI_STRIP[i*3+0]].pos;
 172:                         float3 normal = normalize(cross(U,V));
 173:                         float3 normaldirection = normalize( mul(float4(normal,1.0), _World2Object).xyz);
 174:                         float4 ncolor = float4(_LightColor0.xyz * v[TRI_STRIP[i*3+0]].color.rgb * max(0.0, dot(normaldirection, lightdirection)),1.0);
 175:  
 176:                         //Set the normal and the color based on the color
 177:                         v[TRI_STRIP[i*3+0]].normal = normal;
 178:                         v[TRI_STRIP[i*3+0]].color = ncolor;
 179:                         v[TRI_STRIP[i*3+1]].normal = normal;
 180:                         v[TRI_STRIP[i*3+1]].color = ncolor;
 181:                         v[TRI_STRIP[i*3+2]].normal = normal;
 182:                         v[TRI_STRIP[i*3+2]].color = ncolor;
 183:                         
 184:                         triStream.Append(v[TRI_STRIP[i*3+0]]);
 185:                         triStream.Append(v[TRI_STRIP[i*3+1]]);
 186:                         triStream.Append(v[TRI_STRIP[i*3+2]]);
 187:  
 188:                         triStream.RestartStrip();
 189:                     }                                        
 190:                 }
 191:  
 192:                 //Fragment Shader - with Lighting
 193:                 float4 FS_Main(POSCOLOR_INPUT input) : COLOR
 194:                 {                    
 195:                     half4 texcol = tex2D (_MainTex, input.uv_MainTex);
 196:  
 197:                     return texcol * input.color;                    
 198:                 }                
 199:  
 200:             ENDCG
 201:         }
 202:     } 
 203:     FallBack "Diffuse"
 204: }

With that all done you can now run your scene with the Kinect v2 Running and you should see the wall pushing out when you walk in front of it.

Video Link - https://www.youtube.com/watch?v=cFsPluBZn1s


Conclusion

Using Kinect v2 with Unity3D is extremely easy and the API’s are pretty much identical to the .Net API. Probably the biggest lesson is the benefit of using Shaders to handle the large volume of information between the various image streams you are receiving from the device.

Edit: I have uploaded the zip file of the complete project to codeplex.

Source Code

 https://sharpflipwall.codeplex.com

References

http://wiki.unity3d.com/index.php/Getting_Started_with_Shaders

http://www.bricksntiles.com/textures/

http://answers.unity3d.com/questions/744622/

http://johannes.gotlen.se/blog/2013/02/realtime-marching-cubes/

http://en.wikibooks.org/wiki/Cg_Programming/Unity

https://www.youtube.com/watch?v=bqKULvitmpU

Leave a Comment
  • Please add 3 and 5 and type the answer here:
  • Post
  • Hi, really nice example!

    I tried to run your source code but after I run this I see blank screen and after a while I get this error:

    DestroyBuffer can only be called from the main thread.

    Constructors and field initializers will be executed from the loading thread when loading a scene.

    Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.

    any idea what could do this? thank you.

  • Hi

    Thanks for the feedback. I am able to replicate the error and I am investigating it although my application runs fine with it happening.

    You should see by default a brick wall in camera 1 if you load my unity scene.

    Do you have a Kinect v2 plugged in or a Kinect Studio xef file that you can load that has body index data?

    I have also rechecked in the code to make sure its the latest.

    Shout if you are still stuck.

  • Sorry but I just can't figure out how to properly load the project into unity, it gives me these erorrs:

    Rebuilding Library because the asset database could not be found!

    Script 'AudioSource' has the same name as built-in Unity component.

    AddComponent and GetComponent will not work with this script.

    Script 'Joint' has the same name as built-in Unity component.

    AddComponent and GetComponent will not work with this script.

    Visual Studio Tools: Deleting assembly UnityVS.VersionSpecific.dll as it is out of date.

    Visual Studio Tools: Assembly UnityVS.VersionSpecific.dll generated.

    in scene I can only see Main Camera, MainScript and D. Light and when I click on MainScipr it tells me That the associated scrpt can not be loaded. So I tried to re-add it but it gives me that error with DestroyBuffer. I cant see the Kinect Sensors starts as I can see the IR and the light after I hit Play.

    I dont know if I don't do wrong if I just download the zip, unzip the folder UnityVS.sharpFlipWall go into Scenes and run the Mainscene

    I have the Pro version

    Thank you for your help. I'd really love to see this working and start doing my own games for kinect v2

  • I have uploaded the Unity Folder to CodePlex, please have a look at the download section. I am going to remove the source code for now until I figure out a good way to make it available via codeplex.

  • wow, thank you a lot for sharing this. I tried it and It works like a charm now. I am still getting the DestroyBuffer error as you can see here: dl.dropboxusercontent.com/.../unitykinect.png but I believe it is not a big deal. The shader is a little bit complicated for me to understand. Do you think you will make something without using shader in the future? I will be looking forward. Thanks once more for your help!

  • I would recommend sticking to using shader as they add a huge performance benefit. The gpu allows us to process the volumes of data needed while remaining smooth. I am happy to break down the details of the shader if you like?

  • Yes please that would be awesome! Also thanks for the References - Getting stareted with shader is really good but otherwise there is not much how the shaders works in unity at the internet, so breaking down more details of using your shader for kinect would be really helpful

  • Hi, thanks a lot for posting this example, it's very helpful!

    One question... :-) ...

    Currently, the depth seems limited to either flat or shifted by 3(?), what would I need to adjust so that the depth is correctly matching the varying depth from the sensor?

    Is it simply a type casting error? I can't seem to get any gradient in depth.

    Much appreciated for your time.

    best

  • For more granular depth you would need to leverage an additional step of using the depth index as well as the player index information.

  • Hi, really nice example!

    But I want to know where I can get Kinect Unity Plugin?

  • Plugin is available here, but requires unity pro - www.microsoft.com/.../details.aspx

Page 1 of 1 (11 items)