• Kinect for Windows Product Blog

    Revealing Kinect for Windows v2 hardware

    • 57 Comments

    As we continue the march toward the upcoming launch of Kinect for Windows v2, we’re excited to share the hardware’s final look.

    Sensor

    The sensor closely resembles the Kinect for Xbox One, except that it says “Kinect” on the top panel, and the Xbox Nexus—the stylized green “x”—has been changed to a simple, more understated power indicator:

    Kinect for Windows v2 sensor
    Kinect for Windows v2 sensor

    Hub and power supply

    The sensor requires a couple other components to work: the hub and the power supply. Tying everything together is the hub (top item pictured below), which accepts three connections: the sensor, USB 3.0 output to PC, and power. The power supply (bottom item pictured below) does just what its name implies: it supplies all the power the sensor requires to operate. The power cables will vary by country or region, but the power supply itself supports voltages from 100–240 volts.

    Kinect for Windows v2 hub (top) and power supply (bottom)

    Kinect for Windows v2 hub (top) and power supply (bottom)

    As this first look at the Kinect for Windows v2 hardware indicates, we're getting closer and closer to launch. So stay tuned for more updates on the next generation of Kinect for Windows.

    Kinect for Windows Team

    Key links


  • Kinect for Windows Product Blog

    Swap your face…really

    • 4 Comments

    Ever wish you looked like someone else? Maybe Brad Pitt or Jennifer Lawrence? Well, just get Brad or Jennifer in the same room with you, turn on the Kinect for Windows v2 sensor, and presto: you can swap your mug for theirs (and vice versa, of course). Don’t believe it? Then take a look at this cool video from Apache, in which two developers happily trade faces.

    Swapping faces in real time—let the good times roll

    According to Adam Vahed, managing director at Apache, the ability of the Kinect for Windows v2 sensor and SDK to track multiple bodies was essential to this project, as the solution needed to track the head position of both users. In fact, Adam rates the ability to perform full-skeletal tracking of multiple bodies as the Kinect for Windows v2 sensor’s most exciting feature, observing that it “opens up so many possibilities for shared experiences and greater levels of game play in the experiences we create.”

    Adam admits that the face swap demo was done mostly for fun. That said, he also notes that “the ability to identify and capture a person’s face in real time could be very useful for entertainment-based experiences—for instance, putting your face onto a 3D character that can be driven by your own movements.”

    Adam also stressed the value of the higher definition color feed in the v2 sensor, noting that Apache’s developers directly manipulated this feed in the face swap demo in order to achieve the desired effect. He finds the new color feed provides the definition necessary for full-screen augmented-reality experiences, something that wasn’t possible with the original Kinect for Windows sensor.

    Above all, Adam encourages other developers to dive in with the Kinect for Windows v2 sensor and SDK—to load the samples and play around with the capabilities. He adds that the forums are a great source of inspiration as well as information, and he advises developers “to take a look at what other people are doing and see if you can do something different or better—or both!”

    The Kinect for Windows Team

    Key links

  • Kinect for Windows Product Blog

    Holiday shoppers got the Midas touch

    • 1 Comments

    Ever wonder what you’d look like drenched in gold? December shoppers in Manhattan were captivated by just such images when they paused before an innovative window display for the new men’s cologne Gold Jay Z at Macy’s flagship store in Herald Square. This innovative, engaging display, the creation of advertising agency kbs+ and interactive design firm Future Colossal, employed Kinect for Windows to capture images of window shoppers and flow liquid gold over their silhouettes.

    Window shoppers found it hard to resist creating a gold-clad avatar.

    The experience began when the Kinect for Windows sensor detected that a passer-by had engaged with the display, which showed liquid gold rippling and flowing across a high-resolution screen. The Kinect for Windows sensor then captured a 3D image of the shopper, which artfully emerged from the pool of flowing gold to appear as a silhouette draped in the precious metal. This golden avatar interactively followed the window shopper’s movements, creating a beautiful, sinuous tableau that pulled the passer-by into an immersive experience with the fragrance brand. The Kinect for Windows also provided the shopper a photo of his or her golden doppelganger and a hashtag for sharing it via social media.  

    Kinect for Windows Team

    Key links

  • Kinect for Windows Product Blog

    An MVP’s look at the Kinect for Windows v2 developer preview

    • 0 Comments

    A few months ago, Microsoft Most Valuable Professional (MVP) James Ashley, a leader in developing with Kinect for Windows, wrote a very perceptive blog about Kinect for Windows v2 entitled, Kinect for Windows v2 First Look. James’ blog was so insightful that we wanted to check in with him after being in the Developer Preview program for three months and learn more about his experiences with the preview sensor and his advice to fellow Kinect for Windows developers. Here’s our Q&A with James:

    Microsoft: As a participant in the developer preview program, what cool things have you been doing with the Kinect for Windows v2 sensor and SDK over the past few months? Which features have you used, and what did you do with them?

    James: My advanced technology group at Razorfish has been very interested in developing mixed-media and mixed-technology stories with the Kinect for Windows v2 sensor. We recently did a proof- of-concept digital store with the Windows 8 team for the National Retail Federation (aka “Retail’s BIG Show”) in New York. You've heard of pop-up stores? We took this a step further by pre-loading a shipping container with digital screens, high-lumen projectors, massive arrays of Microsoft Surface tablets, and Perceptive Pixel displays and having a tractor-trailer deposit it in the Javits Center in New York City. When you opened the container, you had an instant retail store. We used the Kinect for Windows v2 sensor and SDK to drive an interactive soccer game built in Unity’s 3D toolset, in which 3D soccer avatars were controlled by the player's full body movements: when you won a game, a signal was sent by using Arduino components to drop a drink from a vending machine.

    Watch the teaser for Razorfish's interactive soccer game

    We also used Kinect for Windows v2 to allow people to take pictures with digital items they designed on the Perceptive Pixel. We then dropped a beach scene they selected into the background of the picture, which was printed out on the spot as well as emailed and pushed to their social networks if they wanted. In creating this experience, the new time-of-flight depth camera in Kinect for Windows v2 proved to be leagues better than anything we were able to do with the original Kinect for Windows sensor; we were thrilled with how well it worked. [Editor’s note: You can learn more about these retail applications in this blog post.]

    Much closer to the hardware, we have also been working with a client on using Kinect for Windows v2 to do precise measurements, to see if the Kinect for Windows v2 sensor can be used in retail to help people get fitted precisely—for instance with clothing and other wearables. Kinect for Windows v2 promises accuracy of 2.5 cm at even 4 meters, so this is totally feasible and could transform how we shop.

    Microsoft: Which features do you find the most useful and/or the most exciting, and why?

    James: Right now, I'm most interested in the depth camera. It has a much higher resolution than some standard time-of-flight cameras currently selling for $8,000 or $9,000. Even though the Kinect for Windows v2 final pricing hasn't been announced yet, we can expect it to be much, much less than that. It's stunning that Microsoft was able to pull off this technical feat, providing both improved quality and improved value in one stroke.

    Microsoft: Have you heard from other developers, and if so, what are they saying about your applications and/or their impressions of Kinect for Windows v2?

    James: I'm on both the MVP list and the developer preview program's internal list, so I've had a chance to hear a lot of really great feedback. Basically, we all had to learn a lot of tricks to make things work the way we wanted with the original Kinect for Windows. With v2, it feels like we are finally getting all the hardware performance we've wanted and then some. Of course, the SDK is still under development and we're obviously still early on with the preview program. People need to be patient.

    Microsoft: Any words of advice or encouragement for other developers about using Kinect for Widows v2?

    James: If you are a C# developer and you haven't made the plunge, now is a good time to start learning Visual C++. All of the powerful interaction and visually intensive things you might want to do are taking advantage of C++ libraries like Cinder, openFrameworks, PCL, and OpenCV. It requires being willing to feel stupid again for about six months, but at the end of that time, you'll be glad you made the effort.

    Our thanks to James for taking time to share his insights and experience with us. And as mentioned at the top of this post, you should definitely read James’ Kinect for Windows v2 First Look blog.

    Kinect for Windows Team

    Key links

  • Kinect for Windows Product Blog

    Course simplifies creation of WPF applications

    • 1 Comments

    KinectInteraction is a set of features, first introduced in Developer Toolkit 1.7, which allows Kinect-enabled applications to incorporate gesture-based interactivity. Developers can use KinectInteraction to create Windows Presentation Foundation (WPF) applications in which the movement of the user’s hand controls an on-screen hand, much like the movement of a mouse controls an on-screen cursor.

    The course teaches how to develop apps that use hand gestures to control an on-screen hand.The course teaches how to develop apps that use hand gestures to control an on-screen hand.

    András Velvart, a Kinect for Windows MVP, has created an online video course that provides step-by-step instructions on how to create such WPF applications and teaches students how to customize the look and feel of the controls provided by Microsoft. It even demonstrates how to completely control the interaction model or use KinectInteraction outside of WPF. The course, aptly named "KinectInteraction with WPF and Beyond,” is available through Pluralsight, an online training service for developers and IT professionals.  

    Kinect for Windows Team

     Key links

     

  • Kinect for Windows Product Blog

    Exploring v2 body imaging capabilities

    • 0 Comments

    In a pair of related blog posts, Zubair Ahmed, a Microsoft Most Valuable Professional nominee and a participant in the Kinect for Windows v2 developer preview program, put his new v2 Kinect for Windows sensor through its paces. In the first post, Zubair demonstrates how to use the body source data captured by the sensor to draw the bones, hands, and joints and overlay them on top of the color frame that comes from the sensor. The post includes the relevant code* and useful tips and tricks. 

    Zubair demonstrates the hand color frame received from the Kinect for Windows sensor.
    Zubair demonstrates the hand color frame received from the Kinect for Windows sensor.

    Zubair’s second post continues his deep dive into the body tracking of the v2 Kinect for Windows sensor. He refines his methods to eliminate a hack he had employed in the original code. In addition, he explains how to merge two body-image color frames and use a single image control to render them. This post not only includes the relevant code* and helpful tips; it also provides a demonstration video.

    Kinect for Windows Team

    Key links

    ________________________

    *This is preliminary software and/or hardware and APIs are preliminary and subject to change.

  • Kinect for Windows Product Blog

    My, you’ve grown…

    • 0 Comments

    That’s what you might be thinking as you scroll through the posts on this site.  That’s because we’ve merged the Kinect for Windows developer and product blogs. This union creates a “one-stop shop” for news about Kinect for Windows: a single source for learning about cool product applications plus the latest developer information. So, yeah, we’re fatter now—just think of it as more to love!          

                    Kinect for Windows Team

  • Kinect for Windows Product Blog

    Super Bowl ad showcases Kinect for Windows during surgery

    • 0 Comments

    It’s not often that we get to tell the Kinect for Windows story to millions of people at the same time, so being featured in a commercial during Super Bowl XLVIII Sunday, February 2, was a thrill. That Microsoft Super Bowl ad showed how technology is #empowering lives, including how GestSure, a Kinect for Windows solution, is helping surgeons and their patients.  Harnessing the power of Kinect for Windows to understand and respond to users’ movements, GestSure allows surgeons to use hand motions to study a patient’s medical images (X-rays as well as MRI and CT scans) on monitors in the operating room.  This eliminates the need for the surgeon to physically manipulate the images using a mouse or keyboard, and thus allows the surgery to continue unimpeded without the doctor having to leave the operating room to view images and spend time scrubbing back in. The result is a better flow of surgery and better care for patients.

    See more stories that celebrate what technology can do, including a short video that showcases GestSure.

    Kinect for Windows Team

    Key links

  • Kinect for Windows Product Blog

    Mysteries of Kinect for Windows Face Tracking output explained

    • 2 Comments

    Since the release of Kinect for Windows version 1.5, developers have been able to use the Face Tracking software development kit (SDK) to create applications that can track human faces in real time. Figure 1, an illustration from the Face Tracking documentation, displays 87 of the points used to track the face. Thirteen points are not illustrated here—more on those points later.

    Figure 1: Tracked Points
    Figure 1: Tracked Points

    You have questions...

    Based on feedback we received via comments and forum posts, it is clear there is some confusion regarding the face tracking points and the data values found when using the SDK sample code. The managed sample, FaceTrackingBasics-WPF, demonstrates how to visualize mesh data by displaying a 3D model representation on top of the color camera image.

    MeshModel - Copy
    Figure 2: Screenshot from FaceTrackingBasics-WPF

    By exploring this sample source code, you will find a set of helper functions under the Microsoft.Kinect.Toolkit.FaceTracking project, in particular GetProjected3DShape(). What many have found was the function returned a collection where the length of the array was 121 values. Additionally, some have also found an enum list, called “FeaturePoint”, that includes 70 items.

    We have answers...

    As you can see, we have two main sets of numbers that don't seem to add up. This is because these are two sets of values that are provided by the SDK:

    1. 3D Shape Points (mesh representation of the face): 121
    2. Tracked Points: 87 + 13

    The 3D Shape Points (121 of them) are the mesh vertices that make a 3D face model based on the Candide-3 wireframe.

    Figure 3: image from http://www.icg.isy.liu.se/candide/img/candide3_rot128.gif
    Figure 3: Wireframe of the Candide-3 model http://www.icg.isy.liu.se/candide/img/candide3_rot128.gif

    These vertices are morphed by the FaceTracking APIs to align with the face. The GetProjected3DShape method returns the vertices as an array of  Vector3DF[]. These values can be enumerated by name using the "FeaturePoint" list. For example, TopSkull, LeftCornerMouth, or OuterTopRightPupil. Figure 4 shows these values superimposed on top of the color frame. 

    FeaturePoints
    Figure 4: Feature Point index mapped on mesh model

    To get the 100 tracked points mentioned above, we need to dive more deeply into the APIs. The managed APIs, provide an FtInterop.cs file that defines an interface, IFTResult, which contains a Get2DShapePoints function. FtInterop is a wrapper for the native library that exposes its functionality to managed languages. Users of the unmanaged C++ API may have already seen this and figured it out. Get2DShapePoints is the function that will provide the 100 tracked points.

    If we have a look at the function, it doesn’t seem to be useful to a managed code developer:

    // STDMETHOD(Get2DShapePoints)(THIS_ FT_VECTOR2D** ppPoints, UINT* pPointCount) PURE;
    void Get2DShapePoints(out IntPtr pointsPtr, out uint pointCount);

    To get a better idea of how you can get a collection of points from IntPtr, we need to dive into the unmanaged function:

    /// <summary>
    /// Returns 2D (X,Y) coordinates of the key points on the aligned face in video frame coordinates.
    /// </summary>
    /// <param name="ppPoints">Array of 2D points (as FT_VECTOR2D).</param>
    /// <param name="pPointCount">Number of elements in ppPoints.</param>
    /// <returns>If the method succeeds, the return value is S_OK. If the method fails, the return value can be E_POINTER.</returns>
    STDMETHOD(Get2DShapePoints)(THIS_ FT_VECTOR2D** ppPoints, UINT* pPointCount) PURE; 

    The function will give us a pointer to the FT_VECTOR2D array. To consume the data from the pointer, we have to create a new function for use with managed code.

    The managed code

    First, you need to create an array to contain the data that is copied to managed memory. Since FT_VECTOR2D is an unmanaged structure, to marshal the data to the managed wrapper, we must have an equivalent data type to match. The managed version of this structure is PointF (structure that uses floats for x and y).

    Now that we have a data type, we need to convert IntPtr to PointF[]. Searching the code, we see that the FaceTrackFrame class wraps the IFTResult object. This also contains the GetProjected3DShape() function we used before, so this is a good candidate to add a new function, GetShapePoints. It will look something like this:

    // populates an array for the ShapePoints
    public void GetShapePoints(ref Vector2DF[] vector2DF)
    {
         // get the 2D tracked shapes
         IntPtr ptBuffer = IntPtr.Zero;
         uint ptCount = 0;
         this.faceTrackingResultPtr.Get2DShapePoints(out ptBuffer, out ptCount);
         if (ptCount == 0)
         {
            
    vector2DF = null;
             return;
         }
     
         // create a managed array to hold the values
         if (vector2DF == null || (vector2DF != null && vector2DF.Length != ptCount))
         {
             vector2DF = new Vector2DF[ptCount];
         }

         ulong sizeInBytes = (ulong)Marshal.SizeOf(typeof(Vector2DF));
         for (ulong i = 0; i < ptCount; i++)
         {
             vector2DF[i] = (Vector2DF)Marshal.PtrToStructure((IntPtr)((ulong)ptBuffer + (i * sizeInBytes)), typeof(Vector2DF));
         }
     } 

    To ensure we are using the data correctly, we refer to the documentation on Get2DShapePoints:

    IFTResult::Get2DShapePoints Method gets the (x,y) coordinates of the key points on the aligned face in video frame coordinates.

    The PointF values represent the mapped values on the color image. Since we know it matches the color frame, there is no need to do apply mapping. You can call the function to get the data, which should align to the color image coordinates.

    The Sample Code

    The modified version of FaceTrackingBasics-WPF is available in the sample code that can be downloaded from CodePlex. It has been modified to allow you to display the feature points (by name or by index value) and toggle the mesh drawing. Because of the way WPF renders, the performance can suffer on machines with lower end graphics cards. I recommend that you only enable these one at a time. If your UI becomes unresponsive, you can block the sensor with your hand to prevent FaceTracking data capturing. Since the application will not detect any face tracked data, it will not render any points, giving you the opportunity to reset the features you enabled by using the UI controls.

    ShapePoints
    Figure 5: ShapePoints mapped around the face

    As you can see in Figure 5, the additional 13 points are the center of the eyes, the tip of the nose, and the areas above the eyebrows on the forehead. Once you enable a feature and tracking begins, you can zoom into the center and see the values more clearly.

    A summary of the changes:

    MainWindows.xaml/.cs:

    • UI changes to enable slider and draw selections

     

    FaceTrackingViewer.cs:

    • Added a Grid control – used for the UI elements
    • Modified the constructor to initialize grid
    • Modified the OnAllFrameReady event
      • For any tracked skeletons, create a canvas and add to the grid. Use that as the parent to put the label controls

    public partial class FaceTrackingViewer : UserControl, IDisposable
    {
         private Grid grid;

         public FaceTrackingViewer()
         {
             this.InitializeComponent();

             // add grid to the layout
             this.grid = new Grid();
             this.grid.Background = Brushes.Transparent;
             this.Content = this.grid;
         }

         private void OnAllFramesReady(object sender, AllFramesReadyEventArgs allFramesReadyEventArgs)
         {
             ...
             // We want keep a record of any skeleton, tracked or untracked.
             if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
             {
                 // create a new canvas for each tracker
                 Canvas canvas = new Canvas();
                 canvas.Background = Brushes.Transparent;
                 this.grid.Children.Add( canvas );
                
                 this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker(canvas));
             }
             ...
         }
    }

    SkeletonFaceTracker class changes:

    • New property: DrawFraceMesh, DrawShapePoints, DrawFeaturePoint, featurePoints, lastDrawFeaturePoints, shapePoints, labelControls, Canvas
    • New functions: FindTextControl UpdateTextControls, RemoveAllFromCanvas, SetShapePointsLocations, SetFeaturePointsLocations
    • Added the constructor to keep track of the parent control
    • Changed the DrawFaceModel function to draw based on what data was selected
    • Updated the OnFrameReady event to recalculate the positions based for the drawn elements
      • If DrawShapePoints is selected, then we call our new function

    private class SkeletonFaceTracker : IDisposable
    {
    ...
        // properties to toggle rendering 3D mesh, shape points and feature points
        public bool DrawFaceMesh { get; set; }
        public bool DrawShapePoints { get; set; }
        public DrawFeaturePoint DrawFeaturePoints { get; set; }

        // defined array for the feature points
        private Array featurePoints;
        private DrawFeaturePoint lastDrawFeaturePoints;

        // array for Points to be used in shape points rendering
        private PointF[] shapePoints;

        // map to hold the label controls for the overlay
        private Dictionary<string, Label> labelControls;

        // canvas control for new text rendering
        private Canvas Canvas;

        // canvas is passed in for every instance
        public SkeletonFaceTracker(Canvas canvas)
        {
            this.Canvas = canvas;
        }

        public void DrawFaceModel(DrawingContext drawingContext)
        {
            ...
            // only draw if selected
            if (this.DrawFaceMesh && this.facePoints != null)
            {
                ...
            }
        }

        internal void OnFrameReady(KinectSensor kinectSensor, ColorImageFormat colorImageFormat, byte[] colorImage, DepthImageFormat depthImageFormat, short[] depthImage, Skeleton skeletonOfInterest)
        {
            ...
            if (this.lastFaceTrackSucceeded)
            {
                ...
                if (this.DrawFaceMesh || this.DrawFeaturePoints != DrawFeaturePoint.None)
                {
                    this.facePoints = frame.GetProjected3DShape();
                }

                // get the shape points array
                if (this.DrawShapePoints)
                {
                    this.shapePoints = frame.GetShapePoints();
                }
            }

            // draw/remove the components
            SetFeaturePointsLocations();
            SetShapePointsLocations();
        }

        ...
    }

    Pulling it all together...

    As we have seen, there are two types of data points that are available from the Face Tracking SDK:

    • Shape Points: data used to track the face
    • Mesh Data: vertices of the 3D model from the GetProjected3DShape() function
    • FeaturePoints: named vertices on the 3D model that play a significant role in face tracking

    To get the shape point data, we have to extend the current managed wrapper with a new function that will handle the interop with the native API.

    Carmine Sirignano
    Developer Support Escalation Engineer
    Kinect for Windows

    Additional Resources:

     

  • Kinect for Windows Product Blog

    Mirror, mirror, on the screen…who’s the fairest on the scene?

    • 0 Comments

    One of the highlights of the recent Consumer Electronics Show was the three-dimensional augmented reality Beauty Mirror shown by ModiFace, a leader in virtual makeover technology. With help from a Kinect for Windows sensor and a PC, the Beauty Mirror enables customers to view the simulated effects of skin-care and beauty products and anti-aging procedures from all angles of their face. It also allows customers to compare the before-and-after results side-by-side. This proprietary technology can simulate the application of blushes, lipsticks, eye shadows, and other makeup products. Moreover, it can display the impact of such anti-aging procedures as dark spot correction, facelift, browlift, cheek volume enhancement, and jaw contouring. According to Parham Aarabi, the CEO of ModiFace, “the Kinect for Windows sensor captures the customer’s image, enabling the creation of real-time, full-3D simulations that utilize ModiFace's patented photorealistic skin and shade simulation technology.”

    Kinect for Windows Team

    Key links

Page 3 of 11 (103 items) 12345»