Microsoft Research is exploring the future of interaction, including direct 3D interactions with augmented reality.

Today’s technology

Let’s take a look at the technology that is already out there:

  • Sensors. Today’s mobile devices have sensors that make Star Trek’s Tricorder look medieval. For example, a typical smart phone has a compass, accelerometers, gyros, GPS and cameras. Combined with a screen, you can already experience augmented reality. For example, the SLARToolkit for Windows Phone and 'Why I can't embrace Augmented Reality' on Ubelly. 
  • Gestures. Kinect gives computers the ability to ‘see’ gestures and track movement.
  • Pico projectors. You can already buy tiny, handheld ‘pico projectors’ such as the HP Notebook Projection Companion and they’re even built into some cameras now. If they were built into phones or laptops, they could project a 3D environment into the physical world.
  • 3D graphics. Computers and even phones have the ability to display convincing 3D graphics and, thanks to KinectFusion, it’s also possible to capture 3D environments from the real world in real time. Checkout this video from Microsoft Research to see a computer build up an awareness of its entire environment and Ben Kacrya’s TED talk about long-range 3D scanning.

Tomorrow’s user interfaces

Each of these technologies gives developers new capabilities and lets them offer users new experiences. The challenge is to figure out what to do with them. This is where Microsoft Research is very active, combining different elements to create new kinds of interactivity, including

  • Holodesk is an experimental system developed by Microsoft Research that brings many of these technologies together to create something very similar to Star Trek’s holodeck. Mixing multiple sensors, physical objects, 3D graphics and intuitive gesture recognition, it lets a user interact with virtual objects almost as if they were real.

  • Wearable Multitouch Interaction, which is an experimental system that letspeople use their hands, arms and legs as graphical interactive surfaces – as the touchscreen of a phone, for example – and interact with them using touch gestures, as in this Microsoft Research video.

Like all research projects, these demos are pre-production versions of soon-to-be-released commercial products. What they do is point to new possibilities and capabilities.

By the way, if you're interested in Augmented Reality, take a look at the new StarChart app for Windows Phone for a pretty cool example.