Keep track of all the latest news and events on developer tools and technologies you care about
I find the concept of Natural User Interface fascinating so after talking to Kris Athi about Surface 2 and the development opportunities it poses, I asked him if he’d mind writing us a post about its history and how you can develop for it, below is the great article he put together for us.
Kris is a Technologist working for Content Master. He develops in WPF, Silverlight and Windows Phone 7. He's also a former Microsoft Student Partner. You can find his blog here.
The Microsoft Surface journey began in June 2002 when thousands of movie goers flooded to see the Steven Spielberg blockbuster, Minority Report. It was that movie that saw the likes of Tom cruise and Colin Farrell using some Impressive wall screen technology in the near future to help them prevent the crimes of the future. It was one of those moments where you see it and think “If only we had that!”
The good news is we don’t have to wait until 2054 for such technology. The ideas seen in the movie reflect the high-level concepts of what Microsoft calls the Natural User Interface (or NUI).
So, what exactly is a NUI? Let’s take a look.
A NUI does not use a keyboard or a mouse. The Interaction with the UI is performed using Touch. This is generally gesture based and offers a more seamless experience with users performing actions they are used to doing in everyday life.
The Microsoft Surface device is a prime example of a NUI implementation. The Microsoft Surface 2 SUR40 (manufactured by Samsung) comes equipped with a 40-inch, 4-inch-thick, 1080p screen and is powered an AMD Athlon II X2 processor and a Radeon HD 6700M graphics card. The SUR40 also uses Gorilla Glass, a type of glass that can withstand more abuse than regular glass.
You can interact with this device directly, and because the device is a large flat panel, it encourages multiuser interaction.
With this sort of technology, a video is worth a thousand words so I encourage you to check out the device in action:
Let us now take a look at the Surface Architecture and software platform
You may be surprised to hear that the operating system powering the device is a standard installation of Windows 7 Embedded 64bit, and this provides all the hardware interaction as it would on a standard PC. The device also has support for interacting with Windows Phone 7.
However, the surface Interface doesn't look anything like a standard Windows 7 interface and that’s because of the Windows integration layer.
This layer is responsible for hiding the Win7 UI. For example, if an error is detected and Win7 throws up an error message, the integration layer catches it and kills it before it is displayed to the user.
This allows the Shell UI, to run without any OS interruption that you might expect on a normal PC. The Shell includes the attract application that is used to entice users over to the device and “Launcher” which is the carousel type menu structure displaying all of the applications installed on the device.
The Surface Vision system uses PixelSense technology. PixelSense allows a display to recognize fingers, hands, and objects placed on the screen, enabling vision-based interaction without the use of cameras. The individual pixels in the display see what’s touching the screen using and that information is immediately processed and interpreted
Let’s take a closer look at the Core layer.
This layer opens up the device in a primitive state and allows you to write applications with any .NET based UI framework.
As the name suggests, only the core functionality is exposed and it requires you to implement things like hit testing from scratch. While this sounds complicated, it can be a good thing as total control can lead to more advanced applications. This layer is not tied to a framework but is mostly used for XNA 4.0, which is the managed game development framework that can be used to make high performance 3D based applications.
The WPF layer is built on the Core layer and enables the typical WPF 4.0 development workflow to be used in Surface development. This means that everything you know and love about WPF development, such as XAML and the use of Expression Blend, are brought into Surface. The SDK ships with various Surface-specific controls to speed up the development process. These include the ScatterView, which holds and displays a collection of objects with built in manipulation and inertia functionality.
As well at these controls, there are also “Surfacefied” versions of the standard WPF controls that are optimised for the platform. For example, using a Surface Textbox will automatically invoke an on screen keyboard to be displayed when it is touched.
However, while the development model may be similar, the target platform is, of course, all new. This means there are several new design patterns you will need to take on board while designing a Surface application.
First, the screen is flat, horizontal and has no mouse and keyboard. Everything is touch driven and if the device is positioned flat, your applications can function as if there is also no “top” and give a 360 degree UI feel that is viewable and workable from any position. You can also place a surface device on the wall to work like a touch screen flat TV.
Surface applications should also be seamless, responsive and never allow the user to become lost. This is to keep users immersed in the new experience. Applications should also be “social” as the nature of the device encourages collaboration between people gathered around the device.
If you’re interested in any of the topics discussed in this short article, check out the getting started page on the Surface website. Here you can find a preview version of the SDK as well as other resources to get you started.
There is a free video/lab based training course that is based on version 1.0 of the platform (with 2.0 currently under development). Each module is divided into video-based information delivery and then a short hands-on lab to put the knowledge into practice.
While Surface development isn’t something the average developer will encounter during their day job, it is something a bit different you may want to take a look at.