Anson Tsao's Blog

Microsoft Surface Toolkit for Windows Touch Beta Release!

AnsonT — Mon, 12 Apr 2010 22:27:25 GMT

Our friends over at Surface just released the beta toolkit built on WPF4.

The toolkit is usable with any multi-touch enabled Windows 7 display, as well as the Multipoint SDK via the custom TouchDevice from my previous blog entry.

Custom Touch Devices

AnsonT — Sat, 30 Jan 2010 17:08:00 GMT

WPF4 allows applications to implement custom touch devices. We added this functionality to allow Surface SDK to be built on WPF multi-touch APIs directly. As we announced at PDC09, the next version of the Surface SDK will implement WPF multi-touch devices with the input from the Surface vision system.

A few weeks ago, Microsoft Multipoint SDK version 1.5 was released. Multipoint SDK supports using up-to 25 USB mice as input devices on the same PC. While the original intent of the Multipoint SDK is to allow multiple students to collaborate and share a single PC, the multiple mice support is perfect for simulating multi-touch input.

In this post, I will explain how to implement custom touch devices in WPF4 and use the Multipoint SDK integration as an example.

TouchDevice

TouchDevice is an abstract class in the System.Windows.Input namespace. An instance of TouchDevice represents one touch contact. To add a new type of touch input to WPF, you need to derive from TouchDevice, override the methods to provide current positions of the touch points, and notify the input system of state changes via the ReportDown/Move/Up and Activate/Deactivate methods on TouchDevice.

Deriving from TouchDevice and TouchPoint

TouchDevice constructor takes an integer device ID as parameter. Device IDs should be unique for your set of touch devices, it is not enforced by the system. For developers, it is sufficient to use the instance of the TouchDevice as a key, and it is guaranteed to be valid and consistent during an interaction cycle.

TouchDevice has 2 public abstract methods that you have to override to provide the position of the touch contact.

TouchPoint GetTouchPoint(IInputElement relativeTo) method returns the current position of the touch contact relative to the coordinate system of ‘relativeTo’ element.

To calculate the transformation from the top-level window coordinate system to the ‘relativeTo’ element’s system, you can calculate the transformation via the Visual.TransformToDescendant() method and Transform() the toucd point accordingly.

TouchPointCollection GetIntermediateTouchPoints(IInputElement relativeTo) method returns a collection of intermediate touch points that was collected but not reported since the last touch event. This is used to aggregate a sequence of high frequency touch positions and not overwhelm the system with individual Touch events. For most applications, it’s sufficient to return an empty collection and call ReportMove() on every detected touch movements.

You can provide additional information about your touch contact by deriving from TouchPoint. For example, Surface also associates elliptical bounds and tag ID with TouchPoint, these attributes are not available via the Windows touch device drivers.

Updating TouchDevice States

Each TouchDevice needs to be associated with a PresentationSource which is typically the active top level window. Next you have to Activate() the device. To report touch point activity, you will be using ReportDown(), ReportMove() and ReportUp() methods. You need to make sure that each Activate() is accompanied by an Deactivate() call, and each ReportDown() has a corresporting ReportUp(), otherwise the input system will throw an exception.

Use multiple mice for multi-touch input

In this example we will be using the MultiPoint SDK to provide multi-touch input. The full source code is here.

First, we need to reference the MultiPoint assembly from the SDK

We will create a MultipointTouchWindow that receives the Multipoint events, and translate into the touch events. With modifications to the Multipoint SDK, we can provide multi-touch event more efficiently, but we will have to wait for a future release of the SDK.

public class MultipointTouchWindow : Window, IMultipointMouseEvents

    public MultipointTouchWindow()

        Loaded += HandleLoaded;

    private void HandleLoaded(object sender, RoutedEventArgs args)

        MultipointSdk.Instance.Initialize(this);

        MultipointSdk.Instance.DeviceArrivalEvent += HandleMultipointDeviceArrival;

        foreach (var device in MultipointSdk.Instance.MouseDeviceList)

            device.DeviceVisual.CursorColor = NextColor();

The window class has to implement the IMultipointMouseEvents interface for the Multipoint SDK to provide mouse events. Default implementations of the events in the interface is sufficient.

The Multipoint SDK instance is initialized after the WPF window is loaded, and the corresponding HWD has been created.

We also assign different colors to the cursor associated with the each mouse device. NextColor() is a simple utility method that returns a different color every time it’s called.

Multipoint works best with full screen windows. In WPF you need to set the WindowState to Maximized and WindowStyle to None.

Next we hook up the preview Multipoint mouse events

static MultipointTouchWindow()

    EventManager.RegisterClassHandler(

                typeof(UIElement),

                MultipointMouseEvents.MultipointPreviewMouseDownEvent,

                (RoutedEventHandler)HandleMultipointMouseDown,

                false );

    EventManager.RegisterClassHandler(typeof(UIElement),

                MultipointMouseEvents.MultipointPreviewMouseMoveEvent,

                (RoutedEventHandler)HandleMultipointMouseMove,

                false);

    EventManager.RegisterClassHandler(typeof(UIElement),

                MultipointMouseEvents.MultipointPreviewMouseUpEvent,

                (RoutedEventHandler)HandleMultipointMouseUp,

                false);

The handlers of these events simply delegate to static methods of the MultipointTouchDevice that we’ll discuss next.

MultipointTouchDevice

MultipointTouchDevice derives from TouchDevice. It maintains a static mapping of Multipoint device Ids to their corresponding TouchDevice instances. As we handle Multipoint mouse events, we retrieve the corresponding MultipointTouchDevice and updates its state.

public static void MouseDown(MultipointMouseEventArgs e)

     var device = GetDevice(e);

     device.SetActiveSource(PresentationSource.FromVisual(e.CurrentWindow));

     device.Position = GetPosition(e);

     device.Activate();

     device.ReportDown();

The MouseMove and MouseUp methods are equally simple, first updating the device’s position, then calling ReportMove() or ReportUp(). We deactivate the device after the a mouse up event.

MultipointTouchDevice.Position is a simple property of the current position in Window coordinate system. GetPosition() is a helper function that retrieves the position from MultipointMouseEventArgs.

public override TouchPoint GetTouchPoint(IInputElement relativeTo)

    Point pt = Position;

    if (relativeTo != null)

        pt = this.ActiveSource.RootVisual.

             TransformToDescendant((Visual)relativeTo).

             Transform(Position);

    var rect = new Rect(pt, new Size(1.0, 1.0));

    return new TouchPoint(this, pt, rect, TouchAction.Move);

The GetTouchPoint() implementation simply transform the Position to the appropriate coordinate system, and return the point as having a 1.0 by 1.0 pixel size.

Pictionaire: Multi-touch table that integrates physical and digital objects from Microsoft Research

AnsonT — Sat, 30 Jan 2010 13:55:00 GMT

Another interesting MSR multi-touch project: Pictionaire This research project combines computer object recognition and Surface style multitouch interactions for collaboration.

It’s based on a modified version of the Surface SDK

It is really easy to add custom touch devices in WPF4 (I’ll write about that in the next post). I'm hoping to see a lot more experimentations around new multi-touch input methods in the community.

Multi-touch in WPF 4 Part 3 – Manipulation Events

AnsonT — Mon, 11 Jan 2010 08:03:00 GMT

One of the most common usage of multi-touch input is for panning, zooming and rotation. With WPF4, the easiest way to implement these gestures is by using the Manipulation events on UIElements. Manipulation events also support simple inertial physics for a more fluid user experience.

You may well wonder why we call these ‘manipulation’ events rather than gestures like Win32’s WM_GESTURE messages. With manipulation events, we interpret the multi-touch input to simulate directly manipulating the on screen elements, as if you are using your fingers to move, rotate and stretch physical objects. Manipulation events reports translation, rotation and scaling transformations simultaneously, whereas with WM_GESTURES, you can only get one of these transformations components at one time. Thus with WM_GESTURE, you cannot pan and zoom at the same time.

There are typically 2 ways of using manipulation events: 1) to interpret the events for panning/zooming/rotating content such as maps and images, 2) to interpret these events for manipulating onscreen elements such as organizing a deck of cards, moving puzzle pieces etc. Depending on your scenario, you will decide which elements to enable manipulation on and which element you use as the manipulation container.

Manipulation Events

Manipulation events on the UIElement is a state machine for the interaction sequence.

Getting Started

There are 3 simple concepts that you need to understand to get started with manipulations.

First, you need to enable the IsManipulationEnabled boolean property on elements that are to be manipulated. Setting this property to true on an UIElement will start hit-testing of touch events on the element and raising manipulation events when you drag the element with one or more touch contacts.

Second, you need to handle the ManipulationStarting event and specify the ManipulationContainer for the interaction. The ManipulationStarting event is a routed event that is raised from an UIElement with IsManipulationEnabled set to true. This event is raised after the first touch down on the UIElement and before any other manipulation events. This event is used to configure the manipulation processing logic.

The ManipulationContainer is an element that act as the frame of reference for the manipulation transformation calculations. You can imagine that the manipulation container is the physical surface that the manipulated elements are moving against. The manipulation container cannot be changed during the interaction sequence between ManipulationStarting and ManipulationCompleted.

Last, you need to handle at least the ManipulationDelta event to respond to the transformations calculated from the finger movements. The ManipulationDelta event reports pan, zoom, and rotate as separate components to the transformation. The transformation is reported both as delta since the last event (DeltaManipulation) and since the the start of the manipulation (CumulativeManipulation).

Basic example

In this example, we enable multi-touch manipulation of a red rectangle element on a canvas. You can move the rectangle with 1 finger and rotate and zoom with 2 or more fingers. The following XAML snippet shows hooking the manipulation events and enabling manipulation on the rectangle.

<Canvas x:Name="_canvas"        
        ManipulationStarting="_canvas_ManipulationStarting"        
        ManipulationDelta="_canvas_ManipulationDelta">    
    <Rectangle IsManipulationEnabled="True"               
               Fill="Red" Width="100" Height="100"/>
</Canvas>

We will be applying render transformation on the rectangle, the transformation is relative to the canvas so we will use the canvas as the manipulation container.

private void _canvas_ManipulationStarting(object sender, 
                              ManipulationStartingEventArgs e)
{
    e.ManipulationContainer = _canvas;
    e.Handled = true;
}

We handle the ManipulationDelta event to apply the render transformation to the rectangle. First we retrieve the rectangle as the original source of the manipulation event. Then we extract the current render transformation as a MatrixTransform.

To calculate the effects of the manipulation, we apply the scale, rotation and translation transformation components from the event argument to the current render transform. In matrix calculations, the order of transformation (multiplication) is important. We need to apply scaling and rotation first centered at the manipulation origin before moving the rectangle based on the translation component.

private void _canvas_ManipulationDelta(object sender,
                                 ManipulationDeltaEventArgs e)
{

    var element = e.OriginalSource as UIElement;

    var transformation = element.RenderTransform 
                                         as MatrixTransform;
    var matrix = transformation == null ? Matrix.Identity : 
                                   transformation.Matrix;
    
    matrix.ScaleAt(e.DeltaManipulation.Scale.X, 
                   e.DeltaManipulation.Scale.Y,
                   e.ManipulationOrigin.X, 
                   e.ManipulationOrigin.Y);
    
    matrix.RotateAt(e.DeltaManipulation.Rotation,
                    e.ManipulationOrigin.X,
                    e.ManipulationOrigin.Y);
    
    matrix.Translate(e.DeltaManipulation.Translation.X, 
                     e.DeltaManipulation.Translation.Y);
    
    element.RenderTransform = new MatrixTransform(matrix);
    e.Handled = true;
}

That’s it!

In the next parts of the series, I’ll talk about inertial movement and single finger rotation with manipulation pivot.

Multi-touch in WPF 4 - Part 2

AnsonT — Fri, 11 Dec 2009 19:09:00 GMT

Raw Touch Events

Touch events are the equivalent of mouse events for multi-touch input. In WPF4, we’ve added touch events to UIElement, UIElement3D and ContentElements. Multi-touch events follow the same patterns as mouse events, with TouchDown, TouchMove and TouchUp, the Preview variants and TouchEnter and TouchLeave. Multi-touch events in WPF are hit-tested and routed through the element tree just like other input events. This is different from the low-level application wide touch event support that is in SL3, I’ll discuss that later.

(For Microsoft Surface SDK developers, TouchDevice corresponds to the Contact class, and Touch events correspond to Contact events).

Each touch contact (finger) is associated with a different TouchDevice. You can use the instance or the Id of the TouchDevice to keep track of the contact between TouchDown and TouchUp events.

WPF also supports the notion of multiple capture for touch events. When an input device is captured to an element, all the input events from that device are routed from that element regardless of whether the input is directly over the element, no hit-testing need to be performed. With multi-touch events, each element can capture multiple TouchDevices and multiple elements can capture simultaneously. This feature allow WPF to support multiple simultaneous active controls, and one of the main differences between the WPF and the native Win32 touch support.

The following is a simple example of handling multi-touch events. In this example, we place a shape on a canvas under each finger, and track the shapes while the fingers are moving.

First we hook up the touch event handlers on the canvas

<Canvas x:Name="_canvas"                    Background="LightYellow"                   TouchDown="_canvas_TouchDown"         TouchMove="_canvas_TouchMove"         TouchUp="_canvas_TouchUp">   </Canvas>

The TouchDown handler creates a shape for the particular touch device, adds the shape to the canvas, moves it under the finger position and stashes it away in a dictionary for later.

private void _canvas_TouchDown(object sender, TouchEventArgs e) {     // New Shape for Touch Device     var shape = CreateSomeShape();     // Get Touch Point relative to _canvas coordinates     var origin = e.GetTouchPoint(_canvas);    // Move the shape to the touch point     shape.RenderTransform = new TranslateTransform                 (origin.Position.X-shape.RenderSize.Width/2,                   origin.Position.Y-shape.RenderSize.Height/2);     // Stash away the shape     _shapes[e.TouchDevice] = shape;     // Add the shape to the canvas     _canvas.Children.Add(shape);     _canvas.InvalidateVisual();     // Capture the touch device     _canvas.CaptureTouch(e.TouchDevice); }

The TouchMove handler retrieves the shape associated with the touch device and moves it under the current finger position

private void _canvas_TouchMove(object sender, TouchEventArgs e) {      if (e.TouchDevice.Captured == _canvas)      {          // Retrieve the shape associated with the device          var shape = _shapes[e.TouchDevice];          // Get current touch point          var origin = e.GetTouchPoint(_canvas);          // Move the shape to the new position          shape.RenderTransform = new TranslateTransform(                   origin.Position.X - shape.RenderSize.Width / 2,                    origin.Position.Y - shape.RenderSize.Height / 2);      } }

The TouchUp handler removes the associated shape from the canvas, and removes it from the dictionary.

private void _canvas_TouchUp(object sender, TouchEventArgs e)   {       // Release the captured device       _canvas.ReleaseTouchCapture(e.TouchDevice);       // Clean up the shape       _canvas.Children.Remove(_trails[e.TouchDevice]);       _shapes.Remove(e.TouchDevice);   }

Multi-touch in Silverlight 3 and 4

Silverlight also supports multi-touch input on Windows 7. Both Silverlight and WPF allows you to retrieve the raw window level multi-touch input via the static Touch.FrameReported event.

GetTouchPoints returns the touch device positions relative to a particular element. Touch Up/Move/Down events are identified in the Action property of the TouchPoint class.

GetPrimaryTouchPoint returns the position of the first finger down in this multi-touch input sequence

Calling SuspendMousePromotionUntilTouchUp will prevent the touch devices from raising the equivalent mouse events. This allows panning to be implemented without also invoking mouse interactions like button clicks or listbox selections.

To support Pan/Zoom/Rotate gestures in SL, Surface will be providing the System.Windows.Input.Manipluations.DLL (the .NET4 core manipulation and inertia processors) as a separate download in the future. This assembly is part of .NET4, and used to implement the WPF4 manipulation events.

Touch User Experience

There are some subtle differences in user experience when dealing with multiple touches that are not immediately obvious. Let’s take Button as an example. With mouse, the button click event is raised when the mouse button is released over the button. However, if you use multiple fingers to press a button, you only want to raise the click event when the LAST finger is lifted from the button. All of the WPF and SL controls are built based on mouse events, luckily with the availability of the Surface SDK for WPF4, you will be able to get the ‘correct’ multi-touch user experience without having to write your own controls.

Stunning data visualization in the AlloSphere

AnsonT — Tue, 08 Dec 2009 21:48:32 GMT

Really amazing work bringing together scientific data, 3D visualization, interactivity, and music.

Multi-touch in WPF 4 – Part 1

AnsonT — Thu, 03 Dec 2009 17:26:17 GMT

During PDC 09, I co-presented a multi-touch session with Robert Levy from Surface. (BTW, You have got to see the monster demo that Robert gave on the Surface unit.) We made a couple of significant announcements for WPF multi-touch developers.

First, in post beta 2 builds of WPF, we will have built-in touch panning support for standard controls (e.g. Listbox, ListView, DataGrid, TextBox, ScrollViewer, etc.). This means that any WPF4 application will get basic touch support for panning without any additional work from the developer.

Second, Robert announced that Surface will be making the Surface multi-touch controls available to all WPF developers to run on Windows 7. This is really great news for WPF developers. Now you can write fully multi-touch themed and optimized applications using the Surface controls that run anywhere. These will be available as a separate download around the time of Visual Studio 2010 release.

I also presented a deep dive into the WPF 4 multi-touch APIs in a chalk talk at the Web Pavilion. Unfortunately that session was not recorded. I will share the slides and samples on this blog, and in future posts I will go into more details about each of the samples.