Stefan Wick's Weblog - Development with Silverlight, WPF and Tablet PC : Tablet Development Gotchas

TabletPC Development Gotchas Part 7: Detecting the input device in Silverlight 2

swick — Sat, 01 Nov 2008 20:07:00 GMT

When writing an application that supports multiple input devices (mouse, stylus, touch) it is often neccessary to detect which type of device is sending the input events.Silverlight 1.0 offered a DeviceType property on the StylusInfo object that gets reported in every mouse event.

In the Silverlight 2 managed API, you will not find an equivelnt object that provides this information. We omitted it on purpose in this release, because adding this to the OM could not be done in a way that would make the Silverlight API a compatible subset of the WPF API. In future versions of both platforms we will provide a consistent way to get to this information.

Omitting this for now is acceptable since we do provide a workaround to get to this information, via the HTML DOM bridge. Here is how you do it (full source attached):

Expose a scriptable method from managed code to handle the DeviceType information:

public Page()

{

InitializeComponent();

HtmlPage.RegisterScriptableObject("page", this);

}

[ScriptableMember]

public void SetDeviceType(string deviceType)

{

_deviceType = deviceType;

switch (_deviceType)

{

case "Mouse":

// add specific code for mouse

break;

case "Stylus":

// add specific code for stylus

break;

case "Touch":

// add specific code for touch

break;

}

Hook up to the MouseLeftButtonDown event in JavaScript and pass the StylusInfo.DeviceType information to managed code when the event fires:

function onSilverlightLoad(sender, args) {

// attach a JavaScript MouseDown event handler

var inkPresenter = sender._element.content.findName("inkPresenter");

inkPresenter.addEventListener("mouseLeftButtonDown", onMouseDown);

}

function onMouseDown(sender, args) {

// get the device type info and pass

// the information to the managed code

var stylusInfo = args.getStylusInfo();

var plugin = sender.getHost();

plugin.content.page.SetDeviceType(stylusInfo.deviceType.toString());

}

</script>

Now, as a proud owner of a Toshiba M700, I can run the solution and collect ink from different devices and the app automatically changes the pen attributes as I switch between input devices. Here is a screenshot showing ink strokes collected with the three different devices available on my Tablet PC:

TabletPC Development Gotchas Part 6: InkCanvas Element Selection/Move/Resize

swick — Tue, 18 Dec 2007 20:46:00 GMT

WPF's InkCanvas element provides a lot of built-in functionality for several common, ink-related tasks like stylus gesture recognition, point and stroke erase, as well as the selection, resizing and moving of ink strokes. The key to those features is the 'EditingMode' property - which is nicely demonstrated in the InkCanvas EditingModes SDK sample (btw, it also demonstrates an implementation of an Undo/Redo stack for ink operations).

One little known, yet very cool feature is not demonstrated in this sample, though: The selection, resizing and moving functioality (i.e. EditingMode="Select") can not only be applied to ink strokes, but also to child elements of the InkCanvas. This can be very handy if you want to build, for example, a note-taking application that can also host text, pictures and other content besides the handwritten ink. You can then use the 'Select" mode to let the user re-arrange and resize all their content.

The "gotcha" I wanted to point in this blog post is a limitation in this feature: Child elements in an InkCanvas may be positioned in a variety of different ways: for example you can position them by setting a "Margin" or by setting any combination of "Canvas.Left/Right/Top/Bottom" attached properties. InkCanvas's selection/moving/resizing feature, however, only works on elements that have absolute positioning set via "Canvas.Left/Top".

Let's look at the following example markup:

<Window x:Class="InkCanvasElementSelection.Window1"

xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

Title="InkCanvasElementSelection"

ResizeMode="NoResize"

Height="450" Width="450">

<Window.Resources>

</Style>

</Window.Resources>

<Grid>

<InkCanvas Name="inkCanvas" EditingMode="Select"

Background="LightGoldenrodYellow"

SelectionChanging="OnSelectionChanging">

<Rectangle.RenderTransform>

</Rectangle.RenderTransform>

</Rectangle>

</InkCanvas>

</Grid>

</Window>

Note that the markup subscribes to the 'SelectionChanging' event on the InkCanvas. Let's leave the implementation of that event handler empty for now to demonstrate the limitation. Also note that the four rectangles use different ways to position themselves on the InkCanvas. Only the blue rectangle uses the "Canvas.Left/Top" attached properties for positioning.

Now you can use the lasso tool to select one or more elements, and then try to move and resize them. You will notice that it only works correctly for the blue rectangle (because that one has been positioned via "Canvas.Left/Top"). For the other rectangles you won't get the desired move or resize results.

Now what can I do if there are child elements that have not been positioned via "Canvas.Left/Top"? Here is a workaround to make this feature work even for those elements: In the 'SelectionChanging' event - which fires whenever you change what content is selected - you can walk the list of child elements and ensure they are positioned via "Canvas.Left/Top". Here is a piece you can add to the above sample to demonstrate this approach:

void OnSelectionChanging(object sender, InkCanvasSelectionChangingEventArgs e)

{

ReadOnlyCollection<UIElement> elements = e.GetSelectedElements();

foreach (UIElement element in elements)

{

// obtain actual location of element relative to InkCanvas

Point locationInInkCanvas = element.TranslatePoint(new Point(0, 0),

inkCanvas);

// set the location via Left/Top properties

element.SetValue(InkCanvas.LeftProperty, locationInInkCanvas.X);

element.SetValue(InkCanvas.TopProperty, locationInInkCanvas.Y);

// un-set right/bottom properties

element.SetValue(InkCanvas.RightProperty, double.NaN);

element.SetValue(InkCanvas.BottomProperty, double.NaN);

// re-translate any render transform

Matrix matRender = element.RenderTransform.Value;

matRender.Translate(-matRender.OffsetX, -matRender.OffsetY);

element.RenderTransform = new MatrixTransform(matRender);

// set margins to zero

if (element is FrameworkElement)

{

((FrameworkElement)element).Margin = new Thickness(0d);

}

Now if re-run the sample you will find all rectangles from the original markup can be selected, moved and resized as expected:

Full source code of this example is included in the attached Visual Studio project.

TabletPC Development Gotchas Part 5: Rendering ink and image to a bitmap using Winforms

swick — Sun, 02 Dec 2007 08:19:00 GMT

Annotating pictures with handwriting or drawings is a fun and useful scenario for digital ink. In many cases the application user wants to create a new image file with the ink annotation "burnt" into the picture, so they can use the result of their work in other contexts and with other applications (e.g. attach the annotated image to an e-mail or upload it to their personal web site or blog, like the image below).

So rendering ink on top of an image is a common scenario for TabletPC/Ink developers - and it should be real easy, right? Well, there is an important 'gotcha' you need to know about if you want to do this from Winforms. But let's start with native code first.

Using the COM Ink API in native code

If you are using the ActiveX version of InkPicture (or the COM InkOverlay component) then rendering its ink to a GDI bitmap is easy. Just get the hDC (handle to device context) of the bitmap and use the InkRenderer.Draw() method:

pInkRenderer->Draw(hDCtargetBitmap, pStrokesToRender);

Using the Winforms Ink API to target XP (Microsoft.Ink v1.7)

In managed Winforms code, the object that corresponds to the hDC is the System.Drawing.Graphics object. So the intuitive approach would be to get the Graphics object of the Winforms target bitmap and then use the Microsoft.Ink.Render.Draw() method to render the desired ink strokes on top of it. Something along the lines of this:

Graphics graphicsTarget = Graphics.FromImage(imageToSave);

inkPicture1.Renderer.Draw(graphicsTarget,

                            inkPicture1.Ink.Strokes);

imageToSave.Save(“InkAndImage.bmp”);

Unfortunately, the result of this operation does not look great: the quality of the ink on the bitmap is not the same as what you see on the screen. Especially if you set some advanced DrawingAttributes (like Transparency or RasterOperations) the resulting bitmap does resemble the image on the screen very well. What is the reason for this behavior? The short version is that the underlying Ink Renderer and the managed Winforms images use GDI in slightly different ways, so the rendering operation has to fall back to a lower quality mode.

Fortunately, there is a workaround: we can take both the Winforms image and our ink strokes and render them to a temporary device-independent-bitmap. Then we'll render the result of that to the actual target bitmap. In order to do this, however, we have to do some manged<->native code interop magic. All that magic is encapsulated in the DibGraphicsBuffer class (see highlighted line below). This helper class is included in the attached sample project. Here is the relevant code to apply the workaround using that helper class:

Graphics graphicsTarget = Graphics.FromImage(imageToSave);

Graphics graphicsSrc = inkPicture1.CreateGraphics();

DibGraphicsBuffer buffer = new DibGraphicsBuffer();

Graphics graphicsTemp = buffer.RequestBuffer(graphicsSrc,

                                               inkPicture1.Image.Width,

                                               inkPicture1.Image.Height);

graphicsTemp.DrawImage(inkPicture1.Image, new Point(0, 0));

inkPicture1.Renderer.Draw(graphicsTemp, inkPicture1.Ink.Strokes);

buffer.PaintBuffer(graphicsTarget, 0, 0);

Using the Winforms Ink API to target Vista (Microsoft.Ink v6.0)

In order to address the limitation of the XP API, the Vista version of Render.Draw() now has an additional overload that makes the above workaround obsolete (of course the workaround still works on Vista). You can now pass in a Bitmap object directly to the API, and your ink gets rendered onto the image in the highest quality with a one-liner:

inkPicture1.Renderer.Draw((Bitmap)imageToSave, inkPicture1.Ink.Strokes);

Attached Code projects:

I have attached two complete code projects to illustrate how to get the best results on both XP and Vista:

1) RenderInkOnBitmapXP - utilizes the DibGraphicsBuffer class to apply the workaround for Windows XP

2) RenderInkOnBitmapVista - uses the new overload to demonstrate the best practice on Windows Vista

TabletPC Development Gotchas Part4: Semantics of GetSystemMetrics(SM_TABLETPC)

swick — Sun, 04 Nov 2007 21:44:00 GMT

If your application needs to find out at runtime whether or not it runs on a Tablet PC, it is recommended to call GetSystemMetrics() and pass in the value 'SM_TABLETPC'.

But what does it actually mean for a computer to be a Tablet PC? Is that determined by the installed hardware components? Or the software components? All of the above? Some of the above?

Those are almost philisophical questions and I won't try to give a complete answer here. But instead I want to shed some light on the semantics of the actual implementation of GetSystemMetrics(SM_TABLETPC). So if you decide to use that API call in your app after reading this post, you know exactly what you are getting.

XP vs. Vista - Semantic Differences

On Windows XP, the return value GetSystemMetrics(SM_TABLETPC) is exclusively based on software considerations. It returns whether or not your computer is running Windows XP Tablet PC Edition. Back in the day, that made a lot of sense, because this edition was exclusively available to OEMs, and they could only install it on computers that satisfied strict hardware requirements. So checking for the OS type implied certain hardware capabilities. It caused a little bit of confusion for MSDN subscribers who installed the Tablet Edition on their desktops for development purposes. Moreover, the hardware requirements changed overtime (to allow new form-factors, such as UMPCs). Thus, on Windows XP, you cannot really rely on GSM(SM_TABLETPC) to tell you about hardware capabilities, it only tells you reliably what OS edition the user is running on.

On Windows Vista the return value of GetSystemMetrics(SM_TABLETPC) is determined based on several things. It returns true if all of the following conditions are true:

1) There is an integrated digitizer, either pen or touch, on the system.

2) The computer is running a Premium SKU of Windows Vista (Home Premium and above). Those SKUs include a license for the TabletPC optional components (which are also installed by default on those SKUs).

3) Tablet PC Input Service is running. Tablet PC Input Service is a new service for Windows Vista that controls Tablet PC input.

This obviously provides greater accuracy than the XP implementation. However, it also can potentially cause some confusion as all three conditions may change at runtime on a given piece of hardware. For example, you can stop the input service. Now when you launch an app that checks GSM(SM_TABLETPC) to enable/disable some features (like Microsoft Word does), you will notice that some of the inking tools are not available. So at times the same computer might be considered a Tablet PC, and sometimes it is not. Hmmm…

Detecting the presence of specific components:

In most cases checking the presence of the required components is more appropriate than using GSM(SM_TABLETPC). For example if your application relies on the Chinese handwriting recognizer, then calling GSM will be useless, because the recognizer could be present on non-Tablet systems - and it could be missing on a Tablet system. The right thing here is to inspect the Recognizers collection to see if it includes a Chinese handwriting recognizer.

There are two articels on MSDN that contain more information about this approach:

Tablet PC Platform Independence (by Dr. Neil Roodyn)

Microsoft Sudoku: Optimizing UMPC Applications for Touch and Ink (by Stephen Toub; scroll down to the 'Platform Detection' section)

TIP: If by any chance you want your Vista Premium desktop computer to be considered a Tablet PC without doing any hardware upgrades, all you need to do is install a virtual driver that reports itself as integrated stylus digitizer. This will cause GSM(SM_TABLETPC) to return true and as a result, light up all Tablet-specific features on your computer. Check the websites of popular external Tablet manufacturers for the availability of those drivers.

Previous post in this series: TabletPC Development Gotchas Part3: Coerce to Factoid (XP vs. Vista)

TabletPC Development Gotchas Part3: Coerce to Factoid (XP vs. Vista)

swick — Sat, 03 Nov 2007 15:50:00 GMT

If you are writing (or maintaining) and application that uses handwriting recognition, please be aware of a subtle behavior difference between XP and Vista when coercing the recognition result to a Factoid (or an input scope or a word list). This may easily be missed in your application testing, but it will potentially result in unhandled exceptions from your application when consumers are starting to use it.

Here is the scoop: normally, when you assign strokes to a RecognizerContext for recognition, it will always be able to return some result - even for random scribbling. However, if you set the 'Coerce' flag to bias the recognizer towards a certain format (e.g. Date, URL, etc.) or word list (e.g. medical terms) then there is a (small) chance that it will not be able to compute a result. On Windows XP, the RecognizerContext returns a null RecognitionResult in this case. On Windows Vista, however, an InvalidOperationException is thrown.

So if you have written an application agains the XP behavior, please update your code to also catch InvalidOperationException if you expect your customers to use your app on Windows Vista. If you are writing a new app today on Vista, please be sure to also check the RecognitionResult for null if you plan to support your product on Windows XP Tablet PCs.

To clarify the issue in code, consider the following example: the goal here is to recognize a given strokes collection as URL. Now if the strokes are random enough, the recognizer won't be able to coerce it to a valid URL (with my handwriting, for example, writing "ooooo" does not yield a result).

private string Recognize(Strokes strokesToRecognize)

{

RecognitionStatus status;

RecognizerContext recoContext = new RecognizerContext();

recoContext.Factoid = "(!IS_URL)";

recoContext.RecognitionFlags = RecognitionModes.Coerce;

recoContext.Strokes = strokesToRecognize;

try

{

RecognitionResult result = recoContext.Recognize(out status);

if (result != null)

{

return result.TopString;

}

else

{

// Windows XP code path

return "Null means NO RESULT on XP";

}

catch (InvalidOperationException)

{

// Windows Vista code path

return "InvalidOperationException means NO RESULT on Vista";

}

So the bottom line is: if you are writing an application that uses handwriting recognition, you should harden your code to handle both a 'null' result as well as an 'InvalidOperationException' from calling RecognizerContext.Recognize().

Next post in this series: TabletPC Development Gotchas Part4: Semantics of GetSystemMetrics(SM_TABLETPC)

Previous post in this series: TabletPC Development Gotchas Part2: Deploying InkPicture applications to Windows XP desktops

TabletPC Development Gotchas Part2: Deploying InkPicture applications to Windows XP desktops

swick — Fri, 02 Nov 2007 02:02:00 GMT

If you are developing applications that use an InkPicture control and you plan to deploy your product to customers running Windows XP desktop OS (i.e. not the Windows XP Tablet PC Edition), please read on:

The TabletPC controls (InkPicture and InkEdit) have been designed for being used on Windows XP TabletPC Edition. On other pre-Vista operating systems (Windows 2000, Windows 2003 and non-Tablet PC editions of Windows XP) you can instantiate these controls but can't use them for ink input. This is documented on MSDN.

So far so good. Now, the confusing part is that ink input in those controls is actually enabled on desktop computers for 'development and testing purposes'. The control determines whether you are 'developing and testing' by checking for the presence of the Tablet PC SDK on the machine at runtime. So, if you didn't read the fine print on MSDN and you do all your testing on development machines that happen to have the Tablet PC SDK installed, you will not notice this limitation until a consumer actually installs the application on a desktop computer without SDK.

Fortunately, this limitation has been removed in Windows Vista. The InkPicture control will collect ink on all Windows Vista SKUs and hardware configurations. Moreover, there also is a solution for deploying to Windows XP desktops. The following two facts are key here:

1) the downlevel deployment limitation does not apply to the InkOverlay component

2) attaching an InkOverlay to a PictureBox controls provides feature parity to InkPicture

So you can go ahead and build your own InkPicture control by extending the PictureBox control with an InkOverlay. Now it is deployable to non-Tablet PC, pre-Vista operating systems. Be sure to expose the InkOverlay methods, properties and events to the consumer of your control. My simple example below does this by making the entire InkOverlay a public property. You can do it more elegantly by exposing each relevant method, property and event individually on the control level - and then basically call through to the inner InkOverlay component.

class MyInkPicture : PictureBox

{

private InkOverlay _inkOverlay;

public MyInkPicture()

{

_inkOverlay = new InkOverlay(this);

_inkOverlay.Enabled = true;

}

public InkOverlay InkOverlay

{

get { return _inkOverlay; }

}

Next post in this series: TabletPC Development Gotchas Part3: Coerce to Factoid (XP vs. Vista)

Previous post in this series: TabletPC Development Gotchas Part1: Handling exceptions in Winforms Ink events

TabletPC Development Gotchas Part1: Handling exceptions in Winforms Ink events

swick — Thu, 01 Nov 2007 17:05:00 GMT

Over the past few years of developing apps using the TabletPC platform, I came across a couple of gotchas that are not exactly obvious to solve or workaround (at least they were not obvious to me). I figured I should blog about those and the respective solutions or workarounds to save others some head-scratching.

Here is the first one - it applies to the Microsoft.Ink assembly for developing inking apps using Winforms. Consider the following trivial app:

using System;

using System.Drawing;

using System.Windows.Forms;

using Microsoft.Ink;

class Program : Form

{

private InkOverlay _inkOverlay;

public Program()

{

_inkOverlay = new InkOverlay(this);

_inkOverlay.Enabled = true;

_inkOverlay.Stroke += new InkCollectorStrokeEventHandler(_inkOverlay_Stroke);

}

void _inkOverlay_Stroke(object sender, InkCollectorStrokeEventArgs e)

{

int foo = 2;

int bar = 5 / (2 - foo);

e.Stroke.DrawingAttributes.Color = Color.Red;

this.Invalidate();

}

static void Main(string[] args)

{

Application.Run(new Program());

}

The goal of the app is that each new stroke turns red once the stroke has been completed. However, when you run the app, this doesn't actually happen. The app works, but the ink remains in the default color (black). If you set a breakpoint on the line that sets the stroke's color to red, the breakpoint never gets hit. If you set a breakpoint at the top of the Stroke handler and step through the code, you will notice that the code execution returns to the app just before the color would be set.

Why is that? Well, in this simple example it's easy to see that there is a bug (division by zero). But why isn't the exception reported to us as an unhandled exception? The answer is not that obvious. It is due to the fact that the Microsoft.Ink assembly is really just a managed wrapper around a COM library (InkObj.dll). Take a look at the callstack when the event handler gets hit and you will see that your handler is being called from native code. So the unhandled, managed DivideByZero exception bubbles up the stack and eventually gets eaten by the native code that sits upstream from your event handler.

Now this behavior can result in some serious headscratching for the application developer. In most real-world cases, the bug is not as easy to spot as the one in my simplistic example. To avoid getting into these kind of issues, I recommend that you put your ink event handling code in try/catch blocks, so you can respond to exceptions thrown by your code, before they get eaten by the native/managed interop. Something along the lines of:

void _inkOverlay_Stroke(object sender, InkCollectorStrokeEventArgs e)

{

try

{

int foo = 2;

int bar = 5 / (2 - foo);

e.Stroke.DrawingAttributes.Color = Color.Red;

this.Invalidate();

}

catch (Exception exc)

{

MessageBox.Show(exc.ToString());

}

With this, the MessageBox will now tell you that there was a DivideByZero exception, and it will tell you where it happened.

Next post in this series: TabletPC Development Gotchas Part2: Deploying InkPicture applications to Windows XP desktops