I am very excited to write my first blog post about Silverlight 2 today, the project I am currently working on. In early March 2008 we shipped a public Beta for Silverlight 2 that you can download (along with all the neccessary tools) from http://silverlight.net/.
Today I want to introduce you to using Ink in your Silverlight 2 application, using the new managed object model. This post has two versions of a basic inking sample attached - a C# version and a VB.NET version. Let's take a look at the XAML first. The relevant piece here is the InkPresenter element, which is our inking surface that will host the ink strokes collected by the user:
<InkPresenter x:Name="inkCtrl" Cursor="Stylus"
MouseLeftButtonDown="inkCtrl_MouseLeftButtonDown"
MouseMove="inkCtrl_MouseMove"
MouseLeftButtonUp="inkCtrl_MouseLeftButtonUp"/>
Now we need to add a little bit of code behind in the mouse event handlers in order to collect strokes from the user's input. Note that the 'GetStylusPoints()' calls actually obtain the high fidelity stylus input when using a stylus, so you get optimal, smooth ink on a Tablet PC - or with an external Tablet input device on your desktop.
public partial class Page : UserControl
{
public Page()
{
InitializeComponent();
}
private void inkCtrl_MouseLeftButtonDown(object sender,
MouseButtonEventArgs e)
{
// capture mouse and create a new stroke
inkCtrl.CaptureMouse();
newStroke = new Stroke();
inkCtrl.Strokes.Add(newStroke);
// set the desired drawing attributes here
newStroke.DrawingAttributes.Color = Colors.Blue;
newStroke.DrawingAttributes.OutlineColor = Colors.Yellow;
newStroke.DrawingAttributes.Width = 6d;
newStroke.DrawingAttributes.Height = 6d;
// add the stylus points
newStroke.StylusPoints.AddStylusPoints(e.GetStylusPoints(inkCtrl));
}
private void inkCtrl_MouseMove(object sender,
MouseEventArgs e)
{
if (newStroke != null)
{
// add the stylus points
newStroke.StylusPoints.AddStylusPoints(e.GetStylusPoints(inkCtrl));
}
}
private void inkCtrl_MouseLeftButtonUp(object sender,
MouseButtonEventArgs e)
{
if (newStroke != null)
{
// add the stylus points
newStroke.StylusPoints.AddStylusPoints(e.GetStylusPoints(inkCtrl));
}
// release mouse capture and we are done
inkCtrl.ReleaseMouseCapture();
newStroke = null;
}
private Stroke newStroke = null;
}
And here is the result at runtime, after collecting a couple of strokeson my Tablet PC:
For a more advanced sample app that uses ink to annotate pictures and has support for erasing ink, please visit the Silverlight 2 Beta1 Gallery and look for the 'Image Snipper' sample. You can also run it directly from the below IFrame (requires the Silverlight 2 Beta1 runtime to be installed on your computer):
In the spirit of the holiday season, I have created a little eCard using the Silverlight v1.0 SDK. Use the buttons to add ice skaters and falling snow to the scene :-)
Have a wonderful Holiday and a happy New Year 2008!
P.S. I am using the Silverlight Streaming Service to host the eCard. If you like it, you can embed it on your site by using the following HTML:
<IFRAME src="http://silverlight.services.live.com/invoke/34348/eCardIceSkater/iframe.html" frameBorder=0 width=545 scrolling=no height=360 mce_src="http://silverlight.services.live.com/invoke/34348/eCardIceSkater/iframe.html"></IFRAME>
The source code for the eCard is attached to this post.
WPF's InkCanvas element provides a lot of built-in functionality for several common, ink-related tasks like stylus gesture recognition, point and stroke erase, as well as the selection, resizing and moving of ink strokes. The key to those features is the 'EditingMode' property - which is nicely demonstrated in the InkCanvas EditingModes SDK sample (btw, it also demonstrates an implementation of an Undo/Redo stack for ink operations).
One little known, yet very cool feature is not demonstrated in this sample, though: The selection, resizing and moving functioality (i.e. EditingMode="Select") can not only be applied to ink strokes, but also to child elements of the InkCanvas. This can be very handy if you want to build, for example, a note-taking application that can also host text, pictures and other content besides the handwritten ink. You can then use the 'Select" mode to let the user re-arrange and resize all their content.
The "gotcha" I wanted to point in this blog post is a limitation in this feature: Child elements in an InkCanvas may be positioned in a variety of different ways: for example you can position them by setting a "Margin" or by setting any combination of "Canvas.Left/Right/Top/Bottom" attached properties. InkCanvas's selection/moving/resizing feature, however, only works on elements that have absolute positioning set via "Canvas.Left/Top".
Let's look at the following example markup:
<Window x:Class="InkCanvasElementSelection.Window1"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="InkCanvasElementSelection"
ResizeMode="NoResize"
Height="450" Width="450">
<Window.Resources>
<Style TargetType="Rectangle">
<Setter Property="Width" Value="100"/>
<Setter Property="Height" Value="100"/>
</Style>
</Window.Resources>
<Grid>
<InkCanvas Name="inkCanvas" EditingMode="Select"
Background="LightGoldenrodYellow"
SelectionChanging="OnSelectionChanging">
<!-- InkCanvas' Child Elements -->
<Rectangle InkCanvas.Left="20" InkCanvas.Top="20" Fill="Blue"/>
<Rectangle InkCanvas.Right="80" InkCanvas.Bottom="30" Fill="Green"/>
<Rectangle Margin="50,150,20,20" Fill="Orange"/>
<Rectangle Fill="Brown">
<Rectangle.RenderTransform>
<TranslateTransform X="280" Y="100"/>
</Rectangle.RenderTransform>
</Rectangle>
</InkCanvas>
</Grid>
</Window>
Note that the markup subscribes to the 'SelectionChanging' event on the InkCanvas. Let's leave the implementation of that event handler empty for now to demonstrate the limitation. Also note that the four rectangles use different ways to position themselves on the InkCanvas. Only the blue rectangle uses the "Canvas.Left/Top" attached properties for positioning.
Now you can use the lasso tool to select one or more elements, and then try to move and resize them. You will notice that it only works correctly for the blue rectangle (because that one has been positioned via "Canvas.Left/Top"). For the other rectangles you won't get the desired move or resize results.
Now what can I do if there are child elements that have not been positioned via "Canvas.Left/Top"? Here is a workaround to make this feature work even for those elements: In the 'SelectionChanging' event - which fires whenever you change what content is selected - you can walk the list of child elements and ensure they are positioned via "Canvas.Left/Top". Here is a piece you can add to the above sample to demonstrate this approach:
void OnSelectionChanging(object sender, InkCanvasSelectionChangingEventArgs e)
{
ReadOnlyCollection<UIElement> elements = e.GetSelectedElements();
foreach (UIElement element in elements)
{
// obtain actual location of element relative to InkCanvas
Point locationInInkCanvas = element.TranslatePoint(new Point(0, 0),
inkCanvas);
// set the location via Left/Top properties
element.SetValue(InkCanvas.LeftProperty, locationInInkCanvas.X);
element.SetValue(InkCanvas.TopProperty, locationInInkCanvas.Y);
// un-set right/bottom properties
element.SetValue(InkCanvas.RightProperty, double.NaN);
element.SetValue(InkCanvas.BottomProperty, double.NaN);
// re-translate any render transform
Matrix matRender = element.RenderTransform.Value;
matRender.Translate(-matRender.OffsetX, -matRender.OffsetY);
element.RenderTransform = new MatrixTransform(matRender);
// set margins to zero
if (element is FrameworkElement)
{
((FrameworkElement)element).Margin = new Thickness(0d);
}
}
}
Now if re-run the sample you will find all rectangles from the original markup can be selected, moved and resized as expected:
Full source code of this example is included in the attached Visual Studio project.
One of my favorite tools on the Tablet PC is the 'Snipping Tool'. I have been using it frequently since the early XP days (back then we shipped it seperately as a Power Toy). Now in Vista, it's part of the main OS and I find myself using it quite often event on my desktop PC.
When I was using the tool recently, I got this idea for a new Silverlight sample app: wouldn't it be cool to snip pieces from various images and put them together as a collage in Silverlight - and then annotate the result with text, drawings and/or handwriting?
Silverlight's imaging and ink APIs make this fairly easy. Here are my results of this effort using Silverlight v1.0 - the sample demonstrates these features:
- Select an image from an animated collection of pictures
- Snip a piece of the selected image (free-form, rectangle or entire picture) and arrange it on the canvas
- Use 'Image Tools' to control zoom, rotation and opacity of individual snippets
- Use 'Text Tools' to add/change text annotations
- Use 'Ink Tools' to add/change drawings and handwritten annotations
- Full screen and embedded mode
Couple of notes and caveats:
- The pictures are currently hardcoded (appropriate for the holiday season :-) ), but you can easily change that in the attached source code to use your own pictures or to pull images from the web.
- When you launch the app from this blog, it opens up in full screen mode (since this is the best mode to view and use it). You can press the 'Esc' key or use the 'Miscellaneous' menu to go back to embedded mode.
- In full screen mode Silverlight disables keyboard input (for security reasons). So if you want to enter some text annotations, please switch to embedded mode.
- The 'Create Bitmap' button doesn't do anything in this version. The idea is to invoke some server-side code here to generate a bitmap of your collage. If there is interest in the community, this might be a good topic for a seperate blog post. But then again: if you are on Windows Vista, you can just use the Snipping Tool in the OS, to grab your collage and save it as bitmap :-)
The sample's complete source code is attached to this post. A live version, hosted via Silverlight Streaming, is included below for you try out. Click the button to launch the app:
I have received several question from folks about my earlier post on Ink Reflections in WPF. People were curious how to accomplish the same or a similar effect in Silverlight, in the absence of the VisualBrush object.
Clearly, WPF's VisualBrush is the main object that enables scenarios like this and it definitely makes their implementation straightforward. However, in Silverlight you can also accomplish effects like this with little bit of extra coding.
First of all, Silverlight has an ImageBrush and a VideoBrush, so as long as the visul you want to mirror is an image or a video, you can apply the same technique as used in my WPF sample and you are done.
If you want to mirror a different type of visual, you will have to manage the mirror yourself, meaning you have to create a secondary, mirrored visual and update it whenever the original changes.
I have created a sample that demonstrates both techniques together. The yellow background with silver border is an image that gets mirrored using an ImageBrush. The ink you draw (and erase) gets mirrored programmatically using some JavaScript code-behind.
The sample's source code is attached. A live version, hosted via Silverlight Streaming, is included below for you try out.
To follow-up on my previous blog post and to complete the story about rendering ink onto pictures and saving the results as a bitmap file, I want to show how this is done in WPF.
In WPF all rendering uses the pipeline - pictures, videos, ink, text, all gets rendered via WPF's 'Media Integration Layer'. As a result, it's very easy and straightforward to combine rendering these different types, both on screen and on a bitmap.
I have attached a WPF sample app that demonstrates how to collect ink on top of an image and then save the combined visual to a bitmap file. In fact, I have used this sample to create the sample picture that is included in my previous post about doing this in Winforms. Here is a screenshot of the sample app (btw, the ink color picker is borrowed from the InkColorPicker SDK sample).
The key class to accomplish our job in WPF is the RenderTargetBitmap class. Here is the relevant code snippet from the sample app:
// render InkCanvas' visual tree to the RenderTargetBitmap
RenderTargetBitmap targetBitmap =
new RenderTargetBitmap((int)inkCanvas1.ActualWidth,
(int)inkCanvas1.ActualHeight,
96d, 96d,
PixelFormats.Default);
targetBitmap.Render(inkCanvas1);
// add the RenderTargetBitmap to a Bitmapencoder
BmpBitmapEncoder encoder = new BmpBitmapEncoder();
encoder.Frames.Add(BitmapFrame.Create(targetBitmap));
// save file to disk
FileStream fs = File.Open(fileName, FileMode.OpenOrCreate);
encoder.Save(fs);
Unlike the Winforms API, there is no behavioral difference between Windows Vista and Windows XP.
Annotating pictures with handwriting or drawings is a fun and useful scenario for digital ink. In many cases the application user wants to create a new image file with the ink annotation "burnt" into the picture, so they can use the result of their work in other contexts and with other applications (e.g. attach the annotated image to an e-mail or upload it to their personal web site or blog, like the image below).
So rendering ink on top of an image is a common scenario for TabletPC/Ink developers - and it should be real easy, right? Well, there is an important 'gotcha' you need to know about if you want to do this from Winforms. But let's start with native code first.
Using the COM Ink API in native code
If you are using the ActiveX version of InkPicture (or the COM InkOverlay component) then rendering its ink to a GDI bitmap is easy. Just get the hDC (handle to device context) of the bitmap and use the InkRenderer.Draw() method:
pInkRenderer->Draw(hDCtargetBitmap, pStrokesToRender);
Using the Winforms Ink API to target XP (Microsoft.Ink v1.7)
In managed Winforms code, the object that corresponds to the hDC is the System.Drawing.Graphics object. So the intuitive approach would be to get the Graphics object of the Winforms target bitmap and then use the Microsoft.Ink.Render.Draw() method to render the desired ink strokes on top of it. Something along the lines of this:
Graphics graphicsTarget = Graphics.FromImage(imageToSave);
inkPicture1.Renderer.Draw(graphicsTarget,
inkPicture1.Ink.Strokes);
imageToSave.Save(“InkAndImage.bmp”);
Unfortunately, the result of this operation does not look great: the quality of the ink on the bitmap is not the same as what you see on the screen. Especially if you set some advanced DrawingAttributes (like Transparency or RasterOperations) the resulting bitmap does resemble the image on the screen very well. What is the reason for this behavior? The short version is that the underlying Ink Renderer and the managed Winforms images use GDI in slightly different ways, so the rendering operation has to fall back to a lower quality mode.
Fortunately, there is a workaround: we can take both the Winforms image and our ink strokes and render them to a temporary device-independent-bitmap. Then we'll render the result of that to the actual target bitmap. In order to do this, however, we have to do some manged<->native code interop magic. All that magic is encapsulated in the DibGraphicsBuffer class (see highlighted line below). This helper class is included in the attached sample project. Here is the relevant code to apply the workaround using that helper class:
Graphics graphicsTarget = Graphics.FromImage(imageToSave);
Graphics graphicsSrc = inkPicture1.CreateGraphics();
DibGraphicsBuffer buffer = new DibGraphicsBuffer();
Graphics graphicsTemp = buffer.RequestBuffer(graphicsSrc,
inkPicture1.Image.Width,
inkPicture1.Image.Height);
graphicsTemp.DrawImage(inkPicture1.Image, new Point(0, 0));
inkPicture1.Renderer.Draw(graphicsTemp, inkPicture1.Ink.Strokes);
buffer.PaintBuffer(graphicsTarget, 0, 0);
Using the Winforms Ink API to target Vista (Microsoft.Ink v6.0)
In order to address the limitation of the XP API, the Vista version of Render.Draw() now has an additional overload that makes the above workaround obsolete (of course the workaround still works on Vista). You can now pass in a Bitmap object directly to the API, and your ink gets rendered onto the image in the highest quality with a one-liner:
inkPicture1.Renderer.Draw((Bitmap)imageToSave, inkPicture1.Ink.Strokes);
Attached Code projects:
I have attached two complete code projects to illustrate how to get the best results on both XP and Vista:
1) RenderInkOnBitmapXP - utilizes the DibGraphicsBuffer class to apply the workaround for Windows XP
2) RenderInkOnBitmapVista - uses the new overload to demonstrate the best practice on Windows Vista
Silverlight makes delivering video content in the browser very easy. Silverlight also has ink support built-in. So why not combine these two features to enable some great new scenarios?
I have put together a little sample using Silverlight v1.0 to record and play back ink annotations, synchronized with the video playback. Below is the key piece of XAML markup for this application. We are using an InkPresenter that hosts the MediaElement with the video and takes care of the ink renderering. Also note the empty storyboard here as this is the magic sauce for the synchronized ink playback.
<InkPresenter Name='inkPresenter' Canvas.Left='10' Canvas.Top='10'
MouseLeftButtonDown='onInkPresenterDown'
MouseMove='onInkPresenterMove'
MouseLeftButtonUp='onInkPresenterUp'>
<InkPresenter.Resources>
<Storyboard Duration="0:0:0" Completed="onStrokePlaybackTimerTick"
x:Name="strokePlaybackTimer" />
</InkPresenter.Resources>
<MediaElement Name='mediaElement' Source='bear.wmv'
Width='720' Height='480'
AutoPlay='False' MediaEnded='onMediaEnded'/>
</InkPresenter>
The relevant pieces in the JavaScript code behind the XAML (in file events.js), are the following functions:
-
ConvertInkToString(strokes) - a function to serialize the point data for the ink strokes into a string for persistence purposes
-
ConvertStringToInk(serializedStylusPoints) - the reverse function that returns a StrokeCollection for a serialized ink string
-
onStrokePlaybackTimerTick() - during playback this function is called on every frame; when it is time to play back a stroke, we do so point by point
The full source code is attached to this post. I have also uploaded the sample to the Silverlight Streaming service, so I can pull it directly into my blog here. To check it out, just click the button below to load the sample. Once the video is loaded, you can click the 'Playback' button to replay my pre-recorded ink annotation - or you can click 'Start Recording' to record your own annotations, and then play them back afterwards.
Thanks to everyone who attended my sessions at the Oredev conference in Malmo, Sweden. Thank you also for all the great questions and feedback after the sessions - much appreciated! I have attached my slide decks to this post for those who are interested.
Some of the demos that I coded or showed during the sessions are already on this blog. I am working on getting the missing ones up here as well. Stay tuned ...
Also the video recordings from all Oredev sessions should become available soon on http://www.oredev.org.
Are you using Windows Journal to jot down your thoughts on your Tablet PC? Did you ever wish to publish your handwritten notes to your blog?
Here is one way to do this: The TabletPC SDK provides a Journal Reader API that provides access to the content of the .jnt file. Now you can read all content and convert it into a 'blog-friendly' format. Since Silverlight offers great ways to present content in an attractive, interactive way, I decided to write a Journal -> Silverlight converter. The result of this test project is now available as SDK sample on microsoft.com/downloads.
The converter is easy enough to use. Just drop down to a command box and run "JntToSilverlight.exe MyNotes.jnt". This will produce a 'MyNotes' folder that contains the resulting Silverlight application. You can test the results now by double-clicking the 'default.htm' file to load the app into your browser.
OK, now how do I get all this onto my blog? Using Silverlight Streaming, it's very easy: just zip it up, upload it to the service and then stick an IFrame into your blog's HTML. See my previous post on this subject.
To summarize the steps:
1) Create a note in Windows Journal
2) Convert note to Silverlight app
3) Upload to Silverlight Streaming service
4) Insert IFrame into blog
This is still a lot of steps and a bit tedious, but it's a proof of concept. Steps 2-4 all have an API they so are automatable and one could create a tool that does all that in one step.
Here is one example of a converted note on my blog (the original .jnt is attached to this post, if you want to compare). Besides flipping through the pages, you can add more ink, move selected content around and erase ink. Also don't miss the full-screen mode and the mouse-wheel zoom feature!
Want to create some fancy looking handwritten text or drawing? Tweaking the standard DrawingAttributes on an ink stroke won't get you very far - and creating a custom ink renderer is a lot of work. Why not just apply some of the WPF BitmapEffects to your ink. All you need to do is add a few lines of markup to an <InkPresenter/> and you are done - see markup snippet below (the complete XAML file is attached to his post):
<InkPresenter.BitmapEffect>
<BitmapEffectGroup>
<DropShadowBitmapEffect
Color="Black" Direction="20" Opacity="0.8" Noise="0.2" Softness="0.2"
ShadowDepth="{Binding ElementName=shadowSlider,Path=Value}"/>
<BlurBitmapEffect
KernelType='Gaussian'
Radius="{Binding ElementName=blurSlider,Path=Value}"/>
<OuterGlowBitmapEffect