Jaime Rodriguez On Windows Store apps, Windows Phone, HTML and XAML
I had to learn DeepZoom recently and along the way I put together some handy notes .. below are the notes organized in a near step-by-step explanation format. This is a very long post, but I hope it has useful insights for any one wanting to do deepzoom so I recommend you read it all. If you must skip, then the outline will help you. imo, part 3 and 5 are the good stuff.
Part 1 – The history and brief explanation on how DeepZoom works.
Part 2 – Constructing a DeepZoom image using Image Composer
Part 3 – Introduction to the DeepZoom object model – goes way beyond the docs I hope
Part 4 – Coding a deepZoom ‘host’ user control with Pan & Zoom
Part 4.1 – Adding a few extra features to our User Control
Part 5 – Lessons learned on the code, documenting the gotchas **must read even if you know Deep Zoom already
Part 6 – Give me the code, just a zip file w/ the goodies
Part 7 – Show me the outcome; what did we build?
A lot of people equate DeepZoom to SeaDragon – they assume SeaDragon was the code name and DeepZoom is the marketing name-. This assumption is not quite right (unless you equate your engine to your car model). Seadragon is an incubation project resulting from the acquisition of Seadragon Software; the Seadragon team is part of the Live organization and are working on several projects (like Photosynth). DeepZoom is an implementation that exposes some of the SeaDragon technology to Silverlight.
DeepZoom provides the ability to smoothly present and navigate large amounts of visual information (images) regardless of the size of the size of the data, and optimizing the bandwidth available to download it.
How does DeepZoom work? DeepZoom accomplishes its goal by partitioning an image (or a composition of images) into tiles. While tiling the image, the composer also creates a pyramid of lower resolution tiles for the original composition.
The image to the right shows you what a pyramid; the original image is lowest in the pyramid, notice how it is tiled into smaller images, also notice the pyramid creates lower resolution images (also tiled). A few of the docs I read said the tiles are 256x256, but from peeking through the files generated by the composer I am not convinced; I do know from reading through the internal stuff that there is some heavy math involved here, so I trust they tile for right size :).
All of this tiling is performed at design-time and gets accomplished using the DeepZoom composer.
At run-time a MultiScaleImage downloads a lower resolution tile of the image first and downloads the other images on demand (as you pan or zoom); DeepZoom make sure the transitions from lower to higher res images are smooth and seamless.
Given all this, how is Deepzoom different than say a ScaleTransform (for zoom) on a high resolution image?
With a ScaleTransform, usually you would download the whole high res image at once; this delays how quickly the end user gets to see the image when the page or application loads. Some times people apply a trick where you use different resolutions images, since you are not tiled you will likely end up downloading several big images (consuming more network bandwidth) or the download time will continue to be high if the initial downloaded image is not small enough, also the transitions from low to higher res are going to be more noticeable unless your write the transitions yourself.
DeepZoom and its tiling make it possible to see bits quicker and can optimize for bandwidth. In the worst case scenario where some one looked at every single one of the tiles at the highest resolution, DeepZoom would have an extra overhead of 33% compared to downloading the single highest resolution image at once, but this ‘worst case’ scenario is almost never hit, most of the time DeepZoom can save you from downloading too much.
Another feature in DeepZoom is its ability to create ‘collections’ from the composite image. This provides you the ability to compose a scene ( group of images ), optimize them for speed & download, but still maintain the ‘autonomy’ and identity of the image, you can programmatically manipulate (or position) these images from within the DeepZoom collection (more on collections in part 4).
Once you have a DeepZoom image, you will need an instance of the MultiScaleImage class in your silverlight application to load that image. Instantiating the MultiScaleImage class can be done from XAML
<MultiScaleImage x:Name="DeepZoom" Source="easter/info.bin" />
or from code:
MultiScaleImage DeepZoom = new MultiScaleImage () ;
DeepZoom.Source = new Uri ( “easter/info.bin”) ;
Before going through the DeepZoom API it makes sense to understand the terminology used:
Now, we navigate through the interesting properties and methods in MultiScaleImage
The interesting methods are:
In my opinion, surprisingly missing from the API were:
The goal here is to code a sample reusable control just to illustrate the points; along the way we will of course implement enough features for our Easter Egg Hunt. [Update: Sorry about belatedness, I started this on 3/22 but had a trip that prevented me from playing around, so I am late from easter]
<UserControl x:Class="DeepZoomSample.Page" xmlns="http://schemas.microsoft.com/client/2007" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" > <Grid x:Name="LayoutRoot" Background="White"> <MultiScaleImage x:Name="DeepZoom" Source="easter/info.bin" /> </Grid> </UserControl>
using System; using System.Windows; using System.Windows.Controls; using System.Windows.Documents; using System.Windows.Ink; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Animation; using System.Windows.Shapes; using System.Windows.Browser;
namespace DeepZoomSample { // this code came from Peter Blois, http://www.blois.us/blog // Code ported by Pete blois from Javascript version at http://adomas.org/javascript-mouse-wheel/ public class MouseWheelEventArgs : EventArgs { private double delta; private bool handled = false;
public MouseWheelEventArgs(double delta) { this.delta = delta; }
public double Delta { get { return this.delta; } }
// Use handled to prevent the default browser behavior! public bool Handled { get { return this.handled; } set { this.handled = value; } } }
public class MouseWheelHelper {
public event EventHandler<MouseWheelEventArgs> Moved; private static Worker worker; private bool isMouseOver = false;
public MouseWheelHelper(FrameworkElement element) {
if (MouseWheelHelper.worker == null) MouseWheelHelper.worker = new Worker();
MouseWheelHelper.worker.Moved += this.HandleMouseWheel;
element.MouseEnter += this.HandleMouseEnter; element.MouseLeave += this.HandleMouseLeave; element.MouseMove += this.HandleMouseMove; }
private void HandleMouseWheel(object sender, MouseWheelEventArgs args) { if (this.isMouseOver) this.Moved(this, args); }
private void HandleMouseEnter(object sender, EventArgs e) { this.isMouseOver = true; }
private void HandleMouseLeave(object sender, EventArgs e) { this.isMouseOver = false; }
private void HandleMouseMove(object sender, EventArgs e) { this.isMouseOver = true; }
private class Worker {
public event EventHandler<MouseWheelEventArgs> Moved;
public Worker() {
if (HtmlPage.IsEnabled) { HtmlPage.Window.AttachEvent("DOMMouseScroll", this.HandleMouseWheel); HtmlPage.Window.AttachEvent("onmousewheel", this.HandleMouseWheel); HtmlPage.Document.AttachEvent("onmousewheel", this.HandleMouseWheel); }
}
private void HandleMouseWheel(object sender, HtmlEventArgs args) { double delta = 0;
ScriptObject eventObj = args.EventObject;
if (eventObj.GetProperty("wheelDelta") != null) { delta = ((double)eventObj.GetProperty("wheelDelta")) / 120;
if (HtmlPage.Window.GetProperty("opera") != null) delta = -delta; } else if (eventObj.GetProperty("detail") != null) { delta = -((double)eventObj.GetProperty("detail")) / 3;
if (HtmlPage.BrowserInformation.UserAgent.IndexOf("Macintosh") != -1) delta = delta * 3; }
if (delta != 0 && this.Moved != null) { MouseWheelEventArgs wheelArgs = new MouseWheelEventArgs(delta); this.Moved(this, wheelArgs);
if (wheelArgs.Handled) args.PreventDefault(); } } } } }
protected double _defaultZoom = 1.3; public double DefaultZoomFactor { get { return _defaultZoom; } set { _defaultZoom = value; } }
private double _currentTotalZoom = 1.0;
public double CurrentTotalZoom { get { return _currentTotalZoom; } set { _currentTotalZoom = value; } }
private double _maxZoomIn = 5000; protected double MaxZoomIn { get { return _maxZoomIn; } set { _maxZoomIn = value; } } private double _maxZoomOut = 0.001;
protected double MaxZoomOut { get { return _maxZoomOut; } set { _maxZoomOut = value; } }
/// <summary> /// Performs a Zoom operation relative to where Image is at. /// Example, call DoZoom twice with a Zoom of 1.25 will lead to an image that is zoomed at /// 1.25 after first time and ( 1.25 * 1.25 for second time, which is a 1.56 /// </summary> /// <param name="relativeZoom"> new zoom level; this is a RELATIVE value not absolute.</param> /// <param name="elementPoint"></param> void DoZoom(double relativeZoom , Point elementPoint) { if ( _currentTotalZoom * relativeZoom < MaxZoomOut || _currentTotalZoom * relativeZoom > MaxZoomIn) return; Point p = DeepZoom.ElementToLogicalPoint(elementPoint); DeepZoom.ZoomAboutLogicalPoint(relativeZoom, p.X, p.Y); this.Zoom = relativeZoom; _currentTotalZoom *= relativeZoom; }
void OnMouseWheelMoved(object sender, MouseWheelEventArgs e) { // e.Delta > 0 == wheel moved away, zoom in if (e.Delta > 0) { DoZoom( DefaultZoomFactor, _lastMousePosition); } else { // Zoom out DoZoom( 1/ DefaultZoomFactor, _lastMousePosition); } }
protected bool _isDragging = false; protected Point _lastDragViewportOrigin; void DeepZoom_MouseLeftButtonDown(object sender, MouseButtonEventArgs e) { this._lastDragViewportOrigin = DeepZoom.ViewportOrigin; this._lastMousePosition = e.GetPosition(DeepZoom); this._isDragging = true; }
void DeepZoom_MouseMove(object sender, MouseEventArgs e) { if (_isDragging) { Point newViewport = _lastDragViewportOrigin; Point currentMousePosition = e.GetPosition(DeepZoom); newViewport.X += (_lastMousePosition.X - currentMousePosition.X) / this.DeepZoom.ActualWidth * this.DeepZoom.ViewportWidth; newViewport.Y += (_lastMousePosition.Y - currentMousePosition.Y) / this.DeepZoom.ActualWidth * this.DeepZoom.ViewportWidth; this.DeepZoom.ViewportOrigin = newViewport; _lastDragViewportOrigin = newViewport; } // NOTE: it is important this be after the isDragging check … // since this updates last position, which is used to compare for dragging. _lastMousePosition = e.GetPosition(DeepZoom); }
In the last sections I took it slow and walked through the code to explain what we were working on. Going forward below will pick up the pace a bit, and the original code will be tweaked to get into a host control with a bit more navigation and troubleshooting advise.
this.PanRight.Click += new RoutedEventHandler(PanRight_Click); this.PanLeft.Click += new RoutedEventHandler(PanLeft_Click); this.Home.Click += new RoutedEventHandler(Home_Click); this.PanBottom.Click += new RoutedEventHandler(PanBottom_Click); this.PanTop.Click += new RoutedEventHandler(PanTop_Click);
void Pan(PanDirection direction) { double percent = PanPercent; if ( UseViewportScaleOnPan ) percent *= this.DeepZoom.ViewportWidth; switch (direction) { case PanDirection.East: this.DeepZoom.ViewportOrigin = new Point(this.DeepZoom.ViewportOrigin.X - Math.Min(percent, this.DeepZoom.ViewportOrigin.X), this.DeepZoom.ViewportOrigin.Y); break; case PanDirection.West: this.DeepZoom.ViewportOrigin = new Point(this.DeepZoom.ViewportOrigin.X + Math.Min(percent, (1.0 - this.DeepZoom.ViewportOrigin.X)), this.DeepZoom.ViewportOrigin.Y); break; case PanDirection.South : this.DeepZoom.ViewportOrigin = new Point(this.DeepZoom.ViewportOrigin.X , this.DeepZoom.ViewportOrigin.Y + Math.Min( percent, 1.0 - this.DeepZoom.ViewportOrigin.Y)); break; case PanDirection.North : this.DeepZoom.ViewportOrigin = new Point(this.DeepZoom.ViewportOrigin.X, this.DeepZoom.ViewportOrigin.Y - Math.Min( percent, this.DeepZoom.ViewportOrigin.Y)); break; } }
/// <summary> /// Zooms in around an ELEMENT Coordinate.. /// I technically did not need this function to abstract and could have called DoZoom directly /// </summary> /// <param name="p"></param> void ZoomIn( Point p ) { if (_useRelatives) DoRelativeZoom(Zoom / DefaultZoomFactor, p, ZoomDirection.In); else DoZoom(DefaultZoomFactor, p ); }
this.LayoutRoot.KeyUp += new KeyEventHandler(DeepZoom_KeyUp);
void DeepZoom_KeyUp(object sender, KeyEventArgs e) { if (e.Handled == true) return; if ( ( RunningOnWindows && ((Keyboard.Modifiers & ModifierKeys.Control) == ModifierKeys.Control) && (e.Key == Key.Add || (e.Key == Key.Unknown && e.PlatformKeyCode == 0xBB))) || ( RunningOnMac && ((Keyboard.Modifiers & ModifierKeys.Control) == ModifierKeys.Control) && (e.Key == Key.Add || (e.Key == Key.Unknown && e.PlatformKeyCode == 0x18 ))) ) { ZoomIn( _lastMousePosition); } else if ( (RunningOnWindows && ((Keyboard.Modifiers & ModifierKeys.Control) == ModifierKeys.Control) && (e.Key == Key.Add || (e.Key == Key.Unknown && e.PlatformKeyCode == 0xBD)))|| ( RunningOnMac && ((Keyboard.Modifiers & ModifierKeys.Control) == ModifierKeys.Control) && (e.Key == Key.Add || (e.Key == Key.Unknown && e.PlatformKeyCode == 0x1B ))) ) { ZoomOut( _lastMousePosition ); } e.Handled = true; }
void PullData(ref MultiScaleImageDummyDataWrapper data, ref MultiScaleImage msi) { data.ViewportWidth = msi.ViewportWidth; data.ViewportOrigin = msi.ViewportOrigin; data.AspectRatio = msi.AspectRatio; data.UseSprings = msi.UseSprings; }
Overall I was quite impressed with DeepZoom. it is pretty cool stuff; I wish I had some cool pictures for a better application, but I did not try since I knew I could not top memorabilia.
My personal advise: do not databind to DeepZoom for now. Pull to it on Motion Finished.
No keyboard input goes into DeepZoom (since it is a FrameworkElement). In order to have keyboard input you must have a Control that has focus; since keyboard events bubble you can handle Keyboard input at a higher level (e.gl LayoutRoot, just check to see if it has not been handled previously).
On my real app –which can’t be shared as it was a customer’s app –. I ran into an issue when using Collections. My images were showing up in the wrong place. I reported it already and they are investigating –during the shower, where I do my best thinking- I came with the theory that is the resolution independence in WPF ( 96 to 72 DPI conversion). I have not confirmed.
I did not discuss collections in the post, so will try it here. Collections are cool because it gives you access to your Subimages so you can manipulate them. Move them around, scale them, animate position and Opacity. For now, beta1 has only one Collection; I think it would be cool to have multiple collections so you can aggregate. This can kind of be simulated via logic, but would be nice if it was in the control. If you simulate it, the advise I was given is do not simulate it by overlaying two MultiScaleImage controls one on top of the other, there are a few known issues with interactions on overlays (though to be honest I tried it and did not run into issues).
UseSprings= true is pretty cool, but pending how quick you want to do your panning/zooming, turning it off can make your app appear more responsive. I would not turn UseSprings off for a consumer facing app, but I would consider doing it for an internal app.. For example, I am doing a Heatmap with lots of data in it, for analytical purposes. Since it is drill through I am considering it.
When panning, make sure you handle MouseLeave on your control.
Handling mouse wheel is not available out of the box is trivial but Peter Blois has a great solution. Do not write the code to handle wheel. Peter’s code works great so far. Check his blog for updates too, he has a nice abstraction now to the same API.
If you skipped section 3, check it out. Understanding the object model is critical and takes 5 mins.
If you are writing a DeepZoom application, I recommend you use the old instantiation via silverlight.js … Click To Activate will eventually go away in IE, but in the mean time it is pretty annoying for an app that is so visual and so focused on mouse navigation.
If subscribing to MultiScaleImage.OpenImageSucceded make sure you do it from your constructor right after initializeComponent. I tried to do it of UserControl.Loaded and when doing a load on a page with image cached that is too late.
If possible try to ‘hold’ any operations until OpenImageSucceded has fired ( no pans, zooms before that). I saw weird results if I try to access properties on MultiScaleImage before this event; in particular if you access the SubImages collection before ImageOpenSucceded, then I would get an empty collection and when ImageOpenSucceeded was fired, the collection would not be overridden; so advise for collections is don’t touch SubImages before the OpenImageSucceeded fires.
Is at Skydrive
You can see it here; it is not visually impressive but I think it shows a bit of what you can do with DeepZoom and most important it is functional code you can quickly refactor and reuse. If I missed a common deepZoom task let me know. I added two extra “easter eggs” beyond the bunny and the eggs above in the walk through.
Thanks to Tim Aidlin who chose the colors and gave me cooler icons for the map; I butchered them a bit when I turned them into controls so don’t hold it against him, he is a gifted designer –you can see his real work on the MIX website and any thing else MIX08 branded.
Below are my scribbles during my flight back from MIX. I considered not publishing this because it feels 'too positive' (or too marketing-like).. if you don't like those, please skip (no harm). I am sharing to see if any one has feedback. I also want to share for people to understand my current thinking - it will explain future blog posts-. that said, don't unsubscribe to the RSS, I won't have many posts like this one. If you only have 3 mins, skip to the "Why I am excited section"..
-------------------------
As I fly back from MIX I am thinking of my daughter's upcoming second birthday ( she was born a few weeks after the very first MIX). Every day I am amazed at how fast she has developed in just two years. This week, I realized another amazing transformation that has happened in same timeframe: Microsoft's client technologies have evolved significantly. At MIX08, Microsoft communicated a very coherent vision for a comprehensive all-inclusive application platform. Microsoft backed up the vision with some pretty impressive customer applications.
The message after Day 1 keynote was:
You could be thinking this is "blue koolaid", but we backed it up with strong partner scenarios. Among the evidence:
That was the executive message.. Why does this excite me so much?
This new convergence on the client stack with XAML + .NET is incredibly powerful; this is one of those 1 + 1 = 3 catalysts.
Here are just a few of my reasons ( in random order):
I know there are challenges to overcome, but I see no major blockers and I do think time is on our side.
That sums up my excitement post MIX. In a few words, I think Silverlight's cross-platform opportunity is going to be the catalyst or accelerator for both RIA and Desktop applications written using .NET and XAML; I am excited about what will come in the next 12 to 18 months.
Check it out here... Source is here!
It is very little code so I did not do a write-up. The code does have comments. Ideas? or Improvements I considered but did not implement (yet )...
Credits: Every thing I know about carousel came from a Lee brimelow's flash carousel tutorial.
Bugs/feedback, leave comments or email.
Ashish did a great job with his demystifying URIs.. post.
I spent a bit of time discussing this topic with him last month due to a regression bug ( fixed before beta1 shipped).. During my 'discovery' I wrote a little test application that summarized it (and tested it all content resolving mechanisms ) ..
What this application does is include 3 images, packaged all possible ways (resource,content,embedded), and shows you how to load them from XAML.
You can see the application here, and the source code is here.
If you pay attention, you will see the sample also loads "external" assemblies NOT in the XAP and it shows you how to cross reference content using the /assembly;component syntax .. which Ashish forgot to mention..
I am going to leave that topic of loading external assemblies for a later post (in hopes ashish tackles it in detail like he has been doing); if you are itching to try it, the source is very small and very easy to follow...
Sorry that my test apps are so ugly. If you can recommend a book (or web reference) on how not to offend the world with ugly colors but at the same time do it with zero overhead to my coding, please send me a link.
If you have Silverlight 2, you can play with the demonstrator at this site
You can also see an internal (=not polished) recording of a walk through of the demonstrator from this silverlight streaming video.
If you want a script that helps you walk through the interactions, you can find one here.
The plan and the source:
Our goal from the beginning has been to release the source code at devcon; since that was yesterday, the source is available today from here.
Warning: the demonstrator grew quickly from a Silverlight 1.1 app with three scenarios to a SL 2.0 application with six or seven scenarios; so we are a bit behind on cleaning up the code. The plan is to do a bit more clean up over the next few days ( or say all of next week) and then put it out at Silverlight.net gallery when the code cleanup churn decreases. Please check the SL gallery or check this blog in a week or so for an updated source. [It takes that long because we do it one hour at a time on evenings or when I have free time at work (which is not much)].
Known issues: Apologies in advance to those outside the US or not running an en english locale. We are aware some of the ' banking concepts' might not apply but we hope some of the interactions are generic enough that are useful to show. We have also not localized the site. Please do report bugs around localization.
Credits:Infusion development did most of the coding. Joe Cleaver, Marley Gray, Joe Rubino and a few others in the MS financials team helped define the scenarios.