Mulling over the list of new features I saw the blur factor that was put into the MSI. The point of this property is to force the MSI to render at a lower level of detail, causing a rendering that looks blurry. This doesn't work great, and it's not a real blur (the edges stay sharp for one), and an upsampling of a lower level of detail doesn't have the same feel as a true gaussian blur, but it's good enough for some effects.
Then Ben Vanik on our team said to me last week "Hey, couldn't we make maps with labels render nicer by tweaking the blur factor?". Of course we can! The problem with maps with labels is that the text never quite looks right. In fact, it almost always looks wrong: unless you render exactly at a power of 2, you get a mix of the upper and lower level of detail, which results in unreadable, ugly labels:
So here's the idea: adjust the blur factor every frame so that you are always rendering an exact level, no matter what size the image is you are displaying. In some scenarios that actually means "sharpening" the image, in others it means "blurring" the image. You always do it as to cancel out the effect of spatial blending between levels. The result is beautiful and can be seen here (click to open new window):
There are two ways this could be done, and one produces less than ideal results. One could adjust the blur factor every frame to render exact levels, which would result in "popping" of tiles when level of detail bopundaries are crossed.
So I decided to go with the second option, which is a "cheap trick": as soon as an animation starts that changes the viewport width, an animation of the same length is start to adjust the blur smoothly so that, by the time the animation settles, the blur factor is set to render exact levels! Nice.
Some other small notes: I decided rendering always the higher level of detail was ugly, so this actually renders the lower level of detail. It's barely noticeable, and it saves 4x the bandwidth.
Source code is here.
The Seadragon team in Live Labs has released a service that lets you easily turn your giant pictures into nice zoomable UIs. In order to try this out I went to the Smithsonian's flickr stream and grabbed a picture to see how it works. Go over to www.seadragon.com to check it out for yourself!
Here is the result:
There are some really neat aspects to this. For one, you get all of the Seadragon goodies that come with the file format:
- Images always load up really fast, no matter how big the image is
- You can zoom and pan smoothly to every detail of the image
There are other parts of this that are really noteworthy. For one, this is an Azure service, so we can see that Azure is well capable of doing image conversion at a massive scale, as well as storage and retrieval. Very nice work guys!
Secondly, and more subtly - this service uses parts of Seadragon AJAX for its rendering. It's smart enought to determine whether to use AJAX (if Silverlight is not installed) or Silverlight (if it is installed and if you use a supported browser/OS combo). For that aspect alone it's a great example of how to bridge AJAX and Silverlight to take advantage of the advantages of both systems (perf will always be a lot nicer in Silverlight than in AJAX, but at least you can be assured that it will work even if Silverlight is not installed). Good example for showing that you can use Seadragon even if you can't count on Silverlight always being installed.
Last week we talked about the collections API and how tomove things around on screen using collections. Today, I want to take a lookunder the hood to see what collections are how they work, and what the benefitsof collections are.
So why were collections created in the first place? Inessence collections are a storage and network optimization.
Storage/NetworkOptimization
Go over to http://memo.hardrock.comone more time (yes, you’ve seen it, but bear with me). You will notice that theguys from Vertigo have upped the size of the collection: you can now view 600images in the collection of Rock memorabilia.
Pay close attention: the collection of all 600 images openspretty much immediately after the application is launched. Some images areblurry for a while depending on your network connection until more detail isloaded. It wouldn’t be possible to show content that quickly if a networkroundtrip was required to load each individual image. Even if each image wasonly a few hundred bytes in size, it would still take considerable amounts oftime to round trip and open each file (remember: modern browsers only allow upto 6 connections at the same time anyway – we would only be able to download 6images at a time, even if each image is loaded in 1/6th of a second,it would still take 100 seconds to load all of the images).
How does Seadragon/Deep Zoom solve this problem? We storeimages in texture atlases, in essence we cram image thumbnails into tiles; andin the same manner as single images, we store the texture atlases as a tiledimage pyramid.
The collection stores images up to level 8 (by default ifyou use deep zoom composer, this is customizable if you use the deep zoom toolsAPI), so images up to 256x256 are stored in a collections. For most largecollections of large images, the size overhead of storing this collection isnegligible.
Let’s take a look at how a sample collection may look:
For this set of 9 images, we are showing, say, level 8, 7,and 6. At the bottom of the pyramid, each image is stored in its own tile. Butif you look just one level up, at level 7 (a quarter of the size of level 8 bythe way), four images share a tile, and at level 6 all images share one tile.
It takes a collection of more than 65536 images to have morethan one tile at the lowest level of detail, where each image only takes up onepixel (thus making space for 256x256=65536 images).
Deep Zoom always loads the tip of the pyramid first, so formost collections <65536 images that means loading a pretty small tile to atleast show *something on screen*. The time it takes to blend in that top levelis usually enough time to load a couple more levels and started blending those.By loading just a handful of tiles in the hardrock demo, users can immediatelysee something on screen, and pretty quickly detail gets blended in as thenetwork downloads more detail in more tiles.
The observant reader will notice a limitation ofcollections: the assumption is that images displayed on screen are at leastsomewhat related. The collection optimization stops working if you makecollections that break that assumption. For example, if you have a collectionof several million images, and you decide to display 1000 of those images thatare all stored on separate tiles even at lower level of details, then deep zoomwill load a lot of tiles that share image data with images that aren’t evenshown on screen.
In our experiments, however, this hasn’t caused problems forcollections around 10000 items in size at all. It really needs to be a prettydegenerate case to cause noticeable “badness”.
Dynamic Collections
When we thought of the collection file format and thestorage thereof, we had another scenario in mind that isn’t properly exposed inSilverlight yet: we wanted to make it possible to quickly add and remove itemsto and from collections. Imagine you have a collection of search results; wewanted to make it possible to create or modify a collection of search resultson the fly without too much performance penalty.
And in fact, adding or removing collection items is O(1),i.e. it’s basically free algorithmically. Of course almost all of the time willbe spent in IO operations getting the tiles written to disk if you use dynamiccollections.
Collections in Deep Zoom meet the following goals:
(1) Images are crammed into tiles in a “prettyoptimal” way (you can see some space between images in the example above, but thatusually compresses away pretty well)
(2) Metadata to access image content in tiles isminimal (it’s the same metadata for each level in the collection, and the onlyinfo theoretically required is the index of the item and the aspect ratio ofthe image)
(3) Minimal space waste: At each level of detail, atmost one tile is less than completely full.
When you look at the requirements above, the strange MortonLayout we use (as documented on MSDN) becomes clear: by using a fractal layout,we can ensure that each thumbnail for an individual image is relativelyspeaking, at the same offset from the top left corner as in any other level.Because of that, we only need to store the metadata for the location of athumbnail once, not again and again for each level. And since we use the MortonLayout, we only need to store a small number, or ordinal. In fact, ifcollections weren’t dynamic, we could use just the position in the list as thatordinal.
So when you need to add an item to a collection, you add itto the end using the Morton number to figure out the image thumbnail’slocation.
If you look at the above image of a collection:
Then you add a few images, it will look like this:
You can easily see how this scales to infinity whilemaintaining all the properties we need from dynamic collections.
I've been getting this question a lot, even from members of my team. Why are the coordinates so weird in deep zoom? They are just totally confusing for laying out images! Why do you have this odd ViewportOrigin and ViewportWidth and why are the numbers I put in there so different from what I think they should be?
These are questions I get every day and I am getting tired of explaining how it works and why it is the way it is. However, I thought it would be fun to start off with the history of why things are the way they are. It's not because we're crazy (maybe a little), but most of it has to do with the way Deep Zoom was created and integrated into Silverlight.
Why Viewports?
So let's start with a little bit of background. Deep Zoom was started in livelabs, Rado and I are not really part of the Silverlight team. Our first modest goal was to get single deep images that render really fast into Silverlight. We thought that was quite a lofty goal and we also thought we'd likely spend most of our time getting the performance up to par given that Silverlight has a software rasterizer.
The scenario we had in mind was for folks to create very deep images, possibly images with certain parts of them being particularly deep. An example of this may be the kinds of images you can easily create with Deep Zoom Composer. Now if you think about single images that are really deep, well then Viewports make a ton of sense. There isn't really a scenario where you zoom out - most of the time you will want to zoom into areas of interest in your large image. So if you have the following image (dotted lines are there for illustration), The viewportOrigin and ViewportWidth coordinates are exactly the coordinates of the image inscribed inside the larger image, in logical coordinates in the larger (original) image's coordinates. Zooming to the small images area requires a straightforward setting for ViewportOrigin and ViewportWidth to the coordinates of the small image (in logical coordinates between 0 and 1).
Then, in the above example, if I want to display the Gray area on screen completely, I set ViewportOrigin = new Point(0,0), and ViewportWidth = 1. Makes perfect send.
If I want to zoom to the green rect, I set ViewportOrigin = new Point(0.5, 0.5), and ViewportWidht = 0.5. Wow, that's easy.
But now collections!
Last year around Christmas Rado said he thinks we'd be able to get collections to work. No way I thought, that sounds impossible. So when we got to designing the API, well we had two choices: either we make SubImages work exactly the same way as MultiScaleImages, or we make SubImages more condusive to laying out on screen like images. The advantage of the latter being the layout being more intuitive for viewing large collections of images, however the disadvantage being that zooming into a SubImage would be completely different and less intuitive from MultiScaleImage single image scenarios. We were stuck between a rock and a hard place. Either we make the API consistent, or we make it intuitive for a specific scenario and less intuitive for another. Since we didn't know how people would use Deep Zoom (turns out people are much more interested in collections - i.e. laying out lots of medium sized images on screen vs really deep images) we frequently hear that the way SubImages work is counter intuitive. And it was designed to be intuitive by being consistent, but I'm not sure that consistency was the right choice in this case! I'd love to hear what people think.
So if you want to lay out a SubImage on screen, you still have to deal with Viewports and ViewportWitdths, something that's designed to work well for zooming in, but not for laying images out (which is really "zooming out").
The best way to think about this is that the top left corner of the image you are displaying is always (0, 0), and the right edge of the image is always at Width = 1. So if you want to zoom out, or make the image smaller, you have to make the ViewportWidth bigger, and you have to move the ViewportOrigin to the top left, i.e. make it negative.
Example:
So you can see, it all makes sense somewhere. By the way - I added some helper functions to the template that ships with Deep Zoom Composer that translates coordinate spaces between the Viewport space and a regular image space if you can't think in Viewport space for laying things out (I can't).
It's been a while since I've posted something here, I realize that. But that's because I only want to blog about cool stuff, and stuff that nobody else has done... Right, so you have to only come here to find the new and cool stuff!
Here's a problem: I want to pan smoothly from one area in a map to another. One way to do that is simply update Viewport Origin and ViewportWidth and let Deep Zoom handle the transition animation. Unfortunately that will lead to animations that look like you are traveling at the speed of light across the globe if you have large distances to cover.
Ideally, you'd want to zoom out a tad bit, nudge over a bit, and zoom back in. Well, thanks to Dan Cory and James Darpinian, we have such a solution. Based on this paper, we implemented this.
You can click around to jump to the various places I've lived in Seattle and Europe.
You can find the source here.
Here's an idea. What if you change the contents of the collection pages for each level of detail? What if you want the collection of images to look different when you are zoomed out vs when you are zoomed in? There are many scenarios I can imagine this being useful for.
Think you want to show a large collection of posters (as I do below). Each single poster is designned to "pop", to stand out in a crowd. Now if you want to display hundreds of these together, they all want to "pop" and stand out. The result? A vegetable, meatball, fish, egg noodle, broccoli soup. It doesn't go together well if every (tiny) item on screen is meantt to stand out from every other imagee on screen.
Or, as a designer friend of mine once said, emphasize everytthing and you end up emphasizing nothing. Well that may not be bad (you don't want to emphasize everything) but the resulting image doesn't look pleasing either.
So I decided to give it a whirl. I created a collection and tweaked the jpg files at each level of detail progressively with Photoshop to make them more blurry, black and white, and edge de-emphasized.
So you zoom out - and you see a black and white pool of things, but as you zoom in the posters become more obvious as the posters they are supposed to be (BTW - I scraped these from IMDB, thanks IMDB!). It's a very subtle effect, but it can be useful to give your web experience that subtle fine touch.
All I had to do was change the content of the JPGs in the collection at every level of detail. Deep Zoom handles all of the blending between levels to make the transitions look really smooth. I think for these types of effect, very subtle, small content changes between levels of detail work best. In my experience completely changing the contents between levels doesn't usually produce a good effect. Or, as Blaise always says - the movements have to be continuous. You can't just change things up dramatically as you move around, but you may very well make continuous changes. My favorite usability engineer (hey Sharma) once said: the only important question to ask is: does the user understand what just happened? If you stick to continuous changes from frame to frame, no doubt they will.
How was this done? Well, if you look at the collection structure:
collection.xml
collection_files
+ 0
0_0.jpg
+ 1
0_0.jpg
...
+ 8
0_0.jpg
There is a tile for each level of detail (from 1x1 pixels up to 512x512 pixels) in the folders under the collection_files folder. I made some small tweaks to the collection to make editing in Photoshop easier, for example I changed the collection tile size to 1024 (you can do this with sparseimagetool quite easily), and I changed the level of the collection to 10, so all of the images would fit into the collection without the need to make additional changes to the individual images at each level.
You can also see how Deep Zoom nicely stores all of the images in a collection in a texture atlas at each level. This is the basis for the collection optimization, we only need to load one image tile for a bunch of images that are displayed together. Also nice because it makes it easy to change things around manually with Photoshop or any other tool.
Mark Dawson pulled it off. He wrote a Silverlight viewer for Photosynth. It uses Deep Zoom when you zoom deeply into images or in 2-D layout. The 3-D work is all based on his kit 3d work.
This is what it looks like:
And you can try it here.
I like to pretend that some of the 2-D layout work is based on my Photosynth 2-D layout stuff below. Well it is, right Mark?
Felix Wang from Microsoft China has created a little project that creates deep zoom tile sets from your flickr streams. It does everything for you and it also created a handsome little application that you can insert in your blog.
Try it out for yourself, it's amazing!
http://mooncake.nxmix.com/
This isn't meant to be a political page, however - Dan Cory has made this great example of a map of the united states, with two states - one the way the political map looks with appropriate coloration by vote, and one where the size is determined by the number of electoral votes each state has.
Press the re-arrange button to switch between the "normal" map view, and a view where the states are sized by electoral vote.
Click below to start the app in a separate window.
How is this done? At a high level, there are two layout files. One that contains the layout you see in the image above, and one that contains the layout by electoral vote. Each state is an individual image, and each image is part of a collection.
Then there is an animation that interpolates ViewportWidth and ViewportOrigin for each image between the layouts.
It's been a while since my last post, but that's because I've been also helping out with the Seadragon AJAX release. You have to check out what we did there. You can now browse single images in web browsers where Silverlight is either not supported or not installed.
Check out Seadragon AJAX in action:
The picture above is actually my own bike. I did this as an example of a sparse image, a technology I am saving for a later post. The above image is actually roughy 1 terra pixel in size. Of course it doesn't have that kind of resolution in every spot, in fact it's really only one terra pixel in the spot where the the image gallery spirals off into seemingly infinity.
Now look below at deep zoom:
So how does this all work, and what's the difference?
If you look at the AJAX version you'll notice a few things. For one, it's much slower (i.e. lower framerate than the deep zoom version). It's not as buttery smooth as deep zoom. The reason for that is the way browser renders images: it's just not that fast. Then, on top of it, there is a ton of blending between the various levels of detail, and all of that has to be done in the browser's DOM. It's not really what the browser was designed to do.
In Silverlight on the other hand, we've had the chance to improve the software rasterizer - the deep zoom rendering not only takes advantage of the ultra fast software rasterizer in Silverlight, we've also been able to speed up the blending significantly by optimizing the blending with direct bitmap manipulation. All of this shows that while AJAX is a great technology for some scenarios, but if you want the last bit of performance, you'll be better off with Silverlight.
Then the next obvious difference is the lack of collections in Seadragon AJAX. For the deep zoom app above I cobbled together all of the images in the Seadragon AJAX gallery and put them into a collection that I posted on Silverlight streaming. So you can see all of the images from the Seadragon AJAX gallery at once in Deep Zoom, and you could move the images independently. BTW - how I built this collection is another whole topic post for another day.
So why did we do Seadragon AJAX? It's quite simple. I've been in a number of meetings where people loved the zooming UIs that Deep Zoom enables, but advertisers usually back off when they hear it requires an ActiveX install. They don't even want to invest in cooler zooming UIs because of that, or they want to wait until Silverlight has a larger deployed base. Well, now there is no reason to wait. You can use Seadragon AJAX as the downlevel version when Silverlight is not installed, then graduate the user to a smoother, richer experience with Silverlight Deep Zoom if it is installed.
And then of course, Seadragon AJAX is so cool, we just had to do it!
Well, I also updated this app now that Silverlight 2 has RTW'ed. The performance improvements we've made in Silverlight are truly stunning. I will reserve the details for a later blog post, but one of the things that deep zoom always does is blend the various levels of detail as they become available over the network. Now if you use Virtual Earth data, as it is the case in the example below, well when you zoom in really fast and the network isn't as fast as you are with the mousewheel, then you may end up having to blend a lot of levels of detail.
Needless to say, even if we have to blend 8 or more layers at the same time, our performance does not drop anymore. We keep up a very high framerate and that's definitely a great thing!
As usual, you can get the Source code here.
Wow, it's been a couple of weeks since I posted last. I was actually over in Germany for Photokina 2008, one of the largest conferences for imaging in the world. It was held in beautiful Cologne, Germany. I showed Photosynth to the awed crowds, it was a fantastic experience being at such a large consumer focused conference showing off Microsoft technology and actually getting ooh's and aaah's.
I guess I got to go because the intersection of folks who worked on Deep Zoom, Photosynth, and who speak German is one person. That's me.
Anyway since then Silverlight has RTM'ed and I had to wait for Streaming to updated, which it has been now so go ahead and check out Photosynth again:
Source is now also available here.
I just got an e-mail from Scott saying that the Photosynth team is looking for people. Well, I figure people who have landed here are folks who may be interested. You know and care about Silverlight and Photosynth, looks like you may be in the right corner!
Anyway, take a look at the job postings here. Enjoy!
If you haven’t already, check out Photosynth, the latest
release from Livelabs showing off this fantastic technology that stitches
images together in 3-D space and lets you navigate your vacation photos. It's better seen than described in words, so don't hesitate to blow off a few minutes looking at the great Synths that have already come in from all over the world.
Well, the wonderful thing about Photosynth and Deep Zoom is
that we share the same file format. A collection created in Photosynth can be
opened in Deep Zoom and the items in the collection can be arranged to a
developer’s liking.
Click below to see the application that lets you view any
photosynth collection you may have found and liked, but don’t have a chance to
look at because you are using a Mac (like me at home) or because you don’t have
a supported browser.
How does one open a Photosynth collection?
Well, search for your favorite Photosynth, right click the
link and copy it, or copy the URL from the address bar, and paste it into the
viewer. The viewer does the rest to open the collection and lay it out on a 2-D
grid.
Some details on the Photosynth format
All this little application really does is analyze the URL
to a collection and pull out the right pieces to reconstruct the URL to the actual
collection.
Let’s take the following example:
http://photosynth.net/view.aspx?cid=ffa4cce7-d8ec-44f8-9eac-ab28b838cdcb
Is the URL to a photosynth of a collection. The resulting
URL to the collection (according to fiddler) is:
http://mslabs-100.vo.llnwd.net/d4/photosynth/M6/collections/ff/a4/cc/ffa4cce7-d8ec-44f8-9eac-ab28b838cdcb.dzc
Note that Photosynth uses the extension “.dzc” (Deep Zoom
Collection) for collections, and “.dzi” (Deep Zoom Image) for images.
So what do you need to do to get to the collection given the
URL to the Synth? Well, first, you extract the collection GUID (ffa4cce7-d8ec-44f8-9eac-ab28b838cdcb).
Next, you need to construct the path to the collection from
the GUID. The first part can always be the same:
http://mslabs-100.vo.llnwd.net/
Next, there is a “d”, followed by a number. This number can’t
be derived from the GUID, it’s different for every synth. So I need to try
numbers between 2 and 8 to see where the collection is.
The next bit (/photosynth/M6/collections/)
is always the same. The next bit (/ff/a4/cc/) is simply
the first six digits of the GUID. Then add the GUID, “.dzc” and you are done!
Note on layouts
One thing to note is that Photosynth does not put layout information into the collection XML file. This layout information is normally used by Deep Zoom to place images in a collection on screen in the right spots (Deep Zoom Composer for example fills in the layout information).
So if you attempt to open a Photosynth collection without laying the images out yourself, you will be in a world of hurt - the default behavior for deep zoom when layout information is missing is to layout all the images in a collection fitting to the container deep zoom is in. And since that can be hundreds of images, it can be pretty slow.
A very common request I have been getting is some sample that shows how you can synchronize content with Deep Zoom.
Click on the image below to start the app. Zoom in to see the various places I've lived.
Download the source here.
So how is this done? Really there are two main concept I show in the app:
Synchronizing other content with Deep Zoom
When you set ViewportOrigin or ViewportWidth (or use the ZoomAboutPoint helper), deep zoom simply kicks off an animation to the new location. This animation happens entirely under the hood and is transparent to the developer. That's very nice for the dead simple zooming application, since the developer doesn't have to worry about making pretty animations when zooming and panning. Pretty animations are built in.
However, this behavior does cause a problem if you want to make other things in Silverlight move along with zooming and panning of the Deep Zoom image.
So what's the trick? Everytime a user pans or zooms, this application sets of a 0-length-storyboard that tracks what Deep Zoom is doing, and adjust size and position for the pins. The 0-length-storyboard does its thing for 1500 ms, which is exactly the amount of time Deep Zoom allocates for each animation.
Using powerlaw scaling
Another nifty little feature that was put into this application is powerlaw scaling. The size of the pins only grows at a fraction of the rate of the rest of the map. This needs to be done, since if the size of the pins is kept constant, our eyes fall victim to an optical illusion: when zooming in, the pins would look like they are shrinking, and when zooming out, the appear to be growing.
Power-law scaling makes this visual effect much less noticeable.
