Mark Dawson pulled it off. He wrote a Silverlight viewer for Photosynth. It uses Deep Zoom when you zoom deeply into images or in 2-D layout. The 3-D work is all based on his kit 3d work.
This is what it looks like:
And you can try it here.
I like to pretend that some of the 2-D layout work is based on my Photosynth 2-D layout stuff below. Well it is, right Mark?