I think that we – and by we, I mean we developers and developer evangelist types at Microsoft – get touch and tablets, or slates, or pads, or whatever you’d like to call them, better than the Ars Technica article Ballmer (and Microsoft) still doesn't get the iPad (written by Peter Bright and posted in the One Microsoft Way section) implies. I believe that over the next few months, you’ll see some interesting touch-related stuff coming from Microsoft, and that we have a responsibility to help developers understand the differences between mouse/keyboard computing and touch computing.
In anticipation of this, I’ve been make my move towards touch- (and other sensor-based) computing over the past little while, by migrating to the following devices:
The idea behind this purposeful move towards touch-equipped devices is to truly understand touch-based interfaces, which UI elements work and which ones don’t, and then to pass the lessons learned to my audience – developers and designers, whether you build for the Microsoft platform or the platforms of the Esteemed Competition.
My own move towards touch-based devices is a microcosmic example of the larger changes taking place at The Empire. The move to touch interfaces is taking place on Microsoft computing platforms of all sizes:
As the Ars Technica article points out, one of the signs that we do get touch is the new interface design of Windows Phone 7. The design philosophy is build around touch (and other sensors), and the WP7 “design bible”, the Windows Phone User Interface Design and Interaction Guide [12 MB PDF], explains this philosophy beyond the mere technical details. Here’s the introduction to its section WP7’s touch interface (any emphasis in the quote below is mine):
Touch input is a core experience of Windows Phone 7 and has inherent differences from traditional keyboard and mouse input systems. Designed for natural and intuitive user interaction, touch input in Windows Phone 7 enables users to interact with application content such as a photo or a web page. Touch input enables simple and consistent user touch gestures that imitate real life behavior, such as panning on a photo to move it. Single-touch gestures make interaction easier with one hand, but multi-touch gestures are also available to provide more advanced gesture functionality. Application developers should strive to create unique and exciting experiences that encourage the discovery of content through the use of touch gestures. Users should enjoy the experience of navigating through the steps of a task as well as the completion of the task itself. Touch gestures should provide a delightful, more colorful, intuitive experience within applications Touch delights the senses as the user gets to see the interaction match the performance. The touch UI should always have aware and responsive performance, just like how real world objects respond to touch immediately, and applications on Windows Phone 7 should as well, by performing the action in real time and by providing immediate feedback that an event or process is occurring. Users should not have to wait as it breaks their immersion, flow, and concentration, especially as their gestures transition from one to the other. For example, a pan may turn into a flick or a tap can become a double tap, and the user should not be aware that the UI is switching gesture support.
Touch input is a core experience of Windows Phone 7 and has inherent differences from traditional keyboard and mouse input systems. Designed for natural and intuitive user interaction, touch input in Windows Phone 7 enables users to interact with application content such as a photo or a web page. Touch input enables simple and consistent user touch gestures that imitate real life behavior, such as panning on a photo to move it. Single-touch gestures make interaction easier with one hand, but multi-touch gestures are also available to provide more advanced gesture functionality.
Application developers should strive to create unique and exciting experiences that encourage the discovery of content through the use of touch gestures. Users should enjoy the experience of navigating through the steps of a task as well as the completion of the task itself. Touch gestures should provide a delightful, more colorful, intuitive experience within applications
Touch delights the senses as the user gets to see the interaction match the performance. The touch UI should always have aware and responsive performance, just like how real world objects respond to touch immediately, and applications on Windows Phone 7 should as well, by performing the action in real time and by providing immediate feedback that an event or process is occurring. Users should not have to wait as it breaks their immersion, flow, and concentration, especially as their gestures transition from one to the other. For example, a pan may turn into a flick or a tap can become a double tap, and the user should not be aware that the UI is switching gesture support.
There’s a great amount of understanding behind the nuances of touch-based interfaces in the Windows Phone User Interface Design and Interaction Guide, and over the next few months, we’ll be covering them in great detail in this blog.
When the Surface, a.k.a. the “Big-Ass Table”, came out, a number of people asked why such a big, expensive thing was built and what practical purpose such a beast would serve.
For starters, there are a number of customers who use it, from casinos in Vegas to bible study classes in megacurches to places closer to home (by which I mean Canada), from the company that did the security for President Obama’s visit to Ottawa to super-sexy Toronto design firm Teehan+Lax to Ontario College of Art and Design to Infusion, who’ve built applications such as Noront Resources’s GSI Surface tool to the security app Falcon Eye.
Equally important are lessons to be learned about input from touch and other sensors from a “concept” machine like the Surface, whose built-in camera systems allow for way more touch points than a resistive or capacitive touch screen will allow, as well as the ability to “see” objects on the tabletop. By being empirical and building such a computer, developing software for it and watching people interact with it, we learn more about touch and sensor-based computing way more than we could from mere theorizing.
I think Des Traynor captured our intent quite nicely in his article about Surface and other Microsoft efforts in the field of user interface:
When the Surface was released two years ago it was chastised by the public. The joke at the time was: “Apple and Microsoft both invest in multi-touch technology, Apple release the iPhone, Microsoft release a $15,000 coffee table!”. But Surface wasn’t about “re-inventing the coffee table”, so much as it was prototyping a vision of the future of computing. There will come a time when “gathering around a laptop” will seem as ridiculous as connecting an ethernet cable; a time when everyone gathers around a multi-user computer to have a meeting or debate a design. With something like surface, Microsoft are preparing for that day.
When the Surface was released two years ago it was chastised by the public. The joke at the time was: “Apple and Microsoft both invest in multi-touch technology, Apple release the iPhone, Microsoft release a $15,000 coffee table!”.
But Surface wasn’t about “re-inventing the coffee table”, so much as it was prototyping a vision of the future of computing. There will come a time when “gathering around a laptop” will seem as ridiculous as connecting an ethernet cable; a time when everyone gathers around a multi-user computer to have a meeting or debate a design. With something like surface, Microsoft are preparing for that day.
A lot of the knowledge from Surface applications have been injected into Windows 7 in the form of the Windows 7 Touch Pack. This pack gives Windows 7 a touch-based API and a set of apps originally designed for the Surface, so that they can run on touch-enabled computers, such as HP’s TouchSmart series, touch-enabled laptops like my own Dell Latitude XT2 as well as any computer connected to one of the new touch-enabled monitors (our manager John Oxley has one in his office).
The Ars Technica article goes on and on about Windows 7’s standard interface controls being too tiny for touch, but a quick look at the Touch Pack apps reveals that they don’t use the standard controls; rather, they use controls better-suited to touch. Here’s a screenshot of Surface Collage, the photo-collage application, running on my XT2:
No standard Windows controls here! You manipulate the photos directly using gestures, and the strip along the bottom is a photo list, which you also manipulate through gestures. The closest thing to a standard Windows control is the “close” button near the upper-right hand corner of the screen, which is larger than the typical “close” button – small enough to be out of the way, yet large enough to click with a finger.
Here’s another app from the Touch Pack, Surface Globe, also running on my XT2:
Once again, no standard Windows 7 controls here, but a map that you directly manipulate, augmented by finger-friendly controls.
The Touch Pack apps all follow this philosophy: when going touch, eschew the standard Windows 7 UI controls in favour of touch-friendly ones, and then back to bog-standard Windows 7 when exiting them. These apps show not just that we understand that touch computing is a different beast from mouse-and-keyboard computing, but that we also understand where they intersect.
We’re working on what I like to call “the touch continuum”, which spans pocket devices such as the Zune HD and Windows Phone, to portable computing with netbooks, laptops and soon, tablets, to desktop and tabletop and wall-sized units. And yes, we get that new types of user input call for new user interfaces and give rise to new usage patterns. We’re aware of the challenges of touch (and other sensor) input and over the next little while, you’ll see our answers to those challenges. And better still, we’ll share what we’ve learned in order to make you better developers and designers of software that use these new interfaces.
This article also appears in Global Nerdy.
My response might be a little facile, but really....
DrPizza: I don't think it's facile at all; in fact, I think it's a very reasonable question that calls for an equally reasonable answer.
Most of the Windows 7 Touch Pack applications run in full-screen mode, hiding all standard Windows parts, the notable exception being Surface Collage. It serves me right for choosing to show a screenshot of glaring exception; it would be only a matter of time until an eagle-eyed reader pointed that out. I have no idea why this exception was made for Collage.
The quick and dirty fix is to restore the taskbar to its default size. In my particular setup, I keep a lot of icons on the taskbar, so I've got the "Use small icons" option turned on. The taskbar probelm can be fixed somewhat by turning that option off, which changes the taskbar height to its full size of 40 pixels. The entire height of any taskbar item, even the system tray icons, is clickable, which means that they're all effectively 40 pixels high. That's larger than the minimum recommended size for a WP7 button, and just a shade smaller than the minimum recommended size for an iPhone button (44 pixels).
The better fix -- and the one that I hope gets incorporated into the Windows UI -- is a task-switcher/application selector that works better with a touch interface, that not only takes advantage of touch, but also of associated human factors such as "muscle memory". Luckily for us, this is a pet topic of Microsoft Research guru and UI supreme master Bill Buxton, and I hope it's something he provides input on.
(Oh, I guess I should explain if it's not clear; I wrote the Ars article)
The thing about the Surface software is, it's fundamentally a bunch of special apps. XP Tablet has had special apps in the past, and that wasn't enough to do the job. Apps are certainly a necessary part of the answer, but it needs to be recognized that they are *just* a part. Sticking a few touch apps onto an operating system that is fundamentally non-touch is what Microsoft has been doing for quite a long time now, and it's not met with success.
Consider the iOS devices. Everything you do on them--whether it be joining a WiFi network, configuring your e-mail, browsing the web, installing applications--is finger-friendly. It's not enough to say "here's OneNote, here's the surface apps, and there's an on-screen keyboard too".
The touch UI needs to encompass everything that a touch computer might reasonably need to do, and from what we've seen so far--with things like Surface--it doesn't.
I mean, I think there are interesting things that can be done with hybrid devices (ones where you can revert to a keyboard and pixel-perfect pointer, or at least a decently accurate one like a digitizer pen), but I don't think that the system can *require* such things. There needs to be a complete self-contained pure-finger environment (shell + apps + APIs), and it needs to be more than skin deep.
So, yes, it needs a task switcher and application selector. But it also needs options screens, network configuration, an e-mail client, a browser, application maintenance, etc. etc., that are touchable.
If Microsoft can't manage this then it won't ever have something that's really comparable to the iPad. Sure, there are those who say that Microsoft and Apple are different companies, and just because Apple has succeeded in a particular market doesn't mean that Microsoft needs to follow. And if Apple were sticking with MP3 players, I would agree.
But iPhones and iPads are both a _kind of computer_. Maybe a limited kind of a computer, with certain kinds of specialization, and as such, they're squarely within MS's core domain. Similarly, if the iPad had kind of fizzled then again, it might not matter much. But indications, admittedly early indications, are that there's a genuine market for these computer-like devices, so it's clearly a field that Microsoft should be playing in.
This is not to say that *everything* needs a finger front-end, necessarily. iOS is again an illustrative example, I think. Behind the scenes, iOS is, more or less, Mac OS X (it started out quite a bit less, but each new revision is incorporating more and more Mac OS X functionality). But that doesn't mean that when designing the network configuration for iOS that Apple had to support every option from Mac OS X. They just provided a limited set of options that covered most of the bases. Other things are off-limits entirely--there's no Finder on iOS, for example.
But everything that's reasonable on a touch device needs to have a true touch UI, and that's not something Microsoft has managed before.
I have heard it suggested that there is an effort to produce such a front-end, a "modern shell" (though I don't really like calling it a "shell", because a shell implies to me something that is rather thin; something that can be chipped away to reveal the interior. This is not possible on the iPhone, nor should it be on a Windows slate IMO), that would be a better model for touch devices. However, it is hard to be confident or excited about such a thing--if it even exists--when Microsoft is saying nothing about it.
And Ballmer's comments--that Microsoft was "tuning" Windows 7 to make it work well on this kind of device--do little to inspire confidence. I think it needs more than just tuning to be good enough on touch devices--at least, in the way that iPad, iPhone, iPod touch, Android, and soon Windows Phone 7 users understand "touch devices".
IOW, I should be able to buy an HP Slate or some comparable device, turn it on, and never once be aware that the OS was designed for something other than my fingertips.
I think that the touch interface is a great thing. Hardware has transformed since the AT, yet most people are still stuck with keyboards and pointing devices like I was using on a DECstation2000 in 1982. McDonnel douglas had some great things, like lamp-and-key menu boxes, spin wheel arrays, and the like - all forgotten.
Finally with a touch interface we can start to really innovate, instead of just having prettier desktop backgrounds and feeble addition of tabs and things to windows. I loved the windowing model in OS9 & OS9000 where each task had a window, and each window was a task and they all ran completely separately, enlarging and shrinking as you or they required.
But where are the other input methods? What about voice control? why can't we tick tounge-click and squeak at the thing to make it do stuff? What about eye tracking - blink twice to click - or what about the real missing link - direct bionic connection? Why can't I think at the thing?
Kirk had a talking computer in 1967. I'm still waiting, despite how much more horsepower a xeon has over a 286.
In answer to Dr Pizza:
"The thing about the Surface software is, it's fundamentally a bunch of special apps. "
That is what infuriated me about early tablet systems. The handwriting and gesture recognition was built into word, not the operating system. You could use it for what microsoft wanted, but not for what I wanted. I use Serif software for drawing - it could not see the tablet extensions at all. If I wanted a computer that only ran the vendors software I'd have bought an Amstrad PCW
No. Kirk had a talking computer in 2266. So Microsoft still has some time to come up with one. And: We can be sure Ballmer won't be around anymore by then (so you probably won't have to start every conversation with your computer by saying "Start").
Joey, you addressed the touch support issue, but one thing you didn't address much is the notion that Windows 7 is generally good operating system to put on tablets.
Let me be clear: The world's best operating system for multi-touch tablets targeting users that need to run legacy Windows applications is... Windows 7. There is a huge market or businesses that are interested in these sexy new tablets (now that iPad has shown what they can really be like) that will want to run their legacy apps on, and for this, Windows 7 multi-touch tablets will sell well. Microsoft can rest easy, nobody can take this market away, not Google, not Apple.
But as an operating system for multi-touch tablets targeting individuals, it is not at all well suited, dare I say not even close. That's why I think Ars is saying that Ballmer and Microsoft doesn't get tablets.
I'll point out a few things about Windows 7... that might cut the mustard for businesses but is awful for individuals faced with the choice of iPad vs. Windows 7 tablets.
User experience: Windows 7 seems to barely scratches the surface in supporting touch. When you need to go into the control panel, you're going to have to deal with standard windows controls. File manager. Task bar icons. Start Menu. There's just so much to "fix". And when I say "fix" I mean, to just "touch-enable", let alone rethinking the stuff to make it "touch-optimized".
Windows 7, despite being adapted over the years to save power, is not really power-saving by iPad standards. All kinds of stuff runs in the background, it supports multi-tasking, needs Windows Defender / virus checkers, etc.
Windows 7 needs power. Users are going to want to run their old apps (because they can). It really needs a full power motherboard, with all the RAM and amenities. If not, users will call it a dud. This takes up space and power.
Did you know iOS devices run with 128, 256, and now 512 Megs of RAM? And the apps are fast. Google Earth runs on iPad with only Did you know that 256 Megs of RAM. Did you know that iPhone 4 is smaller than iPhone 3GS, yet has about the same graphics resolution and power as the iPad, in that small space? Next year's iPad will be thinner, lighter than ever if this is an indication (the battery will be bigger to handle the larger screen light output).
Windows 7 supports USB, and tablet makers will put USB on their tablets. USB devices need drivers. Drivers cause problems.
Windows 7 does not force users to install apps through an App Store. What an incredibly big opportunity for Microsoft, and users, that will be lost. And how can you tack that on without destroying backwards compatibility? You can't. You can't tell users "This edition of Windows can only download apps from the App Store -- the stuff you already have we won't let you install".
To me, I think you should start accepting the fact that Microsoft is going to have to have a different Windows, like a Windows Phone derivative (or Windows Phone itself better yet) for the "other" category of slates that will otherwise not be able to effectively compete with iPad.
Love your article. Thanks for that, it is encouraging nonetheless.
One of the things being ignored by Ballmer in his speeches about Win 7 tablets and software compatibility, is that those tablets won't be very compatible at all. As was mentioned in the previous post, Apple's tablets require less RAM, and that will be true for Android tablets as well. But, Windows 7 requires at least 2GB to function properly. And no matter how fast Intel gets its mobile tablet/phone chips to run, they will still be far to slow for most Windows software.
This all means that Win 7 tablets will be in the same boat as netbooks. That is, they run, in theory all of the software that more powerful laptops run, but that in fact, they can't. These tablets will be even more constrained than netbooks are because they will have even less powerful hardware, and lack the mouse/trackpad. We've been all over the UI problems of Win 7 for tablets already. Just look at the Archos 9 Win 7 tablet reviews to get some idea of how future Win 7 tablets will work; not good.
I'd like to remind everyone that Windows 7 isn't the only software platform option available to us, as evidenced by Winodws Phone. what's far more important, as far as I'm concerned, is the .NET framework, which is quite portable to the smallest of devices.
I don't know. It all sounds cool and all, until you have to spend lots of hours (with a few breaks here and there) manipulating a touchable device without a way to rest your arms. You can sit at a computer keyboard plus mouse for lots of hours because for the most part you can rest your arms on a desk-like surface. So far I haven't seen an example where the elbows would be down on somthing to manipulate the screen. "Minority Report" is all well and good in small bursts, but NOT sustained. Has Microsoft done any studies yet on how much faster work could be done via touch technology so that the rest periods can be more frequent and longer?
I forsee lots of shoulder injuries where the workload didn't allow for a lot of rest, and bosses unwilling to lower the workload because somebody else in another country would be willing to have THEIR arms drop off and take less money doing it. Some ideas need more thought...
It's the whole experience that does not come together on Win7 like it does on iOS. iOS is really a stripped down Mac OS X. iOS is UNIX under the hood with a custom GUI "shell" you can even SSH to it if you jailbreak the iPhone/iPad/iPod. Apple re-did their development API's to support the touch interface but kept the same basic IDE development environment.
I suspect that Windows Phone 7 is running MinWin under the hood with a custom GUI "shell". However, that particular interface is not going to work on a tablet all that well and will need further modification.
The trouble with Microsoft is they have all these different groups doing their own thing and not necessarily collaborating. I mean why are there like 3-4 different File Open dialog boxes depending on the application? One for Office, one for old apps, one for Windows built-in apps, etc. See, there is little in the way of consistency. Going to have to break down the barriers and forget about backwards compatibility to build a tablet to compete with Apple.
Mr. deVilla, take it from me, a developer who's been writing software for Microsoft platforms since WFW 3.11 (among other things) and who now also writes for iOS and OS X, TAKE DR. PIZZA'S ADVICE SERIOUSLY.
He's not just trashing your work, *he's dead on*.
The W7P interface perfectly shows that Microsoft is capable of touch interfaces, but why is it called Windows 7 PHONE and not Windows 7 MOBILE? Was that done purposefully to show this is a "phone only" interface, and don't be believe for one second that we'll port this over to iPad sized computers any time soon because that's Windows 7 territory?
The problem with Microsoft isn't that they're not capable of great innovations, it's that its hamstrung from all the infighting we see going on. Why the complete and utter insistence of styluses are a necessary part of the touch interface? Android nor iOS seem hampered by the lack of a stylus. And, I bet W7P won't be crippled by the lack of a stylus either. Again, the thinking is that this is a political decision to keep the Windows Desktop folks happy.
And why was the Kin so late in its debut? Was there any truth that it was delayed because it had to be rewritten to use Windows CE as an OS underpinning? And, why is there rumors of a new movement to "unify" all of the code in all devices to "Windows"? Will WP7 have to be rewritten to comply? W7P already faces a tough up hill battle. W7P has to prove to be better than iOS and Android if it wants to remain relevant. Zune HD is pretty good, but hasn't been able to compete against the iPod Touch or the iPhone because it isn't better.
Microsoft doesn't need new tech people or new technology. Microsoft is capable of some amazing stuff, but the management needs a good overhaul. Someone needs to break up the various factions in Microsoft and stop all of these nonsense turf battles we've seen over the last few years. Microsoft needs someone with vision at the helm.
DrPizza, I agree with you on the whole, but not on your interpretation of the word 'shell'. Coming from a Unix background, I envisage the shell as being a user interface that allows one to interact with the underlying kernel services. I would hate to see a touch GUI layer slathered on top of a traditional Windows UI, but if the new UI interacts directly with the kernel, this would seem quite reasonable to me.
And when Microsoft does everything the always reasonable and credible DeVilla is sure it will, the result will by default be inaccessible to blind people, a fact that, in Microsoft tradition, can be remedied by purchasing unstable third-party software for more than the price of the tablet.
What does the Esteemed Competition do?