Evan's Weblog of Tech and Life : Tech

Methodology & Observation

EvanF — Fri, 18 Mar 2005 23:34:00 GMT

I haven't talked about methodology in a long time (at least not on here). I was recently in a conversation with someone about doing some research using a remote screen capture tool on a PC. The person (who shall remain nameless -- but you know who you are!) basically recruited participants to try out a build of a new concept for a particular software program and was using a tool that allows all of the interaction on the computer to be encoded to a video file that can be sent back over the internet. This way the researcher could watch a video of how the user interacted with the software while sitting back at her office.

I happened upon her in her office when she was watching the video. During the video I noticed that there were gaps in the video, in that there was a pause in what the user was doing while in the midst of the task trying to be accomplished. We talked a little about what we had seen in the video, and I brought up this observation to her. She didn't think that it was all that relevant -- and I disagreed and suggested that unless she knew why the user wasn't finishing the activity in a continuous fashion, then she really didn’t really understand how the user was doing the task.

For example:

- Was the user trying to get the task done and the kids were fighting in the background?

- Was the user puzzled by what he were trying to do and just sat there trying to figure out what to click next?

- Was the user asking someone for help?

- Did the user forget what they were doing when he was returning to the task and thus sitting motionless for a while before starting up again?

- Did the user really make errors as a result of not remembering where they were at in the task or were the errors the result of something else?

At this point she wasn’t very happy. A lot of planning had gone into setting this study up and getting the software properly instrumented and set up for the participants. At the time she was planning the study, it hadn’t dawned on her that the context in which the user is performing the tasks is as important as the task at hand. (Design solutions to address where to click next are much different than a solution focused on helping the user remember what they just did if they’re being distracted while performing the task.)

Technology is great, it can really help to better understand a situation, but if you’re not getting the full context then you’re missing a part of the bigger picture. A lot of products are built in a vacuum, not because there’s missing data about the user, but rather they don’t take into account the larger ecosystem. If you can capture the user in that context, when they’re doing a task you’re interested in, then you’ll have a much richer set of data about how to design a product.

CES and small computers...

EvanF — Tue, 11 Jan 2005 00:22:00 GMT

While my OQO still hasn't arrived, I was able to borrow one from someone on the team who isn't really using his and thus I wanted to give the device a workout during an event that was in theory designed for the device. Well let's just say I had some technical difficulties with the machine which I'm working on resolve.... but at the highest level and since MSFT is paying me to evaluate the device for now I'll just say that it's an interesting companion.

But on to small computers.... There were 2 devices caught my eye, not from a mobility perspective but rather for a feature/function/size perspective for the home.

First from Aurora Multimedia was the XPC Pro. This is a PC is a full function PC with video and surround sound capabilities in a package 8.5 x 1.75 x 13 with a built in DVD drive. They were showing it off as a second PC to attach to your TV etc, but I think that there are probably some other potential usages that are pretty interesting. However don't have a clue about the performance etc, so maybe I'll get to try one out

Second was basically a complete DVD player/Stereo about the size of a car radio that can be mounted in a drive bay of a PC or used as a standalone component. This was the VPC-2000 from Asour Technology. This reminded me of the days that I worked for Compaq and we had designed a car stereo looking component in the front of the PC that did a lot of the same functions, but at the time was limited by Windows and ultimately failed since the actions didn't happen in real time. You would change the volume and 10 seconds later Windows would respond and then the volume would actually change. However in the implementation that I saw here, I thought it was a well integrated package that really does a great job at consolodating the functionality.

One other interesting product was a thing called the Pocket Surfer from Datawind. This is basically a thin client that's about the size of a checkbook but really thin and light. It uses bluetooth to your cell phone and connects to a backend server to let you browse the internet. A much larger screen than a blackberry or most of the palm or ppc devices out there so it has some interesting applicational uses.

I'll say more about my experiences with the OQO once I finish up my overall evaluation...

A really small computer

EvanF — Tue, 14 Dec 2004 00:19:00 GMT

Over the last week, I've been playing with a development board from a chip manufacturer that is a relatively dated product in that it really isn't all that fast, doesn't use any cutting edge technology, but runs Windows XP at an acceptable level. Sure I'm not going to be going to play any games on this machine, but for doing my general work day in and day out, this is an awesome board.

The dev board is about the size of a standard size hard drive, but the board has a ton of space on it and could easily be miniaturized to encompass less than the volume of a laptop drive! It has all the ports you'd expect USB, audio, VGA etc, but it runs at a very low processing speed compared to anything that you'd buy today. What I find really neat about this device is that there is no fan, the power supply is just about non-existent and I can put my hand right on the CPU while the machines been on for days on end. Okay so it's a little warm, but by no means is it burning hot.

Why do I like this little dev box so much? Well right now I have a second PC at home for my daughters to play games (things like Fredie Fish -- nothing that's a demanding application), but the cheap PC that I put together for them is just so noisy and takes up a lot of space. Just imagine if I built a PC around a board like this, plugged in an external CD (unpowered) and ran the USB cable up to the desk next to the keyboard. An ultra compact machine that does everything that I need it to do.

But it gets better. The cost of the entire board and the existing housing of this device probably is clearly less than $100 maybe as close as $50. If someone were to manufacture this in volume, you could probably have a complete PC (keyboard, mouse, CD, HD, motherboard, etc.) that runs XP, Office and other standard apps plus general games (nothing ultra-intensive) for $100 for the entire package, software extra of course :-)

Would you buy one? Line starts after me!

OQO not on it's way :-(

EvanF — Mon, 06 Dec 2004 23:29:00 GMT

Well appears that the OQO that I had ordered, wasn't really ordered, so I'm not going to get a Hannukah present this year, maybe in time for New Years or perhaps in time to take it for a test run at CES. Having had a few minutes to look at the one that one of my co-workers is using (and is frustrated with) it seems to me that perhaps something like this on the floor of CES would be great particularly when paired with wireless connectivity (either via a cell phone or Wifi). However for CES a camera is real useful too (but no integrated camera). Of course I have to see whether or not this whole pariing of devices works out for this particular application; and of course there is that small little detail of cost-justification...

Oh well, I've got some other devices that I've got to do some more in depth evaluation of.

Having a second or third or fourth computer...

EvanF — Thu, 11 Nov 2004 18:12:00 GMT

I know almost everyone out there who will read this blog has more than 1 computer at their disposal already, so what I really want to know is if you were to give an additional computer to your brother, sister, mom, dad, great aunt Sally, or whomever who only has a single PC today, what would you want them to experience? That is to say, if you were to give them the computer what would you want them to get out of having this second computer that they don't get out of having the original computer that they already have. And if you were to do this, I want to know more than "it's a better computer" than the old one (in that you want to replace the old one with the new one) rather what benefit if any would there be to having these multiple computers for this particular person. Place yourself in their shoes, not your own, of course you want one machine to develop on, one machine to play on, one machine to experiment on, one machine to ...., but does your great aunt Sally?

OQO on it's way?

EvanF — Thu, 11 Nov 2004 18:07:00 GMT

So in my quest to have time on every small form factor machine out there and to figure out what the real end value is behind these devices, I did order an OQO. Appears it's somewhat backordered, as they got more orders than they expected -- which is either good news for them or that they produced a very small number to begin with and anything over that is more than they expected :-) Needless to say, I found JK's comments on what he's been seeing relative to this machine troubling. Particularly the comment about the digitizer. Getting these things to work well is tricky - all kinds of little things interfere with the accuracy of the electro-magnetic digitizer and if you look at the edges on any Tablet PC, you're bound to find a spot or two (or more) where the calibration is just off and you can't do much with it. What worries me here is that since the device is so small, there's only a very limited amount of space that a "human" can target and that will often be close to the edge on a device like this. Thus this would severely limit the overall usefulness of the digitizer itself. Guess we'll just have to wait and see when I get mine.

Tablets on Display...

EvanF — Wed, 20 Oct 2004 16:41:00 GMT

In my previous post, Chris asks why not put this one of a kind Compaq prototype in the display case of all the tablets that is in the building where the tablet team is located. I'll see if I can find a picture of that display case that I can link off to since I can't easily post images here, but it's not as easy as it sounds... Basically there is a glass show case in the lobby of our building (which incidentally is getting plastered with Halloween decorations, but that's a different story) where there are many old Tablets, webpads and other misc hardware that helped to inspire us and let us test out our ideas. The funny thing is that many of the Tablets that we wanted to put into the display case don't fit. The problem is that the glass shelves are of a fixed height and most of the tablets that are primarily designed for a portrait orientation are too tall for the shelves. At one time we had some in there positioned in landscape, but it just looked goofy with a few of them on their sides. So until someone donates a better display case, we'll all have to just hold on to our memories :-)

New iMac is out...

EvanF — Tue, 31 Aug 2004 20:04:00 GMT

Drum roll please.... So the web is abuzz with the new iMac and the latest creation about how it's so thin and just a simple machine. For all of those betting it would be a tablet or at least include a digitizer, you know who you are; I'll take cash, check and credit cards! Of course it doesn't what did you really expect? Would it be great if it did. Sure! But it doesn't.

Actually the new design inspired me to go down to Fry's and pick up the biggest desktop replacement laptop I can get. One preferably with a 17 inch screen (though I'd probably settle for the 15" one that was only $799). So I figured that all of us who aren't Mac users could just engage in 2 minutes of case moding and have our own iMac knock offs. If you don't have a Tablet (like maybe the new Averatec that's low $) you can just buy any old laptop, and remove the screen and then turn it around. Then all you need next is to get a piece of metal to hold it up. I'll probably rumage for a broken LCD display and just rip off the base and then mount the laptop guts to the stand. For the extra $2.50 you could spray paint it white.

It's one of those days and I just couldn't resist.....

Niche products? Here's an example maybe...

EvanF — Sat, 07 Aug 2004 00:41:00 GMT

I was planning on blogging about niche products and how you need to do research to help understand the product in order to determine if it’s worthwhile mainstreaming. But the truth is, I never got up the energy to follow through with my thoughts and wrote four or five drafts of something that I felt wasn’t even worth publishing to myself. So the result is, I’m going to talk about the Sony U50/70 products. I’ve already blogged about these before, wondering who is going to buy one and what it is they are going to do with. The consensus I’ve had is that there are a few people with lots of disposable income will find some cute tricks to do with them like a complete on the go interface to their Sony Aibo. Okay, I’m being saracastic. Fact of the matter is that I’ve only spent about 20 minutes with one of the devices when the interface was all in Japanese. In the next couple of weeks I’m going to be spending a lot more hands on time with one to do a more thorough evaluation. Unfortunately for readers out there, it’s part of my job so I won’t be able to post my evaluation, but I’ll try to at least relate my sense of use over the period of time. I know that Robert is pretty keen on the device as a natural fit for a Tablet, and I’ve read the gushing reviews of it from multiple other sites too…

All that said, I really do believe its part of a niche product at least in the current time frame. It’s expensive and it has what I perceive to be limited appeal for a particular audience. The question that I’m asking in my evaluation of the machine and possibly to anyone who deems to read this, is what are the defining factors in terms of “software” or “hardware/software interaction” that would be necessarily for this machine to move from niche to mass market. That is to say, if the price (which is outrageous) and the overall ergonomics of the device (which may be somewhat lacking) were no longer and issue, what does this machine have to do that would make it appeal to everyone or for everyone to say that this is what “mobile” computing is really about.

As a bonus question, you can apply this same logic to Tablets. What are they missing from a user interaction stand-point that would really make a difference in how you do your job, live your life or what you want?

Before anyone hits me with the obvious things like:
- Battery life
- Instant on
- Weight
- Price
- Different Operating System

Let’s try to focus on those SOFTWARE aspects or software/hardware aspects (e.g. a button for a specific feature) since the ones above are obvious and don’t really need to be stated.

One way to start thinking about this is, what software do you load on every laptop or tablet computer you get? For me, I switch machines often and have a standard suite of things that have to go on… Office (of course), Streets and Trips, Mpeg 2 codec, reVue (program for playing mpeg2’s that were captured from a Replay), OneNote, etc..

Okay so what do you 1) load on your computer, 2) what does something like a Sony U50 have to have to make it worthwhile and 3) what is the tablet lacking that would make all the difference?

Getting machines up to par

EvanF — Sun, 25 Jul 2004 02:01:00 GMT

I promised I was going to write about moving a product from niche to mainstream, but the experience at my in-laws was just an interesting (frustrating?) enough that I had to blog about it.

My in-laws live under some odd budgetary constraints. They definitely do not have a lot of disposable income, yet they seem to spend money on some things (like a cruise or a big vacation to places like Disney World with their son and his wife's family) which seems to me to be a bit over the top given that I feel like they scrimp on some basics (like a decent car and some household items). To me it's just a different situation and people who have a much different set of priorities than I was brought up with. The whole point in saying this is to give the background about the computers in their house.

My father in-law use to work at a computer rental company both selling, servicing and helping to build the business by being the resident IT support staff. He's been retired for at least 7 or 8 years when the company started to grow and got acquired by another company he was laid off and figured he really didn't want to work any more. So he's relatively knowledgeable about computers (or at least you would think so!) Well his home machine was a Pentium 3/550 which was definitely past it's time. It only had 6 gigs on it and was so loaded down with the residuals of past programs that no longer worked and other things that while it worked for surfing the net, it was by no means a good computer to work with. On top of that he had a 17” monitor that was fuzzy as all can be. Plus with his basic configuration he's got USB running from the cable modem directly to the computer so there wasn't a network in the home for me to connect to, so I was stuck looking at the real bad image on the machine trying to do a little bit of work and surfing.

So my brother in-law (his son) picked him up a “new” computer a couple of weeks ago, so I figured that since I'm here and always bored to tears here since I didn't grow up there and basically have to go on what ever errands my wife wants to visit old friends etc, that I'd put together his “new” computer as well as set up a basic network in the house (my treat) so that they could use the old computer as one for the grandkids.

Well it turns out the “new” computer was really a relatively old P3/900 that his son had picked up a computer swap meet for $100. Well for $100 it's not bad, it had all the components was properly imaged with Windows XP, had Office on it plus had a 30 gig hard drive. So I figured that I'd make sure it really was clean and re-image it myself etc. That wasn't too hard of a deal, but I began to think as I waited and waited for the machine to respond every now and then about whether or not it wouldn't have been more expedient and a better purchase to just spend another $300 and get a white box brand new machine with a faster processor better video card etc. His son also several monitors (also used) that he picks up every now and then so that he would replace the current one and bring over another monitor. This is where I don't understand the money expenditures, I for example wouldn't have probably ever bothered to buy such an old machine and would have just spent the money to buy a bare bones new machine, but like I said it's a different lifestyle.

So off I go to Best Buy since that's really one of the few places around here to buy computer related equipment especially since they were having a sale on routers. I pick up a router, some ethernet cables, an ethernet card for the old machine, some more memory for the “new” machine, etc.

So I spent the better part of yesterday putting the new machine together and getting the LAN working. Of course it didn't work out of the box. The D-Link router would see the machines on the network even my laptop which was wirelessly connecting, but just for the life of it would not get an IP address fro the WAN. Nothing I did made any difference, so I had to go call their tech support which luckily didn't take too long, but it was some advanced setting for the network speed that they had me switch which finally did the trick. That was good since I was loosing face, not that it really mattered. I was able to connect then with my laptop and with the old desktop. But the funny thing was that the desktop machine would just keep losing the internet connection every few minutes. You could get to a site or two and then it would just totally stop for no reason at all. Still don't know why that happened, but I suspect it had something to do with ZoneAlarms since I had trouble getting the application to even respond on the machine -- once i disabled that permanently it finally started to at least maintain the connection.

For me however it was frustrating as the old machine was really in a sorry state, but all the knowledge I have of networking etc, didn't allow me to trouble shoot why the connection got dropped etc. At the same time I was setting up the “new” machine and once it was ready I figured that I would just share out the c: drive of the old machine to copy files over as well as remote desktop to it since I only had one monitor at the time. Here again the machine just didn't want to respond when I was doing all the obvious things and the time just kept slipping away. Well after a day of doing this and finally getting it to work long enough to copy over the documents and some install programs for applications he had previously purchased, I finally got it all up and running. At this point I then ran a virus program on the new machine and didn't notice that it also was going to scan the network share, which in retrospect was a good thing since it started to find all sorts of viruses that it just couldn't seem to remove. So off to get the WinXP disks and another machine to pave over....

What I really wonder in doing all this is how often do people (not those in the technology field) have machines that just don't know how to fix and thus end up not fixing and living with something that is sub-optimum or how many people just throw it all out and just buy a new machine and be done and over with it (without ever getting their data off the old machine?) Sure there's a wizard in Windows to help with this migration, but that probably would have been a lot more work or at least had it's own set of issues.

Enough is enough or the world isn't enough?

EvanF — Tue, 20 Jul 2004 18:33:00 GMT

Or maybe it's eight is enough or 58 billion is enough? :-)

In a recent thread which was really aimed at one or two particular people out there, but I figured I'd make the conversation public and put it out for all to read :-) I suggested that rather than complaining about what some (okay maybe only a few) people are seeing as a flailing (or failing) Tablet PC effort and community, I'd entertain constructive feedback, not that I could action upon it, but rather to see what people are really suggesting needs to be done.

Many said "blog" and I asked "blog what"? Lora said just get people started doing it. Some private emails also suggested things along the same lines and started to answer what things people would want to hear about or what people should do. It's good feedback for the Tablet team and I'm hoping that they do something with it.

To be honest I only started the post to make a point to someone and look at where it got me :-) Not that I mind, but after all it's an interesting intellectual discussion which maybe someone else will pick up and run with. Either that or if I get a new job maybe I'll one day have this as input.

Okay, all that said, in addition to Lora's response to blog what, I got one really good response from Layne:

I'm not in the loop, not an evangelist, not an MVP, not even loyal to Microsoft. Heck, I've used Linux since 1995 but I'm attracted to the Tablet PC. It is a fantastic computer. The SP2 is nice.

However, I am amazed at the exclusitivity of your community and the general lack of discourse and discussion. It seems Some people just tell you guys what you want to hear. They are afraid of you and I doubt they are being honest. So, expand your horizons and your reach because you MS guys need new blood and a fresh perspective. You need some 'No Guys' on your team.

Here is a shocker. Take what you believe and turn it upside down. Let me help twist an idea or message that seems constant in the blogs and talks of the past 10 months. As painful as it may be to read, don't get all worried about these next few paragraphs. Instead, digest it and gain some nourishment from it. Toss out the waste. Just think about it and use it as a means to generate more ideas.

****** Random thoughts *****

To be blunt, some people laugh at the constant requests for help. Who are you kidding? Microsoft has billions. It doesn't play well with some people and you need another tactic, a positive message. You look weak. Plus, you don't need an evangelist. You need a person to lead your efforts because if you had a leader then you wouldn't he asking the same questions for 2 years!

Let's expand this and really get some heat going. Why doesn't everyone at MS use a Tablet PC? What message does that send? Maybe no one there believes in it. This idea is fed more when the request for help is seen constantly. If that isn't the case then prove it in action and not words.

One more twist, if you are really helpless then Microsoft is in deep trouble.

Business is really quite easy. Get product into the market. Kill the whining and talk to the manufacturers. Give them incentives. Make deals. Push the product by bundling deals. You figure it out. It is your job. If you can't do the job then quit.

*****

I know that seems hard to face but it is vital to a viable business to consider all angles. Move forward. Take these words and twist them. Work with them to develop a positive message. Advertise Your Weakness as if it is a Strength. Now go get 'em tiger.

Enough Said.

The reason I love this comment is that Layne is speaking his mind without the any intentional malice etc. And he points to a couple of very good insights and a few that I think are somewhat misinformed. First of all, the Tablet PC team as far as I know (remember I don't really have any day to day insights into that team) does have a team of people who are responsible for marketing the product, but this is in a very traditional marketing role or at least as traditional as it can be for a software company. The people in that group don't blog as far as I know and I don't think that they're responsible for “community efforts” other than events. I'm not really sure what their exact charter is, but fostering an individual level community as what happens here doesn't appear to be in center of their focus. I have a lot of respect for that team, it's just that they have some very specific tasks and goals to accomplish and this may not be among them.

That said, a leader for outreach communities would be great. That means that someone would have that goal as a specific goal to accomplish and would have some resources available to do it [1]. And of course MS has so much money in the bank that we should be able to hire someone to do that, right? Wrong. Product teams at Microsoft are traditionally understaffed. I don't think I've ever worked on a team where we had more people than we need or even simply the right number of people. Often times there have been critical things that we haven't been able to research or test simply because we didn't have enough people to do what would be considered the appropriate course of action. So there are times when I've worked extremely long hours or weeks on end just to stay up on the things that we could do. So couldn't we just use some money to hire a vendor or contractor to do it? Nope. While in the past that has been possible, the company is really trying to tighten it's belt and be more fiscally responsible to it's shareholders. That means that getting that extra cash to get individuals to do outside work is just that much harder. I've talked about field trials in past threads, but never how much it cost to do these. Believe me, it's a lot of money. Right now if I wanted to do another field trial this year for a particular purpose, I wouldn't be able to do something that I'd be proud to blog about. It would have to be something that's extremely scaled down and cut every conceivable corner even to just consider getting the funding. Dollars are that tight. Plus would you really want a contractor or outside vendor running your community relationships in this kind of way? I think you'd want someone internally to do it. Also as an aside let's not forget microsoft.com/tabletpc and all the MVPs, they're all doing a great job -- there's definitely a community out there, but not necessarily the Utopian vibrant community that some envision. [1] And then there are paid evangelists for Windows, maybe they should own the community effort here? Any takers :-)

Is Microsoft helpless as Layne indicates we might be given what I've just said? Maybe. At the fundamental level each product group at Microsoft is like a small start up company, clawing it's way for existence in a larger corporation and begging like crazy for a small hand-out from the two giant mainstays in the company. Is this a healthy ecosystem from an industrial organizational stand-point? No. But at the same time the senior leadership a Microsoft recognize this and other issues that I've already mentioned and they're actively working hard to help us to reshape our own culture and processes in order to transform the company in some fundamental ways.

Why doesn't everyone at Microsoft use a tablet? Hmm.. good question. If you want to order an approved laptop class computer at Microsoft, it's a tablet. So if you're getting something new you have to go out of your way to get something that isn't a tablet. But there are also plenty of people who need something different than either because they're evaluating something or because they have a need that the tablet doesn't address yet. Are there people who are down on tablets at Microsoft? Sure. Are there people who are in love with Linux at Microsoft? Sure. We're a bunch of individuals and everyone has their own opinions, and negative impressions and opinions get magnified a lot more than all the positive ones.

So at the end of the day, what does this mean. To me it means that the Tablet team needs to make sure it produces something really cool and usable for Longhorn as well as working on how to sell in more Tablets in the meantime. In addition, maybe they need to utilize all the passionate people out there in the community better and task someone to become the community liaison that puts together a coherent strategy in order to accomplish that.

Comments are welcome, but I'm moving on to another topic next... “How to do user research on products that might make the leap from niche to mainstream?” But this will be several days out as I'm heading with the family on vacation.

Tablet Evangelist?

EvanF — Sat, 17 Jul 2004 14:41:00 GMT

I think some people out there assume that I'm a Tablet Evangelist for Microsoft. After all, I was one of the founding members of the Tablet team, I've written about the Field Trials, I've given some insight into the tablet, and my name starts and is completely contained in the word evangelist. But all that said, I'm just an ordinary guy blogging in his free time on subjects that are near and dear to my heart. My blog is not a reflection of Microsoft or it's opinions (and all that legal jazz). No one at work is asking me to blog, though I have been asked not to blog (and asked why I would do such a thing in general). My performance review (which is coming up) is in no way tied to me blogging and who knows I might actually get a slightly worse review due to blogging. I'm not part of the Tablet PC team (though sit close organizationally). I also mentioned in a previous post that my primary machine isn't even a tablet and I alluded to the fact that I don't necessarily use one on a regularly basis currently.

If that makes me an evangelist, I'd have to say that I'm not a particularly good one or even a model for others out there! Okay but here's the real deal, several people have complained (or maybe it's just one) about the fact that the Tablet team doesn't have an evangelist or doesn't spend the time to evangelize on-line in forums or blogging etc. That all the Microsoft blogs related to tablets that are “relevant“ (I guess I'm not relevant) are relatively stale. That some of the MVPs out there are getting tired of the Tablet gig. That some of the influential community members are ready to call it quits. That the community in general doesn't feel much love from the Tablet team. That the power toys are a joke... and it goes on and on..

Cards on the table (or maybe tablet), I have no real insight into the overall plans of the Tablet team to address this or any issues, but I'm pretty sure that the overall resources available to address any of these is either non-existence or in high demand for other activities. So if I told you that there was say “no money“ and “very little time“ available, what would you want of the Tablet team? I know at least a couple of them read my blog, so why not post your reasonable desires here for building or injecting some more life into the community and maybe someone will do something about it?

I can't and won't promise anything, because as I've said, I'm not an official tablet evangelist (and personally I don't think I'm really one unofficially), so why not start a constructive conversation about “user/community needs” and see where it goes?

Field Trials Part 3.1

EvanF — Fri, 09 Jul 2004 17:10:00 GMT

No this isn't a Windows versioning scheme :-)

Rather I'm using this opportunity to answer some questions:

Lora writes:

Did they go back and field test the revised Journal?

Yes. Next Question.

How much did the concurrent changes in the industry impact the design? Did the elapsed time between the first trial and released product impact how people used the product and their expectations also?

Also Yes.

Okay now for the non-flip analysis/answers. We did in fact conduct 2 additional field trials after the one that I've been blogging about. Both of these were run by my direct reports at the time. One looked specifically at the initial experience and the changes in Journal. The other looked at pre-production real OEM hardware (a convertible) and the effect that Tablet had on the institutional culture and usage over a 9 month - 1 year period. Maybe I'll blog a little about these in the future, but my personal knowledge was quite a bit less since I was only managing the people running the project and not supervising the day to day workings of the projects.

During the development of the Tablet, the mainstreaming of 802.11b had a huge impact on the adoption and reasons that people liked the tablet so much. When Chuck spec'ed out the tablet designs for the prototype we didn't include wireless on board. At Microsoft wireless was just being rolled out selectively in some buildings. At the time we hadn't really thought about the impact. When the first protos came off the line, some of us stuck in 802.11b PC cards into them to access the networks at work or at home, but even that was rare. When we got the units up to the ability to take them to the field trials, one of the sites that we deployed out was playing with wireless in one conference room. We didn't initially give them a wireless card, but the users who used the conference room definitely wanted to try it out and with the help of their IT staff and our infrastructure, we got them up and running. This made a huge difference for some of the users as now they were more likely to take the tablet with them.

We also saw some similar things during later trials and from other data points, that the confluence of wireless with the Tablet was a good thing since many of the early tablet interested people thought that wireless was the reason to have a tablet since it was a better laptop to take with them to different locations. This is very different from those of you who say “the pen” is the reason to use a tablet.

So at the end of the day, having a tablet be wirelessly enabled out of the box just makes a whole lot of sense particularly given that the wireless infrastructure is still booming.

What other questions or other things about the field trials do you want to hear?

Field Trials -- Part 3

EvanF — Thu, 08 Jul 2004 20:20:00 GMT

Car ride home

Charlton Lui was the development manager for the Tablet PC at the time that we did the major field trials. Charlton is a great guy full of a lot of energy and a way of rallying the troops to get things done. It’s probably his really bad sense of humor that he trots out in all situations, but regardless of that fact, I really do like Charlton. Charlton didn’t come out to the field trials until one of the last several weeks. By that time we had something in the neighborhood of 30 or 40% of the team observing and actively shadowing the participants. As I hadn’t been spending much time back in Redmond, I only knew threw the grapevine that a lot of discontent over the direction of the tablet was starting to ferment. Most of this was since many people had seen the field trial participants and were hanging their heads low as to what we were seeing since it wasn’t nearly the rosy optimistic picture of tablet usage that we had been living in our self-enclosed cave at Microsoft. On the way back from a full day of observation I was driving the car and I broached to Charlton that the prototype of what was to become Journal just wasn’t working. Charlton just didn’t want to hear it since it was his “baby”. In fact Charlton had previously had a software company that Microsoft acquired that was specifically based on a pen based note taking application. The heart of the software at this point in time was code re-written and ported based on a lot of the original ideas in that software. This was where many of the basics that created the “word processing” came from. We’re lucky that we didn’t have an accident in the car as it was a pretty heated discussion with lots of discussions about all the usability studies that had been done on the old software and how they had done lots of research and seen that people wanted this functionality etc. It’s not that he was necessarily wrong, but the first priority that we saw from absolutely everybody was that they want to take notes as they take notes on a piece of paper. Once they’re comfortable with being able to do that, then maybe just maybe, they might be more comfortable with something advanced (or at least they aspire for some of the advanced functionality, but don’t necessarily understand the trade-off in the inherent complexity that accompanies it). Charlton just didn’t want to hear that I was suggesting that we pretty much scrap some of the underlying principles in the software and start over again.

This is a pretty common reaction that I’ve seen especially in the software world. Many people whether they are developers, program managers or even usability specialists are not willing to go back to the basics and throw it all away if they think that the model is wrong. Most times people try to put on patches on top from additional functionality to give a different entry point or more and more help topics to try to address these concerns. It takes a lot to let go of your “baby”, but I think that if people did it more often than we’d get better products all the time.

Breakfast

It couldn’t have been too many days later when I arranged to have breakfast with Steve Weil when he came out to observe. Steve was one of the program manager on the tablet group, I don’t remember if he was a lead at the time or not, but basically he was someone who through interactions before I recognized as more of a bigger dreamer. He sees a lofty vision for some product and works as hard as he can to obtain it even if it seems at odds with other goals on the table. So while we’re at the breakfast table I talk to him about the conversation with Charlton plus what I think about the fundamental change that needs to happen for Journal – we need to get rid of the dependency on “lines”. Making this change was pretty radical since it would really require a complete overhaul to the product since absolutely everything in it was based on associating text with the closest line. We talked through a lot of what we saw and how the product would be different. I knew in talking to him that he was game because he recognized that this was the proper thing to do. It was just a matter of figuring out how to get other people on board with it. We started drafting through the different things that needed to happen, how the interaction model would work and what the basic functionality would be for the product. At that point in time things started to really click… Alex Loeb who was the VP (well she wasn’t a VP yet) also had another product that she was responsible for which was a product called MODI (Microsoft Office Document Imaging). This product is all about scanning in documents and letting the text of the document be searchable in the background so the document can be found rather than just OCR’ing the document and dumping it into a Word doc. Maybe I’ll talk more about this product at another time as I also worked on this in detail, but for now Steve was realizing that if they could basically grafted the inking capabilities on top of the MODI application and then started to add back in some of the other note taking functionality. So while I was wrapping up the field trials, he was actively lobbying and working with folks to change the picture of the world for the tablet team.

Wrap up

The last day of observations for the field trial was a Thursday. I flew home late that evening and dragged myself into work the following day to spend some time in the office for the first real time in months. There was lots of commotion going on at the time as we were trying to figure out how to cope with all the information we had gotten from the trials and how we were going to change the plan of record and what it meant for many on the team. Alex had likened the experience to shipping a V1 product and getting the feedback that it wasn’t right so that you could start over to ship the right thing in V2. It really was a hard time for the team in general since many were trying to cope with the fact that we didn’t build the right thing the first time around, but there was definitely an upbeat attitude amongst many since now we felt that we had good, grounded insight into what would work.

Sometime during the afternoon of that Friday, Alex stopped by my office to talk. She wanted to know how things were going and what some of the upcoming plans were. But more importantly she wanted to ask if I could write up something brief for a BillG thinkweek paper. You can all do your own google search on what these are, but basically he spends a week reading important topics from around the company. I said sure, but then she told me that the papers had to be in on Monday! Here was one of the more important documents that I would be writing at Microsoft and I only had the weekend to do it! I’ve written content in other BillG thinkweek papers and reviewed many as well. Often times there are weeks of writing and back and forth discussions on the team about the content etc. So I go off to start writing the document like a madman on Friday night and well into Saturday when I have something that I think is a good draft, but then the unthinkable happens…. my hard drive fails, the machine crashes, I lost everything. I was just about to call Alex and let her know that I just can’t put anything together when I realize that this is a big opportunity so I started over again, trying to recall from memory what I wrote and in the process create an even better document. In retrospect I think losing the original helped me to write a better document the second time around. So late Sunday afternoon I sent the document out for review.

It was interesting in that those who had been to the trials agreed with everything that I had written and really had no substantive comments to add. But the Architects or the “BBC” as we liked to call them (Burt, Butler and Chuck) were not quite bought in nor was Charlton as to the conclusions that I was making. Alex looked it over and we got it over to our resident editor and we made a few quick copy edits and it was sent off to BillG.

There were a couple of unusual things about the think week paper. First off it was around 10 pages in length when Alex was expecting a 2 page executive type summary for Bill rather than some in-depth analysis on some of the major topics. The second thing was that as far as we knew this was the first “user research” document that anyone had given Bill to read as a think week paper. Bill in the past has been semi-notorious for not necessarily agreeing with the user research particularly when it’s research on a product or a vision that has yet to be released or widely available. Thus he was definitely a skeptic and given his passion for tablets (in the first year of the team’s existence we meet with him monthly and exchanged lots of email – so he was like a member of the team) he might also discount the findings.

Well in the end, it turns out that Bill didn’t dismiss the findings, in fact he quoted some of the findings in other meetings and when we met with him after think week he asked for some further follow-ups to some of the information etc. It was a great victory as he saw the value of the project, the value of the results and fully supported the team in making the major overhaul that was required.

So we ended this trial and started planning how we could get more great information like we observed during this time….

Tablet Field Trials

EvanF — Tue, 29 Jun 2004 08:14:00 GMT

I’ve hinted around at what we called the Field Trials in Tablet before and I know that Chris Pratley talked about Field Trials in his work with One Note, but unless you happen to read the HCI (Human Computer Interaction) literature, you might not know what it was that the tablet team did and why it was so important and novel inside of Microsoft.

So what is a Field Trial?
Basically it’s an opportunity to get real world feedback on your product by getting it to a state that a small group of users can use it for a prolonged period of time. In the case of the tablet (when I was still working on it), we ran a 2 week trial, a 5 week trial, a 8 week trial and a 9 month trial.

Before I go too much further… How is a Field Trial different than a Beta?
Well at Microsoft a “Beta” is more about finding bugs in the process as well as getting customer feedback, it’s also usually at a point just before making a release candidate which means that if there are major changes needed in the overall user model, they generally don’t get made or they wait for the next version. Also betas are generally huge with hundreds or hundreds of thousands of people participating in the process; whereas a field trial is relatively small since you’re going to spend a lot of time with each of the individuals participating.

Okay tell me more… Some background
When I first joined the Tablet team, I pitched to the team (all 5 of us) that in order for us to really know if we’re building the right product that we have to get it out into the users hands early enough in the process for us to understand how they want to use it (the plan for the 5 week trial). The way I proposed to do this was to give some users prototype tablets and follow them around so that we can understand how it’s being used, what things are still needed and where usability issues exist. At this same time, Chuck Thacker was busy sketching the schematics for prototypes and I was beginning to make plans for how to take advantage of the fact that we would be building full functional PCs with integrated digitizers without a keyboard and the potential for several buttons. Surprisingly Chuck pushed through the engineering relatively quickly including utilizing a super-secret start-ups new processor (Transmeta) who hadn’t officially announced anything at the time. However actually having these built and debugged took a lot longer than expected but that was a good thing since the logistics of doing a field trial are immense.

Tablets
Chuck had a run of 35 initial tablets built for the team. At that time there were probably around 50 people who were involved with the tablet, but of those 35 units, 30 of them were dedicated for the field trial which meant that the developers who were busy writing prototype code and putting together features and other influentials around the company had to fight over the 5 other tablets that were in circulation. In retrospect it was funny since many of the team members who eventually came out to the field trials had not had any direct experience with one of the working tablets. Of course these devices were extremely fragile as well as the software that we developed and as such we wanted to have spares on hand in case anything failed.

Given delays in getting the hardware we performed an initial trial about 5 months before the first main one with some very limited hardware from a company that we acquired. Wish I could talk more about the company and the hardware but suffice it to say it wasn’t a full-pc but rather a CE style tablet. The main purpose at this time was to understand if users really needed Windows and also for us to get some experience at piloting the methodology and refining it so that when we were ready to roll out the more extensive trial we’d be ready. This one lasted 2 weeks with mixed reactions, some users really like the functionality but wanted XYZ (name your favorite Windows application) to be included while some of the other users had a really hard time just even getting started with the device. We learned quite a bit about the methodology, infrastructure needs, support and how to get buy-in from participants and the corporations they work for.

Corporations
We ran the field trials in a large mid-western city during the winter months of 2001. We wanted to have at least 5 participants from 4 different large companies or corporations involved with the study. The main reason was to have a wide variety of “horizontal” knowledge workers who performed various different tasks in different ways at these different companies. This way we could have a small sample of users who fit this idea of a “meeting go-er” who was central to the Tablet software. So for many months I fly out to this city and had meetings with the IT staff at these companies since we knew that to get into the site we were going to need cooperation from the IT department as well as senior management. Once we got the IT folks interested, then it was on to selling it up the chain and getting them all interested. Unfortunately we had lots of false starts as we’d show the concept to the IT folks using one of the very early prototypes (there was only 2 or 3 of them available at that time) and they’d be really interested and the CEO or CIO thought it would be a distraction. Some times we had the CEO wanting to participate and be one of the people we’d follow around, but the IT folks would simply just dig in their heals and not let us put the equipment on their network. So we went around and around and finally ended up at 3 different companies from which we recruited 21 people.

Users
Once we had the companies, then we went on to start screening users. As you might imagine the IT staffs volunteered themselves as the first candidates and we had to work pretty hard to either get to the fringe of their organizations or get to some of the users that they supported whom they felt would be amenable. This was a pretty elaborate process to as we wanted to interview the people and “shadow” them (follow them around for part of their day) to make sure that these people were relatively “normal” and fit our profile. Almost all of the people we choose were not early adopters of technology nor were these people gadget freaks – we tried to screen them out so as to get those people whom in 2 or 3 years their company would deploy tablets to them as standard gear since it was a tool that would make them more productive, etc. We ended up with a great selection of people ranging from a low level analyst to a couple of corporate executives.

Over the course of the next 5 weeks we interviewed the users, followed them around for parts of the day, send them questionnaires to fill out and had them perform some exercises to assess how much they’ve learned or used of the device. This was a lot of fun as we had multiple teams out in the field and every day we would spend between 8 and 12 hours following around 2-3 different users. At night the teams would get together and we’d all discuss what we had seen, go out for a late dinner and then crash to start it again the next day. Some of the users called us their “entourage” or “posse” as we often had 2 or 3 people following them around as they went about their activity with video and digital cameras. It was quite an experience as almost every meeting was interrupted by at least 10 minutes to either explain who we (the observers) were and/or what the cool device they had was and what it could do. Even when were not there, we often had the user relate about how many demos they had to give during the day and how it was interfering with the “work” that they had to get done. To us they were the “celebs” and we were the paparazzi taking notes and photos of absolutely everything they were doing.

This process went on for 5 weeks which I spent living in a hotel and flying in different team members to observer Mon-Wed and Wed-Fri. I’m still upset about the Business Week article that got it wrong… it said I missed my daughter’s 7^th birthday when in fact I missed her 2^nd birthday, but we did have a party on the weekend. I flew back to Seattle on the weekends or down to San Antonio where my wife and daughter were spending the time with her parents while I was clearly occupied. This was to me one of the most exciting projects I’ve ever done.

Infrastructure
Performing a study like this is no simple matter, we had to insure the IT department that the devices would be compatible on their network, not cause any issues, that their main applications would work, that we’d provide unparalleled support for the users if anything went wrong, etc. In doing this we ended up hiring our own tech support guy for the area who had the job of making sure that the user’s machines were up and running at all times. He was on call 24/7 and we set the goal of less than 2 hours for the user to up and running. Thus having extra units helped since he was able to remove the hard disk and place in the other units; as well as having extra hard drives with the company’s image, providing regular backup service to all the users so that the data could be restored quickly on a backup etc. And his job was pretty rough since I’d be out in the field every day and I think I ended up running into him almost every day while he was fixing someone’s machine. Like I said these were prototype units with prototype software.

And then there was the product team with everyone triaging bugs and making sure that we had the right functionality for the users actually in the build and working on the prototypes. The entire tablet team deserves a lot of credit for pulling this all together, but the testers on the team a great deal of gratitude as they were the ones that were constantly banging on the product and then working with the developers late into the night to get fixes not only before we started but all throughout the trials when we’d find things breaking left and right.

In addition we hired an outside firm to help manage some of the logistics and do some of the reporting of the user data. This way in addition to my skills and those of some of my team, we were able to have multiple sets of eyes out there that are trained in psychology, user research, usability, etc. Thus we could really follow around multiple participants and have a large sample set than if it was just me.

Stay tuned for part 2….