I grew up with “Schoolhouse Rock,” a series of short educational cartoons that played between the normal Saturday morning ones.  They were cute, set to catchy music, and taught us things like grammar and history, etc.   One of my favorites, “I’m Just A Bill,” gave a somewhat sanitized view of how laws get created in the United States.
I’ve been seeing a number of comments lately that basically say, “Here’s a feature request.  When are you going to have it done?”   Thinking back to “I’m Just A Bill,” I decided to do a blog entry that explains how features get done here in the Windows Mobile group.  It’ll be kind of a modern Schoolhouse Rock piece, just without the nice music or the clever cartoons or … well, okay, it won’t be anything at all like that.  But, while not as entertaining, hopefully this will be similarly educational.

The Easy Way

In my spare time, I write a lot of little applications for my Windows Mobile devices.  For instance, my son was learning his multiplication tables, so I wrote a Math Quiz app for him and handed my Smartphone over whenever we had time to kill.  Part of the fun of being able to write apps for your phone is that you can say, “I wish I had this,” and then make it happen.  But the real fun is that writing quick little apps is nothing at all like real software development.  I banged out a fully functional MathQuiz app in an hour and a half, tested it a bit, and then gave it to him to use.  He suggested some things (you haven’t lived until you’ve taken feature requests from your eight year old), I spent another hour adding the things he wanted, and that was that. 

Unfortunately, software development is rarely like this.  The only reason I could get away with such a pain free development cycle is that my target audience was extremely small.  Windows Mobile, on the other hand, has millions and millions of users.  Developing something for one or two people is easy.  Developing something for millions of people never is.

“What” is harder than “How”

Since 2000 we’ve shipped six major releases and countless minor ones.  Here’s a newsflash for you:  there will be a Windows Mobile 6.  And a 7.  And an 8.  And many more after that (who knows if marketing will actually call them those names though).  I can say this with some certainty because, 1) We’ve been very successful and 2) We have so many features on our plates that we’re not going to run out of things to do anytime in the recognizable future.

When we start a project, the first thing we have to decide is what features we’re going to do.  Like I said, this usually of the form “of these 1000 great features, which 100 can we reasonably get done in time?”  Having so many users, each with their own passionate opinions on what’s important, many of them conflicting, makes this a tough choice.  Even if we only had a million users (we’ve got many more than that), if we chose something that 99% of them wanted instead of something that 1% wanted, we’d still annoy 10,000 people.  Remember this when you say, “All of my friends want this too.”  Even if you personally know ten thousand people and they all agree with you, that’s still a tiny percentage of the user base.

But it’s even harder than that.  Sometimes it’s not just “the people who didn’t get this feature will be disappointed.”  Some features are great for most of our users, but make things appreciably worse for some of them.  We’ve had some features that were absolutely critical for half our users, beneficial for forty percent of them, unimportant but not bad for five percent, a net negative for four percent, and absolutely unacceptable for one percent.  On percentages, that would seem like an easy choice, until you have to sit in a room and decide to literally make tens of thousands of users furious at you.

Even the “easy” decisions aren’t.  Mobile Operators wield an enormous amount of power in the phone space.  All they have to say is, “We won’t sell your phones unless you do this,” and the decision is made for us.  But, some of those required changes have made end users angry at us.  Even when we know the change is going to be unpopular, if we tell the Mobile Operator that and they say “tough, do it anyway,” we have to do it.  If we can’t sell our product, we don’t have a product. 

But then it’s easy, right?

Okay, so our friend the feature made it through the gauntlet and has been approved.  Now we just need to write the code and release it, right?  Unfortunately, no, it’s not easy.  Even when the code itself is relatively easy to write, there are a lot of things that make it complicated. 

First, we ship Windows Mobile in around 25 languages.  So, if the feature displays any words, those words need to be translated a few dozen times.  Worse, words in some languages are much longer than words in others.  This means that what fits on the screen in one language might not fit in another.  So it’s not just translating sentences.  It’s also laying the UI out over and over again. 

Both the code and the languages are also made complicated by the wide range of devices we have.  It generally needs to work on both a PocketPC with a touch screen and a Smartphone that doesn’t have one.  It needs to work if you have a lot of buttons or just a DPAD.  It needs to work at 640x480, 480x640, 320x240, 240x320, 480x480, 240x240, and 176x220.  And remember that you need to multiply each of those screen sizes by the number of languages.

Then there’s the security review.  In the good old days, users tried to use their devices in the ways they were intended to be used.  If there was a place to enter the number of dogs you own, we expected you to enter 0 or maybe a number between 1 and 10.  To be safe, we might have covered up to 1000 knowing that almost no one owns 1000 dogs.  Little did we know that some aspiring spammer would find a way to say that he has 4294967295 dogs, and that, due to a subtle bit of programming, that would cause the software to misbehave in a way that enabled him to steal money.  We no longer get to write software that just does what the users need it to do.  We now need to write software that does what users need and is hardened against known and unknown attacks by malicious people who will spend years looking for new ways to beat it.  An attack similar to the four billion dogs one once shut down an airport.  No matter how much users want the feature, security isn’t something we can ignore.

Well, at least you don’t have to test it…

A frequent dig is, “This code was rushed to market and no one bothered to test it.”  Our intrepid friend the feature will tell you that this isn’t true.  We spend a lot more time testing our code than we do writing it.  (The number of features we can do in a release is almost always decided by how many we can effectively test, not by how many we can code.)   A feature will usually have a group of testers specifically assigned to it.  Then, after a week or so of dedicated testing, we’ll do “integration testing” to see how the feature impacts other parts of the system.
After focused testing, the feature is put into a nightly build so that everyone on the team can “dogfood” it.  We have a concept here we call “eating our own dogfood.”  Basically, this means that, throughout the development of the product, all of us are using it on our own phones.   We generally will not ship a product that didn’t have significant dogfood exposure (minimum hundreds of users for multiple months).  When we’ve finished coding all the new features, we’ll spend a couple of months doing both focused testing and dogfood.  We also hold ourselves to doing testing and dogfood of bug fixes.  In general, the final release comes a week after the last bug was fixed, and generally three to four months after the final feature was added.

But then it’s done, right?

No.  Our product is not a phone.  It’s an OS and applications that we ship to our OEM partners.  It’s the OEM’s product, built upon ours, that’s a phone.  The OEMs design and build hardware to use our software, and then they write a ton of their own software for the device.   Often, less than half of the code on Windows Mobile device was written by Microsoft.  We give OEMs early drops of our part so that they can work somewhat in parallel with us, but there will always be a lag between when we finish and when they are able to ship their devices.

When the OEM is finished, they then need to take the device to Mobile Operators for acceptance testing.  They need to do this for every release of every device, need to do it individually for each MO, and need to wait for an available spot in each MO’s busy testing schedule.  Once acceptance testing starts, it tends to take multiple weeks to complete.

The total amount of time that passes between when we release our product and when the finished phone becomes available from the MO varies with a lot of factors.  If it’s a small update to an existing piece of hardware on a phone that the MO is already selling, things go much more quickly than if it’s a major release on a new piece of hardware the MO has never seen before.  I’ve seen this take as little as a month an as much as six. 

And then it’s done

This is the point in the “I’m Just A Bill” cartoon where the senator runs out and exclaims, “He signed ya, Bill.  You’re a law!”  Once the Mobile Operator accepts the phone, it shows up in stores a little while later.  Sometimes there’s a bit of delay for a marketing campaign or to get a bunch of units manufactured, but it happens pretty quickly after that.   

In a nutshell (okay, a really long and verbose nutshell) that’s how ideas become features here in Windows Mobile. 

Mike Calligaro