Software Testing Cage Match: Amazon.com vs. Microsoft

re: Software Testing Cage Match: Amazon.com vs. Microsoft

Seth Eliot — Fri, 14 May 2010 21:23:20 GMT

Great point Ralph.

I think it can be summarized as risk assessment of bug impact.

Hard to detect bugs are higher risk. Bugs without work-arounds are higher risk.

Also systems like banking, aerospace, medical are less tolerant to defects in production, even if you take steps to limit the scope of these defects (such as by using exposure control when TiP).

re: Software Testing Cage Match: Amazon.com vs. Microsoft

ralphcase — Fri, 14 May 2010 21:16:23 GMT

Hi Seth,

I think there are other factors that are important in decided the optimal level of test investment.

You discussed how easy it is to fix a bug, and how that can be quite different depending on the type of product.

Another important question is how easy it is to detect a bug. If a bug causes a server to crash, that's easily detected and reported by monitoring systems. If a bug causes the wrong amount to be deducted from a bank balance, can you detect that in production, or do you need to develop specific test cases to detect such bugs before release?

Yet another variable is how the system behaves when a bug hits. If the effect is that a web page does not display correctly, but when the user refreshes the page it does, that bug is much easier to tolerate in production than a bug that corrupts the database.

If a system is designed from the start to tolerate certain kinds of failures or bugs, that can have a big impact on the ultimate test budget to ensure the system is meeting its requirements.

Ralph Case

re: Software Testing Cage Match: Amazon.com vs. Microsoft

nitinmehra20 — Thu, 29 Apr 2010 04:10:17 GMT

I agree with what Seth has mentioned, our deployment setup is both a boon and a curse. This culture of roll-back-if-it-crashes has been partly fueled by the ease with which we can roll back, fix and redeploy code. Other than the operational cost, their is very little cost involved in a rollback. Thankfully, this is changing in some teams.

This is specially true with regards to Kindle and the SDK teams. We now find ourselves in the same situation as Microsoft, as in we too now own "software in a box". Once released, it is difficult, not to mention prohibitively costly to fix bugs on the Kindle and its associated software development kit.

Though I feel we still need to go some distance to create QA teams which have enough strength in numbers to meet this paradigm shift head on. We need to staff for a "shippable" code base as opposed to a "deployable" one.

re: Software Testing Cage Match: Amazon.com vs. Microsoft

Seth Eliot — Thu, 29 Apr 2010 00:07:37 GMT

Hey Rob, Thanks for your comments.

I think as someone who has been an SDET at both Microsoft and Amazon your input carries a lot of weight here.

re: Software Testing Cage Match: Amazon.com vs. Microsoft

Rob Van Wicklen — Wed, 28 Apr 2010 21:28:33 GMT

Seth, this is a good post. Understanding the cost of fixing bugs is essential to understanding the differences in the way Amazon and Microsoft approach QA.

I like to say that the “pager culture” at Amazon is a big reason why Amazon doesn’t invest more in QA. Developers at Amazon know their pager may go off in the middle of the night and they’ll have to diagnose a problem and deploy a fix. There’s a cost to this, but it doesn’t compare to the cost of patching those boxed software products that Microsoft is famous for.

With “software in a box”, the cost to fix bugs is very large. Fixing a bug means pushing updated bits out to all of the clients running the software. That’s why Microsoft spends so much on QA for Windows… and Windows Update still sends a bunch of hotfixes to my computer every month! When you consider how many computers in the world are running Windows, it’s easy to see how each one of those fixes costs Microsoft a lot of money. Hence, the choice of how much QA to employ comes down to a business decision of what costs less.

With “software as a service”, instead of having each of your customers install your software on their computers, the bulk of your software stays with you and it’s much easier for you to control. To fix a bug, you simply patch your servers and you’re done. The cost is much less than sending a fix out to each client. The result is that at the end of the day, bugs are more tolerable in Amazon’s culture than they are in Microsoft’s culture. Ouch! That was painful to say, but it’s true.

Rob