Perception, Reality, & Incrementally Tuning World-Class Applications

Performance & the 20% Rule

I was reading the post from Ed Glas, Make Your Performance Work Count: The 20% Rule. 

He shared the words of computer science professor & billionaire entrepreneur David Cheriton:

If it is fast and ugly, they will use it and curse you;
if it is slow, they will not use it.

Indeed.

Ed provided this quote in the context of the book Designing and Engineering Time: The Psychology of Time Perception in Software by Steve Seow.  Steve is an expert on performance engineering at Microsoft and has a deep understanding of the human element and psychology behind software performance.  An intriguing take-away is that users do not notice changes in performance below a certain threshold.  Research has shown that for durations under 30 seconds, users do not detect differences up to 18%.  Steve's rule of thumb is that users really won't notice performance changes of +/- 20%.

Ed Glas is a big-time test manager for Microsoft. He relates that when setting goals for performance improvement, or considering performance against a competitive product, the threshold for success should be to gain at least 20% over the baseline measurement.  Likewise when considering whether to fix a performance regression, users likely won't detect the degradation less than 20%.  He suggests we’re better off spending effort elsewhere if the 20% threshold seems out of reach for a given area of testing.

Perception is Reality

I was struck by the (not uncommon) idea that Perception is Reality.  (This has been reinforced by my holiday reading of Stumbling on Happiness by Daniel Gilbert.)  So, to get ooh’s & ah’s from our users or customers—especially before we’ve gained their trust—we must either provide the 20% enhancements Mssrs. Glas & Seow discuss, or set the expectations low (which is sometimes appropriate but hard to do), &/or when incremental changes smaller than the 20% perception threshold we must provide appropriate metrics to document what we did.

My Microsoft Assessment, Consulting, & Engineering (A.C.E.) team & I often provide performance enhancements substantially larger than 20% in terms of all manner of metrics including duration.

World-Class Applications Are the Result of Incremental Improvements

Yet let’s not neglect that like many aspects of development, performance & scalability enhancements are incremental. Dittos for those changes which are significant in magnitude, yet don’t result in a significant performance enhancement visible to the user. For example, I recently documented how a simple configuration change resulted in an immediate, persistent 30% CPU enhancement and reduction in SQL Server wait stats of 78%. The customer was thrilled—yet the user experience was scarcely affected.

In addition, best practices often demand many things the transition to which may not in-&-of-themselves show obvious immediate benefits.  It is my position that incremental improvements in performance & scalability made possible by compliance with best practices provide a foundation fundamentally critical for success.

An analogy would be the forces binding the DNA helix.  Hydrogen bonding is one of the weakest molecular bonds in biology, any individual one of which can be cleaved quite easily.  However, billions of discrete H-bonds result in a virtually indivisible unit the cleavage of which requires specific enzymatic action. 

Likewise the result of multiple tuning iterations will result eventually in only incremental changes beyond the ability of humanoids to appreciate, yet the aggregate effect can be vital to an applications & organization’s success.

Why World-Class Applications Seek Incremental Improvement

A deck from CS193H High Performance Web Sites by Stanford Professor Steve Souders contains this information:

Of course, we go initially for the most bang for our buck, & pick off the low hanging fruit first (& other clichéd metaphors).

This data demonstrates that incremental improvements translate to big bucks in mature, world-class applications.

As an architect out in the real-world, I constantly strived to introduce best practices into our applications.  After several years on the job, incremental improvements were all we had left, yet it still made business sense to pursue them.

A proper, disciplined amalgamation of best practices can result in scalable, robust, performant, world-class applications—a classic example of the whole being greater than the sum of its parts (there's another one of those darned clichés).

Perception Problems are People Problems

So, it’s great to produce 20% & greater performance enhancements, yet such improvements can’t be sustained round after round after round of tuning. 

It is important—isn’t it always?—to perform appropriate expectations setting when only incremental changes are possible.  As Alik Levin shares with us, "No matter how it looks at first, it's always a people problem".

What’s your perception of implementing best practices?
What’s your perception of performance?
Tell me.

Administrivia

Jimmy May, MCDBA, MCSE, MCITP: DBA + DB Dev Senior Performance Consultant: SQL Server
A.C.E.: Assessment Consulting & Engineering Services
http://blogs.msdn.com/jimmymay
Performance is paramount: Asking users to wait is like asking them to leave.

This post was written with the PracticeThis.com plugin for Windows Live Writer