For many years now I've been frustrated by industry development practices that approach software performance in an ad hoc manner rather than as a design issue. Every time I argue in favor of considering a feature’s potential performance issues up front rather than after the fact, the response is consistently one or more of the following:
- Moore's law will solve most performance issues; its not worth our time
- All performance problems are bugs; find and fix them at the end
Or, “premature optimization is the root of all evil” (Hoare’s dictum)
- The performance trade-offs are worth the gains
If you’re a programmer, you’ve no doubt heard these arguments before. They are especially engrained in the culture of commercial software development where performance need only be “good enough”; where time-to-market and new feature development take priority over all else.
In my early career at Microsoft, I joined a team building a WYSIWYG editor designed for programming. The concept was very cool, but the performance of the early prototype was so atrocious that it was practically unusable. In early 2000, a few of us on the team suggested that we needed a fundamental redesign to address performance issues. The immediate response from the “architects” was brutal. We were called myopic for not recognizing that Moore’s law would more efficiently solve any performance problems we could fix. They insisted that the performance tradeoffs of their design were worth the benefits they offered. And they harshly admonished us for not recognizing that performance problems are “bugs” to be discovered and dealt with at the end of a product cycle. Naturally I left the team within weeks of that meeting.
By the end of 2000, after a nearly a decade of very expensive research and development, the project was cancelled owing to a failure to demonstrate a usable prototype. Performance was important after all, and while I lamented the death of a promising idea, I must admit to a little schadenfreude over its demise.
Customers care about performance. They care a lot about performance, and consequently we developers should care about it too, and we should do so from the start. There are critical problems with the three common arguments against an a priori approach to developing efficient software.
Moore’s Law No Longer to the Rescue
In the 1990s, successful development houses typically didn’t invest much design time on performance improvements because Moore’s law often rendered such efforts superfluous. In the early 90s I was on a development team that spent months improving the performance of our software. By the time we finished, CPUs had more than doubled in speed and we lost out to competitors who had instead focused their efforts on additional features. That lesson was hard learned by many in the field, and it accounts for why it’s been so difficult to unlearn.
The problem with depending on Moore’s law is that in recent years we’ve not seen the speed doubling effects of circuit density improvements that we saw in the nineties. Circuit density continues to double every year and a half, or so, but the concomitant growth of CPU speed no longer seems to hold[1]. We can no longer rely on processor improvements to make up for our poor design choices. Moore’s law can no longer rescue us from bad design.
Hoare’s Dictum Misconstrued
Tony Hoare’s dictum, “premature optimization is the root of all evil”, has become the excuse of choice for putting off optimization until the end of a product cycle. To many developers, the dictum has an almost scriptural quality to it, with the power to cut off debate whenever it’s quoted. Unfortunately it’s been misconstrued.
The dictum was popularized by Donald Knuth’s 1974 article “Structured Programming with Goto Statements”[2] in which he explored the use of structured programming to improve program readability and correctness. Although the Böhm-Jacopini theorem had previously proven the equivalence of structured programs, in 1974 most programmers still eschewed structured methods because of perceived performance costs. Knuth’s study showed that in most cases the benefits of readability and correctness outweigh the trivial performance gains of programming unstructured code.
The full version of Hoare’s dictum is "forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" and I generally agree with this. But 97% is not 100%, and 3% of code in a two-million line project is still about sixty thousand lines of code. It’s virtually impossible for a team, at the end of a product cycle, to identify and fix performance problems with sixty thousand lines of code without enormous effort and extensive redesigns. If they instead consider efficiency during early design and throughout coding, their efforts at the end of a product cycle can be focused on more productive tasks like fixing real bugs. Charles Cook, the redoubtable English blogger, put it most aptly:
“A good software developer will do this automatically, having developed a feel for where performance issues will cause problems. An inexperienced developer will not bother, misguidedly believing that a bit of fine tuning at a later stage will fix any problems.” [3]
Put another way, small inefficiencies can really add up and are costly to fix after the fact. In the mid-nineties I joined an enterprise software team that had developed an excellent product that was to save the company millions of dollars. Unfortunately its release was delayed because it was too slow on the computers already deployed in the field. I was hired, in part, to help identify and fix these problems. Upper management was convinced that there was something wrong with the team. After a few weeks of careful analysis, it became clear that the performance problems were NOT the result of algorithmic bottlenecks, poor programming, bad programmers, nor of bad overall design. Indeed, the code was some of the cleanest I’d ever seen, the algorithms solid and the team very competent and bright. The performance problems were instead caused by hundreds of small coding trade-offs, consciously made by developers to improve readability or structure. No single trade-off was measurably bad but in aggregate they created substantial performance problems. It took several months for the team to redesign and rework their individual feature areas to eliminate the overall performance problems.
A Trade-Off Is Not Always A Trade-Up
On large projects, I’ve noticed that every feature team thinks that their ideas are essential and that their particular performance tradeoffs are well worth the overall advantages they offer. While it’s often true that a single design choice may have only a very small cost, in aggregate all those little tradeoffs often add up to huge performance barriers. It’s critically important to measure the cost of each performance trade-off, not only relative to the feature itself but in the context of overall product execution speed. A single design choice may cost only a few thousand cycles every now and then, but a few hundred of these trade-offs can quickly compromise overall performance.
References
[1] http://wi-fizzle.com/compsci/cpu_speed_Page_2.png
[2] http://pplab.snu.ac.kr/courses/adv_pl05/papers/p261-knuth.pdf
[3] http://www.cookcomputing.com/blog/archives/000084.html
Randall Hyde wrote an excellent article for the ACM, titled “The Fallacy of Premature Optimization”. http://www.acm.org/ubiquity/views/v7i24_fallacy.html
I also offer the following as examples where the modern software design practice of putting off efficiency has resulted in very unhappy customers.
- Vista really is slow unless you throw a lot of hardware at it,
Vista: Slow and Dangerous, Business Week Mar 15th, 2007
- I have only one question for Bill Gates. Does he recall a book he wrote, called Business@the Speed of Thought. Because the title is suggestive, and if that’s what he indeed believes in, I would like to know whether or how Microsoft’s latest operating system “Vista” subscribes to this vision.
Why is Vista so slow on uptake?, Business Standard, July 17th 2007
- Vista is actually fairly bug free. Where it falls down is in the performance category.
Top 5 Things Microsoft Must Fix In Windows Vista In 2008,
Information Week, Nov 28th 2007
