Motley says: "Performance stinks, but I'll deal with it later"
Summary
Motley: I'll deal with performance issues when I have something to test. Then I'll run the profiler and just optimize the slow methods.
Maven: Think about performance early in the development cycle. Set goals. Design for performance. Measure, measure, measure. Optimize the scenarios that matter.
______________________________
[Context: Motley is doing some performance testing and optimization on the feature he is building]
Motley: Hey, Mave. Check this out. I've been doing a bunch of performance tuning on this component. I've made these three methods twice as fast as they were before!
Maven: Cool! What tool are you using to measure how fast they are?
Motley: The built-in profiler in Visual Studio rocks. You can run a set of unit tests against your production code and VS will tell you what methods are your bottleneck. Based on that I go in and fix them. For ultimate flexibility, you can choose to either instrument your build with profiling prolog and epilog code in each method, or you can have the VS profiler take frequent samples to see how your application is behaving. I prefer instrumentation - it changes the binary but you get more accurate results.
Maven: Sweet. Why did you just jump to the methods that VS tells you were the slowest?
Motley: Because they're slowest! Duh. I'm not going to go spend valuable time optimizing the fast methods! Have you had your morning coffee today Mave?
Maven: The key is that just because a particular method is slower on a relative basis than other methods for a particular test scenario doesn't mean it's worthy of spending time optimizing. What if the scenario you are optimizing for is rarely used? Is it worth making the 5% of the code that is rarely executed twice as fast? What if the user already perceives that feature to be fast enough given today's hardware?
Motley: Reasonable questions, I guess. But faster is faster. It's never a waste of time to optimize a component.
Maven: With all due respect, Mot, I beg to differ. The time you spend optimizing some component that doesn't really need it from the user perspective is time you could have spent improving code quality, optimizing something that needed it, or building a new piece of functionality to delight the user.
Motley: Grrrr… you mean I am "generating waste", if we talk about being agile. I hate it when you're right.
Maven: Generating waste? Yeah, I didn't think of it that way, but exactly! Nice observation Mot!
Motley: Okay, I'll find out which components need optimizing first and then work on those.
Maven: Great. How are you going to determine which ones need optimization?
Motley: I'll… Hmmm… your telling me I can't just look at absolute numbers. Well… enlighten me Einstein. How am I going to know what to optimize???
Maven: There's nothing wrong with using the profiler for some scenarios as you mentioned previously. Profiling works well when you are executing end-to-end scenario tests that exhibit real-world behavior of the software. For example, use the software in a scenario the way the user would. The bottlenecks that appear in a real-world use case are usually worth fixing. However, if the user already thinks that the software performs well enough for their needs - on standard hardware of course - then focus your efforts elsewhere. You need the voice of the customer (e.g. a user experience person or program manager) to give you that information, however.
There are other types of performance testing you want to do as well that are above the method level. For example, if you are building a web application, you may need to test throughput, response time, and latency. These factors may have more to do with the way you use bandwidth and how machines are configured vs. how individual methods perform.
Motley: I'll have to agree with you there, as scary as that sounds. You have to admit that I was doing the right thing by measuring, however.
Maven: Absolutely! The golden rule of performance tuning is Measure! Measure! Measure! We usually say it three times to ensure the emphasis is made. Don't try to guess at what to optimize as much of the time you'll guess wrong. Leverage a profiler and other testing tools to provide some solid data. Another question: have you been thinking about performance all along?
Motley: What do you mean? I can't fix any performance problems until I know they exist! I deal with performance stuff late in the cycle like any other developer.
Maven: What if you make a bad design decision? I know one team that was trying to determine whether they should redesign their remoting infrastructure from using DCOM to Web Services. Early on they decided to take the plunge and made a heavy investment in web services. Know what happened? Although web services brought benefits in terms of standardization and security, the performance hit they took was huge. Web services infrastructure was tied to the entire application and they couldn't go back.
Motley: Sucks to be them! They should have prototyped or developed using an agile model in small pieces with performance testing throughout.
Maven: Exactly! So why not heed your own advice?
Motley: <silence>
Maven: You need to consider performance from day 1 of your development. Set performance goals to hit. The previous version, if it exists, can be your baseline for performance numbers. Aim to be within at least 10% of those numbers. Think about the scenarios where performance is critical. Plan and budget the resources such as memory and CPU you use accordingly. Prototype new technologies before making a heavy investment in them. If you discover a design flaw related to performance too late, you may not be able to fix it.
Motley: So your best practices for developers when it comes to performance are to:
- Measure, measure, measure - don’t guess at bottlenecks
- Don't leave performance work to too late in the cycle
- Think about performance goals early on, and consider performance when doing design
- Test and tune real customer scenarios
- Prototype technology decisions that may impact performance
What about Test-Driven Development, though? Where does performance work fit in with that methodology?
Maven: The story doesn't change a whole lot. You still think about performance goals and design impacts. As my buddy Matt says, make sure you don't do TDD with the blinders on as a change in one unit or component can have a significant impact on performance in a seemingly unrelated feature of the product. With TDD you are testing units in isolation, and mock objects help with that. You should think about developing performant code according to the contract in your unit. However, you may not be able to predict the impact on your dependencies. The advice there is to integrate early and often, and start performance testing early and often. Don't leave it until the end.
______________________________
Maven's Pointer: At Microsoft, software performance is treated very seriously, and is basically a feature in a project from day 1. You need to make some intelligent design decisions around performance early in a development cycle, but you want to avoid painting yourself into a corner if you make the wrong decision. That's why designing to interfaces is such an important design concept. You want your design to accommodate change if you make the wrong decision or you need to plug in a different algorithm depending on some heuristic. Your sorting algorithm may work fine on small datasets, but what if in the field you run across a huge dataset for which there is a better algorithm?
Maven's Resources:
- No specific resource around designing for performance and tuning performance due to the tight tie to technology. It is recommended that you search your favorite bookstore for books on performance specific to your technology (e.g. SQL, ASP.NET, C#, etc.) or activity (development, testing).
