J.D. Meier's Blog : Performance

Performance Hot Spots

J.D. Meier — Mon, 13 Apr 2009 08:56:01 GMT

I wrote a post about Performance Hot Spots on Shaping Software. This is a follow up to my post on Security Hot Spots. Hot spots are a way to turn Pareto's principle (the 80/20 rule) into action. By focusing on hot spots, you find the levers in the system that produce the greatest results. You can also use the Performance Hot Spots to help you find and share principles, patterns, and practices for performance. Read my post to find the what, why and how of Performance Hot Spots.

patterns & practices Performance Engineering Cheat Sheet

J.D. Meier — Thu, 20 Nov 2008 22:18:27 GMT

We posted our patterns & practices Performance Engineering Cheat Sheet to our Application Architecture Knowledge Base on CodePlex. It’s a bird’s-eye view of applying our performance techniques to the life cycle. The techniques and approach shipped with VSTS/MSF Agile starting in 2005.

Performance Engineering Overlay
Here’s a view that overlays our key performance techniques alongside common software engineering activities:

Key Activities in the Life Cycle
The core activities you should consider performing include the following:

Performance Objectives.
Budgeting.
Performance Modeling.
Performance Design Guidelines.
Performance Design Inspections.
Performance Code Inspections.
Performance Testing.
Performance Health Metrics.
Performance Deployment Inspections.
Capacity Planning.

You can read more about these techniques and how to apply them to your software architecture and your software development life cycle on our Cheat Sheet – patterns & practices Performance Engineering.

Additional Resources
You can find more information on performance engineering at:

My Related Posts

patterns & practices Performance Engineering

J.D. Meier — Tue, 16 Sep 2008 07:49:11 GMT

As part of our patterns & practices App Arch Guide 2.0 project, we're consolidating our information on our patterns & practices Performance Engineering. Our performance engineering approach is simply a collection of performance-focused techniques that we found to be effective for meeting your performance objectives. One of the keys to the effectiveness is our performance frame. Our performance frame is a collection of "hot spots" that organize principles, patterns, and practices, as well as anti-patterns. We use the frame to perform effective performance design and code inspections. Here's a preview of our cheat sheet so far. You'll notice a lot of similarity with our patterns & practices Security Engineering. It's by design so that you can use a consistent approach for handling both security and performance.

Performance Overlay
This is our patterns & practices Performance Overlay:

Key Activities in the Life Cycle
This Performance Engineering approach extends these proven core activities to create performance specific activities. These include:

Performance Objectives. Setting objectives helps you scope and prioritize your work by setting boundaries and constraints. Setting performance objectives helps you identify where to start, how to proceed, and when your application meets your performance goals.
Budgeting. Budget represents your constraints and enables you to specify how much you can spend (resource-wise) and how you plan to spend it.
Performance Modeling. Performance modeling is an engineering technique that provides a structured and repeatable approach to meeting your performance objectives.
Performance Design Guidelines. Applying design guidelines, patterns and principles which enable you to engineer for performance from an early stage.
Performance Design Inspections. Performance design inspections are an effective way to identify problems in your application design. By using pattern-based categories and a question-driven approach, you simplify evaluating your design against root cause performance issues.
Performance Code Inspections. Many performance defects are found during code reviews. Analyzing code for performance defects includes knowing what to look for and how to look for it. Performance code inspections to identify inefficient coding practices that could lead to performance bottlenecks.
Performance Testing. Load and stress testing is used to generate metrics and to verify application behavior and performance under normal and peak load conditions.
Performance Tuning. Performance tuning is an iterative process that you use to identify and eliminate bottlenecks until your application meets its performance objectives. You start by establishing a baseline. Then you collect data, analyze the results, and make configuration changes based on the analysis. After each set of changes, you retest and measure to verify that your application has moved closer to its performance objectives.
Performance Health Metrics. Identity the measures, measurements, and criteria for evaluating the health of your application from a performance perspective.
Performance Deployment Inspections. During the deployment phase, you validate your model by using production metrics. You can validate workload estimates, resource utilization levels, response time, and throughput.
Capacity Planning. You should continue to measure and monitor when your application is deployed in the production environment. Changes that may affect system performance include increased user loads, deployment of new applications on shared infrastructure, system software revisions, and updates to your application to provide enhanced or new functionality. Use your performance metrics to guide your capacity and scaling plans.

Performance Frames
Performance Frames define a set of patterns-based categories that can organize repeatable problems and solutions. You can use these categories to divide your application architecture for further analysis and to help identify application performance issues. The categories within the frame represent the critical areas where mistakes are most often made.

Category	Description
Caching	What and where to cache? Caching refers to how your applications caches data. The main points to be considered are Per user, application-wide, data volatility.
Communication	How to communicate between layers? Communication refers to choices for transport mechanism, boundaries, remote interface design, round trips, serialization, and bandwidth.
Concurrency	How to handle concurrent user interactions? Concurrency refers to choices for Transaction, locks, threading, and queuing.
Coupling / Cohesion	How to structure your application? Coupling and Cohesion refers structuring choices leading to loose coupling, high cohesion among components and layers.
Data Access	How to access data? Data Access refers to choices and approaches for schema design, Paging, Hierarchies, Indexes, Amount of data, and Round trips.
Data Structures / Algorithms	How to handle data? Data Structures and Algorithm refers to choice of algorithms; Arrays vs. collections.
Exception Management	How to handle exceptions? Exceptions management refers to choices / approach for catching, throwing, exceptions.
Resource Management	How to manage resources? Resource Management refers to approach for allocating, creating, destroying, and pooling of application resource
State Management	What and where to maintain state? State management refres to how your application maintains state. The main points to consider are Per user, application-wide, persistence, and location.

Architecture and Design Issues
Use the diagram below to help you think about performance-related architecture and design issues in your application.

The key areas of concern for each application tier are:

Browser. Blocked or unresponsive UI.
Web Server. Using state affinity. Wrong data types. Fetching per request instead of caching. Poor resource management.
Application Server. Blocking operations. Inappropriate choice of data structures and algorithms. Not pooling database connections.
Database Server. Chatty instead of batch processing. Contention, isolation levels, locking and deadlock.

Design Process Principles
Consider the following principles to enhance your design process:

Set objective goals. Avoid ambiguous or incomplete goals that cannot be measured such as "the application must run fast" or "the application must load quickly." You need to know the performance and scalability goals of your application so that you can (a) design to meet them, and (b) plan your tests around them. Make sure that your goals are measurable and verifiable. Requirements to consider for your performance objectives include response times, throughput, resource utilization, and workload. For example, how long should a particular request take? How many users does your application need to support? What is the peak load the application must handle? How many transactions per second must it support? You must also consider resource utilization thresholds. How much CPU, memory, network I/O, and disk I/O is it acceptable for your application to consume?
Validate your architecture and design early. Identify, prototype, and validate your key design choices up front. Beginning with the end in mind, your goal is to evaluate whether your application architecture can support your performance goals. Some of the important decisions to validate up front include deployment topology, load balancing, network bandwidth, authentication and authorization strategies, exception management, instrumentation, database design, data access strategies, state management, and caching. Be prepared to cut features and functionality or rework areas that do not meet your performance goals. Know the cost of specific design choices and features.
Cut the deadwood. Often the greatest gains come from finding whole sections of work that can be removed because they are unnecessary. This often occurs when (well-tuned) functions are composed to perform some greater operation. It is often the case that many interim results from the first function in your system do not end up getting used if they are destined for the second and subsequent functions. Elimination of these "waste" paths can yield tremendous end-to-end improvements.
Tune end-to-end performance. Optimizing a single feature could take away resources from another feature and hinder overall performance. Likewise, a single bottleneck in a subsystem within your application can affect overall application performance regardless of how well the other subsystems are tuned. You obtain the most benefit from performance testing when you tune end-to-end, rather than spending considerable time and money on tuning one particular subsystem. Identify bottlenecks, and then tune specific parts of your application. Often performance work moves from one bottleneck to the next bottleneck.
Measure throughout the life cycle. You need to know whether your application's performance is moving toward or away from your performance objectives. Performance tuning is an iterative process of continuous improvement with hopefully steady gains, punctuated by unplanned losses, until you meet your objectives. Measure your application's performance against your performance objectives throughout the development life cycle and make sure that performance is a core component of that life cycle. Unit test the performance of specific pieces of code and verify that the code meets the defined performance objectives before moving on to integrated performance testing. When your application is in production, continue to measure its performance. Factors such as the number of users, usage patterns, and data volumes change over time. New applications may start to compete for shared resources.

Design Guidelines
This table represents a set of secure design guidelines for application architects. Use this as a starting point for performance design and to improve performance design inspections.

Category	Description
Caching	Decide where to cache data. Decide what data to cache. Decide the expiration policy and scavenging mechanism. Decide how to load the cache data. Avoid distributed coherent caches.
Communication	Choose the appropriate remote communication mechanism. Design chunky interfaces. Consider how to pass data between layers. Minimize the amount of data sent across the wire. Batch work to reduce calls over the network. Reduce transitions across boundaries. Consider asynchronous communication. Consider message queuing. Consider a "fire and forget" invocation model.
Concurrency	Design for loose coupling. Design for high cohesion. Partition application functionality into logical layers. Use early binding where possible. Evaluate resource affinity.
Coupling / Cohesion	How to structure your application? Coupling and Cohesion refers structuring choices leading to loose coupling, high cohesion among components and layers.
Data Structures / Algorithms	Choose an appropriate data structure. Pre-assign size for large dynamic growth data types. Use value and reference types appropriately.
Resource Management	Treat threads as a shared resource. Pool shared or scarce resources. Acquire late, release early. Consider efficient object creation and destruction. Consider resource throttling.
Resource Management	How to manage resources? Resource Management refers to approach for allocating, creating, destroying, and pooling of application resource
State Management	Evaluate stateful versus stateless design. Consider your state store options. Minimize session data. Free session resources as soon as possible. Avoid accessing session variables from business logic.

Additional Resources

Performance Design Guidelines (MSDN)
Performance Modeling (MSDN)
Performance Design Inspection (MSDN)
Performance Testing Guidance for Web Applications (MSDN)

My Related Posts

Performance Techniques, Building Codes, and Approach

J.D. Meier — Mon, 14 Jan 2008 19:02:00 GMT

When you improve the performance of a system, I find it's helpful to think in terms of the techniques you use, the "building codes" you follow, and the overall life cycle approach you use. While there's a lot to know about performance engineering, here's the keys that I've found useful working with customers and experts over the years:

Key Techniques
Techniques are specific methods for producing a result:

Building Codes
Think of "building codes" as the principles, patterns, and checklists for the structure:

Approach
The approach is the methodology you use to orchestrate your efforts:

Performance Engineering

Key MSDN References

While there's certainly more to know, if you use the techniques, building codes, and approach above, you can start improving the performance of your applications immediately.

Using Guidance Explorer from Outlook

J.D. Meier — Tue, 08 Jan 2008 20:15:06 GMT

It looks like Alik Levin has some helpful posts on using Guidance Explorer from Outlook:

Customize Guidance Explorer Inside Outlook 2007 – Find Tech Gold Nuggets Instantly - how to configure Outlook for easier searches using built-in functionality.
Consume patterns&practices Guidance Explorer Via RSS Using Outlook 2007 - how to set up RSS feeds inside Outlook to read GE’s content.

Alik is a long time patterns & practices user. He's intimately familiar with our security and performance bodies of guidance (BOG). He always surprises me with his ability to find every nugget among our vast collection (after all - we have a few thousand pages of patterns & practices security and performance guidance on MSDN.)

As an experienced security and performance consultant, Alik was already very efficient and effective with his customer delivery. I think he was the first person I saw start using Guidance Explorer to build customized guidance for customers. Now that he's taken GE to the next level with his Outlook integration, he's a security and performance machine!

Guidance Share Sweep

J.D. Meier — Wed, 02 Jan 2008 18:48:06 GMT

One of the most important things I did while I was on vacation was sweeping Guidance Share. Guidance Share is where I consolidate my body of software engineering guidance and test user experiences. I redesigned the home page for simpler browsing and findability. It was more pain than pleasure for me, but if it helps the broader community, that's my payback.

Here's a highlight of Guidance Share:

Browsable nuggets for software performance and security
Durable, evolvable frames for security and performance (think of these as maps)
How Tos, Guidelines, Checklists and more

Guidance Share gives me a unique vantage point that I haven't been able to get any other way. The act of building it and evolving it helps me make gain new insights. It also forces me to find ways to be extremely efficient. I then try to carry these lessons over to MSDN and to help shape patterns & practices information models. I don't own the MSDN experience, but I can give input. Guidance Share helps me solidify my recommendations with living proof. It's also let's me quickly experiment with new user experiences.

My biggest lesson learned is how difficult it is to integrate information and make it useful, even when you own it. It's one thing to have a snapshot of information that's useful for a given point in time; it's another to create a stable backdrop with a firm foundation that can evolve over time. The key is factoring volatile from stable information, and enabling them to play well together.

Note that Guidance Share is under construction and there are some obviously empty areas, but it's a work in progress. It's a living knowledge base for software engineering that I periodically sweep to share the best that I've learned.

Enjoy!

Now on Amazon: Performance Testing Guidance for Web Applications

J.D. Meier — Sat, 01 Dec 2007 03:15:07 GMT

Our patterns & practices Performance Testing Guidance for Web Applications book is now available on Amazon.

Videos: Security, Performance Testing, and Visual Studio Team System

J.D. Meier — Thu, 22 Nov 2007 12:17:00 GMT

We posted our ASP.NET 2.0 security videos, performance testing videos, and Visual Studio Team System videos to the MSDN library.

Video Index

Video Index

ASP.NET 2.0 Security Videos
Keith Brown explains the mechanics behind some common input and data validation security issues and how to address them.

Performance Testing Videos
Scott Barber explains the big picture for how to approach performance testing as well as how to do workload modeling and performance reporting.

Visual Studio Team System Videos
Our team takes you through key source control concepts in Team Foundation Server.

I hope we produce more videos in the coming months. I particularly like factoring "explained" videos from the "how to" videos and keeping the videos short and focused. I think we have more work to do here, but I think we've learned key lessons each time we've done a batch of videos.

Now on MSDN: patterns & practices Performance Testing Guidance for Web Applications

J.D. Meier — Sat, 27 Oct 2007 09:59:00 GMT

You can now find our patterns & practices Performance Testing Guidance for Web Applications on MSDN in HTML. It's the same guidance we hosted on CodePlex. CodePlex was our channel for agile release of the guidance. Once we baked the guidance, we ported it to MSDN.

Contents at a Glance
Here's the

Chapters

Download
You can download the patterns & practices Performance Testing Guidance for Web Applications from CodePlex.

Guidance Explorer Scenario
If you want to tailor the guidance for your scenario, you can download Guidance Explorer from CodePlex. Using Guidance Explorer, you can create custom views by dragging and dropping the relevant guidance and then tailoring it as you see fit. You can then save your view or an item to Word or HTML

Performance Testing Videos Posted to CodePlex

J.D. Meier — Sat, 27 Oct 2007 08:20:00 GMT

Today we posted our Performance Testing Videos to CodePlex. They're a companion set for our patterns & practices Performance Testing Guidance for Web Applications.

Video Index
Here's a list of the videos:

Video: What Is - The Core Activities of Performance Testing (Length: 4:32 - Size: 3.02 MB)
Video: What Is - The Core Activities of Performance Testing in Agile Projects (Length: 3:02 - Size: 2.38 MB)
Video: What Is - The Core Activities of Performance Testing in CMMI Projects (Length: 2:55 - Size: 2.15 MB)
Video: What Is - The Distributions of Data for Performance Tests Results (Length: 3:32 - Size: 2.39 MB)
Video: What Is - The Reporting Fundamentals for Performance Test Data (Length: 1:58 - Size: 1.45 MB)
Video: What Is - The Success Criteria for Performance Test Projects (Length: 2:12 - Size: 1.49 MB)
Video: What Is - The Mathematical and Statistical Principles for Performance Testing (Length: 2:47 - Size: 1.83 MB)
Video: What Is - The Modeling Application Usage for Performance Testing (Length: 2:25 - Size: 1.70 MB)

Additional Resources

patterns & practices Performance Testing Guidance (Online KB)
patterns & practices Performance Testing Guidance for Web Applications (Guide)

Scenarios in Practice

J.D. Meier — Mon, 15 Oct 2007 05:21:00 GMT

Scenarios are a practical way to organize and focus your product design, development and release. (We use scenario-driven engineering in patterns & practices)

Key Benefits

Business value. You can use scenarios to evaluate business value. What pain does that scenario address? ... What opportunity does it create? ... etc.
Chunking and prioritizing. You can use scenarios to chunk up and prioritize your product. You can think of versioning in terms of releasing incremental value or scenarios.
Effective design. Walking through a set of customers scenarios and "what ifs" will help you figure out your product's most effective design. It's not a silver-bullet, but it does help make requirements more concrete, or at least put them in perspective. It also opens the door to more precise questions about user experience or engineering challenges..
Focal point for perspectives. You can use scenarios as a focal point for user experience, business and tech perspectives.
Requirements in context. You can use scenarios to put requirements in context. A requirement makes a lot more sense when you see it in action.
Architectural trade-offs. You can use scenarios to evaluate and make architectural trade-offs. You can't evaluate an architecture in a vacuum; you can evaluate against concrete scenarios. There's several flavors of scenario-based evaluations you can leverage.
Customer value. Your customers can appreciate how well you did (or didn't) address their scenario.
Tests for success. You can use scenarios as tests for success. By measuring customer effectiveness against specific scenarios, you have a way to focus your improvements.

Chunking Up a Project Using Scenarios
If you think of your product as helping a user accomplish a goal, it helps cut through the fog. You can think of your product in terms of a list of user goals, business goals and technical goals. What's the minimum set of tasks your user needs to perform with your product? That's your baseline release. What's the incremental value from there? That's how you start to shape your versions and releases.

User Experience, Business and Technical Perspectives
Using scenarios is a good forcing function to bring together the user experience, business, and technical perspectives. For the sake of example, let's say your scenario is "User views the product catalog." From the user experience perspective, you're thinking in terms of "show me a relevant list" and "don't make me wait," From a business perspective, you're thinking in terms of "support a given number of orders per hour" and "lower the total cost of ownership." From a technical standpoint, you're thinking in terms of "requests per second," "minimize resource utilization," ... etc. Well, no wonder getting software right is tough! Luckily, scenarios help you bring together and rationalize the perspectives against a common frame of reference.

Constraints Make More Sense In Context
Constraints are boundaries to operate within, or what the solution must or must not do. In software, I see a lot of generic constraints passed down as policies. When policy meets scenario, you now have a way to evaluate the effectiveness of that constraint. You can also measure whether that scenario is actually meeting the constraint. You can perform scenario-based testing against a set of scenarios with specific constraints.

Walking the Scenarios
Have you ever been sold on a great set of features, only to fall flat when you try to actually do something useful? Obviously, that's the extreme case, but it happens to me. It happens less now because whenever I go to buy something, I walk my usage scenarios. Doing a dry run against a scenario is very revealing. This approach works great on the engineering side too. It's one thing to have generic technical requirements for security or performance. It's another to evaluate a scenario and make appropriate and relevant trade-offs from a user, business and technical perspective. Suddenly, generic technical requirements no longer seem as helpful, do they? They still have their place, but when you're engineering your job is to make the right trade-offs.

Scenarios and Solutions
Given part of my job is to improve customer effectiveness on our platform, I regularly use scenarios as a backdrop for evaluation. As I've gotten more effective, I noticed shifting from features and components to scenarios and solutions is the key.

More Information

My Related Posts

New Release: patterns & practices Performance Testing Guidance for Web Applications

J.D. Meier — Wed, 29 Aug 2007 22:18:00 GMT

We released the final version of our patterns & practices Performance Testing Guidance for Web Applications. This guide provides an end-to-end approach for implementing performance testing. Whether you're new to performance testing or looking for ways to improve your current performance-testing approach, you will gain insights that you can tailor to your specific scenarios. The main purpose of the guide is to be a relatively stable backdrop to capture, consolidate and share a methodology for performance testing. Even though the topics addressed apply to other types of applications, we focused on explaining from a Web application perspective to maintain consistency and to be relevant to the majority of our anticipated readers.

Key Changes Since Beta 1

Added forewords by Alberto Savoia and Rico Mariani.
Integrated more feedback and insights from customer reviews (particularly chapters 1-4, 9, 14, 18)
Integrated learnings from our Engineering Excellence team.
Refactored and revamped the performance testing types.
Revamped and improved the test execution chapter.
Revamped and improved the reporting chapter.
Revamped the stress testing chapter.
Released the guide in HTML pages on our CodePlex Wiki.

Highlights

Learn the core activities of performance testing.
Learn the values and benefits associated with each type of performance testing.
Learn how to map performance testing to agile
Learn how to map performance testing to CMMI
Learn how to identify and capture performance requirements and testing objectives based on the perspectives of system users, business owners of the system, and the project team, in addition to compliance expectations and technological considerations.
Learn how to apply principles of effective reporting to performance test data.
Learn how to construct realistic workload models for Web applications based on expectations, documentation, observation, log files, and other data available prior to the release of the application to production.

Why We Wrote the Guide

To consolidate real-world lessons learned around performance testing.
To present a roadmap for end-to-end performance testing.
To narrow the gap between state of the art and state of the practice.

Scope

Managing and conducting performance testing in both dynamic (e.g., Agile) and structured (e.g., CMMI) environments.
Performance testing, including load testing, stress testing, and other types of performance related testing.
Core activities of performance testing: identifying objectives, designing tests, executing tests, analyzing results, and reporting.

Features of the Guide

Approach for performance testing. The guide provides an approach that organizes performance testing into logical units to help you incrementally adopt performance testing throughout your application life cycle.
Principles and practices. These serve as the foundation for the guide and provide a stable basis for recommendations. They also reflect successful approaches used in the field.
Processes and methodologies. These provide steps for managing and conducting performance testing. For simplification and tangible results, they are broken down into activities with inputs, outputs, and steps. You can use the steps as a baseline or to help you evolve your own process.
Life cycle approach. The guide provides end-to-end guidance on managing performance testing throughout your application life cycle, to reduce risk and lower total cost of ownership (TCO).
Modular. Each chapter within the guide is designed to be read independently. You do not need to read the guide from beginning to end to benefit from it. Use the parts you need.
Holistic. The guide is designed with the end in mind. If you do read the guide from beginning to end, it is organized to fit together in a comprehensive way. The guide, in its entirety, is better than the sum of its parts.
Subject matter expertise. The guide exposes insight from various experts throughout Microsoft and from customers in the field.

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance-Testing Techniques

Chapters

Chapter 1 – Fundamentals of Web Application Performance Testing
Chapter 2 – Types of Performance Testing
Chapter 3 – Risks Addressed Through Performance Testing
Chapter 4 – Web Application Performance Testing Core Activities
Chapter 5 – Coordinating Performance Testing with an Iteration-Based Process
Chapter 6 – Managing an Agile Performance Test Cycle
Chapter 7 – Managing the Performance Test Cycle in a Regulated (CMMI) Environment
Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness
Chapter 9 – Determining Performance Testing Objectives
Chapter 10 – Quantifying End-User Response Time Goals
Chapter 11 – Consolidating Various Types of Performance Acceptance Criteria
Chapter 12 – Modeling Application Usage
Chapter 13 – Determining Individual User Data and Variances
Chapter 14 – Test Execution
Chapter 15 – Key Mathematic Principles for Performance Testers
Chapter 16 – Performance Test Reporting Fundamentals
Chapter 17 – Load-Testing Web Applications
Chapter 18 – Stress-Testing Web Applications

Our Team

Carlos Farre
Prashant Bansode
Scott Barber
Dennis Rea

Contributors and Reviewers

External Contributors and Reviewers: Alberto Savoia; Ben Simo; Cem Kaner; Chris Loosley; Corey Goldberg; Dawn Haynes; Derek Mead; Karen N. Johnson; Mike Bonar; Pradeep Soundararajan; Richard Leeke; Roland Stens; Ross Collard; Steven Woody
Microsoft Contributors / Reviewers: Alan Ridlehoover; Clint Huffman; Edmund Wong; Ken Perilman; Larry Brader; Mark Tomlinson; Paul Williams; Pete Coupland; Rico Mariani

My Related Posts

Performance Threats

J.D. Meier — Tue, 28 Aug 2007 11:42:00 GMT

Rico and I have long talked about performance threats. I finally created a view that shows how you can think of performance issues in terms of vulnerabilities, threats and countermeasures. See Performance Frame v2.

In this case, the vulnerabilities, threats and countermeasures are purely from a technical design standpoint. To rationalize performance against other quality attributes and against goals and constraints, you can use performance modeling and threat modeling. To put it another way, evaluate your design trade-offs against the acceptance criteria for your usage scenarios, considering your user, system, and business goals and constraints.

Performance Inspections

J.D. Meier — Sun, 08 Jul 2007 07:45:00 GMT

In this post, I'll focus on design, code, and deployment inspections for performance. Inspections are a white-box technique to proactively check against specific criteria. You can integrate inspections at key stages in your life cycle, such as design, implementation and deployment.

Keys to Effective Inspections

Know what you're looking for.
Use scenarios to illustrate a problem.
Bound the acceptance criteria with goals and constraints.

Performance Frame
The Performance Frame is a set of categories that helps you organize and focus on performance issues. You can use the frame to organize principles, practices, patterns and anti-patterns. The categories are also effective for organizing sets of questions to use during inspections. By using the categories in the frame, you can chunk up your inspections. The frame is also good for finding low-hanging fruit.

Performance Design Inspections
Performance design inspections focus on the key engineering decisions and strategies. Basically, these are the decisions that have cascading impact and that you don't want to make up on the fly. For example, your candidate strategies for caching per user and application-wide data, paging records, and exception management would be good to inspect. Effective performance design inspections include analyzing the deployment and infrastructure, walking the performance frame, and doing a layer-by-layer analysis. Question-driven inspections are good because they help surface key risks and they encourage curiosity.

While there are underlying principles and patterns that you can consider, you need to temper your choices with prototypes, tests and feedback. Performance decisions are usually trade-offs with other quality attributes, such as security, extensibility, or maintainability. Performance Modeling helps you make trade-off decisions by focusing on scenarios, goals and constraints.

For more information, see Architecture and Design Review of a .NET Application for Performance and Scalability and Performance Modeling.

Performance Code Inspections
Performance code inspections focus on evaluating coding techniques and design choices. The goal is to identify potential performance and scalability issues before the code is in production. The key to effective performance code inspections is to use a profiler to localize and find the hot spots. The anti-pattern is blindly trying to optimize the code. Again, a question-driven technique used in conjunction with measuring is key.

For more information, see Performance Code Inspection.

Performance Deployment Inspections
Performance deployment inspections focus on tuning the configuration for your deployment scenario. To do this, you need to have measurements and runtime data to know where to look. This includes simulating your deployment environment and workload. You also need to know the knobs and switches that influence the runtime behavior. You also need to be bounded by your quality of service requirements so you know when you're done. Scenarios help you prioritize.

My Related Posts

Inspections

J.D. Meier — Mon, 02 Jul 2007 19:30:00 GMT

Inspections are a white-box technique to proactively check against specific criteria. You can integrate inspections as part of your testing process at key stages, such as design, implementation and deployment.

Design Inspections
In a design inspection, you evaluate the key engineering decisions. This helps avoid expensive do-overs. Think of inspections as a dry-run of the design assumptions. Here’s some practices I’ve found to be effective for design inspections:

Use inspections to checkpoint your strategies before going too far down the implementation path.
Use inspections to expose the key engineering risks.
Use scenarios to keep the inspections grounded. You can’t evaluate the merits of a design or architecture in a vacuum.
Use a whiteboard when you can. It’s easy to drill into issues, as well as step back as needed.
Tease out the relevant end-to-end test cases based on risks you identify.
Build pools of strategies (i.e. design patterns) you can share. It’s likely that for your product line or context, you’ll see recurring issues.
Balance user goals, business goals, and technical goals. The pitfall is to do a purely technical evaluation. Designs are always trade-offs.

Code Inspections
In a code inspection, you focus on the implementation. Code inspections are particularly effective for finding lower-level issues, as well as balancing trade-offs. For example, a lot of security issues are implementation level, and they require trade-off decisions. Here’s some practices I’ve found to be effective for code inspections:

Use checklists to share the “building codes.” For example, the .NET Design Guidelines are one set of building codes. There's also building codes for security, performance ... etc.
Use scenarios and objectives to bound and test. This helps you avoid arbitrary optimization or blindly applying recommendations.
Focus the inspection. I’ve found it’s better to do multiple, short-burst, focused inspections than a large, general inspection.
Pair with an expert in the area you’re inspecting.
Build and draw from a pool of idioms (i.e. patterns/anti-patterns)

Deployment Inspections
Deployment is where application meets infrastructure. Deployment inspections are particularly helpful for quality attributes such as performance, security, reliability and manageability concerns. Here’s some practices I’ve found to be effective for deployment inspections:

Use scenarios to help you prioritize.
Know the knobs and switches that influence runtime behavior.
Use checklists to help build and share expertise. Knowledge of knobs and switches tends to be low-level and art-like.
Focus your inspections. I’ve found it more productive and effective to do focused inspections. Think of it as divide and conquer.

Additional Considerations

Set objectives. Without objectives, it's easy to go all over the board.
Keep a repository. In practice, one of the most effective approaches is to have a common share that all teams can use as a starting point. Each team then tailors for their specific project.
Integrate inspections with your quality assurance efforts for continuous improvement.
Identify skill sets you'll need for further drill downs (e.g. detail design, coding, troubleshooting, maintenance.) If you don't involve the right people, you won't produce effective results.
Use inspections as part of your acceptance testing for security and performance.
Use checklists as starting points. Refine and tailor them for your context and specific deliverables.
Leverage tools to automate the low-hanging fruit. Focus manual inspections on more context-sensitive or more complex issues, where you need to make trade-offs.
Tailor your checklists for application types (Web application, Web Service, desktop application, component) and for verticals (manufacturing, financial ... etc.) or project context (Internet-facing, high security, ... etc.)

In the future, I'll post some more specific techniques for security and performance.