Performance, Scalability, and Stability

simonince — Tue, 14 Apr 2009 17:56:00 GMT

I spend quite a lot of time nowadays advising on architecting applications to be scalable, recommending performance testing approaches and processes, and being called in to help an organisation remediate problems with an existing application. It’s one of my favourite sides to the job.

But in spite of spending so much time on this, I still catch myself apparently confusing three terms – Performance, Scalability, and Stability. Why? Well, I’m not really confusing them – it’s just that they’re so closely related I don’t think you can consider any one of them in isolation. Let’s consider my (unofficial and buzzword-free) definition of each;

Performance is about the time taken to execute tasks. The quicker a task completes, the better the application is performing.

Scalability is about the number of tasks a system can execute at the same time. This usually maps almost directly to the number of concurrent users that an application can support.

Stability is about how many failures an application exhibits; whether that is manifested as unexpected or unintended behaviour, users receiving errors, or a catastrophic failure that brings a system down. The fewer failures that are observed the more stable an application is.

I’ve over-simplified a little and left out some other concepts, but I think it serves to illustrate the point. A nice way to visualise this is in the form of a triangle;

This visualisation helps to emphasise that each quality can affect the other. The word “quality” is key here – each of these is basically a “Quality Attribute” (or “Quality of Service” requirement) – although Stability arguably covers a number of attributes. If this term is unfamiliar to you, check out the patterns & practices Application Architecture Guide 2.0, in section II (Design), under Quality Attributes.

It is easy to say they affect each-other, but how exactly does this happen? Let’s consider them one at a time;

Poor Performance affects Scalability

Slow performance means tasks take longer. If they take longer, they are more likely to overlap when multiple users do them at the same time. Therefore improving performance reduces the likelihood of contention for shared resources (such as a database), and therefore increases scalability.

Poor Performance affects Stability

Slow performance means tasks take longer. If they take longer, they are more likely to overlap when multiple users do them at the same time. This is likely to lead to more frequent locks, deadlocks, and resource contention, which is more likely to lead to errors and stability problems. It also means that poor programming practices (such as race conditions and incorrectly shared state) are more likely to be spotted.

Poor Scalability affects Performance

Poor scalability means an application supports fewer concurrent tasks. Therefore tasks may have to queue up if there is no capacity to process them. This usually reduces both perceived and actual performance.

Poor Scalability affects Stability

Poor scalability means an application has two options when demand exceeds capacity; queue requests, or reject requests. Rejecting requests results in an error or unexpected behaviour, and is my definition of instability. Queuing requests leads to reduced performance, but also puts demands on sharing resources, connections, memory, CPU, and more. All this increased demand inevitably leads to further stability issues eventually.

Poor Stability affects Performance

Poor stability generally means three things. Firstly, errors must be propagated throughout the system. .NET exceptions do carry overheads and therefore increase CPU and memory requirements of the application.

Secondly, partial success and partial failure must be handled, usually with database rollbacks but also with manual compensation logic. This inevitably leads to additional resource requirements on the system – whether that is the database server dealing with rollbacks, or the additional processing of compensation logic.

Thirdly, when unexpected results are received the user is likely to retry them. This means the load on parts of an application can literally double (or worse) during times of instability.

The first two of these impacts the actual physical performance of the system, but the third also affects the perceived performance from the user’s point of view.

Poor Stability affects Scalability

Poor stability means that additional resources are used, and users are likely to retry tasks. This means more tasks are being performed by fewer users, and therefore ultimately less users can be supported concurrently as a result.

... wow, what should I do?!

After all that, you might be thinking “wow, what should I do?”

The answer is that you should always consider performance, scalability, and stability when architecting, building, and testing your applications. To be precise, you should;

1. Set objectives for each Quality Attribute (including those I’ve not mentioned in this post).

2. Be aware of how each Quality Attribute affects the others.

3. Invest time, hardware, and money in testing that your application meets the objectives you set for each Quality Attribute.

4. Fix issues that you find through re-architecting, code refactoring or tuning, or by tuning your deployment configuration and/or environment.

5. Do all this as early as possible, as it is considerably cheaper to fix performance, scalability, and stability issues early in the development lifecycle.

p&p Sharepoint Guidance

simonince — Thu, 28 Aug 2008 14:53:00 GMT

Via Blaine’s post (that gives quite a lot of information on what is included and what their plans are), the patterns & practices Sharepoint Guidance is starting to take shape, with a drop to codeplex having arrived this week. Be sure to check it out – this is an area that is becoming more and more popular, so it’s a good chance to get to know the guidance early on.

The drop includes a Reference Implementation so download it and compare it to how you’re working!

Testing WCF Services

simonince — Sun, 27 Jan 2008 18:07:00 GMT

I was recently present at an internal demonstration by Rob Jarratt of a tool to generate unit tests for WCF services from just the message trace files... and it has just been released to codeplex! The tool integrates with Visual Studio 2005 or 2008, and allows creation of a unit test that replays the messages found in the trace file. You can then use the Visual Studio test tools to apply this as load to perform scalability testing of your service layer. There are some neat little features that allow you to edit the values used for each message too to get a more realistic set of messages.

I’d highly recommend checking it out if you’re building an application that uses WCF.

Simon Ince's Blog : Testing