Often when I speak with people that are planning a new solution they have a few different products or platforms in mind. To compare these options, most try something like a “capability matrix” (which goes by many names). Roughly these end up looking like this;
Can be clustered
Shiny management GUI
Command line interface
At the end of the process the product with the most features is flagged as the winner. There are three big problems with this approach;
1. One of the skills in architecting systems is ensuring that the solution is appropriate, and this feature comparison doesn’t address that at all. Do we need to support PHP? Does it matter?
2. Secondly, it doesn’t address the concept of trade-offs; Product One might well claim to be “highly scalable” but it can’t be clustered for availability. Which is more important to us?
3. Lastly, the features are usually taken from products’ marketing material, so don’t align well with each-other for comparison.
To try and address these customers might add weightings to different features, or try and align roughly equivalent functionality as comparable features. But this doesn’t go far enough – we’re still looking from the product’s perspective when instead we should be looking from the point of view of our requirements. This is by far the best way to really target an appropriate architecture.
How do we do this top-down suitability and trade-off analysis? There are a number of methods out there that can help, such as ATAM (Architecture Trade-off Analysis Method) and LAAAM (Lightweight Architecture Alternative Assessment Method). There are many more, but I particularly like LAAAM, as it’s so easy to apply but offers real tangible decision-support quickly. Check out this great walkthrough of how to apply LAAAM.
Using such an approach turns your thinking around into “what is important to me”, and puts it in the architectural language of non-functional requirements. These requirements are broken down into a tree, given real quantifiable form (scenarios that contain Context, Stimulus, and Response), and are then prioritised and weighted according to the solution’s needs. Finally, the product (or approach, platform, framework, etc) alternatives are assessed against each scenario, and some maths is done to score and rank each.
In our example, the tree might look something like this rather contrived sample;
You can see we’ve prioritised Reliability over Performance, and within Reliability given Availability priority over Monitoring abilities (I’m not necessarily promoting that decision, just giving an example!). Under Availability we might then write and weight scenarios, such as;
When a node fails (context) the backup node is online (stimulus) within 1 second (response)
Or under Response Time;
Under normal load (context), pages are served (stimulus) within 3 seconds (response)
Next we would use Rank Order Centroids or something similar to give these priorities a final numeric weight, and then rate Product One and Product Two against how well they can deliver on these scenarios; check out the end of the article linked to above to see how you do this with some simple multiplication.
We would no doubt start to see that the Reliability features of Product Two mean it is more appropriate to our solution than the broad technology support offered by Product One, and we would have quantifiable measures to demonstrate this fact and the various trade-offs that it requires to the stakeholders.
If you’re using the MSF for CMMI template in Team Foundation Server you might be interested to see that LAAAM features under the “Assess Alternatives” stage of the Solution Architecture work-stream, resulting in a “LAAAM Assessment Matrix” work product.
I’d certainly recommend having a look at this kind of approach as I find it more robust than capability matrices, and it can change your perspective on a solution. What’s more, it can be used to gain consensus amongst stakeholders as they see the impact that priorities have on the selection process. If you use such a comparison approach feel free to comment – I’d love to hear your experiences.