Performance testing in the cloud can be quite challenging, the best video I have seen so far was one that our test architect Dennis Bass used to show to people asking about what testing in the cloud is like (I am also borrowing a lot of content from him)
The video says it all: the infrastructure, environment, tools, basically everything you rely on is changing, all while you are building your application
..even more in the cloud, more compute time = more money.
Of course, if your performance sucks, nobody is really going to use your service – but even small performance variations can affect your cost.
For example, while these two code snippets look kind of the same, one of them costs 50% more than the other.
String guid = ServiceGuid.ToString();
(and yes, I know FxCop will scream about my string compare, that’s just an example)
The costs really add up at the end. You need to be really smart about the number and size of your instances
Windows Azure Compute cost is based on size and instances. The formula is simple:
Size *total instances
The critical factor is size. Choices span from "extra small" to "extra large." (For a list of size details, seeWindows Azure Compute, and How To Configure Virtual Machine Sizes.) To complicate this, running many instances of an extra small machine can be more effective than running one instance of an extra large machine. The final choice depends on the application. If the application does intensive processing, then you require an extra large (or large, or medium) instance. In contrast, you can have an application that does little processing but serves many users. In that case, a small or extra small instance that scales out to many instances in response to demand may be all that is needed.
- Estimate the costs of your testing, be realistic about it and set the expectations.
o Testing in the cloud costs thousands, while you can eliminate some of the costs but using simulators or on premise testing, you will need to do final verification on the cloud!
- Add time and resources
o Application and test engineering
o Additional network bandwidth
- Operational testing needs
o Elasticity, Patching
o Backup/Restore, Disaster Recovery
- Test Data
o E.g. Unique Internet Accounts
- Have the right diagnostics in place.
- Keep a heartbeat or ping test running.
- Set realistic expectations
o The cloud is designed for scale, not for performance!
Even more in the cloud, multitenancy, network issues and all kinds of problems really - Dependent Services will go up/down, speed up, slow down, be updated,..
- Create your own Chaos Monkey or use an existing one, NetFlix team have shared their Chaos Monkey for AWS, also you can try Steve Marx’ Chaos money for Azure.
- RETRY: Everything, all the time, until you are 100% sure it is really broken, try to fail gracefully as much as you can (this will be more common than you think!
No matter how reliable your code is, other tenants code might not be as reliable, they might abuse the resources and slow you down, so no matter how confident you are about your code make sure you handle extreme conditions like high CPU and low memory accordignly, this is especially important with SQL Azure.
In the next post, I will be writing about the test tools and strategies for testing a hosted application, stay tuned!