re: Testing in Production (TiP), a common and costly technical malpractice???

Seth Eliot — Mon, 26 Sep 2011 21:13:36 GMT

Hi Steve, thanks for your comment.

I am a fan of production data because I think the diversity of what happens in production can only be approximated in a test lab. With a large enough user base, you are more likely to find those weird edge cases, than relying solely on the expertise of an experienced tester.

And if something won't be seen in production ("bad data" in your example), then why so we even care about it? Testing in Production lets us focus on the code paths that our users use, and product features our users care about.

Now if you are talking about a small user base, then I understand your point. Either way I still favor some up-front testing, but I think it is a balance with how much TiP you do.

re: Testing in Production (TiP), a common and costly technical malpractice???

Steve Croft — Sun, 18 Sep 2011 01:53:27 GMT

I am not a big fan of using production data for testing because it often only exercises particular areas of a system. I'd rather see datasets built during development that test the extremes of the components. Some of these extreme conditions may not be in your production data for a year or two down the road.

Another aspect of testing is finding out how things work when bad data is fed into the system. You won't see any bad data in a production dataset (hopefully).

Production data can be used if you need to test performance and need large volumes of data, just keep the security aspects in mind.

jmho...