Ok, that isn’t what the paper says, it is what I said.  To quote the paper:

“We argue that benchmarks used in the Operating Systems literature for evaluating new designs and techniques are fundamentally unrealistic: they ignore the common case of running multiple applications (or subsystems) on the same machine.”

Again, if you want to skip the blather, you can read the no-cost to my readers paper at:

I always like papers that state the current way of doing things are wrong.  In the case of Multicore OS benchmarks the current benchmarks utilize metrics that assume that the running OS benchmarks concurrently is just wrong.  Nice!

Oh I love this paper, and could write about it for many pages, I just want to use my POS x86 to test this stuff.  But really it is better if you read the researchers remarkable story about how our current process of measuring stuff is just wrong.