In an introductory programming class, I had a professor who explained CPU cache as sitting in a library and having the book in front of you. RAM was like going to a bookshelf. And hitting the hard drive was like ordering it from a store and waiting for it to come.

While those numbers aren't exactly accurate, it does communicate the importance of keeping your data as close to the processor as possible. I recently came across an interesting study by Diomidis Spinellis that shows the relative speeds of different types of memory.

Nominal Worst case Sustained Productivity
Component size latency throughput $1 buys (Bytes read / s / $)
(MB/s) Worst case Best case
L1 D cache 64 KB 1.4ns 19022 10.7 KB 7.91·1012 2.19·1014
L2 cache 512 KB 9.7ns 5519 12.8 KB 1.35·1012 7.61·1013
DDR RAM 256 MB 28.5ns 2541 9.48 MB 3.48·1014 2.65·1016
Hard drive 250 GB 25.6ms 67 2.91 GB 1.22·1011 2.17·1017
 

If you have small chunks of data, the latency difference is huge. L1 cache is roughly 55,000 times faster than the hard drive. If you're dealing with sustained throughput, hitting L1 cache is 283.9 times faster than going to the hard drive.

Draw your own conclusions here, but keep this in mind when you're writing apps. (Note that the cost of memory is a bit outdated in this chart.)