Jeremy Kelly pointed me to this post that he made about a debugging session that the Exchange escalation guys did that discovered a rootkit running on a customers machine.
It is an awesome detective job, and it’s a great example of exactly why (a) Every Developer needs to know Assembly and (b) Why you need to reformat your machine after you’ve been infected.
The ONLY way that they discovered that this machine had been rooted was the fact that the rootkit had a bug. If it hadn’t been for the bug, the poor customer would have never known that he had a problem, until much later.
And yes, stuff like this happens a lot. We’re very fortunate that we have some really talented escalation engineers working here that can diagnose stuff like this, but it’s a part of the skill set that developers and support people need to have.
Way to Go Jeremy, a great read.