I read the following article that Microsoft's Sender ID hits new milestone for stopping spam. In the article, Microsoft has announced that their Sender ID spam filtering technology has blocked over 3.8 billion spam messages. It goes on to say that Microsoft claims that Sender ID has enjoyed a three-fold increase in adoption over the past year.
3.8 billion messages is a lot of spam, I'd say that's not too shabby. But if you will allow me to boast a little bit, some of the anti-spam rules that I have personally written are also pretty impressive. Since October, I checked the stats on four of my special spam rules designed to target certain types of messages. They are responsible for blocking over 4 billion spams. I'm sure it's higher, but the counter in one of the rules stops at 2^31.
This isn't an apples-to-apples comparison since I'm not sure how Microsoft is using this Sender ID in their spam filtering (in EHS, we aren't using it, perhaps Hotmail or Exchange is). My spam rules don't flag the message as spam if it hits the message, but it does contribute a very significant portion to the overall spam score (in fact, the majority). Whether or not Sender ID does the same or works by itself is the question.
Still, I'm a bit of an old-school content filterer. Reputation-based filtering that examines the sender is the way of the future, but a good content filter sure helps.
How many false positives did MS have on that though? I'm guessing they don't measure them since a lot of Hotmail's filters just drop the mail on the floor - no bounce/reject, no mail in the Junk folder, nothing.
It sure helps make their filters look great, but it's practically useless for the poor end user.
Hah, love the title of this one.
It would be great if Microsoft would actually recognize, process, and handle Sender ID records correctly. Then I'd trust their numbers. But they don't, and I don't. It's their own specification, but it's employed so awkwardly and out-of-spec against their inbound mail streams. Very confusing.
Hotmail does have a way of measuring their spam effectiveness and false positive rate, they use something called the feedback loop. They get users to classify their own mail stream as spam or non-spam and then they compare it to the action that the filter would have taken.
So, if the filter classifies mail as spam but the user classifies it as non-spam, that would be a false positive. In this manner they can determine the FP rate and determine how well Sender ID is performing.
I'm not sure how Microsoft is handling Sender ID, but my bet would be that they only junk mail that fail Sender ID tests, not for ambiguous patterns.
For example, in EHS, we have SPF tests, but there are a whole bunch of SPF tests including an SPF pass, no SPF record, SPF soft fail and SPF hard fail. None of these on their own is enough to mark a message as spam. However, we do have a custom spam filter option that if a domain fails an SPF test, it automatically gets filtered as spam. Customers wanted that feature so we added it.
Still, in EHS we differentiate between varying levels of SPF checks and I would suspect that Microsoft's other filtering services do something similar.
BTW, Matt, I forgot to add this to the end of my comments: your point remains, and indeed, it is crucial. Pointing out the efficacy of a new anti-spam method is not revealing unless it is accompanied by the false positive rate.
With my own spam rules that I wrote, the false positive rate is low but it isn't zero. I have to admit that my spam rules could use some work but I have determined that the spam trade-off produces a level of false positives that is acceptable with respect to the constraints in our technology.