Terry Zink's Cyber Security Blog

Discussing Internet security in (mostly) plain English

New spam rules of engagement finally starting to sink in

New spam rules of engagement finally starting to sink in

  • Comments 6

Having been a spam fighter for over two years, and having watched spam evolve very quickly over the previous six months, it is now sinking in to me that the methodology in which we used to use to fight spam is no longer valid.  Whereas before we had a single focus as our primary method of fighting spam, that assumption doesn't hold anymore.  Instead, spam fighting companies must resort to a mult-tiered effort in their attempts to fight spam.  I'm not saying that it took us this long to realize this, or that we only recently changed tactics.  Indeed, Microsoft has done a pretty good job at recognizing this.  What I am saying is beware any spam filtering service that promises a simple solution towards fighting spam, because a simple solution no longer works.

I was browsing through the competition and came across IronPort's strategy, publically available on their web page.  I was impressed with the level of complexity that they use.  I have read some articles that they have and I like the theory behind their Context Adaptive Scanning Engine.  Rather than having different parts of a system add up to make spam determination in isolation, they combine elements of spamminess that stack on top of one another, or rather, with each other, that increases the spamminess.

For example, if an email contains element A (for 100 points) and element B (for 50 points), A + B will make a message more like spam at 150 points.  However, the way IronPort does it, if a message contains A + B, they might assign it 200 points.  This is a much too simplified way of explaining it (and I don't even know if I am reading it or guessing it right) but by considering elements that occur together, this increases the odds that a message is spammy.  The weakness, like any spam filtering service, is that if a spammer figures it out and omits certain parts then the filter can fail.  A malformed file attachment name is not enough to consider a message as spam.  Neither is a message that contains only an image, or an SPF failure.  However, what if a single message contained all three?  That changes things considerably.

We, ourselves, have also shifted our tactics in the past year.  I won't go into any details, but suffice to say we use a multi-tiered approach now.  Our spam filtering service is no longer simple - it is actually quite complex and getting more complicated as we go along.  A simple solution is insufficient, so beware any service that promises simplicity.  Both us and the spammers are smarter and the cat-and-mouse game continues.

Leave a Comment
  • Please add 5 and 7 and type the answer here:
  • Post
  • I'm not an expert, but I personally think the entire approach is wrong.  I think we need to move to the point where public e-mail addresses are the exception rather than the norm, and where a sender needs to ask permission to send you an e-mail (with proper authentication of the identity of the sender), and where inbound e-mails from non-white-listed addresses are not accepted by e-mail servers and clients.

    The current approach, however sophisticated and multi-layer, just becomes an "arms race," with no side ever "winning the war."  In order to win, you have to go back and solve the real root causes, rather than patching up the problems with inelegant filtering strategies on top of a fundamentally incorrect model that doesn't meet key requirements.

    But first, you have to recognize that our Internet e-mail system is seriously broken.  Any time you have 90% of messages being unwanted spam, that is a serously flawed system.  But I don't think that those who are able to fix the problem have yet come to the realization that the problem is more serious than can be solved by the YASFA (yet another spam filter algorithm) approach.

  • Hi Terry --

    'However, what if a single message contained all three?  That changes things considerably.'

    Worth noting that's been the core design of Apache SpamAssassin for the past 5 years now.

    'I won't go into any details, but suffice to say we use a multi-tiered approach now. '

    well, welcome to the fold ;)

    'Both us and the spammers are smarter and the cat-and-mouse game continues.'

    Yes.  BTW, an idea that hasn't gained enough currency yet, is that antispam differs from old-school text classification.  it's a new game entirely -- "adversarial classification", where the text you're classifying is written to evade your classifier.

  • Hi, jmason,

    > BTW, an idea that hasn't gained enough currency

    > yet, is that antispam differs from old-school text

    > classification.  it's a new game entirely --> "adversarial classification", where the text you're

    > classifying is written to evade your classifier.

    That's an interesting concept, and I think it relates somewhat to some of my posts on reputation-based filtering.  As you mention, spammers now have figured out that they are dealing with spam filters and are actively working to evade them.  Like you say, it makes sense to make attempts to classify the sender rather than the content of the message they are sending.

  • Hello, tzagotta,

    > I'm not an expert, but I personally think the

    > entire approach is wrong.  I think we need to move

    > to the point where public e-mail addresses are the

    > exception rather than the norm, and where a sender

    > needs to ask permission to send you an e-mail (with

    > proper authentication of the identity of the

    > sender), and where inbound e-mails from non-white-

    > listed addresses are not accepted by e-mail servers

    > and clients.

    Similar software for this has existed for quite some time, where email software, upon receiving a message from someone, sends an autoreply requesting a sender to authenticate themselves.  Similarly, you can set your email software to not accept anybody in your address book.

    The problem that I can see is do I really want to have to authenticate everyone who wants to send me something?  Imagine if everyone who wanted to telephone me had to first ask permission to do so.  I'd never get a call from anyone new.  On the other hand, 90% of my phone calls aren't spam, either.

    > The current approach, however sophisticated and

    > multi-layer, just becomes an "arms race," with no

    > side ever "winning the war."  In order to win, you

    > have to go back and solve the real root causes.

    In my view, the root causes are the following:

    1. Email is a cheap way to advertise

    2. Some people actually pay for these products (it's a numbers game, 0.1% of 1 million is 1000)

    3. Spammers can make money doing this

    If people stopped buying, spammers would have no motivation to spam.

    > But first, you have to recognize that our Internet

    > e-mail system is seriously broken.  Any time you

    > have 90% of messages being unwanted spam, that is a

    > serously flawed system.  

    I will grant you that.  

    > But I don't think that those who are able to fix

    > the problem have yet come to the realization that

    > the problem is more serious than can be solved by

    > the YASFA (yet another spam filter algorithm)

    > approach.

    It used to be that spam filters would pass through the signal and filter the noise.  Perhaps we ought to look for the signal and assume it's mostly noise on the channel.

  • >> It used to be that spam filters would pass through the signal and filter the noise.  Perhaps we ought to look for the signal and assume it's mostly noise on the channel. <<

    Maybe, but remember also that a spam filter that has "false positives" (real e-mails that end up in the Junk folder) is not helpful to the end user, because the end user ends up wading through all the spam messages to find the one or two that the filter incorrectly categorized.  And it is even worse if you are in a corporate environment where you don't have access to the Junk e-mails, because then the e-mail is "just gone."

  • <a href="http://dildo-pleasure.info/">Dildo</a>">http://dildo-pleasure.info/">Dildo</a> or [url=http://dildo-pleasure.info/]Dildo[/url] or http://dildo-pleasure.info/

Page 1 of 1 (6 items)