Terry Zink: Security Talk

Discussing Internet security in (mostly) plain English

Spam is solved, we can all go home now

Spam is solved, we can all go home now

  • Comments 10

The NewScientist has an article on an interesting new antispam technique.  Here’s an excerpt:

SPAMMERS' own trickery has been used to develop an "effectively perfect" method for blocking the most common kind of spam, a team of computer scientists claims.

Most of the billions of spam messages sent each day originate in networks of compromised computers, called botnets. Unbeknown to their owners, the machines quietly run malicious software in the background that pumps out spam.

Researchers have now come up with a system that deciphers the templates a botnet is using to create spam. These templates are then used to teach spam filters what to look for.

The system, developed by a team at the International Computer Science Institute in Berkeley, California, and the University of California, San Diego, works by exploiting a trick that spammers use to defeat email filters. As spam is churned out, subtle changes are typically incorporated into the messages to confound spam filters. Each message is generated from a template that specifies the message content and how it should be varied. The team reasoned that analysing such messages could reveal the template that created them. And since the spam template describes the entire range of the emails a bot will send, possessing it might provide a watertight method of blocking spam from that bot.

To test their idea, the team installed a previously captured software bot onto a machine. After analysing 1000 emails generated by this compromised machine - less than 10 minutes' work for most bots - the researchers were able to reverse-engineer the template. Knowledge of that template then enabled filters to block further spam from that bot with 100 per cent accuracy.

Knowledge of the spam template enabled filters to block further spam with 100 per cent accuracy.

High accuracy can be achieved by existing spam filters, but sometimes at the cost of blocking legitimate mail. The new system did not produce a single false positive when tested against more than a million genuine messages, says Andreas Pitsillidis, one of the team: "The biggest advantage is this false positive rate."

So, to summarize, a team of researchers downloaded and installed some software that flips a computer into a botnet.  This bot then started spewing out spam and the team was able to capture the spam, analyze, and then write spam rules in order to 100% target the spam run. 

All you have to do is download the malware, capture the spam traffic, and then use the traffic to build an antispam corpus of rules.  In other words, it’s the next step in doing what antispam vendors have been doing since 2002.

In case you can’t tell, I’m not really all that impressed with this spam solution.  Yes, it does have a 100% accuracy rate with no false positives.  But how practical is it in real life?

  1. You have to capture malware from every botnet – There are lots of different botnets out there, not just one.  In order for this solution to be effective at stopping all spam, you would need to capture each type of malware and analyze the spam traffic from all of the botnets, not just a single one.  Different botnets have different spam signatures.

  2. You have to capture multiple versions – Malware from botnets, the more intelligent ones, are auto-updating.  They periodically phone home and upgrade themselves.  And they may not send out traffic in the same ways.  You would have to ensure that the software that you have intercepted is capable of analyzing traffic from versions of botnets that send out spam differently.

  3. Botnets do not just send out spam by themselves – Not all botnets spam.  Some of them break CAPTCHA’s set up by Windows Live (Hotmail), Yahoo and Gmail.  And then, they send out instructions using those compromised accounts to spam from them.  Thus, even if these botnets were intercepted in terms of traffic, they wouldn’t solve the spam problem since botnets have multiple uses.

  4. Botnet software is competitive – Some pieces of malware will erase other pieces of malware in an attempt to monopolize the botnet space.  So, if you have installed one piece of malware, another piece can come and erase it.  You’ll be attempting to capture traffic that doesn’t exist.

Still, this technique is a viable antispam measure if you can capture malware and install it; however, one would need to understand that it is but one tool in the antispam arsenal.  It would have to be supplemented with other techniques like IP reputation and sender reputation.  As to how practical it is, well, I can’t comment on that because I don’t understand botnet malware very well.  But the idea is interesting.

Leave a Comment
  • Please add 8 and 6 and type the answer here:
  • Post
  • One would assume that the steps of installing the botnet software, analyzing it's output and reverse engineering the spam templates is a job for anti-virus companies which then sell us a piece of software that stops spam with 100% accuracy.

    I see nothing impractical about such a solution, I'd buy it today if it was available.

  • Indeed, one would /assume/ that would be the case, but it's not. Also, it's not as practical as it sounds. Currently, this method is being applied by virus analysts that then update their database with the virus signature they discovered, which is rented to anti virus companies. Why this isn't practical is because they can apply their malicious techniques for cloaking even easier than it was with viruses (not saying it was impossible however, google for StormNet and find out why). The programmer of the botnet can easily create a randomizer that generates a random string and lets the email content/body consist of that string. When that happens, the only way to catch spam then is to check the email for certain links and text, but alas, that's the technique being applied right now, rendering this entire new technique to combat spam completely useless.

  • Mark is correct in spirit. However, you can take the idea one step further along the path of increased power and versatility. It should be possible to make the analysis process distributed rather than centralized.

    Essentially every computer with this sort of antivirus installed would work to gather and compute information about the templates it has encountered and then share that information with other clients. The communication could utilize a central server or, to continue the distributed trend, a peer to peer system.

  • As always, the theory is nice but the reality is far different.  To get around this technique, spammers need only to start with a template from a valid email - a template from a large, common, legitimate mailing list.  The false positive rate would then shoot high enough that nobody would use this method of spam filtering.

  • Waylon said:

    "Essentially every computer with this sort of antivirus installed would work to gather and compute information about the templates it has encountered and then share that information with other clients."

    So - in short - this anti-virus software becomes a botnet???

  • Is this not a version of the obvious technique? Know what a spam bot or virus would do and block/prevent it.

    May be over the time some heuristics will be developed and a more effective solution may evolve.

    But that just means that the spammers will be paused with a tougher challenge. At least some of them would break through, pausing a tougher challenge for anti-spam :)

    Is this what we should call a 'vicious circle'?

  • A botnet to strangle botnet's. Interesting, but this would simply create and endless war between botnet companies and this anti-botnet software which would probably end up being sold to one of the big anti-everything security companies (anti-piracy, anti-malware, anti-productive). There is nothing slower at reacting as a bloated company, and nothing more agile than a bunch of renegade software geeks still trying to get back at Gates for taking software mainstream.

    From a fellow UC student, congrats to those at Berkley and San Diego for turning some heads and putting forward hard work on this project. These are some positive strides in the right direction.

  • i seem to recall a article a few years ago which went into how to eliminate spam. I think it was by Davey Winder (though i may be wrong).

    the details were really simple, get the email servers to send the headers for the messages.  when the email is opened by the recipient, the mail server retrives the message from the server.  if it is spam, then the message body will not be there and the email server will report to the reader that the message could be spam.

    i've had a trawl but cannot find the original article, i think it was in PC Pro here in the UK.

  • INSANE !!!

    Please, you can block spam from a host or from an ip, so they change, and you're chasing your tail.

    THE ONLY WAY TO STOP 100% OF SPAM IS TO REGISTER EACH AND EVERY LEGITIMATE EMAIL SENDING (SMTP) SERVER.

    CHECK OUT MY PLAN ON MY WEBSITE.

    AND THE ONLY WAY TO DO THIS IS FOR ALL OF US 'EXPERTS' TO GET TOGETHER AND DEMAND THAT OUR GOVERNMENT MAKES IT HAPPEN.

    I HAVE AN INTERNET EMAIL PATENT, I HAVE BEEN WORKING WITH E(LECTRONIC) MAIL FOR OVER 30 YEARS.  I AM AN EXPERT AND MY WAY WILL WORK.

    ALL THIS OTHER NONSENSE IS JUST SOPHISTRY.

    and pardon me for shouting...

  • Who needs a template if the E-mail service suppliers notice a few hundred notes are sent within a minute from the same IP address? OK, it won't stop the botnets from spitting out garbage, but it would be a way to block the spam until the botnets re-initialize your IP address with each note.

Page 1 of 1 (10 items)