Terry Zink's Cyber Security Blog

Discussing Internet security in (mostly) plain English

Can a computer fight spam better than a human?

Can a computer fight spam better than a human?

  • Comments 8
In this blog, I sometimes try to point out the similarities between stock trading and spam fighting.  I happen to trade stocks (but never take tips I receive by email, not even in The Motley Fool) and I find that the two activities have things in common.

In trading, their are two types: discretionary trading and system trading.  System trading follows a mechanical system (trading based on indicators, fundamental criteria, etc) whereas discretionary trading has an element of subjectivity.  While system trading is based on mechanical signals it is still a reflection of the designer of the system.

The debate that comes up is whether or not a computer can trade better than a human can.  As a trader, I make use of technical patterns and I have read some books that I believe gives my system an edge.  I use many of the classic technical patterns, but I am aware that many of them don't work that well.  I also started learning some new techniques that are based upon reversal patterns (swing trading - buying on weakness and selling on strength).  These are all patterns that have a great deal of subjective elements built in; I trade based on looking at a chart and viewing the trend.

The question in the trading community is whether or not a computer can find better patterns than a human can.  Can a computer find correlative elements to make trading profitable?  The big problem with the stock market is that the market is made up of millions of people.  For a computer to find an underlying order in the market would mean it would have to organize and predict human behaviour - not an easy task.  Perhaps a Bayesian stock picking service is in order.  Still, the concept is intriguing.  Perhaps sometime in the future computers will do all the trading and put guys like me out of business.

But that leads me to spam fighting.  Can a computer fight spam better than a human can?  Can a computer learn to determine what is spam and what is not?  Right now, humans are the ones that ultimately classify spam and computers mimic that behaviour by classifying subsequent messages that share elements of the pre-classified spam.  However, afaik, a computer never classifies a message it has never seen before.  In my next post, I'm going to examine whether or not a computer could ever learn to recognize spam the way a human could. 
Leave a Comment
  • Please add 7 and 7 and type the answer here:
  • Post
  • Why do you suppose a computer HAS to classify spam as a human does? The problem of spam is not one of classification, it is one of AUTHENTICATION. The problem is that you simply cannot tell FOR SURE who sent you a piece of email, except by examining its contents. That, in my opinion, is the fundamental problem of spam -- its not some sort of "AI" problem, it is a simple authentication problem.

    The SMTP protocol is, in my opinion, fundamentally flawed because it DOES NOT include any inherit authentication mechanism. The sender can put any old address in the "From" line he likes.

    The "solution" to spam, therefore, is not to make "smarter AI." It is to drop SMTP and come up with a new, more secure protocol. Of course, coming up with such a protocol is fairly easy. Making it as ubiquitous as SMTP is the hard part!

  • I think Dean is right, e-mail needs some authentification.

    Another way would be to charge for emails to make bulk mailing inattractive.

    And even humans make mistakes regarding spam: valid newsletters might be regarded as being spam if you don't remember subscribing to them.

  • Can a computer denote permission? It's certainly an important measure of what makes something spam or not. Problem is, I have yet to find a bayesian filter that can tell whether or not I willingly gave my email address to Macy's or Minnesota Public Radio.

  • Hello, Dean,

    > Why do you suppose a computer HAS to classify spam

    > as a human does? The problem of spam is not one of

    > classification, it is one of AUTHENTICATION.

    I tend to view authentication as a shortcut towards classification.

    For example, if we know that a sender has a bad reputation, we don't need to authenticate, we can reject mail outright.

    Even if a sender is authenticated, it still doesn't tell me whether or not the content is legitimate.  

    Take a gray-hat mass emailer.  I might know the mail is coming from them, it may even be authenticated but that doesn't mean I want to see it.

    > It is to drop SMTP and come up with a new, more

    > secure protocol. Of course, coming up with such a

    > protocol is fairly easy. Making it as ubiquitous as

    > SMTP is the hard part!

    I agree a more secure protocol is important but ultimately the judge of what I do and do not want to see in my inbox is me, or rather, a computer that is very good at figuring out what I do and do not want to see.

  • > Can a computer denote permission? It's certainly an

    > important measure of what makes something spam or

    > not. Problem is, I have yet to find a bayesian

    > filter that can tell whether or not I willingly

    > gave my email address to Macy's or Minnesota Public

    > Radio.

    Perhaps a custom-Bayesian filter that looks at your historical opt-in lists would be able to figure out, and assign a probability, of whether or not you willingly did it.

    It could sort of be like the movie "Click".  It would remember previous settings.

  • > Take a gray-hat mass emailer.

    With proper authentication, there would be no such thing as "gray-hat" mass emailers. You either subscribe to a mailing list, or you do not. If you did not subscribe, then you don't want to see it.

    (Note: Yes, you still need to allow "unsolicited" email, but it is possible to tell the difference between unsolicited *one-off* email and unsolicited *bulk* email WITHOUT looking at the contents. Once you can do that, filtering based on content is not important any more.)

    I think you're still thinking in terms of the way email works right now -- I'm saying that the way email works right now is fundamentally wrong.

  • > With proper authentication, there would be no such thing as "gray-hat" mass emailers.

    Don't we already have authentication right now in the form of SPF/SenderID/Domain Keys?

    Perhaps you are saying that rather than having these authentication procedures optional, they will be mandatory?

  • > Perhaps you are saying that rather than having these authentication procedures

    > optional, they will be mandatory?

    Correct. For example, if SenderID were mandatory, botnet spam houses would not be able to work -- you can't add 10,000 entries in your DNS record for the 10,000 IPs addresses you "own". Besides, there's a nice paper-trail for the FBI to shut you down ;)

    Anyway, SenderID isn't all that complicated and there are plenty more authentication schemes that can help.

    And as I said above, the problem is not coming up with such mechanisms, its actually MAKING them mandatory ;)

Page 1 of 1 (8 items)