Terry Zink: Security Talk

Discussing Internet security in (mostly) plain English

Interview with Yahoo spam chief

Interview with Yahoo spam chief

  • Comments 6

A few days ago, Yahoo antispam chief Mark Risher hosted a Q&A session with various users and answered their questions, both pre-submitted and live questions.  I thought I'd chime in and take some selected quotes from the session and add my own thoughts to Mark's.

dlippman: Why don't emails with the word "Lottery" and a few other Spam characteristics automatically go into my Spam folder?

Mark: I really wish we could! Catching a specific word is really hard. On the one hand, there are the risks that we’ll catch something legitimate -- “Campus housing lottery this Friday” -- which is what we call a “false positive.” (A “false positive” is any time our filters mistakenly mark something as spam when it isn’t).  On the other hand, if we build a filter for one specific word, there are often about a bazillion other ways the bad guys can spell it and still get their point across. Did you know there were 600,426,974,379,824,381,952 ways to spell \/!@g.r.a? (check out http://cockeyed.com/lessons/viagra/viagra.html)

Indeed.  Back when I was a spam analyst and actively writing and adjusting spam rules every day, we developed some rules of thumb.  Writing spam rules on single words is risky because there are lots and lots of times when supposedly spammy words can occur in legitimate circumstances.  What if a researcher at Pfizer was discussing the latest results of a Viagra test?

This comes up quite often in the sensitive word list.  Some customers want no x-rated words in their inbox and ask us "Why don't you block this word?  Or that word?"  The reason is that in American slang (at least), curse words are used in "legitimate" contexts so you can't block them outright.  Think about how many times you curse in real life, perhaps while doing some coding and can't figure out why your code doesn't produce the correct output...

On the other hand, spammy words that occur in longer phrases are much more suspicious and much less prone to FPs.  Blocking on "Lottery" in the subject line is risky, but blocking on "Win the Spanish lottery!" is much less so.  The rule of thumb in spam filtering is that the less generic the phrase, the more aggressively you can be in your spam weight/evaluation.

brip: Customer Care tells me that I need to forward with full headers if I’m reporting spam, but when I try to do that the headers are never there. What should I do?

Ryan: To forward with headers users have to take two steps. First you will need to reveal the headers for the message. In Classic you can look for a "Full Headers" link just below the bottom right corner of the message. In All-New Mail there is a Header dropdown just above the top right corner of the message.  Once you have exposed the full headers you can copy and paste them into the message as you are forwarding it.

This is something we also tell our customers.  Forwarding full headers is crucial to fighting spam.  Why?  Because the headers of the email tell us much that is not available in the body contents.  While we do write spam rules on body and subject content, the headers tell us the following:

  • Who sent the message (ie, who did they claim to be in the SMTP MAIL FROM)

  • What IP address sent the message

  • The route the mail traveled on its way to you (ie, intermediate hops)

  • Suspicious message headers... what does the Message-ID say?  What does the HELO say?  What do the Received headers say?

  • What did our own spam filter say?  Many filters insert x-headers into the message and receiving those headers let's us know what the filter already said so we know where to start from when we want to block the message.

In other words, there is a much richer set of content in the message headers than is available in the message alone.  Much of the time, simply forwarding a message to the abuse email address loses the message headers, rendering them much less useful to the spam abuse team.

Leave a Comment
  • Please add 7 and 7 and type the answer here:
  • Post
  • PingBack from http://blog.a-foton.ru/2008/08/interview-with-yahoo-spam-chief/

  • Terry, would you focus on spam from within and the reputation of the frontbridge outbound servers for a few weeks please?

    Received: from VA3EHSOBE002.bigfish.com (outbound-va3.frontbridge.com [216.32.180.16]) by localhost (TC-3.1.009); Thu,  7 Aug 2008 19:58:29 +0100

    Subject: Mail Confirmaton(Contact Mr Frank Lennon:franklennon07@hotmail.com)                                                                      

    From: "HARTON, BRENDA" <bharton devry.edu>                                                                                                        

    To: undisclosed-recipients: ;                                                                                                                    

    Attn:                                                                                                                                            

        In regards to our JULY online electronic draw.                                                                                              

    Be informed that  your email has been attached                                                                                                    

    Reference number IR/23/787/67. Which has subsequently won you 1,350,000 Million Euro.                                                            

    Courtesy The Irish National Lottery.                                                                                                              

    THE ONLINE COORDINATOR                                                                                                                            

    Contact:Mr Frank Lennon                                                                                                                          

    Via email: franklennon07@hotmail.com                                                                                                              

    FILL IN FOR CLAIMS WITH THE FOLLOWING INFORMATION.                                                                                                

    Name:..............................................                                                                                              

    Age/Sex:..........................                                                                                                                

    Address:..............................................                                                                                            

    Telephone:.........................................                                                                                              

    Copywright The Irish National Lottery 2008    

  • I'll look into this.  BTW, I did do a series on outbound spam earlier this year, back in January or February.

  • With regards to the fowarding full headers issue.  This is something that is continually difficult to explain and re-explain to users over and over again.  The problem is Outlook.  Unless they are configured to "forward as attachment" there is no easy way to forward full headers.

    Surely you have a tinsy ammount of pull there at the great goliath of the N.W.  Could you possibly weasle in a new feature suggestion for future versions of Outlook that make this easier for the end user?  Perhaps a "Forward spam to I.T." button or some such thing.  At worst, even a dedicated right-click entry would be better than the multiple steps required now.

    Relatedly: they finally made *viewing* full headers easier in OL2007 via right-click->Message Options... in the message pane.  But they should go further and have an option to always display full headers or an easy KB shortcut to display message source (full plaintext content including headers).  See Mozilla Thunderbird's <Ctrl>+<u> feature for enlightenment. :)  OL is a great calendar, collab tool etc... but it's an embarassment for email.  The Exchange team has done a great (albit slow) job over the last 8-10 years in getting the server side more standards compliant and in-line with best current practice in SMTP.  Please evangalize internally as much as you can to help your company bring Outlook up to par in the same ways with MIME and plaintext handling.

  • Following on from my previous post on the interview with the spam chief at Yahoo, I thought I'd respond

  • Jason,

    There is a spam plug-in tool for Outlook which installs a button on your Outlook toolbar that forwards messages to our abuse team with headers inline within the message.  It has some issues with Outlook 2007, though.  Another problem is anyone can download it, so we get submissions from everyone, not just our customers.

    You're right about the headers.  Thunderbird makes it so easy to see the source whereas Outlook is awkward to use.  You'd be surprised at how little pull I have within divisions that aren't my own.

Page 1 of 1 (6 items)