Some of this stuff I couldn't make up if I tried...
With all the hoopla about the David Ritz case (which I will blog about in a future post), I thought I'd remark about a very strange legal requirement about filtering mail. As usual, this unreasonable legal requirement only applies to the EU.
In the EU, you cannot filter mail by inspecting its content.
I am not making that up. When I heard that, I said "Are you serious? How are you supposed to filter mail?" For goodness sakes, by definition, email filtering is based upon content inspection. Apparently, you can only filter mail by doing IP blocking and other high level techniques without actually inspecting the content (I guess also doing SPF checks and whatnot, but I would think you would need some content, namely the MAIL FROM, on which to do that). Now, spam filtering companies have a provision in that we are doing it on behalf of our customers, that is, we are doing it because they want us to do that.
Now you may say "We are using automated techniques to do spam filtering and there is no manual inspection." That actually makes it worse. Using automated techniques to inspect content makes regulators and privacy commissioners feel more uncomfortable about the data is being used, rather than more at ease. Presumably, their point of view is that an automated technique can be more easily used to harvest and extract information. They are really big about protecting PII (Personal Identifiable Information) over there. Too bad they have no clue about the way the email world actually works.
PingBack from http://msdnrss.thecoderblogs.com/2008/01/29/strange-legal-requirements-2/
As long as you don't look at the body of the message, you should be fine.
90%+ accuracy can be achieved by looking at envelope information alone.
Looking at headers for malformed headers is useful too, but a case could be made for this being content checking.
Rejecting based on the SMTP envelope works quite well.
You may want to have a look at "Working Party 29 Opinion 2/2006 on privacy issues related to the provision of email screening services" http://ec.europa.eu/justice_home/fsj/privacy/workinggroup/wpdocs/2006_en.htm
WP 29 is the independent European advisory body on personal data protection.
To summarize the document, I'd read Presentation H of http://www.inbox-outbox.com/Default.asp?page=29 "the screening of emails for the purposes of filtering spam".
WP 29 gives ISPs/ESPs material to legally support the use content filtering in their systems (Article 4, Actirble 7b, Article 10).
The only problem is with false positives. EuroISPA.org representative clearly pin-pointed this problem in Inbox/Outbox conference held in November last year, and stated that ISPs need a "clearer non-liability statement" and that "Good faith spam filtering to protect users should not attract liability".. (Session L presentation http://www.inbox-outbox.com/Default.asp?page=29 )
"90%+ accuracy can be achieved by looking at envelope information alone."
That would be more like 9%+ accuracy.
Any sender can be a bot. You need to look at URLs in the body and see if the sites are served by known spam servers.
While it does sound crazy I agree that inspecting the headers goes a long way. However, spammers are getting more and more sophisticated so looking for keywords and urls in the body will become more and more critical to blocking them.
Yahoo continues filtering by inspecting the headers. Here's Yahoo putting legitimate mail from Yahoo into a spam box instead of inbox, because Yahoo's mail server knows that Yahoo's mail server is a spam sender.
X-Apparently-To: firstname.lastname@example.org via 18.104.22.168; Fri, 01 Feb 2008 20:46:44 +0900
Received-SPF: none (n9.bullet.re3.yahoo.com: domain of email@example.com does not designate permitted sender hosts)
Authentication-Results: mta148.mail.tnz.yahoo.co.jp from=yahoo.com; domainkeys=pass (ok)
Received: from 22.214.171.124 (HELO n9.bullet.re3.yahoo.com) (126.96.36.199)
by mta148.mail.tnz.yahoo.co.jp with SMTP; Fri, 01 Feb 2008 20:46:44 +0900
Received: from [188.8.131.52] by n9.bullet.re3.yahoo.com with NNFMP; 01 Feb 2008 11:46:43 -0000
Received: from [184.108.40.206] by t5.bullet.re3.yahoo.com with NNFMP; 01 Feb 2008 11:46:42 -0000
Received: from [220.127.116.11] by t2.bullet.sp1.yahoo.com with NNFMP; 01 Feb 2008 11:46:42 -0000
Received: from [127.0.0.1] by omp405.mail.sp1.yahoo.com with NNFMP; 01 Feb 2008 11:46:42 -0000
Received: (qmail 35855 invoked by uid 60001); 1 Feb 2008 11:46:42 -0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
I have been bombarded with all sorts of these frauds comming both in email from different sources and spam.
Heres what i found firstname.lastname@example.org, main email address from those person sending, also i have found out that if one visit or registers on for example date finding sites, or finding soul mate, etc etc, they have males and females that have created accounts on yahoo with valid account name , if you email them and start conversation all of a sudden you get hit w/ some emails that they need money and if you send x amt to some name of someone in nigeria lagos, accura.. this is fascinating because today only i have received 40 in less then 2 hours online w/ these frauds.. WOW!!!!
nice one :D