So, according to one Peter Brockmann, challenge/response (C/R) spam filtering is a wonderful thing, and beats all other anti-spam techniques into a cocked hat.Huh? What? How did he come to that conclusion?
I've beaten the "C/R filters are a terrible idea" meme to death, as have many others, so I'm not going to repeat all that here. If you're new to the arguments, take a stroll through these posts (perhaps you should work from the bottom up).
But I was about to write about Peter's methodology. However, it would have been an identical post to the one Justin Mason wrote -- he beat me to the punch. So here are Justin's money quotes:
The “Spam Index” is a proprietary measurement of spam filtering, created by Brockmann and Company. A lower “Spam Index” score is better, apparently, so C/R wins!I too took a close look at his methodology. It is really, really, horribly biased in favour of C/R. Unbelievably so. By orders of magnitude, arguably.
...
However — there’s a fundamental flaw with that “Spam Index” measurement, though; it’s designed to make C/R look good ... The “Spam Index” therefore considers a false negative as about as important as a false positive. However, in real terms, if a user’s legit mail is lost by a spam filter, that’s a much bigger failure than letting some more spam through. When measuring filters, you have to consider false positives as much more serious!
...
[And] the situations where C/R fails are ignored. Is it any wonder C/R wins when the criteria are skewed to make that happen?
The idea is that one can come up with a neat "score" for the performance of a spam filter -- of course, the exact composition and weighting of such a score can sway the results in any direction one chooses.
Statistics aside, asking C/R users if they're happy isn't the be-all and end-all of anti-spam research. C/R users may indeed be happy -- happily unaware that their spam filter is sending spam by replying to innocent third parties who's addresses have been forged by spammers.
(As an aside, I note with amusement that Peter mis-categorizes Commtouch and IronPort as DNSBLs -- which he calls "RBLs", so perhaps Trend Micro should whine at him about trademark infringement.)
So what's going on here? I first came across Peter earlier this month, when I noticed some rather odd edits to the Wikipedia page about Challenge-response spam filtering made by one Pjbrockmann. The edits did rather deviate from Wikipedia's prized "neutral point of view" (NPOV). I also noticed a sneaky link back to his site from the page: naughty-naughty (as a great philosopher once said).
So, let's check out brockmann.com. The About page says, "Brockmann is a Wikipedia contributor." Well, golly, so he is. (Perhaps I should add that to my puff piece too.) His Wikipedia contributions extend to being dinged twice in April and June for spam and non-NPOV (the more recent issue noted above would make it three). Not so great.
Justin alleges that Peter has a relationship with Sendio. I don't know about that, but I do see he also mentions SpamArrest as an example of C/R. But does this (presumed) relationship stop him being objective? As Steve Hunt says, it, "Depends on what you mean by objective":
We are all mere mortals, and my own personal preferences will be very clear in the posts. Actually, my personal preferences and biases pay the bills ... Does that make me less than objective? I don't think so, but use your own judgment ... I commonly won’t expose which vendors I’ve helped because – frankly – it’s none of your business. It doesn’t change my ability to speak frankly and truthfully, and you might look at the list of companies and assume some bias that really doesn’t exist.I like how Steve puts this, but I differ from Steve and Peter in that my personal preference is to maintain a list of clients in public (it's not a complete list, mainly for reasons of confidentiality -- e.g., when I've worked on expert witness contracts). So I guess you might look at that and, "Assume some bias that really doesn’t exist."
But, as an independent adviser/analyst/consultant, I also hope that you'll find that what I have to say is actually true.
7 comments:
Interesting blog. Thank You. I was interested in learning of Peter's methodology when he released his report on the 17th. I was somewhat suspicious. So, I attempted to register on his web site in order to download a copy of his report. I'm still waiting for a response, who knows maybe his acceptance e-mail was justifiably intercepted by my spam filter.
Richi: Thanks for the promotional writeup.
I think you should read the report before you flame it.
520 business people provided details on their email experience including the anti-spam technology class in place in their enterprise. The Spam Index includes time spent dealing with spam, including time spent looking for good email in the junk folder, something that most other methods (% caught, % false-positives) don't consider. You can try it out here where I've automated it as a javascript page.
The goal of the Spam Index is to give users a tool to:
a. compare performance over time - things better/worse this month versus last month?
b. compare experience - how does my performance compare to others?
c. compare technologies - are there any that provide a better experience than others?
d. is there a business reason to invest in anti-spam?
The Spam Index Report and The Problem with Email report address these questions.
I would think that listening to the views of business users can be humbling to all but the arrogant (who are only humbled by listening to themselves). I really don't think you can or should dismiss the facts so quickly:
* Challenge-Response reduces the spam arriving in peoples inboxes.
* It reduces the time they spend dealing with spam.
* It increases user satisfaction with the email experience.
What more do users really want from an anti-spam solution?
BTW - anybody can be a Wikipedia contributor; it's only by editing do the contributors and the wiki itself improve.
-- P
Peter, I feel my post already adequately covered the fallacy of judging C/R by asking its users if they're happy.
I presumed you would be aware that I've spent several days studying your paper, its methodology, and the demo page. I even had to register on your site to access the report. I believe Justin is in a similar position.
I also presumed you would be aware that I, like many many others contribute to Wikipedia. However, I prefer to allow other editors to decide if my reputation is good enough to permit Wikipedia links to my site.
Arrogant? Dissmissive? Moi? Those who know me would probably nod in agreement. Still, I've only been doing this funny email stuff since 1985. What do I know? That's for others to judge.
[Moderation note: I have removed a somewhat ad-hominem portion of this comment]
P
Your "Spam Index" test is a joke.
Dude. Grab what's left of your cedibility and go back to what you know. Spam analysis obviously isn't your area of expertise. TBH it's hard to know what your area of expertise is given that your had like 6 jobs in the last 3 years. Seriously dude. Pick your battles. This ain't one of them!
Richi,
Keep fighting the good fight against C/R. As the holder of a domain name frequently forged into the From: or Reply-To: fields of spam, I can testify for certain that it doesn't work. In fact, whenever I receive a challenge to one of those forged addresses, I make sure to reply to it to make sure the spam gets through. Petty, perhaps, but I'm not being paid to filter C/R users' spam, so I'll pass it through.
Forgot to note in reply to Peter Brockmann's "facts" - here's the one that's the kicker:
* It makes Challenge-Response users into spammers themselves.
In my opinion, throwing that into the mix outweighs any number of other facts. Perhaps he should try telling hia "business users" about that and see how their replies change.
Post a Comment