CEAS Spam Filter Bakeoff

The fourth Conference on Email and Anti-Spam (CEAS) is planning a bakeoff this year. In the CEAS 2007 Live Spam Challenge, the organizers hope to simultaneously inject a live stream of spam and legitimate email into several spam filters over a 24 hour period.

However, fair comparative testing of spam control technologies is extremely difficult -- by some measures, it's impossible. Because some promising filter techniques rely on examining the real-time behaviour of the sending machine, it proves tricky to provide the exact same stream of email to all the filters at the same time.

For example, some filters attempt to "fingerprint" the sending machine's operating system -- the idea being that, say, a Windows 98 PC has no business submitting email direct-to-MX. In a test that replicates an inbound email stream to several servers, it's tricky to allow the receiving filters to send IP packets back to the true originating IP address in such a way that is fair and equitable for all test participants.

In its defense, CEAS recognizes this difficulty by excluding greylisting from the list of permitted techniques. I'll be watching this one with interest.