Hopefully the last on this subject

For the previous post in this series, go here first.

Microsoft is indeed the sole arbiter of who gets to send email to hotmail.com. As is Curt the sole arbiter of who gets to send email to monash.com, and I to richi.co.uk: My server; my rules.

If I chose to reject email from, say, Turkey, because I'm getting a bunch of spam from there, that's entirely my prerogative. I'm not forcing anyone else to reject email from their friends in Turkey.

As far as call-to-action filtering is concerned, I think Brightmail proved that a signature-based approach can be useful. Their approach includes CTA, but is a more general signature filter. It can be labor-intensive however, and suffers from a "zero hour" problem.

It's notable that the two major hosted anti-spam companies (MessageLabs and MX Logic) who use Brightmail use it in conjunction with their existing techniques—not as a replacement. In addition, Symantec (Brightmail's new masters) use the TurnTide IP-level throttling technology in front of Brightmail in their appliances. From accounts, this means that a substantial chunk of the inbound spam never gets as far as the Brightmail filters. This makes perfect sense for the reasons I outlined here.

Ultimately, if Microsoft screws this up and drops too much mail, the market will decide that Hotmail's had it's day. For all its faults, Microsoft's not that stupid. My "faith," if that's the right word, is not about "infallible wisdom and technical perfection," it's about revenues from advertising, partnerships, and the many other ways that MS derives a revenue stream from MSN, direct and indirect. Hotmail ain't a charity any more than Gmail, Monash Information Services, or Richi Jennings Associates are.

Tags: spam, Hotmail, SPF, SenderID, Sender ID.

Yet more on Hotmail's move

If you've not read them yet, you might first want to look at these previous posts:

• The Evil Empire of anti-spam -- Microsoft's Project 1984
• Hotmail Raises the Bar for Senders
• More on Hotmail's move
• Antispam -- focus on the message, not the messenger!

At the risk of perpetuating a good ol' USENET-style flamewar (ad hominem attacks, and all), here are a few thoughts about Curt's latest.

blacklists will inevitably include people who shouldn't actually be on them

Yep. If we look at the sometimes-sorry example of DNS-based RBLs, blacklists (or blocklists, if you must) can cause collateral damage by including too broad an IP netblock, or by including IP addresses that have been reassigned from spammers to legitimate senders. This is one reason why reputation services based solely on IP addresses are a problem. SPF/SenderID/DKIM are a basis to make reputation services that track the senders' behaviour, not their IP addresses.

A good antispam system has 50,000+ rules. To say that there's one rule which is merely a contributing factor like the other 50,000 isn't worthy of an AP story or a press release or an entire Ferris Research implementation report

Microsoft has spent a bunch of time and effort talking about this recently precisely because they want people to know that the ruleset for Hotmail will be changing. I've talked to Craig Spiezle twice about this over the past month. BTW, the "entire" report, authored by Josh, is definitely one of the shorter reports that Ferris Research has published, and I understand has received good feedback from IT customers.

Yes, Microsoft wants people to publish SPF records. That doesn't constitute "forcing SenderID down people's throats." Curt believes they're wasting their time and ours. He's entitled. I disagree.

I believe that antispam filters focusing entirely on the "call to action" can and do get most of the job done with negligible false positives ... I must confess that my opinions are based mainly on research that's slightly over a year old

My take on this: I agree that CTA filtering did indeed seem like a compelling content filtering technique about a year ago. Several vendors made a "big thing" of how it was going to simplify life enormously. It's notable how it's just not talked about now. Perhaps the industry found a gap between "research" and "real life"? Certainly, gathering and acting on the CTA data is very resource-intensive and time-sensitive. A common theme in spam filtering is that there is no one, single, silver bullet to fix this problem. Not CTA, not Bayesian, and certainly not challenge/response.

Examples of the techniques employed at the first stage:

• Valid HELO or EHLO?
• Valid PTR or RDNS?
• Greylisting/tempfailing
• Throttling (prevents illegal pipelining)
• IP reputation/blacklists
• SPF/SenderID/DKIM

More broadly, there's a general problem with content filtering: it's expensive. In a world where 70-90% of the port 25 connection attempts are unwanted, we don't want to be wasting MTA horsepower on receiving the message and performing complex analysis on the content. Moreover, we want to be able to reject the message with an SMTP 5xx code—this allows us to avoid the collateral-damage causing "backscatter" of bounce messages. This means that we need to keep the connection open while we run the rules, which isn't pleasant.

OK, this post is way too long already, and I'm not being paid to write this ;-) To sum up, spam filters are increasingly running an initial set of anti-spam rules at the connection level, before the SMTP DATA transaction even starts. If these rules generate a high enough score, it's 5xx no spam for you, and goodnight Vienna. Only if the filter's unsure will the message make it to the second, content filtering stage. Adding SPF presence checks to the existing SPF rule allows Hotmail and others to reject more spam without expensive content filtering. This shouldn't cause any additional false positives, unless Hotmail does something dumb with the score weights.

More on Hotmail's move

Dr. Curt said, "If the proposed rule at Hotmail were simply that certain limited KINDS of email wouldn't be let through without sender validation -- that I could buy into."

The proposed rule at Hotmail is simply that certain limited kinds of email wouldn't be let through without sender validation. The difference between the current situation and what is proposed from November is that the spam score will be increased somewhat if the sender doesn't have an SPF record. Spam filtering is never a black and white proposition, but based on a score, derived from a weighted sum of several tests' scores. This change adds a new test (actually, strictly speaking, it enhances an existing test).

Yes, this encourages people to publish SPF records. This is a good thing.

No, it doesn't require people lube up their throats to have SenderID rammed down them. That sort of talk just smacks of Microsoft-bashing.

Note that the difference between SPF-classic and SenderID is (mostly) the PRA algorithm. This has problems: mostly political problems, but also technical problems at the corner cases. SPF-classic also has (different) corner-case problems. So did RMX, so does DKIM.

The fact remains that "the industry" broadly agrees that we need strong sender reputation tracking. Right now, all we can do is track the reputation of IP addresses, which is by no means perfect. SPF, SenderID, and DKIM will help us track senders' reputation, by greatly reducing the number of email messages with forged senders.

AOL will make this move too, I'll put money on it. Some commercial products already do (but it's usually configurable). It's the way the industry's going. Get an SPF record and get used to it.

This is so true

Back in the days of the VisorPhone (the predecessor to the Treo line), Handspring added a physical switch to put the phone into "silent" mode. Such was their inexperience with "conventional" mobile phones, that the designers didn't realise that a physical ringer switch was a revolutionary idea. More at TreoCentral.

A brilliant idea - pass it on

Paul "Hopalong" Makepeace links to this great idea: add an ICE (In Case of Emergency) entry to your mobile phonebook. Make it so. And pass it on.

Hotmail Raises the Bar for Senders

Starting in November, Microsoft's Hotmail will extend its checking of SPF records for senders of incoming email. This will raise the bar for email senders in Hotmail's quest to rid its users' inboxes of spam.

Currently, absence of an SPF record does not lead Hotmail to believe that the message is any more or less likely to be spam. From November, Hotmail will see the absence of a record as a sign that a message might be spam.

This move illustrates the steady progress that the industry is making towards a world where email "senders" can't be forged. Stamping out forgery will give us a firmer foundation for monitoring the reputation of senders.

It also highlights the need for domain owners to publish SPF records as soon as possible. On the one hand, 25% of mail using SPF is encouraging, what about the other 75? Do you own a domain? Publish an SPF record already! Ferris Research recently wrote a report showing how to do this.

Expect other email providers such as AOL, Earthlink, and Gmail to follow suit. They will no doubt discuss this at the Email Authentication Summit on July 12th in New York.

Sabre releases AJAX library

Sabre (yes, the travel reservations people, sibling of American Airlines) has released an open source AJAX library, Rico. It's super-cool.

AJAX? Asynchronous JavaScript and XML. The stuff that makes Google Maps and the Scalix web client so cool.

Via A Saber Geek.

Testing w.bloggar

I'm trying out a new offline blogging tool, wbloggar.com. Let's see if this works...

Longhorn error reporting is too chatty?

As I mentioned in today's IT Blogwatch, it seems that Longhorn's error reporting tool tells Microsoft a lot more than it used to. My take: of course it does, it's in beta! Did you really think privacy advocates will let Microsoft get away with this sort of thing in the released product?

Categories: Longhorn, privacy.

Malformed Office docs hide Trojans

David and I talked to MessageLabs' chief architect Paul Fletcher at INBOX/IT before I swanned off on hols. (Yes, for those HP Pinewood or OpenMail old-timers, that Paul Fletcher.) He was talking about a disturbing new trend for high-placed individuals to be carefully targeted with malformed MS Office documents, which contained a Trojan horse. Fiendishly clever.

Seems like the NISCC has figured this out, too.

Categories: spam, virus, malware, NISCC.