Internet and e-mail policy and practice
including Notes on Internet E-mail


2006
Months
Apr

Click the comments link on any story to see comments or add your own.


Subscribe to this blog


RSS feed


Home :: Email


24 Apr 2006

Better automated abuse reporting with ARF Email

Since we know we're not going to find a FUSSP any time soon, anti-spam efforts are concentrating on incremental efforts to make the current mail system, messy though it is, work better. Dealing with abuse reports is a particularly messy and labor-intensive area that desperately needs more automation.

When an ISP or other mail provider receives an abuse report, they typically do triage into categories such as needs immediate attention (phish, child porn or other very illegal content), discard (false alarms due to forged headers, abusive rants), and eventual attention (everything else). Then they deal with the messages in each category. Some providers still have people handling all of abuse mail that comes in, but for any but the smallest that is far too slow and expensive. More likely they do the triage automatically by ad-hoc pattern recognition, looking for various keywords to figure out what kind of report it is, and looking for the Received: headers their mail programs produce to identify copies of mail they sent. For spam reports, if the automatic process can figure out the IP address or the sender, they just count the report and act when there are enough reports for a given IP or sender to indicate trouble. Often, if the message can be recognized as one from a mailing list, it removes the recipient from that list. Needless to say, this process is rather approximate and can make a lot of mistakes.

To make this process less approximate, an ad-hoc group including representatives from some of the largest ISPs have defined a simple common format for abuse messages known as ARF (Abuse Report Format.) Building on existing practice as all the best standards do, an ARF message is a kind of multipart MIME message with three parts.

The first part is unstructured text not intended for computers to interpret, typically a note in case a live person reads the message. The second part is a series of lines in a format resembling mail message headers, a keyword followed by a colon and the corresponding value, intended to be decoded by computers, but still easy for people to read. The third part is a copy of the message about which the message is complaining (or as much of it as the recipient feels like sending back.)

Some fields in the second part such as Original-Mail-From:, Original-Mail-To:, Received-Date:, and Source-IP: contain information that was parsed out of the original message or logged in the incoming SMTP session. The Feedback-Type: field describes the type of report. It usually is abuse to report spam or other e-mail abuse, but it can also be fraud or virus. At the request of bulk mail providers, it can also be opt-out or opt-out-list, with a corresponding Removal-Recipient: field to say who to remove, but unless the recipient network has some reason to believe that the message is from a legitimate mailing list and some reason to do the sender's list management, those are unlikely to appear. The set of fields is intended to be extensible and will probably grow as ARF becomes more widely used.

AOL and Earthlink can provide ARF reports for their feedback loop recipients, and it's spottily used other places. If someone is already using an automated or semi-automated tool to send abuse reports, it's not hard to make it send ARF. I have some perl scripts that send out abuse reports from my network, and it took under half an hour to change them from the previous ad-hoc text format to ARF.

While ARF won't do much on its own against spam, it should help cut down on bogus "this wasn't from us" responses due to triage staff who misread reports. Also, since ARF is intended to be sent and received automatically, it makes it more likely that when a spam run hits a spamtrap and provokes an automated report, that the report will be handled as soon as it's received, and maybe can even stop the spam run while it's in progress.


posted at: 01:08 :: permanent link to this entry :: 3 comments
posted at: 01:08 :: permanent link to this entry :: 3 comments

comments...        (Jump to the end to add your own comment)


I wonder if anybody has prototyped any tools to generate these reports from a piped message. I'd gladly give up my hacked abuse-reporting perl script.

My largest concern with this approach is those ISPs that refuse abuse reports with any attachments, even innocuous message/rfc822.

(by chip 24 Apr 2006 12:44)


Reporting scripts
I have my own hacked perl script. Like I said, it was really easy to adjust it to send ARF. It was just a matter of putting the MIME lines in the header, MIME separator lines between the parts, and a new structured part with the ARF info between the text complaint message and the copy of the spam.

It's true, some clueless ISPs may still reject anything with an attachment, but I haven't seen that for a while. What I see more often is people saying "send me the message headers" because their MUA formatted the enclosed message for display and hid the headers as it does with any other message.

(by John L 25 Apr 2006 18:08)



John,

I've been a tad out of the loop since moving from Bremerton to Dallas to take my new job. It's good to see you promoting ARF. Since $new_job uses WTTW Abacus as its abuse ticketing system, handling of ARF complaints (like the ones your script generates, is a breeze. I would hope that more authors of automated complaint tools move to ARF.

Some years ago, I was breifly involved in a similar, abortive project at MAPS for a complaint format very similar in concept to ARF. I was disappointed that the project died aborning, and quite happy to see ARF in development.

--Brian McNett

(by Brian McNett 10 May 2006 14:51)


Add your comment...

Note: all comments require an email address to send a confirmation to verify that it was posted by a person and not a spambot. The comment won't be visible until you click the link in the confirmation. Unless you check the box below, which almost nobody does, your email won't be displayed, and I won't use it for other purposes.

 
Name:
Email: you@wherever (required, for confirmation)
Title: (optional)
Comments:
Show my Email address
Save my Name and Email for next time

Topics


My other sites

Who is this guy?

Airline ticket info

Taughannock Networks

Other blogs

CAUCE
Criminal Abuse of Domain Names: Bulk Registration and Contact Information Access
52 days ago

A keen grasp of the obvious
My high security debit card
358 days ago

Related sites

Coalition Against Unsolicited Commercial E-mail

Network Abuse Clearinghouse



© 2005-2018 John R. Levine.
CAN SPAM address harvesting notice: the operator of this website will not give, sell, or otherwise transfer addresses maintained by this website to any other party for the purposes of initiating, or enabling others to initiate, electronic mail messages.