On the effectiveness of IP reputation for spam filtering

Holly Esquivel, Aditya Akella, Tatsuya Mori

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

Modern SMTP servers apply a variety of mechanisms to stem the volume of spam delivered to users. These techniques can be broadly classified into two categories: preacceptance approaches, which apply prior to a message being accepted (e.g. IP reputation), and post-acceptance techniques which apply after a message has been accepted (e.g. content based signatures). We argue that the effectiveness of these measures varies based on the SMTP sender type. This paper focuses on the most light-weight pre-acceptance filtering mechanism - IP reputation. We first classify SMTP senders into three main categories: legitimate servers, end-hosts, and spam gangs, and empirically study the limits of effectiveness regarding IP reputation filtering for each category. Next, we develop new techniques that build custom IP reputation lists, which significantly improve the performance of existing IP reputation lists. In compiling these lists, we leverage a somewhat surprising fact that both legitimate domains and spam domains often use the DNS Sender Policy Framework (SPF) in an attempt to pass simple authentication checks. That is, good/bad IP addresses can be systematically compiled by collecting good/bad domains and looking up their SPF resource records. We also evaluate the effectiveness of these lists over time. Finally, we aim to understand the characteristics of the three categories of email senders in depth. Overall, we find that it is possible to construct IP reputation lists that can identify roughly 90% of all spam and legitimate mail, but some of the lists, i.e. the lists for spam gangs, must be updated on a constant basis to maintain this high level of accuracy.

Original languageEnglish
Title of host publication2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010
DOIs
Publication statusPublished - 2010
Externally publishedYes
Event2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010 - Bangalore
Duration: 2010 Jan 52010 Jan 9

Other

Other2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010
CityBangalore
Period10/1/510/1/9

Fingerprint

Servers
Electronic mail
Authentication

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this

Esquivel, H., Akella, A., & Mori, T. (2010). On the effectiveness of IP reputation for spam filtering. In 2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010 [5431981] https://doi.org/10.1109/COMSNETS.2010.5431981

On the effectiveness of IP reputation for spam filtering. / Esquivel, Holly; Akella, Aditya; Mori, Tatsuya.

2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010. 2010. 5431981.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Esquivel, H, Akella, A & Mori, T 2010, On the effectiveness of IP reputation for spam filtering. in 2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010., 5431981, 2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010, Bangalore, 10/1/5. https://doi.org/10.1109/COMSNETS.2010.5431981
Esquivel H, Akella A, Mori T. On the effectiveness of IP reputation for spam filtering. In 2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010. 2010. 5431981 https://doi.org/10.1109/COMSNETS.2010.5431981
Esquivel, Holly ; Akella, Aditya ; Mori, Tatsuya. / On the effectiveness of IP reputation for spam filtering. 2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010. 2010.
@inproceedings{501ea09b275d47f58c6fab25ed37d90a,
title = "On the effectiveness of IP reputation for spam filtering",
abstract = "Modern SMTP servers apply a variety of mechanisms to stem the volume of spam delivered to users. These techniques can be broadly classified into two categories: preacceptance approaches, which apply prior to a message being accepted (e.g. IP reputation), and post-acceptance techniques which apply after a message has been accepted (e.g. content based signatures). We argue that the effectiveness of these measures varies based on the SMTP sender type. This paper focuses on the most light-weight pre-acceptance filtering mechanism - IP reputation. We first classify SMTP senders into three main categories: legitimate servers, end-hosts, and spam gangs, and empirically study the limits of effectiveness regarding IP reputation filtering for each category. Next, we develop new techniques that build custom IP reputation lists, which significantly improve the performance of existing IP reputation lists. In compiling these lists, we leverage a somewhat surprising fact that both legitimate domains and spam domains often use the DNS Sender Policy Framework (SPF) in an attempt to pass simple authentication checks. That is, good/bad IP addresses can be systematically compiled by collecting good/bad domains and looking up their SPF resource records. We also evaluate the effectiveness of these lists over time. Finally, we aim to understand the characteristics of the three categories of email senders in depth. Overall, we find that it is possible to construct IP reputation lists that can identify roughly 90{\%} of all spam and legitimate mail, but some of the lists, i.e. the lists for spam gangs, must be updated on a constant basis to maintain this high level of accuracy.",
author = "Holly Esquivel and Aditya Akella and Tatsuya Mori",
year = "2010",
doi = "10.1109/COMSNETS.2010.5431981",
language = "English",
isbn = "9781424454877",
booktitle = "2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010",

}

TY - GEN

T1 - On the effectiveness of IP reputation for spam filtering

AU - Esquivel, Holly

AU - Akella, Aditya

AU - Mori, Tatsuya

PY - 2010

Y1 - 2010

N2 - Modern SMTP servers apply a variety of mechanisms to stem the volume of spam delivered to users. These techniques can be broadly classified into two categories: preacceptance approaches, which apply prior to a message being accepted (e.g. IP reputation), and post-acceptance techniques which apply after a message has been accepted (e.g. content based signatures). We argue that the effectiveness of these measures varies based on the SMTP sender type. This paper focuses on the most light-weight pre-acceptance filtering mechanism - IP reputation. We first classify SMTP senders into three main categories: legitimate servers, end-hosts, and spam gangs, and empirically study the limits of effectiveness regarding IP reputation filtering for each category. Next, we develop new techniques that build custom IP reputation lists, which significantly improve the performance of existing IP reputation lists. In compiling these lists, we leverage a somewhat surprising fact that both legitimate domains and spam domains often use the DNS Sender Policy Framework (SPF) in an attempt to pass simple authentication checks. That is, good/bad IP addresses can be systematically compiled by collecting good/bad domains and looking up their SPF resource records. We also evaluate the effectiveness of these lists over time. Finally, we aim to understand the characteristics of the three categories of email senders in depth. Overall, we find that it is possible to construct IP reputation lists that can identify roughly 90% of all spam and legitimate mail, but some of the lists, i.e. the lists for spam gangs, must be updated on a constant basis to maintain this high level of accuracy.

AB - Modern SMTP servers apply a variety of mechanisms to stem the volume of spam delivered to users. These techniques can be broadly classified into two categories: preacceptance approaches, which apply prior to a message being accepted (e.g. IP reputation), and post-acceptance techniques which apply after a message has been accepted (e.g. content based signatures). We argue that the effectiveness of these measures varies based on the SMTP sender type. This paper focuses on the most light-weight pre-acceptance filtering mechanism - IP reputation. We first classify SMTP senders into three main categories: legitimate servers, end-hosts, and spam gangs, and empirically study the limits of effectiveness regarding IP reputation filtering for each category. Next, we develop new techniques that build custom IP reputation lists, which significantly improve the performance of existing IP reputation lists. In compiling these lists, we leverage a somewhat surprising fact that both legitimate domains and spam domains often use the DNS Sender Policy Framework (SPF) in an attempt to pass simple authentication checks. That is, good/bad IP addresses can be systematically compiled by collecting good/bad domains and looking up their SPF resource records. We also evaluate the effectiveness of these lists over time. Finally, we aim to understand the characteristics of the three categories of email senders in depth. Overall, we find that it is possible to construct IP reputation lists that can identify roughly 90% of all spam and legitimate mail, but some of the lists, i.e. the lists for spam gangs, must be updated on a constant basis to maintain this high level of accuracy.

UR - http://www.scopus.com/inward/record.url?scp=77952149776&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77952149776&partnerID=8YFLogxK

U2 - 10.1109/COMSNETS.2010.5431981

DO - 10.1109/COMSNETS.2010.5431981

M3 - Conference contribution

AN - SCOPUS:77952149776

SN - 9781424454877

BT - 2010 2nd International Conference on COMmunication Systems and NETworks, COMSNETS 2010

ER -