Unsupervised ensemble anomaly detection using time-periodic packet sampling

Masato Uchida, Shuichi Nawata, Yu Gu, Masato Tsuru, Yuji Oie

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

We propose an anomaly detection method for finding patterns in network traffic that do not conform to legitimate (i.e., normal) behavior. The proposed method trains a baseline model describing the normal behavior of network traffic without using manually labeled traffic data. The trained baseline model is used as the basis for comparison with the audit network traffic. This anomaly detection works in an unsupervised manner through the use of time-periodic packet sampling, which is used in a manner that differs from its intended purpose - the lossy nature of packet sampling is used to extract normal packets from the unlabeled original traffic data. Evaluation using actual traffic traces showed that the proposed method has false positive and false negative rates in the detection of anomalies regarding TCP SYN packets comparable to those of a conventional method that uses manually labeled traffic data to train the baseline model. Performance variation due to the probabilistic nature of sampled traffic data is mitigated by using ensemble anomaly detection that collectively exploits multiple baseline models in parallel. Alarm sensitivity is adjusted for the intended use by using maximum- and minimum-based anomaly detection that effectively take advantage of the performance variations among the multiple baseline models. Testing using actual traffic traces showed that the proposed anomaly detection method performs as well as one using manually labeled traffic data and better than one using randomly sampled (unlabeled) traffic data.

Original languageEnglish
Pages (from-to)2358-2367
Number of pages10
JournalIEICE Transactions on Communications
VolumeE95-B
Issue number7
DOIs
Publication statusPublished - 2012
Externally publishedYes

Fingerprint

Sampling
Testing

Keywords

  • Anomaly detection
  • Packet sampling

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this

Unsupervised ensemble anomaly detection using time-periodic packet sampling. / Uchida, Masato; Nawata, Shuichi; Gu, Yu; Tsuru, Masato; Oie, Yuji.

In: IEICE Transactions on Communications, Vol. E95-B, No. 7, 2012, p. 2358-2367.

Research output: Contribution to journalArticle

Uchida, Masato ; Nawata, Shuichi ; Gu, Yu ; Tsuru, Masato ; Oie, Yuji. / Unsupervised ensemble anomaly detection using time-periodic packet sampling. In: IEICE Transactions on Communications. 2012 ; Vol. E95-B, No. 7. pp. 2358-2367.
@article{a4f9e56f595e4e6baf20aface129a705,
title = "Unsupervised ensemble anomaly detection using time-periodic packet sampling",
abstract = "We propose an anomaly detection method for finding patterns in network traffic that do not conform to legitimate (i.e., normal) behavior. The proposed method trains a baseline model describing the normal behavior of network traffic without using manually labeled traffic data. The trained baseline model is used as the basis for comparison with the audit network traffic. This anomaly detection works in an unsupervised manner through the use of time-periodic packet sampling, which is used in a manner that differs from its intended purpose - the lossy nature of packet sampling is used to extract normal packets from the unlabeled original traffic data. Evaluation using actual traffic traces showed that the proposed method has false positive and false negative rates in the detection of anomalies regarding TCP SYN packets comparable to those of a conventional method that uses manually labeled traffic data to train the baseline model. Performance variation due to the probabilistic nature of sampled traffic data is mitigated by using ensemble anomaly detection that collectively exploits multiple baseline models in parallel. Alarm sensitivity is adjusted for the intended use by using maximum- and minimum-based anomaly detection that effectively take advantage of the performance variations among the multiple baseline models. Testing using actual traffic traces showed that the proposed anomaly detection method performs as well as one using manually labeled traffic data and better than one using randomly sampled (unlabeled) traffic data.",
keywords = "Anomaly detection, Packet sampling",
author = "Masato Uchida and Shuichi Nawata and Yu Gu and Masato Tsuru and Yuji Oie",
year = "2012",
doi = "10.1587/transcom.E95.B.2358",
language = "English",
volume = "E95-B",
pages = "2358--2367",
journal = "IEICE Transactions on Communications",
issn = "0916-8516",
publisher = "Maruzen Co., Ltd/Maruzen Kabushikikaisha",
number = "7",

}

TY - JOUR

T1 - Unsupervised ensemble anomaly detection using time-periodic packet sampling

AU - Uchida, Masato

AU - Nawata, Shuichi

AU - Gu, Yu

AU - Tsuru, Masato

AU - Oie, Yuji

PY - 2012

Y1 - 2012

N2 - We propose an anomaly detection method for finding patterns in network traffic that do not conform to legitimate (i.e., normal) behavior. The proposed method trains a baseline model describing the normal behavior of network traffic without using manually labeled traffic data. The trained baseline model is used as the basis for comparison with the audit network traffic. This anomaly detection works in an unsupervised manner through the use of time-periodic packet sampling, which is used in a manner that differs from its intended purpose - the lossy nature of packet sampling is used to extract normal packets from the unlabeled original traffic data. Evaluation using actual traffic traces showed that the proposed method has false positive and false negative rates in the detection of anomalies regarding TCP SYN packets comparable to those of a conventional method that uses manually labeled traffic data to train the baseline model. Performance variation due to the probabilistic nature of sampled traffic data is mitigated by using ensemble anomaly detection that collectively exploits multiple baseline models in parallel. Alarm sensitivity is adjusted for the intended use by using maximum- and minimum-based anomaly detection that effectively take advantage of the performance variations among the multiple baseline models. Testing using actual traffic traces showed that the proposed anomaly detection method performs as well as one using manually labeled traffic data and better than one using randomly sampled (unlabeled) traffic data.

AB - We propose an anomaly detection method for finding patterns in network traffic that do not conform to legitimate (i.e., normal) behavior. The proposed method trains a baseline model describing the normal behavior of network traffic without using manually labeled traffic data. The trained baseline model is used as the basis for comparison with the audit network traffic. This anomaly detection works in an unsupervised manner through the use of time-periodic packet sampling, which is used in a manner that differs from its intended purpose - the lossy nature of packet sampling is used to extract normal packets from the unlabeled original traffic data. Evaluation using actual traffic traces showed that the proposed method has false positive and false negative rates in the detection of anomalies regarding TCP SYN packets comparable to those of a conventional method that uses manually labeled traffic data to train the baseline model. Performance variation due to the probabilistic nature of sampled traffic data is mitigated by using ensemble anomaly detection that collectively exploits multiple baseline models in parallel. Alarm sensitivity is adjusted for the intended use by using maximum- and minimum-based anomaly detection that effectively take advantage of the performance variations among the multiple baseline models. Testing using actual traffic traces showed that the proposed anomaly detection method performs as well as one using manually labeled traffic data and better than one using randomly sampled (unlabeled) traffic data.

KW - Anomaly detection

KW - Packet sampling

UR - http://www.scopus.com/inward/record.url?scp=84863443672&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84863443672&partnerID=8YFLogxK

U2 - 10.1587/transcom.E95.B.2358

DO - 10.1587/transcom.E95.B.2358

M3 - Article

AN - SCOPUS:84863443672

VL - E95-B

SP - 2358

EP - 2367

JO - IEICE Transactions on Communications

JF - IEICE Transactions on Communications

SN - 0916-8516

IS - 7

ER -