A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs

Yu Cui, Yiping Sun, Takayuki Furuzuki, Gehao Sheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3057-3062
Number of pages6
ISBN (Electronic)9781538666500
DOIs
Publication statusPublished - 2019 Jan 16
Event2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018 - Miyazaki, Japan
Duration: 2018 Oct 72018 Oct 10

Publication series

NameProceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018

Conference

Conference2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018
CountryJapan
CityMiyazaki
Period18/10/718/10/10

Fingerprint

Support vector machines
Ant colony optimization
Macros
Learning systems
Genes
Ants
Anomaly detection
Experiments
Equipment and Supplies
Support vector machine
Research
Support Vector Machine
Module

Keywords

  • Anomaly Detection
  • Ant Colony Optimization
  • Auto-encoder
  • Feature Extraction
  • Log Analysis

ASJC Scopus subject areas

  • Information Systems
  • Information Systems and Management
  • Health Informatics
  • Artificial Intelligence
  • Computer Networks and Communications
  • Human-Computer Interaction

Cite this

Cui, Y., Sun, Y., Furuzuki, T., & Sheng, G. (2019). A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. In Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018 (pp. 3057-3062). [8616515] (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SMC.2018.00519

A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. / Cui, Yu; Sun, Yiping; Furuzuki, Takayuki; Sheng, Gehao.

Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 3057-3062 8616515 (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cui, Y, Sun, Y, Furuzuki, T & Sheng, G 2019, A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. in Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018., 8616515, Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018, Institute of Electrical and Electronics Engineers Inc., pp. 3057-3062, 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018, Miyazaki, Japan, 18/10/7. https://doi.org/10.1109/SMC.2018.00519
Cui Y, Sun Y, Furuzuki T, Sheng G. A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. In Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 3057-3062. 8616515. (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018). https://doi.org/10.1109/SMC.2018.00519
Cui, Yu ; Sun, Yiping ; Furuzuki, Takayuki ; Sheng, Gehao. / A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 3057-3062 (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018).
@inproceedings{6c8cb9cebb7a4e0cbc7f63fc0dc76009,
title = "A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs",
abstract = "Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.",
keywords = "Anomaly Detection, Ant Colony Optimization, Auto-encoder, Feature Extraction, Log Analysis",
author = "Yu Cui and Yiping Sun and Takayuki Furuzuki and Gehao Sheng",
year = "2019",
month = "1",
day = "16",
doi = "10.1109/SMC.2018.00519",
language = "English",
series = "Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "3057--3062",
booktitle = "Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018",

}

TY - GEN

T1 - A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs

AU - Cui, Yu

AU - Sun, Yiping

AU - Furuzuki, Takayuki

AU - Sheng, Gehao

PY - 2019/1/16

Y1 - 2019/1/16

N2 - Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.

AB - Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.

KW - Anomaly Detection

KW - Ant Colony Optimization

KW - Auto-encoder

KW - Feature Extraction

KW - Log Analysis

UR - http://www.scopus.com/inward/record.url?scp=85062220577&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062220577&partnerID=8YFLogxK

U2 - 10.1109/SMC.2018.00519

DO - 10.1109/SMC.2018.00519

M3 - Conference contribution

T3 - Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018

SP - 3057

EP - 3062

BT - Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -