A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs

Yu Cui, Yiping Sun, Takayuki Furuzuki, Gehao Sheng

研究成果: Conference contribution

抄録

Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.

元の言語English
ホスト出版物のタイトルProceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018
出版者Institute of Electrical and Electronics Engineers Inc.
ページ3057-3062
ページ数6
ISBN(電子版)9781538666500
DOI
出版物ステータスPublished - 2019 1 16
イベント2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018 - Miyazaki, Japan
継続期間: 2018 10 72018 10 10

出版物シリーズ

名前Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018

Conference

Conference2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018
Japan
Miyazaki
期間18/10/718/10/10

Fingerprint

Support vector machines
Ant colony optimization
Macros
Learning systems
Genes
Ants
Anomaly detection
Experiments
Equipment and Supplies
Support vector machine
Research
Support Vector Machine
Module

ASJC Scopus subject areas

  • Information Systems
  • Information Systems and Management
  • Health Informatics
  • Artificial Intelligence
  • Computer Networks and Communications
  • Human-Computer Interaction

これを引用

Cui, Y., Sun, Y., Furuzuki, T., & Sheng, G. (2019). A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. : Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018 (pp. 3057-3062). [8616515] (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SMC.2018.00519

A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. / Cui, Yu; Sun, Yiping; Furuzuki, Takayuki; Sheng, Gehao.

Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 3057-3062 8616515 (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018).

研究成果: Conference contribution

Cui, Y, Sun, Y, Furuzuki, T & Sheng, G 2019, A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. : Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018., 8616515, Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018, Institute of Electrical and Electronics Engineers Inc., pp. 3057-3062, 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018, Miyazaki, Japan, 18/10/7. https://doi.org/10.1109/SMC.2018.00519
Cui Y, Sun Y, Furuzuki T, Sheng G. A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. : Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 3057-3062. 8616515. (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018). https://doi.org/10.1109/SMC.2018.00519
Cui, Yu ; Sun, Yiping ; Furuzuki, Takayuki ; Sheng, Gehao. / A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs. Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 3057-3062 (Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018).
@inproceedings{6c8cb9cebb7a4e0cbc7f63fc0dc76009,
title = "A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs",
abstract = "Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.",
keywords = "Anomaly Detection, Ant Colony Optimization, Auto-encoder, Feature Extraction, Log Analysis",
author = "Yu Cui and Yiping Sun and Takayuki Furuzuki and Gehao Sheng",
year = "2019",
month = "1",
day = "16",
doi = "10.1109/SMC.2018.00519",
language = "English",
series = "Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "3057--3062",
booktitle = "Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018",

}

TY - GEN

T1 - A Convolutional Auto-Encoder Method for Anomaly Detection on System Logs

AU - Cui, Yu

AU - Sun, Yiping

AU - Furuzuki, Takayuki

AU - Sheng, Gehao

PY - 2019/1/16

Y1 - 2019/1/16

N2 - Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.

AB - Anomaly detection on system logs is to report system failures with utilization of console logs collected from devices, which ensures the reliability of systems. Most previous researches split logs into sequential time windows and regarded each window as an independent instance for classification using popular machine learning methods like support vector machine(SVM), however, neglected the time patterns under logs. Those approaches also suffer from information loss due to the vector representation, and high dimensionality if there is a large number of log events. To make up these deficiencies, unlike most traditional methods that used a vector to represent a period behavior at the macro level, we construct a 2D matrix to reveal more detailed system behaviors in the time period by dividing each window into sequential subwindows. To provide a more efficient representation, we further use the ant colony optimization algorithm to find a highly-coupled event template as the horizontal index of the 2D window matrix to replace the disordered one. To capture time dependencies, a multi-module convolutional auto-encoder is configured as that different paralleled modules scan among different time intervals to extract information respectively. These features are then concatenated in latent space as the final input, which contains diversified time information, for classification by SVM. The experiments on Blue Gene/L log dataset showed that our proposed method outperforms the state-of-art SVM method.

KW - Anomaly Detection

KW - Ant Colony Optimization

KW - Auto-encoder

KW - Feature Extraction

KW - Log Analysis

UR - http://www.scopus.com/inward/record.url?scp=85062220577&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062220577&partnerID=8YFLogxK

U2 - 10.1109/SMC.2018.00519

DO - 10.1109/SMC.2018.00519

M3 - Conference contribution

AN - SCOPUS:85062220577

T3 - Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018

SP - 3057

EP - 3062

BT - Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -