Network event extraction from log data with nonnegative tensor factorization

Tatsuaki Kimura, Keisuke Ishibashi, Tatsuya Mori, Hiroshi Sawada, Tsuyoshi Toyono, Ken Nishimatsu, Akio Watanabe, Akihiro Shimoda, Kohei Shiomoto

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

Network equipment, such as routers, switches, and RA- DIUS servers, generate various log messages induced by network events such as hardware failures and protocol flaps. In large production networks, analyzing the log messages is crucial for diagnosing network anomalies; however, it has become challenging due to the following two reasons. First, the log messages are composed of unstructured text messages generated in accordance with vendor-specific rules. Second, network events that in- duce the log messages span several geographical locations, network layers, protocols, and services. We developed a method to tackle these obsta- cles consisting of two techniques: statistical template extraction (STE) and log tensor factorization (LTF). The former leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. The latter builds a statistical model that collects spatial-Temporal patterns of log messages. Such spatial-Temporal patterns provide useful in- sights into understanding the impact and patterns of hidden network events. We evaluate our techniques using a massive amount of network log mes- sages collected from a large operating network and confirm that our model fits the data well. We also investigate several case studies that validate the usefulness of our method.

Original languageEnglish
Pages (from-to)1865-1878
Number of pages14
JournalIEICE Transactions on Communications
VolumeE100B
Issue number10
DOIs
Publication statusPublished - 2017 Oct

Keywords

  • Network management
  • Nonnegative tensor factorization (NTF)
  • Syslog

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Network event extraction from log data with nonnegative tensor factorization'. Together they form a unique fingerprint.

Cite this