Spatio-temporal factorization of log data for understanding network events

Tatsuaki Kimura, Keisuke Ishibashi, Tatsuya Mori, Hiroshi Sawada, Tsuyoshi Toyono, Ken Nishimatsu, Akio Watanabe, Akihiro Shimoda, Kohei Shiomoto

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    36 Citations (Scopus)

    Abstract

    Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.

    Original languageEnglish
    Title of host publicationProceedings - IEEE INFOCOM
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages610-618
    Number of pages9
    ISBN (Print)9781479933600
    DOIs
    Publication statusPublished - 2014
    Event33rd IEEE Conference on Computer Communications, IEEE INFOCOM 2014 - Toronto, ON
    Duration: 2014 Apr 272014 May 2

    Other

    Other33rd IEEE Conference on Computer Communications, IEEE INFOCOM 2014
    CityToronto, ON
    Period14/4/2714/5/2

    Fingerprint

    Factorization
    Tensors
    Flaps
    Network layers
    Routers
    Switches
    Hardware
    Network protocols
    Statistical Models

    ASJC Scopus subject areas

    • Computer Science(all)
    • Electrical and Electronic Engineering

    Cite this

    Kimura, T., Ishibashi, K., Mori, T., Sawada, H., Toyono, T., Nishimatsu, K., ... Shiomoto, K. (2014). Spatio-temporal factorization of log data for understanding network events. In Proceedings - IEEE INFOCOM (pp. 610-618). [6847986] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/INFOCOM.2014.6847986

    Spatio-temporal factorization of log data for understanding network events. / Kimura, Tatsuaki; Ishibashi, Keisuke; Mori, Tatsuya; Sawada, Hiroshi; Toyono, Tsuyoshi; Nishimatsu, Ken; Watanabe, Akio; Shimoda, Akihiro; Shiomoto, Kohei.

    Proceedings - IEEE INFOCOM. Institute of Electrical and Electronics Engineers Inc., 2014. p. 610-618 6847986.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Kimura, T, Ishibashi, K, Mori, T, Sawada, H, Toyono, T, Nishimatsu, K, Watanabe, A, Shimoda, A & Shiomoto, K 2014, Spatio-temporal factorization of log data for understanding network events. in Proceedings - IEEE INFOCOM., 6847986, Institute of Electrical and Electronics Engineers Inc., pp. 610-618, 33rd IEEE Conference on Computer Communications, IEEE INFOCOM 2014, Toronto, ON, 14/4/27. https://doi.org/10.1109/INFOCOM.2014.6847986
    Kimura T, Ishibashi K, Mori T, Sawada H, Toyono T, Nishimatsu K et al. Spatio-temporal factorization of log data for understanding network events. In Proceedings - IEEE INFOCOM. Institute of Electrical and Electronics Engineers Inc. 2014. p. 610-618. 6847986 https://doi.org/10.1109/INFOCOM.2014.6847986
    Kimura, Tatsuaki ; Ishibashi, Keisuke ; Mori, Tatsuya ; Sawada, Hiroshi ; Toyono, Tsuyoshi ; Nishimatsu, Ken ; Watanabe, Akio ; Shimoda, Akihiro ; Shiomoto, Kohei. / Spatio-temporal factorization of log data for understanding network events. Proceedings - IEEE INFOCOM. Institute of Electrical and Electronics Engineers Inc., 2014. pp. 610-618
    @inproceedings{dc0ac289d8a446eb816625239f8d1044,
    title = "Spatio-temporal factorization of log data for understanding network events",
    abstract = "Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.",
    author = "Tatsuaki Kimura and Keisuke Ishibashi and Tatsuya Mori and Hiroshi Sawada and Tsuyoshi Toyono and Ken Nishimatsu and Akio Watanabe and Akihiro Shimoda and Kohei Shiomoto",
    year = "2014",
    doi = "10.1109/INFOCOM.2014.6847986",
    language = "English",
    isbn = "9781479933600",
    pages = "610--618",
    booktitle = "Proceedings - IEEE INFOCOM",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",

    }

    TY - GEN

    T1 - Spatio-temporal factorization of log data for understanding network events

    AU - Kimura, Tatsuaki

    AU - Ishibashi, Keisuke

    AU - Mori, Tatsuya

    AU - Sawada, Hiroshi

    AU - Toyono, Tsuyoshi

    AU - Nishimatsu, Ken

    AU - Watanabe, Akio

    AU - Shimoda, Akihiro

    AU - Shiomoto, Kohei

    PY - 2014

    Y1 - 2014

    N2 - Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.

    AB - Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.

    UR - http://www.scopus.com/inward/record.url?scp=84904438214&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84904438214&partnerID=8YFLogxK

    U2 - 10.1109/INFOCOM.2014.6847986

    DO - 10.1109/INFOCOM.2014.6847986

    M3 - Conference contribution

    AN - SCOPUS:84904438214

    SN - 9781479933600

    SP - 610

    EP - 618

    BT - Proceedings - IEEE INFOCOM

    PB - Institute of Electrical and Electronics Engineers Inc.

    ER -