Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations

Tsuyoshi Moriokal, Naohiro Tawara, Tetsuji Ogawa, Atsunori Ogawa, Tomoharu Iwata, Tetsunori Kobayashi

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    1 Citation (Scopus)

    Abstract

    Training recurrent neural network language models (RNNLMs) requires a large amount of data, which is difficult to collect for specific domains such as multiparty conversations. Data augmentation using external resources and model adaptation, which adjusts a model trained on a large amount of data to a target domain, have been proposed for low-resource language modeling. While there are the commonalities and discrepancies between the source and target domains in terms of the statistics of words and their contexts, these methods for domain adaptation make the commonalities and discrepancies jumbled. We propose novel domain adaptation techniques for RNNLM by introducing domain-shared and domain-specific word embedding and contextual features. This explicit modeling of the commonalities and discrepancies would improve the language modeling performance. Experimental comparisons using multiparty conversation data as the target domain augmented by lecture data from the source domain demonstrate that the proposed domain adaptation method exhibits improvements in the perplexity and word error rate over the long short-term memory based language model (LSTMLM) trained using the source and target domain data.

    Original languageEnglish
    Title of host publication2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages6084-6088
    Number of pages5
    Volume2018-April
    ISBN (Print)9781538646588
    DOIs
    Publication statusPublished - 2018 Sep 10
    Event2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Calgary, Canada
    Duration: 2018 Apr 152018 Apr 20

    Other

    Other2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018
    CountryCanada
    CityCalgary
    Period18/4/1518/4/20

    Fingerprint

    Recurrent neural networks
    Statistics

    Keywords

    • Data augmentation
    • Domain adaptation
    • Language models
    • Recurrent neural network

    ASJC Scopus subject areas

    • Software
    • Signal Processing
    • Electrical and Electronic Engineering

    Cite this

    Moriokal, T., Tawara, N., Ogawa, T., Ogawa, A., Iwata, T., & Kobayashi, T. (2018). Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations. In 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings (Vol. 2018-April, pp. 6084-6088). [8462631] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2018.8462631

    Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations. / Moriokal, Tsuyoshi; Tawara, Naohiro; Ogawa, Tetsuji; Ogawa, Atsunori; Iwata, Tomoharu; Kobayashi, Tetsunori.

    2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. Vol. 2018-April Institute of Electrical and Electronics Engineers Inc., 2018. p. 6084-6088 8462631.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Moriokal, T, Tawara, N, Ogawa, T, Ogawa, A, Iwata, T & Kobayashi, T 2018, Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations. in 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. vol. 2018-April, 8462631, Institute of Electrical and Electronics Engineers Inc., pp. 6084-6088, 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018, Calgary, Canada, 18/4/15. https://doi.org/10.1109/ICASSP.2018.8462631
    Moriokal T, Tawara N, Ogawa T, Ogawa A, Iwata T, Kobayashi T. Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations. In 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. Vol. 2018-April. Institute of Electrical and Electronics Engineers Inc. 2018. p. 6084-6088. 8462631 https://doi.org/10.1109/ICASSP.2018.8462631
    Moriokal, Tsuyoshi ; Tawara, Naohiro ; Ogawa, Tetsuji ; Ogawa, Atsunori ; Iwata, Tomoharu ; Kobayashi, Tetsunori. / Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations. 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. Vol. 2018-April Institute of Electrical and Electronics Engineers Inc., 2018. pp. 6084-6088
    @inproceedings{2d5dc82f9237434bbcfeeec7fd7ec0b6,
    title = "Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations",
    abstract = "Training recurrent neural network language models (RNNLMs) requires a large amount of data, which is difficult to collect for specific domains such as multiparty conversations. Data augmentation using external resources and model adaptation, which adjusts a model trained on a large amount of data to a target domain, have been proposed for low-resource language modeling. While there are the commonalities and discrepancies between the source and target domains in terms of the statistics of words and their contexts, these methods for domain adaptation make the commonalities and discrepancies jumbled. We propose novel domain adaptation techniques for RNNLM by introducing domain-shared and domain-specific word embedding and contextual features. This explicit modeling of the commonalities and discrepancies would improve the language modeling performance. Experimental comparisons using multiparty conversation data as the target domain augmented by lecture data from the source domain demonstrate that the proposed domain adaptation method exhibits improvements in the perplexity and word error rate over the long short-term memory based language model (LSTMLM) trained using the source and target domain data.",
    keywords = "Data augmentation, Domain adaptation, Language models, Recurrent neural network",
    author = "Tsuyoshi Moriokal and Naohiro Tawara and Tetsuji Ogawa and Atsunori Ogawa and Tomoharu Iwata and Tetsunori Kobayashi",
    year = "2018",
    month = "9",
    day = "10",
    doi = "10.1109/ICASSP.2018.8462631",
    language = "English",
    isbn = "9781538646588",
    volume = "2018-April",
    pages = "6084--6088",
    booktitle = "2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",

    }

    TY - GEN

    T1 - Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations

    AU - Moriokal, Tsuyoshi

    AU - Tawara, Naohiro

    AU - Ogawa, Tetsuji

    AU - Ogawa, Atsunori

    AU - Iwata, Tomoharu

    AU - Kobayashi, Tetsunori

    PY - 2018/9/10

    Y1 - 2018/9/10

    N2 - Training recurrent neural network language models (RNNLMs) requires a large amount of data, which is difficult to collect for specific domains such as multiparty conversations. Data augmentation using external resources and model adaptation, which adjusts a model trained on a large amount of data to a target domain, have been proposed for low-resource language modeling. While there are the commonalities and discrepancies between the source and target domains in terms of the statistics of words and their contexts, these methods for domain adaptation make the commonalities and discrepancies jumbled. We propose novel domain adaptation techniques for RNNLM by introducing domain-shared and domain-specific word embedding and contextual features. This explicit modeling of the commonalities and discrepancies would improve the language modeling performance. Experimental comparisons using multiparty conversation data as the target domain augmented by lecture data from the source domain demonstrate that the proposed domain adaptation method exhibits improvements in the perplexity and word error rate over the long short-term memory based language model (LSTMLM) trained using the source and target domain data.

    AB - Training recurrent neural network language models (RNNLMs) requires a large amount of data, which is difficult to collect for specific domains such as multiparty conversations. Data augmentation using external resources and model adaptation, which adjusts a model trained on a large amount of data to a target domain, have been proposed for low-resource language modeling. While there are the commonalities and discrepancies between the source and target domains in terms of the statistics of words and their contexts, these methods for domain adaptation make the commonalities and discrepancies jumbled. We propose novel domain adaptation techniques for RNNLM by introducing domain-shared and domain-specific word embedding and contextual features. This explicit modeling of the commonalities and discrepancies would improve the language modeling performance. Experimental comparisons using multiparty conversation data as the target domain augmented by lecture data from the source domain demonstrate that the proposed domain adaptation method exhibits improvements in the perplexity and word error rate over the long short-term memory based language model (LSTMLM) trained using the source and target domain data.

    KW - Data augmentation

    KW - Domain adaptation

    KW - Language models

    KW - Recurrent neural network

    UR - http://www.scopus.com/inward/record.url?scp=85054217601&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85054217601&partnerID=8YFLogxK

    U2 - 10.1109/ICASSP.2018.8462631

    DO - 10.1109/ICASSP.2018.8462631

    M3 - Conference contribution

    AN - SCOPUS:85054217601

    SN - 9781538646588

    VL - 2018-April

    SP - 6084

    EP - 6088

    BT - 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings

    PB - Institute of Electrical and Electronics Engineers Inc.

    ER -