Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations

Tsuyoshi Moriokal, Naohiro Tawara, Tetsuji Ogawa, Atsunori Ogawa, Tomoharu Iwata, Tetsunori Kobayashi

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    2 Citations (Scopus)

    Abstract

    Training recurrent neural network language models (RNNLMs) requires a large amount of data, which is difficult to collect for specific domains such as multiparty conversations. Data augmentation using external resources and model adaptation, which adjusts a model trained on a large amount of data to a target domain, have been proposed for low-resource language modeling. While there are the commonalities and discrepancies between the source and target domains in terms of the statistics of words and their contexts, these methods for domain adaptation make the commonalities and discrepancies jumbled. We propose novel domain adaptation techniques for RNNLM by introducing domain-shared and domain-specific word embedding and contextual features. This explicit modeling of the commonalities and discrepancies would improve the language modeling performance. Experimental comparisons using multiparty conversation data as the target domain augmented by lecture data from the source domain demonstrate that the proposed domain adaptation method exhibits improvements in the perplexity and word error rate over the long short-term memory based language model (LSTMLM) trained using the source and target domain data.

    Original languageEnglish
    Title of host publication2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages6084-6088
    Number of pages5
    Volume2018-April
    ISBN (Print)9781538646588
    DOIs
    Publication statusPublished - 2018 Sep 10
    Event2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Calgary, Canada
    Duration: 2018 Apr 152018 Apr 20

    Other

    Other2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018
    CountryCanada
    CityCalgary
    Period18/4/1518/4/20

      Fingerprint

    Keywords

    • Data augmentation
    • Domain adaptation
    • Language models
    • Recurrent neural network

    ASJC Scopus subject areas

    • Software
    • Signal Processing
    • Electrical and Electronic Engineering

    Cite this

    Moriokal, T., Tawara, N., Ogawa, T., Ogawa, A., Iwata, T., & Kobayashi, T. (2018). Language model domain adaptation via recurrent neural networks with domain-shared and domain-specific representations. In 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings (Vol. 2018-April, pp. 6084-6088). [8462631] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2018.8462631