Multi-Source Domain Generalization Using Domain Attributes for Recurrent Neural Network Language Models

Naohiro Tawara*, Atsunori Ogawa, Tomoharu Iwata, Hiroto Ashikawa, Tetsunori Kobayashi, Tetsuji Ogawa

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Most conventional multi-source domain adaptation techniques for recurrent neural network language models (RNNLMs) are domain-centric. In these approaches, each domain is considered independently and this makes it difficult to apply the models to completely unseen target domains that are unobservable during training. Instead, our study exploits domain attributes, which represent common knowledge among such different domains as dialects, types of wordings, styles, and topics, to achieve domain generalization that can robustly represent unseen target domains by combining the domain attributes. To achieve attribute-based domain generalization system in language modeling, we introduce domain attribute-based experts to a multi-stream RNNLM called recurrent adaptive mixture model (RADMM) instead of domain-based experts. In the proposed system, a long short-term memory is independently trained on each domain attribute as an expert model. Then by integrating the outputs from all the experts in response to the context-dependent weight of the domain attributes of the current input, we predict the subsequent words in the unseen target domain and exploit the specific knowledge of each domain attribute. To demonstrate the effectiveness of our proposed domain attributes-centric language model, we experimentally compared the proposed model with conventional domain-centric language model by using texts taken from multiple domains including different writing styles, topics, dialects, and types of wordings. The experimental results demonstrated that lower perplexity can be achieved using domain attributes.

Original languageEnglish
Pages (from-to)150-160
Number of pages11
JournalIEICE Transactions on Information and Systems
Issue number1
Publication statusPublished - 2022


  • Domain attribute
  • Domain generalization
  • Language model
  • Mixture-of-experts
  • Recurrent adaptive mixture model

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering
  • Artificial Intelligence


Dive into the research topics of 'Multi-Source Domain Generalization Using Domain Attributes for Recurrent Neural Network Language Models'. Together they form a unique fingerprint.

Cite this