Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network

Wataru Hinoshita, Hiroaki Arie, Jun Tani, Hiroshi G. Okuno, Tetsuya Ogata

Research output: Contribution to journalArticle

26 Citations (Scopus)

Abstract

We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences".

Original languageEnglish
Pages (from-to)311-320
Number of pages10
JournalNeural Networks
Volume24
Issue number4
DOIs
Publication statusPublished - 2011 May 1
Externally publishedYes

Keywords

  • Hierarchical linguistic structure
  • Language acquisition
  • Multiple timescale recurrent neural network
  • Self-organization

ASJC Scopus subject areas

  • Cognitive Neuroscience
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network'. Together they form a unique fingerprint.

  • Cite this