Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network

Wataru Hinoshita*, Hiroaki Arie, Jun Tani, Hiroshi G. Okuno, Tetsuya Ogata


研究成果: Article査読

27 被引用数 (Scopus)


We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences".

ジャーナルNeural Networks
出版ステータスPublished - 2011 5月

ASJC Scopus subject areas

  • 認知神経科学
  • 人工知能


「Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。