Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network

Wataru Hinoshita, Hiroaki Arie, Jun Tani, Hiroshi G. Okuno, Tetsuya Ogata

Research output: Contribution to journalArticle

23 Citations (Scopus)

Abstract

We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences".

Original languageEnglish
Pages (from-to)311-320
Number of pages10
JournalNeural Networks
Volume24
Issue number4
DOIs
Publication statusPublished - 2011 May
Externally publishedYes

Fingerprint

Recurrent neural networks
Linguistics
Activation Analysis
Neurons
Chemical analysis
Language
Error correction
Substitution reactions
Chemical activation
Experiments

Keywords

  • Hierarchical linguistic structure
  • Language acquisition
  • Multiple timescale recurrent neural network
  • Self-organization

ASJC Scopus subject areas

  • Artificial Intelligence
  • Cognitive Neuroscience

Cite this

Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network. / Hinoshita, Wataru; Arie, Hiroaki; Tani, Jun; Okuno, Hiroshi G.; Ogata, Tetsuya.

In: Neural Networks, Vol. 24, No. 4, 05.2011, p. 311-320.

Research output: Contribution to journalArticle

@article{f4289c0f00e94adf95cf22bbde06e124,
title = "Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network",
abstract = "We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented {"}characters{"}, those that change more slowly, {"}words{"}, and those that change the slowest, {"}sentences{"}.",
keywords = "Hierarchical linguistic structure, Language acquisition, Multiple timescale recurrent neural network, Self-organization",
author = "Wataru Hinoshita and Hiroaki Arie and Jun Tani and Okuno, {Hiroshi G.} and Tetsuya Ogata",
year = "2011",
month = "5",
doi = "10.1016/j.neunet.2010.12.006",
language = "English",
volume = "24",
pages = "311--320",
journal = "Neural Networks",
issn = "0893-6080",
publisher = "Elsevier Limited",
number = "4",

}

TY - JOUR

T1 - Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network

AU - Hinoshita, Wataru

AU - Arie, Hiroaki

AU - Tani, Jun

AU - Okuno, Hiroshi G.

AU - Ogata, Tetsuya

PY - 2011/5

Y1 - 2011/5

N2 - We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences".

AB - We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences".

KW - Hierarchical linguistic structure

KW - Language acquisition

KW - Multiple timescale recurrent neural network

KW - Self-organization

UR - http://www.scopus.com/inward/record.url?scp=79951944498&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79951944498&partnerID=8YFLogxK

U2 - 10.1016/j.neunet.2010.12.006

DO - 10.1016/j.neunet.2010.12.006

M3 - Article

VL - 24

SP - 311

EP - 320

JO - Neural Networks

JF - Neural Networks

SN - 0893-6080

IS - 4

ER -