Use of a sparse structure to improve learning performance of recurrent neural networks

Hiromitsu Awano; Shun Nishide; Hiroaki Arie; Jun Tani; Toru Takahashi; Hiroshi G. Okuno; Tetsuya Ogata

doi:10.1007/978-3-642-24965-5_36

Use of a sparse structure to improve learning performance of recurrent neural networks

Hiromitsu Awano^*, Shun Nishide, Hiroaki Arie, Jun Tani, Toru Takahashi, Hiroshi G. Okuno, Tetsuya Ogata

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.

Original language	English
Title of host publication	Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings
Pages	323-331
Number of pages	9
Edition	PART 3
DOIs	https://doi.org/10.1007/978-3-642-24965-5_36
Publication status	Published - 2011
Externally published	Yes
Event	18th International Conference on Neural Information Processing, ICONIP 2011 - Shanghai, China Duration: 2011 Nov 13 → 2011 Nov 17

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 3
Volume	7064 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th International Conference on Neural Information Processing, ICONIP 2011
Country/Territory	China
City	Shanghai
Period	11/11/13 → 11/11/17

Keywords

Recurrent Neural Networks
Sparse Coding
Sparse Structure

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-642-24965-5_36

Cite this

Awano, H., Nishide, S., Arie, H., Tani, J., Takahashi, T., Okuno, H. G., & Ogata, T. (2011). Use of a sparse structure to improve learning performance of recurrent neural networks. In Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings (PART 3 ed., pp. 323-331). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7064 LNCS, No. PART 3). https://doi.org/10.1007/978-3-642-24965-5_36

Use of a sparse structure to improve learning performance of recurrent neural networks. / Awano, Hiromitsu; Nishide, Shun; Arie, Hiroaki et al.

Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings. PART 3. ed. 2011. p. 323-331 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7064 LNCS, No. PART 3).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Awano, H, Nishide, S, Arie, H, Tani, J, Takahashi, T, Okuno, HG & Ogata, T 2011, Use of a sparse structure to improve learning performance of recurrent neural networks. in Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings. PART 3 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 3, vol. 7064 LNCS, pp. 323-331, 18th International Conference on Neural Information Processing, ICONIP 2011, Shanghai, China, 11/11/13. https://doi.org/10.1007/978-3-642-24965-5_36

Awano H, Nishide S, Arie H, Tani J, Takahashi T, Okuno HG et al. Use of a sparse structure to improve learning performance of recurrent neural networks. In Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings. PART 3 ed. 2011. p. 323-331. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 3). doi: 10.1007/978-3-642-24965-5_36

Awano, Hiromitsu ; Nishide, Shun ; Arie, Hiroaki et al. / Use of a sparse structure to improve learning performance of recurrent neural networks. Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings. PART 3. ed. 2011. pp. 323-331 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 3).

@inproceedings{be627426bdb149648f244253fdcd49fc,

title = "Use of a sparse structure to improve learning performance of recurrent neural networks",

abstract = "The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.",

keywords = "Recurrent Neural Networks, Sparse Coding, Sparse Structure",

author = "Hiromitsu Awano and Shun Nishide and Hiroaki Arie and Jun Tani and Toru Takahashi and Okuno, {Hiroshi G.} and Tetsuya Ogata",

year = "2011",

doi = "10.1007/978-3-642-24965-5_36",

language = "English",

isbn = "9783642249648",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 3",

pages = "323--331",

booktitle = "Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings",

edition = "PART 3",

note = "18th International Conference on Neural Information Processing, ICONIP 2011 ; Conference date: 13-11-2011 Through 17-11-2011",

}

TY - GEN

T1 - Use of a sparse structure to improve learning performance of recurrent neural networks

AU - Awano, Hiromitsu

AU - Nishide, Shun

AU - Arie, Hiroaki

AU - Tani, Jun

AU - Takahashi, Toru

AU - Okuno, Hiroshi G.

AU - Ogata, Tetsuya

PY - 2011

Y1 - 2011

N2 - The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.

AB - The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.

KW - Recurrent Neural Networks

KW - Sparse Coding

KW - Sparse Structure

UR - http://www.scopus.com/inward/record.url?scp=81855227141&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=81855227141&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-24965-5_36

DO - 10.1007/978-3-642-24965-5_36

M3 - Conference contribution

AN - SCOPUS:81855227141

SN - 9783642249648

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 323

EP - 331

BT - Neural Information Processing - 18th International Conference, ICONIP 2011, Proceedings

T2 - 18th International Conference on Neural Information Processing, ICONIP 2011

Y2 - 13 November 2011 through 17 November 2011

ER -

Use of a sparse structure to improve learning performance of recurrent neural networks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this