Structure discovery of deep neural network based on evolutionary algorithms

Takahiro Shinozaki, Shinji Watanabe

Research output: Chapter in Book/Report/Conference proceedingConference contribution

20 Citations (Scopus)

Abstract

Deep neural networks (DNNs) are constructed by considering highly complicated configurations including network structure and several tuning parameters (number of hidden states and learning rate in each layer), which greatly affect the performance of speech processing applications. To reach optimal performance in such systems, deep understanding and expertise in DNNs is necessary, which limits the development of DNN systems to skilled experts. To overcome the problem, this paper proposes an efficient optimization strategy for DNN structure and parameters using evolutionary algorithms. The proposed approach parametrizes the DNN structure by a directed acyclic graph, and the DNN structure is represented by a simple binary vector. Genetic algorithm and covariance matrix adaptation evolution strategy efficiently optimize the performance jointly with respect to the above binary vector and the other tuning parameters. Experiments on phoneme recognition and spoken digit detection tasks show the effectiveness of the proposed approach by discovering the appropriate DNN structure automatically.

Original languageEnglish
Title of host publication2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages4979-4983
Number of pages5
Volume2015-August
ISBN (Electronic)9781467369978
DOIs
Publication statusPublished - 2015 Aug 4
Externally publishedYes
Event40th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Brisbane, Australia
Duration: 2014 Apr 192014 Apr 24

Other

Other40th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015
CountryAustralia
CityBrisbane
Period14/4/1914/4/24

Fingerprint

Evolutionary algorithms
Tuning
Speech processing
Covariance matrix
Deep neural networks
Genetic algorithms
Experiments

Keywords

  • CMA-ES
  • Deep neural network
  • evolutionary algorithm
  • genetic algorithm
  • structure optimization

ASJC Scopus subject areas

  • Signal Processing
  • Software
  • Electrical and Electronic Engineering

Cite this

Shinozaki, T., & Watanabe, S. (2015). Structure discovery of deep neural network based on evolutionary algorithms. In 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings (Vol. 2015-August, pp. 4979-4983). [7178918] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2015.7178918

Structure discovery of deep neural network based on evolutionary algorithms. / Shinozaki, Takahiro; Watanabe, Shinji.

2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings. Vol. 2015-August Institute of Electrical and Electronics Engineers Inc., 2015. p. 4979-4983 7178918.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Shinozaki, T & Watanabe, S 2015, Structure discovery of deep neural network based on evolutionary algorithms. in 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings. vol. 2015-August, 7178918, Institute of Electrical and Electronics Engineers Inc., pp. 4979-4983, 40th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015, Brisbane, Australia, 14/4/19. https://doi.org/10.1109/ICASSP.2015.7178918
Shinozaki T, Watanabe S. Structure discovery of deep neural network based on evolutionary algorithms. In 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings. Vol. 2015-August. Institute of Electrical and Electronics Engineers Inc. 2015. p. 4979-4983. 7178918 https://doi.org/10.1109/ICASSP.2015.7178918
Shinozaki, Takahiro ; Watanabe, Shinji. / Structure discovery of deep neural network based on evolutionary algorithms. 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings. Vol. 2015-August Institute of Electrical and Electronics Engineers Inc., 2015. pp. 4979-4983
@inproceedings{b48aba36195147b9acbdba9bd7223e89,
title = "Structure discovery of deep neural network based on evolutionary algorithms",
abstract = "Deep neural networks (DNNs) are constructed by considering highly complicated configurations including network structure and several tuning parameters (number of hidden states and learning rate in each layer), which greatly affect the performance of speech processing applications. To reach optimal performance in such systems, deep understanding and expertise in DNNs is necessary, which limits the development of DNN systems to skilled experts. To overcome the problem, this paper proposes an efficient optimization strategy for DNN structure and parameters using evolutionary algorithms. The proposed approach parametrizes the DNN structure by a directed acyclic graph, and the DNN structure is represented by a simple binary vector. Genetic algorithm and covariance matrix adaptation evolution strategy efficiently optimize the performance jointly with respect to the above binary vector and the other tuning parameters. Experiments on phoneme recognition and spoken digit detection tasks show the effectiveness of the proposed approach by discovering the appropriate DNN structure automatically.",
keywords = "CMA-ES, Deep neural network, evolutionary algorithm, genetic algorithm, structure optimization",
author = "Takahiro Shinozaki and Shinji Watanabe",
year = "2015",
month = "8",
day = "4",
doi = "10.1109/ICASSP.2015.7178918",
language = "English",
volume = "2015-August",
pages = "4979--4983",
booktitle = "2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

TY - GEN

T1 - Structure discovery of deep neural network based on evolutionary algorithms

AU - Shinozaki, Takahiro

AU - Watanabe, Shinji

PY - 2015/8/4

Y1 - 2015/8/4

N2 - Deep neural networks (DNNs) are constructed by considering highly complicated configurations including network structure and several tuning parameters (number of hidden states and learning rate in each layer), which greatly affect the performance of speech processing applications. To reach optimal performance in such systems, deep understanding and expertise in DNNs is necessary, which limits the development of DNN systems to skilled experts. To overcome the problem, this paper proposes an efficient optimization strategy for DNN structure and parameters using evolutionary algorithms. The proposed approach parametrizes the DNN structure by a directed acyclic graph, and the DNN structure is represented by a simple binary vector. Genetic algorithm and covariance matrix adaptation evolution strategy efficiently optimize the performance jointly with respect to the above binary vector and the other tuning parameters. Experiments on phoneme recognition and spoken digit detection tasks show the effectiveness of the proposed approach by discovering the appropriate DNN structure automatically.

AB - Deep neural networks (DNNs) are constructed by considering highly complicated configurations including network structure and several tuning parameters (number of hidden states and learning rate in each layer), which greatly affect the performance of speech processing applications. To reach optimal performance in such systems, deep understanding and expertise in DNNs is necessary, which limits the development of DNN systems to skilled experts. To overcome the problem, this paper proposes an efficient optimization strategy for DNN structure and parameters using evolutionary algorithms. The proposed approach parametrizes the DNN structure by a directed acyclic graph, and the DNN structure is represented by a simple binary vector. Genetic algorithm and covariance matrix adaptation evolution strategy efficiently optimize the performance jointly with respect to the above binary vector and the other tuning parameters. Experiments on phoneme recognition and spoken digit detection tasks show the effectiveness of the proposed approach by discovering the appropriate DNN structure automatically.

KW - CMA-ES

KW - Deep neural network

KW - evolutionary algorithm

KW - genetic algorithm

KW - structure optimization

UR - http://www.scopus.com/inward/record.url?scp=84946084313&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84946084313&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2015.7178918

DO - 10.1109/ICASSP.2015.7178918

M3 - Conference contribution

VL - 2015-August

SP - 4979

EP - 4983

BT - 2015 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2015 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -