Sequence discriminative training for low-rank deep neural networks

Yuuki Tachioka, Shinji Watanabe, Jonathan Le Roux, John R. Hershey

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Deep neural networks (DNNs) have proven very successful for automatic speech recognition but the number of parameters tends to be large, leading to high computational cost. To reduce the size of a DNN model, low-rank approximations of weight matrices, computed using singular value decomposition (SVD), have previously been applied. Previous studies only focused on clean speech, whereas the additional variability in noisy speech could make model reduction difficult. Thus we investigate the effectiveness of this SVD method on noisy reverberated speech. Furthermore, we combine the low-rank approximation with sequence discriminative training, which further improved the performance of the DNN, even though the original DNN was constructed using a discriminative criterion. We also investigated the effect of the order of application of the low-rank and sequence discriminative training. Our experiments show that low rank approximation is effective for noisy speech and the most effective combination of discriminative training with model reduction is to apply the low rank approximation to the base model first and then to perform discriminative training on the low-rank model. This low-rank discriminatively trained model outperformed the full discriminatively trained model.

Original languageEnglish
Title of host publication2014 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2014
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages572-576
Number of pages5
ISBN (Electronic)9781479970889
DOIs
Publication statusPublished - 2014 Feb 5
Externally publishedYes
Event2014 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2014 - Atlanta, United States
Duration: 2014 Dec 32014 Dec 5

Other

Other2014 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2014
CountryUnited States
CityAtlanta
Period14/12/314/12/5

    Fingerprint

Keywords

  • Automatic speech recognition
  • Deep neural networks
  • Discriminative training
  • Singular value decomposition

ASJC Scopus subject areas

  • Signal Processing
  • Information Systems

Cite this

Tachioka, Y., Watanabe, S., Le Roux, J., & Hershey, J. R. (2014). Sequence discriminative training for low-rank deep neural networks. In 2014 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2014 (pp. 572-576). [7032182] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/GlobalSIP.2014.7032182