Speech recognition using deep canonical correlation analysis in noisy environments

Shinnosuke Isobe, Satoshi Tamura, Satoru Hayamizu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In this paper, we propose a method to improve the accuracy of speech recognition in noisy environments by utilizing Deep Canonical Correlation Analysis (DCCA). DCCA generates projections from two modalities into one common space, so that the correlation of projected vectors could be maximized. Our idea is to employ DCCA techniques with audio and visual modalities to enhance the robustness of Automatic Speech Recognition (ASR); A) noisy audio features can be recovered by clean visual features, and B) an ASR model can be trained using audio and visual features, as data augmentation. We evaluated our method using an audiovisual corpus CENSREC-1-AV and a noise database DEMAND. Compared to conventional ASR and feature-fusion-based audio-visual speech recognition, our DCCA-based recognizers achieved better performance. In addition, experimental results shows that utilizing DCCA enables us to get better results in various noisy environments, thanks to the visual modality. Furthermore, it is found that DCCA can be used as a data augmentation scheme if only a few training data are available, by incorporating visual DCCA features to build an audio-only ASR model, in addition to audio DCCA features.

Original languageEnglish
Title of host publicationICPRAM 2021 - Proceedings of the 10th International Conference on Pattern Recognition Applications and Methods
EditorsMaria De Marsico, Gabriella Sanniti di Baja, Ana Fred
PublisherSciTePress
Pages63-70
Number of pages8
ISBN (Electronic)9789897584862
Publication statusPublished - 2021
Externally publishedYes
Event10th International Conference on Pattern Recognition Applications and Methods, ICPRAM 2021 - Virtual, Online
Duration: 2021 Feb 42021 Feb 6

Publication series

NameICPRAM 2021 - Proceedings of the 10th International Conference on Pattern Recognition Applications and Methods

Conference

Conference10th International Conference on Pattern Recognition Applications and Methods, ICPRAM 2021
CityVirtual, Online
Period21/2/421/2/6

Keywords

  • Audio-visual processing
  • Canonical correlation analysis
  • Data augmentation
  • Deep learning
  • Noise robustness
  • Speech recognition

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition

Fingerprint

Dive into the research topics of 'Speech recognition using deep canonical correlation analysis in noisy environments'. Together they form a unique fingerprint.

Cite this