Multi-angle lipreading with angle classification-based feature extraction and its application to audio-visual speech recognition

Shinnosuke Isobe*, Satoshi Tamura, Satoru Hayamizu, Yuuto Gotoh, Masaki Nose

*この研究の対応する著者

研究成果: Article査読

抄録

Recently, automatic speech recognition (ASR) and visual speech recognition (VSR) have been widely researched owing to development in deep learning. Most VSR research works focus only on frontal face images. However, assuming real scenes, it is obvious that a VSR system should correctly recognize spoken contents from not only frontal but also diagonal or profile faces. In this paper, we propose a novel VSR method that is applicable to faces taken at any angle. Firstly, view classification is carried out to estimate face angles. Based on the results, feature extraction is then conducted using the best combination of pre-trained feature extraction models. Next, lipreading is carried out using the features. We also developed audio-visual speech recognition (AVSR) using the VSR in addition to conventional ASR. Audio results were obtained from ASR, followed by incorporating audio and visual results in a decision fusion manner. We evaluated our methods using OuluVS2, a multi-angle audio-visual database. We then confirmed that our approach achieved the best performance among conventional VSR schemes in a phrase classification task. In addition, we found that our AVSR results are better than ASR and VSR results.

本文言語English
論文番号182
ジャーナルFuture Internet
13
7
DOI
出版ステータスPublished - 2021 7
外部発表はい

ASJC Scopus subject areas

  • コンピュータ ネットワークおよび通信

フィンガープリント

「Multi-angle lipreading with angle classification-based feature extraction and its application to audio-visual speech recognition」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル