Improvement of Lipreading Performance Using Discriminative Feature and Speaker Adaptation

Takumi Seko, Naoya Ukai, Satoshi Tamura, Satoru Hayamizu

Research output: Contribution to conferencePaperpeer-review

4 Citations (Scopus)

Abstract

In this paper, we apply a general and discriminative feature”GIF” (Genetic Algorithm based Informative feature) to lipreading (visual speech recognition), and improve the lipreading performance using speaker adaptation. The feature extraction method consists of two transforms, which convert an input vector into GIF for recognition. In the speaker adaptation, MAP (Maximum A Posteriori) adaptation is used to adapt a recognition model to a target speaker. Recognition experiments of continuous digit utterances were conducted using an audio-visual corpus CENSREC-1-AV [1] including more than 268,000 lip images. At first, we compared the GIF-based method with the baseline method employing conventional eigenlip features, using two kinds of images: pictures in the database around speakers' mouth, and extracted images only containing lips. Secondly, we evaluated the effectiveness of speaker adaptation for lipreading. The result of comparison shows that the GIF-based approach achieved slightly better than the baseline method. And it is found using the mouth-around images is more suitable than lip-only images. Furthermore, the result of speaker adaptation shows that speaker adaptation significantly improved recognition accuracy in the GIF-based method; after the adaptation, the recognition rate drastically increased from approximately 30% to 70%.

Original languageEnglish
Pages221-226
Number of pages6
Publication statusPublished - 2013
Event2013 International Conference on Auditory-Visual Speech Processing, AVSP 2013 - Annecy, France
Duration: 2013 Aug 292013 Sep 1

Conference

Conference2013 International Conference on Auditory-Visual Speech Processing, AVSP 2013
Country/TerritoryFrance
CityAnnecy
Period13/8/2913/9/1

Keywords

  • CENSREC
  • discriminative feature
  • lip extraction
  • lipreading
  • speaker adaptation

ASJC Scopus subject areas

  • Language and Linguistics
  • Speech and Hearing
  • Otorhinolaryngology

Fingerprint

Dive into the research topics of 'Improvement of Lipreading Performance Using Discriminative Feature and Speaker Adaptation'. Together they form a unique fingerprint.

Cite this