Improvement of recognition of simultaneous speech signals using AV integration and scattering theory for humanoid robots

Kazuhiro Nakadai, Daisuke Matsuura, Hiroshi G. Okuno, Hiroshi Tsujino

Research output: Contribution to journalArticle

44 Citations (Scopus)

Abstract

This paper presents a method to improve recognition of three simultaneous speech signals by a humanoid robot equipped with a pair of microphones. In such situations, sound separation and automatic speech recognition (ASR) of the separated speech signal are difficult, because the signal-to-noise ratio is quite low (around -3 dB) and noise is not stable due to interfering voices. To improve recognition of three simultaneous speech signals, two key ideas are introduced. One is two-layered audio-visual integration of both name (ID) and location, that is, speech and face recognition, and speech and face localization. The other is acoustical modeling of the humanoid head by scattering theory. Sound sources are separated in real-time by an active direction-pass filter (ADPF), which extracts sounds from a specified direction by using the interaural phase/intensity difference estimated by scattering theory. Since features of separated sounds vary according to the sound direction, multiple direction- and speaker-dependent acoustic models are used. The system integrates ASR results by using the sound direction and speaker information provided by face recognition as well as confidence measure of ASR results to select the best one. The resulting system shows an improvement of about 10% on average against recognition of three simultaneous speech signals, where three speakers were located around the humanoid on a 1 m radius half circle, one of them being in front of him (angle 0°) and the other two being at symmetrical positions (±θ) varying by 10° steps from 0° to 90°.

Original languageEnglish
Pages (from-to)97-112
Number of pages16
JournalSpeech Communication
Volume44
Issue number1-4 SPEC. ISS.
DOIs
Publication statusPublished - 2004 Oct
Externally publishedYes

Keywords

  • Active audition
  • Audio-visual integration
  • Robot audition
  • Scattering theory
  • Sound source localization
  • Sound source separation
  • Speech recognition

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering
  • Experimental and Cognitive Psychology
  • Linguistics and Language

Fingerprint Dive into the research topics of 'Improvement of recognition of simultaneous speech signals using AV integration and scattering theory for humanoid robots'. Together they form a unique fingerprint.

  • Cite this