Making a robot recognize three simultaneous sentences in real-time

Shun'Ichi Yamamoto*, Kazuhiro Nakadai, Jean Marc Valin, Jean Rouat, François Michaud, Kazunori Komatani, Tetsuya Ogata, Hiroshi G. Okuno

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

31 Citations (Scopus)


A humanoid robot under real-world environments usually hears mixtures of sounds, and thus three capabilities are essential for robot audition; sound source localization, separation, and recognition of separated sounds. We have adopted the missing feature theory (MFT) for automatic recognition of separated speech, and developed the robot audition system. A microphone array is used along with a real-time dedicated implementation of Geometric Source Separation (GSS) and a multi-channel post-filter that gives us a further reduction of interferences from other sources. The automatic speech recognition based on MFT recognizes separated sounds by generating missing feature masks automatically from the post-filtering step. The main advantage of this approach for humanoid robots resides in the fact that the ASR with a clean acoustic model can adapt the distortion of separated sound by consulting the post-filter feature masks. In this paper, we used the improved Julius as an MFT-based automatic speech recognizer (ASR). The Julius is a real-time large vocabulary continuous speech recognition (LVCSR) system. We performed the experiment to evaluate our robot audition system. In this experiment, the system recognizes a sentence, not an isolated word. We showed the improvement in the system performance through three simultaneous speech recognition on the humanoid SIG2.

Original languageEnglish
Title of host publication2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS
PublisherIEEE Computer Society
Number of pages6
ISBN (Print)0780389123, 9780780389120
Publication statusPublished - 2005
Externally publishedYes

Publication series

Name2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS


  • Automatic missing feature mask generation
  • Continuous speech recognition
  • Missing feature theory
  • Robot audition

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction
  • Control and Systems Engineering


Dive into the research topics of 'Making a robot recognize three simultaneous sentences in real-time'. Together they form a unique fingerprint.

Cite this