Sound and visual tracking for humanoid robot

Hiroshi G. Okuno*, Kazuhiro Nakadai, Tino Lourens, Hiroaki Kitano

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

17 Citations (Scopus)


Mobile robots capable of auditory perception usually adopt the "stop-perceive-act" principle to avoid sounds made during moving due to motor noise. Although this principle reduces the complexity of the problems involved in auditory processing for mobile robots, it restricts their capabilities of auditory processing. In this paper, sound and visual tracking are investigated to compensate each other's drawbacks in tracking objects and to attain robust object tracking. Visual tracking may be difficult in case of occlusion, while sound tracking may be ambiguous in localization due to the nature of auditory processing. For this purpose, we present an active audition system for humanoid robot. The audition system of the highly intelligent humanoid requires localization of sound sources and identification of meanings of the sound in the auditory scene. The active audition reported in this paper focuses on improved sound source tracking by integrating audition, vision, and motor control. Given the multiple sound sources in the auditory scene, SIG the humanoid actively moves its head to improve localization by aligning microphones orthogonal to the sound source and by capturing the possible sound sources by vision. The system adaptively cancels motor noises using motor control signals. The experimental result demonstrates the effectiveness of sound and visual tracking.

Original languageEnglish
Pages (from-to)253-266
Number of pages14
JournalApplied Intelligence
Issue number3
Publication statusPublished - 2004 May
Externally publishedYes


  • Audio-visual integration
  • Audio-visual tracking
  • Computational auditory scene analysis
  • Robot audition

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Artificial Intelligence


Dive into the research topics of 'Sound and visual tracking for humanoid robot'. Together they form a unique fingerprint.

Cite this