Real-time auditory and visual multiple-object tracking for humanoids

Kazuhiro Nakadai, Ken Ichi Hidai, Hiroshi Mizoguchi, Hiroshi G. Okuno, Hiroaki Kitano

研究成果: Conference contribution

93 被引用数 (Scopus)


This paper presents a real-time auditory and visual tracking of multiple objects for humanoid under real-world environments. Real-time processing is crucial for sensorimotor tasks in tracking, and multiple-object tracking is crucial for real-world applications. Multiple sound source tracking needs perception of a mixture of sounds and cancellation of motor noises caused by body movements. However its real-time processing has not been reported yet. Real-time tracking is attained by fusing information obtained by sound source localization, multiple face recognition, speaker tracking, focus of attention control, and motor control. Auditory streams with sound source direction are extracted by active audition system with motor noise cancellation capability from 48KHz sampling sounds. Visual streams with face ID and 3D-position are extracted by combining skin-color extraction, correlation-based matching, and multiple-scale image generation from a single camera. These auditory and visual streams are associated by comparing the spatial location, and associated streams are used to control focus of attention. Auditory, visual, and association processing are performed asynchronously on different PC's connected by TCP/IP network. The resulting system implemented on an upper-torso humanoid can track multiple objects with the delay of 200 msec, which is forced by visual tracking and network latency.

ホスト出版物のタイトルIJCAI International Joint Conference on Artificial Intelligence
出版ステータスPublished - 2001
イベント17th International Joint Conference on Artificial Intelligence, IJCAI 2001 - Seattle, WA, United States
継続期間: 2001 8月 42001 8月 10


Other17th International Joint Conference on Artificial Intelligence, IJCAI 2001
国/地域United States
CitySeattle, WA

ASJC Scopus subject areas

  • 人工知能


「Real-time auditory and visual multiple-object tracking for humanoids」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。