Robot audition from the viewpoint of computational auditory scene analysis

We have been engaged in research on computational auditory scene analysis to attain sophisticated robot/computer human interaction by manipulating real-world sound signals. The objective of our research is the understanding of an arbitrary sound mixture including music and environmental sounds as well as voiced speech, obtained by robot's ears (microphones) embedded on the robot. Three main issues in computational auditory scene analysis are sound source localization, separation, and recognition of separated sounds for a mixture of speech signals as well as polyphonic music signals. The Missing Feature Theory (MFT) approach integrates sound source separation and automatic speech recognition by generating missing feature masks. This robot audition system has been successfully ported to three kinds of robots, SIG2, Robovie R2 and Honda ASIMO. A robot recognizes three simultaneous speeches such as placing a meal order ora referee for RockPaper-Scissors Sound Games with a delay of less than 2 seconds. The real-time beat tracking system is also developed for robot audition. A robot hears music, understands and predicts its musical beats to behave in accordance with the beat times in real-time.