Improvement of audio-visual score following in robot ensemble with human guitarist

Tatsuhiko Itohara, Kazuhiro Nakadai, Tetsuya Ogata, Hiroshi G. Okuno

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Citations (Scopus)

Abstract

Our goal is to create an ensemble between human guitarists and music robots, e.g., singing and playing instruments robots. Such robots need to detect the tempo and beat time of the music. Score following and beat tracking, which requires and does not requires a score, are commonly used for this purpose. Score following is an incremental audio-to-score alignment. Although most score following methods assume that players have a precise score, most scores for guitarists have only melody and chord sequences without any beat patterns. An audio-visual beat tracking for guitarists is reported that improves the accuracy of beat detection. However, the result of this method is still insufficient because it uses only onset information, not pitch information, and because the hand tracking shows low accuracy. In this paper, we report a multimodal score following for a guitar performance, an extension of an audio-visual beat tracking method. The main difference is to use chord sequences to improve tracking of audio signals and depth information to improve tracking of guitar playing. Chord sequences are used for the calculation of chord correlation between the input and a score. Depth information is used in the guitar plane masking by three dimensional Hough transform, for the stable detection of a hand. Finally, the system extracts score positions and tempos by a particle-filter based integration of audio and visual features, The resulting score following system improves the tempo and the score position of a performance by 0.2 [sec] compared to an existing system.

Original languageEnglish
Title of host publication2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012
Pages574-579
Number of pages6
DOIs
Publication statusPublished - 2012 Dec 1
Event2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012 - Osaka, Japan
Duration: 2012 Nov 292012 Dec 1

Publication series

NameIEEE-RAS International Conference on Humanoid Robots
ISSN (Print)2164-0572
ISSN (Electronic)2164-0580

Conference

Conference2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012
CountryJapan
CityOsaka
Period12/11/2912/12/1

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Human-Computer Interaction
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Improvement of audio-visual score following in robot ensemble with human guitarist'. Together they form a unique fingerprint.

Cite this