Improvement of audio-visual score following in robot ensemble with human guitarist

Tatsuhiko Itohara, Kazuhiro Nakadai, Tetsuya Ogata, Hiroshi G. Okuno

研究成果

6 被引用数 (Scopus)

抄録

Our goal is to create an ensemble between human guitarists and music robots, e.g., singing and playing instruments robots. Such robots need to detect the tempo and beat time of the music. Score following and beat tracking, which requires and does not requires a score, are commonly used for this purpose. Score following is an incremental audio-to-score alignment. Although most score following methods assume that players have a precise score, most scores for guitarists have only melody and chord sequences without any beat patterns. An audio-visual beat tracking for guitarists is reported that improves the accuracy of beat detection. However, the result of this method is still insufficient because it uses only onset information, not pitch information, and because the hand tracking shows low accuracy. In this paper, we report a multimodal score following for a guitar performance, an extension of an audio-visual beat tracking method. The main difference is to use chord sequences to improve tracking of audio signals and depth information to improve tracking of guitar playing. Chord sequences are used for the calculation of chord correlation between the input and a score. Depth information is used in the guitar plane masking by three dimensional Hough transform, for the stable detection of a hand. Finally, the system extracts score positions and tempos by a particle-filter based integration of audio and visual features, The resulting score following system improves the tempo and the score position of a performance by 0.2 [sec] compared to an existing system.

本文言語English
ホスト出版物のタイトル2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012
ページ574-579
ページ数6
DOI
出版ステータスPublished - 2012 12 1
イベント2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012 - Osaka, Japan
継続期間: 2012 11 292012 12 1

出版物シリーズ

名前IEEE-RAS International Conference on Humanoid Robots
ISSN(印刷版)2164-0572
ISSN(電子版)2164-0580

Conference

Conference2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012
国/地域Japan
CityOsaka
Period12/11/2912/12/1

ASJC Scopus subject areas

  • 人工知能
  • コンピュータ ビジョンおよびパターン認識
  • ハードウェアとアーキテクチャ
  • 人間とコンピュータの相互作用
  • 電子工学および電気工学

フィンガープリント

「Improvement of audio-visual score following in robot ensemble with human guitarist」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル