An algorithm for tracking guitarists' fingertips based on CNN-segmentation and ROI associated particle filter

Zhao Wang, Jun Ohya

Research output: Contribution to journalArticle


In this paper, we propose a new approach to tracking the fingertips of guitarists by embedding a CNN-based segmentation module and a temporal grouping-based ROI-association module combined with a particle filter. First, a CNN architecture is trained to segment hand area of each frame of input video. Then, four fingertip candidates (fore, middle, ring and little fingertips) on each frame are located by counting the vote number of template matching (TM) and reversed Hough transform (RHT). Furthermore, temporal grouping-based ROI association is applied to removal noise and group the fingertip candidates on consecutive frames. Finally, particles are distributed between associated fingertip candidates on every two adjacent frames for tracking the fingertips of guitarists. Experiments using videos containing multiple persons' guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-the-art tracking algorithm in terms of the hand area segmentation accuracy (98%) and the fingertip tracking mean error (5.16 pixel: 0.22 cm on the guitar neck) as well as computation efficiency.

Original languageEnglish
Article number020506-1
JournalJournal of Imaging Science and Technology
Issue number2
Publication statusPublished - 2019 Jan 1


ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Chemistry(all)
  • Atomic and Molecular Physics, and Optics
  • Computer Science Applications

Cite this