A 3D guitar fingering assessing system based on CNN-Hand pose estimation and SVR-Assessment

Zhao Wang, Jun Ohya

    Research output: Contribution to journalConference article

    Abstract

    This paper proposes a guitar fingering assessing system based on CNN (Convolutional Neural Network) hand pose estimation and SVR (Support Vector Regression) evaluation. To spur our progress, first, a CNN architecture is proposed to estimate temporal 3D position of 16 joints of hand; then, based on a DCT (Discrete Cosine Transform) feature and SVR, fingering of guitarist is scored to interpret how well guitarist played. We also release a new dataset for professional guitar playing analysis with significant advantage in total number of video, professional judgement by expert of guitarist, accurate annotation for hand pose and score of guitar performance. Experiments using videos containing multiple persons' guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-art with (1) low mean error (Euclid distance of 6,1 mm) and high computation efficiency for hand pose estimation; (2) high rank correlation (0.68) for assessing the fingering (C major scale and symmetrical excise) of guitarists.

    Original languageEnglish
    Pages (from-to)2781-2785
    Number of pages5
    JournalIS and T International Symposium on Electronic Imaging Science and Technology
    VolumePart F138660
    DOIs
    Publication statusPublished - 2018 Jan 1
    EventIntelligent Robotics and Industrial Applications using Computer Vision 2018, IRIACV 2018 - Burlingame, United States
    Duration: 2018 Jan 282018 Feb 1

    Fingerprint

    regression analysis
    Neural networks
    annotations
    discrete cosine transform
    Discrete cosine transforms
    Network architecture
    evaluation
    estimates
    Experiments

    ASJC Scopus subject areas

    • Computer Graphics and Computer-Aided Design
    • Computer Science Applications
    • Human-Computer Interaction
    • Software
    • Electrical and Electronic Engineering
    • Atomic and Molecular Physics, and Optics

    Cite this

    A 3D guitar fingering assessing system based on CNN-Hand pose estimation and SVR-Assessment. / Wang, Zhao; Ohya, Jun.

    In: IS and T International Symposium on Electronic Imaging Science and Technology, Vol. Part F138660, 01.01.2018, p. 2781-2785.

    Research output: Contribution to journalConference article

    @article{8837e0d777a0459c807dd9f8d672889d,
    title = "A 3D guitar fingering assessing system based on CNN-Hand pose estimation and SVR-Assessment",
    abstract = "This paper proposes a guitar fingering assessing system based on CNN (Convolutional Neural Network) hand pose estimation and SVR (Support Vector Regression) evaluation. To spur our progress, first, a CNN architecture is proposed to estimate temporal 3D position of 16 joints of hand; then, based on a DCT (Discrete Cosine Transform) feature and SVR, fingering of guitarist is scored to interpret how well guitarist played. We also release a new dataset for professional guitar playing analysis with significant advantage in total number of video, professional judgement by expert of guitarist, accurate annotation for hand pose and score of guitar performance. Experiments using videos containing multiple persons' guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-art with (1) low mean error (Euclid distance of 6,1 mm) and high computation efficiency for hand pose estimation; (2) high rank correlation (0.68) for assessing the fingering (C major scale and symmetrical excise) of guitarists.",
    author = "Zhao Wang and Jun Ohya",
    year = "2018",
    month = "1",
    day = "1",
    doi = "10.2352/ISSN.2470-1173.2018.09.IRIACV-204",
    language = "English",
    volume = "Part F138660",
    pages = "2781--2785",
    journal = "IS and T International Symposium on Electronic Imaging Science and Technology",
    issn = "2470-1173",

    }

    TY - JOUR

    T1 - A 3D guitar fingering assessing system based on CNN-Hand pose estimation and SVR-Assessment

    AU - Wang, Zhao

    AU - Ohya, Jun

    PY - 2018/1/1

    Y1 - 2018/1/1

    N2 - This paper proposes a guitar fingering assessing system based on CNN (Convolutional Neural Network) hand pose estimation and SVR (Support Vector Regression) evaluation. To spur our progress, first, a CNN architecture is proposed to estimate temporal 3D position of 16 joints of hand; then, based on a DCT (Discrete Cosine Transform) feature and SVR, fingering of guitarist is scored to interpret how well guitarist played. We also release a new dataset for professional guitar playing analysis with significant advantage in total number of video, professional judgement by expert of guitarist, accurate annotation for hand pose and score of guitar performance. Experiments using videos containing multiple persons' guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-art with (1) low mean error (Euclid distance of 6,1 mm) and high computation efficiency for hand pose estimation; (2) high rank correlation (0.68) for assessing the fingering (C major scale and symmetrical excise) of guitarists.

    AB - This paper proposes a guitar fingering assessing system based on CNN (Convolutional Neural Network) hand pose estimation and SVR (Support Vector Regression) evaluation. To spur our progress, first, a CNN architecture is proposed to estimate temporal 3D position of 16 joints of hand; then, based on a DCT (Discrete Cosine Transform) feature and SVR, fingering of guitarist is scored to interpret how well guitarist played. We also release a new dataset for professional guitar playing analysis with significant advantage in total number of video, professional judgement by expert of guitarist, accurate annotation for hand pose and score of guitar performance. Experiments using videos containing multiple persons' guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-art with (1) low mean error (Euclid distance of 6,1 mm) and high computation efficiency for hand pose estimation; (2) high rank correlation (0.68) for assessing the fingering (C major scale and symmetrical excise) of guitarists.

    UR - http://www.scopus.com/inward/record.url?scp=85052905106&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85052905106&partnerID=8YFLogxK

    U2 - 10.2352/ISSN.2470-1173.2018.09.IRIACV-204

    DO - 10.2352/ISSN.2470-1173.2018.09.IRIACV-204

    M3 - Conference article

    AN - SCOPUS:85052905106

    VL - Part F138660

    SP - 2781

    EP - 2785

    JO - IS and T International Symposium on Electronic Imaging Science and Technology

    JF - IS and T International Symposium on Electronic Imaging Science and Technology

    SN - 2470-1173

    ER -