Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot

Klaus Petersen, Jorge Solis, Atsuo Takanishi

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    9 Citations (Scopus)

    Abstract

    The Waseda Flutist Robot is able to play the flute at the level of an intermediate human player. This ability opens a wide field of possibilities to research human-robot musical interaction. This research is focused on enabling the flutist robot to interact more naturally with musical partners in the context of a Jazz band. For this purpose a Musical-Based Interaction System (MbIS) has been proposed to enable the robot to process both visual and aural cues coming throughout the interaction with musicians. In a previous publication, we have concentrated on the implementation of visual communication techniques. We created an interaction interface that enabled the robot to detect instrument gestures of partner musicians during a musical performance. Two computer vision approaches were implemented to create a two-skill-level interface for visual human-robot interaction in a musical context. In this paper we focus on the aural perception system of the robot. The method introduced here enables the robot to, a suitable environment provided, detect the tempo and harmony of a partner musician's play, with a specific focus on improvisation. We achieve this by examining the rhythmical and harmonic characteristics of the recorded sound. We apply the same approach to amplitude and frequency spectrum, thus, in the former case tracking amplitude transients. In the latter case, as we focus on communication with monophonic woodwind instruments, we follow the most prominent peak in the frequency spectrum. We specifically use a similar technique for the audio analysis as we did for our previous research on motion tracking. From the experimental results, we have shown that after implementing our algorithm the robot is able to correctly recognize a number of rhythms and harmonies. It is able to engage in a simple form of stimuli and reaction play with a human musician.

    Original languageEnglish
    Title of host publication2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009
    Pages2303-2308
    Number of pages6
    DOIs
    Publication statusPublished - 2009 Dec 11
    Event2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009 - St. Louis, MO
    Duration: 2009 Oct 112009 Oct 15

    Other

    Other2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009
    CitySt. Louis, MO
    Period09/10/1109/10/15

    Fingerprint

    Robots
    Visual communication
    Human robot interaction
    Computer vision
    Acoustic waves
    Communication

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Computer Vision and Pattern Recognition
    • Human-Computer Interaction
    • Control and Systems Engineering

    Cite this

    Petersen, K., Solis, J., & Takanishi, A. (2009). Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009 (pp. 2303-2308). [5354550] https://doi.org/10.1109/IROS.2009.5354550

    Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot. / Petersen, Klaus; Solis, Jorge; Takanishi, Atsuo.

    2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009. 2009. p. 2303-2308 5354550.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Petersen, K, Solis, J & Takanishi, A 2009, Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot. in 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009., 5354550, pp. 2303-2308, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009, St. Louis, MO, 09/10/11. https://doi.org/10.1109/IROS.2009.5354550
    Petersen K, Solis J, Takanishi A. Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009. 2009. p. 2303-2308. 5354550 https://doi.org/10.1109/IROS.2009.5354550
    Petersen, Klaus ; Solis, Jorge ; Takanishi, Atsuo. / Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot. 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009. 2009. pp. 2303-2308
    @inproceedings{10fe6099c9c7407abab471a27df43f64,
    title = "Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot",
    abstract = "The Waseda Flutist Robot is able to play the flute at the level of an intermediate human player. This ability opens a wide field of possibilities to research human-robot musical interaction. This research is focused on enabling the flutist robot to interact more naturally with musical partners in the context of a Jazz band. For this purpose a Musical-Based Interaction System (MbIS) has been proposed to enable the robot to process both visual and aural cues coming throughout the interaction with musicians. In a previous publication, we have concentrated on the implementation of visual communication techniques. We created an interaction interface that enabled the robot to detect instrument gestures of partner musicians during a musical performance. Two computer vision approaches were implemented to create a two-skill-level interface for visual human-robot interaction in a musical context. In this paper we focus on the aural perception system of the robot. The method introduced here enables the robot to, a suitable environment provided, detect the tempo and harmony of a partner musician's play, with a specific focus on improvisation. We achieve this by examining the rhythmical and harmonic characteristics of the recorded sound. We apply the same approach to amplitude and frequency spectrum, thus, in the former case tracking amplitude transients. In the latter case, as we focus on communication with monophonic woodwind instruments, we follow the most prominent peak in the frequency spectrum. We specifically use a similar technique for the audio analysis as we did for our previous research on motion tracking. From the experimental results, we have shown that after implementing our algorithm the robot is able to correctly recognize a number of rhythms and harmonies. It is able to engage in a simple form of stimuli and reaction play with a human musician.",
    author = "Klaus Petersen and Jorge Solis and Atsuo Takanishi",
    year = "2009",
    month = "12",
    day = "11",
    doi = "10.1109/IROS.2009.5354550",
    language = "English",
    isbn = "9781424438044",
    pages = "2303--2308",
    booktitle = "2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009",

    }

    TY - GEN

    T1 - Development of a aural real-time rhythmical and harmonic tracking to enable the musical interaction with the Waseda Flutist robot

    AU - Petersen, Klaus

    AU - Solis, Jorge

    AU - Takanishi, Atsuo

    PY - 2009/12/11

    Y1 - 2009/12/11

    N2 - The Waseda Flutist Robot is able to play the flute at the level of an intermediate human player. This ability opens a wide field of possibilities to research human-robot musical interaction. This research is focused on enabling the flutist robot to interact more naturally with musical partners in the context of a Jazz band. For this purpose a Musical-Based Interaction System (MbIS) has been proposed to enable the robot to process both visual and aural cues coming throughout the interaction with musicians. In a previous publication, we have concentrated on the implementation of visual communication techniques. We created an interaction interface that enabled the robot to detect instrument gestures of partner musicians during a musical performance. Two computer vision approaches were implemented to create a two-skill-level interface for visual human-robot interaction in a musical context. In this paper we focus on the aural perception system of the robot. The method introduced here enables the robot to, a suitable environment provided, detect the tempo and harmony of a partner musician's play, with a specific focus on improvisation. We achieve this by examining the rhythmical and harmonic characteristics of the recorded sound. We apply the same approach to amplitude and frequency spectrum, thus, in the former case tracking amplitude transients. In the latter case, as we focus on communication with monophonic woodwind instruments, we follow the most prominent peak in the frequency spectrum. We specifically use a similar technique for the audio analysis as we did for our previous research on motion tracking. From the experimental results, we have shown that after implementing our algorithm the robot is able to correctly recognize a number of rhythms and harmonies. It is able to engage in a simple form of stimuli and reaction play with a human musician.

    AB - The Waseda Flutist Robot is able to play the flute at the level of an intermediate human player. This ability opens a wide field of possibilities to research human-robot musical interaction. This research is focused on enabling the flutist robot to interact more naturally with musical partners in the context of a Jazz band. For this purpose a Musical-Based Interaction System (MbIS) has been proposed to enable the robot to process both visual and aural cues coming throughout the interaction with musicians. In a previous publication, we have concentrated on the implementation of visual communication techniques. We created an interaction interface that enabled the robot to detect instrument gestures of partner musicians during a musical performance. Two computer vision approaches were implemented to create a two-skill-level interface for visual human-robot interaction in a musical context. In this paper we focus on the aural perception system of the robot. The method introduced here enables the robot to, a suitable environment provided, detect the tempo and harmony of a partner musician's play, with a specific focus on improvisation. We achieve this by examining the rhythmical and harmonic characteristics of the recorded sound. We apply the same approach to amplitude and frequency spectrum, thus, in the former case tracking amplitude transients. In the latter case, as we focus on communication with monophonic woodwind instruments, we follow the most prominent peak in the frequency spectrum. We specifically use a similar technique for the audio analysis as we did for our previous research on motion tracking. From the experimental results, we have shown that after implementing our algorithm the robot is able to correctly recognize a number of rhythms and harmonies. It is able to engage in a simple form of stimuli and reaction play with a human musician.

    UR - http://www.scopus.com/inward/record.url?scp=76249085832&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=76249085832&partnerID=8YFLogxK

    U2 - 10.1109/IROS.2009.5354550

    DO - 10.1109/IROS.2009.5354550

    M3 - Conference contribution

    AN - SCOPUS:76249085832

    SN - 9781424438044

    SP - 2303

    EP - 2308

    BT - 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2009

    ER -