A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars

Haruki Sato, Tatsunori Hirai, Tomoyasu Nakano, Masataka Goto, Shigeo Morishima

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    3 Citations (Scopus)

    Abstract

    This paper presents a system that can automatically add a sound-track to a video clip by replacing and concatenating an existing song's musical bars considering a user's preference. Since a soundtrack makes a video clip attractive, adding a soundtrack to a clip is one of the most important processes in video editing. To make a video clip more attractive, an editor of the clip tends to add a soundtrack considering its timing and climax. For example, editors often add chorus sections to the climax of the clip by re-placing and concatenating musical bars in an existing song. How-ever, in the process, editors should take naturalness of rearranged soundtrack into account. Therefore, editors have to decide how to replace musical bars in a song considering its timing, climax, and naturalness of rearranged soundtrack simultaneously. In this case, editors are required to optimize the soundtrack by listening to the rearranged result as well as checking the naturalness and synchro-nization between the result and the video clip. However, this repe-titious work is time-consuming. [Feng et al. 2010] proposed an automatic soundtrack addition method. However, since this meth-od automatically adds soundtrack with data-driven approach, this method cannot consider timing and climax which a user prefers. Our system takes all the patterns of rearranged musical bars into account and finds the most natural soundtrack considering a user's preference of intention for an audio-visual alignment and a climax of the resulting soundtrack. Specifically, musical sections between user's specified points and the beginning and the ending of the song are automatically interpolated by replacing and concatenat-ing musical bars based on dynamic programming. To consider user's intention for a climax of the soundtrack, the system allows the user to specify the intended climax by an editing interface. The system immediately reflects the intention and the soundtrack will be interactively re-rearranged. These semi-automated pro-cesses of rearranging soundtrack for a video clip help the users adding songs without considering naturalness of the rearranged song by their own.

    Original languageEnglish
    Title of host publicationACM SIGGRAPH 2015 Posters, SIGGRAPH 2015
    PublisherAssociation for Computing Machinery, Inc
    ISBN (Print)9781450336321
    DOIs
    Publication statusPublished - 2015 Jul 31
    EventInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015 - Los Angeles, United States
    Duration: 2015 Aug 92015 Aug 13

    Other

    OtherInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015
    CountryUnited States
    CityLos Angeles
    Period15/8/915/8/13

    Fingerprint

    Dynamic programming
    Acoustic waves

    ASJC Scopus subject areas

    • Computer Graphics and Computer-Aided Design
    • Computer Vision and Pattern Recognition
    • Human-Computer Interaction

    Cite this

    Sato, H., Hirai, T., Nakano, T., Goto, M., & Morishima, S. (2015). A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars. In ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015 [a42] Association for Computing Machinery, Inc. https://doi.org/10.1145/2787626.2792608

    A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars. / Sato, Haruki; Hirai, Tatsunori; Nakano, Tomoyasu; Goto, Masataka; Morishima, Shigeo.

    ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015. Association for Computing Machinery, Inc, 2015. a42.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Sato, H, Hirai, T, Nakano, T, Goto, M & Morishima, S 2015, A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars. in ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015., a42, Association for Computing Machinery, Inc, International Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015, Los Angeles, United States, 15/8/9. https://doi.org/10.1145/2787626.2792608
    Sato H, Hirai T, Nakano T, Goto M, Morishima S. A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars. In ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015. Association for Computing Machinery, Inc. 2015. a42 https://doi.org/10.1145/2787626.2792608
    Sato, Haruki ; Hirai, Tatsunori ; Nakano, Tomoyasu ; Goto, Masataka ; Morishima, Shigeo. / A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars. ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015. Association for Computing Machinery, Inc, 2015.
    @inproceedings{e3df8581e8ba441f91af8d9ec436d593,
    title = "A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars",
    abstract = "This paper presents a system that can automatically add a sound-track to a video clip by replacing and concatenating an existing song's musical bars considering a user's preference. Since a soundtrack makes a video clip attractive, adding a soundtrack to a clip is one of the most important processes in video editing. To make a video clip more attractive, an editor of the clip tends to add a soundtrack considering its timing and climax. For example, editors often add chorus sections to the climax of the clip by re-placing and concatenating musical bars in an existing song. How-ever, in the process, editors should take naturalness of rearranged soundtrack into account. Therefore, editors have to decide how to replace musical bars in a song considering its timing, climax, and naturalness of rearranged soundtrack simultaneously. In this case, editors are required to optimize the soundtrack by listening to the rearranged result as well as checking the naturalness and synchro-nization between the result and the video clip. However, this repe-titious work is time-consuming. [Feng et al. 2010] proposed an automatic soundtrack addition method. However, since this meth-od automatically adds soundtrack with data-driven approach, this method cannot consider timing and climax which a user prefers. Our system takes all the patterns of rearranged musical bars into account and finds the most natural soundtrack considering a user's preference of intention for an audio-visual alignment and a climax of the resulting soundtrack. Specifically, musical sections between user's specified points and the beginning and the ending of the song are automatically interpolated by replacing and concatenat-ing musical bars based on dynamic programming. To consider user's intention for a climax of the soundtrack, the system allows the user to specify the intended climax by an editing interface. The system immediately reflects the intention and the soundtrack will be interactively re-rearranged. These semi-automated pro-cesses of rearranging soundtrack for a video clip help the users adding songs without considering naturalness of the rearranged song by their own.",
    author = "Haruki Sato and Tatsunori Hirai and Tomoyasu Nakano and Masataka Goto and Shigeo Morishima",
    year = "2015",
    month = "7",
    day = "31",
    doi = "10.1145/2787626.2792608",
    language = "English",
    isbn = "9781450336321",
    booktitle = "ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015",
    publisher = "Association for Computing Machinery, Inc",

    }

    TY - GEN

    T1 - A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars

    AU - Sato, Haruki

    AU - Hirai, Tatsunori

    AU - Nakano, Tomoyasu

    AU - Goto, Masataka

    AU - Morishima, Shigeo

    PY - 2015/7/31

    Y1 - 2015/7/31

    N2 - This paper presents a system that can automatically add a sound-track to a video clip by replacing and concatenating an existing song's musical bars considering a user's preference. Since a soundtrack makes a video clip attractive, adding a soundtrack to a clip is one of the most important processes in video editing. To make a video clip more attractive, an editor of the clip tends to add a soundtrack considering its timing and climax. For example, editors often add chorus sections to the climax of the clip by re-placing and concatenating musical bars in an existing song. How-ever, in the process, editors should take naturalness of rearranged soundtrack into account. Therefore, editors have to decide how to replace musical bars in a song considering its timing, climax, and naturalness of rearranged soundtrack simultaneously. In this case, editors are required to optimize the soundtrack by listening to the rearranged result as well as checking the naturalness and synchro-nization between the result and the video clip. However, this repe-titious work is time-consuming. [Feng et al. 2010] proposed an automatic soundtrack addition method. However, since this meth-od automatically adds soundtrack with data-driven approach, this method cannot consider timing and climax which a user prefers. Our system takes all the patterns of rearranged musical bars into account and finds the most natural soundtrack considering a user's preference of intention for an audio-visual alignment and a climax of the resulting soundtrack. Specifically, musical sections between user's specified points and the beginning and the ending of the song are automatically interpolated by replacing and concatenat-ing musical bars based on dynamic programming. To consider user's intention for a climax of the soundtrack, the system allows the user to specify the intended climax by an editing interface. The system immediately reflects the intention and the soundtrack will be interactively re-rearranged. These semi-automated pro-cesses of rearranging soundtrack for a video clip help the users adding songs without considering naturalness of the rearranged song by their own.

    AB - This paper presents a system that can automatically add a sound-track to a video clip by replacing and concatenating an existing song's musical bars considering a user's preference. Since a soundtrack makes a video clip attractive, adding a soundtrack to a clip is one of the most important processes in video editing. To make a video clip more attractive, an editor of the clip tends to add a soundtrack considering its timing and climax. For example, editors often add chorus sections to the climax of the clip by re-placing and concatenating musical bars in an existing song. How-ever, in the process, editors should take naturalness of rearranged soundtrack into account. Therefore, editors have to decide how to replace musical bars in a song considering its timing, climax, and naturalness of rearranged soundtrack simultaneously. In this case, editors are required to optimize the soundtrack by listening to the rearranged result as well as checking the naturalness and synchro-nization between the result and the video clip. However, this repe-titious work is time-consuming. [Feng et al. 2010] proposed an automatic soundtrack addition method. However, since this meth-od automatically adds soundtrack with data-driven approach, this method cannot consider timing and climax which a user prefers. Our system takes all the patterns of rearranged musical bars into account and finds the most natural soundtrack considering a user's preference of intention for an audio-visual alignment and a climax of the resulting soundtrack. Specifically, musical sections between user's specified points and the beginning and the ending of the song are automatically interpolated by replacing and concatenat-ing musical bars based on dynamic programming. To consider user's intention for a climax of the soundtrack, the system allows the user to specify the intended climax by an editing interface. The system immediately reflects the intention and the soundtrack will be interactively re-rearranged. These semi-automated pro-cesses of rearranging soundtrack for a video clip help the users adding songs without considering naturalness of the rearranged song by their own.

    UR - http://www.scopus.com/inward/record.url?scp=84959346776&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84959346776&partnerID=8YFLogxK

    U2 - 10.1145/2787626.2792608

    DO - 10.1145/2787626.2792608

    M3 - Conference contribution

    AN - SCOPUS:84959346776

    SN - 9781450336321

    BT - ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015

    PB - Association for Computing Machinery, Inc

    ER -