A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars

Haruki Sato, Tatsunori Hirai, Tomoyasu Nakano, Masataka Goto, Shigeo Morishima

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    3 Citations (Scopus)

    Abstract

    This paper presents a system that can automatically add a sound-track to a video clip by replacing and concatenating an existing song's musical bars considering a user's preference. Since a soundtrack makes a video clip attractive, adding a soundtrack to a clip is one of the most important processes in video editing. To make a video clip more attractive, an editor of the clip tends to add a soundtrack considering its timing and climax. For example, editors often add chorus sections to the climax of the clip by re-placing and concatenating musical bars in an existing song. How-ever, in the process, editors should take naturalness of rearranged soundtrack into account. Therefore, editors have to decide how to replace musical bars in a song considering its timing, climax, and naturalness of rearranged soundtrack simultaneously. In this case, editors are required to optimize the soundtrack by listening to the rearranged result as well as checking the naturalness and synchro-nization between the result and the video clip. However, this repe-titious work is time-consuming. [Feng et al. 2010] proposed an automatic soundtrack addition method. However, since this meth-od automatically adds soundtrack with data-driven approach, this method cannot consider timing and climax which a user prefers. Our system takes all the patterns of rearranged musical bars into account and finds the most natural soundtrack considering a user's preference of intention for an audio-visual alignment and a climax of the resulting soundtrack. Specifically, musical sections between user's specified points and the beginning and the ending of the song are automatically interpolated by replacing and concatenat-ing musical bars based on dynamic programming. To consider user's intention for a climax of the soundtrack, the system allows the user to specify the intended climax by an editing interface. The system immediately reflects the intention and the soundtrack will be interactively re-rearranged. These semi-automated pro-cesses of rearranging soundtrack for a video clip help the users adding songs without considering naturalness of the rearranged song by their own.

    Original languageEnglish
    Title of host publicationACM SIGGRAPH 2015 Posters, SIGGRAPH 2015
    PublisherAssociation for Computing Machinery, Inc
    ISBN (Print)9781450336321
    DOIs
    Publication statusPublished - 2015 Jul 31
    EventInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015 - Los Angeles, United States
    Duration: 2015 Aug 92015 Aug 13

    Other

    OtherInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015
    CountryUnited States
    CityLos Angeles
    Period15/8/915/8/13

    ASJC Scopus subject areas

    • Computer Graphics and Computer-Aided Design
    • Computer Vision and Pattern Recognition
    • Human-Computer Interaction

    Fingerprint Dive into the research topics of 'A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars'. Together they form a unique fingerprint.

  • Cite this

    Sato, H., Hirai, T., Nakano, T., Goto, M., & Morishima, S. (2015). A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars. In ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015 [a42] Association for Computing Machinery, Inc. https://doi.org/10.1145/2787626.2792608