Face and gesture capturing and cloning for life-like agent

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Face and gesture cloning is essential to make a life-like agent more believable and to give it a personality and a character of target person. To realize cloning, an accurate face capture and motion capture are inevitable to get corpus data about face expressions, speaking scenes and gestures. In this paper, our recent approach to capture the personal feature of face and gesture is presented. For the face capturing, a face location and angles are estimated from video sequence with personal 3D face model and then a synthetic face model data is imposed into frames to realize automatic stand-in system or multimodal translation system.. A stand-in is a common technique for movies and TV programs in foreign languages. The current stand-in that only substitutes the voice channel results awkward matching to the mouth motion. Videophone with automatic voice translation are expected to be widely used in the near future, which may face the same problem without lip- synchronized speaking face image translation. In this paper, we introduce a method to track motion of the face from the video image and then replace the face part or only mouth part with synthesized one which is synchronized with synthetic voice or spoken voice. This is one of the key technologies not only for speaking image translation and communication system, but also for an interactive entertainment system. Also, an interactive movie system is introduced as an application of entertainment system. Capturing and copying a facial expression based on a physics base facial muscle constraint has been already presented[6]. So in this paper, this part is not described. For a gesture capturing, commercially available motion capture products give us fairly precise movements of human body segments but do not measure enough information to define skeletal posture in its entirety. This paper describes how to obtain the complete posture of skeletal structure with the help of marker locations relative to bones that are derived from MRI data sets.

    Original languageEnglish
    Title of host publicationProceedings - IEEE International Workshop on Robot and Human Interactive Communication
    Pages171-176
    Number of pages6
    Publication statusPublished - 2004
    EventRO-MAN 2004 - 13th IEEE International Workshop on Robot and Human Interactive Communication - Okayama
    Duration: 2004 Sep 202004 Sep 22

    Other

    OtherRO-MAN 2004 - 13th IEEE International Workshop on Robot and Human Interactive Communication
    CityOkayama
    Period04/9/2004/9/22

    Fingerprint

    Cloning
    Copying
    Magnetic resonance imaging
    Muscle
    Communication systems
    Bone
    Physics

    ASJC Scopus subject areas

    • Engineering(all)

    Cite this

    Morishima, S. (2004). Face and gesture capturing and cloning for life-like agent. In Proceedings - IEEE International Workshop on Robot and Human Interactive Communication (pp. 171-176)

    Face and gesture capturing and cloning for life-like agent. / Morishima, Shigeo.

    Proceedings - IEEE International Workshop on Robot and Human Interactive Communication. 2004. p. 171-176.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Morishima, S 2004, Face and gesture capturing and cloning for life-like agent. in Proceedings - IEEE International Workshop on Robot and Human Interactive Communication. pp. 171-176, RO-MAN 2004 - 13th IEEE International Workshop on Robot and Human Interactive Communication, Okayama, 04/9/20.
    Morishima S. Face and gesture capturing and cloning for life-like agent. In Proceedings - IEEE International Workshop on Robot and Human Interactive Communication. 2004. p. 171-176
    Morishima, Shigeo. / Face and gesture capturing and cloning for life-like agent. Proceedings - IEEE International Workshop on Robot and Human Interactive Communication. 2004. pp. 171-176
    @inproceedings{18d793f572fe4c2ba5ea02f16bd4aae0,
    title = "Face and gesture capturing and cloning for life-like agent",
    abstract = "Face and gesture cloning is essential to make a life-like agent more believable and to give it a personality and a character of target person. To realize cloning, an accurate face capture and motion capture are inevitable to get corpus data about face expressions, speaking scenes and gestures. In this paper, our recent approach to capture the personal feature of face and gesture is presented. For the face capturing, a face location and angles are estimated from video sequence with personal 3D face model and then a synthetic face model data is imposed into frames to realize automatic stand-in system or multimodal translation system.. A stand-in is a common technique for movies and TV programs in foreign languages. The current stand-in that only substitutes the voice channel results awkward matching to the mouth motion. Videophone with automatic voice translation are expected to be widely used in the near future, which may face the same problem without lip- synchronized speaking face image translation. In this paper, we introduce a method to track motion of the face from the video image and then replace the face part or only mouth part with synthesized one which is synchronized with synthetic voice or spoken voice. This is one of the key technologies not only for speaking image translation and communication system, but also for an interactive entertainment system. Also, an interactive movie system is introduced as an application of entertainment system. Capturing and copying a facial expression based on a physics base facial muscle constraint has been already presented[6]. So in this paper, this part is not described. For a gesture capturing, commercially available motion capture products give us fairly precise movements of human body segments but do not measure enough information to define skeletal posture in its entirety. This paper describes how to obtain the complete posture of skeletal structure with the help of marker locations relative to bones that are derived from MRI data sets.",
    author = "Shigeo Morishima",
    year = "2004",
    language = "English",
    pages = "171--176",
    booktitle = "Proceedings - IEEE International Workshop on Robot and Human Interactive Communication",

    }

    TY - GEN

    T1 - Face and gesture capturing and cloning for life-like agent

    AU - Morishima, Shigeo

    PY - 2004

    Y1 - 2004

    N2 - Face and gesture cloning is essential to make a life-like agent more believable and to give it a personality and a character of target person. To realize cloning, an accurate face capture and motion capture are inevitable to get corpus data about face expressions, speaking scenes and gestures. In this paper, our recent approach to capture the personal feature of face and gesture is presented. For the face capturing, a face location and angles are estimated from video sequence with personal 3D face model and then a synthetic face model data is imposed into frames to realize automatic stand-in system or multimodal translation system.. A stand-in is a common technique for movies and TV programs in foreign languages. The current stand-in that only substitutes the voice channel results awkward matching to the mouth motion. Videophone with automatic voice translation are expected to be widely used in the near future, which may face the same problem without lip- synchronized speaking face image translation. In this paper, we introduce a method to track motion of the face from the video image and then replace the face part or only mouth part with synthesized one which is synchronized with synthetic voice or spoken voice. This is one of the key technologies not only for speaking image translation and communication system, but also for an interactive entertainment system. Also, an interactive movie system is introduced as an application of entertainment system. Capturing and copying a facial expression based on a physics base facial muscle constraint has been already presented[6]. So in this paper, this part is not described. For a gesture capturing, commercially available motion capture products give us fairly precise movements of human body segments but do not measure enough information to define skeletal posture in its entirety. This paper describes how to obtain the complete posture of skeletal structure with the help of marker locations relative to bones that are derived from MRI data sets.

    AB - Face and gesture cloning is essential to make a life-like agent more believable and to give it a personality and a character of target person. To realize cloning, an accurate face capture and motion capture are inevitable to get corpus data about face expressions, speaking scenes and gestures. In this paper, our recent approach to capture the personal feature of face and gesture is presented. For the face capturing, a face location and angles are estimated from video sequence with personal 3D face model and then a synthetic face model data is imposed into frames to realize automatic stand-in system or multimodal translation system.. A stand-in is a common technique for movies and TV programs in foreign languages. The current stand-in that only substitutes the voice channel results awkward matching to the mouth motion. Videophone with automatic voice translation are expected to be widely used in the near future, which may face the same problem without lip- synchronized speaking face image translation. In this paper, we introduce a method to track motion of the face from the video image and then replace the face part or only mouth part with synthesized one which is synchronized with synthetic voice or spoken voice. This is one of the key technologies not only for speaking image translation and communication system, but also for an interactive entertainment system. Also, an interactive movie system is introduced as an application of entertainment system. Capturing and copying a facial expression based on a physics base facial muscle constraint has been already presented[6]. So in this paper, this part is not described. For a gesture capturing, commercially available motion capture products give us fairly precise movements of human body segments but do not measure enough information to define skeletal posture in its entirety. This paper describes how to obtain the complete posture of skeletal structure with the help of marker locations relative to bones that are derived from MRI data sets.

    UR - http://www.scopus.com/inward/record.url?scp=20444439095&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=20444439095&partnerID=8YFLogxK

    M3 - Conference contribution

    AN - SCOPUS:20444439095

    SP - 171

    EP - 176

    BT - Proceedings - IEEE International Workshop on Robot and Human Interactive Communication

    ER -