Video translation system using face tracking and lip synchronization

Shigeo Morishima, Shin Ogata, S. Nakamura

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. Also, we propose a method to track motion of the face from the video image. In this system, movement and rotation of the head is detected by template matching using a 3D personal face wire-frame model. By this technique, an automatic video translation can be achieved.

Original languageEnglish
Title of host publicationProceedings - IEEE International Conference on Multimedia and Expo
PublisherIEEE Computer Society
Pages649-652
Number of pages4
ISBN (Print)0769511988
DOIs
Publication statusPublished - 2001
Externally publishedYes
Event2001 IEEE International Conference on Multimedia and Expo, ICME 2001 - Tokyo
Duration: 2001 Aug 222001 Aug 25

Other

Other2001 IEEE International Conference on Multimedia and Expo, ICME 2001
CityTokyo
Period01/8/2201/8/25

Fingerprint

Synchronization
Wire
Template matching

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications

Cite this

Morishima, S., Ogata, S., & Nakamura, S. (2001). Video translation system using face tracking and lip synchronization. In Proceedings - IEEE International Conference on Multimedia and Expo (pp. 649-652). [1237804] IEEE Computer Society. https://doi.org/10.1109/ICME.2001.1237804

Video translation system using face tracking and lip synchronization. / Morishima, Shigeo; Ogata, Shin; Nakamura, S.

Proceedings - IEEE International Conference on Multimedia and Expo. IEEE Computer Society, 2001. p. 649-652 1237804.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Morishima, S, Ogata, S & Nakamura, S 2001, Video translation system using face tracking and lip synchronization. in Proceedings - IEEE International Conference on Multimedia and Expo., 1237804, IEEE Computer Society, pp. 649-652, 2001 IEEE International Conference on Multimedia and Expo, ICME 2001, Tokyo, 01/8/22. https://doi.org/10.1109/ICME.2001.1237804
Morishima S, Ogata S, Nakamura S. Video translation system using face tracking and lip synchronization. In Proceedings - IEEE International Conference on Multimedia and Expo. IEEE Computer Society. 2001. p. 649-652. 1237804 https://doi.org/10.1109/ICME.2001.1237804
Morishima, Shigeo ; Ogata, Shin ; Nakamura, S. / Video translation system using face tracking and lip synchronization. Proceedings - IEEE International Conference on Multimedia and Expo. IEEE Computer Society, 2001. pp. 649-652
@inproceedings{ff0f75b0d97a45a3b0e0ed61ae1c8f18,
title = "Video translation system using face tracking and lip synchronization",
abstract = "We introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. Also, we propose a method to track motion of the face from the video image. In this system, movement and rotation of the head is detected by template matching using a 3D personal face wire-frame model. By this technique, an automatic video translation can be achieved.",
author = "Shigeo Morishima and Shin Ogata and S. Nakamura",
year = "2001",
doi = "10.1109/ICME.2001.1237804",
language = "English",
isbn = "0769511988",
pages = "649--652",
booktitle = "Proceedings - IEEE International Conference on Multimedia and Expo",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - Video translation system using face tracking and lip synchronization

AU - Morishima, Shigeo

AU - Ogata, Shin

AU - Nakamura, S.

PY - 2001

Y1 - 2001

N2 - We introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. Also, we propose a method to track motion of the face from the video image. In this system, movement and rotation of the head is detected by template matching using a 3D personal face wire-frame model. By this technique, an automatic video translation can be achieved.

AB - We introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. Also, we propose a method to track motion of the face from the video image. In this system, movement and rotation of the head is detected by template matching using a 3D personal face wire-frame model. By this technique, an automatic video translation can be achieved.

UR - http://www.scopus.com/inward/record.url?scp=84908286920&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84908286920&partnerID=8YFLogxK

U2 - 10.1109/ICME.2001.1237804

DO - 10.1109/ICME.2001.1237804

M3 - Conference contribution

AN - SCOPUS:84908286920

SN - 0769511988

SP - 649

EP - 652

BT - Proceedings - IEEE International Conference on Multimedia and Expo

PB - IEEE Computer Society

ER -