FSNet: An Identity-Aware Generative Model for Image-Based Face Swapping

Ryota Natsume, Tatsuya Yatagawa, Shigeo Morishima

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents FSNet, a deep generative model for image-based face swapping. Traditionally, face-swapping methods are based on three-dimensional morphable models (3DMMs), and facial textures are replaced between the estimated three-dimensional (3D) geometries in two images of different individuals. However, the estimation of 3D geometries along with different lighting conditions using 3DMMs is still a difficult task. We herein represent the face region with a latent variable that is assigned with the proposed deep neural network (DNN) instead of facial textures. The proposed DNN synthesizes a face-swapped image using the latent variable of the face region and another image of the non-face region. The proposed method is not required to fit to the 3DMM; additionally, it performs face swapping only by feeding two face images to the proposed network. Consequently, our DNN-based face swapping performs better than previous approaches for challenging inputs with different face orientations and lighting conditions. Through several experiments, we demonstrated that the proposed method performs face swapping in a more stable manner than the state-of-the-art method, and that its results are compatible with the method thereof.

Original languageEnglish
Title of host publicationComputer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers
EditorsGreg Mori, Hongdong Li, C.V. Jawahar, Konrad Schindler
PublisherSpringer-Verlag
Pages117-132
Number of pages16
ISBN (Print)9783030208752
DOIs
Publication statusPublished - 2019 Jan 1
Event14th Asian Conference on Computer Vision, ACCV 2018 - Perth, Australia
Duration: 2018 Dec 22018 Dec 6

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11366 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference14th Asian Conference on Computer Vision, ACCV 2018
CountryAustralia
CityPerth
Period18/12/218/12/6

Fingerprint

Generative Models
Face
Textures
Lighting
Geometry
Latent Variables
Neural Networks
Texture
Deep neural networks
Experiments
Three-dimensional

Keywords

  • Convolutional neural networks
  • Deep generative models
  • Face swapping

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Natsume, R., Yatagawa, T., & Morishima, S. (2019). FSNet: An Identity-Aware Generative Model for Image-Based Face Swapping. In G. Mori, H. Li, C. V. Jawahar, & K. Schindler (Eds.), Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers (pp. 117-132). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11366 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-030-20876-9_8

FSNet : An Identity-Aware Generative Model for Image-Based Face Swapping. / Natsume, Ryota; Yatagawa, Tatsuya; Morishima, Shigeo.

Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. ed. / Greg Mori; Hongdong Li; C.V. Jawahar; Konrad Schindler. Springer-Verlag, 2019. p. 117-132 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11366 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Natsume, R, Yatagawa, T & Morishima, S 2019, FSNet: An Identity-Aware Generative Model for Image-Based Face Swapping. in G Mori, H Li, CV Jawahar & K Schindler (eds), Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11366 LNCS, Springer-Verlag, pp. 117-132, 14th Asian Conference on Computer Vision, ACCV 2018, Perth, Australia, 18/12/2. https://doi.org/10.1007/978-3-030-20876-9_8
Natsume R, Yatagawa T, Morishima S. FSNet: An Identity-Aware Generative Model for Image-Based Face Swapping. In Mori G, Li H, Jawahar CV, Schindler K, editors, Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. Springer-Verlag. 2019. p. 117-132. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-20876-9_8
Natsume, Ryota ; Yatagawa, Tatsuya ; Morishima, Shigeo. / FSNet : An Identity-Aware Generative Model for Image-Based Face Swapping. Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. editor / Greg Mori ; Hongdong Li ; C.V. Jawahar ; Konrad Schindler. Springer-Verlag, 2019. pp. 117-132 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{1d011105bda645e3a16c75e73912234d,
title = "FSNet: An Identity-Aware Generative Model for Image-Based Face Swapping",
abstract = "This paper presents FSNet, a deep generative model for image-based face swapping. Traditionally, face-swapping methods are based on three-dimensional morphable models (3DMMs), and facial textures are replaced between the estimated three-dimensional (3D) geometries in two images of different individuals. However, the estimation of 3D geometries along with different lighting conditions using 3DMMs is still a difficult task. We herein represent the face region with a latent variable that is assigned with the proposed deep neural network (DNN) instead of facial textures. The proposed DNN synthesizes a face-swapped image using the latent variable of the face region and another image of the non-face region. The proposed method is not required to fit to the 3DMM; additionally, it performs face swapping only by feeding two face images to the proposed network. Consequently, our DNN-based face swapping performs better than previous approaches for challenging inputs with different face orientations and lighting conditions. Through several experiments, we demonstrated that the proposed method performs face swapping in a more stable manner than the state-of-the-art method, and that its results are compatible with the method thereof.",
keywords = "Convolutional neural networks, Deep generative models, Face swapping",
author = "Ryota Natsume and Tatsuya Yatagawa and Shigeo Morishima",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-20876-9_8",
language = "English",
isbn = "9783030208752",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer-Verlag",
pages = "117--132",
editor = "Greg Mori and Hongdong Li and C.V. Jawahar and Konrad Schindler",
booktitle = "Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers",

}

TY - GEN

T1 - FSNet

T2 - An Identity-Aware Generative Model for Image-Based Face Swapping

AU - Natsume, Ryota

AU - Yatagawa, Tatsuya

AU - Morishima, Shigeo

PY - 2019/1/1

Y1 - 2019/1/1

N2 - This paper presents FSNet, a deep generative model for image-based face swapping. Traditionally, face-swapping methods are based on three-dimensional morphable models (3DMMs), and facial textures are replaced between the estimated three-dimensional (3D) geometries in two images of different individuals. However, the estimation of 3D geometries along with different lighting conditions using 3DMMs is still a difficult task. We herein represent the face region with a latent variable that is assigned with the proposed deep neural network (DNN) instead of facial textures. The proposed DNN synthesizes a face-swapped image using the latent variable of the face region and another image of the non-face region. The proposed method is not required to fit to the 3DMM; additionally, it performs face swapping only by feeding two face images to the proposed network. Consequently, our DNN-based face swapping performs better than previous approaches for challenging inputs with different face orientations and lighting conditions. Through several experiments, we demonstrated that the proposed method performs face swapping in a more stable manner than the state-of-the-art method, and that its results are compatible with the method thereof.

AB - This paper presents FSNet, a deep generative model for image-based face swapping. Traditionally, face-swapping methods are based on three-dimensional morphable models (3DMMs), and facial textures are replaced between the estimated three-dimensional (3D) geometries in two images of different individuals. However, the estimation of 3D geometries along with different lighting conditions using 3DMMs is still a difficult task. We herein represent the face region with a latent variable that is assigned with the proposed deep neural network (DNN) instead of facial textures. The proposed DNN synthesizes a face-swapped image using the latent variable of the face region and another image of the non-face region. The proposed method is not required to fit to the 3DMM; additionally, it performs face swapping only by feeding two face images to the proposed network. Consequently, our DNN-based face swapping performs better than previous approaches for challenging inputs with different face orientations and lighting conditions. Through several experiments, we demonstrated that the proposed method performs face swapping in a more stable manner than the state-of-the-art method, and that its results are compatible with the method thereof.

KW - Convolutional neural networks

KW - Deep generative models

KW - Face swapping

UR - http://www.scopus.com/inward/record.url?scp=85066959271&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066959271&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-20876-9_8

DO - 10.1007/978-3-030-20876-9_8

M3 - Conference contribution

AN - SCOPUS:85066959271

SN - 9783030208752

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 117

EP - 132

BT - Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers

A2 - Mori, Greg

A2 - Li, Hongdong

A2 - Jawahar, C.V.

A2 - Schindler, Konrad

PB - Springer-Verlag

ER -