High-fidelity facial reflectance and geometry inference from an unconstrained image

Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, Hao Li

    研究成果: Article

    11 引用 (Scopus)

    抄録

    We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and highfrequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions.We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.

    元の言語English
    記事番号162
    ジャーナルACM Transactions on Graphics
    37
    発行部数4
    DOI
    出版物ステータスPublished - 2018 1 1

    Fingerprint

    Textures
    Geometry
    Lighting
    Skin
    Neural networks
    Network architecture

    ASJC Scopus subject areas

    • Computer Graphics and Computer-Aided Design

    これを引用

    Yamaguchi, S., Saito, S., Nagano, K., Zhao, Y., Chen, W., Olszewski, K., ... Li, H. (2018). High-fidelity facial reflectance and geometry inference from an unconstrained image. ACM Transactions on Graphics, 37(4), [162]. https://doi.org/10.1145/3197517.3201364

    High-fidelity facial reflectance and geometry inference from an unconstrained image. / Yamaguchi, Shugo; Saito, Shunsuke; Nagano, Koki; Zhao, Yajie; Chen, Weikai; Olszewski, Kyle; Morishima, Shigeo; Li, Hao.

    :: ACM Transactions on Graphics, 巻 37, 番号 4, 162, 01.01.2018.

    研究成果: Article

    Yamaguchi, S, Saito, S, Nagano, K, Zhao, Y, Chen, W, Olszewski, K, Morishima, S & Li, H 2018, 'High-fidelity facial reflectance and geometry inference from an unconstrained image', ACM Transactions on Graphics, 巻. 37, 番号 4, 162. https://doi.org/10.1145/3197517.3201364
    Yamaguchi, Shugo ; Saito, Shunsuke ; Nagano, Koki ; Zhao, Yajie ; Chen, Weikai ; Olszewski, Kyle ; Morishima, Shigeo ; Li, Hao. / High-fidelity facial reflectance and geometry inference from an unconstrained image. :: ACM Transactions on Graphics. 2018 ; 巻 37, 番号 4.
    @article{9df862b802674be89e3fb4f98df92332,
    title = "High-fidelity facial reflectance and geometry inference from an unconstrained image",
    abstract = "We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and highfrequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions.We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.",
    keywords = "Facial modeling, Image-based modeling, Texture synthesis and inpainting",
    author = "Shugo Yamaguchi and Shunsuke Saito and Koki Nagano and Yajie Zhao and Weikai Chen and Kyle Olszewski and Shigeo Morishima and Hao Li",
    year = "2018",
    month = "1",
    day = "1",
    doi = "10.1145/3197517.3201364",
    language = "English",
    volume = "37",
    journal = "ACM Transactions on Graphics",
    issn = "0730-0301",
    publisher = "Association for Computing Machinery (ACM)",
    number = "4",

    }

    TY - JOUR

    T1 - High-fidelity facial reflectance and geometry inference from an unconstrained image

    AU - Yamaguchi, Shugo

    AU - Saito, Shunsuke

    AU - Nagano, Koki

    AU - Zhao, Yajie

    AU - Chen, Weikai

    AU - Olszewski, Kyle

    AU - Morishima, Shigeo

    AU - Li, Hao

    PY - 2018/1/1

    Y1 - 2018/1/1

    N2 - We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and highfrequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions.We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.

    AB - We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and highfrequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions.We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.

    KW - Facial modeling

    KW - Image-based modeling

    KW - Texture synthesis and inpainting

    UR - http://www.scopus.com/inward/record.url?scp=85056641761&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85056641761&partnerID=8YFLogxK

    U2 - 10.1145/3197517.3201364

    DO - 10.1145/3197517.3201364

    M3 - Article

    AN - SCOPUS:85056641761

    VL - 37

    JO - ACM Transactions on Graphics

    JF - ACM Transactions on Graphics

    SN - 0730-0301

    IS - 4

    M1 - 162

    ER -