3D car shape reconstruction from a contour sketch using GAN and lazy learning

Naoki Nozawa, Hubert P.H. Shum, Qi Feng, Edmond S.L. Ho, Shigeo Morishima

Research output: Contribution to journalArticlepeer-review

Abstract

3D car models are heavily used in computer games, visual effects, and even automotive designs. As a result, producing such models with minimal labour costs is increasingly more important. To tackle the challenge, we propose a novel system to reconstruct a 3D car using a single sketch image. The system learns from a synthetic database of 3D car models and their corresponding 2D contour sketches and segmentation masks, allowing effective training with minimal data collection cost. The core of the system is a machine learning pipeline that combines the use of a generative adversarial network (GAN) and lazy learning. GAN, being a deep learning method, is capable of modelling complicated data distributions, enabling the effective modelling of a large variety of cars. Its major weakness is that as a global method, modelling the fine details in the local region is challenging. Lazy learning works well to preserve local features by generating a local subspace with relevant data samples. We demonstrate that the combined use of GAN and lazy learning produces is able to produce high-quality results, in which different types of cars with complicated local features can be generated effectively with a single sketch. Our method outperforms existing ones using other machine learning structures such as the variational autoencoder.

Original languageEnglish
JournalVisual Computer
DOIs
Publication statusAccepted/In press - 2021

Keywords

  • 3D reconstruction
  • Car
  • Contour sketch
  • Generative adversarial network
  • Lazy learning
  • Sketch-based interface

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of '3D car shape reconstruction from a contour sketch using GAN and lazy learning'. Together they form a unique fingerprint.

Cite this