Joint gap detection and inpainting of line drawings

Kazuma Sasaki, Satoshi Iizuka, Edgar Simo Serra, Hiroshi Ishikawa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Citations (Scopus)

Abstract

We propose a novel data-driven approach for automatically detecting and completing gaps in line drawings with a Convolutional Neural Network. In the case of existing inpainting approaches for natural images, masks indicating the missing regions are generally required as input. Here, we show that line drawings have enough structures that can be learned by the CNN to allow automatic detection and completion of the gaps without any such input. Thus, our method can find the gaps in line drawings and complete them without user interaction. Furthermore, the completion realistically conserves thickness and curvature of the line segments. All the necessary heuristics for such realistic line completion are learned naturally from a dataset of line drawings, where various patterns of line completion are generated on the fly as training pairs to improve the model generalization. We evaluate our method qualitatively on a diverse set of challenging line drawings and also provide quantitative results with a user study, where it significantly outperforms the state of the art.

Original languageEnglish
Title of host publicationProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages5768-5776
Number of pages9
Volume2017-January
ISBN (Electronic)9781538604571
DOIs
Publication statusPublished - 2017 Nov 6
Event30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 - Honolulu, United States
Duration: 2017 Jul 212017 Jul 26

Other

Other30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
CountryUnited States
CityHonolulu
Period17/7/2117/7/26

Fingerprint

Masks
Neural networks

ASJC Scopus subject areas

  • Signal Processing
  • Computer Vision and Pattern Recognition

Cite this

Sasaki, K., Iizuka, S., Simo Serra, E., & Ishikawa, H. (2017). Joint gap detection and inpainting of line drawings. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (Vol. 2017-January, pp. 5768-5776). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CVPR.2017.611

Joint gap detection and inpainting of line drawings. / Sasaki, Kazuma; Iizuka, Satoshi; Simo Serra, Edgar; Ishikawa, Hiroshi.

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Vol. 2017-January Institute of Electrical and Electronics Engineers Inc., 2017. p. 5768-5776.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sasaki, K, Iizuka, S, Simo Serra, E & Ishikawa, H 2017, Joint gap detection and inpainting of line drawings. in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 5768-5776, 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, United States, 17/7/21. https://doi.org/10.1109/CVPR.2017.611
Sasaki K, Iizuka S, Simo Serra E, Ishikawa H. Joint gap detection and inpainting of line drawings. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Vol. 2017-January. Institute of Electrical and Electronics Engineers Inc. 2017. p. 5768-5776 https://doi.org/10.1109/CVPR.2017.611
Sasaki, Kazuma ; Iizuka, Satoshi ; Simo Serra, Edgar ; Ishikawa, Hiroshi. / Joint gap detection and inpainting of line drawings. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Vol. 2017-January Institute of Electrical and Electronics Engineers Inc., 2017. pp. 5768-5776
@inproceedings{3d48416cc8654f46a742a16d2de2a6ac,
title = "Joint gap detection and inpainting of line drawings",
abstract = "We propose a novel data-driven approach for automatically detecting and completing gaps in line drawings with a Convolutional Neural Network. In the case of existing inpainting approaches for natural images, masks indicating the missing regions are generally required as input. Here, we show that line drawings have enough structures that can be learned by the CNN to allow automatic detection and completion of the gaps without any such input. Thus, our method can find the gaps in line drawings and complete them without user interaction. Furthermore, the completion realistically conserves thickness and curvature of the line segments. All the necessary heuristics for such realistic line completion are learned naturally from a dataset of line drawings, where various patterns of line completion are generated on the fly as training pairs to improve the model generalization. We evaluate our method qualitatively on a diverse set of challenging line drawings and also provide quantitative results with a user study, where it significantly outperforms the state of the art.",
author = "Kazuma Sasaki and Satoshi Iizuka and {Simo Serra}, Edgar and Hiroshi Ishikawa",
year = "2017",
month = "11",
day = "6",
doi = "10.1109/CVPR.2017.611",
language = "English",
volume = "2017-January",
pages = "5768--5776",
booktitle = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Joint gap detection and inpainting of line drawings

AU - Sasaki, Kazuma

AU - Iizuka, Satoshi

AU - Simo Serra, Edgar

AU - Ishikawa, Hiroshi

PY - 2017/11/6

Y1 - 2017/11/6

N2 - We propose a novel data-driven approach for automatically detecting and completing gaps in line drawings with a Convolutional Neural Network. In the case of existing inpainting approaches for natural images, masks indicating the missing regions are generally required as input. Here, we show that line drawings have enough structures that can be learned by the CNN to allow automatic detection and completion of the gaps without any such input. Thus, our method can find the gaps in line drawings and complete them without user interaction. Furthermore, the completion realistically conserves thickness and curvature of the line segments. All the necessary heuristics for such realistic line completion are learned naturally from a dataset of line drawings, where various patterns of line completion are generated on the fly as training pairs to improve the model generalization. We evaluate our method qualitatively on a diverse set of challenging line drawings and also provide quantitative results with a user study, where it significantly outperforms the state of the art.

AB - We propose a novel data-driven approach for automatically detecting and completing gaps in line drawings with a Convolutional Neural Network. In the case of existing inpainting approaches for natural images, masks indicating the missing regions are generally required as input. Here, we show that line drawings have enough structures that can be learned by the CNN to allow automatic detection and completion of the gaps without any such input. Thus, our method can find the gaps in line drawings and complete them without user interaction. Furthermore, the completion realistically conserves thickness and curvature of the line segments. All the necessary heuristics for such realistic line completion are learned naturally from a dataset of line drawings, where various patterns of line completion are generated on the fly as training pairs to improve the model generalization. We evaluate our method qualitatively on a diverse set of challenging line drawings and also provide quantitative results with a user study, where it significantly outperforms the state of the art.

UR - http://www.scopus.com/inward/record.url?scp=85044503619&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044503619&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2017.611

DO - 10.1109/CVPR.2017.611

M3 - Conference contribution

AN - SCOPUS:85044503619

VL - 2017-January

SP - 5768

EP - 5776

BT - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -