Abstract
We present an integral framework for training sketch simplification networks that convert challenging rough sketches into clean line drawings. Our approach augments a simplification network with a discriminator network, training both networks jointly so that the discriminator network discerns whether a line drawing is real training data or the output of the simplification network, which, in turn, tries to fool it. This approach has two major advantages: first, because the discriminator network learns the structure in line drawings, it encourages the output sketches of the simplification network to be more similar in appearance to the training sketches. Second, we can also train the networks with additional unsupervised data: by adding rough sketches and line drawings that are not corresponding to each other, we can improve the quality of the sketch simplification. Thanks to a difference in the architecture, our approach has advantages over similar adversarial training approaches in stability of training and the aforementioned ability to utilize unsupervised training data. We show how our framework can be used to train models that significantly outperform the state of the art in the sketch simplification task, despite using the same architecture for inference. We also present an approach to optimize for a single image, which improves accuracy at the cost of additional computation time. Finally, we show that, using the same framework, it is possible to train the network to perform the inverse problem, i.e., convert simple line sketches into pencil drawings, which is not possible using the standard mean squared error loss. We validate our framework with two user tests, in which our approach is preferred to the state of the art in sketch simplification 88.9% of the time.
| Original language | English |
|---|---|
| Article number | 3132703 |
| Journal | ACM Transactions on Graphics |
| Volume | 37 |
| Issue number | 1 |
| DOIs | |
| Publication status | Published - 2018 Jan 1 |
Fingerprint
Keywords
- Convolutional neural network
- Pencil drawing generation
- Sketch simplification
ASJC Scopus subject areas
- Computer Graphics and Computer-Aided Design
Cite this
Mastering sketching : Adversarial augmentation for structured prediction. / Simo Serra, Edgar; Iizuka, Satoshi; Ishikawa, Hiroshi.
In: ACM Transactions on Graphics, Vol. 37, No. 1, 3132703, 01.01.2018.Research output: Contribution to journal › Article
}
TY - JOUR
T1 - Mastering sketching
T2 - Adversarial augmentation for structured prediction
AU - Simo Serra, Edgar
AU - Iizuka, Satoshi
AU - Ishikawa, Hiroshi
PY - 2018/1/1
Y1 - 2018/1/1
N2 - We present an integral framework for training sketch simplification networks that convert challenging rough sketches into clean line drawings. Our approach augments a simplification network with a discriminator network, training both networks jointly so that the discriminator network discerns whether a line drawing is real training data or the output of the simplification network, which, in turn, tries to fool it. This approach has two major advantages: first, because the discriminator network learns the structure in line drawings, it encourages the output sketches of the simplification network to be more similar in appearance to the training sketches. Second, we can also train the networks with additional unsupervised data: by adding rough sketches and line drawings that are not corresponding to each other, we can improve the quality of the sketch simplification. Thanks to a difference in the architecture, our approach has advantages over similar adversarial training approaches in stability of training and the aforementioned ability to utilize unsupervised training data. We show how our framework can be used to train models that significantly outperform the state of the art in the sketch simplification task, despite using the same architecture for inference. We also present an approach to optimize for a single image, which improves accuracy at the cost of additional computation time. Finally, we show that, using the same framework, it is possible to train the network to perform the inverse problem, i.e., convert simple line sketches into pencil drawings, which is not possible using the standard mean squared error loss. We validate our framework with two user tests, in which our approach is preferred to the state of the art in sketch simplification 88.9% of the time.
AB - We present an integral framework for training sketch simplification networks that convert challenging rough sketches into clean line drawings. Our approach augments a simplification network with a discriminator network, training both networks jointly so that the discriminator network discerns whether a line drawing is real training data or the output of the simplification network, which, in turn, tries to fool it. This approach has two major advantages: first, because the discriminator network learns the structure in line drawings, it encourages the output sketches of the simplification network to be more similar in appearance to the training sketches. Second, we can also train the networks with additional unsupervised data: by adding rough sketches and line drawings that are not corresponding to each other, we can improve the quality of the sketch simplification. Thanks to a difference in the architecture, our approach has advantages over similar adversarial training approaches in stability of training and the aforementioned ability to utilize unsupervised training data. We show how our framework can be used to train models that significantly outperform the state of the art in the sketch simplification task, despite using the same architecture for inference. We also present an approach to optimize for a single image, which improves accuracy at the cost of additional computation time. Finally, we show that, using the same framework, it is possible to train the network to perform the inverse problem, i.e., convert simple line sketches into pencil drawings, which is not possible using the standard mean squared error loss. We validate our framework with two user tests, in which our approach is preferred to the state of the art in sketch simplification 88.9% of the time.
KW - Convolutional neural network
KW - Pencil drawing generation
KW - Sketch simplification
UR - http://www.scopus.com/inward/record.url?scp=85041348971&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85041348971&partnerID=8YFLogxK
U2 - 10.1145/3132703
DO - 10.1145/3132703
M3 - Article
AN - SCOPUS:85041348971
VL - 37
JO - ACM Transactions on Graphics
JF - ACM Transactions on Graphics
SN - 0730-0301
IS - 1
M1 - 3132703
ER -