Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification

Satoshi Iizuka, Edgar Simo Serra, Hiroshi Ishikawa

Research output: Contribution to journalArticle

195 Citations (Scopus)

Abstract

We present a novel technique to automatically colorize grayscale images that combines both global priors and local image features. Based on Convolutional Neural Networks, our deep network features a fusion layer that allows us to elegantly merge local information dependent on small image patches with global priors computed using the entire image. The entire framework, including the global and local priors as well as the colorization model, is trained in an end-to-end fashion. Furthermore, our architecture can process images of any resolution, unlike most existing approaches based on CNN. We leverage an existing large-scale scene classification database to train our model, exploiting the class labels of the dataset to more efficiently and discriminatively learn the global priors. We validate our approach with a user study and compare against the state of the art, where we show significant improvements. Furthermore, we demonstrate our method extensively on many different types of images, including black-and-white photography from over a hundred years ago, and show realistic colorizations.

Original languageEnglish
Article numbera110
JournalACM Transactions on Graphics
Volume35
Issue number4
DOIs
Publication statusPublished - 2016 Jul 11

Fingerprint

Color
Photography
Labels
Fusion reactions
Deep neural networks

Keywords

  • Colorization
  • Convolutional neural network

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design

Cite this

Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. / Iizuka, Satoshi; Simo Serra, Edgar; Ishikawa, Hiroshi.

In: ACM Transactions on Graphics, Vol. 35, No. 4, a110, 11.07.2016.

Research output: Contribution to journalArticle

@article{c5e02fc28b0e4f6ead55d403233b8077,
title = "Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification",
abstract = "We present a novel technique to automatically colorize grayscale images that combines both global priors and local image features. Based on Convolutional Neural Networks, our deep network features a fusion layer that allows us to elegantly merge local information dependent on small image patches with global priors computed using the entire image. The entire framework, including the global and local priors as well as the colorization model, is trained in an end-to-end fashion. Furthermore, our architecture can process images of any resolution, unlike most existing approaches based on CNN. We leverage an existing large-scale scene classification database to train our model, exploiting the class labels of the dataset to more efficiently and discriminatively learn the global priors. We validate our approach with a user study and compare against the state of the art, where we show significant improvements. Furthermore, we demonstrate our method extensively on many different types of images, including black-and-white photography from over a hundred years ago, and show realistic colorizations.",
keywords = "Colorization, Convolutional neural network",
author = "Satoshi Iizuka and {Simo Serra}, Edgar and Hiroshi Ishikawa",
year = "2016",
month = "7",
day = "11",
doi = "10.1145/2897824.2925974",
language = "English",
volume = "35",
journal = "ACM Transactions on Graphics",
issn = "0730-0301",
publisher = "Association for Computing Machinery (ACM)",
number = "4",

}

TY - JOUR

T1 - Let there be color!

T2 - Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification

AU - Iizuka, Satoshi

AU - Simo Serra, Edgar

AU - Ishikawa, Hiroshi

PY - 2016/7/11

Y1 - 2016/7/11

N2 - We present a novel technique to automatically colorize grayscale images that combines both global priors and local image features. Based on Convolutional Neural Networks, our deep network features a fusion layer that allows us to elegantly merge local information dependent on small image patches with global priors computed using the entire image. The entire framework, including the global and local priors as well as the colorization model, is trained in an end-to-end fashion. Furthermore, our architecture can process images of any resolution, unlike most existing approaches based on CNN. We leverage an existing large-scale scene classification database to train our model, exploiting the class labels of the dataset to more efficiently and discriminatively learn the global priors. We validate our approach with a user study and compare against the state of the art, where we show significant improvements. Furthermore, we demonstrate our method extensively on many different types of images, including black-and-white photography from over a hundred years ago, and show realistic colorizations.

AB - We present a novel technique to automatically colorize grayscale images that combines both global priors and local image features. Based on Convolutional Neural Networks, our deep network features a fusion layer that allows us to elegantly merge local information dependent on small image patches with global priors computed using the entire image. The entire framework, including the global and local priors as well as the colorization model, is trained in an end-to-end fashion. Furthermore, our architecture can process images of any resolution, unlike most existing approaches based on CNN. We leverage an existing large-scale scene classification database to train our model, exploiting the class labels of the dataset to more efficiently and discriminatively learn the global priors. We validate our approach with a user study and compare against the state of the art, where we show significant improvements. Furthermore, we demonstrate our method extensively on many different types of images, including black-and-white photography from over a hundred years ago, and show realistic colorizations.

KW - Colorization

KW - Convolutional neural network

UR - http://www.scopus.com/inward/record.url?scp=84980049328&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84980049328&partnerID=8YFLogxK

U2 - 10.1145/2897824.2925974

DO - 10.1145/2897824.2925974

M3 - Article

AN - SCOPUS:84980049328

VL - 35

JO - ACM Transactions on Graphics

JF - ACM Transactions on Graphics

SN - 0730-0301

IS - 4

M1 - a110

ER -