Neural network with unbounded activation functions is universal approximator

Sho Sonoda, Noboru Murata

Research output: Contribution to journalArticlepeer-review

55 Citations (Scopus)

Abstract

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.

Original languageEnglish
Pages (from-to)233-268
Number of pages36
JournalApplied and Computational Harmonic Analysis
Volume43
Issue number2
DOIs
Publication statusPublished - 2017 Sep

Keywords

  • Admissibility condition
  • Backprojection filter
  • Bounded extension to L
  • Integral representation
  • Lizorkin distribution
  • Neural network
  • Radon transform
  • Rectified linear unit (ReLU)
  • Ridgelet transform
  • Universal approximation

ASJC Scopus subject areas

  • Applied Mathematics

Fingerprint Dive into the research topics of 'Neural network with unbounded activation functions is universal approximator'. Together they form a unique fingerprint.

Cite this