TY - JOUR

T1 - Constructing infinite deep neural networks with flexible expressiveness while training

AU - Luo, Zhengbo

AU - Sun, Zitang

AU - Zhou, Weilian

AU - Wu, Zizhang

AU - Kamata, Sei ichiro

N1 - Publisher Copyright:
© 2021 Elsevier B.V.

PY - 2022/5/28

Y1 - 2022/5/28

N2 - The depth of the deep neural network (DNN) refers to the number of hidden layers between the input and output layers of an artificial neural network. It usually indicates a certain degree of complexity of the computational cost (parameters and floating point operations per second) and expressiveness once the network structure is settled. In this study, we experimentally investigate the effectiveness of using neural ordinary differential equations (NODEs) as a component to provide further depth in a continuous way to relatively shallower networks rather than stacking more layers (discrete depth), which achieved an improvement with fewer parameters. Experiments are conducted on classic DNNs, the residual networks. Moreover, we construct infinite deep neural networks with flexible complexity based on NODEs, enabling the system to adjust its complexity during training. On a better hidden-space provided by adaptive step DNNs, adaptive step ResNet with NODE (ResODE) is managed to achieve better performances in terms of convergence and accuracy than standard networks, and the improvements are widely observed in popular benchmarks.

AB - The depth of the deep neural network (DNN) refers to the number of hidden layers between the input and output layers of an artificial neural network. It usually indicates a certain degree of complexity of the computational cost (parameters and floating point operations per second) and expressiveness once the network structure is settled. In this study, we experimentally investigate the effectiveness of using neural ordinary differential equations (NODEs) as a component to provide further depth in a continuous way to relatively shallower networks rather than stacking more layers (discrete depth), which achieved an improvement with fewer parameters. Experiments are conducted on classic DNNs, the residual networks. Moreover, we construct infinite deep neural networks with flexible complexity based on NODEs, enabling the system to adjust its complexity during training. On a better hidden-space provided by adaptive step DNNs, adaptive step ResNet with NODE (ResODE) is managed to achieve better performances in terms of convergence and accuracy than standard networks, and the improvements are widely observed in popular benchmarks.

KW - Deep neural networks

KW - Image processing

KW - Neural ordinary differential equations

UR - http://www.scopus.com/inward/record.url?scp=85120804292&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85120804292&partnerID=8YFLogxK

U2 - 10.1016/j.neucom.2021.11.010

DO - 10.1016/j.neucom.2021.11.010

M3 - Article

AN - SCOPUS:85120804292

VL - 487

SP - 257

EP - 268

JO - Neurocomputing

JF - Neurocomputing

SN - 0925-2312

ER -