Rethinking ResNets: improved stacking strategies with high-order schemes for image classification

Zhengbo Luo*, Zitang Sun, Weilian Zhou, Zizhang Wu, Sei ichiro Kamata

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Abstract: Various deep neural network architectures (DNNs) maintain massive vital records in computer vision. While drawing attention worldwide, the design of the overall structure lacks general guidance. Based on the relationship between DNN design and numerical differential equations, we performed a fair comparison of the residual design with higher order perspectives. We show that the widely used DNN design strategy, constantly stacking a small design (usually, 2–3 layers), could be easily improved, supported by solid theoretical knowledge and with no extra parameters needed. We reorganise the residual design in higher order ways, which is inspired by the observation that many effective networks can be interpreted as different numerical discretisations of differential equations. The design of ResNet follows a relatively simple scheme, which is Euler forward; however, the situation becomes complicated rapidly while stacking. We suppose that stacked ResNet is somehow equalled to a higher order scheme; then, the current method of forwarding propagation might be relatively weak compared with a typical high-order method such as Runge–Kutta. We propose HO-ResNet to verify the hypothesis on widely used CV benchmarks with sufficient experiments. Stable and noticeable increases in performance are observed, and convergence and robustness are also improved. Our stacking strategy improved ResNet-30 by 2.15% and ResNet-58 by 2.35% on CIFAR-10, with the same settings and parameters. The proposed strategy is fundamental and theoretical and can, therefore, be applied to any network as a general guideline. Graphical abstract: [Figure not available: see fulltext.].

Original languageEnglish
Pages (from-to)3395-3407
Number of pages13
JournalComplex and Intelligent Systems
Volume8
Issue number4
DOIs
Publication statusPublished - 2022 Aug

Keywords

  • Deep neural networks
  • Image classification
  • Image processing
  • Neural ordinary differential equations

ASJC Scopus subject areas

  • Artificial Intelligence
  • Information Systems
  • Engineering (miscellaneous)
  • Computational Mathematics

Fingerprint

Dive into the research topics of 'Rethinking ResNets: improved stacking strategies with high-order schemes for image classification'. Together they form a unique fingerprint.

Cite this