Overfitting Measurement of Deep Neural Networks Using No Data

Satoru Watanabe, Hayato Yamana

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Overfitting reduces the generalizability of deep neural networks (DNNs). Overfitting is generally detected by comparing the accuracies and losses of training and validation data; however, the detection method requires vast amounts of training data and is not always effective for forthcoming data due to the heterogeneity between training and forthcoming data. The dropout technique has been employed to prevent DNNs from overfitting, where the neurons in DNNs are invalidated randomly during their training. It has been hypothesized that this technique prevents DNNs from overfitting by restraining the co-adaptions among neurons. This hypothesis implies that overfitting of a DNN is a result of the co-adaptions among neurons and can be detected by investigating the inner representation of DNNs. Thus, we propose a method to detect overfitting of DNNs using no training and test data. The proposed method measures the degree of co-adaptions among neurons using persistent homology (PH). The proposed PH-based overfitting measure (PHOM) method constructs clique complexes on DNNs using the trained parameters of DNNs, and the one-dimensional PH investigates the co-adaptions among neurons. Thus, PHOM requires no training and test data to measure overfitting. We applied PHOM to convolutional neural networks trained for the classification problems of the CIFAR-10, SVHN, and Tiny ImageNet data sets. The experimental results demonstrate that PHOM reveals the degree of overfitting of DNNs to the training data, which suggests that PHOM enables us to filter overfitted DNNs without requiring the training and test data.

Original languageEnglish
Title of host publication2021 IEEE 8th International Conference on Data Science and Advanced Analytics, DSAA 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665420990
DOIs
Publication statusPublished - 2021
Event8th IEEE International Conference on Data Science and Advanced Analytics, DSAA 2021 - Virtual, Online, Portugal
Duration: 2021 Oct 62021 Oct 9

Publication series

Name2021 IEEE 8th International Conference on Data Science and Advanced Analytics, DSAA 2021

Conference

Conference8th IEEE International Conference on Data Science and Advanced Analytics, DSAA 2021
Country/TerritoryPortugal
CityVirtual, Online
Period21/10/621/10/9

Keywords

  • Deep neural network
  • Overfitting
  • Persistent homology

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Signal Processing
  • Information Systems and Management
  • Statistics, Probability and Uncertainty

Fingerprint

Dive into the research topics of 'Overfitting Measurement of Deep Neural Networks Using No Data'. Together they form a unique fingerprint.

Cite this