Deep neural networks with mixture of experts layers for complex event recognition from images

Mingyao Li, Seiichiro Kamata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the need for the real-world applications, event recognition from static images has become more and more popular in these years. Although there remain good achievements, recognizing events from images with a complex background like WIDER dataset is still very hard to get good results. In this paper, we show this gap is probably caused by the large discrepancy of data. Most of the existing methods choose to use various modifications on pre-trained CNN network model to solve the problem. Although we follow this thought, after a review of existing methods, we choose two other ways to solve this problem. Firstly, we reveal that a deep one-channel model with end-to-end structure is more suitable to this problem than other multi-channel or multi-task models, which leads we to propose a model under this rule by modifying on one single pre-trained ResNet channel. Secondly, we propose a Mixture of Experts (MoE) neural network layer to overcome the large discrepancy of data. To increase the performance and enhance the specialization of the MoE layer, we also involve a simple neural network transfer method, Elastic Weight Consolidation, to transfer knowledge from SocEID dataset. The result shows that we enhance the accuracy of the WIDER dataset from the state-of-the-art by 9.4% with lower computational time and memory consumption. And some experiments are also listed there to proof the validation of our method.

Original languageEnglish
Title of host publication2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages410-415
Number of pages6
ISBN (Electronic)9781538651612
DOIs
Publication statusPublished - 2019 Feb 12
EventJoint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018 - Kitakyushu, Japan
Duration: 2018 Jun 252018 Jun 28

Publication series

Name2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018

Conference

ConferenceJoint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018
CountryJapan
CityKitakyushu
Period18/6/2518/6/28

Fingerprint

Mixture of Experts
Neural Networks
Discrepancy
Choose
Neural networks
Task Model
Knowledge Transfer
Network layers
Channel Model
Consolidation
Specialization
Real-world Applications
Network Model
Data storage equipment
Deep neural networks
Experiment
Experiments

Keywords

  • CNN
  • Complex Event Recognition
  • Deep Learning
  • Elastic Weight Consolidation
  • Event Recognition
  • Mixture of Experts

ASJC Scopus subject areas

  • Signal Processing
  • Control and Optimization
  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Information Systems

Cite this

Li, M., & Kamata, S. (2019). Deep neural networks with mixture of experts layers for complex event recognition from images. In 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018 (pp. 410-415). [8641027] (2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICIEV.2018.8641027

Deep neural networks with mixture of experts layers for complex event recognition from images. / Li, Mingyao; Kamata, Seiichiro.

2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 410-415 8641027 (2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Li, M & Kamata, S 2019, Deep neural networks with mixture of experts layers for complex event recognition from images. in 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018., 8641027, 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018, Institute of Electrical and Electronics Engineers Inc., pp. 410-415, Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018, Kitakyushu, Japan, 18/6/25. https://doi.org/10.1109/ICIEV.2018.8641027
Li M, Kamata S. Deep neural networks with mixture of experts layers for complex event recognition from images. In 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 410-415. 8641027. (2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018). https://doi.org/10.1109/ICIEV.2018.8641027
Li, Mingyao ; Kamata, Seiichiro. / Deep neural networks with mixture of experts layers for complex event recognition from images. 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 410-415 (2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018).
@inproceedings{9b5ab6ba92dc4b56a2c76d13323288f5,
title = "Deep neural networks with mixture of experts layers for complex event recognition from images",
abstract = "With the need for the real-world applications, event recognition from static images has become more and more popular in these years. Although there remain good achievements, recognizing events from images with a complex background like WIDER dataset is still very hard to get good results. In this paper, we show this gap is probably caused by the large discrepancy of data. Most of the existing methods choose to use various modifications on pre-trained CNN network model to solve the problem. Although we follow this thought, after a review of existing methods, we choose two other ways to solve this problem. Firstly, we reveal that a deep one-channel model with end-to-end structure is more suitable to this problem than other multi-channel or multi-task models, which leads we to propose a model under this rule by modifying on one single pre-trained ResNet channel. Secondly, we propose a Mixture of Experts (MoE) neural network layer to overcome the large discrepancy of data. To increase the performance and enhance the specialization of the MoE layer, we also involve a simple neural network transfer method, Elastic Weight Consolidation, to transfer knowledge from SocEID dataset. The result shows that we enhance the accuracy of the WIDER dataset from the state-of-the-art by 9.4{\%} with lower computational time and memory consumption. And some experiments are also listed there to proof the validation of our method.",
keywords = "CNN, Complex Event Recognition, Deep Learning, Elastic Weight Consolidation, Event Recognition, Mixture of Experts",
author = "Mingyao Li and Seiichiro Kamata",
year = "2019",
month = "2",
day = "12",
doi = "10.1109/ICIEV.2018.8641027",
language = "English",
series = "2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "410--415",
booktitle = "2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018",

}

TY - GEN

T1 - Deep neural networks with mixture of experts layers for complex event recognition from images

AU - Li, Mingyao

AU - Kamata, Seiichiro

PY - 2019/2/12

Y1 - 2019/2/12

N2 - With the need for the real-world applications, event recognition from static images has become more and more popular in these years. Although there remain good achievements, recognizing events from images with a complex background like WIDER dataset is still very hard to get good results. In this paper, we show this gap is probably caused by the large discrepancy of data. Most of the existing methods choose to use various modifications on pre-trained CNN network model to solve the problem. Although we follow this thought, after a review of existing methods, we choose two other ways to solve this problem. Firstly, we reveal that a deep one-channel model with end-to-end structure is more suitable to this problem than other multi-channel or multi-task models, which leads we to propose a model under this rule by modifying on one single pre-trained ResNet channel. Secondly, we propose a Mixture of Experts (MoE) neural network layer to overcome the large discrepancy of data. To increase the performance and enhance the specialization of the MoE layer, we also involve a simple neural network transfer method, Elastic Weight Consolidation, to transfer knowledge from SocEID dataset. The result shows that we enhance the accuracy of the WIDER dataset from the state-of-the-art by 9.4% with lower computational time and memory consumption. And some experiments are also listed there to proof the validation of our method.

AB - With the need for the real-world applications, event recognition from static images has become more and more popular in these years. Although there remain good achievements, recognizing events from images with a complex background like WIDER dataset is still very hard to get good results. In this paper, we show this gap is probably caused by the large discrepancy of data. Most of the existing methods choose to use various modifications on pre-trained CNN network model to solve the problem. Although we follow this thought, after a review of existing methods, we choose two other ways to solve this problem. Firstly, we reveal that a deep one-channel model with end-to-end structure is more suitable to this problem than other multi-channel or multi-task models, which leads we to propose a model under this rule by modifying on one single pre-trained ResNet channel. Secondly, we propose a Mixture of Experts (MoE) neural network layer to overcome the large discrepancy of data. To increase the performance and enhance the specialization of the MoE layer, we also involve a simple neural network transfer method, Elastic Weight Consolidation, to transfer knowledge from SocEID dataset. The result shows that we enhance the accuracy of the WIDER dataset from the state-of-the-art by 9.4% with lower computational time and memory consumption. And some experiments are also listed there to proof the validation of our method.

KW - CNN

KW - Complex Event Recognition

KW - Deep Learning

KW - Elastic Weight Consolidation

KW - Event Recognition

KW - Mixture of Experts

UR - http://www.scopus.com/inward/record.url?scp=85063209687&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063209687&partnerID=8YFLogxK

U2 - 10.1109/ICIEV.2018.8641027

DO - 10.1109/ICIEV.2018.8641027

M3 - Conference contribution

T3 - 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018

SP - 410

EP - 415

BT - 2018 Joint 7th International Conference on Informatics, Electronics and Vision and 2nd International Conference on Imaging, Vision and Pattern Recognition, ICIEV-IVPR 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -