Spatio-temporal predictive network for videos with physical properties

Yuka Aoyagi, Noboru Murata, Hidetomo Sakaino

研究成果: Conference contribution

抄録

In this paper, we propose a spatio-temporal predictive network with attention weighting of multiple physical Deep Learning (DL) models for videos with various physical properties. Previous approaches have been models with multiple branches for difference properties in videos, but the outputs of branches have been simply summed even with properties that change in time and space. In addition, it is difficult to train previous models for sufficient representations of physical properties in videos. Therefore, we propose the design of the spatio-temporal prediction network and the training method for videos with multiple physical properties, motivated by the Mixtures of Experts framework. Multiple spatio-temporal DL branches/experts for multiple physical properties and pixel-wise and expert-wise attention mechanism for adaptively integrating outputs of experts, i.e., Spatial-Temporal Gating Networks (STGNs) are proposed. Experts are trained with a vast amount of synthetic image sequences by physical equations and noise models. Instead, the whole network including STGNs is allowed to be trained only with a limited number of real datasets. Experiments on various videos, i.e., traffic, pedestrian, Dynamic Texture videos, and radar images, show the superiority of our proposed approach compared with previous approaches.

本文言語English
ホスト出版物のタイトルProceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
出版社IEEE Computer Society
ページ2268-2278
ページ数11
ISBN(電子版)9781665448994
DOI
出版ステータスPublished - 2021 6
イベント2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021 - Virtual, Online, United States
継続期間: 2021 6 192021 6 25

出版物シリーズ

名前IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
ISSN(印刷版)2160-7508
ISSN(電子版)2160-7516

Conference

Conference2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2021
国/地域United States
CityVirtual, Online
Period21/6/1921/6/25

ASJC Scopus subject areas

  • コンピュータ ビジョンおよびパターン認識
  • 電子工学および電気工学

フィンガープリント

「Spatio-temporal predictive network for videos with physical properties」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル