Automatic Foreground Detection at 784 FPS for Ultra-High-Speed Human-Machine Interactions

Songlin Du, Peikun Cai, Tingting Hu, Takeshi Ikenaga

Research output: Contribution to journalArticlepeer-review

Abstract

Human-machine interactive systems show increasing demand for analysing fast moving objects in high-frame-rate videos. Robust foreground detection, which is able to reduce large amount of redundant background data from high-frame-rate video, becomes the essence to achieve ultra-high-speed human-machine interactions. This paper proposes a local spatial propagation based background model generation, a local linear illumination correction based background model update, and a regional central coordinates and edge keypoints constrained foreground region reselection. The three proposals make up a robust and hardware-friendly foreground detection method. Experimental results prove that the proposed hardware-friendly algorithm achieves high accuracy and robustness on various kinds of challenging cases. Meanwhile, the hardware implementation utilizes little hardware resources and achieves realtime processing of high-frame-rate (784 frame/second) video with the delay less than 1 ms/frame in image processing core. In addition, a practical system is implemented by combing a PC, a high-speed camera and a field programmable gate array (FPGA) for realworld applications. This work will significatively promote the development and application of high-speed human machine interaction. A demo of the proposed vision system working at 784 FPS is available at https://wcms.waseda.jp/em/5f84f75136a6. Note to Practitioners-This paper was motivated by the problem of high-frame-rate video contains large amount of redundant background pixels which makes ultra-high-speed human-machine interactions inaccessible. Existing approaches are mainly focused on designing complex background models, but processing speed, which is the most important issue for ultra-high-speed human-machine interactions, has received relatively little attention. This paper suggests a robust and hardware-friendly foreground detection algorithm which has been implemented as a hardware system by using an FPGA, a high-frame-rate camera, and a PC. We show that the hardware implementation utilizes less hardware resources and achieves real-time processing speed of 784 FPS with the delay less than 1 ms/frame in the image processing core. This work is a pioneering attempt of ultra-high-speed foreground detection, which will significatively speed up the wide applications of ultra-high-speed human machine interactions.

Original languageEnglish
Pages (from-to)3587-3600
Number of pages14
JournalIEEE Transactions on Automation Science and Engineering
Volume19
Issue number4
DOIs
Publication statusPublished - 2022 Oct 1

Keywords

  • Human-machine interaction
  • foreground detection
  • hardware architecture
  • ultra-high speed

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Automatic Foreground Detection at 784 FPS for Ultra-High-Speed Human-Machine Interactions'. Together they form a unique fingerprint.

Cite this