Widget detection on screenshots using computer vision and machine learning algorithms

Kacper Radzikowski, Karol Chȩciński, Mateusz Forc, Łukasz Lepak, Michał Jabłoński, Wiktor Kuśmirek, Bartłomiej Twardowski, Paweł Wawrzyński, Robert M. Nowak

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper we consider the problem of detecting and recognizing widgets in screenshots of computer programs' graphical user interface (GUI). This problem is fundamental in business process automation. The solution we propose here is based on detecting GUI elements with Canny edge operator, and recognizing already detected GUI elements with classifiers: neural networks, random forests, XGBoost, and others.

Original languageEnglish
Title of host publicationPhotonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019
EditorsRyszard S. Romaniuk, Maciej Linczuk
PublisherSPIE
ISBN (Electronic)9781510630659
DOIs
Publication statusPublished - 2019
Externally publishedYes
EventPhotonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019 - Wilga, Poland
Duration: 2019 May 262019 Jun 2

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
Volume11176
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X

Conference

ConferencePhotonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019
CountryPoland
CityWilga
Period19/5/2619/6/2

Keywords

  • edge detection
  • GUI decomposition
  • image recognition
  • machine learning
  • neural networks
  • widget detection

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Widget detection on screenshots using computer vision and machine learning algorithms'. Together they form a unique fingerprint.

  • Cite this

    Radzikowski, K., Chȩciński, K., Forc, M., Lepak, Ł., Jabłoński, M., Kuśmirek, W., Twardowski, B., Wawrzyński, P., & Nowak, R. M. (2019). Widget detection on screenshots using computer vision and machine learning algorithms. In R. S. Romaniuk, & M. Linczuk (Eds.), Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2019 [111761Q] (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 11176). SPIE. https://doi.org/10.1117/12.2536406