Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

Behrouz Zolfaghari, Reza M. Parizi, Gautam Srivastava*, Yoseph Hailemariam

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

10 Citations (Scopus)

Abstract

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software-defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

Original languageEnglish
Pages (from-to)851-867
Number of pages17
JournalSoftware - Practice and Experience
Volume51
Issue number5
DOIs
Publication statusPublished - 2021 May
Externally publishedYes

Keywords

  • detection
  • flaky testing
  • software
  • tools

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'Root causing, detecting, and fixing flaky tests: State of the art and future roadmap'. Together they form a unique fingerprint.

Cite this