A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests release_cidxhbu75ja6ngjwwfv3dseb7a

by Sarra Habchi, Guillaume Haben, Mike Papadakis, Maxime Cordy, Yves Le Traon

Released as a article .

2021  

Abstract

Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic outcomes that cripple continuous integration and lead developers to investigate false alerts. Industrial reports indicate that on a large scale, the accrual of flaky tests breaks the trust in test suites and entails significant computational cost. To alleviate this, practitioners are constrained to identify flaky tests and investigate their impact. To shed light on such mitigation mechanisms, we interview 14 practitioners with the aim to identify (i) the sources of flakiness within the testing ecosystem, (ii) the impacts of flakiness, (iii) the measures adopted by practitioners when addressing flakiness, and (iv) the automation opportunities for these measures. Our analysis shows that, besides the tests and code, flakiness stems from interactions between the system components, the testing infrastructure, and external factors. We also highlight the impact of flakiness on testing practices and product quality and show that the adoption of guidelines together with a stable infrastructure are key measures in mitigating the problem.
In text/plain format

Archived Files and Locations

application/pdf  244.9 kB
file_ojpfjxj375aahalrl555wumak4
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2021-12-09
Version   v1
Language   en ?
arXiv  2112.04919v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 5f64c35e-67fa-4be9-8087-61173dabcbff
API URL: JSON