A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests
release_cidxhbu75ja6ngjwwfv3dseb7a
by
Sarra Habchi, Guillaume Haben, Mike Papadakis, Maxime Cordy, Yves Le Traon
2021
Abstract
Test flakiness forms a major testing concern. Flaky tests manifest
non-deterministic outcomes that cripple continuous integration and lead
developers to investigate false alerts. Industrial reports indicate that on a
large scale, the accrual of flaky tests breaks the trust in test suites and
entails significant computational cost. To alleviate this, practitioners are
constrained to identify flaky tests and investigate their impact. To shed light
on such mitigation mechanisms, we interview 14 practitioners with the aim to
identify (i) the sources of flakiness within the testing ecosystem, (ii) the
impacts of flakiness, (iii) the measures adopted by practitioners when
addressing flakiness, and (iv) the automation opportunities for these measures.
Our analysis shows that, besides the tests and code, flakiness stems from
interactions between the system components, the testing infrastructure, and
external factors. We also highlight the impact of flakiness on testing
practices and product quality and show that the adoption of guidelines together
with a stable infrastructure are key measures in mitigating the problem.
In text/plain
format
Archived Files and Locations
application/pdf 244.9 kB
file_ojpfjxj375aahalrl555wumak4
|
arxiv.org (repository) web.archive.org (webarchive) |
2112.04919v1
access all versions, variants, and formats of this works (eg, pre-prints)