Surveying the developer experience of flaky tests
flaky tests
human study
Proceedings of the 44th International Conference on Software Engineering – Software Engineering in Practice Track
Abstract
Test cases that pass and fail without changes to the code under test are known as flaky. The past decade has seen increasing research interest in flaky tests, though little attention has been afforded to the views and experiences of software developers. In this study, we utilized a multi-source approach to obtain insights into how developers define flaky tests, their experiences of the impacts and causes of flaky tests, and the actions they take in response to them. To that end, we designed a literature-guided developer survey that we deployed on social media, receiving 170 total responses. We also searched on StackOverflow and analyzed 38 threads relevant to flaky tests, offering a distinct perspective free of any self-reporting bias. Using a mixture of numerical and thematic analyses, this study revealed a number of findings, including (1) developers strongly agree that flaky tests hinder continuous integration; (2) developers who experience flaky tests more often may be more likely to ignore potentially genuine test failures; and (3) developers rate issues in setup and teardown to be the most common causes of flaky tests.Details
Presentation
flake-it/flaky-test-survey-replication-package
Reference
@inproceedings{Parry2022b,
author = {Owain Parry and Gregory M. Kapfhammer and Michael Hilton and Phil
McMinn},booktitle = {Proceedings of the 44th International Conference on Software
Engineering -- Software Engineering in Practice Track},title = {Surveying the developer experience of flaky tests},
year = {2022}
}