Empirically evaluating regression testing techniques: Challenges, solutions, and a potential way forward

The published studies of regression testing methods often contain many of the hallmarks of high quality empirical research. Beyond features like clear descriptions of the methodology and the visualization and statistical analysis of the data sets, certain papers in this field also provide some of the artifacts used in and/or produced by the experiments. Yet, the limited industrial adoption of regression testing techniques is due in part to a lack of comprehensive empirical evaluations. Moreover, the regression testing community has not achieved a level of experimental reproducibility that would fully establish it as a science. After identifying the challenges associated with evaluating regression testing methods, this paper advocates a way forward involving a mutually beneficial increased sharing of the inputs, outputs, and procedures used in experiments.