Research

Flaky Tests

Since flaky tests can pass or fail without any changes to the test code or the code under test, they are an unreliable indicator of software correctness and a major problem for many software developers. To address this challenge, my collaborators and I have developed an automated method that uses machine learning to predict which tests are most likely to be flaky (Parry et al. 2022b) . Along with proposing a methodology for automatically determining which tests have latent flakiness (Parry et al. 2020) , my colleagues and I also published a survey of the literature relevant to flaky test research (Parry et al. 2022a) , and a multi-source investigation into how developers experience the impacts and causes of flaky tests (Parry et al. 2022c) . Some of my recent work in this area suggests that researchers and developers should go beyond any individual metric of “good” or “bad” tests — like test flakiness — as it argues for a more holistic view of a test suite’s health (McMinn et al. 2025) .

My research in this area also shows that flaky test failures often occur in clusters whose members share a common root cause, a phenomenon that my collaborators and I call systemic flakiness, since this result implies that developers can repair several flaky tests by addressing one underlying problem (Parry et al. 2025) . Building on this observation, my colleagues and I revealed that the mutation operators used to seed synthetic faults can themselves induce flakiness, a phenomenon that we call test flimsiness and that calls into question any technique which assumes that a test reacts deterministically to a mutated program (Parry et al. 2026) .

My collaborators and I have also examined how well detection techniques serve the developers who repair flaky tests, since automated rerunning detects the developer-repaired tests in only 40% of the studied commits, which makes it an unsuitable baseline for judging a technique’s usefulness for developers (Parry et al. 2022d) . To reduce the cost of detection without sacrificing its accuracy, my colleagues and I introduced CANNIER, which combines machine learning models with rerunning-based detection to decrease the time cost of detection by an order of magnitude (Parry et al. 2023) . My collaborators and I have also turned these insights into a practical tool, FlakeFighters, an extensible PyTest plugin that combines static and execution data to classify a failing test as a genuine bug or a flaky failure in Python projects (Foster et al. 2026) .

Database Testing

Given the importance of verifying that a database application operates correctly, my PhD dissertation (Kapfhammer 2007) presented and evaluated an approach for testing this type of software; a notable paper derived from my dissertation is (Kapfhammer and Soffa 2003) . Since the relational schema preserves the correctness of the database’s state, my research has also focused on testing this complex artifact, with (McMinn et al. 2015) and (McMinn et al. 2019) being examples of papers. Recent papers such as (Alsharif et al. 2020b) and (Alsharif et al. 2020a) present and study techniques for improving the efficiency and effectiveness of regression testing for relational database schemas by reordering or reducing the schema’s test suites. To make schema testing practical, my collaborators and I have built tools that automatically generate test data for integrity constraints, including DOMINO (Alsharif et al. 2018a) and (Alsharif et al. 2018b) , and SchemaAnalyst, which supports several test data generators and a reproducible data analysis workflow (Alsharif et al. 2018c) and (McMinn, Wright, et al. 2016) .

This line of work also includes database-aware coverage monitoring (Kapfhammer and Soffa 2008) , search-based generation of integrity constraint tests (Kapfhammer et al. 2013) , and the assessment of the efficiency of that search-based approach (Kinneer et al. 2015a) . My research has also built tools that adapted mutation analysis to this domain, enabling the study of efficient mutation for database structures (Wright et al. 2013) , the impact of equivalent and redundant mutants (Wright et al. 2014) , and virtual mutation analysis (McMinn, Kapfhammer, et al. 2016) . Complementing these techniques, my colleagues and I have examined how testers understand SQL test cases (Alsharif et al. 2019) , localized SQL faults (Clark et al. 2011) , detected dynamic invariants that a database should satisfy (Cobb et al. 2011) , and studied the data stores that such applications rely on (Arnold et al. 2002) .

Web Testing

In response to the prevalence and complexity of mobile-ready web sites, my research has developed automated tools for checking the pages in these sites. Some examples of papers describing methods and prototype tools for automatically detecting defects in web pages include (Walsh et al. 2017a) and (Walsh et al. 2017b) . Earlier work in this line detected layout faults that appear after a developer changes a web page (Walsh et al. 2015) . The automated approach introduced in (Althomali et al. 2019) can visually confirm and classify the reported responsive layout failures in a web page. Building on this approach, my colleagues and I have also classified presentation failures that arise from the page’s document object model (Althomali et al. 2021) . The method described in (Walsh et al. 2020) can automatically identify potential regressions from the correct responsive layout of a web page. Finally, recent work introduced a method that automatically repairs a layout failure in a responsive web page by effectively “hiding” the defect from perception (Althomali et al. 2022) . The experiments in the aforementioned papers study real-world web pages, ultimately demonstrating that the presented approaches can automatically detect layout defects in deployed sites.

Mutation Testing

Given the challenges associated with accurately judging the quality of a test suite, my research has designed efficient and useful ways to perform test assessment through mutation analysis. Presenting and evaluating automated techniques that efficiently insert synthetic faults into both real-world Java programs and relational database schemas, some examples of papers include (Just et al. 2012) , (Wright et al. 2014) , and (McMinn et al. 2019) . Finally, recent work explores how the eXtreme mutation testing (XMT) and statement deletion (SDL) mutation operators can detect pseudo-tested methods and statements in a Java program (Maton et al. 2024a) .

To bring these techniques into practice, my collaborators and I implemented PseudoSweep, a tool that automatically identifies pseudo-tested methods and statements in Java projects by combining method and statement deletion (Maton et al. 2024b) . Because developers who inspect covered but unchecked statements must first choose how to compute the corresponding oracle gaps, my colleagues and I empirically compared three calculation approaches and characterized how the discovered gaps differ in prominence, distribution, and fault detection ability (Maton et al. 2025) .

My research has also started to use large language models to create mutants, showing that a hazard-guided approach produces compilable, diverse, and powerful mutants that reveal defects which prior methods fail to detect (Maton et al. 2026a) . To help other researchers compare these emerging techniques, my colleagues and I released multiplex, a modular framework that supports the prototyping and evaluation of large-language-model-based mutant generation modules across different languages, models, and build environments (Maton et al. 2026b) .

Regression Testing

Since software is often incrementally developed, my research on regression testing has created research automated ways to efficiently and effectively run test suites for complex programs. Some examples of papers that present and evaluate regression testing techniques for reordering and reducing test suites include (Walcott et al. 2006) , (Lin et al. 2014) , and (Lin et al. 2017) . Earlier work in this line developed knapsack-based and time-aware methods for prioritizing test suites, including (Alspaugh et al. 2007) , (Rummel et al. 2005) , (Bhadra and Kapfhammer 2008) , and (Kapfhammer 2001) . Papers such as (Kapfhammer and Soffa 2007) , and (Smith et al. 2009) studied how to measure the coverage effectiveness of a prioritization, (Williams and Kapfhammer 2010) compared prioritizations with synthetic test suites, and (Lin et al. 2013) investigated history-based prioritization. My research has also reduced the cost of regression testing with call trees (Smith et al. 2007) , cost-aware reduction (Smith and Kapfhammer 2009) , irreplaceable test cases (Lin et al. 2012) , and methods tailored to database applications (Kapfhammer 2012) , (Kapfhammer 2007) , (Alsharif et al. 2020b) , and (Alsharif et al. 2020a) .

Complementing these techniques, my colleagues and I have studied the selection operators used by search-based prioritization (Conrad et al. 2010) , combined non-redundant mutation operators with test suite prioritization (Just et al. 2012) , built a framework that supports both industrial and academic work on these methods (Kauffman and Kapfhammer 2012) , and surveyed and empirically evaluated the field as a whole (Kapfhammer 2010) , (Kapfhammer 2011) . Leading the way in the realistic assessment of regression testing methods, papers such as (Paterson et al. 2018) and (Paterson et al. 2019) , show how to conduct rigorous regression testing experiments with real program faults. The variance that these experiments must control when they substitute mutants for real faults also motivates my recent work on test flimsiness (Parry et al. 2026) which shows that a mutation can itself make a test outcome non-deterministic.

Search-Based Methods

Using a fitness function as a guide to a solution, search-based methods have shown promise in supporting many software engineering tasks. My work has focused on creating frameworks that support the development of search-based tools, with (McMinn and Kapfhammer 2016) being an example. Other papers like (Conrad et al. 2010) and (Kukunas et al. 2010) describe search-based solutions to software engineering tasks like regression testing or performance optimization. Because the configuration of a search-based generator strongly influences its results, my colleagues and I have revisited how to tune the parameters of a generator (Kotelyanskii and Kapfhammer 2014) , compared search-based techniques on synthetic test suites whose properties are known (Williams and Kapfhammer 2010) , and applied search-based generation to the integrity constraints in relational database schemas with DOMINO (Alsharif et al. 2018a) and SchemaAnalyst (McMinn, Wright, et al. 2016) . Recent papers like (Alsharif et al. 2019) present experimental methodologies that effectively involve humans when studying the usefulness of tests that were generated by search-based techniques. The search-based automation that these techniques exemplify has more recently expanded to use large language models in place of a fitness function, as in my collaborative work on multiplex, which enables the automatic generation of mutants for a program under test (Maton et al. 2026b) .

Performance Evaluation

Given the importance of equipping software engineers with the insights and tools they need to create efficient software, my work has developed tools that automatically assess program performance. Focusing on the empirical evaluation of real-world software components like databases, relevant papers include (Jones and Kapfhammer 2011) , (Burdette et al. 2012) , and (Kinneer et al. 2015a) . Papers such as (Kotelyanskii and Kapfhammer 2014) highlight how my research has investigated the influence that the parameters of a search-based algorithm have on efficiency and effectiveness of methods for automated test data generation. My research has also measured the performance of the building blocks that such systems rely on, including the run-time behavior of GUI creation frameworks (Howell et al. 2003) , the performance of tuple space communication (Fiedler et al. 2005) , and the execution of a test suite inside a resource-constrained environment (Kapfhammer et al. 2005) . Since a theoretical analysis of an algorithm’s worst-case time complexity is difficult to either prove in the first place or to generalize, my colleagues and I have also inferred that complexity automatically from empirical measurements (Kinneer et al. 2015b) .

Research Methods

My surveys of software testing techniques provide a starting point for people exploring this field, with (Kapfhammer 2004) and (Kapfhammer 2010) being examples of such articles. Papers like (Kapfhammer et al. 2016) and (McMinn, Harman, et al. 2016) show how I have articulated a research agenda for the field of software engineering that stresses, for instance, the need for well-tested statistical methods. I have also written papers, like (Alsharif et al. 2018b) and (Alsharif et al. 2018c) , that explain how to replicate my empirical studies of search-based testing techniques. The statistical rigor and replication practices advocated here also shape my recent large-scale studies of flaky tests (Parry et al. 2026) , which analyze millions of mutant-test pairs collected from over half a million test suite executions across 28 Python projects. Because these empirical studies ultimately serve people (e.g., software engineers), my research places a deliberate emphasis on studies that involve humans, examining how testers understand SQL test cases (Alsharif et al. 2019) , how developers experience the effects of flaky tests (Parry et al. 2022c) , how hybrid methods for reducing schema test suites compare in practice (Alsharif et al. 2020a) , and how repaired responsive layouts are evaluated (Althomali et al. 2022) .