Summary: Using Mutation Analysis for Assessing and
Comparing Testing Coverage Criteria
James H. Andrews, Member, IEEE, Lionel C. Briand, Senior Member, IEEE,
Yvan Labiche, Member, IEEE, and Akbar Siami Namin
Abstract---The empirical assessment of test techniques plays an important role in software testing research. One common practice is
to seed faults in subject software, either manually or by using a program that generates all possible mutants based on a set of mutation
operators. The latter allows the systematic, repeatable seeding of large numbers of faults, thus facilitating the statistical analysis of
fault detection effectiveness of test suites; however, we do not know whether empirical results obtained this way lead to valid,
representative conclusions. Focusing on four common control and data flow criteria (Block, Decision, C≠Use, and P≠Use), this paper
investigates this important issue based on a middle size industrial program with a comprehensive pool of test cases and known faults.
Based on the data available thus far, the results are very consistent across the investigated criteria as they show that the use of
mutation operators is yielding trustworthy results: Generated mutants can be used to predict the detection effectiveness of real faults.
Applying such a mutation analysis, we then investigate the relative cost and effectiveness of the above≠mentioned criteria by revisiting
fundamental questions regarding the relationships between fault detection, test suite size, and control/data flow coverage. Although
such questions have been partially investigated in previous studies, we can use a large number of mutants, which helps decrease the
impact of random variation in our analysis and allows us to use a different analysis approach. Our results are then compared with
published studies, plausible reasons for the differences are provided, and the research leads us to suggest a way to tune the mutation
analysis process to possible differences in fault detection probabilities in a specific environment.
Index Terms---Testing and debugging, testing strategies, test coverage of code, experimental design.