What Makes Killing a Mutant Hard

Mutation operators have been studied at length to determine which ones are the "best" at some metric (for example creates the least equivalent mutants, creates hard-to-kill mutants, etc.). These studies though have focused on specific test suites, where the test inputs and oracles are fixed, which leads to results that are strongly influenced by the test suites and thus makes the conclusions potentially less general. In this paper we consider all test inputs and we assume we have no prior knowledge about the likelihood of any specific inputs. We will also show how varying the strength of the oracle have a big impact on the results. We only consider a few mutation operators (mostly relational), only a handful of programs to mutate (amenable to probabilistic symbolic execution), and only consider how likely it is that a mutant is killed. A core finding is that the likelihood of reaching the source line where the mutation is applied, is an important contributor to the likelihood of killing the mutant and when we control for this we can see which operators create mutations that are too easy versus very hard to kill.