On the effectiveness of mutation analysis as a black box testing technique

The technique of mutation testing, in which the effectiveness of tests is determined by creating variants of a program in which statements are mutated, is well known. Whilst of considerable theoretical interest the technique requires costly tools and is computationally expensive. Very large numbers of 'mutants' can be generated for even simple programs. More recently it has been proposed that the concept be applied to specification based (black box) testing. The proposal is to generate test cases by systematically replacing data-items relevant to a particular part of a specification with a data-item relevant to another. If the specification is considered as generating a language that describes the set of valid inputs then the mutation process is intended to generate syntactically valid and invalid statements. Irrespective of their 'correctness' in terms of the specification, these can then be used to test a program in the usual (black box) manner. For this approach to have practical value it must produce test cases that would not be generated by other popular black box test generation approaches. This paper reports a case study involving the application of mutation based black box testing to two programs of different types. Test cases were also generated using equivalence class testing and boundary value testing approaches. The test cases from each method were examined to judge the overlap and to assess the value of the additional cases generated. It was found that less than 20% of the mutation test cases for a data-vetting program were generated by the other two methods, as against 75% for a statistical analysis program. This paper analyses these results and suggests classes of specifications for which mutation based test-case generation may be effective.