Random versus combinatorial effectiveness in software conformance testing: a case study

Combinatorial interaction testing is widely rewarded as a powerful and cost-effective tool for generic debugging of large software implementations. However, its efficacy when applied to the specific task of testing a software for conformance to its specification has not yet been assessed, to the best of our knowledge. For this type of task, we show that the much easier and commonly used random testing approach is a less convenient choice with respect to applying a combinatorial based test suite of comparable size. We also performed a wider set of experiments and found that even much greater random testing efforts won't be able to trigger a comparably wide set of faults, with respect to the combinatorial based testing. The presented results are based on the case study of applying conformance testing to the verifier component of the Java virtual machine. The framework for the combinatorial driven generation of the conformance test suite is also described. In the framework, the test cases are generated by model checking the considered specification, and using a combinatorial coverage criteria targeted to the specification constraints. Results obtained from both types of test suites application are presented and discussed, with their comparison showing the better efficacy of the combinatorial one, and empirically validating the underlying approach.