MERCI - A Method to Evaluate Combinatorial Interaction Testing Tools for Software Product Lines

Testing a system is a routine activity, and it plays an important role in the software quality assurance process. However, testing highly-configurable systems, such as Software Product Lines (SPL), is a rather complex activity, due to the presence of variability in its engineering process, which increases the number of product configurations to test. The underlying idea to make testing feasible in SPL engineering is to select a small but representative subset of products to test, by employing techniques such as combinatorial interaction testing (CIT). This paper presents Method to Evaluate Combinatorial Interaction (MERCI), a novel method to evaluate the adequacy of existing CIT tools for SPL engineering, with respect to three measures: defect detection, test coverage, and test execution length. We carried out an empirical evaluation to compare four CIT tools: ACTS, CATS, PICTMaster and VPTag. The results show that the method may serve as an affordable strategy to evaluate how the CIT tools could behave in an SPL testing scenario.