Detecting and Reducing Redundancy in Software Testing for Highly Configurable Systems

While redundancy in software development has been used as a fault tolerance mechanism that can be useful for increasing software quality, redundancy in test artifacts leads to decreased testing performance and increased effort, mainly due to repetitive testing and high test maintenance costs. The effect of redundancy is especially accentuated in testing software for highly configurable systems, which contain a large number of configurable mutually interacting options that overlap across a test suite and introduce unnecessary redundancy. In this paper we propose a methodology for detecting and reducing redundancy in test suits for highly configurable software. We use equivalence partitioning to segregate test cases covering the same sets of feature interactions, then identify partitions with multiple test cases, and finally we remove those test cases that do not have unique partitions. We evaluate the approach in the industrial case study of testing highly configurable software, and on a large set of test suites developed motivated by the industrial case study. The results show that our approach can reduce test execution time by 35% on average compared to industry practice, without significant effect on fault detection effectiveness.