Industrial Evaluation of Test Suite Generation Strategies for Model-Based Testing

We report on a case study on model based testing for a commercially available telecom software system. A main purpose is to investigate how different strategies for test suite generation affect quality attributes of the generated test suites, in a realistic industrial environment. We develop a functional model in the form of an extended finite state machine, from which we generate test suites using several different (model) coverage criteria, alongside with randomly and manually generated test suites. We compare test suites with respect to fault-detection capability, incurred (source) code coverage, and test generation and execution time. The system under test is a commercially released version, not seeded with any faults, implying that exposed faults are "real" faults that passed previous testing. We did not find clear difference between coverage-based and random test suites. Test suite generation and execution is performed using the tool ERLY MARSH, developed by the first author.