Empirical research on concurrent software testing: A systematic mapping study

Background: Concurrent software testing is a costly and difficult task, especially due to the exponential increase in the test sequences caused by non-determinism. Such an issue has motivated researchers to develop testing techniques that select a subset of the input domain that has a high probability of revealing faults. Academics and industrial practitioners rarely use most concurrent software testing techniques because of the lack of data about their applicability. Empirical evidence can provide an important scientific basis for the strengths and weaknesses of each technique to help researchers and practitioners choose concurrent testing techniques appropriate for their environments. Aim: This paper gathers and synthesizes empirical research on concurrent software testing to characterize the field and the types of empirical studies performed. Method: We performed a systematic mapping study to identify and analyze empirical research on concurrent software testing techniques. We provide a detailed analysis of the studies and their design choices. Results: The primary findings are: (1) there is a general lack of empirical validation of concurrent software testing techniques, (2) the type of evaluation method varies with the type of technique, (3) there are some key challenges to empirical study design in concurrent software testing, and (4) there is a dearth of controlled experiments in concurrent software testing. Conclusions: There is little empirical evidence available about some specific concurrent testing techniques like model-based testing and formal testing. Overall, researchers need to perform more empirical work, especially real-world case studies and controlled experiments, to validate properties of concurrent software testing techniques. In addition, researchers need to perform more analyses and synthesis of the existing evidence. This paper is a first step in that direction.