Crowdsourced software testing: A systematic literature review

Context: Crowdsourced software testing (CST) refers to the use of crowdsourcing techniques in the domain of software testing. CST is an emerging area with its applications rapidly increasing in the last decade. Objective: A comprehensive review on CST has been conducted to determine the current studies aiming to improve and assess the value of using CST as well as the challenges identified by these evaluation studies. Method: We conducted a systematic literature review on CST by searching six popular databases. We identified 50 primary studies that passed our quality assessment criteria and defined two research questions covering the aim of the study. Results: There are three main process activities that the current literature aims to improve, namely selection of suitable testers, reporting of defects, and validation of submitted defects. In addition, there are 23 CST evaluation studies and most of them involve a large group and real crowd. These studies have identified 27 different challenges encountered during the application of crowdsourcing in software testing. Conclusions: The improvements achieved for the specific process activities in CST help explore other unexplored process activities. Similarly, knowing the characteristics of the evaluation studies can direct us on what other studies are worth investigating. Additionally, many of the challenges identified by the evaluation studies represent research problems that need better understanding and alternative solutions. This research also offers opportunities for practitioners to understand and apply new solutions proposed in the literature and the variations between them. Moreover, it provides awareness to the related parties regarding the challenges reported in the literature, which they may encounter during CST tasks.