A systematic literature review of literature reviews in software testing

被引:77
|
作者
Garousi, Vahid [1 ,2 ]
Mantyla, Mika V. [3 ]
机构
[1] Hacettepe Univ, Dept Comp Engn, Software Engn Res Grp, Ankara, Turkey
[2] Maral Software Engn Consulting Corp, Calgary, AB, Canada
[3] Univ Oulu, Fac Informat Technol & Elect Engn, M3S, Oulu, Finland
关键词
Secondary studies; Tertiary study; Software testing; Systematic mapping; Systematic literature reviews; Surveys; CITED ARTICLES; QUALITY; COST;
D O I
10.1016/j.infsof.2016.09.002
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Context: Any newcomer or industrial practitioner is likely to experience difficulties in digesting large volumes of knowledge in software testing. In an ideal world, all knowledge used in industry, education and research should be based on high-quality evidence. Since no decision should be made based on a single study, secondary studies become essential in presenting the evidence. According to our search, over 101 secondary studies have been published in the area of software testing since 1994. With this high number of secondary studies, it is important to conduct a review in this area to provide an overview of the research landscape in this area. Objective: The goal of this study is to systematically map (classify) the secondary studies in software testing. We propose that tertiary studies can serve as summarizing indexes which facilitate finding the most relevant information from secondary studies and thus supporting evidence-based decision making in any given area of software engineering. Our research questions (RQs) investigate: (1) Software-testing specific areas, (2) Types of RQs investigated, (3) Numbers and Trends, and (4) Citations of the secondary studies. Method: To conduct the tertiary study, we use the systematic-mapping approach. Additionally, we contrast the testing topics to the number of Google hits to address a general popularity of a testing topic and study the most popular papers in terms of citations. We furthermore demonstrate the practicality and usefulness of our results by mapping them to ISTQB foundation syllabus and to SWEBOK to provide implications for practitioners, testing educators, and researchers. Results: After a systematic search and voting process, our study pool included 101 secondary studies in the area of software testing between 1994 and 2015. Among our results are the following: (1) In terms of number of secondary studies, model-based approach is the most popular testing method, web services are the most popular system under test (SUT), while regression testing is the most popular testing phase; (2) The quality of secondary studies, as measured by a criteria set established in the community, is slowly increasing as the years go by; and (3) Analysis of research questions, raised and studied in the pool of secondary studies, showed that there is a lack of 'causality' and 'relationship' type of research questions, a situation which needs to be improved if we, as a community, want to advance as a scientific field. (4) Among secondary studies, we found that regular surveys receive significantly more citations than SMs (p = 0.009) and SLRs (p = 0.014). Conclusion: Despite the large number of secondary studies, we found that many important areas of software testing currently lack secondary studies, e.g., test management, role of product risk in testing, human factors in software testing, beta-testing (A/B-testing), exploratory testing,. testability, test stopping criteria, and test-environment development. Having secondary studies in those areas is important for satisfying industrial and educational needs in software testing. On the other hand, education material of ISTQB foundation syllabus and SWEBOK could benefit from the inclusion of the latest research topics, namely search-based testing, use of cloud-computing for testing and symbolic execution. (C) 2016 Elsevier B.V. All rights reserved.
引用
收藏
页码:195 / 216
页数:22
相关论文
共 50 条
  • [1] Systematic literature reviews in software engineering - A systematic literature review
    Kitchenham, Barbara
    Brereton, O. Pearl
    Budgen, David
    Turner, Mark
    Bailey, John
    Linkman, Stephen
    [J]. INFORMATION AND SOFTWARE TECHNOLOGY, 2009, 51 (01) : 7 - 15
  • [2] Testing scientific software: A systematic literature review
    Kanewala, Upulee
    Bieman, James M.
    [J]. INFORMATION AND SOFTWARE TECHNOLOGY, 2014, 56 (10) : 1219 - 1232
  • [4] A Systematic Literature Review of Reviews on the Effectiveness of Chlamydia Testing
    Wong, William Chi Wai
    Lau, Stephanie Tsz Hei
    Choi, Edmond Pui Hang
    Tucker, Joseph D.
    Fairley, Christopher K.
    Saunders, John M.
    [J]. EPIDEMIOLOGIC REVIEWS, 2019, 41 (01) : 168 - 175
  • [5] Automation of systematic literature reviews: A systematic literature review
    van Dinter, Raymon
    Tekinerdogan, Bedir
    Catal, Cagatay
    [J]. INFORMATION AND SOFTWARE TECHNOLOGY, 2021, 136
  • [6] Analysing app reviews for software engineering: a systematic literature review
    Dąbrowski, Jacek
    Letier, Emmanuel
    Perini, Anna
    Susi, Angelo
    [J]. Empirical Software Engineering, 2022, 27 (02)
  • [7] Systematic literature reviews in software engineering
    Wohlin, Claes
    Prikladniki, Rafael
    [J]. INFORMATION AND SOFTWARE TECHNOLOGY, 2013, 55 (06) : 919 - 920
  • [8] Analysing app reviews for software engineering: a systematic literature review
    Jacek Dąbrowski
    Emmanuel Letier
    Anna Perini
    Angelo Susi
    [J]. Empirical Software Engineering, 2022, 27
  • [9] Software product line testing: a systematic literature review
    Agh, Halimeh
    Azamnouri, Aidin
    Wagner, Stefan
    [J]. EMPIRICAL SOFTWARE ENGINEERING, 2024, 29 (06)
  • [10] Software requirements testing approaches: a systematic literature review
    Jemison dos Santos
    Luiz Eduardo G. Martins
    Valdivino A. de Santiago Júnior
    Lucas Venezian Povoa
    Luciana Brasil R. dos Santos
    [J]. Requirements Engineering, 2020, 25 : 317 - 337