Challenges in the peer review of systematic reviews and meta-analyses

Objective: To assess the role of the referees in assisting the peer review process of systematic reviews and meta-analyses. Methods: A one-page questionnaire was mailed to 1391 referees of two journals, the American Journal of Obstetrics and Gynecology and Obstetrics and Gynecology. The referees were asked how often they verified by their own independent analysis 11 key items related to the methodology and statistical analysis of systematic reviews and meta-analyses. Response categories included "always", "frequently" (>50% of the time), "infrequently" (<= 50% of the time) and "never". A second and a third mailing was sent to the non-respondents. Results: 42 mailings were returned because of change of address. Of the remaining 1349 referees, 272 responded (response rate 20%). Of the 272 respondents, 159 (58%) had previously reviewed articles dealing with systematic reviews or meta-analyses. The responses varied according to the key items in the questions but the referees used their own independent analyses "always" in only 2%-17% of the time. The rates of "infrequently" or "never" responses combined together ranged from 51% to 86% for the various key items. Conclusion: The overwhelming majority of the referees do not verify, by their own independent analysis, key items related to methodology and statistical analysis of submitted systematic reviews and meta-analyses.