Evidence Review Group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis

Background: The National Institute for Health and Clinical Excellence (NICE) single technology appraisal (STA) process was set up as a rapid way to appraise new technologies for use within the NHS in England and Wales and has been in place since 2005. Objectives: This study had five primary objectives: (1) to provide a map of the STA process to date; (2) to identify current approaches to the critical appraisal of manufacturers' submissions (MSs) by Evidence Review Groups (ERGs); (3) to identify recurring themes in clarification letters sent to manufacturers; (4) to provide recommendations for possible alternative approaches to be used in the critical appraisal process; and (5) to revise the current ERG report template. Data sources: Data for the mapping of the STA process were obtained from the NICE website (www.nice.org.uk). Data for the analyses of the ERG reports and clarification letters were taken from the reports and letters themselves. Review methods: For the mapping, a spreadsheet was developed to collect data on 22 predefined variables related to timings and outcomes. Simple descriptive statistics were used to analyse the data. For the thematic analysis, a documentary analysis of 30 ERG reports was undertaken. Data on key elements of the MSs, the processes undertaken by ERGs and the strengths and weaknesses of MSs were extracted. A framework of a priori themes was developed. Data were extracted, coded and analysed according to a framework approach. Twenty-one clarification letters were examined and data were extracted using a set of codes to cover report quality, systematic review methods and clinical/economic issues. The current ERG report template was modified and sent to the current ERG teams for comment. All comments were considered and formed the basis for further revisions to the template. Results: Ninety-five STAs were included in the mapping exercise. Many STAs were subject to delay or cancellation for a variety of reasons. The ERG reports highlighted the strengths and weaknesses of MSs to the STA process. Thematic analysis of these data offered a means of clarifying and describing these aspects of the submissions. This analysis generated five themes: process, reporting, satisfaction of objectives, reliability and validity of findings, and content. Points from clarification letters were analysed and presented in four main categories: report quality, systematic review methods, clinical data analysis and economic data analysis. Limitations: Nearly all data were obtained from the NICE website; therefore, any errors in the data on the website will be reflected in the mapping analysis presented in this report. Missing data for the mapping exercise limit the generalisability of the findings. Analyses were limited to what was reported in the ERG reports and the clarification letters. Conclusions: Guidance suggested for manufacturers will help to ensure that more appropriate submissions are received in the future while recommendations provided for ERG teams will help to guide teams to ensure that reporting methods are transparent.