Developing case-specific checklists for standardized-patient-based assessments in internal medicine: A review of the literature

Purpose. To review the literature on the methods used in writing case-specific checklists for studies of internal medicine physicians' performances that were assessed by standardized patients. Method. The authors searched Medline, Embase, Psychlit, and ERIC for articles in English published between 1966 and February 1998. The following search string was used: "[(standardi" or simulat" or programm") near (patient* or client" or consultati")] and internal medicine." The authors then searched the reference lists of papers retrieved from the database searches, as well as those from seven proceedings of the International Ottawa Conference on Medical Education and Assessment. Results. The procedure yielded 29 relevant articles: database searches yielded 14 published reports dealing with case-specific checklists, 11 articles were culled from the reference lists of these papers, and the Ottawa Conference proceedings yielded four articles, Only 12 articles reported specifically on the development of checklists. In general, there were three sources used for developing checklists: panels of experts, the investigators themselves, and responses from expert physicians to written protocols. No article indicated that researchers had relied exclusively on data from the literature to compose their checklists. Only three articles indicated that literature sources had informed their checklist development. All articles except one relied on explicit criteria for the inclusion of items on the checklists. In 21 of the 29 articles, the checklists had been scored by SPs, but the scoring of specific items on the checklists varied according to the purpose of the SP-physician encounter. Only four of the articles made the checklists available or indicated that the checklists could be obtained from the authors. Conclusion. The development of case-specific checklists for SP examinations of physicians' performance has received little attention. To judge the validity of studies of physicians' performances that: use SPs, the development processes for the checklists need to be more fully described to enable readers to evaluate the validity and reliability of the studies.