Interrater Agreement in the Evaluation of Discrepant Imaging Findings With the Radpeer System

被引:54
|
作者
Bender, Leila C. [1 ]
Linnau, Ken F. [1 ]
Meier, Eric N. [2 ]
Anzai, Yoshimi [1 ]
Gunn, Martin L. [1 ]
机构
[1] Univ Washington, Dept Radiol, Seattle, WA 98104 USA
[2] Univ Washington, Dept Biostat, Seattle, WA 98104 USA
关键词
quality assurance; quality improvement; Radpeer; QUALITY IMPROVEMENT; RADIOLOGY QUALITY; PEER ASSESSMENTS; RELIABILITY;
D O I
10.2214/AJR.12.8972
中图分类号
R8 [特种医学]; R445 [影像诊断学];
学科分类号
1002 ; 100207 ; 1009 ;
摘要
OBJECTIVE. The Radpeer system is central to the quality assurance process in many radiology practices. Previous studies have shown poor agreement between physicians in the evaluation of their peers. The purpose of this study was to assess the reliability of the Radpeer scoring system. MATERIALS AND METHODS. A sample of 25 discrepant cases was extracted from our quality assurance database. Images were made anonymous; associated reports and identities of interpreting radiologists were removed. Indications for the studies and descriptions of the discrepancies were provided. Twenty-one subspecialist attending radiologists rated the cases using the Radpeer scoring system. Multirater kappa statistics were used to assess interrater agreement, both with the standard scoring system and with dichotomized scores to reflect the practice of further review for cases rated 3 and 4. Subgroup analyses were conducted to assess subspecialist evaluation of cases. RESULTS. Interrater agreement was slight to fair compared with that expected by chance. For the group of 21 raters, the kappa values were 0.11 (95% CI, 0.06-0.16) with the standard scoring system and 0.20 (95% CI, 0.13-0.27) with dichotomized scores. There was disagreement about whether a discrepancy had occurred in 20 cases. Subgroup analyses did not reveal significant differences in the degree of interrater agreement. CONCLUSION. The identification of discrepant interpretations is valuable for the education of individual radiologists and for larger-scale quality assurance and quality improvement efforts. Our results show that a ratings-based peer review system is unreliable and subjective for the evaluation of discrepant interpretations. Resources should be devoted to developing more robust and objective assessment procedures, particularly those with clear quality improvement goals.
引用
收藏
页码:1320 / 1327
页数:8
相关论文
共 50 条
  • [31] The agreement between magnetic resonance imaging and arthroscopic findings in temporomandibular joint
    Verhelst, P-J
    Vervaeke, K.
    Orhan, K.
    Lund, B.
    Benchimol, D.
    Coucke, W.
    Van der Cruyssen, F.
    De Laat, A.
    Politis, C.
    Jacobs, R.
    INTERNATIONAL JOURNAL OF ORAL AND MAXILLOFACIAL SURGERY, 2021, 50 (05) : 657 - 664
  • [32] Poor Agreement Between Imaging and Histologic and Colonoscopy Findings in Pediatric Patients
    Chapa-Rodriguez, Adrian
    Bhatia, Tavleen
    Buckley, Andrew
    Baker, Susan S.
    Baker, Robert D.
    Alkhouri, Razan H.
    JOURNAL OF PEDIATRIC GASTROENTEROLOGY AND NUTRITION, 2018, 66 (02): : 263 - 267
  • [33] The Interrater and Intrarater Agreement of a Modified Neer Classification System and Associated Treatment Choice for Lateral Clavicle Fractures
    Cho, Chul-Hyun
    Oh, Joo Han
    Jung, Gu-Hee
    Moon, Gi-Hyuk
    Rhyou, In Hyeok
    Yoon, Jong Pil
    Lee, Ho Min
    AMERICAN JOURNAL OF SPORTS MEDICINE, 2015, 43 (10): : 2431 - 2436
  • [34] MEASUREMENT OF AGREEMENT AS AN EVALUATION OF A DIAGNOSTIC-IMAGING METHOD
    BLUM, AG
    FELDMANN, L
    BRESLER, F
    JOUANNY, P
    BRIANCON, S
    REGENT, DM
    RADIOLOGY, 1995, 197 : 478 - 478
  • [35] MEASURING AGREEMENT AS AN EVALUATION OF A DIAGNOSTIC-IMAGING METHOD
    BLUM, A
    FELDMANN, L
    BRESLER, F
    JOUANNY, P
    BRIANCON, S
    REGENT, D
    JOURNAL DE RADIOLOGIE, 1995, 76 (07): : 441 - 443
  • [36] Diagnosing adjustment disorder in patients with cancer: evaluation of the adherence, interrater agreement, and content of a guideline-based interview
    Wijnhoven, Lonneke M. A.
    van Zutphen, Linda
    Custers, Jose A. E.
    van Beek, Florie E.
    Holtmaat, Karen
    Jansen, Femke
    Verdonck-de Leeuw, Irma M.
    Kwakkenbos, Linda
    Prins, Judith B.
    JOURNAL OF PSYCHOSOCIAL ONCOLOGY RESEARCH AND PRACTICE, 2024, 6 (01):
  • [37] Analysis of agreement between specialists for the evaluation of radiological findings of necrotizing enterocolitis
    Scarpa, Erica Cristina
    Lyta, Joao C.
    Lourencao, Pedro L. T. de A.
    Hachem, Andre S.
    da Silva, Geraldo H. S.
    Giacoia, Glauce R. F.
    Ortolan, Erika V. P.
    Silva, Camila de Paula
    da Silveira, Guilherme L.
    Bentlin, Maria R.
    JORNAL DE PEDIATRIA, 2025, 101 (01) : 103 - 109
  • [38] INTER-RATER AGREEMENT ON IMAGING FINDINGS OF CHRONIC PANCREATITIS IN A PEDIATRIC COHORT
    Trout, Andrew T.
    Abu-El-Haija, Maisam
    Anupindi, Sudha A.
    Marine, Megan
    Murati, Michael
    Phelps, Andrew S.
    Rees, Mitchell A.
    Squires, Judy H.
    Ellery, Kate
    Gariepy, Cheryl E.
    Maqbool, Asim
    Mcferron, Brian A.
    Perito, Emily
    Schwarzenberg, Sarah Jane
    Zhang, Bin
    Lowe, Mark
    Uc, Aliye
    GASTROENTEROLOGY, 2021, 160 (06) : S113 - S113
  • [39] Repeatability of the Sirius Imaging System and Agreement With the Pentacam HR
    Nasser, Cornelius K.
    Singer, Reut
    Barkana, Yaniv
    Zadok, David
    Avni, Isaac
    Goldich, Yakov
    JOURNAL OF REFRACTIVE SURGERY, 2012, 28 (07) : 493 - 497
  • [40] Simple procedures to estimate chance agreement and kappa for the interrater reliability of response segments using the Rorschach comprehensive system
    Meyer, GJ
    JOURNAL OF PERSONALITY ASSESSMENT, 1999, 72 (02) : 230 - 255