Inter-rater reliability of the radiographic assessment of simple bone cysts

Purpose To develop and evaluate the reliability of an explicit set of parameters and criteria for simple bone cysts (SBCs) and evaluate the reliability of single versus serial chronological reading methods. Methods Radiographic criteria were developed based on the literature and expert consensus. A single anteroposterior/lateral radiograph from 32 subjects with SBC were evaluated by three radiologists. A second reading was then conducted using revised criteria including a visual schematic. In the third reading the same images were assessed but radiologists had access to images from two additional time points. Inter-rater reliability was assessed after each reading using kappa (kappa) and percentage agreement for categorical and binary parameters and intra-class correlation coefficient (ICC) for continuous parameters. Results Parameters that were revised with more explicit definitions including the visual schematic demonstrated consistent or improved inter-rater reliability with the exception of continuous cortical rim present and cyst location in the metaphysis and mid-diaphysis. Cortical rim displayed only slight reliability throughout (kappa = -0.008 to 0.16). All other categorical parameters had a percentage agreement above 0.8 or a moderate (kappa = 0.41 to 0.60), substantial (kappa = 0.61 to 0.80) or almost perfect inter-rater reliability (kappa = 0.81 to 1.0) in at least one reading. All continuous parameters demonstrated excellent inter-rater reliability (ICC > 0.75) in at least one reading with the exception of scalloping (ICC = 0.37 to 0.70). Inter-rater reliability values did not indicate an obviously superior method of assessment between single and serial chronological readings. Conclusion Explicit criteria for SBC parameters used in their assessment demonstrated improved and substantial inter-rater reliability. Inter-rater reliability did not differ between single and serial chronological readings.