Receiver Operating Characteristic (ROC) Analysis of Image Search-and-Localize Tasks

Rationale and Objectives: Receiver operating characteristic (ROC) analysis for the common image search-and-localize task, in which readers search an image for lesion or lesions not knowing a priori any exists, has been studied for over four decades. However, a satisfactory solution seems elusive. Materials and Methods: We show that the ROC curve predictive of clinical outcomes where readers are penalized appropriately for not correctly localizing known lesions cannot be obtained because it is a missing data problem. Further, this ROC curve is between the case based ROC curve where readers are not penalized and the lesion-based ROC curve where penalty applies. Moreover, the lesion-based ROC curve is the LROC curve proposed by Starr et al. We show maximum-likelihood (ML) estimation of the LROC curve, validation of this procedure with Monte Carlo simulations, and its application to reader ROC datasets. Results: Monte Carlo simulations validated ML estimation of area under the LROC curve (AUC) and its variance. Example applications showed that ML estimate of LROC curve fits experimental datasets. Conclusion: The ROC curve predictive of clinical performance cannot be estimated from reader ROC data alone because it is a missing data problem, and is between the case-based ROC curve where readers are not penalized for not correctly identifying known lesions and the lesion-based ROC curve where penalty applies. The lesion-based ROC curve is the LROC curve proposed by Starr et al. and can be estimated via ML estimation.