Proposal and validation of a method to construct confidence intervals for clinical outcomes around FROC curves for mammography CAD systems

This paper introduces a method for constructing confidence intervals for possible clinical outcomes around the FROC curve of a mammography CAD system. Given the architecture of a CAD classifying machine, there is one and only one system threshold that will yield a desired sensitivity on a certain population. The limited training sample size leads to a sampling error and an uncertainty in determining the optimal system threshold. This leads to an uncertainty in the operating point in the direction along the FROC curve which can be captured by a Bayesian approach where the distribution of possible thresholds is estimated. This uncertainty contributes to a large and spread-out confidence interval which is important to consider when one is intending to make comparisons between CAD algorithms trained on different data sets. The method is validated using a Monte Carlo method designed to capture the effect of correctly determining the system threshold.