On the comparison of FROC curves in mammography CAD systems

We present a novel method for assessing the performance of computer-aided detection systems on unseen cases at a given sensitivity level. The sampling error introduced when training the system on a limited data set is captured as the uncertainty in determining the system threshold that would yield a certain predetermined sensitivity on unseen data sets. By estimating the distribution of system thresholds, we construct a confidence interval for the expected number of false positive markings per image at a given sensitivity. We present two alternative procedures for estimating the probability density functions needed for the construction of the confidence interval. The first is based on the common assumption of Poisson distributed number of false positive markings per image. This procedure also relies on the assumption of independence between false positives and sensitivity, an assumption that can be relaxed with the second procedure, which is nonparametric. The second procedure uses the bootstrap applied to the data generated in the leave-one-out construction of the FROC curve, and is a fast and robust way of obtaining the desired confidence interval. Standard FROC curve analysis does not account for the uncertainty in setting the system threshold, so this method should allow for a more fair comparison of different systems. The resulting confidence intervals are surprisingly wide. For our system a conventional FROC curve analysis yields 0.47 false positive markings per image at 90% sensitivity. The 90% confidence interval for the number of false positive markings per image is (0.28, 1.02) with the parametric procedure and (0.27, 1.04) with the nonparametric bootstrap. Due to its computational simplicity and its allowing more fair comparisons between systems, we propose this method as a complement to the traditionally presented FROC curves. (C) 2005 American Association of Physicists in Medicine.