On the evaluation of microcalcification detection algorithms

We are developing a computer-based microcalcification detection technique - for use in a mass screening context - which prompts the radiologist with the locations of suspicious regions. When evaluating the value of such techniques, ideally we want to measure the resulting improvement in the radiologist's performance, but this is sometimes impractical because radiologists' time is scarce. We argue that ROC or FROC curve analysis alone is not appropriate when assessing such techniques since the intention is that the system works with, rather than replaces, radiologists. We outline an evaluation strategy which comes closer to a meaningful measure of performance in this situation.