Comparison of three-class classification performance metrics: a case study in breast cancer CAD

Receiver Operating Characteristic (ROC) analysis is a widely used method for analyzing the performance of two-class classifiers. Advantages of ROC analysis include the fact that it explicitly considers the tradeoffs in sensitivity and specificity, includes visualization methods, and has clearly interpretable summary metrics. Currently, there does not exist a widely accepted performance method similar to ROC analysis for an N-class classifier (N > 2). The purpose of this study was to empirically compare methods that have been proposed to evaluate the performance of N-class classifiers (N>2). These methods are, in one way or another, extensions of ROC analysis. This report focuses on three-class classification performance metrics, but most of the methods can be extended easily for more than three classes. The methods studied were pairwise ROC analysis, Hand and Till M Function (HTM), one-versus-all ROC analysis, a modified HTM, and Mossman's "Three-Way ROC" method. A three-class classification task from breast cancer computer-aided diagnosis (CADx) is taken as an example to illustrate the advantages and disadvantages of the alternative performance metrics.