Confidence intervals of the prediction ability and performance scores of classifications methods

Chemometricians widely use multivariate classification and class-modeling techniques in identity problems, in multivariate quality control, to obtain homogeneous set for multivariate calibration. The performance of such techniques is measured by parameters related to the prediction rate, the percentage of correct classifications of the objects in the evaluation set, i.e., the objects not used to develop the classification model. Sometimes, the prediction rate is discussed with reference to the no-model rate. In this paper, we suggest to compute a confidence interval for the prediction rate and the derived quantities, as people usually do in the case of measured chemical quantities. The statistical bases of this confidence interval are presented, with an alternative estimate other than those previously known. Tables with the limits of the confidence intervals for some usual cases (two or three categories) are reported. To take into account the effect of the number of samples used to evaluate the prediction ability, a classification performance score is introduced, whose value is zero when the classification rate is compatible with the hypothesis of random assignment. (C) 2001 Elsevier Science B.V. All rights reserved.