Transductive confidence machine for active learning

This paper describes a novel active learning strategy using universal p-value measures of confidence based on algorithmic randomness, and transductive inference. The early stopping criteria for active learning is based on the bias-variance trade-off for classification. This corresponds to that learning instance when the boundary bias becomes positive, and requires one to switch from active to random selection of learning examples. The sign for the boundary bias and the increase in the classification error are two manifestations of the same phenomena, i.e., overtraining. The experimental results presented show the feasibility and usefulness of our novel approach using a non-separable two-class classification problem. Our hybrid learning strategy achieves competitive performance against standard nearest neighbor methods using much fewer training examples.