Semi-supervised learning via constraints

Given partially labeled data consisting of labeled data (y(i), x(i))'s and unlabeled data x(j)'s, semi-supervised learning aims to use unlabeled data, in addition to labeled data, to improve over supervised learning that uses only labeled data. Because most classifications are inherently based on the conditional distribution f(Y vertical bar X) while unlabeled data only contain information on marginal distribution f(X), some modeling assumptions are necessary to connect the two distributions. Here we assume that some characteristics of the marginal distribution of Y, f(Y), are known, and hence are used as extra constraints on the original objective function of supervised learning. As a prototype implementation, for a binary problem with Y = 1 or -1, we assume that the marginal class probability po = Pr(Y = 1) is known, and incorporate it as an additional penalty term into penalized logistic regression. We point out that our proposal is closely related to the approach of combining parametric and empirical likelihoods. We use simulated data and real data to demonstrate that the proposed method can have improved performance over penalized logistic regression without using unlabeled data. The idea can be extended to kernel logistic regression, SVM and even other semi-supervised learning algorithms.