A novel semi-supervised approach for feature extraction

Feature extraction is an essential preprocessing step in machine learning and data mining. Generally, supervised feature extraction algorithms with prior knowledge outperform unsupervised ones without prior knowledge. In particular, nearly all existing supervised feature extraction algorithms employ class labels or pairwise constraints as supervised information. In this paper, we propose to employ another form of supervised information, i.e. Universum, which demonstrates a collection of "non-examples" that do not belong to either class/cluster of interest, but belong to the same domain as the problem at hand. Universum data samples can be obtained easily and are more practical and inexpensive than class labels. We address this topic in feature extraction research and propose a novel semi-supervised approach for feature extraction based on Universum. Experiments are carried out to compare the proposed algorithms with well-known unsupervised and supervised feature extraction algorithms on several UCI data sets. The results show that, with very few Universum data, the proposed algorithms are superior to unsupervised algorithms, and achieve similar or even higher performance than LDA with full class labels on the whole training data.