A general framework for boosting feature subset selection algorithms

Feature selection is one of the most important tasks in many machine learning and data mining problems. Due to the increasing size of the problems, removing useless, erroneous or noisy features is frequently an initial step that is performed before other data mining algorithms are applied. The aim is to reproduce, or even improve, the performance of the data mining algorithm when all the features are used. Furthermore, the selection of the most relevant features may offer the expert valuable information about the problem to be solved. Over the past few decades, many different feature selection algorithms have been proposed, each with its own strengths and weaknesses. However, as in the case of classification, it is unlikely that a single feature selection algorithm would be able to achieve good results across many different datasets and application fields. Furthermore, when we are dealing with thousands of features, the most powerful feature selection methods are frequently too time consuming to be applied. In classification, one of the most successful ways of consistently improving the performance of a single weak learner is to construct ensembles using boosting methods. In this paper, we propose a general framework for feature selection boosting in the same way boosting is applied to classification. The proposed approach opens a new field of research in which to apply the many techniques developed for boosting classifiers. Using 120 datasets, the experiments reported show a clear improvement in several state-of-the-art feature selection algorithms using the proposed methodology.