Data modeling in machine learning based on information-theoretic measures

Data modeling is a key problem in machine learning. In conventional machine learning, a lot of research has been focused on a specific method for a specific environment in which models are selected and built generally by using ad hoc methods, "trial and error" or solely on "expert" knowledge or intuition. As a result, the effectiveness of the models is limited and the research results often do not contribute to the fundamental understanding of the field nor lend themselves to the broader problem domain. This paper aims to provide theoretical foundations as well as useful tools to guide model building and to explain and evaluate model performance by using several information-theoretic measures, namely, entropy, conditional entropy, relative entropy, information gain, and information cost. These measures can characterize the regularity of data set and thus contribute to the data modeling.