Mutual Information Estimation with Random Forests

We present a new method for estimating mutual information based on the random forests classifiers. This method uses random permutation of one of the two variables to create data where the two variables are independent. We show that mutual information can be estimated by the class probabilities of a probabilistic classifier trained on the independent against the dependent data. This method has the robustness and flexibility that random forests offers as well as the possibility to use mixtures of continuous and discrete data, unlike most other approaches for estimating mutual information. We tested our method on a variety of data and found it to be accurate with medium or large datasets yet inaccurate with smaller datasets. On the positive side, our method is capable to estimate the mutual information between sets of both continuous and discrete variables and appears to be relatively insensitive to the addition of noise variables.