Distributed Computation of Pure High-Order Word Dependence

Many data mining methods have been proposed to obtain useful word associations in text documents. However, it is of great challenge to efficiently discover "pure'' high-order (n > 3) word association patterns, especially in the rapidly expanding data collection. Here, by "pure,'' we mean that those words form an inseparable semantic entity, i.e., the high-order dependence that cannot be reduced to the random coincidence of low-order dependence. Our aim is to find pure high-order word associations in large data sets. This paper proposes a distributed pure dependence mining (DPDM) algorithm based on information geometry, which can efficiently mine the pure dependence between words. We also construct a distributed pure dependence mining framework (DPDMF) to mine pure high-order word associations from data sets. Extensive experiments show that DPDM algorithm can significantly improve the efficiency when mining the pure high-order patterns from large data sets. Finally, we apply the extracted high-order word association patterns in the text classification tasks, which will also achieve significant improvement.