Scalable Maximal Discernibility Discretization for Big Data

Discretization of numerical (continuous) attributes is one of the most important data preprocessing tasks in knowledge discovery and data mining. Some of data mining techniques require discretized data. The article aim is to demonstrate that discretization methods based on the discernibility measure to evaluate cuts can be parallelized in Big Data platform Apache Spark. We thus propose a distributed implementation of one of the most well-known discretizers based on rough set methodology. The experimental results in terms of scalability, speedup and sizeup are quite promising.