An improved parallel FP-growth algorithm based on Spark and its application

Frequent itemset mining (FIM) is an important means for data analysis. With the increase of data size, single machine FIM algorithm has the problems of long time-consuming and high memory consumption. Parallel computing of mining algorithm on distributed machine can break through the performance bottleneck of single machine algorithm. In this paper, an improved parallel FP-growth algorithm based on Spark is presented. Firstly, the FP-growth algorithm is improved by matrix technology, compress data set into an information matrix can reduce memory consumption. Then, the improved FP-growth algorithm is parallelized on Spark. Finally, the proposed algorithm is applied to the performance optimization of steam turbine in thermal power plants. The result shows that the proposed algorithm is more efficient than the existing parallel FP-growth algorithm.