An improved parallel FP-growth algorithm based on Spark and its application

被引：0
|
作者：
Miao, Yuhang [1 ]
Lin, Jinxing [1 ]
Xu, Nuo [1 ]
机构：
[1] Nanjing Univ Posts & Telecommun, Coll Automat, Nanjing 210023, Peoples R China
来源：
PROCEEDINGS OF THE 38TH CHINESE CONTROL CONFERENCE (CCC) | 2019年
关键词：
Frequent itemset mining; Big data; Parallel FP-growth; Spark; Steam Turbine;
D O I：
10.23919/chicc.2019.8866373
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
Frequent itemset mining (FIM) is an important means for data analysis. With the increase of data size, single machine FIM algorithm has the problems of long time-consuming and high memory consumption. Parallel computing of mining algorithm on distributed machine can break through the performance bottleneck of single machine algorithm. In this paper, an improved parallel FP-growth algorithm based on Spark is presented. Firstly, the FP-growth algorithm is improved by matrix technology, compress data set into an information matrix can reduce memory consumption. Then, the improved FP-growth algorithm is parallelized on Spark. Finally, the proposed algorithm is applied to the performance optimization of steam turbine in thermal power plants. The result shows that the proposed algorithm is more efficient than the existing parallel FP-growth algorithm.
引用
收藏
页码：3793 / 3797
页数：5
相关论文
共 50 条
  • [41] A Parallel FP-Growth Mining Algorithm with Load Balancing Constraints for Traffic Crash Data
    Yang, Yang
    Tian, Na
    Wang, Yunpeng
    Yuan, Zhenzhou
    [J]. INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL, 2022, 17 (04)
  • [42] An expert recommendation algorithm based on Pearson correlation coefficient and FP-growth
    Wanli Feng
    Quanyin Zhu
    Jun Zhuang
    Shimin Yu
    [J]. Cluster Computing, 2019, 22 : 7401 - 7412
  • [43] Discovery of Incremental Association Rules Based on a New FP-Growth Algorithm
    Kreesuradej, Worapoj
    Thurachon, Wannasiri
    [J]. 2019 IEEE 4TH INTERNATIONAL CONFERENCE ON COMPUTER AND COMMUNICATION SYSTEMS (ICCCS 2019), 2019, : 184 - 188
  • [44] Research on Association Rule Algorithm Based on Distributed and Weighted FP-Growth
    Wang, Huaibin
    Liu, Yuanchao
    Wang, Chundong
    [J]. ADVANCES IN MULTIMEDIA, SOFTWARE ENGINEERING AND COMPUTING, VOL 1, 2011, 128 : 133 - 138
  • [45] Parallel Algorithm of Improved FunkSVD Based on Spark
    Yue, Xiaochen
    Liu, Qicheng
    [J]. KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS, 2021, 15 (05): : 1649 - 1665
  • [46] Analysis of TCM prescription rule of stroke based on FP-growth algorithm
    Wang, Yan
    Qi, Hao
    Huang, Zhengzheng
    [J]. 2020 IEEE INTERNATIONAL CONFERENCE ON BIOINFORMATICS AND BIOMEDICINE, 2020, : 3008 - 3010
  • [47] An expert recommendation algorithm based on Pearson correlation coefficient and FP-growth
    Feng, Wanli
    Zhu, Quanyin
    Zhuang, Jun
    Yu, Shimin
    [J]. CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, 2019, 22 (Suppl 3): : S7401 - S7412
  • [48] An optimized fuzzy based FP-growth algorithm for mining temporal data
    Kumar, B. Praveen
    Padmavathy, T.
    Muthunagai, S. U.
    Paulraj, D.
    [J]. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS, 2024, 46 (01) : 41 - 51
  • [49] Applying a pattern length constraint on the FP-Growth algorithm
    Gyoeroedi, Cornelia
    Gyoeroedi, Robert
    Dersidan, Mihai
    Bandici, Livia
    [J]. SOFA 2009: 3RD INTERNATIONAL WORKSHOP ON SOFT COMPUTING APPLICATIONS, PROCEEDINGS, 2009, : 181 - 184
  • [50] AN OPTIMISED FP-GROWTH ALGORITHM USING MAPREDUCE PARADIGMS
    Wu, Xiuguo
    [J]. JOURNAL OF NONLINEAR AND CONVEX ANALYSIS, 2021, 22 (10) : 2127 - 2137
12345