An Efficient Approach for Mining Association Rules from Sparse and Dense Databases

Association rule mining (ARM) is an important task in data mining. This task is computationally intensive and requires large memory usage. Many existing methods for ARM perform efficiently on either sparse or dense data but not both. We address this issue by presenting a new approach for ARM that runs fast for both sparse and dense databases by detecting the characteristic of data subsets in database and applying a combination of two mining strategies: one is for the sparse data subsets and the other is for the dense ones. Two algorithms, FEM and DFEM, based on our approach are introduced in this paper. FEM applies a fixed threshold as the condition for switching between the two mining strategies while DFEM adopts this threshold dynamically at runtime to best fit the characteristics of the database during the mining process, especially when minimum support threshold is low. Additionally, we present optimization techniques for the proposed algorithms to speed up the mining process, reduce the memory usage and optimize the UO cost. We also analyze in-depth the performance of FEM and DFEM and compare them with several existing algorithms. The experimental results show that FEM and DFEM achieve a significant improvement in execution time and consume less memory than many popular ARM algorithms including the well-known Apriori, FP-growth and Eclat on both sparse and dense databases.