An Efficient Distributed-Computing Framework for Association-Rule-Based Recommendation

The association rule based recommendation model is one of the most widely used commercial recommendation engines in e-commerce websites. A variety of techniques, mainly including eligible rule selection and multiple rules combination, have been developed to create effective recommendation. Unfortunately, the efficiency of the association rule based recommendation has been paid for little attention. In real-life online shopping sites, the concurrency traffic is usually very high, that is, there are a vast amount of users visiting sites simultaneously and persistently adding commodities into their baskets. In the meanwhile, the volume of frequent patterns are usually very large and thus the number of association rules derived from these patterns is much larger because a pattern is able to generate several rules. As the large amount of the association rules and the concurrent access of users, how to match the browsing histories of users with the large set of rules efficiently in order to offer nearly real-time recommendations for massive online users, has become a vital concern which restricts whether the association rule based recommendation model could be used on the real-life e-commerce websites. To address this problem, this paper focuses on the efficiency of the association rule based recommendation and develops a distributed computing framework for improving the computational performance of rule based recommendation, which seamlessly fuses the association rule mining and the recommendation computing. Firstly, based on the summarization of existing rule-based approaches, a tree-type structure called Ordered Patterns Forest (OPF) is designed for the compact representation and storage of frequent patterns, without missing any basic information that will be useful for the subsequent recommendation such as support of a pattern and its nested patterns. Secondly, we transform the two-step rule based recommendation to a series of operations on the data structure, and then develop the corresponding efficient algorithms for these operations which are responsible for mining eligible patterns as well as computing recommendation scores. Specifically, we transform the candidate rules mining into a path searching problem on the OPF and thus present a path searching algorithm running on the single machine. Finally, the real-time recommendation is impossible to be completed in a single machine. Hence, in order to handle the ever-increasing of online customers and patterns, we devise a distributed computing framework in which a novel load balanced strategy for data partitioning is also proposed to reduce the running time of the task that finishes lastly and thus further improves the overall performance. At last, we implement the proposed framework and algorithms on Spark, a widely used memory-based distributed computing engine, and evaluate the framework and algorithms on three real-world datasets, i.e., Accidents, Webdocs and Amazon. The experimental results demonstrate that the efficiency improved by the proposed OPF with the path searching algorithm is more than six times that of the traditional brute force method. Moreover, the proposed load balanced strategy is effective to further improve the performance of the proposed distributed association rule based recommendation framework, which can achieve nearly linear scalability along with the increase of computational nodes. © 2019, Science Press. All right reserved.