MapReduce based computation of the diffusion method in recommender systems

The performance of existing diffusion-based algorithms in recommender systems is still limited by the processing ability of a single computer. In order to conduct the diffusion computation on large data sets, a parallel implementation of the classic diffusion method on the MapReduce framework is proposed. At first, the diffusion computation is transformed from a summation format to a cascade matrix multiplication format, and then, a parallel matrix multiplication algorithm based on dynamic vector is proposed to reduce the CPU and I/O cost on the MapReduce framework, which can also be applied to other parallel matrix multiplication scenarios. Then, block partitioning is used to further improve the performance, while the order of matrix multiplication is also taken into consideration. Experiments on different kinds of data sets have verified the efficiency of the proposed method. Copyright © by HIGH TECHNOLOGY LETTERS PRESS.