Semi-sparse algorithm based on multi-layer optimization for recommender system

Similarity among vectors is basic knowledge required to carry out recommendation and classification in recommender systems, which support personalized recommendation during online interactions. In this paper, we propose a Semi-sparse Algorithm based on Multi-layer Optimization to speed up the Pearson Correlation Coefficient, which is conventionally used in obtaining similarity among sparse vectors. In accelerating the batch of similarity-comparisons within one thread, the semi-sparse algorithm spares out over-reduplicated accesses and judgements on the selected sparse vector by making this vector dense locally. Moreover, a reduce-vector is proposed to restrict using locks on critical resources in the thread-pool, which is wrapped with Pthreads on a multi-core node to improve parallelism. Furthermore, among processes in our framework, a shared zip file is read to cut down messages within the Message Passing Interface package. Evaluation shows that the optimized multi-layer framework achieves a brilliant speedup on three benchmarks, Netflix, MovieLens and MovieLen1600.