Implementation of a Self-Adaptive Real Time Recommendation System using Spark Machine Learning Libraries

被引:0
|
作者
Sunny, Bobin K. [1 ]
Janardhanan, P. S. [2 ]
Francis, Anu Bonia [1 ]
Murali, Reena [1 ]
机构
[1] Rajiv Gandhi Inst Technol, Dept Comp Sci & Engn, Kottayam, Kerala, India
[2] SunTec Business Solut Ltd, Platform & Technol, Technopk, Thiruvananthapuram, Kerala, India
关键词
Spark Streaming; Real-Time Recommendation Engine; Machine Learning;
D O I
暂无
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
Real time recommendation systems have become an essential component of e-commerce web applications. With increasing volume and velocity of data handled by these applications, known as the bigdata problem, traditional recommendation systems that analyze data and update models at regular time intervals would not be able to satisfy this requirement. With the evolution of technologies for processing bigdata in real time, it has become fairly easy to implement real time recommendation systems. Stream-computing is a new computing paradigm for handling the velocity attribute of bigdata which makes it possible to develop real time bigdata applications. This paper gives the details of implementation of a real time recommendation system using Apache Spark, a widely used platform for stream computing. This system is implemented for recommending TV channels to viewers in real time. This becomes a challenging task due to continuous changes in the set of available channels and the context dependent preference of viewers. In channel recommendation scenario, characterized by its dynamic nature, volume of data, and tight time constraints, traditional approaches cannot be used. We have implemented a highly scalable TV channel recommendation system optimized for the processing of real-time data streams originating from set-top boxes. The proposed system implements a self-adaptive approach for model building. The system effectively uses distributed processing power of Apache Spark to make recommendations in real time with scalability to meet the real time constraints with increasing load. The Spark Machine Learning Libraries (Spark MLLib) provide several algorithms which were used for developing the proposed recommendation system. The large amount of data in the system is efficiently managed by the data processing method of Lambda Architecture.
引用
收藏
页数:7
相关论文
共 50 条
  • [1] Implementation of self-adaptive system using the algorithm of neural network learning gain
    Lee, Seong-Su
    Kim, Yong-Wook
    Oh, Hun
    Park, Wal-Seo
    [J]. INTERNATIONAL JOURNAL OF CONTROL AUTOMATION AND SYSTEMS, 2008, 6 (03) : 453 - 459
  • [2] Self-adaptive extreme learning machine
    Gai-Ge Wang
    Mei Lu
    Yong-Quan Dong
    Xiang-Jun Zhao
    [J]. Neural Computing and Applications, 2016, 27 : 291 - 303
  • [3] Self-adaptive extreme learning machine
    Wang, Gai-Ge
    Lu, Mei
    Dong, Yong-Quan
    Zhao, Xiang-Jun
    [J]. NEURAL COMPUTING & APPLICATIONS, 2016, 27 (02): : 291 - 303
  • [4] Self-Adaptive Evolutionary Extreme Learning Machine
    Jiuwen Cao
    Zhiping Lin
    Guang-Bin Huang
    [J]. Neural Processing Letters, 2012, 36 : 285 - 305
  • [5] Self-Adaptive Evolutionary Extreme Learning Machine
    Cao, Jiuwen
    Lin, Zhiping
    Huang, Guang-Bin
    [J]. NEURAL PROCESSING LETTERS, 2012, 36 (03) : 285 - 305
  • [6] Federated Machine Learning as a Self-Adaptive Problem
    Baresi, Luciano
    Quattrocchi, Giovanni
    Rasi, Nicholas
    [J]. 2021 INTERNATIONAL SYMPOSIUM ON SOFTWARE ENGINEERING FOR ADAPTIVE AND SELF-MANAGING SYSTEMS (SEAMS 2021), 2021, : 41 - 47
  • [7] A Self-Adaptive Cuckoo Search Algorithm Using a Machine Learning Technique
    Caselli, Nicolas
    Soto, Ricardo
    Crawford, Broderick
    Valdivia, Sergio
    Olivares, Rodrigo
    [J]. MATHEMATICS, 2021, 9 (16)
  • [8] A Real-Time Self-adaptive Enhancement Method of Infrared System
    Zhang, Bao-Hui
    Xia, Peng-Hao
    Zhang, Jun-ju
    Chang, Ben-kang
    Qian, Yun-sheng
    [J]. COMPUTER SCIENCE FOR ENVIRONMENTAL ENGINEERING AND ECOINFORMATICS, PT 1, 2011, 158 : 111 - 117
  • [9] A Self-adaptive and Real-time Panoramic Video Mosaicing System
    Zeng, Lin
    Deng, Dexiang
    Chen, Xi
    Zhang, Yunlu
    [J]. JOURNAL OF COMPUTERS, 2012, 7 (01) : 218 - 225
  • [10] A self-adaptive and real-time panoramic video mosaicing system
    School of electronic information, Wuhan University, Wuhan, China
    不详
    不详
    [J]. J. Comput, 1 (218-225):