An Energy-Aware Resource Management Strategy Based on Spark and YARN in Heterogeneous Environments

被引:1
|
作者
Shabestari, Fatemeh [1 ]
Navimipour, Nima Jafari [2 ,3 ]
机构
[1] Islamic Azad Univ, Dept Comp Engn, Sofian Branch, Sofian, Iran
[2] Kadir Has Univ, Dept Comp Engn, TR-34083 Istanbul, Turkiye
[3] Natl Yunlin Univ Sci & Technol, Future Technol Res Ctr, Touliu 64002, Taiwan
关键词
Sparks; Yarn; Task analysis; Resource management; Energy efficiency; Energy consumption; Clustering algorithms; Distributed computing; energy management; resource management; scheduling; MAPREDUCE; ALGORITHM; JOBS;
D O I
10.1109/TGCN.2023.3347276
中图分类号
TN [电子技术、通信技术];
学科分类号
0809 ;
摘要
Apache Spark is a popular framework for processing big data. Running Spark on Hadoop YARN allows it to schedule Spark workloads alongside other data-processing frameworks on Hadoop. When an application is deployed in a YARN cluster, its resources are given without considering energy efficiency. Furthermore, there is no way to enforce any user-specified deadline constraints. To address these issues, we propose a new deadline-aware resource management system and a scheduling algorithm to minimize the total energy consumption in Spark on YARN for heterogeneous clusters. First, a deadline-aware energy-efficient model for the considered problem is proposed. Then, using a locality-aware method, executors are assigned to applications. This algorithm sorts the nodes based on the performance per watt (PPW) metric, the number of application data blocks on nodes, and the rack locality. It also offers three ways to choose executors from different machines: greedy, random, and Pareto-based. Finally, the proposed heuristic task scheduler schedules tasks on executors to minimize total energy and tardiness. We evaluated the performance of the suggested algorithm regarding energy efficiency and satisfying the Service Level Agreement (SLA). The results showed that the method outperforms the popular algorithms regarding energy consumption and meeting deadlines.
引用
收藏
页码:635 / 644
页数:10
相关论文
共 50 条
  • [41] Energy-aware scheduling for spark job based on deep reinforcement learning in cloud
    Hongjian Li
    Liang Lu
    Wenhu Shi
    Gangfan Tan
    Hao Luo
    Computing, 2023, 105 : 1717 - 1743
  • [42] Energy-Aware Modeling of Scaled Heterogeneous Systems
    Ami Marowka
    International Journal of Parallel Programming, 2017, 45 : 1026 - 1045
  • [43] Energy-Aware Network Resource Allocation in SDN
    Subbiah, Sankari
    Perumal, Varalakshmi
    PROCEEDINGS OF THE 2016 IEEE INTERNATIONAL CONFERENCE ON WIRELESS COMMUNICATIONS, SIGNAL PROCESSING AND NETWORKING (WISPNET), 2016, : 2071 - 2075
  • [44] Energy-aware resource sharing with mobile devices
    Furthmueller, Jochen
    Waldhorst, Oliver P.
    COMPUTER NETWORKS, 2012, 56 (07) : 1920 - 1934
  • [45] Network-Aware Resource Management Strategy in Cloud Computing Environments
    Abdclaal, Marwa A.
    Ebrahim, Gamal A.
    Anis, Wagdy R.
    PROCEEDINGS OF 2016 11TH INTERNATIONAL CONFERENCE ON COMPUTER ENGINEERING & SYSTEMS (ICCES), 2016, : 26 - 31
  • [46] JEERP: Energy-Aware Enterprise Resource Planning
    Bonino, Dario
    De Russis, Luigi
    Corno, Fulvio
    Ferrero, Gianni
    IT PROFESSIONAL, 2014, 16 (04) : 50 - 56
  • [47] Reliable Energy-Aware Routing Protocol for Heterogeneous WSN Based on Beaconing
    Li Ya
    Wang Pengjun
    Luo Rong
    Yang Huazhong
    Liu Wei
    2014 16TH INTERNATIONAL CONFERENCE ON ADVANCED COMMUNICATION TECHNOLOGY (ICACT), 2014, : 109 - 140
  • [48] Energy-Aware Task Scheduling on Heterogeneous NoC-based MPSoCs
    Abd Ishak, Suhaimi
    Wu, Hui
    Tariq, Umair Ullah
    2017 IEEE 35TH INTERNATIONAL CONFERENCE ON COMPUTER DESIGN (ICCD), 2017, : 165 - 176
  • [49] Energy-Aware Techniques and Location-Based Methodologies in Mobile Environments
    Falcone, Deborah
    Talia, Domenico
    2014 INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING & SIMULATION (HPCS), 2014, : 991 - 994
  • [50] Scalability Evaluation of an Energy-Aware Resource Management System for Clusters of Web Servers
    Kiertscher, Simon
    Schnor, Bettina
    PROCEEDINGS OF THE 2015 INTERNATIONAL SYMPOSIUM ON PERFORMANCE EVALUATION OF COMPUTER AND TELECOMMUNICATION SYSTEMS (SPECTS), 2015,