Albatross: An Efficient Cloud-Enabled Task Scheduling and Execution Framework Using Distributed Message Queues

被引:0
|
作者
Sadooghi, Iman [1 ]
Kumar, Geet [1 ]
Wang, Ke [1 ]
Zhao, Dongfang [1 ]
Li, Tonglin [1 ]
Raicu, Ioan [1 ]
机构
[1] IIT, Dept Comp Sci, Chicago, IL 60616 USA
基金
美国国家科学基金会;
关键词
Data Analytics; Task Scheduling; Distributed Systems; Spark; Hadoop; Distributed Task Execution; Distributed Message Queue;
D O I
暂无
中图分类号
TP39 [计算机的应用];
学科分类号
081203 ; 0835 ;
摘要
Data Analytics has become very popular on large datasets in different organizations. It is inevitable to use distributed resources such as Clouds for Data Analytics and other types of data processing at larger scales. To effectively utilize all system resources, an efficient scheduler is needed, but the traditional resource managers and job schedulers are centralized and designed for larger batch jobs which are fewer in number. Frameworks such as Hadoop and Spark, which are mainly designed for Big Data analytics, have been able to allow for more diversity in job types to some extent. However, even these systems have centralized architectures and will not be able to perform well on large scales and under heavy task loads. Modern applications generate tasks at very high rates that can cause significant slowdowns on these frameworks. Additionally, over-decomposition has shown to be very useful in increasing the system utilization. In order to achieve high efficiency, scalability, and better system utilization, it is critical for a modern scheduler to be able to handle over-decomposition and run highly granular tasks. Further, to achieve high performance, Albatross is written in C/C++, which imposes a minimal overhead to the workload process as compared to languages like Java or Python. We propose Albatross, a task level scheduling and execution framework that uses a Distributed Message Queue (DMQ) for task distribution among its workers. Unlike most scheduling systems, Albatross uses a pulling approach as opposed to the common push approach. The former would let Albatross achieve a good load balancing and scalability. Furthermore, the framework has built in support for task execution dependency on workflows. Therefore, Albatross is able to run various types of workloads, including Data Analytics and HPC applications. Finally, Albatross provides data locality support. This allows the framework to achieve higher performance through minimizing the amount of unnecessary data movement on the network. Our evaluations show that Albatross outperforms Spark and Hadoop at larger scales and in the case of running higher granularity workloads.
引用
收藏
页码:11 / 20
页数:10
相关论文
共 50 条
  • [41] Energy-efficient task scheduling model based on MapReduce for cloud computing using genetic algorithm
    Wang, Xiaoli
    Wang, Yuping
    Zhu, Hai
    [J]. JOURNAL OF COMPUTERS, 2012, 7 (12) : 2962 - 2970
  • [42] ECOS: An efficient task-clustering based cost-effective aware scheduling algorithm for scientific workflows execution on heterogeneous cloud systems
    Dong, Minggang
    Fan, Lili
    Jing, Chao
    [J]. JOURNAL OF SYSTEMS AND SOFTWARE, 2019, 158
  • [43] Efficient multi-attribute precedence-based task scheduling for edge computing in geo-distributed cloud environment
    Chunlin Li
    Chaokun Zhang
    Bingbin Ma
    Youlong Luo
    [J]. Knowledge and Information Systems, 2022, 64 : 175 - 205
  • [44] Efficient multi-attribute precedence-based task scheduling for edge computing in geo-distributed cloud environment
    Li, Chunlin
    Zhang, Chaokun
    Ma, Bingbin
    Luo, Youlong
    [J]. KNOWLEDGE AND INFORMATION SYSTEMS, 2022, 64 (01) : 175 - 205
  • [45] Energy efficient task-pull scheduling using Equal Length Cellular Automata in Distributed Computing
    Mitra, Arnab
    Kundu, Anirban
    Chattopadhyay, Matangini
    [J]. 2014 FOURTH INTERNATIONAL CONFERENCE OF EMERGING APPLICATIONS OF INFORMATION TECHNOLOGY (EAIT), 2014, : 40 - 45
  • [46] QoS enhanced distributed load balancing and task scheduling framework for wireless networks using hybrid optimisation algorithm
    Rajguru, Abhijit A.
    Apte, S. S.
    [J]. INTERNATIONAL JOURNAL OF COMMUNICATION NETWORKS AND DISTRIBUTED SYSTEMS, 2018, 21 (02) : 241 - 265
  • [47] Delay-Aware and Energy-Efficient IoT Task Scheduling Algorithm With Double Blockchain Enabled in Cloud-Fog Collaborative Networks
    Cao, Shaohua
    Zhan, Zijun
    Dai, Congcong
    Chen, Shu
    Zhang, Weishan
    Han, Zhu
    [J]. IEEE INTERNET OF THINGS JOURNAL, 2024, 11 (02): : 3003 - 3016
  • [48] A Joint Resource Allocation, Security with Efficient Task Scheduling in Cloud Computing Using Hybrid Machine Learning Techniques
    Bal, Prasanta Kumar
    Mohapatra, Sudhir Kumar
    Das, Tapan Kumar
    Srinivasan, Kathiravan
    Hu, Yuh-Chung
    [J]. SENSORS, 2022, 22 (03)
  • [49] JHTD: An Efficient Joint Scheduling Framework Based on Hypergraph for Task Placement and Data Transfer Across Geographically Distributed Data Centers
    Jing, Chao
    Dan, Penggao
    [J]. IEEE ACCESS, 2022, 10 : 116302 - 116316
  • [50] Energy-Efficient Task Scheduling for DVFS-enabled Heterogeneous Computing Systems using a Linear Programming Approach
    Zhang, Yujian
    Wang, Yun
    Wang, Hui
    [J]. 2016 IEEE 35TH INTERNATIONAL PERFORMANCE COMPUTING AND COMMUNICATIONS CONFERENCE (IPCCC), 2016,