Efficient Real-time Earliest Deadline First based scheduling for Apache Spark

被引：3
|
作者：
Neciu, Laurentiu-Florin [1 ]
Pop, Florin [1 ,2 ]
Apostol, Elena-Simona [1 ]
Truica, Ciprian-Octavian [1 ]
机构：
[1] Univ Politehn Bucuresti, Bucharest, Romania
[2] Natl Inst Res & Dev Informat ICI Bucharest, Bucharest, Romania
来源：
2021 20TH INTERNATIONAL SYMPOSIUM ON PARALLEL AND DISTRIBUTED COMPUTING (ISPDC) | 2021年
关键词：
Real-time Scheduling; Apache Spark; Real-Time Earliest Deadline First; Resilient Distributed Dataset; Directed Acyclic Graph;
D O I：
10.1109/ISPDC52870.2021.9521640
中图分类号：
TP3 [计算技术、计算机技术];
学科分类号：
0812 ;
摘要：
Apache Spark is a distributed computing framework for fast in-memory data analysis, Machine Learning jobs, and SQL queries that employs Resilient Distributed Dataset for distributing data and Directed Acyclic Graph for scheduling computations. Currently, Spark provides two scheduling policies for tasks pending execution: a First In First Out policy and a FAIR policy, providing no support for deadline-based real-time scheduling. In this paper, we present a new system designed to accept deadlines for heterogeneous Spark Jobs and perform a real-time scheduling policy based on Earliest Deadline First (EDF). To showcase the efficiency of our scheduling policy, we compare and analyze the performance of our solution with the current Spark execution policies in terms of job lateness. We empirically prove that our real-time policy provides much lower lateness given suitable
引用
收藏
页码：97 / 104
页数：8
相关论文
共 50 条
  • [1] Earliest Deadline First Scheduling for Real-Time Computing in Sustainable Sensors
    Chetto, Maryline
    El Osta, Rola
    [J]. SUSTAINABILITY, 2023, 15 (05)
  • [2] Energy efficient fault-tolerant earliest deadline first scheduling for hard real-time systems
    Han, Qiushi
    Niu, Linwei
    Quan, Gang
    Ren, Shaolei
    Ren, Shangping
    [J]. REAL-TIME SYSTEMS, 2014, 50 (5-6) : 592 - 619
  • [3] Optimising earliest deadline first scheduling for parallel real-time tasks on multiprocessors
    Kim, C.
    Cho, H.
    Sung, Y.
    [J]. ELECTRONICS LETTERS, 2019, 55 (25) : 1343 - +
  • [4] Energy efficient fault-tolerant earliest deadline first scheduling for hard real-time systems
    Qiushi Han
    Linwei Niu
    Gang Quan
    Shaolei Ren
    Shangping Ren
    [J]. Real-Time Systems, 2014, 50 : 592 - 619
  • [5] Hybrid earliest deadline first/preemption threshold scheduling for real-time systems
    He, DZ
    Wang, FY
    Li, W
    Zhang, XW
    [J]. PROCEEDINGS OF THE 2004 INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND CYBERNETICS, VOLS 1-7, 2004, : 433 - 438
  • [6] An Algorithm to Reduce the Time Complexity of Earliest Deadline First Scheduling Algorithm in Real-Time System
    Singh, Jagbeer
    Patra, Bichitrananda
    Singh, Satyendra Prasad
    [J]. INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2011, 2 (02) : 31 - 37
  • [7] Earliest Deadline First scheduling with active buffer management for real-time traffic in the Internet
    Hei, XJ
    Tsang, DHK
    [J]. TELECOMMUNICATION SYSTEMS, 2002, 19 (3-4) : 349 - 359
  • [8] Statistical admission control for real-time services under earliest deadline first scheduling
    Quan, Z
    Chung, JM
    [J]. COMPUTER NETWORKS, 2005, 48 (02) : 137 - 154
  • [9] The Earliest Deadline First scheduling with active buffer management for real-time traffic in the Internet
    Hei, XJ
    Tsang, DHK
    [J]. NETWORKING - ICN 2001, PT I, PROCEEDINGS, 2001, 2093 : 45 - 54
  • [10] Earliest Deadline First Scheduling with Active Buffer Management for Real-Time Traffic in the Internet
    Xiaojun Hei
    Danny H.K. Tsang
    [J]. Telecommunication Systems, 2002, 19 : 349 - 359
12345