Efficient Data and Task Co-Scheduling for Scientific Workflow in Geo-distributed Datacenters

被引：5
|
作者：
Chen, Jian [1 ]
Zhang, Jinghui [1 ]
Song, Aibo [1 ]
机构：
[1] Southeast Univ, Sch Comp Sci & Engn, Nanjing 211189, Jiangsu, Peoples R China
来源：
2017 FIFTH INTERNATIONAL CONFERENCE ON ADVANCED CLOUD AND BIG DATA (CBD) | 2017年
基金：
中国国家自然科学基金;
关键词：
datacenter; scientific workflow scheduling; graph partition; linear programming; DEDICATED HETEROGENEOUS MULTICLUSTER; STRATEGY;
D O I：
10.1109/CBD.2017.19
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
Scientific workflow usually needs to be performed in multiple collaborative datacenters for the requirement of accessing community-wide resources. However, the movements of initial input data and intermediate data across geo-distributed datacenters would hinder efficient execution of large-scale dataintensive scientific workflows. In this paper, a novel scheduling approach based on graph partition is proposed for the execution of data-intensive scientific workflow in geo-distributed datacenters, aiming at the optimization of the overall data transfer cost. Simulations show that our algorithm significantly reduces the overall geo-distributed data transfer and demonstrate its effectiveness.
引用
收藏
页码：63 / 68
页数：6
相关论文
共 50 条
  • [21] Scheduling Jobs across Geo-Distributed Datacenters with Max-Min Fairness
    Chen, Li
    Liu, Shuhao
    Li, Baochun
    Li, Bo
    [J]. IEEE INFOCOM 2017 - IEEE CONFERENCE ON COMPUTER COMMUNICATIONS, 2017,
  • [22] Leveraging Endpoint Flexibility When Scheduling Coflows across Geo-distributed Datacenters
    Li, Wenxin
    Yuan, Xu
    Li, Keqiu
    Qi, Heng
    Zhou, Xiaobo
    [J]. IEEE CONFERENCE ON COMPUTER COMMUNICATIONS (IEEE INFOCOM 2018), 2018, : 873 - 881
  • [23] Cost Optimization for Time-Bounded Request Scheduling in Geo-Distributed Datacenters
    Wei, Xiaohui
    Li, Lanxin
    Wang, Xingwang
    Liu, Yuanyuan
    [J]. CLOUD COMPUTING AND SECURITY, PT II, 2017, 10603 : 601 - 610
  • [24] MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale
    Choudhury, Arnab
    Wang, Yang
    Pelkonen, Tuomas
    Srinivasan, Kutta
    Jain, Abha
    Lin, Shenghao
    David, Delia
    Soleimanifard, Siavash
    Chen, Michael
    Yadav, Abhishek
    Tijoriwala, Ritesh
    Samoylov, Denis
    Tang, Chunqiang
    [J]. PROCEEDINGS OF THE 18TH USENIX SYMPOSIUM ON OPERATING SYSTEMS DESIGN AND IMPLEMENTATION, OSDI 2024, 2024, : 563 - 580
  • [25] Efficient Geo-Distributed Data Processing with Rout
    Jayalath, Chamikara
    Eugster, Patrick
    [J]. 2013 IEEE 33RD INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS (ICDCS), 2013, : 470 - 480
  • [26] Electricity and Carbon-aware Task Scheduling in Geo-distributed Internet Data Centers
    Wang, Peng
    Liu, Wenyu
    Cheng, Ming
    Ding, Zhaohao
    Wang, Yi
    [J]. 2022 IEEE/IAS INDUSTRIAL AND COMMERCIAL POWER SYSTEM ASIA (I&CPS ASIA 2022), 2022, : 1416 - 1421
  • [27] Calantha: Content Distribution across Geo-Distributed Datacenters
    Li, Yangyang
    Zhang, Linchao
    Jia, Yue
    Liao, Yong
    Xie, Haiyong
    [J]. 2017 IEEE CONFERENCE ON COMPUTER COMMUNICATIONS WORKSHOPS (INFOCOM WKSHPS), 2017, : 724 - 729
  • [28] Cost-Aware Big Data Processing Across Geo-Distributed Datacenters
    Xiao, Wenhua
    Bao, Weidong
    Zhu, Xiaomin
    Liu, Ling
    [J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2017, 28 (11) : 3114 - 3127
  • [29] A Framework of Hypergraph-Based Data Placement Among Geo-Distributed Datacenters
    Yu, Boyang
    Pan, Jianping
    [J]. IEEE TRANSACTIONS ON SERVICES COMPUTING, 2020, 13 (03) : 395 - 409
  • [30] Optimizing Network Transfers for Data Analytic Jobs Across Geo-Distributed Datacenters
    Chen, Li
    Liu, Shuhao
    Li, Baochun
    [J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2022, 33 (02) : 403 - 414
12345