Resource Scheduling and Data Locality for Virtualized Hadoop on IaaS Cloud Platform

被引:3
|
作者
Tao, Dan [1 ]
Wang, Bingxu [1 ]
Lin, Zhaowen [2 ,3 ,4 ]
Wu, Tin-Yu [5 ]
机构
[1] Beijing Jiaotong Univ, Sch Elect & Informat Engn, Beijing 100044, Peoples R China
[2] Beijing Univ Posts & Telecommun, Network & Informat Ctr, Inst Network Technol, Beijing 100876, Peoples R China
[3] Beijing Univ Posts & Telecommun, Sci & Technol Informat Transmiss & Disseminat Com, Beijing 100876, Peoples R China
[4] Beijing Univ Posts & Telecommun, Natl Engn Lab Mobile Network Secur 2013 2685, Beijing 100876, Peoples R China
[5] Natl Ilan Univ, Dept Comp Sci & Informat Engn, Yilan 26041, Taiwan
关键词
Hadoop; Resource scheduling; Data locality; IaaS; MAPREDUCE;
D O I
10.1007/978-3-319-42553-5_28
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
With cloud computing technology becoming more mature, it is urgent to combine big data processing tool Hadoop with IaaS cloud platform. In this paper, we firstly propose a new Dynamic Hadoop Cluster on IaaS (DHCI) architecture, which includes four key modules: monitoring module, scheduling module, virtual machine management module and virtual machine migration module. The load of both physical hosts and virtual machines are collected by the monitoring module, and can be used for designing resource scheduling and data locality solutions. Secondly, we present a load feedback based resource scheduling scheme. The resource allocation can be avoided on overburdened physical hosts or the strong scalability of virtualized cluster can be achieved by fluctuating the amount of virtual machines (VMs). Thirdly, we reuse the method of VM migration and propose a dynamic migration based data locality scheme. We migrate computation nodes to different host(s) or rack(s) where the corresponding storage nodes are deployed to satisfy the requirement of data locality. We evaluate our solutions in a realistic scenario based on Openstack. Massive experimental results demonstrate the effectiveness of our solutions that contribute to balance workload and performance improvement, even under heavy-loaded cloud system conditions.
引用
收藏
页码:332 / 341
页数:10
相关论文
共 50 条
  • [1] A Load Feedback based Resource Scheduling Algorithm for IaaS Cloud Platform
    Wang, Bingxu
    Tao, Dan
    Lin, Zhaowen
    [J]. 2016 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS-TAIWAN (ICCE-TW), 2016, : 133 - 134
  • [2] A virtual machine based task scheduling approach to improving data locality for virtualized Hadoop
    Sun, Ruiqi
    Yang, Jie
    Gao, Zhan
    He, Zhiqiang
    [J]. 2014 IEEE/ACIS 13TH INTERNATIONAL CONFERENCE ON COMPUTER AND INFORMATION SCIENCE (ICIS), 2014, : 291 - 296
  • [3] Moving average fuzzy resource scheduling for virtualized cloud data services
    Priya, V
    Babu, C. Nelson Kennedy
    [J]. COMPUTER STANDARDS & INTERFACES, 2017, 50 : 251 - 257
  • [4] Virtualized Resource Scheduling in Cloud Computing Environments: An Review
    Lin, Jianpeng
    Cui, Delong
    Peng, Zhiping
    Li, Qirui
    He, Jieguang
    Guo, Mian
    [J]. 2020 IEEE CONFERENCE ON TELECOMMUNICATIONS, OPTICS AND COMPUTER SCIENCE (TOCS), 2020, : 310 - 315
  • [5] Shareability and locality aware scheduling algorithm in Hadoop for mobile cloud computing
    Wei, Hsin-Wen
    Wu, Tin-Yu
    Lee, Wei-Tsong
    Hsu, Che-Wei
    [J]. Journal of Information Hiding and Multimedia Signal Processing, 2015, 6 (06): : 1215 - 1230
  • [6] An improved task scheduling algorithm based on cache locality and data locality in Hadoop
    Zhang, Peng
    Li, Chunlin
    Zhao, Yahui
    [J]. 2016 17TH INTERNATIONAL CONFERENCE ON PARALLEL AND DISTRIBUTED COMPUTING, APPLICATIONS AND TECHNOLOGIES (PDCAT), 2016, : 244 - 249
  • [7] Cloud curriculum resource management platform based on Hadoop
    Zhao, Yu
    Liu, Hongxin
    [J]. MEASUREMENT & CONTROL, 2020, 53 (9-10): : 1782 - 1790
  • [8] Energy and resource efficient workflow scheduling in a virtualized cloud environment
    Neha Garg
    Damanpreet Singh
    Major Singh Goraya
    [J]. Cluster Computing, 2021, 24 : 767 - 797
  • [9] Energy and resource efficient workflow scheduling in a virtualized cloud environment
    Garg, Neha
    Singh, Damanpreet
    Goraya, Major Singh
    [J]. CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, 2021, 24 (02): : 767 - 797
  • [10] Profit-Driven Resource Scheduling for Virtualized Cloud Systems
    Ye, Shiyang
    Wang, Tao
    Zhang, Wenbo
    Zhong, Hua
    [J]. 2014 IEEE/ACIS 13TH INTERNATIONAL CONFERENCE ON COMPUTER AND INFORMATION SCIENCE (ICIS), 2014, : 263 - 268