PARTITIONER: A Distributed HDFS Metadata Server Cluster

被引：3
|
作者：
Xue, Ruini [1 ]
Ao, Lixiang [1 ]
Gao, Shengli [1 ]
Guan, Zhongyang [1 ]
Lian, Lupeng [1 ]
机构：
[1] Univ Elect Sci & Technol China, Sch Comp Sci & Engn, Chengdu 610054, Peoples R China
来源：
2014 INTERNATIONAL CONFERENCE ON CYBER-ENABLED DISTRIBUTED COMPUTING AND KNOWLEDGE DISCOVERY (CYBERC) | 2014年
关键词：
distributed file system; metadata management; scalability; HDFS;
D O I：
10.1109/CyberC.2014.38
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
The Hadoop Distributed File System (HDFS) provides scalable, high throughput and fault-tolerant file service for Hadoop distributed frameworks. However, its single NameNode architecture can lead the metadata server to be a performance bottleneck and a single point of failure (SPOF). In this study, we designed and implemented PARTITIONER, a distributed HDFS metadata server cluster, which can unite multiple NameNodes and provide single unique namespace for clients. To support the distributed metadata management, a novel subtree based metadata partitioning and lookup scheme is proposed. Experiment results indicated that this method can eliminate SPOF while expanding metadata capacity, resulting in improved scalability and availability of the distributed file system.
引用
收藏
页码：167 / 174
页数：8
相关论文
共 50 条
  • [31] Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems
    Quanqing XU
    Rajesh Vellore ARUMUGAM
    Khai Leong YONG
    Yonggang WEN
    YewSoon ONG
    Weiya XI
    [J]. Frontiers of Computer Science., 2015, 9 (06) - 918
  • [32] Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems
    Xu, Quanqing
    Arumugam, Rajesh Vellore
    Yong, Khai Leong
    Wen, Yonggang
    Ong, Yew-Soon
    Xi, Weiya
    [J]. FRONTIERS OF COMPUTER SCIENCE, 2015, 9 (06) : 904 - 918
  • [33] Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems
    Quanqing Xu
    Rajesh Vellore Arumugam
    Khai Leong Yong
    Yonggang Wen
    Yew-Soon Ong
    Weiya Xi
    [J]. Frontiers of Computer Science, 2015, 9 : 904 - 918
  • [34] A Distributed File System Based on HDFS
    Liu, Jun
    Leng, Fang-Ling
    Li, Shi-Qi
    Bao, Yu-Bin
    [J]. Dongbei Daxue Xuebao/Journal of Northeastern University, 2019, 40 (06): : 795 - 800
  • [35] Automatic partitioner for behavior level distributed logic simulation
    Chang, KH
    Kang, JY
    Wang, HW
    Tu, WT
    Yeh, YJ
    Kuo, SY
    [J]. FORMAL TECHNIQUES FOR NETWORKED AND DISTRIBUTED SYSTEMS - FORTE 2005, 2005, 3731 : 525 - 528
  • [36] HBA: Distributed metadata management for large cluster-based storage systems
    Zhu, Yifeng
    Jiang, Hong
    Wang, Jun
    Xian, Feng
    [J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2008, 19 (06) : 750 - 763
  • [37] Research of Metadata Management Method of Hierarchical Storage System Based on HDFS
    Liu, Xiaoyu
    Xia, Libin
    Jiang, Xiaowei
    Sun, Gongxing
    [J]. Computer Engineering and Applications, 2023, 59 (17) : 257 - 265
  • [38] Design and Implementation of Two-Level Metadata Server in Small-Scale Cluster File System
    LIU Yuling
    [J]. Wuhan University Journal of Natural Sciences, 2006, (06) : 1939 - 1942
  • [39] CuSP: A customizable streaming edge partitioner for distributed graph analytics
    Hoang, Loc
    Dathathri, Roshan
    Gill, Gurbinder
    Pingali, Keshav
    [J]. Operating Systems Review (ACM), 2021, 55 (01): : 47 - 60
  • [40] The Design of Distributed File System Based on HDFS
    Wang, Yannan
    Zhang, Shudong
    Liu, Hui
    [J]. APPLIED MATERIALS AND TECHNOLOGIES FOR MODERN MANUFACTURING, PTS 1-4, 2013, 423-426 : 2733 - 2736
12345