PARTITIONER: A Distributed HDFS Metadata Server Cluster

被引：3
|
作者：
Xue, Ruini [1 ]
Ao, Lixiang [1 ]
Gao, Shengli [1 ]
Guan, Zhongyang [1 ]
Lian, Lupeng [1 ]
机构：
[1] Univ Elect Sci & Technol China, Sch Comp Sci & Engn, Chengdu 610054, Peoples R China
来源：
2014 INTERNATIONAL CONFERENCE ON CYBER-ENABLED DISTRIBUTED COMPUTING AND KNOWLEDGE DISCOVERY (CYBERC) | 2014年
关键词：
distributed file system; metadata management; scalability; HDFS;
D O I：
10.1109/CyberC.2014.38
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
The Hadoop Distributed File System (HDFS) provides scalable, high throughput and fault-tolerant file service for Hadoop distributed frameworks. However, its single NameNode architecture can lead the metadata server to be a performance bottleneck and a single point of failure (SPOF). In this study, we designed and implemented PARTITIONER, a distributed HDFS metadata server cluster, which can unite multiple NameNodes and provide single unique namespace for clients. To support the distributed metadata management, a novel subtree based metadata partitioning and lookup scheme is proposed. Experiment results indicated that this method can eliminate SPOF while expanding metadata capacity, resulting in improved scalability and availability of the distributed file system.
引用
收藏
页码：167 / 174
页数：8
相关论文
共 50 条
  • [1] HDFS distributed metadata management research
    Xiong, An-ping
    Ma, Jin-yong
    [J]. PROCEEDINGS OF THE 2015 INTERNATIONAL CONFERENCE ON APPLIED SCIENCE AND ENGINEERING INNOVATION, 2015, 12 : 956 - 961
  • [2] Design and Implementation of a Non-Shared Metadata Server Cluster for Large Distributed File Systems
    Yun, Jong-Hyeon
    Park, Yong-Hun
    Lee, Seok-Jae
    Jang, Su-Min
    Yoo, Jae-Soo
    [J]. CSA 2008: INTERNATIONAL SYMPOSIUM ON COMPUTER SCIENCE AND ITS APPLICATIONS, PROCEEDINGS, 2008, : 343 - 346
  • [3] Clover: A distributed file system of expandable metadata service derived from HDFS
    Wang, Youwei
    Zhou, Jiang
    Ma, Can
    Wang, Weiping
    Meng, Dan
    Kei, Jason
    [J]. 2012 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING (CLUSTER), 2012, : 126 - 134
  • [4] Memory based metadata server for cluster file systems
    Xing, Jing
    Xiong, Jin
    Ma, Jie
    Sun, Ninghui
    [J]. GCC 2008: SEVENTH INTERNATIONAL CONFERENCE ON GRID AND COOPERATIVE COMPUTING, PROCEEDINGS, 2008, : 287 - +
  • [5] An Improvement of Static Subtree Partitioning in Metadata Server Cluster
    Tan Zhipeng
    Zhou Wei
    Sun Jianliang
    Zhan Tian
    Cui Jie
    [J]. INTERNATIONAL JOURNAL OF DISTRIBUTED SENSOR NETWORKS, 2012,
  • [6] DMooseFS: Design and Implementation of Distributed Files System with Distributed Metadata Server
    Yu, Jiongyu
    Wu, Weigang
    Li, Huaguan
    [J]. IEEE ASIA PACIFIC CLOUD COMPUTING CONGRESS 2012, 2012, : 42 - 47
  • [7] Social Hash Partitioner: A Scalable Distributed Hypergraph Partitioner
    Kabiljo, Igor
    Karrer, Brian
    Pundir, Mayank
    Pupyrev, Sergey
    Shalita, Alon
    [J]. PROCEEDINGS OF THE VLDB ENDOWMENT, 2017, 10 (11): : 1418 - 1429
  • [8] Scalable Distributed Metadata Server Based on Nonblocking Transactions
    Hiraga, Kohei
    Tatebe, Osamu
    Kawashima, Hideyuki
    [J]. JOURNAL OF UNIVERSAL COMPUTER SCIENCE, 2020, 26 (01) : 89 - 106
  • [9] PPMDS: A Distributed Metadata Server based on Nonblocking Transactions
    Hiraga, Kohei
    Tatebe, Osamu
    Kawashima, Hideyuki
    [J]. 2018 FIFTH INTERNATIONAL CONFERENCE ON SOCIAL NETWORKS ANALYSIS, MANAGEMENT AND SECURITY (SNAMS), 2018, : 202 - 208
  • [10] Optimized Approach (SPCA) for Load Balancing in Distributed HDFS Cluster
    Manjula K.
    Meenakshi Sundaram S.
    [J]. SN Computer Science, 2020, 1 (2)
12345