HDFS Copy Storage Improvement Strategy

被引：0

作者：

Zhang, Jing ^{[1
]}

Sun, Hongbo ^{[1
]}

Yuan, ShiJing ^{[1
]}

机构：

[1] Nanjing Univ Posts & Telecommun, 9 Wenyuan Rd, Nanjing, Jiangsu, Peoples R China

来源：

PROCEEDINGS OF 2019 3RD INTERNATIONAL CONFERENCE ON CLOUD AND BIG DATA COMPUTING (ICCBDC 2019) | 2019年

关键词：

Distributed file system; Copy policy; Response time; Cluster equalization;

D O I：

10.1145/3358505.3358520

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

To improve the reliability of data storage, HDFS adopts the multicopy storage strategy of Rack-Aware. However, it does not consider the data block's access heat during the replication process, resulting in that in the case of high concurrent access, the client's request for the hot file cannot be responded quickly, and the storage cluster's I/O throughput rate is significantly reduced. Aiming at the defects of HDFS default replica replication strategy, this paper proposes a replica replication strategy based on block heat The results show that the improved strategy is better than the HDFS default strategy in load balancing performance, the average response time of read requests of the improved strategy is 28.46% lower than the default strategy, and the response speed is significantly better than the HDFS default strategy.

引用

页码：71 / 77

页数：7

共 50 条

[31] QoSC: A QoS-Aware Storage Cloud Based on HDFS
Yang, Bowei
Song, Guanghua
Zheng, Yao
Wu, Yue
[J]. 2015 INTERNATIONAL SYMPOSIUM ON SECURITY AND PRIVACY IN SOCIAL NETWORKS AND BIG DATA (SOCIALSEC 2015), 2015, : 32 - 38
[32] HDFS Heterogeneous Storage Resource Management based on Data Temperature
Subramanyam, Rohith
[J]. 2015 INTERNATIONAL CONFERENCE ON CLOUD AND AUTONOMIC COMPUTING (ICCAC), 2015, : 232 - 235
[33] Research of multi-Namenode cloud storage model based on HDFS
College of Computer and Information Science, Chongqing Normal University, Chongqing, China
[J]. Intl. J. Adv. Comput. Technolog., 2012, 18 (482-489):
[34] Research on Data Storage and Processing Optimization Based on Federation HDFS and Spark
Chen, Fangzhou
Li, Peng
Xu, He
Xie, Wenkang
[J]. COMPLEX, INTELLIGENT, AND SOFTWARE INTENSIVE SYSTEMS, 2019, 772 : 1047 - 1057
[35] COMBINING STORAGE AND HARD COPY
VANHOOMI.K
[J]. ELECTRONIC ENGINEER, 1973, 32 (01): : 13 - 14
[36] Research of Metadata Management Method of Hierarchical Storage System Based on HDFS
Liu, Xiaoyu
Xia, Libin
Jiang, Xiaowei
Sun, Gongxing
[J]. Computer Engineering and Applications, 2023, 59 (17) : 257 - 265
[37] Fault Tolerant Erasure Coded Replication for HDFS Based Cloud Storage
Ko, Aye Chan
Zaw, Wint Thida
[J]. 2014 IEEE FOURTH INTERNATIONAL CONFERENCE ON BIG DATA AND CLOUD COMPUTING (BDCLOUD), 2014, : 104 - 109
[38] The Dynamic Replication Mechanism of HDFS Hot File based on Cloud Storage
Li, Mingyong
Ma, Yan
Chen, Meilian
[J]. INTERNATIONAL JOURNAL OF SECURITY AND ITS APPLICATIONS, 2015, 9 (08): : 439 - 448
[39] Researching on the Placement of Data Replicas in the System of HDFS Cloud Storage Cluster
Bao, Guangbin
Yu, Chaojia
Zhao, Hong
Luan, Yangyang
[J]. PROCEEDINGS OF 2013 CHINESE INTELLIGENT AUTOMATION CONFERENCE: INTELLIGENT AUTOMATION & INTELLIGENT TECHNOLOGY AND SYSTEMS, 2013, 255 : 259 - 269
[40] Unifying HDFS and GPFS: Enabling Analytics on Software-Defined Storage
Raghavendra, Ramya
Dewan, Pranita
Srivatsa, Mudhakar
[J]. MIDDLEWARE '16: PROCEEDINGS OF THE 17TH INTERNATIONAL MIDDLEWARE CONFERENCE, 2016,

← 1 2 3 4 5 →