Performance Evaluation of HDFS in Big Data Management

被引:0
|
作者
Dev, Dipayan [1 ]
Patgiri, Ripon [1 ]
机构
[1] Natl Inst Technol, Dept Comp Sci & Engn, Silchar, Silchar, India
关键词
HDFS; NameNode; DataNode; JobTracker; Metadata;
D O I
暂无
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
Size of the data used in today's enterprises has been growing at exponential rates from last few years. Simultaneously, the need to process and analyze the large volumes of data has also increased. To handle and for analysis of large datasets, an open-source implementation of Apache framework, Hadoop is used now-a-days. For managing and storing of all the resources across its cluster, Hadoop possesses a distributed file system called Hadoop Distributed File System(HDFS).HDFS is written completely in Java and is depicted in such a way that in can store Big Data more reliably, and can stream those at high processing time to the user applications. In recent days, Hadoop is used widely by popular organizations like Yahoo, Facebook and various online shopping market venders. On the other hand, Experiments on Data-Intensive computations are going on to parallelize the processing of data. None of them could actually achieve a desirable performance. Hadoop, with its Map-Reduce parallel data processing capability can achieve these goals efficiently [1]. This paper initially provides an overview of the HDFS in details. Later on, the paper reports the experimental work of Hadoop with the big data and suggests the various factors that affects the Hadoop cluster performance. Paper concludes with providing the different real field challenges of Hadoop in recent days and scope for future work
引用
收藏
页数:7
相关论文
共 50 条
  • [1] Big Data Analytics: Performance Evaluation for High Availability and Fault Tolerance using MapReduce Framework with HDFS
    Verma, Jai Prakash
    Mankad, Sapan H.
    Garg, Sanjay
    [J]. 2018 FIFTH INTERNATIONAL CONFERENCE ON PARALLEL, DISTRIBUTED AND GRID COMPUTING (IEEE PDGC), 2018, : 770 - 775
  • [2] Big Data Management Performance Evaluation in Hadoop Ecosystem
    Liu, Qing
    Fu, Yinjin
    Ni, Guiqiang
    Mei, Jianmin
    [J]. 2017 3RD INTERNATIONAL CONFERENCE ON BIG DATA COMPUTING AND COMMUNICATIONS (BIGCOM), 2017, : 413 - 421
  • [3] BIG DATA RETRIEVAL USING HDFS WITH LZO COMPRESSION
    Prasanth, T.
    Aarthi, K.
    Gunasekaran, M.
    [J]. PROCEEDINGS OF THE 2019 INTERNATIONAL CONFERENCE ON ADVANCES IN COMPUTING & COMMUNICATION ENGINEERING (ICACCE-2019), 2019,
  • [4] Development and Evaluation of a Big Data Framework for Performance Management in Mobile Networks
    Martinez-Mosquera, Diana
    Navarrete, Rosa
    Lujan-Mora, Sergio
    [J]. IEEE ACCESS, 2020, 8 : 226380 - 226396
  • [5] Energy Conservation Strategy for Big News Data on HDFS
    Zhong, Jiang
    Chen, Hao
    Yang, Lei
    [J]. BIG DATA TECHNOLOGY AND APPLICATIONS, 2016, 590 : 59 - 73
  • [6] HDFS efficiency storage strategy for big data in smart city
    Xiang, Min
    Jiang, Yuzhou
    Xia, Zhong
    Xu, Longzhang
    Huang, Chunmei
    [J]. 2019 CHINESE AUTOMATION CONGRESS (CAC2019), 2019, : 2394 - 2398
  • [7] Evaluation of Data Management Systems for Geospatial Big Data
    Amirian, Pouria
    Basiri, Anahid
    Winstanley, Adam
    [J]. COMPUTATIONAL SCIENCE AND ITS APPLICATIONS - ICCSA 2014, PT V, 2014, 8583 : 678 - +
  • [8] A Performance Evaluation of Classification Algorithms for Big Data
    Hai, Mo
    Zhang, You
    Zhang, Youjin
    [J]. 5TH INTERNATIONAL CONFERENCE ON INFORMATION TECHNOLOGY AND QUANTITATIVE MANAGEMENT, ITQM 2017, 2017, 122 : 1100 - 1107
  • [9] Research on performance evaluation and optimization of college budget management under the background of big data
    Zhao, Wei
    Li, Xiangying
    Zhou, Liping
    [J]. APPLIED MATHEMATICS AND NONLINEAR SCIENCES, 2023,
  • [10] From Big Data to Smart Data: Application to performance management
    Souifi, Amel
    Boulanger, Zohra Cherfi
    Zolghadri, Marc
    Barkallah, Maher
    Haddar, Mohamed
    [J]. IFAC PAPERSONLINE, 2021, 54 (01): : 857 - 862