HDFS efficiency storage strategy for big data in smart city

被引：0
|
作者：
Xiang, Min [1 ]
Jiang, Yuzhou [1 ]
Xia, Zhong [1 ]
Xu, Longzhang [1 ]
Huang, Chunmei [1 ]
机构：
[1] Chongqing Univ Posts & Telecommun, Key Lab Ind Internet Things & Networked Control, Chongqing, Peoples R China
来源：
2019 CHINESE AUTOMATION CONGRESS (CAC2019) | 2019年
基金：
国家重点研发计划;
关键词：
big data; IMES; smart city; BP neural network; response time; load balancing;
D O I：
10.1109/cac48633.2019.8996571
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
With the rapid development of smart city and artificial intelligence technology, massive city-related information has been increased. The traditional storage strategy for big data can easily lead to hot nodes, which makes it difficult to meet requirement of big data storage efficiency. A Hadoop Distributed File System (HDFS) storage strategy for data based on the response time of data nodes (DNs) is proposed. Four parameters of memory utilization, network distance, bandwidth utilization and CPU utilization of DNs arc token as evaluation indicators of the strategy. Then the master node evaluates the response time of each data node (DN) based on BP neural network. Finally, the DNs with shorter response time would be chosen for data storage. The simulation results show that the proposed strategy can realize the distributed storage for big data and avoid the emergence of hot nodes. Furthermore, the strategy can effectively improve the response time of DNs and the load balancing of cluster.
引用
收藏
页码：2394 / 2398
页数：5
相关论文
共 50 条
  • [1] Energy Conservation Strategy for Big News Data on HDFS
    Zhong, Jiang
    Chen, Hao
    Yang, Lei
    [J]. BIG DATA TECHNOLOGY AND APPLICATIONS, 2016, 590 : 59 - 73
  • [2] City Management Support and Smart City Strategy as Success Factors in Adopting Big Data Technologies for Smart Cities
    Pivar, Jasmina
    [J]. SMART GOVERNMENTS, REGIONS AND CITIES, 2020, : 167 - 183
  • [3] Big Data and Smart City Platforms
    Oktug, Sema F.
    Yaslan, Yusuf
    [J]. 2017 25TH SIGNAL PROCESSING AND COMMUNICATIONS APPLICATIONS CONFERENCE (SIU), 2017,
  • [4] The role of big data in smart city
    Hashem, Ibrahim Abaker Targio
    Chang, Victor
    Anuar, Nor Badrul
    Adewole, Kayode
    Yaqoob, Ibrar
    Gani, Abdullah
    Ahmed, Ejaz
    Chiroma, Haruna
    [J]. INTERNATIONAL JOURNAL OF INFORMATION MANAGEMENT, 2016, 36 (05) : 748 - 758
  • [5] HDFS Optimization Strategy Based On Hierarchical Storage of Hot and Cold Data
    Guan, Yuxin
    Ma, Zhiqiang
    Li, Leixiao
    [J]. 11TH CIRP CONFERENCE ON INDUSTRIAL PRODUCT-SERVICE SYSTEMS, 2019, 83 : 415 - 418
  • [6] HDFS Copy Storage Improvement Strategy
    Zhang, Jing
    Sun, Hongbo
    Yuan, ShiJing
    [J]. PROCEEDINGS OF 2019 3RD INTERNATIONAL CONFERENCE ON CLOUD AND BIG DATA COMPUTING (ICCBDC 2019), 2019, : 71 - 77
  • [7] Big Data, Big Rhetoric in Toronto's Smart City
    Tierney, T. F.
    [J]. ARCHITECTURE AND CULTURE, 2019, 7 (03) : 351 - 363
  • [8] Big Data Processing Platform for smart city
    El Mendili, Saida
    El Bouzekri El Idrissi, Younes
    Hmina, Nabil
    [J]. 2018 INTERNATIONAL SYMPOSIUM ON ADVANCED ELECTRICAL AND COMMUNICATION TECHNOLOGIES (ISAECT), 2018,
  • [9] Big Data, Technical Communication, and the Smart City
    Frith, Jordan
    [J]. JOURNAL OF BUSINESS AND TECHNICAL COMMUNICATION, 2017, 31 (02) : 168 - 187
  • [10] The Development of Smart City in the Era of Big Data
    Guo, Ziyu
    [J]. NEW INDUSTRIALIZATION AND URBANIZATION DEVELOPMENT ANNUAL CONFERENCE: THE INTERNATIONAL FORUM ON NEW INDUSTRIALIZATION DEVELOPMENT IN BIG-DATA ERA, 2015, : 555 - 558
12345