Design and development of real-time query platform for big data based on hadoop

被引：0
|
作者：
Liu, Xiaoli [1 ]
Xu, Pandeng [2 ]
Liu, Mingliang [3 ]
Zhu, Guobin [4 ]
机构：
[1] Key Laboratory of Earthquake Geodesy, Institute of Seismology, CEA, Wuhan,430071, China
[2] Jiangxi Branch of China Telecom, Nancang,330046, China
[3] The Chinese Institute of Electronics, Beijing,100036, China
[4] International School of Software, Wuhan University, Wuhan,430079, China
来源：
High Technology Letters | 2015年 / 21卷 / 02期
关键词：
Column-oriented database - Design and Development - Distributed computing platform - Extraction transformation loadings - Hadoop - Massive data - Multi-source spatial data - Real time;
D O I：
10.3772/j.issn.1006-6748.2015.02.017
中图分类号：
学科分类号：
摘要：
This paper designs and develops a framework on a distributed computing platform for massive multi-source spatial data using a column-oriented database (HBase). This platform consists of four layers including ETL (extraction transformation loading) tier, data processing tier, data storage tier and data display tier, achieving long-term store, real-time analysis and inquiry for massive data. Finally, a real dataset cluster is simulated, which are made up of 39 nodes including 2 master nodes and 37 data nodes, and performing function tests of data importing module and real-time query module, and performance tests of HDFS's I/O, the MapReduce cluster, batch-loading and real-time query of massive data. The test results indicate that this platform achieves high performance in terms of response time and linear scalability. ©, 2015, Inst. of Scientific and Technical Information of China. All right reserved.
引用
收藏
页码：231 / 238
相关论文
共 50 条
  • [31] A scheme of structured data compression and query on Hadoop platform
    Ding, Xiangwu
    Tian, Bo
    Li, Yefeng
    2015 Third International Conference on Digital Information, Networking, and Wireless Communications (DINWC), 2015, : 160 - 164
  • [32] Attack Models for Big Data Platform Hadoop
    Li, Ningwei
    Gao, Hang
    Liu, Liang
    Zhang, Fan
    Wang, Wenxuan
    2019 IEEE 5TH INTL CONFERENCE ON BIG DATA SECURITY ON CLOUD (BIGDATASECURITY) / IEEE INTL CONFERENCE ON HIGH PERFORMANCE AND SMART COMPUTING (HPSC) / IEEE INTL CONFERENCE ON INTELLIGENT DATA AND SECURITY (IDS), 2019, : 154 - 159
  • [33] Research on Industry Data Analysis Model Based on Hadoop Big Data Platform
    Xu, Hongsheng
    Fan, Ganglong
    Li, Ke
    PROCEEDINGS OF THE 7TH INTERNATIONAL CONFERENCE ON EDUCATION, MANAGEMENT, INFORMATION AND COMPUTER SCIENCE (ICEMC 2017), 2017, 73 : 783 - 787
  • [34] Analysis of Big Data Platform with OpenStack and Hadoop
    Li, Xiaoyan
    Lu, Zhihui
    Wang, Nini
    Wu, Jie
    Huang, Shalin
    ADVANCES IN SERVICES COMPUTING, 2016, 10065 : 375 - 390
  • [35] Design of a Hadoop Based Data Platform for Auto Aftermarket
    Shen, Yi
    PROCEEDINGS OF THE 2015 INTERNATIONAL CONFERENCE ON MANAGEMENT, EDUCATION, INFORMATION AND CONTROL, 2015, 125 : 1425 - 1431
  • [36] Railway Big Data Real-time Processing Based on Storm
    Guo, Shihang
    Zhang, Lichen
    PROCEEDINGS OF THE 2016 2ND WORKSHOP ON ADVANCED RESEARCH AND TECHNOLOGY IN INDUSTRY APPLICATIONS, 2016, 81 : 536 - 539
  • [37] The Design and Implementation of Real-time HILS Based on RTX Platform
    Chai Lina
    Zhou Qiang
    2012 10TH IEEE INTERNATIONAL CONFERENCE ON INDUSTRIAL INFORMATICS (INDIN), 2012, : 276 - 280
  • [38] A Hadoop/MapReduce based platform for supporting health big data analytics
    Kuo A.
    Chrimes D.
    Qin P.
    Zamani H.
    Studies in Health Technology and Informatics, 2019, 257 : 229 - 235
  • [39] Real-time multi-path traffic flow assignment algorithm based on Hadoop platform
    Duan, Zong-Tao, 1600, Chang'an University (27):
  • [40] Design of real-time monitoring platform for internet of things based on cloud platform
    Wang, Shengjie
    Yan, Hairong
    PROCEEDINGS OF 2020 IEEE 5TH INFORMATION TECHNOLOGY AND MECHATRONICS ENGINEERING CONFERENCE (ITOEC 2020), 2020, : 61 - 64
12345