Design and development of real-time query platform for big data based on hadoop

被引：0
|
作者：
Liu, Xiaoli [1 ]
Xu, Pandeng [2 ]
Liu, Mingliang [3 ]
Zhu, Guobin [4 ]
机构：
[1] Key Laboratory of Earthquake Geodesy, Institute of Seismology, CEA, Wuhan,430071, China
[2] Jiangxi Branch of China Telecom, Nancang,330046, China
[3] The Chinese Institute of Electronics, Beijing,100036, China
[4] International School of Software, Wuhan University, Wuhan,430079, China
来源：
High Technology Letters | 2015年 / 21卷 / 02期
关键词：
Column-oriented database - Design and Development - Distributed computing platform - Extraction transformation loadings - Hadoop - Massive data - Multi-source spatial data - Real time;
D O I：
10.3772/j.issn.1006-6748.2015.02.017
中图分类号：
学科分类号：
摘要：
This paper designs and develops a framework on a distributed computing platform for massive multi-source spatial data using a column-oriented database (HBase). This platform consists of four layers including ETL (extraction transformation loading) tier, data processing tier, data storage tier and data display tier, achieving long-term store, real-time analysis and inquiry for massive data. Finally, a real dataset cluster is simulated, which are made up of 39 nodes including 2 master nodes and 37 data nodes, and performing function tests of data importing module and real-time query module, and performance tests of HDFS's I/O, the MapReduce cluster, batch-loading and real-time query of massive data. The test results indicate that this platform achieves high performance in terms of response time and linear scalability. ©, 2015, Inst. of Scientific and Technical Information of China. All right reserved.
引用
收藏
页码：231 / 238
相关论文
共 50 条
  • [41] A real-time computer vision-based platform for fabric inspection part 2: platform design and real-time implementation
    Zhou, Jian
    Li, Guanzhi
    Wan, Xianfu
    Wang, Jun
    JOURNAL OF THE TEXTILE INSTITUTE, 2016, 107 (02) : 264 - 272
  • [42] Real-time Video Copy Detection Based on Hadoop
    Li, Jing
    Lian, Xuquan
    Wu, Qiang
    Sun, Jiande
    2016 SIXTH INTERNATIONAL CONFERENCE ON INFORMATION SCIENCE AND TECHNOLOGY (ICIST), 2016, : 492 - 497
  • [43] Open-Source Big Data Platform for Real-Time Geolocation in Smart Cities
    Moreno-Bernal, Pedro
    Alan Cervantes-Salazar, Carlos
    Nesmachnow, Sergio
    Manuel Hurtado-Ramirez, Juan
    Alberto Hernandez-Aguilar, Jose
    SMART CITIES (ICSC-CITIES 2021), 2022, 1555 : 207 - 222
  • [44] Query Object Detection in Big Video Data on Hadoop Framework
    Raju, U. S. N.
    Varma, N. Kishan
    Pariveda, Harikrishna
    Reddy, Kotte Abhilash
    2015 1ST IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA BIG DATA (BIGMM), 2015, : 284 - 285
  • [45] Real-time Twitter data analysis using Hadoop ecosystem
    Rodrigues, Anisha P.
    Chiplunkar, Niranjan N.
    COGENT ENGINEERING, 2018, 5 (01): : 1 - 16
  • [46] NEAR REAL-TIME PROCESSING OF PROTEOMICS DATA USING HADOOP
    Hillman, Chris
    Ahmad, Yasmeen
    Whitehorn, Mark
    Cobley, Andy
    BIG DATA, 2014, 2 (01) : 44 - 49
  • [47] A survey on data stream, big data and real-time
    Gomes E.H.A.
    Plentz P.D.M.
    De Rolt C.R.
    Dantas M.A.R.
    International Journal of Networking and Virtual Organisations, 2019, 20 (02) : 143 - 167
  • [48] Developing a Real-Time Data Analytics Framework using Hadoop
    Cha, Sangwhan
    Wachowicz, Monica
    2015 IEEE INTERNATIONAL CONGRESS ON BIG DATA - BIGDATA CONGRESS 2015, 2015, : 657 - 660
  • [49] Toward real-time data query systems in HEP
    Pivarski, Jim
    Lange, David
    Jatuphattharachat, Thanat
    18TH INTERNATIONAL WORKSHOP ON ADVANCED COMPUTING AND ANALYSIS TECHNIQUES IN PHYSICS RESEARCH (ACAT2017), 2018, 1085
  • [50] RTSTREAM: Real-time query processing for data streams
    Wei, Yuan
    Son, Sang H.
    Stankovic, John A.
    NINTH IEEE INTERNATIONAL SYMPOSIUM ON OBJECT AND COMPONENT-ORIENTED REAL-TIME DISTRIBUTED COMPUTING, PROCEEDINGS, 2006, : 141 - 150
12345