Design and development of real-time query platform for big data based on hadoop

被引：0

作者：

Liu, Xiaoli ^{[1
]}

Xu, Pandeng ^{[2
]}

Liu, Mingliang ^{[3
]}

Zhu, Guobin ^{[4
]}

机构：

[1] Key Laboratory of Earthquake Geodesy, Institute of Seismology, CEA, Wuhan,430071, China

[2] Jiangxi Branch of China Telecom, Nancang,330046, China

[3] The Chinese Institute of Electronics, Beijing,100036, China

[4] International School of Software, Wuhan University, Wuhan,430079, China

来源：

High Technology Letters | 2015年 / 21卷 / 02期

关键词：

Column-oriented database - Design and Development - Distributed computing platform - Extraction transformation loadings - Hadoop - Massive data - Multi-source spatial data - Real time;

D O I：

10.3772/j.issn.1006-6748.2015.02.017

中图分类号：

学科分类号：

摘要：

This paper designs and develops a framework on a distributed computing platform for massive multi-source spatial data using a column-oriented database (HBase). This platform consists of four layers including ETL (extraction transformation loading) tier, data processing tier, data storage tier and data display tier, achieving long-term store, real-time analysis and inquiry for massive data. Finally, a real dataset cluster is simulated, which are made up of 39 nodes including 2 master nodes and 37 data nodes, and performing function tests of data importing module and real-time query module, and performance tests of HDFS's I/O, the MapReduce cluster, batch-loading and real-time query of massive data. The test results indicate that this platform achieves high performance in terms of response time and linear scalability. ©, 2015, Inst. of Scientific and Technical Information of China. All right reserved.

引用

页码：231 / 238

共 50 条

[1] Design and development of real-time query platform for big data based on hadoop
刘小利
Xu Pandeng
Liu Mingliang
Zhu Guobin
High Technology Letters, 2015, 21 (02) : 231 - 238
[2] Study of CDR Real-time Query Based on Big Data Technologies
Gao, Zhiheng
Chen, Kang
Bi, Lingyan
PROGRESS IN MECHATRONICS AND INFORMATION TECHNOLOGY, PTS 1 AND 2, 2014, 462-463 : 845 - +
[3] Design and Implementation of Real-Time Video Big Data Platform based on Spark Streaming
Chen, Hongjun
Luo, Fuqiang
Zhao, Liheng
Li, Yao
INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND APPLICATION ENGINEERING (CSAE), 2017, 190 : 733 - 739
[4] Platform for real-time data analysis and visualization based on Big Data methods
Ferreira, Gabriel
Alves, Paulo
de Almeida, Simone
PROCEEDINGS OF 2021 16TH IBERIAN CONFERENCE ON INFORMATION SYSTEMS AND TECHNOLOGIES (CISTI'2021), 2021,
[5] Design and Implementation of Meteorological Big Data Platform Based on Hadoop and Elasticsearch
Yin, He
Deng Fengdong
2019 IEEE 4TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND BIG DATA ANALYSIS (ICCCBDA), 2019, : 705 - 710
[6] Development and Application of Personal Hadoop-Based Big Data Platform
Wu G.
Lin F.
Chang W.-Y.
Tsai W.-F.
Lin S.-C.
Yang C.-T.
Journal of the Chinese Institute of Civil and Hydraulic Engineering, 2018, 30 (02): : 107 - 120
[7] Hadoop Based Real-Time Big Data Architecture for Remote Sensing Earth Observatory System
Rathore, M. Mazhar
Ahmad, Awais
Paul, Anand
Daniel, Alfred
2015 6TH INTERNATIONAL CONFERENCE ON COMPUTING, COMMUNICATION AND NETWORKING TECHNOLOGIES (ICCCNT), 2015, : 204 - 210
[8] Soft Real-Time Hadoop Scheduler for Big Data Processing in Smart Cities
Barbieru, Ciprian
Pop, Florin
IEEE 30TH INTERNATIONAL CONFERENCE ON ADVANCED INFORMATION NETWORKING AND APPLICATIONS IEEE AINA 2016, 2016, : 863 - 870
[9] Real-time Big Data Technologies of Energy Internet Platform
Wang Guilan
Zhou Guoliang
Zhao Hongshan
Liu Hongyang
2016 IEEE INTERNATIONAL CONFERENCE ON POWER SYSTEM TECHNOLOGY (POWERCON), 2016,
[10] Power Big Data platform Based on Hadoop Technology
Chen, Jilin
Liu, Nana
Chen, Yong
Qiu, Weijiang
PROCEEDINGS OF THE 2016 6TH INTERNATIONAL CONFERENCE ON MACHINERY, MATERIALS, ENVIRONMENT, BIOTECHNOLOGY AND COMPUTER (MMEBC), 2016, 88 : 571 - 576

← 1 2 3 4 5 →