Design and development of real-time query platform for big data based on hadoop

被引：1

作者：

刘小利 ^{[1
]}

Xu Pandeng ^{[2
]}

Liu Mingliang ^{[3
]}

Zhu Guobin ^{[4
]}

机构：

[1] Key Laboratory of Earthquake Geodesy,Institute of Seismology,CEA

[2] Jiangxi Branch of China Telecom

[3] The Chinese Institute of Electronics

[4] International School of Software,Wuhan University

来源：

High Technology Letters | 2015年 / 21卷 / 02期

关键词：

big data; massive data storage; real-time query; Hadoop; distributed computing;

D O I：

暂无

中图分类号：

TP311.13 [];

学科分类号：

1201 ;

摘要：

This paper designs and develops a framework on a distributed computing platform for massive multi-source spatial data using a column-oriented database(HBase).This platform consists of four layers including ETL(extraction transformation loading) tier,data processing tier,data storage tier and data display tier,achieving long-term store,real-time analysis and inquiry for massive data.Finally,a real dataset cluster is simulated,which are made up of 39 nodes including 2 master nodes and 37 data nodes,and performing function tests of data importing module and real-time query module,and performance tests of HDFS’s I/O,the MapReduce cluster,batch-loading and real-time query of massive data.The test results indicate that this platform achieves high performance in terms of response time and linear scalability.

引用

页码：231 / 238

页数：8

共 50 条

[1] Study of CDR Real-time Query Based on Big Data Technologies
Gao, Zhiheng
Chen, Kang
Bi, Lingyan
[J]. PROGRESS IN MECHATRONICS AND INFORMATION TECHNOLOGY, PTS 1 AND 2, 2014, 462-463 : 845 - +
[2] Design and Implementation of Real-Time Video Big Data Platform based on Spark Streaming
Chen, Hongjun
Luo, Fuqiang
Zhao, Liheng
Li, Yao
[J]. INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND APPLICATION ENGINEERING (CSAE), 2017, 190 : 733 - 739
[3] Platform for real-time data analysis and visualization based on Big Data methods
Ferreira, Gabriel
Alves, Paulo
de Almeida, Simone
[J]. PROCEEDINGS OF 2021 16TH IBERIAN CONFERENCE ON INFORMATION SYSTEMS AND TECHNOLOGIES (CISTI'2021), 2021,
[4] Design and Implementation of Meteorological Big Data Platform Based on Hadoop and Elasticsearch
Yin, He
Deng Fengdong
[J]. 2019 IEEE 4TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND BIG DATA ANALYSIS (ICCCBDA), 2019, : 705 - 710
[5] Hadoop Based Real-Time Big Data Architecture for Remote Sensing Earth Observatory System
Rathore, M. Mazhar
Ahmad, Awais
Paul, Anand
Daniel, Alfred
[J]. 2015 6TH INTERNATIONAL CONFERENCE ON COMPUTING, COMMUNICATION AND NETWORKING TECHNOLOGIES (ICCCNT), 2015, : 204 - 210
[6] Soft Real-Time Hadoop Scheduler for Big Data Processing in Smart Cities
Barbieru, Ciprian
Pop, Florin
[J]. IEEE 30TH INTERNATIONAL CONFERENCE ON ADVANCED INFORMATION NETWORKING AND APPLICATIONS IEEE AINA 2016, 2016, : 863 - 870
[7] Real-time Big Data Technologies of Energy Internet Platform
Wang Guilan
Zhou Guoliang
Zhao Hongshan
Liu Hongyang
[J]. 2016 IEEE INTERNATIONAL CONFERENCE ON POWER SYSTEM TECHNOLOGY (POWERCON), 2016,
[8] Power Big Data platform Based on Hadoop Technology
Chen, Jilin
Liu, Nana
Chen, Yong
Qiu, Weijiang
[J]. PROCEEDINGS OF THE 2016 6TH INTERNATIONAL CONFERENCE ON MACHINERY, MATERIALS, ENVIRONMENT, BIOTECHNOLOGY AND COMPUTER (MMEBC), 2016, 88 : 571 - 576
[9] Design and Development of a Medical Big Data Processing System Based on Hadoop
Yao, Qin
Tian, Yu
Li, Peng-Fei
Tian, Li-Li
Qian, Yang-Ming
Li, Jing-Song
[J]. JOURNAL OF MEDICAL SYSTEMS, 2015, 39 (03)
[10] Design and Development of a Medical Big Data Processing System Based on Hadoop
Qin Yao
Yu Tian
Peng-Fei Li
Li-Li Tian
Yang-Ming Qian
Jing-Song Li
[J]. Journal of Medical Systems, 2015, 39

← 1 2 3 4 5 →