Parallel Accessing Massive NetCDF Data Based on MapReduce

被引：0
|
作者：
Zhao, Hui [1 ,2 ]
Ai, SiYun [3 ]
Lv, ZhenHua [4 ]
Li, Bo [1 ]
机构：
[1] Key Lab Trustworthy Comp Shanghai, Shanghai, Peoples R China
[2] East China Normal Univ Shanghai, Shanghai, Peoples R China
[3] Nanyang Technol Univ Singapore, Sch EEE Commun Software & Network, Singapore, Singapore
[4] East China Normal Univ, Dept Geog, Minist Educ, Key Lab Geograph Informat Sci, Shanghai, Peoples R China
来源：
WEB INFORMATION SYSTEMS AND MINING | 2010年 / 6318卷
基金：
美国国家科学基金会;
关键词：
NetCDF; MapReduce; Data intensive; Parallel access;
D O I：
暂无
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
As a Network Common Data Format, NetCDF has been widely used in terrestrial, marine and atmospheric sciences. A new paralleling storage and access method for large scale NetCDF scientific data is implemented based on Hadoop. The retrieval method is implemented based on MapReduce. The Argo data is used to demonstrate our method. The performance is compared under a distributed environment based on PCs by using different data scale and different task numbers. The experiments result show that the parallel method can be used to store and access the large scale NetCDF efficiently.
引用
收藏
页码：425 / +
页数：3
相关论文
共 50 条
  • [1] Parallel labeling of massive XML data with MapReduce
    Choi, Hyebong
    Lee, Kyong-Ha
    Lee, Yoon-Joon
    [J]. JOURNAL OF SUPERCOMPUTING, 2014, 67 (02): : 408 - 437
  • [2] Parallel labeling of massive XML data with MapReduce
    Hyebong Choi
    Kyong-Ha Lee
    Yoon-Joon Lee
    [J]. The Journal of Supercomputing, 2014, 67 : 408 - 437
  • [3] Parallel Processing of Massive EEG Data with MapReduce
    Wang, Lizhe
    Chen, Dan
    Ranjan, Rajiv
    Khan, Samee U.
    Kolodziej, Joanna
    Wang, Jun
    [J]. PROCEEDINGS OF THE 2012 IEEE 18TH INTERNATIONAL CONFERENCE ON PARALLEL AND DISTRIBUTED SYSTEMS (ICPADS 2012), 2012, : 164 - 171
  • [4] Accessing and Distributing Large Volumes of NetCDF Data
    Devarakonda, Ranjeet
    Wei, Yaxing
    Thornton, Michele
    [J]. 2016 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2016, : 3966 - 3967
  • [5] Efficient Random Data Accessing in MapReduce
    Mittal, Mamta
    Singh, Hari
    Paliwal, K. K.
    Goyal, Lalit Mohan
    [J]. 2017 INTERNATIONAL CONFERENCE ON INFOCOM TECHNOLOGIES AND UNMANNED SYSTEMS (TRENDS AND FUTURE DIRECTIONS) (ICTUS), 2017, : 552 - 556
  • [6] Parallel Map Matching on Massive Vehicle GPS Data Using MapReduce
    Huang, Jian
    Qiao, Shaoqing
    Yu, Haitao
    Qie, Jinhui
    Liu, Chunwei
    [J]. 2013 IEEE 15TH INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING AND COMMUNICATIONS & 2013 IEEE INTERNATIONAL CONFERENCE ON EMBEDDED AND UBIQUITOUS COMPUTING (HPCC_EUC), 2013, : 1498 - 1503
  • [7] PARALLEL GRAPH PATTERN MATCHING IN MASSIVE NETWORKS BASED ON MAPREDUCE
    Ding, Yuanxiong
    Chen, Qun
    [J]. FIFTH INTERNATIONAL CONFERENCE ON ADVANCED COMPUTER THEORY AND ENGINEERING (ICACTE 2012), 2012, : 1 - 5
  • [8] Analysis of Massive Industrial Data using MapReduce Framework for Parallel Processing
    Aly, Mohab
    Yacout, Soumaya
    Shaban, Yasser
    [J]. 2017 ANNUAL RELIABILITY AND MAINTAINABILITY SYMPOSIUM, 2017,
  • [9] Parallel area navigation enhanced information extraction algorithm based on massive historical GNSS data and MapReduce
    Li, Dengao
    Wu, Gang
    Zhao, Jumin
    Niu, Wenhui
    Li, Shuai
    [J]. 2015 THIRD INTERNATIONAL CONFERENCE ON ADVANCED CLOUD AND BIG DATA, 2015, : 52 - 57
  • [10] Parallel similarity joins on massive high-dimensional data using MapReduce
    Ma, Youzhong
    Meng, Xiaofeng
    Wang, Shaoya
    [J]. CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE, 2016, 28 (01): : 166 - 183
12345