Development of NCL equivalent serial and parallel python']python routines for meteorological data analysis

被引:2
|
作者
Gharat, Jatin [1 ]
Kumar, Bipin [2 ]
Ragha, Leena [1 ]
Barve, Amit [3 ]
Jeelani, Shaik Mohammad [2 ]
Clyne, John [4 ]
机构
[1] Ramrao Adik Inst Technol, Comp Engn, Navi Mumbai, India
[2] Minist Earth Sci, Indian Inst Trop Meteorol, HPCS, Dr Homi Bhabha Rd, Pune 411008, Maharashtra, India
[3] Parul Univ, Parul Inst Engn & Technol, Comp Sci & Engn, Vadodara, Gujarat, India
[4] Natl Ctr Atmospher Res, Computat & Informat Syst Lab, POB 3000, Boulder, CO 80307 USA
关键词
National Center for Atmospheric Research command; language functions; !text type='python']python[!/text] routines; parallel [!text type='python']python[!/text] version; climate data analysis; high-performance computing; PERFORMANCE; MODELS;
D O I
10.1177/10943420221077110
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
The NCAR Command Language (NCL) is a popular scripting language used in the geoscience community for weather data analysis and visualization. Hundreds of years of data are analyzed daily using NCL to make accurate weather predictions. However, due to its sequential nature of execution, it cannot properly utilize the parallel processing power provided by High-Performance Computing systems (HPCs). Until now very few techniques have been developed to make use of the multi-core functionality of modern HPC systems on these functions. In the recent trend, open-source languages are becoming highly popular because they support major functionalities required for data analysis and parallel computing. Hence, developers of NCL have decided to adopt Python as the future scripting language for analysis and visualization and to enable the geosciences community to play an active role in its development and support. This study focuses on developing some of the widely used NCL routines in Python. To deal with the analysis of large datasets, parallel versions of these routines are developed to work within a single node and make use of multi-core CPUs to achieve parallelism. Results show high accuracy between NCL and Python outputs and the parallel versions provided good scaling compared to their sequential counterparts.
引用
收藏
页码:337 / 355
页数:19
相关论文
共 50 条
  • [21] Data management routines for reproducible research using the G-Node Python']Python Client library
    Sobolev, Andrey
    Stoewer, Adrian
    Pereira, Michael
    Kellner, Christian J.
    Garbers, Christian
    Rautenberg, Philipp L.
    Wachtler, Thomas
    [J]. FRONTIERS IN NEUROINFORMATICS, 2014, 8
  • [22] reciprocalspaceship: a Python']Python library for crystallographic data analysis
    Greisman, Jack B.
    Dalton, Kevin M.
    Hekstra, Doeke R.
    [J]. JOURNAL OF APPLIED CRYSTALLOGRAPHY, 2021, 54 : 1521 - 1529
  • [23] Scalable Multimedia Content Analysis on Parallel Platforms Using Python']Python
    Gonina, Ekaterina
    Friedland, Gerald
    Battenberg, Eric
    Koanantakool, Penporn
    Driscoll, Michael
    Georganas, Evangelos
    Keutzer, Kurt
    [J]. ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS, 2014, 10 (02)
  • [24] Efficient development of high performance data analytics in Python']Python
    Alvarez Cid-Fuentes, Javier
    Alvarez, Pol
    Amela, Ramon
    Ishii, Kuninori
    Morizawa, Rafael K.
    Badia, Rosa M.
    [J]. FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2020, 111 : 570 - 581
  • [25] A New Parallel Python']Python Tool for the Standardization of Earth System Model Data
    Paul, Kevin
    MickelsonT, Sheri
    Dennis, John M.
    [J]. 2016 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2016, : 2953 - 2959
  • [26] DESIGN AND DEVELOPMENT OF A VIRTUAL LABORATORY IN PYTHON']PYTHON FOR THE TEACHING OF DATA ANALYSIS AND MATHEMATICS IN GEOLOGY: GEOPY
    Lopez Gonzalez-Nieto, P.
    Gomez Flechoso, M. A.
    Mocoroa, M. E. Arribas
    Munoz Martin, A.
    Garcia Lorenzo, M. L.
    Cabrera Gomez, G.
    Alvarez Gomez, J. A.
    Caso Fraile, A.
    Orosco Dagan, J. M.
    Merinero Palomares, R.
    Lahoz-Beltra, R.
    [J]. 14TH INTERNATIONAL TECHNOLOGY, EDUCATION AND DEVELOPMENT CONFERENCE (INTED2020), 2020, : 2236 - 2242
  • [27] The Lompe code: A Python']Python toolbox for ionospheric data analysis
    Hovland, A. O.
    Laundal, K. M.
    Reistad, J. P.
    Hatch, S. M.
    Walker, S. J.
    Madelaire, M.
    Ohma, A.
    [J]. FRONTIERS IN ASTRONOMY AND SPACE SCIENCES, 2022, 9
  • [28] Refrapy: A Python']Python program for seismic refraction data analysis
    Cavalcanti Bezerra Guedes, Victor Jose
    Ramalho Maciel, Susanne Taina
    Rocha, Marcelo Peres
    [J]. COMPUTERS & GEOSCIENCES, 2022, 159
  • [29] MEG and EEG data analysis with MNE-Python']Python
    Gramfort, Alexandre
    Luessi, Martin
    Larson, Eric
    Engemann, Denis A.
    Strohmeier, Daniel
    Brodbeck, Christian
    Goj, Roman
    Jas, Mainak
    Brooks, Teon
    Parkkonen, Lauri
    Haemaelaeinen, Matti
    [J]. FRONTIERS IN NEUROSCIENCE, 2013, 7
  • [30] GuPPy, a Python']Python toolbox for the analysis of fiber photometry data
    Sherathiya, Venus N.
    Schaid, Michael D.
    Seiler, Jillian L.
    Lopez, Gabriela C.
    Lerner, Talia N.
    [J]. SCIENTIFIC REPORTS, 2021, 11 (01)