END-TO-END PROCESS ORCHESTRATION OF EARTH OBSERVATION DATA WORKFLOWS WITH APACHE AIRFLOW ON HIGH PERFORMANCE COMPUTING

被引：3
|
作者：
Tian, Liang [1 ]
Sedona, Rocco [1 ,2 ]
Mozaffari, Amirpasha [2 ]
Kreshpa, Enxhi [2 ]
Paris, Claudia [3 ]
Riedel, Morris [1 ,2 ]
Schultz, Martin G. [2 ]
Cavallaro, Gabriele [1 ,2 ]
机构：
[1] Univ Iceland, Sch Engn & Nat Sci, IS-107 Reykjavik, Iceland
[2] Forschungszentrum Julich, Julich Supercomp Ctr, D-52428 Julich, Germany
[3] Univ Twente, NL-7514 AE Enschede, Netherlands
来源：
IGARSS 2023 - 2023 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM | 2023年
基金：
欧盟地平线“2020”;
关键词：
Workflows; Deep Learning (DL); High-Performance Computing (HPC); remote sensing data;
D O I：
10.1109/IGARSS52108.2023.10283416
中图分类号：
P [天文学、地球科学];
学科分类号：
07 ;
摘要：
Earth Observation (EO) data processing faces challenges due to large volumes, multiple sources, and diverse formats. To address this issue, this paper presents a scalable and parallelizable workflow using Apache Airflow, capable of integrating Machine Learning (ML) and Deep Learning (DL) models with Modular Supercomputing Architecture (MSA) systems. To test the workflow, we considered the production of large-scale Land-Cover (LC) maps as a case study. The workflow manager, Airflow, offers scalability, extensibility, and programmable task definition in Python. It allows us to execute different steps of the workflow in different High-Performance Computing (HPC) systems. The workflow is demonstrated on the Dynamical Exascale Entry Platform (DEEP) and J <spacing diaeresis>ulich Research on Exascale Cluster Architectures (JURECA) hosted at the J <spacing diaeresis>ulich Supercomputing Centre (JSC), a platform that incorporates heterogeneous JSC systems.
引用
收藏
页码：711 / 714
页数：4
相关论文
共 50 条
  • [31] END-TO-END SIMULATOR OF GEOSYNCHRONOUS SAR DATA FOR SYSTEM PERFORMANCE ASSESSMENT
    Giudici, Davide
    Leanza, Antonio
    Guarnieri, Andrea Monti
    Recchia, Andrea
    IGARSS 2018 - 2018 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2018, : 5659 - 5662
  • [32] Detecting and Localizing End-to-End Performance Degradation for Cellular Data Services
    Ahmed, Faraz
    Erman, Jeffrey
    Ge, Zihui
    Liu, Alex X.
    Wang, Jia
    Yan, He
    IEEE INFOCOM 2016 - THE 35TH ANNUAL IEEE INTERNATIONAL CONFERENCE ON COMPUTER COMMUNICATIONS, 2016,
  • [33] Research on the Performance of an End-to-End Intelligent Receiver with Reduced Transmitter Data
    Wang, Mingbo
    Wang, Anyi
    Zhang, Yuzhi
    Chai, Jing
    APPLIED SCIENCES-BASEL, 2022, 12 (22):
  • [34] Data-flow in mobile edge computing networks: end-to-end performance analysis using stochastic network calculus
    Zhu Xiaorong
    Jing Chuanfang
    Shi Jindou
    Wang Yong
    Ho Chifong
    The Journal of China Universities of Posts and Telecommunications, 2022, (01) : 81 - 92
  • [35] Data-flow in mobile edge computing networks: end-to-end performance analysis using stochastic network calculus
    Xiaorong Z.
    Chuanfang J.
    Jindou S.
    Yong W.
    Chifong H.
    Journal of China Universities of Posts and Telecommunications, 2022, 29 (01): : 81 - 92
  • [36] End-to-End Event Classification of High-Energy Physics Data
    Andrews, M.
    Paulini, M.
    Gleyzer, S.
    Poczos, B.
    18TH INTERNATIONAL WORKSHOP ON ADVANCED COMPUTING AND ANALYSIS TECHNIQUES IN PHYSICS RESEARCH (ACAT2017), 2018, 1085
  • [37] HIDE & SEEK: End-to-end packages to simulate and process radio survey data
    Akeret, J.
    Seehars, S.
    Chang, C.
    Monstein, C.
    Amara, A.
    Refregier, A.
    ASTRONOMY AND COMPUTING, 2017, 18 : 8 - 17
  • [38] End-to-End Performance Evaluation in High-Speed Wireless Networks
    Vega, Maria Torres
    Zou, Shihuan
    Mocanu, Decebal Constantin
    Tangdiongga, Eduward
    Koonen, A. M. J.
    Liotta, Antonio
    2014 10TH INTERNATIONAL CONFERENCE ON NETWORK AND SERVICE MANAGEMENT (CNSM), 2014, : 344 - 347
  • [39] Orchestrating real-time IoT workflows in a fog computing environment utilizing partial computations with end-to-end error propagation
    Georgios L. Stavrinides
    Helen D. Karatza
    Cluster Computing, 2021, 24 : 3629 - 3650
  • [40] Orchestrating real-time IoT workflows in a fog computing environment utilizing partial computations with end-to-end error propagation
    Stavrinides, Georgios L.
    Karatza, Helen D.
    CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, 2021, 24 (04): : 3629 - 3650
12345