Performance analysis and data reduction for exascale scientific workflows

被引：0
|
作者：
Kelly, Christopher [1 ,2 ]
Xu, Wei [1 ,3 ]
Pouchard, Line C. [4 ]
Van Dam, Hubertus [1 ,2 ]
Islam, Tanzima Z. [5 ]
Yoo, Shinjae [1 ,3 ]
Van Dam, Kerstin Kleese [1 ,6 ]
机构：
[1] Brookhaven Natl Lab, Upton, NY USA
[2] Computat Sci Dept, Computat Sci Initiat, Upton, NY USA
[3] Facebook Artificial Intelligence Res, New York, NY USA
[4] Ctr Comp Res, Sandia Natl Labs, Albuquerque, NM USA
[5] Texas State Univ, Dept Comp Sci, San Marcos, TX USA
[6] Natl Secur Directorate, Data Sci & Analyt Grp, Richland, WA 99354 USA
来源：
INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS | 2025年
关键词：
Chimbuko; in-situ data reduction; scientific workflows; performance instrumentation; scalability; ANOMALY DETECTION; SYSTEM; FRAMEWORK; NWCHEM;
D O I：
10.1177/10943420251316253
中图分类号：
TP3 [计算技术、计算机技术];
学科分类号：
0812 ;
摘要：
Chimbuko is the first in situ, scalable, workflow-level performance analysis tool for trace-level analysis and visualization of application performance. This tool was developed by the Co-design Center for Online Data Analysis and Reduction and funded by the U.S. Department of Energy's Exascale Computing Project. We provide a detailed description of Chimbuko's architecture and illustrate our online and offline visualization with multiple use cases. We also present results for the deployment and scalability of the tool as applied to a high-energy physics workflow running at large scale on the Frontier supercomputer.
引用
收藏
页数：26
相关论文
共 50 条
  • [1] Outcome-Preserving Input Reduction for Scientific Data Analysis Workflows
    Anh Duc Vu
    Kehrer, Timo
    Tsigkanos, Christos
    PROCEEDINGS OF THE 37TH IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING, ASE 2022, 2022,
  • [2] End-to-end online performance data capture and analysis for scientific workflows
    Papadimitriou, George
    Wang, Cong
    Vahi, Karan
    da Silva, Rafael Ferreira
    Mandal, Anirban
    Liu, Zhengchun
    Mayani, Rajiv
    Rynge, Mats
    Kiran, Mariam
    Lynch, Vickie E.
    Kettimuthu, Rajkumar
    Deelman, Ewa
    Vetter, Jeffrey S.
    Foster, Ian
    FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2021, 117 : 387 - 400
  • [3] End-to-end online performance data capture and analysis for scientific workflows
    Papadimitriou, George
    Wang, Cong
    Vahi, Karan
    da Silva, Rafael Ferreira
    Mandal, Anirban
    Liu, Zhengchun
    Mayani, Rajiv
    Rynge, Mats
    Kiran, Mariam
    Lynch, Vickie E.
    Kettimuthu, Rajkumar
    Deelman, Ewa
    Vetter, Jeffrey S.
    Foster, Ian
    Future Generation Computer Systems, 2021, 117 : 387 - 400
  • [4] ExaWorks: Workflows for Exascale
    Al-Saadi, Aymen
    Ahn, Dong H.
    Babuji, Yadu
    Chard, Kyle
    Corbett, James
    Hategan, Mihael
    Herbein, Stephen
    Jha, Shantenu
    Laney, Daniel
    Merzky, Andre
    Munson, Todd
    Salim, Michael
    Titov, Mikhail
    Turilli, Matteo
    Uram, Thomas D.
    Wozniak, Justin M.
    PROCEEDINGS OF 16TH WORKSHOP ON WORKFLOWS IN SUPPORT OF LARGE-SCALE SCIENCE (WORKS21), 2021, : 50 - 57
  • [5] A Codesign Framework for Online Data Analysis and Reduction at the Exascale
    Mehta, Kshitij
    Allen, Bryce
    Wolf, Matthew
    Logan, Jeremy
    Suchyta, Eric
    Choi, Jong
    Takahashi, Keichi
    Yakushin, Igor
    Munson, Todd
    Foster, Ian
    Klasky, Scott
    PROCEEDINGS OF WORKS19: THE 2019 14TH IEEE/ACM WORKFLOWS IN SUPPORT OF LARGE-SCALE SCIENCE (WORKS), 2019, : 11 - 20
  • [6] OpenAlea: Scientific Workflows Combining Data Analysis and Simulation
    Pradal, Christophe
    Fournier, Christian
    Valduriez, Patrick
    Cohen-Boulakia, Sarah
    PROCEEDINGS OF THE 27TH INTERNATIONAL CONFERENCE ON SCIENTIFIC AND STATISTICAL DATABASE MANAGEMENT, 2015,
  • [7] Online performance monitoring and analysis of grid scientific workflows
    Truong, HL
    Fahringer, T
    ADVANCES IN GRID COMPUTING - EGC 2005, 2005, 3470 : 1154 - 1164
  • [8] Data reduction in scientific workflows using provenance monitoring and user steering
    Souza, Renan
    Silva, Vitor
    Coutinho, Alvaro L. G. A.
    Valduriez, Patrick
    Mattoso, Marta
    FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2020, 110 (110): : 481 - 501
  • [9] Exacution: Enhancing Scientific Data Management for Exascale
    Klasky, Scott
    Suchyta, Eric
    Ainsworth, Mark
    Liu, Qing
    Whitney, Ben
    Wolf, Matthew
    Choi, Jong
    Foster, Ian
    Kim, Mark
    Logan, Jeremy
    Mehta, Kshitij
    Munson, Todd
    Ostrouchov, George
    Parashar, Manish
    Podhorszki, Norbert
    Pugmire, David
    Wan, Lipeng
    2017 IEEE 37TH INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS (ICDCS 2017), 2017, : 1927 - 1937
  • [10] Dynamic instrumentation, performance monitoring and analysis of Grid scientific workflows
    Truong H.-L.
    Fahringer T.
    Dustdar S.
    Journal of Grid Computing, 2005, 3 (1-2) : 1 - 18
12345