Automatic Parallelization of Python']Python Programs for Distributed Heterogeneous Computing

被引:2
|
作者
Shirako, Jun [1 ]
Hayashi, Akihiro [1 ]
Paul, Sri Raj [2 ]
Tumanov, Alexey [1 ]
Sarkar, Vivek [1 ]
机构
[1] Georgia Inst Technol, Atlanta, GA 30332 USA
[2] Intel Corp, Austin, TX 78746 USA
来源
关键词
Parallelizing compilers; !text type='Python']Python[!/text] language; Parallel computing; Heterogeneous computing; Distributed computing;
D O I
10.1007/978-3-031-12597-3_22
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
This paper introduces a new approach to automatic ahead-of-time (AOT) parallelization and optimization of sequential Python programs for execution on distributed heterogeneous platforms. Our approach enables AOT source-to-source transformation of Python programs, driven by the inclusion of type hints for function parameters and return values. These hints can be supplied by the programmer or obtained by dynamic profiler tools; multi-version code generation guarantees the correctness of our AOT transformation in all cases. Our compilation framework performs automatic parallelization and sophisticated high-level code optimizations for the target distributed heterogeneous hardware platform. It introduces novel extensions to the polyhedral compilation framework that unify user-written loops and implicit loops present in matrix/tensor operators, as well as automated selection of CPU vs. GPU code variants. Finally, output parallelized code generated by our approach is deployed using the Ray runtime for scheduling distributed tasks across multiple heterogeneous nodes in a cluster, thereby enabling both intra-node and inter-node parallelism. Our empirical evaluation shows significant performance improvements relative to sequential Python in both single-node and multi-node experiments, with a performance improvement of over 20,000x when using 24 nodes and 144 GPUs in the OLCF Summit supercomputer for the Space-Time Adaptive Processing (STAP) radar application.
引用
收藏
页码:350 / 366
页数:17
相关论文
共 50 条
  • [1] Scaling Astroinformatics: Python']Python plus Automatic Parallelization
    Mueller, Stefan C.
    Alonso, Gustavo
    Csillaghy, Andre
    [J]. COMPUTER, 2014, 47 (09) : 41 - 47
  • [2] Parallel distributed computing using Python']Python
    Dalcin, Lisandro D.
    Paz, Rodrigo R.
    Kler, Pablo A.
    Cosimo, Alejandro
    [J]. ADVANCES IN WATER RESOURCES, 2011, 34 (09) : 1124 - 1139
  • [3] Python']Python Programmers Have GPUs too Automatic Python']Python Loop Parallelization with Staged Dependence Analysis
    Jacob, Dejice
    Trinder, Phil
    Singer, Jeremy
    [J]. PROCEEDINGS OF THE 15TH ACM SIGPLAN INTERNATIONAL SYMPOSIUM ON DYNAMIC LANGUAGES (DLS '19), 2019, : 42 - 54
  • [4] Automatic Optimization of Python']Python Skeletal Parallel Programs
    Loulergue, Frederic
    Philippe, Jolan
    [J]. ALGORITHMS AND ARCHITECTURES FOR PARALLEL PROCESSING (ICA3PP 2019), PT I, 2020, 11944 : 183 - 197
  • [5] Parallelization in Python
    Khot, Tejas
    [J]. XRDS: Crossroads, 2017, 23 (03): : 56 - 58
  • [6] Playdoh: A lightweight Python']Python library for distributed computing and optimisation
    Rossant, Cyrille
    Fontaine, Bertrand
    Goodman, Dan F. M.
    [J]. JOURNAL OF COMPUTATIONAL SCIENCE, 2013, 4 (05) : 352 - 359
  • [7] Python']Python Code Parallelization, Challenges and Alternatives
    Gonzalez, Justo
    Taylor, Julian
    Castro, Sandra
    Kern, Jeff
    Knudstrup, Jens
    Zampieri, Stefano
    Manning, Alisdair
    Bhatnagar, Sanjay
    Davis, Lindsey
    Golap, Kumar
    Jacobs, Jim
    Nakazato, Takeshi
    Petry, Dirk
    Pokorny, Martin
    Rao, Urvashi
    Robnett, James
    Schiebel, Darrell
    Sugimoto, Kanako
    Tsutsumi, Takahiro
    Wells, Akeem
    Williams, Stewart J.
    [J]. ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXVI, 2019, 521 : 515 - 518
  • [8] Enabling Python']Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs
    Amela, Ramon
    Ramon-Cortes, Cristian
    Ejarque, Jorge
    Conejero, Javier
    Badia, Rosa M.
    [J]. PROCEEDINGS OF PYHPC'17: 7TH WORKSHOP ON PYTHON FOR HIGH-PERFORMANCE AND SCIENTIFIC COMPUTING, 2017,
  • [9] Scientific computing with python']python
    Beazley, DM
    [J]. ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS IX, 2000, 216 : 49 - 58
  • [10] Python']Python for scientific computing
    Oliphant, Travis E.
    [J]. COMPUTING IN SCIENCE & ENGINEERING, 2007, 9 (03) : 10 - 20