Evaluating and Optimizing Parallel LU-SGS for High-order CFD simulations on the Tianhe-2 supercomputer

The inherent strong data dependency of LU-SGS poses tough challenges for shared-memory parallelization. The popular pipeline solution for parallel LU-SGS in CFD, achieves impressive parallel scalability on early multi-core processors. However, recent experiences show that the scalability of pipeline LU-SGS drops dramatically on emerging many-core processors such as Xeon Phi due to high startup and emptying overheads and severe load imbalance. We discover that increasingly large pipeline depth tremendously hinder the applicability of pipeline LU-SGS in realistic parallel CFD simulations on many-core processors. Aiming at alleviating these performance issues, we propose a novel improved pipeline LU-SGS algorithm, which organizes threads hierarchically using nested OpenMP to construct a sub-pipeline in each original pipeline stage to further exploit LU-SGS's parallelism. We implement and evaluate it in our in-house high-order CFD software HOSTA on Xeon and Xeon Phi. For a given 256 x 256 x 256 workload, improved method achieves over 20% performance gains on Xeon Phi than traditional pipeline approach, a further 38% performance boost are observed on Xeon Phi when varies the dimension sizes. Related problems in realistic CFD simulations such as domain decomposition and algorithmic parameter tuning are also discussed. Generally, our work is applicable to all Gauss-Seidel like methods with intrinsic strong data dependency.