On the Relaxed Synchronization for Massively Parallel Numerical Algorithms

This paper presents a novel relaxed synchronization strategy for generic numerical algorithms executed in distributed and parallel computing systems. Large problems are efficiently solved if they can be parallelized. However, as the number of processing elements increases, the communication, necessary to synchronize intermediate computation across processing elements, increases and soon becomes a serious bottleneck. This is a critical concern if future multicore machines are to be useful for scientific computing. In this paper, we analyze the convergence of numerical algorithms in the dynamical system framework and introduce a relaxed synchronization technique. This reduces the synchronization bottleneck through periodic communications across processing elements. Instead of synchronizing after every iterations, the proposed framework synchronizes at a certain period. We provide the condition to determine an appropriate synchronization period. It is shown that with this relaxation, the numerical algorithm converges faster to the same fixed-point value than the conventional implementation. The validity and efficiency of the proposed algorithm is verified by numerical example.