Memory Hierarchy Aware I/O Scheduling Under Contention for Hybrid Storage Based HPC

Scientific computing involved with handling enormous computation and processing huge volume of data poses a serious challenge to high performance computing (HPC). The significant progress of computational power further widens the historical gap between processing speed and storage latency. Introducing solid-state-drives (SSDs) as a burst buffer or cache to memory hierarchy in hybrid storage based HPC can effectively improve the I/O performance. Meanwhile, increasing the number of processes to get greater parallelism also can reduce the final execution time. However, these methods are unable to adequately utilize the system resources when there is competition for system resources like SSDs, I/O network. In this paper, we first analyze the effects of I/O congestion between different applications and SSDs contention between different processes. Then, a memory hierarchy aware I/O scheduling method, which is able to detect contentions and schedules I/O to relieve this situation, is proposed to improve overall I/O performance. The theoretical analysis and extensive experiments prove that our method is effective and efficient. The results from the real-world scientific simulations show this I/O scheduling technique achieves up to 20% execution performance improvement.