A performance study of two-phase I/O

Massively parallel computers are increasingly being used to solve large, I/O intensive applications in many different fields. For such applications, the I/O subsystem represents a significant obstacle in the way of achieving good performance. While massively parallel architectures do, in general, provide parallel I/O hardware, this alone is not sufficient to guarantee good performance. The problem is that in many applications each processor initiates many small I/O requests rather than fewer larger requests, resulting in significant performance penalties due to the high latency associated with I/O. However, it is often the case that in the aggregate the I/O requests are significantly fewer and larger. Two-phase I/O is a technique that captures and exploits this aggregate information to recombine I/O requests such that fewer and larger requests are generated, reducing latency and improving performance. In this paper, we describe om efforts to obtain high performance using two-phase I/O. In particular, we describe our first implementation which produced a sustained bandwidth of 78 MBytes per second, and discuss the steps taken to increase this bandwidth to 420 MBytes per second.