Parallel I/O performance of fine grained data distributions

Fine grained data distributions are widely used to balance computational loads across complete processes in parallel scientific applications. When a fine grained data distribution is used in memory, performance of I/O intensive applications can be limited not only by disk speed bur also by message passing, because a large number of small messages may be generated by the implementation strategy used in the underlying parallel file system or parallel I/O library. Combining (or packetizing) a set of small messages into a large message is generally known to speed up parallel I/O. However, overall I/O performance is affected not only by small messages bur also by other factors like cyclic block size and interconnect characteristics. We describe small message combination and communication scheduling for fine grained delta distributions in the Panda parallel I/O library mid analyze I/O performance on parallel platforms having different interconnects: IBM SP2, IBM workstation cluster connected by FDDI and Pentium It cluster connected by Myrinet.