A compiler driven out-of-core programming approach for optimizing data locality in loop nests

Most scientific programs have large input and output data sets that require out-of-core programming or the use of virtual memory management (VMM). Often, VMM is not an effective approach because it results frequently in substantial performance reduction. In contrast, compiler driven I/O management allows a program's data sets to be retrieved explicitly in parts, called blocks or tiles. In this paper, we offer an out-of-core programming schema to optimize locality of disk accesses in stepped problems by choosing a good combination of data tiling and loop transformations. The experimental results pro-Ode strong evidence that an out-of-core programming approach combined with non-standard data mappings and corresponding loop transformations can improve the performance of out-of-core problems by one order of magnitude or more.