High-Performance Serverless Data Transfer over Wide-Area Networks

We propose a serverless data movement architecture in which we bypass data transfer nodes, the filesystem stack, and the host system stack and directly move the data from one disk array controller to another in order to obtain the highest end-to-end performance. Under the current data movement architecture, separate data transfer nodes arbitrate data transfer by input/output to parallel file systems over local-area networks and parallel file systems read/write actual data from/to disks through disk controllers. Our new architecture embeds parallel file system servers and data transfer processes into a disk controller to eliminate the data path between a data transfer process and a parallel file system. This prevents the network between those two entities from being a bottleneck in end-to-end data transfer. In addition, we propose a parallel data-layout-aware data transfer, where multiple embedded data transfer processes send segments of a file concurrently while considering data layout in disks to improve data transfer performance. Our experimental results show that our proposed architecture is feasible and outperforms the traditional architecture significantly.