R3S: RDMA-based RDD Remote Storage for Spark

In the past few years, Spark has seen rapid adoption by organizations that require high-performance data processing. This growth stems from its significant performance improvement over past frameworks such as Hadoop and MapReduce. Spark achieves its performance improvement by aggressively storing intermediate computation in memory in order to avoid slow disk accesses. However, Spark's performance advantage disappears when Spark nodes do not have enough local memory to store the intermediate computations, requiring results to be written to disk or recomputed. In this paper, we introduce R3S, an RDMA-based in-memory RDD storage layer for Spark. R3S leverages high-bandwidth networks and low-latency one-sided RDMA operations to allow Spark nodes to efficiently access intermediate output from a remote node. R3S can use this flexibility to treat the memory available on the different machines in the cluster as a single memory pool. This enables more efficient use of memory and can reduce job completion time. R3S can also work together with the cluster manager to add storage-only nodes to the Spark cluster. This can benefit workloads where adding memory has a far larger impact on performance than adding processing units. Our prototype reduces job completion time by 29% compared to Spark using Tachyon as the RDD storage layer on a machine learning benchmark.