StorageFlow: SDN-Enabled Efficient Data Regeneration for Distributed Storage Systems

Distributed Storage Systems (DSS) have seen increasing levels of deployment in data centers and in cloud storage provider networks. They provide efficient and cost-effective ways to store large amounts of data and ensure resilience to failures by using mirroring and coding schemes at the block and file level. While mirroring techniques provide an efficient way to recover lost data, they require excessive amounts of storage space. Coding techniques provide equivalent resilience while reducing the amount of required storage space. However, the current recovery process for coded data is not efficient due to the need to transfer large amounts of data to regenerate the data lost as a result of a failure. This contributes to large delays and excessive amount of network traffic resulting in significant performance bottlenecks. In this paper, we propose a new architecture for efficient data regeneration in distribution storage systems. The key idea of our framework is to enable network switches to perform network coding operations, i.e., combine multiple packets received over incoming links and forward the resulting packets towards the destination. Our framework includes a transport-layer reverse multicast protocol that takes advantage of network coding to minimize the amount of transferred data. The new architecture is implemented using the principles of Software Defined Networking (SDN). To enable the switches to perform network coding operations, we extend the SDN packet processing pipeline using a principled abstract model that requires minimum changes to the existing SDN frameworks. Our testbed experiments show that the proposed architecture results in significant performance gains.