Multicast-based Replication for Hadoop HDFS

The Hadoop HDFS is a popular open-source distributed storage system, which serves as the foundation of many important big-data technologies. The performance of data replication is crucial to HDFS, since it accounts for a major portion of network traffic in the entire cluster. In this research, we propose to enable multicast-based replication, which is expected to use less network bandwidth than the native TCP-based pipelined replication method. We developed a congestion-controlled reliable multicast socket (the CCRMSocket) for HDFS and evaluated its performance with our multi-rack test platform. The experimental result shows that our multicast implementation can effectively save bandwidth and peacefully coexist with TCP traffic. We also developed a simulator (the HFlowSim) to further study the impact of multicast-based replication to a large-scale Hadoop system. The simulation result suggests that multicast-based replication can systematically improve a Hadoop system by accelerating the big jobs.