A new replica placement strategy based on multi-objective optimisation for HDFS

Distributed storage systems like the Hadoop distributed file system (HDFS) constitute the core infrastructure of cloud platforms which are well poised to deal with big-data. An optimised HDFS is critical for effective data management in terms of reduced file service time and access latency, improved file availability and system load balancing. Recognising that the file-replication strategy is key to an optimised HDFS, this paper focuses on the file-replica placement strategy while simultaneously considering storage and network load. Firstly, the conflicting relationship between storage and network load is analysed and a bi-objective optimisation model is built, following which a multi-objective optimisation memetic algorithm based on decomposition (MOMAD) and its improved version are used. Compared to the default strategy in HDFS, the file-replica placement strategies based on multi-objective optimisation provide more diverse solutions. And competitive performance could be obtained by the proposed algorithm.