Shard Level Transaction Based Cluster Management for Online Distributed Storage

Nowadays, distributed systems have been used everywhere, some for online applications, such as Apache HBase [1] and Cassandra [2], and some for offline applications, such as HDFS [3] and GFS. These systems achieve good result and largely reduce the cost. Due to the fact that distributed system is generally composed of lots of machines and failures occur recurrently, cluster management module is a must to handle these failures, thus provides availability and scalability. Low latency and high read and write performance is needed in many circumstances, such as in search engine and game field. Distributed storage for these systems need special ways to do cluster management. In this paper, a shard-level transaction based cluster management solution is proposed to handle such systems, and it can be easily extended.