A Novel Distributed File System Using Blockchain Metadata

Cluster computing has become an inevitable part of data processing as the huge volume of data being produced from different sources like online social media, IoT, mobiledata, sensor data, black box data and so on increases in an exponentially fast manner. Distributed File System defines different methods to distribute, read and eliminate the files among different cluster computing nodes. It is found that popular distributed file systems such as Google File System and Hadoop Distributed File System store metadata centrally. This creates a chance for a Single Point of Failure that arises the need for backup and alternative solutions to recover the metadata on the failure of the metadata server. Also, the name node server is built using expensive and reliable hardware. For small and medium clusters, it is not cost effective to maintain expensive name node server. Even though cheap commodity hardware may substitute the name node functionality, they are prone to hardware failure. This paper proposes a novel distributed file system to distribute files over a cluster of machines connected in a Peer-to-Peer network. The most significant feature of the file system is its capability to distribute the metadata using distributed consensus, using hash values. Although the distributed metadata is visible to the public, the methodology ensures that it is immutable and irrefutable. As part of the in-depth research, the proposed file system has been successfully tested in the Google Cloud Platform. Also, the basic operations like read, write, and delete on Distributed File System with distributed metadata are compared with that of Hadoop Distributed File System based on distribution time on the same cluster setup. The novel distributed file system provides better results compared to the existing methodologies.