An Efficient and Metadata-Aware Big Data Storage Architecture

This paper introduces a hash partitioning-based file compaction design to improve the efficiency of storing and accessing small files in big data storage systems. The proposed approach consists of a file compaction tool and an access interface. The compaction tool merges a group (usually a directory) of small files into a set of "big files" to reduce the metadata required to be maintained in the on-chip memory. The data locality and tree structure of those small files are preserved. The access interface is designed to provide transparent access to the small files in the big files. Experimental results confirm that the proposed approach lead to a significantly enhancement in terms of namespace usage and access speed.