Network music big data platform storage system construction

To build the network music big data platform, We need to collect a lot of music resources from the Internet, And store it on big data platforms. Therefore, it is a key problem to build a storage system that is high performance, scalable and capable of supporting big data. The network music data is very large and the metadata structure is not consistent, HBase and other distributed databases are widely used to build it. However, single RowKey index query in HBase is inefficient, it is difficult to meet the demand of big data platform for real-time or quasi-real-time query and statistical analysis. To this end, this paper proposes a solution, First, we migrated to HBase through Sqoop, which stored the music resources stored in multiple relational databases; Then use Phoenix to establish a secondary index query mechanism, build the Phoenix + HBase storage structure, and store management of the heterogeneous music resources; Finally, the read-write performance and query performance of the storage architecture are tested in the Spark parallel computing framework. Compared with the experimental results, the efficiency of Phoenix + HBase query performance is significantly higher than that of single computer and direct call to HBase, and it can operate unstructured data and structured data, which is in line with the requirement of the storage system of music big data platform.