Research on Real-time Analysis and Hybrid Encryption of Big Data

With the increasing of massive multi-source heterogeneous big data, the higher demand on the speed of data analysis has been raised. Based on the analysis of the existing problems of Hadoop MapReduce and Apache Spark, this paper proposes to use Spark Streaming for real-time processing of massive stream data, and builds a prototype cluster, which proves the real-time analysis of log data (online log and offline log) and the real-time processing of network attack data. At the same time, a hybrid encryption method combining ECC encryption with block data encryption and a secret key generation method combining semiconductor noise source and chaotic sequence are proposed. The simulation results show that the efficiency and security of encryption are effectively guaranteed.