Big Data Processing on Volunteer Computing

In order to calculate the node big data contained in complex networks and realize the efficient calculation of complex networks, based on voluntary computing, taking ICE middleware as the communication medium, the loose coupling distributed framework DCBV based on voluntary computing is proposed. Then, the Master, Worker, and MiddleWare layers in the framework, and the development structure of a DCBV framework are designed. The task allocation and recovery strategy, message passing and communication mode, and fault tolerance processing are discussed. Finally, to calculate and verify parameters such as the average shortest path of the framework and shorten calculation time, an improved accurate shortest path algorithm, the N-SPFA algorithm, is proposed. Under different datasets, the node calculation and performance of the N-SPFA algorithm are explored. The algorithm is compared with four approximate shortest-path algorithms: Combined Link and Attribute (CIA), Lexicographic Breadth First Search (LBFS), Approximate algorithm of shortest path length based on center distance of area division (CDZ), and I Iub Vertex of area and Core Expressway (I IEA-CE). The results show that when the number of CPU threads is 4, the computation time of the DCBV framework is the shortest (514.63 ms). As the number of CPU cores increases, the overall computation time of the framework decreases gradually. For every 2 additional CPU cores, the number of tasks increases by 1. When the number of Worker nodes is 8 and the number of nodes is 1, the computation time of the framework is the shortest (210,979 ms), and the JO statistics data increase with the increase of Worker nodes. When the datasets are Undirected01 and Undirected02, the computation time of the N-SPFA algorithm is the shortest, which is 4520 ms and 7324 ms, respectively. I Iowever, the calculation time in the ca-condmat_undirected dataset is 175,292 ms, and the performance is slightly worse. Overall, however, the performance of the N-SPFA and SPFA algorithms is good. Therefore, the two algorithms are combined. For networks with less complexity, the computational scale coefficient of the SPFA algorithm can be set to 0.06, and for general networks, 0.2. When compared with other algorithms in different datasets, the pretreatment time, average query time, and overall query time of N-SPFA algorithm are the shortest, being 49.67 ms, 5.12 ms, and 94,720 ms, respectively. The accuracy (1.0087) and error rate (0.024) are also the best. In conclusion, voluntary computing can be applied to the processing of big data, which has a good reference significance for the distributed analysis of large-scale complex networks.