Hybrid in-network computing and distributed learning for large-scale data processing

Recently, the value of big data generated, processed, and shared by mobile devices and intelligent Internet of Things devices has significantly increased. The speed of data generation is accelerating, and the amount of data that needs to be collected and learned in a distributed database is increasing. However, the conventional data processing methods, centralized and distributed computing, are reaching their limits owing to the trade-off between data processing speed and learning accuracy. To solve this problem, in this study, a hybrid in -network computing (HINC) method that can process data quickly and accurately is proposed. The efficiency of the proposed HINC method was verified by comparing and analyzing its data loss, throughput, and accuracy to those of the conventional method. The experimental results show that in an environment with many nodes in the network, HINC reduces the data loss, which means data that could not be processed because it exceeded the available throughput of the node by 20.94% compared to the conventional method. Furthermore, in an envi-ronment with full workload, HINC reduced data loss by an average of 55.6% compared with the conventional centralized computation method. It also improved throughput by an average of 18.91% and model evaluation accuracy by 15.7% compared with the conventional fully distributed computing model.