Detecting Anomaly in Large-scale Network using Mobile Crowdsourcing

In this paper, we propose a tree modeling-based data mining method to detect anomalies from crowdsourced network data. We design an algorithm to extract potential network anomalies from decision trees. Moreover, we propose a criteria to evaluate the severity of anomaly in terms of three factors: standard deviation, weight sum and impurity decrease. To enhance generalization performance, we randomly generate sample subspace of the original dataset as the input for each subtree and compact detected anomalies from all subtrees. We carry out experiments based on the crowdsourced network measurement dataset containing live million samples, which contains round trip time (RTT) from more than 5,000 users. Experiments show that the proposed method can effectively detect high-latency network anomalies. Moreover, the random forest-based approach can achieve an improvement of approximately 25% of generalization performance compared to the single decision tree approach.