An Outlier Detection Algorithm Based on the Degree of Sharpness and Its Applications on Traffic Big Data Preprocessing

Outlier detection is one important research area of data mining, which plays key roles in data preprocessing, equipment fault diagnosis, credit fraud detection, traffic incident detection etc. This paper is devoted to a new outlier detection algorithm based on the degree of sharpness. The proposed algorithm takes a new way to solve the outlier detection problem, which employs a measure in image processing, degree of sharpness, to detect the outliers. Compared to the classical outlier detection methods with statistical learning, the proposed algorithm has no iterative processes. It generates a smooth curve to describe the overall distribution of the data firstly, and then computes the sharpness of degree for each data point. Finally, the outliers are recognized as they have larger values of the degree of sharpness. Also, some practical applications on traffic big data are presented to prove the effectiveness of the proposed algorithm.