A Data Cleaning Method on Massive Spatio-Temporal Data

In open conditions of Internet of Things, massive data would be rapidly accumulated from sensors in low quality. On huge size raw data, the correction for consistency is time-consuming and inaccurate to achieve, and the validation for legality is difficult to guarantee without prior knowledge. In this paper, time-based clustering and rule-based filtering for data cleaning is proposed on massive bus IC card data, which guarantees the consistency and legality among spatio-temporal attributes. Implemented through Hadoop MapReduce and evaluated on real data set, our method shows its efficiency and accuracy in extensive conditions.