Hadoop-Based Power Grid Data Quality Verification and Monitoring Method

In recent years, with the continuous improvement of automation and information of the power grid industry, the scale of power grid data is also expanding. Power grid data quality monitoring is the most important link in the power grid big data processing business. The large-scale power grid data makes the data quality monitoring technology currently used in the power grid industry difficult to meet the needs of daily production management and business decision-making. Big data technology provides good technical means and support platforms for solving power grid big data processing. In this paper, a data quality monitoring and verification scheme for the power grid using a big data platform is proposed, and distributed data storage management and parallel verification rule execution technology based on the Hadoop platform is studied. Different data verification index mechanisms and parallel verification algorithms are designed, and verification research is carried out for typical scenarios of full data and incremental data quality verification. For the full amount of data, the HDFS-based full table data index and the MapReduce-based parallel verification rules execution algorithm are used, and for the incremental data, HBase is used to complete the data storage, indexing, and parallel verification. On this basis, a verification system is designed based on the verification data set and verification rules. Through testing the marketing table and GIS table data set, it shows that the proposed technical method effectively improves the data quality verification processing performance, greatly shortens the data verification time, meets the needs of real-time power grid data verification, and provides a system solution with good scalability.