Multi-source Integration and Storage Optimization Method for Big Data of Power Distribution and Utilization

被引：0
|
作者：
Wang L. [1 ]
Zhao T. [1 ]
Zhang Y. [1 ]
Su Y. [2 ]
Tian S. [3 ]
机构：
[1] Department of Electrical Engineering, Shanghai Jiao Tong University, Shanghai
[2] Electric Power Research Institute, Shanghai Municipal Electric Power Company of State Grid, Shanghai
[3] China Electric Power Research Institute, Beijing
来源：
Gaodianya Jishu/High Voltage Engineering | 2018年 / 44卷 / 04期
关键词：
Big data of power distribution and utilization; Data integration; Hadoop; Hash bucket storage; Parallel association query;
D O I：
10.13336/j.1003-6520.hve.20180329012
中图分类号：
学科分类号：
摘要：
In the face of massive, heterogeneous and fast growing big data of power distribution and utilization, how to apply big data technology to improve the breadth, depth, and accuracy of power distribution and utilization business becomes a new opportunity and challenge in power industry. In order to solve the two major problems of big data with the multi-source integration and efficient storage, according to the composition and characteristics of big data of power distribution and utilization,we adopt standardized metadata and corresponding data dictionaries to realize the standardized integration of multi-source data of power distribution and utilization. On the basis of data integration, the optimization method of big data storage is studied based on the Hadoop platform. A Hash bucket algorithm considering data correlation is proposed. The algorithm realizes the centralized storage of related data, so as to enhance the efficiency of data query and processing. On the basis of data storage optimization, the parallel association query for multi-source big data of power distribution and utilization based on MapReduce is realized. Tests on a Hadoop platform show that, after optimization of hash bucket storage, the time of the multi-source data parallel association query is significantly shortened than traditional Hadoop method. © 2018, High Voltage Engineering Editorial Department of CEPRI. All right reserved.
引用
收藏
页码：1131 / 1139
页数：8
相关论文
共 26 条
  • [1] Song Y., Zhou G., Zhu Y., Present status and challenges of big data processing in smart grid, Power System Technology, 37, 4, pp. 927-935, (2013)
  • [2] Zhang D., Miao X., Liu L., Et al., Research on development strategy for smart grid big data, Proceedings of the CSEE, 35, 1, pp. 2-12, (2015)
  • [3] Managing big data for smart grids and smart meters[R/OL]
  • [4] Liu K., Sheng W., Zhang D., Et al., Big data application requirements and scenario analysis in smart distribution network, Proceedings of the CSEE, 35, 2, pp. 287-293, (2015)
  • [5] Big data challenges for the grid: EPRI survey results and analysis Data analytics and applications demonstration newsletter[R/OL]
  • [6] Peng X., Deng D., Cheng S., Et al., Key technologies of electric power big data and its application prospects in smart grid, Pro-ceedings of the CSEE, 35, 3, pp. 503-511, (2015)
  • [7] Zhang S., Mao D., Wang B., Application of big data processing technology in fault diagnosis and early warning of wind turbine gearbox, Automation of Electric Power Systems, 40, 14, pp. 129-134, (2016)
  • [8] Gu S., Wang J., Feng W., Et al., Statistical and mining analysis of lightning detection data in power grid, High Voltage Engineering, 42, 11, pp. 3383-3391, (2016)
  • [9] Yan Y., Sheng G., Chen Y., Et al., An method for anomaly detection of state information of power equipment based on big data analysis, Proceedings of the CSEE, 35, 1, pp. 52-59, (2015)
  • [10] Mo W., Xu Z., Xiao F., Et al., Research on temporal-spatial correlation of power disturbance events based on random matrix theory, High Voltage Engineering, 43, 7, pp. 2386-2393, (2017)
123