On construction of a big data warehouse accessing platform for campus power usages

With the emerging of the Internet of Things (IoT) technology, we can analyze and real-time monitoring the power consumption data of buildings or equipment. However, over time, many power data will accumulate to even the terabyte level. Traditional data processing techniques will not handle. It is a challenge to monitor and process the data in a reasonable time. In this paper, we construct an efficient and real-time power monitoring platform, based on open source techniques. The power-data of buildings or equipment is provided through smart meters equipped inside campus buildings. The technologies of data collected and stored on the server side were handled by the Hadoop ecosystem and the data process to Hive data warehouse is executed by the Spark. On the particulars of system evaluation, we make some experiments to compare with the performance of record query, use three different technologies separately: Apache Hive, Apache Spark, and Impala had been evaluated regarding query-response performance. Moreover, for the performance estimation on the data ETL processing and SQL functions. The relevant experiments have been conducted on the same three software modules as well. (C) 2019 Elsevier Inc. All rights reserved.