Raw Data Processing Framework for IoT

The Internet of Things IoT sensors continuously generate large amount of data. Storing and managing the streaming IoT sensor data is a great challenge. IoT applications require real-time analytical query processing on stored data. To make any IoT data queriable, it needs to be loaded into any database management system traditionally. Where the high data insertion time DIT becomes the first bottle-neck. Storing data in its raw format is the most convenient solution for scientific experiments and variety of other applications where high data velocity and unknown or varying schemas exist in the approaching data. Therefore raw data storage and management techniques need to be explored to utilize very low DIT for handling streaming IoT data. Most of the work with raw data processing is dealing with the static data and how to improve query performance on the stored data. This work tries to explore possibilities in applying raw data processing technique to streaming IoT sensor data. A Raw data processing framework is proposed to handle the streaming IoT sensor data. The work tries to exploit the open source system setup to handle the streaming data. The experimental setup consists of PostgresRAW PgRAW a first mature NoDB system. Results of PgRAW shown 99.5% improvement in DIT as compared to PostgreSQL PgSQL. While the query execution time QET performance improved by 57% over the data scaling experiment from 1M to 4M for PgRAW over PgSQL.