A stream processing architecture for heterogeneous data sources in the Internet of Things

The number of Internet of Things (IoT) and smart devices capable of producing, consuming and exchanging information is constantly increasing. It is estimated there will be around 30 billion of them in 2020. In most cases, the structures of the information produced by such devices are completely different, thus providing heterogeneous information. This is becoming a challenge for researchers working on IoT, who need to perform homogenisation and pre-processing tasks before using the IoT data. This paper aims to provide an architecture for processing and analysing data from heterogeneous sources with different structures in IoT scopes, allowing researchers to focus on data analysis, without having to worry about the structure of the data sources. This architecture combines the real-time stream processing paradigm for information processing and transforming, together with the complex event processing for information analysis. This provides us with capability of processing, transforming and analysing large amounts of information in real time. The results obtained from the evaluation of a real-world case study about water supply network management show that the architecture can be applied to an IoT water management scenario to analyse the information in real time. Additionally, the stress tests successfully conducted for this architecture highlight that a large incoming rate of input events could be processed without latency, resulting in efficient performance of the proposed architecture. This novel software architecture is adequate for automatically detecting situations of interest in the IoT through the processing, transformation and analysis of large amounts of heterogeneous information in real time. © 2020 Elsevier B.V.