Machine learning based network intrusion detection for data streaming IoT applications

In recent years, Internet of Things (IoT) technologies have been widely used in many fields such as surveillance, healthcare, smart metering and environment monitoring. This extensive usage leads to massive data management and a complexity in data analysis. A huge number of IoT sensors are deployed for monitoring task and send continuously their collected data to gateways. IoT applications are analyzing these data flows and making real time decisions about specific monitored events (fire, flood, terrorist attacks, etc.). Anomalies that may be related to sensor failures or network intrusions are affecting such decisions. Therefore, they should be detected and eliminated as soon as they arrive. This task requires real time data processing detectors for making accurate and fast predictions. In this paper, we design an architecture for a real time network intrusion detection system for IoT streaming data. The system was developed, deployed and tested with the two leading stream processing frameworks (Apache fink and Apache Spark Streaming). We used two different public datasets and different machine learning algorithms. Results show considerable throughputs and high detection accuracy especially for Apache Flink.