NEPTUNE: Real Time Stream Processing for Internet of Things and Sensing Environments

Improvements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing environments. Data streams generated in such settings must keep pace with generation rates and be processed in real time. Challenges in accomplishing this include: high data arrival rates, buffer overflows, context-switches, and object creation overheads. We propose a holistic framework that addresses the CPU, memory, network, and kernel issues involved in stream processing. Our prototype, Neptune, builds on our Granules cloud runtime. The framework maximizes bandwidth utilization in the presence of small messages via the use of buffering and dynamic compactions of packets based on payload entropy. Our use of thread-pools and batched processing reduces context switches and improves effective CPU utilizations. NEPTUNE alleviates memory pressure that can lead to swapping, page faults, and thrashing through efficient reuse of objects. To cope with buffer overflows we rely on flow control and throttling the preceding stages of a processing pipeline. Our benchmarks demonstrate the suitability of the Neptune and we contrast our performance with Apache Storm, the dominant stream-processing framework developed by Twitter. At a single node, we are able to achieve a processing rate of similar to 2 million stream packets per-second. In a distributed setup, we achieved a rate of similar to 100 million packets per-second.