Roda: A Flexible Framework for Real-Time On-demand Data Aggregation

It is critical to aggregate data from multiple sources to support real-time decision making in several fields, such as anti-telecommunications fraud detection. However, as data sources are distributed, heterogeneous and autonomous, it is challenging to ensure that data aggregation satisfies the requirements of real-time, on-demand and flexibility. In this paper, we propose a real-time on-demand data aggregation (Roda) framework, which is designed to be flexible enough to support the dynamic joining of new data sources, the immediate updating of aggregation rules and the quick adaptation to data velocity. We implement a prototype of Roda based on Kafka and Docker using the overlay network technique. To evaluate the effectiveness and performance of Roda, we conduct a series of experiments based on real trace data. The experiment results show that Roda can guarantee data aggregation latency at the millisecond scale, easily achieving our design goals.