Window-based multiple continuous query algorithm for data streams

As more data are being collected and analyzed in real time, stream processing is attracting greater attention. In traditional network, economic and financial analysis, or processing of sensors data in Internet of Things, efficient and timely methods of handling continuous data generation are required. Data are being produced at higher frequency, and the volume of data to be processed within a particular period of time is increasing rapidly. This is especially true for continuous window aggregation, which involves intensive computation. An increase in the number of query windows also generates scalability problems involving aggregate queries. Traditional query handling algorithms can perform many repeated operations. In this study, to enable window-based shared processing of continuous queries, a window reuse algorithm for a multi-query environment based on pace and results is proposed: the multiple continuous query algorithm (MCQA). The aggregation is simplified by gradually shrinking the set of multiple values so that the operation is reduced at each step, eventually achieving result sharing. The algorithm is implemented with the Storm stream processing framework. Experiments prove that the MCQA performance is more efficient and effectively reduces memory usage.