A Probabilistic Sketch for Summarizing Cold Items of Data Streams

Conventional sketches on counting stream item frequencies use hash functions for mapping data items to a concise structure, e.g., a two-dimensional array, at the expense of overcounting due to hashing collisions. Despite the popularity, it is still challenging to handle cold (low-frequency) items, especially when the space is limited. The cold items can be misreported as hot (high-frequency) items as the accumulation of error in hashing collisions, leading to the estimation accuracy degrading. We find that a streaming item can be split into a set of compactly stored basic elements, which can be recomposed in a probabilistic manner to estimate the frequency of an item. Thus, we design a novel decomposition and recomposition framework, called the XY-sketch, which estimates the frequency of a stream item by estimating the probability of basic elements appearing in the data stream. By improving the estimation accuracy of cold items, we show that advanced streaming queries, such as top-k queries and heavy change queries. Throughout, we conduct theoretical analysis and optimizations under space constraints. Experiments on real datasets are conducted to examine the effectiveness of our proposals.