Incremental bloom filters

A Bloom filter is a randomized data structure for performing approximate membership queries. It is being increasingly used in networking applications ranging front security to routing, in peer to peer networks. In order to meet a given false positive rate, the amount of memory required by it bloom filter is it function of the number of elements in the set. We consider the problem of minimizing the memory requirements in cases where the number elements in the set is not known in advance but the distribution or moment information of the number of elements is known. We show how to exploit such information too minimize the expected amount of memory required For the filter. We also show how this approach can significantly reduce memory requirement when bloom filters are constructed For multiple sets in parallel. We show analytically as well as experiments on synthetic and trace data that our approach leads to one to three orders of magnitude reduction in memory compared to it standard Bloom filter.