Time-decaying Bloom Filters for data streams with skewed distributions

Kai Cheng, Limin Xiang, Haiyan Xu, Mizuho Iwaihara, Mukesh M. Mohania

Research output: Contribution to conferencePaper

37 Citations (Scopus)

Abstract

Bloom Filters are space-efficient data structures for membership queries over sets. To enable queries for multiplicities of multi-sets, the bitmap in a Bloom Filter is replaced by an array of counters whose values increment on each occurrence. In a data stream model, however, data items arrive at varying rates and recent occurrences are often regarded as more significant than past ones. In most data stream applications, it is critical to handle this "time-sensitivity". Furthermore, data streams with skewed distributions are common in many emerging applications, e.g., traffic engineering and billing, intrusion detection, trading surveillance and outlier detection. For such applications, it is inefficient to allocate counters of uniform size to all buckets. In this paper, we present Time-decaying Bloom Filters (TBF), a Bloom Filter that maintains the frequency count for each item in a data stream, and the value of each counter decays with time. For data streams with highly skewed distributions, we proposed further optimization by allowing dynamically allocating free counters to the "large" items. We performed preliminary experiments to verify the optimization.

Original languageEnglish
Pages63-69
Number of pages7
Publication statusPublished - 2005 Oct 31
Event15th International Workshop on Research Issues in Data Engineering: Stream Data Mining and Applications, RIDE-SDMA 2005 - Tokyo, Japan
Duration: 2005 Apr 32005 Apr 4

Conference

Conference15th International Workshop on Research Issues in Data Engineering: Stream Data Mining and Applications, RIDE-SDMA 2005
CountryJapan
CityTokyo
Period05/4/305/4/4

ASJC Scopus subject areas

  • Software
  • Engineering (miscellaneous)
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Time-decaying Bloom Filters for data streams with skewed distributions'. Together they form a unique fingerprint.

  • Cite this

    Cheng, K., Xiang, L., Xu, H., Iwaihara, M., & Mohania, M. M. (2005). Time-decaying Bloom Filters for data streams with skewed distributions. 63-69. Paper presented at 15th International Workshop on Research Issues in Data Engineering: Stream Data Mining and Applications, RIDE-SDMA 2005, Tokyo, Japan.