SeqCDC: Hashless Content-Defined Chunking for Data Deduplication

Data deduplication is critical to cloud storage providers and is widely employed to conserve server-side storage space. Data chunking is an important aspect of deduplication, being directly responsible for storage space savings and end-to-end system throughput. While deduplication systems deployed in production favor larger chunk sizes, existing data chunking algorithms are slow and offer minimal throughput increases with increasing chunk size. We present SeqCDC, a chunking algorithm that leverages contentbased data skipping and lightweight boundary judgement to improve chunking throughputs. SeqCDC's chunking throughput is higher at larger chunk sizes. Our evaluation shows that SeqCDC can improve chunking throughput by 1.5x - 3.1x over the stateof-the-art while achieving similar space savings benefits, across a variety of datasets.