Data De duplication Using N0SQL Databases in Cloud

Data de-duplication has an important role in reducing storage consumption to make it affordable to manage in today's explosive data growth. Numerous DD methodologies like chunking and, delta encoding are available to optimize the use of storage. These technologies approach DD at file and/or sub-file level but this approach has never been optimal for NoSQL DBs. This method proposes data De-Duplication in NoSQL Databases (DDNSDB) which makes use of a DD approach at a higher level of abstraction. The main goals of this research is, to maximally reduce the amount of duplicates in one type of NoSQL DBs, namely the key-value store, to maximally increase the process performance such that the backup window is marginally affected, and to design with horizontal scaling in mind such that it would run on a Cloud Platform competitively. Also, by following an optimized adapted Map Reduce architecture, DDNSDB proves to have competitive performance advantage in a horizontal scaling cloud environment compared with a vertical scaling environment.