ODscan: On-demand database scan approach to mining large itemsets

The discovery of frequently occurring patterns in large databases is commonly referred to as the market basket analysis problem Solving the problem requires high PO overhead (to perform several scans of the database), large memory resources (to tentatively store candidate patterns until they are actually counted through a database scan), and intensive computatons (to perform subset testing among patterns). Most commercial implementations and published algorithms for solving the problem are based in one way or another on the Apriori principle. We propose an algorithm, ODscan, which is also Apriori-based but is unique in that it attempts to balance the cost of I/O and the memory requirements during the derivation process. In principle, ODscan requires only two scans of the database. Additional scans may be needed only when user-controlled memory requirements are exceeded In that case, a scan results in freeing some of the memory before resuming the process of derivation.