Projection-Based Partitioning for Large, High-Dimensional Datasets

Recent work in the field of cluster analysis has focused on designing algorithms that address the issue of ever growing datasets and provide meaningful solutions for data with high cardinality and/or dimensionality, under the natural restriction of limited resources. Within this line of research, we propose a method drawing on the principles of projection pursuit and grid partitioning, which focuses on reducing computational requirements for large datasets without loss of performance. To achieve that, we rely on procedures such as sampling of objects, feature selection, and quick density estimation using histograms. The present algorithm searches for low-density points in potentially favorable one-dimensional projections, and partitions the data by a hyperplane passing through the best split point found. Tests on synthetic and reference data indicate that our method can quickly and efficiently recover clusters that are distinguishable from the remaining objects on at least one direction; linearly nonseparable clusters are usually subdivided. The solution is robust in the presence of noise in moderate levels, and when the clusters are partially overlapping. An implementation of the algorithm is available online, as supplemental material.