Using Higher Dimensionalities to Identify Abnormal Behavior in Noisy Data Sets

In cluster analysis, distinguishing abnormal behavior from noise is an important problem that has remained unresolved for many years. This paper presents a general approach for distinguishing abnormal behavior from noise and for identifying abnormal behavior in noisy data sets. The approach is an application of a new, complete linkage hierarchical clustering method for n.(n-1)/2-level hierarchical sequences and a means for finding meaningful levels of such hierarchical sequences prior to performing a cluster analysis. These technologies were designed with small-n, large-m data sets in mind. Based on four broadly applicable assumptions, the approach uses higher dimensionalities to reveal inherent structure in noisy data sets and find meaningful levels in the corresponding hierarchical sequences. The new clustering method is used to construct only the cluster sets that correspond to these levels. Results from a first experiment show how the effects of noise are attenuated as the dimensionality of the data points increases. Results from a second experiment show how meaningful cluster sets can have real world meanings that are useful for identifying abnormal behavior.