Assessing clustering methods using Shannon's entropy

Unsupervised clustering techniques are crucial for effectively partitioning datasets into meaningful subgroups. In this paper, we introduce a novel clustering fuzziness metric based on Shannon's entropy, which quantifies the level of uncertainty in clustering assignments. This metric is employed to develop a statistical test for evaluating clustering algorithms and to propose parametric corrections to enhance their performance. We provide empirical evidence showing that our approach significantly improves clustering quality compared to well-known algorithms such as Fuzzy K-Means (FKM) and Fanny. For instance, applying our correction method led to a notable decrease in Jensen-Shannon Divergence (JSD) values, indicating enhanced clustering accuracy. Our methodology is demonstrated through comprehensive experiments on both simulated and real-world datasets. Additionally, our approach proves effective in determining the optimal number of clusters, showing superiority over traditional methods. These results highlight the practical benefits of our metric and correction techniques for improving clustering performance.