Let’s make some noise: The Critical State of External Evaluation of Clusterings with Noise

Detecting noise is an important and beneficial feature of frequently used clustering algorithms like DBSCAN or HDBSCAN. However, the external evaluation of clusterings containing noise labels is handled inconsistently in the literature: as common external cluster validity indices (CVIs) like NMI or ARI are not defined for noise labels, a variety of ways to handle noise has emerged, most without proper foundations or explicit description. In this survey, we examine i) how papers presenting noise-detecting clustering methods evaluate their methods, and ii) the datasets that have been used. We define four desirable properties for noise-aware external CVIs and perform experiments across a wide range of existing noise-handling strategies. By highlighting where these properties are violated, we show that evaluation in this field is currently neither uniform nor scientifically adequate. Based on these findings, we propose two new noise-aware CVIs, $NAR{I}_{\kappa }$ and $NAR{I}_{\phi }$ (Noise-Aware ARI), combining ARI on cluster points with Cohen's $\kappa$ and the Matthews–Pearson $\phi$ coefficient on the binary noise classification, respectively. We show that both satisfy all four properties and confirm on real data, with held-out minority classes and foundation-model-annotated images as noise, that flawed strategies certify degenerate clusterings with perfect scores while the proposed measures do not.