- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper reviews how clustering solutions with noise labels are evaluated using supervised scores and shows that common noise handling conventions can produce misleading scores. It formalizes four desirable properties for noise-aware external evaluation and proposes two scores combining ARI on mutually non-noise points with a separate measure of noise-detection agreement. Through theoretical analysis and experiments, the paper compares these measures with existing alternatives and illustrates how the employed scores can affect clustering assessment and parameter selection.
Critical strengths
S1. The paper provides a valuable review of current evaluation practices for clustering with noise, highlighting inconsistencies and limitations of available benchmarks. It formalizes desirable properties for noise-aware evaluation and analyzes existing and proposed measures against them.
S2. The proposed scores explicitly distinguish clustering agreement from noise-detection quality. This discrimination seems sensible and helpful since clustering and noise detection are different tasks.
Critical weaknesses
W1. I do not consider that this problem is currently of particular importance, since several popular clustering algorithms do not provide noise labels. Moreover, it seems that real datasets of this type are not yet available.
W2. Since we are speaking of external measures, the practical usefulness is questionable, since clustering is unsupervised and does not assume the existence of cluster (and noise) labels.
W3. Although it is discussed in the paper, the superiority against singleton ARI and AMI is not clear in practice. Better scores do not imply better decisions based on those scores.
Questions for authors
Q1. I do not see any reason to present two scores differing only in the evaluation function (\kappa and \phi) for noise detection.
Q2. In which practically meaningful cases does NARI improve evaluation over singleton-based ARI/AMI?
Q3. Why not reporting two separate measures (cluster validity and noise detection) instead of combining them into one composite score? This would provide a clearer view on the quality of the clustering solution. The combination of the two as a weighted average is rather straightforward.
5 comments
Q3
Reporting clustering agreement and noise-detection quality separately is useful for interpreting a result, and our implementation already returns both components, their associated sample counts, and the combined score. This decomposability is an explicit advantage of NARI.
Simply averaging scores does not resolve how noise labels enter the clustering component. Standard ARI and AMI compare partitions without assigning a special meaning to noise labels. Applying them to clusterings containing noise therefore requires an additional convention, such as treating noise as a cluster, encoding noise points as singletons, filtering them, or applying a filtering penalty. Our analysis demonstrates that these choices have substantive consequences for the resulting evaluation.
NARI explicitly defines the two evaluation tasks: ARI measures agreement on the objects that both labelings identify as cluster objects, while κ or φ evaluates the binary noise assignments over all observations. Every disagreement about whether an object is noise is therefore assessed explicitly. The combined score provides a reproducible criterion for benchmarking and parameter selection when the components favor different configurations, while retaining access to both components for interpretation. The weighting is also consequential. It depends on both labelings, assigning the clustering component the fraction of common cluster objects and the noise component the complementary fraction. Appendix C demonstrates that using ground-truth-only or prediction-only weights can produce perfect scores despite severe noise-detection errors. Our contribution therefore includes the precise definition and theoretical analysis of a noise-aware evaluation criterion, together with evidence of its consequences for practical selection decisions. We will make these complementary roles of the component scores and the aggregate more explicit in the manuscript.
W1
Noise rejection is a central capability of DBSCAN, HDBSCAN, and OPTICS, and our survey documents its use in newer methods (see Table 2). Reliable evaluation matters both when comparing noise-detecting methods and when comparing them with methods that assign every observation to a cluster. The availability of algorithms without noise rejection therefore makes consistent evaluation across these outputs particularly relevant.
The evaluation problem also arises on standard benchmarks whose reference labels contain no noise: predicted noise labels still require an explicit evaluation rule. Filtering these observations can hide the rejection of substantial parts of the reference clustering and yield a perfect score on the retained subset. Thus, the relevance of noise-aware evaluation extends beyond benchmarks with ground-truth noise.
We have further strengthened the empirical evidence with an astronomical case study using Gaia DR3 observations around NGC 869 and NGC 884. Identifying stellar clusters within a field-star background is an established scientific application of clustering with noise. We use published memberships from Hunt and Reffert (2023, updated 2024), whose catalogue was constructed using HDBSCAN with additional statistical and photometric validation. Our dataset consists of 8,834 stars: 662 catalogue members and 8,172 background-labelled stars treated as noise.
On the same five scaled astrometric features, standard K-Means with k=2 assigns all 8,172 background stars to clusters and merges the members of both dominant stellar clusters into one predicted cluster. This result is identical across ten seeds with 20 initializations each. Nevertheless, NMI-LNP-I assigns it a near-perfect score of 0.9964. Both NARI variants score this solution at 0.00125, compared with 0.8198 and 0.8239 for their selected HDBSCAN solution, which makes 259 noise-classification errors and correctly matches 601 catalogue members. Furthermore, Purity-LNP-I assigns its maximum score of 1 to an HDBSCAN partition containing 2,194 clusters and 5,752 noise-classification errors.
These findings demonstrate both the relevance of noise rejection in a real scientific application and the practical consequences of how it is evaluated. As reviewer njDT also highlights, these evaluation choices affect parameter selection and method comparisons.
W2
Clustering can be unsupervised while its output is evaluated against reference labels in a benchmark. External evaluation is standard practice for assessing whether a method recovers known structures and for comparing competing algorithms. Our contribution addresses the reliability of this established practice when clustering outputs contain noise labels.
Table 2 already surveys 21 publications, 20 of which report external evaluation measures. Beyond this focused survey of noise-handling practices, the following 16 examples from KDD, ICLR, ICML, and NeurIPS illustrate the widespread use of reference-based evaluation in recent research. They cover deep, graph, multi-view, ensemble, and combinatorial clustering, as well as cell-clustering applications.
| Authors | Paper | Venue | External measures |
|---|---|---|---|
| Leiber et al. | The DipEncoder: Enforcing Multimodality in Autoencoders | KDD 2022 | NMI, ARI |
| Yang et al. | Effective Clustering on Large Attributed Bipartite Graphs | KDD 2024 | ACC, NMI, ARI |
| Liu et al. | Revisiting Modularity Maximization for Graph Clustering: A Contrastive Learning Perspective | KDD 2024 | ACC, NMI, ARI, F1 |
| Lin et al. | Spectral Subspace Clustering for Attributed Graphs | KDD 2025 | ACC, NMI, ARI |
| Liu et al. | Deep Temporal Graph Clustering | ICLR 2024 | ACC, NMI, ARI, F1 |
| Jia et al. | Towards Calibrated Deep Clustering Network | ICLR 2025 | ACC, NMI, ARI |
| Miklautz et al. | Breaking the Reclustering Barrier in Centroid-based Deep Clustering | ICLR 2025 | ACC, NMI, ARI |
| Li & Jia | ConMix: Contrastive Mixup at Representation Level for Long-tailed Deep Clustering | ICLR 2025 | ACC, NMI, ARI |
| Svirsky & Lindenbaum | Interpretable Deep Clustering for Tabular Data | ICML 2024 | ACC, NMI, ARI |
| Dai et al. | Multi-View Clustering by Inter-cluster Connectivity Guided Reward | ICML 2024 | NMI, Purity, ARI |
| Wang et al. | MC-GTA: Metric-Constrained Model-Based Clustering using Goodness-of-fit Tests with Autocorrelations | ICML 2024 | ARI, NMI |
| Zhang et al. | Generalization Performance of Ensemble Clustering: From Theory to Algorithm | ICML 2025 | NMI, ARI, Purity |
| Arutyunova et al. | Approximately Pareto-optimal Solutions for Bi-Objective k-Clustering | NeurIPS 2024 | NMI, RI, Fβ |
| Chen et al. | Bridging Gaps: Federated Multi-View Clustering in Heterogeneous Hybrid Views | NeurIPS 2024 | ACC, NMI, ARI |
| Guo et al. | Robust Contrastive Multi-view Clustering against Dual Noisy Correspondence | NeurIPS 2024 | ACC, NMI, ARI |
| Um et al. | Gene-Gene Relationship Modeling Based on Genetic Evidence for Single-Cell RNA-Seq Data Imputation | NeurIPS 2024 | ARI, NMI, CA |
ACC and CA denote clustering accuracy. ACC and CA denote clustering accuracy.
Our work strengthens the methodological foundation of these comparisons. When an algorithm rejects observations as noise, its evaluation requires an explicit treatment of those labels. Our theoretical analysis and experiments demonstrate that this choice can change parameter selection and method rankings, and can even assign perfect scores to degenerate solutions. Ensuring that external evaluation reflects both clustering agreement and noise-detection quality therefore has direct practical value.
W3
The practical evidence concerns the clustering configurations selected by each evaluation criterion. As detailed in our response to Q2, both NARI variants select the same CLUTO-t4.8k configuration with 133 noise-classification errors, compared with 261 for singleton ARI and 138 for singleton AMI. This configuration also recovers more genuine cluster objects and contains fewer spurious noise-only clusters. These improvements are assessed directly against the reference labels and demonstrate better parameter-selection decisions. The additional Gaia case study in Q2 illustrates how the choice of evaluation criterion affects the recovery of stellar cluster members in a real scientific application.
Dear Reviewer R1uJ, thank you for your review and insightful questions. Before addressing your questions individually, we would first like to briefly clarify that we strongly disagree with both W1 and W2. The fact that some clustering algorithms do not produce noise labels does not diminish the relevance of evaluating methods that explicitly distinguish noise points from cluster members. Moreover, the unsupervised nature of clustering does not diminish the importance of external evaluation, which is essential for benchmarking algorithms and comparing methods with each other.
Q1
The two variants instantiate a common noise-aware evaluation framework using two established notions of binary agreement. Cohen’s κ measures agreement beyond the level expected from the observed marginal frequencies, whereas φ, equivalently the Matthews correlation coefficient, measures correlation between the binary noise indicators.
Their different normalizations matter when predicted and reference noise proportions differ. When these proportions coincide, the coefficients agree. Otherwise, they can respond differently to over- or under-detection of noise. Both variants retain the same theoretical guarantees of our construction.
Including both also demonstrates that the principal findings are reproduced with two established choices for the noise component. We will clarify the shared NARI construction and explain the distinct interpretations of its κ- and φ-based variants.
Q2
A concrete example is parameter selection in our existing DBSCAN sweep on CLUTO-t4.8k (Section 3.6). NARIκ and NARIφ both select the same DBSCAN configuration. The resulting selections compare as follows:
Noise-classification errors count both ground-truth noise assigned to clusters and genuine cluster objects rejected as noise. Correctly matched cluster objects are counted after optimal one-to-one cluster-label matching, out of 7,236 ground-truth cluster objects. Both NARI variants therefore select a configuration with fewer noise-classification errors, fewer spurious noise-only clusters, and more correctly recovered cluster objects than either singleton baseline.
We have additionally completed an astronomical case study using Gaia DR3 observations around NGC 869 and NGC 884. We construct the reference labeling from published catalogue memberships (Hunt and Reffert, 2024), assigning unmatched sources the reference noise label. The field contains 8,834 stars: 662 catalogue members and 8,172 background-labelled objects. In the DBSCAN/HDBSCAN parameter search on this field, singleton ARI selects a configuration that correctly matches 534 catalogue members and rejects 128 members as noise. Both NARI variants select a configuration that correctly matches 601 members and rejects only 60. This recovers 67 additional members, with total noise-classification errors of 259 versus 245. Singleton AMI selects the same configuration as both NARI variants. This example demonstrates a practically relevant difference in member recovery, complementing the improvement over both singleton baselines on CLUTO.
These comparisons assess the selected clusterings directly against reference labels. More generally, NARI explicitly evaluates noise rejection: singleton encoding makes a rejected noise point indistinguishable from a one-point cluster, whereas NARI preserves this distinction. We will incorporate these comparisons into the manuscript to make their practical implications for parameter selection explicit.