- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies anomaly-free model selection for kNN-based anomaly detectors. The authors derive an AUC lower bound and analyze how embedding-space geometry affects anomaly-score variability. They further investigate pseudo-anomaly-based performance estimators using different embedding-space pseudo-anomaly constructions.
Critical strengths
- Useful theoretical insights into anomaly-score variability. The paper analyzes how local density, intrinsic dimensionality, and domain mismatch affect kNN anomaly scores. Although the connection to model selection remains limited, this analysis provides a useful theoretical perspective on the behavior of distance-based anomaly detectors and the effects of local density normalization.
Critical weaknesses
This paper is a follow-up work of AutoUAD (ICLR 2025). The technical contribution is essentially incremental, and the connection between the theoretical framework and the model selection target is weak.
-
The title should refelect the application scope of the paper. The experiment mainly focus on the audio data. If the author claim the method is a general model selection method, then it should at least include results on ADBench.
-
Limited methodological novelty. The proposed framework largely follows existing semi-internal evaluation approaches, particularly AutoUAD. The AUC Bound estimator is a monotonic transformation of NPD, while Pseudo AUC closely resembles existing synthetic-anomaly-based evaluation strategies. The main methodological difference lies in the construction of pseudo-anomalies in embedding space, which primarily involves simple feature recombination or the use of auxiliary normal samples. Moreover, the paper lacks a direct comparison with the pseudo-anomaly generation strategy of AutoUAD, making it difficult to establish the added value of these modifications. The author should at least compare the performance with existing model selection methods using internal evaluation.
-
Weak connection between theoretical analysis and model selection. The geometric analysis mainly explains how local density, intrinsic dimensionality, and domain mismatch affect anomaly-score variability, but does not establish how these properties influence model-ranking reliability, which is the central objective of the paper. In particular, the proposed estimators and pseudo-anomaly constructions do not explicitly exploit the analyzed geometric properties, and the best-performing Diverse Pseudo AUC is largely independent of the theoretical analysis. Furthermore, the experiments do not directly validate the relationship between these geometric properties and model-selection performance. Consequently, the practical contribution of the theoretical extension remains unclear.
Questions for authors
-
Model selection under domain shift. The paper emphasizes domain shift, but the reported results largely aggregate source- and target-domain performance. Could the authors clarify whether the proposed estimators aim to select detectors that generalize well to the target domain? If so, could they provide separate source- and target-domain ranking correlations and selection regrets to better evaluate this capability?
-
Detector-agnostic applicability. The authors describe the proposed estimators as detector-agnostic, but the experiments only consider kNN-based scoring and its density-normalized variant. Could the authors clarify whether the framework can be directly applied to other anomaly detection methods, such as reconstruction-based or one-class classification approaches, and provide additional validation?
-
Dependence on semantic metadata. The best-performing Diverse pseudo-anomaly strategy relies on cross-domain, cross-class, and cross-attribute samples, which require additional semantic information. How practical is this assumption in general anomaly-free model selection scenarios? Could the authors further investigate performance when such metadata or auxiliary samples are unavailable?
-
Validity of the AUC lower bound. In Section 4, the authors state that no special handling is required when the mean score separation because the bound depends on . However, the derivation in Section 3.1 explicitly assumes . Could the authors clarify whether the lower bound remains valid when this assumption is violated?
We thank the reviewer for the thoughtful comments and critical feedback. We revised the manuscript and highlighted the changes:
Critical Weakness 0: Title and application scope
We agree that the original title could suggest broader applicability than demonstrated by our experiments. We have therefore revised the title, abstract, introduction, and related work to explicitly reflect the focus on kNN-based anomalous sound detection using pretrained audio representations. An evaluation on ADBench would require a different candidate pool and experimental protocol and is outside the scope of this work.
Critical Weakness 1: Limited methodological novelty
We acknowledge the close relationship between the AUC Bound and AutoUAD's NPD criterion: both induce the same candidate ranking when applied to identical score distributions. We position the bound primarily as a theoretical contribution, providing a direct derivation of an explicit AUC lower bound. Our additional contributions are the geometric analysis of kNN score variability, the analysis of local density normalization, and the systematic investigation of pseudo-anomaly constructions for selecting pretrained audio representations and pooling configurations. In the revised manuscript, we have also included AutoUAD's Gaussian pseudo-anomaly construction as a baseline. Our metadata-free Feature construction combined with the AUC Bound achieves lower overall regret (4.97 vs. 5.67).
Critical Weakness 2: Weak connection between theoretical analysis and model selection
We have clarified the relationship between the theoretical and empirical contributions. The AUC bound shows that detection performance depends jointly on mean score separation and inlier- and outlier-score variances. Hence, inlier-score variance alone does not capture all quantities relevant to performance. The geometric analysis characterizes how local density, intrinsic dimensionality, and domain mismatch affect kNN score variability, and explains how LDN mitigates some of these effects. Complementing this theoretical analysis, we systematically investigate several embedding-space pseudo-anomaly constructions, including metadata-free and metadata-based approaches, and evaluate their effectiveness for model selection across different candidate systems. To strengthen the empirical connection, we added detection-performance comparisons of kNN and LDN, an estimator-based comparison of the two scoring functions (Section 6.3), and further experiments on domain mixing (Appendix A.4).
Question 1: Model selection under domain shift
Our objective is to select a single detector that performs well across both source and target domains, rather than optimizing either domain in isolation. The selection procedure does not use domain labels or anomalous validation data. We have clarified this objective and added domain-wise model-selection results to Appendix A.4, reporting ranking correlations, pairwise ranking accuracies, and selection regrets separately for the source and target domains.
Question 2: Detector-agnostic applicability
The moment-based AUC inequality is formulated for scalar anomaly scores and can, in principle, be applied to other detector families. However, our geometric analysis and empirical evaluation specifically concern kNN and LDN. This focus reflects current practice in first-shot ASD: multiple top-performing systems in the DCASE 2025 and 2026 challenges use frozen pretrained audio representations with nearest-neighbor or distance-based scoring (Saengthong & Shinozaki, 2025; Fujimura et al., 2026). We have clarified this empirical scope and do not claim validation across other detector families.
Question 3: Dependence on semantic metadata
We agree that metadata-based constructions such as Cross-Attribute and Diverse are not always practical. We therefore explicitly distinguish metadata-based and metadata-free constructions in the revised results and discussion. We recommend Feature as the default when semantic metadata is unavailable: it constructs pseudo-anomalies directly from the reference embeddings, requires neither semantic labels nor auxiliary samples, and outperforms AutoUAD's Gaussian pseudo-anomaly baseline overall. Metadata-based constructions can provide further improvements when the required information is available.
Question 4: Validity of the AUC lower bound
We agree that the original statement was incorrect: the non-trivial lower bound assumes positive mean score separation (μΔ>0), and squaring the mean difference does not remove this assumption. We have corrected the implementation, repeated all experiments, and now assign a bound of zero whenever μΔ≤0. Appendix A.3 reports the frequency of such cases, which occur predominantly for Sequence and Element pseudo-anomalies, only rarely for Feature, and not at all for the remaining constructions. The correction does not materially affect the main conclusions.