acceptodds
Under review as a conference paper at ICLR 2027

Can We Predict Anomaly Detection Performance from Embedding-Space Geometry?

Abstract

Anomaly detection systems are often trained using normal data alone, while model selection and evaluation typically require labeled anomalies. We study whether anomaly detection performance can be predicted without access to anomalous data. For kNN-based detectors, we derive a lower bound on the area under the ROC curve (AUC) that relates detection performance to the separation between inlier and outlier scores and to their respective variances. Under a local scaling model, we use this bound to characterize how density variation, intrinsic-dimensional heterogeneity, and cross-domain mismatch contribute to score variability. We then investigate anomaly-free model selection and show that inlier score variance alone does not reliably predict performance across different representations. To address this limitation, we introduce simple pseudo-anomaly probes that provide a reference for estimating relative score separation. Experiments on the DCASE 2022–2025 benchmarks, spanning four embedding models and 208 candidate systems, show that pseudo-anomaly-based estimators substantially improve anomaly-free model selection. In particular, diverse pseudo-anomalies enable anomaly-free model selection to outperform conventional development-set selection under domain shift. These results show that embedding-space geometry contains predictive information about anomaly detection performance while also highlighting the representation-dependent nature of inlier-only performance estimates.

Primary area:
learning theory
Keywords:
Anomaly Detection, Unsupervised Learning, Model Selection, Learning Theory, Nearest-Neighbor Methods, Representation Geometry, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qWJVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies anomaly-free model selection for kNN-based anomaly detectors. The authors derive an AUC lower bound and analyze how embedding-space geometry affects anomaly-score variability. They further investigate pseudo-anomaly-based performance estimators using different embedding-space pseudo-anomaly constructions.

Critical strengths
  • Useful theoretical insights into anomaly-score variability. The paper analyzes how local density, intrinsic dimensionality, and domain mismatch affect kNN anomaly scores. Although the connection to model selection remains limited, this analysis provides a useful theoretical perspective on the behavior of distance-based anomaly detectors and the effects of local density normalization.
Critical weaknesses

This paper is a follow-up work of AutoUAD (ICLR 2025). The technical contribution is essentially incremental, and the connection between the theoretical framework and the model selection target is weak.

  1. The title should refelect the application scope of the paper. The experiment mainly focus on the audio data. If the author claim the method is a general model selection method, then it should at least include results on ADBench.

  2. Limited methodological novelty. The proposed framework largely follows existing semi-internal evaluation approaches, particularly AutoUAD. The AUC Bound estimator is a monotonic transformation of NPD, while Pseudo AUC closely resembles existing synthetic-anomaly-based evaluation strategies. The main methodological difference lies in the construction of pseudo-anomalies in embedding space, which primarily involves simple feature recombination or the use of auxiliary normal samples. Moreover, the paper lacks a direct comparison with the pseudo-anomaly generation strategy of AutoUAD, making it difficult to establish the added value of these modifications. The author should at least compare the performance with existing model selection methods using internal evaluation.

  3. Weak connection between theoretical analysis and model selection. The geometric analysis mainly explains how local density, intrinsic dimensionality, and domain mismatch affect anomaly-score variability, but does not establish how these properties influence model-ranking reliability, which is the central objective of the paper. In particular, the proposed estimators and pseudo-anomaly constructions do not explicitly exploit the analyzed geometric properties, and the best-performing Diverse Pseudo AUC is largely independent of the theoretical analysis. Furthermore, the experiments do not directly validate the relationship between these geometric properties and model-selection performance. Consequently, the practical contribution of the theoretical extension remains unclear.

Questions for authors
  1. Model selection under domain shift. The paper emphasizes domain shift, but the reported results largely aggregate source- and target-domain performance. Could the authors clarify whether the proposed estimators aim to select detectors that generalize well to the target domain? If so, could they provide separate source- and target-domain ranking correlations and selection regrets to better evaluate this capability?

  2. Detector-agnostic applicability. The authors describe the proposed estimators as detector-agnostic, but the experiments only consider kNN-based scoring and its density-normalized variant. Could the authors clarify whether the framework can be directly applied to other anomaly detection methods, such as reconstruction-based or one-class classification approaches, and provide additional validation?

  3. Dependence on semantic metadata. The best-performing Diverse pseudo-anomaly strategy relies on cross-domain, cross-class, and cross-attribute samples, which require additional semantic information. How practical is this assumption in general anomaly-free model selection scenarios? Could the authors further investigate performance when such metadata or auxiliary samples are unavailable?

  4. Validity of the AUC lower bound. In Section 4, the authors state that no special handling is required when the mean score separation μΔ<0\mu_\Delta<0 because the bound depends on μΔ2\mu_\Delta^2. However, the derivation in Section 3.1 explicitly assumes μΔ>0\mu_\Delta>0. Could the authors clarify whether the lower bound remains valid when this assumption is violated?

1 comment

We thank the reviewer for the thoughtful comments and critical feedback. We revised the manuscript and highlighted the changes:

Critical Weakness 0: Title and application scope

We agree that the original title could suggest broader applicability than demonstrated by our experiments. We have therefore revised the title, abstract, introduction, and related work to explicitly reflect the focus on kNN-based anomalous sound detection using pretrained audio representations. An evaluation on ADBench would require a different candidate pool and experimental protocol and is outside the scope of this work.

Critical Weakness 1: Limited methodological novelty

We acknowledge the close relationship between the AUC Bound and AutoUAD's NPD criterion: both induce the same candidate ranking when applied to identical score distributions. We position the bound primarily as a theoretical contribution, providing a direct derivation of an explicit AUC lower bound. Our additional contributions are the geometric analysis of kNN score variability, the analysis of local density normalization, and the systematic investigation of pseudo-anomaly constructions for selecting pretrained audio representations and pooling configurations. In the revised manuscript, we have also included AutoUAD's Gaussian pseudo-anomaly construction as a baseline. Our metadata-free Feature construction combined with the AUC Bound achieves lower overall regret (4.97 vs. 5.67).

Critical Weakness 2: Weak connection between theoretical analysis and model selection

We have clarified the relationship between the theoretical and empirical contributions. The AUC bound shows that detection performance depends jointly on mean score separation and inlier- and outlier-score variances. Hence, inlier-score variance alone does not capture all quantities relevant to performance. The geometric analysis characterizes how local density, intrinsic dimensionality, and domain mismatch affect kNN score variability, and explains how LDN mitigates some of these effects. Complementing this theoretical analysis, we systematically investigate several embedding-space pseudo-anomaly constructions, including metadata-free and metadata-based approaches, and evaluate their effectiveness for model selection across different candidate systems. To strengthen the empirical connection, we added detection-performance comparisons of kNN and LDN, an estimator-based comparison of the two scoring functions (Section 6.3), and further experiments on domain mixing (Appendix A.4).

Question 1: Model selection under domain shift

Our objective is to select a single detector that performs well across both source and target domains, rather than optimizing either domain in isolation. The selection procedure does not use domain labels or anomalous validation data. We have clarified this objective and added domain-wise model-selection results to Appendix A.4, reporting ranking correlations, pairwise ranking accuracies, and selection regrets separately for the source and target domains.

Question 2: Detector-agnostic applicability

The moment-based AUC inequality is formulated for scalar anomaly scores and can, in principle, be applied to other detector families. However, our geometric analysis and empirical evaluation specifically concern kNN and LDN. This focus reflects current practice in first-shot ASD: multiple top-performing systems in the DCASE 2025 and 2026 challenges use frozen pretrained audio representations with nearest-neighbor or distance-based scoring (Saengthong & Shinozaki, 2025; Fujimura et al., 2026). We have clarified this empirical scope and do not claim validation across other detector families.

Question 3: Dependence on semantic metadata

We agree that metadata-based constructions such as Cross-Attribute and Diverse are not always practical. We therefore explicitly distinguish metadata-based and metadata-free constructions in the revised results and discussion. We recommend Feature as the default when semantic metadata is unavailable: it constructs pseudo-anomalies directly from the reference embeddings, requires neither semantic labels nor auxiliary samples, and outperforms AutoUAD's Gaussian pseudo-anomaly baseline overall. Metadata-based constructions can provide further improvements when the required information is available.

Question 4: Validity of the AUC lower bound

We agree that the original statement was incorrect: the non-trivial lower bound assumes positive mean score separation (μΔ>0\mu_\Delta > 0), and squaring the mean difference does not remove this assumption. We have corrected the implementation, repeated all experiments, and now assign a bound of zero whenever μΔ≤0\mu_\Delta \leq 0. Appendix A.3 reports the frequency of such cases, which occur predominantly for Sequence and Element pseudo-anomalies, only rarely for Feature, and not at all for the remaining constructions. The correction does not materially affect the main conclusions.

Reviewer_y2tgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors in this paper propose an approach to select the model when there are no labelled anomalies present. They construct pseudo anomalous data points, calculate AUC using the normal and pseudo anomalous data points (using KNN and LDN methods), and then compare it with ground truth AUC. They use 208 candidate systems (4 pre-trained models and 52 pooling configurations per embedding model) that the model selection method must choose from. The report metrics such as Spearman rank correlation, pair-wise ranking accuracy and regret. The results show that the proposed method is able to achieve higher metrics as compared to when models are selected randomly, fixed selection method define by the authors and inlier variance method. They also mention that the performance depends on pseudo-anomaly construction.

Critical strengths
  1. The paper presents a method of model selection when no labelled anomalies are available, which is very realistic and practical setting for anomaly detection.
  2. The authors evaluate the proposed method 208 candidate systems.
Critical weaknesses
  1. The authors discuss various related works, but none are used as the baselines. It’s difficult to understand how this method advances beyond the baselines and how the performance changes. They just use random selection, fixed selection, inlier variance, but no direct comparison with closely related works is done(for eg: AutoUAD).
  2. The performance varies significantly across pseudo anomaly construction methods. The strongest methods rely on domain/class/attribute information but meta-data free alternative provides weaker gains. This raises a pratical question: how do we select pseudo anomaly construction method when no semantic metadata is available?
  3. The performance gain in diverse scenario may be attributed to more number of pseudo anomalies present. Can authors do an ablation study, where they fix the number of pseudo anomalies across each method?
Questions for authors

See above

1 comment

We thank the reviewer for the thoughtful comments and critical feedback. We substantially revised our submission and uploaded the manuscript with highlighted changes. Here are detailed replies:

Critical Weakness 1/Question 1: Comparison with AutoUAD

We agree that a direct comparison with AutoUAD is important. We therefore included its Gaussian pseudo-anomaly construction as a baseline in our experiments. In particular, the AUC Bound applied to AutoUAD's pseudo-anomalies induces the same candidate ranking as AutoUAD's NPD criterion when applied to identical score distributions. Our metadata-free Feature construction achieves a lower overall regret than the AutoUAD baseline (4.97 vs. 5.67), demonstrating that the choice of pseudo-anomaly construction matters for model selection.

Critical Weakness 2/Question 2: Pseudo anomaly selection without metadata

We agree that the reliance on semantic metadata limits the applicability of constructions such as Cross-Attribute and Diverse. We have therefore made our recommendation explicit: Feature is our recommended default when semantic metadata is unavailable. It achieves competitive performance without requiring domain, class, or attribute labels, and outperforms AutoUAD's Gaussian pseudo-anomaly baseline in overall model-selection performance. When semantic metadata is available, constructions such as Cross-Attribute and Diverse can provide further improvements. We now discuss this practical distinction more explicitly in the revised manuscript.

Critical Weakness 3/Question 3: Controlling number of pseudo anomalies

We agree that the number of pseudo-anomalies should be controlled to ensure a fair comparison. We therefore repeated all experiments using exactly the same number of pseudo-anomalies for every construction, namely (M=1000), matching the size of the reference set. Whenever a construction yielded more samples, we randomly subsampled it to 1000. All results reported in the revised manuscript are based on this controlled sample count. The resulting performance is similar to that obtained with the original sample counts, and the main conclusions remain unchanged.

Reviewer_3NmrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether anomaly detection performance can be predicted without anomalous validation data. For kNN-based detectors, it derives an AUC lower bound from Cantelli's inequality in terms of mean score separation and score variances, and uses a local scaling analysis to attribute inlier score variability to density variation, intrinsic-dimension heterogeneity, and domain mismatch, with local density normalization as a case study. Since inlier variance alone does not compare well across representations, the paper constructs pseudo-anomalies directly in embedding space and defines two model-selection criteria, AUC Bound and Pseudo AUC. On DCASE 2022–2025 with four pretrained audio embeddings and 208 candidate systems, Diverse Pseudo AUC gives the strongest selection signal and outperforms development-set-based Fixed Selection on the evaluation sets.

Critical strengths

The problem is practically important, and the decomposition of inlier score variance into density, intrinsic-dimension, and domain-mismatch components is a clean way to organize what normal data can and cannot reveal; it also gives a principled explanation for why local density normalization helps. The empirical study is careful by the standards of this area, with 174 model-selection tasks, 208 candidate systems, bootstrap confidence intervals, and a comparison of pseudo-anomaly constructions that includes unsuccessful alternatives rather than only the winning configuration.

Critical weaknesses
  1. The contribution is narrower than the framing suggests, and the evaluation is disconnected from the current state of the art. The title asks a general question about predicting anomaly detection performance, and Section 4 calls the estimators detector-agnostic, but the theory is specific to kNN geometry and the candidate pool contains only two kNN variants over audio embeddings. No deep detection methods (!) and none of the recent anomaly detection foundation models (!) appear as candidates or baselines, so we learn nothing about whether the criteria can rank the systems a practitioner would actually consider today. The increment over prior anomaly-free selection is also modest — the authors note the AUC Bound ranks candidates identically to AutoUAD's NPD on the same scores, and Pseudo AUC parallels SWSA — and the strongest pseudo-anomalies require semantic metadata that is often unavailable.

  2. Within this narrow scope, there are also two concrete problems. The treatment of negative mean separation in the AUC Bound criterion is incorrect: Section 3.1 derives the bound under positive mean separation, but Section 4 claims negative separation needs no special handling because the expression uses its square. Squaring removes direction — with constant inlier scores of one and outlier scores of zero, the true AUC is zero while the expression evaluates to one — so the criterion is not a valid lower bound in this regime and could favor candidates that rank pseudo-anomalies in the wrong direction. The paper reports neither how often this occurs across its tasks and candidates nor the impact on the AUC Bound results in Table 1. This is a mismatch between a stated condition of the theory and how the criterion is actually used, not a presentation slip.

Finally, the anomaly-free claim is under-documented at the pipeline level: Appendix A.2 states that EAT preprocessing thresholds were selected on development sets without specifying the objective or whether anomaly labels were involved. Given that the headline result is an advantage over label-based Fixed Selection, the role of labels at every stage should have been documented. A narrow and modestly incremental contribution, combined with an incorrect criterion and an under-documented protocol, does not meet the bar for acceptance.

Questions for authors
  • How frequently is the estimated mean pseudo-outlier score at or below the mean inlier score across tasks and candidates, and do the AUC Bound rankings, selected systems, and regrets in Table 1 change if such cases are assigned the trivial bound of zero?

  • Were real anomaly labels used to select the EAT preprocessing thresholds, and does Diverse Pseudo AUC retain its evaluation-set advantage over Fixed Selection without development-set-tuned preprocessing?

  • Finally, how do the proposed criteria behave when the candidate pool includes strong modern detectors, such as deep anomaly detection methods or anomaly detection foundation models, rather than kNN variants alone? If the criteria are not expected to apply beyond neighborhood-based scoring, please state the intended scope explicitly.

1 comment

We thank the reviewer for the thoughtful comments and critical feedback. We substantially revised our submission and uploaded the manuscript with highlighted changes. Here are detailed replies:

Critical Weakness 1/Question 3: Scope of the contribution and framing

We agree that the original title and framing could suggest a broader study of anomaly detection model selection than what we actually investigate. We have therefore revised the title, abstract, introduction, and discussion of related work to explicitly define the scope: anomaly-free selection among pretrained audio representations and temporal pooling configurations for first-shot anomalous sound detection using nearest-neighbor-based scoring.

Critical Weakness 1/Question 3: Relevance of the candidate pool and current research practice

We do not claim that the proposed criteria generalize beyond nearest-neighbor-based scoring, and a comparison with other detector families and anomaly detection foundation models is beyond the scope of this work. However, we strongly disagree with the characterization that our candidate pool is disconnected from current research practice. All candidates use deep pretrained audio encoders (OpenL3, BEATs, EAT, and Dasheng), with kNN and LDN as scoring backends. Deep pretrained audio representations, in particular BEATs, combined with nearest-neighbor-based scoring constitute a state-of-the-art paradigm in first-shot anomalous sound detection, as demonstrated by top-ranked systems in the DCASE 2025 and 2026 Task 2 challenges (Saengthong & Shinozaki, 2025; Fujimura et al., 2026). SVDD-style one-class compactness objectives have been shown to yield near-chance performance on noisy industrial ASD benchmarks (Wilkinghoff & Kurth, 2024). Moreover, under squared Euclidean distance, SVDD's distance-to-center score is, in fact, ranking-equivalent to averaging squared distances to all training samples (i.e., kNN with (k=N)). Conventional autoencoder baselines are similarly uncompetitive, substantially underperforming representation-learning approaches across DCASE benchmarks (Fujimura et al., 2025). The competitiveness of the candidate pool is now explicitly stated in the introduction of the revised manuscript.

Critical Weakness 1: Novelty relative to existing anomaly-free selection methods

We agree that the AUC Bound is closely related to AutoUAD's NPD and induces the same candidate ranking when applied to identical score distributions. We have clarified this relationship and position the bound primarily as a theoretical contribution, providing a direct derivation of an explicit AUC lower bound rather than proposing a fundamentally new ranking criterion. The contributions beyond this connection are the geometric analysis of kNN score variability under domain shift, the analysis of local density normalization, and the systematic investigation of embedding-space pseudo-anomaly constructions for pretrained representation selection. We have also added the pseudo-anomaly construction presented in AutoUAD as a baseline in our experiments.

Critical Weakness 2/Question 1: Treatment of negative mean separation

We agree. We investigated this issue in Appendix A.3 and found that non-positive mean separations occur predominantly for Sequence and Element pseudo-anomalies. Across development and evaluation tasks, their frequency ranges from 3.82% to 9.84% with kNN and from 23.86% to 31.74% with LDN. They occur only rarely for Feature pseudo-anomalies (at most 0.05%) and not at all for the remaining constructions. We now explicitly assign zero to the AUC Bound whenever the estimated mean separation is non-positive (μΔ≤0\mu_\Delta \leq 0) and have repeated all experiments using this corrected implementation. The correction does not materially affect the overall ranking performance or the main conclusions of the paper.

Critical Weakness 3/Question 2: Underdocumented anomaly-free claim for EAT pre-processing

We agree that the original EAT preprocessing was insufficiently documented. We adopted the preprocessing procedure, including its thresholds, from the literature rather than selecting them ourselves. However, since these thresholds may have been tuned using validation data containing anomalies, we repeated all experiments using raw EAT embeddings without preprocessing to eliminate any potential dependence on anomalous validation data. The updated results retain the evaluation-set advantage of Diverse Pseudo AUC over Fixed Selection with both kNN and LDN.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.