acceptodds
Under review as a conference paper at ICLR 2027

Distributional Measurement-based Model Selection for Anomaly Detection

Abstract

Anomaly detection (AD) is a fundamental task in data analysis with broad applications in high-stakes domains. Although numerous AD algorithms have been developed over the past decades, this rich methodological landscape makes detector selection increasingly challenging. Existing automated AD methods mainly rely on hyperparameter optimization or meta-learning. Hyperparameter optimization (HO) is fully unsupervised and requires no historical tasks, but its effectiveness depends on surrogate internal criteria and it is often computationally expensive for large-scale datasets. Meta-learning methods transfer selection knowledge from historical tasks by constructing task-level meta-features, yet existing approaches primarily use coarse statistical summaries, which may fail to capture subtle structural and relational patterns that are critical in complex AD tasks. To address these limitations, we propose a distribution-aware meta-learning framework for automated AD model selection. Instead of representing each dataset only by statistical summaries, our method constructs informative and task-guided meta-features through distributional measurements across datasets. Specifically, we first build sample-wise similarity graphs to map heterogeneous datasets into a unified relational representation space. We then compute graph-based distances or similarities between datasets, thereby capturing fine-grained structural and distributional information. Extensive experiments across diverse AD tasks and two modalities show that our method consistently achieves superior or comparable detection performance compared with existing model selection techniques, while substantially reducing online selection time compared with the HO-based approaches. Moreover, the constructed meta-features are robust to domain shifts, allowing our framework to select strong detectors for new tasks from domains not observed in the historical task set.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model Selection, Unsupervised Anomaly Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qZugOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GOOD, a meta-learning framework that selects both an anomaly detection algorithm and its hyperparameter configuration for a new unlabeled dataset. Each dataset is represented by sample-wise similarity graphs built with Gaussian kernels at multiple scales, and WL similarities and Minimum Factorization Distances against historical graphs form a task representation. An MLP meta-learner maps this representation to the predicted performance of 358 candidate (algorithm, configuration) pairs. Evaluation covers 47 tabular and 61 image-embedding tasks derived from ADBench, with leave-one-out experiments, representation ablations, and domain and modality transfer studies. The main contribution is the graph-induced task representation within an established offline-online meta-selection framework.

Critical strengths

The graph-induced task representation is a concrete and well-validated answer to a real bottleneck in meta-learning-based selection. The tabular AUPRC result is the strongest evidence: GOOD reaches 64.44% against 57.79% for ISAC, 56.31% for MetaOOD, and 54.80% for ELECT, with paired tests supporting the differences, and the AUPRC metric is the informative one under class imbalance. The analysis in Section 4.2, showing that WL similarity tracks detector-ranking similarity substantially better than MetaOD features, gives direct quantitative support for why the representation works, and the authors appropriately describe this relationship as inductive rather than guaranteed.

The empirical study is more complete than is typical for this line of work. Scale and measurement ablations, alternative graph constructions, meta-learner comparisons, and the domain-exclusion and cross-modality transfer experiments together examine the contribution from several angles instead of reporting a single aggregate table. The finding that performance is largely insensitive to excluding same-domain datasets from the history, and drops only about 2% under full modality exchange, is practically important for deployment.

Critical weaknesses
  1. The paper does not position the problem against the current generation of tabular and anomaly detection foundation models. The candidate pool contains only classical and earlier deep detectors, and no foundation model appears as a candidate, a baseline, or even a discussion point. Since these models also operate without labels on the new task, the practical question a reader is left with is when one should run meta-learned selection over a legacy pool at all, rather than applying a strong foundation model directly. The paper should at least report where the pool's empirical upper bound stands relative to such models, and discuss whether foundation models can be included as pool members.

Relatedly, the method assumes a strictly label-free new task; it neither uses nor discusses the common intermediate case where a few labeled anomalies are available, in which direct candidate evaluation may dominate. The boundary of the intended setting remains undefined.

  1. Two further issues affect the reliability of specific claims. MetaOD's tabular AUROC of 65.18 is almost ten points below simply running KNN (74.74), which is inconsistent with MetaOD's published behavior and suggests a baseline implementation or adaptation problem; since Table 2 is the main evidence, this needs verification.

  2. As a minor presentation request, Table 1 is difficult to read at its current font size and Figure 4 could be made more compact.

Questions for authors
  • How does the pool and its empirical upper bound compare to current tabular and anomaly detection foundation models on the same tasks, and can such models be added as pool members?

  • Can the framework exploit a small number of labeled anomalies on the new task, or should it be understood as strictly label-free?

  • Please describe the per-fold construction of the reference-based features, including whether the held-out task's reference column, graph-threshold contribution, and preprocessing statistics are excluded, and identify any intentionally transductive steps.

  • Please also verify the MetaOD configuration underlying Table 2, since its reported performance falls well below both its published results and a default KNN detector.

1 comment

Thank you for the detailed response. The additional UniOD comparison and the fold-wise feature-construction clarification address much of my concern.

One remaining issue is that the UniOD results appear to be taken from the original paper rather than reproduced under the same experimental environment as GOOD. If so, I recommend explicitly characterizing this as a cross-paper comparison and clarifying differences in task splits, preprocessing, metric implementation, and runtime accounting. If computationally feasible, a unified rerun would provide stronger evidence. It would also be useful to consider FoMo-0D [1] and OutFormer [2] as additional external baselines or potential pool members, since they are directly relevant zero-shot tabular anomaly detection foundation models.

The weakly supervised extension is a reasonable future direction, but it could be made more concrete. In particular, future work could specify how labels modify the graph construction, how the sampling rate p is selected, how target-task labels are prevented from leaking into training, and how performance varies with the number of labeled anomalies.

These remaining points concern positioning and future work rather than the validity of the main findings. I will therefore maintain my weak acceptance score.

[1] Shen, Wen, and Akoglu. FoMo-0D: A Foundation Model for Zero-shot Tabular Outlier Detection. TMLR, 2025.

[2] Ding, Wen, Klüttermann, and Akoglu. From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detection. ICML, 2026.

Reviewer_GkMQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GOOD selects an anomaly detection algorithm and hyperparameter configuration for an unlabeled target dataset using performance information from labeled historical tasks. It constructs sample similarity graphs at multiple Gaussian kernel scales, concatenates inter-dataset Weisfeiler-Lehman similarities and Minimum Factorization Distances, and trains a regressor to predict candidate performance. Evaluation covers 358 configurations on 47 tabular and 61 image tasks, alongside representation ablations, transfer experiments, correlation analyses, and runtime comparisons.

Critical strengths
  1. GOOD combines multi-scale sample similarity graphs with complementary WL/MFD comparisons to construct dataset representations for performance prediction. Replacing MetaOD’s original descriptors with these features increases reported AUROC from 65.18% to 72.36% and AUPRC from 47.94% to 51.34%, providing controlled evidence that the representation contributes useful selection information.
  2. Within the common candidate pool, GOOD reports 64.44% mean AUPRC across 47 tabular tasks, exceeding ISAC’s 57.79% with p=0.0116p=0.0116. Changing the historical collection from tabular tasks to image tasks reduces tabular AUROC and AUPRC by 1.80 and 2.39 percentage points, respectively, providing additional evidence of transfer within the evaluated setting of tabular features and fixed ResNet18 image embeddings.
Critical weaknesses
  1. LOTUS [ref.1] already uses low-rank Gromov-Wasserstein dataset distances to transfer detectors and hyperparameters from the nearest historical task. GOOD’s distinctive change is the combination of multi-scale WL/MFD comparisons with learned performance regression. However, Table 2 omits LOTUS, and Table 12 only substitutes measurements within GOOD, leaving the additional capability or empirical benefit of this design over the closely related prior selector insufficiently established.
  2. The main empirical gains are difficult to assess because the evaluation does not clearly separate selecting the meta-learner design from measuring its performance. Appendix G chooses NN with MSE based on the aggregate leave-one-out results in Table 10, while the described 20% validation split does not establish that regressor-family and loss selection were isolated from the outer evaluation. Since the case for the method rests substantially on improved selection performance, uncertainty about selection optimism directly limits the strength of that evidence.

REF:

  1. Singh, P., & Vanschoren, J. (2023). AutoML for outlier detection with optimal transport distances. In 32nd International Joint Conference on Artificial Intelligence, IJCAI 2023 (pp. 7175-7178). International Joint Conferences on Artificial Intelligence (IJCAI).
Questions for authors
  1. Were the regressor family and loss in Table 10 fixed independently of the outer evaluation or selected within each outer training fold, and what performance estimate preserves that separation?
  2. Can you provide the exact AUPRC implementation and detector-score vectors for the six CIFAR-10 entries reported as 0.5476 in Table 16? Clarifying integration and ties is necessary to interpret these results.
  3. Under the same historical tasks and candidate pool, how does GOOD compare with the low-rank Gromov-Wasserstein nearest-task selector in LOTUS, and what benefit is specifically attributable to learning from the combined historical comparisons?
  4. Was the WL kernel normalized or computed on equally sampled graphs, and does its relationship with detector-ranking similarity persist after controlling for sample count?
  5. Appendix F proposes a lightweight Nyström-based variant, but Table 9 reports only detection performance. Can you clarify how the graph required by WL is constructed from the Nyström representation and provide total runtime and peak memory measurements, including graph construction, to substantiate the claimed computational savings?
1 comment

The additional baseline comparisons, particularly with LOTUS, strengthen the evidence for GOOD’s performance, and the rebuttal addresses most of my concerns. My main reservation is that selecting the meta-learner across evaluation folds may have inflated the reported gains. On balance, I maintain my rating of 3.

Reviewer_KUmJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GOOD, a meta-learning framework for unsupervised anomaly detection model selection. Instead of relying on traditional statistical meta-features, it maps heterogeneous datasets into sample-wise similarity graphs and extracts meta-features using Weisfeiler-Lehman (WL) kernels and Minimum Factorization Distance (MFD). A meta-learner is then trained to predict the best detector and hyperparameter configuration for a new, unlabeled dataset, aiming to avoid expensive online hyperparameter optimization (HPO). Experiments on tabular and image testbeds show improved selection accuracy over existing baselines. However, the framework's reliance on O(n^2 )sample-wise graph construction during online inference severely limits its scalability for large datasets, contradicting its primary motivation of minimizing online cost.

Critical strengths

Problem formulation is practically relevant. The paper tackles the important and challenging problem of unsupervised anomaly detection model selection. Given the high sensitivity of anomaly detectors to hyperparameters and the lack of labels in real-world tasks, automating the selection of both algorithms and hyperparameter configurations is a meaningful direction.

Critical weaknesses
  1. The method fails to fulfill its primary motivation of avoiding high online inference costs. The paper's core selling point is overcoming the "high online inference cost of HO." However, Figure 1.(a) shows that when the sample size reaches approximately 1,400, GOOD's inference time catches up with and nearly matches MetaOD. Since the method is only faster on small datasets, the claim that this graph-based replacement for statistical meta-features is superior remains unsubstantiated.

  2. Figure 1(b) shows GOOD outperforms MetaOD (Sta-LDM) by about 15 points in AUROC (81.06 vs 65.18) and 16 points in AUPRC. While this is a significant improvement, MetaOD's biggest advantage is its lightweight nature (just over 100 seconds for 10,000 samples). To achieve this 15-point gain, GOOD's inference time on large samples becomes several times higher than MetaOD. In a real-world scenario, one could use MetaOD to select a slightly suboptimal model in 100 seconds and still have ample time to run it on the data. In contrast, GOOD spends 500 seconds just constructing meta-features for selection. This fundamentally questions whether the framework truly achieves the goal of "avoiding expensive online search."

  3. The baseline comparison for 2026 is missing:

    (a) Comparison with UniOD (ICLR 2026). The paper does not discuss recent frameworks like UniOD, which bypasses the model selection paradigm entirely by training a universal model for direct inference on new datasets. UniOD requires no historical graph comparisons and achieves comparable AUROC (around 78–79%). In large-sample scenarios, if GOOD takes hundreds of seconds for graph kernel comparisons while UniOD performs a direct forward pass in seconds with similar performance, what is the practical significance of GOOD's complex meta-feature graph computation?

    (b) Comparison with MetaEns (ICML 2026). MetaEns performs ensemble selection using only statistical summaries of detector scores, completely avoiding the construction of n×n sample-wise graphs. Consequently, its online computational overhead is extremely low. If MetaEns can select highly effective models (or compact ensembles) within seconds, does GOOD's O(n^2) graph kernel computation add any necessary value, or is it essentially redundant?

[1] Fu, Dazhi, and Jicong Fan. "Uniod: a universal model for outlier detection across diverse domains." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Phan, Hong-Phuc, et al. "Automatic Unsupervised Ensemble Outlier Model Selection."Forty third International Conference on Machine Learning. 2026.

Questions for authors

For details, please refer to ``Critical Weaknesses''.

3 comments

Dear Reviewer KUmJ,

Thank you very much for your time and effort throughout the review discussion.

We would like to kindly ask whether you could consider our response, in which we clarified in detail the research motivation, main contributions and empirical results and comparisons with the recent baselines.

We would sincerely appreciate your feedback you may have. Your insights are highly valuable and will help us improve our work.

Thank you again for your careful consideration.

Sincerely,

Authors

Reviewer_KUmJOpenReview ↗

Response

Thank you for the author's reply. I will increase my score.

Dear Reviewer,

We sincerely appreciate your feedback and support.

Regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.