- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies label-free OOD detection using frozen pretrained representations across vision and language models. The authors compare the Mahalanobis detector and ReSCOPED (a diffusion-based detector trained on ID representations). They observe that both methods improve with stronger pretrained representations.
Critical strengths
- The paper investigates an interesting question about the relationship between pretrained representations and their ability to detect novelties based on a predefined ID set.
- Using both global and local distribution modeling to show the same takeaway is useful.
- The experiments have good breadth. The authors cover multiple models and OOD sets across vision and language.
Critical weaknesses
I find the research question relevant and the empirical observations are also potentially interesting. My main concern is not with the label-free setting, but I find the comparisons and scientific claims made in the paper not aligned with the problem being studied.
-
Problem formulation and positioning: In the conventional OOD detection setting, a classifier is trained on known classes and the objective is to identify inputs for which the classifier should abstain from making predictions [1]. For example, a digit classifier may confidently classify an image of a frog as a digit, although the input belongs outside its intended distribution. In this context, having a supervised classifier and its associated distribution is part of the problem formulation. Therefore, it is misleading when the authors present the use of class labels or supervised training as a limitation of prior work.
-
Ambiguity in the ID/OOD definition and confounding factors: Again in the typical OOD detection setting, the reference distribution is closely tied to the training distribution of the learned model and the motivation is to understand whether the model is seeing something different from what it can answer. However, in this paper, the models (DINO, Qwen) are trained on large-scale datasets whose relationship to the evaluated ID/OOD sets is unclear. This paper bypasses the problem of having a limited learner. Additionally, when the training distribution of the model is detached from the problem formulation, the motivation for a separate calibration set for thresholding also becomes less clear. In the case of iid and sufficiently large data, choosing the 99th percentile for Mahalanobis using the full ID data or a separate calibration/validation set should make little difference. For this reason, I like ReSCOPED better as it brings the limited data distribution seen by the learner back into the formulation.
-
Scientific novelty: I find the main observation that stronger pretrained representations lead to better ID/OOD discrimination somewhat expected, especially given prior observations on the benefits of large-scale pretraining and frozen representations for OOD detection [2, 3]. The relationship between representation geometry and Mahalanobis OOD performance has also been studied in [4]. The observation that the performance gap between Mahalanobis and ReSCOPED decreases with stronger representations is potentially interesting. However, I am not convinced that comparing only two detectors is sufficient to support the broader claim that the choice of the detector becomes less important with stronger representations. The paper also provides limited insight into why this gap decreases or under what conditions a simple global detector becomes sufficient.
-
Benchmarks: I appreciate the authors for using OpenOOD near-OOD experiments. I was expecting to see the corresponding far-OOD results as well. Is there a reason to exclude them and use CIFAR10 and CelebA instead? The results on these smaller benchmarks look highly saturated.
Questions for authors
See weaknesses.
Minor:
- In L072, the motivation does not really follow from the previous observation.
References:
[1] Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024). Generalized out-of-distribution detection: A survey. International Journal of Computer Vision, 132(12), 5635–5662. https://doi.org/10.1007/s11263-024-02117-4
[2] Fort, Stanislav, Jie Ren, and Balaji Lakshminarayanan. "Exploring the limits of out-of-distribution detection." Advances in neural information processing systems 34 (2021): 7068-7081.
[3] Zhao, Shizhen, et al. "Equipping vision foundation model with mixture of experts for out-of-distribution detection." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
[4] Janiak, Denis, Jakub Binkowski, and Tomasz Kajdanowicz. "A geometry-based view of Mahalanobis OOD detection." arXiv preprint arXiv:2510.15202 (2025).
3 comments
W3. Scientific novelty
We disagree with the characterization of our contribution as simply showing that stronger pretrained representations improve OOD detection. We explicitly discuss the cited prior work and distinguish our contributions from it. Fort et al. demonstrate the benefits of supervised fine-tuning for near-OOD detection, which we directly revisit in Section 4.1 and Figure 3, showing that modern frozen representations can achieve comparable performance without fine-tuning. We also acknowledge Zhao et al.'s findings on frozen DINOv2 representations in our introduction and cite Janiak et al.'s analysis of Mahalanobis representation geometry in Section 2.1.
Our central contribution is instead to investigate whether improvements in pretrained representations reduce performance differences between fundamentally different detectors, and whether this relationship persists across modalities and nonsaturated benchmarks. As discussed in our response to FtWE (W1/Q1), we deliberately compare global Mahalanobis and local ReSCOPED as two contrasting probes of identical frozen representations.
On ImageNet-200, their average absolute AUROC gap decreases from 0.093 to 0.037 to 0.005 across DINOv3-S/B/L. On SST, it decreases from 0.175 to 0.058 to 0.002 across Qwen3-0.6B/1.7B/4B. These results demonstrate convergence even when detection remains nontrivial.
We also do not claim that all detectors become equivalent. Additional detector families would broaden the generality of our findings, while measurements in representation space could provide deeper mechanistic insight. As discussed in our response to FtWE (W3/Q3), our contribution is to establish the empirical relationship, which motivates further investigation into the representation properties responsible for it.
Finally, our results identify empirical settings in which simple global detection becomes competitive, while determining the precise geometric conditions responsible is an important direction for future work.
W4. Benchmark selection
Our historical CIFAR/SVHN/CelebA experiments demonstrate that many previously challenging OOD benchmarks approach saturation with modern frozen representations. We provide these comparisons to establish continuity with prior diffusion-based OOD literature, their previously reported results, and to show that unsupervised OOD detection using frozen representations can saturate benchmarks for previously hard tasks.
Because these benchmarks often saturate, we deliberately prioritize even harder near-OOD evaluations from OpenOOD v1.5. The ImageNet-200 comparisons against SSB-hard and NINCO remain substantially below saturation, allowing us to investigate whether detector performance convergence persists when meaningful detection difficulty remains.
OpenOOD classifies its far-OOD benchmarks as easier detection tasks. While including them would broaden our evaluation, we prioritized the harder near-OOD benchmarks because they provide a more informative test of our central hypothesis, without the confounding effects of AUROC saturation.
Minor comment
We will clarify the transition at line 72.
References
[1] Yang et al., https://doi.org/10.1007/s11263-024-02117-4
[2] Lang et al., https://arxiv.org/abs/2305.03236
I thank the authors for their response. While the rebuttal clarifies some points, my main concerns remain.
First, I agree that label-free OOD detection is a legitimate setting. However, the central issue is the distinction between detecting deviations from an arbitrary downstream ID distribution and detecting inputs that are unfamiliar to OR unreliable for the pretrained backbone. The authors define OOD independently of the backbone's pretrained distribution and it remains unclear how the resulting detection scores relate to the model's reliability or capabilities.
Second, the comparisons with prior methods do not properly consider the substantial external data used to pretrain the backbones. Therefore, framing the additional information required by supervised OOD detectors as a limitation is misleading without accounting for the substantial external data used to pretrain the backbones.
Finally although the convergence between Mahalanobis and ReSCOPED is interesting, comparing only two detectors is not sufficient to support the broad conclusion that detector choice becomes less important with stronger representations.
I appreciate the additional discussion and explanations but I remain unconvinced that the evidence sufficiently supports paper's broader claims. Therefore I will maintain my original score.
We thank the reviewer for recognizing the relevance of our research question and the breadth of our experiments. We address the critical weaknesses and corresponding questions together below.
W1. Problem formulation
We agree that conventional classifier-based OOD detection often concerns identifying inputs that differ from the classifier’s training distribution, since performing predictions on such inputs may be unreliable. However, this is not the only formulation of OOD detection. Indeed, the reviewer's cited survey [1] introduces a generalized OOD detection framework encompassing anomaly detection, novelty detection, open-set recognition, OOD detection, and outlier detection. The survey also explicitly notes that density-based OOD detection does not require ID classification (Figure 1).
Furthermore, Lang et al. [2] explicitly categorize OOD detection methods according to the availability of OOD data and ID labels (Section 3). Section 3.3 specifically addresses OOD detection when neither OOD data nor ID labels are available, noting that this setting applies to non-classification tasks and remains relatively underexplored in NLP.
We also disagree that presenting the requirements for class labels or supervised training as limitations of prior work is misleading. These are additional information and training requirements that fundamentally restrict the settings in which such methods can be applied. In many applications, particularly those involving pretrained foundation models and language models, task-specific class labels are unavailable or impractical to obtain. Requiring supervised fine-tuning also introduces additional computational costs and necessitates maintaining task-adapted models.
Our paper investigates OOD detection when only unlabeled downstream ID data and a frozen pretrained backbone are available. As discussed in our introduction, removing the requirements for class labels and backbone fine-tuning is a central motivation for our work. While supervised classifier-based OOD detection is a legitimate formulation, its additional information and training requirements are real limitations in many practical settings. These limitations directly motivate the label-free setting investigated in our paper.
W2. ID/OOD definition and pretraining distribution
We disagree that using broadly pretrained representations creates ambiguity in the ID/OOD definition. As discussed in W1, our setting deliberately does not require a task-specific supervised learner. Modern foundation models provide representations that can be reused across downstream tasks without retraining the backbone. However, broad pretraining does not eliminate the need for OOD detection. As discussed in Section 2.2, even generative models trained to estimate data likelihood can assign higher likelihood to OOD samples than ID samples.
We also disagree with the distinction between Mahalanobis and ReSCOPED. Both are fitted using exactly the same downstream ID training representations. Mahalanobis estimates their mean and covariance, while ReSCOPED trains a diffusion observer on them. Both therefore model the downstream ID distribution, differing only in how they model its structure.
In our setting, the downstream ID distribution is defined by the data used to fit the detector, independently of the backbone's pretraining distribution.
Finally, our primary results report AUROC, which requires no calibration threshold. The 99th-percentile threshold in Figure 4 illustrates a particular operating point on the ROC curve for LLM gatekeeping.