acceptodds
Under review as a conference paper at ICLR 2027

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

Abstract

Models trained with deep learning often fail to signal when inputs fall outside their training data manifold, leading to unreliable predictions under distribution shift. Prior work suggests that effective out-of-distribution (OOD) detection often requires some combination of class-conditional modeling, sophisticated detectors, and specialized models obtained through supervised fine-tuning. We ask whether, in modern pretrained models, label-free OOD detection is increasingly limited by the representation being probed rather than the sophistication of the detector. Across 59 backbone–task pairings spanning vision and language, we compare two complementary label-free detectors: a global Mahalanobis estimator fit on unlabeled latent representations, and ReSCOPED, a lightweight, diffusion-based typicality estimator operating on the same features at a local level. As pretrained representations scale, both detectors improve substantially, previously challenging benchmarks approach saturation even with simple label-free detection, and the performance gap between the two methods shrinks. This same trend persists on harder, non-saturated vision and language benchmarks. These results suggest that label-free OOD detection depends strongly on the geometry exposed by frozen pretrained backbones, reducing the importance of detector choice as backbone scale increases and enabling efficient deployment directly on frozen models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Out-of-distribution detection, Unsupervised learning, Typicality, Generative modeling, Diffusion models, Representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DGaDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies label-free OOD detection using frozen pretrained representations across vision and language models. The authors compare the Mahalanobis detector and ReSCOPED (a diffusion-based detector trained on ID representations). They observe that both methods improve with stronger pretrained representations.

Critical strengths
  • The paper investigates an interesting question about the relationship between pretrained representations and their ability to detect novelties based on a predefined ID set.
  • Using both global and local distribution modeling to show the same takeaway is useful.
  • The experiments have good breadth. The authors cover multiple models and OOD sets across vision and language.
Critical weaknesses

I find the research question relevant and the empirical observations are also potentially interesting. My main concern is not with the label-free setting, but I find the comparisons and scientific claims made in the paper not aligned with the problem being studied.

  • Problem formulation and positioning: In the conventional OOD detection setting, a classifier is trained on known classes and the objective is to identify inputs for which the classifier should abstain from making predictions [1]. For example, a digit classifier may confidently classify an image of a frog as a digit, although the input belongs outside its intended distribution. In this context, having a supervised classifier and its associated distribution is part of the problem formulation. Therefore, it is misleading when the authors present the use of class labels or supervised training as a limitation of prior work.

  • Ambiguity in the ID/OOD definition and confounding factors: Again in the typical OOD detection setting, the reference distribution is closely tied to the training distribution of the learned model and the motivation is to understand whether the model is seeing something different from what it can answer. However, in this paper, the models (DINO, Qwen) are trained on large-scale datasets whose relationship to the evaluated ID/OOD sets is unclear. This paper bypasses the problem of having a limited learner. Additionally, when the training distribution of the model is detached from the problem formulation, the motivation for a separate calibration set for thresholding also becomes less clear. In the case of iid and sufficiently large data, choosing the 99th percentile for Mahalanobis using the full ID data or a separate calibration/validation set should make little difference. For this reason, I like ReSCOPED better as it brings the limited data distribution seen by the learner back into the formulation.

  • Scientific novelty: I find the main observation that stronger pretrained representations lead to better ID/OOD discrimination somewhat expected, especially given prior observations on the benefits of large-scale pretraining and frozen representations for OOD detection [2, 3]. The relationship between representation geometry and Mahalanobis OOD performance has also been studied in [4]. The observation that the performance gap between Mahalanobis and ReSCOPED decreases with stronger representations is potentially interesting. However, I am not convinced that comparing only two detectors is sufficient to support the broader claim that the choice of the detector becomes less important with stronger representations. The paper also provides limited insight into why this gap decreases or under what conditions a simple global detector becomes sufficient.

  • Benchmarks: I appreciate the authors for using OpenOOD near-OOD experiments. I was expecting to see the corresponding far-OOD results as well. Is there a reason to exclude them and use CIFAR10 and CelebA instead? The results on these smaller benchmarks look highly saturated.

Questions for authors

See weaknesses.

Minor:

  • In L072, the motivation does not really follow from the previous observation.

References:

[1] Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024). Generalized out-of-distribution detection: A survey. International Journal of Computer Vision, 132(12), 5635–5662. https://doi.org/10.1007/s11263-024-02117-4

[2] Fort, Stanislav, Jie Ren, and Balaji Lakshminarayanan. "Exploring the limits of out-of-distribution detection." Advances in neural information processing systems 34 (2021): 7068-7081.

[3] Zhao, Shizhen, et al. "Equipping vision foundation model with mixture of experts for out-of-distribution detection." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[4] Janiak, Denis, Jakub Binkowski, and Tomasz Kajdanowicz. "A geometry-based view of Mahalanobis OOD detection." arXiv preprint arXiv:2510.15202 (2025).

3 comments

We thank the reviewer for recognizing the relevance of our research question and the breadth of our experiments. We address the critical weaknesses and corresponding questions together below.

W1. Problem formulation

We agree that conventional classifier-based OOD detection often concerns identifying inputs that differ from the classifier’s training distribution, since performing predictions on such inputs may be unreliable. However, this is not the only formulation of OOD detection. Indeed, the reviewer's cited survey [1] introduces a generalized OOD detection framework encompassing anomaly detection, novelty detection, open-set recognition, OOD detection, and outlier detection. The survey also explicitly notes that density-based OOD detection does not require ID classification (Figure 1).

Furthermore, Lang et al. [2] explicitly categorize OOD detection methods according to the availability of OOD data and ID labels (Section 3). Section 3.3 specifically addresses OOD detection when neither OOD data nor ID labels are available, noting that this setting applies to non-classification tasks and remains relatively underexplored in NLP.

We also disagree that presenting the requirements for class labels or supervised training as limitations of prior work is misleading. These are additional information and training requirements that fundamentally restrict the settings in which such methods can be applied. In many applications, particularly those involving pretrained foundation models and language models, task-specific class labels are unavailable or impractical to obtain. Requiring supervised fine-tuning also introduces additional computational costs and necessitates maintaining task-adapted models.

Our paper investigates OOD detection when only unlabeled downstream ID data and a frozen pretrained backbone are available. As discussed in our introduction, removing the requirements for class labels and backbone fine-tuning is a central motivation for our work. While supervised classifier-based OOD detection is a legitimate formulation, its additional information and training requirements are real limitations in many practical settings. These limitations directly motivate the label-free setting investigated in our paper.

W2. ID/OOD definition and pretraining distribution

We disagree that using broadly pretrained representations creates ambiguity in the ID/OOD definition. As discussed in W1, our setting deliberately does not require a task-specific supervised learner. Modern foundation models provide representations that can be reused across downstream tasks without retraining the backbone. However, broad pretraining does not eliminate the need for OOD detection. As discussed in Section 2.2, even generative models trained to estimate data likelihood can assign higher likelihood to OOD samples than ID samples.

We also disagree with the distinction between Mahalanobis and ReSCOPED. Both are fitted using exactly the same downstream ID training representations. Mahalanobis estimates their mean and covariance, while ReSCOPED trains a diffusion observer on them. Both therefore model the downstream ID distribution, differing only in how they model its structure.

In our setting, the downstream ID distribution is defined by the data used to fit the detector, independently of the backbone's pretraining distribution.

Finally, our primary results report AUROC, which requires no calibration threshold. The 99th-percentile threshold in Figure 4 illustrates a particular operating point on the ROC curve for LLM gatekeeping.

W3. Scientific novelty

We disagree with the characterization of our contribution as simply showing that stronger pretrained representations improve OOD detection. We explicitly discuss the cited prior work and distinguish our contributions from it. Fort et al. demonstrate the benefits of supervised fine-tuning for near-OOD detection, which we directly revisit in Section 4.1 and Figure 3, showing that modern frozen representations can achieve comparable performance without fine-tuning. We also acknowledge Zhao et al.'s findings on frozen DINOv2 representations in our introduction and cite Janiak et al.'s analysis of Mahalanobis representation geometry in Section 2.1.

Our central contribution is instead to investigate whether improvements in pretrained representations reduce performance differences between fundamentally different detectors, and whether this relationship persists across modalities and nonsaturated benchmarks. As discussed in our response to FtWE (W1/Q1), we deliberately compare global Mahalanobis and local ReSCOPED as two contrasting probes of identical frozen representations.

On ImageNet-200, their average absolute AUROC gap decreases from 0.093 to 0.037 to 0.005 across DINOv3-S/B/L. On SST, it decreases from 0.175 to 0.058 to 0.002 across Qwen3-0.6B/1.7B/4B. These results demonstrate convergence even when detection remains nontrivial.

We also do not claim that all detectors become equivalent. Additional detector families would broaden the generality of our findings, while measurements in representation space could provide deeper mechanistic insight. As discussed in our response to FtWE (W3/Q3), our contribution is to establish the empirical relationship, which motivates further investigation into the representation properties responsible for it.

Finally, our results identify empirical settings in which simple global detection becomes competitive, while determining the precise geometric conditions responsible is an important direction for future work.

W4. Benchmark selection

Our historical CIFAR/SVHN/CelebA experiments demonstrate that many previously challenging OOD benchmarks approach saturation with modern frozen representations. We provide these comparisons to establish continuity with prior diffusion-based OOD literature, their previously reported results, and to show that unsupervised OOD detection using frozen representations can saturate benchmarks for previously hard tasks.

Because these benchmarks often saturate, we deliberately prioritize even harder near-OOD evaluations from OpenOOD v1.5. The ImageNet-200 comparisons against SSB-hard and NINCO remain substantially below saturation, allowing us to investigate whether detector performance convergence persists when meaningful detection difficulty remains.

OpenOOD classifies its far-OOD benchmarks as easier detection tasks. While including them would broaden our evaluation, we prioritized the harder near-OOD benchmarks because they provide a more informative test of our central hypothesis, without the confounding effects of AUROC saturation.

Minor comment

We will clarify the transition at line 72.

References

[1] Yang et al., https://doi.org/10.1007/s11263-024-02117-4

[2] Lang et al., https://arxiv.org/abs/2305.03236

I thank the authors for their response. While the rebuttal clarifies some points, my main concerns remain.

First, I agree that label-free OOD detection is a legitimate setting. However, the central issue is the distinction between detecting deviations from an arbitrary downstream ID distribution and detecting inputs that are unfamiliar to OR unreliable for the pretrained backbone. The authors define OOD independently of the backbone's pretrained distribution and it remains unclear how the resulting detection scores relate to the model's reliability or capabilities.

Second, the comparisons with prior methods do not properly consider the substantial external data used to pretrain the backbones. Therefore, framing the additional information required by supervised OOD detectors as a limitation is misleading without accounting for the substantial external data used to pretrain the backbones.

Finally although the convergence between Mahalanobis and ReSCOPED is interesting, comparing only two detectors is not sufficient to support the broad conclusion that detector choice becomes less important with stronger representations.

I appreciate the additional discussion and explanations but I remain unconvinced that the evidence sufficiently supports paper's broader claims. Therefore I will maintain my original score.

Reviewer_CdV6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper reports that scaling frozen pretrained vision and language representations enables strong label-free OOD detection without class labels or finetuning, pushing previously hard benchmarks toward saturation even with a simple global Mahalanobis detector. Comparing this global method with ReSCOPED, a new diffusion-based detector, the authors find that detector differences are large and dataset-dependent for weaker representations but shrink as backbones scale. They also identify a consistent plateau in diffusion noise level where ReSCOPED performs best across Transformer models and modalities. Finally, they illustrate a practical, low-latency application as a prefill-stage OOD gatekeeper for LLMs.

Critical strengths
  1. A controlled study with a useful takeaway.

Holding two very different detectors fixed while varying the frozen backbone across 59 pairings, the paper shows consistently that stronger representations make detector choice matter less, even on harder near-OOD benchmarks. A simple label-free Mahalanobis detector on frozen features can match a more sophisticated method, shifting focus from detector design to the representations being probed.

Critical weaknesses
  1. Scale is confounded.

Scaling conflates model size with changes in training data, objectives, and architecture (e.g., DINOv1 and DINOv3), and the size range is modest (DINOv3 S/B/L, Qwen3 up to 4B), so the evidence is correlational rather than isolating scale as the cause. The paper also offers no theoretical account of why representation geometry improves or why the denoising plateau emerges, leaving the central claim largely descriptive.

  1. Uneven baselines and unclear added value of ReSCOPED.

The vision baselines are mostly generative models trained from scratch on raw pixels, while the proposed detectors use DINOv3 pretrained on ~1.7B images, and no baseline on the same frozen features (e.g., k-NN) or OpenOOD baseline on ImageNet-200 is included, so the gains largely reflect the backbone rather than the detectors. Combined with thin language evaluation (two benchmarks, 8 pairings) and ReSCOPED rarely beating the simpler Mahalanobis, the case for ReSCOPED as a contribution beyond a contrast probe remains weak.

Questions for authors
  1. How do your detectors compare to other detectors on the same frozen features?

Since the baselines in Table 1 are mostly pixel-space generative models trained from scratch, what does the comparison actually claim? Adding simple feature-space baselines such as k-NN, and standard OpenOOD baselines for the ImageNet-200 tasks would show whether ReSCOPED and Mahalanobis are strong detectors or just benefit from DINOv3.

  1. What exactly do you mean by scaling?

Can you separate within-family size sweeps (DINOv3 S/B/L, Qwen3) from cross-generation (DINOv1 to DINOv3) and cross-family (BERT vs. Qwen3) comparisons, where data, objectives, and architecture also change? How does distillation in the smaller models affect the interpretation?

  1. What is the true computational cost?

Please report backbone parameter counts, pretraining data size, and wall-clock time including the backbone forward pass. The #F + #J metric treats a DINOv3 L pass the same as a small diffusion-model pass, which makes the efficiency claims hard to interpret.

  1. How was the noise level chosen, and how does it relate to Figure 5?

σ was selected on SVHN vs. CelebA, which is also a reported test task, so is there leakage? Please also clarify the inconsistency between σ ≈ 0.09 (main text) and 0.099 (appendix), and how the plateau “near t = 5” in Figure 5 maps to σ.

  1. When should a practitioner use ReSCOPED over Mahalanobis?

Mahalanobis is simpler, cheaper, and matches or beats ReSCOPED in most settings, and ReSCOPED lags clearly on SST with smaller backbones. The LLM gatekeeping use case should be evaluated quantitatively (e.g., detection accuracy versus prefix length) rather than only through the qualitative example in Figure 4.

4 comments

We thank the reviewer for their assessment. Please find our response below, and note that overlapping issues are addressed in our FtWE response.

W1/Q2. Scaling and mechanism We distinguish two forms of representation improvement in our paper. DINOv1/v2/v3 ViT-B all have approximately 85–86M parameters, so cross-generation comparisons examine improvements in pretrained representations without increasing backbone size. Although pretraining data, objectives, and some architectural details differ, these comparisons demonstrate advances at approximately fixed model capacity. Separately, within-family comparisons span DINOv3-S/B/L (~21–300M parameters, ~14x) and Qwen3-0.6B/1.7B/4B (~6x). Together, these experiments examine representation improvements across both pretraining generations and model sizes, without claiming that parameter count alone causes the observed gains. While we absolutely would have liked to have further isolated these effects, we do not have the compute resources required to do the necessary experiments that would fully isolate changes to training data, objectives, and minor architecture changes for Qwen and DINO style models. Such experiments would have required full-scale training of these models with careful isolation applied and likely substantial hyperparameter tuning as well.

The AUROC gap shrinks on ImageNet-200 across DINOv3 (0.093->0.037->0.005) and SST across Qwen3 (0.175->0.058->0.002). BERT/Qwen3 adds historical context. Our finding concerns improvements in pretrained representations, not size alone, but we were careful about our wording because what constitutes a “better” representation is a very active research area. As mentioned in our response to FtWE (W3/Q3), no complete theory is needed to establish these empirical trends. We hope our work motivates the OOD detection community to commit to studying representations alongside improved detectors.

W2/Q1/Q3/Q5. Baselines, deployment cost, and detector roles

We apologize if our presentation caused confusion. The key contributions block at the end of our introduction identifies Mahalanobis and ReSCOPED as illustrative global and local detectors, not ReSCOPED as a standalone algorithmic contribution or a universally preferred method.

Table 1's pixel-space baselines are intended to provide historical context, not as an argument of universal detector superiority. Our finding is that modern pretrained representations enable strong label-free detection even with simple estimators. When a model backbone is already being used to accomplish downstream tasks (e.g. self-driving or image segmentation), its intermediate features can be reused for OOD detection without a separate large generative model. As mentioned in the paper, Mahalanobis fits ID mean/covariance statistics; ReSCOPED trains a small diffusion model observer on the same features.

This also partly answers Q3. When the backbone already is used for inference for downstream tasks, feature extraction is already being done. As a result the only remaining costs are that Mahalanobis adds the representation distance computation and ReSCOPED adds a diffusion observer forward pass and JVP, which is 5.4 ms of added latency on Qwen3-4B (using an RTX 4090). We can make this clearer in the final paper. Likewise, #F + #J follows SCOPED [6] and counts evaluations, not hardware-normalized runtime - we agree that this is a coarse comparison and can make this clearer alongside the argument around backbones already being used for downstream applications. While we could measure backbone inference times for all models in our study, these measurements would not be directly comparable to prior methods, because wall-clock measurements are not provided for them. This is why we reported the 5.4 ms additional latency for ReSCOPED on Qwen3-4B to provide a concrete indication of the marginal computational overhead when the backbone is already used for downstream inference. To provide further computational context as requested by the reviewer, the published model size and pretraining corpus figures [1-5] are DINOv1-B 85M/1.28M images, DINOv2-B 86M/142M images, DINOv3-S/B/L 21/86/300M with a 1.689B-image teacher corpus, BERT 110M/~3.3B tokens, and Qwen3 0.6/1.7/4B with 36T pretraining tokens.

As explained in our response to FtWE (W1/Q1, W2/Q2), both detectors use identical features across 59 backbone-task pairings. k-NN would test broader generality and OpenOOD baselines add context, without changing their convergence on nonsaturated ImageNet-200. As for the comment on thin language evaluations, the language OOD literature has fewer standardized benchmarks with directly comparable published results under our fully label-free, frozen-backbone setting than the vision literature. We selected CLINC150 and SST because they provide established protocols and published baselines. While additional language OOD datasets exist, many prior evaluations use different assumptions, including supervised adaptation or additional representation training. Extending our evaluation to new benchmarks while using the same algorithmic baselines for comparison would therefore require reproducing these methods. In lieu of this effort, we provided our language experiments as cross-modal evidence for the trends observed in vision, rather than an exhaustive language OOD benchmark.

With regard to Q5, ReSCOPED is meant to contrast in locality and complexity with Mahalanobis - our intention was never to propose it as a superior method. In our experiments it sometimes beats Mahalanobis (BERT/CLINC150: 0.815 vs. 0.791; DINOv3-B/ImageNet-200: 0.834 vs. 0.797), but performs worse on smaller Qwen3 SST and nearly matches at 4B. Where performance is comparable, we recommend the simpler-to-implement Mahalanobis probe. In regards to evaluating the LLM gatekeeping case qualitatively, Table 2 evaluates detector performance against the full query, rather than against the partial queries shown in Figure 4. A streaming benchmark would be a worthwhile future direction of study that we could mention in a future work discussion section, but we think for this work it would expand the scope beyond our central representation study.

Q4. Noise selection and Figure 5

We selected sigma=0.099 using SVHN vs. CelebA and held it fixed for all subsequent experiments. Since this pair also appears in Appendix Table 3 for evaluation, we acknowledge that selecting sigma on this task introduces leakage, but when we introduce our choice of sigma in Section 3.2 we immediately include a caveat saying why the practical impact of the leakage is negligible. Figure 5 demonstrates that performance remains high across a broad range of noise levels across all 59 configurations, including both vision and language, which is further investigated in Section 5.2. This suggests that our results are not dependent on choosing a narrowly optimized noise level or leakage. For the final paper, we are happy to run another held-out data pair that isn’t included in our results to minimize confusion for readers.

The sigma=0.09 reported in the main text was a typo and should have matched the sigma=0.099 specified in Appendix E. We will use 0.099 consistently in the revised manuscript. Figure 5 uses a reversed, zero-indexed 20-step EDM noise schedule [7]. The selected sigma=0.099 corresponds to the fifth point in Figure 5, within the broad plateau near t=5. We will clarify the sigma-to-t mapping in the appendix of the final paper.

References

[1] Caron et al., https://arxiv.org/abs/2104.14294

[2] Oquab et al., https://arxiv.org/abs/2304.07193

[3] Siméoni et al., https://arxiv.org/abs/2508.10104

[4] Devlin et al., https://arxiv.org/abs/1810.04805

[5] Yang et al., https://arxiv.org/abs/2505.09388

[6] Barkley et al., https://arxiv.org/abs/2510.01456

[7] Karras et al., https://arxiv.org/abs/2206.00364

Reviewer_CdV6OpenReview ↗

Answer to Author's comments

I thank the authors for their detailed response.

W1/Q2: Scaling and mechanism

Separating cross-generation comparisons at fixed size from within-family size sweeps is helpful. As the cross-generation and cross-family comparisons also change data, objectives and architecture, the evidence for scaling rests on the within-family sweeps; there, my question on distillation in DINOv3-S/B/L is not answered. On the small-scale benchmarks, most tasks are near saturation and the detector gap is already small at S, so they say little about convergence. On ImageNet-200 and SST, the gap shrinks, but each trend rests on three model sizes. On ImageNet-200, the leading detector changes from S to B. On SST, the convergence comes from ReSCOPED catching up, while Mahalanobis does not improve with Qwen3 scale (0.939 / 0.934 / 0.951) and remains below BERT (0.965). This supports a descriptive observation rather than a scaling result. The mechanism is deferred to the response to reviewer FtWE, which declares it out of scope, leaving the geometric interpretation without direct measurements.

W2/Q1: Baselines on the same features

The clarification of ReSCOPED's role and the rationale for the language evaluation are noted. However, apart from the BERT baselines, no detector other than Mahalanobis and ReSCOPED is evaluated on the same frozen features. Since the central claim is that detector choice matters less as representations improve, such baselines are its direct test, and two detectors are a minimal one. Neither response provides them, although k-NN is inexpensive and is reported to be competitive in work the paper cites. The authors state that they do not claim the trend holds for all detector families, yet the abstract and conclusion still present it generally.

Q3: Computational cost

The added model sizes and pretraining-data figures are helpful. However, the argument that only the detector adds cost assumes a backbone already running downstream, which does not hold for the Table 1 comparison. There, #F counts a DINOv3 forward pass the same as a pass through a baseline's diffusion model, and wall-clock time including the backbone is still not reported.

Q4: Noise level

The clarifications resolve my confusion. The mapping of σ=0.099 to t=4 is consistent with a standard EDM schedule. Since ReSCOPED is saturated on SVHN vs. CelebA (≈1.000 AUROC, Table 3), the leakage is likely negligible, but this task also gives little guidance for choosing σ within the plateau. The robustness of σ therefore rests on Figures 5 and 7 rather than on the selection itself. The held-out selection is promised but not reported.

Q5: ReSCOPED vs. Mahalanobis, and gatekeeping

Given ReSCOPED's clarified role as a contrast probe, the recommendation to prefer Mahalanobis is reasonable. However, the cases where ReSCOPED is better can only be identified with OOD data, so no practical selection criterion is given. For full-prompt gating, Table 2 and the reported latency provide support. The prefix-level scoring in Figure 4 remains qualitative, and its quantitative evaluation is deferred to future work.

Overall, the research question is relevant, and I appreciate the authors' directness about compute limits. However, the rebuttal adds no new experiments, and the main concerns remain:

  • no baselines beyond the two detectors on the same frozen vision and Qwen3 features
  • scaling evidence confounded across generations and families, and limited to three sizes with unaddressed distillation within families
  • geometric interpretations without direct measurements of the representations
  • no quantitative evaluation of prefix-level gatekeeping

The promised changes do not affect the main conclusions. I therefore maintain my score.

Reviewer_FtWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether stronger pretrained representations reduce the need for complex OOD detectors. It compares label-free Mahalanobis detection with ReSCOPED, a diffusion-based detector, on frozen vision and language backbones. The results suggest that larger pretrained models generally improve OOD detection and narrow the performance gap between the two detectors.

Critical strengths

1.The paper explores an interesting question: whether stronger pretrained representations reduce the need for complex OOD detectors.

2.Experiments cover both vision and language, including challenging near-OOD benchmarks.

3.The approach requires neither class labels nor backbone fine-tuning, making it useful when labeled data is unavailable.

Critical weaknesses

1.The paper uses only the diffusion-based ReSCOPED as a representative complex detector, and its representativeness requires further validation. Comparing it solely with Mahalanobis seems insufficient to conclude that detector complexity generally becomes less important as pretrained models scale. Additional types of detectors would help establish whether this trend holds more broadly.

2.In Table 1, results on ImageNet-200 are missing for all other baselines, although this benchmark provides important evidence for challenging OOD scenarios. Comparing only Mahalanobis and ReSCOPED is insufficient to fully assess their relative performance on this benchmark. Adding representative baselines under the same label-free setting would make the comparison more complete and convincing.

3.The authors interpret the results as indicating that stronger representations make OOD information more accessible to both global and local methods. However, the evidence mainly consists of AUROC values and detection score distributions, with no direct measurements of the representation geometry itself.

4.Some figures, particularly the bar charts, are difficult to read due to the color choices and font sizes.

Questions for authors

1.Does the observed trend hold for other types of OOD detectors beyond Mahalanobis and ReSCOPED?

2.Could you include additional label-free baselines on ImageNet-200 to provide a more complete comparison?

3.Could you provide direct measurements of representation geometry to support your interpretation of the results?

4.Could you improve the readability of the figures, particularly the colors and font sizes in the bar charts?

1 comment

We thank the reviewer for recognizing the relevance of our research question and the breadth of our evaluation. We address the critical weaknesses and corresponding questions together below.

W1/Q1. Generality across detector families

Our objective is not to establish that all OOD detectors become equivalent as representations scale, but to investigate whether improvements in pretrained representations reduce the importance of detector choice, and whether this relationship persists across modalities and nonsaturated benchmarks.

We deliberately compare global Mahalanobis, which models the ID distribution through a single covariance matrix, with ReSCOPED, which learns a diffusion-based score function to estimate local score-curvature typicality. These represent fundamentally different approaches to modeling the same representation distribution, allowing us to examine whether increasingly informative representations diminish the performance advantages of more sophisticated detectors. Importantly, both operate on identical frozen features, allowing us to directly examine how their relative performance changes with backbone quality. The observed convergence persists across vision and language, including nonsaturated ImageNet-200 and SST benchmarks.

We do not claim that ReSCOPED represents every sophisticated detector, nor that the trend necessarily holds for all detector families. Additional detectors would strengthen the generality of our findings, but our experiments already establish that performance differences between these two distinct detector constructions diminish as representations improve.

W2/Q2. Additional ImageNet-200 baselines

The ImageNet-200 experiments address whether detector convergence is an artifact of AUROC saturation, rather than serving as comparisons against state-of-the-art methods on those benchmarks.

On SSB-hard and NINCO, the average absolute AUROC gap between Mahalanobis and ReSCOPED decreases from 0.093 to 0.037 to 0.005 across DINOv3-S/B/L, despite detection performance remaining substantially below saturation.

We acknowledge that additional label-free baselines could provide additional evidence to support the central hypothesis, i.e. that differences in performance between methods decline with increasing representation scale. However, additional baselines would not change the observed relationship between Mahalanobis and ReSCOPED, and the finding that a simple detector can match a more sophisticated one as representations improve is itself a substantial contribution.

We therefore believe the existing ImageNet-200 experiments support the specific convergence finding, while additional baselines would broaden its generality.

W3/Q3. Representation geometry

We agree that direct geometric measurements could provide additional mechanistic insight. However, our paper is an empirical investigation of how pretrained representation improvements affect OOD detection performance and the relative behavior of different detector constructions, not a mechanistic study of why these improvements occur.

Our experiments directly investigate this question by applying two fundamentally different detectors to identical frozen representations across backbone families and scales. The resulting AUROC trends establish the observed performance improvements and detector convergence, including on nonsaturated benchmarks. Figure 2 further illustrates changes in ID/OOD score separation as representations improve.

Identifying the precise geometric properties responsible for these trends is a separate research question. Indeed, an important implication of our work is that greater attention should be directed toward the properties of pretrained representations themselves, instead of improving detector design.

Our findings provide a foundation for investigating which geometric properties enable effective OOD detection and how representation learning might be improved to expose OOD-relevant information more effectively. We view the reviewer's suggested analysis as a valuable direction motivated by our findings and are happy to include the suggestion into a future work discussion in the final manuscript.

W4/Q4. Figure readability

We agree and will improve the font sizes and color distinguishability, particularly in the bar charts, in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.