acceptodds
Under review as a conference paper at ICLR 2027

Beyond Confidence Thresholding: Mitigating Cache Corruption in Test-Time OOD Detection

Abstract

Out-of-distribution (OOD) detection aims to identify test samples deviating from in-distribution (ID) data. Recent test-time adaptation methods tackle this challenge by dynamically updating visual caches to retrieve negative labels that delineate the ID decision boundary. However, these methods implicitly assume balanced test streams and suffer catastrophic performance degradation under realistic, ID-dominated streaming deployments in challenging Near-OOD scenarios. In this work, we reveal that this failure stems from an inherent objective misalignment between margin-driven negative label retrieval and confidence-based cache updates. By projecting high-dimensional representations onto a scalar confidence metric, existing cache updates discard local neighborhood geometry, mistakenly purging informative boundary negatives in favor of coarse, uninformative concepts. To resolve this dilemma, we propose a graph-driven training-free framework that replaces heuristic scalar thresholding with a mutual -NN bipartite graph constructed between visual embeddings and textual concepts, introducing a topological connectivity gate to govern cache admission via geometric proximity. By preserving the cross-modal manifold structure, our approach reliably mines informative negative labels that are acutely sensitive to the fine-grained boundary between ID and Near-OOD data. Extensive evaluations on challenging Near-OOD benchmarks demonstrate that our framework effectively prevents boundary collapse, outperforming state-of-the-art baselines by up to 10% AUROC under severe streaming imbalances while remaining highly competitive in balanced settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
OOD Detection, VLM-OOD Detection, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AJfBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies test-time OOD detection with vision-language models under ID-dominated streams and attributes cache corruption to a mismatch between confidence-based admission and hard-negative retrieval. It proposes a training-free mutual-kNN visual-text graph with multi-scale disparity scores and a topological cache gate, reporting strong Near-OOD results across imbalance ratios and backbones but weaker ImageNet Far-OOD performance.

Critical strengths
  • The studied problem is important.
  • The reported Near-OOD gains are consistent across imbalance ratios and backbones.
  • The evaluation includes useful ablations and semantic-alignment visualizations.
Critical weaknesses
  • The studied problem is important.
  • The reported Near-OOD gains are consistent across imbalance ratios and backbones.
  • The evaluation includes useful ablations and semantic-alignment visualizations.
Questions for authors
  1. The realistic-streaming motivation is not established. Section 4.2 creates imbalance by subsampling OOD datasets, but does not specify stream order, bursts, batch composition, cache resets, update timing, or full-spectrum mixing. Please provide the complete online protocol and justify how it represents deployment streams.

  2. The claimed cause of cache failure is not isolated. Ours(TANL−Cache)Ours (TANL-Cache) and the R1R1 comparison do not form a complete negative-label-selector × cache-updater control. Please report this factorial ablation and the true ID/OOD/boundary composition of both caches over time.

  3. The cache gate may be undefined or degenerate for small batches. Eqs. (8)–(9) use k=70k=70 on a current batch of size BB, but BB is unreported; when B<kB<k, visual Top-kk is undefined or contains all samples, and for B=1B=1 the skeleton connects every text to the same image. What BB was used, and how stable is the gate across BB, CC, and NactiveN_active?

  4. The theoretical result does not derive the algorithm. Eqs. (3)–(4) provide an algebraic analogy to message passing, while Appendix A.1 assumes the exterior-shell model and the similarity orderings required by Eqs. (22)–(30). Which assumptions are verified, and what result specifically supports mutual-kkNN, the skeleton, and Eq. (18)?

  5. The necessity of several components remains unclear. Table 2 ablates disparity scores and kk, but not mutual-kkNN versus the Top-1 skeleton, text-based KMeans initialization, or a simple cosine/kkNN cache gate. Please include these controls to distinguish the proposed mechanism from a simpler geometric rule.

  6. The conclusions should reflect the observed Near-/Far-OOD trade-off. Table 3 shows lower ImageNet Far-OOD AUROC and higher FPR@95 than TANL for every backbone, whereas Table 4 shows the opposite trend on CIFAR-100. Please explain this inconsistency and scope the robustness claim accordingly.

  7. The statistical evidence is difficult to interpret. Table 1 averages over five different OOD ratios, and several Figure 2 margin error bars cross zero despite the claim of “strictly positive and wide” margins. Please provide per-ratio numerical results, the source of randomness, confidence intervals or significance tests, and a more qualified mechanism claim.

  8. The computational and reproducibility details are incomplete. The paper omits NcorN_cor, graph-recomputation frequency, NactiveN_active, runtime, peak memory, latency, the negative-text corpus, prompt template, KMeans details, and stream seeds. Please report these items and compare efficiency with TANL and the other test-time baselines.

Reviewer_dgRSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies a major limitation with previous test-time adaptation OOD detection studies, that the performance of these methods degrade substantially in imbalanced cases where the ID data is dominant. To resolve this issue, they propose a novel mechanism.

Critical strengths
  1. The described limitation of previous test-time adaptation methods is valid and quite interesting.
  2. The proposed method is novel and involves valid mechanisms.
Critical weaknesses
  1. The main issue is that the paper is very hard to follow. Several parts use complex terminology without providing intuitive, easy-to-follow explanations. The equations could be explained in simpler terms, with clearer descriptions of the purpose. The connection between the message-passing interpretation and the proposed method is also unclear.

  2. The reasoning behind the claim that ID-dominated test sets lead to boundary ID samples entering the positive cache is not convincing (lines 118–120). Because most incoming samples are ID, there is even a higher chance that more confident ID samples be selected. The authors should explain why ID dominance increases the positive cache’s similarity to informative hard-negative concepts.

  3. For the experiment, ratios like 2% might be very small to evaluate. Retaining only 2% of an OOD dataset such as NINCO may leave very few OOD samples, particularly relative to the large ID dataset. Moreover, Figure 1 suggests that the proposed method performs slightly below TANL on SSB-hard when retaining 20% or more of the original OOD data.

  4. It is unclear how the full-spectrum (FS) results are computed. Section 4.1 lists ImageNet-C, ImageNet-R, and ImageNet-V2, but does not clearly explain whether their results are averaged into a single score. The construction of the FS streams and the aggregation procedure should be specified.

  5. Another important limitation is the lower performance on ImageNet Far-OOD benchmarks relative to several baselines. Although the authors explicitly target Near-OOD detection and acknowledge this trade-off, maintaining strong Far-OOD performance remains important for broader applicability. The degradation is also not marginal. For example, at the 10% OOD sub-sampling level, Figure 3 shows a gap of about 4–5% AUROC relative to AdaNeg on the Far-OOD. Furthermore, dataset-specific AUROC and FPR95 results for iNaturalist, Textures, and OpenImage-O are not reported, making it difficult to assess whether the degradation is consistent across far-OOD datasets.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_8v7cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that cache-based test-time negative-label OOD detectors for CLIP (e.g., AdaNeg [1], TANL [2]) collapse on Near-OOD data when OOD samples are rare in the test stream. It attributes this to confidence-thresholded cache admission, which it claims is misaligned with margin-driven negative-label mining. It proposes a training-free fix: mutual-k-NN bipartite graphs between cached image features and text labels are used (i) to rank negative labels by graph-masked cache disparities and (ii) to route each test sample into the positive or negative cache according to its graph-masked similarity to ID versus active negative labels. On ImageNet-1K with SSB-hard/NINCO subsampled to 2–100%, the paper reports gains of up to ~10 AUROC over prior test-time methods at low OOD proportions, at some cost on Far-OOD.

References

  • [1] Y. Zhang and L. Zhang. AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models. NeurIPS, 2024.
  • [2] Y. Zhang, M. Varma, Y. Gao, J.-B. Delbrouck, J. Liu, C. Wang, and C. Langlotz. Activation Matters: Test-time Activated Negative Labels for OOD Detection with Vision-Language Models. CVPR, 2026.
Critical strengths
  • The method is clearly specified, with every step defined by an explicit equation (graph construction, the three disparity scores, label retrieval, scoring, cache routing and initialization; Eqs. 5–19). The per-proportion curves (Figs. 1, 3) show directly where it helps (low-ratio Near-OOD) and where it does not (Far-OOD).
Critical weaknesses
  • The "implicitly presume balanced arrivals" premise is contradicted by AdaNeg [1], which already studies ID:OOD imbalance up to 1000:1, attributes the degradation to "misclassified samples in the memory" and fixes it with a ratio-adaptive gap (AdaGap), and by ZS-NTTA [3], whose detector (pseudo-labelled with the OWTTT variance threshold [4] that TANL [2] adopts) "tends to misclassify numerous clean samples as noisy ones" on all-ID streams; the method itself recombines TANL's cache-disparity score with classical mutual-k-NN graphs [5] and the k-NN graphs over test samples and text/class prototypes already used for training-free VLM adaptation [6, 7] and OOD scoring [8]. Neither AdaGap nor simple prior-robust admission rules (e.g., a threshold calibrated on OpenOOD's 5,000 held-out ID images [9] or minimum-error thresholding [10]) are compared, so the paper cannot show that graph topology, rather than any admission rule that does not assume a bimodal score history, is what fixes the failure.
  • The ablations do not isolate the claimed cause: with R1 alone the gate roughly matches TANL on Near-OOD (78.01 vs. 77.47 AUROC) but loses ≈6 Far-OOD AUROC (88.42 vs. 94.6), R1+R2+R3 under TANL's cache trails TANL by ≈6.5 AUROC even on the full SSB-hard stream (Fig. 1), and Fig. 2 gives the widest ID–OOD margin to R1, so the gains may come from ensembling three NegLabel [11] scores rather than from graph structure, and cache corruption itself is never measured. The theory does not justify the design: the Sec. 3.2 "equivalence" is the generic log-sum-exp gradient, whose message-passing interpretation was already established [12] and which implies nothing about mutual-k-NN sparsification at inference, and Appx. A.1 is a qualitative narrative (e.g., Eqs. 26–27 drop the ID terms of the score) rather than a proof.
  • The x-axis is the fraction of each OOD set retained, not the stream OOD rate (with OpenOOD's 45,000 ID test images [9], NINCO spans only 0.26–11.6% OOD versus 2.1–52% for SSB-hard), and the batch size B, on which the gate's graph depends (for B = 1 it reduces to a plain mean-similarity comparison), the stream order, and hyperparameter selection (k is swept on the test sets) are all unreported. The Far-OOD cost is described as "1–2% AUROC", yet the method is last of five methods at every Far-OOD proportion (Fig. 3), has 6.8–8.6 points higher FPR95 than TANL on all four backbones (Table 3), and also trails TANL at the near-balanced SSB-hard 100% point (≈82.1 vs. ≈83.0 AUROC).

References:

  • [3] C. Cao, Z. Zhong, Z. Zhou, T. Liu, Y. Liu, K. Zhang, and B. Han. Noisy Test-Time Adaptation in Vision-Language Models. ICLR, 2025.
  • [4] Y. Li, X. Xu, Y. Su, and K. Jia. On the Robustness of Open-World Test-Time Training: Self-Training with Dynamic Prototype Expansion. ICCV, 2023.
  • [5] M. R. Brito, E. L. Chávez, A. J. Quiroz, and J. E. Yukich. Connectivity of the Mutual k-Nearest-Neighbor Graph in Clustering and Outlier Detection. Statistics & Probability Letters, 35(1):33–42, 1997.
  • [6] V. Stojnić, Y. Kalantidis, and G. Tolias. Label Propagation for Zero-shot Classification with Vision-Language Models. CVPR, 2024.
  • [7] Y. Li, Y. Su, A. Goodge, K. Jia, and X. Xu. Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model. ICLR, 2025.
  • [8] T. Chen, Y. Li, A. Goodge, F. Teng, X. Yang, T. Li, and X. Xu. Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation. ICCV, 2025.
  • [9] J. Zhang, J. Yang, P. Wang, H. Wang, Y. Lin, H. Zhang, Y. Sun, X. Du, Y. Li, Z. Liu, Y. Chen, and H. Li. OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. Journal of Data-centric Machine Learning Research (DMLR), 2024.
  • [10] J. Kittler and J. Illingworth. Minimum Error Thresholding. Pattern Recognition, 19(1):41–47, 1986.
  • [11] X. Jiang, F. Liu, Z. Fang, H. Chen, T. Liu, F. Zheng, and B. Han. Negative Label Guided OOD Detection with Pretrained Vision-Language Models. ICLR, 2024.
  • [12] Y. Wang, Q. Zhang, T. Du, J. Yang, Z. Lin, and Y. Wang. A Message Passing Perspective on Learning Dynamics of Contrastive Learning. ICLR, 2023.
Questions for authors
  • On the same streams, how does the method compare with AdaNeg + AdaGap [1] and with TANL [2] under prior-robust admission (a threshold calibrated on OpenOOD's held-out ID images [9], minimum-error thresholding [10], or the true stream OOD rate as an oracle)? If these recover TANL's performance at low ratios, what remains of the diagnosis that confidence-based updates "discard local neighborhood geometry"?
  • Can you report a factorial of {TANL admission, proposed gate, gate with A ≡ 1} × {TANL's ranking, R1, R2, R3, R1+R2+R3} with per-ratio Near- and Far-OOD AUROC and FPR95, an ensemble control (three R1-ranked label sets with matched |I_active|), and direct cache statistics (the true-OOD fraction of X_pos and X_neg over the stream)?
  • What batch size B is used, how do results change for B ∈ {1, …, 512} and when re-indexed by the actual stream OOD rate (including a balanced NINCO stream), and were k, β, L and τ chosen on OpenOOD's validation splits [9] rather than on the test sets? Why are the reproduced DDE [13] and ANTS [14] results at 100% (≈80.5/≈82.0 and ≈74.0/≈79.6 AUROC on SSB-hard/NINCO, Fig. 1) roughly 3–5 points below the values their authors report?

References:

  • [13] W. Zhu, Y. Zhang, L. Xu, X. Jin, W. Zeng, and L. Zhang. Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs. arXiv:2606.25758, 2026.
  • [14] W. Zhu, Y. Zhang, X. Jin, W. Zeng, and L. Zhang. ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoning. CVPR, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.