acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fixed Geometry: Reliable Test-Time Adaptation of Vision-Language Models

Abstract

Under test-time distribution shift, vision–language models can fail along two distinct reliability axes: the pretrained similarity geometry may misweight target- relevant feature directions, while augmented views may remain confidently wrong. We introduce a composite heteroscedastic reference model whose MAP rule exposes two distinct reliability roles: directional comparison precision and view-specific observation precision. Guided by these roles, GeoRel (Geometry– Reliability Factorized Test-Time Adaptation) learns a shared positive-semidefinite comparison geometry from weak cross-modal class evidence through discrimina- tive attraction, separation, and redundancy conditioning, while ranking augmented observations through scale-free agreement between frozen-space query consistency and adaptive predictive certainty. EMA separately stabilizes noisy target-visual evidence used by the geometry-adapted prediction. Across natural-shift and cross- dataset benchmarks with RN50 and ViT-B/16, GeoRel achieves strong and broadly competitive performance. The resulting principle is simple: adapt the comparison geometry, but anchor view reliability in frozen space.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Adaptation, Vision-Language Models, Cross-Modal Geometry, Metric Learning, Online Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zHxNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method, GeoRel for online test time adaptation for CLIP but with batch size set to one. GeoRel splits adaptation into two parts - (1) the geometry that compares class evidence, and (2) the reliability of augmented views. On top of ReTA, it adds multiple components such as:

  • a shared diagonal-plus-low-rank metric
  • trained with attraction term
  • margin and redundancy losses
  • EMA visual prototypes
  • agreement-sensitive reliability score
  • aggregation rule
  • a two-branch fusion

GeoRel gains about 1 pp over ReTA on the TPT benchmark suite.

Critical strengths
  1. The two-role framing, which separates comparison geometry from view reliability, offers an interesting perspective on VLM-TTA.
  2. The authors justify most design choices. They compare shared and modality-specific transforms at equal parameter counts, which shows that the benefit of sharing does not come from added capacity.
  3. Gains hold across both backbones and both benchmark families.
Critical weaknesses
  1. GeoRel adds a learned metric with three losses, EMA prototypes, a rank-based view score and two-branch fusion to ReTA. Eq. (2) is proposed at the principle as it separates two reliability roles: M_t determines where class evidence is compared by weighting feature directions, while {γ_i} determine which observations dominate inference. It remains unclear how much of the improvement the principle itself accounts for.
  2. Some components do not follow directly from the two-role principle. The paper describes EMA as added stabilization mechanism and branch fusion as an operational design choice. Each choice is reasonable engineering, but together they make the link between the principle and the full method less direct.
  3. Since TTA offers no target validation set, it becomes an important point to understand how to choose hyperparameters and how many hyperparameters to choose to judge the method's effectiveness. With a mean gain near 1 pp on a heavily reused benchmark suite, this protocol determines whether the gain reflects the method or the tuning.
  4. The paper reports no latency or memory. When comparing with a method like ZERO which reaches competitive accuracy without optimization, one can not omit this study.
Questions for authors
  1. What does the minimal method (frozen CLIP, learned W, anchored rank weighting) achieve against CLIP, TDA and ReTA?
  2. Please report a cumulative table (base, +geometry, +reliability, +EMA, +fusion) on Final accuracy for all datasets.
  3. How do the authors select the new hyperparameters? Do they stay fixed across datasets and backbones?
  4. Please share also the standard deviation for all methods alongside the reported mean as the gains are in a marginal bracket and are important in this setting.
  5. What are the per-image latency and memory against ReTA, TDA and ZERO?
Reviewer_pMzrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoRel extends ReTA by learning a shared diagonal-plus-low-rank comparison transform and selecting reliable augmented views. Geometry learning uses prototype attraction, class separation, and decorrelation. View selection combines frozen-space query consistency with predictive certainty. Final predictions fuse an EMA-supported clean-query branch with selected-view outputs. Evaluation covers ImageNet, four natural shifts, and ten additional datasets using CLIP-RN50 and ViT-B/16.

Critical strengths
  • Direct support for geometry adaptation: Table 6 reports higher Primary accuracy on all six datasets, averaging a 0.90-percentage-point gain. Shared transforms also win 25 of 32 comparisons at matched parameter counts, supporting the shared-geometry design.

  • Broad empirical gains: GeoRel achieves the highest reported accuracy in 24 of 30 dataset–backbone comparisons. Relative to ReTA, natural-shift averages improve by 0.93/0.97 percentage points and cross-dataset averages by 1.31/0.83 points for RN50/ViT-B/16, respectively.

Critical weaknesses
  • Selector comparisons do not isolate the proposed contribution: Figure 10 uses accuracy against random selection but CWR@5 against certainty-only selection. CWR@5 counts only errors with certainty ranks of at least 0.8; reducing it does not establish lower overall error. Matched accuracy comparisons are needed to identify the benefits of frozen-space consistency and the proposed rank fusion.

  • Incomplete reproducibility and uncertainty reporting: Appendices B–C lack complete settings for geometry losses, optimization, EMA, and prediction fusion. Hyperparameter selection, dataset-specific settings, and matched baseline reruns are unclear. Three-seed averages lack variability estimates, limiting confidence in gains near one percentage point. Robustness to incorrect early cache entries also remains unresolved.

Questions for authors
  • Under identical views, adaptation states, (K), and prediction fusion, how do certainty-only, consistency-only, arithmetic-rank fusion, and adapted-space consistency compare in selected-view, Auxiliary, and Final accuracy?

  • What are the complete defaults and selection procedures? Are settings fixed across datasets and backbones, were target labels used, and when are adaptation states reset?

  • How is cache-logit correction computed in Eqs. (14) and (20)? Are queries and cache keys transformed by the current (W)? When does cache admission occur, and do geometry losses update prompt residuals?

  • Which baselines were rerun, and what is the variability of paired GeoRel–ReTA gains under matched seeds, stream orders, and augmentations?

  • How robust is geometry learning to systematic early misclassifications, and what explains the weaker results on EuroSAT and Food101?

Reviewer_ed9wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates test-time adaptation of vision-language models under distribution shifts and proposes GeoRel, which separately models “how to compare features” and “which augmented views are worth trusting.” Building on ReTA’s prompt residuals and cache mechanism, the method introduces a shared diagonal-plus-low-rank transformation and learns the comparison geometry through cross-modal prototype attraction, inter-class margins, and a decorrelation loss. Meanwhile, it combines view consistency in the frozen CLIP space with adaptive prediction certainty through rank normalization and geometric fusion to select reliable augmented views, which are then fused with the primary branch using EMA prototypes. The method offers a potentially valuable approach.

Critical strengths
  1. The paper discusses feature comparison geometry and augmented-view reliability separately, accurately identifying that “high prediction confidence” does not necessarily mean that an augmented view preserves valid semantics. This provides a useful perspective for understanding test-time adaptation of VLMs.
  2. The paper provides valuable mechanism-level validation. Experiments on the geometry switch, parameter-matched comparisons, reliability stratification, and fixed-state replay establish meaningful connections between the method’s design and the observed results, offering stronger evidence than reporting final accuracy alone.
  3. The authors explicitly clarify the differences between the reference model and the actual algorithm, and acknowledge risks such as pseudo-label contamination, EMA retaining erroneous information, and reliability signals failing under severe distribution shifts. The appendix also reports cases where fusion introduces errors, presenting a relatively balanced assessment.
Critical weaknesses
  1. Metric learning, pseudo-label caching, EMA prototype updates, consistency-based filtering, and multi-branch fusion all have precedents in related work. The paper needs to demonstrate more clearly that its core contribution goes beyond combining existing components, using targeted comparisons to establish the additional benefits of geometric adaptation and reliability ranking over alternative approaches.
  2. The paper uses a heteroscedastic Gaussian model to explain comparison geometry and observation reliability. However, the actual algorithm learns geometry through a discriminative loss and constructs reliability scores using consistency and prediction certainty. The current theory primarily provides design motivation and does not establish a theoretical connection to the performance of the actual algorithm.
  3. Geometric adaptation relies on pseudo-labels and visual evidence stored in the cache, but these pseudo-labels may contain systematic biases. EMA can smooth updates, but it does not necessarily eliminate persistent error feedback. The paper lacks sufficient quantitative experimental analysis of pseudo-label contamination, error accumulation, and performance under extreme distribution shifts.
Questions for authors
  1. Metric learning, pseudo-label caching, EMA prototype updates, consistency-based filtering, and multi-branch fusion all have precedents in related work. The paper needs to demonstrate more clearly that its core contribution goes beyond combining existing components, using targeted comparisons to establish the additional benefits of geometric adaptation and reliability ranking over alternative approaches.
  2. The paper uses a heteroscedastic Gaussian model to explain comparison geometry and observation reliability. However, the actual algorithm learns geometry through a discriminative loss and constructs reliability scores using consistency and prediction certainty. The current theory primarily provides design motivation and does not establish a theoretical connection to the performance of the actual algorithm.
  3. Geometric adaptation relies on pseudo-labels and visual evidence stored in the cache, but these pseudo-labels may contain systematic biases. EMA can smooth updates, but it does not necessarily eliminate persistent error feedback. The paper lacks sufficient quantitative experimental analysis of pseudo-label contamination, error accumulation, and performance under extreme distribution shifts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.