acceptodds
Under review as a conference paper at ICLR 2027

AGI-TPT: Adaptive Geometry-Informed Test-Time Prompt Tuning for Calibrated Zero-Shot Vision-Language Models

38 views

Abstract

Test-time prompt tuning (TPT) adapts a vision–language model such as CLIP to each test image by lowering the entropy of its prediction. It improves accuracy but makes the model over-confident, and when the prompt is carried along a stream of test images it collapses. We give a mechanism: entropy minimization pulls together the classes that compete for an image. We propose AGI-TPT, a regularizer whose target adapts to the number of classes N and the feature dimension D, and whose push adapts to each pair’s current state. It pushes apart only the pairs of class text features whose cosine is above the attainable level c⋆(N,D), the smallest largest cosine that N unit vectors in D dimensions can have, and it applies no force to pairs below it. The level is computed from N and D, not tuned. For free class vectors we prove that the zeros of the loss are the maximally separated configurations, that its low-rank local minima are global, and that merged classes are never local minima (N≤2D−1). We also show that at a single TPT step, spread regularizers differ mainly in strength: at equal strength AGI-TPT coincides with O-TPT’s penalty, and a random reversal of TPT’s step of the same strength comes within 0.13 ECE of it. The geometry matters when adaptation continues. There, among the prompt-tuning methods with a single template that we compare, AGI-TPT gives the best accuracy and calibration. With ViT-B/16 on 11 datasets, online, it reaches 68.8% average accuracy at ECE 1.38; at a single step it lowers the ECE of A-TPT, the best-calibrated published method, from 2.61 to 1.81 under the same protocol, and under sustained adaptation from 2.9 (A-TPT with a drift remedy) to 1.4. On four ImageNet shifts it reaches 65.3% accuracy at ECE 1.40. It keeps more of CLIP’s class-similarity structure than any baseline, and its calibration does not drift along the stream. Code will be made publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, CLIP, Test-Time Prompt Tuning, Zero-Shot Learning, Model Calibration, Equiangular Tight Frames, Representation Learning, Multimodal Learning, Trustworthy AI, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pNhtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AGI-TPT adds a one-sided squared-hinge penalty on pairwise text-feature cosines to test-time prompt tuning. Its stopping level is motivated by attainable spherical-code separation for the class count and feature dimension, with certified levels used in the more difficult regime. The paper analyzes the limited ability of one-step comparisons to distinguish geometric objectives and evaluates calibration, accuracy, and preservation of class geometry under sustained online adaptation.

Critical strengths

The distinction between one-step strength effects and sustained changes in class geometry is a useful insight, and the paper backs it up with matched-strength controls. I also like the idea of stopping the separation penalty for pairs that have reached the chosen level, since the goal is to preserve useful class relationships while avoiding collapse. The online experiments, component comparisons, and label-shift tests give the method a much stronger empirical case than a one-step calibration result alone would. The paper also makes a real effort to spell out the limits of the free-feature theory and to show cases where spreading classes can hurt.

Critical weaknesses
  1. Could we separate the pairwise force from the actual update a bit more clearly? Proposition 10 gives a nonzero pairwise derivative below the attainable level, but at a regular simplex those tangent contributions can cancel, leaving the regularizer's total gradient at zero. I'm therefore not quite sure whether the 'persistent offset' in S1.1 means a nonzero loss value or a gradient that keeps moving the features. Checking that small example and showing the actual prompt-space regularizer gradient along one existing online run would help connect the stopping argument to the observed stability; this doesn't require a new general convergence guarantee for prompts.

  2. I was a little unsure how the fixed-random-direction control in S5.3 gets to an override mass of 0.58. With a nonnegative scale, only coordinates where the direction opposes the entropy gradient can ever be overridden, so scaling a data-independent symmetric random direction still gives an expected maximum mass of 0.5. A particular direction can exceed 0.58, so I'm not saying the reported run is impossible. Could you explain how directions are sampled or selected when they cannot reach the target, and report the mass actually reached? That would make this specific matched-strength comparison much easier to interpret.

  3. Could we double-check the EuroSAT reachability row in Table S11? If its columns describe the same initial and final configurations, the initial mean cosine of 0.74, final maximum of 0.10, and 47% of pairs below the level imply a loss-ratio upper bound of about 0.53 * (0.10 + 1/9)^2 / (0.74 + 1/9)^2 = 0.0326, whereas the table gives 0.046. I may be missing a difference in how the columns are aggregated, so it would help to explain that or recompute them from the same Gram matrices. This matters because the table is one of the main links between the free-feature theory and what the shared prompt actually reaches.

Questions for authors

For the three points above, could you clarify the meaning of the persistent offset and the simplex cancellation case, the sampling rule and achieved strength of the fixed random direction, and the aggregation used for the EuroSAT reachability row? Those clarifications would help me assess the mechanism and its supporting diagnostics.

Reviewer_TP2qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates class-feature collapse and calibration degradation during test-time prompt tuning (TPT). The authors propose AGI-TPT, which introduces a geometry-informed separation threshold c∗(N,D)c^*(N,D) and a one-sided ReLU regularizer to selectively separate class features. Theoretical analyses and experiments across multiple benchmarks demonstrate improved calibration, preservation of class-feature geometry, and stability during online adaptation.

Critical strengths

S1. Clear motivation and theoretically grounded design. The paper analyzes how entropy minimization may bring competing class features closer and introduces a selective geometric regularizer.

S2. Meaningful experimental validation. The paper provides mechanistic interventions, sustained online adaptation experiments, and evaluations across multiple CLIP backbones.

Critical weaknesses

W1. The theoretical guarantees concern freely optimizable class vectors rather than practical shared-prompt optimization. Moreover, AGI-TPT and strength-adjusted O-TPT achieve similar one-step ECE (1.81% vs. 1.86%), leaving the independent benefit of the proposed geometry less clear.

W2. Despite the theoretical complexity analysis, empirical latency, throughput, and peak memory measurements are missing. Robustness and calibration under controlled distribution shifts, such as PUG-ImageNet, also remain unexplored.

Questions for authors

Q1. Could the authors verify whether the theoretical geometric properties hold under shared-prompt optimization and clarify the benefits over simpler regularizers during sustained online adaptation?

Q2. Could the authors provide empirical efficiency comparisons and consider evaluating accuracy and ECE on PUG-ImageNet under controlled visual variations?

I appreciate the authors' efforts and look forward to their responses. I am open to revisiting my assessment and adjusting my rating based on the clarifications and evidence provided during the rebuttal.

Reviewer_YFp7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies overconfidence and collapse in test-time prompt tuning of vision–language models. It proposes AGI-TPT, a squared-hinge regularizer that penalizes pairwise class-text similarities only when they exceed a geometry-dependent threshold. The threshold corresponds to the optimal worst-pair cosine for N unit vectors in D dimensions when N ≤ 2D, and a certified attainable upper bound otherwise.

The paper also analyzes how fixed-weight regularizers affect the first Adam step, showing that much of the one-step calibration improvement can be explained by update strength. It develops geometric guarantees for freely optimized class vectors and evaluates the method under both episodic and sustained online adaptation. The reported results show improved accuracy and calibration, with ablations supporting the importance of a stopping threshold and a one-sided penalty.

Critical strengths
  1. The one-step analysis provides a useful methodological insight. Proposition 1 and the strength-matched comparisons in Appendix G.3 (Table 7) show that calibration gains can largely reflect regularization strength, rather than geometric design. The authors’ inclusion of their own method in this analysis strengthens the argument.
  2. The online experiments provide convincing empirical support through tuned baselines and controlled ablations. Across six datasets, AGI-TPT achieves 56.8% accuracy and 1.4 ECE, compared with 52.4% and 2.9 for A-TPT with SAR. Table 11 in Appendix G.6 further supports the importance of the stopping threshold and one-sided penalty.
Critical weaknesses
  1. The handling of AdamW’s moment estimates across test images is unclear from the protocol description. Clarifying this detail would improve the reproducibility of the online experiments.
  2. Cumulative ECE and final-prompt reevaluation may mask temporary calibration deterioration. More direct temporal evidence would strengthen the “no drift” claim.
Questions for authors
  1. Are AdamW’s first- and second-moment estimates retained across test images or reset? Please specify the treatment for AGI-TPT and the reproduced online baselines.
  2. Can you provide, or point to existing, windowed ECE results from the sequential predictions, particularly under label shifts and repeated streams? Please specify the window sizes.
  3. Section 7 acknowledges possible over-separation of genuinely confusable classes. On Aircraft or Cars, do the accuracy and calibration gains also hold for highly confusable class groups, for example those identified by zero-shot CLIP’s confusion matrix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.