acceptodds
Under review as a conference paper at ICLR 2027

RA-TPT: Stopping Logit Inflation at Test time

Abstract

Vision-Language Models (VLMs) enable strong zero-shot transfer, yet their predictive probabilities can become miscalibrated under distribution shift. Test-Time Prompt Tuning (TPT) adapts continuous prompts using unlabeled target inputs and can improve target-domain accuracy; however, its entropy-minimization objective may increase predictive concentration for incorrect or ambiguous samples. We propose Reliability-Aware Test-Time Prompt Tuning (RA-TPT), a logit-space approach for improving the calibration-accuracy trade-off during test-time adaptation. RA-TPT replaces entropy minimization with an uncertainty-aware bounded-margin objective and a pseudo-label anchor that provide a self-training signal while limiting margin-expansion pressure on uncertain inputs. It further applies uncertainty-routed centered-logit dispersion regularization to discourage excessive relative logit variation when predictions are uncertain, with a complementary mean-logit regularizer. A soft conflict-aware gradient projection attenuates the regularization-gradient component that opposes the separability gradient. Across 31 benchmarks spanning standard classification, ImageNet distribution shifts, biomedical imaging, and remote sensing, RA-TPT improves the accuracy-calibration trade-off in CLIP-style adaptation, substantially reducing Expected Calibration Error (ECE) versus entropy-based TPT and matching calibration-aware baselines at competitive accuracy. Because the objective acts purely on logits, it further transfers to non-VLM test-time adaptation, matching strong corruption-robustness baselines on CIFAR-10/100-C and ImageNet-C.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Adaptation, Test-Time Prompt Tuning, Vision-Language Models, Model Calibration, Expected Calibration Error, Trustworthy AI, Logit-Space Regularization, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CP8DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Reliability-Aware Test-Time Prompt Tuning (RA-TPT) to improve the calibration–accuracy trade-off of vision-language models under distribution shift. RA-TPT addresses overconfident predictions caused by test-time entropy minimization by adaptively controlling logit separation according to sample uncertainty. It combines an uncertainty-aware bounded margin, centered-logit dispersion regularization, and a soft conflict-aware gradient projection. Experiments on 31 benchmarks show improved calibration with competitive accuracy across CLIP-based, domain-specific, and non-VLM models.

Critical strengths
  • The proposed method directly operates in the logit space, which the authors identify as the source of the issue, and adaptively adjusts the regularization according to the uncertainty of each sample, rather than applying the same regularization uniformly to all samples.

  • The method achieves the lowest average ECE among the compared approaches. Although the difference in ECE compared with SoC is not statistically significant, the improvement in accuracy is statistically significant. Its VRAM usage is comparable to that of TPT, while incurring only about a 10% increase in latency. Compared with O-TPT, it reduces memory usage by 55% on ImageNet-Sketch with RN50.

  • The experimental evaluation is extensive, covering 31 benchmarks, multiple backbones and domains (CLIP RN50/ViT-B/16, BioMedCLIP, and RemoteCLIP), as well as non-VLM models.

Critical weaknesses
  • In Table 4, combining LsepL_{\mathrm{sep}} and LregL_{\mathrm{reg}} reduces ECE from 8.67 to 4.29, while gradient projection further reduces it only to 4.17. Similarly, uncertainty routing and adaptive margin in Tables 5–6 provide relatively small additional ECE improvements. The ablations do not provide statistical significance tests for these gains, making the necessity of these additional components unclear.
  • The paper does not quantitatively establish whether RA-TPT specifically reduces confidence on incorrect predictions rather than reducing confidence more generally. While the confidence distributions of incorrect predictions provide qualitative evidence, a corresponding analysis of correct predictions or quantitative measures of confidence-based error detection (e.g., correctness AUROC or risk–coverage analysis) would more directly support this claim.
  • RA-TPT uses predictive entropy to estimate uncertainty, but a highly confident incorrect prediction can have low entropy and consequently receive weak centered-logit dispersion regularization. Since excessive predictive concentration on incorrect predictions is a key motivation of the paper, an analysis of such cases would better support the uncertainty-routed regularization.
  • RA-TPT achieves the lowest average ECE, but its ECE improvement over SoC is not statistically significant. Since improving the calibration–accuracy trade-off is the main objective, this weakens the evidence for a clear calibration improvement over the strongest baseline.
Questions for authors
  • How does the uncertainty routing handle confidently incorrect predictions, which may have low predictive entropy?
  • Are the relatively small improvements from uncertainty routing, adaptive margin, and gradient projection statistically significant?
  • Does RA-TPT selectively reduce confidence on incorrect predictions, or does it reduce confidence more generally?
Reviewer_J8icOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Reliability-Aware Test-Time Prompt Tuning (RA-TPT), a logit-space approach for improving the calibration–accuracy trade-off during test-time adaptation. The method replaces entropy minimization with an uncertainty-aware bounded-margin objective and a pseudo-label anchor. It also introduces uncertainty-routed centered-logit dispersion regularization, a complementary mean-logit penalty, and soft conflict-aware gradient projection to reduce interference between adaptation objectives. The method is evaluated across 31 benchmarks spanning standard image classification, ImageNet distribution shifts, biomedical imaging, and remote sensing. The authors also extend the approach to conventional vision classifiers under corruption shifts. The reported results show substantial reductions in Expected Calibration Error (ECE) relative to entropy-based TPT while keeping competitive classification accuracy. Nevertheless, the paper addresses an important problem in test-time adaptation and presents a coherent, interpretable approach to controlling excessive predictive confidence.

Critical strengths
  1. The paper identifies an important limitation of entropy-minimization TPT: it can increase predictive confidence on incorrect or ambiguous examples under distribution shift. This highlights why accuracy alone is insufficient for evaluating test-time adaptation.
  2. RA-TPT combines bounded-margin separation, uncertainty-routed logit regularization, and gradient conflict attenuation. These components address different aspects of adaptation and can be evaluated individually.
  3. The distinction between centered-logit dispersion and the mean-logit component is well motivated. In particular, centering separates relative logit variation from common additive shifts that do not affect softmax probabilities.
  4. The evaluation covers various vision-language backbones, 31 benchmarks, specialized biomedical and remote-sensing domains, and conventional vision classifiers under corruption shifts. This provides enough to justify the strength.
  5. For CLIP ViT-B/16, the reported average ECE decreases from 11.50 to 3.99 across ten standard datasets and from 11.70 to 4.56 across four ImageNet variants compared with standard TPT. These improvements are achieved with relatively small reductions in average accuracy.
  6. The paper also reports computational overhead and memory consumption, helping assess practical applicability.
Critical weaknesses

None

Questions for authors
  1. The paper reports overall calibration improvements but does not specifically analyze whether RA-TPT improves calibration on initially incorrect predictions. This analysis would help validate the paper’s claim that the method addresses excessive confidence in incorrect predictions. Could the authors provide a justification/analysis of RA-TPT's calibration performance on initially incorrect predictions to clarify whether it effectively addresses overconfidence in these cases?
  2. How do the view-coherence factor and gradient-projection sharpness affect RA-TPT's performance, and why were these hyperparameter settings selected? Please justify.
  3. Why were multiple random seeds not used to assess the variability of the reported results, and how consistent are the improvements across runs?
  4. How does RA-TPT perform on initially incorrect predictions, and can the method inadvertently reinforce incorrect pseudo-labels?
  5. Could the authors justify the choice of 20 bins for ECE calculation and include the binning details in the main experimental setup to improve clarity and reproducibility?
  6. Could the authors consistently use “reduces ECE” rather than “improves ECE” throughout the manuscript for greater clarity and precision?
Reviewer_xBonOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RA-TPT to reduce overconfidence during test-time prompt tuning. It combines an uncertainty-aware margin objective, logit regularization, and soft gradient projection. Experiments across general, biomedical, and remote-sensing datasets show improved calibration over entropy-based TPT while maintaining competitive accuracy.

Critical strengths

1.The paper addresses a practical problem: test-time adaptation can improve accuracy while making confidence less reliable.

2.The logit-space formulation is flexible and applies to both VLM prompt tuning and standard vision classifiers.

3.Experiments cover multiple backbones and diverse domains, showing substantial calibration gains over entropy-based TPT at competitive accuracy.

Critical weaknesses

1.The methodological contribution seems incremental. Logit magnitude control has been studied in [1], and the projection is a smoothed, asymmetric variant of PCGrad [2]. Applying these ideas to unlabeled TPT is useful, but the contribution beyond this combination is not fully clear. In particular, soft projection provides only a small gain over hard projection.

2.The interaction between the objectives needs stronger support. The regularizer limits overconfidence, while the projection weakens the part of its gradient that opposes self-training. When the pseudo-label is confidently wrong, this opposing gradient may provide useful corrective pressure. Showing less gradient conflict does not establish that the removed conflict was harmful.

[1] Mitigating Neural Network Overconfidence with Logit Normalization. ICML, 2022.

[2] Gradient Surgery for Multi-Task Learning. NeurIPS, 2020.

Questions for authors

1.You acknowledge that confident errors can be reinforced. For samples already confidently misclassified before adaptation, how often does RA-TPT correct the prediction or reduce its confidence? Could you compare these outcomes with and without projection under the default one-step setting?

2.Tables 5 and 7 compare routing and projection variants. What do the proposed choices add beyond well-tuned constant or linear weighting and hard projection? Are the benefits consistent across individual datasets?

3.The hyperparameters are selected on a 5-shot ImageNet subset and then fixed. How sensitive are individual target domains to these choices, especially biomedical and remote-sensing datasets? Can the same defaults be used reliably without labeled target-domain validation?

4.Although results are averaged over three runs, the ablation tables do not show variability. Could you report seed-level differences or confidence intervals for accuracy and ECE, particularly for soft versus hard projection (4.17 versus 4.24 ECE)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.