- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Reliability-Aware Test-Time Prompt Tuning (RA-TPT) to improve the calibration–accuracy trade-off of vision-language models under distribution shift. RA-TPT addresses overconfident predictions caused by test-time entropy minimization by adaptively controlling logit separation according to sample uncertainty. It combines an uncertainty-aware bounded margin, centered-logit dispersion regularization, and a soft conflict-aware gradient projection. Experiments on 31 benchmarks show improved calibration with competitive accuracy across CLIP-based, domain-specific, and non-VLM models.
Critical strengths
-
The proposed method directly operates in the logit space, which the authors identify as the source of the issue, and adaptively adjusts the regularization according to the uncertainty of each sample, rather than applying the same regularization uniformly to all samples.
-
The method achieves the lowest average ECE among the compared approaches. Although the difference in ECE compared with SoC is not statistically significant, the improvement in accuracy is statistically significant. Its VRAM usage is comparable to that of TPT, while incurring only about a 10% increase in latency. Compared with O-TPT, it reduces memory usage by 55% on ImageNet-Sketch with RN50.
-
The experimental evaluation is extensive, covering 31 benchmarks, multiple backbones and domains (CLIP RN50/ViT-B/16, BioMedCLIP, and RemoteCLIP), as well as non-VLM models.
Critical weaknesses
- In Table 4, combining and reduces ECE from 8.67 to 4.29, while gradient projection further reduces it only to 4.17. Similarly, uncertainty routing and adaptive margin in Tables 5–6 provide relatively small additional ECE improvements. The ablations do not provide statistical significance tests for these gains, making the necessity of these additional components unclear.
- The paper does not quantitatively establish whether RA-TPT specifically reduces confidence on incorrect predictions rather than reducing confidence more generally. While the confidence distributions of incorrect predictions provide qualitative evidence, a corresponding analysis of correct predictions or quantitative measures of confidence-based error detection (e.g., correctness AUROC or risk–coverage analysis) would more directly support this claim.
- RA-TPT uses predictive entropy to estimate uncertainty, but a highly confident incorrect prediction can have low entropy and consequently receive weak centered-logit dispersion regularization. Since excessive predictive concentration on incorrect predictions is a key motivation of the paper, an analysis of such cases would better support the uncertainty-routed regularization.
- RA-TPT achieves the lowest average ECE, but its ECE improvement over SoC is not statistically significant. Since improving the calibration–accuracy trade-off is the main objective, this weakens the evidence for a clear calibration improvement over the strongest baseline.
Questions for authors
- How does the uncertainty routing handle confidently incorrect predictions, which may have low predictive entropy?
- Are the relatively small improvements from uncertainty routing, adaptive margin, and gradient projection statistically significant?
- Does RA-TPT selectively reduce confidence on incorrect predictions, or does it reduce confidence more generally?