acceptodds
Under review as a conference paper at ICLR 2027

Tempering Confidence: Temperature Scaling for Calibrating Test-Time Prompt Tuning

Abstract

Test-time prompt tuning (TPT) enables adaptation on a single test instance, achieving improved accuracy but often sacrificing calibration performance. Most existing calibration methods introduce additional regularization terms to promote dispersion across text embeddings and reduce calibration error, yet these methods often suffer from a drop in accuracy. Motivated by the well-calibrated nature of zero-shot predictions, we propose CoTS, a simple yet effective post-hoc calibration method that preserves accuracy. Specifically, CoTS applies temperature scaling to minimize the confidence gap between adapted and zero-shot predictions. To fully exploit the potential of multiple augmentations during adaptation, we introduce a weak-strong ensemble strategy that further boosts accuracy. We then apply CoTS to this ensemble, termed E-CoTS, to maintain its well-calibrated property. Extensive experiments on diverse datasets and backbones show that our approaches effectively mitigate miscalibration without compromising primary accuracy. For instance, E-CoTS reduces the average expected calibration error of TPT from 11.90% to 5.38% on ImageNet variants, while even increasing accuracy from 60.74% to 62.95%. Moreover, when integrated with existing calibration methods, E-CoTS usually enhances both accuracy and calibration simultaneously. Code is available at [https://anonymous.4open.science/r/CoTS/](https://anonymous.4open.science/r/CoTS/).

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-time adaptation, calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tUAgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To address overconfident predictions in TPT, the authors use temperature scaling to calibrate confidence and reduce calibration error while preserving the predicted classes. They further propose a weighted ensemble of predictions from strongly and weakly augmented views to improve classification performance.

Critical strengths

The paper is well organized and easy to read, with a clear line of reasoning. The proposed method for calibrating TPT confidence is simple and effective, with potential for further development. The experimental evaluation is reasonably comprehensive, and the authors provide source code to support reproducibility.

Critical weaknesses

(1) The relationship between the objective in Eq.5 and calibration error is not sufficiently justified. Minimizing this objective does not, in general, guarantee a reduction in ECE. It would be helpful to clarify the assumptions under which confidence matching is expected to improve calibration. A natural alternative worth comparing is the following distribution-matching objective:

\tau^*=\arg \min _{\tau>0}\frac{1}{|\Omega|}\sum _{i\in\Omega}D _{\text{KL}}\left(\mathbf{p} _i^{\text{zs}}\\;\middle\\|\\;\hat{\mathbf{p}} _i^{\text{tpt}}(\tau)\right).

Moreover, this objective is convex when expressed in terms of the inverse temperature β=1τ\beta=\frac{1}{\tau}.

(2) The reported Brier scores are inconsistent with the stated formula. Eq.9 in the appendix defines the Brier score. Let ci∗c _i^* denote the ground-truth class of sample ii. If sample ii is misclassified, there must be another class c′≠ci∗c' \neq c _i^* such that pi,c′≥pi,ci∗p _{i,c'} \geq p _{i,c _i^*}. Therefore,

∑c=1K(pi,c−yi,c)2≥(pi,c′−0)2+(pi,ci∗−1)2≥pi,ci∗2+(pi,ci∗−1)2≥12.\sum _{c=1}^K(p _{i,c}-y _{i,c})^2 \geq (p _{i,c'}-0)^2+(p _{i,c _i^*}-1)^2 \geq p _{i,c _i^*}^2+(p _{i,c _i^*}-1)^2 \geq \frac{1}{2}.

Consequently, the Brier score should satisfy

BS≥12(1−ACC).\text{BS}\geq\frac{1}{2}\left(1-\text{ACC}\right).

However, Table 1 reports an accuracy of 23.2723.27\\% for CoTS on Aircraft, while Table 17 reports a Brier score of 14.1414.14\\%, which is clearly below the lower bound derived above:

12(1−23.27\frac{1}{2}(1-23.27\\%)\approx38.37\\%.

I also inspected the provided source code with the help of an AI agent and could not find an implementation of the Brier score calculation.

(3) ECE measures the magnitude of miscalibration but does not indicate whether confidence is too high or too low. The authors should explicitly distinguish between overconfidence and underconfidence in the experiments, given that addressing overconfidence is one of the paper’s central contributions.

Questions for authors
  1. What theoretical guarantees does Eq.5 provide, and why was this particular optimization objective chosen?
  2. Please clarify exactly how the Brier score was calculated in the experiments.
  3. It would also be helpful to see the distribution of the optimized temperature τ∗\tau^*. Comparing it with the initial temperature could shed light on whether the method actually corrects TPT's overconfidence.
4 comments

Thank you for your thoughtful comments and valuable suggestions. We address each point below and present the corresponding revisions and additional results. If any concerns remain unresolved, we appreciate the opportunity for further discussion and clarification.

W1&Q1: The relationship between the objective in Eq.5 and calibration error is not sufficiently justified. Minimizing this objective does not, in general, guarantee a reduction in ECE. It would be helpful to clarify the assumptions under which confidence matching is expected to improve calibration. Compare with the KL loss. What theoretical guarantees does Eq.5 provide, and why was this particular optimization objective chosen?

A: Thank you for the question. Eq. (5) is motivated by our CoR analysis: correcting confidence alone substantially reduces ECE while preserving TPT predictions, indicating that confidence shift is a major component of TPT miscalibration. Therefore, the loss function directly matches the maximum confidence used in top-label ECE and learns a temperature without changing the predicted class. KL loss matches the entire distribution, including non-top classes, and is therefore less directly aligned with ECE, which is computed using top-label confidence and accuracy. Nevertheless, following your suggestion, we additionally compare Eq. (5) with the KL objective under the same seed (seed=0)(\text{seed}=0) and report the results below. The results indicate that directly matching maximum confidence is more effective.

MethodObjectiveFine-grained ECEImageNet Variants ECE
CoTSOur Loss4.155.17
KL Loss4.175.81
E-CoTSOur Loss4.185.47
KL Loss4.656.53

W2&Q2: The reported Brier scores are inconsistent with the stated formula. Eq.9 in the appendix defines the Brier score. Please clarify exactly how the Brier score was calculated in the experiments.

A: We apologize for the confusion. We have updated the public code to include the implementations of all evaluation metrics reported in the paper. Tables 17 & 18 report the top-label binary Brier score, calculated as BStop=1S∑i=1S(pi−oi)2,BS_{top}=\frac{1}{S}\sum_{i=1}^{S}(p_i-o_i)^2, where (pi)(p_i) denotes the top-label confidence of sample (i)(i), and (oi∈{0,1})(o_i\in\{0,1\}) indicates whether its top-1 prediction is correct. And Eq. (9) mistakenly gives the multiclass Brier-score formula. We will correct the formula and metric name in the revision. We also recomputed the multiclass Brier score using the full (KK)-class probability vector and report the results below. These results demonstrate the effectiveness of our methods under both top-label binary and multiclass Brier scores. We have included the results for both top-label binary and multiclass Brier scores in Appendix C.6 of the revised manuscript.

MethodImgNetDTDFlowersFood101SUN397AircraftPetsCaltechUCF101EuroSATCarsAverage
Zero-Shot45.6670.2843.6224.2850.8885.9417.8211.2645.1269.9444.9446.34
TPT46.5075.7446.2423.9451.4291.8620.3410.8045.1476.3644.3848.42
C-TPT44.9672.4241.6225.1050.8885.9817.1811.0246.3875.2645.0246.90
Penalty46.5475.3445.7424.4651.3491.6425.1612.3446.5071.1444.6248.62
SaLS46.2074.2245.6423.7651.1490.8420.1610.9044.8673.6444.1047.76
O-TPT44.9070.1441.3625.6451.7886.1617.3211.0047.5875.7045.5847.02
A-TPT44.8270.8041.1425.2451.7286.6217.7211.9247.8869.9445.3846.66
SoC45.4071.8642.8424.5451.5485.8417.1011.3044.7869.9645.2046.40
CoTS44.7070.0843.3423.4449.3886.6818.7211.6043.1671.8843.9246.08
E-CoTS43.7869.8843.8823.3848.8486.4019.0010.4242.8872.5242.8645.80
Method-A-V-R-SkAverage
Zero-Shot68.5852.7635.7467.5656.16
TPT66.0453.9633.7071.1056.20
C-TPT66.6052.6634.2468.6455.54
Penalty67.3253.8234.1871.1456.62
SaLS65.3653.5233.3470.6255.72
O-TPT67.9452.2835.7267.2455.80
A-TPT67.1052.1834.7867.4055.36
SoC67.0652.6034.6867.5655.48
CoTS63.2451.6233.4267.0053.82
E-CoTS57.9450.3831.9666.1851.62

W3: ECE measures the magnitude of miscalibration but does not indicate whether confidence is too high or too low. The authors should explicitly distinguish between overconfidence and underconfidence in the experiments, given that addressing overconfidence is one of the paper’s central contributions.

A: We appreciate this helpful suggestion. In the updated manuscript, we indicate the direction of miscalibration using the signed confidence–accuracy gap. In Appendix D.7, Tables 30 and 31, light red and light green cells denote overconfidence and underconfidence, respectively. This analysis complements ECE by showing whether the calibration error arises from excessive or insufficient confidence.

Q3: It would also be helpful to see the distribution of the optimized temperature. Comparing it with the initial temperature could shed light on whether the method actually corrects TPT's overconfidence.

A: Thank you for the suggestion. We additionally visualize the distribution of the optimized temperatures relative to the initial value (T=1T=1). Most optimized temperatures are greater than 1, indicating that CoTS generally softens the TPT logits and reduces their excessive confidence. This provides further evidence that CoTS primarily corrects TPT’s overconfidence. We have added this analysis to Appendix D.6 of the updated manuscript.

MetricImgNetDTDFlower102Food101SUN397AircraftPetsCaltech101UCF101EuroSATCars-A-V-R-Sk
Avg. Temp.1.271.431.391.181.291.521.381.291.421.651.321.291.271.271.27
T>1T>1 (%)88.9788.0098.0581.5689.9595.2996.5991.4895.7298.6296.1890.0588.9289.4987.39

Thank you for the clarifications and the effort you put into the additional experiments. These have largely addressed my main concerns, and I continue to support acceptance. After reading the other reviewers’ comments and the subsequent discussion, I share some remaining reservations about the cross-view calibration assumption and the justification for using the average pairwise cosine similarity of class text embeddings as the fusion weight. If a score of 3.5 were available, I would choose it. For now, I will keep my rating at 3.

A: We sincerely thank you for considering raising the score to 3.5 and for your continued support toward acceptance. We address the two remaining concerns below.

  1. Cross-view calibration assumption.

Our motivation is that these strong views are directly used to update the prompt in TPT, so their confidence changes directly reflect the distortion introduced by TPT. CoTS therefore estimates a shared temperature from these views and applies it to the weak-view prediction produced by the same updated prompt. Using multiple views also provides a more stable estimate and Figure 3(a) further shows that this strategy outperforms weak-view estimation, supporting the design through both the TPT mechanism and empirical results.

  1. Similarity-based fusion weight.

The similarity-based weight provides a label-free alternative to fixed weights, rather than an optimal weight for every dataset. Figure 4 shows that no fixed weight is consistently optimal, while our approach remains competitive without dataset-specific tuning. Intuitively, higher text similarity makes strong-view selection less discriminative and the weak view relatively more reliable, whereas lower similarity makes strong views more complementary. As shown in Figure 5, most datasets have a weak-to-strong accuracy ratio close to one and are relatively insensitive to the exact weight. For datasets with larger weak-strong differences, text similarity follows the expected direction. Thus, text similarity serves as a practical adaptive proxy rather than an exact optimal mapping.

Thank you again for your constructive feedback.

Reviewer_vQ8yOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoTS, a post-calibration method that estimates temperature parameters for each image by aligning TPT confidence scores with zero-shot confidence scores in selected enhanced views. Building on this, E-CoTS further combines the predictions from temperature-scaled weak and weak-strong ensemble strategy views and introduces text-similarity-based weights for aggregation. Experiments were conducted on fine-grained datasets and ImageNet variants, covering a variety of backbone network architectures and prompt initialization methods, to evaluate the models’ accuracy and calibration performance.

Critical strengths

CoTS provides label-free, post-hoc calibration without changing TPT’s predicted classes. In the reported ViT-B/16 experiments, it reduces average ECE from 11.30% to 4.28% on fine-grained datasets and from 11.90% to 5.30% on ImageNet variants while keeping TPT’s accuracy.

Critical weaknesses

W1. The calibration rationale and incremental contribution remain insufficiently substantiated. CoR supports zero-shot confidence as an empirical reference, but does not establish confidence inflation as the dominant cause of miscalibration or distributional inconsistency as the cause of CoR’s residual error. For identical weak-view TPT logits, exact positive-temperature matching to the same zero-shot maximum confidence would reproduce CoR’s predicted labels, confidence scores, and top-label ECE. Moreover, ZERO [1] shows that zero-shot CLIP can be miscalibrated on confidence-filtered augmented views. Good calibration on original images therefore does not, by itself, validate CoTS’s use of zero-shot confidence on selected strong views as the fitting target in Eq. (5), or explain why the resulting temperature should calibrate the final weak-view or ensemble prediction. The reliability of this reference under the actual view-selection procedure and its applicability to the final output remain insufficiently justified. Finally, the comparison with SaLS does not adequately isolate the benefit of the proposed confidence-matching criterion from that of multi-view estimation and selection.

W2. The argument regarding “adaptive fusion” and the incremental benefits it provides lacks sufficient support. In fixed-weight ablation experiments, similarity-based adaptive weights did not demonstrate any significant performance advantage; furthermore, Figure 6 fails to establish a mapping relationship between “text similarity” and “appropriate fusion weights.” Additional fusion may even lead to a decline in CoTS performance: in fine-grained experiments using CoOp initialization, the accuracy dropped from 65.67% to 65.47%, while the ECE rose from 5.61% to 6.10%; a similar pattern was observed when using the RN50 backbone network—accuracy decreased, and the ECE did not improve. Furthermore, on the fine-grained dataset, the zero-shot setting using the same fusion method achieved results of 65.21/4.00 (accuracy/ECE), while the full E-CoTS model achieved 65.22/4.31. These comparative results are not yet sufficient to fully demonstrate the added value provided by this specific weighting rule and the complete process.

Reference: [1] Frustratingly Easy Test-Time Adaptation of Vision-Language Models, NeurIPS 2024.

Questions for authors

Q1. When a positive temperature precisely matches the zero-shot maximum confidence on the same weak view, CoTS and CoR should have identical predicted classes, confidences, and top-label ECE. Please explain the source of the discrepancy in the weak view results in Figure 3(a), and how this discrepancy supports the attribution that "distributional discrepancy leads to the poorer calibration of CoR."

Q2. After standardizing the available views, view selection, and subsequent fusion scheme, which gains of CoTS over SaLS can be attributed to the maximum confidence matching criterion itself, and how does the existing evidence disentangle these gains from those brought by multi-view information?

Q3. Considering the competitive performance of fixed weights and that Figure 6 does not establish a correspondence between similarity and suitable fusion weights, what evidence supports directly using the average pairwise text similarity as the numerical fusion weight?

Q4. Figure 3(a) evaluates the selection of temperature-fitting views via the CoTS output; is this selection still appropriate on the final E-CoTS fused output? Please also report the end-to-end runtime of the full E-CoTS to evaluate its marginal gain versus computational cost relative to CoTS and zero-shot fusion.

7 comments

Thank you for your valuable comments. We address each point below and summarize the corresponding revisions and additional experiments. If any concerns remain unresolved, we appreciate the opportunity for further discussion and clarification.

W1: CoR supports zero-shot confidence as an empirical reference, but does not establish confidence inflation as the dominant cause of miscalibration or distributional inconsistency as the cause of CoR’s residual error.

A: Thank you for raising this concern. We agree that describing confidence inflation as the dominant cause based solely on CoR sounds too strong. CoR markedly reduces ECE by modifying only confidence while preserving TPT predictions, indicating that confidence shift is indeed a major empirical component of TPT miscalibration. This result provides strong empirical support for treating confidence as an important target for calibration improvement. We also agree that attributing CoR's residual error to a "fundamental inconsistency" was too strong. Our intended point is that combining TPT predictions with zero-shot confidence may introduce a semantic mismatch. We have revised the text to describe this mismatch as a potential limitation of confidence replacement, rather than a causal explanation for the residual calibration error. These revisions are provided in Lines 183-185 of the updated manuscript.

W1&Q1: For identical weak-view TPT logits, exact positive-temperature matching to the same zero-shot maximum confidence would reproduce CoR’s predicted labels, confidence scores, and top-label ECE. Please explain the source of the discrepancy in the weak view results in Figure 3(a), and how this discrepancy supports the attribution that "distributional discrepancy leads to the poorer calibration of CoR.

A: Great question! We agree that, with a sufficiently large number of optimization steps, the weak view result should converge to CoR. In Figure 3(a), however, "CoTS w/ Weak View" follows our unified protocol of 50 optimization steps and therefore does not exactly solve for the temperature. To verify this explanation, we increased the optimization budget from 50 to 100 steps, as shown below. This brings the ECE close to that of CoR and confirms that the original discrepancy arises from finite-step optimization. Accordingly, this discrepancy does not support attributing CoR’s residual ECE to distributional inconsistency. More generally, CoR combines TPT-predicted labels with zero-shot confidence, creating a potential semantic mismatch, and does not fully exploit augmented views used during adaptation. These potential limitations motivate the development of CoTS. We have carefully revised this discussion in Lines 183-185 of the updated manuscript.

MethodFine-grained Acc.Fine-grained ECEImageNet Variants Acc.ImageNet Variants ECE
CoTS w/ Weak View (step=100)65.255.2460.726.03
CoR65.255.2560.726.02

W1: Moreover, ZERO [1] shows that zero-shot CLIP can be miscalibrated on confidence-filtered augmented views. Good calibration on original images therefore does not, by itself, validate CoTS’s use of zero-shot confidence on selected strong views as the fitting target in Eq. (5), or explain why the resulting temperature should calibrate the final weak-view or ensemble prediction.

A: Thank you for pointing this out. We agree with ZERO [1] that confidence-filtered strong views can be substantially miscalibrated. We use these views because they directly participate in TPT adaptation and therefore reflect the confidence shift introduced by prompt adaptation. Importantly, CoTS does not require zero-shot confidence on these views to be perfectly calibrated. Instead, Eq. (5) compares zero-shot and TPT confidence on the same selected strong views and uses their relative confidence gap to estimate the adaptation-induced shift. CoTS treats this shift as a sample-specific effect shared across views involved in TPT adaptation and applies the resulting temperature to the final weak view or ensemble prediction. Figure 3(a) and our additional E-CoTS results under different view-selection strategies in our response to Q4 empirically show that our strategy performs better on the final predictions than estimating the temperature from the weak view alone.

[1] Frustratingly Easy Test-Time Adaptation of Vision-Language Models, NeurIPS 2024.

W1&Q2: Finally, the comparison with SaLS does not adequately isolate the benefit of the proposed confidence-matching criterion from that of multi-view estimation and selection. After standardizing the available views, view selection, and subsequent fusion scheme, what gains of CoTS over SaLS can be attributed to the maximum confidence matching criterion itself, and how does the existing evidence disentangle these gains from those brought by multi-view information?

A: We humbly disagree. CoTS follows TPT and prior calibration methods (C-TPT, O-TPT) by reusing the augmented views generated for TPT adaptation without introducing extra information. More importantly, Figure 2 provides a controlled single-view comparison, where both CoR and SaLS operate on the same weak-view TPT prediction without multi-view estimation, selection, or fusion. As discussed in our response to Q1, weak-view CoTS converges to CoR with sufficient optimization steps. Therefore, CoR’s improvement over SaLS in this controlled setting can be attributed to the maximum-confidence matching criterion rather than multi-view information, demonstrating that our confidence-matching strategy is more effective than the logit-range adjustment used by SaLS.

MethodFine-grained ECEImageNet variants ECE
CoR5.256.02
SaLS9.8511.14

W2: In fixed-weight ablation experiments, similarity-based adaptive weights did not demonstrate any significant performance advantage.

A: Thank you for the comment. Similarity-based adaptive weighting is designed to avoid using an additional validation set to select a fixed weight. As shown below, no fixed weight consistently outperforms similarity-based weight across all settings. For example, (α=0.5\alpha=0.5) performs best on ImageNet variants but is worse on fine-grained datasets, whereas (α=1\alpha=1) improves fine-grained ECE but substantially reduces ImageNet variants accuracy. In contrast, similarity-based weight provides competitive performance without dataset-specific tuning.

alphaFine-grained Acc.Fine-grained ECEImageNet Variants Acc.ImageNet Variants ECE
0.164.557.4263.009.37
0.364.706.1763.046.52
0.564.924.9563.105.11
0.765.214.3162.895.46
0.965.254.1961.725.55
165.254.1560.725.17
sim65.284.1862.995.47

Although it does not always achieve the best result, it consistently provides competitive performance across different datasets and remains close to the best fixed weight. Therefore, its main advantage is not a consistent numerical improvement over every fixed weight, but a practical, label-free mechanism for adaptive weighting.

W2&Q3: Figure 6 fails to establish a mapping relationship between “text similarity” and “appropriate fusion weights. Considering the competitive performance of fixed weights and that Figure 6 does not establish a correspondence between similarity and suitable fusion weights, what evidence supports directly using the average pairwise text similarity as the numerical fusion weight?

A: We agree that Figure 6 may not establish an exact mapping between text similarity and the optimal fusion weight. Nevertheless, it suggests a moderate positive relationship between text similarity and the weak-strong accuracy ratio. We computed the Pearson correlation between text similarity and the weak-strong accuracy ratio and obtained a moderate positive correlation (r=0.526r=0.526, p=0.044p=0.044). In particular, for most datasets, the accuracy ratio is close to one, indicating comparable reliability between weak and strong views, making performance relatively insensitive to the fusion weight. For the two datasets with larger deviations, similarity adjusts the weight in the appropriate direction: higher similarity emphasizes the weak view, whereas lower similarity emphasizes the strong view. This result supports using similarity as a simple, label-free proxy for the relative reliability of weak and strong views.

W2: Additional fusion may even lead to a decline in CoTS performance: in fine-grained experiments using CoOp initialization, the accuracy dropped from 65.67% to 65.47%, while the ECE rose from 5.61% to 6.10%; a similar pattern was observed when using the RN50 backbone network—accuracy decreased, and the ECE did not improve. Furthermore, on the fine-grained dataset, the zero-shot setting using the same fusion method achieved results of 65.21/4.00 (accuracy/ECE), while the full E-CoTS model achieved 65.22/4.31.

A: Thank you for highlighting these cases. We acknowledge that E-CoTS may not outperform CoTS in every setting. Nevertheless, it provides improvements in most cases and remains competitive with existing calibration methods. We present it as an optional, orthogonal accuracy-enhancing extension rather than a uniformly superior replacement for CoTS.

Table 25 shows that our ensemble strategy substantially improves zero-shot accuracy and outperforms ZERO's voting strategy. Interestingly, the accompanying reduction in ECE is an unexpected by-product, possibly because the large accuracy gain improves the alignment between confidence and correctness. However, on underconfident datasets, increasing accuracy without adjusting confidence may enlarge the calibration gap, as shown below. Even in these cases, E-CoTS achieves a favorable accuracy-calibration trade-off by combining ensemble with explicit temperature scaling. Thus, the ensemble primarily targets accuracy, whereas CoTS explicitly calibrates predictions after test-time adaptation. We have clarified this observation in the revised manuscript in Appendix D.9.

Generally, we believe these results validate CoTS for effective calibration and our ensemble strategy for improving accuracy, while E-CoTS combines both to achieve a competitive accuracy–calibration trade-off.

MetricMethodFood101SUN397CarsImageNet-R
Acc.ZS83.6662.5865.5673.99
ZS+Ensemble (ours)84.4764.6667.9477.21
E-CoTS84.5465.6767.5777.88
ECEZS2.042.244.413.55
ZS+Ensemble (ours)2.502.676.184.70
E-CoTS1.292.593.842.94

Q4: Figure 3(a) evaluates the selection of temperature-fitting views via the CoTS output; is this selection still appropriate on the final E-CoTS fused output? Please also report the end-to-end runtime of the full E-CoTS to evaluate its marginal gain versus computational cost relative to CoTS and zero-shot fusion.

A: Figure 3(a) was designed to isolate the choice of views for temperature estimation in CoTS and therefore did not include the ensemble component. Following the reviewer’s suggestion, we additionally evaluate the same view-selection strategies on the final E-CoTS output, as reported below. These results support that our top-6 strong views selection is still appropriate for E-CoTS.

View SelectionFine-grained Acc.Fine-grained ECEImageNet Variants Acc.ImageNet Variants ECE
w/ All Views65.34.263.05.7
w/ Top-6 Views65.34.263.05.5
w/ Weak View65.34.862.85.7
w/ Strong Views65.34.263.05.7
w/ Top-6 Strong Views65.34.263.05.5

As for the end-to-end runtime of E-CoTS, our method is designed to calibrate predictions after test-time adaptation and should therefore be compared primarily with other TPT-based calibration methods. As shown below, CoTS achieves comparable or lower runtime than these methods and introduces only a small overhead over the original TPT, with most of the total runtime arising from TPT adaptation itself. Besides, zero-shot fusion applies our ensemble strategy directly to zero-shot predictions without test-time adaptation, making a direct runtime comparison with TPT-based methods less meaningful. We have updated the end-to-end runtime analysis to Lines 506--510 of the revised manuscript.

Time (s/sample)Zero-Shot FusionTPTC-TPTO-TPTA-TPTSoCCoTSE-CoTS
DTD0.79710.96911.11171.09241.08851.09991.0148 (+0.0457)1.0223
ImageNet-A0.25690.45220.50650.54020.54210.51040.4987 (+0.0465)0.5139
Reviewer_4X8gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Confidence-based Temperature Scaling (CoTS), a post-hoc calibration method for test-time prompt tuning (TPT) in CLIP-style vision-language models. Instead of using labeled validation data, CoTS learns a temperature that aligns the confidence of TPT predictions with the corresponding zero-shot confidence over selected strongly augmented views. The paper also proposes E-CoTS, which combines temperature-scaled predictions from a weakly augmented view and selected strong views using a weight derived from the average pairwise cosine similarity of text embeddings.

The method is evaluated on eleven fine-grained datasets and four ImageNet variants, primarily with ViT-B/16, and is also examined with CoOp initialization and other episodic test-time adaptation methods. On fine-grained datasets, CoTS reduces average ECE from 11.30% for TPT to 4.28% while preserving average accuracy at 65.19%. On ImageNet variants, E-CoTS increases average accuracy from 60.74% to 62.95% and reduces ECE from 11.90% to 5.38%. The paper further reports that E-CoTS can complement several existing calibration methods.

Overall, the paper addresses a practical calibration problem in episodic test-time adaptation with a simple, label-free post-hoc approach. Its emphasis on retaining accuracy while improving confidence calibration is valuable, although the method’s dependence on zero-shot confidence as a reference and the scope of its evaluation merit further discussion.

Critical strengths
  1. The paper identifies a clear and relevant problem: TPT can improve classification accuracy while substantially increasing overconfidence and ECE. The motivation is supported by comparisons with zero-shot CLIP and existing calibration approaches.
  2. CoTS is conceptually simple and operates after prompt adaptation, avoiding changes to the prompt-optimization objective. It does not require labeled validation data, which is well suited to the episodic test-time setting.
  3. The confidence-alignment objective is clearly specified, and the ablation study compares view selection and loss formulations. The reported results support the use of selected strong augmentations and an L2 confidence-alignment objective.
  4. The evaluation covers a decent number of fine-grained datasets and ImageNet variants, with additional experiments using CoOp initialization and other episodic TTA methods. This gives useful evidence beyond a single dataset or adaptation configuration.
  5. The reported results are promising. CoTS reduces average ECE on fine-grained datasets from 11.30% to 4.28% without changing TPT’s average accuracy, while E-CoTS improves ImageNet-variant average accuracy from 60.74% to 62.95% and lowers ECE from 11.90% to 5.38%. The paper also investigates integration with existing calibration methods.
  6. The weak-strong ensemble uses a task-adaptive weight based on text-embedding similarity rather than requiring manual selection of a fixed ensemble weight. This is a practical design choice, and the results suggest that it offers a decent accuracy-calibration trade-off.
Critical weaknesses

Minor Weaknesses

The manuscript is generally understandable and well organized. The following clarifications and presentation improvements would help:

  1. Define all method names (CoR, CoTS, and E-CoTS) consistently at first use and ensure the distinction between confidence replacement and temperature-based calibration remains clear.
  2. In the implementation details, clarify precisely which logits/views are used to select the top 10% of views for temperature optimization and whether the same selection rule is used for every baseline.
  3. Report the number of calibration bins and the exact ECE implementation in the main experimental setup, rather than leaving readers to infer these details from supplementary material.
  4. Use consistent terminology for “strongly augmented views,” “weakly augmented view,” and “zero-shot confidence” throughout the paper.
  5. A short discussion of limitations, especially dependence on the zero-shot confidence reference and extra test-time optimization would make the practical scope of the method clearer.
Questions for authors
  1. CoTS relies on zero-shot confidence as a label-free reference. Could the authors provide further justification for the reliability of this reference under substantial distribution shifts, particularly when zero-shot and adapted predictions differ significantly?
  2. The authors needs to provide an analysis or any related justifcation of the sensitivity of ECE to different bin counts and binning strategies?
  3. Computational Efficiency: CoTS requires 50 optimization steps to determine the temperature for each test instance, which may introduce additional computational overhead. Could the authors clarify the impact of this process on inference time and memory consumption?
4 comments

Thank you for your careful review and constructive comments. Below, we respond to each concern in turn and describe the corresponding revisions and additional experiments. If any concerns remain unresolved, we appreciate the opportunity for further discussion and clarification.

W1: Define all method names (CoR, CoTS, and E-CoTS) consistently at first use and ensure the distinction between confidence replacement and temperature-based calibration remains clear.

A: We agree. As stated in Lines 170-175, 199-200, and 247-249, we define CoR, CoTS, and E-CoTS at first use and clarify that CoR directly replaces the TPT confidence, whereas CoTS calibrates the TPT logits through temperature scaling.

W2: In the implementation details, clarify precisely which logits/views are used to select the top 10% of views for temperature optimization and whether the same selection rule is used for every baseline.

A: As stated in Lines 301–304, we specify that the top 10% views refer to the Top 6 confident views selected exclusively from strong views, excluding the weak view. The same augmentation and view-selection protocol is used for all applicable baselines.

W3: Report the number of calibration bins and the exact ECE implementation in the main experimental setup, rather than leaving readers to infer these details from supplementary material.

A: Thanks for the reminder. We have moved this detail from the appendix to the main text. As stated in Lines 296-298, we compute top-label ECE using 20 equal-width confidence bins.

W4: Use consistent terminology for "strongly augmented views," "weakly augmented view," and "zero-shot confidence" throughout the paper

A: Thanks for the reminder. In the revised manuscript, we consistently use the terms "strongly augmented views," "weakly augmented view," and "zero-shot confidence" throughout the manuscript.

W5: A short discussion of limitations, especially dependence on the zero-shot confidence reference and extra test-time optimization would make the practical scope of the method clearer.

A: Following your suggestion, we have added a discussion of these limitations to the revised manuscript. CoTS is heavily built on the observation that zero-shot confidence provides a suitable calibration target. Nevertheless, extensive experiments demonstrate its effectiveness across datasets, even when zero-shot predictions are less well calibrated. In addition, temperature optimization introduces a small computational overhead beyond standard TPT.

Q1: CoTS relies on zero-shot confidence as a label-free reference. Could the authors provide further justification for the reliability of this reference under substantial distribution shifts, particularly when zero-shot and adapted predictions differ significantly?

A: Good question! To examine the reliability of zero-shot confidence under substantial distribution shifts, we evaluate CoTS on datasets with relatively large accuracy gaps between zero-shot and TPT predictions. The results are shown below.

MetricMethodImageNet-AUCF101SUN397DTD
ACCZero-Shot47.8365.1962.5844.33
TPT54.6368.2065.5047.12
ECEZero-Shot8.342.952.248.24
TPT15.2011.5711.2320.96
+ CoTS7.853.113.516.67

These results indicate that zero-shot confidence remains a useful label-free calibration reference even when zero-shot and adapted predictions exhibit substantial performance differences.

Q2: The authors need to provide an analysis or any related justification of the sensitivity of ECE to different bin counts and binning strategies.

A: We appreciate this helpful suggestion and have conducted a sensitivity analysis using different bin counts under seed 0. Our main evaluation uses 20 equal-width bins, following prior work.

Dataset (ECE)BinsZero-ShotTPTC-TPTSoCCoTSE-CoTS
Fine-grained104.0411.205.074.083.904.08
154.1611.235.154.154.064.23
204.3311.255.234.304.164.18
304.4411.335.394.434.384.49
504.6511.385.704.704.814.83
ImageNet Variants104.8012.036.124.725.105.47
154.9112.026.704.775.155.48
204.9912.066.744.835.175.47
304.9712.076.724.905.265.53
505.0212.127.015.025.415.55

The results under different binning strategies support that CoTS remains competitive with other methods across different settings, while it is not sensitive to the specific ECE configuration. We have added these results to Appendix D.5 in the updated manuscript.

Q3: Computational Efficiency: CoTS requires 50 optimization steps to determine the temperature for each test instance, which may introduce additional computational overhead. Could the authors clarify the impact of this process on inference time and memory consumption?

A: Thank you for the question. We evaluated the end-to-end inference time per sample. The results have been updated in Table 9 (below). CoTS has a runtime comparable to other calibration methods, and the additional overhead introduced by temperature optimization is relatively small compared with the overall cost of TPT adaptation. Moreover, CoTS optimizes only a single scalar temperature and reuses the computed logits, resulting in negligible additional memory consumption.

Time (s/sample)TPTC-TPTO-TPTA-TPTSoCCoTSE-CoTS
DTD0.96911.11171.09241.08851.09991.0148 (+0.0457)1.0223
ImageNet-A0.45220.50650.54020.54210.51040.4987 (+0.0465)0.5139
AuthorsOpenReview ↗

Inviting further discussion

Dear Reviewer,

Thank you again for your valuable time and constructive feedback. We would like to kindly follow up on our responses and ask whether they have adequately addressed your concerns. To ensure sufficient time to address any remaining issues before the discussion period concludes, we would greatly appreciate your feedback at your earliest convenience. We would be happy to provide any further clarification and continue the discussion if needed.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer,

Thank you again for your valuable comments. We have revised the manuscript and provided additional analyses addressing your concerns. With less than one day remaining, we would sincerely appreciate your feedback on our responses.\color{#1455A3}{\mathbf{\text{With less than one day remaining, we would sincerely appreciate your feedback on our responses.}}} We remain happy to provide any further clarification before the discussion concludes.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.