- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
To address overconfident predictions in TPT, the authors use temperature scaling to calibrate confidence and reduce calibration error while preserving the predicted classes. They further propose a weighted ensemble of predictions from strongly and weakly augmented views to improve classification performance.
Critical strengths
The paper is well organized and easy to read, with a clear line of reasoning. The proposed method for calibrating TPT confidence is simple and effective, with potential for further development. The experimental evaluation is reasonably comprehensive, and the authors provide source code to support reproducibility.
Critical weaknesses
(1) The relationship between the objective in Eq.5 and calibration error is not sufficiently justified. Minimizing this objective does not, in general, guarantee a reduction in ECE. It would be helpful to clarify the assumptions under which confidence matching is expected to improve calibration. A natural alternative worth comparing is the following distribution-matching objective:
\tau^*=\arg \min _{\tau>0}\frac{1}{|\Omega|}\sum _{i\in\Omega}D _{\text{KL}}\left(\mathbf{p} _i^{\text{zs}}\\;\middle\\|\\;\hat{\mathbf{p}} _i^{\text{tpt}}(\tau)\right).
Moreover, this objective is convex when expressed in terms of the inverse temperature .
(2) The reported Brier scores are inconsistent with the stated formula. Eq.9 in the appendix defines the Brier score. Let denote the ground-truth class of sample . If sample is misclassified, there must be another class such that . Therefore,
Consequently, the Brier score should satisfy
However, Table 1 reports an accuracy of for CoTS on Aircraft, while Table 17 reports a Brier score of , which is clearly below the lower bound derived above:
I also inspected the provided source code with the help of an AI agent and could not find an implementation of the Brier score calculation.
(3) ECE measures the magnitude of miscalibration but does not indicate whether confidence is too high or too low. The authors should explicitly distinguish between overconfidence and underconfidence in the experiments, given that addressing overconfidence is one of the paper’s central contributions.
Questions for authors
- What theoretical guarantees does Eq.5 provide, and why was this particular optimization objective chosen?
- Please clarify exactly how the Brier score was calculated in the experiments.
- It would also be helpful to see the distribution of the optimized temperature . Comparing it with the initial temperature could shed light on whether the method actually corrects TPT's overconfidence.
4 comments
W3: ECE measures the magnitude of miscalibration but does not indicate whether confidence is too high or too low. The authors should explicitly distinguish between overconfidence and underconfidence in the experiments, given that addressing overconfidence is one of the paper’s central contributions.
A: We appreciate this helpful suggestion. In the updated manuscript, we indicate the direction of miscalibration using the signed confidence–accuracy gap. In Appendix D.7, Tables 30 and 31, light red and light green cells denote overconfidence and underconfidence, respectively. This analysis complements ECE by showing whether the calibration error arises from excessive or insufficient confidence.
Q3: It would also be helpful to see the distribution of the optimized temperature. Comparing it with the initial temperature could shed light on whether the method actually corrects TPT's overconfidence.
A: Thank you for the suggestion. We additionally visualize the distribution of the optimized temperatures relative to the initial value (). Most optimized temperatures are greater than 1, indicating that CoTS generally softens the TPT logits and reduces their excessive confidence. This provides further evidence that CoTS primarily corrects TPT’s overconfidence. We have added this analysis to Appendix D.6 of the updated manuscript.
| Metric | ImgNet | DTD | Flower102 | Food101 | SUN397 | Aircraft | Pets | Caltech101 | UCF101 | EuroSAT | Cars | -A | -V | -R | -Sk |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Avg. Temp. | 1.27 | 1.43 | 1.39 | 1.18 | 1.29 | 1.52 | 1.38 | 1.29 | 1.42 | 1.65 | 1.32 | 1.29 | 1.27 | 1.27 | 1.27 |
| (%) | 88.97 | 88.00 | 98.05 | 81.56 | 89.95 | 95.29 | 96.59 | 91.48 | 95.72 | 98.62 | 96.18 | 90.05 | 88.92 | 89.49 | 87.39 |
Thank you for the clarifications and the effort you put into the additional experiments. These have largely addressed my main concerns, and I continue to support acceptance. After reading the other reviewers’ comments and the subsequent discussion, I share some remaining reservations about the cross-view calibration assumption and the justification for using the average pairwise cosine similarity of class text embeddings as the fusion weight. If a score of 3.5 were available, I would choose it. For now, I will keep my rating at 3.
A: We sincerely thank you for considering raising the score to 3.5 and for your continued support toward acceptance. We address the two remaining concerns below.
- Cross-view calibration assumption.
Our motivation is that these strong views are directly used to update the prompt in TPT, so their confidence changes directly reflect the distortion introduced by TPT. CoTS therefore estimates a shared temperature from these views and applies it to the weak-view prediction produced by the same updated prompt. Using multiple views also provides a more stable estimate and Figure 3(a) further shows that this strategy outperforms weak-view estimation, supporting the design through both the TPT mechanism and empirical results.
- Similarity-based fusion weight.
The similarity-based weight provides a label-free alternative to fixed weights, rather than an optimal weight for every dataset. Figure 4 shows that no fixed weight is consistently optimal, while our approach remains competitive without dataset-specific tuning. Intuitively, higher text similarity makes strong-view selection less discriminative and the weak view relatively more reliable, whereas lower similarity makes strong views more complementary. As shown in Figure 5, most datasets have a weak-to-strong accuracy ratio close to one and are relatively insensitive to the exact weight. For datasets with larger weak-strong differences, text similarity follows the expected direction. Thus, text similarity serves as a practical adaptive proxy rather than an exact optimal mapping.
Thank you again for your constructive feedback.
Thank you for your thoughtful comments and valuable suggestions. We address each point below and present the corresponding revisions and additional results. If any concerns remain unresolved, we appreciate the opportunity for further discussion and clarification.
A: Thank you for the question. Eq. (5) is motivated by our CoR analysis: correcting confidence alone substantially reduces ECE while preserving TPT predictions, indicating that confidence shift is a major component of TPT miscalibration. Therefore, the loss function directly matches the maximum confidence used in top-label ECE and learns a temperature without changing the predicted class. KL loss matches the entire distribution, including non-top classes, and is therefore less directly aligned with ECE, which is computed using top-label confidence and accuracy. Nevertheless, following your suggestion, we additionally compare Eq. (5) with the KL objective under the same seed (seed=0) and report the results below. The results indicate that directly matching maximum confidence is more effective.
A: We apologize for the confusion. We have updated the public code to include the implementations of all evaluation metrics reported in the paper. Tables 17 & 18 report the top-label binary Brier score, calculated as BStop=S1∑i=1S(pi−oi)2, where (pi) denotes the top-label confidence of sample (i), and (oi∈{0,1}) indicates whether its top-1 prediction is correct. And Eq. (9) mistakenly gives the multiclass Brier-score formula. We will correct the formula and metric name in the revision. We also recomputed the multiclass Brier score using the full (K)-class probability vector and report the results below. These results demonstrate the effectiveness of our methods under both top-label binary and multiclass Brier scores. We have included the results for both top-label binary and multiclass Brier scores in Appendix C.6 of the revised manuscript.