- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TS3, a calibration methods for LLM on multi-class classification problems. TS3 is a three stage method. The first stage is sets a reference global temperature for all samples , following Temperature scaling from Guo et al. (2017). The second introduces a per-sample temperature-shift , fitted on a decomposition of the calibration dataset computed using sample features similarity. Then the last stage focuses on "low confidence" samples (corresponding to samples where the learned temperature shift is close to zero) and assign them a global temperature. The authors then compare their methods to several baselines, using ECE as the main metric.
Critical strengths
Topic is important.
Overall the paper is well written and easy to read.
Critical weaknesses
I have several remarks for the authors.
- Stage 3 seems quite counter intuitive. Why use a different partitioning from stage 2 for fitting ?
- Regarding the experiments and baselines, several aspects are unclear.
- In table 1, it seems that your method is the best overall, and TS is the second. I find this very surprising that other per-sample methods perform that poorly, even behind TS baseline. After checking, I have some concerns regarding the choice of the baselines. Thermometer (Thermo) Shen et al. (2024) is unsupervised and Density-Aware Calibration (DAC) Tomani et al. (2023) targets calibration under distribution shift, which is not exactly the same setup. A quick search returns paper such as Calibrating Language Models with Adaptive Temperature Scaling Xie et al 2024 or Calibrating LLM Confidence by Probing Perturbed Representation Stability Khanmohammadi et al (2025) which might provide relevant baselines against which this work should be positioned / compared.
- Hyperparameters choice and calibration process for all methods are not discussed sufficiently enough. Since baselines results are not so competitive I think further details on how the experiments were conducted is expected.
- Regarding my first point about partitioning for stage 3, I think further ablation on how stage 3 is constructed could help better understand the method, especially since based on Table 2 it often accounts for most of the ECE reduction.
- I did not find any concrete implementation details about the mapping which is not explained.
My main concerns are therefore:
- unclear positioning / relevant baselines are missing.
- lack of necessary experiment details.
Questions for authors
See weaknesses.
2 comments
W4. Regarding my first point about partitioning for stage 3, I think further ablation on how stage 3 is constructed could help better understand the method, especially since based on Table 2 it often accounts for most of the ECE reduction.
Response. Table 2 already ablates Stage 3 in two ways: removing it (the Stage 2 row) and removing the uncertainty regularizer that defines its noisy subset (w/o reg.) are both worse than TS3 in all 12 settings. We added two further variants. (1) The first variant fits one temperature per shift group (positive, negative, near-zero); it is worse than TS3 in all four settings and worse than global TS in three of them (table in our response to Q1 of Reviewer wS2G). (2) When , which holds for all our tasks, TS3 fits on the near-zero and positively shifted samples together. The second variant fits on the near-zero samples only. On BIG-Bench with Qwen-7B-I, it reaches 0.0312, against 0.0445 for Stage 2 and 0.0304 for TS3 (table below). Most of the Stage 3 gain therefore comes from re-calibrating the noisy region, and including the positively shifted samples gives a further small improvement. The reason for the shift-based subset is given in W1.
| BIG-Bench, Qwen-7B-I, ECE (10 bins) | TS | Stage 2 | Stage 3 (near-zero only) | TS3 |
|---|---|---|---|---|
| Average over 21 tasks | 0.0443 | 0.0445 | 0.0312 | 0.0304 |
W5. I did not find any concrete implementation details about the mapping which is not explained.
Response. In A.3 we specify that the mapping is a 2-layer MLP (hidden dimension 256 and 128, ReLU) that takes the last-token hidden state as input and outputs a scalar temperature shift. We will state this explicitly in Section 4.2. All results are reproducible from the uploaded code.
Dear Reviewer,
Thank you for the constructive feedback.
Response. Stage 2 partitions by purity. Stage 3 partitions by the predicted shift. The two partitions describe the same region in two forms. Purity defines the reliable/noisy region during training, but it requires labels and cannot be computed at inference time. The trained ψθ outputs a shift for every samples without requiring labels, so it is a label-free estimate of the regions at inference time (compare Figure 5b and 5d). We fit τˉ on the shift-based subset during training, rather than purity-based subset, because the same shift-based rule will be applied at inference (Eq. 6), so τˉ is learned on the same kind of samples that receive it.
When M≥C, the subset also includes positively shifted samples because of the dead bin issue (Lemma A.2); we therefore estimate them jointly through τˉ (Proposition 5.5). Moreover, purity only separates reliable from noisy regions; it does not indicate the direction of the shift, so these positively shifted samples can be identified only through the shift-based rule. This is a refinement on top of the noisy-region re-calibration; its effect is quantified in our response in W4. We will include this clarification in the revision.
Response. Our baselines cover the most recent methods for LLM calibration (Thermometer, DACA) and strong methods from image classification (DAC, TvA). Since the original Thermometer targets unseen tasks without labels, for a fair comparison we adapted it to our in-domain setting: an MLP predicts a sample-dependent temperature within each task. This makes it closely related to ATS ([4] Xie et al., 2024); the two coincide when the ATS smoothing weight is α=0. We therefore discussed ATS in the paper but did not include it as a separate baseline. As requested, we have now added ATS and CCPS ([5] Khanmohammadi et al., 2025). ATS uses the same implementation as Thermometer with α=0.5 (the optimal setting from the ATS paper). TS3 outperforms both in all 8 settings. We will include these results in the revision.
Response. Every baseline was tuned with the same protocol as TS3 and is reported at its best selected configuration, so the gap is not caused by under-tuned baselines. For each method, we grid-search the hyperparameters (for TS3, including β and k) and select the configuration with the lowest NLL on the calibration set. Methods with an auxiliary network share the same MLP (2 hidden layers of 256 and 128 units, ReLU), and all trainable methods use Adam with a learning rate of 1e-4 and a weight decay of 1e-6. All results are reproducible from the uploaded code, and we will add these details to the appendix.
References:
[4] Xie, Johnathan, et al. "Calibrating language models with adaptive temperature scaling." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
[5] Khanmohammadi, Reza, et al. "Calibrating LLM confidence by probing perturbed representation stability." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.