- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies confidence calibration of deep networks and LoRA-fine-tuned LLMs, whose predictions become increasingly overconfident as training/fine-tuning proceeds. The authors report a consistent phenomenon across models: ECE is minimized at an early stage of training (or fine-tuning) and then rises, while accuracy improves only marginally (Fig. 1a). A feature-space analysis further suggests that the early model retains "uncertainty awareness" in both its predictions and its feature structure, which is gradually lost with continued training (Fig. 1b).
To prevent this, the paper proposes EUA-Cal, which freezes an early checkpoint as an uncertainty anchor and adds two regularizers to the remaining training: (i) Early Prediction Regularization (EPR), which penalizes the KL divergence between the current predictive distribution and the frozen early predictions; and (ii) Prototype Structure Regularization (PSR), which penalizes the KL divergence between the current prediction and a soft target distribution q, constructed from cosine similarities between (early) features and confidence-weighted early class prototypes. The overall objective is the cross-entropy loss plus the EPR and PSR terms, weighted by alpha and beta respectively.
The claimed contributions are (1) the "early model is better calibrated" observation and an analysis attributing it to uncertainty awareness in early predictions and features; (2) the EUA-Cal method, which uses the early model as a reusable uncertainty anchor without architectural change or extra inference cost; and (3) an empirical demonstration that EUA-Cal outperforms prior calibration methods.
Validation is empirical rather than theoretical: 8 models (ResNet-50/101, Wide-ResNet-26-10, DenseNet-121, ViT-B/32, and LoRA-fine-tuned Llama3-8B, Llama3.1-8B, Qwen3-8B-Base) over 6-7 datasets, covering image classification (CIFAR-10/100, Tiny-ImageNet) and multiple-choice QA (SciQ, ARC-C, ARC-E; CSQA for distribution shift). Experiments span in-distribution calibration, distribution shift, OOD detection, ablations isolating EPR and PSR (including an early-vs-late anchor control), and sensitivity to alpha, beta, and tau, with mean/std over 3 seeds. The authors conclude that EUA-Cal achieves state-of-the-art calibration on both CNNs and LoRA-fine-tuned LLMs while largely preserving accuracy.
Critical strengths
S1. Simple, architecture-agnostic, drop-in method. EUA-Cal needs no architecture change and no extra inference passes: the early-model targets can be precomputed in a single forward pass, and the method applies unchanged to CNNs and to LoRA-fine-tuned LLMs. The two regularizers are intuitive and cheap. For practitioners this is a genuinely usable contribution.
S2. Broad and well-organized empirical study. 8 models x 6-7 datasets, covering calibration, distribution shift (CIFAR-10-C/100-C, ARC-E to ARC-C/SciQ/CSQA), OOD detection, ablations on EPR/PSR, anchor-model selection (Fig. 7), and // sensitivity, with mean std over 3 seeds. Crucially, the phenomenon itself is reproduced consistently across CNN and LLM backbones and across image classification and MCQ - that breadth is itself evidence that the effect is real rather than a single-benchmark artifact.
Critical weaknesses
W1. PSR's framing is not matched by its implementation (clarification needed). The similarity is computed from the frozen early feature and the frozen early prototypes ; therefore is a fixed distribution for the entire run. As implemented, PSR does not regularize the current feature space - it is an output-level KL toward a fixed soft target, formally the same family of term as EPR, differing only in how the target was constructed. Statements such as "PSR exploits uncertainty reflected in the early feature space" therefore read stronger than the implementation supports. This is easy to fix and worth fixing, because it sharpens the claimed novelty: either (a) state the mechanism precisely, or (b) add the more interesting variant in which the similarity is computed from current features (aligning current features to early prototypes) and ablate both variants.
W2. Two standard reference points are absent from the comparison tables. (i) Post-hoc Temperature Scaling (Guo et al. 2017) - cited in Related Work but never evaluated, despite being the de-facto baseline and essentially free; Beta/field-aware calibration are likewise cited but not run. (ii) The plain early-stopped checkpoint's own ECE/ACC is never reported in the tables, even though the entire narrative rests on "the early model is better calibrated." Note that the CIFAR-10 numbers already suggest EUA-Cal beats both trivial alternatives (0.54 vs. 1.65 early-stop vs. 3.36 final, S4), which is encouraging - but this needs to be shown explicitly across all models and datasets, ideally alongside a simple entropy/logit-penalty regularizer. Adding these would make the empirical case complete.
Questions for authors
-
Can you confirm that no test data entered any selection (anchor epoch, , , ), and report NLL/Brier to corroborate the near-perfect CNN calibration?
-
In PSR, is computed from the frozen early feature or the current feature? If early, what does PSR add over EPR beyond target construction - and can you ablate a current-feature variant?
8 comments
Weakness 2 and Question 3: Comparison with more calibration baselines.
Thanks for your suggestion. We provide a comparison of our method with these baselines. As shown in Table R20 and Table R21, our EUA-Cal method consistently achieves the lowest ECE and AECE across all six datasets, outperforming early stopping, post-hoc calibration methods, and logit regularization. Moreover, EUA-Cal achieves the highest accuracy on all three image classification datasets and SciQ, with only marginal accuracy reductions on ARC-C and ARC-E.
Compared with the early model, EUA-Cal benefits from continued training while preserving early uncertainty awareness. Compared with post-hoc calibration methods that rescale confidence and logit-penalty regularization that penalizes high-confidence predictions, EUA-Cal preserves sample-level uncertainty through regularization, thereby mitigating overconfidence more effectively. These results demonstrate that EUA-Cal provides more effective calibration than these simpler methods. The corresponding results and analyses have been added to Appendix C.3 (Tables 15 and 16) of the revised paper.
Table R20: Comparison with more calibration baselines on ResNet-50.
| Method | CIFAR-10 | CIFAR-100 | Tiny-ImageNet | ||||||
|---|---|---|---|---|---|---|---|---|---|
| ECE↓ | AECE↓ | ACC↑ | ECE↓ | AECE↓ | ACC↑ | ECE↓ | AECE↓ | ACC↑ | |
| Vanilla | 3.36±0.17 | 3.31±0.19 | 94.57±0.15 | 9.19±0.16 | 9.07±0.26 | 76.75±0.62 | 5.54±0.44 | 5.52±0.43 | 64.45±0.45 |
| TS | 0.94±0.07 | 1.08±0.18 | 94.57±0.15 | 4.26±0.31 | 4.14±0.36 | 76.75±0.62 | 2.75±0.41 | 2.73±0.57 | 64.45±0.45 |
| Beta Calibration | 0.96±0.11 | 1.14±0.18 | 94.57±0.15 | 4.98±0.16 | 4.87±0.21 | 76.75±0.62 | 4.75±0.32 | 4.68±0.25 | 64.45±0.45 |
| Neural Calibration | 1.50±0.12 | 1.38±0.26 | 94.57±0.15 | 6.69±0.06 | 6.65±0.03 | 76.75±0.62 | 11.10±0.08 | 11.08±0.10 | 64.45±0.45 |
| Early Model | 1.84±0.44 | 1.79±0.49 | 91.30±0.64 | 5.93±0.31 | 5.86±0.37 | 72.34±0.53 | 1.91±0.06 | 1.95±0.11 | 55.31±0.22 |
| Early Model w/ TS | 0.85±0.24 | 0.69±0.42 | 91.30±0.64 | 2.11±0.13 | 2.03±0.16 | 72.34±0.53 | 1.88±0.20 | 1.79±0.24 | 55.31±0.22 |
| Logit Penalty (L2) | 2.66±0.31 | 2.89±0.28 | 94.33±0.30 | 8.96±0.37 | 9.09±0.28 | 77.21±0.76 | 4.96±0.32 | 4.88±0.39 | 64.75±0.93 |
| Ours | 0.54±0.12 | 0.50±0.09 | 94.63±0.05 | 1.28±0.11 | 1.18±0.21 | 77.66±0.32 | 1.38±0.17 | 1.29±0.11 | 65.31±0.47 |
Table R21: Comparison with more calibration baselines on Llama3-8B. The early model is the same pretrained Llama3-8B, yielding zero std.
| Method | ARC-C | ARC-E | SciQ | ||||||
|---|---|---|---|---|---|---|---|---|---|
| ECE↓ | AECE↓ | ACC↑ | ECE↓ | AECE↓ | ACC↑ | ECE↓ | AECE↓ | ACC↑ | |
| Vanilla | 17.24±0.25 | 17.12±0.31 | 81.31±0.64 | 6.54±0.03 | 6.48±0.03 | 92.81±0.12 | 6.42±0.38 | 6.10±0.41 | 93.40±0.52 |
| TS | 6.71±0.45 | 7.68±0.50 | 81.31±0.64 | 4.18±0.19 | 4.08±0.11 | 92.81±0.12 | 3.80±0.32 | 3.84±0.22 | 93.40±0.52 |
| Beta Calibration | 6.52±0.17 | 7.38±0.27 | 81.31±0.64 | 4.29±0.34 | 4.30±0.30 | 92.81±0.12 | 3.64±0.19 | 3.80±0.46 | 93.40±0.52 |
| Neural Calibration | 13.61±0.51 | 13.31±0.75 | 81.31±0.64 | 5.95±0.30 | 5.92±0.26 | 92.81±0.12 | 5.88±0.37 | 5.78±0.31 | 93.40±0.52 |
| Early Model | 4.37 | 4.67 | 75.68 | 6.89 | 6.38 | 89.06 | 5.82 | 4.75 | 91.00 |
| Early Model w/ TS | 5.03 | 5.04 | 75.68 | 1.76 | 1.84 | 89.06 | 2.88 | 2.75 | 91.00 |
| Logit Penalty (L2) | 16.93±0.53 | 17.24±0.74 | 80.06±0.52 | 5.53±0.25 | 5.92±0.33 | 92.27±0.31 | 4.17±0.08 | 4.81±0.42 | 93.63±0.15 |
| Ours | 3.60±0.34 | 4.17±0.60 | 80.94±0.47 | 1.67±0.19 | 1.79±0.29 | 92.72±0.19 | 2.19±0.24 | 2.57±0.49 | 93.90±0.28 |
Question 1: Clarification of data usage and additional calibration metrics.
We confirm that no test data were used in any selection. Instead, following standard practice, we used a separate validation set to select the anchor checkpoint, , and . The parameter was determined by minimizing the training NLL of prototype predictions. All baseline methods were tuned using the same validation set to ensure a fair comparison.
To further demonstrate the calibration performance of our method, we provide more evaluation metrics, including NLL and Brier Score. As shown in Table R22 and Table R23, our EUA-Cal achieves the lowest NLL and Brier Score across all six datasets, confirming its effectiveness beyond ECE evaluation. The corresponding results and analyses have been added to Appendix C.2 (Tables 13 and 14) of the revised paper.
Table R22: Comparison of NLL (×10) and Brier Score (%) on ResNet-50. Best results are bolded and second-best results are underlined.
| Method | CIFAR-10 | CIFAR-100 | Tiny-ImageNet | |||
|---|---|---|---|---|---|---|
| NLL↓ | Brier↓ | NLL↓ | Brier↓ | NLL↓ | Brier↓ | |
| Vanilla | 2.08±0.11 | 8.45±0.28 | 9.64±0.20 | 34.30±0.79 | ||
| MbLS | 2.30±0.18 | 8.73±0.22 | 9.68±0.19 | 33.99±0.67 | 14.48±0.16 | 47.23±0.57 |
| BalCAL | 2.09±0.06 | 8.83±0.24 | 9.53±0.15 | 14.54±0.14 | 47.32±0.34 | |
| PLP | 1.92±0.04 | 8.93±0.23 | 35.60±0.90 | 16.57±0.82 | 51.46±1.31 | |
| MaC-Cal | 9.30±0.05 | 33.73±0.14 | 16.89±0.06 | 52.25±0.10 | ||
| Ours | 1.60±0.03 | 8.00±0.13 | 7.66±0.05 | 30.79±0.07 | 13.74±0.10 | 45.50±0.17 |
Table R23: Comparison of NLL (×10) and Brier Score (%) on Llama3-8B. Best results are bolded and second-best results are underlined.
| Method | ARC-C | ARC-E | SciQ | |||
|---|---|---|---|---|---|---|
| NLL↓ | Brier↓ | NLL↓ | Brier↓ | NLL↓ | Brier↓ | |
| Vanilla | 11.50±0.25 | 34.80±0.33 | 4.40±0.03 | 13.25±0.14 | 5.03±0.11 | 12.82±0.63 |
| MbLS | 13.03±1.97 | 36.38±3.29 | 5.50±0.31 | 15.40±0.58 | 5.32±0.06 | 13.56±0.32 |
| CSAM | 10.43±0.24 | 32.97±0.33 | 4.13±0.58 | |||
| EDL | 7.14±0.10 | 33.64±0.46 | 3.15±0.05 | 13.29±0.16 | 2.98±0.06 | 12.12±0.27 |
| IB-EDL | 12.67±0.20 | 2.93±0.04 | 12.02±0.27 | |||
| Ours | 6.08±0.08 | 28.36±0.30 | 2.39±0.05 | 11.65±0.38 | 2.08±0.03 | 10.38±0.13 |
Question 4: Distinction from KL-to-reference regularization in RLHF/PPO.
EPR shares the general form of KL-to-reference regularization, but EUA-Cal differs from RLHF/PPO in its objective and use of early predictions and feature structures. While RLHF/PPO uses KL regularization to constrain policy updates during reward optimization, EUA-Cal aims to preserve early uncertainty for confidence calibration. PSR further incorporates uncertainty information from early feature structures by constructing confidence-weighted class prototypes and deriving fixed prototype predictions from the similarities between features and prototypes. By combining EPR and PSR, EUA-Cal jointly leverages uncertainty awareness from early predictions and feature structures to mitigate overconfidence.
Question 5: Statistical significance of calibration improvements.
We conducted paired statistical tests on ARC-C using Llama3-8B, with three runs per method. We compute ECE using 20 equal-width bins and define the average paired ECE improvement as
(ECE_{baseline, s} -ECE_{ours, s}).$$ We use 10,000 paired bootstrap resamples to estimate 95% confidence intervals and 10,000 paired permutations to obtain two-sided p-values, followed by Holm correction. As shown in Table R24, EUA-Cal significantly outperforms MIT-A (p < 0.001), while its improvement over IB-EDL is not statistically significant. **Table R24: Paired statistical significance tests of ECE improvements on Llama3-8B / ARC-C.** |Method|ECE (%) ↓|Δ (pp)|95% CI (pp)|Adjusted p-value| |:---:|:---:|:---:|:---:|:---:| |EUA-Cal|**3.577**|–|–|–| |IB-EDL|3.912|0.335|[-1.426, 1.530]|0.637| |MIT-A|6.358|2.781|[0.946, 3.917]|**0.0006**|
Thanks for your time and effort in handling this paper. We appreciate your recognition of our practical method, extensive experiments, and consistent findings. We respond to your comments point by point.
Weakness 1 and Question 2: Clarification and ablation of PSR.
We appreciate the reviewer's insightful suggestion. We confirm that PSR constructs fixed soft targets from frozen early features and prototypes, rather than directly regularizing the current feature space. PSR derives additional supervision from uncertainty reflected in the early feature space. We have revised the description in the Abstract and Introduction (pp. 1-2) to clarify this distinction.
Following the reviewer's suggestion, we further provide a current feature variant, which computes similarities between current features and frozen early prototypes. As shown in Table R18 and Table R19, our EUA-Cal performs better in calibration against this variant across all six datasets. These results support our design choice.
Table R18: Comparison on ResNet-50.
Table R19: Comparison on Llama3-8B.