acceptodds
Under review as a conference paper at ICLR 2027

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Abstract

Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose **EUA-Cal**, a novel method that exploits the **E**arly model as an **U**ncertainty **A**nchor for **Cal**ibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods. Code is available in the supplementary materials.

Primary area:
interpretability and explainable AI
Keywords:
uncertainty calibration, calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_6A5dOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies confidence calibration of deep networks and LoRA-fine-tuned LLMs, whose predictions become increasingly overconfident as training/fine-tuning proceeds. The authors report a consistent phenomenon across models: ECE is minimized at an early stage of training (or fine-tuning) and then rises, while accuracy improves only marginally (Fig. 1a). A feature-space analysis further suggests that the early model retains "uncertainty awareness" in both its predictions and its feature structure, which is gradually lost with continued training (Fig. 1b).

To prevent this, the paper proposes EUA-Cal, which freezes an early checkpoint as an uncertainty anchor and adds two regularizers to the remaining training: (i) Early Prediction Regularization (EPR), which penalizes the KL divergence between the current predictive distribution and the frozen early predictions; and (ii) Prototype Structure Regularization (PSR), which penalizes the KL divergence between the current prediction and a soft target distribution q, constructed from cosine similarities between (early) features and confidence-weighted early class prototypes. The overall objective is the cross-entropy loss plus the EPR and PSR terms, weighted by alpha and beta respectively.

The claimed contributions are (1) the "early model is better calibrated" observation and an analysis attributing it to uncertainty awareness in early predictions and features; (2) the EUA-Cal method, which uses the early model as a reusable uncertainty anchor without architectural change or extra inference cost; and (3) an empirical demonstration that EUA-Cal outperforms prior calibration methods.

Validation is empirical rather than theoretical: 8 models (ResNet-50/101, Wide-ResNet-26-10, DenseNet-121, ViT-B/32, and LoRA-fine-tuned Llama3-8B, Llama3.1-8B, Qwen3-8B-Base) over 6-7 datasets, covering image classification (CIFAR-10/100, Tiny-ImageNet) and multiple-choice QA (SciQ, ARC-C, ARC-E; CSQA for distribution shift). Experiments span in-distribution calibration, distribution shift, OOD detection, ablations isolating EPR and PSR (including an early-vs-late anchor control), and sensitivity to alpha, beta, and tau, with mean/std over 3 seeds. The authors conclude that EUA-Cal achieves state-of-the-art calibration on both CNNs and LoRA-fine-tuned LLMs while largely preserving accuracy.

Critical strengths

S1. Simple, architecture-agnostic, drop-in method. EUA-Cal needs no architecture change and no extra inference passes: the early-model targets can be precomputed in a single forward pass, and the method applies unchanged to CNNs and to LoRA-fine-tuned LLMs. The two regularizers are intuitive and cheap. For practitioners this is a genuinely usable contribution.

S2. Broad and well-organized empirical study. 8 models x 6-7 datasets, covering calibration, distribution shift (CIFAR-10-C/100-C, ARC-E to ARC-C/SciQ/CSQA), OOD detection, ablations on EPR/PSR, anchor-model selection (Fig. 7), and α\alpha/β\beta/τ\tau sensitivity, with mean ±\pm std over 3 seeds. Crucially, the phenomenon itself is reproduced consistently across CNN and LLM backbones and across image classification and MCQ - that breadth is itself evidence that the effect is real rather than a single-benchmark artifact.

Critical weaknesses

W1. PSR's framing is not matched by its implementation (clarification needed). The similarity si,ks_{i,k} is computed from the frozen early feature hith^{t}_{i} and the frozen early prototypes μk\mu_{k}; therefore qq is a fixed distribution for the entire run. As implemented, PSR does not regularize the current feature space - it is an output-level KL toward a fixed soft target, formally the same family of term as EPR, differing only in how the target was constructed. Statements such as "PSR exploits uncertainty reflected in the early feature space" therefore read stronger than the implementation supports. This is easy to fix and worth fixing, because it sharpens the claimed novelty: either (a) state the mechanism precisely, or (b) add the more interesting variant in which the similarity is computed from current features (aligning current features to early prototypes) and ablate both variants.

W2. Two standard reference points are absent from the comparison tables. (i) Post-hoc Temperature Scaling (Guo et al. 2017) - cited in Related Work but never evaluated, despite being the de-facto baseline and essentially free; Beta/field-aware calibration are likewise cited but not run. (ii) The plain early-stopped checkpoint's own ECE/ACC is never reported in the tables, even though the entire narrative rests on "the early model is better calibrated." Note that the CIFAR-10 numbers already suggest EUA-Cal beats both trivial alternatives (0.54 vs. 1.65 early-stop vs. 3.36 final, S4), which is encouraging - but this needs to be shown explicitly across all models and datasets, ideally alongside a simple entropy/logit-penalty regularizer. Adding these would make the empirical case complete.

Questions for authors
  1. Can you confirm that no test data entered any selection (anchor epoch, α\alpha, β\beta, τ\tau), and report NLL/Brier to corroborate the near-perfect CNN calibration?

  2. In PSR, is qq computed from the frozen early feature hith^{t}_{i} or the current feature? If early, what does PSR add over EPR beyond target construction - and can you ablate a current-feature variant?

8 comments

Thanks for your time and effort in handling this paper. We appreciate your recognition of our practical method, extensive experiments, and consistent findings. We respond to your comments point by point.

Weakness 1 and Question 2: Clarification and ablation of PSR.

We appreciate the reviewer's insightful suggestion. We confirm that PSR constructs fixed soft targets from frozen early features and prototypes, rather than directly regularizing the current feature space. PSR derives additional supervision from uncertainty reflected in the early feature space. We have revised the description in the Abstract and Introduction (pp. 1-2) to clarify this distinction.

Following the reviewer's suggestion, we further provide a current feature variant, which computes similarities between current features and frozen early prototypes. As shown in Table R18 and Table R19, our EUA-Cal performs better in calibration against this variant across all six datasets. These results support our design choice.

Table R18: Comparison on ResNet-50.

MethodCIFAR-10CIFAR-100Tiny-ImageNet
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Current0.75±0.170.77±0.2194.66±0.202.86±0.132.64±0.4077.20±0.171.99±0.341.91±0.3065.51±0.20
Ours0.54±0.120.50±0.0994.63±0.051.28±0.111.18±0.2177.66±0.321.38±0.171.29±0.1165.31±0.47

Table R19: Comparison on Llama3-8B.

MethodARC-CARC-ESciQ
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Current5.66±1.175.56±1.2580.74±0.432.49±0.422.79±0.1492.73±0.064.52±0.214.73±0.2692.97±0.06
Ours3.60±0.344.17±0.6080.94±0.471.67±0.191.79±0.2992.72±0.192.19±0.242.57±0.4993.90±0.28

Weakness 2 and Question 3: Comparison with more calibration baselines.

Thanks for your suggestion. We provide a comparison of our method with these baselines. As shown in Table R20 and Table R21, our EUA-Cal method consistently achieves the lowest ECE and AECE across all six datasets, outperforming early stopping, post-hoc calibration methods, and logit regularization. Moreover, EUA-Cal achieves the highest accuracy on all three image classification datasets and SciQ, with only marginal accuracy reductions on ARC-C and ARC-E.

Compared with the early model, EUA-Cal benefits from continued training while preserving early uncertainty awareness. Compared with post-hoc calibration methods that rescale confidence and logit-penalty regularization that penalizes high-confidence predictions, EUA-Cal preserves sample-level uncertainty through regularization, thereby mitigating overconfidence more effectively. These results demonstrate that EUA-Cal provides more effective calibration than these simpler methods. The corresponding results and analyses have been added to Appendix C.3 (Tables 15 and 16) of the revised paper.

Table R20: Comparison with more calibration baselines on ResNet-50.

MethodCIFAR-10CIFAR-100Tiny-ImageNet
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Vanilla3.36±0.173.31±0.1994.57±0.159.19±0.169.07±0.2676.75±0.625.54±0.445.52±0.4364.45±0.45
TS0.94±0.071.08±0.1894.57±0.154.26±0.314.14±0.3676.75±0.622.75±0.412.73±0.5764.45±0.45
Beta Calibration0.96±0.111.14±0.1894.57±0.154.98±0.164.87±0.2176.75±0.624.75±0.324.68±0.2564.45±0.45
Neural Calibration1.50±0.121.38±0.2694.57±0.156.69±0.066.65±0.0376.75±0.6211.10±0.0811.08±0.1064.45±0.45
Early Model1.84±0.441.79±0.4991.30±0.645.93±0.315.86±0.3772.34±0.531.91±0.061.95±0.1155.31±0.22
Early Model w/ TS0.85±0.240.69±0.4291.30±0.642.11±0.132.03±0.1672.34±0.531.88±0.201.79±0.2455.31±0.22
Logit Penalty (L2)2.66±0.312.89±0.2894.33±0.308.96±0.379.09±0.2877.21±0.764.96±0.324.88±0.3964.75±0.93
Ours0.54±0.120.50±0.0994.63±0.051.28±0.111.18±0.2177.66±0.321.38±0.171.29±0.1165.31±0.47

Table R21: Comparison with more calibration baselines on Llama3-8B. The early model is the same pretrained Llama3-8B, yielding zero std.

MethodARC-CARC-ESciQ
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Vanilla17.24±0.2517.12±0.3181.31±0.646.54±0.036.48±0.0392.81±0.126.42±0.386.10±0.4193.40±0.52
TS6.71±0.457.68±0.5081.31±0.644.18±0.194.08±0.1192.81±0.123.80±0.323.84±0.2293.40±0.52
Beta Calibration6.52±0.177.38±0.2781.31±0.644.29±0.344.30±0.3092.81±0.123.64±0.193.80±0.4693.40±0.52
Neural Calibration13.61±0.5113.31±0.7581.31±0.645.95±0.305.92±0.2692.81±0.125.88±0.375.78±0.3193.40±0.52
Early Model4.374.6775.686.896.3889.065.824.7591.00
Early Model w/ TS5.035.0475.681.761.8489.062.882.7591.00
Logit Penalty (L2)16.93±0.5317.24±0.7480.06±0.525.53±0.255.92±0.3392.27±0.314.17±0.084.81±0.4293.63±0.15
Ours3.60±0.344.17±0.6080.94±0.471.67±0.191.79±0.2992.72±0.192.19±0.242.57±0.4993.90±0.28

Question 1: Clarification of data usage and additional calibration metrics.

We confirm that no test data were used in any selection. Instead, following standard practice, we used a separate validation set to select the anchor checkpoint, α\alpha, and β\beta. The parameter τ\tau was determined by minimizing the training NLL of prototype predictions. All baseline methods were tuned using the same validation set to ensure a fair comparison.

To further demonstrate the calibration performance of our method, we provide more evaluation metrics, including NLL and Brier Score. As shown in Table R22 and Table R23, our EUA-Cal achieves the lowest NLL and Brier Score across all six datasets, confirming its effectiveness beyond ECE evaluation. The corresponding results and analyses have been added to Appendix C.2 (Tables 13 and 14) of the revised paper.

Table R22: Comparison of NLL (×10) and Brier Score (%) on ResNet-50. Best results are bolded and second-best results are underlined.

MethodCIFAR-10CIFAR-100Tiny-ImageNet
NLL↓Brier↓NLL↓Brier↓NLL↓Brier↓
Vanilla2.08±0.118.45±0.289.64±0.2034.30±0.7913.89±0.20‾\underline{13.89\pm0.20}45.75±0.69‾\underline{45.75\pm0.69}
MbLS2.30±0.188.73±0.229.68±0.1933.99±0.6714.48±0.1647.23±0.57
BalCAL2.09±0.068.83±0.249.53±0.1532.82±0.53‾\underline{32.82\pm0.53}14.54±0.1447.32±0.34
PLP1.92±0.048.93±0.239.29±0.41‾\underline{9.29\pm0.41}35.60±0.9016.57±0.8251.46±1.31
MaC-Cal1.76±0.02‾\underline{1.76\pm0.02}8.06±0.04‾\underline{8.06\pm0.04}9.30±0.0533.73±0.1416.89±0.0652.25±0.10
Ours1.60±0.038.00±0.137.66±0.0530.79±0.0713.74±0.1045.50±0.17

Table R23: Comparison of NLL (×10) and Brier Score (%) on Llama3-8B. Best results are bolded and second-best results are underlined.

MethodARC-CARC-ESciQ
NLL↓Brier↓NLL↓Brier↓NLL↓Brier↓
Vanilla11.50±0.2534.80±0.334.40±0.0313.25±0.145.03±0.1112.82±0.63
MbLS13.03±1.9736.38±3.295.50±0.3115.40±0.585.32±0.0613.56±0.32
CSAM10.43±0.2432.97±0.334.13±0.5812.24±1.84‾\underline{12.24\pm1.84}2.64±0.03‾\underline{2.64\pm0.03}11.04±0.12‾\underline{11.04\pm0.12}
EDL7.14±0.1033.64±0.463.15±0.0513.29±0.162.98±0.0612.12±0.27
IB-EDL6.47±0.08‾\underline{6.47\pm0.08}31.30±0.48‾\underline{31.30\pm0.48}2.96±0.03‾\underline{2.96\pm0.03}12.67±0.202.93±0.0412.02±0.27
Ours6.08±0.0828.36±0.302.39±0.0511.65±0.382.08±0.0310.38±0.13

Question 4: Distinction from KL-to-reference regularization in RLHF/PPO.

EPR shares the general form of KL-to-reference regularization, but EUA-Cal differs from RLHF/PPO in its objective and use of early predictions and feature structures. While RLHF/PPO uses KL regularization to constrain policy updates during reward optimization, EUA-Cal aims to preserve early uncertainty for confidence calibration. PSR further incorporates uncertainty information from early feature structures by constructing confidence-weighted class prototypes and deriving fixed prototype predictions from the similarities between features and prototypes. By combining EPR and PSR, EUA-Cal jointly leverages uncertainty awareness from early predictions and feature structures to mitigate overconfidence.

Question 5: Statistical significance of calibration improvements.

We conducted paired statistical tests on ARC-C using Llama3-8B, with three runs per method. We compute ECE using 20 equal-width bins and define the average paired ECE improvement as

(ECE_{baseline, s} -ECE_{ours, s}).$$ We use 10,000 paired bootstrap resamples to estimate 95% confidence intervals and 10,000 paired permutations to obtain two-sided p-values, followed by Holm correction. As shown in Table R24, EUA-Cal significantly outperforms MIT-A (p < 0.001), while its improvement over IB-EDL is not statistically significant. **Table R24: Paired statistical significance tests of ECE improvements on Llama3-8B / ARC-C.** |Method|ECE (%) ↓|Δ (pp)|95% CI (pp)|Adjusted p-value| |:---:|:---:|:---:|:---:|:---:| |EUA-Cal|**3.577**|–|–|–| |IB-EDL|3.912|0.335|[-1.426, 1.530]|0.637| |MIT-A|6.358|2.781|[0.946, 3.917]|**0.0006**|
Reviewer_uW54OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EUA-Cal, a calibration method that uses an early training checkpoint as a fixed reference for subsequent training. It combines KL regularization toward the early model's predictions with a second regularization term based on confidence-weighted prototypes constructed from early features. The authors evaluate the method on image classification and multiple-choice question answering with LLMs, reporting accuracy and calibration results. Additional experiments examine robustness under distribution shift and the contributions of individual components.

Critical strengths
  • The method is clearly described. Eqs. (1)–(6) define the two regularization terms alongside the cross-entropy loss. Table 5 evaluates EPR and PSR separately and compares early and late anchors. These experiments help establish the contributions of the individual components to the reported calibration improvements.

  • The calibration improvements are demonstrated across multiple architectures and tasks. For ResNet-50 on CIFAR-100, Table 1 shows that ECE decreases from 9.19% to 1.28%, while accuracy increases from 76.75% to 77.66%. The appendix provides additional results on CNNs, transformers, and LLMs, while Section 4.3 examines performance under distribution shift. These experiments provide evidence of the method's effectiveness, although some settings show accuracy trade-offs.

  • The paper provides useful observations about how calibration changes during training. Figure 1 shows that ECE can reach its minimum relatively early, even when accuracy continues to improve. The feature visualizations further illustrate how low-confidence samples become less prominent as training progresses. These observations help motivate the use of early checkpoints for calibration.

Critical weaknesses
  • The experiments do not clearly distinguish the preservation of early uncertainty from the general effect of soft-target regularization. Since the targets are fixed, the objective in Eq. (6) can be rewritten as cross-entropy against a weighted mixture of the ground-truth labels, early predictions, and prototype predictions, up to a constant and scaling factor. The paper does not include a temperature-scaled cross-entropy baseline or a direct comparison with the early checkpoint itself. These comparisons would help establish whether the proposed method offers benefits beyond conventional confidence regularization.

  • The training budget is not sufficiently clear, which makes it difficult to assess the fairness and reproducibility of the comparisons. Algorithm 1 first trains the model for tt epochs and then performs another TT epochs of optimization, while Appendix A.2 specifies 200 training epochs on CIFAR and Appendix A.3 selects t=60t=60. It is unclear whether the total training budget is 200 or 260 epochs, or whether the learning-rate schedule is continued or restarted. The authors should clarify the actual training protocol and whether the baselines use matched optimization budgets.

  • The evaluation relies primarily on ECE and AECE, which provide only a limited view of calibration quality. Both metrics depend on confidence binning and may not capture differences in the full predictive distribution. Although the paper reports strong improvements in these metrics, additional evaluation using proper scoring rules such as negative log-likelihood and Brier score would help determine whether EUA-Cal improves probabilistic predictions more generally, rather than mainly reducing confidence-based calibration error.

Questions for authors
  • Could the authors provide results for temperature-scaled cross-entropy and the early checkpoint itself, using the same evaluation and selection protocol? Together with the early/late anchor ablation, these comparisons would help clarify whether the gains come from preserving early uncertainty or from soft-target regularization more generally.

  • Does EUA-Cal use TT total epochs or t+Tt+T epochs, and are the model weights, optimizer state, and learning-rate schedule continued or restarted after selecting the anchor? Please also clarify how α\alpha and β\beta are selected for each setting and how the fifth answer option is handled for CSQA, since Appendix A.2 specifies only A/B/C/D. These details are important for assessing the fairness and reproducibility of the experiments.

5 comments

Thanks for your time and effort in handling this paper. We appreciate your recognition of the clarity of our methodology, extensive experiments, and useful observation. We respond to your comments point by point.

Weakness 1 and Question 1: distinguishing the preservation of early uncertainty from the general effect of soft-target regularization.

Following your valuable suggestions, we report the performance of temperature-scaled cross-entropy (TS-CE) and the early checkpoint itself in Table R14 and Table R15. Our EUA-Cal method achieves superior performance in calibration across all six datasets, while maintaining competitive accuracy. The corresponding results and analyses have been added to Appendix C.3 (Tables 15 and 16) of the revised paper. Moreover, we have provided a more detailed ablation study in Table 5 of the revised paper. EUA-Cal variants with late or softened late models lead to slightly higher accuracy in most settings, but their calibration performance is substantially worse. These results suggest that our EUA-Cal method benefits more from the preserved early uncertainty.

Table R14: Comparison with temperature-scaled cross-entropy and the early checkpoint on ResNet-50.

CIFAR-10CIFAR-100Tiny-ImageNet
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
TS-CE4.18±0.154.17±0.1494.98±0.2416.45±0.3516.44±0.3477.54±0.1020.04±0.6920.06±0.6665.22±0.35
Early Model1.84±0.441.79±0.4991.30±0.645.93±0.315.86±0.3772.34±0.531.91±0.061.95±0.1155.31±0.22
Ours0.54±0.120.50±0.0994.63±0.051.28±0.111.18±0.2177.66±0.321.38±0.171.29±0.1165.31±0.47

Table R15: Comparison with temperature-scaled cross-entropy and the early checkpoint on Llama3-8B. The early model is the same pretrained Llama3-8B, yielding zero std.

ARC-CARC-ESciQ
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
TS-CE19.92±0.2519.50±0.3479.69±0.247.06±0.077.07±0.0392.70±0.055.96±0.165.87±0.1593.83±0.25
Early Model4.37±04.67±075.68±06.89±06.38±089.06±05.82±04.75±091.00±0
Ours3.60±0.344.17±0.6080.94±0.471.67±0.191.79±0.2992.72±0.192.19±0.242.57±0.4993.90±0.28

Weakness 2 and Question 2: Training budget and details.

The final model is trained for TT epochs after a separate tt-epoch pre-learning stage. After pre-learning to epoch tt, EUA-Cal restarts training rather than continuing from the early checkpoint. For DNNs, after pre-learning for predetermined tt epochs, we freeze the early model and restart training from initialization for the full training schedule, e.g., 200 epochs for ResNet-50/101. For LLMs, we use either the pretrained model or a model obtained after 5-15 fine-tuning steps as the frozen early model.

For a fair comparison, all models are trained with the same training schedule and optimization settings. EUA-Cal requires an additional tt-epoch pre-learning stage, but the early model is prepared only once and shared across three runs. EUA-Cal incurs modest training overhead and no additional inference overhead.

We select α\alpha and β\beta from {0.1, 0.2, 0.3} through validation set tuning, while τ\tau is optimized by minimizing the training NLL of prototype predictions. For a fair comparison, all compared baselines are also tuned on validation sets, and follow the same training schedule. More training details and hyperparameter settings have been added to Appendices A.2 and A.3 of the revised paper.

“Appendix A.2 specifies only A/B/C/D” is a typo. Thank you for pointing this out. For CSQA, we use all five answer options (A/B/C/D/E), while the other datasets use (A/B/C/D, or E when applicable). We have corrected this description in Appendix A.2 (p. 14).

Weakness 3: Additional evaluation.

Thanks for your suggestions. We further report more evaluation metrics, including NLL and Brier Score. As shown in Table R16 and Table R17, our EUA-Cal achieves the lowest NLL and Brier Score across all six datasets, confirming its effectiveness beyond ECE evaluation. The corresponding results and analyses have been added to Appendix C.2 (Tables 13 and 14) of the revised paper.

Table R16: Comparison of NLL (×10) and Brier Score (%) on ResNet-50. Best results are bolded and second-best results are underlined.

MethodCIFAR-10CIFAR-100Tiny-ImageNet
NLL↓Brier↓NLL↓Brier↓NLL↓Brier↓
Vanilla2.08±0.118.45±0.289.64±0.2034.30±0.7913.89±0.20‾\underline{13.89\pm0.20}45.75±0.69‾\underline{45.75\pm0.69}
MbLS2.30±0.188.73±0.229.68±0.1933.99±0.6714.48±0.1647.23±0.57
BalCAL2.09±0.068.83±0.249.53±0.1532.82±0.53‾\underline{32.82\pm0.53}14.54±0.1447.32±0.34
PLP1.92±0.048.93±0.239.29±0.41‾\underline{9.29\pm0.41}35.60±0.9016.57±0.8251.46±1.31
MaC-Cal1.76±0.02‾\underline{1.76\pm0.02}8.06±0.04‾\underline{8.06\pm0.04}9.30±0.0533.73±0.1416.89±0.0652.25±0.10
Ours1.60±0.038.00±0.137.66±0.0530.79±0.0713.74±0.1045.50±0.17

Table R17: Comparison of NLL (×10) and Brier Score (%) on Llama3-8B. Best results are bolded and second-best results are underlined.

MethodARC-CARC-ESciQ
NLL↓Brier↓NLL↓Brier↓NLL↓Brier↓
Vanilla11.50±0.2534.80±0.334.40±0.0313.25±0.145.03±0.1112.82±0.63
MbLS13.03±1.9736.38±3.295.50±0.3115.40±0.585.32±0.0613.56±0.32
CSAM10.43±0.2432.97±0.334.13±0.5812.24±1.84‾\underline{12.24\pm1.84}2.64±0.03‾\underline{2.64\pm0.03}11.04±0.12‾\underline{11.04\pm0.12}
EDL7.14±0.1033.64±0.463.15±0.0513.29±0.162.98±0.0612.12±0.27
IB-EDL6.47±0.08‾\underline{6.47\pm0.08}31.30±0.48‾\underline{31.30\pm0.48}2.96±0.03‾\underline{2.96\pm0.03}12.67±0.202.93±0.0412.02±0.27
Ours6.08±0.0828.36±0.302.39±0.0511.65±0.382.08±0.0310.38±0.13

We have clarified the training procedure in Algorithm 1 (p. 6) and the implementation and hyperparameter settings in Appendices A.2-A.3 (pp. 14-15), expanded the anchor-model ablation in Section 4.4 (Table 5, p. 9), and added the simpler calibration comparisons in Appendix C.3 (Tables 15 and 16, p. 21) and NLL/Brier results in Appendix C.2 (Tables 13 and 14, p. 21). The corresponding revisions are highlighted in red.

Dear Reviewer uW54,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_EBbGOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This is a well-written and technically interesting paper that addresses an important problem: modern neural networks often become more accurate but less calibrated as training continues. The proposed EUA-Cal method is conceptually coherent, empirically effective, and evaluated across a broad range of image-classification models and fine-tuned language models. I therefore recommend acceptance, subject to minor revisions addressing several issues related to experimental comparisons, reproducibility, and methodological clarification.

Critical strengths

The paper is clearly written and addresses an important problem in modern deep learning: continued training may improve accuracy while progressively worsening confidence calibration. The central idea of using an early checkpoint as an uncertainty anchor is intuitive and practically relevant. The proposed EPR and PSR components are conceptually complementary, operating at both the prediction and feature levels. The empirical evaluation is broad, covering multiple image datasets, architectures, fine-tuned LLMs, distribution shifts, OOD detection, ablations, and hyperparameter sensitivity. The results consistently show substantial calibration improvements while generally preserving competitive accuracy. Overall, the paper provides useful insight into the relationship between training progression, feature structure, and uncertainty awareness.

Critical weaknesses

The main weakness is that several important experimental and methodological details require clarification. In particular, the paper should compare EUA-Cal directly with the selected early checkpoint, standard early stopping, and temperature scaling. Because the method relies critically on choosing an early anchor model, the exact anchor-selection procedure and the extent of validation-set tuning should be specified more carefully. Reproducibility would also benefit from fuller details about the feature representations, pooling strategy, prompting, and answer-token probabilities used in the LLM experiments. Finally, additional calibration metrics such as NLL, Brier score, and reliability diagrams would strengthen the evaluation. These concerns appear fixable and do not undermine the paper’s central contribution.

Questions for authors

Clarification requests that could materially affect the assessment

  1. Anchor selection and comparison with the early model: How is the early anchor checkpoint selected—by validation ECE, accuracy-ECE tradeoff, or a predetermined epoch? How does EUA-Cal compare directly with using that early checkpoint as the final model? This affects whether the gains come from EUA-Cal itself or simply from early stopping.
  2. Comparison with standard post-hoc calibration: Why is temperature scaling absent from the baseline comparisons? Since it is a strong and inexpensive calibration method, its inclusion could materially affect the claimed practical advantage of EUA-Cal.
  3. Validation-set tuning and experimental fairness: Are the anchor checkpoint, α\alpha, β\beta, and τ\tau selected using validation data for every dataset and model? Were competing baselines tuned with comparable access to validation data and comparable computational budgets?
  4. LLM feature construction: Which hidden layer and pooling strategy are used to construct the LLM features for PSR? How are answer-token probabilities and multi-token answer choices handled? These details are necessary to assess whether the LLM results are reproducible and whether PSR is being applied consistently.
  5. Accuracy-calibration tradeoff: In some distribution-shift and LLM results, EUA-Cal substantially improves calibration while reducing accuracy. How do the authors define “maintaining competitive accuracy,” and do the conclusions remain supported when considering NLL, Brier score, or selective prediction performance?
  6. Training procedure ambiguity: After pre-learning to epoch tt, does Algorithm 1 continue training from the checkpoint at tt, or does it restart training? This clarification is important for reproducing the reported results and interpreting the computational cost. Suggestions for future work I would not raise additional future-work questions as conditions for acceptance. The questions above are the ones most likely to affect the current evaluation of the paper’s novelty, fairness, soundness, and reproducibility.
5 comments

Thanks for your time and effort in handling this paper. We appreciate your recognition of the clarity of our presentation, motivation, methodological design, extensive evaluation, and promising results. We respond to your comments point by point.

Weakness: Experimental and methodological details.

We appreciate the reviewer's valuable comments. To address these concerns, we have conducted additional experiments comparing EUA-Cal with early stopping and temperature scaling, evaluated additional calibration metrics, and clarified the anchor selection, hyperparameter tuning, LLM implementation details, and training procedure. We provide detailed responses below.

Question 1: Anchor selection and comparison with the early model.

The early anchor checkpoint is predetermined based on validation ECE and accuracy curves. Specifically, we select an early checkpoint before validation ECE begins to increase consistently, when further training yields only marginal accuracy gains. We have provided the details of anchor selection in Appendix A.3 of the revised paper.

As shown in Table R8 and Table R9, we compare our EUA-Cal method with using that early checkpoint as the final model. Our EUA-Cal method consistently improves both calibration and accuracy across all six datasets. This demonstrates that our method achieves better calibration while preserving early uncertainty awareness and improving accuracy. The corresponding results and analyses have been added to Appendix C.3 (Tables 15 and 16) of the revised paper.

Table R8: Comparison with early model on ResNet-50 (mean±std %).

CIFAR-10CIFAR-100Tiny-ImageNet
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Early Model1.84±0.441.79±0.4991.30±0.645.93±0.315.86±0.3772.34±0.531.91±0.061.95±0.1155.31±0.22
Ours0.54±0.120.50±0.0994.63±0.051.28±0.111.18±0.2177.66±0.321.38±0.171.29±0.1165.31±0.47

Table R9: Comparison with early model on Llama3-8B (mean±std %). The early model is the same pretrained Llama3-8B, yielding zero std.

ARC-CARC-ESciQ
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Early Model4.37±04.67±075.68±06.89±06.38±089.06±05.82±04.75±091.00±0
Ours3.60±0.344.17±0.6080.94±0.471.67±0.191.79±0.2992.72±0.192.19±0.242.57±0.4993.90±0.28

Question 2: Comparison with standard post-hoc calibration.

Thanks for your suggestion. As shown in Table R10 and Table R11, we compare our EUA-Cal method with temperature scaling (TS). Our EUA-Cal method achieves better calibration while maintaining competitive accuracy. Compared with TS, which only rescales confidence post hoc, EUA-Cal preserves sample-level uncertainty through regularization, thereby mitigating overconfidence more effectively. The corresponding results and analyses have been added to Appendix C.3 (Tables 15 and 16) of the revised paper.

Table R10: Comparison with temperature scaling on ResNet-50 (mean±std %).

CIFAR-10CIFAR-100Tiny-ImageNet
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
TS0.94±0.071.08±0.1894.57±0.154.26±0.314.14±0.3676.75±0.622.75±0.412.73±0.5764.45±0.45
Ours0.54±0.120.50±0.0994.63±0.051.28±0.111.18±0.2177.66±0.321.38±0.171.29±0.1165.31±0.47

Table R11: Comparison with temperature scaling on Llama3-8B (mean±std %).

ARC-CARC-ESciQ
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
TS6.71±0.457.68±0.5081.31±0.644.18±0.194.08±0.1192.81±0.123.80±0.323.84±0.2293.40±0.52
Ours3.60±0.344.17±0.6080.94±0.471.67±0.191.79±0.2992.72±0.192.19±0.242.57±0.4993.90±0.28

Question 3: Validation-set tuning and experimental fairness.

We appreciate the reviewer's concern. The epoch tt of the early checkpoint is predetermined based on validation ECE and accuracy curves, and fixed across three runs. We select α\alpha and β\beta from {0.1, 0.2, 0.3} through validation set tuning, while τ\tau is optimized by minimizing the training NLL of prototype predictions. For a fair comparison, all compared baselines are also tuned on validation sets, and follow the same training schedule. Detailed hyperparameter settings have been added to Appendix A.3 of the revised paper.

Question 4: LLM feature construction.

For PSR, we use the final layer hidden state of the last non-padding input token as the feature representation, without pooling across tokens.

Following IB-EDL [R1], we formulate multiple-choice question answering as single-token label prediction. We extract the next-token logits corresponding to the valid answer labels (A/B/C/D, or E when applicable) and apply softmax to obtain class probabilities. Multi-token answer choices are included in the input prompt, while predictions are made over single-token option labels, avoiding the need to aggregate probabilities across multiple tokens.

[R1] Li et al., Calibrating LLMs with Information-Theoretic Evidential Deep Learning, ICLR 2025.

Question 5: Accuracy-calibration trade-off.

We appreciate the reviewer's feedback. We admit that our previous statement was not sufficiently precise and have revised it in Section 4.2 (p. 7) and Section 5 (p. 10) accordingly. In our distribution-shift experiments, ARC-C and SciQ show accuracy decreases, while EUA-Cal achieves the lowest AECE on all three datasets. In most settings, EUA-Cal maintains competitive accuracy. We also acknowledge that the accuracy-calibration trade-off should be considered in practical applications.

To provide a more comprehensive evaluation, we further report NLL, Brier Score, and selective prediction performance (AURC) in Table R12 and Table R13. EUA-Cal achieves the lowest NLL and Brier Score across all six datasets, confirming its effectiveness beyond ECE evaluation. For selective prediction, EUA-Cal achieves the best AURC on all three image datasets and the second best on ARC-E and SciQ. The corresponding results and analyses have been added to Appendix C.2 (Tables 13 and 14) of the revised paper. We have also added reliability diagrams to Appendix B.2 (Figure 7) of the revised paper.

Table R12: Comparison of NLL (×10), Brier Score (%), and AURC (×10310^3) on ResNet-50. Best results are bolded and second-best results are underlined.

MethodCIFAR-10CIFAR-100Tiny-ImageNet
NLL↓Brier↓AURC↓NLL↓Brier↓AURC↓NLL↓Brier↓AURC↓
Vanilla2.08±0.118.45±0.286.27±0.339.64±0.2034.30±0.7965.06±2.32‾\underline{65.06\pm2.32}13.89±0.20‾\underline{13.89\pm0.20}45.75±0.69‾\underline{45.75\pm0.69}133.27±3.70
MbLS2.30±0.188.73±0.226.99±0.889.68±0.1933.99±0.6765.32±1.9314.48±0.1647.23±0.57137.76±3.53
BalCAL2.09±0.068.83±0.246.80±0.359.53±0.1532.82±0.53‾\underline{32.82\pm0.53}65.40±1.8714.54±0.1447.32±0.34131.98±2.67‾\underline{131.98\pm2.67}
PLP1.92±0.048.93±0.235.90±0.389.29±0.41‾\underline{9.29\pm0.41}35.60±0.9075.49±4.9816.57±0.8251.46±1.31162.67±12.62
MaC-Cal1.76±0.02‾\underline{1.76\pm0.02}8.06±0.04‾\underline{8.06\pm0.04}5.75±0.04‾\underline{5.75\pm0.04}9.30±0.0533.73±0.1469.18±0.1116.89±0.0652.25±0.10159.11±0.81
Ours1.60±0.038.00±0.135.19±0.157.66±0.0530.79±0.0759.08±0.4213.74±0.1045.50±0.17127.66±3.19

Table R13: Comparison of NLL (×10), Brier Score (%), and AURC (×10310^3) on Llama3-8B. Best results are bolded and second-best results are underlined.

MethodARC-CARC-ESciQ
NLL↓Brier↓AURC↓NLL↓Brier↓AURC↓NLL↓Brier↓AURC↓
Vanilla11.50±0.2534.80±0.3364.72±1.67‾\underline{64.72\pm1.67}4.40±0.0313.25±0.1412.22±0.175.03±0.1112.82±0.638.83±0.09
MbLS13.03±1.9736.38±3.2966.46±2.745.50±0.3115.40±0.5818.27±2.115.32±0.0613.56±0.3212.04±0.26
CSAM10.43±0.2432.97±0.3355.95±1.484.13±0.5812.24±1.84‾\underline{12.24\pm1.84}10.68±0.762.64±0.03‾\underline{2.64\pm0.03}11.04±0.12‾\underline{11.04\pm0.12}6.26±0.10
EDL7.14±0.1033.64±0.4670.72±4.253.15±0.0513.29±0.1620.32±1.942.98±0.0612.12±0.2713.27±2.64
IB-EDL6.47±0.08‾\underline{6.47\pm0.08}31.30±0.48‾\underline{31.30\pm0.48}74.39±2.862.96±0.03‾\underline{2.96\pm0.03}12.67±0.2013.85±0.472.93±0.0412.02±0.2713.55±1.41
Ours6.08±0.0828.36±0.3068.99±1.872.39±0.0511.65±0.3811.70±0.34‾\underline{11.70\pm0.34}2.08±0.0310.38±0.138.11±0.25‾\underline{8.11\pm0.25}

Question 6: Training procedure ambiguity.

After pre-learning to epoch tt, Algorithm 1 restarts training rather than continuing from the early checkpoint. For DNNs, after pre-learning for predetermined tt epochs, we freeze the early model and restart training from initialization for the full training schedule, e.g., 200 epochs for ResNet-50/101. For LLMs, we use either the pretrained model or a model obtained after 5-15 fine-tuning steps as the frozen early model. For a fair comparison, all models are trained with the same training schedule and optimization settings. EUA-Cal requires an additional tt-epoch pre-learning stage, but the early model is prepared only once and shared across three runs. EUA-Cal incurs modest training overhead and no additional inference overhead. We have revised Algorithm 1 to show the restart and added the anchor-selection and feature-extraction details to Appendix A.3.

We have clarified the training procedure in Algorithm 1 (p. 6) and the implementation details in Appendices A.2-A.3 (pp. 14-15), and added the early-model and temperature-scaling comparisons in Appendix C.3 (Tables 15 and 16, p. 21). We have also added reliability diagrams in Appendix B.2 (Figure 7, p. 16) and results and analysis for NLL, Brier Score, and AURC in Appendix C.2 (Tables 13 and 14, p. 21). The corresponding revisions are highlighted in red.

Reviewer_DkzTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose using the early more calibrated signal as regularization to help the calibration of the model.

Critical strengths
  1. The motivation and design is straightforward.

  2. The results looks promising.

Critical weaknesses
  1. No generative setup, mainly in classification setting.

  2. No distinguish preserved uncertainty information from the regularizing effect of softer targets. In Table 5, the late and early changes both the uncertainty strength and the sample wise uncertainty difference, a more detailed analysis is needed.

  3. Some exp details are missing.

  4. The epoch is selected on validation sets, but do not give a precise search grid or an explicit rule for trading off ECE against accuracy.

  5. Hyperparameters selection protocol is not specified.

Questions for authors

NAN.

10 comments

Thanks for your time and effort in handling this paper. We appreciate your recognition of our motivation, design, and promising results. We respond to your comments point by point.

Weakness 1: Generative setup.

We appreciate the reviewer's concern. To address this limitation, we conduct new experiments in generative settings. In Table R5, we fine-tune Qwen2.5-1.5B-Instruct and Llama3-8B-Instruct on GSM8K. For evaluation, we define the confidence of each sample as the average probability of all generated tokens, and compute ACC, ECE, and Brier Score. Since PSR is designed for classification tasks, we apply the EPR variant in these experiments. Our method performs better than the baseline, demonstrating the potential for generative tasks. The corresponding results and analyses have been added to Appendix C.4 (Table 17) of the revised paper.

Table R5: Generative calibration results on GSM8K.

ModelMethodACC↑ECE↓Brier↓
Qwen2.5-1.5BVanilla61.1125.1529.09
Qwen2.5-1.5BOurs62.5423.7828.21
Llama3-8BVanilla69.0714.6722.86
Llama3-8BOurs71.7211.2321.10

Weakness 2: Distinguishing early uncertainty from soft target regularization.

We appreciate the reviewer's suggestion. To further distinguish preserved uncertainty information from the regularizing effect of softer targets, we conduct a more detailed ablation study. We add softened late and the shuffled early models as anchor models, where the softened late model softens the late model's predictions through temperature scaling, and the shuffled early model randomly shuffles early features across samples to disrupt their correspondence.

Table R6: Ablation study of different anchor models on ResNet-50 and Llama3-8B.

EPRPSRAnchor ModelCIFAR-10CIFAR-100Tiny-ImageNetARC-CARC-ESciQ
ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑
√√Late2.1494.786.2478.113.6366.159.7180.126.6492.855.1194.30
√√Softened Late2.4694.763.7077.791.8965.778.3879.354.9391.163.5693.20
√√Shuffled Early1.3494.503.4075.806.8464.675.5879.354.5490.493.1894.10
√√Early0.5494.631.2877.661.3865.313.6080.941.6792.722.1993.90

As shown in Table R6, the variant with the softened late model improves calibration over the late model on most datasets, but still underperforms the variant with the early model. Moreover, the variant with the shuffled early model degrades calibration on most datasets by disrupting their uncertainty information, highlighting the importance of uncertainty information preserved in the early model. These results suggest that the regularizing effect of softer targets without early uncertainty information is insufficient, and our EUA-Cal method specifically leverages the uncertainty information preserved in the early model for calibration. The corresponding results have been added to Table 5 of the revised paper.

Weakness 3: More experimental details.

Thanks for your suggestion. We have clarified the experimental setup in Algorithm 1 and Appendices A.2 and A.3 of the revised paper.

Early model selection: The early epoch is predetermined based on validation ECE and accuracy curves. We select an early checkpoint before validation ECE begins to increase consistently, when further training yields only marginal accuracy gains. These early models are fixed across all three runs for each model and dataset. For more details, please see the response to Weakness 4.

Training Procedure: For DNNs, after pre-learning for predetermined tt epochs, we freeze the early model and restart training from initialization for the full training schedule, e.g., 200 epochs for ResNet-50/101. For LLMs, we use either the pretrained model or a model obtained after 5-15 fine-tuning steps as the frozen early model. All compared models use the same training schedule to ensure fairness. EUA-Cal needs an extra tt epochs for early model setup, with the early model fixed across three runs, with modest training overhead and no additional inference cost.

Hyperparameter selection: We select α\alpha and β\beta from {0.1, 0.2, 0.3} via validation set tuning. τ\tau is determined by minimizing the NLL of prototype predictions on the training set. All compared methods are also tuned on the validation set to ensure a fair comparison. For more details, please see the response to Weakness 5.

Feature extractor: For DNNs, we extract the feature representations from the penultimate layer of the early model, before the final classification layer. For LLMs, we extract the final layer hidden state of the last non-padding input token as the feature representation, without pooling across tokens.

Class probability computation: Following IB-EDL [R1], we formulate multiple-choice question answering as single-token label prediction. We extract the next-token logits corresponding to the valid answer labels (A/B/C/D, or E when applicable) and apply softmax to obtain class probabilities.

If you have any further questions about the experimental details, we would be glad to clarify.

Weakness 4: How to select the epoch on validation sets.

We appreciate the reviewer's concern. The early epoch is predetermined based on validation ECE and accuracy curves. Specifically, we select an early checkpoint before validation ECE begins to increase consistently, when further training yields only marginal accuracy gains.

For DNNs, ECE typically increases after the first learning rate decay, so we select epoch 60 for CIFAR-10/100 and epoch 40 for Tiny-ImageNet. For ViTs, ECE increases noticeably after approximately 5 epochs, so we select epoch 5. For LLMs, pretrained models generally exhibit lower ECE, with ECE increasing after 5-15 fine-tuning steps. Thus, we use either the pretrained model or a checkpoint at step 5 to 15. These checkpoints are fixed across all three runs for each model and dataset. Details of epoch selection are reported in Table R7. We have added these selection details to Appendix A.3 of the revised paper.

Weakness 5: Hyperparameter selection protocol.

We appreciate the reviewer's concern. For EUA-Cal, we select α\alpha and β\beta from {0.1, 0.2, 0.3} via validation set tuning. All compared methods are also tuned on the validation set to ensure a fair comparison. We determine τ\tau by minimizing the NLL of prototype predictions on the training set, i.e.,

τ∗=arg⁡min⁡τ>0[−1N∑i=1Nlog⁡qi,yi(τ)],(xi,yi)∈Dtrain,\tau^*=\arg\min_{\tau>0}\left[-\frac{1}{N}\sum_{i=1}^{N}\log q_{i,y_i}(\tau)\right],\quad (x_i,y_i)\in\mathcal{D}_{\mathrm{train}},

where NN is the number of training samples, qi,yi(τ)q_{i,y_i}(\tau) is the prototype prediction (Eq. 4 of the paper). Table R7 summarizes the selected regularization weights and early checkpoints for each model and dataset. We have added the detailed hyperparameter selection protocol to Appendix A.3 of the revised paper.

Table R7: Hyperparameter settings of EUA-Cal across different models and datasets.

ModelDatasetαβt (epoch/step)
ResNet-50CIFAR-100.30.260
CIFAR-1000.30.360
Tiny-ImageNet0.10.240
ResNet-101CIFAR-100.30.260
CIFAR-1000.30.360
Tiny-ImageNet0.10.340
Wide-ResNet-26-10CIFAR-100.30.260
CIFAR-1000.30.260
DenseNet-121CIFAR-100.30.160
CIFAR-1000.30.160
ViT-B/32CIFAR-100.20.25
CIFAR-1000.30.25
Llama3-8BARC-C0.30.2Pretrained
ARC-E0.20.2Pretrained
SciQ0.10.2Pretrained
Llama3.1-8BARC-C0.20.25
ARC-E0.30.210
SciQ0.20.15
Qwen3-8B-BaseARC-C0.20.15
ARC-E0.20.115
SciQ0.20.115

We have added the generative results and analysis in Appendix C.4 (Table 17, p. 21), expanded the anchor-model ablation in Section 4.4 (Table 5, p. 9), and clarified the implementation and hyperparameter details in Algorithm 1 (p. 6) and Appendices A.2-A.3 (pp. 14-15, including Table 6). The corresponding revisions are highlighted in red.

Reviewer_H6hUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes EUA-Cal, a calibration method that mitigates overconfidence by leveraging uncertainty information preserved during the early stages of training. The authors observe that, as training progresses, calibration performance deteriorates despite marginal improvements in accuracy. To address this issue, they introduce EPR and PSR. Experiments across various image classification models and LLMs demonstrate improved calibration performance compared to existing methods.

Critical strengths
  1. Extensive experiments across diverse models and datasets The authors conduct experiments on various architectures, demonstrating improvements in ECE and AECE across multiple datasets.
Critical weaknesses
  1. The proposed EPR is closely related to conventional knowledge distillation, as it aligns the current model's predictions with those of an early model using KL divergence. Although the paper includes a distillation baseline, it does not provide a controlled comparison with knowledge distillation using the same early checkpoint as the teacher. Therefore, it remains unclear to what extent the improvements can be attributed to the proposed regularization framework rather than the use of a well-calibrated early model as a teacher.

  2. The central observation of this paper is that early-stage models are better calibrated than later-stage models. Although the paper analyzes calibration performance at different training stages, the main experimental results do not provide a direct comparison with simply using the selected early checkpoint through early stopping under the same evaluation protocol. Furthermore, a direct comparison with the widely used temperature scaling method is missing. These comparisons are important for assessing whether the proposed method provides substantial benefits over simpler calibration strategies.

Questions for authors
  1. What are the key methodological differences between EUA-Cal and knowledge distillation using the same early checkpoint as the teacher?

  2. What advantages does EUA-Cal offer over directly using the selected early model or applying temperature scaling?

  3. How can the authors further verify that PSR specifically leverages uncertainty information preserved in the early feature space? Although the ablation study demonstrates the effectiveness of PSR, it remains unclear whether the improvement comes specifically from early-stage uncertainty information or from prototype-based regularization more generally.

8 comments

Thanks for your time and effort in handling this paper. We appreciate your recognition of our extensive experiments and consistent calibration improvements. We respond to your comments point by point.

Weakness 1 and Question 1: Key methodological differences between EUA-Cal and knowledge distillation using the same early checkpoint as the teacher.

We appreciate the reviewer’s question. EUA-Cal differs from knowledge distillation (KD) with an early teacher in its motivation and its additional use of early feature structures. EUA-Cal consists of two complementary components: Early Prediction Regularization (EPR) and Prototype Structure Regularization (PSR). Only EPR is similar to KD with an early teacher.

Although EPR is similar to KD using the same early checkpoint, their motivations differ. Our key insight is that the early model preserves uncertainty awareness in both the predictions and the feature structure. However, this uncertainty is gradually lost as training continues, resulting in overconfidence. To the best of our knowledge, we are the first to explicitly investigate this phenomenon from the perspective of uncertainty preservation in early predictions and feature structures. Motivated by this insight, we propose EUA-Cal to preserve the early uncertainty awareness for calibration, rather than simply transferring teacher predictions.

Specifically, EPR preserves the uncertainty awareness in early predictions by KL divergence, which is closely related to KD using the same early checkpoint. However, EUA-Cal further introduces PSR to derive additional supervision from uncertainty awareness reflected in the early feature structure. Thus, EUA-Cal jointly leverages predictive and feature structural uncertainty awareness of the early model to mitigate overconfidence, rather than relying solely on the output distributions of the early teacher.

Table R1: Comparison with knowledge distillation using the same early checkpoint as the teacher.

VariantCIFAR-10CIFAR-100Tiny-ImageNetARC-CARC-ESciQ
ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑
KD (early teacher)0.7594.693.3077.572.5065.487.4781.102.6791.254.2193.30
EUA-Cal0.5494.631.2877.661.3865.313.6080.941.6792.722.1993.90

We further compare EUA-Cal with knowledge distillation (KD) using the same early checkpoint as the teacher in Table R1. Compared to KD with an early teacher, our EUA-Cal achieved better calibration performance, reducing ECE from 0.75 to 0.54 on CIFAR-10, 3.30 to 1.28 on CIFAR-100, and 7.47 to 3.60 on ARC-C, and so on. These results demonstrate that the improvements of EUA-Cal are not solely from the predictions of a well-calibrated early model, but also benefit from the complementary uncertainty information provided by PSR.

Weakness 2 and Question 2: Comparison with simpler calibration strategies.

Thanks for the suggestion. Table R2 and Table R3 show the comparison of EUA-Cal with the selected early model and temperature scaling on ResNet-50 and Llama3-8B, respectively. Our EUA-Cal method achieves superior performance in calibration against simpler calibration strategies across all six datasets, while maintaining competitive accuracy. Compared with the early model, EUA-Cal benefits from continued training while preserving early uncertainty awareness. Compared with TS, which only rescales confidence post hoc, EUA-Cal preserves sample-level uncertainty through regularization, thereby mitigating overconfidence more effectively. The corresponding results and analyses have been added to Appendix C.3 (Tables 15 and 16) of the revised paper.

Table R2: Comparison with early model and temperature scaling on ResNet-50 (mean±std %).

CIFAR-10CIFAR-100Tiny-ImageNet
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Early Model1.84±0.441.79±0.4991.30±0.645.93±0.315.86±0.3772.34±0.531.91±0.061.95±0.1155.31±0.22
TS0.94±0.071.08±0.1894.57±0.154.26±0.314.14±0.3676.75±0.622.75±0.412.73±0.5764.45±0.45
Ours0.54±0.120.50±0.0994.63±0.051.28±0.111.18±0.2177.66±0.321.38±0.171.29±0.1165.31±0.47

Table R3: Comparison with early model and temperature scaling on Llama3-8B (mean±std %). The early model is the same pretrained Llama3-8B, yielding zero std.

ARC-CARC-ESciQ
ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑ECE↓AECE↓ACC↑
Early Model4.37±04.67±075.68±06.89±06.38±089.06±05.82±04.75±091.00±0
TS6.71±0.457.68±0.5081.31±0.644.18±0.194.08±0.1192.81±0.123.80±0.323.84±0.2293.40±0.52
Ours3.60±0.344.17±0.6080.94±0.471.67±0.191.79±0.2992.72±0.192.19±0.242.57±0.4993.90±0.28

Question 3: How to further verify that PSR specifically leverages uncertainty information preserved in the early feature space.

To verify that PSR specifically leverages uncertainty information preserved in the early feature space, we conduct a more detailed ablation study. Specifically, we compare PSR variants with different anchor models, including late, early, softened late, and shuffled early models. The softened late model softens the late model's predictions through temperature scaling, matching the average prediction entropy of the early model. The shuffled early model randomly shuffles early features across samples, disrupting their correspondence.

Table R4: Ablation study of PSR with different anchor models on ResNet-50 and Llama3-8B.

VariantCIFAR-10CIFAR-100Tiny-ImageNetARC-CARC-ESciQ
ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑ECE↓ACC↑
PSR + Late3.6194.4410.4175.365.3564.4610.5782.427.2492.346.2893.30
PSR + Softened Late2.4694.763.7077.792.6965.778.3879.354.9391.163.5693.20
PSR + Shuffled Early2.0894.662.4976.632.2864.6516.7680.128.7390.826.0493.60
PSR + Early0.8394.672.2377.372.6264.417.7981.074.6791.332.4493.10

As shown in Table R4, the PSR variant with the softened late model improves calibration over the late model, but still underperforms the variant with the early model. Moreover, the PSR variant with the shuffled early model degrades calibration on most datasets by disrupting their uncertainty information, highlighting the importance of uncertainty information in the early feature space. These results suggest that prototype-based regularization without early uncertainty information is insufficient, and PSR specifically leverages the uncertainty information preserved in the early feature space for calibration.

We have clarified the role of early feature uncertainty in the Abstract and Introduction (pp. 1-2), expanded the anchor-model ablation in Section 4.4 (Table 5, p. 9), and added the early-model and temperature-scaling comparisons in Appendix C.3 (Tables 15 and 16, p. 21). The corresponding revisions are highlighted in red.

Dear Reviewer H6hU,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 3 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revisions Summary

Dear Reviewers, Area Chairs, and Program Chairs,

We appreciate all the reviewers' constructive feedback. We summarize our contributions, the manuscript revisions, and responses to key reviewer questions.

1. Our Contributions

Our key insight is that the early model preserves uncertainty awareness in both the predictions and the feature structure. However, this uncertainty is gradually lost as training continues, resulting in overconfidence.

Motivated by this insight, we propose EUA-Cal to preserve early uncertainty through two regularizations: Early Prediction Regularization (EPR) and Prototype Structure Regularization (PSR). EUA-Cal jointly leverages predictive and feature structural uncertainty awareness of the early model to mitigate overconfidence. Extensive experiments on eight models and six datasets demonstrate that EUA-Cal outperforms state-of-the-art methods.

Reviewer uW54 recognized the usefulness of our observation, Reviewers DkzT and EBbG acknowledged our motivation and methodological design, and Reviewer 6A5d highlighted the practicality of our method. Moreover, all five reviewers acknowledged our extensive experiments or promising results.

2. Revisions to the manuscript

The following revisions are highlighted in red in the revised manuscript:

  • Methodological clarification (Abstract and Introduction): We clarified the role of early feature structures in PSR.

  • Extended ablation studies (Section 4.4): We added ablation of softened late and shuffled early anchors.

  • Additional baselines (Appendix C.3): We compared EUA-Cal with early stopping, post-hoc calibration, and training-time regularization baselines.

  • Additional evaluation (Appendices B.2 and C.2): We added NLL, Brier Score, AURC, and reliability diagrams.

  • Extended applications (Appendix C.4): We added generative and RLVR experiments.

  • Implementation and efficiency (Algorithm 1 and Appendices A.2, A.3 and D): We clarified training procedures, hyperparameter selection, LLM implementation, and computational overhead.

3. Responses to Key Reviewer Questions

  • How does EUA-Cal differ from KD? EUA-Cal differs from KD in its motivation to preserve early uncertainty for calibration and its additional use of early feature structures through PSR. EUA-Cal achieves lower ECE than KD using the same early checkpoint across all six datasets (Reviewer H6hU).

  • Do the improvements stem from early uncertainty rather than softer targets or better feature quality? Our additional ablations show that the variants with softened late model and shuffled early model yield inferior calibration. Moreover, late models achieve higher prototype classification accuracy but worse prototype calibration, while early models provide more effective supervision for calibration. These results support the importance of preserved early uncertainty (Reviewers H6hU and DkzT).

  • How does EUA-Cal compare with simpler calibration methods? EUA-Cal achieves the lowest ECE and AECE across all six datasets while maintaining competitive accuracy (Reviewers H6hU, EBbG, uW54, and 6A5d).

  • Do the improvements extend beyond ECE? EUA-Cal achieves the lowest NLL and Brier Score, further supporting its effectiveness (Reviewers EBbG, uW54, and 6A5d).

  • Can EUA-Cal extend to generative tasks and RLVR? Additional experiments demonstrate its potential in both settings. On GSM8K, EUA-Cal improves calibration across different confidence estimators (Reviewers DkzT and 6A5d).

  • Are the comparisons fair and reproducible? No test data are used for any selection. All methods use the same validation set for tuning and follow the same final training schedule and optimization settings. We have clarified the training procedure, anchor selection, hyperparameter tuning, and implementation details in the revised manuscript (Reviewers DkzT, EBbG, uW54, and 6A5d).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.