acceptodds
Under review as a conference paper at ICLR 2027

Calibrating LLMs by Reducing Overconfidence via Instance-Adaptive Smoothing

Abstract

Large language models (LLMs) play a growing role in real world decisions, emphasizing a need to ensure accurate reflection of their uncertainty to minimize risk. While the uncertainty calibration of neural networks precedes LLMs, the uniqueness of the setting leads to observations that cannot directly be addressed through existing methods, such as the emergence of severe overconfidence past the base LLM pre-training phase. While label smoothing (LS) have been suggested as a possible remedy due to its ability to mitigate overconfidence, approaches targeted to traditional machine learning settings can be ill-suited to the demands of LLMs, for example multiple training and alignment stages. We identify why this may fail to consider specific nuances unique to LLMs, in particular how the decomposition of the traditional label smoothing loss fails to eliminate overconfidence during misclassifications, leading to possible degradations in the generative abilities of models that are essential for further alignment and tuning. To this end, we introduce _**H**olistic **S**moothing **Hi**nges_ (___HoSHi___), an instance-adaptive smoothing approach during supervised fine-tuning to enable calibration while retaining strong generative capability. ___HoSHi___ acts as an adjustment to the learning objective that can mitigate previous concerns and enable the stable training of language models that remain calibrated at each stage. Results after different stages demonstrates the effectiveness of our approach, highlighting possible avenues towards better uncertainty-aware LLMs that are more interpretable and effective for public use.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language models, uncertainty calibration, instruction tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GKdVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the logits-based confidence calibration of LLMs across the post-training pipeline, in which a model is first supervised fine-tuned (SFT) and then further trained with DPO and RLVR. Label smoothing (LS) during SFT is a well-known, cheap remedy for the overconfidence that SFT introduces. However, the authors observe that LS-tuned models fail to improve during subsequent DPO and RLVR training. They attribute this to the LS regularizer lowering the ground-truth logit even for tokens the model mispredicts. To address this, they propose Holistic Smoothing Hinges (HoSHi), an SFT objective that is identical to LS when the target token is ranked first, and otherwise shifts the penalty from the target logit to the logits ranked above it, spread uniformly among them. They motivate the uniform penalty with a lower bound on the LM head’s output entropy under a bounded logit norm, arguing that probability mass among the wrongly top-ranked tokens is close to flat. Across three base models, two SFT datasets and three multiple-choice benchmarks, HoSHi matches LS calibration after SFT. On Llama3-8B further trained with DPO and RLVR, it remains trainable, stays better calibrated than the no-smoothing baseline, and preserves downstream performance on 12 benchmarks better than MaxSup.

Critical strengths
  1. Well-motivated and practically relevant question. The paper studies the logits-based confidence calibration across the full post-training pipeline (SFT → DPO → RLVR) and asks whether an SFT-time calibration fix interferes with later alignment. Since label smoothing is well known to be cheap and is recommended for calibrating instruction-tuned LLMs, the observation that it may impair subsequent DPO and RLVR training (Fig. 3) is surprising and directly relevant to practitioners.
  2. A simple, drop-in method with clear positioning. HoSHi uses the same single hyperparameter as LS, adds only O(K) overhead, and reduces exactly to LS whenever the target token is ranked first.
Critical weaknesses
  1. Most of the theoretical analysis restates prior results without attribution. Prop. 3.2 and Lemma 4.1 have already been proposed in Huang et al. (ICML 2025), and Thm 4.2 is exactly the same as Thm 4.2 in Huang et al. (ICML 2025); the appendix credits only Zhang & Sabuncu (2020) and Zhai et al. (2023) for proof techniques, which suggests the results themselves are new. Please explicitly attribute these results and indicate which parts of the analysis are new; as written, the theoretical contribution claimed in C1 and C2 largely reduces to the HoSHi objective itself.
  2. Several reported results coincide exactly with previously published numbers. For example, in Table 1, every ECE/RMS value reported for HoSHi is identical to the value reported for label smoothing (β = 0.1) in Huang et al. (ICML 2025, Table 1) under the same models, SFT datasets, and β, while the “Smoothing” row in this paper reports different and worse values. Separately from prior work, the HoSHi SFT results for the same Alpaca/Llama3-8B setting are inconsistent: Table 9 reports ECE 0.093, while Table 11 reports ECE 0.073, which matches the LS row in Table 9. Please clarify the source of these numbers and provide per-run results (ideally with seeds and training logs). Until then, Table 1 cannot be taken as evidence that HoSHi matches LS in calibration.
Questions for authors
  1. Provenance of Table 1. Could you clarify whether the HoSHi results in Table 1 were obtained from new training runs? Their ECE/RMS values coincide exactly with the label-smoothing (β = 0.1) values reported in Huang et al. (ICML 2025, Table 1), while the “Smoothing” row in this paper reports different values.
  2. Inconsistent HoSHi results for the same setting. For SFT on Alpaca with Llama3-8B, which HoSHi result is correct: Table 9 (ECE 0.093, RMS 0.092) or Table 11 (ECE 0.073, RMS 0.094)? If Table 9 is correct, does the claim in Appendix G.3 that HoSHi without tuning matches Reference + DACA still hold at SFT stage?
  3. Novelty of the theory. Which of Def. 3.1, Prop. 3.2, Prop. 3.3, Lemma 4.1, and Thm 4.2 do you regard as new relative to Huang et al. (2025), Liu et al. (2022), and Zhai et al. (2023)? How do Lemma 3.4, Cor. 3.5, and the error-amplification argument differ from the decomposition in MaxSup (Zhou et al., 2025)?
Reviewer_WdXwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that label smoothing (LS) during SFT improves calibration but impairs learning in later DPO/RLVR stages. The authors attribute this to how LS treats classes ranked above the target. They propose HoSHi, which uses the target logit as a hinge and spreads the smoothing penalty evenly over the over-ranked classes; it reduces to LS on correct predictions. Experiments cover SFT on three base models and DPO/RLVR on Llama3-8B, with calibration evaluated on MMLU, HellaSwag and ARC-Easy and performance on a downstream suite.

Critical strengths

The paper addresses a relevant question, calibration across multi-stage post-training rather than SFT alone. It proposes a simple, low-cost method and evaluates it broadly across models, training stages and downstream tasks.

Critical weaknesses

Large parts of the paper might closely follow Huang et al. (ICML 2025), including the background, Props. 3.2 and 3.3, Lemma 4.1 and Thm. 4.2, and Figs. 1 and 2. Many passages seems near-verbatim, yet that paper is cited only in passing. In Table 1, all 18 HoSHi ECE/RMS values seem quite close to the LS results reported there. What is new relative to that work, and were these experiments newly run? I would like the authors to further clarify.

Questions for authors

see weakness

Reviewer_CyKdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that label smoothing (LS) applied during LLM supervised fine-tuning improves calibration but impairs the model’s ability to learn in later DPO and RLVR stages. It attributes this to how LS treats logits ranked above the target on misclassified tokens. The proposed method, HOSHI, is an instance-adaptive variant of LS. On misclassified tokens it moves the smoothing pressure onto the Z logits ranked above the target, and on correctly predicted tokens it reduces to standard LS. The authors motivate the design with an entropy lower bound for the LM head. They evaluate SFT on Mistral-7B, Llama3-8B and Gemma2-2B, and run Llama3-8B through DPO and RLVR. After SFT, HOSHI’s calibration is comparable to LS and MaxSup, and after alignment its ECE is lower than the reference.

Critical strengths

The paper studies calibration across the full multi-stage post-training pipeline instead of a single stage. Its observation that an SFT-time calibration intervention can affect downstream trainability, shown through target-rank statistics and DPO/RLVR training curves, is practically relevant and has received little attention.

The method is simple, can be dropped into existing training code, and adds only O(K) overhead. The evaluation is broad: SFT and alignment stages, a wide downstream benchmark suite, combinations with post-hoc methods (TS, DACA), free-form generation, and image classification.

Critical weaknesses

The evidence for the central claim, that HOSHI preserves post-SFT trainability where LS does not, is thin and partly inconsistent. All alignment-stage results use only Llama3-8B. LS is missing from Figure 6 and Tables 2 and 4, so the LS-vs-HOSHI comparison runs across separate figures whose axis scales differ (150k vs. 2000k RLVR steps).

The stated mechanism does not match what the method actually changes. The paper attributes the failure of LS to a weakened corrective update on classes ranked above the target. Under LS, however, this extra gradient is β/K per class (about 8e-7 for a 128k vocabulary), which is negligible at LLM scale. The substantive change HOSHI makes is to remove the target’s (1−β) discount on misclassified tokens and to add a β/Z push on over-ranked classes. Consistent with this, the region ablation (Table 6) suggests that most of the calibration gain comes from the LS component on correctly predicted tokens, where HOSHI is identical to LS. The theoretical support is also limited: Theorem 4.2 is close to vacuous for realistic norms, and it bounds the entropy of the full distribution rather than the misclassified-region entropy that the design relies on (Figure 5).

Questions for authors

Does the limitation section exceed the page limit for ICLR submissions?

Q1. Can you report LS through DPO and RLVR under exactly the same setup as Figure 6 and Table 2, ideally on a second model? Please also report the fraction of misclassified response tokens in the SFT data and the distribution of Z, since HOSHI is identical to LS on all other tokens.

Q2. Can you add an ablation that applies LS on correctly predicted tokens and plain CE on misclassified tokens? This would separate the effect of removing the target discount from the effect of the β/Z penalty. Please also give precise definitions of the two Table 6 variants and report results over multiple seeds.

Q3. Eq. 9 defines s using ℓt≤ℓiℓ_t ≤ ℓ_i, while Eq. 10 and Appendix A.7 use a strict inequality, which leaves Z=0 for correct predictions. Which definition was implemented? Please also explain the discrepancy between Table 11 and Table 9, and why LS outperforms HOSHI on ECE in Table 9.

Q4. For realistic values of ρ, D and K, is the bound in Theorem 4.2 non-vacuous? How does a lower bound on full-distribution entropy support the misclassified-region entropy shown in Figure 5, and which model and data were used for that figure?

Q5. All methods share β=0.1, but the same β gives each method a different regularization strength. If β is tuned per method to reach similar ECE, does HOSHI still show an advantage in accuracy and downstream trainability? In particular, does LS with a smaller β avoid the DPO/RLVR failure shown in Figure 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.