acceptodds
Under review as a conference paper at ICLR 2027

The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation

Abstract

On-policy distillation (OPD) is an increasingly important paradigm for post-training language models. However, we identify a pervasive Scaling Law of Miscalibration: while OPD effectively improves task accuracy, it systematically traps models in severe overconfidence. We trace this failure to an information mismatch: teacher supervision is formed under privileged context available during training, whereas the deployed model must report confidence using only deployment-time information. We formalize this perspective theoretically, showing that teacher-conditioned success is generally not a valid target for deployment-time confidence and that helpful privileged context induces entropy collapse and a systematic optimism bias. To address this, we propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts, replaces self-reported confidence with this student-grounded target, and distills the revised response through the same self-distillation pipeline. Experiments across various models and domains show that CaOPD achieves Pareto-optimal calibration while maintaining competitive capability, generalizing robustly under out-of-distribution and continual learning. Our findings highlight that capability distillation does not imply calibrated confidence, and that confidence should be treated as an essential objective in post-training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Calibration, Confidence, Uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AbWcOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3(+2)
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CaOPD, which aims to mitigate the miscalibration introduced by OPSD training. CaOPD decouples capability learning from confidence supervision by replacing the confidence signal with a student-grounded estimate of “true” confidence, computed from empirical Monte Carlo rollouts. This revised confidence target is then distilled alongside the original capability objective, allowing the model to preserve task performance while improving calibration. This work applies 4 baselines and two datasets to validate the effectiveness of the proposed CaOPD, along with a series solid ablations and analysis of the proposed algorithm.

Critical strengths

The problem discussed is common and important for recent widely-used OPSD.

The proposed CaOPD framework is simple and well motivated, and its effectiveness is supported by both theoretical analysis and extensive empirical results across models, domains, and training settings.

Critical weaknesses

The paper appears to overgeneralize a failure mode specific to privileged-context OPSD as a general limitation of OPD.

The theoretical analysis relies critically on the teacher having additional privileged information unavailable to the student, which is not a defining property of OPD itself. For example, OPD can be used as strong-to-weak models, or Multi-teacher OPD, the supervision signal can from multiple sources while they dont see additional information.

As a result, claims that OPD inherently induces overconfidence seem broader than what the theory and experiments directly support.

Questions for authors

whether the identified miscalibration is intended to apply to OPD broadly or specifically to privileged-context OPSD methods such as SDPO/SDFT? If it is for OPD, I believe another set of experiments which using stronger model as teacher should be discussed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AbWc

Thank you for the positive assessment and for the precise question; it identified the main weakness in our framing. Changes in the revised PDF are in blue.

Q: Does the miscalibration apply to OPD broadly, or to privileged-context OPSD such as SDPO/SDFT? Our claims and experiments concern privileged-context OPSD. We agree that the original framing overgeneralized, and we have revised the abstract, Sec. 1, 2.1 and 3 to state the scope as OPSD and to define OPD and OPSD explicitly. The title will become "... in On-Policy Self-Distillation" in the final version (title changes are not permitted during the discussion period).

What carries over to OPD with a stronger teacher or multiple teachers. Following your question, a new paragraph in Sec. 3.2 states what each result requires:

  • Props. 1 and 2 require privileged context (I(R;Z|X) > 0 and I(A;Z|X) > 0). As you point out, they do not apply to same-input strong-to-weak or multi-teacher OPD.
  • Prop. 3 only requires that the teacher succeed more often than the student. In strong-to-weak OPD the gap comes from capability rather than context: if a stronger teacher's confidence tracks its own success probability μ_T(x), imitating it biases the student upward by E[μ_T(X) − μ_S(X)] > 0. Even a perfectly calibrated teacher yields an overconfident student. In multi-teacher OPD, the same holds for each domain expert on its domain.
  • CaOPD's target μ̂(x) is computed from student rollouts only, so it does not depend on where the gap comes from and applies unchanged.

A stronger teacher could instead judge the student's own trajectory (acting partly as a verifier), so the size of this effect is an empirical question. We therefore state the strong-teacher case as an analytical extension, together with the corresponding limitation, rather than claim it empirically; testing it is a natural next step.

We hope the clarified scope addresses your concern. If so, we would be grateful if you would consider raising your score. Thank you again for the constructive review.

Thank you for the reply and the latest editing, my main concern about the scope of OPD vs. OPSD is addressed. I will raise my confidence and informative score accrodingly.

AuthorsOpenReview ↗

thanks for your response

Thank you for reading our response and the revision, and for raising your confidence and informativeness scores. We are glad that the OPD vs. OPSD scope concern is resolved. We will keep this scope throughout the final version, including changing the title to "... in On-Policy Self-Distillation". Thank you again for your constructive review.

Reviewer_xc9cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of overconfidence problem of LLMs. The authors claim that the modern post-training methodology, on-policy self-distillation (OPSD), is the root cause for the above problem. The authors analyze the impact of OPSD through detailed mathematical analysis. The the authors propose CaOPD, an improved method to jointly optimizing the capability and calibration of LLMs through OPSD. The authors conduct experiments on Sci QA and ToolUse tasks to show the effectiveness of the proposed method.

Critical strengths
  1. Either overconfidence issue of LLMs or OPSD is an important research problem.
Critical weaknesses
  1. The paper is generally difficult to follow, particularly for researchers working on on-policy distillation (OPD). I would first point out that OPD and OPSD are distinct settings. OPD refers to training a student to match the teacher’s distribution on the same input, and is commonly used for rapid capability learning and multi-expert capability merging. In contrast, this paper studies OPSD, in which the teacher is constructed using privileged information. The authors should clearly differentiate between these two settings. Second, the authors claim that OP(S)D is the modern post-training paradigm and the root cause of the overconfidence issue in current frontier LLMs (Lines 62–83). On this point, I would argue that the current post-training paradigm for frontier LLMs is built on Reinforcement Learning (RL) and OPD, not OPSD. Frontier LLMs are typically developed using RL and MOPD. OPSD, in my view, is not yet a mature or stable algorithm and still deserves more exploration. Given this, I do not see the logic behind this paper’s effort to improve OPSD so as to alleviate overconfidence in frontier LLMs. Therefore, I have major concerns on the paper's soundness.

  2. The experiments are not convincing (though I appreciate the ablations). The authors only include two toy tasks and only conduct experiments on 7B/8B models, which is not enough. The authors only take one naive baseline from OPSD paradigm (SDPO), and the empirical improvement seems marginal.

Questions for authors

See weakness above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xc9c

Thank you for the review. We agree with your two conceptual points (OPD vs. OPSD, and the frontier-model claim) and have revised the paper accordingly. We also clarify the experimental scope, part of which may not have been sufficiently visible in the original version. Changes in the revised PDF are in blue.

1a. OPD and OPSD are distinct settings. Agreed, and our paper studies OPSD. The revised Sec. 1 and 2.1 now define both: in OPD a separate teacher sees the same input as the student; in OPSD the teacher is the model itself conditioned on privileged information (SDFT, SDPO, and OPSD of Zhao et al.). The abstract, introduction, Sec. 2–3 and conclusion now state the scope as OPSD, and the title will read "... in On-Policy Self-Distillation" in the final version (ICLR does not permit title changes during the discussion period).

1b. Frontier LLMs and the "root cause" claim (L62–83). We agree that this passage overstated our claim, and we have removed it. We do not argue that frontier models' overconfidence originates from OPSD. Fig. 1 documents that verbalized overconfidence is widespread among current LLMs, which is a measurement rather than an attribution, and the revised text says so explicitly. The question we study is narrower: when a model is post-trained with OPSD, what happens to its confidence? We find that OPSD improves accuracy while saturating verbalized confidence, explain why (the teacher's confidence is a non-identifiable, upward-biased target), and fix it.

1c. Why improve OPSD? We agree that OPSD "still deserves more exploration"; this paper is such an exploration. OPSD is used for continual learning from demonstrations (SDFT), RL with rich feedback (SDPO), reasoning (Zhao et al.), context distillation (Ye et al.) and, more recently, agentic RL (e.g., arXiv:2608.14945). These methods share the privileged-teacher structure, so they share the confidence failure mode we identify, and CaOPD fixes it at no inference cost. The analysis also reaches beyond OPSD: Prop. 3 only requires the teacher to succeed more often than the student, so the same optimism bias arises in OPD with a stronger teacher (new paragraph in Sec. 3.2). Our experiments include RL as well: GRPO is also miscalibrated (Tool Use ECE 0.304 on Qwen3-8B, Table 3), and CaOPD is better calibrated than GRPO in all four model×domain settings and more accurate in three.

2a. "Two toy tasks." Science Q&A (SciKnowEval Chemistry, level L-3) and ToolAlpaca are the evaluation domains shared by the two OPSD methods we study: SDFT reports Science Q&A, Tool Use and Medical; SDPO reports scientific reasoning, tool use and LiveCodeBench. Using them ensures faithful reproductions of both baselines. Neither is saturated: base Qwen3-8B reaches 31.0% on Science Q&A and 64.7% on Tool Use. We additionally test OOD transfer and continual training across the two domains (Sec. 5.4).

2b. "Only 7B/8B models." Sec. 5.5 (Fig. 4) and App. E.4 evaluate the Qwen3 family at 0.6B, 1.7B, 4B, 8B, 14B and 32B. The main tables use two model families (Qwen3-8B, OLMo-3-7B-Instruct).

2c. "Only one naive OPSD baseline (SDPO)." We evaluate two OPSD methods, SDFT and SDPO (both from January 2026), each with and without CaOPD, plus GRPO and two calibration-aware RL methods, RLCR and CAR (Tables 2–3). Because CaOPD is a drop-in change to OPSD, the decisive comparison is the same OPSD backbone with and without CaOPD, which we report for both backbones.

2d. "The improvement seems marginal." Across the 8 settings of Tables 2–3 (2 OPSD backbones × 2 models × 2 domains), relative to the same OPSD baseline:

MetricMeanRange
ECE, relative reduction37%10% to 59%
Brier score, relative reduction34%24% to 46%
SPR, absolute change+0.16+0.07 to +0.47 (8/8 improve)
Accuracy, change in points+1.6−0.4 to +4.7

Against calibration-aware RL, CaOPD has the best Brier score and SPR in all four model×domain settings and is 7.0–18.7 points more accurate. Under continual training, ECE drops from 0.513 (SDFT) to 0.126 (App. Table 7). The smallest gain (Qwen3-8B Science Q&A with SDPO, ECE 0.157 → 0.141) occurs where SDPO is already comparatively well calibrated; even there, the Brier score falls by 24% and SPR rises by 0.10.

Presentation. We revised the introduction and Sec. 2.1 to define the setting explicitly.

We hope these revisions address your concerns about scope and soundness, and we would be glad to answer any further questions.

Dear Reviewer xc9c,

As the discussion period closes on November 18 (AoE), we wanted to check whether our response and the revised PDF address your concerns. In brief: (1) we agree that OPD and OPSD are distinct settings; the revision defines both and states the scope as OPSD, and Reviewer AbWc, who raised the same point, has confirmed that it is addressed; (2) we removed the passage that attributed the overconfidence of frontier LLMs to OPD; (3) the paper evaluates six Qwen3 scales from 0.6B to 32B (Sec. 5.5, Fig. 4) and compares against two OPSD methods (SDFT, SDPO), GRPO, RLCR and CAR (Tables 2–3).

We would be glad to clarify anything further before the deadline. Thank you for your time.

Reviewer_ZT1UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CaOPD to tackle the overconfidence problem introduced by OPD. The authors argue that the student becomes overconfident through training on teacher outputs, where teacher has access to privileged information. They provide a theoretical analysis of this information asymmetry and propose CaOPD. The method estimates the student's success probability using multiple on-policy rollouts and replaces the confidence targets in the distillation process with this empirical estimate, while retaining teacher supervision for reasoning tokens.

Critical strengths
  1. This paper proposes a simple solution to a practical and relevant problem.
  2. The empirical results are quite strong. The paper demonstrate substantial reductions in calibration error while maintaining competitive task accuracy.
Critical weaknesses

The biggest weakness of this paper is that CaOPD does not directly address the underlying overconfidence induced by on-policy distillation.

The paper attributes OPD-induced overconfidence to an information asymmetry between the teacher and student, arguing that privileged teacher conditioning leads to artificially sharp token distributions and entropy collapse. However, CaOPD primarily corrects the student's verbalized confidence rather than the predictive distributions underlying its reasoning process.

In particular, the student continues to imitate the privileged teacher's token distributions during reasoning. Replacing the confidence targets at the end of a trajectory does not necessarily correct the overly sharp predictions learned from the teacher. A student could learn to report well-calibrated confidence scores while its reasoning-token distributions remain overconfident.

This raises a fundamental concern about whether CaOPD addresses the root cause identified in the paper or merely corrects its manifestation in verbalized confidence. The reported ECE, Brier Score, and SPR improvements demonstrate better confidence reporting, but do not establish that the underlying predictive overconfidence or entropy collapse has been resolved. This distinction substantially limits the strength of the paper's central claims.

Questions for authors
  1. Does CaOPD change the entropy or calibration of the student's next-token distributions during reasoning, compared with standard OPD? Can the authors provide measurements that directly examine the proposed entropy-collapse mechanism?
  2. Given that CaOPD retains privileged-teacher supervision for reasoning tokens, what mechanism prevents the student from inheriting the teacher's overly sharp predictions? Is there evidence that confidence-target replacement indirectly corrects these distributions?
  3. Would it be more accurate to characterize CaOPD as a method for calibrating verbalized confidence, rather than addressing the underlying overconfidence induced by OPD?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZT1U

Thank you for the thoughtful review and for describing CaOPD as "a simple solution to a practical and relevant problem" with "quite strong" results. Your central question, whether CaOPD addresses the root cause or calibrates verbalized confidence, is well taken and has made the paper more precise. We answer Q3 first, then Q2 and Q1. Changes in the revised PDF are in blue.

Q3. Is CaOPD a method for calibrating verbalized confidence? Yes. CaOPD calibrates the model's verbalized confidence, its explicit single-pass estimate of its own probability of success, and it leaves OPSD's objective at reasoning positions unchanged by design. The revised abstract, Sec. 1, Sec. 4.2 and conclusion now say this explicitly, and we removed the phrases "addresses the root cause" and "directly resolving the entropy collapse". What CaOPD does correct at its source is the confidence target: OPSD supervises the confidence segment with the teacher's confidence, which cannot be recovered from x (Prop. 1) and is biased upward (Prop. 3). CaOPD replaces it with an unbiased estimate of μ(x), the optimal x-measurable target (Prop. 1); it does not rescale outputs post hoc.

Q2. What prevents the student from inheriting the teacher's sharp reasoning-token distributions? Nothing, intentionally.

(a) Sharpening toward the privileged teacher at reasoning positions is how OPSD transfers capability: SDPO reaches 80.6% on Qwen3-8B Science Q&A vs. 74.5% for GRPO (Table 3). Flattening it would reintroduce the capability tax of RL-based calibration (RLCR and CAR are 14.5–18.7 points below CaOPD in that setting).

(b) Low entropy at reasoning positions is not by itself miscalibration. Calibration concerns probability statements about correctness; a next-token distribution over a derivation is not such a statement, and a sharp distribution that reliably produces correct derivations is desirable.

(c) Our results indicate that reasoning-token sharpening is not what drives verbalized overconfidence. CaOPD and SDPO optimize essentially the same objective at reasoning positions (Eq. 6; the privileged reasoning content is unchanged), and their accuracy trajectories coincide (Fig. 2, left). Yet CaOPD's verbalized confidence is much better calibrated (Tool Use ECE 0.298 → 0.133) and more discriminative (SPR improves in all 8 settings of Tables 2–3). The dominant pathway is the direct imitation of the teacher's confidence target, which CaOPD removes.

We revised Sec. 3 accordingly (new paragraph in Sec. 3.2): Prop. 2 describes the sharpening induced by privileged conditioning; at reasoning positions this is the capability signal, while at confidence positions it becomes an explicit probability claim (a value near 1.0) that the deployed model cannot justify, which is the only place CaOPD intervenes.

Q1. Measurements of the entropy-collapse mechanism. At reasoning positions, CaOPD and SDPO optimize the same per-token objective, so CaOPD does not change reasoning-token entropy by design; their matching accuracy trajectories (Fig. 2, left) are consistent with this. At the confidence positions, where CaOPD intervenes, the paper reports the relevant measurements: OPSD drives mean verbalized confidence toward saturation (up to 0.996, Table 1) and collapses its distribution to a point mass at 1.0 (Fig. 6, right), and this saturation persists at every scale we evaluate (Fig. 4, right). Under CaOPD, single-pass confidence is dispersed and tracks accuracy (Table 1, Fig. 6), and SPR, which gives no credit to ties, rises from 0.085 to 0.555 (Qwen3-8B, Tool Use).

On token-level calibration. Where white-box access exists and calibrated token probabilities are needed, post-hoc temperature scaling (Guo et al., 2017) is a cheap fix that is orthogonal to and composes with CaOPD. Verbalized confidence has no comparable post-hoc fix: it is generated text, and once it saturates at 1.0, any monotone recalibration map leaves all ties in place, so discrimination cannot be recovered after training. This is why we target the verbalized channel, which is also the one available through APIs and to downstream agents in a single pass. We added this discussion to App. C.7 and the Limitations.

We hope this resolves the concern about the central claim, which we have narrowed accordingly. If it does, we would be grateful if you would consider revisiting your score, and we are happy to answer follow-up questions.

Dear Reviewer ZT1U,

As the discussion period closes on November 18 (AoE), we wanted to check whether our response and the revised PDF address your concern. In brief: we agree that CaOPD calibrates verbalized confidence (your Q3), and the revised Sec. 1, Sec. 3.2, Sec. 4.2 and conclusion now make this scope explicit. CaOPD keeps OPSD's objective at reasoning positions unchanged and replaces only the confidence target, which Props. 1 and 3 show is non-identifiable and biased upward. Because CaOPD and SDPO share the reasoning-position objective, yet only CaOPD's verbalized confidence is calibrated (e.g., Tool Use ECE 0.298 → 0.133; SPR improves in all 8 settings of Tables 2–3), our results indicate that this target, rather than reasoning-token sharpening, is the main driver of verbalized overconfidence in OPSD.

We would be glad to clarify anything further before the deadline. Thank you for your time.

Thank you to the authors for the detailed response and for updating the manuscript.

I appreciate the clarifications regarding the scope of CaOPD; specifically, that the method explicitly targets verbalized confidence rather than altering the next-token entropy or predictive distributions during reasoning. The revised framing in Sections 1, 3.2, and 4.2 makes the paper much clearer and avoids overstating what the technique achieves.

However, my core concern remains: because CaOPD leaves the student's objective at reasoning positions unchanged, the underlying model continues to imitate the sharp predictions learned from the privileged teacher. While replacing the confidence target effectively calibrates the verbalized output channel, it acts as a secondary layer rather than resolving the predictive overconfidence inherent in the distillation process itself.

While the empirical gains on verbalized confidence calibration are solid, I believe this distinction limits the theoretical impact of the paper regarding how on-policy distillation affects token-level uncertainty. Therefore, I would like to maintain my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. We are encouraged that reviewers found the problem "practical and relevant" with "quite strong" empirical results (ZT1U), the method "simple and well motivated" with "extensive empirical results across models, domains, and training settings" and "solid ablations" (AbWc), and the problem important (xc9c).

Two concerns were shared across reviews. We agree with both and have revised the paper; changes in the revised PDF are highlighted in blue.

1. Scope: privileged-context on-policy self-distillation (OPSD), not OPD in general (AbWc, xc9c). All of our experiments, and the core of our analysis, concern OPSD, where the teacher is the model itself conditioned on information unavailable at deployment (demonstrations in SDFT; feedback or successful rollouts in SDPO). The original framing overgeneralized to OPD. We have (i) defined OPD and OPSD explicitly (Sec. 1, 2.1); (ii) stated the scope as OPSD in the abstract, introduction, Sec. 3 and conclusion; and (iii) removed the passage attributing the overconfidence of frontier models to OPD. Fig. 1 documents that phenomenon but does not attribute it to any training stage. In the final version, we will harmonize the remaining terminology and change the title to "... in On-Policy Self-Distillation" (ICLR does not permit title changes during the discussion period).

A new paragraph in Sec. 3.2 ("What the analysis covers") states which results require privileged context. Props. 1 and 2 do. Prop. 3 only requires that the teacher succeed more often than the student, so its optimism argument extends to OPD with a stronger teacher: even a perfectly calibrated stronger teacher yields an overconfident student. CaOPD's target uses student rollouts only and applies unchanged. We present this as an analytical extension, not an empirical claim.

2. What CaOPD calibrates (ZT1U). CaOPD calibrates the model's verbalized confidence, its explicit single-pass estimate of P(correct | x), and leaves OPSD's objective at reasoning positions unchanged by design. The revised abstract, Sec. 1, Sec. 4.2, App. C.7 and Limitations state this explicitly, and we removed the wording suggesting that CaOPD resolves entropy collapse. Our results indicate that the dominant source of verbalized overconfidence in OPSD is the confidence target itself (Props. 1 and 3), which CaOPD replaces: CaOPD and SDPO optimize essentially the same objective at reasoning positions (Eq. 6), yet CaOPD's verbalized confidence is substantially better calibrated and more discriminative (e.g., Tool Use ECE 0.298 → 0.133 on Qwen3-8B).

Experimental scope (xc9c). The paper evaluates two model families (Qwen3, OLMo-3), six Qwen3 scales from 0.6B to 32B (Sec. 5.5, Fig. 4, App. E.4), two OPSD methods (SDFT, SDPO), GRPO, and two calibration-aware RL methods (RLCR, CAR), plus OOD transfer and continual training. Averaged over the 8 backbone × model × domain settings in Tables 2–3, CaOPD reduces ECE by 37% and Brier score by 34% relative to the OPSD baseline, improves SPR in all 8 settings, and changes accuracy by +1.6 points on average (at most −0.4 in any setting).

Summary of revisions (blue in the PDF):

  • Scope and terminology (OPD vs. OPSD): abstract, Sec. 1, 2.1, 3, 5.2 heading, conclusion; the frontier-model attribution in Sec. 1 is removed.
  • New paragraph in Sec. 3.2: which results need privileged context, the role of Prop. 2, and the extension to a stronger teacher.
  • What CaOPD calibrates: abstract, Sec. 1, Sec. 4.2, App. B.2, App. C.7, Limitations.

We respond to each reviewer individually below and welcome further questions before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.