acceptodds
Desk rejected at ICLR 2027

Learning from What the Teacher Trusts: Confidence-Aligned On-Policy Distillation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation is a promising paradigm for aligning Large Language Models. However, standard objectives compel the student to indiscriminately mimic the teacher, forcing the model to learn from low-agreement or noisy trajectories generated during exploration. While recent approaches attempt to mitigate this via external reward models or step-wise teacher intervention, they introduce significant computational and communication overhead. In this work, we propose Teacher-Confidence Aligned On-Policy Distillation (TCAD), a scalable, rewardfree paradigm that leverages the teacher’s intrinsic sequence-level confidence as a relative ranking signal among rollouts for the same prompt. By dynamically reweighting trajectories based on model-relative teacher certainty, TCAD softly down-weights low-agreement rollouts without requiring auxiliary reward models. To address the approximation error caused by vocabulary truncation in large-scale distillation, we introduce support-symmetric candidate construction and probability-proportional Monte Carlo optimization over a compact event space. These mechanisms reduce the distribution mismatch between teacher and student supports while optimizing a compact support-plus-tail divergence with a controllable Monte Carlo sample-count trade-off. Extensive experiments across diverse benchmarks demonstrate that our proposed method consistently improves over standard on-policy distillation baselines, achieving strong alignment efficiency without external supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-policy Distillation, Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZuPLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TCAD uses relative, length-normalized teacher likelihood to weight student-generated trajectories during distillation. It estimates forward KL, reverse KL and JSD on a compact union of teacher and student token supports, with a residual tail accounting for omitted probability mass. An analysis describes the compact objective’s tail approximation and separates loss-value unbiasedness from gradient correctness. Ablations examine trajectory weighting and sample counts. Experiments use within-family teacher/student pairs from Qwen dense and DeepSeek MoE, with three independent training seeds and checkpoints fixed at 500 updates. For FKL, the four-domain macro average rises from 52.23 to 53.92 and from 37.08 to 38.68 in those two families.

Critical strengths

S1. The fixed-support ablation isolates the trajectory weights: with the compact support unchanged, removing them lowers the Qwen macro average from 53.92 to 52.47. This tells us more than the full-method comparison alone, because it ties a specific part of the gain to weighting in this recipe.

S2. The analysis makes clear what the compact estimate guarantees. Compact KL is a lower bound with an explicit conditional-tail gap, and the paper distinguishes an unbiased loss value from an unbiased parameter gradient. The exact-enumeration and sample-count controls are also useful: closer local gradients do not consistently translate into better final scores.

Critical weaknesses

W1. The RKL and JSD results need a more explicit account of the training update. The paper already distinguishes value unbiasedness from gradient correctness, and the derivation includes the student-proposal term. However, the actual surrogate losses and detach choices are not specified in the algorithm. Since the proposal depends on the student for these objectives, those details determine which parameter gradient is applied. Teacher-sampled FKL at fixed support avoids this specific dependence.

W2. The significance claimed for the main results cannot be checked from the reported summaries. The paper states three independent training seeds, greedy decoding and benchmark-level paired tests, and the macro means can be recalculated, but the per-seed benchmark vectors and their aggregation into the tests are not supplied. The item counts of 1,773 for HumanEval, 1,192 for GPQA and 480 per AIME year also do not say which variants are included or whether the units are unique questions or repetitions.

Questions for authors

Q1. Could you state the surrogate losses actually used for RKL and JSD, including how the student-proposal gradient is handled and which terms are detached? On a small fixed support, how closely do the sample-average gradients match the exact-enumeration gradients for these two objectives?

Q2. What do the benchmark item counts include, that is, which variants were evaluated and how many items are unique questions versus repetitions? Could you also share the per-seed benchmark scores and the exact vectors and aggregation that entered the paired tests?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ZuPL (Part 1)

Thank you for your careful assessment and constructive feedback. Below, we provide point-to-point responses to your questions and the weaknesses you identified.


To W1: Clarification of the RKL/JSD training updates, surrogate objectives, and detach operations.

We clarify the actual gradient computation for RKL and JSD, distinguishing Monte Carlo value estimation from parameter-gradient estimation.

For a fixed generation state and compact support, let:

q(a)=\bar p_\theta(a\mid s_t),\qquad m(a)=\frac{p(a)+q(a)}{2}.$$ Here, $p$ is independent of $\theta$, while $q$ depends on $\theta$. All derivations assume fixed support without differentiating through discrete Top-K selection. We denote stop-gradient by $\operatorname{sg}[\cdot]$. **1. Reverse KL (RKL)** The RKL objective and Monte Carlo value estimator are: $$D_{\mathrm{RKL}}(q\|p) = \sum_{a\in\mathcal A_K}q(a)\log\frac{q(a)}{p(a)}, \qquad a_m\sim\operatorname{sg}[q],$$ $$\widehat D_{\mathrm{RKL}} = \frac{1}{M}\sum_{m=1}^{M} \log\frac{q(a_m)}{p(a_m)}.$$ Accounting for the student-dependent proposal, the full gradient and corresponding surrogate are: $$\nabla_\theta D_{\mathrm{RKL}} = \mathbb E_{a\sim q} \left[ \left(\log\frac{q(a)}{p(a)}+1\right) \nabla_\theta\log q(a) \right],$$ $$\widetilde L_{\mathrm{RKL}} = \frac{1}{M}\sum_{m=1}^{M} \operatorname{sg}\left[ \log\frac{q(a_m)}{p(a_m)}+1 \right]\log q(a_m).$$ **2. Jensen–Shannon Divergence (JSD)** The JSD objective and Monte Carlo value estimator are: $$D_{\mathrm{JSD}}(p,q) = \frac12D_{\mathrm{KL}}(p\|m) + \frac12D_{\mathrm{KL}}(q\|m),$$ $$\widehat D_{\mathrm{JSD}} = \frac{1}{2M}\sum_{m=1}^{M} \left[ \log\frac{p(a_m)}{m(a_m)} + \log\frac{q(b_m)}{m(b_m)} \right], \qquad a_m\sim p,\quad b_m\sim q.$$ Since $2m=p+q$, the mixture-derivative terms cancel, yielding: $$\nabla_\theta D_{\mathrm{JSD}} = \frac12\mathbb E_{a\sim q} \left[ \log\frac{q(a)}{m(a)} \nabla_\theta\log q(a) \right],$$ $$\widetilde L_{\mathrm{JSD}} = \frac{1}{2M}\sum_{m=1}^{M} \operatorname{sg}\left[ \log\frac{q(b_m)}{m(b_m)} \right]\log q(b_m).$$ **3. Actual Implementation** Both objectives combine the original Monte Carlo value with the corresponding surrogate: $$L_{\mathrm{RKL}} = \operatorname{sg}[\widehat D_{\mathrm{RKL}}] + \widetilde L_{\mathrm{RKL}} - \operatorname{sg}[\widetilde L_{\mathrm{RKL}}],$$ $$L_{\mathrm{JSD}} = \operatorname{sg}[\widehat D_{\mathrm{JSD}}] + \widetilde L_{\mathrm{JSD}} - \operatorname{sg}[\widetilde L_{\mathrm{JSD}}].$$ This preserves the original divergence value in the forward pass and uses the surrogate gradient during backpropagation. Our implementation follows this formulation: sampled indices are non-differentiable, the student proposal and score-function coefficients are detached, and gradients flow through student log-probabilities. The teacher remains frozen, while the student-side tail-probability gradient is retained. Under a fixed generation state and compact support, these estimators are unbiased with respect to the corresponding exact compact-divergence gradients. This guarantee does not extend to discrete support selection or on-policy rollout sampling. --- ### **To W2: Statistical significance, three training seeds, and benchmark-level paired tests.** For each method, benchmark scores are averaged across three independently trained checkpoints (macro-average standard deviation below 0.5 percentage points). Two-sided paired t-tests are then conducted across the 15 benchmark-level mean scores, rather than treating the 45 seed-benchmark combinations as independent observations. For evaluation, AIME uses 16 generations per question ($T=0.7$), with individual responses scored and averaged; GPQA uses the unweighted mean of Extended, Main, and Diamond accuracies; and HumanEval uses greedy decoding over 164 unique problems. The original counts of 1,773 for HumanEval and 1,192 for GPQA represent evaluation records rather than unique questions. We will clarify these distinctions in Table 6. Based on the unrounded benchmark-level mean scores, all six two-sided paired t-tests are significant at $p<0.01$: | Model | Objective | Mean Improvement | Paired t-test p-value | |---|---|---:|---:| | Qwen | FKL | +1.69 | $6.85\times10^{-8}$ | | Qwen | RKL | +2.93 | $1.67\times10^{-5}$ | | Qwen | JSD | +1.69 | $4.93\times10^{-8}$ | | DeepSeek | FKL | +1.59 | $4.96\times10^{-5}$ | | DeepSeek | RKL | +1.49 | $8.89\times10^{-3}$ | | DeepSeek | JSD | +1.54 | $1.24\times10^{-4}$ | All p-values use unrounded benchmark-level mean scores and paired comparisons across the same 15 benchmarks. Finally, Appendix C.1 originally described all benchmarks as using deterministic greedy decoding, which does not reflect AIME's actual sampling protocol. We will revise it in the camera-ready version to specify AIME's setting and distinguish it from the other benchmark configurations.
AuthorsOpenReview ↗

Response to Reviewer ZuPL (Part 2)

To Q1: Can you quantify the difference between sample-averaged RKL/JSD gradients and exact enumeration gradients?

For a fixed compact support, exact enumeration computes the exact gradient over all K+1K+1 compact events, whereas Monte Carlo approximates it by averaging gradient contributions from MM samples drawn with replacement from the student distribution.

For RKL, define the per-sample gradient contribution as:

\left( \log\frac{q(a)}{p(a)}+1 \right) \nabla_\theta\log q(a).$$ The Monte Carlo estimator is: $$\widehat g_{\mathrm{RKL}} = \frac1M\sum_{m=1}^{M}h_R(a_m), \qquad a_m\sim q.$$ Consequently: $$\mathbb E[\widehat g_{\mathrm{RKL}}] = \sum_{a\in\mathcal A_K}q(a)h_R(a) = g_{\mathrm{RKL}}^{\mathrm{Exact}}.$$ Similarly, for JSD, define: $$h_J(a)= \frac12 \log\frac{q(a)}{m(a)} \nabla_\theta\log q(a).$$ The Monte Carlo estimator is: $$\widehat g_{\mathrm{JSD}} = \frac1M\sum_{m=1}^{M}h_J(b_m), \qquad b_m\sim q.$$ Its expectation is: $$\mathbb E[\widehat g_{\mathrm{JSD}}] = g_{\mathrm{JSD}}^{\mathrm{Exact}}.$$ Thus, both Monte Carlo gradient estimators are unbiased with respect to their corresponding exact compact divergence gradients. With finite sampling, individual estimates may still differ from the exact gradient due to sampling variance: $$\operatorname{Cov}(\widehat g) = \frac{1}{M}\operatorname{Cov}(h).$$ Therefore, the estimator variance decreases proportionally to $1/M$, while its standard deviation decreases proportionally to $1/\sqrt M$. The finite-sample discrepancy arises from Monte Carlo sampling variance, rather than from omitting the gradient contribution induced by the student-dependent proposal. The gradient cosine similarity reported in Appendix D.3 evaluates local gradient alignment specifically for FKL. Since RKL and JSD use different proposal distributions and score-function estimators, this diagnostic cannot be interpreted as an empirical gradient error or cosine similarity measurement for RKL/JSD. Our theoretical guarantee is gradient unbiasedness under a fixed generation state and compact support, rather than exact agreement between every finite-sample estimate and its exact enumeration counterpart. -------------------------------------------------------------------------------------------------------------------- ### **To Q2: Clarification of benchmark item counts, three training seeds, and the inputs to paired tests.** For each method, we average benchmark scores across three independent training seeds. The three seeds assess training randomness, whereas paired t-tests are conducted over the same 15 benchmarks, rather than treating the 45 seed-benchmark combinations as independent observations. The evaluation protocols for AIME, GPQA, and HumanEval are detailed in our response to W2 above. Briefly, AIME uses 16 independent generations per question at $T=0.7$, GPQA uses the unweighted average of the Extended, Main, and Diamond subset accuracies, and HumanEval uses greedy decoding over its 164 unique programming problems. The reported values for HumanEval and GPQA represent evaluation records rather than unique questions. Specifically, we evaluate the Extended, Main, and Diamond subsets separately and compute the final GPQA score as the unweighted arithmetic mean of their individual accuracies. The three subsets contain 546, 448, and 198 evaluation records, respectively, totaling 1,192 records across the subsets rather than 1,192 unique questions. The six two-sided paired t-test results reported above are based on unrounded benchmark-level mean scores, with paired comparisons across the same 15 benchmarks. All six comparisons satisfy $p<0.01$. We will revise Appendix C.1 and Table 6 to clarify the benchmark-specific decoding configurations, unique question counts, and evaluation records, as well as the statistical aggregation procedure. -------------------------------------------------------------------------------------------------------------------- We sincerely appreciate the reviewer's constructive feedback and the opportunity to clarify these aspects. We hope that our responses adequately address the concerns regarding the theoretical properties and practical implementation of our Monte Carlo gradient estimators, as well as the evaluation protocol. **We hope that these clarifications resolve the main concerns and encourage the reviewer to consider a more positive assessment of our work.**

Thank you for your detailed explanation of the RKL/JSD surrogate model and the separation operation, which addresses my concerns at the formulation level. However, two questions I raised remain missing:

  1. an empirical RKL/JSD comparison of sample-averaged gradients versus precisely enumerated gradients;
  2. benchmark scores for each seed and unrounded paired test vectors.

The p-values ​​for FKL cosine diagnostics and reporting answer different questions. Could you provide these checks and inputs?

Ablation experiments showed that teacher confidence weighting played a beneficial role in this formulation (accuracy of Qwen3-4B increased from 52.47% to 53.92%), but this is not a direct comparison with standard OPD.

I still consider this a practical supplement to the established refinement objectives, and therefore these clarifications improve the technical description, but also leave me with reservations about its suitability. Furthermore, I would like the authors to submit an updated PDF to acknowledge your revisions.

Reviewer_QbjDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TCAD, which weights student-generated responses by the teacher’s length-normalized likelihood. It combines this weighting with a compact support-plus-tail distribution and Monte Carlo sampling for token-level distillation.

Critical strengths
  • The main idea is simple and practical: reuse teacher likelihood to weight rollouts without an extra reward model.

  • Experiments cover two teacher–student pairs, three divergence objectives, and 15 benchmarks.

  • The ablations separately examine confidence weighting, support selection, and Monte Carlo sampling.

Critical weaknesses
  • Appendix A.2 acknowledges that RKL/JSD need a gradient correction, but never specifies what the experiments actually use. This leaves a key implementation detail unresolved. This should be solved by detailed formula or publishing codes, or at lease double check which gradient is the final used one.

  • The evaluation protocol is unclear: Table 6 lists 1,773 HumanEval samples and 480 AIME problems per year, without explaining how these counts fit the stated greedy-decoding setup.

  • The confidence probe has reporting inconsistencies: it claims all statistics are within-query but includes a global correlation. The reported 88.56% of positive-correlation groups also needs an explanation given the maximum of 100 groups.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer QbjD (Part 1)

Thank you for your careful assessment and constructive feedback. Below, we provide point-to-point responses to the weaknesses you identified.


To W1: Clarification of RKL/JSD gradient estimation and surrogate objectives.

We thank the reviewer for noting that the original manuscript did not sufficiently clarify RKL/JSD gradient estimation. We distinguish Monte Carlo divergence-value estimation from parameter-gradient estimation and provide the corresponding surrogate objectives and stop-gradient operations.

For a fixed generation state sts_t and compact event space AK\mathcal A_K, let:

q(a)=\bar p_\theta(a\mid s_t),\qquad m(a)=\frac{p(a)+q(a)}{2}.$$ Here, $p$ is independent of $\theta$, whereas $q$ depends on $\theta$. All derivations use the fixed compact support without differentiating through discrete Top-K selection. We denote stop-gradient by $\operatorname{sg}[\cdot]$. **1. Forward KL (FKL)** Since the teacher sampling distribution is independent of $\theta$, the FKL gradient is: $$\nabla_\theta D_{\mathrm{FKL}} = -\mathbb E_{a\sim p} \left[ \nabla_\theta\log q(a) \right].$$ No student-dependent proposal correction is needed. **2. Reverse KL (RKL)** The RKL objective is: $$D_{\mathrm{RKL}}(q\|p) = \sum_{a\in\mathcal A_K} q(a)\log\frac{q(a)}{p(a)}.$$ Its Monte Carlo value estimator samples from the detached student distribution: $$a_m\sim\operatorname{sg}[q], \qquad \widehat D_{\mathrm{RKL}} = \frac{1}{M}\sum_{m=1}^{M} \log\frac{q(a_m)}{p(a_m)}.$$ Accounting for the student-dependent proposal, the full gradient is: $$\nabla_\theta D_{\mathrm{RKL}} = \mathbb E_{a\sim q} \left[ \left( \log\frac{q(a)}{p(a)}+1 \right) \nabla_\theta\log q(a) \right].$$ We therefore use the following score-function surrogate: $$\boxed{ \widetilde L_{\mathrm{RKL}} = \frac{1}{M}\sum_{m=1}^{M} \operatorname{sg} \left[ \log\frac{q(a_m)}{p(a_m)}+1 \right] \log q(a_m) }$$ The sampled indices are non-differentiable, the score-function coefficient is detached, and gradients are propagated through the student log-probability. **3. Jensen–Shannon Divergence (JSD)** The JSD objective is: $$D_{\mathrm{JSD}}(p,q) = \frac12D_{\mathrm{KL}}(p\|m) + \frac12D_{\mathrm{KL}}(q\|m).$$ Its Monte Carlo value estimator uses teacher-side and student-side proposals: $$a_m\sim p,\qquad b_m\sim q,$$ $$\widehat D_{\mathrm{JSD}} = \frac{1}{2M}\sum_{m=1}^{M} \log\frac{p(a_m)}{m(a_m)} + \frac{1}{2M}\sum_{m=1}^{M} \log\frac{q(b_m)}{m(b_m)}.$$ Since $2m=p+q$, the mixture-derivative terms cancel when differentiating the two KL terms jointly, yielding: $$\nabla_\theta D_{\mathrm{JSD}} = \frac12 \mathbb E_{a\sim q} \left[ \log\frac{q(a)}{m(a)} \nabla_\theta\log q(a) \right].$$ The corresponding student-side score-function surrogate is: $$\boxed{ \widetilde L_{\mathrm{JSD}} = \frac{1}{2M}\sum_{m=1}^{M} \operatorname{sg} \left[ \log\frac{q(b_m)}{m(b_m)} \right] \log q(b_m) }$$ Both teacher-side and student-side samples contribute to value estimation. For gradient estimation, the student proposal is detached during sampling, the score-function coefficient is stop-gradient, and gradients flow through the student log-probability. **4. Forward-Pass Value and Backward-Pass Gradient** For both divergences, we combine the original Monte Carlo value and the corresponding surrogate: $$L_{\mathrm{RKL}} = \operatorname{sg}[\widehat D_{\mathrm{RKL}}] + \widetilde L_{\mathrm{RKL}} - \operatorname{sg}[\widetilde L_{\mathrm{RKL}}],$$ $$L_{\mathrm{JSD}} = \operatorname{sg}[\widehat D_{\mathrm{JSD}}] + \widetilde L_{\mathrm{JSD}} - \operatorname{sg}[\widetilde L_{\mathrm{JSD}}].$$ This preserves the original divergence value in the forward pass while using the surrogate gradient during backpropagation. Our actual training implementation follows these objectives: sampled indices are non-differentiable, the student proposal and score-function coefficients are detached, and gradients flow through the student log-probabilities. The teacher remains frozen, while the student-side gradient of the tail probability is retained. Thus, the implementation explicitly accounts for student-dependent proposal gradients rather than naively differentiating the sampled integrand. Under a fixed generation state and compact support, the resulting Monte Carlo gradient estimators are unbiased with respect to the exact gradients of their corresponding compact divergences. This guarantee does not extend to differentiation through discrete support selection or the on-policy rollout sampling distribution.
AuthorsOpenReview ↗

Response to Reviewer QbjD (Part 2)

To W2: Clarification of evaluation protocols and sample counts for HumanEval, GPQA, and AIME.

We agree that the original Sample Size column in Table 6 did not adequately distinguish unique test questions, evaluation records across benchmark subsets, and repeated generations.

For small mathematical benchmarks, a single stochastic generation may lead to considerable evaluation variability. We therefore use repeated sampling to obtain more stable performance estimates. This practice is supported by OpenCompass, which recommends multiple evaluation runs and averaging for datasets with limited sample sizes. EvalScope also supports repeated generations and aggregation for mathematical reasoning tasks [1, 2]. Specifically, AIME 2024 and AIME 2025 each contain 30 unique questions. We use a sampling temperature of 0.7 and generate 16 independent responses for each question. Each response is scored individually, and the final accuracy is computed by averaging correctness over all generations. Thus, the reported 480 evaluation instances per benchmark represent 30×1630\times16 generations, rather than 480 unique questions. This estimates average single-generation accuracy, rather than Pass@16 or majority voting. All compared methods use the same generation and aggregation protocol. For HumanEval, we use the original benchmark containing 164 unique programming problems, evaluated with deterministic greedy decoding. The reported value in the original Table 6 does not represent the number of unique HumanEval problems. We will revise Table 6 to explicitly report the 164 unique questions and distinguish them from the evaluation instances recorded by the evaluation pipeline.

For GPQA, we evaluate the Extended, Main, and Diamond subsets separately and compute the final score as the unweighted arithmetic mean of their individual accuracies. The subsets contain 546, 448, and 198 evaluation records, respectively. Thus, 1,192 represents the sum of the evaluation records across the three subsets, rather than the number of unique questions. The final aggregation is:

Acc_{\mathrm{Extended}}+ Acc_{\mathrm{Main}}+ Acc_{\mathrm{Diamond}} }{3}.$$ We acknowledge that Appendix C.1 originally described all benchmarks as using deterministic greedy decoding, which does not accurately reflect the repeated stochastic sampling protocol used for AIME. We will revise Appendix C.1 in the camera-ready version to explicitly distinguish the AIME evaluation setting from the configurations used for the remaining benchmarks. **References** [1] OpenCompass Contributors. Tutorial for Evaluating Reasoning Models. OpenCompass Documentation. [2] ModelScope Team. Quick Start. EvalScope Documentation. -------------------------------------------------------------------------------------------------------------------- ### **To W3: Clarification of global correlation and positive-correlation group statistics in the Confidence Probe.** The point-biserial correlation in Table 3 ($r=0.5573$) is computed globally over all 800 rollouts. In contrast, the mean within-prompt Spearman correlation ($\rho=0.5431$) is computed separately within each prompt containing both correct and incorrect responses and then averaged over the 96 valid prompts. Pairwise ranking accuracy ($A=81.77\%$) is likewise computed by comparing correct and incorrect rollouts within the same prompt. For the proportion of prompts with positive within-prompt correlations ($P$), 96 of the 100 prompts contain both correct and incorrect responses. Among these 96 mixed prompts, 85 exhibit a positive correlation between teacher confidence and response correctness, yielding $P=85/96$. We will report this proportion in Table 3 and Appendix D.1, while further clarifying the distinction between global point-biserial correlation and within-prompt ranking statistics. -------------------------------------------------------------------------------------------------------------------- We hope that our clarifications have adequately addressed your concerns regarding RKL/JSD gradient estimation, practical implementation, benchmark evaluation protocols, and statistical reporting. We sincerely appreciate your constructive feedback, which has helped us improve the technical clarity and rigor of our paper. **In light of these clarifications, we hope you will find our responses satisfactory and consider giving the paper a more positive rating.**
AuthorsOpenReview ↗

Gentle Reminder Regarding Our Rebuttal

Dear Reviewer QbjD,

We hope you have had a chance to review our rebuttal and the clarifications provided in response to your comments. As the rebuttal period is approaching its deadline, we would greatly appreciate any feedback you may have.

We have carefully addressed the concerns raised in your review and revised the manuscript accordingly. We sincerely hope our responses have resolved your questions and clarified the contributions of our work. We would be grateful if you could share your thoughts and consider a more positive assessment of the paper. Thank you very much for your time and consideration !

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer QbjD,

We would like to briefly follow up on our previous comment as the rebuttal period is approaching its deadline. We would appreciate your consideration of our responses and any further feedback you may wish to provide before the deadline. We sincerely hope that our clarifications have addressed your concerns and would welcome your further assessment of our work.

Thank you again for your time and consideration.

Reviewer_FuT9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TCAD, a reward-free on-policy distillation method with confidence reweighting plus symmetric Top-K and MC estimation. Overall, the idea is simple and the evaluation does show it helps over vanilla OPD

Critical strengths
  1. The motivation and method are sound. The blind imitation story makes sense, and confidence weighting + Dual-Union + compact divergence fit naturally.

  2. The evaluations, ablations, and results across dense/MoE and FKL/RKL/JSD show consistent gains on TCAD, demonstrating its effectiveness over various setups

Critical weaknesses
  1. The motivation sounds right but there's no direct evidence. It would be better to have a case study showing that fitting the teacher on wrong trajectories actually hurts, e.g. what happens to loss/gradients on low- vs high-agreement rollouts.
  2. No comparison with other distillation methods. The paper discusses reward-based and teacher-intervention approaches but only runs vanilla OPD + SFT + RSO, so it's unclear how it stacks up against the most relevant prior work.
Questions for authors

In addition to the weaknesses,

  1. In Table 1, why does the student beat the teacher so much on Math (Qwen teacher 35.75 vs Ours 41.68)?
  2. How is teacher ranking ensured to be reliable, or how robust is TCAD to the ranking? It's basically average log-likelihood, and the probe itself has ~18% pairwise errors, it would be better to have an analysis of when it systematically fails.
  3. In Table 4, why does MC (53.92 at M=256) beat exact enumeration (52.35), but M=512 has higher gradient cosine (0.96) yet lower score (53.69)? If fidelity keeps going up, why doesn't performance follow?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FuT9 (Part I)

Thank you for your careful assessment and constructive feedback. Below, we provide point-to-point responses to your questions and the weaknesses you identified.


To W1: Insufficient motivation and analysis of loss/gradient contributions from incorrect trajectories.

We emphasize that our motivation is not based on the assumption that supervision on incorrect trajectories is inherently harmful. Matching the teacher distribution on an incorrect prefix may still provide useful corrective signals. Rather, our concern is that standard OPD assigns uniform weights to all rollouts without distinguishing their relative training utility.

Importantly, confidence-based ranking and gradient harmfulness are distinct notions. The Confidence Probe evaluates whether teacher likelihood can rank the relative quality of rollouts within the same prompt; it does not directly establish whether a particular rollout induces a harmful gradient. The former provides a basis for relative supervision weighting, whereas the latter depends on how the resulting gradient affects student parameter updates and downstream performance. TCAD does not filter rollouts based on correctness labels. Instead, it retains all rollouts and adjusts their gradient contributions through sequence-level confidence weights. For the sampled rollouts, with the confidence weights treated as stop-gradient coefficients, the resulting gradient can be expressed as:

= \sum_i w_i \frac{1}{|y^{(i)}|} \sum_t \nabla_\theta L_{\mathrm{tok}}(s_t^{(i)};\theta)$$ Thus, even low-confidence or incorrect rollouts can provide potentially useful corrective supervision, while their relative contributions to optimization are adjusted. This design is supported by two complementary results. First, the Confidence Probe achieves 81.77% within-prompt pairwise ranking accuracy and a mean within-prompt Spearman correlation of 0.5431, supporting the usefulness of teacher confidence as a relative ranking signal. Second, removing confidence weighting reduces the Qwen macro-average from 53.92% to 52.47% (Table 2), providing empirical evidence for the contribution of this component to downstream performance. The examples in Appendix D.4 further illustrate how teacher confidence can distinguish responses with different information densities. They are not intended to establish that fitting incorrect trajectories is inherently harmful. We will further clarify this distinction and the scope of the supporting evidence in the revised manuscript. -------------------------------------------------------------------------------------------------------------------- ### **To W2: Lack of comparisons with other knowledge distillation methods.** We compare TCAD with SFT, RSO, and standard OPD using FKL, RKL, and JSD objectives across Dense-to-Dense and MoE-to-MoE architectures. These approaches differ primarily in how supervision signals are obtained and utilized: SFT imitates teacher-generated demonstrations; RSO selects teacher-preferred responses through rejection sampling; and reward-based distillation uses outcome rewards or reward models to evaluate generated responses. In contrast, TCAD does not require an additional reward model or discard low-confidence rollouts through rejection sampling. It retains all student-generated trajectories and softly reweights their contributions based on teacher-assigned confidence scores, followed by token-level divergence optimization over a compact support. We acknowledge that reward-model-based distillation and teacher-intervention are relevant research directions, but they are not directly evaluated baselines in our experiments. Accordingly, our empirical claims are limited to the methods evaluated in this work, rather than asserting superiority over all existing reward-based approaches. We will clarify the scope of our comparisons in the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer FuT9 (Part 2)

To Q1: Why does the Student substantially outperform the Teacher on mathematical benchmarks (35.75 vs. 41.68)?

The Qwen3-4B-Base student initially achieves a mathematical macro-average of 27.80%, below the Qwen3-32B-Instruct teacher's 35.75%. After training with OPD-FKL and TCAD, the student reaches 41.68%, surpassing the teacher by 5.93 percentage points. Importantly, this improvement should be interpreted in the context of both the standard OPD contribution and the additional benefit of TCAD. Standard OPD-FKL already improves the student's mathematical macro-average from 27.80% to 40.07%, while TCAD further increases it to 41.68%, a gain of 1.61 percentage points over the corresponding OPD baseline. Thus, we do not attribute the entire improvement over the teacher to TCAD alone. The improvement is also not uniform across all tasks. The student remains below the teacher on AIME 2024 (30.55% vs. 32.46%), but outperforms it on AIME 2025 (24.59% vs. 22.85%), GSM8K (54.84% vs. 44.40%), and MATH (56.74% vs. 43.28%). Therefore, this result reflects a higher average performance on the evaluated mathematical benchmarks, rather than a universal advantage over the teacher.

Similar student-over-teacher improvements have also been reported in recent studies of Qwen3-based on-policy distillation. The Qwen3 Technical Report shows that teacher-guided learning on student-generated trajectories improves Qwen3-8B's AIME Pass@64 from 90.0% to 93.3% on AIME 2024 and from 83.3% to 86.7% on AIME 2025. Dr. OPD further reports that a Qwen3-1.7B student achieves a five-benchmark mathematical average of 35.0%, surpassing its Qwen3-4B teacher's 34.1% through selectively weighted teacher supervision. Together, these studies provide supporting evidence that teacher-guided learning on student-generated trajectories can enable smaller models to outperform their teachers on specific target benchmarks [1, 2].

In our setting, the student learns from its own rollouts through teacher-confidence weighting and compact token-level distillation. It also receives approximately 100K instruction samples, including 30% mathematics- and code-related data, with benchmark decontamination. Therefore, the observed performance reflects the combined effect of task-oriented post-training and teacher-guided on-policy learning. We emphasize that this result does not imply that the smaller student has surpassed the teacher in general capabilities. Rather, it shows that a smaller model can achieve higher average performance on specific target benchmarks through task-oriented post-training and teacher-guided learning on its own generated trajectories.

References

[1] Qwen Team. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025.

[2] Zhenyu Wang et al. Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models. arXiv preprint arXiv:2609.38025, 2026.


To Q2: Reliability of teacher ranking and robustness to ranking errors.

We use the teacher's length-normalized sequence log-likelihood as a relative ranking signal within each prompt, rather than an absolute correctness measure. Across 100 prompts with 8 rollouts each (800 total), the Confidence Probe yields a global point-biserial correlation of 0.5573, a mean within-prompt Spearman correlation of 0.5431 across 96 mixed prompts, a positive-correlation proportion of 85/96 (88.54%), and a within-prompt pairwise ranking accuracy of 81.77%. The global correlation measures the overall confidence-correctness association, whereas the within-prompt metrics assess relative ranking consistency among rollouts for the same prompt.

TCAD does not assume perfect teacher ranking or discard low-confidence rollouts. Instead, softmax-normalized weights retain all rollouts while adjusting their relative contributions. Removing confidence weighting reduces the macro-average by 1.45 percentage points (Table 2), supporting the effectiveness of this mechanism.


To Q3: Why does M=256 outperform Exact Enumeration, while M=512 achieves higher gradient cosine similarity but lower final performance?

Exact Enumeration, MC-256, and MC-512 achieve downstream scores of 52.35%, 53.92%, and 53.69%, with initialization-time gradient cosine similarities of 1.00, 0.90, and 0.96, respectively. The cosine similarity in Appendix D.3 measures local gradient alignment across 100 fixed token-level states, not the entire training trajectory or final performance. Although increasing MM reduces Monte Carlo variance and improves local gradient alignment, it does not guarantee better outcomes in non-convex optimization. The 0.23-point difference between MC-256 and MC-512 is an empirical observation, not evidence that MC-256 is universally optimal.

I have read the authors response and prefer to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.