acceptodds
Under review as a conference paper at ICLR 2027

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

Abstract

On-policy distillation offers dense, per-token supervision for training reasoning models, but it remains unclear when this signal helps and when it hurts. Which teacher should be used, and in self-distillation, which context should serve as the supervisory signal? Answering these questions typically requires costly training runs whose aggregate metrics obscure what happens at individual tokens. We introduce a training-free diagnostic framework that operates per token, per question, and per teacher. We derive an ideal per-node gradient, the update that maximally increases the student's probability of success, and develop a scalable targeted-rollout algorithm to estimate it even for long reasoning chains. The gradient alignment score, the cosine similarity between this ideal gradient and a distillation gradient, quantifies how closely a configuration approximates the ideal signal. In distillation training runs with external teachers, the teacher ranking under the better-performing training configuration largely agrees with the score. Across self-distillation settings and external teachers, distillation guidance is more aligned with the ideal on incorrect rollouts than on correct ones, and the best distillation context depends jointly on the student's capacity and the task, motivating per-task, per-token diagnostics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, self-distillation, knowledge distillation, reasoning models, gradient alignment, reinforcement learning, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MTtFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free, per-token diagnostic for on-policy distillation. It derives an ideal per-node gradient, ∂L/∂z_j = P_θ(j)(P_succ(j) − P̄_succ), the logit direction that maximally raises the student's probability of reaching a correct answer from that node, and shows Dr. GRPO recovers it in expectation. It then shows GKD, the single-sample estimator and MiniLLM all produce per-node gradients of the shared form P_θ(j)(f_j − f̄), differing only in f, which makes them comparable by direction. The gradient alignment score is the cosine between the ideal and distillation gradients at each branching node. Success probabilities come from targeted rollouts under exponentially growing depth windows, with one enriched generation tree shared across all teacher configurations. The framework is applied to Qwen3-0.6B and Qwen3-1.7B across eight teacher configurations on BoolQ and MMLU (around 264K decision points) plus AIME 2025 case studies, and the resulting teacher ranking is checked against real distillation runs.

Critical strengths
  1. The unification is clean and correct. Showing that reward-based and distillation objectives share the local structure P_θ(j)(f_j − f̄) and differ only in f gives one frame for comparing GRPO-style and KD-style updates at token granularity. The corollary the authors draw from the shared P_θ(j) factor, that the teacher can amplify continuations the student already partly believes in but cannot inject new ones in a single step, is a useful mechanistic statement and it rationalises several scattered findings about overly capable teachers.

  2. The finding that distillation guidance is more aligned with the ideal on incorrect rollouts than on correct ones, significant under the SR-weighted cosine in all four settings, is concrete and directly actionable: weight distillation toward failing rollouts. The teacher-independent tree sharing that makes an eight-configuration comparison affordable is a well-judged piece of experimental engineering.

Critical weaknesses
  1. The score's dynamic range is too narrow for the ranking claims placed on it. In Table 1 the eight configurations span .000 to .053 with 95% CIs of roughly ±.02 to ±.04 that overlap almost entirely. At 0.6B on BoolQ, Qwen3-4B, 8B and 14B score .018, .018 and .016: two exact ties inside a confidence interval twice the size of the spread. In Figure 4 the fraction of aligned decision points runs 0.501 to 0.533 against a 0.500 chance line. "The teacher ranking largely agrees with the score" is then an ordering over quantities that are not statistically distinguishable, and no pairwise significance is reported.

  2. The validation against training is thin and partly self-defeating. Table 2 covers three external teachers, one dataset and two learning rates. The ranking flips at 1e-6. MMLU cannot validate anything because every teacher lowers accuracy. The paper itself notes the learning-rate effect is two to three times the between-teacher spread. Meanwhile the self-distillation configurations, which carry the most interesting claims in Table 1 and Figure 4, are validated only as "preliminary" runs in the appendix, so the headline conclusion about self-distillation versus external teachers is not checked against training at all.

  3. The cost works against the stated purpose, and the benchmarks do not match the use case. The pitch is avoiding expensive training runs, but the diagnostic needs 45K to 200K targeted rollouts per question, which is why AIME appears only as case studies. That limits practical use to short-chain tasks. BoolQ (True/False) and MMLU (four-way multiple choice) are exactly where OPD matters least, and where P_succ sits against a high chance floor, so the per-node success estimates are less informative than they would be in the long-chain regime the method is motivated by.

Questions for authors
  • What is the actual GPU cost of scoring one BoolQ question relative to one distillation training run at the same scale? A direct comparison would settle whether the diagnostic is cheaper than what it replaces.
  • With S_u typically holding only two or three tokens, how sensitive are the aggregate scores to N_sig and to the τ = 0.02 inclusion threshold?
  • Does the correct/incorrect alignment gap survive conditioning on nodes with comparable P̄_succ, or is part of it explained by correct and incorrect paths sitting at different points on the success-probability scale?
1 comment

We are very grateful for an exceptionally careful and constructive review that captures the paper precisely. We appreciate the recognition of the clean unification, the mechanistic corollary of the shared P_theta(j) factor, the actionable incorrect versus correct finding, and the teacher-independent tree sharing. We address each point below.

  1. Overlapping CIs. Comparisons are paired: all configurations are scored on the same tree, nodes, and questions. The marginal CIs in Table 1 include between-question variance that cancels in pairwise differences, so overlap does not imply indistinguishability. We will add question-level paired tests (Holm-corrected) and state rankings only for resolved pairs. Where the score shows a tie, training shows a near tie: the 0.6B external teachers score .018/.018/.016 and gain .072/.061/.059. Where the score separates widely, so does training: SELF-1C versus SELF-1C1W scores .047 versus .019 and trains to +.045 versus -.014. We will describe the 0.6B external teachers as indistinguishable.

Small magnitude. Empirically, the mean cosine and the fraction of aligned nodes move together (SELF-1C, 0.6B BoolQ: .526 aligned, mean .047), so a score near 0.05 means the teacher favors the better branch at slightly more than half of the outcome-relevant forks. That this is only slightly above chance is the paper's message: not all teacher guidance is useful. The useful signal concentrates on failing rollouts (Figure 3) and on a subset of nodes whose selection raises the mean signal 10 to 15 times (Table 8).

  1. Learning-rate flip. This illustrates our motivation: training outcomes mix teacher quality with hyperparameters, setup, and implementation details, and the learning-rate effect is two to three times the teacher spread. At 1e-6, gains are smaller for every teacher and both students, so 1e-5 is the practitioner-relevant setting; there, the 1.7B ordering on final accuracy matches the score fully. The score has no training hyperparameters and gives a stable reference that single runs cannot.

Self-distillation. The main claim, that correct-only contexts beat wrong-demonstration contexts, holds in all 8 training comparisons, and the score orders all 8 the same way (one-sided sign test p of about 0.004); within-setting Spearman is 0.67 to 1.00. Summarization is confirmed too: for 1.7B, the score ranks SELF-SUM-1C best on both datasets, and it trains best on both. We agree the two teacher families were trained under different protocols (LoRA versus full fine-tuning); we will add matched runs and move this validation to the main text.

MMLU. We agree MMLU cannot validate the ranking, as every teacher lowers accuracy there. The score ranks configurations by per-token signal quality; absolute gain also depends on the training setup and the factors above. We will state this scope explicitly.

  1. Cost and benchmark fit. A fair training comparison needs sweeps, seeds, and a verified pipeline per candidate; the score needs one teacher-independent tree per student, then one forward pass per configuration. We agree cost grows for long chains; exponential depth windows make the budget scale with the chosen number of windows rather than sequence length, which enabled AIME traces of up to 30K tokens. On the chance floor: the cosine is scale-invariant, so compressed P_succ differences do not bias the ideal direction, and the SR-weighted cosine emphasizes forks with larger differences.

Q1: We will add a per-question cost comparison with a training run and a ranking versus budget analysis.

Q2: As noted in Section 2.5, raising N_sig to 40 or 60 did not meaningfully change our conclusions. We will add the full table for N_sig in {20, 40, 60} and stricter tau, reporting ranking stability and the incorrect versus correct gap.

Q3: An excellent question. Existing results suggest the gap is not a scale artifact: the cosine is invariant to the smaller ideal-gradient magnitude, the gap appears for nearly every teacher (Table 12), and it persists on AIME. We will add the analysis stratified by mean P_succ. The practical recipe holds either way: rollout correctness is observable at training time while mean P_succ is not, so weighting by outcome is the implementable version.

Thank you again; this review will directly improve our statistical presentation and the scope of our claims.

Reviewer_VAigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a diagnostic framework for evaluating the effectiveness of teacher contribution at token-level. It derives an ideal gradient based on the probability of generating correct answers and introduces a gradient alignment score which measures the cosine similarity between the distillation gradient. To address the computation complexity, the author proposed to calculate this score through targeted student rollouts. Experiments across different teacher configurations, student model sizes, and reasoning benchmarks suggest that distillation guidance is more beneficial on incorrect trajectories, while the effectiveness of different teachers and self-distillation contexts depends on the student's capacity and the task.

Critical strengths
  • The paper provides a diagnostic framework to investigate when OPD helps or harms model performance, an important question given its increasing adoption.
  • The paper presents several interesting empirical findings and observations.
Critical weaknesses
  • Although the framework provides interesting diagnostic insights, it remains unclear how practitioners can use these findings to improve OPD training or select suitable teachers in practice.
  • The experiments primarily focus on relatively simple benchmarks (BoolQ and MMLU), with only a few case studies on complex mathematical reasoning.
Questions for authors

See weakness

1 comment

We sincerely thank the reviewer for the careful and positive assessment. We are glad the reviewer sees our diagnostic framework as addressing an important question given the growing adoption of OPD, and found our empirical findings and observations interesting. We address the two weaknesses raised below.

How practitioners can use the findings

Our current results support three concrete uses, which we will collect in a dedicated practical guidance paragraph in the revision.

  1. Screen teacher configurations before training. Training outcomes mix teacher quality with many hidden factors, such as learning rate, schedule, training setup, and implementation details; in our own runs, the learning-rate effect is two to three times the spread between teachers (Table 2). The score lets practitioners check whether a teacher carries useful signal without tuning hyperparameters or debugging a training pipeline, and it reliably flags harmful context designs. Adding a wrong demonstration lowers the score in all 8 settings, and those contexts also train worse in all 8 (Tables 1 and 7). Within each self-distillation setting, the score correlates with training gain at Spearman 0.67 to 1.00. In 3 of 4 settings, the top-scored self-distillation context is also the best one in training (on MMLU, where every context lowers accuracy, this means the least harmful one). Because the generation tree depends only on the student, each extra candidate configuration costs a single teacher forward pass.

  2. Focus distillation on failing rollouts. Distillation is consistently more aligned with the ideal on incorrect rollouts. The gap is significant in all four settings (Figure 3) and appears for 15 of 16 teacher and student pairs (Table 12). This leads to a simple recipe that needs no diagnostic at training time, since rollout correctness is observable: upweight the distillation loss on failing rollouts. The oracle analysis (Table 8) shows how much is at stake: restricting the teacher signal to aligned tokens raises the mean signal by 10 to 15 times while keeping about half of the tokens.

  3. Choose the teacher per task and per student. The best configuration shifts with student capacity and task. Summarized demonstrations help the 1.7B student but not the 0.6B student. External teachers match the best self-distillation context only for 1.7B. Wrong demonstrations hurt on BoolQ and MMLU but rank first on some hard AIME questions. A fixed recipe therefore does not transfer, and running the diagnostic on the target domain is informative.

On benchmark difficulty

We chose BoolQ and MMLU deliberately: they let us estimate success probabilities densely across hundreds of questions (about 264K decision points), which the statistical comparisons require. They are also not saturated for these students: the untrained baselines are .750 and .810 on BoolQ and .514 and .678 on MMLU. The analysis uses only outcome-relevant forks: a node counts only if at least two children each have at least 20 visits and their success rates differ. So even on these benchmarks, every scored decision is one where the token choice changes the outcome. The AIME 2025 case studies, with traces of 5K to 30K tokens, show that the method scales to long mathematical reasoning without modification, and the finding that alignment is higher on incorrect rollouts replicates on every AIME question we analyzed. We also include AIME training validation (Table 6): for the 1.7B student the teacher ordering matches the score exactly, and for the 0.6B student the lowest-scored teacher is also the weakest in training. We will extend the analysis to additional AIME questions and discuss full-scale evaluation on harder long-chain benchmarks in the revision.

We thank the reviewer again for the constructive feedback and hope these clarifications address the concerns.

Reviewer_wfNCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study when on-policy distillation provides useful or harmful supervision to students. They derive an ideal local gradient based on the student’s probability of reaching a correct answer and compare it with the distillation gradient using a gradient alignment score. By using targeted rollouts, they estimate this score at individual token positions to evaluate different teachers and self-distillation contexts without additional training. Experiments with two Qwen students on BoolQ, MMLU, and AIME questions show that guidance is generally more aligned on incorrect rollouts, while the best teacher or context depends on student capacity and the task. Comparisons with actual training runs show partial agreement between the score and training outcomes.

Critical strengths

The authors an important question of how to find the more aligned teacher without actually training a student. The paper is well motivated.

The authors provide some promising results.

Critical weaknesses

The results are not conclusive and not comprehensive.

There are some questions I have regarding the validity of the method.

Please see questions for details.

Questions for authors

The main experiments are only done on a single and slightly outdated family of models Qwen 3 (for both students and teachers), which limits the generalizability of the findings.

The main 2 benchmarks BoolQ and MMLU are simple, so results seem to be saturated as well, as shown through delta accuracy results. I would suggest a different setting to show a larger gap going much beyond the statistical error.

It seems that the score can sometimes identify the better teacher, but it does not reliably answer the question, whether training with that teacher will actually help the model.

When is this method computationally worth it? Computing the tree requires a lot of rollouts, while training on smaller dataset guiding the data selection might be cheaper.

I am not sure whether the proposed token-level logit alignment score reliably predicts helpful updates to the model’s parameters. I suggest checking whether higher scores predict improvements in accuracy after each training update.

2 comments

We sincerely thank the reviewer for the thoughtful review and for recognizing that our paper addresses an important and well-motivated question: identifying a better-aligned teacher without training the student. We also appreciate the constructive questions; in particular, the question on logit versus parameter updates helped us state the theoretical basis of the score more clearly.

Conclusiveness. Our central finding holds in every setting: distillation guidance is more aligned with the ideal on incorrect than on correct rollouts. It is significant in all four settings under the SR-weighted cosine (all p < 10^-5; Table 10) and in three of four under the mean cosine, holds for 15 of 16 teacher and student pairs on BoolQ (the one exception is a tie, not a reversal; Table 12), and replicates on every AIME question analyzed (5K to 30K tokens, versus about 200 for BoolQ).

Q1: Model family. We agree, and we will add additional model families in the revision. The current choice reflects a constraint of logit-level OPD: teacher and student must share a tokenizer, so external teachers are same-family, as in the Qwen3 pipeline itself. Within it, we cover two student sizes, three teacher sizes (4B to 14B), and five self-distillation contexts. The method needs only student rollouts and one teacher forward pass, so it applies to any family without modification.

Q2: Benchmarks. The benchmarks are not saturated for these students: baselines are .750/.810 on BoolQ and .514/.678 on MMLU, and distillation moves BoolQ accuracy by up to +.089 (Table 2). Moreover, a node is scored only if at least two children have at least 20 visits and different success rates, so every scored decision is one where the token choice changes the outcome. BoolQ and MMLU allow dense estimation across hundreds of questions (about 264K decision points), and AIME shows the method scales to long chains. We will extend the analysis to more AIME questions and discuss harder long-chain benchmarks in the revision.

Q3: Does the score tell whether training will help? Training outcomes entangle teacher quality with many hidden factors: learning rate, schedule, LoRA versus full fine-tuning, implementation details, and even bugs. In Table 2, the learning-rate effect is two to three times the spread between teachers, and the 1.7B ranking changes with it. The score measures teacher signal quality independent of these factors, so practitioners need not find the best hyperparameters or debug a pipeline to learn whether a teacher helps; it also serves as a sanity check when a training run disagrees with it. Where confounders are controlled, it agrees with training:

  • Correct-only contexts beat wrong-demonstration contexts in training in all 8 comparisons, and the score orders all 8 the same way (one-sided sign test p of about 0.004).
  • Within-setting Spearman correlation is 0.67 to 1.00 (Table 7).
  • For 1.7B external teachers, the top-scored teacher (Qwen3-8B) gains most at lr 1e-5, and the full ordering matches on final accuracy.

Q4: Compute. A fair training comparison is not one run: given the learning-rate sensitivity, it needs sweeps, seeds, and a verified pipeline per candidate. The score needs one teacher-independent tree per student, and each extra configuration costs one forward pass (about 7 times cheaper across our eight). It also shows where the teacher helps, which training cannot: for example, restricting to aligned tokens raises the mean signal 10 to 15 times using about half of the tokens (Table 8). We will add an explicit GPU cost comparison and a ranking versus budget analysis.

Q5: Logit versus parameter space. At node u, both objectives backpropagate through the same Jacobian J_u. A small step delta_theta = eta * J_u^T g_D changes the ideal objective, to first order, by

Delta L_ideal(u) = eta * g_ideal^T (J_u J_u^T) g_D

The logit inner product is thus the first-order predictor of improvement, weighted by the kernel J_u J_u^T, which depends only on the student. All teachers are therefore compared under the same geometry, and the score isolates the teacher's contribution. Training (Section 4.3) is the end-to-end check; following the reviewer's suggestion, we will also report the exact parameter-space cosine on sampled nodes and its agreement with the logit score.

We thank the reviewer again and would be glad to clarify anything further.

Reviewer_wfNCOpenReview ↗

Thank you for your reply!

I would like to thank the authors for their reply. I acknowledge reading the author's response and will take that into consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.