- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper organizes LLM distillation along two axes: the source of training prefixes and the direction of token-level KL divergence. It interprets the four combinations through connections to SFT, DAgger-style imitation learning, offline policy optimization, and on-policy distillation. Experiments distill Qwen3-4B/8B teachers into a Qwen3-0.6B student, examining mathematical reasoning, predictive entropy, response length, and subsequent GRPO training under short and long distillation horizons.
Critical strengths
The distillation-to-RL evaluation addresses a consequential practical question. Comparing warm starts through subsequent GRPO training goes beyond reporting distillation endpoints. The observed differences between standalone performance and continued trainability could help practitioners choose initialization procedures for multi-stage post-training.
The study examines several relevant dimensions under a common model family. The four objective combinations, two teacher scales, two training horizons, and joint reporting of accuracy, entropy, and length provide useful empirical coverage. Shared tokenization and the stated full-vocabulary KL computation also help reduce confounding from token alignment and vocabulary truncation.
Critical weaknesses
The theoretical unification conflates fixed-prefix identities with optimization under a student-dependent prefix distribution. Appendix E, page 18, lines 925–934, discards teacher entropy when claiming equality of the global minimizers of student-prefix forward KL and cross-entropy. However,
Although teacher entropy is fixed at an individual prefix, its expectation generally depends on through state visitation. A two-step counterexample makes the distinction explicit. Let the teacher select branches and uniformly, while the student selects with trainable probability . At the second step, let the teacher distributions be after and after , with the student already matching both and only trainable. Forward KL is minimized at , whereas cross-entropy adds to the first-step cross-entropy and is minimized at approximately . The later discussion of the supervised subproblem under frozen rollout distributions is valid, but does not establish the earlier global equivalence.
The same distinction affects the sequence-level reverse-KL interpretation. For , the full gradient includes both terms:
=\sum_t\mathbb{E}\left[ \nabla_\theta\ell_\theta(s_t) +\ell_\theta(s_t)\sum_{i<t}\nabla_\theta\log q_\theta(y_i\mid s_i) \right].$$ Proposition 1 is correct at a fixed prefix, but ordinary backpropagation through KL on sampled, fixed prefixes supplies only the first term. Consequently, it does not by itself establish optimization of the student-dependent sequence objective. Appendix D additionally describes a single-sample log-ratio reward implementation, while Section 3 and Appendices G.4/H.2 describe exact full-vocabulary KL. These descriptions need to be reconciled through explicit pseudocode specifying sampling, stop-gradient operations, and whether future log-ratio rewards affect earlier-token updates. This issue affects the central interpretation of the algorithms.Questions for authors
How should the global argmin equivalence in Appendix E be corrected when the prefix distribution depends on the student parameters? Which conclusions require only a frozen-prefix surrogate, and which require the full student-dependent objective?
Does reverse-KL training differentiate exact vocabulary-level KL on fixed sampled prefixes, use sampled log-ratio policy-gradient updates, or combine both?
What are the exact horizons, entropy threshold, stage budgets, transition decisions, stopping rule, gating split, and effective token/compute totals used for Table 3?
Do the curriculum gains persist against matched-budget fixed intermediate horizons, the same ungated schedule, and validation-based early stopping?
What is the precise contribution beyond GKD's distillation framework, and how do the proposed mixtures compare with AKL and entropy-aware on-policy distillation methods such as EOPD? A direct comparison under the same teacher, student, data, and budget would materially affect the assessment of the claimed methodological advance.
7 comments
Part 2/7
Q2. Is the sequence-level interpretation of reverse KL consistent with the actual training update?
Response
Thank you for your careful reading and for asking us to clarify the theoretical interpretation and implementation. Our RL-style connection follows from the token-level gradient identity, and the implementation comparisons distinguish fixed-prefix updates from full sequence-gradient estimators.
-
Our token-level reverse-KL update has an explicit policy-gradient interpretation and follows the OPD mechanism of supervising student-visited prefixes. Proposition 1(ii) shows that, at a fixed prefix, the reverse-KL descent direction equals the policy-gradient ascent direction with the teacher–student log-probability difference as scalar reward. Subsequent student-generated prefixes also receive teacher supervision, allowing the student to improve its continuations on the states it visits. GKD [1] adopts this supervision mechanism and distinguishes it from differentiating through the student's sampling process. Fu et al. [2] analyze the bias–variance tradeoff between token-level and sequence-level updates, and their synthetic experiments show that stronger future-reward coupling can increase gradient variance and destabilize training. We will distinguish the sequence-KL decomposition from the token-level policy-gradient identity in Section 2.3 and expand the discussion of these references.
-
The main experiments directly differentiate full-vocabulary KL, and the implementation controls distinguish gradient targets and sampling variance. At each sampled prefix, we compute reverse KL over the full vocabulary and backpropagate through the student distribution. Sampled prefixes and teacher outputs are held fixed, and the main implementation uses no additional return-to-go policy-gradient term. Controls comprise sampled-token policy gradients, sequence-level updates using cumulative future log-ratios, and full-vocabulary KL with a prefix-distribution correction. Table 1 uses an FP64 autoregressive model with vocabulary size 3, horizon 4, and seed 42. We enumerate all 81 trajectories to compute each estimator's expected gradient and variance under the same student distribution. Distances use the relative L2 norm against each reference gradient, and variance is over one sampled trajectory. Table 2 uses the same instruction-tuned Qwen3-4B teacher, Qwen3-0.6B-Base student, 4096-token training horizon, 1000 steps, and the evaluation protocol in the main paper. We will align Appendix D, Section 3, and Appendices G.4/H.2 with the sampling, stop-gradient, and update steps in Part 3.
Table 1. Gradient targets and sampling variance in the enumerable model. Distances compare expected gradients; variance is normalized to full-vocabulary token-level KL.
| Implementation | Distance to frozen-prefix gradient | Distance to sequence gradient | Relative variance |
|---|---|---|---|
| Full-vocabulary token-level KL | 1.51e−16 | 0.149 | 1.00 |
| Sampled-token policy gradient | 1.65e−16 | 0.149 | 3.33 |
| Full-vocabulary KL + prefix correction | 0.155 | 4.36e−16 | 3.24 |
| Sequence-level policy gradient | 0.155 | 2.35e−16 | 11.48 |
Table 2. Implementation comparison under matched training settings. Metrics are means across four mathematical benchmarks.
| Implementation | Avg@k (%) | Pass@k (%) | Mean response length |
|---|---|---|---|
| Full-vocabulary token-level KL (our implementation) | 36.4 | 48.3 | 8,185 |
| Sampled-token policy gradient | 35.8 | 47.5 | 8,100 |
| Full-vocabulary KL + prefix correction | 35.3 | 47.0 | 7,950 |
| Sequence-level policy gradient | 34.7 | 46.1 | 7,900 |
Table 1 confirms the respective gradient targets and shows that full-vocabulary token-level KL has the lowest variance, while sequence-level policy gradients have 11.48× higher variance in this diagnostic. Our full-vocabulary implementation achieves the highest Avg@k and Pass@k in Table 2, improving over sampled-token policy gradients by 0.6 and 0.8 percentage points, respectively. Sequence-level policy gradients perform worst, with Avg@k and Pass@k lower by 1.7 and 2.2 percentage points than our implementation. This weaker performance is consistent with the noise introduced by coupling early-token updates to stochastic future log-ratios, as analyzed in [2]. We will add these comparisons to the revised manuscript.
References
[1] R. Agarwal et al. On-policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.
[2] Y. Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv:2603.25562, 2026.
Part 3/7
Q2 (continued). Is the sequence-level interpretation of reverse KL consistent with the actual training update?
Response
The main experiments directly differentiate full-vocabulary KL, and the implementation controls distinguish gradient targets and sampling variance. At each sampled prefix, we compute reverse KL over the full vocabulary and backpropagate through the student distribution. Sampled prefixes and teacher outputs are held fixed, and the main implementation uses no additional return-to-go policy-gradient term. Controls comprise sampled-token policy gradients, sequence-level updates using cumulative future log-ratios, and full-vocabulary KL with a prefix-distribution correction.
The pseudocode specifies the four updates for one fixed-length trajectory, with batch reduction omitted. sg denotes stop-gradient, suffix_sum accumulates from the current position to the end, and c is a KL cost. All losses use gradient descent.
y = sg(sample_student(prompt))
s = prefixes(prompt, y)
logp = sg(teacher_log_probs(s))
logq = student_log_probs(s)
K = sum_over_vocab(exp(logq) * (logq - logp))
a = gather_generated_token(logq, y)
c = sg(gather_generated_token(logq - logp, y))
loss_full_vocab = sum(K)
loss_token_pg = sum(c * a)
loss_sequence_pg = sum(suffix_sum(c) * a)
# Add only the future-state correction to the exact local KL gradient.
future_K = sg(suffix_sum(K) - K)
loss_combined = loss_full_vocab + sum(future_K * a)
Q3. Please specify the curriculum schedule, gating rules, and effective training budget used for main-paper Table 3.
Response
Thank you for asking us to specify these details. The entropy gate controls horizon growth within a fixed 1,000-update budget, and the resource accounting includes both training and gate evaluation.
- The horizon increases only when the entropy gate passes; after a failed gate, training continues at the last stable horizon until the budget is exhausted. We use the configuration in Table 1 for both teacher sizes. At the end of each stage, we compute the student's mean per-token predictive entropy on the fixed gating split and advance only if . In both teacher settings, the first five stages use horizons 128, 256, 512, 1024, and 2048 for 120 updates each. The gate fails at the end of the 2048-token stage, after update 600, and the remaining 400 updates use the last stable horizon of 1024 tokens.
Table 1. Curriculum and gating configuration.
| Setting | Value |
|---|---|
| Candidate horizons | 128 → 256 → 512 → 1024 → 2048 → 4096 |
| Updates per stage / gate interval | 120 |
| Regular evaluation interval | 30 updates |
| Entropy threshold | 0.06 |
| Total updates | 1,000 |
| Batch size | 32 |
| Gating split | 128 held-out DeepScaleR prompts, seed 42, reused at every gate check |
| Failed gate | Continue at the last stable horizon |
The gating prompts are excluded from training. The schedule is implemented as follows, with the same model and optimizer carried through the stages.
horizons = [128, 256, 512, 1024, 2048, 4096]
last_stable = horizons[0]
used = 0
for horizon in horizons:
train(horizon=horizon, steps=120)
used += 120
H = mean_student_entropy(gating_prompts, horizon=horizon)
if H < 0.06:
break
last_stable = horizon
train(horizon=last_stable, steps=1000 - used)
- Each run completes 1,000 updates, and total GPU-hours include generation, teacher scoring, student optimization, and gate evaluation. Table 2 reports the effective training response tokens and total GPU-hours on a single H100 GPU with 94 GB of memory. Effective tokens count non-padding response tokens participating in the student KL loss, excluding prompt and gating-evaluation tokens. The gating overhead is included in total GPU-hours.
Table 2. Effective training response tokens and total GPU-hours.
| Teacher | Effective training response tokens | Total GPU-hours |
|---|---|---|
| Qwen3-4B | 26.0M | 12.5 |
| Qwen3-8B | 26.5M | 16.0 |
We will add the configuration, stage transitions, pseudocode, and resource accounting to Section 5.2 and Appendix G.
Part 1/7
Q1. The theoretical unification conflates fixed-prefix identities with optimization under a student-dependent prefix distribution.
Response
Thank you for your careful reading and for raising this question. The DAgger correspondence holds for supervised updates on fixed student prefixes, and our methods and empirical findings do not rely on a global argmin equivalence.
Forward KL and soft-label cross-entropy have identical gradients and minimizers when the prefix distribution is fixed during the supervised update. Let μkt denote the prefix distribution generated by qθk, held fixed while optimizing θ. The expected losses satisfy LFKL(θ;μk)=LCE(θ;μk)−Ck, where Ck=∑tEst∼μktH(pT(⋅∣st)) is constant with respect to θ. The identity also holds for a fixed mixture of historical rollout distributions, establishing the soft-label DAgger supervised subproblem with data aggregation. Using only current-rollout data gives a DAgger-style variant. We will distinguish the rollout parameters θk from the optimization parameters θ, state the argmin equivalence in Appendix E under a fixed prefix distribution, and clarify this scope in Section 2.3.
KL mixing, the length curriculum, and the empirical comparisons do not require a global argmin equivalence. KL mixing combines token-level losses, while the length curriculum uses predictive entropy to control the training horizon. Accuracy, entropy, response length, and downstream RL performance are measured directly from the trained checkpoints.