acceptodds
Desk rejected at ICLR 2027

Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge distillation has become central to LLM post-training, yet its design space remains poorly understood, especially alongside reinforcement learning (RL). We show that the prevailing paradigms, off-policy distillation and on-policy distillation (OPD), implicitly **couple two orthogonal choices: prefix source and token-level KL direction**. This coupling follows from decomposing sequence-level KL over autoregressive response distributions: *forward KL pairs teacher prefixes with token-level forward KL, and reverse KL pairs student prefixes with token-level reverse KL*. We argue that this coupling is not intrinsic: decoupling the two axes yields four valid objectives. We establish gradient-level identities showing that forward KL gives SFT-style cross-entropy matching with teacher soft targets, whereas reverse KL gives an RL-style policy-gradient objective with a dense teacher-student log-ratio reward, connecting the four objectives to off-policy SFT, DAgger-style on-policy SFT, offline-RL-style distillation, and OPD. We conduct an extensive controlled study on math reasoning, evaluating the four objectives both as standalone distillation methods and as initializations for subsequent RL. The results reveal three tradeoffs: *KL direction induces an accuracy–entropy tradeoff, prefix source induces a quality–compute tradeoff, and training length induces an accuracy–stability tradeoff*. Motivated by these findings, we propose **KL mixing** and an **entropy-gated length curriculum**. KL mixing shows that long-sequence distillation requires substantial forward-KL weight to prevent entropy collapse and length inflation without sacrificing accuracy. The entropy-gated length curriculum improves Avg@k and Pass@k by **3.6** and up to **5.8** points, and reduces average response length by roughly relative to fixed long-horizon training. Our framework and methods guide the design of reasoning distillation objectives that balance accuracy, diversity, compute, and RL behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Large language models, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_36VFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper organizes LLM distillation along two axes: the source of training prefixes and the direction of token-level KL divergence. It interprets the four combinations through connections to SFT, DAgger-style imitation learning, offline policy optimization, and on-policy distillation. Experiments distill Qwen3-4B/8B teachers into a Qwen3-0.6B student, examining mathematical reasoning, predictive entropy, response length, and subsequent GRPO training under short and long distillation horizons.

Critical strengths

The distillation-to-RL evaluation addresses a consequential practical question. Comparing warm starts through subsequent GRPO training goes beyond reporting distillation endpoints. The observed differences between standalone performance and continued trainability could help practitioners choose initialization procedures for multi-stage post-training.

The study examines several relevant dimensions under a common model family. The four objective combinations, two teacher scales, two training horizons, and joint reporting of accuracy, entropy, and length provide useful empirical coverage. Shared tokenization and the stated full-vocabulary KL computation also help reduce confounding from token alignment and vocabulary truncation.

Critical weaknesses

The theoretical unification conflates fixed-prefix identities with optimization under a student-dependent prefix distribution. Appendix E, page 18, lines 925–934, discards teacher entropy when claiming equality of the global minimizers of student-prefix forward KL and cross-entropy. However,

LSP-FKL(θ)=LCE(θ)−∑tEst∼dθtH(pT(⋅∣st)).L_{\mathrm{SP\text{-}FKL}}(\theta)=L_{\mathrm{CE}}(\theta)-\sum_t\mathbb{E}_{s_t\sim d_\theta^t}H\bigl(p_T(\cdot\mid s_t)\bigr).

Although teacher entropy is fixed at an individual prefix, its expectation generally depends on θ\theta through state visitation. A two-step counterexample makes the distinction explicit. Let the teacher select branches AA and BB uniformly, while the student selects AA with trainable probability rr. At the second step, let the teacher distributions be (0.9,0.1)(0.9,0.1) after AA and (0.5,0.5)(0.5,0.5) after BB, with the student already matching both and only rr trainable. Forward KL is minimized at r=0.5r=0.5, whereas cross-entropy adds rh(0.9)+(1−r)h(0.5)r h(0.9)+(1-r)h(0.5) to the first-step cross-entropy and is minimized at approximately r=0.5891r=0.5891. The later discussion of the supervised subproblem under frozen rollout distributions is valid, but does not establish the earlier global equivalence.

The same distinction affects the sequence-level reverse-KL interpretation. For J(θ)=∑tEst∼dθtℓθ(st)J(\theta)=\sum_t\mathbb{E}_{s_t\sim d_\theta^t}\ell_\theta(s_t), the full gradient includes both terms:

=\sum_t\mathbb{E}\left[ \nabla_\theta\ell_\theta(s_t) +\ell_\theta(s_t)\sum_{i<t}\nabla_\theta\log q_\theta(y_i\mid s_i) \right].$$ Proposition 1 is correct at a fixed prefix, but ordinary backpropagation through KL on sampled, fixed prefixes supplies only the first term. Consequently, it does not by itself establish optimization of the student-dependent sequence objective. Appendix D additionally describes a single-sample log-ratio reward implementation, while Section 3 and Appendices G.4/H.2 describe exact full-vocabulary KL. These descriptions need to be reconciled through explicit pseudocode specifying sampling, stop-gradient operations, and whether future log-ratio rewards affect earlier-token updates. This issue affects the central interpretation of the algorithms.
Questions for authors

How should the global argmin equivalence in Appendix E be corrected when the prefix distribution depends on the student parameters? Which conclusions require only a frozen-prefix surrogate, and which require the full student-dependent objective?

Does reverse-KL training differentiate exact vocabulary-level KL on fixed sampled prefixes, use sampled log-ratio policy-gradient updates, or combine both?

What are the exact horizons, entropy threshold, stage budgets, transition decisions, stopping rule, gating split, and effective token/compute totals used for Table 3?

Do the curriculum gains persist against matched-budget fixed intermediate horizons, the same ungated schedule, and validation-based early stopping?

What is the precise contribution beyond GKD's distillation framework, and how do the proposed mixtures compare with AKL and entropy-aware on-policy distillation methods such as EOPD? A direct comparison under the same teacher, student, data, and budget would materially affect the assessment of the claimed methodological advance.

7 comments

Q1. The theoretical unification conflates fixed-prefix identities with optimization under a student-dependent prefix distribution.

Response

Thank you for your careful reading and for raising this question. The DAgger correspondence holds for supervised updates on fixed student prefixes, and our methods and empirical findings do not rely on a global argmin equivalence.

  1. Forward KL and soft-label cross-entropy have identical gradients and minimizers when the prefix distribution is fixed during the supervised update. Let μkt\mu_k^t denote the prefix distribution generated by qθkq_{\theta_k}, held fixed while optimizing θ\theta. The expected losses satisfy L~FKL(θ;μk)=L~CE(θ;μk)−Ck\widetilde L_{\mathrm{FKL}}(\theta;\mu_k)=\widetilde L_{\mathrm{CE}}(\theta;\mu_k)-C_k, where Ck=∑tEst∼μktH(pT(⋅∣st))C_k=\sum_t\mathbb E_{s_t\sim\mu_k^t}H(p_T(\cdot\mid s_t)) is constant with respect to θ\theta. The identity also holds for a fixed mixture of historical rollout distributions, establishing the soft-label DAgger supervised subproblem with data aggregation. Using only current-rollout data gives a DAgger-style variant. We will distinguish the rollout parameters θk\theta_k from the optimization parameters θ\theta, state the argmin equivalence in Appendix E under a fixed prefix distribution, and clarify this scope in Section 2.3.

  2. KL mixing, the length curriculum, and the empirical comparisons do not require a global argmin equivalence. KL mixing combines token-level losses, while the length curriculum uses predictive entropy to control the training horizon. Accuracy, entropy, response length, and downstream RL performance are measured directly from the trained checkpoints.

Q2. Is the sequence-level interpretation of reverse KL consistent with the actual training update?

Response

Thank you for your careful reading and for asking us to clarify the theoretical interpretation and implementation. Our RL-style connection follows from the token-level gradient identity, and the implementation comparisons distinguish fixed-prefix updates from full sequence-gradient estimators.

  1. Our token-level reverse-KL update has an explicit policy-gradient interpretation and follows the OPD mechanism of supervising student-visited prefixes. Proposition 1(ii) shows that, at a fixed prefix, the reverse-KL descent direction equals the policy-gradient ascent direction with the teacher–student log-probability difference as scalar reward. Subsequent student-generated prefixes also receive teacher supervision, allowing the student to improve its continuations on the states it visits. GKD [1] adopts this supervision mechanism and distinguishes it from differentiating through the student's sampling process. Fu et al. [2] analyze the bias–variance tradeoff between token-level and sequence-level updates, and their synthetic experiments show that stronger future-reward coupling can increase gradient variance and destabilize training. We will distinguish the sequence-KL decomposition from the token-level policy-gradient identity in Section 2.3 and expand the discussion of these references.

  2. The main experiments directly differentiate full-vocabulary KL, and the implementation controls distinguish gradient targets and sampling variance. At each sampled prefix, we compute reverse KL over the full vocabulary and backpropagate through the student distribution. Sampled prefixes and teacher outputs are held fixed, and the main implementation uses no additional return-to-go policy-gradient term. Controls comprise sampled-token policy gradients, sequence-level updates using cumulative future log-ratios, and full-vocabulary KL with a prefix-distribution correction. Table 1 uses an FP64 autoregressive model with vocabulary size 3, horizon 4, and seed 42. We enumerate all 81 trajectories to compute each estimator's expected gradient and variance under the same student distribution. Distances use the relative L2 norm against each reference gradient, and variance is Eτ∣g^(τ)−Eτg^(τ)∣22\mathbb E_\tau\\|\hat g(\tau)-\mathbb E_\tau\hat g(\tau)\\|_2^2 over one sampled trajectory. Table 2 uses the same instruction-tuned Qwen3-4B teacher, Qwen3-0.6B-Base student, 4096-token training horizon, 1000 steps, and the evaluation protocol in the main paper. We will align Appendix D, Section 3, and Appendices G.4/H.2 with the sampling, stop-gradient, and update steps in Part 3.

Table 1. Gradient targets and sampling variance in the enumerable model. Distances compare expected gradients; variance is normalized to full-vocabulary token-level KL.

ImplementationDistance to frozen-prefix gradientDistance to sequence gradientRelative variance
Full-vocabulary token-level KL1.51e−160.1491.00
Sampled-token policy gradient1.65e−160.1493.33
Full-vocabulary KL + prefix correction0.1554.36e−163.24
Sequence-level policy gradient0.1552.35e−1611.48

Table 2. Implementation comparison under matched training settings. Metrics are means across four mathematical benchmarks.

ImplementationAvg@k (%)Pass@k (%)Mean response length
Full-vocabulary token-level KL (our implementation)36.448.38,185
Sampled-token policy gradient35.847.58,100
Full-vocabulary KL + prefix correction35.347.07,950
Sequence-level policy gradient34.746.17,900

Table 1 confirms the respective gradient targets and shows that full-vocabulary token-level KL has the lowest variance, while sequence-level policy gradients have 11.48× higher variance in this diagnostic. Our full-vocabulary implementation achieves the highest Avg@k and Pass@k in Table 2, improving over sampled-token policy gradients by 0.6 and 0.8 percentage points, respectively. Sequence-level policy gradients perform worst, with Avg@k and Pass@k lower by 1.7 and 2.2 percentage points than our implementation. This weaker performance is consistent with the noise introduced by coupling early-token updates to stochastic future log-ratios, as analyzed in [2]. We will add these comparisons to the revised manuscript.

References

[1] R. Agarwal et al. On-policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

[2] Y. Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv:2603.25562, 2026.

Q2 (continued). Is the sequence-level interpretation of reverse KL consistent with the actual training update?

Response

The main experiments directly differentiate full-vocabulary KL, and the implementation controls distinguish gradient targets and sampling variance. At each sampled prefix, we compute reverse KL over the full vocabulary and backpropagate through the student distribution. Sampled prefixes and teacher outputs are held fixed, and the main implementation uses no additional return-to-go policy-gradient term. Controls comprise sampled-token policy gradients, sequence-level updates using cumulative future log-ratios, and full-vocabulary KL with a prefix-distribution correction.

The pseudocode specifies the four updates for one fixed-length trajectory, with batch reduction omitted. sg denotes stop-gradient, suffix_sum accumulates from the current position to the end, and c is a KL cost. All losses use gradient descent.

y = sg(sample_student(prompt))
s = prefixes(prompt, y)
logp = sg(teacher_log_probs(s))
logq = student_log_probs(s)

K = sum_over_vocab(exp(logq) * (logq - logp))
a = gather_generated_token(logq, y)
c = sg(gather_generated_token(logq - logp, y))

loss_full_vocab = sum(K)
loss_token_pg = sum(c * a)
loss_sequence_pg = sum(suffix_sum(c) * a)

# Add only the future-state correction to the exact local KL gradient.
future_K = sg(suffix_sum(K) - K)
loss_combined = loss_full_vocab + sum(future_K * a)

Q3. Please specify the curriculum schedule, gating rules, and effective training budget used for main-paper Table 3.

Response

Thank you for asking us to specify these details. The entropy gate controls horizon growth within a fixed 1,000-update budget, and the resource accounting includes both training and gate evaluation.

  1. The horizon increases only when the entropy gate passes; after a failed gate, training continues at the last stable horizon until the budget is exhausted. We use the configuration in Table 1 for both teacher sizes. At the end of each stage, we compute the student's mean per-token predictive entropy on the fixed gating split and advance only if Hm≥Hmin⁡=0.06H_m\geq H_{\min}=0.06. In both teacher settings, the first five stages use horizons 128, 256, 512, 1024, and 2048 for 120 updates each. The gate fails at the end of the 2048-token stage, after update 600, and the remaining 400 updates use the last stable horizon of 1024 tokens.

Table 1. Curriculum and gating configuration.

SettingValue
Candidate horizons128 → 256 → 512 → 1024 → 2048 → 4096
Updates per stage / gate interval120
Regular evaluation interval30 updates
Entropy threshold Hmin⁡H_{\min}0.06
Total updates1,000
Batch size32
Gating split128 held-out DeepScaleR prompts, seed 42, reused at every gate check
Failed gateContinue at the last stable horizon

The gating prompts are excluded from training. The schedule is implemented as follows, with the same model and optimizer carried through the stages.

horizons = [128, 256, 512, 1024, 2048, 4096]
last_stable = horizons[0]
used = 0
for horizon in horizons:
    train(horizon=horizon, steps=120)
    used += 120
    H = mean_student_entropy(gating_prompts, horizon=horizon)
    if H < 0.06:
        break
    last_stable = horizon
train(horizon=last_stable, steps=1000 - used)
  1. Each run completes 1,000 updates, and total GPU-hours include generation, teacher scoring, student optimization, and gate evaluation. Table 2 reports the effective training response tokens and total GPU-hours on a single H100 GPU with 94 GB of memory. Effective tokens count non-padding response tokens participating in the student KL loss, excluding prompt and gating-evaluation tokens. The gating overhead is included in total GPU-hours.

Table 2. Effective training response tokens and total GPU-hours.

TeacherEffective training response tokensTotal GPU-hours
Qwen3-4B26.0M12.5
Qwen3-8B26.5M16.0

We will add the configuration, stage transitions, pseudocode, and resource accounting to Section 5.2 and Appendix G.

Reviewer_D1f1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper decomposes LLM distillation along two axes: the source of training prefixes (teacher/student-generated) and the direction of token-level KL divergence (forward/reverse). The authors compare the four resulting objectives in both standalone distillation and subsequent RL training. Based on the observed trade-offs among accuracy, entropy, computational cost, and training stability, they propose two methods: KL mixing and an entropy-gated length curriculum.

Critical strengths

The four-way comparison is useful, especially because the authors also run RL from the distilled checkpoints. Figures 3 and 5 show that a model with higher accuracy immediately after distillation can perform worse after GRPO. This part is interesting.

Critical weaknesses

The proposed methods are sensitive to their hyperparameters (Figure 7), but the paper didn't show how to set them. In Figure 7, only the forward-heavy mixture (λ=0.2\lambda = 0.2) works well, while other mixtures still show length inflation, but the authors didn't say how to set the λ\lambda beforehand. For the curriculum, the threshold Hmin⁡H_{\min} is neither reported nor ablated, so its sensitivity is unknown.

Questions for authors
  1. Recent studies, including Fast and Effective On-policy Distillation from Reasoning Prefixes (arXiv:2602.15260), Are Full Rollouts Necessary for On-Policy Distillation? (arXiv:2605.31490), and Less is More: Early Stopping Rollout for On-Policy Distillation (arXiv:2605.27028), have shown that shorter rollouts can achieve comparable or even better performance than full-length OPD. However, this paper finds that longer distillation generally improves accuracy. Could the authors explain this discrepancy? In particular, could the difference be due to the 128 token setting being too short?

  2. In Table 2, the 4096 token reverse-KL students have average response lengths of 8,189–8,192 tokens on benchmarks, could the authors explain why this happen? It implies all the reponses reaches the max response length, could the author share what is the trajectory looks like? How to parse the answer in this setting? Is the gain in Table 3 partly come from avoiding truncation rather than from better reasoning?

  3. How can we tell whether gradually increasing the training length is what helps? The curriculum is compared only with fixed 4096-token training, so it is unclear whether the entropy gate actually matters. Could the authors add an ablation, for example, the same length schedule without the entropy gate, or fixed training at the curriculum's final length, to show that entropy gating is responsible for the improvement?

3 comments

W (λ). How can λ be set beforehand, and how sensitive is KL mixing to its choice?

Response

Thank you for asking about coefficient selection. In the studied long-horizon setting, KL mixing is insensitive to moderate changes over λ=0.15-0.30, supporting λ=0.20 as a practical default. Here λ is the reverse-KL weight, so the default assigns 80% weight to forward KL. Section 5.1 identifies the importance of substantial forward-KL weight for stabilizing long-sequence distillation; the relevant sensitivity test therefore examines neighboring values within this regime. Table 1 compares λ=0.15, 0.20, 0.25, and 0.30 using the instruction-tuned Qwen3-4B teacher and Qwen3-0.6B-Base student on DeepScaleR, with student prefixes, a 4096-token horizon, batch size 32, 1000 updates, and the same evaluation and subsequent GRPO protocols. Across this range, the maximum differences in Avg@k, Pass@k, and post-RL Avg@k are only 0.3, 0.4, and 0.3 percentage points, respectively. Entropy remains between 0.272 and 0.326, and mean response length varies from 3860 to 4320 tokens, showing gradual changes rather than abrupt length inflation. We will add this sensitivity analysis and the default coefficient recommendation to Section 5.1.

Table 1. Sensitivity to the reverse-KL weight in 4096-token distillation after 1000 updates. Accuracy metrics are percentages; results use the four-benchmark evaluation.

Reverse-KL weight λAvg@k ↑Pass@k ↑EntropyMean response lengthPost-RL Avg@k ↑
0.1537.052.10.326386040.3
0.2037.152.00.305398040.4
0.2536.851.80.290410040.2
0.3036.951.70.272432040.1

W (entropy threshold). What threshold is used, and how sensitive is the curriculum to its choice?

Response

Thank you for asking us to specify the threshold and its sensitivity. We use 0.06 as the default entropy threshold; in the studied setting, performance is insensitive to moderate changes over 0.05-0.08. At each stage end, we compare the student's mean per-token predictive entropy on fixed held-out prompts with the threshold and increase the horizon only if the gate passes. After a failed gate, training continues at the last stable horizon until 1000 updates are complete. Table 2 varies the threshold using the instruction-tuned Qwen3-4B teacher and Qwen3-0.6B-Base student on DeepScaleR with student-prefix reverse KL, keeping the candidate horizons, stage budgets, optimizer, batch size 32, four-benchmark evaluation, and subsequent GRPO protocol fixed. Within 0.05-0.08, Avg@k ranges from 39.8 to 40.1 and post-RL Avg@k from 40.6 to 40.8, with maximum differences of only 0.3 and 0.2 percentage points, respectively. Mean response length remains between 2630 and 2840 tokens, while final evaluation entropy ranges from 0.074 to 0.107, supporting a useful neighborhood around the default rather than requiring a precisely tuned threshold. We will report the default threshold, gating rule, and sensitivity analysis in Section 5.2 and Appendix G.

Table 2. Sensitivity to the entropy threshold after 1000 updates. Accuracy metrics are percentages; entropy is evaluated after training.

Entropy thresholdAvg@k ↑Final evaluation entropyMean response lengthPost-RL Avg@k ↑
0.0339.20.042510037.8
0.0539.80.074284040.6
0.0640.00.085268840.8
0.0740.10.096266040.7
0.0839.90.107263040.6
0.0938.90.118260340.3

Q1. Why do our length findings differ from short-rollout OPD studies, and could the difference arise because 128 tokens is too short?

Response

Thank you for asking us to reconcile these findings. These studies demonstrate the value of truncating rollouts for efficient distillation in their settings; they do not establish that shorter rollouts have higher accuracy across all settings, so their results are compatible with ours. Our study uses instruction-tuned Qwen3-4B/8B teachers and a Qwen3-0.6B-Base student on DeepScaleR, with 1000 distillation updates. Fast and Effective [1] uses Qwen3-1.7B-Base/8B-Base students; its Section 5.1 explicitly finds that shorter prefixes are advantageous under limited compute, while longer prefixes can achieve higher accuracy with sufficient training through prefix scheduling. Are Full Rollouts Necessary [2] studies reasoning-trained 1.5B students and shows that progressive or truncated rollouts retain comparable performance at substantially lower training cost. ESR [3] studies larger students, primarily with LoRA, and reduces rollout cost while improving training stability. We will clarify these differences in Section 4.3 and the related work.


Q2 (generation behavior and answer parsing). Why do the 4096-token reverse-KL students generate responses near the 8192-token limit, what do these trajectories look like, and how are their answers parsed?

Response

Thank you for asking us to clarify the generation behavior and scoring. The affected fixed-4096 reverse-KL outputs repeat until the evaluation cap without producing an extractable final answer. The 4096-token limit refers to distillation training, while benchmark evaluation allows up to 8192 generated tokens. In these repetition-degenerate outputs, repeated content continues to the cap without a valid \boxed{} final answer, explaining their near-limit lengths. We use Math-Verify [4] to extract and parse mathematical answers and check their equivalence to the reference answers. An output from which no valid final answer can be extracted is scored as incorrect; when extraction succeeds, correctness is determined by verification against the reference. This description concerns the repetition-degenerate outputs, rather than every evaluated response. We will add representative output excerpts showing the repeated spans, endings, and extraction results, and specify the answer-extraction and scoring procedure in Section 3 and the experimental appendix.


Q2 (gain attribution). Is the gain in main-paper Table 3 partly from avoiding truncation rather than from better reasoning?

Response

Thank you for asking about the source of the improvement. Avoiding repetitive degeneration and reliably completing a solution with a valid final answer are themselves improvements in reasoning performance. The issue in the affected 4096-token reverse-KL students is that generation becomes trapped in repetition and continues to the 8192-token evaluation cap without an extractable final answer. This behavior prevents the model from completing the mathematical task. With the same generation cap and Math-Verify scoring procedure, a model that avoids such loops and produces a verifiably correct answer demonstrates better end-to-end reasoning performance. The improvement therefore includes the ability to sustain productive reasoning and reach a correct conclusion, rather than merely a change in whether the evaluator truncates the response. We will clarify this interpretation alongside the repeated-output examples and answer-scoring protocol in the revised manuscript.

References

[1] D. Zhang et al. Fast and Effective On-policy Distillation from Reasoning Prefixes. arXiv:2602.15260, 2026.

[2] Y. Zhang et al. Are Full Rollouts Necessary for On-Policy Distillation?. arXiv:2605.31490, 2026.

[3] Z. Zhou et al. Less is More: Early Stopping Rollout for On-Policy Distillation. arXiv:2605.27028, 2026.

[4] Hugging Face. Math-Verify.

Q3. Does entropy gating improve over the same length schedule without gating and fixed training at the curriculum's final horizon?

Response

Thank you for asking us to isolate the effect of entropy gating. The entropy-gated curriculum outperforms both the same schedule without gating and fixed training at its final stable horizon under matched training steps. We compare the instruction-tuned Qwen3-4B teacher and Qwen3-0.6B-Base student using student-prefix reverse KL, the same data, optimizer, evaluation, and downstream RL protocol, with 1000 updates and batch size 32. The ungated control follows the same candidate horizons and stage budgets but continues expanding regardless of entropy; Fixed-1024 uses the curriculum's final stable horizon of 1024 throughout training. Relative to the ungated schedule, the curriculum improves Avg@k, Pass@k, and post-RL Avg@k by 1.5, 1.9, and 3.0 percentage points; relative to Fixed-1024, the gains are 2.4, 2.9, and 3.4 points, respectively (Table 1). To isolate the intervention, we also branch from the same checkpoint at update 600, when entropy is below 0.06, and run each branch for the remaining 400 updates. Continuing the predetermined increase yields Avg@k 38.5 and mean response length 5760, whereas returning to the last stable horizon of 1024 yields 40.0 and 2688. We will report these controls and the checkpoint-branching comparison in Section 5.2 and the experimental appendix.

Table 1. Controls for entropy gating after 1000 updates. Accuracy metrics are percentages.

MethodAvg@k ↑Pass@k ↑Mean response length ↓Post-RL Avg@k ↑
Fixed-102437.651.2552037.4
Same schedule, no entropy gate38.552.2576037.8
Entropy-gated curriculum40.054.1268840.8
Reviewer_Y2oVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a unified view of SFT, DAgger-style on-policy SFT, offline-RL-style distillation, and OPD by decoupling prefix source and token-level KL direction.

The authors conduct experiments to investigate the four combinations of prefix source and KL direction. Their results suggest that reverse KL can lead to higher average accuracy while reducing entropy, whereas forward KL can preserve entropy. Student prefixes yield better performance, whereas teacher prefixes offer greater computational efficiency. Long-sequence distillation can improve accuracy but cause severe length inflation, whereas short-sequence distillation is more stable but less accurate.

Motivated by these findings, the authors propose two methods: (1) mixing forward and reverse KL, and (2) an entropy-gated length curriculum in which the training horizon increases only when the current predictive entropy exceeds a threshold.

The authors evaluate both methods and find that KL mixing can help avoid entropy collapse while preserving accuracy, and that the entropy-gated length curriculum can generally improve Avg@k and Pass@k while substantially reducing response length.

Critical strengths
  1. The experimental design is informative. The authors conduct useful controlled comparisons across different settings, helping clarify the effects of various experimental factors on model behavior.

  2. Simple methods show promising results. KL mixing offers a balance between generation accuracy and stability, while the entropy-gated length curriculum provides a way to improve accuracy and substantially reduce response length.

Critical weaknesses
  1. The novelty is limited. GKD has already explored varying the data source and mixing divergences.

  2. The comparison between teacher and student prefixes has a limited connection to the proposed methods.

  3. The experimental evaluation of the proposed methods is insufficient. For example, comparing the entropy-gated curriculum with a schedule that progressively increases the training length without entropy gating would help determine whether the entropy gate itself is useful.

Questions for authors
  1. Could the authors clarify how this work differs in emphasis from GKD, which has already explored different data sources and mixed divergence objectives?

  2. Is there a comparison between the entropy-gated curriculum and an ungated schedule that progressively increases the training length? If such results are available, could the authors report them to clarify whether entropy gating provides additional improvements?

2 comments

W1 / Q1. How does this work differ in emphasis from GKD, which already explores different data sources and mixed divergence objectives?

Response

Thank you for asking us to clarify our contribution. Our work unifies four distillation objectives through a common learning interpretation and, to our knowledge, provides the first controlled study of how these objectives affect subsequent RL in reasoning models. We reveal three tradeoffs linking accuracy, entropy, compute, and stability, and translate these findings into KL mixing and an entropy-gated length curriculum.

  1. Our framework explains the conventional distillation pairings and unifies the learning interpretations of all four objectives. GKD [1] allows flexible choices of data source and divergence. We derive the conventional teacher-prefix/forward-KL and student-prefix/reverse-KL pairings from sequence-KL decomposition, then separate these choices at the token level. Fixed-prefix gradient identities connect forward KL to soft-label cross-entropy and reverse KL to a policy-gradient form with dense teacher–student log-ratio feedback. Together with prefix source, these identities connect the four objectives to off-policy SFT, DAgger-style supervision, offline-RL-style distillation, and OPD.

  2. Our controlled study reveals how distillation choices shape long-reasoning behavior and subsequent RL trainability. GKD studies quality–diversity tradeoffs and joint RL–distillation optimization; we evaluate all four distilled initializations under the same subsequent GRPO protocol and characterize their accuracy, entropy, and response-length dynamics. KL direction induces an accuracy–entropy tradeoff: reverse KL improves standalone accuracy but concentrates the student distribution, whereas forward KL better preserves entropy and diversity. Prefix source induces a quality–compute tradeoff: student prefixes improve accuracy under matched training steps, whereas teacher prefixes reduce cost through reusable offline trajectories and cached logits. Training length induces an accuracy–stability tradeoff: longer horizons improve accuracy but can induce entropy collapse and length inflation under reverse KL, whereas shorter horizons are more stable but less accurate. Crucially, standalone and post-RL rankings can reverse, making continued trainability a central criterion for choosing distillation objectives.

  3. Our methods turn these findings into concrete controls for accuracy, diversity, and stability. KL mixing identifies the importance of substantial forward-KL weight in long-sequence distillation to stabilize entropy and response length while retaining accuracy. The entropy-gated curriculum adjusts the training horizon according to the student's predictive entropy. Relative to fixed-4096 training, it improves Avg@k by 3.6 points, Pass@k by up to 5.8 points, and reduces average response length to approximately one third.

We will clarify these contributions and their distinctions from GKD in the introduction and related work.

References

[1] R. Agarwal et al. On-policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

W2. The comparison between teacher and student prefixes has a limited connection to the proposed methods.

Response

Thank you for asking us to clarify this connection. Our controlled study reveals three connected findings: i) KL direction induces an accuracy–entropy tradeoff: reverse KL improves standalone accuracy, while forward KL better preserves entropy and diversity; ii) prefix source induces a quality–compute tradeoff: student prefixes improve accuracy under matched training steps, while teacher prefixes reduce cost through offline reuse; iii) training length induces an accuracy–stability tradeoff: longer horizons improve accuracy but can exacerbate entropy collapse and length inflation. Together, these findings motivate retaining student-prefix supervision while using KL mixing and an entropy-gated length curriculum to balance accuracy, diversity, and stability.

  1. The prefix-source finding supports retaining student-prefix supervision in both methods. With KL direction, horizon, and training steps fixed, student prefixes improve accuracy by providing supervision on states the student encounters (Section 4.2). Both methods retain this supervision setting.

  2. KL mixing directly addresses the accuracy–entropy tradeoff identified by the KL-direction comparison. Reverse KL improves standalone accuracy but reduces entropy, while forward KL better preserves entropy and diversity (Section 4.1). Mixing the two objectives combines these benefits, with substantial forward-KL weight stabilizing entropy and response length during long-sequence distillation (Section 5.1).

  3. The entropy-gated length curriculum directly addresses the accuracy–stability tradeoff identified by the training-length comparison. Longer horizons improve accuracy but can exacerbate entropy collapse and length inflation under reverse KL, while shorter horizons are more stable but less accurate (Section 4.3). The curriculum therefore increases the horizon only while predictive entropy remains at or above the threshold; after a failed gate, training continues at the last stable horizon, preserving further learning without continuing horizon expansion (Section 5.2).

We will make this connection between the prefix-source analysis and method design explicit in Sections 4.2 and 5.


W3 / Q2. Does entropy gating provide additional improvements over an ungated schedule that progressively increases the training length?

Response

Thank you for requesting this direct comparison. Entropy gating improves distillation accuracy and downstream RL performance while reducing training time relative to an ungated progressive-length schedule. We compare the instruction-tuned Qwen3-4B teacher and Qwen3-0.6B-Base student using student-prefix reverse KL, the same data, optimizer, decoding, and downstream RL protocol. Both runs use 1000 updates, batch size 32, and the same candidate horizons and stage progression; the ungated control continues increasing the horizon regardless of entropy, while our curriculum returns to the last stable horizon after a failed gate. Compared with the ungated schedule, our curriculum improves Avg@k, Pass@k, and post-RL Avg@k by 1.5, 1.9, and 3.0 percentage points, respectively (Table 1). Completing the same 1000 updates takes 12.5 versus 16.0 GPU-hours, a 21.9% reduction in training time, including monitoring. To isolate the intervention, we also branch from the same checkpoint at update 600, when entropy is below 0.06, and run each branch for the remaining 400 updates. Continuing the predetermined increase yields Avg@k 38.5 and response length 5760, whereas returning to the last stable horizon of 1024 yields 40.0 and 2688. This comparison supports using the student's current state to decide when to stop increasing the horizon and continue learning at a stable length. We will add the ungated control, checkpoint-branching comparison, and training costs to Section 5.2 and the experimental appendix.

Table 1. Ungated scheduling versus the entropy-gated curriculum after 1000 updates. GPU-hours include monitoring and exclude downstream RL.

MethodGPU-hours ↓Avg@k ↑Pass@k ↑Mean response length ↓Post-RL Avg@k ↑
Same schedule, no entropy gate16.038.552.2576037.8
Entropy-gated curriculum12.540.054.1268840.8
Reviewer_YvzeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper separates prefix source from token-level KL direction in LLM distillation, producing four objectives. Fixed-prefix gradient identities connect these objectives to SFT, DAgger-style imitation, offline-RL-style optimization, and OPD. Experiments with a Qwen3-0.6B student compare two teacher sizes, short and long training horizons, and subsequent GRPO. Based on observed accuracy, entropy, and response-length tradeoffs, the paper proposes forward/reverse KL mixing and an entropy-gated length curriculum.

Critical strengths
  1. Evaluating distillation as an initialization for subsequent RL is practically useful. The paper shows that stronger standalone distillation performance does not necessarily imply better downstream RL behavior. Jointly tracking accuracy, entropy, and response length provides a more informative evaluation than endpoint accuracy alone.

  2. The experimental matrix is systematic. Four objectives, two teacher sizes, two training horizons, and RL follow-up experiments provide a useful comparative study. Full-vocabulary KL computation also reduces confounding from top-k approximations.

Critical weaknesses
  1. The novelty of the central framework is overstated, and comparisons with closely related work are insufficient. GKD already treats the student-data fraction and divergence as separately selectable factors and studies forward/reverse KL and quality–diversity tradeoffs. The broad claim that the off-diagonal objectives have not been systematically studied is therefore difficult to sustain. The contribution could instead lie in the specific long-reasoning and downstream-RL analysis.

  2. The practical contribution is insufficiently specified and isolated. Section 5.2 does not provide concrete values for HminH_{min}, the horizon sequence, steps per stage, or whether training terminates or continues after a failed gate. There are also no matched fixed-intermediate-horizon or ungated-schedule controls, so Table 3 does not isolate entropy gating from ordinary horizon selection.

Questions for authors
  1. Which objectives, analyses, or empirical findings are new relative to GKD?

  2. Can the authors narrow the claim about previously unstudied off-diagonal objectives and directly compare with existing entropy-aware combinations of KL objectives?

  3. Are the teachers base or instruction-tuned models?

  4. Is reverse KL optimized through an exact full-vocabulary gradient, a single-sample REINFORCE estimator, or different implementations across experiments?

  5. What exact horizon schedule, entropy threshold, checking frequency, and stopping rule produced Table 3?

4 comments

W1 / Q1 / Q2 (novelty claim). Which objectives, analyses, or empirical findings are new relative to GKD?

Response

Thank you for asking us to clarify our contribution. Our work unifies four distillation objectives through a common learning interpretation and, to our knowledge, provides the first controlled study of how these objectives affect subsequent RL in reasoning models. We reveal three tradeoffs linking accuracy, entropy, compute, and stability, and translate these findings into KL mixing and an entropy-gated length curriculum.

  1. Our framework explains the conventional distillation pairings and connects the four objectives to distinct learning regimes. GKD [1] allows independent choices of data source and divergence. Our framework derives the conventional teacher-prefix/forward-KL and student-prefix/reverse-KL pairings from sequence-KL decomposition. Fixed-prefix gradient identities then connect forward KL to soft-label cross-entropy and reverse KL to a policy-gradient form with dense teacher–student log-ratio feedback. Combining these identities with prefix source connects the four objectives to off-policy SFT, DAgger-style supervision, offline-RL-style distillation, and OPD. Our use of “systematic study” concerns the joint characterization of all four objectives, including the off-diagonal combinations, under a common reasoning-distillation and subsequent-RL protocol. We will make this scope explicit in the introduction and related work.

  2. Our controlled comparisons reveal how distillation choices shape long-reasoning behavior and subsequent trainability. GKD studies quality–diversity tradeoffs and joint RL–distillation optimization; we compare the four distilled initializations under the same subsequent GRPO protocol and examine training-length effects. KL direction induces an accuracy–entropy tradeoff: reverse KL improves standalone accuracy but concentrates the student distribution, whereas forward KL better preserves entropy and diversity. Prefix source induces a quality–compute tradeoff: student prefixes improve accuracy under matched training steps, whereas teacher prefixes reduce cost through reusable trajectories and cached logits. Training length induces an accuracy–stability tradeoff: longer horizons improve accuracy but can induce entropy collapse and length inflation under reverse KL, whereas shorter horizons are more stable but less accurate. Crucially, standalone and post-RL rankings can reverse, making continued trainability a central criterion for choosing distillation objectives.

  3. Our methods turn these findings into concrete designs for balancing accuracy, diversity, and stability. KL mixing identifies the importance of substantial forward-KL weight in long-horizon distillation to stabilize entropy and response length while retaining accuracy. The entropy-gated curriculum adjusts the training horizon according to the student's state. Relative to fixed-4096 training, it improves Avg@k by 3.6 points, Pass@k by up to 5.8 points, and reduces average response length to approximately one third.

References

[1] R. Agarwal et al. On-policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

Q2. Can the authors narrow the claim about previously unstudied off-diagonal objectives and directly compare with existing entropy-aware combinations of KL objectives?

Response

Thank you for asking us to clarify the scope and provide direct comparisons. Regarding the statement about off-diagonal objectives, we clarify that our use of “systematic study” refers to the joint characterization of all four objectives, including the off-diagonal combinations, in a controlled reasoning-distillation and subsequent-RL setting. For the requested comparison with entropy-aware KL combinations, we compare the methods in Table 1 using instruction-tuned Qwen3-4B → Qwen3-0.6B-Base on DeepScaleR with student prefixes, a 4096-token horizon, batch size 32, 1000 updates, and the same evaluation and downstream GRPO protocols. GPU-hours report the resulting distillation cost, including generation, teacher scoring, and student optimization. Our forward-heavy mixture achieves higher Pass@k and post-RL accuracy than AKL and EOPD under matched training steps. With reverse-KL weight λ=0.20\lambda=0.20, Avg@k is 37.1, compared with 37.0 for AKL and 36.9 for EOPD. Pass@k improves by 2.7 points over AKL and 1.4 over EOPD, and post-RL Avg@k improves by 0.7 and 0.3 points, respectively (Table 1). Higher predictive entropy and shorter responses accompany these gains, supporting the accuracy–diversity–stability balance sought in long-horizon distillation. We will make this scope explicit in the introduction and related work and add these comparisons to the revised manuscript.

Table 1. KL-combination methods after 1000 updates. Accuracy and length metrics use the four-benchmark evaluation; entropy is mean student predictive token entropy.

MethodAvg@k ↑Pass@k ↑EntropyMean response length ↓Post-RL Avg@k ↑GPU-hours ↓
Forward KL36.050.00.450294338.413.6
Reverse KL36.448.30.010818530.617.2
GKD, generalized JSD [3]36.250.20.245450039.314.4
AKL [1]37.049.30.220470039.714.5
EOPD [2]36.950.60.270425040.114.1
KL mixing (λ=0.20\lambda=0.20)37.152.00.305398040.414.2

W2 (configuration) / Q5. What exact horizon schedule, entropy threshold, checking frequency, and stopping rule produced main-paper Table 3?

Response

Thank you for asking us to specify the curriculum. The horizon increases only when the entropy gate passes; after a failed gate, training continues at the last stable horizon until 1000 updates are complete. We use the settings in Table 2 for both teacher sizes. At the end of each stage, we compute the student's mean per-token predictive entropy on the fixed held-out prompts and advance only if Hm≥Hmin⁡=0.06H_m \geq H_{\min}=0.06. In both teacher settings, the first five stages use horizons 128, 256, 512, 1024, and 2048 for 120 updates each; the gate fails after update 600, and the remaining 400 updates use the last stable horizon of 1024. The gating prompts are excluded from training. We will specify the schedule, entropy threshold, checking frequency, and failed-gate handling in Section 5.2 and Appendix G.

Table 2. Curriculum and gating configuration.

SettingValue
Candidate horizons128 → 256 → 512 → 1024 → 2048 → 4096
Updates per stage / gate interval120
Entropy threshold Hmin⁡H_{\min}0.06
Total updates1,000
Batch size32
Gating split128 held-out DeepScaleR prompts, seed 42, reused at every gate check
Failed gateContinue at the last stable horizon

References

[1] T. Wu et al. Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. COLING 2025.

[2] W. Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML 2026.

[3] R. Agarwal et al. On-policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.

W2 (controls). main-paper Table 3 does not isolate entropy gating from ordinary horizon selection.

Response

Thank you for asking us to isolate the effect of entropy gating. Our empirical results show that the entropy-gated curriculum outperforms both fixed intermediate horizons and the same length schedule without gating in distillation accuracy and downstream RL performance, while taking less time than the ungated schedule. These comparisons distinguish the benefit of entropy-guided horizon adaptation from choosing a favorable fixed length or simply increasing the length over time. The comparisons use Qwen3-4B → 0.6B student-prefix reverse KL with the same data, optimizer, decoding, and downstream RL protocol. All distillation runs use 1000 updates and batch size 32; GPU-hours report the resulting training cost, including monitoring and excluding downstream RL.

  1. The curriculum outperforms fixed intermediate horizons under matched training steps. Table 1 compares fixed training horizons from 128 to 4096 tokens with the entropy-gated curriculum after 1000 updates. The strongest fixed horizon for standalone Avg@k is 2048, whereas the strongest for downstream RL is 1024. Against these respective baselines, the curriculum improves standalone Avg@k and Pass@k by 0.9 and 1.2 percentage points, and post-RL Avg@k by 3.4 points. Its average response length is 2688, compared with 7250 for fixed-2048. These comparisons support adapting the training horizon beyond selecting a favorable fixed length.

Table 1. Fixed horizons and the entropy-gated curriculum after 1000 updates. Accuracy metrics are percentages.

Training horizon / methodAvg@k ↑Pass@k ↑Mean response length ↓Post-RL Avg@k ↑
Fixed-12834.547.1364233.9
Fixed-51236.049.5432036.3
Fixed-76837.150.7502037.0
Fixed-102437.651.2552037.4
Fixed-153638.652.3642037.3
Fixed-204839.152.9725036.7
Fixed-307238.151.7774034.5
Fixed-409636.448.3818530.6
Entropy-gated curriculum40.054.1268840.8
  1. Entropy gating improves on a predetermined length increase by preventing further horizon expansion when the student becomes unstable. The ungated control follows the same candidate horizons and stage progression but continues increasing the horizon regardless of entropy. It improves over persistent fixed-4096 training, yet remains below the curriculum by 1.5 points in Avg@k and 3.0 points after RL (Table 2). Completing the same 1000 updates takes 12.5 GPU-hours for entropy gating versus 16.0 for the ungated schedule, a 21.9% reduction in training time, including monitoring. To isolate the intervention, we also branch from the same checkpoint at update 600, when entropy is below 0.06, and run each branch for the remaining 400 updates. Continuing the predetermined increase yields Avg@k 38.5 and response length 5760, whereas returning to the last stable horizon of 1024 yields 40.0 and 2688. This comparison supports using the student's current state to decide when to stop increasing the horizon and continue learning at a stable length.

Table 2. Ungated scheduling versus the entropy-gated curriculum after 1000 updates. GPU-hours include monitoring and exclude downstream RL.

MethodGPU-hours ↓Avg@k ↑Pass@k ↑Mean response length ↓Post-RL Avg@k ↑
Same schedule, no entropy gate16.038.552.2576037.8
Entropy-gated curriculum12.540.054.1268840.8

We will add the fixed-horizon and ungated-schedule controls, checkpoint-branching comparisons, and training costs to Section 5.2 and the experimental appendix.


Q3. Are the teachers base or instruction-tuned models?

Response

Thank you for asking us to clarify the teacher model versions. Both teachers are instruction-tuned models: Qwen3-4B and Qwen3-8B. We will explicitly identify their instruction-tuned status in Section 3 and Appendix G.1 and use consistent teacher model names throughout the manuscript.

Q4. Is reverse KL optimized through an exact full-vocabulary gradient, a single-sample REINFORCE estimator, or different implementations across experiments?

Response

Thank you for asking us to clarify the implementation. All main reverse-KL experiments use the same full-vocabulary loss and directly backpropagate through the student distribution. At each collected prefix, we sum reverse KL over the entire vocabulary, holding the prefix and teacher outputs fixed. Thus, the gradient is exact for the token-level KL at a given prefix, while the prefixes themselves are sampled from the teacher or student according to the objective. We use neither a single-sample REINFORCE estimator nor an additional return-to-go policy-gradient term. The policy-gradient form in Proposition 1(ii) is an equivalent gradient interpretation at a fixed prefix, rather than a different implementation used in some experiments. The pseudocode below uses sg for stop-gradient and averages over valid supervised positions. We will make the sampling and gradient computation explicit and consistent in Appendix D, Section 3, and Appendices G.4/H.2.

s = sg(collected_prefixes)
logp = sg(teacher_log_probs(s))
logq = student_log_probs(s)
kl = sum_over_vocab(exp(logq) * (logq - logp))
loss = mean_over_valid_positions(kl)
update_student(loss)  # gradient descent

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.