- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses a trade-off in on-policy distillation (OPD) between the true reverse KL long-horizon supervision and optimization stability. Token-level OPD provides stable but biased local supervision, while sequence-level OPD accounts for future credit at the cost of horizon-dependent variance. They propose OPD, which introduces a discount factor to interpolate between the two regimes, and prove that the variance of the sequence-level credit can grow quadratically with the remaining horizon, whereas the OPD credit admits a horizon-independent bound for < 1. As a second contribution, they introduce reward-compatible bounded mixing (RBM), which normalizes the discounted credit, bounds it via a softsign transform, and adds a verifiable outcome reward, so that the reward determines the sign of the update while the teacher signal modulates its magnitude.
Critical strengths
- Theorem 3.3 which makes the bias-variance trade-off precise: sequence-level credit variance can grow quadratically with the remaining horizon, whereas any admits a horizon-independent bound. Prior OPD methods navigate this trade-off differently, a provable characterization could be a useful reference point given how central OPD has become in post-training.
Critical weaknesses
-
The main comparisons are confounded, and the ablation suggests the two contributions are largely orthogonal. In Tables 1 and 2, γOPD is evaluated with verifiable rewards, while most baselines (OPD, ExOPD, TOPD) do not use verifier rewards; the headline improvements therefore cannot be attributed to temporal credit assignment, which is the paper's central story. Without (i) OPD-without-RBM rows in the main tables and (ii) RBM applied to the strongest baselines, it is impossible to tell whether the contribution is temporal credit assignment, reward mixing, or their interaction. This is important because it goes to the core of what the paper claims to contribute.
-
I am questioning the novelty of the contributions and their significance. Both propositions 3.1 and 3.2 are known [1]. Works introducing the reward in distillation also, with a more principled Lagrangian optimization [2] (Eq. 5), [3]. [1] showed you can deal with the variance of the unbiased loss by decomposing the gradient in 2 terms: a local one (where you can even compute the exact gradient over the whole expectation over vocabulary, but you don't have to, if you are memory constrained), and a remaining long-term one. It remains unclear why introducing there is justified in terms of the quantity that is optimized and where does it converges to.
[1] https://arxiv.org/abs/2306.08543
Questions for authors
-
Does temporal credit assignment alone outperform the strongest distillation baselines (including [1] when using the whole expectation?)? Relatedly, would combining verifiable rewards with the strongest non-vanilla baselines (e.g., ExOPD, AOPD) catch up with OPD?
-
Theorem 3.3 gives , which at the default = 0.99 seems a vacuous bound (10⁴ ). It is looser than the sequence-level bound unless the remaining horizon exceeds ~100 tokens. As γ also acts as a truncation (credit from the majority of a 16K-token response is effectively discarded), does Theorem 3.3 actually explain the empirical ordering , or is the observed benefit of discounting better understood as an implicit truncation of long-horizon credit rather than the variance control captured by the bound?
-
By construction, sign(Â) = R, so every token in an incorrect response receives negative credit. How do you avoid punishing correct intermediate reasoning steps within failed responses?
3 comments
[W2] — Novelty of the contributions
R3:
TL;DR: Our contribution is the systematic investigation in long-chain reasoning distillation with LLMs, together with a gradient-level analysis revealing that token-level and sequence-level OPD can have equivalent objective values but differ from the stop-gradient treatment of the autoregressive prefix distribution.
Action: We will narrow the novelty claims and clarify the scope of the analysis.
Thank you for this important comment. We agree that some of the underlying ideas have precedents in prior work, and we will clarify the scope of our contributions.
Regarding Propositions 3.1 and 3.2, MiniLLM [1] has indeed derived a decomposition of the sequence-level reverse-KL gradient into a local vocabulary-level term and a long-term return-to-go term. We do not claim this general gradient decomposition as novel. Instead, our analysis provides an explicit explanation of the gradient difference between practical token-level OPD and sequence-level OPD, particularly arising from the stop-gradient treatment of the autoregressive prefix distribution.
More importantly, our focus is on developing a computationally practical and stable approach for large-scale LLM distillation. Unlike explicitly estimating the local gradient over the full vocabulary or introducing additional model computations, our OPD directly applies discounted future credit exponentially to tokens in standard sampled rollouts. This is intended to balance temporal credit assignment and optimization variance under realistic LLM training constraints.
Regarding the choice of , it controls the effective temporal horizon of credit assignment. In reinforcement learning, discount factors are typically chosen close to 1 when long-term returns are important. This is particularly relevant to long-chain reasoning, where the contribution of an early token may only become evident through rewards received much later in the reasoning trajectory. Therefore, we choose close to 1 to preserve long-range credit signals while gradually discounting distant future contributions. This allows the model to benefit from future-aware supervision without treating all future tokens equally, providing a balance between long-horizon credit assignment and optimization stability.
We thank you again for this helpful suggestion. In the revised manuscript, we will tone down our claims regarding the introduction of the discounted estimator, and relabel Propositions 3.1 and 3.2 as Lemmas 3.1 and 3.2, respectively. Our main emphasis will be on the practical interpretation of the gradient differences, the discounted temporal credit assignment mechanism, and its systematic application to long-chain reasoning distillation.
[Q3] — Credit assignment in incorrect responses
R4:
TL;DR: Our ultimate goal is to improve task performance beyond teacher imitation; we prioritize verifiable rewards as the primary signal determining whether a response should be reinforced or suppressed, while OPD signals modulate the strength of token-level updates.
Thank you for this insightful observation. Indeed, our goal is not to avoid penalizing potentially correct intermediate steps in failed responses, but rather to mitigate their penalties through teacher guidance. Since our ultimate goal is to improve task performance beyond teacher imitation, we prioritize verifiable rewards as the primary signal determining whether a response should be reinforced or suppressed, while OPD signals modulate the strength of token-level updates.
Importantly, without normalization, the potentially large teacher-derived advantage may dominate the reward and even reverse its intended direction. Our bounded normalization prevents this from happening. In the additional two-benchmark ablation, γOPD without reward mixing or normalization has an average accuracy of 55.69%; adding reward mixing without normalization gives 55.74%, while the full configuration reaches 57.56%. These results are consistent with the benefit of combining reward mixing and bounded normalization in this setting.
References
[1] Gu, Y., et al. MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.
[2] Lin, W., et al. On-Policy Distillation with Verifiable Reward. arXiv:2608.24696, 2026.
[3] Ding, Y., et al. SAF-OPD: Stable Advantage Fusion for On-Policy Distillation. arXiv:2607.29209, 2026.
[4] Yang, W., et al. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv:2602.12125, 2026.
Thanks for the additional experiments, which partly address my concerns. However, γ's gain over RBM is mostly on AIME24, the contribution remains incremental given the prior work. I am raising my score to 2.
Thank you for your careful reading and constructive comments. We address your questions below.
[W1] [Q1] — Additional Comparisons
R1:
Thank you for this helpful suggestion. We additionally compare MiniLLM [1], OPDVR [2], SAF-OPD [3], and ExOPD [4] + RBM under the same verl framework to provide a more consistent training setting.
As shown in the table, γOPD achieves the highest average pass rate (82.41%) and the second-highest average accuracy (69.49%), using only one rollout per prompt. In comparison, SAF-OPD achieves a slightly higher average accuracy (70.10%) but requires a rollout group size of 8, leading to approximately 6× longer training time. These results highlight the favorable performance–efficiency trade-off of γOPD. Furthermore, γOPD outperforms ExOPD + RBM in both average accuracy and average pass rate, suggesting that our discounted credit assignment strategy provides additional benefits beyond RBM augmentation.
We sincerely appreciate the reviewer's suggestion to include these closely related baselines. These additional comparisons further demonstrate the effectiveness of γOPD, particularly in terms of pass rate and computational efficiency under a substantially lower rollout budget.
[Q2] — Variance bound and discounting
R2:
We thank the reviewer for this insightful comment. This distinction is particularly relevant to our reasoning tasks, where response lengths are typically around 8K tokens. For example, with Nt=8000 and γ=0.99, the respective upper bounds are approximately 104σΔ2 and 6.4×107σΔ2, demonstrating a substantial reduction in the worst-case variance bound.
The bound is consistent with the possibility of improved stability, but does not by itself establish the cause of the behavior observed in Figure 4. In particular, the sequence-level variant exhibits substantially larger gradient norms and entropy collapse, whereas sub-unit discount factors lead to more stable optimization dynamics.
We also agree that the empirical ordering γ=0.99>γ=0.9>γ=0 cannot be explained by variance reduction alone. We interpret the empirical advantage of γ=0.99 as resulting from a balance between variance control and the preservation of useful temporal credit, rather than variance reduction alone. We will clarify this distinction in the revised manuscript.