acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation

Abstract

On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off between objective fidelity and optimization stability. Token-level OPD provides stable but local supervision, whereas sequence-level OPD captures future credit at the cost of horizon-dependent variance. We establish a unified temporal-credit view of these formulations, showing that practical token-level OPD can be interpreted as a temporal approximation to the sequence-level reverse-KL gradient. Building on this connection, we propose OPD, which uses discounted temporal credit assignment to balance long-horizon supervision and optimization stability, while admitting a horizon-independent variance bound. We further develop a reward-compatible bounded mixing (RBM) mechanism for that balances verifiable outcome feedback with the discounted OPD advantage to move beyond purely teacher-dependent optimization. Experiments on mathematical and code reasoning demonstrate consistent improvements over existing OPD methods across vanilla, size-mismatched, and multi-teacher distillation settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On policy distillation, Large language model, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G58cOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a trade-off in on-policy distillation (OPD) between the true reverse KL long-horizon supervision and optimization stability. Token-level OPD provides stable but biased local supervision, while sequence-level OPD accounts for future credit at the cost of horizon-dependent variance. They propose γ\gammaOPD, which introduces a discount factor γ\gamma to interpolate between the two regimes, and prove that the variance of the sequence-level credit can grow quadratically with the remaining horizon, whereas the γ\gammaOPD credit admits a horizon-independent bound for γ\gamma < 1. As a second contribution, they introduce reward-compatible bounded mixing (RBM), which normalizes the discounted credit, bounds it via a softsign transform, and adds a verifiable outcome reward, so that the reward determines the sign of the update while the teacher signal modulates its magnitude.

Critical strengths
  1. Theorem 3.3 which makes the bias-variance trade-off precise: sequence-level credit variance can grow quadratically with the remaining horizon, whereas any γ<1\gamma < 1 admits a horizon-independent bound. Prior OPD methods navigate this trade-off differently, a provable characterization could be a useful reference point given how central OPD has become in post-training.
Critical weaknesses
  1. The main comparisons are confounded, and the ablation suggests the two contributions are largely orthogonal. In Tables 1 and 2, γOPD is evaluated with verifiable rewards, while most baselines (OPD, ExOPD, TOPD) do not use verifier rewards; the headline improvements therefore cannot be attributed to temporal credit assignment, which is the paper's central story. Without (i) γ\gammaOPD-without-RBM rows in the main tables and (ii) RBM applied to the strongest baselines, it is impossible to tell whether the contribution is temporal credit assignment, reward mixing, or their interaction. This is important because it goes to the core of what the paper claims to contribute.

  2. I am questioning the novelty of the contributions and their significance. Both propositions 3.1 and 3.2 are known [1]. Works introducing the reward in distillation also, with a more principled Lagrangian optimization [2] (Eq. 5), [3]. [1] showed you can deal with the variance of the unbiased loss by decomposing the gradient in 2 terms: a local one (where you can even compute the exact gradient over the whole expectation over vocabulary, but you don't have to, if you are memory constrained), and a remaining long-term one. It remains unclear why introducing γ\gamma there is justified in terms of the quantity that is optimized and where does it converges to.

[1] https://arxiv.org/abs/2306.08543

[2] https://openreview.net/forum?id=3zKtaqxLhW

[3] https://arxiv.org/abs/2509.22921

Questions for authors
  1. Does temporal credit assignment alone outperform the strongest distillation baselines (including [1] when using the whole expectation?)? Relatedly, would combining verifiable rewards with the strongest non-vanilla baselines (e.g., ExOPD, AOPD) catch up with γ\gammaOPD?

  2. Theorem 3.3 gives Var≤σ2(1−γ)2Var \leq \frac{ \sigma^2 }{(1 - \gamma)^2}, which at the default γ\gamma = 0.99 seems a vacuous bound (10⁴σ2\sigma^2 ). It is looser than the sequence-level bound unless the remaining horizon exceeds ~100 tokens. As γ also acts as a truncation (credit from the majority of a 16K-token response is effectively discarded), does Theorem 3.3 actually explain the empirical ordering γ=0.99>γ=0.9>γ=0\gamma = 0.99 > \gamma = 0.9 > \gamma = 0, or is the observed benefit of discounting better understood as an implicit truncation of long-horizon credit rather than the variance control captured by the bound?

  3. By construction, sign(Â) = R, so every token in an incorrect response receives negative credit. How do you avoid punishing correct intermediate reasoning steps within failed responses?

3 comments

Thank you for your careful reading and constructive comments. We address your questions below.

[W1] [Q1] — Additional Comparisons

R1:

TL;DR: We add comparisons with MiniLLM, OPDVR, SAF-OPD, and ExOPD + RBM, where γOPD achieves the highest average pass rate and competitive average accuracy using one rollout per prompt.

Thank you for this helpful suggestion. We additionally compare MiniLLM [1], OPDVR [2], SAF-OPD [3], and ExOPD [4] + RBM under the same verl framework to provide a more consistent training setting.

MethodGroup SizeAvg. Acc.Avg. PassAIME24 Acc.AIME24 PassAIME25 Acc.AIME25 PassAMC23 Acc.AMC23 PassMATH500 Acc.MATH500 Pass
OPDVR167.8878.8957.8180.0051.0466.6793.2897.5069.4071.40
MiniLLM167.7881.7954.9083.3353.9673.3392.7397.5069.5573.00
SAF-OPD870.1080.4161.4680.0056.1573.3393.7597.5069.0570.80
ExOPD + RBM168.6579.6760.0080.0052.8170.0092.5897.5069.2071.20
γ\gammaOPD (Ours)169.4982.4160.9486.6754.1766.6792.8997.5069.9578.80

As shown in the table, γOPD achieves the highest average pass rate (82.41%) and the second-highest average accuracy (69.49%), using only one rollout per prompt. In comparison, SAF-OPD achieves a slightly higher average accuracy (70.10%) but requires a rollout group size of 8, leading to approximately 6× longer training time. These results highlight the favorable performance–efficiency trade-off of γOPD. Furthermore, γOPD outperforms ExOPD + RBM in both average accuracy and average pass rate, suggesting that our discounted credit assignment strategy provides additional benefits beyond RBM augmentation.

We sincerely appreciate the reviewer's suggestion to include these closely related baselines. These additional comparisons further demonstrate the effectiveness of γOPD, particularly in terms of pass rate and computational efficiency under a substantially lower rollout budget.

[Q2] — Variance bound and discounting

R2:

TL;DR: The theorem gives a smaller worst-case variance bound for sufficiently long horizons, but it does not establish the observed ordering among discount factors.

Action: We will distinguish the bound from the empirical interpretation in the revised manuscript.

We thank the reviewer for this insightful comment. This distinction is particularly relevant to our reasoning tasks, where response lengths are typically around 8K tokens. For example, with Nt=8000N_t=8000 and γ=0.99\gamma=0.99, the respective upper bounds are approximately 104σΔ210^4\sigma_\Delta^2 and 6.4×107σΔ26.4\times10^7\sigma_\Delta^2, demonstrating a substantial reduction in the worst-case variance bound.

The bound is consistent with the possibility of improved stability, but does not by itself establish the cause of the behavior observed in Figure 4. In particular, the sequence-level variant exhibits substantially larger gradient norms and entropy collapse, whereas sub-unit discount factors lead to more stable optimization dynamics.

We also agree that the empirical ordering γ=0.99>γ=0.9>γ=0\gamma=0.99>\gamma=0.9>\gamma=0 cannot be explained by variance reduction alone. We interpret the empirical advantage of γ=0.99\gamma=0.99 as resulting from a balance between variance control and the preservation of useful temporal credit, rather than variance reduction alone. We will clarify this distinction in the revised manuscript.

[W2] — Novelty of the contributions

R3:

TL;DR: Our contribution is the systematic investigation in long-chain reasoning distillation with LLMs, together with a gradient-level analysis revealing that token-level and sequence-level OPD can have equivalent objective values but differ from the stop-gradient treatment of the autoregressive prefix distribution.

Action: We will narrow the novelty claims and clarify the scope of the analysis.

Thank you for this important comment. We agree that some of the underlying ideas have precedents in prior work, and we will clarify the scope of our contributions.

Regarding Propositions 3.1 and 3.2, MiniLLM [1] has indeed derived a decomposition of the sequence-level reverse-KL gradient into a local vocabulary-level term and a long-term return-to-go term. We do not claim this general gradient decomposition as novel. Instead, our analysis provides an explicit explanation of the gradient difference between practical token-level OPD and sequence-level OPD, particularly arising from the stop-gradient treatment of the autoregressive prefix distribution.

More importantly, our focus is on developing a computationally practical and stable approach for large-scale LLM distillation. Unlike explicitly estimating the local gradient over the full vocabulary or introducing additional model computations, our γ\gammaOPD directly applies discounted future credit exponentially to tokens in standard sampled rollouts. This is intended to balance temporal credit assignment and optimization variance under realistic LLM training constraints.

Regarding the choice of γ\gamma, it controls the effective temporal horizon of credit assignment. In reinforcement learning, discount factors are typically chosen close to 1 when long-term returns are important. This is particularly relevant to long-chain reasoning, where the contribution of an early token may only become evident through rewards received much later in the reasoning trajectory. Therefore, we choose γ\gamma close to 1 to preserve long-range credit signals while gradually discounting distant future contributions. This allows the model to benefit from future-aware supervision without treating all future tokens equally, providing a balance between long-horizon credit assignment and optimization stability.

We thank you again for this helpful suggestion. In the revised manuscript, we will tone down our claims regarding the introduction of the discounted estimator, and relabel Propositions 3.1 and 3.2 as Lemmas 3.1 and 3.2, respectively. Our main emphasis will be on the practical interpretation of the gradient differences, the discounted temporal credit assignment mechanism, and its systematic application to long-chain reasoning distillation.


[Q3] — Credit assignment in incorrect responses

R4:

TL;DR: Our ultimate goal is to improve task performance beyond teacher imitation; we prioritize verifiable rewards as the primary signal determining whether a response should be reinforced or suppressed, while OPD signals modulate the strength of token-level updates.

Thank you for this insightful observation. Indeed, our goal is not to avoid penalizing potentially correct intermediate steps in failed responses, but rather to mitigate their penalties through teacher guidance. Since our ultimate goal is to improve task performance beyond teacher imitation, we prioritize verifiable rewards as the primary signal determining whether a response should be reinforced or suppressed, while OPD signals modulate the strength of token-level updates.

Importantly, without normalization, the potentially large teacher-derived advantage may dominate the reward and even reverse its intended direction. Our bounded normalization prevents this from happening. In the additional two-benchmark ablation, γOPD without reward mixing or normalization has an average accuracy of 55.69%; adding reward mixing without normalization gives 55.74%, while the full configuration reaches 57.56%. These results are consistent with the benefit of combining reward mixing and bounded normalization in this setting.

References

[1] Gu, Y., et al. MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.

[2] Lin, W., et al. On-Policy Distillation with Verifiable Reward. arXiv:2608.24696, 2026.

[3] Ding, Y., et al. SAF-OPD: Stable Advantage Fusion for On-Policy Distillation. arXiv:2607.29209, 2026.

[4] Yang, W., et al. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv:2602.12125, 2026.

Thanks for the additional experiments, which partly address my concerns. However, γ's gain over RBM is mostly on AIME24, the contribution remains incremental given the prior work. I am raising my score to 2.

Reviewer_7jiWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The work interprets token- and sequence-level OPD through temporal credit assignment, proposes gammaOPD using discounted future log-ratios, and introduces RBM to combine bounded teacher guidance with verifier rewards. It evaluates the method on mathematical and code reasoning under vanilla, size-mismatched, and multi-teacher distillation settings.

Critical strengths

gammaOPD and RBM are straightforward to implement and introduce negligible computational overhead, making the approach potentially useful in large-scale OPD pipelines.

Critical weaknesses

The central methodological novelty is not established. gammaOPD is algebraically identical to the discounted estimators already presented by Fu et al. [1] (Eq. 3) and Zhang et al. [2] (Eq. 5), despite both papers being cited without this overlap being acknowledged. RBM also closely resembles SAF-OPD [3] and OPDVR [4], yet these closest alternatives are not directly compared.

References:

[1] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. Yuqiang Fu, et al.

[2] Are Full Rollouts Necessary for On-Policy Distillation? Yaocheng Zhang, et al.

[3] SAF-OPD: Stable Advantage Fusion for On-Policy Distillation.

[4] On-policy Distillation with Verifiable Reward.

Questions for authors

Equation (7) appears algebraically identical to the gamma-discounted estimators in Fu et al. (2026, Eq. 3) and Zhang et al. (2026, Eq. 5), including the same gamma=0/1 endpoints. Could the authors clarify, formula by formula, what is technically new in gammaOPD, and distinguish RBM from the bounded teacher–verifier fusion mechanisms in SAF-OPD and OPDVR?

References:

[1] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. Yuqiang Fu, et al.

[2] Are Full Rollouts Necessary for On-Policy Distillation? Yaocheng Zhang, et al.

3 comments

Thank you for your careful reading and constructive comments. We address your questions below.

[W1] [Q1] — Novelty clarification

R1:

Thank you for carefully pointing this out. We clarify our contributions and distinctions from two perspectives.

TL;DR:

  1. Our contribution is not the introduction of the discounted estimator itself, but its systematic investigation in long-chain reasoning distillation with LLMs, together with a gradient-level analysis revealing that token-level and sequence-level OPD can have equivalent objective values but different gradients because the practical token-level implementation treats the autoregressive prefix distribution as a stop-gradient.
  2. Our RBM introduces a distinct combination of future-aware temporal credit assignment and response-adaptive bounded additive fusion, differing from SAF-OPD's scheduled local-advantage fusion and OPDVR's verifier-conditioned hard gating.

1. Novelty of the Discounted Estimator

Specifically, [1] provides a proof-of-concept validation of the discounted estimator in a one-dimensional continuous-control environment, using a three-layer MLP with approximately 4K parameters without investigating its effectiveness in OPD with LLMs. In [2], Zhang et al. mention the discounted formulation only in the Preliminaries, without providing a corresponding derivation or empirical experiments of the estimator, while their proposed POPD/TOPD methods focus on improving efficiency through varying rollout generation lengths.

In contrast, we systematically investigate the discounted estimator in practical long-chain reasoning distillation with LLMs. In Table 3, γ\gammaOPD improves average accuracy over the vanilla OPD baseline. Beyond this empirical investigation, we provide an explicit gradient-level analysis showing that token-level and sequence-level OPD can have equivalent objective values but different gradients due to the stop-gradient treatment of the autoregressive prefix distribution. This identifies the source of their different temporal credit assignments, beyond simply attributing the difference to the omission of future-token signals [3].

2. Distinction from Concurrent Work

We also acknowledge two concurrent works, SAF-OPD [5] and OPDVR [4], which emerged around August 2026 and independently explored combining teacher supervision with verifiable rewards. Specifically, SAF-OPD [5] combines local token-level OPD advantages with GRPO advantages, employing top-kk sparsification, tanh⁡\tanh compression, and training-stage warm-up/annealing to regulate teacher contributions. OPDVR [4], on the other hand, employs verifier-conditioned ReLU gating to discard teacher signals that conflict with trajectory-level correctness.

In contrast, our RBM operates on γ\gamma-discounted, future-aware token advantages and applies response-wise adaptive normalization followed by softsign compression before additive fusion with the binary verifier reward. Rather than sparsifying or discarding conflicting teacher signals, RBM retains them as bounded, token-dependent modifiers of reward-driven updates. Therefore, our distinctive contribution lies in integrating future-aware temporal credit assignment with response-adaptive bounded additive fusion, distinguishing our approach from SAF-OPD's scheduled local-advantage fusion and OPDVR's verifier-conditioned hard gating.

We thank you again for this helpful suggestion. In the revised manuscript, we will tone down our claims regarding the introduction of the discounted estimator, relabel Propositions 3.1 and 3.2 as Lemmas 3.1 and 3.2, respectively, and explicitly clarify the distinctions between our work and prior studies.

[W1] [Q1] — Additional Comparisons

R2:

TL;DR: We add OPDVR, SAF-OPD, and ExOPD + RBM. Using one rollout per prompt, γOPD achieves the highest average pass rate and competitive average accuracy.

Thank you for the suggestion. We have added comparisons with OPDVR [4], SAF-OPD [5], and ExOPD [3] augmented with RBM.

MethodGroup SizeAvg. Acc.Avg. PassAIME24 Acc.AIME24 PassAIME25 Acc.AIME25 PassAMC23 Acc.AMC23 PassMATH500 Acc.MATH500 Pass
OPDVR167.8878.8957.8180.0051.0466.6793.2897.5069.4071.40
SAF-OPD870.1080.4161.4680.0056.1573.3393.7597.5069.0570.80
ExOPD + RBM168.6579.6760.0080.0052.8170.0092.5897.5069.2071.20
γOPD169.4982.4160.9486.6754.1766.6792.8997.5069.9578.80

As shown in the table, γOPD achieves the highest average pass rate (82.41%) and the second-highest average accuracy (69.49%), using only one rollout per prompt. In comparison, SAF-OPD achieves a slightly higher average accuracy (70.10%) but requires a rollout group size of 8, leading to approximately 6× longer training time. These results highlight the favorable performance–efficiency trade-off of γOPD. Furthermore, γOPD outperforms ExOPD + RBM in both average accuracy and average pass rate, suggesting that our discounted credit assignment strategy provides additional benefits beyond RBM augmentation.

We sincerely appreciate the reviewer's suggestion to include these closely related baselines. These additional comparisons further demonstrate the effectiveness of γOPD, particularly in terms of pass rate and computational efficiency under a substantially lower rollout budget.

References

[1] Fu, Y., et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. COLM 2026.

[2] Zhang, Y., et al. Are Full Rollouts Necessary for On-Policy Distillation? arXiv:2605.31490, 2026.

[3] Yang, W., et al. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv:2602.12125, 2026.

[4] Lin, W., et al. On-policy Distillation with Verifiable Reward. arXiv:2608.24696, 2026.

[5] Ding, Y., et al. SAF-OPD: Stable Advantage Fusion for On-Policy Distillation. arXiv:2607.29209, 2026.

Thank you for your response.

Your response also confirms one point: the Discounted Estimator is not novel, and this work is more of an investigation.

Given this, I will keep my current rating (score: 1).

Reviewer_7amgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the trade-off between token-level and sequence-level on-policy distillation (OPD) for large language models (Section 1–3). The authors first formalize token-level OPD as a prefix-stop-gradient approximation of the sequence-level reverse-KL objective (Section 3.1 Eq. (4a)–(4b)), showing that sequence-level OPD weights each token score by the future return-to-go of log-ratios −∑t′=tTΔt′-\sum _ {t'=t}^T \Delta _ {t'} (Proposition 3.1 Eq. (5)) whereas token-level OPD uses only the immediate log-ratio −Δt-\Delta _ t (Proposition 3.2 Eq. (6)). To interpolate between these two regimes, the paper introduces γ\gammaOPD (Section 3.2 Eq. (7)), which applies a discount factor γ∈[0,1]\gamma \in [0, 1] to future log-ratios (At(γ)≜−∑t′=tTγt′−tΔt′A _ t^{(\gamma)} \triangleq -\sum _ {t'=t}^T \gamma^{t'-t} \Delta _ {t'}) and satisfies a scalar credit-variance bound Var(At(γ))≤σ2/(1−γ)2\mathrm{Var}(A _ t^{(\gamma)}) \le \sigma^2 / (1 - \gamma)^2 for γ∈[0,1)\gamma \in [0, 1) (Theorem 3.3). To combine teacher guidance with binary verifier rewards R(x,y)∈{−1,1}R(\boldsymbol{x}, \boldsymbol{y}) \in \lbrace -1, 1 \rbrace, the authors propose Reward-Compatible Bounded Mixing (RBM) (Section 3.3 Eq. (8)–(9), Appendix D Algorithm 1), which normalizes At(γ)A _ t^{(\gamma)} by its per-response mean absolute value and applies softsign(u)=u/(1+∣u∣)∈(−1,1)\mathrm{softsign}(u) = u / (1 + |u|) \in (-1, 1) so that the final advantage A^t(γ)\widehat{A} _ t^{(\gamma)} preserves the sign of R(x,y)R(\boldsymbol{x}, \boldsymbol{y}). Experiments on Qwen3-4B and Qwen3-1.7B students distilled from Qwen3-4B-Math and Qwen3-4B-Code teachers across same-size, size-mismatched (Section 4.2 Table 1), and multi-teacher settings (Table 2) report consistent aggregate improvements over OPD, ExOPD, REOPOLD, AOPD, and TOPD with negligible compute overhead (0.1064% step time, Appendix F.1 Table 5).

Critical strengths
  1. Clean unification of token-local and sequence-level OPD with a sign-preserving reward fusion rule. Proposition 3.1 and Proposition 3.2 clearly explain how stopping gradients through the prefix distribution truncates future log-ratio credit, and RBM (Eq. (8)) guarantees sign(A^t(γ))=R(x,y)\mathrm{sign}(\widehat{A} _ t^{(\gamma)}) = R(\boldsymbol{x}, \boldsymbol{y}) while preserving relative within-trajectory token modulation at virtually zero additional compute cost (0.1064% step time and 0.00046% memory in Table 5).

  2. Consistent empirical gains across three distillation regimes. Evaluating same-size (4B →\to 4B), size-mismatched (4B →\to 1.7B, Table 1), and multi-teacher (4B-Math + 4B-Code →\to 4B, Table 2) settings shows consistent aggregate improvements over the five included OPD baselines (OPD, ExOPD, REOPOLD, AOPD, TOPD), and all reported table averages and relative improvement percentages check out arithmetically.

Critical weaknesses
  1. Overstated novelty of the discounted estimator (Eq. (7)) and missing comparisons to the closest variance-reduction and advantage-fusion OPD baselines. Discounted future log-ratio credit At(γ)=−∑t′=tTγt′−tΔt′A _ t^{(\gamma)} = -\sum _ {t'=t}^T \gamma^{t'-t} \Delta _ {t'} interpolating between γ=0\gamma = 0 (token-level) and γ=1\gamma = 1 (sequence-level) has already been introduced in prior work cited by the authors (Fu et al., 2026; TOPD, Zhang et al., 2026). Meanwhile, the paper cites SAF-OPD (Ding et al., 2026)—which specifically addresses magnitude mismatch when fusing RLVR rewards with OPD advantages—and control-variate OPD (Oh et al., 2026), yet omits both from the experimental baselines, and does not report full benchmark numbers for γ=1\gamma = 1 with RBM in Tables 1–2.

  2. Incomplete experimental controls, potential validation-on-test tuning on AIME24, narrow model-scale coverage, and missing multi-seed statistics.

    • Potential tuning on AIME24: Figure 4 (p. 8) explicitly labels AIME24 Acc (avg@32) as "validation accuracy" when comparing γ∈{0,0.9,0.99,1.0}\gamma \in \lbrace 0, 0.9, 0.99, 1.0 \rbrace, and the paper never specifies what dataset was passed as VAL_FILE (tested every 10 steps in launcher_common.sh line 230) or whether γ=0.99\gamma = 0.99 and the reported checkpoints were selected on AIME24.
    • Incomplete 232^3 ablation in Table 3: Table 3 (p. 8, evaluated only on AIME24 and AIME25) tests only 5 of the 8 combinations of (γ,Mix,Norm)(\gamma, \text{Mix}, \text{Norm}), omitting (+, -, +) (γ\gammaOPD + bounded normalization without verifier reward), (-, -, +) (Norm only), and (-, +, -) (Mix only), and bundling mean-absolute scaling with softsign\mathrm{softsign}.
    • Scale and domain gaps: Experiments are limited to Qwen3-1.7B and Qwen3-4B with 4B teachers (without clarifying whether student checkpoints are -Base or post-trained, or why enable_thinking=False is set in launcher_common.sh line 177), omitting larger Qwen3-8B/14B/32B scales; code distillation is evaluated only in the 4B multi-teacher setting (Table 2), with no single-teacher or 1.7B code experiments; and all tables report single-seed runs without confidence intervals.
Questions for authors
  1. Differentiation from prior discounted OPD and missing nearest-neighbor baselines: How does the discounted estimator in Eq. (7) (prior to RBM) differ mathematically from the discounted return-to-go OPD formulation in Fu et al. (2026) and TOPD (Zhang et al., 2026)? Can you provide controlled comparisons against SAF-OPD (Ding et al., 2026), control-variate OPD (Oh et al., 2026), and γ=1\gamma = 1 + RBM in Table 1, as well as the missing (+, -, +) (γ\gamma + Norm without reward), (-, -, +), and (-, +, -) rows in Table 3?

  2. Validation split (VAL_FILE), checkpoint selection, and training step counts: Was AIME24 used as VAL_FILE to tune γ=0.99\gamma = 0.99 (Figure 4) or select checkpoints? Please specify the exact dataset used for VAL_FILE, the exact training step at which Tables 1–3 are evaluated (reconciling TOTAL_TRAINING_STEPS=150 in launcher_common.sh with the 40-step curves in Figure 3, 70-step curves in Figure 4, and 100-step profiling in Appendix F.1), and whether Qwen3-4B / Qwen3-1.7B are -Base or post-trained checkpoints.

  3. Code-versus-paper alignment in the Supplementary ZIP: In launcher_common.sh (lines 181–183, 209) and dp_actor.py (lines 942, 977), training uses adv_estimator=grpo with rollout.n=1 and rollout_is=token (rollout_is_threshold=5.0). In the upstream veRL commit you used, what exact values does model_inputs["advantages"] take when rollout.n=1 (is it strictly ±1\pm 1 as required by Eq. (8)), and how much does token-level rollout IS clipping contribute to the stability shown in Figure 3?

3 comments

Thank you for your careful reading and constructive comments. We address your questions below.

[W1] [Q1] — Differentiation from prior discounted OPD

R1:

TL;DR: Prior work has explored discounted estimators. Our contribution lies in systematically investigating discounted credit assignment in practical LLM distillation and analyzing the gradient differences between token-level and sequence-level OPD.

Thank you for pointing this out. We acknowledge that discounted estimators have been discussed in prior work [1, 2]. Their practical behavior in long-chain reasoning distillation with LLMs remains less explored.

Specifically, [1] provides a proof-of-concept validation of the discounted estimator in a one-dimensional continuous-control environment, using a three-layer MLP with approximately 4K parameters. It does not investigate its effectiveness in OPD with LLMs. In [2], Zhang et al. mention the discounted formulation only in the Preliminaries, without providing a corresponding derivation or empirical experiments of the estimator, while their proposed POPD/TOPD methods focus on improving efficiency through varying rollout generation lengths.

In contrast, we systematically investigate the discounted estimator in practical long-chain reasoning distillation with LLMs. In Table 3, γ\gammaOPD improves average accuracy over the vanilla OPD baseline. Beyond this empirical investigation, we provide an explicit gradient-level analysis showing that token-level and sequence-level OPD can have equivalent objective values but different gradients due to the stop-gradient treatment of the autoregressive prefix distribution. This identifies the source of their different temporal credit assignments, beyond simply attributing the difference to the omission of future-token signals [3].

We thank you again for this helpful suggestion. In the revised manuscript, we will tone down our claims regarding the introduction of the discounted estimator, relabel Propositions 3.1 and 3.2 as Lemmas 3.1 and 3.2, respectively, and explicitly clarify the distinctions between our work and prior studies.


[W1] [Q1] — Missing nearest-neighbor baselines

R2:

TL;DR: We add OPDVR, SAF-OPD, and ExOPD + RBM. Using one rollout per prompt, γOPD achieves the highest average pass rate and competitive average accuracy.

Thank you for the suggestion. We have extended our experimental comparisons to include recent methods released around August 2026, OPDVR [4] and SAF-OPD [5], as well as ExOPD [3] augmented with our proposed RBM.

MethodGroup SizeAvg. Acc.Avg. PassAIME24 Acc.AIME24 PassAIME25 Acc.AIME25 PassAMC23 Acc.AMC23 PassMATH500 Acc.MATH500 Pass
OPDVR167.8878.8957.8180.0051.0466.6793.2897.5069.4071.40
SAF-OPD870.1080.4161.4680.0056.1573.3393.7597.5069.0570.80
ExOPD + RBM168.6579.6760.0080.0052.8170.0092.5897.5069.2071.20
γOPD169.4982.4160.9486.6754.1766.6792.8997.5069.9578.80

As shown in the table, γOPD achieves the highest average pass rate (82.41%) and the second-highest average accuracy (69.49%), using only one rollout per prompt. In comparison, SAF-OPD achieves a slightly higher average accuracy (70.10%) but requires a rollout group size of 8, leading to approximately 6× longer training time. These results highlight the favorable performance–efficiency trade-off of γOPD. Furthermore, γOPD outperforms ExOPD + RBM in both average accuracy and average pass rate, suggesting that our discounted credit assignment strategy provides additional benefits beyond RBM augmentation.

We sincerely appreciate the reviewer's suggestion to include these closely related baselines. These additional comparisons further demonstrate the effectiveness of γOPD, particularly in terms of pass rate and computational efficiency under a substantially lower rollout budget.

[W1] [Q2] — Incomplete ablation, and does not report full benchmark numbers for γ=1 with RBM

R3:

TL;DR: We complete the full 232^3 factorial ablation by adding the three missing configurations and reporting full benchmark results for γ=1\gamma=1 with RBM. The results demonstrate the individual contributions and complementary effects of all components, while revealing the instability of γ=1\gamma=1 on long responses and highlighting the necessity of discounting with γ=0.99\gamma=0.99.

We agree that a full factorial ablation provides a more comprehensive assessment of individual components and their interactions. Following your suggestion, we have supplemented all three previously missing configurations: (+,−,+)(+,-,+) (i.e., γ\gammaOPD with bounded normalization but without reward mixing), Mix-only (−,+,−)(-,+,-), and Norm-only (−,−,+)(-,-,+).

γ\gammaMixNormAIME24AIME25Avg.Δ\Delta Avg.
−−−56.4650.8353.650.00
✓−−58.2953.0855.69+2.04
−✓−57.8150.0053.91+0.26
−−✓57.6051.9854.79+1.14
✓✓−58.8152.6755.74+2.09
✓−✓57.1953.0255.11+1.46
−✓✓57.5553.7555.65+2.00
✓✓✓60.9454.1757.56+3.91

Overall, γ\gamma alone provides the largest improvement over vanilla OPD compared with Mix or Norm individually. Combining Mix and Norm yields further gains, demonstrating their complementary effects in RBM. The full combination achieves the best performance, while removing any individual component leads to a performance drop, validating the contributions of all three components and their effectiveness when integrated.

Following the reviewer's suggestion, we also report the full benchmark results for γ=1\gamma=1 with RBM, as shown below. Setting γ=1\gamma=1 reduces average accuracy to 61.11%, falling 5.93 percentage points below vanilla OPD. This result is consistent with the hypothesis that undiscounted accumulation over long responses can amplify fluctuations.

MethodAvg. Acc.Avg. PassAIME24 Acc.AIME24 PassAIME25 Acc.AIME25 PassAMC23 Acc.AMC23 PassMATH500 Acc.MATH500 Pass
Vanilla OPD67.0477.6356.4680.0050.8363.3392.8195.0068.0572.20
γOPD (γ = 1)61.1179.2750.3183.3341.5663.3385.47100.0067.1070.40
γOPD (γ = 0.99)69.4982.4160.9486.6754.1766.6792.8997.5069.9578.80

We appreciate the reviewer's suggestion for further strengthening our ablation study.


[W2] [Q2] — Validation split (VAL_FILE), checkpoint selection, and training step counts

R4:

TL;DR: All methods are trained for 100 steps and selected using AIME24 validation performance. We report results excluding AIME24 to avoid evaluating on the selection set.

Thank you for raising this important point. For a fair comparison, we train all methods for the same number of steps (100 steps) and select the checkpoint with the best validation performance for final evaluation. This evaluation protocol is also consistent with common practice in post-training methods, such as [6] and [7].

We also appreciate your suggestion regarding the evaluation protocol. Since AIME24 is used as the validation set for checkpoint selection, we exclude it from the final evaluation to avoid potential data leakage. γOPD consistently achieves the best overall performance in both settings, which further supports the effectiveness of γOPD compared with existing methods.

Qwen3-4B-Math → Qwen3-4B

MethodAvg. Acc.Avg. PassAIME25 Acc.AIME25 PassAMC23 Acc.AMC23 PassMATH500 Acc.MATH500 Pass
OPD70.5676.8450.8363.3392.8195.0068.0572.20
ExOPD71.5178.3452.5063.3393.1397.5068.9074.20
REOPOLD71.3277.8151.6763.3393.5997.5068.7072.60
AOPD71.7179.0652.5066.6793.2897.5069.3573.00
TOPD71.6977.7451.6763.3393.4497.5069.9572.40
γOPD72.3480.9954.1766.6792.8997.5069.9578.80

Qwen3-4B-Math → Qwen3-1.7B

MethodAvg. Acc.Avg. PassAIME25 Acc.AIME25 PassAMC23 Acc.AMC23 PassMATH500 Acc.MATH500 Pass
OPD57.5667.4331.2540.0076.0292.5065.4069.80
ExOPD57.8168.0831.6743.3376.7292.5065.0568.40
REOPOLD58.4268.2830.8343.3377.7392.5066.7069.00
AOPD59.2469.7132.5043.3379.3895.0065.8570.80
TOPD58.5267.2230.8336.6779.1495.0065.6070.00
γOPD61.3271.6235.0046.6780.8695.0068.1073.20

[Q3] — Code-versus-paper alignment in the Supplementary ZIP

R5:

TL;DR: With one rollout per prompt, the singleton-group implementation yields an advantage approximately equal to the binary verifier reward. All methods use the same rollout importance-sampling configuration.

Action: We will clarify the implementation and the absence of a dedicated importance-sampling ablation in the supplement.

Thank you for carefully examining the implementation details.

All algorithms in our work are implemented based on the ExOPD [3] codebase, which is built upon the upstream veRL commit d62da4950573d7a4b7ef2362337952e7ab59e78d.

Regarding the first concern, standard OPD does not require multiple rollouts per prompt or group-relative comparisons. Therefore, we set rollout.n=1. In the corresponding veRL implementation of GRPO, singleton groups are handled by setting the group means to 0 and the standard deviation to 1. Consequently, when the verifier returns a binary reward r∈−1,+1r\in{-1,+1}, the resulting raw advantage is

A=r−01+10−6≈±1.A=\frac{r-0}{1+10^{-6}}\approx \pm 1.

Thus, the resulting advantage is within 10−610^{-6} of the required ±1\pm 1 verifier-reward signal and is broadcast to all valid response tokens, with padding positions masked out.

Regarding token-level rollout importance sampling (IS), rollout_is_threshold=5.0 is a standard veRL default for correcting the distribution mismatch between the rollout and training policies by clipping excessively large importance weights. Importantly, we apply the same rollout IS configuration to all methods to ensure a consistent experimental setup. The same configuration is also adopted in [3]. Due to limited computational resources, we were unable to conduct additional dedicated training runs to isolate its effect. We will clarify this implementation detail and limitation in the revised supplementary material.


[W2] — Model scale, domains, and seeds

R6:

TL;DR: Our experimental configurations follow established OPD protocols [3] for consistent and fair comparisons, covering representative OPD scenarios.

Thank you for pointing out these limitations. Our experimental settings largely follow established OPD protocols [3] to ensure consistency with prior work. Specifically, we use instruction-tuned Qwen3-1.7B and Qwen3-4B checkpoints as students, and an RL-post-trained Qwen3-4B model as the teacher, with enable_thinking=False to match its non-thinking generation mode, consistent with ExOPD [3]. Importantly, we use the same discount factor γ=0.99\gamma=0.99 across all experimental settings without per-setting tuning. Our experiments cover three representative OPD scenarios: vanilla distillation (4B-to-4B), size-mismatch distillation (4B-to-1.7B), and multi-teacher distillation across mathematical and code reasoning. γ\gammaOPD consistently improves over existing OPD baselines across these settings, demonstrating its effectiveness beyond a specific model scale or reasoning domain.

Although our main results are reported using a single seed, we have additionally verified across different seeds in an independent setting that performance varies only marginally. Due to the substantial computational cost of LLM distillation, we report single-run results, following the experimental practice of related OPD works [3, 7, 8, 9].

References

[1] Fu, Y., et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. COLM 2026.

[2] Zhang, Y., et al. Are Full Rollouts Necessary for On-Policy Distillation? arXiv:2605.31490, 2026.

[3] Yang, W., et al. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv:2602.12125, 2026.

[4] Lin, W., et al. On-policy Distillation with Verifiable Reward. arXiv:2608.24696, 2026.

[5] Ding, Y., et al. SAF-OPD: Stable Advantage Fusion for On-Policy Distillation. arXiv:2607.29209, 2026.

[6] He, B., et al. JustRL: Scaling a 1.5B LLM with a Simple RL Recipe. arXiv:2512.16649, 2025.

[7] Zhang, D., et al. Fast and Effective On-Policy Distillation from Reasoning Prefixes. Findings of ACL 2026.

[8] Xing, X., et al. Trust Region On-Policy Distillation. arXiv:2606.01249, 2026.

[9] Kang, W., et al. AsyncOPD: How Stale Can On-Policy Distillation Be? NeurIPS 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.