- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents β-OPSD, an on-policy self-distillation method motivated by a KL-regularized policy optimization objective. The resulting ideal target is a geometric interpolation between a reference policy and a privileged teacher. In practice, the method interpolates the logits of the current student and a fixed teacher, increases the teacher weight during training, and uses return-to-go signals to propagate subsequent distillation discrepancies to earlier tokens. The evaluation covers mathematical and coding tasks with Qwen3 models ranging from 1.7B to 8B. The method improves over the sampled reverse-KL OPSD baseline in the reported main results. The paper also includes comparisons with alternative objectives and ablations of the target, credit assignment, interpolation endpoints, and weight schedule.
Critical strengths
-
A coherent and relatively simple training design. The paper connects regularized policy optimization with a practical distillation target and combines it with future credit assignment. This provides a useful explanation of how teacher guidance can be moderated without requiring an external correctness reward.
-
A substantial experimental and ablation effort. The evaluation includes both mathematics and coding across three model sizes. The appendix also reports results that qualify the main claims, including cases where forward-KL distillation or local credit assignment performs better. This makes the empirical picture more informative.
Critical weaknesses
-
The methodological advance over closely related work is not sufficiently established. DemoPSD already constructs a geometric teacher–student target, while SR-OPSD also develops a reference-based geometric target. DistIL examines future credit assignment in distillation. β-OPSD differs in its formulation and implementation, but the current discussion and comparisons do not clearly establish what additional insight or practical benefit its particular combination provides. The β interpretation is useful, although the weight schedule appears primarily to be an empirical design choice.
-
The experiments do not yet establish a sufficiently clear practical advantage. On 4B mathematics, the average improvement over the base model is only 0.74 percentage points. On 8B coding, the full-vocabulary forward-KL OPSD baseline reaches 28.44, compared with 26.26 for β-OPSD; the corresponding computational costs are not reported. Moreover, RTG improves the overall coding result only at 4B, while local credit assignment performs better at 1.7B and 8B. Figure 1 supports reduced late-stage degradation, but does not demonstrate an improvement over the baseline’s best checkpoint. Together, these results leave the benefit of the full combination insufficiently established.
Questions for authors
-
What is the main methodological distinction from DemoPSD and SR-OPSD beyond the choice of interpolation endpoints and weighting rule? How does the proposed RTG formulation advance the future-credit-assignment perspective discussed in DistIL?
-
On 8B coding, full-vocabulary forward-KL OPSD outperforms β-OPSD. What are the GPU-hour and memory costs of these two methods under comparable settings? How would their performance compare when each uses an appropriately selected checkpoint?
-
Could you clarify why Figure 1 reports approximately 48.1 versus 46.6 at step 100, whereas Table 4 reports 57.87 versus 56.11 for the stated 4B mathematical evaluation? Also, does the interpolation schedule span the full 200 training steps or the first 100 steps used for the main evaluation?
-
Under a fixed-prefix, unfiltered local-gradient formulation, interpolation with the current student can scale the reverse-KL teacher signal by the teacher weight. Could a baseline with matched update strength explain part of the observed improvement? Given that RTG also reduces overall coding performance at two model sizes, what evidence best isolates the benefit of the full proposed combination?
6 comments
Q2. On Qwen3-8B coding, how do β-OPSD and full-vocabulary forward-KL compare under checkpoint selection, evaluation protocols, GPU-hours, and memory?
Answer: Full-vocabulary forward-KL is better under the original coding protocol; the ordering reverses when public function/class interfaces are included. In controlled measurements, β-OPSD reduces peak allocated update memory by 22.1% but uses 6.5% more GPU-hours.
We added instrumented, matched runs because the original experiments did not record the GPU-memory measurements needed for this comparison. We also re-evaluated the existing checkpoints under a common evaluation protocol; these fresh scores are reported separately from the submitted results.
Under the original protocol, forward-KL performs better at both step 100 and the validation-selected step-50 checkpoint:
| Qwen3-8B LiveCodeBench Pass@1 | β-OPSD | Forward-KL |
|---|---|---|
| Original protocol, step 100 | 25.97% | 28.53% |
| Original protocol, validation-selected step 50 | 26.64% | 28.06% |
| Interface-aware protocol, same selected checkpoints | 45.50% | 43.60% |
The original protocol does not include the public function/class interfaces in the prompt. When we evaluate the same selected checkpoints with these interfaces included, both methods improve substantially and the ordering reverses: β-OPSD reaches 45.50% versus 43.60% for forward KL (+1.90 points).
| Resource measurement | β-OPSD | Forward-KL |
|---|---|---|
| Matched 100-update H200 training, 1K completion cap | 4.58 GPU-hours | 4.30 GPU-hours |
| Controlled update-stage peak allocated memory, 8K padded input | 40.87 GiB | 52.46 GiB |
Regarding resources, β-OPSD reduces update-stage peak allocated memory by 22.1% but uses 6.5% more GPU-hours in the matched run (4.58 vs. 4.30). We therefore claim a memory benefit only, not a training speedup, and will qualify the efficiency claim accordingly. We retain both evaluation protocols.
Our interpolated target is compatible with full-vocabulary forward KL. At each prefix, we interpolate the reference and teacher logits to obtain a distribution over the entire vocabulary. Our current RTG estimator uses the sampled token's log-ratio. The same interpolated distribution can directly replace the teacher distribution in the full-vocabulary forward-KL loss. A DistIL-style variant could likewise use cross-entropies at later prefixes as the return-to-go signal in the full-vocabulary setting (Agrawal et al., Eq. 5). We leave both variants to future work.
Q3. Why do Figure 1 and Table 4 disagree in absolute Qwen3-4B math scores, and does the 0.5-to-0.8 schedule span 100 or 200 steps?
Answer: Figure 1 is a separate 100-step training-dynamics experiment; the main runs have a 200-step budget. The teacher weight anneals over the full run in either setting, so the main step-100 checkpoint is only halfway through its schedule.
Figure 1 presents a separate training-dynamics experiment, rather than the runs underlying Figure 3 and Table 4. It uses Qwen3-4B on OpenThoughts Math 30K, trained for 100 steps on one H200 with seed 42, an effective batch size of 32, and a 1,024-token completion limit.
| Method | Training objective in Figure 1 |
|---|---|
| Vanilla OPSD | Sampled-token reverse-KL loss (Tinker implementation), fixed teacher, no RTG |
| β-OPSD | Current-student reference, teacher-target weight scheduled from 0.5 to 0.8, discounted RTG with |
The absolute scores therefore correspond to different experimental configurations and should not be read as reproducing the main-result scores. Figure 1 illustrates checkpoint-level behavior in this particular setting; the observed post-step-50 degradation is not a claim that vanilla OPSD always degrades after step 50. We will make this distinction explicit and add these settings to the caption.
On the schedule: the teacher weight is annealed from 0.5 to 0.8 over the full training run in both settings. In Figure 1 the run is 100 steps, so the schedule completes at step 100. In the main experiments the run is 200 steps and we report the step-100 checkpoint, so those results are taken midway through the schedule.
Q4. Can scaling the teacher gradient explain interpolation at a fixed prefix, and what does interpolation add when RTG is used? Does RTG consistently help coding?
Answer: Fixed-prefix local-gradient scaling is correct, but under RTG the interpolated target contributes an additional continuation-dependent Rényi-divergence term. A schedule-matched control isolates +1.20 points at 4B and +0.55 at 8B on math; RTG is not consistently helpful on coding.
We agree that, under the reviewer's fixed-prefix assumptions, scaling the teacher signal reproduces the expected local gradient of the interpolated target. The distinction under return-to-go arises because β-OPSD normalizes its target separately at each prefix, and an earlier token determines which later prefixes are visited.
At each training step, we use the current student as the detached reference. Applying softmax to the interpolated student and teacher logits gives the target in Eqs. (7) and (58):
Although the student and teacher distributions each sum to one, their tokenwise geometric average can sum to less than one. The denominator normalizes this average over the vocabulary:
Substituting the normalized target into the sampled log-ratio gives
=w\log\frac{\pi(a\mid h)}{p_T(a\mid h)}+\log Z_w(h).$$ **For a local update**, the log normalizer is the same for every candidate token at a fixed prefix. Its contribution vanishes in the expected local gradient because the expected score gradient under the student is zero. This recovers the reviewer's scaling observation (Appendix B.5, Eq. 60). **For return-to-go**, the difference accumulates along the sampled continuation. Let the superscript T denote the unscaled teacher-only return. By Eq. (11), $$G_t^{\mathrm{interp}}-wG_t^T =\sum_{s=t}^L\gamma^{s-t}\log Z_w(h_s).$$ Each normalizer has a closed form $\log Z_w(h)=-(1-w)D_w(p_T(\cdot\mid h)\|\pi(\cdot\mid h))\leq 0$: a Rényi divergence of order $w$ between teacher and student, which is zero only when the two next-token distributions coincide. The difference above is therefore a discounted sum of teacher–student Rényi divergences along the sampled continuation, computed from the full distributions at each prefix and not from the sampled tokens. For example, suppose token A leads to a prefix where the student and teacher agree, and token B leads to one where they disagree. Matched scaling charges both continuations $w$ times the teacher log-ratio. β-OPSD charges A the same amount but reduces the charge for B by $(1-w)D_w$ at the disagreeing prefix. In expectation over the next token, the per-prefix cost is $\mathrm{KL}(\pi\|q_w)=w\mathrm{KL}(\pi\|p_T)-(1-w)D_w(p_T\|\pi)$ under β-OPSD and $w\mathrm{KL}(\pi\|p_T)$ under matched scaling. Matched scaling thus shrinks the teacher penalty uniformly, while interpolation shrinks it most at prefixes where the privileged teacher is far from the student. This is the sense in which the interpolated target differs from a matched update strength. **β-OPSD gains 1.20 percentage points at 4B and 0.55 at 8B over matched teacher-only RTG.** The matched control uses the same RTG estimator and scales each sampled teacher log-ratio by β-OPSD's teacher weight at that step. Scores are averaged across the three math benchmarks in Table 4. | Method | Qwen3-4B avg@12 (%) | Qwen3-8B avg@12 (%) | |---|---:|---:| | Teacher-only, reweighted local reverse KL | 53.61 | 56.48 | | Teacher-only, reweighted log-ratio + RTG | 56.67 | 59.63 | | **β-OPSD, logit interpolation + RTG** | **57.87 (+1.20)** | **60.18 (+0.55)** | These gains suggest a benefit from prefix-dependent normalization, which β-OPSD incorporates directly by applying log-softmax to the interpolated logits. **RTG on code, and what isolates the combination.** We agree that RTG is not consistently beneficial on LiveCodeBench: it improves the overall result at 4B, while local credit assignment is better at 1.7B and 8B. We do not claim RTG as a contribution in its own right. Future credit appears in prior work (DistIL), and in our derivation it is the estimator prescribed by the sequence-level gradient. Our primary contribution is the β-family and its interpolated target, and the two interact as follows. With local credit, the target acts through the teacher weight $w=1/\beta$, as the reviewer notes. With RTG, the target additionally contributes the Rényi term above, and the matched control isolates that term on math (+1.20 points at 4B, +0.55 at 8B). We will report both credit-assignment variants side by side in the main text and state that the benefit of RTG is domain-dependent.
We thank the reviewer for highlighting the connection between regularized policy optimization and distillation, and for recognizing the value of our experimental efforts. We have answered all four original questions and completed the requested matched-scaling control, matched resource measurements, and checkpoint/protocol comparison. Bottom line: interpolation + RTG has a theoretically distinct prefix-dependent correction, with +1.20/+0.55 points over matched scaling on 4B/8B math; β-OPSD uses 22.1% less update-stage peak allocated memory than full-vocabulary forward-KL in the controlled comparison, but 6.5% more GPU-hours; and the benefit of RTG is domain-dependent. We agree that both geometric targets and future credit have been discussed as well in concurrent related work. Our contribution is to derive a target over complete responses, characterize what changes when we implement it token by token, and identify the signal this implementation propagates along a rollout.
Q1. How does β-OPSD differ methodologically from DemoPSD and SR-OPSD's geometric targets, and from DistIL's future credit assignment?
Answer: β-OPSD starts from the KL-regularized RL objective underlying vanilla OPSD, recovers the sequence-level reverse-KL objective at β=1, and derives the optimal policy over complete responses for general β. Its connection between interpolation and sequence-level RTG produces a distinct prefix-normalization term (Q4).
Our starting question is what KL-regularized RL objective underlies vanilla OPSD. We recover its sequence-level reverse-KL objective at β=1 and derive the optimal policy over complete responses for general β.
Two procedural differences are not in the table. DemoPSD builds its teacher prompt from a correct rollout and skips prompts without one (Alg. 1), whereas β-OPSD supplies the reference solution directly and can train on those prompts. DistIL propagates cross-entropies over many candidate tokens, whereas β-OPSD propagates one log-ratio per position, for the token the student actually sampled.
DemoPSD and SR-OPSD appeared on arXiv on 2 July and 10 August 2026, shortly before the submission deadline, and we regard them as concurrent work. We will expand this comparison in the revision.
Our contribution is a unified view of OPSD and KL-regularized RL that explains how geometric targets and return-to-go credit assignment arise in the same framework. Both ideas appear in related work. We show that vanilla OPSD's sequence-level objective is the β=1 case of a broader policy optimization problem. Solving it for general β yields a geometric target with teacher weight 1/β. The sequence-level distillation gradient then gives the return-to-go update, in which each token receives credit for current and future mismatches.
What is specific to β-OPSD is the interaction of the two. Because the target is normalized at every prefix and credit is propagated along the rollout, the return carries a Rényi-divergence term that neither a local geometric target nor future credit toward the un-interpolated teacher produces (also see our answer to Q4). We agree that the particular schedule (0.5 to 0.8) is an empirical choice: the derivation fixes the form of the target and the meaning of its weight, not how the weight is annealed.