acceptodds
Under review as a conference paper at ICLR 2027

β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

11 views

Abstract

On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the β=1 member of a broader policy-optimization family, where β weights the KL penalty anchoring the student to a reference policy. This equivalence turns β from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce β-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of β selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that β-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, reinforcement learning, policy optimization, distillation, post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EaUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents β-OPSD, an on-policy self-distillation method motivated by a KL-regularized policy optimization objective. The resulting ideal target is a geometric interpolation between a reference policy and a privileged teacher. In practice, the method interpolates the logits of the current student and a fixed teacher, increases the teacher weight during training, and uses return-to-go signals to propagate subsequent distillation discrepancies to earlier tokens. The evaluation covers mathematical and coding tasks with Qwen3 models ranging from 1.7B to 8B. The method improves over the sampled reverse-KL OPSD baseline in the reported main results. The paper also includes comparisons with alternative objectives and ablations of the target, credit assignment, interpolation endpoints, and weight schedule.

Critical strengths
  1. A coherent and relatively simple training design. The paper connects regularized policy optimization with a practical distillation target and combines it with future credit assignment. This provides a useful explanation of how teacher guidance can be moderated without requiring an external correctness reward.

  2. A substantial experimental and ablation effort. The evaluation includes both mathematics and coding across three model sizes. The appendix also reports results that qualify the main claims, including cases where forward-KL distillation or local credit assignment performs better. This makes the empirical picture more informative.

Critical weaknesses
  1. The methodological advance over closely related work is not sufficiently established. DemoPSD already constructs a geometric teacher–student target, while SR-OPSD also develops a reference-based geometric target. DistIL examines future credit assignment in distillation. β-OPSD differs in its formulation and implementation, but the current discussion and comparisons do not clearly establish what additional insight or practical benefit its particular combination provides. The β interpretation is useful, although the weight schedule appears primarily to be an empirical design choice.

  2. The experiments do not yet establish a sufficiently clear practical advantage. On 4B mathematics, the average improvement over the base model is only 0.74 percentage points. On 8B coding, the full-vocabulary forward-KL OPSD baseline reaches 28.44, compared with 26.26 for β-OPSD; the corresponding computational costs are not reported. Moreover, RTG improves the overall coding result only at 4B, while local credit assignment performs better at 1.7B and 8B. Figure 1 supports reduced late-stage degradation, but does not demonstrate an improvement over the baseline’s best checkpoint. Together, these results leave the benefit of the full combination insufficiently established.

Questions for authors
  1. What is the main methodological distinction from DemoPSD and SR-OPSD beyond the choice of interpolation endpoints and weighting rule? How does the proposed RTG formulation advance the future-credit-assignment perspective discussed in DistIL?

  2. On 8B coding, full-vocabulary forward-KL OPSD outperforms β-OPSD. What are the GPU-hour and memory costs of these two methods under comparable settings? How would their performance compare when each uses an appropriately selected checkpoint?

  3. Could you clarify why Figure 1 reports approximately 48.1 versus 46.6 at step 100, whereas Table 4 reports 57.87 versus 56.11 for the stated 4B mathematical evaluation? Also, does the interpolation schedule span the full 200 training steps or the first 100 steps used for the main evaluation?

  4. Under a fixed-prefix, unfiltered local-gradient formulation, interpolation with the current student can scale the reverse-KL teacher signal by the teacher weight. Could a baseline with matched update strength explain part of the observed improvement? Given that RTG also reduces overall coding performance at two model sizes, what evidence best isolates the benefit of the full proposed combination?

6 comments

We thank the reviewer for highlighting the connection between regularized policy optimization and distillation, and for recognizing the value of our experimental efforts. We have answered all four original questions and completed the requested matched-scaling control, matched resource measurements, and checkpoint/protocol comparison. Bottom line: interpolation + RTG has a theoretically distinct prefix-dependent correction, with +1.20/+0.55 points over matched scaling on 4B/8B math; β-OPSD uses 22.1% less update-stage peak allocated memory than full-vocabulary forward-KL in the controlled comparison, but 6.5% more GPU-hours; and the benefit of RTG is domain-dependent. We agree that both geometric targets and future credit have been discussed as well in concurrent related work. Our contribution is to derive a target over complete responses, characterize what changes when we implement it token by token, and identify the signal this implementation propagates along a rollout.

Q1. How does β-OPSD differ methodologically from DemoPSD and SR-OPSD's geometric targets, and from DistIL's future credit assignment?

Answer: β-OPSD starts from the KL-regularized RL objective underlying vanilla OPSD, recovers the sequence-level reverse-KL objective at β=1\beta=1, and derives the optimal policy over complete responses for general β\beta. Its connection between interpolation and sequence-level RTG produces a distinct prefix-normalization term (Q4).

Our starting question is what KL-regularized RL objective underlies vanilla OPSD. We recover its sequence-level reverse-KL objective at β=1\beta=1 and derive the optimal policy over complete responses for general β\beta.

DimensionDemoPSDSR-OPSDDistILβ-OPSD (ours)
Question addressedLimiting privileged-information leakageStabilizing a moving self-teacherLearning from rich feedback via distributional imitationWhich KL-regularized RL objective OPSD optimizes, and its solution for general β\beta
Interpolated targetTeacher–student geometric combination, with the teacher weight reduced where the two disagreeGeometric interpolation between the self-teacher and a frozen copy of the initial studentNone; the student matches the feedback-conditioned teacherGeometric interpolation between a reference policy and the privileged teacher, with teacher weight 1/β1/\beta
Level at which the target is definedPer tokenPer token, at fixed student-generated prefixesn/aComplete responses (sequence level), then implemented per token
Training signalLocal token loss, downweighted on disagreement (Eq. 13)Forward Rényi divergence at each prefixForward cross-entropy, over the teacher's top-100 tokens in its experiments (§6.4.2)Sampled-token log-ratio to the interpolant (reverse KL)
Future creditNoNoYes: later teacher–student cross-entropiesYes: discounted RTG of log-ratios, which carries the prefix normalizers (Q4)

Two procedural differences are not in the table. DemoPSD builds its teacher prompt from a correct rollout and skips prompts without one (Alg. 1), whereas β-OPSD supplies the reference solution directly and can train on those prompts. DistIL propagates cross-entropies over many candidate tokens, whereas β-OPSD propagates one log-ratio per position, for the token the student actually sampled.

DemoPSD and SR-OPSD appeared on arXiv on 2 July and 10 August 2026, shortly before the submission deadline, and we regard them as concurrent work. We will expand this comparison in the revision.

Our contribution is a unified view of OPSD and KL-regularized RL that explains how geometric targets and return-to-go credit assignment arise in the same framework. Both ideas appear in related work. We show that vanilla OPSD's sequence-level objective is the β=1\beta=1 case of a broader policy optimization problem. Solving it for general β\beta yields a geometric target with teacher weight 1/β1/\beta. The sequence-level distillation gradient then gives the return-to-go update, in which each token receives credit for current and future mismatches.

What is specific to β-OPSD is the interaction of the two. Because the target is normalized at every prefix and credit is propagated along the rollout, the return carries a Rényi-divergence term that neither a local geometric target nor future credit toward the un-interpolated teacher produces (also see our answer to Q4). We agree that the particular schedule (0.5 to 0.8) is an empirical choice: the derivation fixes the form of the target and the meaning of its weight, not how the weight is annealed.

Q2. On Qwen3-8B coding, how do β-OPSD and full-vocabulary forward-KL compare under checkpoint selection, evaluation protocols, GPU-hours, and memory?

Answer: Full-vocabulary forward-KL is better under the original coding protocol; the ordering reverses when public function/class interfaces are included. In controlled measurements, β-OPSD reduces peak allocated update memory by 22.1% but uses 6.5% more GPU-hours.

We added instrumented, matched runs because the original experiments did not record the GPU-memory measurements needed for this comparison. We also re-evaluated the existing checkpoints under a common evaluation protocol; these fresh scores are reported separately from the submitted results.

Under the original protocol, forward-KL performs better at both step 100 and the validation-selected step-50 checkpoint:

Qwen3-8B LiveCodeBench Pass@1β-OPSDForward-KL
Original protocol, step 10025.97%28.53%
Original protocol, validation-selected step 5026.64%28.06%
Interface-aware protocol, same selected checkpoints45.50%43.60%

The original protocol does not include the public function/class interfaces in the prompt. When we evaluate the same selected checkpoints with these interfaces included, both methods improve substantially and the ordering reverses: β-OPSD reaches 45.50% versus 43.60% for forward KL (+1.90 points).

Resource measurementβ-OPSDForward-KL
Matched 100-update H200 training, 1K completion cap4.58 GPU-hours4.30 GPU-hours
Controlled update-stage peak allocated memory, 8K padded input40.87 GiB52.46 GiB

Regarding resources, β-OPSD reduces update-stage peak allocated memory by 22.1% but uses 6.5% more GPU-hours in the matched run (4.58 vs. 4.30). We therefore claim a memory benefit only, not a training speedup, and will qualify the efficiency claim accordingly. We retain both evaluation protocols.

Our interpolated target is compatible with full-vocabulary forward KL. At each prefix, we interpolate the reference and teacher logits to obtain a distribution over the entire vocabulary. Our current RTG estimator uses the sampled token's log-ratio. The same interpolated distribution can directly replace the teacher distribution in the full-vocabulary forward-KL loss. A DistIL-style variant could likewise use cross-entropies at later prefixes as the return-to-go signal in the full-vocabulary setting (Agrawal et al., Eq. 5). We leave both variants to future work.

Q3. Why do Figure 1 and Table 4 disagree in absolute Qwen3-4B math scores, and does the 0.5-to-0.8 schedule span 100 or 200 steps?

Answer: Figure 1 is a separate 100-step training-dynamics experiment; the main runs have a 200-step budget. The teacher weight anneals over the full run in either setting, so the main step-100 checkpoint is only halfway through its schedule.

Figure 1 presents a separate training-dynamics experiment, rather than the runs underlying Figure 3 and Table 4. It uses Qwen3-4B on OpenThoughts Math 30K, trained for 100 steps on one H200 with seed 42, an effective batch size of 32, and a 1,024-token completion limit.

MethodTraining objective in Figure 1
Vanilla OPSDSampled-token reverse-KL loss (Tinker implementation), fixed teacher, no RTG
β-OPSDCurrent-student reference, teacher-target weight scheduled from 0.5 to 0.8, discounted RTG with γ=0.99\gamma=0.99

The absolute scores therefore correspond to different experimental configurations and should not be read as reproducing the main-result scores. Figure 1 illustrates checkpoint-level behavior in this particular setting; the observed post-step-50 degradation is not a claim that vanilla OPSD always degrades after step 50. We will make this distinction explicit and add these settings to the caption.

On the schedule: the teacher weight is annealed from 0.5 to 0.8 over the full training run in both settings. In Figure 1 the run is 100 steps, so the schedule completes at step 100. In the main experiments the run is 200 steps and we report the step-100 checkpoint, so those results are taken midway through the schedule.

Q4. Can scaling the teacher gradient explain interpolation at a fixed prefix, and what does interpolation add when RTG is used? Does RTG consistently help coding?

Answer: Fixed-prefix local-gradient scaling is correct, but under RTG the interpolated target contributes an additional continuation-dependent Rényi-divergence term. A schedule-matched control isolates +1.20 points at 4B and +0.55 at 8B on math; RTG is not consistently helpful on coding.

We agree that, under the reviewer's fixed-prefix assumptions, scaling the teacher signal reproduces the expected local gradient of the interpolated target. The distinction under return-to-go arises because β-OPSD normalizes its target separately at each prefix, and an earlier token determines which later prefixes are visited.

At each training step, we use the current student as the detached reference. Applying softmax to the interpolated student and teacher logits gives the target in Eqs. (7) and (58):

qw(a∣h)=π(a∣h)1−wpT(a∣h)wZw(h).q_w(a\mid h)=\frac{\pi(a\mid h)^{1-w}p_T(a\mid h)^w}{Z_w(h)}.

Although the student and teacher distributions each sum to one, their tokenwise geometric average can sum to less than one. The denominator normalizes this average over the vocabulary:

Zw(h)=∑bπ(b∣h)1−wpT(b∣h)w.Z_w(h)=\sum_b\pi(b\mid h)^{1-w}p_T(b\mid h)^w.

Substituting the normalized target into the sampled log-ratio gives

=w\log\frac{\pi(a\mid h)}{p_T(a\mid h)}+\log Z_w(h).$$ **For a local update**, the log normalizer is the same for every candidate token at a fixed prefix. Its contribution vanishes in the expected local gradient because the expected score gradient under the student is zero. This recovers the reviewer's scaling observation (Appendix B.5, Eq. 60). **For return-to-go**, the difference accumulates along the sampled continuation. Let the superscript T denote the unscaled teacher-only return. By Eq. (11), $$G_t^{\mathrm{interp}}-wG_t^T =\sum_{s=t}^L\gamma^{s-t}\log Z_w(h_s).$$ Each normalizer has a closed form $\log Z_w(h)=-(1-w)D_w(p_T(\cdot\mid h)\|\pi(\cdot\mid h))\leq 0$: a Rényi divergence of order $w$ between teacher and student, which is zero only when the two next-token distributions coincide. The difference above is therefore a discounted sum of teacher–student Rényi divergences along the sampled continuation, computed from the full distributions at each prefix and not from the sampled tokens. For example, suppose token A leads to a prefix where the student and teacher agree, and token B leads to one where they disagree. Matched scaling charges both continuations $w$ times the teacher log-ratio. β-OPSD charges A the same amount but reduces the charge for B by $(1-w)D_w$ at the disagreeing prefix. In expectation over the next token, the per-prefix cost is $\mathrm{KL}(\pi\|q_w)=w\mathrm{KL}(\pi\|p_T)-(1-w)D_w(p_T\|\pi)$ under β-OPSD and $w\mathrm{KL}(\pi\|p_T)$ under matched scaling. Matched scaling thus shrinks the teacher penalty uniformly, while interpolation shrinks it most at prefixes where the privileged teacher is far from the student. This is the sense in which the interpolated target differs from a matched update strength. **β-OPSD gains 1.20 percentage points at 4B and 0.55 at 8B over matched teacher-only RTG.** The matched control uses the same RTG estimator and scales each sampled teacher log-ratio by β-OPSD's teacher weight at that step. Scores are averaged across the three math benchmarks in Table 4. | Method | Qwen3-4B avg@12 (%) | Qwen3-8B avg@12 (%) | |---|---:|---:| | Teacher-only, reweighted local reverse KL | 53.61 | 56.48 | | Teacher-only, reweighted log-ratio + RTG | 56.67 | 59.63 | | **β-OPSD, logit interpolation + RTG** | **57.87 (+1.20)** | **60.18 (+0.55)** | These gains suggest a benefit from prefix-dependent normalization, which β-OPSD incorporates directly by applying log-softmax to the interpolated logits. **RTG on code, and what isolates the combination.** We agree that RTG is not consistently beneficial on LiveCodeBench: it improves the overall result at 4B, while local credit assignment is better at 1.7B and 8B. We do not claim RTG as a contribution in its own right. Future credit appears in prior work (DistIL), and in our derivation it is the estimator prescribed by the sequence-level gradient. Our primary contribution is the β-family and its interpolated target, and the two interact as follows. With local credit, the target acts through the teacher weight $w=1/\beta$, as the reviewer notes. With RTG, the target additionally contributes the Rényi term above, and the matched control isolates that term on math (+1.20 points at 4B, +0.55 at 8B). We will report both credit-assignment variants side by side in the main text and state that the benefit of RTG is domain-dependent.
Reviewer_5vp4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

On-policy self distillation has to be used in situations where there are no suitable stronger teachers available but since the student and the teacher are essentially the same model, getting this to work in practice is hard. Beta-OPSD describes a way to balance teacher supervision and trusting the model's own policy (guided self-distillation). The factor beta controls the amount of supervision to retain potentially best of both models. The paper develops theory to convert a costly RL objective into a cheaper and manageable distillation objective that retains the gains and shows that OPSD is the beta=1 KL regularized case. Beta-OPSD develops a framework wherein the RL objective is converted into a closed form distillation objective that selectively balances when to learn from self and when to learn from the teacher. Experiments support the claims and the paper goes from an idea, to proving out the theory, providing the tuition and then demonstrating it in practice.

Critical strengths

Mixing token-level feedback and sequence-level feedback is a useful setting and return-to-go credit assignment enables this.

Knowing what to retain and what to lose in self-distillation is tricky and this paper provides a more principled way to control this

Empirical evidence is exhaustive

Closed form solution derivation (and mathematical justification throughout) is well developed

Critical weaknesses

The experimentation support results in math domain (authors already point this out), but in practice, models tend to be built for more than one domain, so this may limit the applicability of the method (not yet clear).

Questions for authors

Does the teacher model have access to additional information (context, hints, examples)?

What happens when both teacher and student fail on a particular class of problems? There is no learning signal (or perhaps more correctly, very weak signal) in this case, but the failures may not be identical, so updates may still happen.

2 comments

We thank the reviewer for highlighting the closed-form derivation and the role of return-to-go in connecting token-level feedback to sequence-level learning. We have addressed all three questions. Bottom line: our experiments cover both math and code; the frozen privileged teacher sees the reference solution in addition to the student-generated prefix; and the method can provide token-level supervision even when a generated answer is incorrect.

Q1. Does the method extend beyond math reasoning, and what evidence is available for coding and multi-domain applicability?

Answer: We already train on coding and evaluate on LiveCodeBench v6; β-OPSD improves coding pass@1 at all three evaluated model sizes. Joint math–code training remains untested.

We train on DeepCoder and evaluate on LiveCodeBench v6. Compared with our sampled reverse-KL vanilla OPSD baseline, β-OPSD improves overall pass@1 by 1.52, 1.90, and 2.18 percentage points on Qwen3-1.7B, 4B, and 8B, respectively (Figure 4 and Table 5). We train separately on math and code. We agree that whether a single model trained on both domains retains these gains in each remains open.

Q2. Does the privileged teacher receive additional context, hints, examples, or other information unavailable to the student?

Answer: Yes. The frozen teacher additionally sees the reference solution; the student does not.

At each student-generated prefix, both models see the problem and the tokens generated so far. The frozen teacher additionally sees the reference solution. At inference, the student solves the problem independently (Section 4.1).

Q3. If both student and teacher fail on a class of problems, can training still provide a learning signal?

Answer: A training signal can still arise from differences in their token distributions, even if the sampled final answer is incorrect.

During training, only the student generates a response. The teacher uses the reference solution to score the sampled tokens along the student's trajectory. Even if the teacher would produce an incorrect final answer itself, it may still assign higher probability than the student to a correct intermediate step. This difference enters the interpolated target and can influence earlier token updates through return-to-go (Eqs. 10–12). With the current student as the reference and a positive teacher weight, the log-ratio mismatch at a prefix is zero for all tokens only when the teacher and student assign exactly the same next-token distribution there. Because the reference solution can shift the teacher's token probabilities, we would expect such exact agreement at every prefix of a rollout to be rare.

Thank you for the clear and to the point response. Thanks also for the pointer to the existing DeepCoder/LiveCoderBench results in Fig 4 & Table 5, clarification on the frozen teacher with access to reference solution and how the token-level sampling preserves a learning signal even when the teacher is wrong. These points resolve my questions on mechanics and single-domain coding coverage but joint domain training is hard (and often needed in practice) and there is no new experimental evidence along these lines. Having already accounted for technical soundness but slightly limited applicability and the need for these explanations, I remain positive about the paper and would like to hold my score.

Reviewer_gMLEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces β-OPSD, a method for on-policy self-distillation with a privileged teacher. It derives a geometric reference–teacher distribution from a KL-regularized policy objective, implements a token-level approximation through logit interpolation, and uses discounted return-to-go of token log-ratio mismatches for credit assignment. The default implementation combines a dynamic student reference with a fixed teacher and increases the teacher weight from 0.5 to 0.8. Experiments on Qwen3 models from 1.7B to 8B report improvements over the paper's vanilla OPSD baseline on mathematical reasoning and LiveCodeBench, together with comparisons against related distillation methods and an efficiency study.

Critical strengths
  1. A coherent and practical formulation. The paper connects a KL-regularized objective to an implementable interpolation rule and incorporates future mismatches into token updates. It explicitly discusses the gap between the exact sequence-level solution and its practical approximation, which helps delimit the theoretical claim.
  2. Useful empirical coverage and relevant controls. The main results cover three model sizes and both mathematical and code reasoning. Table 1 includes TRB, TOP-D, and Veto under a shared protocol, while Table 2 holds return-to-go fixed when changing the target. The efficiency experiment reports GPU-hours and acknowledges that equal optimizer steps do not imply equal compute. These comparisons provide meaningful evidence beyond a comparison against vanilla OPSD alone.
Critical weaknesses
  1. The incremental contribution and the mechanism behind the gains are insufficiently isolated. Equation (3) can be rewritten as maximizing −KL(π∥pT)−(β−1)KL(π∥πref)-\mathrm{KL}(\pi\Vert p_T)-(\beta-1)\mathrm{KL}(\pi\Vert\pi_{\mathrm{ref}}), making the geometric optimum a familiar weighted-KL construction. Related logit-space intermediate targets already exist in Veto (Jang et al., 2026), although its product-of-experts target is not identical to this paper's normalized-weight interpolation. I appreciate the direct comparison in Table 1. Nevertheless, Table 2 changes both the target and the scale of the teacher mismatch. With a detached current-student reference, the expected local reverse-KL gradient at a fixed prefix is simply multiplied by the teacher weight. Return-to-go can introduce additional effects, so this is not an equivalence claim for the complete algorithm. A teacher-only return-to-go control with the same mismatch scaling is needed to establish what the proposed target contributes beyond this simpler change.

  2. The evidence for robust improvement and stabilization remains incomplete. Several aggregate gains are modest: 1.66–1.76 percentage points for the larger models on math and 1.52–2.18 points on coding. The main results do not report variation across independent training runs, and Appendix C.2 states that the reported checkpoints are trained with a single seed (42), so the reported standard errors reflect only decoding variability. All main comparisons use a single checkpoint (step 100), although Figure 1 suggests that this choice matters: in that plot, vanilla OPSD reaches about 48.8% at step 50, above the 48.1% that β-OPSD reaches at step 100. Moreover, Figure 1's Qwen3-4B math curve ends at approximately 46.6% for vanilla OPSD and 48.1% for β-OPSD, whereas Figure 3 reports 56.11% and 57.87% at step 100; these may reflect different protocols, but the main text does not explain the difference. Finally, the comparison with concurrent methods (Table 1) and the target ablation (Table 2) are reported only for Qwen3-1.7B math, the setting with the largest gain (+5.74), so it is unclear whether these controls hold at 4B and 8B, where the gains are about 1.7 points. Because Figure 1 motivates the stabilization claim, these gaps limit confidence that the method improves training consistently rather than in a particular run, checkpoint, or configuration.

Questions for authors
  1. (Requested analysis) Can the authors compare against Teacher + Return-to-Go with ρt=wklog⁡[πθ(yt∣ht)/pT(yt∣ht,c)]\rho_t=w_k\log[\pi_\theta(y_t\mid h_t)/p_T(y_t\mid h_t,c)], using the same wkw_k schedule, discount, normalization, optimizer, and training budget? This would help isolate the effect of the interpolant's prefix-dependent normalization from simply scaling teacher mismatches.
  2. (Clarification) In Table 1, were the re-implemented baselines (TRB, TOP-D, and Veto) trained with the same return-to-go estimator as β-OPSD, or with their original token-level objectives? Since return-to-go improves most math settings in the ablation, this determines whether the gap to Veto reflects the target itself.
  3. (Clarification) What accounts for the different step-100 Qwen3-4B math scores in Figures 1 and 3? Please specify the benchmark aggregation, training configuration, and evaluation protocol for Figure 1, and clarify whether it represents the main experimental setup.
  4. (Requested analysis) Can the authors provide results from independent training seeds for a representative setting, including the claimed post-step-50 degradation of vanilla OPSD? Evaluation-sampling variability and training variability should be distinguished.
  5. (Requested analysis) Appendix Table 6 indicates training for 200 steps with checkpoints saved at steps 25, 50, 75, 100, and 200, but all main comparisons use step 100. How do the methods compare at step 200 and at each method's best checkpoint, given that Figure 1 shows vanilla OPSD at step 50 exceeding β-OPSD at step 100?
  6. (Requested analysis) Can the comparison with concurrent methods (Table 1) and the target ablation (Table 2) be reported for Qwen3-4B or Qwen3-8B, where the gains over vanilla OPSD are smaller?
3 comments

We thank the reviewer for recognizing the connection between our KL-regularized objective and its practical implementation, and for highlighting the value of the controlled comparisons and GPU-hour measurements. We have answered all six questions and completed the additional requested analyses: matched-scaling controls on Qwen3-4B/8B; RTG-matched TRB and Veto comparisons; two additional independent training seeds; checkpoint comparisons through step 200; and expanded 4B/8B baselines. Bottom line: interpolation improves math avg@12 over matched teacher-only RTG by +1.20 points at 4B and +0.55 points at 8B (one seed per cell), while the seed and checkpoint analyses show a more modest, setting-dependent stability benefit. We agree that the standard errors in the submission capture decoding variability from a fixed training seed, and that training stability requires separate evidence across seeds and checkpoints.

Q1. Does the interpolated target improve performance beyond simply scaling teacher mismatches, when teacher-only return-to-go (RTG) uses the same schedule, discount, normalization, optimizer, and training budget?

Answer: Yes. Interpolation improves math avg@12 by 1.20 percentage points at 4B and 0.55 at 8B over a matched teacher-only RTG control.

We multiplied each sampled student–teacher log-ratio by the scheduled teacher weight before computing RTG. We kept the weight schedule (0.5 to 0.8), RTG discount (0.99), per-response length normalization, optimizer, and training budget identical.

MethodQwen3-4B math avg@12 (%)Qwen3-8B math avg@12 (%)
Teacher-only RTG with matched scaling56.6759.63
β-OPSD (Table 4)57.8760.18
Gain (percentage points)+1.20+0.55

The distinction comes from normalizing the geometric target at each prefix. Although the student and teacher distributions each sum to one, their tokenwise geometric average need not. To see what interpolation adds beyond scaling the teacher signal, fix a training step kk and write w=wkw=w_k, with the current student as the detached reference. Let ht=(x,y<t)h_t=(x,y_{<t}) be the current prefix, π\pi the current student policy, and pTp_T the teacher policy, which additionally conditions on the reference solution. Interpolating the detached student logits and the teacher logits with teacher weight ww gives the target

Z_w(h_t)=\sum_{b\in\mathcal V}\pi(b\mid h_t)^{1-w}p_T(b\mid h_t)^w.$$ For a token $y_t\sim\pi(\cdot\mid h_t)$, define the teacher-only log-ratio as $\rho_t^{\mathrm{teacher}}:=\log[\pi(y_t\mid h_t)/p_T(y_t\mid h_t)]$. Substituting the interpolated target into the β-OPSD log-ratio gives, by Eqs. (7) and (10), $$\rho_t^{\beta\text{-OPSD}}:=\log\frac{\pi(y_t\mid h_t)}{q_w(y_t\mid h_t)} =w\rho_t^{\mathrm{teacher}}+\log Z_w(h_t).$$ As you observe, at a fixed prefix $h_t$ the offset $\log Z_w(h_t)$ is the same for every candidate token, so its contribution to the expected local policy gradient is zero. The offset has a closed form. It is a Rényi divergence of order $w$ between the teacher and the student at that prefix: $$\log Z_w(h)=-(1-w)D_w(p_T(\cdot\mid h)\|\pi(\cdot\mid h))\leq 0,$$ with equality only when the two next-token distributions coincide. Writing $G_t=\sum_{s=t}^L\gamma^{s-t}\rho_s$ for the discounted return of Eq. (11), the β-OPSD return therefore decomposes as $$G_t^{\beta\text{-OPSD}}=wG_t^{\mathrm{teacher}} -(1-w)\sum_{s=t}^L\gamma^{s-t}D_w(p_T(\cdot\mid h_s)\|\pi(\cdot\mid h_s)).$$ The first term is the matched-scaling control. The second term is the correction introduced by the interpolated target. - **It changes credit only through later prefixes.** The $s=t$ term is constant across candidate tokens and vanishes in expectation, which recovers your local observation. The $s>t$ terms depend on $y_t$ through the prefixes it leads to. - **It is a full-vocabulary quantity.** $D_w$ is computed from the complete student and teacher next-token distributions at each visited prefix, so unlike $\rho_s^{\mathrm{teacher}}$ it does not depend on which token was sampled there. - **Its direction is known.** Averaged over the sampled token, the per-prefix cost is $\mathrm{KL}(\pi\|q_w)=w\mathrm{KL}(\pi\|p_T)-(1-w)D_w(p_T\|\pi)$ under β-OPSD and $w\mathrm{KL}(\pi\|p_T)$ under matched scaling. The two coincide where student and teacher agree and differ most where they disagree strongly. Matched scaling therefore shrinks the teacher penalty uniformly, whereas interpolation charges an earlier token less for leading to prefixes where the privileged teacher is far from the student. In the matched RTG comparison above, retaining this term improved math avg@12 by 1.20 percentage points at 4B and 0.55 points at 8B (one training seed per cell).

Q2. Were TRB, TOP-D, and Veto evaluated using their original credit-assignment objectives or β-OPSD's RTG, and how do comparisons change with RTG matched?

Answer: Table 1 uses each method's original credit-assignment rule. With RTG matched, β-OPSD leads Veto + RTG by 3.33 points at 4B and 3.51 at 8B, and TRB + RTG by 2.78 points at 4B; at 8B, TRB + RTG is essentially tied.

TRB and Veto use their original token-level objectives. TOP-D already adds the mean of subsequent token rewards to the current reward (Xie et al., §3.2, Algorithm 1). For each baseline, we used the best-performing hyperparameters reported in its paper, including interpolation weights where applicable.

To test whether RTG explains the gap, we also trained Veto and TRB with the same discounted RTG estimator as β-OPSD (full table under Q6). RTG helps TRB substantially (+1.76 points at 4B, +3.24 at 8B) and leaves Veto essentially unchanged (+0.28, −0.09). With RTG matched, β-OPSD exceeds Veto + RTG by 3.33 points at 4B and 3.51 at 8B, and TRB + RTG by 2.78 points at 4B. At 8B, TRB + RTG is within 0.09 points of β-OPSD, which we do not regard as a meaningful difference.

Q3. Why do Figure 1 and Figure 3/Table 4 report different Qwen3-4B step-100 math scores? Do they use the same training and evaluation setup?

Answer: Figure 1 is a separate training-dynamics experiment, not the runs underlying Figure 3 and Table 4; the configurations differ.

Figure 1 presents a separate training-dynamics experiment, rather than the runs underlying Figure 3 and Table 4. It uses Qwen3-4B on OpenThoughts Math 30K, trained for 100 steps on one H200 with seed 42, an effective batch size of 32, and a 1,024-token completion limit. The main-result runs instead use a 200-step training budget and are evaluated at the step-100 checkpoint.

MethodTraining objective in Figure 1
Vanilla OPSDSampled-token reverse-KL loss (Tinker implementation), fixed teacher, no RTG
β-OPSDCurrent-student reference, teacher-target weight scheduled from 0.5 to 0.8, discounted RTG with γ=0.99\gamma=0.99

The absolute scores therefore correspond to different experimental configurations and should not be read as reproducing the main-result scores. Figure 1 illustrates checkpoint-level behavior in this particular setting; the observed post-step-50 degradation is not a claim that vanilla OPSD always degrades after step 50. We will make this distinction explicit and add these settings to the caption.

Q4. Do independent training seeds support the reported performance and post-step-50 degradation, rather than only decoding-sampling variability?

Answer: We repeated Figure 1 with two additional seeds (42, 43, 44). β-OPSD's mean step-100 math avg@12 is 0.71 percentage points higher; vanilla OPSD declines by 1.08 points on average from step 50 to 100.

We repeated the Figure 1 experiment with two additional training seeds, yielding three seeds (42, 43, and 44). All runs use Qwen3-4B, 100 training steps, a 1,024-token completion limit, and the same evaluation protocol. We report Avg@12 mean ± sample standard deviation across training seeds.

Step-100 performance

MethodAIME24AIME25HMMT25Average
Vanilla OPSD0.6037 ± 0.01850.4870 ± 0.01160.3130 ± 0.01250.4679 ± 0.0054
β-OPSD, RTG γ=0.99\gamma=0.990.6056 ± 0.02100.5037 ± 0.00850.3157 ± 0.01970.4750 ± 0.0147

β-OPSD's mean is 0.71 percentage points higher. These results suggest a modest average improvement in this setting.

Vanilla OPSD: Step 50 → 100

Training seedStep 50 averageStep 100 average
420.48890.4657
430.47410.4741
440.47310.4639
Mean ± standard deviation0.4787 ± 0.00880.4679 ± 0.0054

Vanilla OPSD's mean performance decreases by 1.08 percentage points between steps 50 and 100. Two seeds decline, while one remains unchanged.

Q5. How do β-OPSD and vanilla OPSD compare at step 200 and at their respective best saved checkpoints, rather than only at step 100?

Answer: In the examined Qwen3-4B run, β-OPSD leads by 0.55 points at step 200 and 0.28 points when comparing each method's best saved checkpoint.

On Qwen3-4B math avg@12, β-OPSD's best saved checkpoint outperforms vanilla OPSD's best by 0.28 percentage points under a 200-step training budget. β-OPSD reaches 57.22% at step 200, while vanilla OPSD peaks at 56.94% at both steps 50 and 100. Comparing both methods at step 200, β-OPSD leads by 0.55 points (57.22% vs. 56.67%).

The table shows all saved checkpoints from the seed-43 runs. We report avg@12 (%) averaged over AIME 2024, AIME 2025, and HMMT 2025. β-OPSD uses RTG with γ=0.99\gamma=0.99.

MethodStep 25Step 50Step 75Step 100Step 200
Vanilla OPSD55.9356.9456.6756.9456.67
β-OPSD56.1156.8555.6557.1357.22

Q6. Do concurrent-method comparisons and the target ablation extend to Qwen3-4B and 8B, where the original gains were smaller?

Answer: Yes. At 4B, β-OPSD leads the strongest concurrent baseline by 2.13 points. At 8B, TRB + RTG comes within 0.09 points of β-OPSD.

The table reports math avg@12 (%) averaged over AIME 2024, AIME 2025, and HMMT 2025.

MethodQwen3-4BQwen3-8B
TOP-D (native RTG)55.7458.06
Veto54.2656.76
Veto + RTG54.5456.67
TRB53.3356.85
TRB + RTG55.0960.09
β-OPSD57.8760.18

For the target ablation, Q1 reports the teacher-only control at both scales. Holding the teacher-weight schedule and RTG estimator fixed, logit interpolation improves avg@12 by 1.20 points at 4B and 0.55 points at 8B.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary for All Reviewers and the Area Chair

We thank all three reviewers for their careful assessment and constructive suggestions. We have responded to all 13 questions in the original reviews and completed the requested follow-up analyses, including matched-scaling controls, expanded Qwen3-4B/8B comparisons, independent training seeds, checkpoint analyses, and matched resource measurements. The bottom line is that these additions isolate a benefit from interpolation beyond simple teacher-signal scaling, strengthen the controlled comparisons, and clarify where the empirical claims must be qualified. The reviewer-requested weighting-schedule visualization from the subsequent discussion is being prepared for the manuscript revision; we do not present it as completed here.

Summary of additional evidence

Reviewer concernAdditional analysis or experimentMain finding
Is interpolation merely teacher-signal scaling? (gMLE Q1; EaUU Q4)Teacher-only RTG control with matched weight schedule, discount, normalization, optimizer, and training budget at 4B and 8Bβ-OPSD improves math avg@12 by +1.20 pp (4B) and +0.55 pp (8B); one training seed per cell
Are concurrent-method comparisons confounded by RTG? (gMLE Q2, Q6)Added TRB+RTG and Veto+RTG; expanded the comparison to 4B and 8BAt 4B, β-OPSD leads the strongest concurrent baseline by 2.13 pp; at 8B, TRB+RTG is within 0.09 pp
Are training-dynamics observations seed-dependent? (gMLE Q4)Two additional training seeds, for three seeds (42, 43, 44) in the Figure 1 settingStep-100 math avg@12: 47.50 ± 1.47% (β-OPSD) vs. 46.79 ± 0.54% (vanilla); vanilla drops 1.08 pp on average from step 50 to 100
Does step-100 selection hide a better baseline checkpoint? (gMLE Q5)Compared saved checkpoints through step 200 in the examined 4B runβ-OPSD leads by +0.55 pp at step 200 and +0.28 pp at each method's best saved checkpoint
What is the forward-KL tradeoff on 8B coding? (EaUU Q2)Matched H200 training/resource measurements and evaluation of selected checkpoints with and without public function/class interfacesβ-OPSD uses 22.1% less update-stage peak allocated memory, but 6.5% more GPU-hours; forward-KL wins under the original protocol, while β-OPSD leads 45.50% vs. 43.60% under the interface-aware protocol
Why do Figure 1 and the main tables differ? (gMLE Q3; EaUU Q3)Clarified experiment configurations, objectives, budgets, and schedulesFigure 1 is a separate 100-step training-dynamics setting; the main-result runs have a 200-step budget and report the step-100 checkpoint

1. The interpolated target contributes beyond matched teacher scaling

The reviewer-requested controlled comparison separates target interpolation from teacher-signal scaling. With the same RTG estimator and teacher-weight schedule, β-OPSD improves Qwen3-4B and Qwen3-8B math avg@12 by +1.20 and +0.55 percentage points, respectively, over teacher-only RTG with matched scaling.

We further characterize why these objectives differ. At a fixed prefix, the expected local gradient toward the normalized interpolated target has the same teacher-weight scaling identified by the reviewers. Under sequence-level RTG, however, the target's per-prefix normalizers accumulate along sampled continuations. Their log values correspond to Rényi-divergence terms, giving a continuation-dependent correction that is absent from uniform teacher-signal scaling. The matched control measures the effect of retaining this correction. We emphasize that the measured improvements are based on one training seed per cell.

2. We expanded and better controlled the empirical comparisons

The expanded 4B/8B baselines distinguish target effects from credit-assignment effects. We preserved TRB, TOP-D, and Veto's original objectives in the initially reported comparisons, and separately evaluated TRB and Veto using β-OPSD's discounted RTG estimator. RTG improves TRB by +1.76 points (4B) and +3.24 points (8B), while changing Veto by only +0.28 and −0.09 points. β-OPSD leads the strongest concurrent baseline by 2.13 points at 4B; at 8B, TRB+RTG (60.09) is effectively tied with β-OPSD (60.18). We do not treat the 0.09-point difference as meaningful.

The new training-seed and checkpoint results support a more qualified stability statement. Across three independent training seeds in the Figure 1 setting, β-OPSD averages 47.50 ± 1.47% versus 46.79 ± 0.54% for vanilla OPSD. Vanilla OPSD's mean drops 1.08 points from steps 50 to 100 (two seeds decline, one is unchanged). In a separate main-experiment configuration, β-OPSD's best saved 4B checkpoint leads vanilla OPSD's best by 0.28 points and leads by 0.55 points at step 200. These are modest differences, not evidence of universal or statistically robust stability gains.

3. We clarified the 8B coding and efficiency comparisons

β-OPSD has a measured update-memory advantage, not a speed advantage, over full-vocabulary forward-KL in the matched setup. Under matched 100-update H200 runs, it uses 40.87 GiB versus 52.46 GiB in peak allocated update memory (−22.1%) and 4.58 versus 4.30 GPU-hours (+6.5%).

In the original LiveCodeBench evaluation protocol, forward-KL performs better at both step 100 and the validation-selected checkpoint. When public function/class interfaces are included for the same selected checkpoints, β-OPSD reaches 45.50% versus 43.60% for forward-KL. We will retain and clearly distinguish both evaluation protocols rather than replacing the original results. We also acknowledge that RTG improves the overall coding result only at 4B, whereas local credit assignment performs better at 1.7B and 8B.

4. We clarified the methodological scope and experimental setup

Our methodological contribution is the policy-optimization derivation and its connection to sequence-level credit assignment, rather than claiming that geometric interpolation or future credit was individually introduced here. We show that vanilla OPSD corresponds to the β=1\beta=1 case of a KL-regularized policy-optimization family, derive its optimal policy for general β\beta over complete responses, and identify the prefix-normalization correction that appears when the token-level interpolated target is combined with RTG. We have provided a detailed comparison with DemoPSD, SR-OPSD, and DistIL as closely related/concurrent directions.

We clarified that Figure 1 and the main results use different configurations, why their absolute scores differ, and that the 0.5-to-0.8 teacher-weight schedule spans the full run: 100 steps in Figure 1 and 200 steps in the main experiments. The particular schedule is an empirical choice, not a theoretical consequence of the derivation. We also answered the questions about privileged teacher information (the teacher additionally receives the reference solution), coding results across model sizes, and nonzero token-level signals on incorrect student trajectories, while acknowledging that joint math–code training remains untested.

Summary of Manuscript Revisions

We have incorporated all additional experiments, analyses, and clarifications from the rebuttal into the revised manuscript (with all updates highlighted in red):

  • Matched controls and larger-scale comparisons: We added the matched-scaling teacher-only controls in Table 2 (Section 4.4.1, p. 9) and expanded our 4B/8B comparisons against TRB, Veto, and TOP-D in Table 14 (Appendix E.5, p. 29). Section 4.2 analyzes these findings, highlighting that adding RTG brings TRB (60.09%) within 0.09 points of β\beta-OPSD (60.18%) in mean math avg@12 at the 8B scale.

  • Theoretical analysis and related work: Section 2.3 (Eq. 13, p. 5) now details how the prefix-dependent normalization term shapes credit assignment for earlier tokens under return-to-go (RTG). Additionally, Section 3 clarifies the conceptual and algorithmic distinctions between our approach and DemoPSD, SR-OPSD, and DistIL.

  • Training-seed sensitivity and checkpoint dynamics: Appendix C.2 clarifies that the standard errors in the main tables capture variance across evaluation decoding runs. In Appendix E.6 (p. 30), Tables 15–16 evaluate training randomness across three independent seeds under the 100-update Figure 1 protocol, while Table 17 tracks all saved checkpoints across a full 200-update run.

  • 8B coding performance vs. efficiency trade-offs: In Appendix D (p. 26), Table 10 compares β\beta-OPSD against full-vocabulary forward-KL OPSD across multiple checkpoints and prompt protocols, while Table 11 profiles training GPU-hours and peak memory footprint. Section 4.3 synthesizes the performance–efficiency trade-offs between the two methods.

We appreciate the reviewers' feedback; it has led to a substantially more complete and carefully qualified evaluation of β-OPSD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.