acceptodds
Under review as a conference paper at ICLR 2027

Self-Distilled Policy Gradient

Abstract

On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. We instantiate OPD as an auxiliary full-vocabulary student-to-teacher reverse Kullback–Leibler divergence (KL) loss. For any fixed sampled prefix and with the privileged branch detached, we show that its student-side gradient is identical to a policy-gradient update whose centered token advantage is a centered log teacher/student ratio. We propose SDPG, a self-distilled policy-gradient framework that combines standard-deviation-normalized group-relative verifier advantages, exact full-vocabulary OPD, and reference-policy KL regularizationing. Positive-advantage gating and a warmup-decay schedule control when the privileged signal is trusted. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines.

Primary area:
reinforcement learning
Keywords:
self-distillation, reinforcement learning, policy gradient

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5HEyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SDPG, a loss for LLM reinforcement learning with verifiable reward (RLVR) and privileged information. Based on its derivation of the self-distillation loss as a policy gradient whose advantage appears to reinforce wrong trajectories, they suggest SDPG, a loss that combines the following components:

  1. a REINFORCE loss with group-level advantage from the verifiable reward.
  2. an auxiliary on-policy self-distillation from privileged information loss (reverse KL), gated on positive-advantage only and weighted by a warmup-decay schedule.
  3. an auxiliary on-policy KL loss to a reference policy, whose reverse and forward formulations can both be applied to on-policy samples.

Alongside the derivation of these losses, they claim that this combination of the losses fixes failure cases in on-policy self-distillation from privileged information (OPSD) and beats GRPO and other OPSD baselines.

The authors report results on math benchmarks (AIME24, 25, and AMC23) from two RL training runs on a math dataset (DAPO-Math-17k), one with Qwen3-4B and one with Qwen3-1.7B, and compare to GRPO and RLSD (which uses self-distillation to weigh reward-based advantages) and OPCD (classical on-policy self-distillation with full vocab). They also perform two ablations, removing each of the auxiliary losses.

Critical strengths

SDPG appears to continue improving in a stable manner on both runs with Qwen3 4B and 1.7B, while other self-distillation baselines seem to plateau or collapse (Figure 2abdc, Figure 6abcd). RLVR and self-distillation methods are known to suffer from instabilities and plateaus, preventing them from scaling. A recipe that allows further scaling is an important contribution to the field.

The motivation to schedule the self-distillation feedback as an auxiliary loss so that it mainly kicks in at the beginning, to limit and reduce the bias as training progresses, is interesting.

Critical weaknesses

The critical weakness of the paper is that it reads as an incomplete study. It identifies a relevant limitation in previous methods (the distillation loss can reinforce wrong trajectories), designs a complex solution to address it (3 objectives: reward + gated and scheduled distillation + reference regularization), but fails to sufficiently show or explain how each of the components in the proposed solution helps and how it outperforms prior work. Some motivated solutions do not discuss their flipside and there is insufficient experimental evidence or depth in its analysis to conclude.

  • Why replace the GRPO loss with a REINFORCE loss? Is there off-policy in the experiments? Otherwise, GRPO is itself REINFORCE.
  • Why start with the reference-regularized framework (RPG)? Why is regularization important? The no-reference-KL ablation seems to outperform SDPG, making its contribution unclear, and the discussion is missing relevant work discussing its need with GRPO, at least for math (e.g. in [1]). Does GRPO use KL-regularization in the experiments?
  • The on-policy forward/reverse KL to the reference policy is already present in [2] and [3]; the incremental contribution needs clarification.
  • The scheduled and gated self-distillation loss should be compared without regularization to the other self-distillation baselines, RLSD and OPCD, to isolate the fix to the main identified challenge (SD loss can reinforce bad trajectories)
  • Also, the gating and scheduling are not individually ablated or studied more carefully. A major advantage of SDPO is that even when the student can't find solutions, the teacher can steer it towards solutions, but a flipside of gating on positive advantages is that it removes this signal and can drastically reduce efficiency.
  • How does performance change when changing the distillation loss schedule? When gating the distillation on negative advantages?
  • And, the no-distillation ablation seems to match SDPG, giving no clear benefit of distillation. This is surprising.
  • No hyperparameter search or documented selection procedure, and no statistical significance. How were the hyperparameters for SDPG selected?
  • Is there any interplay between the auxiliary losses?
  • No limitations are discussed in the main text, including the flipside of the suggested fixes.

[1] Liu, Zichen, et al. "Understanding r1-zero-like training: A critical perspective." arXiv preprint arXiv:2503.20783 (2025). [2] Tang, Yunhao, and Rémi Munos. "On a few pitfalls in KL divergence gradient estimation for rl." arXiv preprint arXiv:2506.09477 (2025). [3] Zhang, Yifan, et al. "On the design of kl-regularized policy gradient algorithms for llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

see above

4 comments

We appreciate the reviewer's detailed analysis but respectfully disagree with the characterization of SDPG as an arbitrary three-loss recipe. And the responses are shown below.

Q1. Why REINFORCE instead of GRPO? Are there off-policy updates?

A1. We do not claim that REINFORCE and GRPO are fundamentally distinct. GRPO is a group-advantage policy-gradient method, with clipping becoming relevant when the optimized policy moves away from the frozen rollout policy. Sec. 3.2 expresses the fresh-rollout score-function form so that its outcome signal can be placed beside the fixed-prefix distillation gradient. This makes the two sources comparable; it is not an attempt to replace GRPO's underlying principle. The verified 1.7B verifier objective directly differentiates the clipped ratio against the cached rollout policy, with group-relative advantages. Each full actor batch schedules eight minibatch updates in one epoch. 4B logs corroborate RPGD and clipping; the original 4B source revision remains unverified. The detached expression in the printed Eq. (C.2) should not be conflated with exact PPO clipping, and the appendix is clarified accordingly. None of this changes the local OPD identity in Prop. A.1.

Q2. Why RPG and reference KL? If α=0 is competitive, why is regularization important? Did GRPO use KL?

A2. A regularizer does not need to increase every unregularized evaluation score to have a distinct optimization role. Reference KL penalizes departure from a fixed pretrained distribution; the intended population-level anchoring role is formalized, for example, by TV(p,pref)≤DKL(p∣pref)/2TV(p,p_{ref})\le\sqrt{D_{KL}(p|p_{ref})/2}. Thus, maximizing reward plus OPD minus a KL penalty is a different criterion from maximizing accuracy alone. The α=0 ablation actually shows that the anchor has a measurable behavioral effect: generation length and entropy change appreciably(Figure 5). We do not need to equate shorter answers with invalid reasoning to establish this role. The anchor is a deliberately chosen conservative component, not a claim that mathematics RL fails without it. We discuss Liu et al.(2025) in this context. Both GRPO baselines used reference-KL reward shaping with coeff. 0.001 and disabled a separate actor KL loss. RLSD uses sampled-token teacher reweighting in the verifier update. The original 4B RLSD patch remains unavailable.

Q3. Forward and reverse on-policy reference KL already appear in [2] and [3].

A3. We do not claim priority for forward/reverse reference KL or the score-function identity. They are the anchoring machinery of the framework, not its unique privileged-supervision mechanism. SDPG's method-specific contribution is a separate verifier-gated, full-vocabulary privileged teacher-student term, which is neither a reference KL nor the sampled-token reweighting used by RLSD. As with Tang&Munos(2025) and Zhang et al.(2026), the validity of a sampled KL update depends on the gradient estimator and sampling convention. We specify these conventions more precisely without changing the UFKL/URKL variants.

Q4. Why not compare reference-free SDPG with reference-free RLSD/OPCD?

A4. Appendix E.1 already includes α=0, which retains outcome learning, verifier gating, and full-vocabulary OPD, while removing the reference anchor. Its competitive AIME accuracy shows the core SDPG supervision mechanism works without relying on reference KL. This is exactly the conceptual isolation relevant to whether the anchor is essential to making the new signal learnable. A cross-method no-anchor control could refine quantitative attribution, but its absence does not negate the existing within-method isolation; RLSD and OPCD also differ in their basic gradient objective, not only in reference regularization.

Q5. Does positive gating remove the teacher signal when the student cannot solve any problem?

A5. A privileged teacher evaluated on a prefix containing an earlier reasoning error is not automatically an oracle that repairs that history: next-token distribution matching cannot retroactively edit previous tokens. Indiscriminate OPD on such prefixes can therefore create local learning pressure without outcome validation. Under binary rewards, our gate has the exact property mi=1m_i=1 iff response i is correct and the group contains at least 1 incorrect response. The gate targets mixed-success prompts and excludes failed rollouts from the auxiliary imitation objective. Under conditionally independent rollouts of success probability p, the active-group probability is 1−pG−(1−p)G1-p^G-(1-p)^G; hence the limitation at extreme difficulty is explicit and predictable. At the first logged 1.7B update, all-wrong groups account for at least 76.6%(URKL) and 77.7%(UFKL) of 256 groups. Weighted OPD coverage is bounded by 0.49–3.22% and 0.68–3.03% of 2,048 replies, respectively. Fig. 6 shows that pure OPCD's stronger reliance on distillation is not automatically beneficial. We added more detailed materials on this issue in App. F.3.

Q6. What if we gate negative advantages or change the schedule?

A6. Negative gating does not solve the all-wrong-group problem under the specified group-relative estimator: if all rewards are zero, every advantage is zero and both $\mathbf 1

A 0A\>0

$ and $\mathbf 1

A<0A<0

$ are inactive. We found no matched strict-negative or all-wrong fallback run. The adaptive pilot compares two weighting rules with the hard gate off and βbase=0.005\beta_{base}=0.005; its early results are in the supplementary comment. The available schedule runs change multiple factors. Negative gating on mixed-outcome groups instead favors failed trajectories; it could help if the teacher supplies reliable corrective guidance, but that is not guaranteed by privileged conditioning. Positive gating makes the opposite, verifier-grounded choice. The schedule likewise has a specific role: reduce persistent imitation of an unavailable privileged conditional policy while preserving its earlier dense guidance. With the reported parameters it warms up for 50 steps and decays thereafter, without an extended plateau. The choice has a clear mechanism; no claim of unique hyperparameter optimality is required.

Q7. Why can \beta=0 nearly match final SDPG if distillation is important?

A7. The relevant distinction is learning dynamics versus the final checkpoint alone. Figure 5 reports that removing OPD loses the early AIME24/AIME25 accuracy advantage. The auxiliary coefficient is then intentionally annealed to zero at step 400, so the end of SDPG training increasingly resembles reward-plus-anchor optimization. Late-stage catch-up of a β=0\beta=0 model is therefore consistent with a front-loaded learning benefit, not a contradiction of OPD's role. The paper demonstrates an early step-matched benefit, not an unmeasured FLOP-normalized speedup. The fact that the training paths differ even when some endpoints converge is informative evidence about the auxiliary signal.

Q8. Hyperparameters and significance?

A8. The paper reports the exact optimization configuration (α=βbase=10−3\beta_{base}=10^{-3}, Twarm=50T_{warm}=50, Tdecay=350T_{decay}=350, T=400). Their roles are transparent: anchor strength, privileged-gradient strength, and its temporal allocation. Our claim is not that these constants are universally optimal. The logged main setting is α=βbase\beta_{base}=0.001 with 50 warmup and 350 decay steps. The parameter selection procedure is unverified. At step 400, 4B AIME24 differs by only +0.21 pp, while AMC23 gaps are larger at both sizes. The logs lack aligned per-problem scores and verified independent training repeats, so evaluation and seed uncertainties remain separate and unquantified.On reliability, mean@32 is a per-checkpoint sampled-answer statistic, not training-seed variance. We do not use the 4B AIME24 0.002 gap as a significance claim; we instead point to the larger AIME25/AMC23 differences and their replication as observations at both model scales, not a formal multi-seed test.

Q9. Is there interaction among the objectives?

A9. Yes; the interaction is the design. With gout,gOPDandgrefg_{\mathrm{out}}, g_{\mathrm{OPD}} and g_{\mathrm{ref}} denoting the respective gradients, g=gout+β(k)gOPD+\+αgrefg=g_{\mathrm{out}}+\beta(k)g_{\mathrm{OPD}}^+\+\alpha g_{\mathrm{ref}}. The verifier gate removes the OPD contribution on non-positive-advantage trajectories, directly preventing one class of reward–distillation conflict. This does not mathematically guarantee that gradients on correct trajectories are always aligned: all three terms share parameters. The schedule controls the relative strength of the auxiliary gradient over time, and reference KL opposes excessive policy drift. This is a coordinated specification of global outcome selection, local distribution transfer, and policy anchoring, not three independent mechanisms required to improve each accuracy column monotonically.

Q10. Why is there no main-text limitations discussion?

A10. We add a short main-text scope paragraph that makes the tradeoffs explicit without changing the claims: gating intentionally forgoes OPD on all-wrong groups; the privileged branch has extra computation; reference anchoring can trade off with accuracy on a single benchmark; and mean@32 does not quantify between-seed variability. These facts are fully consistent with the theoretical selection and fixed-prefix gradient properties, and with the reported step-matched comparisons. They identify the regime in which the method is designed to operate rather than contradicting the method.

Reviewer_5HEyOpenReview ↗

Thank you for the answers

I have read the answers of the authors and thank them for taking the time. However, I feel in some cases my question has just been reformulated and the gist of the critical weaknesses remain. I therefore maintain my score, which seems to also echo what the other reviewers propose.

We thank the reviewer for the follow-up. However, we respectfully disagree with the characterization that our responses merely reformulate the original concerns. We believe this overlooks several concrete theoretical distinctions and empirical findings already provided in the paper and our rebuttal.

  1. The methodological contribution is substantive, not a reformulation of existing objectives. SDPG differs from RLSD in its optimization mechanism: RLSD reweights sampled-token verifier advantages, whereas SDPG introduces an independent, verifier-gated, full-vocabulary distillation gradient. Moreover, under binary rewards, our positive-advantage gate has an exact property: privileged distillation is applied only to verifier-correct trajectories in mixed-outcome groups. This is a mathematically defined supervision-selection mechanism, not merely a heuristic description.

  2. The existing experiments provide direct evidence for the proposed mechanism. The no-OPD ablation in Figure 5 loses the early AIME24/AIME25 accuracy advantage, demonstrating the contribution of privileged distillation to early optimization. Since the distillation coefficient is deliberately annealed to zero, similar final accuracy does not contradict this role. Furthermore, the improvements over RLSD are not limited to marginal differences. On AMC23, SDPG-UFKL improves final accuracy from 0.813 to 0.858 on Qwen3-4B, and from 0.573 to 0.637 on Qwen3-1.7B. Figure 6 also shows that OPCD suffers substantial performance degradation, whereas SDPG maintains stronger performance. Although these comparisons do not isolate every stabilizer, they provide substantive evidence for the effectiveness of the proposed framework across model scales.

  3. We have also addressed the concrete optimization concern rather than merely restating the formulation. The verified 1.7B implementation directly differentiates the dual-clipped importance-ratio loss using cached rollout-policy log probabilities, with eight optimizer updates per actor batch. The 4B training histories independently record RPGD mode, matching clipping settings, and nonzero clipping fractions, corroborating clipped-ratio optimization. And we have accordingly specified the correction to Eq. (C.2). This implementation issue is separate from, and does not affect, the fixed-prefix OPD gradient identity in Proposition A.1.

  4. Missing fine-grained ablations should not be conflated with an absence of methodological or empirical contributions. We has done a lot of ablation study and added more ablation experiments to the Appendices E and F. The existing theoretical analysis, OPD and reference-KL ablations, and cross-scale benchmark comparisons already address the central methodological hypothesis.

We believe a clear rejection should be grounded in identifiable methodological flaws or evidence that the central contribution is unsupported, rather than solely in the absence of every conceivable component-level ablation. Our response has provided a precise distinction from existing methods, corrected the disputed implementation formula, and identified direct supporting evidence from the existing experiments. We respectfully ask that these substantive points be evaluated on their technical merits.

Reviewer_TWUgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SDPG, which combines verifier-based reinforcement learning with full-vocabulary on-policy self-distillation. The authors derive a policy-gradient interpretation of reverse-KL distillation and introduce positive-advantage gating, a warmup-decay schedule, and reference KL regularization. Experiments on Qwen3-4B and Qwen3-1.7B demonstrate improved mathematical reasoning performance and training stability over GRPO and self-distillation baselines.

Critical strengths
  • Combining verifier-based rewards with full-vocabulary distillation is a reasonable approach. The improvements are fairly consistent across both model sizes, and the training dynamics indicate better stability than RLSD and OPCD.

  • The ablation studies provide useful insights into the roles of OPD and reference KL regularization, showing their different effects on convergence, entropy, and response length.

Critical weaknesses
  • The methodological novelty appears limited. OPSD already employs full-vocabulary distillation, while RLSD combines self-distillation with verifier-based rewards. The policy-gradient equivalence primarily reformulates the reverse-KL gradient rather than introducing a fundamentally new optimization method.

  • Important ablations for positive-advantage gating and the warmup-decay schedule are missing. Furthermore, full-vocabulary OPD introduces additional computation, but the paper does not report FLOPs or training-time comparisons. It is therefore unclear whether the relatively modest improvements over RLSD justify the additional computational cost.

Questions for authors
  • What are the individual contributions of positive-advantage gating and the warmup-decay schedule?

  • How does SDPG compare with RLSD under matched training FLOPs or GPU-hour budgets?

3 comments

We appreciate the novelty and efficiency questions. Here are our responses.

Q1. Are OPSD's full-vocabulary KL, RLSD's verifier reward and the reverse-KL identity already known? What is the actual methodological distinction?

A1. The distinction is the interaction rule and the gradient information available at selected prefixes. OPSD performs privileged distribution matching, but its distillation term is not selected using a binary verifier. RLSD introduces privileged information through a sampled-token multiplicative adjustment to the verifier advantage (Appendix B, Eqs. (B.3)–(B.7)); it does not optimize a separate full-vocabulary teacher-student KL objective. SDPG has an independent term$\mathcal L_{\mathrm{OPD}}^+ =\mathbb E!\left[\frac{1}{\sum_i|y_i|}\sum_{i,t} \mathbf 1[A_i>0]D_{\mathrm{KL}}!\left(p_{i,t}|\operatorname{SG}[q_{i,t}]\right)\right].$ Its fixed-prefix gradient depends on all token probabilities, not solely on the realized yi,ty_{i,t}. The verifier decides where this complete distribution-level signal is permitted to act, while the exact KL determines how the student distribution moves there. The local gradient identity in Proposition A.1 is an exact interpretation of that design, not a claim to have invented the score-function trick itself. The contrast is therefore structural: verifier-conditioned full-vocabulary distribution matching versus either unconditioned distribution matching or sampled-token outcome-gradient reweighting. The experimental distinctions are also substantial. Table 1 gives 4B AMC23 Last 0.863/0.858 for the two SDPG variants versus RLSD 0.813. Table 2 gives 1.7B AMC23 Last 0.620/0.637 versus RLSD 0.573. The pure OPCD baseline deteriorates strongly late in the 1.7B run (Figure 6). These are observations consistent with the motivation for combining dense distillation with verifier grounding; they are not presented as a controlled causal estimate of gating alone.

Q2. What do gating and the schedule contribute individually?

A2. They have mathematically nonoverlapping functions. Existing runs do not isolate the positive OPD gate or the warmup/decay schedule. β=0 removes OPD altogether; the closest vocabulary comparison is preliminary and appears in the supplementary comment. Scheduling weights that selected signal over time: with the reported parameters T=400, Twarm=50T_{warm}=50, and Tdecay=350T_{decay}=350, β(k)=βbasek/50\beta(k)=\beta_{base}k/50, if 0≤k≤50 and β(k)=βbase(400−k)/350\beta(k)=\beta_{base}(400-k)/350 if 50<k≤400. This schedule progressively removes pressure to imitate a privileged conditional distribution that is not itself available at deployment. A useful formal illustration is that, for a fixed deployable prefix s and a family of detached privileged targets qc(a)q_c(a), the solution to min⁡pEcDKL(p∣qc)\min_p\mathbb E_c D_{KL}(p|q_c) is generally p∗(a)∝exp⁡(Ec[log⁡qc(a)]),p^*(a)\propto\exp\left(\mathbb E_c[\log q_c(a)]\right),rather than any single privileged teacher. This is a mathematical reason not to require persistent perfect agreement with each unavailable privileged context; it does not assert that our specific decay schedule is uniquely optimal. Unlike scheduling, gating directly excludes imitation on verifier-rejected trajectories. The ablation β=0 already loses SDPG's early AIME accuracy advantage (Figure 5). These facts establish distinct mechanisms and support the full design; additional one-factor runs would quantify the isolated effect sizes rather than define the method.

Q3. What happens under equal FLOPs/GPU-hours with RLSD?

A3. The published comparison is matched in optimizer steps, not asserted to be matched in FLOPs. This distinction does not make the comparison uninformative: it establishes an optimization trajectory on the common reported protocol. The privileged branch adds computation, but the extra vocabulary-wide work purchases a precisely defined estimator advantage. If g(a,s) is a valid sampled-token representation of the fixed-prefix OPD gradient, gfull(s)=Ea∼p(⋅∣s)g(a,s),Cov(g^N∣s)=1NCov(g(a,s)∣s)g_{full}(s)=E_{a\sim p(\cdot\mid s)}g(a,s), Cov(\hat g_N\mid s)=\frac1NCov(g(a,s)|s) for N conditionally independent token samples, while the exact vocabulary sum has zero token-sampling variance at that prefix. This is a Rao–Blackwellization-style rationale, not a claim of zero trajectory variance or proven wall-clock superiority. SDPG and RLSD both involve privileged information, so their actual cost difference depends on how the implementations reuse logits and activations; it cannot be reliably inferred from the loss names alone. We keep the concrete conclusion warranted by the present data—step-matched learning improvements—and state the compute accounting separately where existing logs are available Under the logged step-400 RLSD GPU-hour cap, the last eligible SDPG checkpoints score 0.846/0.851 versus 0.813 on 4B AMC23 and 0.628/0.592 versus 0.573 on 1.7B AMC23(UFKL/URKL). Both trail on AIME24. This retrospective GPU-hour comparison does not measure FLOPs. No FLOP-matched superiority is claimed. We have updated the results in Appendix F.1.

Thank you for the detailed rebuttal and clarifications.

I appreciate the clearer distinction between SDPG and existing methods, as well as the additional computational analysis.

However, my main concerns remain. The individual contributions of positive-advantage gating and the warmup-decay schedule are still not experimentally isolated. Moreover, the additional GPU-hour comparisons do not establish a consistent efficiency advantage over RLSD.

While the empirical results are encouraging, I still find the methodological novelty and evidence supporting the key design choices somewhat limited.

Therefore, I will maintain my current rating.

We thank the reviewer for the follow-up. We appreciate the acknowledgment of our methodological clarification and additional computational analysis. We would like to emphasize two points that we believe are important for the final assessment.

First, the compute-matched results provide more positive evidence than the characterization of an inconsistent efficiency advantage may suggest. In Table 4, under comparable total GPU-hour budgets, SDPG-UFKL outperforms RLSD on four of the six benchmark/model-scale comparisons. Most notably, on AMC23, SDPG-UFKL improves accuracy from 0.813 to 0.846 on 4B (153.74 vs. 156.74 GPU-hours) and from 0.573 to 0.628 on 1.7B (111.49 vs. 112.55 GPU-hours). These gains are achieved despite the additional full-vocabulary computation. While the benefits are task-dependent, the results demonstrate that the additional computation can translate into meaningful improvements under matched budgets, rather than merely faster progress per optimization step.

Second, we believe it is important to distinguish the empirical attribution of individual stabilizers from the methodological contribution of SDPG. Positive-advantage gating and the coefficient schedule have mathematically distinct roles: the former restricts privileged distillation to verifier-correct trajectories in mixed-outcome groups, while the latter controls the temporal influence of privileged supervision. More fundamentally, SDPG differs from RLSD by optimizing an independent full-vocabulary distillation objective rather than reweighting sampled-token verifier advantages. This distinction does not depend on an individual ablation of either stabilizer.

We agree that additional single-factor ablations would provide finer empirical attribution. However, the combination of a distinct optimization mechanism, the existing OPD/reference-KL ablations, consistent improvements across model scales, and the newly reported compute-matched gains provides substantive support for the proposed framework.

We therefore respectfully recommend to evaluate SDPG's methodological novelty and empirical effectiveness separately from the remaining question of fine-grained component attribution.

Reviewer_SdtLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SDPG combines an outcome based policy gradient with full vocabulary distillation from a privileged branch of the same model and a reference policy KL term. Distillation is applied only to responses with positive group relative advantage, and its weight is warmed up and then decayed. The paper derives a policy gradient interpretation of the reverse KL derivative at a fixed sampled prefix. Experiments on Qwen3 4B and 1.7B compare the resulting training procedure with GRPO, RLSD, and, at the smaller scale, OPCD on three mathematics benchmarks.

Critical strengths
  1. The formulation separates three sources of supervision clearly: the verifier chooses which outcomes to reinforce, privileged context supplies a dense token distribution, and the reference policy limits movement from the starting model. Algorithm 1 and the detached teacher definition in Equation 3.4 make the intended computation reasonably easy to follow.

  2. The gains are not confined to a favorable intermediate checkpoint. Table 1 reports both the last and best results, and both SDPG variants improve on the main baselines at the 400 step endpoint. The AMC23 result is particularly notable: the forward KL variant reaches 0.858 compared with 0.813 for RLSD. Appendix E.2 finds a similar ordering at 1.7B, where the corresponding endpoint scores are 0.637 and 0.573.

  3. Including complete training curves is helpful for this problem. Figure 2 shows differences in entropy and response length that endpoint accuracy alone would conceal. Appendix E also removes the OPD and reference KL terms separately, so the empirical case is not based solely on comparing the full method with other named algorithms.

Critical weaknesses
  1. It is not clear which update produced the main results. Section 3.2 describes an on policy update without PPO clipping, while Appendix C.3 gives an approximation for reused rollouts. In Equation C.2, page 19, the entire clipped loss from C.1 is detached and multiplied by the token log probability. For a positive advantage with an importance ratio above the upper clipping boundary, ordinary PPO stops that token's contribution to the gradient; C.2 still produces an update proportional to the clipped advantage. This could be intentional weight capping, but it does not preserve the usual clipping behavior. Please identify the objective used for Tables 1 and 2 and the number of optimizer updates per rollout batch. The printed formula establishes a discrepancy, not that the unseen training code necessarily contains it.

  2. The experiments do not isolate several decisions presented as central to SDPG. Appendix E.1 tests removing OPD and removing the reference KL, but not positive advantage gating, the warmup and decay schedule, or full vocabulary versus sampled token distillation within the same objective. In particular, Section 4.2 attributes stability to gating and scheduling without separating their effects. Retaining the verifier objective and reference KL while changing only the gate, schedule, or estimator would be more informative than another model scale. As it stands, the results support the combined recipe more clearly than its explanation.

  3. Reliability is difficult to judge for the smaller differences. The tables report mean@32, which averages sampled answers rather than independent training runs; they do not give uncertainty across training seeds. On 4B AIME24, the final score is 0.380 for either SDPG variant versus 0.378 for RLSD, whereas the separation on AMC23 is much larger. Those are materially different levels of evidence. Please report paired uncertainty over evaluation problems and variation across a modest number of training seeds. Additional answers to the same evaluation problems cannot by themselves establish training stability.

  4. Training steps are not a sufficient measure of the cost of this comparison. Algorithm 1 evaluates a privileged branch and a full vocabulary KL at sampled prefixes, while Section 4.1 also uses privileged information generated by Gemini 2.5 Pro. These are legitimate sources of supervision, but their costs and availability need to be distinguished from verifier only training. Please report elapsed training time, peak memory, and the cost of preparing the privileged contexts, and confirm which baselines receive exactly the same contexts. The faster progress in Figure 2 has not yet been translated into a measured compute advantage.

Questions for authors
  1. The motivation for unnormalized KL conflates a distribution objective with a sampled surrogate. For normalized student and reference policies, the mass correction in Equation 2.4 is identically zero, so UKL and KL are the same population divergence. Different sampled surrogates can have different variance and stationarity properties. Section 3.4 and Appendix C.1 describe an inherent bias of normalized KL, even though C.1 also derives unbiased expected gradient identities for its normalized KL surrogates. Please identify the particular biased estimator and sampling convention rather than attributing the problem to KL itself.

  2. Appendix E.1 says that removing the reference KL achieves comparable or slightly higher AIME accuracy, but interprets shorter responses and rising entropy as departure from coherent reasoning. Neither observation alone establishes that diagnosis. A direct error or coherence analysis would support it; otherwise, describe these as changes in generation behavior rather than demonstrated degradation.

  3. How many groups contribute privileged supervision over the course of training? With binary rewards and the stated gate, both uniformly incorrect and uniformly correct groups have no positive advantage. Appendix A discusses the edge cases, but reporting the active fraction would show whether the distillation term remains substantial or largely disappears as training progresses.

  4. Define the quantity plotted as reward in Figures 2, 5, and 6. Section 2.1 illustrates correctness rewards as one and zero, whereas the plotted reward can be negative and Figure 2 calls it “group relative reward.” Binary rewards can use other encodings, so negative values are not inherently inconsistent; the missing detail is the actual transformation, penalty, or averaging rule used here.

  5. Section 4.1 gives the training rollout temperature as 1.0. Please also specify the evaluation temperature and checkpoint evaluation interval used for mean@32 and the Best columns. In Appendix E.2, correct the statement that UFKL wins five of the six columns: Table 2 shows four UFKL wins and two URKL wins.

3 comments

We thank the reviewer for the technically careful assessment. Here are our responses to your question.

Q1. Section 3.2 gives an on-policy score-function update, whereas Appendix C.3 appears to reuse rollouts and detach a clipped loss. Which update produced Tables 1–2? Is Appendix Eq. (C.2) equivalent to PPO clipping?

A1. These are two levels of description: Eq. (3.3) specifies the fresh-rollout reward-gradient surrogate, which describes the fresh-policy case; Appendix C.3 addresses possible behavior-policy staleness when optimizing a frozen rollout batch. Neither changes the defining full-vocabulary OPD term of Eq. (3.4). The reviewer is correct about the specific algebraic distinction in the printed Eq. (C.2): the gradient of $SG

ℓi,tclip\ell_{i,t}^{clip}

\log\pi_\theta(y_{i,t}| s_{i,t})$ is generally not the gradient of the PPO clipped objective. In particular, coefficient detachment can leave a nonzero gradient beyond the PPO clipping boundary. The appropriate correction is to distinguish a genuine clipped importance-ratio objective from a detached-coefficient approximation, rather than to conflate them. We have replaced Eq. (C.2) with the directly differentiated ratio loss verified in the 1.7B source. The 4B logs corroborate the RPGD mode and clipping, while the original commit remains unverified. Proposition A.1 concerns the separate fixed-prefix OPD term. The 1.7B source verifies a directly differentiated dual-clipped verifier ratio loss using cached rollout-policy log probabilities. Both 4B histories record RPGD, matching clipping settings and nonzero clipping fractions. A full actor batch schedules eight optimizer updates in one epoch. The original 4B SDPG commits remain unverified.

Q2. Why are the gate, schedule and exact vocabulary estimator not separately ablated?

A2. Their roles can be distinguished analytically, without treating them as three interchangeable heuristics. Let K of G binary-reward responses be correct. Since Ai=(Ri−μG)/(σG+ϵstd)A_i=(R_i-\mu_G)/(\sigma_G+\epsilon_{std}), ∑i=1G1[Ai>0]=K,1[0<K<G].\sum_{i=1}^{G}\mathbf 1[A_i>0]=K,\mathbf 1[0<K<G]. The gate selects correct replies from mixed-outcome groups. The schedule reduces the weight of privileged guidance over time. The full-vocabulary sum computes the exact conditional gradient for a fixed prefix and teacher. Fig.5 compares the full recipe with β=0 and shows an early AIME advantage. Pure OPCD deteriorates late in the 1.7B run (Fig.6 and Tab.2). We found no gate-only or schedule-only run. The closest 1.7B vocabulary comparison used different commits, and its sampled branch stopped at step 202. The comment gives the step-200 results.

Q3. Mean@32 is not training-seed uncertainty; what about the AIME24 near tie? Can the small margins establish reliability?

A3. We agree with the distinction between the evaluation estimand and independent training runs, but a near tie in one column should not be conflated with the entire evidence. We do not use the 4B AIME24 Last margin 0.380-0.378=0.002 as a significance claim. The larger observed differences are, for example, 4B AMC23 Last 0.863(SDPG-URKL) versus 0.813(RLSD), 4B AIME25 Last 0.327(SDPG-UFKL) versus 0.300, and 1.7B AMC23 Last 0.637(SDPG-UFKL) versus 0.573(RLSD). At step 400, 4B AIME24 was 38.02% for both SDPG variants versus 37.81% for RLSD (+0.21 pp). AMC23 gaps were +4.45 to +5.00 pp at 4B and +4.69 to +6.41 pp at 1.7B. Aligned problem scores and verified independent training seeds were unavailable, so both uncertainties remain unquantified. Mean@32 averages sampled-answer correctness across the evaluated problems; it is not a multi-training-seed confidence interval. We make no claim of statistical significance for the 0.002 AIME24 margin or of measured between-seed variance. Our claim is the observed improvement pattern under the specified, step-matched evaluation, not a claim that repeated decoding establishes training-seed stability.

Q4. How costly is the privileged branch, and do baselines receive the same information?

A4. We agree that optimization-step efficiency and end-to-end compute efficiency are different questions. Our paper demonstrates the former in the reported trajectories; it does not infer a GPU-hour advantage solely from Fig.2. Privileged contexts are prepared offline with Gemini 2.5 Pro and correctness filtering (Sec. 4.1, App. A.4). During training, SDPG evaluates a context-conditioned branch sharing the student parameters, which saves the need for a separately parameterized teacher but not the additional computation. The full-vocabulary reduction is also not free. The cost has a clear statistical purpose: for the same prefix and detached teacher, full-vocabulary summation eliminates the additional conditional token-sampling variance of a Monte Carlo gradient estimator. Both cost and estimator quality should be kept in view. The comparison should also distinguish verifier-only GRPO (no privileged context needed) from privileged methods. Within each size, SDPG and RLSD point to the same privileged-data filename, as does 1.7B OPCD; file identity is unverified. Training plus periodic evaluation used 52–61% more GPU-hours at 4B and 38–41% more at 1.7B than RLSD. We recovered rollout, actor, eval, and actor allocator peaks and added to App. F.1. Offline preparation was unlogged.

Q5. Is the motivation for UKL incorrectly attributed to normalized KL itself?

A5. The reviewer raises a useful population-objective versus sample-gradient distinction. For normalized p and q, UKL(p∥q)=DKL(p∥q)+∑a[q(a)−p(a)]=DKL(p∥q).UKL(p\|q)=D_{KL}(p\|q)+\sum_a[q(a)-p(a)]=D_{KL}(p\|q). Our reason for considering the UFKL/URKL forms is the differentiated on-policy sample surrogate, not a different population divergence. With r_\theta(a)=\pi_\theta(a\mid s)/\pi_{\mathrm{ref}}(a\mid s) and s fixed, ∇θDKL(πθ∥πref)=Ea∼πθ[log⁡rθ(a) ∇θlog⁡πθ(a∣s)],\nabla_\theta D_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{ref}}) =\mathbb E_{a\sim\pi_\theta}\left[\log r_\theta(a)\,\nabla_\theta\log\pi_\theta(a\mid s)\right],

∇θDKL(πref∥πθ)=Ea∼πθ[(1−rθ(a)−1)∇θlog⁡πθ(a∣s)].\nabla_\theta D_{\mathrm{KL}}(\pi_{\mathrm{ref}}\|\pi_\theta) =\mathbb E_{a\sim\pi_\theta}\left[(1-r_\theta(a)^{-1})\nabla_\theta\log\pi_\theta(a\mid s)\right].

Holding a sampled action fixed, the derivatives of 12(log⁡rθ)2\tfrac12(\log r_\theta)^2 and rθ−1+log⁡rθr_\theta^{-1}+\log r_\theta produce the two corresponding factors. By contrast, directly differentiating a sampled log⁡rθ\log r_\theta gives only ∇log⁡πθ\nabla\log\pi_\theta, whose on-policy expectation is zero. The issue is which estimator is differentiated under which sampling and stop-gradient convention. We correct the shorthand “inherent bias of normalized KL” accordingly while preserving the UFKL/URKL constructions. This also makes the connection to Tang and Munos (2025) and Zhang et al. (2026) precise.

Q6. Do shorter responses and rising entropy without reference KL prove incoherence?

A6. They should be read as behavioral effects of removing an active regularizer, not as a stand-alone proof that every shorter reasoning trace is incoherent. Reference KL serves a different optimization purpose from accuracy: for normalized distributions, Pinsker's inequality gives

TV⁡(πθ,πref)≤12DKL(πθ∥πref).\operatorname{TV}(\pi_\theta,\pi_{\mathrm{ref}}) \leq\sqrt{\tfrac12D_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{ref}})}.

It therefore supplies a principled population-level measure of departure from a fixed reference, even though the coefficient \alpha is a soft constraint and no numerical bound is established for the trained model from the present plots. The \alpha=0 ablation's comparable/higher AIME accuracy is fully compatible with that distinct anchoring role: maximizing a regularized objective need not maximize each unregularized benchmark. Figure 5 shows the anchor materially changes length/entropy behavior.

Q7. What fraction of groups actually contributes OPD?

A7. The fraction has an exact reward-defined characterization, which we now state explicitly. Let KjK_j be the number of correct responses in group j. Then group j is active iff 0<Kj<G0<K_j<G, and it contributes exactly $K_j $ positively gated responses. Accordingly, across the actual collected groups, Fgroup=1N∑j=1N1[0<Kj<G],Fresponse=1NG∑j=1NKj1[0<Kj<G].F_{\mathrm{group}}=\frac{1}{N}\sum_{j=1}^{N}\mathbf 1[0<K_j<G],\qquad F_{\mathrm{response}}=\frac{1}{NG}\sum_{j=1}^{N}K_j\mathbf 1[0<K_j<G]. Under an illustrative, conditionally independent Bernoulli model with per-prompt success probability p, these become 1−pG−(1−p)G1-p^G-(1-p)^G and p(1−pG−1)p(1-p^{G-1}). The four histories bound coverage among 128/256 logged groups and 1,024/2,048 replies at 4B/1.7B, respectively. Exact counts and pre-filter coverage are unavailable. Weighted OPD coverage is zero at step 400 because β = 0. See the comment for the bounds.

Q8. Why is the reward curve negative if correctness rewards are binary?

A8. The binary outcome label Ri∈0,1R_i\in\\{0,1\\} and the training metric plotted as “reward” must not be confused. The group-relative advantage Ai=(Ri−μG)/(σG+ϵstd)A_i=(R_i-\mu_G)/(\sigma_G+\epsilon_{\mathrm{std}}) can be negative. However, because its within-group mean is zero, the nonzero plotted curves cannot be identified as the unweighted mean of all such advantages. The appropriate response is an exact metric definition rather than guessing a penalty or transformation. Figures 2(d), 5(d), and 6(d) use critic/score/mean: sum verifier token scores per reply, then average ±1 scores over non-aborted batch replies before KL shaping, with no extra plot-time outcome filtering. Figures 2 and 5 use a centered seven-step mean; Figure 6 uses EMA weight 0.04. We have added explanation on this in Appendix F.4.

Q9. Evaluation temperature, checkpoint interval and Table 2 wording?

A9. The rollout temperature of 1.0 is already reported in Section 4.1; evaluation decoding needs to be specified separately. Evaluation used 32 sampled answers per problem, temperature 1, top-p 1, no top-k cutoff, and 4,096/2,048 response tokens at 4B/1.7B. Last is step 400; Best is the per-benchmark maximum over steps 10–400 at intervals of 10. Table 2 UFKL AIME24 Best rounds to 0.211 (not 0.212); rankings are unchanged. In Tables 1 and 2, “Last” is step 400 and “Best” is the maximum over evaluated checkpoints. And we have also updated for the statement in E.2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We have uploaded a revised PDF. New and changed text is in red. The title, the section structure and the theorem numbering are unchanged, and the main text stays within 9 pages.

Main text

  • Sec. 2.3: we rephrase the last paragraph to remove the unverified wording on lower variance
  • Sec. 3.2: we correct the Eq. (3.2) and add more explanatory words
  • Sec. 4.1: we add more details on the differences between the experiments of Qwen3-1.7B and Qwen3-4B.
  • Sec. 4.2: we add more explanations on the function of positive-advantage gating
  • Sec. 6: we add more conclusions on the functions of features including positive-advantage and full-vocabulary

Appendix

  • Sec. A.3: we correct the differentiation equation and rephrase our wording on the overstrong explanatory words.
  • Sec. C.1: we add more explanatory words to clarify the relation between normalized KL variants and unnormalized KL variants
  • Sec. C.3: we correct Eq. (C.2) and definition of ρi,t\rho_{i,t}, and correct the explanations of the final expression of SDPG approximation.
  • Sec. E.1/E.2: we correct the overclaiming on the performance of the proposed methods and fix a bug in the results (0.212$\to$0.211, due to rounding errors.)
  • Sec. F: we added more details and ablation studies of the experiments, including the GPU memory usage, execution time, ablation on the features as well as the explanation on the reasonability of the grouped advantages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.