- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SDPG, a loss for LLM reinforcement learning with verifiable reward (RLVR) and privileged information. Based on its derivation of the self-distillation loss as a policy gradient whose advantage appears to reinforce wrong trajectories, they suggest SDPG, a loss that combines the following components:
- a REINFORCE loss with group-level advantage from the verifiable reward.
- an auxiliary on-policy self-distillation from privileged information loss (reverse KL), gated on positive-advantage only and weighted by a warmup-decay schedule.
- an auxiliary on-policy KL loss to a reference policy, whose reverse and forward formulations can both be applied to on-policy samples.
Alongside the derivation of these losses, they claim that this combination of the losses fixes failure cases in on-policy self-distillation from privileged information (OPSD) and beats GRPO and other OPSD baselines.
The authors report results on math benchmarks (AIME24, 25, and AMC23) from two RL training runs on a math dataset (DAPO-Math-17k), one with Qwen3-4B and one with Qwen3-1.7B, and compare to GRPO and RLSD (which uses self-distillation to weigh reward-based advantages) and OPCD (classical on-policy self-distillation with full vocab). They also perform two ablations, removing each of the auxiliary losses.
Critical strengths
SDPG appears to continue improving in a stable manner on both runs with Qwen3 4B and 1.7B, while other self-distillation baselines seem to plateau or collapse (Figure 2abdc, Figure 6abcd). RLVR and self-distillation methods are known to suffer from instabilities and plateaus, preventing them from scaling. A recipe that allows further scaling is an important contribution to the field.
The motivation to schedule the self-distillation feedback as an auxiliary loss so that it mainly kicks in at the beginning, to limit and reduce the bias as training progresses, is interesting.
Critical weaknesses
The critical weakness of the paper is that it reads as an incomplete study. It identifies a relevant limitation in previous methods (the distillation loss can reinforce wrong trajectories), designs a complex solution to address it (3 objectives: reward + gated and scheduled distillation + reference regularization), but fails to sufficiently show or explain how each of the components in the proposed solution helps and how it outperforms prior work. Some motivated solutions do not discuss their flipside and there is insufficient experimental evidence or depth in its analysis to conclude.
- Why replace the GRPO loss with a REINFORCE loss? Is there off-policy in the experiments? Otherwise, GRPO is itself REINFORCE.
- Why start with the reference-regularized framework (RPG)? Why is regularization important? The no-reference-KL ablation seems to outperform SDPG, making its contribution unclear, and the discussion is missing relevant work discussing its need with GRPO, at least for math (e.g. in [1]). Does GRPO use KL-regularization in the experiments?
- The on-policy forward/reverse KL to the reference policy is already present in [2] and [3]; the incremental contribution needs clarification.
- The scheduled and gated self-distillation loss should be compared without regularization to the other self-distillation baselines, RLSD and OPCD, to isolate the fix to the main identified challenge (SD loss can reinforce bad trajectories)
- Also, the gating and scheduling are not individually ablated or studied more carefully. A major advantage of SDPO is that even when the student can't find solutions, the teacher can steer it towards solutions, but a flipside of gating on positive advantages is that it removes this signal and can drastically reduce efficiency.
- How does performance change when changing the distillation loss schedule? When gating the distillation on negative advantages?
- And, the no-distillation ablation seems to match SDPG, giving no clear benefit of distillation. This is surprising.
- No hyperparameter search or documented selection procedure, and no statistical significance. How were the hyperparameters for SDPG selected?
- Is there any interplay between the auxiliary losses?
- No limitations are discussed in the main text, including the flipside of the suggested fixes.
[1] Liu, Zichen, et al. "Understanding r1-zero-like training: A critical perspective." arXiv preprint arXiv:2503.20783 (2025). [2] Tang, Yunhao, and Rémi Munos. "On a few pitfalls in KL divergence gradient estimation for rl." arXiv preprint arXiv:2506.09477 (2025). [3] Zhang, Yifan, et al. "On the design of kl-regularized policy gradient algorithms for llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
see above
4 comments
Q6. What if we gate negative advantages or change the schedule?
A6. Negative gating does not solve the all-wrong-group problem under the specified group-relative estimator: if all rewards are zero, every advantage is zero and both $\mathbf 1
$ and $\mathbf 1
$ are inactive. We found no matched strict-negative or all-wrong fallback run. The adaptive pilot compares two weighting rules with the hard gate off and ; its early results are in the supplementary comment. The available schedule runs change multiple factors. Negative gating on mixed-outcome groups instead favors failed trajectories; it could help if the teacher supplies reliable corrective guidance, but that is not guaranteed by privileged conditioning. Positive gating makes the opposite, verifier-grounded choice. The schedule likewise has a specific role: reduce persistent imitation of an unavailable privileged conditional policy while preserving its earlier dense guidance. With the reported parameters it warms up for 50 steps and decays thereafter, without an extended plateau. The choice has a clear mechanism; no claim of unique hyperparameter optimality is required.
Q7. Why can \beta=0 nearly match final SDPG if distillation is important?
A7. The relevant distinction is learning dynamics versus the final checkpoint alone. Figure 5 reports that removing OPD loses the early AIME24/AIME25 accuracy advantage. The auxiliary coefficient is then intentionally annealed to zero at step 400, so the end of SDPG training increasingly resembles reward-plus-anchor optimization. Late-stage catch-up of a model is therefore consistent with a front-loaded learning benefit, not a contradiction of OPD's role. The paper demonstrates an early step-matched benefit, not an unmeasured FLOP-normalized speedup. The fact that the training paths differ even when some endpoints converge is informative evidence about the auxiliary signal.
Q8. Hyperparameters and significance?
A8. The paper reports the exact optimization configuration (α=, , , T=400). Their roles are transparent: anchor strength, privileged-gradient strength, and its temporal allocation. Our claim is not that these constants are universally optimal. The logged main setting is α==0.001 with 50 warmup and 350 decay steps. The parameter selection procedure is unverified. At step 400, 4B AIME24 differs by only +0.21 pp, while AMC23 gaps are larger at both sizes. The logs lack aligned per-problem scores and verified independent training repeats, so evaluation and seed uncertainties remain separate and unquantified.On reliability, mean@32 is a per-checkpoint sampled-answer statistic, not training-seed variance. We do not use the 4B AIME24 0.002 gap as a significance claim; we instead point to the larger AIME25/AMC23 differences and their replication as observations at both model scales, not a formal multi-seed test.
Q9. Is there interaction among the objectives?
A9. Yes; the interaction is the design. With denoting the respective gradients, . The verifier gate removes the OPD contribution on non-positive-advantage trajectories, directly preventing one class of reward–distillation conflict. This does not mathematically guarantee that gradients on correct trajectories are always aligned: all three terms share parameters. The schedule controls the relative strength of the auxiliary gradient over time, and reference KL opposes excessive policy drift. This is a coordinated specification of global outcome selection, local distribution transfer, and policy anchoring, not three independent mechanisms required to improve each accuracy column monotonically.
Q10. Why is there no main-text limitations discussion?
A10. We add a short main-text scope paragraph that makes the tradeoffs explicit without changing the claims: gating intentionally forgoes OPD on all-wrong groups; the privileged branch has extra computation; reference anchoring can trade off with accuracy on a single benchmark; and mean@32 does not quantify between-seed variability. These facts are fully consistent with the theoretical selection and fixed-prefix gradient properties, and with the reported step-matched comparisons. They identify the regime in which the method is designed to operate rather than contradicting the method.
Thank you for the answers
I have read the answers of the authors and thank them for taking the time. However, I feel in some cases my question has just been reformulated and the gist of the critical weaknesses remain. I therefore maintain my score, which seems to also echo what the other reviewers propose.
We thank the reviewer for the follow-up. However, we respectfully disagree with the characterization that our responses merely reformulate the original concerns. We believe this overlooks several concrete theoretical distinctions and empirical findings already provided in the paper and our rebuttal.
-
The methodological contribution is substantive, not a reformulation of existing objectives. SDPG differs from RLSD in its optimization mechanism: RLSD reweights sampled-token verifier advantages, whereas SDPG introduces an independent, verifier-gated, full-vocabulary distillation gradient. Moreover, under binary rewards, our positive-advantage gate has an exact property: privileged distillation is applied only to verifier-correct trajectories in mixed-outcome groups. This is a mathematically defined supervision-selection mechanism, not merely a heuristic description.
-
The existing experiments provide direct evidence for the proposed mechanism. The no-OPD ablation in Figure 5 loses the early AIME24/AIME25 accuracy advantage, demonstrating the contribution of privileged distillation to early optimization. Since the distillation coefficient is deliberately annealed to zero, similar final accuracy does not contradict this role. Furthermore, the improvements over RLSD are not limited to marginal differences. On AMC23, SDPG-UFKL improves final accuracy from 0.813 to 0.858 on Qwen3-4B, and from 0.573 to 0.637 on Qwen3-1.7B. Figure 6 also shows that OPCD suffers substantial performance degradation, whereas SDPG maintains stronger performance. Although these comparisons do not isolate every stabilizer, they provide substantive evidence for the effectiveness of the proposed framework across model scales.
-
We have also addressed the concrete optimization concern rather than merely restating the formulation. The verified 1.7B implementation directly differentiates the dual-clipped importance-ratio loss using cached rollout-policy log probabilities, with eight optimizer updates per actor batch. The 4B training histories independently record RPGD mode, matching clipping settings, and nonzero clipping fractions, corroborating clipped-ratio optimization. And we have accordingly specified the correction to Eq. (C.2). This implementation issue is separate from, and does not affect, the fixed-prefix OPD gradient identity in Proposition A.1.
-
Missing fine-grained ablations should not be conflated with an absence of methodological or empirical contributions. We has done a lot of ablation study and added more ablation experiments to the Appendices E and F. The existing theoretical analysis, OPD and reference-KL ablations, and cross-scale benchmark comparisons already address the central methodological hypothesis.
We believe a clear rejection should be grounded in identifiable methodological flaws or evidence that the central contribution is unsupported, rather than solely in the absence of every conceivable component-level ablation. Our response has provided a precise distinction from existing methods, corrected the disputed implementation formula, and identified direct supporting evidence from the existing experiments. We respectfully ask that these substantive points be evaluated on their technical merits.
We appreciate the reviewer's detailed analysis but respectfully disagree with the characterization of SDPG as an arbitrary three-loss recipe. And the responses are shown below.
A1. We do not claim that REINFORCE and GRPO are fundamentally distinct. GRPO is a group-advantage policy-gradient method, with clipping becoming relevant when the optimized policy moves away from the frozen rollout policy. Sec. 3.2 expresses the fresh-rollout score-function form so that its outcome signal can be placed beside the fixed-prefix distillation gradient. This makes the two sources comparable; it is not an attempt to replace GRPO's underlying principle. The verified 1.7B verifier objective directly differentiates the clipped ratio against the cached rollout policy, with group-relative advantages. Each full actor batch schedules eight minibatch updates in one epoch. 4B logs corroborate RPGD and clipping; the original 4B source revision remains unverified. The detached expression in the printed Eq. (C.2) should not be conflated with exact PPO clipping, and the appendix is clarified accordingly. None of this changes the local OPD identity in Prop. A.1.
A2. A regularizer does not need to increase every unregularized evaluation score to have a distinct optimization role. Reference KL penalizes departure from a fixed pretrained distribution; the intended population-level anchoring role is formalized, for example, by TV(p,pref)≤DKL(p∣pref)/2. Thus, maximizing reward plus OPD minus a KL penalty is a different criterion from maximizing accuracy alone. The α=0 ablation actually shows that the anchor has a measurable behavioral effect: generation length and entropy change appreciably(Figure 5). We do not need to equate shorter answers with invalid reasoning to establish this role. The anchor is a deliberately chosen conservative component, not a claim that mathematics RL fails without it. We discuss Liu et al.(2025) in this context. Both GRPO baselines used reference-KL reward shaping with coeff. 0.001 and disabled a separate actor KL loss. RLSD uses sampled-token teacher reweighting in the verifier update. The original 4B RLSD patch remains unavailable.
A3. We do not claim priority for forward/reverse reference KL or the score-function identity. They are the anchoring machinery of the framework, not its unique privileged-supervision mechanism. SDPG's method-specific contribution is a separate verifier-gated, full-vocabulary privileged teacher-student term, which is neither a reference KL nor the sampled-token reweighting used by RLSD. As with Tang&Munos(2025) and Zhang et al.(2026), the validity of a sampled KL update depends on the gradient estimator and sampling convention. We specify these conventions more precisely without changing the UFKL/URKL variants.
A4. Appendix E.1 already includes α=0, which retains outcome learning, verifier gating, and full-vocabulary OPD, while removing the reference anchor. Its competitive AIME accuracy shows the core SDPG supervision mechanism works without relying on reference KL. This is exactly the conceptual isolation relevant to whether the anchor is essential to making the new signal learnable. A cross-method no-anchor control could refine quantitative attribution, but its absence does not negate the existing within-method isolation; RLSD and OPCD also differ in their basic gradient objective, not only in reference regularization.
A5. A privileged teacher evaluated on a prefix containing an earlier reasoning error is not automatically an oracle that repairs that history: next-token distribution matching cannot retroactively edit previous tokens. Indiscriminate OPD on such prefixes can therefore create local learning pressure without outcome validation. Under binary rewards, our gate has the exact property mi=1 iff response i is correct and the group contains at least 1 incorrect response. The gate targets mixed-success prompts and excludes failed rollouts from the auxiliary imitation objective. Under conditionally independent rollouts of success probability p, the active-group probability is 1−pG−(1−p)G; hence the limitation at extreme difficulty is explicit and predictable. At the first logged 1.7B update, all-wrong groups account for at least 76.6%(URKL) and 77.7%(UFKL) of 256 groups. Weighted OPD coverage is bounded by 0.49–3.22% and 0.68–3.03% of 2,048 replies, respectively. Fig. 6 shows that pure OPCD's stronger reliance on distillation is not automatically beneficial. We added more detailed materials on this issue in App. F.3.